Run two models back to back in Ollama and something peculiar happens: the second model loads in about the time a browser tab takes to appear, and when you return to the first, it is warm again. That feeling of infinite memory is the work of one Go file - server/sched.go, 1,851 lines - which decides what stays resident, what gets evicted, which GPU group receives each model, and how long a loaded runner lingers before being shown the door. This is our third visit to the ollama repository; the first two tours covered the server surface and the manifest machinery, so this time we go where the real memory management lives: the scheduler that treats your VRAM like a landlord treats apartments, with pre-flight fit checks, eviction notices, and a very precise definition of how long a tenant may stay after its last request.

As always in this series, what follows is an educational tour of published source code.

Overview: HTTP handlers schedule requests through getRunner; the pending loop reuses, evicts, or loads; each load predicts VRAM, selects a GPU placement, and pre-flights the fit; completions release reference counts into keep-alive timers; device discovery refreshes the candidate list.

Reading the overview from left to right:

• server/routes.go, 3,653 lines, is the API surface: GenerateHandler at line 308 and EmbedHandler at 1082 both end at the same scheduler call, while PullHandler at 1480 and ShowHandler at 1745 handle the manifest side of the house.
• server/sched.go defines LlmRequest at line 31 and the Scheduler at 62, initialized by InitScheduler at 131 with a loaded map keyed by schedulerModelKey at 152 so identical requests share one runner.
• getRunner at line 218 is the entry every request passes: it floors the context at 4 tokens (219-221), raises it to 2,048 for vision models at 223-226, resolves the context-shift flag at 228-231, and either reuses a live runner at 250 or enqueues into pendingReqCh at 254 - returning ErrMaxQueue when the channel is full.
• The pending loop, processPending at line 274, is the landlord: it blocks the queue while it works, consults envconfig.MaxRunners at 275, reuses via needsReload checks at 305-313, and otherwise refreshes the GPU list at 320-326 before attempting a load.
• Scheduler.load at line 545 does the heavy lifting: it loads GGUF metadata at 576, asks the llm package to PredictServerVRAM at 585, and calls selectLlamaServerPlacement at 586 to choose which devices will host the model.
• The placement engine selectLlamaServerPlacement at line 1014 walks candidate groups - bestSingleGPUFit at 1102, multi-GPU spreads, or main-GPU preferences at 1081 - scoring each by available memory after the load.
• discover/runner.go keeps the candidate list honest: GPUDevices at line 36 returns refreshed devices with delta detection via sameRefreshDevice at 372, filters integrated GPUs at 382-435, and bootstraps driver libraries through a Metal retry path at 488 or a status watchdog at 564.
• server/model.go is what gets loaded: parseFromModel at line 27 turns manifest layers into modelLayer entries, and detectChatTemplate at 99 picks the template that the runner will apply.

Why You Need This

Every local-inference tool eventually invents a memory manager, and comparing them is the fastest way to understand the design space. llama.cpp ships as a library and leaves residency policy to whoever embeds it; vLLM assumes a server with one big model and continuous batching, so its scheduler packs sequences, not models; and llamafile sidesteps the question by being a single self-contained binary for one model. Ollama’s situation is unique: a desktop daemon that must juggle chat models, embedding models, and vision models simultaneously on whatever hardware the user happens to own, with zero configuration. The answer in sched.go is a two-queue design - processPending at 274 for arrivals and processCompleted at 416 for departures - plus a reference-counted runner table. It is the same shape an operating system uses for process scheduling, applied to GPU residency, and reading it is a masterclass in resource management under uncertainty.

The eviction dance is the part worth studying twice. When a request arrives for a model that is not resident, the scheduler first checks whether the load can succeed: load() predicts VRAM at 585, picks a placement at 586, and, for the second-and-later model case, runs a pre-flight check at 598-605 that evicts before spawning if the prediction exceeds eighty percent of free memory at 601. If the load was tried and crashed with an OOM anyway, the OOM retry path at 365-375 evicts every resident model, waits for each unload to finish, and retries exactly once - a second crash falls through to fail-fast. Every eviction goes through the same ritual at 386-397: stop the expire timer, zero the session duration, and push the runner onto expiredCh once its reference count drains. The wisdom here is that OOM handling is separated into predicted-OOM (cheap, act early) and actual-OOM (expensive, recover once, then give up), a distinction most hand-rolled model servers get wrong by retrying forever against a wall that never moves.

The scheduler also quietly encodes years of community bug reports. The architecture list at 556-559 pins numParallel to 1 for model families that corrupt output under parallel decoding, with the issue number quoted in the comment; embedding models are forced to parallel 1 at 549-552 because there is nothing to batch; and vision models get their 2,048-token context floor at 223-226 because multimodal prompts genuinely do not fit below it. On the discovery side, integrated GPUs are filtered out unless explicitly admitted at 382-435 because iGPU memory reporting lied to earlier releases, and mmap is disabled under host memory pressure at 1242-1296 with a documented headroom constant at 1297. None of this is glamorous, and all of it is the difference between a tool that works on your machine and one that works on everyone’s machine - which is, ultimately, the only product Ollama sells.

Detail: routes feed a keyed runner table; the pending loop manages auto concurrency limits, free-space refresh, and placement; loads predict, clamp, and pre-flight; discovery filters and bootstraps devices; completions arm keep-alive timers.

The keep-alive story is subtler than the OLLAMA_KEEP_ALIVE env var suggests. When a request finishes, processCompleted at 416 decrements the runner’s refCount at 433 and, at zero, branches on session duration: zero means expire immediately at 434-441, while a positive duration arms a time.AfterFunc timer at 442-454 - and if a timer already exists, it is Reset rather than recreated at 455-458, with expiresAt recorded for status reporting. useLoadedRunner at 524 does the mirror image on arrival: increment refCount at 527, stop any pending expire timer at 528-531, and refresh sessionDuration from the request at 532-534 so a per-request keep_alive overrides the daemon default. The net effect is that a model stays warm exactly as long as the most recent request asked for, and the first byte of the next request never waits for weights to cross the PCIe bus. It is a tiny state machine - idle, referenced, expiring - but getting it wrong produces either VRAM exhaustion or the famous “model reloads every request” support thread.

Placement is the other half of the puzzle, and it reads like a tiny bin-packing solver with opinions. selectLlamaServerPlacement at 1014 first honors explicit placement hints, then considers a single-GPU fit at 1102 (preferred, because tensor parallelism across consumer GPUs is rarely worth the latency), then multi-GPU groups scored by betterPlacementGPU at 1120 and bestGPUGroupByAvailableMemory at 1128. The prediction feeding it comes from PredictServerVRAM at 585 using the context size computed by effectiveLlamaServerContext at 584 with numParallel folded in - which is why the parallel clamps at 546-559 matter for placement too. Two memory-shaping knobs then adjust the launch: automaticGenerationBatch at 875 sizes the decode batch to what the leftover VRAM can hold (generationBatchFits at 923, stepping down at 950-968 when it does not), and reduceAutoNumCtxForLoadOOM at 817 shrinks an auto-sized context on failure. mmap gets the same treatment at 1180-1296: defaults per OS, then a host-pressure override at 1242 with headroom at 1297, because on 32-GB machines the page cache is part of the memory story.

Discovery is the foundation the whole edifice stands on, and it is refreshingly defensive. GetSystemInfo at discover/gpu.go line 21 provides the baseline memory picture; GPUDevices at runner.go line 36 layers runtime detection on top, with sameRefreshDevice at 372 preventing spurious re-reporting between requests - important because the scheduler calls getGpuFn on every pending request at 325. Bootstrap is wrapped in failure recovery: bootstrapDevicesWithMetalRetry at 488 handles macOS driver hiccups, bootstrapDevicesWithStatusWatchdog at 564 guards the Windows/CUDA library path with a timeout, and filterOverlapByLibrary at 438 deduplicates devices that appear through multiple driver interfaces - the classic CUDA-plus-Vulkan double-count that once made users see twice the VRAM they owned. updateFreeSpace at sched.go 344 subtracts resident models from the picture, so the numbers the placement solver sees reflect what is actually free, not what the card shipped with.

The model layer closes the loop. parseFromModel at server/model.go line 27 resolves a name into manifest blobs and produces modelLayer entries - weights, projector, template - that detectChatTemplate at 99 post-processes; the gguf metadata that load() reads at 576 comes from the same files, giving the scheduler its model family, context training length, and tensor count before a single byte of VRAM is claimed. PullHandler at routes.go 1480 and CreateBlobHandler at 2209 maintain those blobs; ListHandler at 2131 reports what is installed. It is a clean separation: routes know names, model.go knows layers, sched.go knows memory, and discover knows hardware - four vocabularies meeting at one scheduler, which is exactly why a new backend can be added without touching the eviction logic.

From Install to a Warm Model

The README gets you running; the scheduler’s knobs are environment variables (commands abridged):

curl -fsSL https://ollama.com/install.sh | sh
ollama serve
# scheduler controls:
export OLLAMA_MAX_LOADED_MODELS=3      # resident model cap (auto = per-GPU default)
export OLLAMA_KEEP_ALIVE=10m           # default session duration per model
export OLLAMA_NUM_PARALLEL=2           # decode parallelism per runner
ollama run llama3.2 "hello"
ollama run nomic-embed-text "embed me"  # loads alongside, parallel=1 forced

Each loaded runner keeps its keep-alive timer reset on every request, so a model in steady use never unloads; stop talking and it expires exactly on schedule.

Honest limits: the scheduler is host-side only - it launches and monitors llama-server processes but does not batch across models, so two models share a GPU by time-slicing residency, not by compute interleaving; the VRAM predictor is an estimate, and exotic quantization mixes can still surprise it, which is why the fail-fast after one OOM retry exists; placement assumes homogeneous devices within a group and can misjudge split CPU+GPU layouts on unusual driver stacks; keep-alive timers are daemon-local, so a crash discards the residency policy along with the weights; and the whole design presumes one daemon per machine - multi-user GPU sharing is left to the tools built on top of it.

Enjoyed this post? Never miss out on future posts by following us