Ask a million developers what “running a model locally” looks like and most will describe one command: ollama run. The project by the same name is the software behind that moment, and its repository is a masterclass in hiding enormous complexity behind a calm exterior. What looks like a friendly installer is actually a Go server that manages model storage, GPU discovery, process supervision, wire-format compatibility with three API ecosystems, and a carefully patched C++ engine — all in one binary you can start with ollama serve.

The source rewards the patient reader. There is a 3,202-line router file where every endpoint lives in one visible table. There is a scheduler that measures free VRAM before promising a load, and quietly evicts your oldest model when a new one needs the room. There is a directory of per-family output parsers — Qwen, DeepSeek, Gemma, GLM, and more — because every model family leaks its reasoning and tool calls in a slightly different dialect. None of it is magic; all of it is engineering you can borrow. As always in this series, what follows is an educational tour of published source code.

From the command line through the Gin router, scheduler, and engine supervisor, down to the manifests and parsers that make models behave.

Reading the overview from left to right:

• The CLI main is the entry verb table — serve, run, pull, create — where every subcommand ends up calling the local HTTP API.
• The route table registers the native /api endpoints plus OpenAI and Anthropic translations in one readable block.
• The OpenAI compatibility layer converts /v1 payloads into internal types so existing tooling just works.
• The scheduler keeps pending load requests, watches free VRAM, and evicts idle runners under pressure.
• The engine supervisor builds launch options, spawns the llama.cpp subprocess, and streams its status back.
• The compat bridge is C++ glue that keeps a pinned llama.cpp build speaking Ollama’s interface.
• The manifest store maps human names to content-addressed config and layer blobs, OCI-style.
• The output parser registry picks a per-family parser that extracts thinking blocks and tool calls from raw generations.

Why You Need This

First, it is the clearest public example of a product-shaped inference server. Most serving codebases you can read are research-shaped: a monolith that assumes a data center. Ollama assumes a laptop. The scheduler in server/sched.go is built around questions like “can this model fit alongside what is already loaded,” “should mmap be disabled because host RAM is tight,” and “which runner should be unloaded first.” Those functions — maybeDisableMmapForHostPressure, findRunnerToUnload, waitForVRAMRecovery — are the unglamorous decisions every local tool gets wrong at least once. Reading them is a free course in memory management for consumer hardware.

The scheduling loop itself is equally instructive. Two goroutines run the show: one drains the pending queue, matching each load request against current free memory and the runners already resident; the other processes completed requests, releasing slots and updating free-space estimates after every change. Requests carry their own keep-alive duration, so a chat session can hold a model warm for minutes while an idle one expires in seconds, and when a runtime-level out-of-memory error surfaces, the scheduler expires the offending runner and retries rather than taking the whole server down. Queue plus lease plus recovery is the entire reliability story, expressed in under 1,700 lines.

Second, the compatibility strategy is worth stealing. Ollama speaks three wire protocols from one handler core: its native /api/chat and /api/generate, the OpenAI /v1 family including the newer responses and audio endpoints, and Anthropic’s /v1/messages via a dedicated middleware. The translations happen at the edge — openai/openai.go and middleware/anthropic.go convert foreign payloads into the internal api/types.go vocabulary, and then every downstream component deals with exactly one request shape. When your own service grows a second API dialect, this is the pattern: translate at the boundary, never in the core.

Third, the model-storage design answers a question every tool eventually faces: how do you ship gigabytes of weights safely? The answer here is content addressing all the way down. A model name resolves through manifest/manifest.go to digested blobs, downloads go through a resumable transfer layer with sparse-file support on Windows, and the GGUF reader in fs/gguf/gguf.go inspects key-value metadata so the scheduler can size a load before moving a byte. Verifiability is not an add-on; it is the address scheme. The transfer layer is worth its own sentence of praise: every blob download is digest-checked on arrival, resumes across interrupted connections, and on Windows lands in a sparse file so a partially fetched twenty-gigabyte model does not secretly reserve the full size on disk. That single detail — resumable, verified, space-honest downloads — is the difference between a tool people trust with their bandwidth and one they delete after the first failed pull on hotel Wi-Fi.

The detail view: CLI internals, wire middleware, scheduling, device discovery, output parsing, and the storage layer, with the seams between them.

The command surface deserves a closer look because it has quietly become a launchpad, not just a CLI. Beyond the familiar verbs, cmd/launch/launch.go — 1,481 lines — knows how to configure external coding agents such as Claude Code, Codex, OpenCode, and Droid to point at local models, with per-agent modules that rewrite their configuration files. There is a full-screen terminal picker in cmd/tui/selector.go, an interactive REPL in cmd/interactive.go that can save and restore conversation context, and a benchmark harness in cmd/bench/bench.go that measures throughput across your installed models. The lesson: a binary that people already trust is the perfect place to grow orchestration features.

The runtime side is a study in supervising something you do not fully control. llm/server.go implements a LlamaServer interface whose Load method starts the engine subprocess with a filtered environment — the code visibly redacts any variable whose name hints at a credential before passing it down, a small touch that says a lot about the threat model. Device discovery runs through discover/runner.go, which bootstraps CUDA, ROCm, Metal, and CPU paths, sometimes compiling a small native probe binary (discover/native_probe*.go) so the machine can report its own GPUs without crashing the parent process. A pinned llama.cpp version at the repository root is bridged by the llama-ollama-compat.cpp sources, keeping upstream engine updates an explicit, reviewable event rather than a silent drift. Configuration flows through envconfig/config.go, a single 410-line registry where every OLLAMA_ environment variable is typed, defaulted, and documented — one file to grep when behavior surprises you. And model creation is a path, not a cliff: cmd/create_safetensors.go converts a directory of HuggingFace-style weights into GGUF layers on the fly, while the Modelfile grammar composes them into a new manifest. Import, customize, publish — the whole lifecycle is inspectable Go.

Output handling is the newest battleground, and the source treats it as a first-class subsystem. The model/parsers directory holds a registry of per-family parsers — Qwen3, Qwen3-VL, DeepSeek, Gemma, GLM, Cohere, Olmo, and more — each responsible for peeling reasoning blocks and tool-call syntax out of raw tokens. A generic fallback lives in thinking/parser.go, while harmony/harmonyparser.go, 503 lines, implements the channel-based harmony format used by gpt-oss models; the router checks model metadata and switches to it when appropriate. Prompts flow the opposite direction through template/template.go, which renders each model’s own chat template with the injected history. Parse early, parse once, and the rest of the server never sees dialect-specific strings. The same discipline extends to tool use: tools/tools.go normalizes whatever call syntax a family emits into one internal tool-call type, so a handler can dispatch a function call without knowing whether it arrived wrapped in Qwen tags or harmony channels. If you maintain any product that serves multiple model families, this registry-of-parsers shape — one directory, one interface, one entry per family — is the pattern that keeps the chaos contained.

From Install to First Token

The README’s own path is two commands:

curl -fsSL https://ollama.com/install.sh | sh
ollama run gemma4

Behind that second command, the CLI calls POST /api/chat on port 11434, the scheduler finds room for the weights, the engine subprocess loads them, and tokens begin streaming. The same conversation is available from any OpenAI client pointed at /v1/chat/completions, or from a Claude-flavored client via /v1/messages, and the REST shape is simple enough to drive with plain curl. Because the API contract in api/types.go is one file, it is realistic to read the whole request surface in an afternoon — then go further by writing a Modelfile, parsed by the 677-line grammar in parser/parser.go, to customize a model’s system prompt, parameters, and adapter layers into a new manifest you can push and pull like a container image. For developers who prefer SDKs, the official Python and JavaScript libraries wrap the same endpoints, so the quickstart above and a ten-line script exercise identical server code.

Honest limits: this is a serving and management system, not a training or fine-tuning framework — model creation means wrapping existing weights, not computing gradients. The cloud features mix remote calls into the local story, which is clearly flagged but changes the mental model from “everything on this machine” to “this machine first.” Hardware discovery remains inherently fragile across drivers and vendors; the probe-and-watchdog design contains that fragility but cannot eliminate it. And the compatibility layers, while disciplined, necessarily trail new features in the OpenAI and Anthropic APIs by some weeks. None of that dims the core lesson: take one calm binary, put a strict contract in front of it, schedule every scarce resource deliberately, and “runs on my laptop” stops being a miracle and becomes a design.

Enjoyed this post? Never miss out on future posts by following us