Our first visit to InternLM/lmdeploy stayed above the waterline: the CLI, the lite quantization toolkit, the OpenAI-compatible server, and the fork where a request picks either the readable PyTorch engine or the compiled TurboMind core. This visit goes under the waterline, into the C++ half that EuroSys-grade numbers were built on. The questions here are the classic serving-engine questions — where do requests wait, who decides what runs next, how are KV-cache pages handed out and reused — answered with a design that is unusually clean: one engine thread that owns all mutable state, a scheduler that plans several steps ahead of the GPU, and a two-level block system where logical sequences borrow from pooled physical pages. As always, this is an educational walk through published source, Apache-2.0 licensed, on the main branch.

The design decision everything else hangs on is single-threaded ownership. Instead of locks sprinkled through the request path, TurboMind gives the engine one internal thread that pops from a Gateway, and everything else — Python, signal delivery, result forwarding — talks to that thread through queues. Reading the loop below is therefore the most direct way to understand the whole system.

The TurboMind core: the Python TurboMind class reaches through the _turbomind pybind module into the Engine; requests arrive via the Gateway, the Engine's internal thread validates, accepts, cancels and retires them, the Scheduler plans against block pools and a prefix trie, and the ModelExecutor consumes prepared BatchData on its own thread.

Reading the overview from left to right:

• The Python side is TurboMind: it loads weights, calls _create_engine, and hands out per-stream instances from create_instance.
• The boundary is a pybind module: PYBIND11_MODULE exposes engine handles, request structs, and even multimodal inputs, with config classes auto-generated via a for_each trick at line 379.
• Gateway is the doorsill: push and pop for requests, cancel for aborts, and a dedicated signal thread so notifications never block the caller.
• The engine loop at Engine::Impl::InternalThreadEntry is the whole scheduling brain in one while-true.
• The Scheduler plans each step: admission, prefix matching, resume versus continue, publication.
• Blocks live in two pools — CacheBlock physical pages managed by CacheBlockPool, and LogicalBlock sequences that borrow them.

The bridge and the boundaries

Between Python and C++ sits more than a translator. The bind.cpp module defines three worlds: engine handles, the request vocabulary — requests, sequences, signals, multimodal inputs — and configuration. The configuration part is the most elegant: C++ config structs register their own fields, so a helper at lines 379 through 390 can expose every option to Python without a hand-written def per attribute — add a field to the struct and the binding follows. Multimodal inputs get first-class treatment too, with per-model item types such as QwenVitItem carrying grid dimensions and token spans, and tensor data crossing by dlpack capsule rather than by copy. On the Python side, TurboMind.init resolves the model source, _process_weights and _create_weight move tensors into the layout the kernels expect, and runtime weight updates or sleep/wakeup travel the same bridge through update_params and wakeup.

The Gateway deserves a closer look because it is the only object on the hot path that Python threads touch concurrently. push enqueues a request and wakes the signal thread; pop drains it under a free-slot budget, optionally blocking; notify is how the engine thread talks back — signals for finished, failed, or cancelled requests are handed to a separate signal thread so that a slow consumer of results never stalls the engine loop itself. Cancellation is also asynchronous: cancel marks a request, and the engine observes it in its own time, which is why the loop below has an explicit Cancel step rather than an interrupt.

The engine loop, one thread to rule the batch

InternalThreadEntry starts with pinning chores — CUDA device, a stream, allocators — then enters the loop that is the engine. Only rank zero talks to the Gateway: the pop at line 797 requests as many free slots as the batch can offer and blocks only when the batch is completely empty, a detail that keeps idle GPUs from spinning hot. The popped requests pass Validate, which checks everything the scheduler would otherwise discover mid-step — token limits, session resumption, multimodal payload shapes — and rejects early with per-request signals. Tensor-parallel ranks then synchronize at line 804 only when the batch is empty, and the full request set is broadcast so every rank runs the identical step.

The body of the iteration is a fixed order that reads like a checklist: Accept turns validated requests into sequences, Cancel applies aborts, gateway notify flushes the resulting signals, Retire frees finished sequences, and only then — with the active count re-reduced across data-parallel groups — does Schedule run. Two quiet guardians follow: FailStalledHeadOfLine catches the case where a queue head cannot fit and everything behind it starves, and cache-stats logging is throttled so observability never becomes load. The ordering matters: because retire precedes schedule, freed pages are reusable in the very same step.

The Scheduler: admission, prefixes, and partial siblings

Where the engine loop is a metronome, the Scheduler is the strategist. Each pass builds a ScheduleState and walks three phases. Required admission at line 1105 handles sequences already holding resources; optional admission at line 1219 brings new prompts in only if that cannot hurt anyone admitted. Prompt admission itself is where the KV cache gets interesting: AdmitPrompt ensures the sequence has blocks, then MatchPrompt walks the prompt against the prefix trie to find already-materialized blocks, and IndexMissingBlocks decides what must still be computed.

The subtle machinery is for shared and resumed sequences. PlanResume rehydrates a sequence from a stored session, SetupPartialSiblings splits a shared prefix where two requests diverge mid-block, and a trio — SetProducers, CheckProducers, MarkProduced — tracks which in-flight sequence will fill a block that others want, so cache reuse does not have to wait for a sequence to finish. Publication of checkpoint blocks is planned separately at PlanPublication, with a clamp that keeps a forward pass from outrunning its context. The step ends with ReplayMemory applying the plan to the block system and CommitResults handing tokens back.

Underneath the scheduler sits a small object system with real engineering in it. A LogicalBlockPtr and a CacheBlockPtr are reference-counted handles that know their pools, so release paths are exception-safe by construction. The physical pool answers the classic eviction question cheaply: SortedBlocks produces candidates in reuse-friendly order, and a Stamp computed over a set of blocks gives the trie a fingerprint to index. Lifecycle is explicit rather than implicit: Invalidate drops a block whose owner rewrote it, Deallocate returns memory to the allocator, and the logical pool’s Recycle fires the trie-erase callback that keeps the prefix index honest — the comment there even notes the pool deliberately stays prefix-agnostic.

The detail view: Python frontend and pybind bridge on the left; the engine tier of Gateway, engine loop, request structs, and BatchData; the scheduler tier with block pools, the prefix trie, and the block implementation file; the memory layer of pages and slabs underneath; and the ModelExecutor thread on the right.

Walking the detail diagram, a token’s journey touches every tier. The request struct itself is compact — Request carries inputs and a generation config, Sequence is the long-lived state, and CacheCopy records when a shared physical block must be duplicated before divergence. The block pools are pooled twice over: LogicalBlockPool recycles logical objects through a callback that erases the sequence from the prefix trie on recycle, while physical pages come from PageAllocator and the pool storage itself is slab-allocated so steady-state serving never wakes the general-purpose heap. On the other end, ModelExecutor runs its own internal thread with a Run that consumes the BatchData the engine filled — container structs like ResolvedCopy carry the block copies that must happen before attention reads.

Splitting the engine thread from the executor thread is not ceremony; it is latency hiding. The scheduler can plan step N+1 — including block allocation and copies — while the GPU is still finishing step N inside the model, and the two only meet at the BatchData hand-off. The same separation protects the reverse path: interrupts such as Interrupt mark a sequence for the engine to observe at a step boundary, and Finalize runs cleanup once the executor has drained, so a cancelled request never frees memory that attention kernels are still reading. It is the same discipline that made the single-threaded engine safe in the first place, applied one layer down.

From Install to Your First Model

Install is pip install lmdeploy, then lmdeploy serve api_server <model> picks TurboMind by default. What this tour adds is the operational reading: create_instance per stream is cheap because the engine, not the instance, owns state; the schedule metrics surfaced by get_schedule_metrics are the engine’s own counters about admission and cache pressure, worth watching before raising max_batch_size; and prefix reuse is automatic — the trie and producer tracking above are why repeated system prompts do not multiply prefill cost.

Honest limits: this walk covered the orchestration core, not the kernels — attention, MoE, and the linear layers under models/llama are their own mountain, and the PyTorch engine shares none of this C++ machinery. We read, not benchmarked; the design claims here are structural. And the engine thread model is a real contract: user code that blocks inside callbacks stalls the whole batch, which is precisely why the Gateway keeps its signal thread. For placement in the ecosystem, vLLM and SGLang solve the same block-management problem in Python, TensorRT-LLM splits the role across MPI processes, and OpenVINO shows what one-graph-per-stream looks like without any server at all. TurboMind’s answer — one thread, one plan, pooled pages — is the most conservative and arguably the easiest to reason about under load.

Enjoyed this post? Never miss out on future posts by following us