LMDeploy from the InternLM team at Shanghai AI Laboratory occupies an interesting spot in the inference landscape. It ships two independent inference engines — a pure-PyTorch engine you can read and hack like any research codebase, and TurboMind, a C++ and CUDA engine whose benchmark numbers were deemed strong enough to earn a EuroSys 2027 paper — and hides the choice behind one async engine interface and one OpenAI-compatible server.

That dual-engine design is what makes the repository worth a slow read. The Python side under lmdeploy/pytorch shows exactly how paging, continuous batching, and fused mixture-of-experts kernels are expressed in modern PyTorch and Triton, while the native side under src/turbomind shows the same problems solved with a hand-written scheduler and architecture-specific kernels. Above both sits a serving layer that speaks OpenAI and Anthropic protocols and a proxy that fans requests across multiple model instances. Licensed under Apache-2.0, the project supports far more than InternLM checkpoints: DeepSeek, Qwen, Kimi, Llama, and a growing list of vision-language models all run through the same paths. As always in this series, what follows is an educational tour of published source code.

The overview shows the fork in the road: one async engine that delegates to either the readable PyTorch engine or the compiled TurboMind core, depending on the backend you pick at launch.

Reading the overview from left to right:

• The unified CLI parses every subcommand — chat, serve, convert — and maps arguments onto engine configuration.
• The lite toolkit calibrates a model on sample data and produces AWQ or GPTQ quantized checkpoints before deployment.
• The OpenAI API server is a FastAPI app that accepts chat-completions requests and streams tokens back over SSE.
• Request shapes are defined by protocol models, which also carry usage accounting for token billing dashboards.
• The async engine is the heart of the serving layer: it queues requests, forms batches, and forwards them to whichever backend was selected.
• Inside the PyTorch engine, the model agent — the largest Python file in the repo at over 1,700 lines — manages device buffers, cache updates, and forward passes for every step.
• The CUDA kernel library implements paged attention, KV cache filling, and fused MoE in Triton and PyTorch.
• On the native side, the TurboMind scheduler plans prefill and decode waves, honoring priority policies and KV memory limits in about 1,400 lines of C++.

Why You Need This

Throughput on commodity GPUs is the first reason. TurboMind’s kernels are tuned per GPU architecture — the linear attention kernels for Hopper-class cards are one example of that attention to silicon — and the project has published strong MoE and mixed-precision numbers on hardware as ordinary as V100-class cards. If you serve open models to real users from rented GPUs, the difference between engines is measured directly in your monthly bill, and having both engines behind one switch means you can benchmark on your own traffic instead of trusting someone else’s chart.

The second reason is the serving surface. The async engine exposes OpenAI chat-completions plus an Anthropic adapter, so clients written for either hosted provider keep working when pointed at your own box. The model proxy goes a step further: it can front several model instances at once, which turns a single deployment into a small routing layer for A/B testing two models or spilling overflow traffic from a small model to a big one.

The third reason is educational symmetry. Because both engines exist side by side, you can read the paged attention kernel in Python one hour and the C++ scheduler the next, comparing how each handles the same batch-scheduling decisions. The prefill scheduler in the PyTorch engine and its C++ counterpart make the same trade-offs — admission control, chunked prefill, block accounting — in two languages, which is a rare and honest way to learn serving systems.

The detail graph separates the six layers: CLI, serving and API, the PyTorch engine, the TurboMind native stack, the quantization toolkit, and the vision-language wrapper.

Trace a request from the top. The CLI dispatches serve into the API server, which validates the payload against protocol models and hands it to the async engine. A session manager keeps interactive chat history resumable across reconnects. The engine’s backend switch — the two edges labeled engine=pytorch and engine=turbomind in the diagram — is the whole architectural story: identical request state flows into either implementation.

On the PyTorch side, the engine loop advances batches and hands per-session streaming to engine instances. The model agent is where the interesting memory work happens: it builds positional and block tensors through the inputs maker, writes new keys and values via fill_kv_cache, and dispatches attention to the right backend — standard paged attention for most models, MLA for DeepSeek-style latent attention. Model definitions such as deepseek_v2.py wire expert routing to the fused MoE kernel.

On the TurboMind side, a checkpoint converter rewrites Hugging Face weights into the native tensor layout before the pybind bridge instantiates the C++ engine. The scheduler owns admission and wave planning, consulting a prefix trie to reuse shared prompt caches, and the largest CUDA file in the tree — moe_utils_v2.cu — implements the gather, scatter, and GEMM helpers that expert routing needs.

From Install to a Running Server

A working GPU box with recent CUDA drivers is enough to start. Install with pip install lmdeploy, then launch a terminal chat with lmdeploy chat internlm/internlm2_5-7b-chat — the CLI pulls the checkpoint from the Hugging Face hub, converts weights on first run, and starts streaming. For an HTTP service, run lmdeploy serve api_server internlm/internlm2_5-7b-chat and point any OpenAI-compatible client at port 23333. Useful dials at launch include --tp for tensor parallelism across GPUs and --cache-max-entry-count to cap how much VRAM the KV cache may claim. For quantized deployments, run the lite flow first: lmdeploy lite calibrate followed by lmdeploy lite auto_awq produces a compressed checkpoint that both engines load directly.

Honest limits: TurboMind’s CUDA path is NVIDIA-centric, so AMD and other accelerators effectively route you to the PyTorch engine, which trades some peak throughput for portability. The quantization toolkit covers weight-only flows well, but the calibration stage wants a representative dataset to produce good scales, and the resulting checkpoints are engine-specific artifacts. Finally, the codebase moves fast — model support and kernel paths shift between releases, so pin a version in production and expect to re-verify flags when you upgrade. None of that dims the core appeal: two complete engines, one clean interface, and a codebase that teaches serving systems in two languages at once.

Enjoyed this post? Never miss out on future posts by following us