DeepSeek-V3 is a 671-billion-parameter Mixture-of-Experts language model that activates only 37 billion of those parameters per token. Trained on 14.8 trillion tokens with an FP8 mixed-precision recipe, it reached results comparable to the strongest closed models of its generation while costing a fraction of the usual GPU budget. The architecture combines two ideas DeepSeek had refined over earlier releases: Multi-head Latent Attention (MLA), which compresses keys and values into a small latent vector, and DeepSeekMoE, which pairs many fine-grained routed experts with a couple of always-on shared experts.
Most labs ship a paper, a checkpoint, and a handshake. DeepSeek did something rarer: the repository contains a compact, readable reference implementation that actually loads those 671 billion parameters and generates text with them. The inference/ folder is a handful of Python files — a model definition, a set of Triton FP8 kernels, a generation loop, and two weight-conversion scripts. It is not a serving framework; it is an executable specification of how DeepSeek-V3 fits together, closer in spirit to nanoGPT than to a production stack like vLLM.
That makes the code unusually valuable to read. Every architectural claim in the paper — the latent attention, the sigmoid routing with a per-expert bias, the 128-element FP8 tiles — is visible as a few lines of PyTorch you can trace with a debugger. This tour walks through that source with two diagrams generated from the real file tree. Every node in the diagrams links back to its file on GitHub, so you can click through while reading.
What the Repo Gives You
The checkout is tiny: inference/model.py holds the entire architecture, inference/kernel.py holds three Triton kernels for FP8 arithmetic, inference/generate.py is the entry point, and inference/convert.py plus inference/fp8_cast_bf16.py reshape Hugging Face checkpoints into the sharded format the demo expects. Four JSON configs (16B, 236B, 671B, and a V3.1 variant) feed hyperparameters into a single ModelArgs dataclass. That is the whole surface area.
One design choice stands out while reading: the code is commented like a teaching artifact. Every class carries a docstring explaining its arguments, the MoE forward pass documents its bincount-based dispatch, and the FP8 kernels walk through their pointer arithmetic line by line. For a model at the research frontier, the level of polish is more reminiscent of an educational repository than a research dump — a deliberate contrast with the weight files themselves, which are as opaque as safetensors get.
The full DeepSeek-V3 demo at a glance: the generation loop drives a stack of blocks, each combining latent attention with either a dense or an expert feed-forward, all flowing through quantization-aware linear layers and Triton FP8 kernels.
Reading the overview from left to right:
- The journey starts at
generate()(generate.py, L31), an interactive chat loop that forwards only the new tokens at each step and keeps everything else in KV caches. Transformer(model.py, L738) is the top-level module: embedding, the layer stack, a final norm, and a parallel vocabulary head.- Each
Block(model.py, L696) is pre-normed attention followed by pre-normed feed-forward — nothing exotic at the macro level. - The attention itself is
MLA(model.py, L396), Multi-head Latent Attention, the component that makes KV caching for this model radically cheaper. - On the feed-forward side, layers one and up use
MoE(model.py, L636), while layer zero uses a plain denseMLP(model.py, L500). Gate(model.py, L535) scores all 256 routed experts per token and picks the top eight, with a learned bias that keeps expert usage balanced without an auxiliary loss.- Every projection funnels through the quantization-aware
Linear(model.py, L166) layer and its dispatch functionlinear()(model.py, L131). - In FP8 mode that dispatch calls
act_quant(kernel.py, L38) to quantize activations, thenfp8_gemm(kernel.py, L175) for the matmul. - Before any of this runs,
convert.py(convert.py, L33) must remap Hugging Face weight names and shard the tensors across your GPUs.
From Install to Your First Model
The demo is deliberately spartan: Linux, Python 3.10, and four pinned dependencies — torch 2.4.1, triton 3.0.0, transformers 4.46.3, and safetensors 0.4.5, all listed in inference/requirements.txt. Clone the repository, create an environment, install those four packages, and you have the code side done in minutes. The heavy part is the weights: roughly 685 GB for the FP8 checkpoint on Hugging Face, which includes the main model plus the Multi-Token Prediction modules.
Because the team trained natively in FP8, the public release contains FP8 weights only. If you want to poke at BF16 tensors instead — say, to experiment without quantization in the loop — the repo includes the conversion script for exactly that, and it is a two-flag command:
python fp8_cast_bf16.py --input-fp8-hf-path /path/to/fp8_weights --output-bf16-hf-path /path/to/bf16_weights
Next, the demo needs weights in its own sharded layout rather than the Hugging Face layout. The convert.py script performs the renames and the sharding in one pass — the mapping table at the top of the file is worth reading on its own, since it is effectively a Rosetta stone between the Hugging Face parameter names (q_a_proj, kv_b_proj, gate_proj) and the demo’s internal names (wq_a, wkv_b, w1):
python convert.py --hf-ckpt-path /path/to/DeepSeek-V3 --save-path /path/to/DeepSeek-V3-Demo --n-experts 256 --model-parallel 16
Finally, launch generation with torchrun. The README’s example spans two nodes with eight GPUs each, but a single node works too as long as the model-parallel factor you used during conversion matches your GPU count. Pass the converted checkpoint and the 671B config, and you land in an interactive prompt where every line you type is completed by the full model:
torchrun --nnodes 2 --nproc-per-node 8 --node-rank $RANK --master-addr $ADDR generate.py --ckpt-path /path/to/DeepSeek-V3-Demo --config configs/config_671B.json --interactive --temperature 0.7 --max-new-tokens 200
Tensor parallelism is not bolted on afterward — it is woven into the layer definitions themselves. ParallelEmbedding (model.py, L89) shards the vocabulary across ranks, ColumnParallelLinear (model.py, L208) splits output features, and RowParallelLinear (model.py, L237) splits input features and finishes with an all-reduce. Even the expert list is partitioned: each rank owns 256 divided by world-size experts and silently skips tokens routed elsewhere.
The Three Ideas That Make V3 Fast
The detail view expands each region into individual classes: the MLA cache variants, the Gate's grouped top-k selection, the parallel linear hierarchy, and the two Triton kernels behind every FP8 matmul.
Latent attention. Standard multi-head attention caches one key and one value per head per position, which for a 671B model with 128 heads is a staggering amount of memory. MLA instead projects each token’s KV state into a single low-rank latent vector (rank 512 in the 671B config) and caches just that plus a small rope-carrying piece. The MLA (model.py, L396) forward pass has two modes: a naive mode that materializes full keys and values, and the default absorb mode, where the up-projection weights are folded into the attention scores themselves so the cache never needs expanding. Rotary embeddings enter through precompute_freqs_cis (model.py, L297), which bakes in YaRN-style frequency interpolation for long contexts, and apply_rotary_emb (model.py, L378), which applies them as complex multiplications.
Balanced routing without a balancing loss. The Gate (model.py, L535) module is only a few dozen lines but encodes one of the paper’s headline contributions. It scores experts with a sigmoid, adds a learnable per-expert bias purely for load balancing, and — because the bias never enters the gate value — the routing objective stays untouched by the balancing pressure. Group-limited selection then picks the top groups before choosing the final eight experts. Downstream, MoE (model.py, L636) dispatches token subsets to each local Expert (model.py, L601), adds the shared experts’ output to every token unconditionally, and all-reduces across ranks.
FP8 with per-block scales. Every weight is stored as float8_e4m3fn accompanied by a scale grid covering 128-element blocks. At runtime the linear() (model.py, L131) dispatcher either dequantizes weights with weight_dequant (kernel.py, L89) and runs an ordinary BF16 matmul, or stays fully in FP8: act_quant (kernel.py, L38) quantizes the activations per 128-element tile (clamping each tile’s max and dividing by 448, the e4m3 ceiling), and fp8_gemm (kernel.py, L175) accumulates products of FP8 operands rescaled by their block scales. The GEMM kernel is autotuned over dozens of tile configurations, all in about 200 lines of Triton — a compact companion to the tensor-core tricks used by engines like TensorRT-LLM.
The same files scale across sizes, too. Alongside the 671B configuration, the configs folder ships 16B and 236B variants plus one for the V3.1 revision, and because every dimension — expert count, activated experts, MLA ranks, number of dense layers — arrives as data through ModelArgs, you can instantiate a much smaller sibling of the architecture on a single GPU and watch tensors flow through the identical code paths. The default ModelArgs in the source is itself a miniature: 27 layers, 64 routed experts, two shared experts, 16 heads. Reading the architecture at that scale before renting an eight-GPU node is time well spent.
Honest Limits
This is demonstration code, and DeepSeek says so plainly. The MoE dispatch loops over experts in a Python for loop, which is perfect for clarity and hopeless for throughput — every serious deployment instead uses a fused grouped GEMM. Sampling offers temperature only, there is no continuous batching, no speculative decoding despite the MTP modules riding along in the checkpoint, and the maximum batch size defaults to eight. The interactive prompt keeps exactly one conversation. The demo also fixes its random seed, caps CPU threads at eight, and silences printing on non-zero ranks — small touches, but each is a decision you would have to revisit in real serving.
Environment requirements are strict, too: Linux with Python 3.10, pinned one-year-old package versions, and no official Windows or macOS path. The Hugging Face Transformers library cannot load these checkpoints directly, so any experiment that starts from AutoModelForCausalLM needs a different plan. If your goal is serving the model rather than reading it, the README points to SGLang, LMDeploy, vLLM, TensorRT-LLM, and LightLLM as the supported engines; the reference code is for study, not for traffic. The tokenizer handling is the one place the demo leans on an external library, pulling in AutoTokenizer from Transformers for chat templating rather than shipping its own.
None of that diminishes why this code matters. Between the paper’s equations and the checkpoint’s opaque tensors sits this small, honest implementation, and it answers questions the paper leaves open: exactly where the bias enters the gate, exactly what gets cached in absorb mode, exactly how a 128-element scale tile becomes a float. Few frontier models ship with documentation this executable.
Related Tours
If you want to build intuition for the components, our earlier source tours cover the dense-model counterparts: the original Llama 2 implementation for classic multi-head attention, nanoGPT for a minimal trainable stack, and the serving-side view in vLLM and TensorRT-LLM. Reading a sparse MoE right after a dense model makes the routing machinery click in a way neither alone can. And for the training-side story — how FP8 held up over 14.8 trillion tokens, and how the auxiliary-loss-free balance was validated — the technical report linked from the repository remains the best companion to the code. Enjoyed this post? Never miss out on future posts by following us 
