Two years of quantization work have convinced most teams that 4-bit is the practical floor for running large language models on everyday hardware. Microsoft’s BitNet refuses that floor entirely. Its weights live in {-1, 0, +1}, roughly 1.58 bits each, and the repository we are touring today is the official runtime for them. We looked at what BitNet is back in April in the conceptual overview; this time we walk the source itself, from a HuggingFace checkpoint all the way to lookup-table matrix multiplications executing inside a patched llama.cpp.

The interesting part of this codebase is not the transformers — those are ordinary — but the machinery around them. A build-time code generator emits AVX2 GEMM kernels as C++ text, a conversion tool reorders ternary weights into exactly the layout those kernels want, and a thin C++ shim overrides ggml’s mul_mat so llama.cpp runs BitNet kernels without knowing it. There is also a parallel GPU path where a pure PyTorch model calls hand-written CUDA kernels through ctypes. If you have read our tours of DeepSpeed’s ragged inference engine, TVM’s Relax VM, or Triton, this sits at a very different point on the same spectrum: almost no abstraction, maximal specialization per model shape.

Everything below links to real files on main. Diagrams first, then sections.

The BitNet ternary stack: setup and entry scripts at the left, checkpoint conversion and kernel codegen in the middle, generated AVX2/NEON kernels feeding a patched llama.cpp, with a pure-PyTorch CUDA detour below.

Reading the overview from left to right:

• setup_env.py orchestrates everything once: it quantizes the checkpoint, invokes codegen, and compiles a patched llama.cpp (compile).
• run_inference.py and run_inference_server.py are thin wrappers that drive the resulting llama-cli and llama-server binaries.
• convert-hf-to-gguf-bitnet.py packs ternary weights into three on-disk layouts: I2_S, TL1 and TL2.
• codegen_tl2.py and codegen_tl1.py emit AVX2 and NEON lookup-table kernels; gemm-config.h holds the tiling knobs.
• The generated headers live in preset_kernels and are included by ggml-bitnet-lut.cpp, the mul_mat override.
• On GPU, gpu/model.py implements the transformer in PyTorch and calls bitnet_kernels.cu through ctypes.
• e2e_benchmark.py times the whole thing end to end.

One Checkpoint, Three Layouts

The conversion tool, convert-hf-to-gguf-bitnet.py, is the quiet hero of this repository. It subclasses the llama.cpp GGUF exporter — see the Model base class at line 48 — but its BitnetModel subclass at line 1031 carries a scale_map (line 1077) that matches each linear layer to its precomputed activation scale. Ternary models throw away weight magnitudes, so the activation scales become the only thing keeping the forward pass calibrated; losing one layer’s scale is a silent quality disaster, and the map is how the tool avoids it.

Three output layouts fall out of one pass. The first is I2_S: quantize_to_i2_s maps {-1, 0, 1} to {0, 1, 2} (line 700), packs four ternary values into every byte (lines 705-725), and appends a 32-byte float32 scale tail per superblock. That is the format llama-quantize understands, which is why prepare_model in setup_env.py can finish the job by invoking the stock quantizer with the I2_S ftype (lines 145-152). Clever reuse: BitNet gets a standard GGUF file instead of inventing a container format.

The other two layouts, TL1 and TL2, are not about compression at all — they are about access order. transform_to_tl1 and transform_to_tl2 reorder and re-encode weights so that the lookup-table kernels of the next section can stream them sequentially. The preprocessing helpers (TL1 at line 483, TL2 at line 601) split each weight matrix into the block shapes the generated code will iterate over. If you have ever wondered why you cannot just run the stock gguf through the LUT path, this is why: the weights on disk are pre-shuffled for a specific kernel.

Codegen: Writing AVX2 Kernels at Build Time

Instead of hand-writing SIMD kernels for every supported model, BitNet writes Python that writes C++. codegen_tl2.py is the x86 half. Its gen_tbl_impl emits functions like three_tbl_impl_0 (line 287) — precomputed 256-entry lookup tables where each entry is the dot product of one 8-bit activation chunk with all possible ternary weight patterns. Since a ternary byte only takes 3^9-ish meaningful patterns, a multiply-accumulate chain collapses into table lookups plus integer adds.

The GEMM emitters, three_qgemm_lut and two_qgemm_lut, bake the model’s exact block sizes into the emitted loops, and gen_top_api wraps it all with the ggml_preprocessor glue (line 534) that ggml will call. The per-layer constructors from gen_ctor_code wire kernel selection to layer dimensions. docs/codegen.md shows the knobs: --BM, --BK, --bm control the tiling that gets hard-coded into the emitted loops.

How specific is the output? Compare the checked-in preset TL2 header for the 3B model: it guards itself with GGML_BITNET_X86_TL2, defines BK2 32 (line 24), and opens with hand-managed aligned allocators and _mm256 merge helpers (line 26). Every preset directory in preset_kernels/ is a frozen codegen output for one released model. Meanwhile gemm-config.h supplies compile-time tile constants per ISA — AVX2 gets row block 4, column block 128, parallel 4 (lines 3-10) — with NEON-DOTPROD and portable fallbacks below.

This is the same trade TVM made in our Relax VM tour, just at zero abstraction cost: TVM compiles arbitrary schedules through a compiler stack; BitNet stamps out one template with tuned constants. Less general, dramatically simpler, and for a fixed family of ternary transformer shapes, that simplicity wins.

The ggml mul_mat Override

The generated kernels need a host, and the host is ggml-bitnet-lut.cpp — about two hundred lines of glue that hooks into llama.cpp’s compute graph. Initialization (ggml_bitnet_init) allocates per-tensor extra state, and do_permutate applies a runtime row permutation to the in-memory weights, aligning them with the access pattern the generated loops assume.

Dispatch is guarded twice before any kernel runs. can_mul_mat checks the tensor type and device, and mul_mat_get_wsize computes the exact scratch size the qgemm needs — llama.cpp allocates work buffers once per graph, so this sizing function is what lets the LUT path request a different workspace than the stock matmul would. Only then does mul_mat run: it pulls the packed qlut data (line 107), the lut_scales and lut_biases (lines 108-109) that were stashed during init, and hands everything to the shape-matched generated function. Scales and biases ride along because the lookup tables hold raw integer accumulations; dequantization happens after the adds, in float.

Note what is absent: no dispatcher framework, no backend registry, no plugin API. When we toured ONNX Runtime we counted the layers of indirection between a node and its kernel; here it is one function with an if-chain. The cost is equally clear — the file knows llama.cpp internals intimately, so upstream ggml changes can break it, which is exactly why the repo pins its 3rdparty llama.cpp checkout.

The GPU Detour: Pure PyTorch Plus ctypes

The gpu/ directory is a second, independent implementation. Here the transformer is plain PyTorch — Attention, FeedForward, TransformerBlock built on xformers kernels — except every linear is a BitLinearKernel. Its forward quantizes activations to int8 per token (lines 68-71), then calls bitnet_int8xint2_linear, which loads libbitnet.so via ctypes (line 19) and passes raw data_ptr()s plus the current CUDA stream. No torch extension bindings, no autograd bookkeeping — inference only, and it shows.

The CUDA side mirrors the CPU story at smaller scale. bitnet_kernels.cu exports one C symbol, bitlinear_int8xint2, whose body is a chain of shape checks: each known (M, N, K) launches ladder_int8xint2_kernel with tile parameters chosen offline for that layer’s dimensions (lines 4-33). Weights arrive pre-packed as int8 (four ternary values per byte), so the kernel does its own unpack and int8-times-int2 accumulation, writing bf16 output. setup.py builds it as a bare CUDAExtension.

Around it, FastGen does what torch.compile was built for in our Executorch tour: compile_prefill and compile_generate trace the model into CUDA graphs, then generate_all replays them token by token with a KV cache managed by make_cache. The honest caveat: this path only supports the exact layer shapes hard-coded in the kernel dispatch — feed it an unknown width and the kernel prints a warning and skips the GEMM.

From Install to Your First Model

The intended flow is one command: setup_env.py detects your platform (system_info at line 90), pulls the model (get_model_name at line 93), quantizes it, runs codegen via gen_code, and invokes cmake with your chosen compiler (lines 220-222). After that, run_inference.py defaults to the bundled bitnet_b1_58-3B I2_S gguf with a 2048-token context and temperature 0.8, or you can serve it through run_inference_server.py on localhost:8080. Quality checking is built in too — PerplexityTester scores checkpoints against wikitext, and e2e_benchmark.py wraps timed CLI runs.

Being straight about the limits, because the source is: the generated and preset kernels are shape-pinned, so new model widths require a codegen run (docs/codegen.md shows the TL1 and TL2 invocations) and ideally a tuning sweep with GemmTuner, which searches configurations into kernel_config.ini (generate_configurations at line 262). Batch size 1 is the design point; the CUDA kernels literally check M == 1. And the patched llama.cpp is pinned, not merged — you are building a fork by design.

That trade is the whole point. BitNet buys CPU-side 3B-model inference at interactive speeds by refusing generality everywhere else: weights pre-shuffled on disk, kernels stamped out per model, one entry point that does everything. Compare it with the serving-grade generality of vLLM, the embedding focus of Text Embeddings Inference, the int8 CPU story of faster-whisper, or the self-hosted API surface of LocalAI — BitNet is the narrowest of the family and the only one betting the model format itself on ternary weights. For the conceptual background, start with our first BitNet post; for the other end of the efficiency spectrum, see our inference index.

Enjoyed this post? Never miss out on future posts by following us