Every retrieval system has a generation model that gets the applause and an embedding model that does the work, and the embedding side is exactly where huggingface/text-embeddings-inference lives. TEI — as everyone calls it — is a Rust server whose entire purpose is to turn text into vectors, vectors into similarity scores, and query-document pairs into reranking scores, at throughputs that made it the default backing store for Hugging Face’s own inference endpoints. Unlike the general-purpose serving engines this series has toured, from vLLM to SGLang to the ragged engine of DeepSpeed, TEI narrows its ambition to a small set of request shapes and then optimizes them without mercy: it holds no chat templates, samples nothing, and generates nothing beyond a fixed pool of tokens. That narrowness is what makes its source so readable — two crates of Rust, one candle-based backend, and a model zoo that reimplements fourteen architectures without a single line of PyTorch. As always, this is an educational tour of published source on the main branch, Hugging Face’s own license, and readers wanting the generation-side counterpart can compare text-generation-inference.

The one-sentence summary: an axum router translates five API dialects into calls on Infer, Infer orchestrates a tokenization pool, a Queue, and a blocking backend call, and the backend is CandleBackend, a dispatch table over hand-written model implementations that end in CUDA flash attention.

Overview of TEI: CLI arguments flow into router::run, which wires the hub downloader, tokenization pool, queue, and candle backend before serving HTTP; the Infer facade batches requests through the queue into the backend, whose Model trait dispatches to BertModel and the CUDA flash-attention path.

Reading the overview from left to right:

• main.rs parses the CLI into an Args struct and main at line 204 validates the combination before anything boots.
• router::run is the assembly line: download at line 106, tokenization at line 265, backend at line 285, queue at line 322, Infer at line 330.
• the HTTP server registers the whole route table at lines 1761 through 1774.
• request types carry PredictRequest at line 215 and RerankRequest at line 243.
• Infer::new spawns batching_task at line 525 and backend_task at line 547.
• Queue::next_batch assembles pending Entries into one batch.
• the tokenization pool encodes inputs with truncation and length limits through encode at line 68.
• download_artifacts fetches weights, config, and pooling metadata from the hub.
• CandleBackend implements the Backend trait with embed at line 656 and predict at line 711.
• the Model trait is the dispatch contract every zoo entry satisfies.
• BertModel is the reference citizen, its forward at line 706 returning pooled embeddings and raw token states.
• flash_attn_varlen picks the attention kernel by GPU compute capability.

One router, every protocol

The router’s first trick is breadth: a single process speaks the dialects of half the embedding industry. The core routes are native — embed for pooled or sparse vectors, rerank for cross-encoder scoring, predict for classification labels, and similarity, which is a two-vector cosine computed server-side. Around them sit the compatibility shims that made adoption frictionless: openai_embed serves the /v1/embeddings shape so OpenAI-SDK clients work unmodified, and vertex_compatibility covers Google’s contract, with the route table showing the wiring plainly. There are also tokenize and decode debug endpoints that expose the raw tokenizer, and a metrics route feeding Prometheus. Behind HTTP there is an alternative transport — a tonic gRPC server sharing the same Infer facade — and the server code even carries conditional rewrites for SageMaker and Vertex deployment conventions, remapping / and /invocations to the right handlers. The design point worth stealing: the protocol layer is deliberately thin, and every route funnels into the same three facade verbs, so adding a dialect costs a struct and a handler, not a new runtime.

The batch machine

The core crate is a compact lesson in Rust async architecture. Requests arrive as futures, and the first stop is admission control: try_acquire_permit draws from a semaphore sized by max concurrent requests, so overload degrades to queueing at the door instead of memory pressure inside. Tokenization never blocks the async runtime because it runs in a dedicated pool — Tokenization::new spins worker threads whose count defaults to one less than the machine’s core count, and the run() wiring makes that default explicit and tunable. Encoded requests become Entry values appended to the Queue, and the whole batching policy is one function: next_batch waits for at least one entry, then drains up to the model’s batch capacity without padding concerns — sequences ride through the whole flow with their true lengths. Two spawned tasks close the loop: batching_task pulls NextBatches and sends them down a channel, while backend_task receives them and issues the actual model call through spawn_blocking, because candle’s synchronous execution would otherwise stall the async reactor. Responses return through per-request oneshot channels, and the facade computes its own metrics — InferMetadata carries tokenization and queue timings for the Prometheus histograms. It is the same producer-consumer shape this series found in faster-whisper’s generator loop and ONNX Runtime’s thread pools, expressed in twenty lines of channels and semaphores.

The candle backend and its model zoo

Candle is Hugging Face’s minimal tensor framework, and TEI’s backend crate is effectively a candle showcase. CandleBackend::new inspects the model type and configuration, then builds exactly one architecture object; the Backend impl reports max_batch_size at line 640 — the flash variants advertise no limit because they never pad — and dispatches embed and predict calls into the zoo. The zoo itself is the Model trait: implement is_padded, embed, and optionally predict, and you are in the family. BERT is the reference implementation to read first: BertEmbeddings at line 42 handles tokens and positions, BertAttention at line 117 carries the query-key-value projection with either absolute or rotary position treatment, BertEncoder at line 340 stacks the layers, and the heads make one model serve three businesses — a classification head for predict, a SPLADE head at line 468 for sparse retrieval, and plain pooling for dense embeddings. The rest of the zoo reads as a curriculum in embedding architecture: ModernBertModel with its alternating attention and every-fourth-layer norm scheme, DistilBertModel as the distilled student, GTEModel adding a dense projection for Matryoshka-style truncation, MPNetModel with disentangled position attention, NomicBertModel for long contexts, and Qwen3Model showing that decoder-style LLMs now double as embedders, bringing rotary layers from the shared rotary module and RMSNorm with them. Shared plumbing — Linear, LayerNorm, and a cublasLt wrapper that fuses GEMM with bias and activation epilogues — keeps every model file short.

CUDA realities: compute caps and honest limits

The GPU path is where TEI stops being portable and starts being fast. flash_attn_varlen is the gatekeeper: it reads the runtime compute capability once, and the dispatch at lines 39 through 115 routes Turing’s 7.5 to flash attention v1 with its no-ALiBi, no-windowing caveats, while everything from Ampere 8.0 through Blackwell calls the windowed v2 kernel, with and without ALiBi slopes. The varlen in the name matters: because the batching layer passes true sequence lengths instead of padded ones, the kernel processes exactly the tokens that exist, which is why the flash model variants can advertise unbounded batch sizes. CUDA-only feature flags gate all of this — build without the feature and the code fails loudly rather than silently degrading to slow paths, a decision that shows up repeatedly in FlashBertModel, the fused CUDA rewrite of the BERT forward that skips the vanilla module’s intermediate tensors. Be honest about the boundaries: TEI serves one model per process, scales by running more containers, has no multi-GPU tensor parallelism, and will not chat — the gRPC and HTTP surfaces are strictly embed, rerank, predict, and similarity. That is the same trade the whole series keeps meeting, from ExecuTorch’s mobile constraints to Triton’s kernel specialization: narrow the surface, then make the narrow thing as fast as the hardware allows.

From Install to Your First Model

The fastest route is Docker: docker run --gpus all -p 8080:80 -v /data:/data ghcr.io/huggingface/text-embeddings-inference:1.8 --model-id BAAI/bge-base-en-v1.5, and the server comes up already warm, having fetched weights through download_artifacts on first use. Then curl :8080/embed -d '{"inputs":"the quiet workhorse"}' returns your first vector, :8080/rerank scores a query against candidates, and :8080/v1/embeddings answers an OpenAI-format client without it knowing the difference. The fine print is worth reading in the source rather than the README: pooling strategy is inferred from the model’s config but can be forced, the char-limit truncation in the tokenization pool quietly protects the queue from pathological inputs, and the memory floor is real — a 7 billion parameter embedder like Qwen3 needs a GPU with room to spare, while BERT-class models fit comfortably on modest cards. Where it fits, though, it is the shortest path from model name to production vectors we have measured. Pair this tour with the training-side machinery of ONNX Runtime’s execution plans, the speech sibling faster-whisper, the LLM-serving counterparts lmdeploy and MLC, and the local stack at LocalAI — the pattern repeats everywhere: production inference is a queue, a batcher, and a kernel, wearing a competent HTTP face.

Enjoyed this post? Never miss out on future posts by following us