Before Chatbot Arena became the leaderboard everyone cites in model release tweets, there had to be a machine that could host seventy models at once, shuffle anonymous battles to millions of visitors, and record every vote. That machine is FastChat, the lm-sys platform that describes itself as “an open platform for training, serving, and evaluating large language model based chatbots.” Its README claims over ten million served chat requests and more than 1.5 million human votes, and the code in fastchat/serve/ is the part that made those numbers possible. This post is a source tour of that serving stack: not the training scripts and not the leaderboard analytics, but the plumbing that turns a Hugging Face checkpoint into an OpenAI-compatible endpoint.

FastChat earns a careful reading for three reasons. First, its conversation template registry in fastchat/conversation.py became a de-facto community standard — dozens of projects imported it (or, to the maintainers’ dismay, copy-pasted it) rather than reinventing per-model prompt formats. Second, its model adapter layer in fastchat/model/model_adapter.py is a compact study in extensibility: more than a hundred small adapter classes map a model path onto the right loader, tokenizer quirks, and default template. Third, its architecture is genuinely distributed: a controller process tracks workers, workers host engines, and FastAPI front doors route to them — a shape you can still recognize in modern stacks, even as dedicated servers like TGI (/tgi/) or vLLM’s own engine server (/vllm-2/) have absorbed much of this workload with continuous batching baked in.

The diagram below is the map we will follow. On the left are the two client surfaces — the Gradio web UI that powers the Arena experience and the OpenAI-compatible REST server that lets you point openai-python at localhost. In the middle sits the controller, a registry with heartbeats and two dispatch policies. On the right, the workers: a shared scaffold class, the default Hugging Face worker, and the vLLM variant, all resting on the conversation template registry and the adapter registry.

The FastChat serving mesh: two FastAPI front doors (Gradio web UI and the OpenAI-compatible API server) ask the controller for worker addresses, then stream directly from model workers, which load weights through the adapter registry and render prompts through the conversation template registry.

Reading the overview from left to right:

• Conversation, L48 — the dataclass that turns a message list into a model-specific prompt string via get_prompt, L76; every request in the system funnels through it.
• get_model_adapter, L158 — the registry lookup that matches a model path to one of 100+ adapter classes.
• load_model, L201 — the device and quantization dispatch that every local worker calls at startup.
• BaseModelWorker, L27 — the worker scaffold: controller registration, heartbeats, and the FastAPI stream endpoint.
• ModelWorker, L38 — the default Hugging Face transformers worker, including embeddings.
• VLLMWorker, L31 — the high-throughput variant that drives an async vLLM engine instead.
• generate_stream, L62 — the hand-rolled, token-by-token generation loop used by the default worker.
• Controller, L64 — the worker registry with lottery and shortest-queue dispatch plus heartbeat-based garbage collection.
• create_chat_completion, L412 — the /v1/chat/completions handler where external clients enter the mesh.

The Conversation Template Layer

The single most-copied file in the repository defines one dataclass and one enum. SeparatorStyle (conversation.py, L16) enumerates roughly two dozen prompt formats — ADD_COLON_TWO for Vicuna, LLAMA2, LLAMA3, CHATML, RWKV, GEMMA and friends — and Conversation carries everything needed to render a chat: a system_template, role names, message history, sep/sep2 strings, stop_str, and stop_token_ids. The branching get_prompt method then assembles the exact byte sequence each model family expects, including the [INST] ... [/INST] wrapping for Llama 2 and the <|start_header_id|> headers for Llama 3.

Templates are registered into a module-level dict with register_conv_template, L698 and retrieved by name with get_conv_template, L708, which returns a copy — so a request can mutate its conversation state without corrupting the registry. The Vicuna template itself is a study in minimalism (vicuna_v1.1, L773): a system sentence, USER/ASSISTANT roles, a single-space separator, and </s> as the round-terminating second separator. You can reproduce a real client prompt in four lines with verified public APIs:

from fastchat.conversation import get_conv_template

conv = get_conv_template("vicuna_v1.1")
conv.append_message(conv.roles[0], "Hello!")
conv.append_message(conv.roles[1], None)
print(conv.get_prompt())
# A chat between a curious user and an artificial intelligence assistant.
# The assistant gives helpful, detailed, and polite answers to the user's questions.
# USER: Hello! ASSISTANT

This layer is also the contract between the API server and the workers. The REST server does not know how Vicuna formats prompts; it asks the worker for its template over HTTP (get_conv, L387), re-materializes a fresh Conversation from the serialized fields inside get_gen_params, L266, appends the OpenAI-style messages, and calls get_prompt(). Template knowledge stays in exactly one place.

The Adapter Registry: One Path In, Any Model Out

When a worker starts it calls load_model, L201, which first resolves an adapter through the registry. The base class BaseModelAdapter, L97 matches everything and simply loads via transformers’ AutoModelForCausalLM; subclasses override match() on path substrings and override loading or the default template. Registration is a decorator — register_model_adapter, L152 appends an instance to a module-level list — and lookup in get_model_adapter tries the model directory’s basename first, then the full path, then raises. VicunaAdapter, L702 is the canonical example: match vicuna, load a Llama-family model, and default to the vicuna_v1.1 template.

load_model then handles the environment matrix: CPU (optionally with Intel AMX/AVX-512 bf16 via IPEX), CUDA with multi-GPU device_map="auto" sharding through accelerate (/accelerate/), Apple MPS with a non-in-place attention patch for older transformers versions, Ascend NPU, and XPU. Quantization is pluggable the same way — 8-bit weight compression through load_compress_model, L109, AWQ checkpoints through load_awq_quantized, L24, and GPTQ through its sibling module. Two more registry helpers close the loop: get_conversation_template, L398 picks the right template for a path, and get_generate_stream_function, L404 swaps in family-specific stream implementations for ChatGLM, Falcon, CodeT5p, ExLlama and others. The default worker in /transformers/ (/transformers/) terms is therefore a thin shell around two lookups plus the generation loop.

From Install to Your First Model

The documented install is one line, with extras for workers and the web UI:

pip3 install "fschat[model_worker,webui]"

Three terminals give you the documented minimal serving stack. Terminal one starts the coordinator (its defaults are localhost:21001 with shortest_queue dispatch, per create_controller, L353):

python3 -m fastchat.serve.controller

Terminal two loads the model and self-registers; wait until you see “Uvicorn running on …”:

python3 -m fastchat.serve.model_worker --model-path lmsys/vicuna-7b-v1.5

Terminal three smoke-tests the whole chain without a browser, exactly as the README prescribes:

python3 -m fastchat.serve.test_message --model-name vicuna-7b-v1.5

test_message asks the controller for the worker address, posts to /worker_generate_stream, and prints the reply — if this works, the mesh is healthy. To expose an OpenAI-compatible endpoint, run the documented API server:

python3 -m fastchat.serve.openai_api_server --host localhost --port 8000

then call it with the standard SDK shape:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "vicuna-7b-v1.5", "messages": [{"role": "user", "content": "Hello!"}]}'

The documented throughput upgrade is a one-command swap: replace model_worker with python3 -m fastchat.serve.vllm_worker --model-path lmsys/vicuna-7b-v1.5 and leave the controller and API server untouched — the vLLM worker satisfies the same worker HTTP contract, so the front doors cannot tell the difference. That drop-in design is FastChat’s best idea, and it is why the Arena could mix local weights, quantized engines, and pure API models behind one interface.

Inside the Request Path

Follow one /v1/chat/completions call. FastAPI validates the body into ChatCompletionRequest, L58 and checks the optional static API key in check_api_key, L109. create_chat_completion, L412 then resolves a worker through get_worker_address, L367, which POSTs to the controller’s own /get_worker_address; the controller’s get_worker_address, L156 either draws a speed-weighted lottery or picks the shortest queue — the queue number arrives via worker heartbeats (receive_heart_beat, L209) and stale workers are reaped by remove_stale_workers_by_expiration, L219.

Back in the API server, get_gen_params builds the prompt from the fetched template and length-checks it against the worker’s tokenizer budget in check_length, L155. Streaming goes through chat_completion_stream_generator, L486, which POSTs to the worker and re-encodes the zero-byte-delimited JSON chunks into proper OpenAI data: delta events. On the worker side the shared endpoint api_generate_stream, L197 acquires a concurrency semaphore, then dispatches into the subclass’s generate_stream_gate, L170 — in the default worker (model_worker’s gate, L104) that yields json.dumps(...).encode() + b"\0" chunks from the generation loop, where prepare_logits_processor, L45 has already installed temperature, repetition, top-p and top-k warpers. The vLLM variant does the same dance but converts parameters into a SamplingParams object and iterates the engine’s async result stream (vllm generate_stream, L67), including client-disconnect aborts. Workers keep the controller updated every 45 seconds from WORKER_HEART_BEAT_INTERVAL in constants.py, which also defines the shared ErrorCode enum used in every error payload.

Arena and MT-Bench: The Eval Side

The serving mesh feeds the research side. In battle mode, the anonymous Arena block pairs two models — local ones via the worker mesh, and API-only ones (OpenAI, Anthropic, Gemini, Mistral, Cohere, and more) through the stream iterators in fastchat/serve/api_provider.py. Every upvote, downvote, and flag lands in the monitor directory, the same code paths that produced the LMSYS-Chat-1M and Arena-33k datasets. On the automated side, the fastchat/llm_judge/ package implements MT-bench: make_judge_pairwise, L137 builds judge objects over question/answer files, and run_judge_single, L135 prompts a strong model to grade answers on a 1-10 scale, with reference answers optionally injected for math and reasoning questions. The Judge class (Judge, L59) is deliberately small — the interesting engineering is in the match bookkeeping around it. Training shares the same foundations: fastchat/train/train.py fine-tunes Vicuna-style models on the conversation templates you just saw, with attention kernel swaps for flash-attention (/flash-attention/) wired in as monkey patches.

Honest Limits

The architecture shows its age in specific, inspectable ways. The repository’s own news section ends in early 2024, and the project now reads as a legacy-but-stable platform: newer stacks — vLLM’s native server, SGLang, TGI, or Triton Inference Server (/triton-server/) — handle continuous batching, token streaming, and multi-GPU scheduling inside one process, while FastChat’s default model_worker path generates tokens one forward pass at a time with a Python loop. For interactive demos that is fine; for production throughput it is not, which is precisely why the vLLM worker exists.

The multi-process design also has real operational cost. A minimal deployment is already three processes plus a front door; each worker is its own FastAPI app with heartbeats, and the controller must be reachable by everything. There is no built-in service discovery beyond HTTP, no dynamic scaling, and authentication is a single static API key. Restarts matter too: the controller is in-memory state, so a crash orphans workers until they re-register.

Template maintenance is a second chronic cost. Keeping two dozen separator styles and a hundred-plus hand-written templates in sync with the hubs’ own chat_template files is unwinnable over time, and the file’s opening plea — contribute changes back instead of copying — is a scar from years of forks drifting. Quantization and engine coupling is the third: the vLLM worker contains compatibility shims for engine-internal renames (tokenizer groups, SamplingParams fields that later versions removed), so FastChat pins you to a band of vLLM releases rather than tracking its head. Finally, the non-streaming paths (generate_gate, embeddings) simply drain the stream generator to completion, which is honest and simple but wasteful under concurrency — a reminder that FastChat was built to answer “can we serve this model at all?” in 2023, not “at what p99?”.

Enjoyed this post? Never miss out on future posts by following us