The newest generation of text-to-speech systems stopped pretending to be signal processing and started being language models, and fish-speech is one of the clearest open examples of that shift. Its flagship model, Fish Audio S2, is a 4-billion-parameter transformer trained on over ten million hours of audio in more than eighty languages, and it treats speech exactly the way an LLM treats text: raw audio is compressed into a vocabulary of discrete tokens, a decoder-only transformer predicts the next tokens, and generation streams out token by token. The results put it at the top of public leaderboards — the lowest word-error rates on Seed-TTS Eval in both Chinese and English, ahead of closed systems — while the feature that gets the attention is expressive control: bracketed natural-language tags like [whisper], [excited], or [laughing] embedded anywhere in the text steer prosody at the sub-word level. What makes the repository worth a tour is that every layer of that stack is small enough to read. As always in this series, what follows is an educational tour of published source code.

The architecture splits cleanly in half, joined by the token vocabulary. The semantic half is a Dual-AR transformer — a master decoder that predicts a coarse first codebook, with a slave head that predicts the remaining fine codebooks in parallel — wrapped by a generation loop that handles sampling, streaming, and interruptions. The codec half is a modified DAC, a convolutional encoder-decoder with residual vector quantization across ten codebooks at roughly twenty-one hertz, that converts raw audio to tokens and back. The inference engine glues the two: it loads a reference clip, builds the cloning prompt, runs the transformer, and decodes tokens to waveform. Around them sit the tooling that makes the system reproducible — packed protobuf datasets, extraction and building scripts, LoRA fine-tuning, quantization — and the serving layer, an OpenAI-compatible API plus a Gradio WebUI. The catch is the license: code and weights ship under the Fish Audio Research License, not an OSI license, which shapes who can use it.

Fish Speech at a glance: the inference engine drives the Dual-AR transformer through its generation loop while encoding references through the DAC codec and its RVQ quantizer, LoRA adapts the model, packed datasets feed training, and the API server plus WebUI put it all behind endpoints.

Reading the overview from left to right:

• The inference engine is the public class that turns text plus a reference into audio.
• The reference loader cuts and encodes the voice you clone from.
• The Dual-AR transformer is the semantic model with slow and fast codebook heads.
• The generation loop samples tokens and streams semantics as they arrive.
• LoRA attaches low-rank adapters for cheap fine-tuning.
• The DAC codec converts raw audio to tokens and back.
• The RVQ quantizer defines the audio vocabulary across ten codebooks.
• The API server exposes OpenAI-style TTS endpoints with streaming.

Why You Need This

The first reason is the Dual-AR design, because it is the whole trick of making audio generation fast without losing quality. llama.py implements a decoder-only transformer — rotary embeddings, grouped-query attention, standard LLM plumbing — but the output head is split in two: a slow head that autoregressively predicts the first, coarsest codebook, and a fast head that predicts the remaining codebooks in parallel conditioned on it. Since the RVQ codebooks form a hierarchy from coarse to fine, this gives near-LLM generation speed at ten times the token density. The generation loop is where pragmatics live — top-p and top-k sampling, repetition windows, a semantic-id for interruptible streaming, and the bracket-tag parsing that turns [whisper] in your text into conditioning the model was trained to obey. The Lightning module shows the training objectives, and LoRA with its ready-made rank configs is how the community adapts the model to new voices on single GPUs.

The second reason is the codec, because the token vocabulary is where speech quality is won or lost. modded_dac.py is a modified version of the Descript Audio Codec: convolutional downsampling to a twenty-one-hertz latent rate, then the residual vector quantizer — ten stacked codebooks where the first captures the coarsest shape and each subsequent one quantizes the residual error, exactly the hierarchy the Dual-AR head exploits. The codec ships with its own inference wrapper, and the fork exists precisely because the original DAC’s rate and codebook layout were re-tuned for speech; reading it alongside the transformer explains why this model can stream at conversational latency where older codecs could not. The preprocessing tools close the loop: extract_vq.py batch-encodes raw audio into token files, the split tool divides the output for training, and the codec’s own configs show how it is pretrained before the semantic model ever runs.

The third reason is the data and serving engineering, which is what turns a model into a product. Training data flows through a protobuf format — the text-data definitions and the packed semantic dataset that reads them — built by the dataset builder from extracted tokens and transcripts, a design that ships terabytes of tokenized speech as sequential archives. On the serving side, the API views expose OpenAI-compatible chat-style TTS routes with server-sent-event streaming, the model manager keeps checkpoints loaded, and the typed Python client shows the full request surface. The WebUI wraps the same engine in Gradio for local cloning sessions, and quantize.py shrinks weights for smaller GPUs — deployment options for every budget.

The detail view: the engine group with reference loading and the API client pair; the semantic model with its training module and weight tools; the codec with quantizer and wrappers; the data group from extraction to packed archives; the serving group of views, manager and WebUI; and the config layer.

The detail view shows how little glue the design needs. The engine is thin because the codec and the transformer share one vocabulary — the VQ manager just loads codebooks, and the reference loader runs the same encode path that training used, so cloning prompts are always in-distribution. The serving stack separates concerns cleanly: the entry point parses flags, views handles HTTP, api_utils.py encodes responses, and the WebUI can call the engine directly through its own inference module when no server is wanted. The config layer is equally deliberate — a base YAML extended by fine-tune and codec variants — and utilities like the text cleaner and spectrogram helpers stay minimal because the model consumes tokens, not handcrafted features.

From Install to a Cloned Voice

The quickstart runs through the WebUI. Install with uv or pip, then python tools/run_webui.py — the tool downloads the S2 checkpoint on first start, opens a browser page, and you paste text, drop in a reference clip, and hit generate; enable streaming to hear audio begin before synthesis completes. The same stack works headless: python tools/api_server.py starts the OpenAI-compatible server, and any client that speaks the OpenAI audio API format — or the bundled api_client.py — can POST text with a reference voice and receive streamed audio. Fine-tuning is a documented sequence: extract tokens from your dataset, build the packed archive, then train with the LoRA configs from a single GPU. One caution: the weights are large, the first download is many gigabytes, and a CUDA GPU is effectively required for interactive use. The bracket-tag system is learned, not enforced — tags outside the model’s training distribution may be ignored — so expressive control works best with natural phrasing.

Honest limits: the license is the big one. The Fish Audio Research License is not OSI-approved, restricts commercial deployment, and the maintainers state they will act against violations — for commercial voice products you are directed to their hosted API. The WebUI is functional rather than polished, multi-speaker conversation generation is exposed through the API but thinly documented in the repo, and the codec checkpoint, being tuned for speech, is not a general audio codec. The benchmark numbers come from the technical report rather than independent replication. None of this changes the educational value: this is the most readable end-to-end example of the audio-token paradigm — codec, transformer, streaming, and control tags — in active development today.

Enjoyed this post? Never miss out on future posts by following us