Music source separation is the task of taking one mixed recording and pulling it apart into its ingredients: drums, bass, the rest of the instruments, and the vocal. It sounds like an audio party trick, but it is a genuinely hard prediction problem — the model has to listen to a dense mixture and hallucinate a signal that was never isolated on the recording. Demucs, maintained by Meta’s research org on GitHub as facebookresearch/demucs, is one of the two or three models that defined this field, and it is the engine behind a large share of the “stem splitter” tools you have seen online. The current flagship checkpoint, htdemucs, is a hybrid: a convolutional U-Net over the raw waveform, a second U-Net over the spectrogram, and a transformer sitting at the bottom where the two representations exchange information. That combination is why the paper calls it the Hybrid Transformer Demucs.

What makes the repository worth a source tour is how small and legible it is. Everything that matters lives in about thirty Python files under a single demucs/ package — no service layers, no plugin registries, no tangled abstraction. You can read the whole inference path in one sitting: a CLI in separate.py, a friendly wrapper in api.py, a checkpoint resolver in pretrained.py and repo.py, the shift/split/overlap logic in apply.py, and the model itself in htdemucs.py, hdemucs.py, transformer.py, and spec.py. Training is equally compact: train.py builds a solver, and solver.py runs the epochs. The same organization also ships AudioCraft for generative audio (our sibling tour lives at /audiocraft/), so if you have read that one, the code style here — plain torch, explicit shapes, long honest docstrings — will feel familiar.

This post walks the library end to end as of the main branch (October 2026): first the architecture, then the checkpoint-to-stems inference path, then a working quickstart, and finally the limits you will hit in production. Every symbol and line number below was verified against the actual checkout, and both diagrams were generated from the real dependency structure of the code. The whole thing stands on PyTorch (/pytorch/), so tensor-level details translate directly.

The Demucs system at a glance: a thin CLI and API shell around a checkpoint registry, an inference engine that segments and blends, and the hybrid model family (waveform U-Net, hybrid convolutions, cross-domain transformer) that does the actual prediction.

Reading the overview from left to right:

• separate.main, L102 — the demucs console command: parses flags, constructs a Separator, then loops over tracks and writes one stem file per source.
• Separator, L53 — the Python-facing class that owns the model, device, and segment settings; its separate_audio_file, L293 method is the workhorse.
• get_model, L59 — turns a name like htdemucs into a fully loaded nn.Module by delegating to the repository layer.
• AnyModelRepo, L148 — resolves names against the remote index packaged in the wheel, or against a local directory if you pass one.
• BagOfModels, L29 — a weighted ensemble container; the popular htdemucs_ft variant is a bag of four fine-tuned checkpoints whose predictions are averaged.
• apply_model, L145 — the segmentation engine: random shifts, overlapping chunks, triangular cross-fade weights, and per-source blending.
• HTDemucs.forward, L527 — one forward pass: STFT, dual U-Net, cross-domain transformer, masking, and inverse STFT.
• CrossTransformerEncoder, L526 — the bottleneck block where the waveform and spectrogram branches finally talk to each other.
• AudioFile, L28 — the ffmpeg-backed reader that decodes any input format and resamples on the fly; save_audio, L236 handles the writing side.

The detailed call graph. Use it as a map while reading the sections below — the numbers in the captions match the line numbers in the current `main` branch.

What Demucs Actually Is

The library ships several model generations under one roof, and the registry treats them uniformly. Demucs itself (Demucs, L219) is the original: a purely waveform-domain U-Net with a BLSTM bottleneck, encoder blocks built from convolution, a DConv residual branch (DConv, L86), and skip connections back up the decoder. HDemucs (HDemucs, L338) added the spectrogram branch. HTDemucs (HTDemucs, L27) is the v4 model that adds the transformer and remains the default — DEFAULT_MODEL, L24 is literally the string htdemucs, and the loader prints a notice reminding you the default changed from the older MDX checkpoints.

All sources are predicted jointly: the standard four are drums, bass, other, and vocals. A forward pass returns a tensor of shape [batch, sources, channels, samples], and the loss during training is computed in the time domain, backpropagating through the masking and inverse STFT. That design choice — end-to-end differentiable decoding rather than a separate spectral loss — is what makes the hybrid branches cleanly combinable.

The Hybrid Front End: Waveform and Spectrogram

HTDemucs.forward starts by padding the mixture to the training segment length — ten seconds at 44.1 kHz by default — because convolution alignment is easier when every call sees the same geometry. Then _spec (HTDemucs._spec, L420) pads the signal by three quarters of the kernel, calls spectro, L11 (a thin wrapper over torch.stft with a Hann window), and trims the edges so the STFT output lines up sample-for-sample with the waveform branch. The FFT size is 4096 with a hop of 1024.

A detail that matters: the released model uses complex as channels (cac=True), so instead of predicting a magnitude mask and running EM refinement, _magnitude (HTDemucs._magnitude, L452) reshapes the complex spectrogram into real channels, the network predicts full complex spectra for each source, and _mask (HTDemucs._mask, L463) simply reinterprets them — the OpenUnmix Wiener path in _wiener (HTDemucs._wiener, L480) stays available for the older non-CaC configurations. The two branches then run through matched stacks of HEncLayer (HEncLayer, L69) and HDecLayer (HDecLayer, L256) blocks — each a strided convolution, a depthwise DConv residual, and a 1×1 rewrite — with the frequency branch compressing along frequency until it collapses to the time axis, while the waveform branch strides in time in parallel. A learned frequency embedding (ScaledEmbedding, L43, injected around line 360 of htdemucs.py) breaks the frequency-axis symmetry, and MultiWrap (MultiWrap, L160) lets outer layers process frequency bands with shared weights.

The Cross-Domain Transformer Bottleneck

At the bottom of both U-Nets — four downsampling stages deep, 48 base channels with a growth factor of two, so 384 channels — the model inserts the transformer. If bottom_channels is set, a pair of 1×1 convolutions first projects both branches to a common width; then crosstransformer runs t_layers = 5 layers per branch. Each layer is a CrossTransformerEncoderLayer (CrossTransformerEncoderLayer, L380) in pre-norm style with a residual LayerScale (LayerScale, L236), and the attention inside is a custom MultiheadAttention (MultiheadAttention, L725) that supports structured sparsity: the mask can be diagonal-band (diag), random, or global-token patterns, mixed freely, which is what keeps 10000-position sequences affordable.

The point of the block is the cross part: after self-attention within each branch, the spectrogram stream queries the waveform stream and vice versa, so low-level timing cues from the waveform can disambiguate spectral overlaps and vice versa. Both updated sequences are returned to the decoder, which mirrors the encoder with skip connections on each side and finally sums the two reconstructed sources in the time domain. It is a clean answer to a classic question in this literature — you rarely have to choose between time-domain and frequency-domain modeling when you can afford both.

From Checkpoint to Stems: The Inference Path

The loader is refreshingly explicit. get_model builds a RemoteRepo from a files.txt index that ships inside the package (REMOTE_ROOT, L21 points into demucs/remote), a BagOnlyRepo that parses the bag YAML files you saw above, and an AnyModelRepo that tries both; there is even a special case that detects a missing diffq dependency from the import error. Once loaded, the model is set to eval and returned.

The engine around the forward pass is where the practical quality lives. apply_model handles three orthogonal tricks. First, bags: it runs each sub-model and averages the estimates with per-source weights. Second, shifts (API default 1): the mixture is randomly shifted by up to half a second, the whole prediction is repeated, and outputs are unshifted and averaged — a cheap test-time ensemble that measurably improves SDR. Third, splitting: the track is cut into model.segment-long chunks with overlap = 0.25, each chunk is predicted independently, and a triangular weighting kernel (built around line 271 of apply.py) cross-fades the joints before a final normalization. On CPU you can pass num_workers and the chunks are submitted to a thread pool. The top-level Separator wraps all of this and handles decoding through AudioFile.read and encoding through save_audio, which supports wav, flac, and mp3 with clip-mode control.

From Install to Your First Model

Install with pip install demucs (or clone the repo and pip install -e .), and make sure ffmpeg is on your PATH — the I/O layer shells out to it for both probing and decoding. Then separate your first track:

demucs -n htdemucs "your_song.mp3"

You will find the four stems under separated/htdemucs/your_song/vocals.wav and friends. Two variations worth trying immediately: htdemucs_ft is the fine-tuned bag of four checkpoints — noticeably better on vocals, roughly four times slower because it averages four full predictions — and --two-stems=vocals writes just the vocal and a karaoke-style complement. The equivalent Python is short:

from demucs.api import Separator

sep = Separator(model="htdemucs")
origin, stems = sep.separate_audio_file("your_song.mp3")
print(stems.keys(), sep.samplerate)

If you use torchaudio (/torchaudio/), note the cross-pollination: torchaudio ships its own HDemucs implementation, and Demucs’s training script can wrap it directly via TorchHDemucsWrapper (TorchHDemucsWrapper, L36) for benchmarking against the same checkpoint metadata. Training your own variant runs through hydra configs: python -m demucs.train builds datasets (Wavset, Wavset, L107, random-crops pre-extracted wav tracks), constructs the optimizer, and hands everything to Solver.train (Solver.train, L172), whose per-epoch loop lives in _run_one_epoch (Solver._run_one_epoch, L291).

Honest Limits

Memory is the first wall. The hybrid model processes ten-second 44.1 kHz stereo segments with two parallel U-Nets plus attention at the bottom, and on consumer GPUs you will hit out-of-memory long before you expect; the segmented apply_model path exists precisely because full-track inference does not fit. CPU-only separation works but is slow — plan minutes per song, not seconds, and use --jobs to parallelize chunks. On Apple silicon the code even detours complex tensors through the CPU because MPS lacks complex support (see the explicit workaround in HTDemucs.forward), so expect the same order of slowness there.

Second, scope: Demucs is a music separator. Speech enhancement, speaker diarization, and transcription belong to other tools entirely — for transcript-oriented audio work, see our tours of /whisperx/ or /pyannote-audio/. Third, quality is checkpoint-dependent: the htdemucs name changed the default model, and the maintainers themselves note it can underperform the older MDX models on some material, which is why bag variants and the --shifts flag exist as tuning knobs rather than conveniences. Finally, training is a serious undertaking — it assumes MUSDB18-HQ, the Dora experiment runner, and multi-GPU patience. For what it is, though, the codebase is a model of restraint: a real production model you can read in an afternoon, and one of the best illustrations that hybrid architectures are not a compromise but a deliberate design.

Enjoyed this post? Never miss out on future posts by following us