LLM Training Pipeline: From Pretraining to RLHF and DPO

A capable LLM like Llama-3 or GPT-4 is not born in a single training run. It is built in three distinct stages, each with different data, objectives, and compute requirements. The first stage – pretraining – teaches the model language, facts, and basic reasoning by predicting the next token across trillions of words. The second stage – supervised fine-tuning (SFT) – teaches it to follow instructions and hold conversations. The third stage – RLHF or DPO – aligns the model’s outputs with human preferences for helpfulness, honesty, and harmlessness.

Understanding this pipeline is essential for anyone who wants to fine-tune models, evaluate training options, or reason about why models behave the way they do. This is the fifth post in our LLM internals series, following our coverage of tokenization, attention, KV-cache, and inference phases.

The Three-Stage Pipeline

Understanding the Diagram

The diagram above shows the complete training pipeline with three stages side by side, followed by a comparison table and a conceptual loss curve. Let us examine each stage in detail.

Stage 1: Pretraining (The Foundation)

Pretraining is where the model learns the fundamentals. The training data consists of trillions of tokens scraped from the internet: web pages (Common Crawl), books, code repositories (GitHub), Wikipedia, scientific papers, and more. For Llama-3, this dataset comprises approximately 15 trillion tokens – roughly 60 terabytes of text.

The Objective: Next-Token Prediction

The model is given a sequence of tokens and asked to predict the next one. If the input is “The cat sat on the”, the model should output a high probability for “mat” (or “floor”, “chair”, etc.). The loss function is cross-entropy:

L = -sum log P(token_i | tokens_0..i-1)

This seemingly simple task forces the model to learn grammar, facts, reasoning patterns, code structure, and world knowledge. To predict the next token accurately across diverse text, the model must build an internal representation of how language works.

Training Details

Pretraining consumes approximately 90% of the total training compute. For a 7B parameter model, this means roughly 6,000 H100-days of GPU time. Key hyperparameters include:

• Learning rate: ~3e-4 with a warmup schedule (starts low, rises, then decays)
• Batch size: ~4 million tokens (achieved through gradient accumulation)
• Context length: 4K-8K tokens per sequence
• Optimizer: AdamW with weight decay
• Precision: BF16 mixed precision (saves memory while maintaining stability)

The Output: A Base Model

After pretraining, you have a “base model” – a powerful next-token predictor that has absorbed vast knowledge but cannot follow instructions. If you ask a base model “What is the capital of France?”, it might respond with “What is the capital of Germany?” (completing your question with more questions) rather than answering. The base model completes text; it does not converse.

Stage 2: Supervised Fine-Tuning (SFT)

SFT transforms the base model into an instruction-following assistant. The training data changes from raw text to high-quality instruction-response pairs: a human-written prompt and an ideal response. The diagram shows an example pair where the human asks “Explain what a database index is” and the assistant provides a clear explanation.

The Objective: Response Prediction

The loss function is the same cross-entropy as pretraining, but it is only computed on the response tokens – the prompt tokens are masked. This teaches the model to generate good responses given prompts, without trying to learn to generate the prompts themselves:

L = -sum log P(response_tokens | prompt_tokens)

Training Data Sources

SFT data comes from several sources:

• Human-written demonstrations (expensive but high quality)
• Synthetic data generated by larger models (e.g., GPT-4 generating training pairs for a smaller model)
• Distilled from existing conversation datasets
• Open-source datasets like OpenOrca, FLAN, and UltraChat

The dataset is much smaller than pretraining: typically 100K to 10M examples, compared to 15 trillion tokens. Despite this small size, SFT is remarkably effective because the model has already learned language and knowledge during pretraining – it just needs to learn the format of instruction following.

Training Details

SFT uses a much lower learning rate (~2e-5, about 10x lower than pretraining) to avoid destroying the knowledge learned during pretraining. Training runs for only 1-3 epochs to prevent overfitting. The total compute is approximately 5% of pretraining.

The Output: An Instruct Model

After SFT, the model can follow instructions and hold conversations. However, it may produce harmful, biased, or low-quality outputs because it has only learned to mimic the format of helpful responses, not the values underlying them. An SFT model might happily provide instructions for harmful activities if asked in the right way.

Stage 3: RLHF or DPO (Alignment)

The alignment stage teaches the model to be helpful, harmless, and honest. There are two main approaches:

RLHF (Reinforcement Learning from Human Feedback)

RLHF, introduced by OpenAI in the InstructGPT paper, works in two steps:

Step 1: Train a Reward Model (RM). Human raters compare two or more model responses to the same prompt and choose the preferred one. A separate reward model is trained on these comparisons to predict which response a human would prefer. The RM takes a (prompt, response) pair and outputs a scalar score.

Step 2: Optimize the Policy with PPO. The SFT model (now called the “policy”) generates responses, the reward model scores them, and the Proximal Policy Optimization (PPO) algorithm updates the policy to maximize the reward. A KL divergence penalty keeps the policy close to the SFT model to prevent it from gaming the reward model.

RLHF is powerful but complex: it requires training a separate reward model, sampling from the policy during training (expensive), and tuning multiple hyperparameters (PPO learning rate, KL penalty coefficient, reward model learning rate). GPT-4, Claude 2, and Llama-2 Chat all use RLHF.

DPO (Direct Preference Optimization)

DPO, introduced by Rafailov et al. in 2023, simplifies the alignment process dramatically. The key insight is that the reward model can be expressed in terms of the policy itself, eliminating the need for a separate RM. DPO directly optimizes the policy using preference pairs (prompt, chosen response, rejected response) with a simple binary classification loss:

L = -log sigmoid(beta * (
  log_pi(chosen) - log_pi_ref(chosen)
  - log_pi(rejected) + log_pi_ref(rejected)
))

This increases the probability of the preferred response and decreases the probability of the rejected response, relative to a reference model (the SFT model). DPO is simpler, more stable, and faster to train than RLHF, while achieving comparable or better results. Llama-3, Mistral, and Zephyr all use DPO.

Training Data for Alignment

Both RLHF and DPO require preference data: tuples of (prompt, chosen_response, rejected_response). This data is collected by:

• Human raters comparing model outputs (expensive, slow)
• AI feedback (e.g., Constitutional AI where a model critiques its own outputs)
• Preference synthesis from existing datasets

The dataset is typically 100K to 1M preference pairs – much smaller than SFT data.

The Comparison Table

The diagram’s comparison table highlights the key differences across stages:

Aspect	Pretraining	SFT	RLHF / DPO
Goal	Learn language & knowledge	Follow instructions	Align with preferences
Data volume	~15T tokens	~100K-10M examples	~100K-1M preferences
Compute	~90% of total	~5% of total	~5% of total
Learning rate	~3e-4	~2e-5	~5e-7
Can chat?	No	Yes (basic)	Yes (aligned)
Safety?	No	Minimal	Yes

The most important insight: pretraining dominates compute (90%) and is where the model acquires its knowledge. SFT and RLHF are relatively cheap (5% each) but transform the model’s behavior. This is why “fine-tuning” a pretrained model is so much cheaper than training from scratch.

The Loss Curve

The bottom of the diagram shows a conceptual loss curve. Pretraining shows a large, gradual loss decrease over months of training. SFT produces a small additional drop. RLHF/DPO produces an even smaller change in loss but a significant change in behavior quality – the loss reduction is small because the model is already good at generating text; alignment is about changing what it chooses to generate, not improving its raw predictive ability.

Why Three Stages Instead of One?

You might wonder: why not train on instruction-response data from the start? The answer is data efficiency and capability.

Pretraining on raw text is data-efficient: The internet contains trillions of tokens of raw text but only millions of high-quality instruction pairs. By pretraining on raw text, the model learns language patterns, facts, and reasoning from the vast majority of available data. If you trained only on instruction pairs, the model would see 1000x less data and would be far less knowledgeable.

SFT adds format without destroying knowledge: By fine-tuning with a low learning rate on a small dataset, SFT teaches the model the conversational format without overwriting the knowledge learned during pretraining. The model already “knows” what France’s capital is; SFT just teaches it to answer the question rather than continue the text.

Alignment adds values without reducing capability: RLHF/DPO makes tiny adjustments to the model’s output distribution, steering it toward helpful and safe responses. The change in the loss function is small, but the change in user experience is dramatic – the model becomes reliable enough for production deployment.

What Can Go Wrong at Each Stage

Pretraining failures: If the training data contains too much low-quality content (spam, duplicates, toxic text), the model will learn those patterns. If the learning rate is too high, the model can diverge (loss spikes to infinity). If the context length is too short, the model cannot learn long-range dependencies.

SFT failures: If the SFT data is too homogeneous (all from one source), the model will overfit to that style. If the learning rate is too high, the model “forgets” pretraining knowledge (catastrophic forgetting). If you train for too many epochs, the model memorizes the training data and loses generalization ability.

Alignment failures: If the reward model (in RLHF) is poorly calibrated, the policy will exploit its weaknesses (reward hacking). If the KL penalty is too low, the model drifts too far from the SFT model and becomes incoherent. If the preference data is biased, the model will reflect those biases. DPO avoids the reward model issue but can still suffer from distribution shift if the preference data does not match the model’s output distribution.

Practical Implications

For fine-tuning: If you want to customize a model for your use case, you typically start from an SFT model (not a base model) and apply either continued SFT on your data or DPO with your preference pairs. Fine-tuning a base model requires significant SFT data to first teach it the conversational format.

For cost estimation: The 90/5/5 compute split means that training a model from scratch is dominated by pretraining costs. If you can start from a pretrained base model (like Llama-3-8B-Base), your fine-tuning costs are only 10% of training from scratch. If you start from an SFT model (like Llama-3-8B-Instruct), your alignment costs are only 5%.

For model selection: Base models are useful for research and custom pipelines where you want full control over the output format. SFT models are useful for general instruction following. Aligned models (RLHF/DPO) are what you want for production chatbot deployments where safety and helpfulness matter.

For evaluation: Each stage requires different evaluation metrics. Base models are evaluated on perplexity (how well they predict text) and downstream benchmarks (MMLU, HellaSwag). SFT models are evaluated on instruction-following benchmarks (AlpacaEval, MT-Bench). Aligned models are evaluated on safety benchmarks (TruthfulQA, BBQ) and human preference ratings.

Further Reading

• Training Language Models to Follow Instructions with Human Feedback (Ouyang et al., 2022) – The InstructGPT paper that introduced RLHF for LLMs
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Rafailov et al., 2023) – The DPO paper that simplified alignment

Related Posts

• LLM Attention Mechanism: The Heart of the Transformer
• LLM Tokenization: How Text Becomes Numbers
• LLM Decode Deep Dive: KV-Cache, GPU VRAM, and the Memory Bottleneck
• LLM Prompt vs Decode: Understanding the Two Phases of LLM Inference

Conclusion

The three-stage training pipeline – pretraining, SFT, and RLHF/DPO – is the recipe that transforms random weights into a helpful assistant. Each stage has a distinct purpose: pretraining builds knowledge, SFT builds conversational ability, and alignment builds trustworthiness. The compute distribution (90/5/5) means that most of the cost is in pretraining, which is why open-source base models are so valuable – they let you skip 90% of the cost and focus on customization.

RLHF and DPO represent two philosophies of alignment. RLHF explicitly models human preferences through a reward model and optimizes against it with reinforcement learning. DPO implicitly captures preferences through a clever mathematical reformulation that eliminates the reward model entirely. As of 2026, DPO has become the preferred method for open-source models due to its simplicity and stability, while RLHF remains common in large-scale proprietary training where the complexity is justified by scale.

Understanding this pipeline demystifies the process of creating LLMs. A model is not “trained” in one shot – it is carefully sculpted through stages that each add a layer of capability. This staged approach is what makes modern LLMs both knowledgeable (from pretraining) and usable (from SFT and alignment).

Enjoyed this post? Never miss out on future posts by following us