Reinforcement learning from human feedback is the most distributed-shaped problem in LLM training: you need a policy being trained, a reference model frozen in time, a reward model scoring outputs, a critic estimating values, and a generation engine producing rollouts - each with different resource needs, all cooperating every step. OpenRLHF, the framework behind the REINFORCE++ line of work and a teaching case in CMU’s graduate NLP course, answers with a single-controller architecture: one Python process holds the training logic while Ray actors hold the models, and a fleet of vLLM engines does nothing but generate. Release 0.11.3 under an Apache-2.0 license fits all of this into a remarkably small codebase, which is exactly why it rewards a careful read.

The whole system speaks one data language. A dataclass called Experience - prompts, responses, log probabilities, rewards, values, losses - flows from the rollout engines through a replay buffer into the trainers, and every algorithm from PPO to GRPO to RLOO is a different way of filling its fields and weighting its updates. If you want to understand how production RLHF systems actually share state across heterogeneous model workers, this is the clearest public map available. As always in this series, what follows is an educational tour of published source code.

Overview: CLI entries feed the trainers; models and losses form the mathematical core; Ray actors and vLLM engines carry the distributed execution while a DeepSpeed strategy manages sharding.

Reading the overview from left to right:

• openrlhf/cli/train_ppo_ray.py is the 771-line PPO entry whose train at line 19 parses the deeply nested argument tree and assembles the actor group.
• openrlhf/datasets/prompts_dataset.py defines PromptDataset at line 48 with preprocessing at line 26 for chat templates and mixing probabilities.
• openrlhf/models/actor.py wraps the policy in Actor at line 54 with a ZeRO-3 weight-mapping patch at line 16.
• openrlhf/models/loss.py is the algorithm catalog: PolicyLoss at line 116, ValueLoss at line 262, PairWiseLoss at line 301, DPOLoss at line 329.
• openrlhf/trainer/ppo_trainer.py holds BasePPOTrainer at line 148 and PPOTrainer at line 450 of its 599 lines.
• openrlhf/trainer/ray/ppo_actor.py is the distributed training loop, ActorPPOTrainer at line 37 of 683 lines.
• openrlhf/trainer/ray/vllm_engine.py defines RolloutRayActor at line 34 and the fleet factory create_vllm_engines at line 209.
• openrlhf/utils/deepspeed/deepspeed.py implements DeepspeedStrategy at line 37, the 788-line configuration engine behind every model actor.

Why You Need This

The framework field splits into two camps: monoliths that own everything and thin wrappers that delegate everything. OpenRLHF takes a third path that is worth copying - the coordination problem is solved once, in one place, and the algorithms ride on top. Where TRL (the library unsloth patches) embeds RL into the Hugging Face Trainer, and LLaMA-Factory exposes DPO as one stage among many, OpenRLHF makes the distributed loop itself the product. Reading it clarifies what PPO actually requires at scale: synchronized weight broadcasts to inference engines, advantage estimation over batched experiences, and a KL leash to the reference model.

The weight-sync problem is the deepest lesson here. During RLHF, the policy changes every step, but the vLLM engines that generate rollouts hold their own copy of the weights. OpenRLHF solves this with explicit broadcast paths - training step finishes, weights flow to every rollout engine over NCCL, generation resumes - and the source shows the machinery without abstraction: initialization of the sync group, per-engine broadcast calls, and sleep modes that let training and inference share GPUs. Anyone who has wondered how hybrid engines work will find the answer written in plain PyTorch and Ray.

Finally, the algorithms are honest about being small. Once experiences are collected, PPO, REINFORCE++, GRPO, and RLOO differ by a few lines of advantage estimation and loss masking - the repo’s own notes explain how REINFORCE++-baseline outperformed GRPO in stability for reasoning models. That insight, that the algorithm is the easy part and the plumbing is the hard part, is the reason this codebase exists, and it pairs naturally with the evaluation tour of lm-evaluation-harness for scoring the resulting models, or with axolotl’s async GRPO trainers as a different point on the same design space. Even non-RL parts - the DeepSpeed integration, the sequence-length balancing - are useful references alongside deepspeed and colossalai.

Detailed view: CLI entries with datasets, the model and loss core, single-process trainers, the experience machinery, Ray actors and rollout engines, and infrastructure utilities.

Start where a run starts. train at line 19 of cli/train_ppo_ray.py builds a DeepspeedStrategy, loads PromptDataset from datasets/prompts_dataset.py, and creates a RayActorGroup at line 202 of trainer/ray/launcher.py - the same class that will later spawn reference, reward, and critic actors (BaseModelActor at line 54, ReferenceModelActor at line 105, RewardModelActor at line 158 of that file). The single-controller idea lives in this split: launcher.py knows how to place models on GPUs, while the trainer process knows the algorithm and calls the actors remotely.

The experience path is the heart. RemoteExperienceMaker at line 19 of trainer/ppo_utils/experience_maker.py sends prompts to the rollout engines, collects log probabilities from the policy and reference models, scores with the reward model, and packs everything into the Experience dataclass at line 29 of trainer/ppo_utils/experience.py - a 283-line module that also handles batching (make_experience_batch at line 199), padding removal at line 236, and sequence balancing at line 250. Experiences land in NaiveReplayBuffer at line 17 of trainer/ppo_utils/replay_buffer.py, and the KL leash is controlled by AdaptiveKLController at line 4 of trainer/ppo_utils/kl_controller.py - all 29 lines of it, because a controller only needs a target and an update rule.

The distributed loop then takes over in trainer/ray/ppo_actor.py. ActorPPOTrainer at line 37 initializes the vLLM sync group at line 108, runs the generation-training alternation in ppo_train at line 153, executes each gradient step in training_step at line 259, and pushes refreshed weights to every rollout engine through broadcast_to_vllm at line 409. The concrete worker, PolicyModelActor at line 494, wraps the Actor model with state offloading - offload_states at line 658 lets the policy sleep while another model uses its GPUs - and trainer/ray/vllm_engine.py mirrors the other side: RolloutRayActor at line 34 hosts a vLLM engine per Ray process, while create_vllm_engines at line 209 sizes the fleet from tensor-parallel and engine-count flags. The single-process cousin, PPOTrainer at line 450 of trainer/ppo_trainer.py after BasePPOTrainer at line 148, shares the same experience machinery for colocated setups, and the async variant in trainer/ppo_trainer_async.py overlaps generation with training.

Small helpers round out the loop with the same economy. prepare_datasets at line 26 of trainer/ppo_trainer.py splits train and evaluation prompt sources, compute_eval_metrics at line 82 runs evaluation rollouts between training phases, batch_vllm_engine_call at line 314 of trainer/ray/vllm_engine.py fans a method call out across the engine fleet while keeping only rank zero’s return value, and aggregate_loss at line 11 of models/loss.py merges policy, value, and KL terms into one weighted objective so a training step is a single backward call.

The math lives in two files. models/loss.py is the full algorithm catalog in 364 lines: SFTLoss at line 87 for supervised runs, PolicyLoss at line 116 implementing the PPO-clip objective with optional dual clipping, ValueLoss at line 262 for the critic, PairWiseLoss at line 301 and LogExpLoss at line 316 for reward modeling, and DPOLoss at line 329 for direct preference optimization. models/model.py builds the value-headed siblings of any causal LM through get_llm_for_sequence_regression at line 21, with the reward variant at _get_reward_model line 177 and the critic at _get_critic_model line 245 - the trick that lets any pretrained checkpoint become a reward model by appending a score head. The Actor wrapper at line 54 of models/actor.py handles generation-mode switching and packing-aware forward passes, with ring-attention support from models/ring_attn_utils.py for long contexts.

Around the core sit the supporting trainers and utilities. trainer/sft_trainer.py (SFTTrainer at line 13), trainer/dpo_trainer.py (DPOTrainer at line 12), and trainer/rm_trainer.py (RewardModelTrainer at line 12) cover the pre-RL stages of the alignment recipe, each with a matching CLI in cli/ and dataset in datasets/ - reward_dataset.py holds the chosen/rejected loader for RM training. Agent-based reinforcement learning gets utils/agent.py: AgentExecutorBase at line 13, MultiTurnAgentExecutor at line 32, and SingleTurnAgentExecutor at line 188 turn arbitrary reward functions into rollout sources, which is how the repo trains on verifiable math and code rewards. Infrastructure details that quietly decide performance get their own files too - DeepspeedStrategy at line 37 of utils/deepspeed/deepspeed.py owns ZeRO stages, packing, and sleep modes, while karmarkar_karp at line 24 of utils/seqlen_balancing.py partitions variable-length sequences across micro-batches so one 4k-token sample cannot stall a whole step.

From Install to an Aligned Model

The README’s recommended path runs in Docker with vLLM:

pip install openrlhf[vllm]
ray start --head --node-ip-address 0.0.0.0 --num-gpus 8

Then submit a PPO run with hybrid engine settings:

ray job submit --address="http://127.0.0.1:8265" \
  -- python3 -m openrlhf.cli.train_ppo_ray \
  --ref.num_nodes 1 --ref.num_gpus_per_node 8 \
  --actor.model_name_or_path OpenRLHF/Llama-3-8b-sft-mixture \
  --reward.model_name_or_path OpenRLHF/Llama-3-8b-rm-700k \
  --vllm.num_engines 4 --vllm.tensor_parallel_size 2 \
  --train.batch_size 128 --rollout.batch_size 1024 \
  --zero.stage 3 --adam.offload True

Algorithm variants are single flags: --algo.advantage.estimator group_norm becomes GRPO, reinforce becomes REINFORCE++, rloo becomes RLOO.

Honest limits: the framework assumes you can afford a Ray cluster and knows it - the README recommends Docker and multi-GPU nodes; vLLM version pinning matters because the weight-sync path touches engine internals; and the argument surface is enormous, since every strategy knob is a CLI flag. What you get in exchange is a complete, readable map of distributed RLHF that fits in an afternoon.

Enjoyed this post? Never miss out on future posts by following us