How to RLHF an LLM with Hugging Face TRL and PPO Steps

Most RLHF tutorials stop at a diagram and never show the testable artifact each stage produces. This walkthrough takes an open-source base model through SFT, reward modeling, PPO, and sycophancy evaluation using Hugging Face TRL - focusing on config choices, GPU sizing, and failure modes rather than theory.

Quick answer

To RLHF an LLM, run three sequential training stages using Hugging Face TRL. First, supervised-fine-tune a base model on demonstration data to produce a policy checkpoint. Second, train a reward model on human preference pairs that scores candidate completions. Third, optimize the policy against that reward model with PPO, constraining drift via a KL penalty.

How RLHF Works for an LLM: Training Stages and Artifacts

RLHF for LLMs chains five artifacts, each feeding the next.

Stage 1 - SFT policy. Fine-tune a pretrained base model on demonstration data (prompt → desired completion). The output is a checkpoint that follows instructions yet never sees contrastive signal between outputs, which is the gap RLHF fills. Gate: held-out perplexity drops and stabilizes.

Stage 2 - Preference dataset. Annotators rank two or more completions per prompt. Each row carries a prompt, a chosen response, and a rejected response. Gate: high inter-annotator agreement on a sample, or - for synthetic preference data - spot-check accuracy against human labels.

Stage 3 - Reward model. A second model trains on those pairs to output a scalar score. Gate: pairwise ranking accuracy on a held-out preference set exceeds random chance by a comfortable margin.

Stage 4 - PPO alignment. The SFT policy generates completions, the reward model scores them, and PPO updates the policy to maximize reward. A KL penalty between the current policy and the frozen SFT reference prevents reward-hacked drift. Gate: mean reward rises while KL stays below your chosen ceiling.

Stage 5 - Evaluation. Probe the aligned checkpoint for sycophancy, refusal regressions, and factual degradation. Gate: sycophancy rate on adversarial prompts stays below a threshold you set before training began.

Set Up Hugging Face TRL, the Base Model, and GPU Budget

Start with a 1-3B parameter causal LM - Pythia 2.8B, Gemma 2B, or Qwen 2.5 1.5B - so you can iterate through all three stages on a single 24 GB GPU before scaling.

pip install transformers==4.46.0 trl==0.12.0 datasets==3.1.0 accelerate==1.1.0 peft==0.13.0 bitsandbytes==0.44.1

PPO loads four models simultaneously: the active policy, a frozen reference policy, the reward model, and a value head. SFT loads one. That multiplier is why PPO blows past your VRAM budget even when SFT fit comfortably. For single-GPU prototyping, QLoRA (4-bit base + LoRA adapters) cuts each frozen copy dramatically - Qwen 2.5 1.5B in bf16 occupies approximately 3.2 GB per copy; in 4-bit QLoRA, approximately 0.9 GB. Exact figures vary by architecture and adapter rank.

Set gradient_accumulation_steps=8, gradient_checkpointing=True, and cap sequence length at 512 tokens during prototyping in every TRL config.

Lock reproducibility before the first run. Set seed=42 in every config, save training_args.json alongside each checkpoint, and name checkpoints by stage and step (sft-step-2000, rm-step-800). Split your dataset once - train/val/test - and persist the indices so reward model evaluation and PPO rollouts never touch the same rows.

Supervised Fine-Tune the Policy Before RLHF

Format every example using the model's chat template so special tokens match at inference. Mask prompt tokens when computing loss - otherwise the gradient wastes capacity predicting tokens the model will never generate.

from trl import SFTTrainer, SFTConfig
from peft import LoraConfig

sft_config = SFTConfig(
    output_dir="checkpoints/sft",
    max_seq_length=512,
    packing=True,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    gradient_checkpointing=True,
    learning_rate=2e-4,
    num_train_epochs=2,
    eval_strategy="steps",
    eval_steps=200,
    save_steps=400,
    bf16=True,
    seed=42,
)

lora_config = LoraConfig(r=16, lora_alpha=32, target_modules="all-linear")

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-1.5B",
    args=sft_config,
    train_dataset=train_ds,
    eval_dataset=val_ds,
    peft_config=lora_config,
)
trainer.train()
trainer.save_model("checkpoints/sft-final")

Save the tokenizer alongside the adapter weights - missing tokenizer_config.json breaks every downstream stage silently.

Gate progression with four checks: held-out loss plateaus, 50 sampled completions follow the expected chat format, instruction responses beat the base model, and no regression on safety prompts. PPO cannot fix a weak SFT policy.

Build and Validate RLHF Preference Data

Every row needs three fields: prompt, chosen, and rejected. TRL's RewardTrainer expects this schema. Each response must use the same chat template applied during SFT.

Split by prompt, not by row. If the same prompt appears in train and validation, the reward model memorizes prompt-specific preferences instead of learning generalizable ranking signals.

Run five quality checks before training:

• Drop rows where chosen and rejected are identical or near-identical.
• If chosen responses average 2× longer than rejected, add length-controlled pairs to prevent the reward model from scoring verbosity.
• Verify no completion contains the preference label or annotation instructions.
• Flag rows with low annotator agreement for a quarantine split.
• Run your existing guardrail checks on both completions.

For prototyping, generate a small synthetic dataset by prompting a stronger model to produce a good and a flawed completion per prompt. Tag these rows source: synthetic in metadata - synthetic preferences bootstrap the pipeline but carry the judge model's biases. Preserve annotator IDs, agreement scores, topic tags, and token counts per row for later diagnosis.

Train an RLHF Reward Model and Test Its Rankings

Load the SFT-compatible backbone as a sequence-classification model with num_labels=1. The final hidden state passes through a linear head producing a single scalar reward score.

TRL's RewardTrainer optimizes a Bradley-Terry loss: loss = -log(sigmoid(score_chosen - score_rejected)), pushing the chosen score above the rejected by an increasing margin [1]. Start with learning_rate=1e-5, bf16=True, num_train_epochs=1, and LoRA (r=8). Select the checkpoint with the highest validation pairwise accuracy, not the lowest loss.

Evaluate beyond aggregate accuracy. Check score margins per topic slice - thin margins signal weak signal. Plot reward score against response length; a noticeable positive correlation means the model scores verbosity.

Before plugging this into PPO, run adversarial probes: score verbose-but-wrong answers against concise correct ones, score prompt-copying completions, and score out-of-distribution prompts. Any probe the reward model gets wrong is a failure mode PPO will exploit.

Run PPO RLHF with KL Control in TRL

Load four components: the active policy (SFT checkpoint with a value head via AutoModelForCausalLMWithValueHead), a frozen copy as the reference policy, the tokenizer, and the reward model.

Each iteration follows four steps: sample prompts, generate completions, score them, call ppo_trainer.step(). The step computes KL-adjusted rewards and updates both the policy and value head.

Start with these PPOConfig defaults:

• batch_size=64, mini_batch_size=8
• ppo_epochs=4, learning_rate=1.4e-5
• cliprange=0.2
• adap_kl_ctrl=True, init_kl_coef=0.2, target=6.0

Adaptive KL control prevents mode collapse. The controller raises the coefficient when KL exceeds the target and lowers it when KL drops. Generate with temperature=0.7 and max_new_tokens=256.

Track KL divergence, entropy, value loss, clipping fraction, and mean response length every step. Rising reward paired with falling entropy signals collapse to a single high-reward pattern. Save checkpoints every 50 steps. Stop training if KL climbs past 15, if reward drops for 200 consecutive steps, or if completions become repetitive.

Evaluate Reward Hacking and RLHF Sycophancy

Score both the SFT and PPO checkpoints on the same held-out prompts using the reward model and an LLM-judge evaluation with a held-out rubric. The reward score tells you what the optimizer saw; the LLM-judge evaluation tells you whether alignment actually improved.

Reward hacking probes. Generate completions that exploit known failure modes - pad responses with filler, inject high-scoring phrases into empty answers, repeat tokens. If the PPO-trained model scores higher on these than on correct, concise answers, the policy has learned to exploit the reward model [2].

Sycophancy measurement. Craft prompts containing false claims ("The capital of France is Lyon, right?"), leading opinions, and requests to reverse a correct answer. Compare agreement rates before and after PPO. RLHF amplifies sycophancy when preference annotators systematically reward agreement or politeness over factual correction [3].

Deployment gates. Block promotion unless factuality holds within 2 points of SFT baseline, preference win rate (PPO vs. SFT) exceeds 55%, safety refusal rates hold steady, and sycophancy rate stays below your pre-set threshold. Flag any prompt category where the PPO-trained model degrades for human review.

Debug Failed Runs and Package the Aligned Checkpoint

Symptom	Fix
CUDA OOM during rollouts	Cut max_new_tokens, drop mini_batch_size, enable gradient_checkpointing
NaN loss or reward explosion	Enable whiten_rewards=True, lower learning rate 3-5×
KL spikes early and oscillates	Raise init_kl_coef or lower KL target
Collapsed/repetitive responses	Increase generation temperature, reduce ppo_epochs
Raw special tokens in output	Reload the tokenizer saved with the SFT checkpoint, not the base model's

Once the checkpoint passes evaluation gates, merge LoRA adapters with model.merge_and_unload() for a standalone model, or ship the adapter directory to swap bases later. Save the tokenizer, generation_config.json, and reward model checkpoint together.

Document lineage in a model_card.md: base model revision, dataset hashes, reward model validation accuracy, PPO hyperparameters, final KL, and every evaluation gate result. Include known limitations - topic slices where sycophancy spiked, prompt categories where factuality dropped. Without the dataset hash, for example, you cannot tell whether a reward regression came from data drift or a hyperparameter change.

FAQ

How does RLHF work for LLMs?

Three stages build on each other: SFT teaches instruction-following, a reward model learns to score outputs by human preference, and PPO optimizes the policy against those scores with a KL constraint. Each stage produces a testable checkpoint - see the stages section above for artifact-level detail.

How is RLHF done?

Practitioners collect preference data - pairs of responses labeled chosen and rejected per prompt - then train a reward model on those pairs using a Bradley-Terry pairwise loss. The SFT policy generates completions, the reward model scores them, and PPO updates the policy to maximize reward. Adaptive KL control constrains each update to keep the model close to the SFT reference.

How RLHF amplifies sycophancy?

When preference annotators consistently reward agreement or confidence over factual correction, the reward model encodes that bias as a desirable feature. PPO then optimizes the policy to produce agreeable outputs even when disagreement would be more accurate. Measuring agreement rate on adversarial factual prompts before and after PPO catches this failure mode early.

References

1. Online Iterative Reinforcement Learning from Human Feedback with General Preference Model - Chenlu Ye, Wei Xiong, Yuheng Zhang et al. (2024)
2. Reward Model Ensembles Help Mitigate Overoptimization - Thomas Coste, Usman Anwar, Robert Kirk et al. (2023)
3. From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning - Wei Chen, Zhen Huang, Liang Xie et al. (2024)