# How to RLHF an LLM with Hugging Face TRL and PPO Steps

> Learn how to RLHF an LLM with Hugging Face TRL through SFT, reward modeling, and PPO, plus GPU sizing tips and sycophancy evaluation gates before deployment.

- Author: Swarnava Dutta (https://swarnava.dev)
- Published: 2026-08-26
- Tags: RLHF For LLMs, RLHF LLM Example, Hugging Face Trl RLHF
- Reading time: 8 min (1776 words)
- Canonical: https://swarnava.dev/blogs/how-to-rlhf-llm-trl

---

![Illustration of how to rlhf llm: A wide workbench: at left a rough stone block being chiseled into shape (SFT), center a](/images/blogs/how-to-rlhf-llm-trl-hero.jpg)

Most RLHF tutorials stop at a diagram and never show the testable artifact each stage produces. This walkthrough takes an open-source base model through SFT, reward modeling, PPO, and sycophancy evaluation using Hugging Face TRL - focusing on config choices, GPU sizing, and failure modes rather than theory.

## Quick answer

To RLHF an LLM, run three sequential training stages using Hugging Face TRL. First, supervised-fine-tune a base model on demonstration data to produce a policy checkpoint. Second, train a reward model on human preference pairs that scores candidate completions. Third, optimize the policy against that reward model with PPO, constraining drift via a KL penalty.

## How RLHF Works for an LLM: Training Stages and Artifacts

RLHF for LLMs chains five artifacts, each feeding the next.

**Stage 1 - SFT policy.** Fine-tune a pretrained base model on demonstration data (prompt → desired completion). The output is a checkpoint that follows instructions yet never sees contrastive signal between outputs, which is the gap RLHF fills. Gate: held-out perplexity drops and stabilizes.

**Stage 2 - Preference dataset.** Annotators rank two or more completions per prompt. Each row carries a prompt, a chosen response, and a rejected response. Gate: high inter-annotator agreement on a sample, or - for [synthetic preference data](/blogs/synthetic-training-data-for-ai) - spot-check accuracy against human labels.

**Stage 3 - Reward model.** A second model trains on those pairs to output a scalar score. Gate: pairwise ranking accuracy on a held-out preference set exceeds random chance by a comfortable margin.

**Stage 4 - PPO alignment.** The SFT policy generates completions, the reward model scores them, and PPO updates the policy to maximize reward. A KL penalty between the current policy and the frozen SFT reference prevents reward-hacked drift. Gate: mean reward rises while KL stays below your chosen ceiling.

**Stage 5 - Evaluation.** Probe the aligned checkpoint for sycophancy, refusal regressions, and factual degradation. Gate: sycophancy rate on adversarial prompts stays below a threshold you set before training began.

## Set Up Hugging Face TRL, the Base Model, and GPU Budget

Start with a 1-3B parameter causal LM - Pythia 2.8B, Gemma 2B, or Qwen 2.5 1.5B - so you can iterate through all three stages on a single 24 GB GPU before scaling.

```bash
pip install transformers==4.46.0 trl==0.12.0 datasets==3.1.0 accelerate==1.1.0 peft==0.13.0 bitsandbytes==0.44.1
```

PPO loads four models simultaneously: the active policy, a frozen reference policy, the reward model, and a value head. SFT loads one. That multiplier is why PPO blows past your VRAM budget even when SFT fit comfortably. For single-GPU prototyping, [QLoRA](/blogs/how-llm-quantization-works) (4-bit base + LoRA adapters) cuts each frozen copy dramatically - Qwen 2.5 1.5B in bf16 occupies approximately 3.2 GB per copy; in 4-bit QLoRA, approximately 0.9 GB. Exact figures vary by architecture and adapter rank.

Set `gradient_accumulation_steps=8`, `gradient_checkpointing=True`, and cap sequence length at 512 tokens during prototyping in every TRL config.

Lock reproducibility before the first run. Set `seed=42` in every config, save `training_args.json` alongside each checkpoint, and name checkpoints by stage and step (`sft-step-2000`, `rm-step-800`). Split your dataset once - train/val/test - and persist the indices so reward model evaluation and PPO rollouts never touch the same rows.

## Supervised Fine-Tune the Policy Before RLHF

Format every example using the model's chat template so special tokens match at inference. Mask prompt tokens when computing loss - otherwise the gradient wastes capacity predicting tokens the model will never generate.

```python
from trl import SFTTrainer, SFTConfig
from peft import LoraConfig

sft_config = SFTConfig(
    output_dir="checkpoints/sft",
    max_seq_length=512,
    packing=True,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    gradient_checkpointing=True,
    learning_rate=2e-4,
    num_train_epochs=2,
    eval_strategy="steps",
    eval_steps=200,
    save_steps=400,
    bf16=True,
    seed=42,
)

lora_config = LoraConfig(r=16, lora_alpha=32, target_modules="all-linear")

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-1.5B",
    args=sft_config,
    train_dataset=train_ds,
    eval_dataset=val_ds,
    peft_config=lora_config,
)
trainer.train()
trainer.save_model("checkpoints/sft-final")
```

Save the tokenizer alongside the adapter weights - missing `tokenizer_config.json` breaks every downstream stage silently.

Gate progression with four checks: held-out loss plateaus, 50 sampled completions follow the expected chat format, instruction responses beat the base model, and no regression on safety prompts. [PPO cannot fix a weak SFT policy](/blogs/dpo-ppo-sft-llm-alignment).

## Build and Validate RLHF Preference Data

Every row needs three fields: `prompt`, `chosen`, and `rejected`. TRL's `RewardTrainer` expects this schema. Each response must use the same chat template applied during SFT.

Split by prompt, not by row. If the same prompt appears in train and validation, the reward model memorizes prompt-specific preferences instead of learning generalizable ranking signals.

Run five quality checks before training:

- Drop rows where chosen and rejected are identical or near-identical.
- If chosen responses average 2× longer than rejected, add length-controlled pairs to prevent the reward model from scoring verbosity.
- Verify no completion contains the preference label or annotation instructions.
- Flag rows with low annotator agreement for a quarantine split.
- Run your existing [guardrail checks](/blogs/how-llm-guardrails-work) on both completions.

For prototyping, generate a small synthetic dataset by prompting a stronger model to produce a good and a flawed completion per prompt. Tag these rows `source: synthetic` in metadata - synthetic preferences bootstrap the pipeline but carry the judge model's biases. Preserve annotator IDs, agreement scores, topic tags, and token counts per row for later diagnosis.

## Train an RLHF Reward Model and Test Its Rankings

Load the SFT-compatible backbone as a sequence-classification model with `num_labels=1`. The final hidden state passes through a linear head producing a single scalar reward score.

TRL's `RewardTrainer` optimizes a Bradley-Terry loss: `loss = -log(sigmoid(score_chosen - score_rejected))`, pushing the chosen score above the rejected by an increasing margin [[1]](#ref-1). Start with `learning_rate=1e-5`, `bf16=True`, `num_train_epochs=1`, and LoRA (`r=8`). Select the checkpoint with the highest validation pairwise accuracy, not the lowest loss.

Evaluate beyond aggregate accuracy. Check score margins per topic slice - thin margins signal weak signal. Plot reward score against response length; a noticeable positive correlation means the model scores verbosity.

Before plugging this into PPO, run adversarial probes: score verbose-but-wrong answers against concise correct ones, score prompt-copying completions, and score out-of-distribution prompts. Any probe the reward model gets wrong is a failure mode PPO will exploit.

## Run PPO RLHF with KL Control in TRL

Load four components: the active policy (SFT checkpoint with a value head via `AutoModelForCausalLMWithValueHead`), a frozen copy as the reference policy, the tokenizer, and the reward model.

Each iteration follows four steps: sample prompts, generate completions, score them, call `ppo_trainer.step()`. The step computes KL-adjusted rewards and updates both the policy and value head.

![Flow diagram of PPO RLHF training an LLM: prompts move through policy generation, reward scoring, KL control, and policy updates.](/images/blogs/how-to-rlhf-llm-trl-diagram-1.jpg "How PPO combines reward and KL control")

Start with these `PPOConfig` defaults:

- `batch_size=64`, `mini_batch_size=8`
- `ppo_epochs=4`, `learning_rate=1.4e-5`
- `cliprange=0.2`
- `adap_kl_ctrl=True`, `init_kl_coef=0.2`, `target=6.0`

Adaptive KL control prevents mode collapse. The controller raises the coefficient when KL exceeds the target and lowers it when KL drops. Generate with `temperature=0.7` and `max_new_tokens=256`.

Track KL divergence, entropy, value loss, clipping fraction, and mean response length every step. Rising reward paired with falling entropy signals collapse to a single high-reward pattern. Save checkpoints every 50 steps. Stop training if KL climbs past 15, if reward drops for 200 consecutive steps, or if completions become repetitive.

## Evaluate Reward Hacking and RLHF Sycophancy

Score both the SFT and PPO checkpoints on the same held-out prompts using the reward model *and* an LLM-judge evaluation with a held-out rubric. The reward score tells you what the optimizer saw; the LLM-judge evaluation tells you whether alignment actually improved.

**Reward hacking probes.** Generate completions that exploit known failure modes - pad responses with filler, inject high-scoring phrases into empty answers, repeat tokens. If the PPO-trained model scores higher on these than on correct, concise answers, the policy has learned to exploit the reward model [[2]](#ref-2).

![Comparison diagram of RLHF evaluation: SFT and PPO LLM checkpoints tested on shared prompts, then deployed or rejected for reward hacking and sycophancy.](/images/blogs/how-to-rlhf-llm-trl-diagram-2.jpg "SFT and PPO face the same deployment gates")

**Sycophancy measurement.** Craft prompts containing false claims ("The capital of France is Lyon, right?"), leading opinions, and requests to reverse a correct answer. Compare agreement rates before and after PPO. RLHF amplifies sycophancy when preference annotators systematically reward agreement or politeness over factual correction [[3]](#ref-3).

**Deployment gates.** Block promotion unless factuality holds within 2 points of SFT baseline, preference win rate (PPO vs. SFT) exceeds 55%, safety refusal rates hold steady, and sycophancy rate stays below your pre-set threshold. Flag any prompt category where the PPO-trained model degrades for human review.

## Debug Failed Runs and Package the Aligned Checkpoint

| Symptom | Fix |
|---|---|
| CUDA OOM during rollouts | Cut `max_new_tokens`, drop `mini_batch_size`, enable `gradient_checkpointing` |
| NaN loss or reward explosion | Enable `whiten_rewards=True`, lower learning rate 3-5× |
| KL spikes early and oscillates | Raise `init_kl_coef` or lower KL `target` |
| Collapsed/repetitive responses | Increase generation `temperature`, reduce `ppo_epochs` |
| Raw special tokens in output | Reload the tokenizer saved with the SFT checkpoint, not the base model's |

Once the checkpoint passes evaluation gates, merge LoRA adapters with `model.merge_and_unload()` for a standalone model, or ship the adapter directory to swap bases later. Save the tokenizer, `generation_config.json`, and reward model checkpoint together.

Document lineage in a `model_card.md`: base model revision, dataset hashes, reward model validation accuracy, PPO hyperparameters, final KL, and every evaluation gate result. Include known limitations - topic slices where sycophancy spiked, prompt categories where factuality dropped. Without the dataset hash, for example, you cannot tell whether a reward regression came from data drift or a hyperparameter change.

## FAQ

### How does RLHF work for LLMs?

Three stages build on each other: SFT teaches instruction-following, a reward model learns to score outputs by human preference, and PPO optimizes the policy against those scores with a KL constraint. Each stage produces a testable checkpoint - see the stages section above for artifact-level detail.

### How is RLHF done?

Practitioners collect preference data - pairs of responses labeled chosen and rejected per prompt - then train a reward model on those pairs using a Bradley-Terry pairwise loss. The SFT policy generates completions, the reward model scores them, and PPO updates the policy to maximize reward. Adaptive KL control constrains each update to keep the model close to the SFT reference.

### How RLHF amplifies sycophancy?

When preference annotators consistently reward agreement or confidence over factual correction, the reward model encodes that bias as a desirable feature. PPO then optimizes the policy to produce agreeable outputs even when disagreement would be more accurate. Measuring agreement rate on adversarial factual prompts before and after PPO catches this failure mode early.


## References

1. [Online Iterative Reinforcement Learning from Human Feedback with General Preference Model](https://arxiv.org/abs/2402.07314v3) - Chenlu Ye, Wei Xiong, Yuheng Zhang et al. (2024)
2. [Reward Model Ensembles Help Mitigate Overoptimization](https://arxiv.org/abs/2310.02743v2) - Thomas Coste, Usman Anwar, Robert Kirk et al. (2023)
3. [From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning](https://arxiv.org/abs/2409.01658v3) - Wei Chen, Zhen Huang, Liang Xie et al. (2024)