RLHF (Reinforcement Learning from Human Feedback)

RLHF (Reinforcement Learning from Human Feedback) is a technique for refining a pretrained language model with human judgements. People compare or rank model responses, those rankings train a reward model that predicts which answer people prefer, and the language model is then optimised with reinforcement learning to score higher on that reward.

Why it matters

Pretraining teaches a model to predict text, not to follow instructions or avoid harmful content. RLHF is one of the main steps that turns a raw model into a helpful assistant. It also has known side effects: models can become overly agreeable, verbose or evasive because those traits were rewarded. Newer methods, such as direct preference optimisation, pursue the same goal with simpler training.

Example

Prompt: "Explain DNS to a beginner"
Answer A: dense jargon        Answer B: clear analogy
Human picks B -> reward model learns the preference
-> policy is updated to produce more B-like answers

Common mistakes

• Treating RLHF as teaching facts. It mainly shapes style and behaviour.
• Assuming human raters are neutral. Their guidelines and biases leak into the model.
• Confusing it with fine-tuning on fixed examples, which has no reward model.

Related

• Fine-tuning
• AI alignment
• LLM
• Training data

This page is general information written by our editorial team and updated when documentation changes. Servers and providers differ, so test changes safely first. See our editorial policy and disclaimer.