RLHF (Reinforcement Learning from Human Feedback)

RLHF (Reinforcement Learning from Human Feedback) is a technique for refining a pretrained language model with human judgements. People compare or rank model responses, those rankings train a reward model that predicts which answer people prefer, and the language model is then optimised with reinforcement learning to score higher on that reward.

Why it matters

Pretraining teaches a model to predict text, not to follow instructions or avoid harmful content. RLHF is one of the main steps that turns a raw model into a helpful assistant. It also has known side effects: models can become overly agreeable, verbose or evasive because those traits were rewarded. Newer methods, such as direct preference optimisation, pursue the same goal with simpler training.

Example

Prompt: "Explain DNS to a beginner"
Answer A: dense jargon        Answer B: clear analogy
Human picks B -> reward model learns the preference
-> policy is updated to produce more B-like answers

Common mistakes

• Treating RLHF as teaching facts. It mainly shapes style and behaviour.
• Assuming human raters are neutral. Their guidelines and biases leak into the model.
• Confusing it with fine-tuning on fixed examples, which has no reward model.

Related

• Fine-tuning
• AI alignment
• LLM
• Training data

यह पेज सामान्य जानकारी है, जिसे हमारी संपादकीय टीम ने लिखा है और दस्तावेज़ीकरण बदलने पर अपडेट किया जाता है। सर्वर और प्रदाता अलग-अलग होते हैं, इसलिए बदलाव पहले सुरक्षित तरीके से जाँचें। देखें: संपादकीय नीति और अस्वीकरण.