Strategyproof Reinforcement Learning from Human Feedback

Synopsis

Shows existing RLHF is not strategyproof; even one strategic labeler can cause major misalignment. Proposes "Pessimistic Median of MLEs" achieving approximate strategyproofness.

Keywords

·strategyproof ·RLHF ·manipulation ·mechanism design

Related entries

• How RLHF Amplifies Sycophancy
  February 1, 2026 · arXiv
• Language Models Learn to Mislead Humans via RLHF
  September 19, 2024 · ICLR 2025
• Towards Understanding Sycophancy in Language Models
  October 20, 2023 · ICLR 2024
• Training Language Models to Follow Instructions with Human Feedback
  March 4, 2022 · NeurIPS 2022
• Learning to summarize from human feedback
  September 2, 2020 · NeurIPS 2020
• Deep Reinforcement Learning from Human Preferences
  June 12, 2017 · NeurIPS 2017