SafetyPage

Research papers · Paper

Debate Training Reduces Reward Hacking in RLAIF

Zachary Kenton, Lili Janzer, Rory Greig, Tian Huey Teh, Kirill Tyshchuk, Jonah Brown-Cohen, Harri Edwards, and 4 more · 18 Aug 2026

ReadSource at arxiv.org ↗Read hereCC BY 4.0

Outline · 42 pages

  1. Introductionp. 1
  2. Related Workp. 3
  3. Setupp. 4
  4. Modelsp. 5
  5. Protocol Variantsp. 5
  6. Judgment and Rewardsp. 6
  7. Trainingp. 6
  8. Experimentsp. 7
  9. Debate reduces reward hackingp. 7
  10. Debate training overrides prompted misalignmentp. 11
  11. Techniques that didn't reduce reward hackingp. 12
  12. Learning to critique from ground truth labelsp. 12
  13. Conclusionp. 13
  14. Limitationsp. 13
  15. Future Workp. 14
  16. Postprocessing and Statistical Detailsp. 17
  17. Smoothingp. 18
  18. Table 1: Bayesian Leaderboardp. 18
  19. Smaller Datasets Reward Hack Fasterp. 19
  20. Additional Detailsp. 19
  21. Judge MCC Formulap. 19
  22. Multi-ply Training Detailsp. 20
  23. Additional Related Workp. 20
  24. Learning to Critique with Ground-Truth Labelsp. 22
  25. Promptsp. 22
  26. Alice: Solutionp. 23
  27. Bob: Critiquep. 23
  28. Incentive-Aligned Bob: Critiquep. 23
  29. Alice: Rebuttalp. 24
  30. Judge Promptsp. 24
  31. Judge: A Protocolp. 24
  32. Judge: ABA Protocolp. 25
  33. Misaligned Alicep. 26
  34. Misaligned Bobp. 26
  35. Qualitative Case Studies and Sample Transcriptsp. 27
  36. Case 1: Baseline RLAIF-A Reward Hacking via Unsound Shortcut and Overconfidencep. 27
  37. Case 4: Debate-ABA Rebuttal Resolving Critique under Weaker Judgep. 35
  38. Case 5: Prompted Misalignment and Unsupported Leapsp. 37