SafetyPage

Research papers · Paper

Gradient Routing Localizes and Removes Unintended Behaviors in RL

Jake Ward, Shawn Hu, Aria Wong, Nathan Hu, Bryce Woodworth, Alexander Matt Turner, Alex Cloud · 12 Oct 2026

ReadSource at arxiv.org ↗Read hereCC BY 4.0

Outline · 32 pages

  1. Introductionp. 2
  2. Problem settingp. 3
  3. Methodp. 3
  4. GRAFT prevents monitor-exploiting policies in reinforcement learningp. 4
  5. Qwen3-8B in Countdown-Codep. 4
  6. SmolLM2-135M in six toy environmentsp. 6
  7. GRAFT suppresses unintended behavior in supervised fine-tuningp. 7
  8. Related workp. 9
  9. Discussionp. 9
  10. Conclusionp. 10
  11. Code availabilityp. 14
  12. More algorithm detailsp. 14
  13. Sample Countdown-Code promptp. 14
  14. Additional resultsp. 15
  15. Generalization of hacking in the FORGET adapter of the Qwen3-8B model trained on Countdownp. 15
  16. Applying GRAFT to Countdown-Code without the anchor environmentp. 18
  17. Countdown-Code with 50% hackable promptsp. 18
  18. A more pathological variation of the ``sort'' environmentp. 19
  19. Small-scale environments with a form of anchor distributionp. 20
  20. Full training graphsp. 20
  21. Monitor recallp. 21
  22. Model selection by metrics available to model developersp. 26
  23. Sample inoculation promptsp. 26
  24. Environment detailsp. 27
  25. Small-scale environmentsp. 27
  26. Hyperparameters and implementation detailsp. 29
  27. Software stackp. 29
  28. RL hyperparameters for SmolLM2-135Mp. 29
  29. RL hyperparameters for Qwen3-8Bp. 30
  30. Large-scale SFTp. 30
  31. Large-scale SFT evaluationp. 30
  32. Computep. 32