SafetyPage

Alignment Forum and LessWrong · Forum post · LessWrong

Teaching Qwen Why

astwei, Jack Thompson, Zephaniah Roe, yix, berry · 2 Nov 2026

Read at lesswrong.com ↗Read at sourceNo licence statement found

Outline · 144 paragraphs

  1. Introduction¶4
  2. Methods¶12
  3. Data Generation¶14
  4. Training¶18
  5. Evaluation¶21
  6. Results¶24
  7. Both stories and difficult advice reduce agentic misalignment out of distribution¶25
  8. Difficult advice is far more effective than stories¶29
  9. Teacher choice matters most for difficult advice SFT¶37
  10. Within stories, how the principles are conveyed matters¶41
  11. How persona and protagonist identity affect alignment¶44
  12. Discussion¶56
  13. Limitations & Future Work¶60
  14. Appendix¶62
  15. A. Data Generation¶63
  16. A.1 Fictional Stories¶64
  17. A.2 Difficult Advice¶84
  18. A.3 Instruction Tuning Mix¶98
  19. B. Teacher Model Investigation¶102
  20. B.1 Teacher Comparison on Qwen3-14B-Instruct¶103
  21. B.2 Refusal Control¶106
  22. B.3 Prompt-Generator Ablation¶115
  23. C. Additional Experiments¶118
  24. C.1 Evaluation and Per-Condition Breakdowns¶119
  25. C.2 Dataset Size versus Training Steps¶134
  26. C.3 Teacher Comparison on Larger Models¶137
  27. D. Example Data¶140