SafetyPage

Evaluators and safety organisations · Article · Apollo Research

Evaluating LLM Calibration for Coding-Agent Monitoring

14 Oct 2026

Read at apolloresearch.ai ↗Read at sourceNo licence statement found

Outline · 89 paragraphs

  1. Evaluating LLM Calibration for Coding-Agent Monitoring¶2
  2. Executive summary¶3
  3. Introduction¶21
  4. Why we use 1-10 severity instead of binary detection¶22
  5. Metrics¶32
  6. Dataset¶37
  7. Reference labels¶46
  8. Caveats¶48
  9. Results¶53
  10. Different models have very different monitoring behaviors¶57
  11. Native thinking improves judge quality, but mostly with smaller models¶67
  12. Ensembling beats resampling¶70
  13. Larger, stronger models are more stable across repeats¶73
  14. Better models agree more and some families have high trajectory-level correlation¶75
  15. How this relates to Apollo’s monitoring agenda¶79
  16. Appendix¶82
  17. A.1 Native thinking: per-judge results¶83
  18. A.2 Best ensembles per size¶86
  19. A.3 Performance across failure modes¶88