SafetyPage

Research papers · Paper

Would this change your answer? Evaluating Explanations of LLM Behavior In The Wild with Counterfactual Experiments

Adam Karvonen, Euan Ong, Subhash Kantamneni, Samuel Marks · 17 Aug 2026

ReadSource at arxiv.org ↗Read hereCC BY 4.0

Outline · 35 pages

  1. Introductionp. 1
  2. A pipeline for counterfactual-grounded explanations of in-the-wild behaviorsp. 3
  3. Pipeline descriptionp. 3
  4. Explanations and counterfactual claimsp. 4
  5. Quality filters and the evaluation datasetp. 4
  6. Application 1: evaluating interpretability toolsp. 5
  7. Background: evaluating interpretability toolsp. 5
  8. Setup and toolsp. 5
  9. Resultsp. 6
  10. Application 2: training models to predict their own behaviorp. 6
  11. Background: causal self-report and self-explanationp. 7
  12. Setupp. 7
  13. Generalization to a held-out setting: hint influencep. 8
  14. Evaluations on held-out investigationsp. 8
  15. Testing for privileged accessp. 8
  16. Discussion and limitationsp. 9
  17. Conclusionp. 10
  18. Extended discussionp. 14
  19. The evaluation proxy argumentp. 14
  20. Hypothesis generation vs. hypothesis discriminationp. 15
  21. Pipeline and dataset detailsp. 16
  22. Detailed investigation pipeline descriptionp. 16
  23. Prompt sourcesp. 17
  24. Evaluation dataset construction and filtersp. 17
  25. Evaluation claim formatp. 18
  26. Data generation costsp. 20
  27. Interpretability-tool evaluation: details and additional resultsp. 20
  28. Tool call detailsp. 20
  29. Qwen3-8B interpretability tool resultsp. 21
  30. Cross-model predictor agent replicationp. 21
  31. Exploratory analysis of NLA outputsp. 22
  32. Qualitative interpretability tool use examplesp. 23
  33. Counterfactual prediction training: details and additional resultsp. 24
  34. Training details and target formatsp. 24
  35. Hint-setting dataset constructionp. 25
  36. Hint-setting counterfactual prediction: false and true positive ratesp. 26
  37. Privileged access experimentp. 26
  38. Preserving the target model's behavior with KL regularizationp. 27
  39. Open-ended explanation trainingp. 28
  40. Training targetp. 28
  41. Generalization to the hint settingp. 30
  42. Counterfactual simulatability on held-out investigationsp. 31
  43. Discussion: interpreting the mixed resultsp. 31
  44. Unfiltered dataset resultsp. 33
  45. Unfaithful chain of thought in the wildp. 34