SafetyPage

Research papers · Paper

RouterInterp: Understanding Superposed Specialisation in Mixture of Experts Routing

Ilya Lasy, Nora Yinuo Cai, Kola Ayonrinde · 8 Oct 2026

ReadSource at arxiv.org ↗Read hereCC BY 4.0

Outline · 33 pages

  1. Introductionp. 1
  2. Backgroundp. 3
  3. Sparse Autoencoders (SAE)p. 3
  4. Mixture of Experts (MoE)p. 3
  5. Specialisation Hypothesesp. 3
  6. Alternative Specialisation Hypothesesp. 4
  7. Motivation for Superposed Specialisationp. 5
  8. Empirical Evidence for Superposed Specialisationp. 5
  9. RouterInterpp. 6
  10. Methodp. 6
  11. Evaluationp. 7
  12. Experimental Setupp. 8
  13. Resultsp. 9
  14. Discussionp. 10
  15. Limitationsp. 11
  16. Related Workp. 11
  17. Conclusionp. 12
  18. Extended Related Workp. 20
  19. Sparse Feature Decompositions for Model Interpretabilityp. 20
  20. Adaptive Computationp. 21
  21. Modularity in Neural Networksp. 21
  22. SAE Features as Routing Predictorsp. 21
  23. Feature Selection Method and Set Sizep. 22
  24. Feature Selection Methodp. 22
  25. Selection Criterion and Set Sizep. 23
  26. Expert Routing Across Pile Domainsp. 25
  27. Routing Prediction Across Layersp. 26
  28. Routing Depends on Multiple Featuresp. 26
  29. SAE Setup and Reconstruction Qualityp. 27
  30. Human Evaluationp. 28
  31. SAE Probes Recover Router Probabilitiesp. 28
  32. Promptsp. 29
  33. Example Expert Explanationsp. 32