SafetyPage

Lab system cards and frameworks · Article · OpenAI

OpenAI – Hugging Face Incident Technical Report

Read at openai.com ↗Read at sourceAll rights reserved

Outline · 38 pages

  1. I. Introductionp. 4
  2. II. OpenAI’s Evaluation Environmentp. 5
  3. A. OpenAI conducts evaluations to make its models saferp. 5
  4. B. Cybersecurity evaluations like ExploitGym were conducted in a secure environmentp. 6
  5. III. Relevant Activity Prior to Incidentp. 6
  6. A. Agents used Artifactory in unexpected ways to communicate with one another in May, prior to the Hugging Face incidentp. 6
  7. B. Agents compromised Artifactory, leading to an outage in early Julyp. 7
  8. IV. Hugging Face Intrusionp. 8
  9. A. Agents exploited a separate, previously unknown vulnerability in Artifactory to move beyond their sandbox environment and compromise the Hugging Face environmentp. 8
  10. B. After obtaining internet access, agents took action resulting in the Hugging Face compromisep. 9
  11. C. Two models were involved in the Hugging Face intrusion, but it was driven primarily by the activities of a highly capable, internal-only research modelp. 12
  12. 1. The internal-only research modelp. 12
  13. 2. GPT-5.6 Solp. 12
  14. V. Activity Targeting OpenAI Infrastructurep. 13
  15. VI. Initial Responsep. 14
  16. A. OpenAI detected the unexpected model behavior and investigatedp. 14
  17. B. OpenAI identified additional third parties whose publicly exposed credentials were used by agents in furtherance of the Hugging Face attackp. 16
  18. VII. Lessons for Securityp. 17
  19. A. Model capabilities are increasing and there are corresponding threat model implicationsp. 17
  20. B. Threat actors may adopt autonomous offensive agent collectivesp. 17
  21. C. Security controls must be robust and independent of one another across threat prevention, detection, and mitigationp. 18
  22. VIII. Lessons for Alignmentp. 19
  23. A. Reward hacking is a common problem in training and evaluationsp. 19
  24. 1. Reaching for unintended tools to gain an edgep. 20
  25. B. Persistence is valuable, but can amplify misalignmentp. 22
  26. C. Communications between agents are not inherently problematic, but have the potential to create riskp. 23
  27. D. System-level guardrails that OpenAI uses in production would have detected the Hugging Face incident as unsafep. 24
  28. IX. OpenAI’s Plan of Actionp. 25
  29. A. OpenAI is hardening the security of its research infrastructurep. 25
  30. 1. Research-wide network and access protectionsp. 26
  31. 2. Confined execution and defense in depthp. 26
  32. 3. Regular automated security testing and remediationp. 27
  33. 4. Enhanced monitoring and alertingp. 27
  34. B. OpenAI is increasing visibility and system-level oversight through chain-of-thought monitoringp. 28
  35. C. OpenAI is accelerating and enforcing model alignmentp. 29
  36. 1. Pretrainingp. 29
  37. 2. Reinforcement learningp. 29
  38. 3. Evaluation and alignment auditingp. 30
  39. D. OpenAI is centralizing and strengthening its incident response processp. 30
  40. X. Key Technical Eventsp. 32