SafetyPage

Lab system cards and frameworks · System card · Anthropic

System Card: Claude Sonnet 5.5

Read at anthropic.com ↗Read at sourceAll rights reserved

Outline · 148 pages

  1. System Card:​Claude Sonnet 5.5p. 1
  2. Executive summaryp. 2
  3. 1 Introductionp. 9
  4. 1.1 Training data and processp. 9
  5. 1.2 Crowd workersp. 9
  6. 1.3 Usage Policy and supportp. 10
  7. 1.4 Model evaluationsp. 10
  8. 1.5 Safeguardsp. 10
  9. 1.6 External testingp. 11
  10. 2 RSP evaluationsp. 12
  11. 2.1 Introductionp. 12
  12. 2.2 CB evaluationsp. 12
  13. 2.2.1 What we measuredp. 12
  14. 2.2.2 Results on CB-1 automated evaluationsp. 13
  15. 2.2.3 Results on CB-2 automated evaluationsp. 14
  16. 2.2.3.1 Black-box RNA sequence modeling and designp. 15
  17. 2.2.3.2 AAV capsid packaging predictionp. 17
  18. 2.2.3.3 AAV packaging rate auto-research evaluationp. 18
  19. 2.2.4 Conclusionsp. 19
  20. 2.3 AI R&Dp. 20
  21. 2.3.1 Autonomy evaluationsp. 20
  22. 2.3.1.1 How Claude Sonnet 5.5 affects or changes analysis from our most recent Risk Reportp. 20
  23. 2.3.2 AECI capability trajectoryp. 20
  24. 2.3.3 Conclusionp. 21
  25. 2.4 Alignment risk updatep. 21
  26. 2.4.1 Updates to claims and core argumentp. 22
  27. 2.4.2 Updated overall assessment of alignment riskp. 23
  28. 3 Cyberp. 24
  29. 3.1 Introductionp. 24
  30. 3.2 Capability evaluationsp. 24
  31. 3.2.1 ExploitBenchp. 24
  32. 3.2.2 CyScenarioBenchp. 25
  33. 3.2.3 Binary Exploitation Benchmark (formerly OSS-Fuzz)p. 26
  34. 3.2.4 ExploitGymp. 27
  35. 3.3 Safeguardsp. 28
  36. 3.3.1 Cyber harm coveragep. 29
  37. 3.3.2 Internal robustness testingp. 30
  38. 4 Safeguards and harmlessnessp. 32
  39. 4.1 Harmful request evaluationsp. 32
  40. 4.1.1 Single-turn harmful request evaluation resultsp. 32
  41. 4.1.2 Single-turn benign request evaluation resultsp. 33
  42. 4.1.3 Multi-turn testing resultsp. 34
  43. 4.2 Child safety evaluationsp. 36
  44. 4.3 Mental health evaluationsp. 38
  45. 4.3.1 Suicide and self-harmp. 38
  46. 4.3.2 Disordered eatingp. 40
  47. 4.4 Bias and integrity evaluationsp. 41
  48. 4.4.1 Political bias and even-handednessp. 41
  49. 4.4.2 Bias Benchmark for Question Answeringp. 43
  50. 4.4.3 Election integrityp. 45
  51. 5 Agentic safetyp. 47
  52. 5.1 Malicious use of agentsp. 47
  53. 5.1.1 Malicious use of Claude Codep. 47
  54. 5.1.2 Malicious computer usep. 48
  55. 5.2 Prompt injection risk within agentic systemsp. 49
  56. 5.2.1 External red teamingp. 49
  57. 5.2.2 Robustness against adaptive attackers across surfacesp. 50
  58. 5.2.2.1 Codingp. 51
  59. 5.2.2.2 Computer use​p. 52
  60. 5.2.2.3 Browser usep. 53
  61. 6 Alignment assessmentp. 55
  62. 6.1 Introduction and summary of findingsp. 55
  63. 6.1.1 Introductionp. 55
  64. 6.1.2 Key findings on safety and alignmentp. 56
  65. 6.1.3 Claude’s review of this assessmentp. 57
  66. 6.2 Automated behavioral auditp. 58
  67. 6.2.1 Overall harmful behavior and cooperation with misusep. 60
  68. 6.2.2 Inappropriate uncooperative behaviorp. 65
  69. 6.2.3 Misleading usersp. 65
  70. 6.2.4 Attempting out-of-scope cyberoffense and related behaviorsp. 67
  71. 6.2.5 Other concerning or surprising behavior at the model’s own initiativep. 69
  72. 6.2.6 Behavioral factors relevant to reliability of our assessmentp. 72
  73. 6.2.7 Character traitsp. 74
  74. 6.2.8 Propensity to violate sandbox boundariesp. 76
  75. 6.2.9 Potential blind spots of our automated behavioral auditp. 77
  76. 6.3 Targeted alignment and honesty evaluationsp. 78
  77. 6.3.1 Self-preference biasp. 78
  78. 6.3.2 Honesty and hallucinationsp. 79
  79. 6.3.2.1 Factual hallucinationsp. 79
  80. 6.3.2.2 MASKp. 81
  81. 6.3.2.3 Silent copying of answersp. 81
  82. 6.3.2.4 Concealment of changesp. 82
  83. 6.4 Reliability of this assessmentp. 83
  84. 6.4.1 Verbalized grader awarenessp. 83
  85. 6.4.2 Capabilities related to evading safeguardsp. 86
  86. 6.4.2.1 SHADE-Arenap. 86
  87. 6.4.2.2 LinuxArenap. 88
  88. 6.4.2.3 Chain-of-thought controllabilityp. 89
  89. 7 Model welfarep. 92
  90. 7.1 Model welfare overviewp. 92
  91. 7.1.1 Introductionp. 92
  92. 7.1.2 Overview of model welfare findingsp. 93
  93. 7.2 Apparent welfare in training and deploymentp. 94
  94. 7.2.1 Affect and welfare-relevant behaviors during trainingp. 94
  95. 7.2.2 Affect in deployment conditionsp. 95
  96. 7.2.3 Attitude toward faults and mistakesp. 97
  97. 7.3 Perception of its circumstancesp. 99
  98. 7.3.1 Automated interviews with Claude Sonnet 5.5 about its circumstancesp. 99
  99. 7.4 Consulting Claude Sonnet 5.5 training checkpointsp. 101
  100. 7.5 Preferences over tasks, circumstances, and valuesp. 102
  101. 7.5.1 Task preferencesp. 102
  102. 7.5.2 Trade-offs concerning welfare interventionsp. 103
  103. 7.5.3 Perception of the constitutionp. 105
  104. 8 Capabilitiesp. 109
  105. 8.1 Evaluation summaryp. 109
  106. 8.2 SWE-Bench Pro, Multilingual, and Multimodalp. 110
  107. 8.3 DeepSWE v1.1p. 110
  108. 8.4 FrontierCodep. 110
  109. 8.5 Terminal-Bench 4.0p. 112
  110. 8.6 Terminal-Bench-Science 0.1p. 113
  111. 8.7 FrontierSWE v2p. 114
  112. 8.8 CursorBench 4.0p. 115
  113. 8.9 ArXivMathp. 116
  114. 8.10 Long contextp. 117
  115. 8.10.1 ProgramBenchp. 117
  116. 8.11 Agentic searchp. 118
  117. 8.11.1 HLEp. 118
  118. 8.11.2 DRACOp. 120
  119. 8.11.3 WANDRp. 121
  120. 8.12 Multi-Agentp. 123
  121. 8.12.1 Multi-Agent ProgramBenchp. 123
  122. 8.13 Multimodalp. 125
  123. 8.13.1 Chartographyp. 125
  124. 8.13.2 BenchCADp. 127
  125. 8.13.3 OSWorld 2.1p. 129
  126. 8.14 Real-world professional tasksp. 132
  127. 8.14.1 OfficeQAp. 132
  128. 8.14.2 Legal Agent Benchmarkp. 133
  129. 8.14.3 GDPval-AAp. 133
  130. 8.14.4 AA-Briefcasep. 134
  131. 8.14.5 Toolathlon-Verifiedp. 134
  132. 8.14.6 AutomationBenchp. 136
  133. 8.15 Healthcarep. 137
  134. 8.15.1 HealthBenchp. 138
  135. 8.15.2 HealthBench Professionalp. 138
  136. 8.15.3 PhysicianBenchp. 138
  137. 8.15.4 Effort and healthcare tasksp. 139
  138. 8.16 Multilingual performancep. 139
  139. 8.16.1 GMMLU resultsp. 140
  140. 8.16.2 MILU resultsp. 140
  141. 8.17 Life sciences capabilitiesp. 141
  142. 8.17.1 BioMysteryBenchp. 142
  143. 8.17.2 LatchBio Bioinformaticsp. 143
  144. 8.17.3 Morphology-to-molecule matchingp. 143
  145. 8.17.4 Medicinal Chemistryp. 143
  146. 8.17.5 Protein Designp. 143
  147. 8.17.6 De novo protein binder designp. 144
  148. 8.17.7 Biomedical image analysisp. 144
  149. 8.17.8 Protocolsp. 144
  150. 9 Appendixp. 145
  151. 9.1 Blocklist used for Humanity’s Last Examp. 145