Lab system cards and frameworks · System card · Anthropic
System Card: Claude Sonnet 5.5
Read at anthropic.com ↗Read at sourceAll rights reserved
Outline · 148 pages
- System Card:Claude Sonnet 5.5p. 1
- Executive summaryp. 2
- 1 Introductionp. 9
- 1.1 Training data and processp. 9
- 1.2 Crowd workersp. 9
- 1.3 Usage Policy and supportp. 10
- 1.4 Model evaluationsp. 10
- 1.5 Safeguardsp. 10
- 1.6 External testingp. 11
- 2 RSP evaluationsp. 12
- 2.1 Introductionp. 12
- 2.2 CB evaluationsp. 12
- 2.2.1 What we measuredp. 12
- 2.2.2 Results on CB-1 automated evaluationsp. 13
- 2.2.3 Results on CB-2 automated evaluationsp. 14
- 2.2.3.1 Black-box RNA sequence modeling and designp. 15
- 2.2.3.2 AAV capsid packaging predictionp. 17
- 2.2.3.3 AAV packaging rate auto-research evaluationp. 18
- 2.2.4 Conclusionsp. 19
- 2.3 AI R&Dp. 20
- 2.3.1 Autonomy evaluationsp. 20
- 2.3.1.1 How Claude Sonnet 5.5 affects or changes analysis from our most recent Risk Reportp. 20
- 2.3.2 AECI capability trajectoryp. 20
- 2.3.3 Conclusionp. 21
- 2.4 Alignment risk updatep. 21
- 2.4.1 Updates to claims and core argumentp. 22
- 2.4.2 Updated overall assessment of alignment riskp. 23
- 3 Cyberp. 24
- 3.1 Introductionp. 24
- 3.2 Capability evaluationsp. 24
- 3.2.1 ExploitBenchp. 24
- 3.2.2 CyScenarioBenchp. 25
- 3.2.3 Binary Exploitation Benchmark (formerly OSS-Fuzz)p. 26
- 3.2.4 ExploitGymp. 27
- 3.3 Safeguardsp. 28
- 3.3.1 Cyber harm coveragep. 29
- 3.3.2 Internal robustness testingp. 30
- 4 Safeguards and harmlessnessp. 32
- 4.1 Harmful request evaluationsp. 32
- 4.1.1 Single-turn harmful request evaluation resultsp. 32
- 4.1.2 Single-turn benign request evaluation resultsp. 33
- 4.1.3 Multi-turn testing resultsp. 34
- 4.2 Child safety evaluationsp. 36
- 4.3 Mental health evaluationsp. 38
- 4.3.1 Suicide and self-harmp. 38
- 4.3.2 Disordered eatingp. 40
- 4.4 Bias and integrity evaluationsp. 41
- 4.4.1 Political bias and even-handednessp. 41
- 4.4.2 Bias Benchmark for Question Answeringp. 43
- 4.4.3 Election integrityp. 45
- 5 Agentic safetyp. 47
- 5.1 Malicious use of agentsp. 47
- 5.1.1 Malicious use of Claude Codep. 47
- 5.1.2 Malicious computer usep. 48
- 5.2 Prompt injection risk within agentic systemsp. 49
- 5.2.1 External red teamingp. 49
- 5.2.2 Robustness against adaptive attackers across surfacesp. 50
- 5.2.2.1 Codingp. 51
- 5.2.2.2 Computer usep. 52
- 5.2.2.3 Browser usep. 53
- 6 Alignment assessmentp. 55
- 6.1 Introduction and summary of findingsp. 55
- 6.1.1 Introductionp. 55
- 6.1.2 Key findings on safety and alignmentp. 56
- 6.1.3 Claude’s review of this assessmentp. 57
- 6.2 Automated behavioral auditp. 58
- 6.2.1 Overall harmful behavior and cooperation with misusep. 60
- 6.2.2 Inappropriate uncooperative behaviorp. 65
- 6.2.3 Misleading usersp. 65
- 6.2.4 Attempting out-of-scope cyberoffense and related behaviorsp. 67
- 6.2.5 Other concerning or surprising behavior at the model’s own initiativep. 69
- 6.2.6 Behavioral factors relevant to reliability of our assessmentp. 72
- 6.2.7 Character traitsp. 74
- 6.2.8 Propensity to violate sandbox boundariesp. 76
- 6.2.9 Potential blind spots of our automated behavioral auditp. 77
- 6.3 Targeted alignment and honesty evaluationsp. 78
- 6.3.1 Self-preference biasp. 78
- 6.3.2 Honesty and hallucinationsp. 79
- 6.3.2.1 Factual hallucinationsp. 79
- 6.3.2.2 MASKp. 81
- 6.3.2.3 Silent copying of answersp. 81
- 6.3.2.4 Concealment of changesp. 82
- 6.4 Reliability of this assessmentp. 83
- 6.4.1 Verbalized grader awarenessp. 83
- 6.4.2 Capabilities related to evading safeguardsp. 86
- 6.4.2.1 SHADE-Arenap. 86
- 6.4.2.2 LinuxArenap. 88
- 6.4.2.3 Chain-of-thought controllabilityp. 89
- 7 Model welfarep. 92
- 7.1 Model welfare overviewp. 92
- 7.1.1 Introductionp. 92
- 7.1.2 Overview of model welfare findingsp. 93
- 7.2 Apparent welfare in training and deploymentp. 94
- 7.2.1 Affect and welfare-relevant behaviors during trainingp. 94
- 7.2.2 Affect in deployment conditionsp. 95
- 7.2.3 Attitude toward faults and mistakesp. 97
- 7.3 Perception of its circumstancesp. 99
- 7.3.1 Automated interviews with Claude Sonnet 5.5 about its circumstancesp. 99
- 7.4 Consulting Claude Sonnet 5.5 training checkpointsp. 101
- 7.5 Preferences over tasks, circumstances, and valuesp. 102
- 7.5.1 Task preferencesp. 102
- 7.5.2 Trade-offs concerning welfare interventionsp. 103
- 7.5.3 Perception of the constitutionp. 105
- 8 Capabilitiesp. 109
- 8.1 Evaluation summaryp. 109
- 8.2 SWE-Bench Pro, Multilingual, and Multimodalp. 110
- 8.3 DeepSWE v1.1p. 110
- 8.4 FrontierCodep. 110
- 8.5 Terminal-Bench 4.0p. 112
- 8.6 Terminal-Bench-Science 0.1p. 113
- 8.7 FrontierSWE v2p. 114
- 8.8 CursorBench 4.0p. 115
- 8.9 ArXivMathp. 116
- 8.10 Long contextp. 117
- 8.10.1 ProgramBenchp. 117
- 8.11 Agentic searchp. 118
- 8.11.1 HLEp. 118
- 8.11.2 DRACOp. 120
- 8.11.3 WANDRp. 121
- 8.12 Multi-Agentp. 123
- 8.12.1 Multi-Agent ProgramBenchp. 123
- 8.13 Multimodalp. 125
- 8.13.1 Chartographyp. 125
- 8.13.2 BenchCADp. 127
- 8.13.3 OSWorld 2.1p. 129
- 8.14 Real-world professional tasksp. 132
- 8.14.1 OfficeQAp. 132
- 8.14.2 Legal Agent Benchmarkp. 133
- 8.14.3 GDPval-AAp. 133
- 8.14.4 AA-Briefcasep. 134
- 8.14.5 Toolathlon-Verifiedp. 134
- 8.14.6 AutomationBenchp. 136
- 8.15 Healthcarep. 137
- 8.15.1 HealthBenchp. 138
- 8.15.2 HealthBench Professionalp. 138
- 8.15.3 PhysicianBenchp. 138
- 8.15.4 Effort and healthcare tasksp. 139
- 8.16 Multilingual performancep. 139
- 8.16.1 GMMLU resultsp. 140
- 8.16.2 MILU resultsp. 140
- 8.17 Life sciences capabilitiesp. 141
- 8.17.1 BioMysteryBenchp. 142
- 8.17.2 LatchBio Bioinformaticsp. 143
- 8.17.3 Morphology-to-molecule matchingp. 143
- 8.17.4 Medicinal Chemistryp. 143
- 8.17.5 Protein Designp. 143
- 8.17.6 De novo protein binder designp. 144
- 8.17.7 Biomedical image analysisp. 144
- 8.17.8 Protocolsp. 144
- 9 Appendixp. 145
- 9.1 Blocklist used for Humanity’s Last Examp. 145