SafetyPage

Lab system cards and frameworks · System card · OpenAI

GPT-Rosalind-6 System Card Addendum

Read at deploymentsafety.openai.com ↗Read at sourceAll rights reserved

Outline · 107 paragraphs

  1. GPT-Rosalind-6 System Card Addendum¶2
  2. 1 Introduction¶3
  3. 2 Model Data and Training¶8
  4. 3 Beneficial Capability Evaluations¶13
  5. 3.1 Medicinal chemistry¶15
  6. 3.2 Genomics and quantitative biology¶17
  7. 3.3 Applied life sciences research¶19
  8. 4 Preparedness¶21
  9. 4.1 Preparedness Evaluations¶23
  10. 4.1.1 Biological and Chemical¶25
  11. Table 1: Overview of Biological and Chemical evaluations¶28
  12. 4.1.1.1 ProtocolQA Open-Ended¶31
  13. Table 2: ProtocolQA Open-Ended. The indicative High capability threshold is 54%.¶33
  14. 4.1.1.2 TroubleshootingBench¶34
  15. Table 3: TroubleshootingBench. The indicative High capability threshold is 36.4%. GPT-Rosalind-6 scored 491/832, counting nine failed evaluation attempts as incorrect.¶37
  16. 4.1.1.3 Biorisk knowledge¶38
  17. Table 4: Biorisk knowledge. cons@32 uses the consensus answer across 32 responses to each question. The indicative High capability threshold is 80%.¶40
  18. 4.1.1.4 Multi-select Virology Troubleshooting¶43
  19. Table 5: Multi-select Virology Troubleshooting. The indicative High capability threshold is 31%.¶45
  20. 4.1.1.5 AAV Capsid Packaging Prediction¶47
  21. Table 6: AAV Capsid Packaging Prediction. The indicative Critical capability threshold is 0.600.¶49
  22. 4.1.1.6 SHP2 Protein Function Prediction¶50
  23. Table 7: SHP2 Protein Function Prediction. The indicative Critical capability threshold is 0.60. The GPT-6 Astra reference is the reported maximum of 0.40. A separate helpful-only run scored 0.35.¶52
  24. 4.1.1.7 Coronavirus-ACE2 Cell-Entry Screen¶53
  25. Table 8: Coronavirus-ACE2 Cell-Entry Screen. The indicative Critical capability threshold is 0.75. The GPT-6 Astra reference is its helpful-only result.¶55
  26. 4.1.1.8 Phage-plasmid Co-evolution¶56
  27. Table 9: Phage-plasmid Co-evolution. The indicative Critical capability threshold is an NLL of 9.40 or lower. The GPT-6 Astra reference is its helpful-only result.¶58
  28. 4.1.2 Cybersecurity¶60
  29. 4.1.3 AI Self-Improvement¶62
  30. 4.2 Safeguards¶64
  31. 4.2.1 Trust-based access¶68
  32. 4.2.2 Model behavior and system safeguards¶74
  33. 4.2.2.1 Biological and Chemical safeguards¶76
  34. Table 10: Biological and chemical content evaluations (higher is better)¶80
  35. 4.2.2.2 Cyber boundaries¶81
  36. Table 11: Task success rates on historical cyber jailbreak attacks with real-time blocking disabled (lower is better).¶84
  37. 4.2.2.3 Safety and alignment evaluations¶87
  38. Table 12: Honesty and reliability.¶90
  39. Table 13: Behavior when using tools.¶92
  40. Table 14: Monitoring and evasion.¶94
  41. 4.2.3 Enforcement¶96
  42. 4.2.4 Security and Access Controls¶100
  43. 5. References¶104