SafetyPage

Government · Government · NIST

2024 NIST GenAI (Pilot Study): Text-to-Text Evaluation Overview and Results

24 Jun 2025

ReadSource at nvlpubs.nist.gov ↗Read herePublic domain (US government work)

Outline · 61 pages

  1. Introductionp. 10
  2. Related Workp. 11
  3. Evaluation Frameworkp. 13
  4. Evaluation Managementp. 15
  5. Goal and Function within Evaluationp. 15
  6. Registration and Licensing processp. 16
  7. Analytics Backendp. 17
  8. Evaluation Pipelinep. 17
  9. Implementation Overviewp. 17
  10. Scoring Processp. 18
  11. Text-to-Text Validation and Testingp. 18
  12. Text-to-Text Baseline Modelsp. 21
  13. Administrative and Regulatory Requirementsp. 22
  14. Registrationp. 22
  15. Data Usage Agreementsp. 23
  16. Data Transfer Agreement for Generatorsp. 23
  17. Human Subjects Research Determinationp. 23
  18. Software Usage Approvalsp. 23
  19. Tasksp. 23
  20. Text-to-Text Generators (T2T-G)p. 24
  21. Text-to-Text Discriminator (T2T-D)p. 24
  22. Discriminator Metricsp. 25
  23. Receiver Operating Characteristic (ROC)p. 25
  24. Area Under the ROC curve (AUC)p. 25
  25. True Positive Rate (TPR) at False Positive Rate (FPR)p. 25
  26. Detection Error Tradeoff (DET) and Equal Error Rate (EER)p. 25
  27. Assessment of the Quality of AI-generated Summariesp. 27
  28. Data and Submissionsp. 29
  29. Datap. 29
  30. Submissionsp. 31
  31. Generatorsp. 31
  32. Discriminatorsp. 33
  33. Data Analyses and Resultsp. 35
  34. Results for Test Set-1 (D-Round 1)p. 35
  35. Results for Test Set-2 (D-Round 2)p. 41
  36. Results for Test Set-3 (D-Round 3)p. 45
  37. Evolution of Discriminatorsp. 49
  38. Strengths, Limitations, and Challengesp. 50
  39. Strengthsp. 50
  40. Limitationsp. 51
  41. Challengesp. 51
  42. Conclusionp. 52
  43. Recommendations for Future Workp. 52