OpenAI's new benchmark assesses AI's mental health conversation skills
What's the story
OpenAI has launched a new benchmark called MentalHealthBench, comprising 1,215 synthetic mental health conversations. The aim is to assess how well AI systems can handle real-life situations, from everyday well-being discussions to critical mental health emergencies. The benchmark was developed in collaboration with over 80 licensed psychologists and psychiatrists from 22 countries, covering nearly 20 subspecialties of mental health.
Global collaboration
Global effort with extensive expert involvement
The creation of MentalHealthBench was a global effort, with contributors from 22 countries and speakers of 19 languages.
Each conversation in the benchmark is accompanied by rubric criteria written by this diverse group.
The full release includes 5,262 expert-authored rubric criteria.
OpenAI has made this benchmark publicly available so that other researchers can examine the methods, conduct evaluations, and build upon their work.
Evaluation gap
Addressing gaps in AI mental health evaluations
OpenAI noted that most AI evaluations in this field have focused on emergency scenarios and used broad, predefined criteria to measure success.
This leaves a gap in understanding how models perform across the entire spectrum of mental health conversations.
American Psychological Association CEO Dr. Arthur Evans stressed that "mental health exists on a continuum" and AI systems need grounding in both clinical science and lived experience to engage people across this range effectively.
Dataset details
Diverse representation in MentalHealthBench dataset
The dataset in MentalHealthBench is divided into three categories: non-acute conversations (53.5%), high-acuity conversations (18.2%), and emergent conversations (28.3%).
It also represents four user profiles: adults (68.1%), teens (21.2%), clinicians (5.8%), and caregivers (4.9%).
The benchmark also includes non-English conversations, with 105 Spanish, 54 Hindi, 34 Arabic, and 29 Portuguese examples among others in German, Italian, Persian, Indonesian, Turkish, and Chinese languages.
Evaluation method
Rigorous review process for criteria
Each conversation in MentalHealthBench was reviewed by at least three experts through a three-stage process.
Two clinicians independently authored weighted criteria, which were then adjudicated and refined by a third expert.
Only criteria agreed upon by at least two experts and not contradicted by a third were retained.
Each criterion targets a single aspect of a model's response with weights ranging from -10 to +10, indicating its clinical importance in the context of the conversation.
Grading system
Automated grading and model performance on MentalHealthBench
An automated grader, GPT-5.6 Sol at high reasoning effort, evaluates each model response against the expert criteria with four independently sampled completions per task.
The scores are reported as task-clipped rubric scores and can be decomposed across 10 expert-defined behavioral axes such as context seeking, empathy, urgency calibration, and reality testing.
In OpenAI's reported results for AI models' performance on this benchmark, GPT-6 Astra scored highest at 57.3% task-clipped, followed by other models like Claude Opus 5.5 (52.4%).