OpenAI tweaks Astra's benchmarks, giving it a boost over Anthropic
What's the story
OpenAI has made several changes to the evaluation benchmarks for its latest model, GPT-6 Astra, since its initial announcement last week. The updates have shown Astra performing better in some cases while models from rival company Anthropic have seen a decline in their scores. The changes come amid a highly competitive AI landscape where companies are constantly updating their large language models (LLMs) to stay ahead.
Technical issues
Blog post rollout faced technical issues
The rollout of OpenAI's blog post was marred by technical glitches. The company had initially scheduled the post to go live at 2:00pm ET but it took nearly two hours for it to be widely accessible online.
When OpenAI's X account tweeted out the blog post, the link didn't work properly and returned an error message. Later, CEO Sam Altman acknowledged these issues in a tweet and shared the link himself.
Metric adjustments
Initial blog retracted for undisclosed reasons
OpenAI had published the blog post shortly after 2:00pm but later retracted it for undisclosed reasons. The company said these were unrelated to benchmark performance figures.
When the blog was republished, it featured different evaluation metrics that appeared to favor Astra.
Some of these figures have continued to change even after publication, highlighting the difficulty of using standardized benchmark tests to measure LLM performance and concerns over potential manipulation.
Defense
OpenAI defends its actions
An OpenAI spokesperson defended the company's actions, saying "We care deeply about getting evaluations right."
They explained that most evaluations have noise within a few percentage points based on factors like the exact checkpoint and scaffold used in reporting.
The spokesperson added that they made fixes for their launch blog to ensure numbers represent the best estimate of available model performance, allowing users to make meaningful comparisons.
Metric revisions
Astra's hallucination rate halved before being restored to original value
The first internet archive snapshot of the blog post showed Astra's hallucination rate at 4.2%. This remained unchanged for several snapshots before being halved to 2% in a later version.
The scores for Astra's predecessor, GPT-5.6 Sol, also decreased from 12.2% to 9.4%.
However, OpenAI has continued to modify this metric; as of now, the hallucination rates are back up to their original values of 4.2% and 12.2%.
Score boost
ExploitBench score for Sol boosted significantly
OpenAI seems to have given a significant boost to GPT-5.6 Sol on its internal version of the ExploitBench cybersecurity evaluation, increasing its score from 5.5% in the first version to 11.5% in later versions.
The company said it is currently looking into reverting that number back to 5.5% as the higher result reflects a reasoning level not commercially available for Sol.
More
Astra appeared significantly stronger at math than other models
OpenAI says Astra is particularly strong at mathematics, a capability the company highlights in the opening paragraph of its announcement.
Astra's own score on the FrontierMath Tier 4 (v2) evaluation remained unchanged at 97.6% across the snapshots.
However, OpenAI briefly changed the scores listed for GPT-5.6 Sol and Anthropic's latest model, Fable 5.1, making Astra appear significantly stronger at math than both models.
Updated scores
The updates and latest scores for Fable and Sol
In the first snapshot, taken at 2:23pm on September 3, Fable 5.1 had a score of 87.8%. By 5:17pm., that figure had fallen by nearly 10 percentage points to 78%. It has since returned to 83%.
GPT-5.6 Sol saw a similar change, with its score dropping from 83% to 80.5% before returning to 83% today.
Evaluation differences
Caveats around evaluation metrics and test conditions
Different research teams at OpenAI manage different metrics and report them to a central team for publication.
The company is transparent about these numbers being achieved under optimal conditions and may differ slightly from the models available in the production ChatGPT product most users can access.
"Evaluation scores are the maximum at any effort," reads a disclaimer on the blog, with further caveats provided in footnotes for each metric.