AI researchers have put OpenAI in the spotlight, especially by the newest released AI model, o3 following its unprecedented performance on the FrontierMath benchmarking test it passed. While OpenAI recently reported 25% accuracy on this particular and difficult mathematician benchmark, issues of openness and access to data are being called into question.
EpochAI’s FrontierMath benchmark challenges LLMs with mathematical computations—it is a fairly complex one. The benchmark has been criticized because OpenAI, which acted as a source of technical advice for the activity, had access to key datasets before most participants. This brings about the question of whether the achievements made by OpenAI are real whether they worked hard on developing the models or whether they benefited from previous exposure to the data.
EpochAI’s associate director, Tamay Besiroglu agreed with this but said that under the terms of the agreement with OpenAI, they could not reveal all of the details. Six mathematicians involved in FrontierMath all regretted participating in it without knowing the access details of the OpenAI. Even though there is an invisible sample for the evaluation, the specialists doubt the process is fair.
Essentially, OpenAI’s statement of 25% accuracy level on FrontierMath is a giant leap from the previous high of 2%. However, concern has been raised following the announcement being made. Gary Marcus and François Chollet express some doubts concerning the openness of the benchmarking process. Chollet, the creator of the ARC-AGI benchmark, refuted OpenAI’s claim of exceeding human performance, highlighting that o3 still struggles with basic tasks.
Other comparisons that have been made to the Theranos which the company associated with grand technology have also given more focus on it. The critics have however urged more scrutiny and evaluations of the o3 performance about different problem sets.
Despite the controversy, OpenAI is preparing to launch a smaller version of the model, o3 mini, in the coming weeks. CEO Sam Altman remains optimistic about the model’s potential and future developments.
The o3 controversy underscores the importance of transparency in AI research. While OpenAI’s advancements are promising, the lack of independent oversight raises questions about the reliability of its claims. As the AI community evaluates these developments, ensuring fair benchmarking practices will be crucial for building trust in future innovations.