.png){kind=logo}
A research team from Hao AI Lab at the University of California San Diego has uniquely tested artificial intelligence models—by making them play Super Mario Bros. Unlike traditional benchmarks, this real-time gaming test evaluated how well AI systems adapt to dynamic environments.
The results revealed a clear divide: non-reasoning models, such as Claude 3.7, excelled in quick reactions, while reasoning models, including OpenAI’s GPT-4o, struggled with delays. The findings raise important questions about AI evaluation methods and the balance between speed and reasoning in real-world applications.
The researchers implemented GamingAgent as a framework to let AI models determine Mario's movements through the game. The performance results showed that Claude 3.7 from Anthropic occupied first place, with Claude 3.5 in second place. The models from Google, Gemini 1.5 Pro and OpenAI, GPT-4o, encountered significant difficulties in this setup.
The research used an emulator instead of running the 1985 version of the game. GamingAgent provided the AI system with fundamental gameplay directions and digital images from the game screen.
Afterwards, the AI systems produced Python code that helped Mario navigate through obstacles and enemies in the game. This trial scenario assessed the AI adaptation and planning capabilities. Through quick gameplay, the researchers observed that models demonstrated strong and weak points that may not become apparent during regular testing.
Claude 3.7 and 3.5 showed superior performance in the fast-paced elements of Super Mario Bros. The non-reasoning models achieved fast reaction times toward game events such as enemy dodging or jumping over gaps. The OpenAI GPT-4o reasoning model demonstrated difficulty when dealing with the task's quick response requirements. The models processed their decision-making processes which took several seconds, causing significant ramifications in a game where timing determines success or failure.
The difference in performance demonstrates the main distinction between AI systems that use reasoning methods and those without reasoning capabilities. The strength of reasoning models lies within structured problems but these systems demonstrate limitations when dealing with fast-changing dynamic settings that require immediate responses.
The Hao AI Lab team identified that Super Mario Bros. revealed crucial weaknesses of AI systems, which generated new insights about their capabilities. According to the study findings speed and adaptation ability are equally important as raw computational power when solving problems in specific situations.
Using video games to assess AI is not new but it sparks ongoing discussion among experts. However, certain experts doubt whether gaming expertise demonstrates real technological advancement. Games contain complex challenges but offer simpler problems than what happens in real-life situations.
OpenAI research scientist Andrej Karpathy declared the need for new evaluation metrics to be an “evaluation crisis.” According to him, there exists confusion about which measurement standards accurately assess modern AI capabilities.
This experiment from Hao AI Lab contributes important evidence to controversial discussions in the field. In this study, AI demonstrates its interactive abilities yet critics point out that games supply perpetual training data that cannot be easily replicated in real environments.
The research demonstrates that innovative assessment tools such as Super Mario Bros. demonstrate value for AI testing beyond conventional methodology limitations. The research findings propel discussions regarding efficient evaluation methods for contemporary artificial intelligence systems.
