Artificial intelligence researcher Andrej Karpathy showed how large language model testing is changing, replacing traditional coding benchmarks with real-world software engineering tasks.
In a recent experiment, Karpathy asked Anthropic’s Claude Opus 5 to transform the first paragraph from ‘The Lord of the Rings’ by JRR Tolkien into a 3D interactive website, showcasing how far AI systems have come while also underscoring their limitations.
Rather than ask the AI model to produce small snippets of code and static images, Karpathy gave Claude Opus 5 a token limit of one million, which would translate to about $10, and told the AI system to create the scene using the Three.js JavaScript library, a common platform used for browser 3D graphics.
This test represents a new direction in how frontier AI is being tested. Conventional testing methods have been limited to specific tests that involve completing individual tasks like writing code and text. However, Karpathy tested the model’s ability to think, write thousands of lines of code, allocate resources, and provide an interactive product.
The above example shows an industry trend regarding the evaluation of AI through an end-to-end workflow process. As AI models improve at performing tasks, engineers want to evaluate their performance in solving engineering problems.
Even though Claude did manage to produce an interactive 3D scene, the experiment also pointed out the shortcomings of the most advanced AI models. According to Karpathy, visual rendering and scene composition are two tasks that are still challenging for AI.
Another problem with the experiment was how to assess the outputs of the technology in question. While the output could be automatically tested and evaluated, visual data cannot be reliably assessed without human intervention.
Karpathy’s experiment illustrates how expectations from large language models are expanding beyond answering questions or writing code. Developers now expect AI systems to build complete applications, reason across multiple steps, and generate interactive digital experiences with minimal human intervention.
As AI capabilities continue to improve, such real-world engineering challenges are expected to play a bigger role in measuring model performance, exposing not only what these systems can accomplish but also the technical gaps that remain before they can consistently deliver production-ready software.