.png){kind=logo}
A software engineer specializing in enterprise AI reliability built and open-sourced structured-eval, giving teams a way to catch a wrong AI decision before it ever reaches a business one
Gartner recently warned that, in the near term, "ungoverned decisions using LLMs will cause financial or reputational loss for enterprises." The fix, Gartner said, is to redesign analytic workflows around a "required evaluation stage," a step most AI pipelines skip. Their other release predicts that by 2027, 40% of enterprises will demote or decommission autonomous AI agents, and that the governance gaps behind those failures will typically surface only after a production incident, not before. Strip away the different framing, and both predictions describe that AI systems are already making structured, machine-actionable decisions inside enterprises.
Kirill Pechurin has built a way to check whether any of those decisions was correct, the kind of check a pipeline can run before the output reaches the system that acts on it. A senior software engineer at Arctera, an Astana-based engineering company with roughly 400 IT staff, he built and open-sourced structured-eval, a free tool, published under the Apache-2.0 license so any engineer can use and adapt it, built specifically to catch structured LLM outputs that parse correctly but contain the wrong values. He also led the development of an LLM-based system for procurement, delivery, and equipment-accounting automation, including a spec-reconciliation tool that cut manual work by roughly 30% and exposed the problem this article is about.
What follows is a look at Kirill's work to turn Gartner's "required evaluation stage" from a policy line into a method that enterprise AI and data engineering teams can build.
Most conversation about AI reliability centers on hallucination rates in open-domain question answering, like a chatbot inventing a citation or a search summary getting a fact wrong. Those numbers have improved. But those benchmarks test a model on clean questions and expect an answer in plain sentences (not what most business systems actually ask of it). In production, a model's output usually isn't a sentence a person reads but a structured record (a JSON entry that gets written straight into a database, a compliance check, or a purchase order, often without a human looking at it first).
“A mistake doesn’t look like 'the model said something false' in a way a reader could notice anymore,” Kirill says. “It's now one buried detail, nested deep inside an otherwise correct-looking record, that doesn't match reality. No one skimming a chat transcript would catch it. But the system downstream will still act on it.”
That's a significant blind spot in evaluation tooling. A model can pass every open-domain hallucination benchmark on the market and still ship a record with a wrong quantity, a fabricated part number, or fields that contradict each other. It happens because nothing in the pipeline checks individual fields against a source of truth, only whether the output parsed as valid JSON.
Kirill ran into this gap in back-office automation. At Arctera, his procurement tools included a component that automatically reconciled technical specifications against vendors' commercial proposals, and another that parsed vendor-supplied equipment specifications into structured records the rest of the procurement workflow could consume. The system measurably worked, cutting manual operations by roughly 30% in a decision workflow that ran at high frequency. But that description also matches the risk Gartner is warning about: an LLM's structured output feeding directly into a real business decision, with no human necessarily reading the raw model response before it's acted on.
"A reconciliation error doesn't throw an exception," Pechurin says. "The tool just quietly matches a spec to the wrong vendor line, and by the time anyone notices, the equipment's already been ordered."
In such a kind of workflow, asking whether the output looks plausible is the wrong question. Asking whether a specific field matches the source document is the right one, and where almost nobody had built the tooling to ask it systematically.
Building the procurement system left Kirill with an unsolved problem: the evaluation layer was missing; it didn't exist yet, so he built one and released it under an Apache-2.0 license. The result, structured-eval, is a software tool specifically designed to assess the quality of structured JSON or YAML LLM outputs. Instead of stopping at whether the output parsed into valid JSON, it compares values field by field at any level of nesting, checks outputs against a source document to catch fabricated or unsupported values, and enforces logical consistency across fields through a rules layer built for the purpose. It also supports running evaluations across a full dataset rather than one output at a time, and testing whether a model gives a stable answer when the same prompt is run repeatedly.
"Running the same prompt twice and getting two different structured outputs is routine," Pechurin says. "Most evaluation tooling doesn't even ask that question."
The framework is designed to sit alongside existing LLM-evaluation tools such as deepeval and LangSmith instead of replacing them.
Pechurin frames the underlying idea like this: "There's no such thing as one hallucination score. A quantity field and a description field fail in completely different ways, so each field gets scored with whatever comparison actually fits it, then those get combined."
A quantity field might need exact or near-exact numeric matching; a description field might tolerate paraphrase and reward it with a token-level similarity score; a status field might need strict enumeration matching with zero tolerance. Letting each field carry its own comparison method, then combining those into a composite score, is a much closer match to how a human reviewer actually audits a structured record than a single blended accuracy number ever is.
That kind of infrastructure is, in effect, the practical answer to the industry pattern. It's cheaper to build field-level verification into a pipeline once than to discover, after a procurement error or a compliance gap, that nobody was checking. The organizations less likely to face the kind of after-the-fact governance failure Gartner describes, which is decommissioning agents after the damage is already done, are the ones treating structured-output verification as a first-class engineering problem rather than an afterthought bolted onto a chatbot.
It also isn't the first time reliability work of this kind has produced a large, measurable result rather than a marginal one. Earlier in his career, at the legal and real-estate services firm Consul Group, Kirill redesigned how a 1-million-plus-document Elasticsearch index handled updates. The documents spanned several different data schemas, and the existing approach rewrote an entire document even when a single field changed. However, this design worked until the dataset got large enough to make it expensive. When Kirill restructured the update mechanism to touch only the changed segment of each document, he cut query latency from 5,000 milliseconds to 500 milliseconds, a tenfold improvement, and the fix held up as load on the system continued to grow. It's a different problem than structured-output verification, but the same idea: not to assume a system behaves correctly at scale, but to measure it and rebuild the part that doesn't.
An LLM can be reliable in general and still be the wrong tool for a procurement pipeline if nobody checks whether its specific fields match the specific documents it was given. Kirill Pechurin built his tool to close this gap, giving enterprises a concrete way to check AI-driven procurement, finance, and compliance decisions before they ship.
