Automated test suites break all the time, and usually not because the software is failing. A button moves, a label is renamed, a developer refactors a page, and a test that checked the right thing yesterday can no longer find what it is looking for. For years, fixing that meant an engineer had to stop and repair the test by hand. A newer class of tools promises to do the repair automatically, using large language models to rewrite the broken locator.
Suneet Malhotra, a Senior Member of the IEEE and an independent researcher in AI, software quality, and test automation, set out to measure how well that promise holds up. His research on LLM-based self-healing test infrastructure, under peer review at the Journal of Systems and Software and available as an open preprint (https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6963910), attaches hard numbers to a technology that is usually described in marketing language rather than empirical terms.
Malhotra’s standing in the field extends beyond this single paper. He currently has four manuscripts under peer review: the self-healing and cross-layer-observability study described here at the Journal of Systems and Software; a specification-enrichment study at the e-Informatica Software Engineering Journal; an embodied-oversight framework at IEEE Pervasive Computing; and a fault-injection field study of unattended LLM-agent systems submitted to a reliability-engineering workshop at IEEE ISSRE 2026. A companion visual-oracle benchmark, described below, is available as an open, citable preprint. He has presented this line of work at BrowserStack Breakpoint 2026 and BrowserStack World Tour 2025. He serves on the Artifact Evaluation Committee for ASE 2026 (the IEEE/ACM International Conference on Automated Software Engineering) and as an ethics reviewer for NeurIPS 2026, and has been invited to the Program Committee of the Software Engineering in Practice track at ICSE 2027 — one of the field's principal venues for practitioner research.
The results come in two findings that have to be read together. Across two complex public web applications, Supabase Studio and Grafana, and two different model families, the LLM-based healer recovered between 55 and 68 percent of broken locators. That was well ahead of the alternatives tested in the same setup: a text-similarity heuristic at 20 to 34 percent, the self-healing tool Healenium at 23 percent, and a plain broken selector at zero. The second finding complicates the first. About 26 percent of the time, the healer produced a repair that passed but was wrong.
Malhotra summarizes the result this way: “LLM healing recovers 55 to 68 percent of broken test locators, beating the industry tool and a text heuristic, but it false-heals about 26 percent of the time, so it suits assisted triage rather than unsupervised CI.”
A 26 percent false-heal rate is not a rounding error. In a CI/CD pipeline that runs hundreds of tests a day, an incorrect automatic repair is harder to catch than an outright failure. The test still passes and the locator still looks fixed, even though the repair now points at the wrong element and the check no longer verifies what it was written to verify.
That is why Malhotra’s research stops short of endorsing the technology for unsupervised use. It does not argue that LLM-based healing is useless; it argues that the conditions under which it is useful are specific. Assisted triage, where an engineer reviews a proposed repair before it is committed, is a workable mode. Unsupervised CI, where repairs are committed automatically, is not ready for that role yet.
The false-heal figure came from an evaluation in which every proposed repair was checked against a known-correct target. That let the benchmark separate a fix that restored the intended behavior from one that merely made the test pass. The LLM healer recovered more broken locators than any baseline tested, but a higher recovery rate did not translate into reliable repairs, and the benchmark quantifies that gap directly.
A second strand of Malhotra’s research asks what would have to change for these tools to become more trustworthy. His answer, developed in a companion line of work on specification enrichment with a public reference implementation, is that part of the problem lives upstream.
Test locators break for a reason: the application changed, and the test did not carry enough of its original intent to adapt. A richer specification, one that encodes more of the original design intent than a bare selector, gives a healer more to work with when a failure occurs. The more the system knows about what a test was meant to verify, the better its odds of proposing a repair that preserves that intent instead of restoring a selector that only happens to match the current page.
That reframes self-healing as a question of engineering discipline rather than tool choice. The deciding factor is less which product a team buys than whether the specifications underneath the test suite are detailed enough to support reliable automated repair.
Malhotra published his agent-harness architecture as an open-source repository at github.com/SuneetMalhotra/agent-harness (DOI: 10.5281/zenodo.20576685) under an MIT license. It connects framework authoring, a multi-agent pipeline built on the Model Context Protocol, and self-healing execution through a single shared event store, so that what the execution layer learns about flaky elements can inform the layers above it. In a paired comparison, the version of the healer that could read this shared history recovered slightly more locators than the version that could not, 68 percent against 60 percent, but Malhotra reports that the difference was not statistically significant at the sample size tested and treats the benefit as an open question for a larger study. A small calibration run on a public reference application produced a preliminary visual-assertion agreement of Cohen’s κ = 0.667 against a seeded ground truth, a number he presents as directional given its 24-image sample. Only the web execution path has been formally evaluated; the mobile and hardware-in-the-loop tiers are part of the design but were not tested.
That self-healing work is one half of a connected research program. A separate study, distinct from the self-healing benchmark described here, is available as an open, citable preprint. That paper measures how reliably two independent AI visual judges agree with a seeded ground truth, including how often they raise a false alarm on a screen that is actually correct, and it reports substantial inter-judge agreement of Cohen’s κ = 0.679. The benchmark is archived publicly at github.com/SuneetMalhotra/visual-oracle-bench with the DOI (https://doi.org/10.5281/zenodo.20620870), alongside a preregistered phase-two replication protocol (OSF DOI 10.17605/OSF.IO/CSKUY).
Malhotra is Senior Manager of Test Engineering at Motorola Solutions. The affiliation is listed for identification only; the research described here is his own, carried out on public infrastructure and independent of any employer systems or data. He has nearly two decades of engineering practice across web, mobile, and hardware-in-the-loop systems, with earlier roles at Tinder(Match Group), Amazon, Ticketmaster and Spokeo.
For teams weighing these tools, the benchmark offers a baseline rather than a verdict. Recovery in the 55-to-68 percent range is worth having, but the 26 percent false-heal rate means a person still needs to review each repair before it is committed. Without that step, an automated fix can leave a test passing for the wrong reason, which costs a team more time to untangle than the broken test it replaced.
Related: Malhotra discusses this research program on The Agentic Quality Podcast and on the Real Python Podcast