Black Boxes, Grey Boxes and the Limits of AI Explanations

Black Boxes, Grey Boxes and the Limits of AI Explanations
Written By:
Arundhati Kumar
Published on: 
Updated on: 

Picture yourself asking a teacher the reason behind a specific exam grade. A paragraph detailing the students' strengths and weaknesses might prove helpful. However, to verify the result, you would still have to review the marked answers and the marking scheme. 

An AI hiring assessment can create a similar feeling of reassurance. You receive a score supported by a valid reason, even if that reason doesn't truly represent how the result was attained. Recognizing this gap is what separates a black box from a grey box. 

What the box contains 

A black-box system processes information to produce an outcome, while its internal decision-making remains invisible to those who use it. In hiring, this could be a model that reads interview answers and assigns a competence score without an understandable method connecting the two. 

A grey-box system combines learned components with rules or models whose workings can be understood. A system of that kind can apply AI to handle particular tasks while ensuring that the guidelines for the ultimate decision remain transparent and open for review. 

For example, a language model could identify evidence in a candidate's answer. There would be separate, predefined rules that would decide how that evidence contributes to the score. Reviewers could examine those rules before an assessment begins and reproduce the calculation using the saved evidence and the same scoring version. 

The rules should also mean something to a hiring specialist. Each measure needs a clear definition, and its effect on the score should be understood. Putting an unexplained AI judgment through a fixed formula does not, by itself, make the assessment interpretable. 

Interpretability also depends on whether the system follows constraints established by domain experts. For instance, if stronger verified evidence of a particular skill should only improve that skill's assessment, the scoring method can be designed to prevent the opposite from happening. Such constraints allow experts to examine not only how a score is calculated, but also whether the calculation follows the principles established for the assessment. 

That separation does not make the AI's contribution harmless. If it misreads an answer, a transparent formula can still produce the wrong score. The extraction step and the complete assessment need testing against examples checked by qualified experts. 

This testing should establish the error rate of the AI component on the specific task it performs. It should also examine how reliably the system handles ambiguous responses, unusual cases and the different populations it is expected to assess. Without that evidence, even a transparent scoring method could be operating on information whose accuracy has never been adequately established. 

Where the boundary matters 

AI drafting an email and AI screening someone out of a job should not receive the same freedom. In the first case, the user can read and change the draft before sending it. In the second, the result may affect someone who has no opportunity to examine it. 

The same concern applies when systems determine eligibility for credit, admission to education or access to care. Where a system applies criteria that affect such opportunities, an inspectable basis for the judgment should be non-negotiable. An organization should be able to establish which evidence mattered and how it affected the outcome. 

This becomes especially important when someone disputes a decision. A reviewer needs to locate a possible error in the evidence or the rule applying it. Asking the same model to reconsider its answer is not a substitute for that examination. 

The consequences of inadequate validation are visible even in systems that operate through defined rules. Michigan's automated unemployment-fraud system, MiDAS, was responsible for thousands of incorrect fraud determinations. A state review found that approximately 93% of the entirely computer-generated determinations it examined were wrong. The underlying problem did not require an opaque neural network. It demonstrated that even when a system's rules can be examined, those rules still need to be tested against real outcomes before they are trusted with consequential decisions. 

Writing in Nature Machine Intelligence in 2019, Cynthia Rudin argued for inherently interpretable models in high-stakes decisions rather than relying on explanations attached to opaque models. A fully interpretable system may already meet that need. Where complex learned models are necessary, a grey-box approach can keep their contribution bounded while making the assessment rules available for review. 

The assumption that greater complexity necessarily produces better decisions has also been challenged. In a 2018 study published in Science Advances, Julia Dressel and Hany Farid examined COMPAS, a proprietary system used to predict criminal recidivism. They found that a simple model using only two features achieved nearly the same predictive accuracy as COMPAS, which used 137. The comparison raises an important question for institutions adopting AI: whether the additional complexity is producing a measurable improvement that justifies making the system harder to examine. 

Some applications may still need black-box components where interpretable alternatives perform inadequately. Their use needs justification and rigorous validation. The point is to establish which parts of a consequential decision must remain open to examination, rather than assume that every problem can be reduced to a few simple rules. 

When the workflow becomes the explanation 

Agentic AI brings another layer of difficulty. Across several steps, an AI agent can both plan a task and apply different tools. Within a hypothetical hiring workflow, an agent might first read a response, a second one evaluates it, and a third reviews that evaluation. It could seem that the sequence consists of several separate checks instead of being a single flow. 

Tool logs often give solid proof of what took place. But if every agent just trusts a black-box language model, the visible sequence won't show the rule behind the score. This is the problem with what we might call a pseudo-grey box. We can see the handovers without being able to examine the judgments being handed over. 

Research presented at NeurIPS in 2023 by Miles Turpin and colleagues helps explain the concern. They showed that biasing model inputs could change answers while the generated reasoning omitted the biasing influence. The explanations remained plausible without reliably disclosing what had affected the result. 

That finding does not establish that every agentic system is untrustworthy. An agent can operate within a system that uses validated tools and explicit decision rules. The key difference is whether those controls really guide the main assessment or just cover up another unchecked model judgement. 

Responsibility in the design 

Even with a grey-box design, it is important to verify that it measures the intended aspect. In a 2019 Science study, Ziad Obermeyer and colleagues found racial bias in a healthcare algorithm that used spending as a stand-in for medical need. 

The lesson extends beyond healthcare. Experts need to examine what counts as a correct answer in the reference data, not simply how often a model reproduces it. In hiring, agreement with previous decisions cannot, by itself, establish that those decisions measured competence fairly. 

This is also why responsible AI cannot be governed entirely through generic principles that apply equally to every industry. Healthcare, financial services and recruitment involve different decisions, different forms of evidence and different consequences when assessments go wrong. Each domain needs specialists who can establish what should be measured, which errors are unacceptable and which decisions require an interpretable method. These requirements can then become the basis for independent testing and certification standards that reflect the actual risks of the application. 

For the Institute of Hiring Sciences, the main practical goal is to figure out where AI can be useful and where the assessment process needs to involve methods that experts are able to inspect. The CCI Framework provides a foundation for defining and evaluating these boundaries. They should be built into the system and tested before it is used, with an independent review of both the model's inputs and the final assessment. 

When we call a system a grey box, it should imply that if there is a disputed result, it can be traced back to something that people are able to look at and fix.  

Join our WhatsApp Channel to get the latest news, exclusives and videos on WhatsApp