.png){kind=logo}
An engineer who builds AI systems over operational data argues that the largest cost decision is made before anyone writes a prompt
The error arrived before the answer did. The system had been handed a stretch of operational data, logs, alerts, historical records, and asked to work out what had gone wrong. Instead of a diagnosis it returned a hard context-length failure. The model had a token limit and the request had run past it.
Cut until it fits. That is the obvious response, and it is the one Yashaswini Nalla warns against. She designs AI systems that analyse operational data so teams can diagnose problems faster, and she treats blind truncation as the trap rather than the solution. It gets the request under the limit and breaks the thing the request was for. "You end up cutting the wrong things and the model reasons badly on incomplete context," she says.
This is true as many teams move from using language models (lms) as experimental tools to lms on their cloud bills. The maturity of the practices for managing cloud costs have been built around a few familiar approaches: instance types; caching; committed use discounts. Nalla’s point is that all of these approaches are “downstream” of where the decisions are actually made regarding how much money is spent. Teams therefore turn to these approaches because they can’t clearly see the upstream decision.
The upstream decision is one that Nalla asks before any prompts are written. Is this task best accomplished by a probabilistic model, or is it best accomplished by a deterministic pipeline? In an environment where almost all teams want to apply some form of AI to the things they are currently building, the question is becoming increasingly unpopular. Nalla continues to ask the question regardless.
The upstream decision creates a divide. Pattern matching, correlation, looking up against a known reference: all of this is rule based and remains within a deterministic pipeline where it can be inspected and where the cost is fixed and predictable. A rule that fires in exactly the same way each time doesn’t require a polite request and doesn’t require payment via tokens. The portion of the task that requires reasoning — for example, creating a story out of ambiguous or missing information — is what gets sent to the model. Determining the appropriate split at the outset, rather than sending everything down the pipe via a model by default, is what she gives credit to for maintaining a system that is inexpensive and explainable versus making it expensive and opaque.
Nalla’s point can be summarized in a single sentence. AI should be viewed as the last mile option once all other deterministic options have been explored — not simply as the default response to a problem that hasn’t been considered yet.
What follows is unglamorous, according to Nalla. The process of determining what elements get pulled out and sent to the model prior to any of them reaching a prompt is something that never makes the pitch deck. While it may not be as glamorous as the AI architecture that sits atop it, Nalla believes this process is responsible for determining if a system can be reasonably priced to run in production.
When feeding operational data to a model, there exists an instinctive urge to feed everything: full records; complete detail; all fields available. Most of that information represents noise that the model does not require in order to reason correctly. Pulling only the elements that matter while leaving behind the detailed but low signal information greatly decreases prompt size compared to feeding raw data, according to Nalla. Instead of viewing the process as an optimized one-off, she views it as a repeatable process.
The logic required to build extraction was not the hard part; tuning it was. Since there existed no method of anticipating which fields consistently mattered toward a correct outcome and which were weighted toward being worthless, the work became a series of experiments, run and rerun until results were produced rather than intuition. While weighted toward being unimportant, dead weight rarely appears obvious from outside looking-in which is why the question had to be resolved through testing versus inspection.
While Nalla never attempted to shorten payload, she did attempt to create denser ones. According to Nalla, the process required more iterative and unglamourous steps than people expect.
According to Nalla, limiting what reaches the model to structured attributes relevant to the task vs. Sending raw unfiltered data meaningfully reduced both prompt size and frequency of context-limit failures without losing any signal necessary for the model to reason well.
In general, according to Nalla, where do teams look when the bill arrives? There exists quiet assumption that cost problems associated with AI are infrastructure issues solved by selecting better instance types adding caching negotiating better rates. Based on her experience, however, the larger lever exists above all of this within whether the model should be viewed as a place to dump everything that might conceivably be needed or an expensive reasoning step that only sees what it requires.
“The discipline that saves the most money isn’t a FinOps dashboard,” says Nalla. “it’s deciding line by line what earns the right to be sent down stream.”
It’s a demanding standard because it forces teams to justify every field they pass along at exactly the same time that easier move would have been passing them all now and sort out later. As context windows grow and per-token prices fall, the desire to skip over the exercise will grow directly with them. Nalla’s point suggests that the context window was never really the true constraint – only the first visible enough to force a decision. Systems that send a model only what it needs are less expensive to run and also easier to reason about when things go wrong. These two properties tend to arrive together or not at all.
