.png){kind=logo}
Walk into any digital product organization today and you will find the same ritual. Before a feature ships, it goes through an A/B test. Half the users see the change, half don't, and the difference decides its fate. The practice has become so embedded that experiment readouts function as something close to institutional truth that if the topline metric didn't move, the idea didn't work, and the team moves on.
But a growing body of practitioners argues that this reflex is quietly wrong a significant share of the time, and that two-sided digital marketplaces are the most exposed of all. The reason is structural: on a marketplace, no two customers see the same page. Assortment is personalized, inventory is supplier-dependent, merchandising is dynamic, and much of the interface renders conditionally. In that environment, being placed in a test group tells you almost nothing about whether a customer ever encountered the thing being tested.
Most AB testing platforms enroll users the moment a session begins, and that enrollment becomes the criteria for every analysis that follows. But a large share of modern features are conditional such as a module below the fold, a component gated behind an interaction, a recommendation block that surfaces only for certain inventory or certain users. Practitioners call this dilution, and the scale is not marginal: for conditionally rendered features, trigger rates commonly fall between 20% and 40% reported by major online marketplaces, meaning most of the "treated" population never experienced the treatment at all. This dilution effect not only causes digital technology companies to ship bad features but also kill good ones. Marketplaces absorb this worse than most. Unlike many simpler digital products, marketplaces combine personalization, dynamic inventory, supplier availability and context-dependent user journeys. Two customers assigned to the same experiment may encounter very different experiences. As personalization becomes more sophisticated, the gap between experiment assignment and actual exposure can widen.
While the scientific theory behind the remedy is well known, yet most organizations struggle to adopt it due to engineering challenges. Online e-commerce marketplaces face the hardest version of the problem. They must account for exposure across search, browse, product pages, recommendations and checkout flows, often while eligibility changes throughout the experiment.
This is where the work of Shashank Agarwal, a technical lead on the experimentation platform team at Wayfair, stands out. He devised a more practical way to solve the problem at scale. Rather than rewiring how users are assigned to experiments, the industry standard approach, and one that adds significant engineering complexity, he build a post-processing layer over telemetry the platform already collects. Assignment stays untouched. The full dataset is preserved. And the analysis runs after an experiment concludes, when exposure can be established from what customers actually did. In a Wayfair engineering post titled "When Assigned Isn't Exposed: How Wayfair Uses Trigger Analysis to Recover Hidden Test Signals", Agarwal sets out the architecture in full, along with the safeguards that make it trustworthy at scale.
Agarwal's work has not only helped Wayfair save millions of dollars but also provided a path for other digital marketplaces to recover experimental signal without the engineering complexity that has kept a well-understood technique out of production at many companies. Because his design sits in the data layer rather than the assignment path, it is stack agnostic. Any online marketplace with exposure telemetry and a similar dilution problem can follow the same pattern, guardrails included. For marketplaces operating personalized and conditionally rendered experiences, it offers a way to recover experiment signal while preserving the statistical safeguards of the original randomized test.
By publishing the architecture, implementation considerations and statistical guardrails behind the system, Agarwal has made this capability accessible to practitioners confronting the same problem across e-commerce and other digital products.
Every trend in digital product development like more personalization, more conditional rendering, more progressive disclosure widens the gap between who is assigned to an experiment and who actually sees it. The assumption underneath most experimentation programs is eroding a little further every quarter.
Companies that keep reading experiments at the topline will keep discarding their best ideas without ever knowing they did. The ones that treat exposure as a first-class part of measurement will be working from a sharper picture of reality using the same data, the same platform and the same tests they are already running.
