A retailer introduces AI-generated product recommendations. Over the following quarter, customers who interact with them spend more than customers who do not. The dashboard labels their purchases as AI-generated revenue, and the project appears to have a substantial return.
Those customers may already have been more likely to buy. Some may have responded to a promotion, while others would have found the same products through search. The retailer has measured revenue associated with the feature without establishing how much the feature added.
Assessing revenue impact requires a comparison with what would probably have happened under an alternative experience. That comparison needs to be planned alongside the implementation, with a clear account of what is being tested and how confidently the evidence can support a causal claim.
Defining How the Intervention Could Create Value
The retailer should begin with a specific hypothesis. Recommendations might help returning customers discover relevant products, leading to additional purchases or fewer abandoned visits. That account connects a technical capability to behaviour the business can observe.
The team can then examine the links. Are suitable recommendations available? Do customers notice them? Does discovery change, and does the change affect purchases? Each question helps locate where the proposed mechanism is working or breaking down.
A click is useful diagnostic evidence without proving a revenue benefit. A customer may click a recommendation and buy an item they would otherwise have found elsewhere. Similarly, higher order value may coexist with fewer customers placing an order.
The business case should therefore identify the outcome to assess and the supporting measures needed to interpret it. For this retailer, that might include revenue per eligible customer over a defined period, with conversion and order composition helping explain the result.
Comparing Experiences Fairly
Where feasible, a randomised experiment can assign eligible customers to the recommendation experience or the existing experience. Assignment should persist appropriately so that repeated visits do not expose one customer to conflicting experiences.
The analysis should compare the assigned groups rather than selecting only customers who clicked the feature. Clicking is behaviour the treatment may influence; restricting the analysis to clickers can select a group that differs from the comparison for other reasons.
The choice of assignment unit matters too. Microsoft Research discusses the challenges of tenant-level randomisation, illustrating how the structure of a product affects experiment design. A consumer retailer may assign individual customers, while a business service may need to consider interactions within an account.
The team should establish its primary outcome, comparison period, sample needs, and analysis approach before interpreting results. An analyst can help assess whether the design can detect a commercially meaningful effect and whether customers’ interactions create spillovers between groups.
Reporting an Incremental Estimate
Consider a hypothetical test in which the recommendation group averages $52 in revenue per assigned customer and the comparison group averages $50 over the same period. The observed difference is $2 per customer. It does not establish that the feature created every purchase in the recommendation group.
The estimate also has uncertainty. The analysis needs to show whether the result is precise enough to support the decision and whether data quality or experiment implementation problems affect its interpretation.
The duration matters. A short promotion might bring purchases forward rather than increase spending over a longer period. Returns and cancellations may also arrive after the initial transaction. The reporting window should reflect the effect the business wants to understand.
The dashboard should distinguish observed revenue, estimated incremental effect, and any projection to a broader rollout. Extending a result to other customers or seasons introduces assumptions that should remain visible.
Working With Less Controlled Comparisons
Randomisation may be impractical for a system that changes inventory decisions across the business or affects a shared sales process. The team then needs an evaluation design suited to those conditions.
The Magenta Book’s analytical methods guidance describes approaches to impact evaluation, including comparison-based and theory-based methods. Their suitability depends on the intervention and the assumptions the available evidence can support.
A staged rollout may provide a useful comparison, but the rollout order alone does not make the groups equivalent. Regions selected early may have stronger managers, different customers, or other changes underway. Historical trends and concurrent events need examination.
A before-and-after revenue increase can still be informative. It becomes a stronger causal claim only when the analysis credibly addresses alternative explanations. If pricing and marketing changed at the same time, the report should state what can and cannot be separated.
Recognising Contribution Within a Wider Change
Some initiatives deliberately combine AI with cleaner data, revised workflows, training, and new commercial policies. The organisation may be able to assess the combined intervention more credibly than the isolated model.
If the retailer introduces recommendations alongside a redesigned product page, a comparison against the old page estimates the effect of that package. It does not separately identify which element caused the improvement. Separating those effects would require a design that varies the relevant components.
UK guidance on evaluating AI interventions discusses using different evidence sources and theory-based approaches to investigate how change occurred. A contribution claim should be supported by an examined mechanism and consideration of other explanations.
This can produce a useful conclusion without assigning an exact revenue share to the model. The organisation can explain that the new operating arrangement improved results, describe evidence of the AI component’s role, and identify what remains uncertain.
Continuing From Revenue to Economic Return
Even a credible revenue improvement needs a cost assessment. Recommendations may promote discounted items, increase fulfilment expenses, or create returns. The retailer should examine incremental margin and the costs of delivering the changed experience.
Relevant costs include inference, data services, integration, evaluation, monitoring, maintenance, and any human review. A pilot with extensive developer support should not assume that support will disappear at scale without evidence.
Other AI initiatives may have a different value mechanism. A support assistant might release staff capacity rather than produce direct sales. Multiplying time saved by salary rates estimates a value for that time; it does not establish an equivalent reduction in expenditure.
The organisation should record what the capacity enabled, such as reducing a backlog or accommodating growth without planned hiring. This keeps the financial claim connected to what actually changed instead of treating all benefits as interchangeable revenue or savings.
Using Measurement to Guide the Next Decision
The initial goals should identify what result would justify expansion, further investigation, revision, or closure. The threshold depends on expected value, cost, uncertainty, and the consequences of a mistaken commitment.
If recommendation quality improves but customer behaviour does not, further model work needs a reason to expect a different result. If sales improve but margin falls, the objective or recommendation rules may need revision. If the estimate is too uncertain, a better-designed test may be more useful than a larger deployment.
Measurement should continue after rollout where practical. Customer behaviour, catalogue quality, and operating costs can change, so an earlier result should not be assumed permanent.
For the retailer, useful attribution explains the additional outcome the intervention plausibly created, the strength of the comparison, and the cost of producing it. That gives leadership a basis for investment without giving AI credit for every transaction in which it appeared.
