A distributor needs to order stock six weeks before it arrives. Two forecasting systems perform well in a supplier demonstration. One has the lower average error across the catalogue, while the other handles the distributor’s busiest products more reliably. Choosing between them requires a clear account of what will be predicted, when the prediction is made, and how the forecast will influence an order.
Predictive accuracy is relevant evidence. It becomes useful when the comparison matches the decision the organisation needs to make. A model should not win because it received information unavailable to its competitor, was tested at an easier horizon, or benefited from a metric that conceals expensive errors. A fair evaluation makes those choices visible before the results are interpreted.
Specify the Forecasting Task Precisely
For the distributor, define the quantity: weekly units demanded for each product at each location. Specify the forecast horizon and when orders are committed. A next-day prediction cannot establish six-week planning performance, and a forecast of total company sales may not support item-level replenishment.
Decide whether the historical target measures demand or observed sales. If a product sold out, recorded sales may understate what customers wanted. A model evaluated only against sales can appear accurate while reproducing the effect of limited availability. The team needs to identify censored periods and decide how those cases will be treated in training and evaluation.
Define how new products, discontinued lines, promotions, returns, and missing records enter the comparison. Excluding them can be reasonable for a bounded first release, but the resulting scope must be explicit. Report what proportion of real planning work falls outside it and how the remaining items will be handled.
Record the decision rule too. An order might use a forecast alongside current inventory, expected deliveries, service targets, and supplier constraints. This lets the team distinguish prediction error from a failure in the stock policy or inventory record. The model is one input to that complete process.
Recreate the Information Available at the Time
Evaluation should respect the moment each prediction would have been made. A forecast produced in early March can use information known then, including scheduled events that were already available. It cannot legitimately use later sales totals, a corrected inventory value entered in April, or the realised effect of a promotion that had not yet occurred.
This is the core of leakage prevention. Information can enter through features, preprocessing, selection, or repeated tuning on evaluation results. The scikit-learn guidance on common pitfalls explains why transformations should be fitted using the appropriate training data rather than learning from a held-out test set.
Preserve historical availability where possible. A current data warehouse may contain corrected values without showing when the corrections became known. Using those values in a historical test can make the past easier than the real operating task. If the records cannot establish availability, state the limitation and test sensitivity to plausible delays or missing inputs.
Leakage is not simply the presence of seasonal patterns. A model can legitimately learn that demand rises near a recurring holiday from earlier years. The problem is giving it future information that would not have been available at the prediction date, not allowing it to learn recurring relationships.
Use Time-Based Comparisons and Simple Baselines
Evaluate successive forecasting origins: train on earlier information, predict the required horizon, then advance the origin and repeat. This tests several historical operating periods rather than relying on one unusually easy holdout. Keep the evaluation design the same for all candidates and reserve a final period from model selection where practical.
Hyndman and Athanasopoulos describe time-series cross-validation in their forecasting textbook. The important condition is that each test forecast uses earlier observations. A random split can be unsuitable when it gives the training process access to future periods relative to the observations being predicted.
Include a simple baseline, such as the most recent value or the corresponding seasonal value where that makes sense. Also compare with the current planning method. A complex system that improves on a weak demonstration baseline may still add little to an experienced planner’s existing process.
Historical evaluation is informative, but it cannot establish performance under every future change. Include periods representative of important operating conditions and describe what the history lacks. A forecast tested only during stable demand needs further evidence before the organisation depends on it through a major product or market change.
Choose Error Measures That Fit the Task
Classification accuracy counts the proportion of correct class predictions. Demand forecasts usually predict quantities, so they require measures of numerical error. A report describing a forecast as “95% accurate” needs to explain exactly how that percentage was calculated and over which cases and horizon.
Mean absolute error averages the size of forecast errors in the target’s units. Root mean squared error gives larger errors more weight through squaring before averaging and taking a square root. Neither is universally superior. The choice should reflect which differences in errors matter, alongside a report of important failure cases.
Consider an illustrative two-period demand sequence of 100 and 100 units. Model A predicts 80 and 120, while model B predicts 100 and 135. Model A has mean absolute error of 20 units. Model B has 17.5, but its root mean squared error is about 24.7 units, compared with 20 for A. The ranking changes because the second measure penalises B’s concentrated larger error more heavily.
Percentage errors can help compare scales but become problematic when actual demand is zero or very small. Avoid silently dropping those periods, especially for intermittent-demand products. The evaluation should state the handling rule and report affected coverage. A combined catalogue measure also needs an explicit weighting scheme so that the result’s relationship to business volume is clear.
Examine Where and When the Models Fail
Break results down by forecast horizon, product group, location, demand level, and other relevant operating conditions. The distributor may care particularly about the six-week horizon for items with inflexible lead times. A good average across one- and two-week forecasts cannot substitute for that evidence.
Assess uncertainty in the comparison. Candidate models evaluated on the same cases should be compared on those paired results, with attention to dependence between periods and products. A small observed difference may be unstable across evaluation windows. Report variation and the amount of evidence rather than declaring a winner from an extra decimal place.
Review large and persistent errors. A model that consistently underpredicts one product family creates a different planning problem from one with occasional balanced errors. Examine whether failures arise from the model, a missing promotion schedule, an inventory correction, or a mismatch between the target and the ordering process.
Distribution changes do not guarantee deterioration, and a fixed model does not automatically decay each month. Monitor the relevant performance and input conditions so that changes can be investigated. Historical tests support a bounded conclusion about expected use, with uncertainty about conditions they did not represent.
Evaluate Uncertainty Alongside the Point Forecast
A point estimate of 100 units conceals the range of plausible demand. For replenishment, the cost of too little stock may differ from the cost of excess inventory. A suitable predictive distribution, quantile, or interval can inform that decision if its reliability has been evaluated for the task.
Check coverage and usefulness together. An interval that contains most outcomes because it is extremely wide may offer little planning value. An apparently precise interval that frequently misses realised demand is misleading. Examine performance at the required horizon and across relevant product types, with sample-size limitations stated.
For classification applications such as predicting late deliveries, probability calibration is a related consideration. Among comparable cases assigned a probability near 0.7, the event should occur at roughly that frequency. The scikit-learn calibration guide distinguishes probability reliability from other predictive properties. Good ranking does not by itself make probabilities suitable for cost-based decisions.
Do not treat statistical uncertainty as a complete account of risk. A model may not capture an unrepresented supplier shutdown or a new policy. Planning still needs procedures for events outside the evaluated conditions and for missing or unreliable source data.
Compare the Resulting Decisions Before Scaling
Apply each candidate to the same ordering policy where feasible, then compare stockouts, excess inventory, service levels, and operating costs under explicit assumptions. A historical simulation can identify consequences, but it needs care around lead times, stock constraints, and demand that was not observed because items sold out.
A shadow run can test current data availability, latency, integration, and planner interpretation without making the candidate responsible for orders. It cannot fully demonstrate how decisions will affect customer outcomes when the proposed actions are not executed. A controlled, bounded live trial may provide that further evidence when appropriate.
The AI success-metrics article follows this transition from prediction quality to service outcomes. Include the cost of operating each candidate, the effort needed to review exceptions, and the ability to recover when forecasts are unavailable. The AI ROI guide provides a consistent way to account for those costs and benefits.
Choose the model that meets the defined requirements with a defensible balance of predictive evidence, operational fit, and cost. Record the uncertainty and the conditions that trigger reassessment. The comparison then supports an actual stock decision, rather than an abstract contest in which the lowest reported error wins regardless of what it means.
