A manufacturer tests a vision model that identifies damaged packaging. The demonstration includes a coloured overlay highlighting areas associated with a damage prediction. Most of the colour falls near a torn corner, which seems reassuring. On another image, it appears around the inspection station’s date stamp. The prediction is correct in both cases, but the second image raises a question about what the model has learned.
Visual explanations can help investigate that question. They do not turn a model into a transparent description of its own knowledge, and a convincing overlay does not establish that a prediction is correct. Their practical value comes from generating specific hypotheses that the team can test against the model, the images, and the production task.
Establish What the Picture Represents
Start with the predicted outcome and the explanation method. A heatmap might represent sensitivity to small pixel changes, the effect of covering image regions, or a class-related signal derived from internal feature maps. These are different measurements. A colour scale without a method and target is insufficient to interpret the result.
Grad-CAM, for example, uses gradients flowing into a convolutional layer to produce a coarse localisation map for a target concept. It can indicate image regions relevant to that target under the method. It is not a precise segmentation of the defect, a record of every computation, or a probability that each highlighted pixel is damaged.
For the packaging model, ask whether the explanation targets the damage class, the predicted class, or another output. Show the original image beside the overlay. Record how colours are normalised and whether the scale is comparable across images. If every map is independently rescaled to fill the colour range, an intense colour in one image need not represent a stronger effect than a pale region in another.
Keep these details available in the investigation record. Otherwise two reviewers can reach different conclusions because they assume the same visual convention means the same thing across methods and examples.
Follow the Image Through the Actual Pipeline
The explanation needs to describe the image the model received. Production pipelines may crop, resize, rotate, compress, or normalise an image before inference. An overlay displayed on the original photograph can mislead if its coordinates do not match those transformations. A defect outside the model’s crop cannot have contributed as visible image evidence to that prediction.
Record the source image, preprocessing configuration, model version, output, explanation method, and relevant settings. For the packaging station, include the camera and lighting setup where these affect interpretation. The point is to reconstruct the evaluated case, with appropriate controls on any sensitive material, rather than to collect every available record indefinitely.
Check basic implementation errors first. A heatmap assigned to the wrong image, a reversed class label, or a display that stretches a low-resolution map incorrectly can produce a plausible-looking result. Verifying the data path may resolve an apparent modelling problem before anyone begins a more complicated investigation.
This complements the broader guide to testing explanation evidence. For images, geometry and preprocessing are part of that evidence, alongside the prediction and the explanation algorithm.
Use Unexpected Regions as Testable Leads
The highlighted date stamp suggests a possible shortcut. Perhaps damaged packages were photographed on one day and intact packages on another. The date could then correlate with the label without representing damage. That is a hypothesis about this dataset, not a conclusion established by the overlay alone.
Inspect how images and labels were collected. Were rejected items photographed at a separate station? Did one operator handle most defects? Were multiple photographs of the same package split between training and evaluation? These details can create relationships that look useful in testing but fail when the collection process changes.
Examine groups of examples, including correct predictions, missed defects, and false alarms. A single unusual map can reveal a problem, but it cannot establish how common the problem is. Compare across cameras, packaging types, shifts, and relevant defect categories. Include difficult intact packages so the investigation does not focus only on visually obvious damage.
A model can also use legitimate context. A region near a seam might contain evidence of a tear even when the strongest colour does not cover the tear itself. Ask an inspection specialist what information is available there before labelling every unexpected region a spurious feature.
Test Whether the Map Responds to the Model
A useful explanation should depend on the behaviour it claims to describe. A map that looks similar after the model’s learned parameters are randomised deserves scrutiny. In Sanity Checks for Saliency Maps, Adebayo and colleagues tested dependence on model parameters and training labels and found that some methods failed those checks in the evaluated settings.
The result matters because visual plausibility is a weak validation method. Image edges and object outlines can look meaningful even when they reveal little about the trained predictor. A sharp, attractive map is not necessarily a more faithful explanation than a coarse one.
Run appropriate checks on the explanation pipeline in a controlled environment. Compare targets where their behaviour should differ, assess what changes when model parameters change, and verify that the displayed result belongs to the intended output. The checks need to match the method; they are evidence about particular properties, not a universal certificate of explanation quality.
Also assess sensitivity to ordinary input changes. A small brightness adjustment should be evaluated against both the prediction and the map. If an explanation changes substantially while the prediction is stable, investigate whether the method is locally sensitive, whether the display normalisation exaggerates the difference, or whether the model is using different evidence.
Design Perturbations That Answer a Useful Question
Covering the date stamp and measuring the prediction can help test whether that region matters. However, replacing it with a black rectangle introduces a new visual feature. If the score changes, the model might be responding to the unnatural patch as well as the removed information. Interpret the result in light of how the intervention changed the image.
Where practical, collect matched photographs under controlled conditions. Photograph intact and damaged packages with varied dates, backgrounds, and station arrangements. This can separate the defect from collection artefacts more convincingly than editing one image. The experimental design should preserve the features that inspection genuinely requires.
| Investigation | Evidence it can provide |
|---|---|
| Hide or replace a region | Sensitivity to the chosen image intervention |
| Vary the camera or background | Performance under a specified environmental change |
| Inspect collection and label records | Possible shortcuts or mistaken labels |
| Evaluate a corrected model on held-out cases | Whether the change improves the intended task |
Do not confuse a successful image intervention with a full causal account of the model’s internal computation. It can support a narrower finding: changing a particular part of the input under specified conditions changed an output. That may be enough to guide the next investigation step.
Choose a Fix Based on the Failure
If the collection process created a date-related shortcut, the remedy might involve collecting balanced examples, revising the image pipeline, or excluding irrelevant metadata from the camera view. If the problem is a mislabeled defect category, the team needs a label correction and a review of affected examples. If the overlay is displayed incorrectly, retraining the predictor will not repair the interface.
Cropping out an unwanted region can be useful, but verify that it does not remove legitimate inspection evidence or create a fragile dependence on camera alignment. A new crop is part of the deployed pipeline and needs evaluation with the same care as a model change.
After any correction, evaluate predictions on held-out cases that were not used to design the fix. Check missed defects, false alarms, and performance across relevant conditions. A heatmap that looks more sensible is supporting information; improved inspection performance is the outcome the manufacturer needs.
Preserve failure examples as regression cases. They help test whether later releases reintroduce the same issue, but they should not become the entire evaluation set. A team can learn to solve its favourite demonstration images while remaining unreliable on future packages.
Decide What the Reviewer Actually Needs
An engineer investigating shortcuts may need several maps and configuration details. A production inspector may need the original image, the proposed defect category, a route to inspect the item, and clear controls for accepting or rejecting the suggestion. More visualisation is not automatically more useful at the inspection station.
Test the review interface with correct and incorrect predictions in a safe evaluation. Can inspectors find real defects, reject false alarms, and recognise when the image is insufficient? Compare the workflow with a suitable baseline. Higher confidence in the model is not an adequate substitute for improved inspection outcomes. The article on ergonomic AI interfaces examines how to make that review practical.
The team should be able to report what the visual explanation suggested, how the hypothesis was tested, and what the investigation established. Sometimes the result will be an actionable data problem; sometimes the evidence will remain inconclusive. Either is more useful than treating a coloured overlay as the model’s final word about itself.
