Explainable AI for medical scans: a check, not a diagnosis

Explainable AI, or XAI, is often sold as a way to make a black-box scanner model trustworthy. That oversells the tools. Grad-CAM, attention rollout, and similar overlays highlight regions that influenced a score. They do not certify pathology, and they do not replace a radiologist or pathologist. I treat them as debugging: if the heatmap sits on a ruler, a label, or an empty corner, the pipeline is wrong even if the accuracy is high.

Medical images make this especially sharp. Ultrasound speckle, overlapping grey levels in brain scans, and stain variation on bone-marrow fields all produce pretty maps that can still be meaningless. A published mini-review I wrote with Kok Swee Sim on explainable AI for breast ultrasound sets out current methods and the limits that remain. The short version for teaching is: an explanation is a hypothesis about the model, not a finding in the patient.

Good practice is narrow. Fix the random seed when you generate maps. Compare a correct case with a confident error. Ask whether the highlighted region is anatomically plausible. If two XAI methods disagree, report both rather than picking the prettier picture. Never put a heatmap in a paper or a dashboard without the corresponding image and the stated method.

XAI does not turn an unpublished model into a clinical device. It is one more control, alongside a documented split, an external test set when you can get one, and a sentence that says what the system must not be used for.