A heatmap is not a confidence score: why medical AI needs explainability and uncertainty

A radiologist opens an AI result for a mammogram. A red heatmap sits on the suspicious region. It looks convincing. It also says nothing about whether the model is right. Explainability and uncertainty answer two different questions, and a system that offers only one of them leaves the most important question open: should I trust this particular result?

Two questions, two toolkits

Explainability methods give an account of a single prediction. In imaging, the usual output is a heatmap laid over the image. Grad-CAM is a common choice, and LIME and SHAP are others. Uncertainty quantification attaches a measure of doubt to each prediction. You will meet Monte Carlo dropout, deep ensembles, Bayesian neural networks, and conformal prediction.

My own research keeps returning to one question: when can we trust what a model tells us? Neither toolkit answers it alone.

Why a heatmap alone can mislead

A heatmap is easy to trust because it looks like reasoning. Two findings should slow you down.

• Explanations can disagree. Rafferty and colleagues applied LIME, RISE, and SHAP to a model that classifies mammograms as benign or malignant. They found that the three techniques consistently disagreed with each other and with the medical truth. When your tools disagree, the picture is not evidence.
• Some maps do not depend on the model. Adebayo and colleagues randomized the weights of trained networks and checked whether the saliency maps changed. For some popular methods, they barely did. Grad-CAM was not one of them: its maps changed as they should.

The lesson is not that heatmaps are useless. A heatmap should pass a test before it earns trust, and passing that particular test is necessary, not sufficient.

Why a confidence number alone can mislead

The other half of the trust problem is the number the model gives you. Modern neural networks tend to be overconfident, as Guo and colleagues showed. A prediction of 95 percent does not mean the model is right 95 percent of the time.

The gap has a name: calibration. A model is calibrated when its confidence matches how often it is actually right. Expected calibration error is the usual summary. It is also the idea behind the name of this site: confidence that matches the evidence.

Two kinds of uncertainty

Not all doubt is the same, and the difference changes what you do about it.

• Aleatoric uncertainty comes from the data itself. Dense breast tissue can hide a lesion, and a poor image is a poor image. More training data does not remove it.
• Epistemic uncertainty comes from what the model has not learned. A case unlike the training data, a new scanner, a different population. More diverse data can reduce it.

The first says the image may not contain the answer. The second says the model should not be trusted here. For a fuller map of the field, see the survey by Gawlikowski and colleagues.

The methods you will meet

• Monte Carlo dropout. Keep dropout switched on at test time, pass the image through several times, and look at how much the answers vary. It was proposed by Gal and Ghahramani. It is cheap and works on an existing model, but it is an approximation.
• Deep ensembles. Train several models and compare their answers. Often reliable, at the price of training and running several networks.
• Bayesian neural networks. Treat the weights as distributions instead of single values. Principled, and usually the most expensive.
• Conformal prediction. Instead of one label, return a set of plausible labels, sized so that the true label is included at a chosen rate. CONFINE is one example, and it adds example-based explanations to the prediction sets.

For a plain comparison of these methods, their cost, and how to test them, see uncertainty quantification explained for researchers.

More expensive is not automatically better. In the screening study below, the uncertainty metrics that took more computation did not separate good predictions from bad ones any better.

A worked example from breast screening

The clearest evidence I know for why uncertainty matters comes from a screening study, not from a benchmark. Verboom and colleagues built a model that outputs a probability of malignancy and a measure of its own uncertainty. The rule was simple. When the model was confident, it decided on recall. When it was not, two radiologists read the examination as usual. They tested this retrospectively on screening examinations from the Dutch national programme.

• Radiologists still read 61.9 percent of examinations. The model alone handled the rest.
• Performance held. The recall rate was 23.6 per 1000 examinations, against 23.9 for standard double reading, and the cancer detection rate did not differ significantly.
• Confidence meant something. The model's AUC was 0.96 on the examinations it was sure about and 0.87 on the others.
• The gap was large where it mattered. On the uncertain examinations, the model reached a sensitivity of 44.4 percent while radiologists reached 63.6 percent at the same specificity. On the certain ones, the model matched the radiologists.

The model on its own was less sensitive than radiologists. Uncertainty still made it useful, because it told the workflow which cases to keep away from it. The uncertain examinations were also more often from dense breasts, the same cases that are hard for people.

The limits matter as much as the result. The study was retrospective, at a single screening unit with mammography systems from a single vendor. The intervals around the sensitivity figures are wide, because cancers are rare. And reading fewer examinations does not automatically mean less reading time. The authors call for a prospective trial.

How the two fit together

This is my view, not a trial result. I would design the output in a fixed order. First the uncertainty. Then, if the model is confident, the explanation. If the model is not confident, the case goes to a human, and a heatmap on that case is a hypothesis, not evidence. Put the number before the picture.

What I check when a paper claims trustworthy AI

These are the questions I bring to a manuscript that uses XAI, UQ, or both.

The general version of these checks is in how to read a research paper like a reviewer.

Scope of this note

This is a practical guide, not a systematic review. It covers the classification of mammograms and screening workflows. Every study mentioned is listed below with its reference.