Selective prediction: when a medical AI should say "I am not sure"

Most classifiers are forced to answer every time. A selective classifier is allowed to say: I am not sure, please look at this one yourself. The idea is old, and in medical imaging it fits how care already works, because a human reader is part of the process anyway.

The idea in one paragraph

A selective classifier has a "reject option". For each input it either gives a prediction or abstains. The idea goes back at least to Chow, who studied the trade-off between error and rejection in 1970. Two numbers describe the trade:

• Coverage. The share of cases the model answers.
• Selective risk. The error rate on the cases it does answer.

As the model abstains more, coverage falls, and the error on the remaining cases usually falls too. The curve that plots one against the other is the risk coverage curve. The question is never "how accurate is the model", but "how accurate is it at this level of coverage".

How one is built

There are two ingredients.

1. A confidence or uncertainty score for each case. For example the highest class probability, the disagreement among the members of an ensemble, or the spread of repeated predictions with dropout. See aleatoric versus epistemic uncertainty for where such scores come from, and uncertainty quantification explained for the main methods.
2. A threshold. Cases below it are deferred to a human.

Everything depends on the score ranking the cases correctly: the cases it is least sure about should be the ones with the most errors. A poorly calibrated or uninformative score gives you a threshold that abstains on the wrong cases. For the calibration side, see model calibration and ECE explained.

Geifman and El-Yaniv showed how to build a selective classifier from an already trained deep network. The user sets a target risk, and the classifier rejects inputs as needed to keep the risk at that level with high probability. As an example, they report a guaranteed 2 percent top 5 error on ImageNet with 99.9 percent probability, at almost 60 percent test coverage. The example is from natural images, not medicine, but it shows what the trade looks like: a strong guarantee, paid for with coverage.

Why it fits medical imaging

A deferred case does not mean a wrong answer. It means a human reads the case, which is how the work is already organized. The goal is reliable assistance, not automation: the model handles the cases it is confident about, and people spend their attention where it is needed.

Six checks before you claim it works

1. Choose the threshold on validation data. Report results on a separate test set. Never tune the threshold on the test set.
2. Report the whole risk coverage curve. A hand picked point is easy to flatter. Whenever you state an accuracy, state the coverage it comes with: 98 percent accuracy at 40 percent coverage is a very different claim from 98 percent at full coverage.
3. Show that the deferred cases really are harder. The error rate among them should be clearly higher than among the answered cases.
4. Look at who gets deferred. Check by subgroup, such as age, breast density, scanner, or site. Abstention should not pile up on one group.
5. Compare against simple baselines. At least the highest class probability as the score, and random abstention at the same coverage.
6. Describe the workflow honestly. Who reads the deferred cases, and how much extra work does that create? A model that abstains too often is not usable, however good the answered cases look.

What selective prediction does not do

• It does not make the model better on the cases it answers. It makes the answered set easier by removing the hard cases.
• It does not repair a miscalibrated score. If the score is wrong, the threshold is wrong.
• It gives no guarantee when the data shift. A threshold chosen on one hospital's images may behave differently on another's, so test on external data when you can.

This is the question behind much of my own research on breast cancer detection in mammography: not only how accurate a model is, but whether it knows when it should not be trusted. The publications are listed on the research page.