Under review

Subtype Robustness Is Not Just Accuracy: Calibration Under Unseen Subtype Shift

Hanyu Su, Carlota Julbe i Juanola, Yibo Hu

An image classifier can become far less accurate on a new variety of a category it already knows while its confidence falls far less than its accuracy.

Two broad categories, VEHICLE and ANIMAL. The model trains on airplane and automobile for VEHICLE and on bird and cat for ANIMAL, then is tested on a ship and a deer, which it never saw in training. Outcomes. Ship: true VEHICLE, predicted VEHICLE, confidence high, correct. Deer: true ANIMAL, predicted VEHICLE, confidence high, a high-confidence error.
An illustration from the paper. The unseen ship is placed in the right category. The unseen deer is called a vehicle, with the same high confidence. Adapted from Figure 1 of the paper; highlights added.

The model in this example has to name only the broad category. A ship is still a vehicle and a deer is still an animal, but neither kind of object appeared in training. Such an example is an unseen subtype: a new variety inside a category the model already knows. The deer shows the failure this paper looks for. The model is wrong, and it is as sure of itself as when it is right.

To measure how common this is, we trained image classifiers on some subtypes of each category and tested them on subtypes held out from training. The benchmarks are built from ImageNet, BREEDS, iNaturalist and CIFAR-100, and we tested five architectures (two of them on the low-resolution CIFAR-100). The figure below shows four of the benchmarks: IN-25 (25 broad ImageNet categories), Liv-17 and NL-26 (the BREEDS Living-17 and Nonliving-26 tasks) and iNat-25 (25 iNaturalist orders). For each test image we record whether the predicted category is correct and how much probability the model puts on its answer, its confidence. A model whose average confidence matches its accuracy is called calibrated; the gap between the two is, roughly, its calibration error.

Accuracy falls much more than confidence on unseen subtypes
Seen subtypesUnseen subtypes
Bar chart of accuracy on four benchmarks. On IN-25, boxed, accuracy is 91 on seen and 61 on unseen subtypes, 30 points lower. Bar chart of average confidence on the same benchmarks. On IN-25, boxed, confidence is 93 on seen and 82 on unseen subtypes, 11 points lower.
One architecture, ResNet-50 trained from scratch, on four of the benchmarks; CIFAR-100 is not shown. Boxed: IN-25. Adapted from Figure 2 of the paper; highlights added.

Compare the boxed pairs. On IN-25, accuracy fell from 91.0% on seen subtypes to 60.7% on unseen ones, while average confidence fell only from 93.4% to 82.3%. The other benchmarks follow the same pattern: the orange bars fall further below the blue ones for accuracy than for confidence. The calibration error grew from seen to unseen subtypes for every model on every benchmark.

Lower accuracy alone does not explain this. For ResNet-50 on the four benchmarks in the figure, we blurred, added noise to and otherwise corrupted images of seen subtypes until accuracy fell by the same amount as on unseen subtypes (on iNat-25, whose accuracy drop is small, only a few corruption types qualified). At that matched loss of accuracy, the corrupted images lowered confidence by 17.3 points on average, compared with 9.9 points for unseen subtypes. Confidence falls when an image looks damaged, much less when a normal-looking image shows an unfamiliar kind of object.

These models learned only the broad category labels and started from random weights. In a control on IN-25, starting from an ImageNet-pretrained network made confidence more responsive and reduced the gap, but did not remove it. Recalibrating confidence on familiar subtypes also narrowed the gap without closing it. For a classifier that will meet new varieties of known categories, an accuracy check needs a confidence check next to it.