Subtype Robustness Is Not Just Accuracy: Calibration Under Unseen Subtype Shift
An image classifier can become far less accurate on a new variety of a category it already knows while its confidence falls far less than its accuracy.
The model in this example has to name only the broad category. A ship is still a vehicle and a deer is still an animal, but neither kind of object appeared in training. Such an example is an unseen subtype: a new variety inside a category the model already knows. The deer shows the failure this paper looks for. The model is wrong, and it is as sure of itself as when it is right.
To measure how common this is, we trained image classifiers on some subtypes of each category and tested them on subtypes held out from training. The benchmarks are built from ImageNet, BREEDS, iNaturalist and CIFAR-100, and we tested five architectures (two of them on the low-resolution CIFAR-100). The figure below shows four of the benchmarks: IN-25 (25 broad ImageNet categories), Liv-17 and NL-26 (the BREEDS Living-17 and Nonliving-26 tasks) and iNat-25 (25 iNaturalist orders). For each test image we record whether the predicted category is correct and how much probability the model puts on its answer, its confidence. A model whose average confidence matches its accuracy is called calibrated; the gap between the two is, roughly, its calibration error.
Compare the boxed pairs. On IN-25, accuracy fell from 91.0% on seen subtypes to 60.7% on unseen ones, while average confidence fell only from 93.4% to 82.3%. The other benchmarks follow the same pattern: the orange bars fall further below the blue ones for accuracy than for confidence. The calibration error grew from seen to unseen subtypes for every model on every benchmark.
Lower accuracy alone does not explain this. For ResNet-50 on the four benchmarks in the figure, we blurred, added noise to and otherwise corrupted images of seen subtypes until accuracy fell by the same amount as on unseen subtypes (on iNat-25, whose accuracy drop is small, only a few corruption types qualified). At that matched loss of accuracy, the corrupted images lowered confidence by 17.3 points on average, compared with 9.9 points for unseen subtypes. Confidence falls when an image looks damaged, much less when a normal-looking image shows an unfamiliar kind of object.
These models learned only the broad category labels and started from random weights. In a control on IN-25, starting from an ImageNet-pretrained network made confidence more responsive and reduced the gap, but did not remove it. Recalibrating confidence on familiar subtypes also narrowed the gap without closing it. For a classifier that will meet new varieties of known categories, an accuracy check needs a confidence check next to it.