Under review

Conformity Breaks Conformal Prediction

Yibo Hu, Hanyu Su

A model’s shortlist of possible answers can be calibrated to contain the right answer 90% of the time when the model answers alone. When the model first reads scripted peers who all back the same wrong answer, that promise no longer holds, and it fails worst on uncertain questions the shortlist used to cover.

Suppose a model faces a four-option science question and, instead of committing to one option, returns a short list: options B and C. If the right answer is C, the list covers it. A statistical method called conformal prediction sets one probability cutoff, using questions the model answered on its own, so that lists built with that cutoff cover the right answer on a promised share of questions, here 90%. That share is the list’s coverage. We study the standard version of the method, which keeps every option whose probability clears that single cutoff. The promise assumes the model scores questions at use time the way it did during calibration.

We kept the questions and the cutoff fixed and changed only what the model read before answering: a transcript in which six peers each assert an answer, in lines like “Mary: I think this answer is (B).” The transcript is text we wrote, not a discussion among running models. When all six peers asserted the same wrong answer, coverage fell from 90% to 74%, scoring four open-weight models one by one against a shared cutoff on ARC-Challenge and TruthfulQA. When the peers were split, or all asserted the right answer, coverage stayed above 90%. The question was the same in every case; what moved was the probability the model gave the right answer.

To see where the losses land, the paper follows a specific slice of questions, the targeted band: the tenth of questions on which the model’s solo confidence in the right answer sat just above the cutoff. These are the most uncertain questions the shortlist still covers. Questions below the cutoff were mostly uncovered to begin with, so they have nothing left to lose.

Bar chart of coverage. POOLED: gray bar about 0.87, blue bar about 0.76, red bar about 0.47, boxed in red. Gemma-9B: gray about 0.87, blue bar on the 90% line, boxed in green, red about 0.71.
Dashed line: the promised 90%. Gray: targeted band without peers; blue: all questions with unanimous wrong peers; red: targeted band with those peers. Red box: four models pooled. Green box: Gemma-9B, whose overall coverage stays at 90%. Adapted from Figure 2 of the paper; highlights added.

The figure treats the four models as one system, the way a deployer would certify it, so its overall number differs from the 74% above. In the POOLED group, the blue bar, overall coverage under wrong peers, is 76%: below the promise, but not alarming. The gray and red bars show the targeted band. Without peers, 87% of those questions were covered; with wrong peers, 47% (red box). Gemma-9B shows the pattern most plainly: its overall coverage sits right on the 90% line (green box) while its targeted band falls well below it.

Those uncertain questions are the ones a shortlist is supposed to help with most. Someone who controls the peer messages does not need the answer key to find them: picking the questions the deployed system itself reports as least confident works. A common safety rule, act automatically when the shortlist shrinks to one option and ask a person otherwise, can then act on the peers’ wrong answer instead of asking.

The calibration method itself did not fail; its assumption did. The guarantee holds when calibration and use look alike, and peer messages change how the model scores the same question. Calibrating on answers given under peer pressure restores coverage, but only if one knows in advance which inputs carry pressure, and it works by widening the lists until the system hands almost every decision to a person. A guarantee measured on a model answering alone has to be checked again once the model reads other agents.