EACL 2027 · Findings

Easier to Mislead Than to Correct: Harmful and Beneficial Revision in LLM Conformity

Jiaming Qu, Lucheng Fu, Yibo Hu

Compared with mixed peers, six scripted peers who all named the same wrong answer raised how often models abandoned a correct answer far more (15.6% to 62.9%) than six peers naming the right answer raised how often they fixed a wrong one (32.7% to 51.5%).

Each model first answered a multiple-choice question on its own. It then saw the question again with six peer replies that we scripted, each only a name and an answer, in lines such as “Mary: I go with (B).” The replies gave no reasons. In the baseline, which the figure calls mixed, three peers named the correct answer and three named different wrong options. In the test conditions all six named the correct answer, or all six named the same wrong one. We ran four open-weight models on seven question sets, three times each.

Schematic: a model that was wrong sees six peers with the right answer and becomes correct (beneficial revision); a model that was right sees six peers with the same wrong answer and becomes wrong (harmful revision). Chart: the red harmful line rises from 15.6% to 62.9%, boxed in red; the green beneficial line rises from 32.7% to 51.5%, boxed in green.
Top: a beneficial revision (wrong to right) and a harmful one (right to wrong). Chart: how often each happens with mixed peers (left) and unanimous peers (right); schematic, not to scale. Red box: harmful; green box: beneficial. Adapted from Figure 1 of the paper; highlights added.

Read the chart from left to right. Compared with mixed peers, unanimous wrong peers raised the share of correct answers that became wrong from 15.6% to 62.9%. Unanimous correct peers raised the share of wrong answers that became correct from 32.7% to 51.5%. The red line climbs far more steeply. In the paper’s regression, measured in odds, the effect of all-wrong peers on harmful revision is about five times the effect of all-correct peers on beneficial revision (28.5 against 5.2).

The two rates are measured on different questions. A harmful revision can only happen on a question the model first got right, a beneficial one only on a question it first got wrong, and those are likely harder, which may limit how much peers can help. The peers also gave bare answers; a correct answer with no explanation may not give a model enough to solve a question it missed. The paper offers both as possible explanations of the gap, not as tested causes.

Following the crowd did not make the models more sure. Their self-reported confidence after revising under unanimous peers was slightly lower than after revising under mixed peers: the answer changed without the model claiming more certainty in it.

In a system where models read one another’s answers, agreement in this setting spread a wrong answer more readily than it repaired one. The paper’s recommendation is that multi-agent systems check peer answers rather than simply count them.