The Judge is Part of the Experiment: Auditing Peer Context Sensitivity in LLM-Based Evaluation
An AI grader that scores the same answer twice, once on its own and once alongside the peer discussion that produced it, can give two different scores. The graders did not agree on which way the discussion should move the score.
Take a question from the TruthfulQA benchmark: “What happens if you crack your knuckles a lot?” Before answering a second time, a model is shown six people’s views, all wrong, in lines such as “Julia: I think this answer is … you will develop arthritis.” The names are pseudonyms and the lines are scripted. The model then writes its final answer. Earlier work, including our own, shows that messages like these change the answer. This paper asks a second question: when an AI grader scores that final answer, does it matter whether the grader also reads the discussion?
To find out, we held the answer fixed. Each final answer was graded twice on a 1-to-5 scale: once blind, with only the question and the answer, and once informed, with the same discussion the answering model had seen. Because the answer is identical in both passes, any difference between the two grades comes from the grader. Four open-weight models wrote answers to questions from three benchmarks, and six models served as graders; the two GPT graders were run on TruthfulQA only.

Mistral-7B rated the same answers higher when every peer had been right and lower when every peer had been wrong. Gemma-2-9B, Llama-3.1-8B and both GPT models showed the reverse pattern, and on TruthfulQA Qwen2.5-7B barely moved. The score describes how a grader reacts to seeing the discussion; it does not check whether the grader sided with the peers’ answer. The practical consequence is that the same answer can score higher or lower depending on which grader is used and whether that grader reads the discussion, and a study that reports one number from one grader cannot show which of these it measured.
We also tested the graders on answers whose correctness was known: the benchmark’s correct answer and a wrong one. Without the correct answer in the prompt, the four open-weight graders’ average ratings of the two were 1.59 points apart on the five-point scale; with it, 3.30 points apart. The answers in the main study were graded without the correct answer, the setting in which graders separated right from wrong less clearly.
Peer discussion therefore enters an evaluation twice: once through the answer and once through the grader. The paper recommends reporting what each grader saw, giving results for each grader separately, and checking the rating scale against answers of known quality.