Work on AI conformity counts how often a model changes its answer under group pressure. We argued that the count hides what matters and split it in two: changes that break a correct answer, and changes that repair a wrong one. The two directions are nothing alike. Facing a unanimous wrong group, harmful changes climb from 16% to 63%, while helpful changes barely move. Influence is not one dial. It is cheap to use for harm and expensive to use for good.

Every benchmark in this area shows the model an answer together with a peer who endorses it, and nobody had run the control that removes the peer. We built it: no speaker, no authority, no conversation, just a bare candidate answer sitting in the context. Two thirds of the effect survives. These benchmarks are mostly not measuring social influence, so a large literature has been crediting peer pressure for something a stray plausible answer does on its own.

If a model could tell when changing its mind is a mistake, we could suppress only those changes. We tested that idea to destruction: probes on internal states, three independent steering methods, and a reinforcement-learning policy trained to be selective. The probes predict that a model will revise much better than which way, and every method moved harmful and helpful revisions together. Helpfulness and gullibility do not separate along any axis we could find, which says the fix is not a better filter.

Safety systems increasingly ask several models to review content and take a majority vote, on the theory that independent errors cancel out. We built that panel and broke it. One wrong message shown to all six reviewers makes the panel flag every harmless item across six datasets, up from 43%. The mechanism matters: independence is not lost. The message nudges each reviewer the same small amount, and a majority vote amplifies a shared nudge instead of cancelling it. Redundancy defends against independent errors, and a shared context is not one.

Nearly all of this research grades models by having another model act as judge, which assumes the judge is a neutral instrument. We designed an evaluation that measures judge and subject together, on open-ended answers rather than multiple choice. The judge conforms to the same social cues it is grading. Results produced this way carry the instrument's bias inside them.

A network reporting 33% confidence might be looking at evidence pointing several ways at once, or at nothing it recognises at all. Those call for opposite responses, and a single probability collapses them into the same number. We built a classifier on evidence theory that reports them separately, so a system can tell a genuinely hard case from a case it was never built for.

Measuring uncertainty only helps if a system acts on it. We brought the evidence-based approach into text classification and used it to decide what not to answer, generating adversarial out-of-distribution examples during training so the classifier learns where its competence ends. It stays reliable on the inputs that make ordinary classifiers fail silently.

Benchmarks test models either on categories they trained on or on inputs that are visibly broken. We built the case in between. A model learns the category animal from birds and cats, then meets a deer: an ordinary photo, the right category, but a kind of animal it never saw in training. We call that a subtype shift, and it describes most of what deployment actually looks like. Accuracy drops sharply and confidence stays flat, so the model gets much worse with no signal that it has. Recalibrating afterwards closes the gap only if you already know the input is an unseen subtype, which is exactly what a deployed system cannot know.

Content-moderation models are compared on whichever benchmark each paper happened to choose, so the numbers mean little side by side. We built a benchmark that puts seven existing safety datasets behind one protocol and scores every system on identical items, releasing the per-item decisions and confidences. Across six leading models the pattern holds: they say they are sure 94% of the time and are right 74% of the time. Rescaling confidence fixes almost none of that. Letting a model route its shakiest cases to a person fixes a lot.

Defenders cannot patch everything, so one strategy is to fill a system with believable decoy documents and let an intruder burn time on them. Earlier methods swapped nouns and produced text that fell apart on a close read. We mask the parts of a real document that carry the sensitive meaning and infill them in context, so the result stays coherent while being false. We showed people one real research abstract beside three of ours. They could not reliably pick the real one.

Research on multi-agent attacks assumes that some agent misbehaves. We found a failure where none does. Verification status travels as words inside a message, so when a summarizer compresses that message the word unverified is dropped as ordinary redundancy, and the agent downstream reads a bare claim as a checked one. We measured it across nine models and two harm benchmarks, and showed routine memory compression does the same. Verification is a relationship between a claim and the process that checked it, and text cannot carry a relationship.

The standard defence for agent systems is a runtime monitor that inspects every message and tool call. We formalised what that design can never catch, and built a testbed that demonstrates it. Split an attack across agents and each fragment looks more innocent than ordinary traffic while the assembled trajectory is plainly malicious, and locally benign fragments walk past a real deployed injection detector. This is structural rather than a matter of training a better classifier: the harm exists in the composition, and a per-step monitor never sees a composition.

If no single step reveals a distributed attack, can an unfinished run? We built a working distributed-backdoor system, collected 2,082 runs over five models and two task domains, and trained a detector on partial traces. It catches 99% of successful attacks with a median of five steps still left to run, at a cost of 10% false alarms on safe runs. We also show why the obvious version fails: the per-step features that look useful are tracking message length rather than malice.

People ask chatbots real medical questions, so we built a benchmark from one: given what you have already taken today, is another dose safe? It needs tracking of time and dose across a day rather than recall of a fact, and it has a checkable right answer. Models are most confident on exactly the ambiguous cases where the correct move is to refuse. The failure is not being confidently wrong in general. It is a failure to abstain.

Health advice is a place where a wrong answer costs something real, and most of the world does not ask in English. We built a cross-lingual evaluation of health question answering and found the quality of the answer depends on the language it was asked in. Reliability is not a fixed property of a model. It is a property of a model and the people using it.

Political scientists track wars and protests by reading the news and recording each event by hand. Medicine, science and law each had a language model pretrained on their own literature; social science did not. We built one, on 34GB of government reports and conflict news. It beats general-purpose BERT across event classification and entity recognition, and the margin grows as labelled data shrinks, which is the regime this field actually lives in. It has been adopted well beyond our group, including by a United Nations effort that extended it to Spanish and Arabic.

Human coders in this field work from a codebook, a manual defining every category and how to apply it. Instead of collecting thousands of labelled examples, we handed the model that manual, pairing natural-language inference with the definitions the humans already use. It codes political events with no labelled training data, which matters because the categories change every time the research question does.

Having proposed codebook-driven coding ourselves, we went back to check whether it works for the reason we claimed. We wrote a better codebook, watched accuracy rise, then tested whether the model was executing the rules at all: swapping labels, rewriting definitions, probing whether it followed the manual or matched surface patterns. It was not following the manual. For social science that distinction is the whole point, because a number produced by an instrument that ignores the codebook cannot support the claim it is quoted for.

Platforms increasingly ask ordinary users to deal with misinformation, and the evidence on whether that works sat in three separate literatures. We reviewed 88 papers under one framework by separating what the crowd is actually asked to do: label the content, judge whether a correction worked, or write the counter-message. Crowds are cheap and they scale, and each role only pays off under conditions a platform has to design for on purpose.

Billions of people write online by mixing two languages inside one sentence. We built matched pairs that hold the content fixed and change only that surface form, then sent both through moderation pipelines. The decision changes. Whatever these systems key on, it is not only the meaning.