Research Blog

Social Influence and Collective Decision-Making

Language models now read one another's answers, vote together and grade each other's work. We measure how this contact changes their answers, and when agreement among models is no longer a reliable signal.

Monitoring and Security in Multi-Agent Systems

When AI agents pass work to one another, an attack can be split across many steps. We study attacks and failures that no single agent or single check can detect, and how monitoring can catch them.

Reliability under Uncertainty and Distribution Shift

Does an AI model know what it does not know? We study when a model's confidence matches how often it is right, how that confidence breaks down under distribution shift, when new data differ from the data the model was trained on, and how to build models that report their own uncertainty.

AI for Health and Public-Interest Domains

People ask chatbots about medication, and social scientists use AI to measure conflict and politics. We evaluate whether these systems are reliable enough for that use, and build tools for it.

EACL 2027

When Better Codebooks Are Not Enough: Predictive Performance and Behavioral Reliability in LLM Political Event Coding

A richer codebook raised a model’s score at coding political news, but not once category names were replaced by neutral placeholders.

Read more →
Under review

Can I Take Another Dose? DoseBench: Evaluating LLM Decision-Making in OTC Dosing QA

Two chatbots with about the same accuracy on dosing questions differ sharply in how often they approve a dose the label rules out.

Read more →
ACL 2024

Leveraging Codebook Knowledge with NLI and ChatGPT for Zero-Shot Political Relation Classification

Rewritten as short statements a model can check, a political science codebook classifies political relations without labeled training data.

Read more →
WWW 2024

Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries

Asked the same health questions in four languages, GPT-3.5 answered less completely and less consistently outside English, worst in Hindi.

Read more →
ACM TKDD 2024

A Survey on the Role of Crowds in Combating Online Misinformation: Annotators, Evaluators, and Creators

Ordinary users fight misinformation in three roles: labeling it, showing by their reactions whether corrections work, and writing them.

Read more →
NAACL 2022

ConfliBERT: A Pre-trained Language Model for Political Conflict and Violence

Trained on about 7 billion words about conflict and politics, ConfliBERT scored higher than BERT on all 18 conflict-research tasks tested.

Read more →