ACM TKDD 2024

A Survey on the Role of Crowds in Combating Online Misinformation: Annotators, Evaluators, and Creators

Bing He, Yibo Hu, Yeon-Chang Lee, Soyoung Oh, Gaurav Verma, Srijan Kumar

Ordinary internet users take on three different jobs in countering online misinformation: they label it, they are the readers whose reactions show whether a correction works, and they write corrections themselves.

This survey organizes the research on those efforts around the three roles. We searched the Scopus database on September 22, 2022 for English-language papers published after 1999, screened the 3,956 papers the search returned, and kept 88 that study crowds working against misinformation.

Box labeled Crowd Annotator with four news-item icons marked False!, Fact, False!, Fact.

Annotators mark posts or articles as false or factual. They work alone, with many people’s labels combined afterward; together, as in Twitter’s Birdwatch program (now Community Notes), where contributors attach notes with supporting material and rate one another’s notes; or alongside a machine classifier whose labels they check. Group membership shapes these labels. Allen and colleagues found that Birdwatch contributors tended to annotate tweets from the opposing party negatively and to rate those contributors’ notes as less helpful.

The annotator box from the paper’s overview figure: news items labeled false or factual. Adapted from Figure 2 of the paper.
Box labeled Crowd Evaluator with icons labeled Comments/Replies, Interview and Survey.

Evaluators are the readers whose reactions show whether a correction works. In lab experiments they read a false claim and a correction and rate how believable the claim is; they also answer interviews and surveys, and their replies, stances and sharing are analyzed afterward. Small choices of wording matter. Pillai and colleagues found that readers shown a bare negation such as “This is wrong.” often remembered the claim and forgot the negation.

The evaluator box from the paper’s overview figure: comments and replies, interviews, and surveys. Adapted from Figure 2 of the paper.
Box labeled Crowd Creator with icons of chat bubbles, a web page being edited and a frowning face, labeled Counter-Misinformation.

Creators write their own corrections in posts, replies and comments. Here researchers study the messages and the people: what the corrections say, how they spread, and who chooses to write them. Giachanou and colleagues found that users who post fact-checks write with more positive and causal language than users who spread false news, who write more informally.

The creator box from the paper’s overview figure: posts, replies and edited content that counter misinformation. Adapted from Figure 2 of the paper.

Each role answers a different question and needs a different kind of evidence. Studies of annotators ask who participates and how their labels are combined. Studies of evaluators ask what makes a correction persuasive, and for which audience. Studies of creators ask what volunteers already do on their own. Across all three roles the survey reaches the same conclusion: crowds are far more numerous than professional fact-checkers, but the quality of their labels, judgments and corrections varies, so crowd efforts work best combined with quality control and expert oversight.