A Survey on the Role of Crowds in Combating Online Misinformation: Annotators, Evaluators, and Creators
Ordinary internet users take on three different jobs in countering online misinformation: they label it, they are the readers whose reactions show whether a correction works, and they write corrections themselves.
This survey organizes the research on those efforts around the three roles. We searched the Scopus database on September 22, 2022 for English-language papers published after 1999, screened the 3,956 papers the search returned, and kept 88 that study crowds working against misinformation.
Annotators mark posts or articles as false or factual. They work alone, with many people’s labels combined afterward; together, as in Twitter’s Birdwatch program (now Community Notes), where contributors attach notes with supporting material and rate one another’s notes; or alongside a machine classifier whose labels they check. Group membership shapes these labels. Allen and colleagues found that Birdwatch contributors tended to annotate tweets from the opposing party negatively and to rate those contributors’ notes as less helpful.
Evaluators are the readers whose reactions show whether a correction works. In lab experiments they read a false claim and a correction and rate how believable the claim is; they also answer interviews and surveys, and their replies, stances and sharing are analyzed afterward. Small choices of wording matter. Pillai and colleagues found that readers shown a bare negation such as “This is wrong.” often remembered the claim and forgot the negation.
Creators write their own corrections in posts, replies and comments. Here researchers study the messages and the people: what the corrections say, how they spread, and who chooses to write them. Giachanou and colleagues found that users who post fact-checks write with more positive and causal language than users who spread false news, who write more informally.
Each role answers a different question and needs a different kind of evidence. Studies of annotators ask who participates and how their labels are combined. Studies of evaluators ask what makes a correction persuasive, and for which audience. Studies of creators ask what volunteers already do on their own. Across all three roles the survey reaches the same conclusion: crowds are far more numerous than professional fact-checkers, but the quality of their labels, judgments and corrections varies, so crowd efforts work best combined with quality control and expert oversight.