NAACL 2022

ConfliBERT: A Pre-trained Language Model for Political Conflict and Violence

Yibo Hu, MohammadSaleh Hosseini, Erick Skorupa Parolin, Javier Osorio, Latifur Khan, Patrick T. Brandt, Vito D'Orazio

ConfliBERT is a language model trained on about 7 billion words of text about conflict, politics and diplomacy. Tested on 18 tasks from 12 conflict-research datasets, it scored higher than the general-purpose BERT on every one.

Political scientists who track protests, attacks and diplomacy have more news than they can read, and labeling examples by hand takes expert time. ConfliBERT is a model the same size as BERT-Base that learned the language of this field before seeing any labeled task. Its training text comes from United Nations and other organization reports, think tanks, government records such as the Foreign Relations of the United States, decades of international news, and Wikipedia articles on politics and war.

Training text, compared with BERT and other field-specific models
Table of BERT-style models and their training text. BERT: Wikipedia and books, 3.3 billion words, 16 GB. ConfliBERT: organization and government reports and news, 7 billion words, 34 GB.
Green box: ConfliBERT, trained on about twice as much text as the original BERT. Adapted from Table 1 of the paper; highlights added.

The table also shows the two ways such a model can be built. “Cont” continues training an existing BERT on the new text; “SCR” trains a new model from scratch with a vocabulary drawn from it. We built ConfliBERT both ways (“Both”). We then fine-tuned each version on 18 tasks that conflict researchers actually perform: finding political news and protest reports, sorting terrorist attacks and police actions by type, telling past events from planned ones, tagging crime categories, coding interactions between political actors, and pulling perpetrators, victims and weapons out of reports. Every result is the average of ten runs.

ConfliBERT’s gain over BERT on each of 18 tasks
Horizontal bar chart with one bar per task, all to the right of zero, ranging from about 0.1 to about 3.2 points. Most bars are dark (p below 0.05 or 0.01). Three light bars, SATP-BC, InSightC. and MUC-4, are boxed in red.
Each bar is one task; darker bars mean stronger evidence that the gain is not chance. Red boxes: the three tasks where the gain was not statistically significant. Adapted from Figure 1 of the paper; highlights added.

The figure compares the uncased from-scratch ConfliBERT with the better of the two BERT versions (with and without capital letters) on each task. Bar length is how many points higher ConfliBERT scored on the task’s F1 measure, out of 100; the shading gives the p-value of that difference. Every bar points right: ConfliBERT scored higher on all 18 tasks, from a fraction of a point to about three points, and the difference was statistically significant at p<0.1 in all but three. The largest gains came on police-event reports from India and on extracting entities from reports about the conflict in Syria and Iraq (IndiaP. and Re3d).

The paper also finds that the advantage grows when only a few labeled examples are available, which is the usual situation in conflict research. ConfliBERT is publicly released, so researchers can fine-tune it for their own English-language conflict data.