ConfliBERT: A Pre-trained Language Model for Political Conflict and Violence
ConfliBERT is a language model trained on about 7 billion words of text about conflict, politics and diplomacy. Tested on 18 tasks from 12 conflict-research datasets, it scored higher than the general-purpose BERT on every one.
Political scientists who track protests, attacks and diplomacy have more news than they can read, and labeling examples by hand takes expert time. ConfliBERT is a model the same size as BERT-Base that learned the language of this field before seeing any labeled task. Its training text comes from United Nations and other organization reports, think tanks, government records such as the Foreign Relations of the United States, decades of international news, and Wikipedia articles on politics and war.
The table also shows the two ways such a model can be built. “Cont” continues training an existing BERT on the new text; “SCR” trains a new model from scratch with a vocabulary drawn from it. We built ConfliBERT both ways (“Both”). We then fine-tuned each version on 18 tasks that conflict researchers actually perform: finding political news and protest reports, sorting terrorist attacks and police actions by type, telling past events from planned ones, tagging crime categories, coding interactions between political actors, and pulling perpetrators, victims and weapons out of reports. Every result is the average of ten runs.
The figure compares the uncased from-scratch ConfliBERT with the better of the two BERT versions (with and without capital letters) on each task. Bar length is how many points higher ConfliBERT scored on the task’s F1 measure, out of 100; the shading gives the p-value of that difference. Every bar points right: ConfliBERT scored higher on all 18 tasks, from a fraction of a point to about three points, and the difference was statistically significant at p<0.1 in all but three. The largest gains came on police-event reports from India and on extracting entities from reports about the conflict in Syria and Iraq (IndiaP. and Re3d).
The paper also finds that the advantage grows when only a few labeled examples are available, which is the usual situation in conflict research. ConfliBERT is publicly released, so researchers can fine-tune it for their own English-language conflict data.