Leveraging Codebook Knowledge with NLI and ChatGPT for Zero-Shot Political Relation Classification
The codebook that political scientists write for human annotators can teach a model to classify political relations in news without any labeled training examples, once its category definitions are rewritten as short statements the model can check against a sentence.
Event coding turns a news sentence into a record of who did what to whom. A widely used scheme, PLOVER, sorts each interaction between a source and a target into categories such as REQUEST, PROTEST and THREATEN, and its codebook defines each one for human coders. The definitions are subtle. Students who “said they would stage mass demonstrations” have not protested yet: the codebook calls this a threat.
The model behind these scores was trained for natural language inference: given two sentences, it judges whether the first supports the second, and the score is how strongly it does. It has never seen a PLOVER label. Our system, ZSP, supplies the knowledge by turning each codebook definition into about five statements and asking the model to check them. The red box shows why plain statements are not enough: the sentence supports both “requested” and “protested” almost equally. ZSP therefore decides in three steps. It first keeps the few categories whose statements score highest. It then asks whether the event happened or was only threatened, promised or proposed, which the codebook calls the event’s mode; this step finds the green row. Last, it applies the codebook’s rules for categories that overlap, as in row (c), which drops REQUEST whenever PROTEST is among the top candidates. In the table, <S> and <T> are the source and target, the digit after each label is its broad group (2 = material cooperation, 3 = verbal conflict, 4 = material conflict), and THREAT. is short for THREATEN. The paper calls the sentence the premise and the statements hypotheses.
On 1,033 test sentences, ZSP scored 82.4 in macro F1, an accuracy measure out of 100 that gives every category equal weight. GPT-4, prompted with summaries of the same codebook’s categories, scored 61.5, and Universal PETRARCH, a widely used dictionary-based coder, scored 46.3 even under the most favorable way of scoring it. Models trained on all 1,050 labeled sentences scored between 81.6 and 86.7. In the figure, where each point averages five runs, the trained models fall well below the dotted line when they have only a tenth of the data and the strongest of them, ConfliBERT (CBERT, a model pretrained on conflict text), needs about half of the data to move ahead of ZSP.
Trained models still lead when enough labels exist: ConfliBERT scored highest here, and on a coarser four-group version of the task the trained models stayed clearly ahead. The result concerns one English event-coding scheme. Within it, a codebook written for people carried enough knowledge to stand in for a labeled dataset, and a change to the scheme becomes an edit to the statements or rules rather than a new round of labeling.