ACL 2024

Leveraging Codebook Knowledge with NLI and ChatGPT for Zero-Shot Political Relation Classification

Yibo Hu, Erick Skorupa Parolin, Latifur Khan, Patrick T. Brandt, Javier Osorio, Vito D'Orazio

The codebook that political scientists write for human annotators can teach a model to classify political relations in news without any labeled training examples, once its category definitions are rewritten as short statements the model can check against a sentence.

Event coding turns a news sentence into a record of who did what to whom. A widely used scheme, PLOVER, sorts each interaction between a source and a target into categories such as REQUEST, PROTEST and THREATEN, and its codebook defines each one for human coders. The definitions are subtle. Students who “said they would stage mass demonstrations” have not protested yet: the codebook calls this a threat.

One sentence, checked against statements from the codebook
A sentence about Indonesian students who said they would stage demonstrations, gold label THREATEN. Plain statements: requested scores 92.7, protested against 92.5, provided aid 0.8. With mode added, threatened to protest against scores 97.3 and gives the correct label.
A news sentence (correct label THREATEN) scored out of 100 against statements written from the codebook. Red box: “requested” and “protested” nearly tie. Green box: a threatened protest scores highest, giving the right label. Adapted from Table 2 of the paper; highlights added.

The model behind these scores was trained for natural language inference: given two sentences, it judges whether the first supports the second, and the score is how strongly it does. It has never seen a PLOVER label. Our system, ZSP, supplies the knowledge by turning each codebook definition into about five statements and asking the model to check them. The red box shows why plain statements are not enough: the sentence supports both “requested” and “protested” almost equally. ZSP therefore decides in three steps. It first keeps the few categories whose statements score highest. It then asks whether the event happened or was only threatened, promised or proposed, which the codebook calls the event’s mode; this step finds the green row. Last, it applies the codebook’s rules for categories that overlap, as in row (c), which drops REQUEST whenever PROTEST is among the top candidates. In the table, <S> and <T> are the source and target, the digit after each label is its broad group (2 = material cooperation, 3 = verbal conflict, 4 = material conflict), and THREAT. is short for THREATEN. The paper calls the sentence the premise and the statements hypotheses.

No labeled examples versus trained models
Line chart of macro F1 against the share of 1,050 training examples. ZSP is a flat dotted line near 82. Trained models start between about 37 and 60 at 10 percent of the data and reach the dotted line between about 30 and 100 percent.
Green box: ZSP, which uses no labeled examples. Other lines: four models trained on 10% up to all 1,050 labeled sentences; the red box marks them at 10%. Adapted from Figure 3(c) of the paper; highlights added.

On 1,033 test sentences, ZSP scored 82.4 in macro F1, an accuracy measure out of 100 that gives every category equal weight. GPT-4, prompted with summaries of the same codebook’s categories, scored 61.5, and Universal PETRARCH, a widely used dictionary-based coder, scored 46.3 even under the most favorable way of scoring it. Models trained on all 1,050 labeled sentences scored between 81.6 and 86.7. In the figure, where each point averages five runs, the trained models fall well below the dotted line when they have only a tenth of the data and the strongest of them, ConfliBERT (CBERT, a model pretrained on conflict text), needs about half of the data to move ahead of ZSP.

Trained models still lead when enough labels exist: ConfliBERT scored highest here, and on a coarser four-group version of the task the trained models stayed clearly ahead. The result concerns one English event-coding scheme. Within it, a codebook written for people carried enough knowledge to stand in for a labeled dataset, and a change to the scheme becomes an edit to the statements or rules rather than a new round of labeling.