EACL 2027

When Better Codebooks Are Not Enough: Predictive Performance and Behavioral Reliability in LLM Political Event Coding

Zixian He, Bharath Raahul Murugesan, Patrick T. Brandt, Yibo Hu

Giving a language model a richer codebook raised its score at coding political news. It did not make the model do better once the category names were replaced by neutral placeholders, the test of whether a model is working from the definitions it was given.

Three sentences about an Algerian activist and the Algerian security forces. 'Announced that he would begin a hunger strike' is coded THREATEN; 'staged a hunger strike' is coded PROTEST; 'ended his hunger strike' is coded YIELD.
Three example sentences from the paper. Boxed: the verb that decides the label, and its label at the codebook’s finest level (Root); Binary and Quad are coarser levels. Adapted from Figure 2 of the paper; highlights added.

Political scientists turn news reports into data with codebooks: fixed categories, each with a written definition and rules for borderline cases. The paper illustrates the PLOVER codebook with the three sentences above, about the same activist and the same security forces. All three mention a hunger strike, yet announcing it is a THREATEN event, staging it is PROTEST, and ending it is YIELD. The category names are ordinary English words, but in the codebook each one means what its definition says.

We gave four open-weight models (Gemma-2-9B, Qwen2.5-7B, Mistral-7B and Llama-3.1-8B) 1,033 annotated news examples, each with a marked source and target, and asked for one of the codebook’s 15 root categories. Scores in this post are F1 scores, from 0 to 1, which reward a model for finding the cases of each category without mislabeling other cases as that category. With the real category names, more codebook content raised the score. Averaged equally over the 15 categories (macro-F1), a compact codebook with short definitions, each category’s coarser group and a few general instructions scored 0.574. An enriched codebook that adds examples and rules for confusable categories scored 0.633.

A score like this cannot say what the model used. A model may choose PROTEST because it read the definition, or because the word “protest” fits the sentence. To separate the two, we replaced every category name with a neutral placeholder such as LABEL_1 and left each definition word for word as it was. The correct answer becomes the placeholder attached to the matching definition. A model that works from the definitions should hardly notice the change. With placeholders we report weighted F1, which counts each category in proportion to how often it occurs.

Ranked with the real names, ranked with placeholders
Two bar panels. With real names (macro-F1): Enriched 0.633, RAG 0.588, Compact 0.574, ICL 0.490, CoT 0.477. With placeholder names (weighted F1): Enriched 0.519, RAG 0.522, Compact 0.566, ICL 0.152, CoT 0.625.
Means over four open-weight models. Top: macro-F1 with real names. Bottom: weighted F1 with placeholders. The scores differ, so compare the order of methods within a panel, not values across panels. Redrawn from Tables 2, 3 and 4 of the paper.

Follow the red bars. With the real names, the enriched codebook ranked first. With placeholders it fell below the compact codebook, 0.519 against 0.566 weighted F1, and RAG, which retrieves parts of the same enriched codebook, did likewise. The richer codebook raised the score without making the models work more from its definitions. The green bars run the other way: step-by-step reasoning over the enriched codebook (CoT) ranked last with the real names and first with placeholders, at 0.625. In-context learning (ICL), which gives one labeled example per category, is the extreme case. It never saw the full definitions, and with placeholder names it fell to 0.152, so its correct answers had depended largely on the familiar names.

These are averages over four models, and the paper reports the gap between the enriched and compact codebooks as a descriptive difference. Two API models, GPT-4o-mini and GPT-5.4-mini, also kept high scores with placeholder names, and for them the enriched codebook scored slightly higher than the compact one. The test checks one thing: whether correct answers survive once the names lose their everyday meaning. It does not show that a model applies every rule in the codebook. A study that uses a language model as a coder can run this check alongside its usual evaluation, since it needs only the codebook and the existing labeled examples.