EMNLP 2022 · Findings

Controllable Fake Document Infilling for Cyber Deception

Yibo Hu, Yu Lin, Erick Skorupa Parolin, Latifur Khan, Kevin Hamlen

A decoy document that rewrites only chosen phrases of a real one was as hard for readers to pick out as text written freely by a language model, and it changed the original by a moderate amount instead of drifting away from it.

One defense against data theft is to store a real document beside convincing fakes, so that an intruder who takes them all must work out which one is genuine. A useful fake has two jobs: it has to pass as real, and it has to change the details that matter. Here is one research abstract and two kinds of fake.

One abstract, two kinds of fake
Original, with the phrases our method chose to replace…We first construct a nonnegative dictionary based on prototype elements from the training images; this problem is formulated as a regularized non-negative matrix factorization. Incorporating the dictionary as a prior in a convex reconstruction problem, we then find an approximate solution with a sparse representation in the dictionary…
Fake from our method…We first construct a collection of missing patches based on images from the training images; this problem is formulated as a regularized non-negative matrix factorization. Incorporating the dictionary as a prior in the whole dictionary, we then find a similar estimate for missing patches in the dictionary…
Fake written freely by GPT-2…We describe and examine an algorithm for tomographic image reconstruction where prior knowledge about the solution is available in the form of training images. Instances were reconstructed from their images using image and pixel centroids. The concept of image reconstruction provides several advantages over previous techniques, such as indexing the solution to a representation with integral or submaximal number of cepstrates, …
Excerpts from one research abstract; every version keeps its title and first sentence. Blue: phrases chosen for replacement. Red: new text. Rows E, F and B of Table 1 of the paper, set as text; wording and colors as printed.

Our method, Fake Document Infilling, picks important phrases, blanks them out, and has a language model trained to fill blanks write a replacement using the text on both sides of each gap. During generation it is discouraged from writing the original phrase back. The result keeps the structure and topic of the abstract while the method itself (a dictionary, an approximate solution) is swapped for something plausible but wrong. GPT-2, fine-tuned on each dataset and given only the opening, writes on fluently and moves away from the original.

We tested the fakes on research abstracts, patent abstracts and Wall Street Journal news stories with more than 40 computer science students. Each reviewer saw one real document and three fakes made by one method and tried to pick the real one. Averaged over the three datasets, they found it 91% of the time when the fakes came from random word edits, 67% for GPT-2 and 66% for our method. On news stories alone, the GPT-2 fakes were harder to spot than ours. But a fake can be hard to spot simply because it no longer resembles the original, so we also asked how much each fake changed. Reviewers compared each fake with the known original and rated it F1 inadequate (only insignificant changes), F2 marginal (changes that leave the facts largely intact), F3 moderate (important changes that stay coherent and realistic) or F4 excessive (so many changes that the fake drifts from the original topic or facts). In the figure, EDA is random word edits, WEF is WE-FORGE, an earlier method that swaps nouns, GPT is GPT-2 writing freely, and FDI is our method.

How much each method changed the original
Stacked bar chart of fakeness ratings for four methods, about 82 samples each. GPT is mostly excessive (red box). FDI is mostly moderate (green box). EDA and WEF are spread across all four levels.
Bars count fakes per method at each rating, from F1 (inadequate) to F4 (excessive). Green box: our method’s moderate (F3) ratings. Red box: GPT-2’s excessive ratings. Adapted from Figure 2 of the paper; highlights added.

This is where the two kinds of fake part ways. Reviewers rated 61.0% of our method’s fakes as moderate, the level they also preferred overall, while most GPT-2 fakes changed too much. Control over which phrases change is meant to keep a decoy close enough to the original to be believed and far enough from it to hide the real details; in this test it did both. These results concern short English documents judged by students.