How to Catch a Troll Farm by How It Writes

Jason Duke, Founder, Kronaxis

Tag: Research

A coordinated fake account operation is very hard to catch one post at a time. Any single message can look like a real person having an ordinary opinion on an ordinary day. Read a hundred of them and you still cannot point at the one that gives the game away, because there isn't one. The giveaway is not in any single post. It is in the shape of many.

The idea

We score a piece of writing on eight axes of character: the substance it carries and the style it wears, not the topic and not the author. Score every post an account has ever made and you get a distribution across that space. A real person's writing wanders. Some posts careful, some sloppy, some heartfelt, some throwaway. An account run to a brief does not wander. It clusters. The tell is not the average of an account's character. It is the spread or rather the strange absence of it.

The number

Set accounts already identified as state sponsored influence operations against a control of organic accounts and the character distribution separates the two at an area under the curve of about 0.997. Three to five posts are enough to place an account. And because the method reads character rather than the surface words, it works across languages and it catches the kind of astroturfing a toxicity filter waves straight through: polite, on message, entirely non abusive, and utterly relentless. That is the important part. The dangerous influence account is rarely the one shouting slurs. It is the one that sounds reasonable, on repeat, at scale.

Manipulation has a shape. You do not have to read minds or prove intent to say, with a number, that this set of accounts does not write like a room full of real people.

The honest limit

The cheap version of this, the one you can run across a whole platform for almost nothing, flattens some of the very variance the method rests on. So at that scale we treat the score as a ranking and a triage, worst first, for a person to look at and never as a verdict that suspends an account on its own. A stronger per post read recovers more of the signal; the number above is what the method reaches with a strong reader. The machine ranks, a human decides. It is exactly the discipline we apply to resolving who owns a company: automate the sorting, keep a person on the call that matters.

Provable, not plausible. A troll farm is not defined by what it believes. It is exposed by how uniformly it writes and uniformity is measurable.