Research: measuring how much disagreement is about words

This is a plan in progress. Nothing here is a finding yet. We will publish the method, the results and the mistakes as we go.

The question

Of all disagreements, what share are really about words? And in how many of those do the people involved never realize it? The second number is the heart of the idea behind this publication. We also want to know whether the share differs between partners, coworkers, companies and their customers, and public debate.

As far as we can find, nobody has published a figure. Philosophers call verbal disputes “ubiquitous”, but that’s a claim, not a measurement.¹²

How we plan to measure it

The unit we count is a disagreement episode: a stretch of conversation where two or more people take positions they treat as incompatible.

For each episode, the plan is to:

1. Find the disagreement.
2. Find the pivot terms: the words, phrases or slogans each side might be using differently.
3. Run the elimination test: restate each side’s position without the pivot term, and judge whether the restated positions still conflict. This is David Chalmers’s “method of elimination”.¹
4. Classify what’s left: a disagreement about facts, about values, or about how a word should be used.

We also record whether the participants noticed the meaning problem themselves: neither noticed, one noticed and used it, or both noticed. The headline number we’re after is the share of word-based disagreements where nobody noticed.

Steps 2 to 4 will be done by language models, using prompts we publish.

What would make the number credible

• People first. In each setting, at least two people will label a few hundred episodes independently, and we’ll report how often they agree. If people can’t agree on the labels, the categories need fixing before any model is used.
• Models are checked against people. We only scale up once a model agrees with human labelers about as well as they agree with each other.
• More than one model, with a report of how much the headline number moves between them.
• A check for bias toward “you actually agree”. We’ll include disagreements we know are real, and check that the models don’t explain them away.
• Everything open: prompts, code, labeled examples and results.

An experiment

The strongest evidence would come from an experiment. Pairs of people who disagree have a structured conversation. One group discusses normally. The other first does a taboo round: each person restates their position without the key term. We then measure how much the disagreement shrinks in each group. We’ll also ask people beforehand whether they think their disagreement is about words.

Where things stand

We have run a first small pilot on 50 episodes from formal settings (the US Congress, Intelligence Squared debates and the Supreme Court), with model labels only and no human labels yet. In 22 of the 24 episodes that had a word-based component, the participants did notice the term problem. That cuts against “nobody notices”, at least in formal, adversarial settings where speakers are trained to scrutinize terms. It may be the setting, how the excerpts were cut, or a bias in the model’s labels. Next come human labels, then a pilot on informal conversation at work and at home.

Known limits

• What counts as a disagreement changes the answer, and online arguments over-represent some kinds of dispute.
• Labels involve judgment. Even people won’t always agree on what’s verbal. We’ll report that disagreement rather than hide it.
• Privacy. Public posts are still written by people. We’ll report results in aggregate and take care before quoting anyone.