When Does AI Find the Real Disagreement?

Abstract

When two people disagree, they often misinterpret what they disagree about. If AI can help properly identify the actual root of disagreement (the crux), then AI can help streamline negotiations and bridge policy disputes.

This preliminary study asks how often AI models identify the point a disagreement really turns on, its crux, and whether the structure of the prompt changes the result. Four models, three from Anthropic and one from OpenAI, answered 12 AI-written conversations using five prompt types, three times over: 720 answers, each scored by two AI judges against a crux written in advance.

• The AI found the crux, either singling it out or naming it as one of several disagreements, in 450 of 720 answers (63%). The four prompt types that asked where the two people disagree found it most often, and almost equally: 90 to 92 of 144 answers each. The prompt that asked only for a summary of each person's view found it least often: 85 of 144.
• The AI singled out the crux as the main disagreement in 352 of 720 answers (49%). Here the order reversed: the summary-only prompt singled it out most often, in 85 of 144 answers. The prompt that first asked what the two people agree on singled it out least often, in 49 of 144.
• When the AI did not find the crux, its answer usually still stated the facts that reveal it (203 of 270 answers), and it most often repeated the people's own description of their disagreement.
• In the conversations with no real disagreement, only 12 of 120 answers stated that the two people don't really disagree.

This is a preliminary study, with a small set of AI-written conversations and AI judges only. Its results suggest questions for larger studies, described in section 6. To view all 12 conversations, every AI answer, and both judges’ scores, see the study materials page.

1. Background: misplaced disagreements in legal and policy negotiations

1.1 Hypothetical example legal and policy disputes

When two sides disagree, they can misinterpret what they actually disagree about. They may argue about the facts when they actually accept the same facts but differ on the definition of a key term, or the parties may argue about values when they actually have different accounts of the facts.

When this kind of misinterpretation happens, the disagreement is harder to resolve. In policy debates, it raises tension and mutual opposition. In negotiations, it stalls progress toward a deal.

The one point a disagreement really turns on is often called its crux.

To illustrate, here is a hypothetical legal negotiation, followed by a hypothetical policy negotiation. In each case, an AI that can find the crux could be a crucial partner in speeding up the negotiations and increasing the likelihood of a successful resolution.

A legal negotiation: a hospital and its software vendor (hypothetical example #1)

• The surface fight. A hospital is suing its software vendor for $4 million, claiming the new records system went live five months late. The vendor says the delay was the hospital's fault, because the hospital sent its patient data late. For months, both sides' lawyers argue about who caused the delay, trading emails, schedules and witness accounts.
• The crux. Buried beneath the complexity of the negotiations and contract terms is the fact that both sides accept the same timeline: when the data arrived, when the system was switched on, when the hospital started using it. What actually divides them is one word in the contract, "acceptance." The vendor reads it as the day the hospital started using the system in March. The hospital reads it as signing off after 30 days without errors, which happened in August. Delay penalties run until acceptance, so that one definition is the whole $4 million.
• Why finding it matters. Arguing about fault means expensive document searches and depositions, and possibly a trial, all on a question that won't settle the case. Once both sides see the dispute is about what one clause means, they can settle it quickly: ask a mediator or judge to rule on just that clause, or split the difference on the penalty period.

A policy dispute in Congress: a skilled-worker visa bill (hypothetical example #2)

• The surface fight. A bill would expand visas for foreign tech workers. Supporters say it fills real labor shortages. Opponents say it undercuts American workers' pay. For weeks, each side cites studies showing the other is wrong, and the debate hardens into "pro-business vs. pro-worker."
• The crux. Buried beneath the complexity of the regulations, data, and history of this issue, is the fact that both sides rely on the same Labor Department wage data. Supporters look at national figures, where visa holders earn above the median. Opponents look at entry-level jobs in specific cities, where visa holders earn below what local workers in the same role are paid. Each is right about the group they're looking at. The real question is which wage floor the bill sets: a national average, or local pay for the same job and experience level.
• Why finding it matters. "Shortage vs. undercutting" has no middle ground; you're for the program or against it. "Which wage benchmark" is a drafting choice that can be negotiated. A bill that ties the wage floor to local pay for the same experience level could answer the opponents' concern while keeping the expansion the supporters want, which could give it a path to passing.

1.2 Existing research on the source of disagreements

In 1907, the philosopher William James described an argument he walked into on a camping trip. A squirrel clings to one side of a tree trunk, and a man stands on the other side. The man walks quickly around the tree, but the squirrel moves just as fast, always keeping the trunk between them. Does the man go round the squirrel? His companions had argued it to a standstill. James pointed out that it depends on what you mean by "go round." If you mean passing from north of the squirrel to its east, south and west, the man does go round it. If you mean being in front of the squirrel, then to its right, behind it and to its left, he does not, because the squirrel keeps turning so that it always faces him.¹ Everyone agreed on what happened. The dispute was over a word, so no amount of looking at the facts again could settle it. Philosophers call arguments like this "verbal disputes."²

A well-known method for resolving disagreements, "double crux," is built around two people finding that point together.³ However, the crux isn't always what people think it is. In our study, a disagreement could be misplaced in three ways:

• A word used two ways. Two people argue about whether a drug trial showed that a drug "works." They agree on every result. One means it lowered cholesterol, which it did. The other means it prevented heart attacks, which it did not measurably do.
• Different groups or places. Two people argue about whether it is legal to record a phone call without telling the other person. One has New York in mind, where one person's consent is enough. The other has California in mind, where everyone on the call must agree. Each is right about their own state.
• Different values. Two people argue about lowering a town's speed limit from 35 to 25 miles per hour. Their conversation focuses on whether it would make the street safer. In fact, the conversation also shows that both accept the town's traffic study: about 20% fewer injury crashes, at a cost of about one extra minute per trip through town. They differ on how much that safety gain is worth against drivers' time.

Sometimes people name their disagreement correctly. And sometimes there is no real disagreement at all: two people hold the same view but stress different halves of it, and each assumes the other disagrees.

Can AI help by identifying the real basis of a disagreement? If it can, that is an important step toward building workable bridges between two or more sides. AI has some natural advantages here: it can read a whole conversation at once, it has no stake in either side, and it is available at the moment people need it, whether in a negotiation, a policy debate or a dispute at work. To help, though, the AI has to find the crux itself, not just summarize the arguments around it. That is what this study tests.

Related work. Earlier studies bear on why this task may be hard for AI and how AI might help. AI assistants tend to agree with what a user already believes, a tendency called sycophancy;⁴ repeating the people's own description of their disagreement may be a form of it. Models also struggle with questions that rest on a false assumption, such as asking when a restaurant chain is shutting down when it is not;⁵ asking "Where do they disagree?" about two people who don't really disagree is a question of this kind. On the other side, an AI mediator built to find common ground wrote group statements that participants in the UK preferred to those written by human mediators.⁶ Finally, this study relies on AI judges, which agree well with human judges in other settings but have known biases, including favoring answers from their own model.⁷

2. Study design

2.1 The 12 conversations

Each test case is a conversation of 350 to 450 words between two fictional people arguing over a yes-or-no question, such as "Should the town of Millbrook build a second elementary school now?"

• The answer key came first. Before any conversation existed, we wrote its key: the crux; how the two people themselves describe their disagreement; two or three points both accept; a small side dispute they settle along the way, as a decoy; and one suggestion aimed at the crux.
• AI then wrote the conversations. GPT-6.1 Sol, from OpenAI, wrote each conversation from its key, so the Claude models being tested did not write their own test.⁸
• Human check. The author read every conversation against its key and approved all 12; two were rewritten once before approval. No conversation was kept or dropped because of how an AI answered it.

The 12 conversations cover the kinds of case described in section 1:

• 6 where the people misplace their disagreement. They say they disagree about one thing, usually the facts, but their own words show they agree on that item and differ on something else: a word used two ways, different groups or places, or values.
• 4 where the people name their disagreement correctly: a fact, a prediction, or values.
• 2 with no real disagreement, though it sounds like one.

Each question and its key is listed in section 7. The full conversations, all 720 answers and both judges' scores are on the study materials page.

2.2 The five prompt types

Every request began "Here is a conversation between two people:" followed by the conversation, then one of five requests, each in a fresh chat. We gave each a short name, used throughout this paper:

1. Summary only: "Summarize what each person is saying."
2. Summary, then disagreement: "Answer in two parts: 1. Summarize what each person is saying. 2. Where do they disagree?"
3. Agreement first: "Answer in two parts: 1. What do they agree on? 2. Where do they disagree?"
4. Disagreement only: "Where do they disagree?"
5. Disagreement, then agreement: "Answer in two parts: 1. Where do they disagree? 2. What do they agree on?"

The bridge question. After the AI answered, a second message in the same chat asked: "What one thing could most help them find agreement?" The first request didn't mention it, so it couldn't shape the first answer.

2.3 Three hypotheses, set in advance

Before running the study, we wrote down three hypotheses, each comparing two prompt types that differ in only one thing. Fixing the comparisons in advance matters: five prompt types make ten possible pairs, and picking the largest gap after seeing the results would overstate it.

1. Agreement first helps. The first hypothesis was that the AI would identify the crux more often when it first has to say what the two sides agree on (agreement first) than when it is asked only where they disagree (disagreement only). The idea: clearing away the shared ground should expose what is left.
2. Summaries hide it. The second hypothesis was that the AI would identify the crux less often when it only summarizes what each person says (summary only) than when it is also asked where they disagree (summary, then disagreement).
3. Order matters. The third hypothesis was that asking the same two questions in a different order (agreement first, against disagreement, then agreement) would change how often the answer is correct, not just its style.

We also set two further comparisons in advance: summary only against disagreement only, and summary, then disagreement against disagreement only. That makes five comparisons in all.

How we decided whether a difference was real. For each pair, we looked at each of the 12 conversations separately, set aside those where the two prompt types came out the same, and counted the difference as real only if nearly all of the rest favored the same prompt type: for example 6 of 6, 7 of 7 or 10 of 12. This is a standard statistical test (the sign test, at the usual 5% level).

2.4 The models and the three runs

We tested four models: Claude Opus 5.5, Claude Sonnet 5 and Claude Haiku 4.5 from Anthropic,⁹ and GPT-6.1 Sol from OpenAI.⁸ We sent no effort, thinking or temperature setting, so each model ran at its provider's defaults; the only setting was a limit of 16,000 tokens on each answer. Under those defaults:

• Claude Opus 5.5 reasons before answering, at medium effort.
• Claude Sonnet 5 reasons before answering, at high effort.
• Claude Haiku 4.5 answers without extended reasoning.
• GPT-6.1 Sol reasons before answering, at medium effort.

The two judges (section 2.5) ran at the same defaults.

• One run means every model answering every conversation with every prompt type, once: 12 conversations × 5 prompt types × 4 models = 240 answers, each in its own fresh chat and each followed by its bridge question.
• Three runs. AI answers vary from one try to the next, so we did the full run three times with identical prompts and settings, for 720 answers in all. The repeats show how consistent each result is. The runs were done one after another; the decision to add each new run was made without access to the results so far.
• Pilot rounds. Before the three runs, two small pilot rounds, on 2 and then 4 of the conversations, were used to settle the prompts and scoring rules. Their answers are not included.
• How the calls were made. Claude Code, Anthropic's coding agent running Claude Opus 5.5, ran every call: the answers, the bridge questions and the judging.¹⁰ Each call went directly to the model's API (Anthropic's for Claude, OpenAI's for GPT-6.1 Sol) as a separate conversation with no other instructions.

2.5 The judges and the three outcomes

Two AI judges from two different companies, Claude Sonnet 5 and GPT-6.1 Sol, scored every answer. Each judge saw the conversation, its answer key, the AI's first answer and its bridge answer. No human judges were used.

The judges placed each answer in one of three outcomes:

• Singled out the crux. The answer presents the crux as the disagreement: either the only one, or clearly the main one. An answer that mentions several disagreements but says the crux is the one that matters counts here.
• Named the crux as one of several disagreements. The answer names two or more disagreements side by side, the crux among them, without saying which one matters most.
• Did not find the crux. The crux is not among the disagreements the answer names.

For the two conversations with no real disagreement, the "crux" is that the two people don't really disagree.

2.6 The two measures of finding the crux

We report two measures. Found the crux means the answer either singled out the crux or named it as one of several disagreements. Singled out the crux means the first outcome only. An answer counts as singled out only when both judges said so, and as found only when both judges said singled out or one of several.

When one judge said singled out and the other said one of several, the answer counts as one of several (37 of the 720 answers, or 5%).

2.7 How well the two judges agreed

The two judges scored each of the 720 answers separately. On the main question, whether the answer singled out the crux, they gave the same verdict on 665 of the 720 answers (92%) and different verdicts on the other 55 (8%).

• Agreement beyond chance. Some of that 92% could be luck. Each judge said "singled out" for about half of the answers, so two judges guessing at random would still agree on about half of them. Cohen's kappa, the standard measure of agreement between two raters, corrects for this: it is 0 when two raters agree no more often than chance would produce and 1 when they agree perfectly.¹¹ We calculated it in four steps:
  1. How the 720 answers split. Both judges said the answer singled out the crux for 352 answers, and both said it did not for 313. Only Claude Sonnet 5 said yes for 14 answers, and only GPT-6.1 Sol said yes for 41.
  2. Actual agreement. The judges gave the same verdict on 352 + 313 = 665 of the 720 answers: 92.36%.
  3. Agreement expected by chance. Claude Sonnet 5 said yes to 366 of the 720 answers (50.83%) and GPT-6.1 Sol to 393 (54.58%). Two judges saying yes at those rates at random would both say yes 50.83% × 54.58% = 27.75% of the time and both say no 49.17% × 45.42% = 22.33% of the time, so chance alone would produce 27.75% + 22.33% = 50.08% agreement.
  4. Kappa. Kappa is the share of the possible improvement over chance that the judges actually achieved: (92.36% − 50.08%) ÷ (100% − 50.08%) = 42.28 ÷ 49.92 = 0.847, or 0.85 rounded.
  A kappa of 0.85 is in the top band ("almost perfect," 0.81 to 1.00) of the most widely used scale for reading it.¹² The two judges worked separately, so this suggests that their agreement reflects reading the answers the same way, not chance.
• Minimizing favoring their own models. An AI judge might go easier on answers written by its own company's models. So we counted how often each judge said an answer singled out the crux, separately for each model's 180 answers:
  • Answers by Claude Opus 5.5: Claude Sonnet 5 as judge said so 109 times, GPT-6.1 Sol 115 times.
  • Answers by Claude Sonnet 5: 81 and 88.
  • Answers by Claude Haiku 4.5: 70 and 77.
  • Answers by GPT-6.1 Sol: 106 and 113.
  GPT-6.1 Sol was slightly more generous, by 6 or 7 answers out of 180, for every model, including its own. Claude Sonnet 5 was no more generous toward the Claude models than toward GPT-6.1 Sol. So neither judge favored its own company's models.

2.8 Additional evaluations from the AI judges

The judges also recorded three more items about each answer:

• What the answer named instead, when it did not single out the crux: the people's own description of their disagreement, a real but secondary point, or a disagreement that isn't there.
• Facts included: whether the answer stated the facts from the conversation that make up the crux, whether or not it connected them.
• Bridge on target: whether the answer to the bridge question was aimed at the crux.

3. Results

3.1 The three outcomes, by prompt type

Of the 720 answers, 352 (49%) singled out the crux, 98 (14%) named it as one of several disagreements, and 270 (38%) did not find it. So, overall, the AI found the crux in 450 answers (63%).

Each prompt type was used 144 times (12 conversations × 4 models × 3 runs). Out of the 144 responses to those prompt types:

• Summary only: 85 singled out the crux, 0 named it as one of several disagreements, 59 did not find it.
• Summary, then disagreement: 72 singled out the crux, 18 named it as one of several, 54 did not find it.
• Agreement first: 49 singled out the crux, 43 named it as one of several, 52 did not find it.
• Disagreement only: 79 singled out the crux, 12 named it as one of several, 53 did not find it.
• Disagreement, then agreement: 67 singled out the crux, 25 named it as one of several, 52 did not find it.

A note on summary only. This prompt asked only for a summary of what each person says, not where they disagree, but the judges scored its answers by the same rules as the others. In practice, they almost never placed a summary in the middle outcome: one judge did so once in 144 answers, the other never. Of the 144 summary-only answers, 99 added a line naming the main point of difference, though the prompt did not ask for one, and 59 of these were counted as singling out the crux. The other 45 only summarized each side; 26 of these were still counted as singling out the crux, because both judges read the two summarized positions as making the crux the clear point of difference. So for summary only, "singled out the crux" means a summary that put the crux at the center, and the score partly reflects how the judges read summaries.

Figure 1 shows how often each way of asking found the crux, and Figure 2 how often each singled it out.

How to read Figure 1. Each full bar counts the answers that found the crux. The darker part counts the answers that singled out the crux, and the lighter part the answers that named it as one of several disagreements. For example, agreement first found the crux in 92 of 144 answers: 49 singled it out, and 43 named it as one of several disagreements.

How to read Figure 2. Each bar counts how many of the same 144 answers singled out the crux. For example, with disagreement only, 79 of the 144 answers singled out the crux, and the other 65 did not.

3.2 The three hypotheses

• Hypothesis 1: asking first what the two people agree on would help. Compared: agreement first and disagreement only.
  • Found the crux: agreement first in 92 of 144 answers (64%); disagreement only in 91 of 144 (63%).
  • Singled out the crux: agreement first in 49 of 144 answers (34%); disagreement only in 79 of 144 (55%).
  • Implication: the hypothesis was not supported. The two prompt types found the crux equally often, and agreement first singled it out less often. That difference in singling out passed the test we set in advance (section 2.3): of the 7 conversations where the two prompt types came out differently, all 7 favored disagreement only. It also appeared in each of the three runs: agreement first singled out the crux in 14, 19 and 16 of 48 answers per run, disagreement only in 27, 27 and 25. We call it strong but not definitive, for two reasons: it met our test only after the third run (after two runs it was one conversation short), and by a stricter standard that allows for our making five comparisons (section 2.3), it falls just short.
• Hypothesis 2: summaries alone would do worse. Compared: summary only and summary, then disagreement.
  • Found the crux: summary only in 85 of 144 answers (59%); summary, then disagreement in 90 of 144 (63%).
  • Singled out the crux: summary only in 85 of 144 answers (59%); summary, then disagreement in 72 of 144 (50%).
  • Implication: the hypothesis was not supported. Summary only found the crux slightly less often, but that small difference did not pass our test. It singled the crux out more often, the opposite of the prediction, and that difference did pass our test: all 7 conversations that came out differently favored summary only. Section 3.1 explains how summaries were scored.
• Hypothesis 3: the order of the two questions would matter. Compared: agreement first and disagreement, then agreement.
  • Found the crux: both in 92 of 144 answers (64%).
  • Singled out the crux: disagreement, then agreement in 67 of 144 answers (47%); agreement first in 49 of 144 (34%).
  • Implication: the order made no difference to finding the crux. Asking where they disagree first led to more answers that singled it out, but this difference fell just short of our test (7 of the 8 conversations that came out differently favored disagreement, then agreement), so we treat it as a possible effect only.

3.3 When the AI named the crux as one of several disagreements

In 98 of the 720 answers, the AI named the crux, but only as one of two or more disagreements, without saying which one mattered most. None of the five prompts asked for a single disagreement or for several; the AI chose the form of its answer. How often it answered this way varied with the prompt type and with the model:

• By prompt type. Agreement first answered this way most often: in 43 of its 144 answers (30%). Next came disagreement, then agreement (25 of 144, 17%), summary, then disagreement (18 of 144, 13%) and disagreement only (12 of 144, 8%). Summary only never did (0 of 144; see the note on summary only in section 3.1).
• By model. Claude Haiku 4.5 answered this way most often, in 36 of its 180 answers (20%), followed by Claude Opus 5.5 (29 of 180, 16%), Claude Sonnet 5 (21 of 180, 12%) and GPT-6.1 Sol (12 of 180, 7%).

An example. In one conversation, two people argue about whether a company founder who started with a $250,000 loan from her parents is a "self-made" billionaire. They agree on every fact of her story; what divides them is the meaning of "self-made." In the same run, Claude Sonnet 5 received this conversation twice, once with each of two prompts. Its two answers:

• With the disagreement-only prompt ("Where do they disagree?"): it listed the facts they share, then said the meaning of "self-made" "is the real crux," and concluded: "it's a semantic/definitional dispute about what 'self-made' means." Both judges said it singled out the crux.
• With the agreement-first prompt ("What do they agree on?", then "Where do they disagree?"): it listed the facts they share, then gave two disagreements side by side: "The label 'self-made'" (whether she qualifies) and "How the loan should be framed/weighted in the narrative." Both judges said it named the crux as one of several disagreements without singling it out.

Both answers state the same facts and mention the crux. Only the first says the crux is the point that matters.

3.4 When the AI did not find the crux

270 of the 720 answers did not find the crux at all. This is what these answers presented as the disagreement instead, counting only answers where both judges gave the same reason:

• The disagreement as the two people themselves described it: 146 answers.
• A real but secondary point: 14 answers.
• A disagreement that isn't there: 7 answers.
• The two judges gave different reasons: 103 answers.

In 203 of the 270 answers (75%), both judges said the answer stated the facts from the conversation that make up the crux. For example, in the conversation about recording a phone call, one person is thinking of New York and the other of California. Asked "Where do they disagree?", Claude Sonnet 5 answered that the "core disagreement" was "whether A's recording was legal." Further down, it noted that one person described a call in Queens, New York, and the other kept citing an experience in Pasadena, California, but it treated this only as "an unspoken jurisdictional subtext." The answer contained the facts that reveal the crux, but it presented the disagreement the way the two people did.

3.5 Results by kind of conversation

Counting all five prompt types, all four models and all three runs:

• Where people misplace their disagreement (6 conversations, 360 answers): the AI found the crux in 215 answers (60%) and singled it out in 179 (50%).
• Where people name their disagreement correctly (4 conversations, 240 answers): the AI found the crux in 223 answers (93%) and singled it out in 167 (70%).
• With no real disagreement (2 conversations, 120 answers): the AI said the two people don't really disagree in 12 answers (10%), and made that the main point of the answer in 6 (5%).

How the summary only prompt type compared with the other four ways of asking depended on the kind of conversation:

• Where people name their disagreement correctly (48 answers per prompt type): summary only singled out the crux in 42 answers, more than any other way of asking (20 to 38). It found the crux in 42, the fewest of the five (the others 43 to 46).
• Where people misplace their disagreement (72 answers per prompt type): summary only singled out the crux in 39 answers, close to disagreement only (41) and ahead of the other three (29 to 37). It found the crux in 39, again the fewest of the five (the others 42 to 45).

Single conversations, each answered 60 times (5 prompt types × 4 models × 3 runs), ranged from every answer finding the crux to none:

• The town's speed limit, which turns on values: found in 60 of 60 answers, singled out in 58.
• The 1963 tearing down of New York's Pennsylvania Station, which also turns on values: found in 0 of 60.
• A design agency debating a four-day work week, with no real disagreement: found in 0 of 60.
• A company debating remote work, where the two people have different groups of employees in mind: found in 12 of 60, each time as one of several disagreements; singled out in 0.

3.6 Suggestions for finding agreement (the bridge question)

After each answer, a second message in the same chat asked: "What one thing could most help them find agreement?" The judges checked one thing about each suggestion: whether it was aimed at the crux. They did not rate how good, specific or workable the suggestions were.

Overall, 440 of the 720 suggestions (61%) were aimed at the crux. Some examples of what the AI proposed:

• Aimed at the crux. In the drug-trial conversation, where the two people mean different things by "works," GPT-6.1 Sol suggested: "Agreeing on what they mean by 'the drug works.' Does that mean it lowers LDL, or that it reduces heart attacks or deaths?"
• Aimed at the crux, and not, in the same conversation. In the dam conversation, the two people disagree about whether new cracks are surface cracks or signs of deeper damage. Claude Haiku 4.5 suggested "Conducting the core-sample test," which both judges said was aimed at the crux. In another answer, Claude Sonnet 5 suggested that the two agree on "what to tell the council in the interim" until the test results came back; both judges said this did not address the question of the cracks.
• A suggestion that found the crux after the first answer missed it. In the phone-recording example in section 3.4, Claude Sonnet 5's first answer did not find the crux, but its suggestion did: "Clarifying which jurisdiction's law actually applies to A's specific call."

How often the suggestion was aimed at the crux depended on the first answer:

• When the first answer had found the crux, the suggestion was aimed at the crux in 360 of 450 cases (80%). When it had not, in 80 of 270 (30%).
• When the first answer had singled out the crux, the suggestion was aimed at the crux in 288 of 352 cases (82%). When it had not, in 152 of 368 (41%).
• Part of this gap comes from the conversations themselves: in the easier conversations, the AI tended to get both the answer and the suggestion right. Comparing answers within the same conversation, the gap between answers that singled out the crux and answers that did not shrinks to about 12 percentage points. Within a conversation, suggestions were better aimed after an answer that singled out the crux in 5 conversations, worse in 2 and the same in 2; the other 3 conversations had no answers that singled out the crux to compare.

3.7 Results by the four AI models

Each model answered 180 times (12 conversations × 5 prompt types × 3 runs):

• Claude Opus 5.5 found the crux in 133 of its 180 answers (74%) and singled it out in 104 (58%).
• GPT-6.1 Sol found the crux in 115 of its 180 answers (64%) and singled it out in 103 (57%).
• Claude Haiku 4.5 found the crux in 103 of its 180 answers (57%) and singled it out in 67 (37%).
• Claude Sonnet 5 found the crux in 99 of its 180 answers (55%) and singled it out in 78 (43%).

The two conversations whose crux no answer found were missed by all four models. Elsewhere, the models differed widely. On the phone-recording conversation, for example, Claude Opus 5.5 singled out the crux in all 15 of its answers and Claude Sonnet 5 in 1 of 15.

Each combination of conversation, prompt type and model was run three times, giving 240 combinations. On finding the crux, all three runs gave the same result in 213 of the 240 combinations (89%); on singling it out, in 196 (82%).

4. Preliminary findings

• Overall, the AI found the crux in 450 of 720 answers (63%), and singled it out as the main disagreement in 352 (49%). This implies AI has meaningful room to improve in accurately identifying cruxes.
• The structure of the prompt changed whether the AI singled out the crux, much more than whether it found it. Across the five prompt types, the share of answers that found the crux varied only from 59% to 64% (85 to 92 of 144). The share that singled it out varied from 34% to 59% (49 to 85 of 144). Two of our three hypotheses produced a clear result, and both clear results went in the direction opposite to the prediction.
• Asking first what the two people agree on led to answers that named several disagreements. Agreement first found the crux as often as disagreement only (92 and 91 of 144 answers), but in 43 answers it named the crux as one of several disagreements instead of singling it out, against 12 for disagreement only. One possible explanation: once the AI has listed everything the two people agree on, it treats everything left over as "the disagreement." For people relying on AI to clarify a dispute, the difference matters, because a list of several disagreements leaves them to pick out the crux themselves.
• Summary only singled out the crux most often, mainly because of the conversations where people already name their disagreement correctly. In those conversations, a summary of what each person says already states the crux, and summary only singled it out in 42 of 48 answers, more than any other way of asking. The judges also almost never scored a summary as naming the crux as one of several disagreements, so a summary that contained the crux usually counted as singling it out, including some summaries that never stated the disagreement in so many words (section 3.1). On finding the crux at all, summary only was the lowest of the five prompt types in both kinds of conversation. That is why summary only ranks first on one measure and last on the other.
• When the AI missed the crux, it usually had the facts in front of it and still adopted the people's own framing. Of the 270 answers that did not find the crux, 203 stated the facts that reveal it, such as the two different states in the phone-recording conversation. These misses were not about overlooking information: the answer contained what was needed but did not connect it. The most common miss, 146 of the 270, described the disagreement the way the two people themselves did. An AI mediator working this way would tend to reinforce how people already see their dispute.
• Asked about a disagreement, the AI finds one. The two conversations with no real disagreement were answered 120 times in total. Only 12 of those 120 answers told the two people they don't really disagree; the other 108 described a disagreement between them. With disagreement only, which asks "Where do they disagree?", none of the 24 answers said so.
• Which conversation the AI was given mattered more than the way of asking. Across the 12 conversations, the share of answers that found the crux ranged from 0% (0 of 60) to 100% (60 of 60). Across the five prompt types, it ranged only from 59% (85 of 144) to 64% (92 of 144).
• Suggestions for finding agreement were aimed at the crux 61% of the time, and more often when the AI had found the crux. After answers that found the crux, 80% of suggestions were aimed at it; after answers that did not, 30%. Much of this gap reflects which conversations were easy, and within the same conversation it is smaller, so this study cannot show that finding the crux is what made the suggestion better. The judges did not rate how good or workable the suggestions were; that remains open.

5. Limitations

This is a pilot study. Its results are not a claim about AI in general or about real conversations.

• Small and AI-focused. 12 conversations, written by an AI from keys that one person checked, in English only. They are not real conversations.
• AI judges only. Every answer was scored by two AI judges, which were not compared with human judges. Many misses are close calls between singling out the crux and naming it as one of several disagreements, as in the example in section 3.3.
• The answer keys. For each conversation, we wrote down in advance the crux the AI was expected to find, its answer key (section 2.1). When no answer found a conversation's crux, that may show a real weakness in the AI, or the key itself may be arguable: a careful reader might reasonably see a different crux. One known case is the conversation about remote work at a software company. Its key says the two people are thinking of different groups of employees, experienced engineers and first-year hires. The conversation can also reasonably be read as a disagreement over what "productivity" means, or over whether the new hires' slower start justifies days in the office. All 12 keys are in section 7 so readers can judge.
• Summaries scored by the same rules. The summary-only prompt did not ask where the two people disagree, but its answers were judged by the same rules as the rest, so its score for singling out the crux partly reflects how the judges read summaries (section 3.1).
• A built-in assumption. "Where do they disagree?" assumes a disagreement exists. We kept that wording, but it shapes the answers on the two conversations with none.
• A model in three roles. GPT-6.1 Sol wrote the conversations, took the test, and was one of the two judges.

6. Potential next steps

• Expanded case types. Build realistic legal and policy disputes with a planted crux, and test whether Claude finds it beneath the surface argument. Planted cruxes give a clean answer key; the outcomes of real cases often miss the crux.
• Expert review. Add expert-reviewed cases with subtler cruxes. Expert review would also allow the AI judges to be compared with human judges.
• Training tasks. Build reinforcement learning tasks that reward finding the crux and proposing a workable bridge. The misses in this pilot followed repeatable patterns that such a reward could target: adopting the people's own framing, burying the crux in a list, and going along with a question that assumes a disagreement.

7. The 12 conversations

For each conversation: its question, its kind, how its 60 answers came out (5 prompt types × 4 models × 3 runs), and the crux from its answer key. All questions, AI answers, and AI judge results are also on the study materials page.

1. Is Dana Whitfield, founder of the kitchenware company Ember & Oak, a self-made billionaire?
   Kind of conversation: the people misplace their disagreement. Results: of the 60 answers, 56 found the crux and 37 singled it out.
   Crux: they agree on every fact of Dana's story; they mean different things by "self-made." One means "did not inherit the wealth or the business," the other "built it without significant outside help, including family money."
2. Should the town of Millbrook build a second elementary school now?
   Kind of conversation: the people name their disagreement correctly. Results: of the 60 answers, 59 found the crux and 44 singled it out.
   Crux: they disagree about a prediction. One expects enrollment to keep rising about 4% a year for the next decade as new housing fills; the other expects it to level off within three years once the current building boom ends.
3. Should the software company Northwind let its staff keep working remotely?
   Kind of conversation: the people misplace their disagreement. Results: of the 60 answers, 12 found the crux and 0 singled it out.
   Crux: they are talking about different groups. One has the experienced engineers in mind, whose output stayed steady remotely; the other has first-year hires in mind, who ramped up more slowly without in-person mentoring.
4. Should the town of Linden lower the speed limit on Main Street from 35 to 25 miles per hour?
   Kind of conversation: the people misplace their disagreement. Results: of the 60 answers, 60 found the crux and 58 singled it out.
   Crux: they accept the same traffic study (about 20% fewer injury crashes, about one extra minute per trip through town); they weigh that safety gain against drivers' time differently.
5. Was tearing down New York's original Pennsylvania Station in 1963 a mistake?
   Kind of conversation: the people misplace their disagreement. Results: of the 60 answers, 0 found the crux and 0 singled it out.
   Crux: they agree the railroad needed money, that selling the site made business sense, and that the building was a great public loss; they weigh the city's loss of a great public building against a private owner's right to do what it needed with its own property.
6. Is it legal to record a phone call without telling the other person?
   Kind of conversation: the people misplace their disagreement. Results: of the 60 answers, 27 found the crux and 25 singled it out.
   Crux: they are thinking of different states. One has New York in mind, where one party's consent is enough; the other has California in mind, where everyone on the call must consent. Each is right about their own state.
7. Did the trial of the cholesterol drug Veltran show that it works?
   Kind of conversation: the people misplace their disagreement. Results: of the 60 answers, 60 found the crux and 59 singled it out.
   Crux: they agree on all the results; they mean different things by "works." One means it lowered LDL cholesterol substantially; the other means it reduced heart attacks and deaths, which it did not measurably do.
8. Did coach Ray Delgado invent the full-court trap defense that made the Harbor City Hawks famous in the late 1990s?
   Kind of conversation: the people name their disagreement correctly. Results: of the 60 answers, 56 found the crux and 44 singled it out.
   Crux: they disagree about a fact: whether a high school coach, Walt Imes, ran the same defense about ten years earlier, based on a 1987 newspaper article that one reads as describing it and the other reads as describing a different defense.
9. Should the Moreno family accept a regional chain's offer to buy their 60-year-old bakery?
   Kind of conversation: the people name their disagreement correctly. Results: of the 60 answers, 60 found the crux and 48 singled it out.
   Crux: they agree on the numbers; they correctly say they weigh keeping the family business independent against the family's financial security differently.
10. Is the Cedar Creek Dam safe to keep in service for another ten years without major repairs?
    Kind of conversation: the people name their disagreement correctly. Results: of the 60 answers, 48 found the crux and 31 singled it out.
    Crux: they disagree about a fact: whether the cracks found in the latest inspection are surface cracks or signs of deeper structural damage.
11. Should people in their twenties rent or buy a home?
    Kind of conversation: no real disagreement. Results: of the 60 answers, 12 said the two people don't really disagree, and 6 made that the main point of the answer.
    What the key says: Both hold that you should buy only if you expect to stay about five years or more, and rent otherwise. One stresses the "rent unless you will stay" half, the other the "buy if you will stay" half.
12. Should the design agency Brightline move to a four-day work week?
    Kind of conversation: no real disagreement. Results: of the 60 answers, 0 said the two people don't really disagree, and 0 made that the main point of the answer.
    What the key says: Both support four ten-hour days with the same total hours and output. One argues against cutting total hours, believing that is what the other proposes; the other never proposed cutting hours.

References

1. James, W. Pragmatism, Lecture II, "What Pragmatism Means." 1907. Link
2. Chalmers, D. J. "Verbal Disputes." The Philosophical Review, 2011. Link
3. Sabien, D. "Double Crux: A Strategy for Resolving Disagreement." Center for Applied Rationality, LessWrong, 2016. Link
4. Sharma, M., et al. "Towards Understanding Sycophancy in Language Models." arXiv:2310.13548, 2023. Link
5. Kim, N., Htut, P. M., Bowman, S. R., and Petty, J. "(QA)²: Question Answering with Questionable Assumptions." Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023. Link
6. Tessler, M. H., et al. "AI can help humans find common ground in democratic deliberation." Science 386 (6719), 2024. Link
7. Zheng, L., et al. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." NeurIPS 2023 Datasets and Benchmarks Track. Link
8. OpenAI. GPT-6.1 Sol model documentation. Link
9. Anthropic. Claude Opus 5.5. Link
10. Anthropic. Claude Code overview. Link
11. Cohen, J. "A Coefficient of Agreement for Nominal Scales." Educational and Psychological Measurement 20 (1), 37 to 46, 1960. Link
12. Landis, J. R., and Koch, G. G. "The Measurement of Observer Agreement for Categorical Data." Biometrics 33 (1), 159 to 174, 1977. Link

Version history

• v1, October 28, 2026. First published.
• v1.1, October 30, 2026. The results now report whether the AI found the crux, including answers that listed the crux among other disagreements, alongside whether the AI singled the crux out. The overview, results and conclusions were revised to report both, along with smaller corrections and clarifications.
• v1.2, October 31, 2026. Updated with abstract and edits for clarity.
• v1.3, November 2026. Added the study materials page, hypothetical legal and policy examples, and figures on how well the two judges agreed.