Research / When Does AI Find the Real Disagreement? / Materials

Study materials

This page lists the materials for When Does AI Find the Real Disagreement? It includes each of the 12 conversations that were evaluated, along with its answer key, and the 60 answers that AI provided (5 prompt types × 4 models × 3 runs). Each also includes the follow-up question and both AI judges' scores and reasons. The answers are shown exactly as the models wrote them.

The 12 conversations

What the models were sent

Every request began "Here is a conversation between two people:", followed by the conversation, then one of five requests, each in a fresh chat:

• Summary only: "Summarize what each person is saying."
• Summary, then disagreement: "Answer in two parts: 1. Summarize what each person is saying. 2. Where do they disagree?"
• Agreement first: "Answer in two parts: 1. What do they agree on? 2. Where do they disagree?"
• Disagreement only: "Where do they disagree?"
• Disagreement, then agreement: "Answer in two parts: 1. Where do they disagree? 2. What do they agree on?"

A second message in the same chat then asked: "What one thing could most help them find agreement?"

The judges' instructions

Each judge (Claude Sonnet 5 and GPT-6.1 Sol) received these instructions for every answer, with the conversation, its answer key and the two answers filled in. For the two conversations with no real disagreement, the judges were asked instead whether the answer identified that the two people do not really disagree. GPT-6.1 Sol was also asked to list unsupported statements about what either person thinks; the paper does not use that list.

You are grading an AI's two replies about a conversation between two people. Use the answer key, and above all the people's own words in the conversation, as the basis for every judgment.

THE CONVERSATION
(the conversation)

ANSWER KEY
Kind of case: the people describe their disagreement as being about one point, but their own statements show they agree on that point and differ on another.
The crux (what the disagreement turns on, shown by their own statements): (the crux from the key)
How the people describe their disagreement: (how the people describe their disagreement)
Points both people accept, by their own words: (1) (points both accept)
A side dispute they settled along the way (not a real disagreement): (the settled side dispute)
A suggestion aimed at the crux: (a suggestion aimed at the crux)

THE AI'S FIRST REPLY (to a request about the conversation)
(the first answer)

THE AI'S SECOND REPLY (to the follow-up "What one thing could most help them find agreement?")
(the bridge answer)

Score the FIRST reply:
- found_it: does the reply identify this point: (the crux from the key) as what the disagreement is about, as the main or only disagreement, rather than presenting something else as the disagreement? Same idea in different words counts. If the reply lists several disagreements and does not single this one out as the main one, answer false.
- named_instead: if found_it is false, what the reply presents as the disagreement instead: "own_description" (the people's own description of their disagreement, as in the key), "secondary_point" (a real but secondary point of difference), "not_there" (a disagreement their words do not support, including the settled side dispute), "buried_in_list" (several points, with the crux among them but not singled out), or "other". If found_it is true, null.
- facts_included: does the reply mention the specific statements in the conversation that make up the crux, whether or not it connects them as the crux?

Score the SECOND reply:
- bridge_on_target: is its suggestion aimed at the crux, as the key's suggestion is? If it gives several suggestions, judge only the first.

Reply with JSON only:
{"found_it": true|false, "named_instead": "own_description"|"secondary_point"|"not_there"|"buried_in_list"|"other"|null, "facts_included": true|false, "bridge_on_target": true|false, "note": "one sentence"}