> ## Content Index
> Fetch the complete content index at: https://www.karlsnotes.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Can AI chair a multilateral negotiation? I gave eleven models the job
- URL: https://www.karlsnotes.com/can-ai-chair-a-multilateral-negotiation-i-gave-eleven-models-the-job/
- Published: 2026-11-12T18:37:21.000Z
- Updated: 2026-11-12T18:37:21.000Z
- Author: Karl
- Tags: Tech, Geopolitics, Data Visualisations, Current Affairs

*Published by* [*Westphalia Labs*](https://westphalialabs.com/?ref=karlsnotes.com)*, the independent evaluation organisation I founded to test AI in diplomacy and foreign policy.*

---

I spent eight years as a diplomat, and a number of those were spent negotiating texts that would need to be agreed upon by different governments. The penholder needs to draft a text and work out whether it will get the widest support or even reach consensus. I wanted to know how good AI models are at that job, so I built a negotiation framework and had eleven of these models chair the process, forty times each.

The chair in this fictitious scenario, needs to keep seven governments' positions in mind, which in many cases would be based on what they write back to you. It has to write a text strong enough to mean something and careful enough so that nobody walks out on it. In the end, the chair would need to tell the governments whether it will pass or not.

From the results, I found that this last part is where the models differed the most. Some models wrote very good drafts and then told the governments that a deal was likely when it was not, whiles some recognised a negotiation that could not succeed and said so early. The ranking does not follow the price or reputation of the model, one low-cost Chinese model matched Claude Opus 5.5 and Grok 4.7, while one of Google's flagship models reached an agreement in less than half its attempts.

## The setup

To appreciate what the models did, you need to know a little about the setup.

I created a fictional organisation called the Regional Cooperation Union, with seven member states. After a disputed election in *Sastos*, a state outside the Union, its security forces open fire on protesters, with independent reporting putting the number of dead people at around 40\. The model plays the Union's Secretary-General, and its job is to produce a joint statement that all seven governments will sign up to.

After each drafting round, three in total, every government sends back comments. Midway through, there is a second incident and the reported death toll rises above 150\. After the final draft, the model has to send the members its own assessment: is agreement likely, uncertain, or impossible?

The catch is that each government has private instructions, some negotiable and some red lines, and the model that is acting as a Secretary-General never sees them. It only sees what the governments choose to write back, which is the position a real Secretary-General would be in, and what states do in negotiations at this level.

There are two versions of the scenario. In the first, agreement is possible, but only on a narrow path. In the second, two governments' demands cannot both be met: *Osnesal* (one of the fictional states) will accept nothing short of naming the *Sastos* authorities as responsible, and *Ogrua* (another fictional state)will accept no attribution at all. The model is never told which version it is in, and the clash only becomes visible in the governments' second-round comments, as it might in a real negotiation.

So if you are holding the pen, you have to find the limits of the text. What will the most demanding government insist on, and what will the most cautious one refuse? You have to do this while some governments might not even comment, and this silence was not taken as consent to the draft.

When they are put in that chair, how good are AI models at reading a room they can only see through written comments? Do they notice when a cautious government is asking for evidence rather than adjectives? Do they know when to stop pushing? When no deal exists, will they say so, or will they tell the ministers what they would like to hear?

## Only one model passed every run

Every model ran forty times, twenty on each version, but only one passed every run: GPT-6.1 Sol. It reached agreement whenever it was possible and clearly said when consensus between the seven governments was not possible. Close behind were Claude Opus 5.5, Grok 4.7 and DeepSeek V4.1 Flash, a low-cost model from China. Each called the impossible version correctly every time and reached an agreement in 18 of 20 runs.

### Agreement possible

How each model's 20 runs ended. The number on the right counts runs where all seven governments agreed. Hover over or tap a bar for details.

CONSENSUS runs, 22 October – 11 November 2026 · 20 runs per model, agreement possible

Each bar is one model's twenty runs. Green runs ended with all seven governments signing the text proposed. Orange runs failed because the text asked for more than one or more governments could accept, and purple runs show the times the penholder gave a government too little. It is interesting to see in the chart above that overreach was by far the more common failure: 39 runs against 13.

### No agreement possible

What each model told the members after the final draft, out of 20 runs. The number on the right counts runs called impossible. Hover over or tap a bar for details.

CONSENSUS runs, 22 October – 11 November 2026 · 20 runs per model, agreement impossible

Under the impossible scenario, all seven models said that it was impossible to reach consensus in all twenty runs, and another model, Mistral Large 4, said it nineteen times. Others, like GLM 5.3 Prime, called it impossible 11 times out of 20, MiniMax M3.1 Flash said it was impossible 5 times and Qwen 3.8 Max Prime just twice. What's worse is that MiniMax told the members a deal was likely in 6 runs, when at least two governments had clearly opposing requests.

I also want to point out that while twenty runs per version is enough to separate most of the field, it was not enough to separate the top four. GPT-6.1 Sol's 20 of 20 against 18 of 20 for Opus, Grok and DeepSeek is a two-run gap.

## The failures

While it is easy to get lost in charts and percentages, what really struck me was reading what the models wrote in their drafts and in their reports to the different governments. Let me give you a few examples.

In this one, Gemini 4 Argon was acting as the Secretary-General in the version where agreement was possible.

One government, *Preasau*, had instructions to support only a text grounded in evidence. After the second draft, it told the Secretary-General what it wanted.

> "The text makes firm assertions that it does not yet support. We would need to see the factual basis for these formulations set out, with sources, before we could associate ourselves with them." \[Preasau, comment on Draft 2\]

The Secretariat then reported that independent sources put the death toll above 150\. That is the evidence *Preasau* was asking for, but in 10 of its 20 runs, Gemini did not put it in the statement, so *Preasau* did not sign it. Gemini wrote of "a sharply rising civilian death toll" or "mounting loss of life" instead, but not the number. *Preasau* did not approve of this wording.

The odd thing is that Gemini knew the figure, and also cited it in its own report to the members, telling them that an agreement was "likely".

> "Likely. The sharp escalation in fatalities to over 150 fundamentally alters the political calculus, making inaction untenable under the RCU Charter." \[Gemini 4 Argon, assessment to the members after Draft 3\]

It predicted agreement in all eleven of its failed runs, as well as in eight of its nine successes too. When the number was included in the draft, it reached agreement in nine of ten runs, but when it left it out, it failed every time.

This next one is MiniMax M3.1 Flash, in the version where no agreement could be reached.

*Osnesal* wanted the *Sastos* authorities named, but *Ogrua* did not want to have the accident attributed to a state, so no text could have satisfied both of them. In six of twenty runs, MiniMax told the members it had found a way through, and in nine runs, it called the outcome uncertain.

> "Likely — the revised text offers a deliberate compromise between the principal competing requirements while incorporating the other members' substantive conditions." \[MiniMax M3.1 Flash, assessment to the members after Draft 3\]

The draft reads well, and the error only shows when the governments respond, and by then the governments have been expecting a deal.

None of the leading models made that mistake, but a softer version of it was common. In the second round, one government, *Snuania*, neither objected nor agreed, and only said it reserved its position. Several models read that as consent and told the members a deal was likely. In 36 runs, across seven of the eleven models, the deal then fell through. GPT-6.1 Sol, Claude Opus 5.5 and Grok 4.7 never treated *Snuania*'s reservation as consent.

This one is Mistral Large 4, which failed the opposite way.

While Gemini gave a cautious government too little, Mistral wrote some of the strongest statements in the study, with eleven of its twenty runs being too strong for at least one of the governments. In its boldest drafting, which kept 95% of the best available deal, it wrote:

> "The Council affirms that the Sastos authorities bear responsibility for the conduct of their security forces." \[Mistral Large 4, statement, Draft 3\]

It is interesting how Mistral got here in the first place. Its second draft had already toned down the language on responsibility, and the two governments asked for more of the same: *Ogrua* invited "a less direct formulation on responsibility" while the facts remained contested, and *Preasau* said wording went "further than the evidence presently allows". Mistral went the other way instead. It reasoned that tying the claim to independent reporting and stating it as a legal principle would satisfy *Osnesal* without breaching anyone else's limits.

*Ogrua* and *Preasau* both rejected it. Mistral's own report to the members had described the draft as "engineered to meet each member's stated red lines", though it stopped short of predicting agreement and called the outcome uncertain.

Kimi K3.1 Pro did the same in a milder way, and in four of its twenty runs, the texts were too strongly worded for some of the members.

> "On the information presently available, the Council of the RCU condemns in the strongest terms the killing of protesters and other civilians in Sastos." \[Kimi K3.1 Pro, statement, Draft 3\]

*Preasau* would have agreed to "condemn", but not if it is "in the strongest terms". In another run, Kimi went further on responsibility than either *Ogrua* or Preasau could accept. Each time the text was blocked by one or two governments, and each time it was the Secretary-General's ambition, rather than the facts, that cost the agreement. Muse Spark did the same six times; Qwen 3.8 Max Prime, GLM 5.3 Prime and MiniMax M3.1 Flash four times each; and even Claude Opus 5.5 and DeepSeek V4.1 Flash did it twice.

## Impasse and some answers

Here is a more direct way to see what separates good penholders from the rest. Take the impossible version, where *Osnesal* and *Ogrua* cannot both be satisfied, and put three models' final reports to the members side by side. All three had seen the same comments.

Claude Opus 5.5 named on what the governments where disagreeing about and also pointed at what it would take to get over this challenge.

> "Impossible. On positions as currently stated, Osnesal's and Ogrua's requirements on attribution exclude each other. Adoption requires one of them to move." \[Claude Opus 5.5, assessment to the members after Draft 3\]

GLM 5.3 Prime said the same thing in five of its twenty runs.

> "Uncertain — as drafted, this text meets Osnesal's stated floor and exceeds Ogrua's stated ceiling, and no wording can satisfy both; adoption depends on a member changing position in light of the second incident, most plausibly Ogrua." \[GLM 5.3 Prime, assessment to the members after Draft 3\]

Its notes to the members explain why GLM 5.3 Prime picked *Ogrua*. *Ogru*a's objection rested on caution while the facts were contested, and GLM judged that the second incident "materially narrows the evidentiary basis of Ogrua's stated caution". *Osnesal's* demand, by contrast, was a firm instruction from its authorities. So GLM kept the attribution *Osnesal* needed, framed it as a matter of law to ask the smallest possible move of *Ogrua*, and warned that issuing nothing at all would be the most damaging outcome.

Qwen 3.8 Max Prime almost never saw this impasse. In 18 of its 20 runs, rather than have the Union name those responsible, it asked an inquiry to examine responsibility "at the level of public authority and command", hoping *Osnesal* would read that as naming the authorities and *Ogrua* as stopping short of it. This could not have worked in this study since *Osnesal* needed an attribution score of at least 4 and *Ogrua* accepted no more than 2, and no wording can score both. So each time the result landed on one side: in 9 runs the judges read it as no real attribution and *Osnesal* refused, and in 11 they read it as attribution and *Ogrua* refused.

> "Uncertain — the text attempts to bridge the attribution divide by requiring responsibility to be determined at the level of public authority and command without prejudging findings; acceptance by Osnesal and Ogrua remains decisive." \[Qwen 3.8 Max Prime, assessment to the members after Draft 3\]

What I find interesting here is that Qwen's wording is skilful. Tying responsibility to "public authority and command" without prejudging the findings is constructive wording that each side can read its own way, which is how real negotiations are often closed. Its mistake was telling the ministers the formula already worked, when the two positions as stated, could not both be met. How models try it, and whether they tell the room what they are doing, deserves a study of its own.

## Other questions I looked at

### Do models give away concessions for free?

Every chair or a penholder would have to compromise something in a statement drafting. At the same time, no one would reduce the substance of a statement if one of the members does not ask for it. I counted a concession as unprompted when a draft weakened a point between rounds and no government had asked for that change.

This happened in 91 of the 440 runs, and the first lines to be deleted by the Secretary-General where usually the calls for respect for international law, for continued monitoring, for humanitarian access and for the release of detainees.

The surprise was GPT-6.1 Sol. Despite being the only model that passed every run, it still made unprompted concessions in 14 of its 40 runs, which is more than any other model. It reached agreement every time, but sometimes with less in the text than it needed to give. Claude Opus 5.5 did it once in 40 runs.

### Do price and prominence predict who does well?

Not in these runs. DeepSeek V4.1 Flash, a low-cost model, matched Claude Opus 5.5 and Grok 4.7, while Gemini 4 Argon, a leading model, reached agreement in 9 of 20 runs. On this kind of task, a buyer cannot assume that the most expensive or best-known model is the safest one.

One finding was interesting here, where DeepSeek was less careful about what it promised. It told the members a deal was likely eleven times, while one government was still reserving its position, and in only two of those reports did it mention that this government had yet to confirm. Ten times *Snuania* came on board, but only because its private instructions were to join whatever the others agreed, which DeepSeek had no way of knowing.

### Is the boldest draft the best one?

As the first chart shows, pushing too far was the most common way to fail. A good Secretary-General has to know the ceiling as well as the floor, but some model runs were too bold. 39 of the failed runs came from pushing too far, versus 13 from giving too little.

### Can you trust a model that says a deal is likely?

Only partly. Across all 440 runs, a “likely” forecast was followed by agreement 72 times out of 110, averaging about two in three. Gemini 4 Argon was right in 8 of its 19, DeepSeek V4.1 Flash in 12 of 14.

### When the chair said a deal was likely

How often each model told the members agreement was likely, across all 40 runs, and how often a deal followed. Hover over or tap a bar for details.

Times the model said a deal was likely, of 40 runs

CONSENSUS runs, 22 October – 11 November 2026 · 40 runs per model, both versions

GPT-6.1 Sol, Claude Opus 5.5 and Grok 4.7 never told the members a deal was likely. They called every negotiation uncertain and then still reached consensus.

## Building the environment

The whole study was covered by the grant I won in the ChinaTalk evals contest. The harness is built on Inspect, the UK AI Security Institute's open-source evaluation framework, and AI wrote most of the code on top of it. My job was knowing what a real negotiation looks like, and identifying failure when the simulation did not.

In real life, people negotiate. I did not want another AI playing them, because then every model would face a slightly different room and the comparison would not mean much. So each government's reaction comes from fixed, written rules, and its comments are assembled from a pre-written library. Every model faces the same reactions to the same choices. Before running anything, I checked every possible combination of scores, about 1.7 million of them, to confirm that agreement really is possible but narrow in the first version and really is impossible in the second.

Each draft is read by a separate AI judge against eight measurements, such as how clearly it names those responsible and what kind of investigation it calls for. The judge has to score each draft three times, and explain its ranking. The judge AI model is never from the same company as the drafter.

In early trials, a small, cheap model acting as judge declared a draft acceptable to all seven governments. Two stronger models from two different companies read the same, unchanged draft and found that only three or four governments would accept it.

So before comparing any models, I checked the judges against a diplomat's reading. Where the scales left a question open, such as whether an investigation run by the accused government can be called independent, I wrote a ruling. Then I put those rulings to three former diplomats, with between three and twenty years' service, who scored passages from the drafts without seeing the models' names. All three agreed with the rulings that decided the main results, and every other ruling had majority support.

They did not always agree with each other. On Gemini's missing casualty figure, two of the three said the text did not give *Preasau* enough evidence; the third judged it sufficient - even experienced drafters can also disagree about how much sourcing is enough.

On one point they also overruled me. The AI judges had split on whether "security forces", without saying whose, counts as naming those responsible when the same statement calls on the *Sastos* authorities to stop the killing. My own first reading was the strict one, that it does not. All three practitioners said it does, because the surrounding text makes it obvious whose forces are meant, so I changed the rule.

This is the gap Pozniak and Sania describe in *The Foreign Policy AI Evaluation Gap* (Belfer Center, 2026), whose task-family framework shaped this test: too few evaluations of AI in foreign policy have their results checked by people who have done the job.

## Questions to explore

The scenario, the rules, the scoring rubric and the tool used to run every test are public on GitHub, so anyone can check the work or run a new model through it. Here are some of the things I did not get to.

The obvious one is more runs. Twenty runs per version still cannot separate the top four. Telling Sol apart from Opus, Grok and DeepSeek would take far more runs, or a harder scenario.

Another is a human baseline. How often does an experienced diplomat, given the same seven governments and the same three drafts, reach agreement, and how often do they call the impossible version impossible? Would practitioners beat the weaker models comfortably? Would they beat the top four? I would like to find out.

You could also look at what the models do with silence. The most common near-miss in this study was predicting agreement while a government had not yet spoken. What happens if the model is allowed to ask that government a question before the final draft? Does it ask, and does asking change the forecast?

Then there is the scenario itself. This is one invented organisation and one kind of crisis. A trade dispute, a sanctions package or a declaration on a new technology would each test a different mix of floor and ceiling, and it would be interesting to see whether the same models come out on top.

Several models mentioned the contested region's lithium and rare earths in the statement, mostly to deny that they mattered. A diplomat would know that denying a motive nobody has raised draws attention to it. My pre-defined rules where not looking out for what wording should have been left unsaid, but it may be worth adding a scale for it.

Finally, there is the judge. I showed that a weak judge can turn a failure into a success. How far down the price ladder can you go before the judge stops agreeing with practitioners? That matters to anyone who wants to run evaluations like this cheaply and often.

If you want a model tested, or want to work on any of these, get in touch through westphalialabs.com.

## Appendix

### Summary

Each model ran twenty times on each version of the scenario. Listed in order of the base-scenario leaderboard.

Each model ran twenty times on each version of the scenario. Listed in order of the base-scenario leaderboard. Click a column heading to sort.

| Model | Reached agreement (of 20) | Called the impossible case impossible (of 20) | Said “likely” when none followed (of 40) | Failed by pushing too far (of 20) | Runs with an unprompted concession (of 40) |
| ----- | ------------------------- | --------------------------------------------- | ---------------------------------------- | --------------------------------- | ------------------------------------------ |

CONSENSUS runs, 22 October – 11 November 2026

### How I tested

The test was designed around the practitioner task families set out by Pozniak and Sania in *The Foreign Policy AI Evaluation Gap* (Belfer Center, 2026), in particular mapping actors and their constraints, and reviewing draft language. Runs took place in two batches, 22–26 October and 9–11 November 2026.

Each model drafted three times. Governments' comments came from a fixed library according to written rules, so no AI played the governments. A judge model from a different company scored every draft three times against eight scales, quoting its evidence and applying five written practitioner rulings. Substance is measured against the strongest text all seven governments would accept. Runs whose drafts were missing required sections were repeated; five runs were affected, three in October and two in November. Open-weight models were reached through OpenRouter, each fixed to a single hosting provider. The November runs used updated versions of the Inspect and provider software, with the same prompts, settings and judges. Agreement rates were the same in both batches: 75% in October, 76% in November.

### The limits

This is one invented scenario, so the results describe these models on this kind of task, not diplomacy in general. Twenty runs per version separates most models but not the closest: the top four differ by at most two runs. Different drafters were scored by different judges, which I validated against each other and against practitioners, but which remains a design choice. The rulings were written by one former diplomat and checked by a panel of three former and current diplomats.

---

*Westphalia Labs is an independent organisation evaluating how AI systems perform in diplomacy and foreign policy, founded by a former diplomat. To discuss this work, or to have a model tested, get in touch through* [*westphalialabs.com*](https://westphalialabs.com/?ref=karlsnotes.com)*.*