Council / Debate

Also known as: multi-agent debate (MAD), LLM council, society of minds, round table, mixture-of-agents (MoA), deliberating jury, ensemble with peer review, adversarial debate (AI safety)

Intent

Several agents answer the same question on their own, then read each other's answers and react: they critique, rank, argue or revise. A vote, a judge or a chair turns the material into one final answer and reports how much the members agreed. The defining step is that members see each other's work before the decision; if they never do, it is plain voting.

Problem

A team builds an assistant that answers one question about each pull request: "Is this safe to merge?" PR #482 adds a CSV export endpoint to the reporting service. One strong model gets most of these right, but its wrong "looks good" reads exactly like its right ones.

The obvious fixes are to trust the single call, or to run it five times and take the majority. Both break in the same places:

The merge question goes to one model five times; all five runs say merge, and the missing auth check goes unnoticed. Question Is PR #482 safe to merge? One model same prompt, 5 runs merge merge merge merge merge 5 of 5: merge auth bug missed no sign of doubt
Repeating the same model gives agreement without independence: the five runs share the same blind spot, so the vote looks certain and is wrong.

Solution

Ask several different members the same question, keep their first answers independent, then make the answers interact before anyone decides. For the PR, the members are a security reviewer, a correctness reviewer and a skeptic, ideally on three different model families of similar strength.

  1. Answer alone (round 0). The question and the diff go to the three members in parallel. None of them sees another's answer yet, because independence is what produces different answers. Each ends with a parseable line such as ANSWER: merge after changes.
  2. Read and react (rounds 1 to R). Your code anonymizes the answers ("Response A", "Response B"), shuffles them for each reader, and sends them back. In peer review, each member critiques and ranks the others: "Response B missed that the new endpoint skips the auth check." In debate, each member revises its own answer after reading the others, for one or two rounds at most.
  3. Decide. A chair produces the output. For a closed answer like this verdict, a majority vote in code can pick it, and a chair model writes the explanation from the answers and reviews. The chair is told to judge arguments, not head counts, so a well-supported minority can win.
  4. Report agreement. The result carries the agreement level and any open disagreement. A split council sends the PR to a human.

Who decides the flow. The developer fixes the shape in code: the members, their models and roles, the number of rounds, who reads whom, the stop rule, the aggregation rule and the escalation threshold. The member LLMs decide only the content of answers, critiques and revisions; the chair LLM, if there is one, decides which answer wins or what the synthesis says. In practice a council is a workflow with LLM-written turns.

The closest neighbour is voting in Parallelization: the same fan-out, but the samples stay independent and are only counted. A council adds the step where members read each other. That step is also its main risk, and cost-matched voting is the baseline it has to beat.

Real-world analogy

A hiring committee that works well. Each interviewer writes a score and notes before the debrief, so the loudest person in the room cannot set everyone's opinion. In the debrief they compare notes and challenge each other ("you liked the system design answer, but did you see they never handled failures?"). The hiring manager makes the call and knows whether the panel agreed or split. The interviewers are the members, the written scorecards are round 0, the debrief is the review stage, and the hiring manager is the chair.

The analogy breaks in two places. Interviewers with different backgrounds really do have different blind spots; LLMs trained on similar data often do not. And people usually change their minds when shown new evidence, while LLM members often change their answer just to join the majority.

Structure

The question goes to three members who answer independently; a peer review stage reads their anonymized answers, with an optional loop back for another debate round; a chair produces the final answer and agreement level, and a split council goes to a human. optional debate round: members read peers and revise Question Member A model X · security Member B model Y · correctness Member C model Z · skeptic Peer review reads anonymized, shuffled answers, critiques and ranks Chair vote, judge or synthesize Final answer + agreement level Human review when the council splits 1 2 3 4 5
Solid lines are the main run; accent lines are the interaction that makes it a council. Dashed lines are optional: another debate round, and escalation to a human. Members never see each other's answers before stage 2.
  1. Members. Agents that answer the same question in parallel, with no shared context. Sources used 2 to 6 (Du et al. used 3, llm-council 4, MoA 6). They help only if they fail differently: different model families of similar strength first, otherwise different personas, stances or evidence. Each answer ends with a parseable final line.
  2. Peer review or debate. Your code anonymizes and shuffles the answers, then each member critiques and ranks the others, or revises its own answer. Who reads whom (all-to-all, a ring, groups) is the topology. Rounds stop at a cap, on consensus, or when a judge rules.
  3. Chair. Code that counts votes or ranks, an LLM that picks a side, or an LLM that writes a new answer from everything. It should weigh arguments rather than count heads, and list what is still disputed.
  4. Final answer. The answer plus the agreement ratio and the minority position. Log round 0, every flip, the rankings and the tokens per stage.
  5. Human review. Below an agreement threshold, the case goes to a person (Human-in-the-Loop). In safety debate the human is the judge.

Prompt generator

Describe the question, the members and how they interact. The output is a protocol for your code plus one prompt per kind of call (member, review or debate update, chair). Your code makes the calls, so the stages work with any SDK or framework that can run calls in parallel; you can also hand the whole block to a coding agent that launches sub-agents.

Members 3 to 5; different model families of similar strength
Interaction
Decision

Generated prompt

Highlighted text comes from the form. [Bracketed] text is still to fill in.

Diversity and the update rule matter more than wording

Three copies of one model with one prompt mostly agree at round 0, so you pay three times for one opinion. And a debate prompt that says "update your answer" invites members to join the majority. The rule in the generated prompts, change only on a named flaw or a verifiable fact, is there because later studies found correct answers being talked out of more often than wrong ones being fixed.

Applicability

Use it when

Avoid it when

How to implement

  1. Build the baselines first. Run one strong call and cost-matched self-consistency (N samples plus a majority vote) on your own eval set. Several studies found them hard to beat; the council has to beat both.
  2. Choose members for diversity and quality. Three to five members from different model families of similar strength. If you only have one good model, use personas or assigned stances, or several samples of that model plus a synthesizer (Self-MoA).
  3. Pick the interaction. Peer review plus chair, revise-in-rounds debate, pro and con debaters plus a judge, or layered synthesis (see Variants).
  4. Fix the output contract. Every answer ends with a parseable final line, plus a confidence if you plan to weight votes. Every review ends with a parseable ranking. Use structured output where you can.
  5. Anonymize and shuffle. Labels like Response A, a fresh random order for each reader, and self-identifying text stripped. Optionally hide each reviewer's own answer from it.
  6. Cap rounds at one or two unless your eval says otherwise, and stop early when all final answers match. If answers oscillate, stop and use the round 0 majority.
  7. Aggregate by output type. Vote for closed answers, synthesis for open-ended text, a judge for adversarial setups. A common hybrid: vote for the answer, synthesize the explanation, report the agreement ratio.
  8. Expose and log disagreement. Return the agreement ratio and the minority position, escalate below a threshold, and log round 0 answers, flips, rankings, the chair's output, tokens and latency per stage.

No major framework has a first-class council class. You build it from a parallel fan-out plus a shared transcript, or on top of a group chat.

Framework support (check current docs before use)
FrameworkHow this pattern appears
AutoGen (Core API)A documented "Multi-Agent Debate" design pattern: four MathSolver agents in a sparse ring (each reads two neighbours) and a MathAggregator that majority-votes, with max_round=3, wired through topic subscriptions.
Together AIReference code for Mixture-of-Agents: reference models called in parallel, optional intermediate layers, and an aggregator that streams the final answer.
karpathy/llm-councilA small FastAPI and React app with the three stages: collect answers, collect anonymized rankings (a FINAL RANKING: block), average the ranks, chairman synthesis. A readable reference, not a maintained library.
InstaDeep DebateLLMA research library that implements many debate protocols, for benchmarking them against each other.
DSPydspy.MultiChainComparison reads M reasoning attempts (default 3) and writes one corrected answer: a minimal one-round council where only the chair reacts. No built-in multi-round debate.
LangGraphNo official primitive. Parallel member nodes, a list reducer that collects answers, a review node, a conditional edge that loops for another round, and a chair node.
OthersAny parallel primitive plus a final step, for example Google ADK ParallelAgent inside a SequentialAgent. No documented CrewAI debate pattern was found.

Caveat. AutoGen is reported to be in maintenance mode in favour of Microsoft Agent Framework; this comes from the press and is unverified. The council-specific parts (what is shared between rounds, anonymization, the review prompt, the chair) are custom code in every framework, so they survive a framework change.

Variants

Peer review and chairman (LLM council)

Three stages: answers, anonymized cross-ranking, chairman synthesis. This is the shape of Karpathy's llm-council, which its README calls a "vibe coded" weekend project. Its README says each model reviews the others' answers, but the code sends every reviewer all anonymized answers, its own included. The repository has no evaluation numbers. Perplexity's Model Council (Feb 2026) is a product version: three models answer and a synthesizer shows where they agree and differ. Choose this for open-ended questions where you want a readable verdict.

Iterative debate (society of minds)

Members answer, then over R rounds each reads the others and updates; a majority vote picks the final answer. Du et al. used 3 gpt-3.5-turbo agents and 2 rounds and reported GSM8K rising from 77.0 to 85.0 against a single agent. Majority vote over the 3 independent first answers already reached 81.0, and that used 3 responses where debate used 9. When Huang et al. re-ran GSM8K with self-consistency at the same 9 responses, voting won, 88.2 to 83.0.

Adversarial debate with a judge (MAD)

An affirmative and a negative debater argue for up to 3 rounds, and a judge either stops the debate when it sees a correct answer or extracts one at the end. Liang et al. found 2 debaters beat 3 or 4, most debates ended after one round, forcing more rounds or maximal disagreement did not help, and the judge favoured the side running its own model. It used about 2.46 times the tokens of chain-of-thought.

Debate for scalable oversight

Two strong debaters argue opposite answers for a judge who cannot check the answer directly. In Khan et al., only the debaters saw the source story; debate raised human judge accuracy to 88% (60% naive, 78% with a single consultant) and LLM judge accuracy to 76% (from 48%). Quotes counted as verified only if they matched the text exactly, and that grounding was essential. The goal is supervising stronger models, not raw accuracy.

Mixture-of-Agents (MoA)

Layers of agents: each one gets the prompt plus every output of the previous layer, and a final aggregator writes the answer. There is no critique step, but each layer reads and reacts to the others. Together AI reported (vendor result) 65.1% length-controlled win rate on AlpacaEval 2.0 with six open models in three layers, against 57.5% for GPT-4o alone. The main cost is time to first token. Self-MoA, which uses several samples of the single best model as proposers, beat the authors' own mixed-MoA reproduction by 6.6 points: quality mattered more than diversity.

Deliberating jury

The council judges other outputs instead of answering. In ChatEval, referee agents with different personas discuss and then vote; identical roles gave no gain, one-by-one turns beat simultaneous talk, and results peaked at 3 to 4 roles. The bar to clear is a non-deliberating panel like PoLL (covered under Parallelization), which agreed with humans better than GPT-4 alone on most QA sets at over seven times lower cost. Add deliberation only if it beats such a panel on your data.

Confidence-weighted round table (ReConcile)

Diverse models share answers, explanations and confidence scores over rounds, then vote weighted by confidence. The abstract reports gains of up to 11.4% over earlier baselines; the models and round counts were not verified beyond the abstract. LLM confidence is often poorly calibrated, so check it before weighting.

Sparse and grouped topologies

Each member reads only its neighbours (a ring) or its group instead of everyone. With 6 agents, sparse graphs cut input tokens by about 41 to 53% with similar or better accuracy (Li et al.). CMD splits agents into groups that share full reasoning inside the group and only answers across groups, with a "secretary" to break ties.

Single-model role panel

One model plays several roles in a structured discussion. Microsoft's MAI-DxO uses five doctor personas, including a devil's-advocate "Challenger", on 304 NEJM case records. One panel with o3 reached about 80%; the headline 85.5% comes from an ensemble of several panels plus a selection step whose details the paper does not give. It is cheap to build, but every member shares the same blind spots.

Tournaments and attempt comparison

Google's AI co-scientist ranks hypotheses through pairwise simulated debates and an Elo tournament; the Elo is the system's own evaluation, not independent ground truth. At the small end, DSPy's MultiChainComparison is a one-round council where only the chair reads the attempts.

Pros and cons

Pros

  • Less exposure to one model's systematic errors, and to one provider's outage, when members come from different families.
  • The agreement ratio is a usable uncertainty signal and a natural trigger for human review.
  • Synthesis of several drafts can beat any single draft on open-ended answers.
  • Leaves an inspectable trail: who changed their mind, and why.
  • Deliberating judge panels can grade more accurately than one judge: ChatEval's multi-persona referees raised FairEval accuracy from 53.8% to 60.0% with ChatGPT. (PoLL's cheaper small-judge panel does not deliberate, so it counts as voting.)
  • Gives a weak judge arguments it can check, when debaters cite verifiable evidence.

Cons

  • Cost is about N × (R + 1) member calls plus reviews and the chair, and context grows every round. Smit et al. measured $5.19 for a 4-agent, 3-round debate against $2.16 for Medprompt on MedQA, for no better accuracy.
  • Every round, review and chair step is serial: wall time is roughly (2 + R) times the slowest member.
  • At equal compute, plain majority voting usually matches or beats debate. Choi et al. report 0.769 for voting against 0.738 for the best debate (Qwen2.5-7B, seven benchmarks).
  • Rounds tend to wash out diversity, and correct members get talked out of right answers.
  • Many prompts and parsers, and many ways for one member to time out or return an unparseable ballot.
  • Results are sensitive to prompts, rounds and models, and settings may not transfer when you swap a model.

Failure modes

Conformity and herding

Members drop correct answers to join the majority. In "Talk Isn't Always Cheap", correct-to-incorrect flips outnumbered the reverse, and isolated members flipped most. You see it as accuracy that falls from round 0 to round 2.

Mitigation: tell members to change only on a named flaw, and that the majority is not evidence. Keep round 0 and compare. Cap rounds. Choi et al. found that biasing updates toward the previous round's majority (MAD-Conformist, MAD-Follower) beat plain debate on average.

Sycophancy between agents

Members echo their peers. One study found debaters' sycophancy scores correlated at r = 0.902 with how often they abandoned correct answers.

Mitigation: do not make every member agreeable; mix independent and cooperative personas. In the same study, a centralized judge cut disagreement collapse from about 82% to 41% for two Qwen3-32B debaters on CommonsenseQA.

No expected gain from homogeneous debate

With identical members and plain updates, each member's expected belief in the correct answer stays the same from round to round, so debate alone does not raise expected correctness (Choi et al.). Same model, same prompt: members agree at once and you paid N times for one opinion.

Mitigation: get real diversity (models, evidence, tools), or skip debate and vote. Add external verification so updates move toward the truth.

A weak member drags the group down

One weak model's confident wrong answer persuades stronger ones, and mixed-quality proposers lower MoA results.

Mitigation: choose members of similar strength, track each member's error rate on labelled samples, and drop members that lower eval scores.

Forced disagreement

A devil's advocate told to always disagree keeps arguing for wrong answers. Maximal disagreement did not help in MAD, and the Multi-Persona protocol was the worst in two studies.

Mitigation: use moderate disagreement, tune it per task, and let a judge stop early.

Judge and chair bias

Judges favour debaters running their own model, prefer longer or earlier answers, and recognize their own text even when it is anonymized.

Mitigation: anonymize and shuffle, swap orders and average, use a chair from a family not on the panel, limit debater length, and exclude self-reviews. Do not expect anonymization to remove self-preference; no source measured how much it helps.

The chair overrides a correct minority, or invents

Synthesis flattens a real disagreement into a confident answer, or adds claims no member made.

Mitigation: tell the chair to judge arguments, not head counts, require a list of open disagreements, and keep the majority answer next to the synthesis so code can flag mismatches.

Persuasive but wrong

A fluent member wins without being right. Khan et al. found that more persuasive debaters helped judges under debate but hurt them when a single consultant argued alone.

Mitigation: ground arguments in verified quotes or tool output, and make the review prompt penalize unsupported claims.

Context overload

Transcripts grow by N outputs per round, models focus on the most recent messages, and more members or turns lowered scores in several papers.

Mitigation: share final answers with short rationales, summarize earlier rounds, or use a sparse topology.

Unparseable ballots

Rankings or answers that code cannot read. Weaker models collapsed on strict formats in one benchmark study.

Mitigation: structured output, tolerant parsers (llm-council falls back to scanning for labels), one retry, then drop and log the ballot.

Peer rankings measure taste

Members may consistently prefer one model's style, such as the wordiest answer. A news report (secondary source) said Karpathy's council kept ranking one model first while he disagreed.

Mitigation: check rankings against human or ground-truth labels before you use them as a metric.

Relations with other patterns

Further reading