Council / Debate
Also known as: multi-agent debate (MAD), LLM council, society of minds, round table, mixture-of-agents (MoA), deliberating jury, ensemble with peer review, adversarial debate (AI safety)
Intent
Several agents answer the same question on their own, then read each other's answers and react: they critique, rank, argue or revise. A vote, a judge or a chair turns the material into one final answer and reports how much the members agreed. The defining step is that members see each other's work before the decision; if they never do, it is plain voting.
Problem
A team builds an assistant that answers one question about each pull request: "Is this safe to merge?" PR #482 adds a CSV export endpoint to the reporting service. One strong model gets most of these right, but its wrong "looks good" reads exactly like its right ones.
The obvious fixes are to trust the single call, or to run it five times and take the majority. Both break in the same places:
- Correlated blind spots. Sampling one model several times reduces random error, not systematic error. If the model does not notice that the new endpoint skips the auth middleware, most of the five samples will not notice either.
- Self-review defends the first answer. Asking the same model to re-check its approval tends to confirm it. Liang et al. named this "Degeneration-of-Thought" and designed debate to get around it.
- No uncertainty signal. Five near-identical approvals look like confidence. Nothing tells you this PR was a coin flip that a human should look at.
- No argument trail. A reviewer who cannot check the diff in depth has nothing to verify, only a verdict.
Solution
Ask several different members the same question, keep their first answers independent, then make the answers interact before anyone decides. For the PR, the members are a security reviewer, a correctness reviewer and a skeptic, ideally on three different model families of similar strength.
- Answer alone (round 0). The question and the diff go to the three members in parallel. None of them sees another's answer yet, because independence is what produces different answers. Each ends with a parseable line such as
ANSWER: merge after changes. - Read and react (rounds 1 to R). Your code anonymizes the answers ("Response A", "Response B"), shuffles them for each reader, and sends them back. In peer review, each member critiques and ranks the others: "Response B missed that the new endpoint skips the auth check." In debate, each member revises its own answer after reading the others, for one or two rounds at most.
- Decide. A chair produces the output. For a closed answer like this verdict, a majority vote in code can pick it, and a chair model writes the explanation from the answers and reviews. The chair is told to judge arguments, not head counts, so a well-supported minority can win.
- Report agreement. The result carries the agreement level and any open disagreement. A split council sends the PR to a human.
Who decides the flow. The developer fixes the shape in code: the members, their models and roles, the number of rounds, who reads whom, the stop rule, the aggregation rule and the escalation threshold. The member LLMs decide only the content of answers, critiques and revisions; the chair LLM, if there is one, decides which answer wins or what the synthesis says. In practice a council is a workflow with LLM-written turns.
The closest neighbour is voting in Parallelization: the same fan-out, but the samples stay independent and are only counted. A council adds the step where members read each other. That step is also its main risk, and cost-matched voting is the baseline it has to beat.
Real-world analogy
A hiring committee that works well. Each interviewer writes a score and notes before the debrief, so the loudest person in the room cannot set everyone's opinion. In the debrief they compare notes and challenge each other ("you liked the system design answer, but did you see they never handled failures?"). The hiring manager makes the call and knows whether the panel agreed or split. The interviewers are the members, the written scorecards are round 0, the debrief is the review stage, and the hiring manager is the chair.
The analogy breaks in two places. Interviewers with different backgrounds really do have different blind spots; LLMs trained on similar data often do not. And people usually change their minds when shown new evidence, while LLM members often change their answer just to join the majority.
Structure
- Members. Agents that answer the same question in parallel, with no shared context. Sources used 2 to 6 (Du et al. used 3, llm-council 4, MoA 6). They help only if they fail differently: different model families of similar strength first, otherwise different personas, stances or evidence. Each answer ends with a parseable final line.
- Peer review or debate. Your code anonymizes and shuffles the answers, then each member critiques and ranks the others, or revises its own answer. Who reads whom (all-to-all, a ring, groups) is the topology. Rounds stop at a cap, on consensus, or when a judge rules.
- Chair. Code that counts votes or ranks, an LLM that picks a side, or an LLM that writes a new answer from everything. It should weigh arguments rather than count heads, and list what is still disputed.
- Final answer. The answer plus the agreement ratio and the minority position. Log round 0, every flip, the rankings and the tokens per stage.
- Human review. Below an agreement threshold, the case goes to a person (Human-in-the-Loop). In safety debate the human is the judge.
Prompt generator
Describe the question, the members and how they interact. The output is a protocol for your code plus one prompt per kind of call (member, review or debate update, chair). Your code makes the calls, so the stages work with any SDK or framework that can run calls in parallel; you can also hand the whole block to a coding agent that launches sub-agents.
Generated prompt
Highlighted text comes from the form. [Bracketed] text is still to fill in.
Three copies of one model with one prompt mostly agree at round 0, so you pay three times for one opinion. And a debate prompt that says "update your answer" invites members to join the majority. The rule in the generated prompts, change only on a named flaw or a verifiable fact, is there because later studies found correct answers being talked out of more often than wrong ones being fixed.
Applicability
Use it when
- You have several strong, different models of similar quality. Their errors are less correlated, which is the only source of gain. Self-MoA found mixing helps only when the models are close in quality.
- The output is open-ended. For analysis, advice or long answers there is nothing to vote on, and the MoA authors found that synthesizing several drafts beat having an LLM pick one.
- You need an evaluation signal with less single-model bias. Panels of judges from different families agree with humans more than one judge does; evaluation is where councils have the cleanest evidence.
- A weaker judge has to decide. When a human or a small model cannot check the answer directly, two debaters who cite checkable evidence give it something to verify.
- Disagreement is useful output. You want to know which cases to escalate, and a wrong answer costs much more than a few extra calls.
Avoid it when
- The answer is a closed label or number and you have one model. Self-consistency at equal compute is usually as good or better, and simpler. Use voting in Parallelization.
- A grounded checker exists. Tests, a compiler, retrieval or a schema beat any panel opinion. Use Evaluator–Optimizer with that checker, or best-of-N with it.
- You have not tried a good single prompt with examples. One study found a single agent with demonstrations matched multi-agent discussion.
- Latency matters. Every round, the review and the chair are serial steps, and layered MoA cannot stream until the last layer.
- Your extra members are much weaker than your best model. A weak member drags strong ones down. Sample the best model instead.
- The task is mostly recall or commonsense. In "Talk Isn't Always Cheap", every CommonsenseQA configuration got worse after debate.
How to implement
- Build the baselines first. Run one strong call and cost-matched self-consistency (N samples plus a majority vote) on your own eval set. Several studies found them hard to beat; the council has to beat both.
- Choose members for diversity and quality. Three to five members from different model families of similar strength. If you only have one good model, use personas or assigned stances, or several samples of that model plus a synthesizer (Self-MoA).
- Pick the interaction. Peer review plus chair, revise-in-rounds debate, pro and con debaters plus a judge, or layered synthesis (see Variants).
- Fix the output contract. Every answer ends with a parseable final line, plus a confidence if you plan to weight votes. Every review ends with a parseable ranking. Use structured output where you can.
- Anonymize and shuffle. Labels like
Response A, a fresh random order for each reader, and self-identifying text stripped. Optionally hide each reviewer's own answer from it. - Cap rounds at one or two unless your eval says otherwise, and stop early when all final answers match. If answers oscillate, stop and use the round 0 majority.
- Aggregate by output type. Vote for closed answers, synthesis for open-ended text, a judge for adversarial setups. A common hybrid: vote for the answer, synthesize the explanation, report the agreement ratio.
- Expose and log disagreement. Return the agreement ratio and the minority position, escalate below a threshold, and log round 0 answers, flips, rankings, the chair's output, tokens and latency per stage.
No major framework has a first-class council class. You build it from a parallel fan-out plus a shared transcript, or on top of a group chat.
| Framework | How this pattern appears |
|---|---|
| AutoGen (Core API) | A documented "Multi-Agent Debate" design pattern: four MathSolver agents in a sparse ring (each reads two neighbours) and a MathAggregator that majority-votes, with max_round=3, wired through topic subscriptions. |
| Together AI | Reference code for Mixture-of-Agents: reference models called in parallel, optional intermediate layers, and an aggregator that streams the final answer. |
| karpathy/llm-council | A small FastAPI and React app with the three stages: collect answers, collect anonymized rankings (a FINAL RANKING: block), average the ranks, chairman synthesis. A readable reference, not a maintained library. |
| InstaDeep DebateLLM | A research library that implements many debate protocols, for benchmarking them against each other. |
| DSPy | dspy.MultiChainComparison reads M reasoning attempts (default 3) and writes one corrected answer: a minimal one-round council where only the chair reacts. No built-in multi-round debate. |
| LangGraph | No official primitive. Parallel member nodes, a list reducer that collects answers, a review node, a conditional edge that loops for another round, and a chair node. |
| Others | Any parallel primitive plus a final step, for example Google ADK ParallelAgent inside a SequentialAgent. No documented CrewAI debate pattern was found. |
Caveat. AutoGen is reported to be in maintenance mode in favour of Microsoft Agent Framework; this comes from the press and is unverified. The council-specific parts (what is shared between rounds, anonymization, the review prompt, the chair) are custom code in every framework, so they survive a framework change.
Variants
Peer review and chairman (LLM council)
Three stages: answers, anonymized cross-ranking, chairman synthesis. This is the shape of Karpathy's llm-council, which its README calls a "vibe coded" weekend project. Its README says each model reviews the others' answers, but the code sends every reviewer all anonymized answers, its own included. The repository has no evaluation numbers. Perplexity's Model Council (Feb 2026) is a product version: three models answer and a synthesizer shows where they agree and differ. Choose this for open-ended questions where you want a readable verdict.
Iterative debate (society of minds)
Members answer, then over R rounds each reads the others and updates; a majority vote picks the final answer. Du et al. used 3 gpt-3.5-turbo agents and 2 rounds and reported GSM8K rising from 77.0 to 85.0 against a single agent. Majority vote over the 3 independent first answers already reached 81.0, and that used 3 responses where debate used 9. When Huang et al. re-ran GSM8K with self-consistency at the same 9 responses, voting won, 88.2 to 83.0.
Adversarial debate with a judge (MAD)
An affirmative and a negative debater argue for up to 3 rounds, and a judge either stops the debate when it sees a correct answer or extracts one at the end. Liang et al. found 2 debaters beat 3 or 4, most debates ended after one round, forcing more rounds or maximal disagreement did not help, and the judge favoured the side running its own model. It used about 2.46 times the tokens of chain-of-thought.
Debate for scalable oversight
Two strong debaters argue opposite answers for a judge who cannot check the answer directly. In Khan et al., only the debaters saw the source story; debate raised human judge accuracy to 88% (60% naive, 78% with a single consultant) and LLM judge accuracy to 76% (from 48%). Quotes counted as verified only if they matched the text exactly, and that grounding was essential. The goal is supervising stronger models, not raw accuracy.
Mixture-of-Agents (MoA)
Layers of agents: each one gets the prompt plus every output of the previous layer, and a final aggregator writes the answer. There is no critique step, but each layer reads and reacts to the others. Together AI reported (vendor result) 65.1% length-controlled win rate on AlpacaEval 2.0 with six open models in three layers, against 57.5% for GPT-4o alone. The main cost is time to first token. Self-MoA, which uses several samples of the single best model as proposers, beat the authors' own mixed-MoA reproduction by 6.6 points: quality mattered more than diversity.
Deliberating jury
The council judges other outputs instead of answering. In ChatEval, referee agents with different personas discuss and then vote; identical roles gave no gain, one-by-one turns beat simultaneous talk, and results peaked at 3 to 4 roles. The bar to clear is a non-deliberating panel like PoLL (covered under Parallelization), which agreed with humans better than GPT-4 alone on most QA sets at over seven times lower cost. Add deliberation only if it beats such a panel on your data.
Confidence-weighted round table (ReConcile)
Diverse models share answers, explanations and confidence scores over rounds, then vote weighted by confidence. The abstract reports gains of up to 11.4% over earlier baselines; the models and round counts were not verified beyond the abstract. LLM confidence is often poorly calibrated, so check it before weighting.
Sparse and grouped topologies
Each member reads only its neighbours (a ring) or its group instead of everyone. With 6 agents, sparse graphs cut input tokens by about 41 to 53% with similar or better accuracy (Li et al.). CMD splits agents into groups that share full reasoning inside the group and only answers across groups, with a "secretary" to break ties.
Single-model role panel
One model plays several roles in a structured discussion. Microsoft's MAI-DxO uses five doctor personas, including a devil's-advocate "Challenger", on 304 NEJM case records. One panel with o3 reached about 80%; the headline 85.5% comes from an ensemble of several panels plus a selection step whose details the paper does not give. It is cheap to build, but every member shares the same blind spots.
Tournaments and attempt comparison
Google's AI co-scientist ranks hypotheses through pairwise simulated debates and an Elo tournament; the Elo is the system's own evaluation, not independent ground truth. At the small end, DSPy's MultiChainComparison is a one-round council where only the chair reads the attempts.
Pros and cons
Pros
- Less exposure to one model's systematic errors, and to one provider's outage, when members come from different families.
- The agreement ratio is a usable uncertainty signal and a natural trigger for human review.
- Synthesis of several drafts can beat any single draft on open-ended answers.
- Leaves an inspectable trail: who changed their mind, and why.
- Deliberating judge panels can grade more accurately than one judge: ChatEval's multi-persona referees raised FairEval accuracy from 53.8% to 60.0% with ChatGPT. (PoLL's cheaper small-judge panel does not deliberate, so it counts as voting.)
- Gives a weak judge arguments it can check, when debaters cite verifiable evidence.
Cons
- Cost is about N × (R + 1) member calls plus reviews and the chair, and context grows every round. Smit et al. measured $5.19 for a 4-agent, 3-round debate against $2.16 for Medprompt on MedQA, for no better accuracy.
- Every round, review and chair step is serial: wall time is roughly (2 + R) times the slowest member.
- At equal compute, plain majority voting usually matches or beats debate. Choi et al. report 0.769 for voting against 0.738 for the best debate (Qwen2.5-7B, seven benchmarks).
- Rounds tend to wash out diversity, and correct members get talked out of right answers.
- Many prompts and parsers, and many ways for one member to time out or return an unparseable ballot.
- Results are sensitive to prompts, rounds and models, and settings may not transfer when you swap a model.
Failure modes
- Conformity and herding
-
Members drop correct answers to join the majority. In "Talk Isn't Always Cheap", correct-to-incorrect flips outnumbered the reverse, and isolated members flipped most. You see it as accuracy that falls from round 0 to round 2.
Mitigation: tell members to change only on a named flaw, and that the majority is not evidence. Keep round 0 and compare. Cap rounds. Choi et al. found that biasing updates toward the previous round's majority (MAD-Conformist, MAD-Follower) beat plain debate on average.
- Sycophancy between agents
-
Members echo their peers. One study found debaters' sycophancy scores correlated at r = 0.902 with how often they abandoned correct answers.
Mitigation: do not make every member agreeable; mix independent and cooperative personas. In the same study, a centralized judge cut disagreement collapse from about 82% to 41% for two Qwen3-32B debaters on CommonsenseQA.
- No expected gain from homogeneous debate
-
With identical members and plain updates, each member's expected belief in the correct answer stays the same from round to round, so debate alone does not raise expected correctness (Choi et al.). Same model, same prompt: members agree at once and you paid N times for one opinion.
Mitigation: get real diversity (models, evidence, tools), or skip debate and vote. Add external verification so updates move toward the truth.
- A weak member drags the group down
-
One weak model's confident wrong answer persuades stronger ones, and mixed-quality proposers lower MoA results.
Mitigation: choose members of similar strength, track each member's error rate on labelled samples, and drop members that lower eval scores.
- Forced disagreement
-
A devil's advocate told to always disagree keeps arguing for wrong answers. Maximal disagreement did not help in MAD, and the Multi-Persona protocol was the worst in two studies.
Mitigation: use moderate disagreement, tune it per task, and let a judge stop early.
- Judge and chair bias
-
Judges favour debaters running their own model, prefer longer or earlier answers, and recognize their own text even when it is anonymized.
Mitigation: anonymize and shuffle, swap orders and average, use a chair from a family not on the panel, limit debater length, and exclude self-reviews. Do not expect anonymization to remove self-preference; no source measured how much it helps.
- The chair overrides a correct minority, or invents
-
Synthesis flattens a real disagreement into a confident answer, or adds claims no member made.
Mitigation: tell the chair to judge arguments, not head counts, require a list of open disagreements, and keep the majority answer next to the synthesis so code can flag mismatches.
- Persuasive but wrong
-
A fluent member wins without being right. Khan et al. found that more persuasive debaters helped judges under debate but hurt them when a single consultant argued alone.
Mitigation: ground arguments in verified quotes or tool output, and make the review prompt penalize unsupported claims.
- Context overload
-
Transcripts grow by N outputs per round, models focus on the most recent messages, and more members or turns lowered scores in several papers.
Mitigation: share final answers with short rationales, summarize earlier rounds, or use a sparse topology.
- Unparseable ballots
-
Rankings or answers that code cannot read. Weaker models collapsed on strict formats in one benchmark study.
Mitigation: structured output, tolerant parsers (llm-council falls back to scanning for labels), one retry, then drop and log the ballot.
- Peer rankings measure taste
-
Members may consistently prefer one model's style, such as the wordiest answer. A news report (secondary source) said Karpathy's council kept ranking one model first while he disagreed.
Mitigation: check rankings against human or ground-truth labels before you use them as a metric.
Relations with other patterns
- Parallelization: owns voting, self-consistency, best-of-N and non-deliberating judge panels. Stage 1 of a council is a parallel fan-out, and cost-matched voting is the baseline a council must beat.
- Evaluator–Optimizer: one generator and one critic in a loop, including a single LLM judge with a rubric. A council is several peers critiquing each other. Prefer the evaluator loop when a grounded checker exists.
- Orchestrator–Workers: the lead splits a task into different subtasks; council members all work on the same question. An orchestrator can call a council for one high-stakes subtask.
- Supervisor: control returns to a central agent after every worker. A supervisor can call a council as a tool for one decision; the council's chair aggregates only once, at the end.
- Plan-and-Execute: search-based planning (Tree of Thoughts, LATS) also compares candidates, but by scoring and backtracking inside one agent, not by peers arguing.
- Group Chat: often confused with debate, and Azure uses "debate" and "council" as names for group chats. Group chat is open turn-taking with a speaker selector; a council has fixed rounds, independent first answers, anonymization and an explicit decision rule. You can run a debate on a group-chat framework.
- Blackboard: a debate transcript is a simple shared workspace, but blackboard agents contribute different expertise to one solution instead of competing answers to one question.
- Human-in-the-Loop: the agreement ratio is a natural trigger for human review, and in safety debate the human is the judge.
- ReAct: any member can itself be a ReAct loop with tools.
- Swarm / Handoffs, Routing, Prompt Chaining, Hierarchical teams: not neighbours in shape, but a council can sit as one step of a chain, one route, or one team in a hierarchy.
Further reading
- Improving Factuality and Reasoning in Language Models through Multiagent Debate (Du et al., 2023): the paper that popularized revise-in-rounds debate, with its prompts and scaling results.
- Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models? (Choi, Zhu, Li, NeurIPS 2025): the controlled comparison against majority voting, and why homogeneous debate has no expected gain.
- Talk Isn't Always Cheap: Understanding Failure Modes in Multi-Agent Debate (Wynn, Satija, Hadfield, 2025): flip analysis, conformity, and weak members dragging strong ones down.
- Mixture-of-Agents Enhances Large Language Model Capabilities (Wang et al., Together AI, 2024): layered synthesis, and why synthesizing beats picking one answer.
- Debating with More Persuasive LLMs Leads to More Truthful Answers (Khan et al., 2024): debate as oversight for a judge who cannot see the evidence, and verified quotes.
- karpathy/llm-council (GitHub): a short, readable implementation of answers, anonymized ranking and chairman synthesis.