When AI Agents Disagree, Don’t Just Make Them Vote
Three confident AIs can still agree on the wrong answer. The useful part of an AI council is the feedback loop between them.
The idea sounds brilliant: instead of asking one AI, ask five. Let them debate. Make them vote. Take the majority answer.
Welcome to the AI Council. It makes for a great demo. It can also give you five variations of the same mistake.
The useful question is not “How many agents do we have?” It is “What information is capable of changing an agent’s mind?”
More agents can help
Multi-agent debate is a legitimate research direction. One agent proposes an answer, another challenges it, a third looks for inconsistencies and the original agent revises. The pattern resembles peer review.
But there is a dangerous leap between “multiple agents can improve results” and “five copies of the same model automatically produce truth.”
AI can create an echo chamber
Suppose Agent A confidently gives the wrong answer. Agent B reads Agent A before doing independent work and decides it sounds plausible. Agent C sees two agents agreeing and treats that agreement as additional evidence.
Five seconds later, your sophisticated AI council has manufactured consensus around an error.
The useful loop looks different
A better system separates roles. The second component does not merely ask whether it agrees. It receives a different job: find what could make this answer wrong.
- Researcher: retrieves source material the generator did not see.
- Calculator: recomputes numerical claims.
- Database tool: checks the actual customer record.
- Policy checker: compares a proposed action with company rules.
- Human reviewer: handles high-impact or low-confidence decisions.
The feedback contains new evidence, not just another opinion.
A travel-planning example
A user asks: “I land at 6:15 PM. Can I make the 7:30 PM ferry?”
Agent one checks the ferry timetable and says yes. Agent two looks at the same timetable and agrees.
Two votes. Still wrong.
Neither checked immigration time, baggage collection, airport-to-port travel time or the ferry boarding cutoff.
Feedback needs something observable
A feedback loop becomes valuable when the system can measure an outcome.
- An email agent drafts a message → did a human approve it?
- A sales agent qualifies a lead → did the lead convert?
- A support agent answers → did the customer reopen the ticket?
- An extraction agent produces JSON → did schema validation pass?
- A recommendation engine proposes an item → did the user select it?
Without the second half, you do not have a feedback loop. You have generation.
Self-critique is useful — but not enough
“Check your answer again” can improve output. But the same model created the mistake and may fail to recognise it. Strong verification introduces independence: different prompts, different evidence, different tools, sometimes different models and, for sufficiently important decisions, a human.
The verifier should have access to something the original generator did not.
When not to build an AI council
Multi-agent workflows create token cost, latency, orchestration logic and new failure modes. If one well-prompted agent using two reliable tools reaches the required quality, adding five agents is not sophistication. It is overhead.
The rule we use
Do not add another AI because you want another opinion. Add another component because it provides a different capability.
AI should not argue for the sake of arguing. It should produce information capable of changing the final answer.
References & further reading
Build with DestinPQ
Production AI should survive the real world.
We design AI agents, integrations and full-stack products around real business constraints: reliability, observability, safety, cost and maintainability.
Talk to DestinPQ →