All research

/en/research · what we measured

Agents debated and voted, but a single model proved more accurate. Why we switched to state exchange

Three experts' vote yielded 0.848 vs. 0.893 for a single model, a debate between two models performed 7.7 p.p. [−13.0; −2.3] worse than one judge, and an ideal exchange would add no more than 2 p.p. We explain why opinion exchange doesn't work and describe a method where agents exchange not verdicts, but states: what they know, what they've done, and what they're unsure about. The method itself has not been measured yet, and we say so plainly.

Authors: slavb18
  • AI
  • LLM
  • Agents
  • Multi-Agent
  • Research
  • HRTech

Three experts vote on a verdict and achieve 0.848. A single model, trained on the same data, achieves 0.893. Two strong agents debate before a judge, and the judge makes more mistakes than without the debate.

For six months, we tried to assemble a council of agents that would make better decisions than an individual. The council never won a single time. Below are three experiments that demonstrated this, and the method we arrived at afterward.

Task and Testbed

The decision is binary: decide from a technical interview transcript whether a candidate is a senior or not. Transcripts are synthetic, with known ground truth. First, a generator sets a hidden candidate profile (level, where they bluff, what skills they know), then a dialogue is written based on this profile. The correct answer is known by construction, so accuracy is calculated without human annotators.

The held-out sample contains 112 interviews, plus 149 with skills models haven't seen during training (ClickHouse, Linux). A total of 261 records. Confidence intervals are bootstrapped across records everywhere, and hypotheses were pre-registered before running.

Experiment 1. Council of Experts vs. One Model

We took Qwen2.5-3B in 4 bits and trained three LoRA adapters for it, each on its own third of the training set. A fourth adapter, full, was trained on the entire dataset.

accuracy, held-out accuracy, new skills
base model without adapter 0.402 0.571
best of three experts 0.857 0.866
voting by three experts 0.848 0.832
one model on all data 0.893 0.899

Voting loses even to the best of its participants. Then we had the experts revise their verdict after seeing a colleague's opinion, either as text or via hidden states. The net effect of revision (corrected minus corrupted) was close to zero in all variants: from −0.019 to +0.006, and all intervals include zero.

However, the reverse check worked. When a "colleague" deliberately stated the opposite of the truth, the experts yielded in 36.7% of cases. A model that was right saw another's incorrect opinion in the prompt and changed its answer. We analyzed this experiment and the attempt to find a "retained opinion" within the model in detail in the article "Expert Yielded, but Probe Says They Stood Their Ground. Model Without an Opinion Reads the Same".

It turns out that another's error easily passes through the opinion channel, while corrections are almost non-existent.

Experiment 2. How much can be gained at all?

Perhaps the exchange was poorly designed? We calculated the ceiling: an oracle that, for each interview, knows which expert is correct and takes their answer.

Over a single model, such an oracle adds +0.9 p.p. [−2.7; 4.5] on the held-out sample and +2.0 p.p. [−1.3; 6.0] on new skills. No exchange can be better than this, and even its gain is indistinguishable from zero.

The reason lies in error correlation. Experts disagree in only 6% of pairs. Yule's Q between their errors is 0.95-0.98: if one made a mistake, almost certainly all of them did. Three adapters on the same base and same labels make mistakes like a single model, and they have nothing to correct each other with.

Hence the first rule, which later formed the basis of the method: exchange only makes sense if the participants have different content. Three copies of the same knowledge, no matter how they are combined, yield the same knowledge.

Experiment 3. Debate of Different Models

So, different models are needed. We set up a classic debate scheme: two models defend assigned sides, a judge reads their arguments, and makes a decision. deepseek-v4.1-flash and gpt-oss-120b played the sides, and Qwen 27B was the judge. Each record was processed twice, with side swapping, so that the position, not the model, won. Quotes from the transcript referenced by the debaters were verified by code, and the judge saw a flag indicating whether the quote was confirmed or not.

variant accuracy (n = 261)
voting by three models 0.843
quorum: each sees arguments of two others and revises 0.835
judge alone with transcript 0.820
judge with transcript and debate 0.743
judge with debate only, no transcript 0.640
judge with one advisor 0.498

The debate performs worse than a single judge by −7.7 p.p. [−13.0; −2.3]. It corrected 6.3% of verdicts and corrupted 14.0%. For bluffing candidates, the debate's false "senior" rate was 37.5%, compared to 28.1% for a single judge: a convincing defender pulls through someone who should have been rejected.

The most unpleasant finding was in quote verification. The side defending an incorrect conclusion cited confirmed quotes just as often as the side with the correct one. The quote is genuine, but the conclusion drawn from it is false. Verifying quotes against the source does not catch this.

The debate is still better than one advisor, +14.2 p.p. [+7.5; +20.9]. But this is a comparison with a situation where half of the advice is intentionally incorrect. A quorum over the best single model yielded +1.5 p.p. [−2.3; +5.8], which is not proven.

What Unites the Three Failures

In all three experiments, agents transmitted a conclusion to each other: a verdict, a vote, an argument in favor of a side. A conclusion compresses all reasoning into a single word, and it is transmitted as a conviction. The recipient cannot verify it; they can only accept or reject it. They accept it based on the force of conviction, and thus yield under pressure in a third of cases.

However, what was actually needed for the decision did not flow through the channel:

  • what facts the agent saw and their source;
  • what it had already checked and how;
  • where it lacked data.

Human consultations are structured similarly when they work. A doctor in a case review does not say, "I vote for surgery." They state what analyses they looked at, what they ruled out, and what examination is missing, and the decision is assembled from these reports.

Method: State Exchange

Agents work in a shared room but do not exchange verdicts. Each publishes its state, a three-part record.

What I know. Facts with source indication: an utterance in the transcript, a document, an answer from an external system. Without evaluations. "Candidate mentioned MergeTree and ReplacingMergeTree, utterances 14 and 17" fits here. "Knows ClickHouse well" is already a conclusion; it's not written into the state.

What I've done. What actions the agent has already performed: what question it asked, what it checked, what tool it called, and what it received. This is needed so that two agents don't do the same thing and so that it's clear what each fact rests upon.

What I'm unsure about. What is missing for a decision: what fact contradicts another, what question remains unanswered, where data ran out. This is the most valuable part, and agents in a debate hide it: admitting uncertainty means losing the round.

The one who assembles the decision reads the states. Facts from different agents combine into a single picture, contradictions between them become apparent and are resolved by source, not by who is more convincing. If the sum of states still contains unaddressed "unsure," this is visible before the decision, not after an error.

How this differs from debate and voting

voting debate state exchange
what is transmitted verdict argument for an assigned side facts, actions, gaps
verifiable by source no quote, but not conclusion every fact
pressure on recipient majority rhetorical force no verdict to pressure with
uncertainty lost in the vote hidden transmitted explicitly
gain from copies of one model none (experiment 2) none none: different content needed

State exchange does not cure error correlation. If three agents have the same data and the same base model, their states will coincide, and the method will yield the same as a single model. A gain is possible where participants have different things: one read the transcript, another checked the take-home assignment, a third looked at the candidate's project history.

What We Don't Yet Know

We have not yet measured state exchange itself. The numbers above relate to voting, revision, and debate. They explain why we moved away from these schemes but do not prove that the new scheme is better. The next step: pre-registration of an experiment with state exchange, and we will publish the result, including negative ones.

The data is synthetic. Candidate competence in our transcripts is visible from the first turn, and the task is almost equally solved by TF-IDF with logistic regression (0.893 on the held-out sample, like the best LLM). On such a dataset, methods don't differentiate well. The difference between decision schemes needs to be checked on real interviews where competence indicators are not lexical.

Each experiment was run once per model. Intervals are honest regarding records but not regarding the randomness of training.

In Experiment 3, sides and judge operated on cloud weights at full precision. On a local card with quantized models, the picture might be worse.

What This Implies for Those Building Agent Systems

Several agents based on one model and the same data are no more accurate than one: for us, the ceiling of ideal exchange came out at two percentage points with an interval crossing zero. Debate should be checked against a single judge on your own data; for us, it lost by 7.7 percentage points, and quote verification did not distinguish an honest debater from a dishonest one.

When agents do have something to share, let them share what they've seen and what they don't know, and let the one who assembles the decision make the final verdict.

Early experiments with a council of LoRA experts and latent exchange are open: preprint Latent Debate Between LoRA Experts on Zenodo and code on github.com/iconicompany/latent-debate.

All research