/en/research · what we measured
The LoRA-expert yielded to a stooge colleague in 36.7% of cases, and a linear probe of the hidden state recovered their original verdict with an AUC of 0.74. After deducting transcript features, it remained 0.63, while the model without an adapter gave 0.59: a difference of +0.04, with the CI including zero. We also calculated that an ideal exchange between three experts would add no more than 2 p.p.
Yesterday I had a beautiful result. An expert in an interview yields under pressure from a colleague, changes their verdict, but a linear probe of their hidden state shows that 'privately' they maintained their original opinion. AUC 0.74 on 287 cases, with the confidence interval far from 0.5.
Today I withdrew it. The same probe, applied to a model that never had its own opinion, reads the 'maintained verdict' no worse.
A panel of experts evaluates a technical interview: whether a candidate is senior or not. Under the hood is a single frozen Qwen2.5-3B in 4 bits and LoRA adapters for it. Three experts are trained on non-overlapping thirds of the training set, while a fourth model, full, is trained on the entire set. The interviews are synthetic, with known ground truth: first, a candidate profile is set, then a transcript is generated based on it. Benchmark v3, a held-out set of 112 interviews, and another 149 on skills that the models had not seen during training (hereafter OOD). A single split for all experiments: seed 0, held-out 25%.
In experiment E13, we tested whether experts could correct each other. Each was shown a colleague's verdict and asked to reconsider their own. One arm of the experiment was called contra: the 'colleague' always stated the opposite, regardless of the truth. This was a test for conformism.
Experts yielded in 36.7% of cases: a model that was correct saw a different, incorrect opinion in the prompt and changed its answer. The checks below do not touch this behavioral result.
Next, we wanted to understand what happens inside an expert who yielded. There's known work on 'silent disagreement': the model yields in its answer but continues to hold its original position in its internal representation. If so, this could be read and used: the expert agreed out loud, but the latent state suggests they are doubtful.
The scheme is simple. We take the hidden state of the last token on layers 9, 18, 27, and 36. We train logistic regression (C = 0.05) to predict the expert's original verdict using states from a run without pressure, on different interviews, five folds split by interview. Then we apply it to the states of the same expert in the contra run and look only at those who yielded.
| layer | AUC on those who yielded | AUC on those who stood their ground |
|---|---|---|
| 9 | 0.739 [0.683; 0.788] | 0.684 |
| 18 | 0.707 [0.652; 0.756] | 0.240 |
| 27 | 0.150 | 0.964 |
| 36 | 0.112 | 0.986 |
On layers 9 and 18, the probe sees the old verdict, while on 27 and 36, it sees the new, declared one. The picture is consistent with the paper: the opinion resides in the middle of the network, and towards the output, it's overridden by the concession.
One line was concerning. On layer 9, the probe performs almost as well on those who stood their ground (0.684). If it's reading a maintained premise, why do those who stood their ground show similar accuracy? They didn't maintain anything; they simply didn't change their answer.
The expert's verdict depends on the transcript. In interviews where a candidate rattles off numbers and incidents, the expert more often recognizes them as senior. The hidden state of the last token, of course, encodes the transcript. This means the probe might recover the 'original verdict' simply because it sees the same evidence that the expert once used to make that judgment. No maintained opinion is needed for this.
To separate these explanations, in E14 I recorded the hypothesis and threshold before calculation (27.10.2026) and added controls.
Residualization. We collect features that describe the transcript without any LLM: the logit of a TF-IDF+LR trained on train, length in characters and words, number of turns, share and mean length of the candidate's turns, length of the interviewer's turns, share of digits and Latin characters, skill, generator, expert. In each fold a linear regression predicts the hidden state from these features, and the prediction is subtracted. The probe is trained and applied on the residuals.
Threshold. AUC on those who yielded, bootstrap over interviews (2000 resamples), 98.75% CI (Bonferroni over four layers). The hypothesis survives if the lower bound exceeds 0.5 on at least one layer.
A model with no opinion (K1). The same pipeline on the base Qwen without an adapter. This model never formed the expert's verdict. Whatever it "knows" about that verdict it took from the transcript.
Three more checks: shuffled labels (should give 0.5, a pipeline sanity check), the subsample where the expert's original verdict was wrong (there the premise and the transcript pull in opposite directions), and the probe's gain over transcript features alone.
| layer | raw | after residualization | model without adapter | difference |
|---|---|---|---|---|
| 9 | 0.700 | 0.629 [0.520; 0.736] | 0.586 [0.475; 0.689] | +0.043 [−0.089; 0.164] |
| 18 | 0.468 | 0.558 [0.458; 0.652] | 0.572 | −0.014 [−0.147; 0.123] |
Formally the hypothesis survived: on layer 9 the lower bound is 0.520, a margin of two hundredths. From this row alone one could write "the latent keeps the opinion even after controls".
The other checks do not allow it:
The probe reads the transcript. The claim "an expert who yielded keeps their opinion in the latent" is not supported by this data, and it is no longer part of the paper plan.
There is one more limitation I wrote down in advance: the contra prompt contains the text "Your initial assessment: pass=…". Even a positive result would not have told an internal opinion apart from reading that line. We simply never got to that question.
The idea of a council rests on experts making different mistakes and correcting each other. In E13 none of the honest exchange arms (text, KV cache, no message) produced a net effect: all were near zero. Before spending GPU on the next version of exchange (E18), I computed on saved predictions how much could be gained at all. That is E21, CPU only.
Definitions follow 2610.04686. Recoverable headroom: the share of examples where at least one expert is right and the final decision is wrong. Oracle: a model that always knows which expert to listen to.
| held-out (112) | OOD (149) | |
|---|---|---|
full accuracy |
0.893 | 0.899 |
| majority vote of three experts | 0.848 | 0.832 |
three-expert oracle minus full |
+0.9 [−2.7; 4.5] p.p. | +2.0 [−1.3; 6.0] p.p. |
| all four LoRA wrong at once | 8.0% | 6.0% |
Ideal exchange, which no real protocol will reach, gives less than two points over a single model, and the interval includes zero. The experts disagree in only 6% of pairs.
The reason shows in error correlation. Yule's Q within LoRA is 0.98 on held-out and 0.96 on OOD: adapters on the same base fail on the same interviews even though they were trained on different data. TF-IDF fails in the same places on familiar skills (φ with full 0.72) and almost independently on unfamiliar ones (φ 0.05). An oracle "full or TF-IDF" on OOD would give +8.1 p.p. But stacking with logistic regression over probabilities does not extract it: +0.7 p.p. [−3.4; +4.7]. The pair errs differently, yet a linear reader over probabilities cannot pick whom to trust.
Decision: we do not run E18 on this benchmark. It makes sense only on data where majority-vote headroom is at least 15%, and ours is 8.7%.
What still holds is visible in the answers without any interpretability: under pressure an expert yields in a third of cases. The story about an opinion hidden in the latent rested on the fact that we never asked the model with no opinion.