/en/research · what we measured
We tested conformal prediction as a 'when to involve a human' rule in senior screening. Coverage on unfamiliar skills is 0.913, 4% of interviews are escalated to a human, but among automatically promoted candidates, there are 14.9% errors. A simple probability threshold of 0.9 from the model did not violate coverage in any of the five shifts: the preregistered hypothesis was disproven.
Conformal prediction promises: 90% of model answers will be correct, even if the model is poorly calibrated. In candidate screening, this looks like a ready-made rule for when to involve a human. We tested it on interviews for skills that the model had not seen during training. The promise was fulfilled: coverage of 0.913. However, among candidates automatically advanced by the system, 15% turned out to be non-seniors: the guarantee applies to a different metric.
The model reads a transcript of a technical interview and decides whether the candidate is a senior or not. This is a full LoRA-adapter on Qwen2.5-3B from previous experiments, with an accuracy of approximately 0.89 on the held-out set. The question is which interviews it can be trusted with without human intervention.
The policy is asymmetric, and this is important for everything that follows:
The conformal part constructs a set of possible labels for each interview. Nonconformity s = 1 − p̂(y | x), the threshold q̂ is taken as the quantile ⌈(n+1)(1−α)⌉/n from the calibration set, α = 0.1. The set {1} means promote, {0} means reject for confirmation, {0, 1} or empty means escalation. Guarantee: the true label falls into the set in at least 90% of cases, if the calibration and new data are identically distributed.
Interviews are synthetic, with known ground truth (benchmark v3). Calibration was performed on 112 interviews from the held-out set. Test: 149 interviews on two skills the model had not seen: ClickHouse and Linux networking. No new model inferences were run; everything was computed using saved probabilities from E13, taking about a minute on a CPU.
Preregistration was recorded on 2026-10-27 before the first run. Five fixed 'calibration → test' shifts: S1 held-out → both new skills; S2 and S3 for one new skill each; S4 and S5 interviews generated by a model not present in calibration (deepseek-v4.1-flash and qwen3.8-27b).
Thresholds:
| threshold | meaning |
|---|---|
| P1: coverage on S1 ≥ 0.88 | guarantee holds on unfamiliar skills (0.90 with 2 percentage points tolerance) |
| P2: escalation on S1 ≤ 50% | no more than half are sent to a human |
| P3: simple threshold violates coverage by ≥ 5 percentage points in 3 out of 5 shifts | this is why conformal calibration is needed |
| disproof: CP is better than threshold by less than 2 percentage points on average in terms of coverage | then the hypothesis is dismissed |
A simple threshold is taken as a baseline from 2604.07667: the model answers itself if max(p, 1−p) ≥ 0.9, otherwise escalation. No calibration, model probability is taken as is.
| threshold | value | outcome |
|---|---|---|
| P1 | 0.913 [0.86; 0.95] | fulfilled |
| P2 | escalation 4.0%, autonomy 45.0%, batch rejection 51.0% | fulfilled |
| P3 | 0 out of 5: threshold coverage 0.973-0.987 | not fulfilled |
| disproof | CP − threshold = −6.24 percentage points | triggered |
The first two rows look like a success. The guarantee holds on new skills; one in 25 interviews is sent to a human.
The third row disproves the central hypothesis. A simple threshold of 0.9 did not violate coverage in any shift; on the contrary, it achieved higher coverage. Our LoRA-adapter keeps probabilities close to the poles, so the 0.9 threshold rarely triggers and over-insures: 18-27% escalation versus 3-8% for the conformal method. The methods differ in the balance of 'how many we hand over to a human / how many errors we make ourselves'.
Now for the unpleasant part. Here's what happens with the automatic part of the conformal method:
| shift | coverage | escalation | error among promoted |
|---|---|---|---|
| S1, both new skills | 0.913 | 4.0% | 14.9% (10 out of 67) |
| S2, ClickHouse | 0.932 | 2.7% | 8.6% |
| S3, Linux networking | 0.893 | 5.3% | 21.9% |
| S4, new generator deepseek | 0.859 | 2.9% | 18.4% |
| S5, new generator qwen | 0.973 | 8.0% | 0.0% |
| threshold 0.9 on S1, for comparison | 0.973 | 22.2% | 7.8% (4 out of 51) |
Marginal coverage counts all responses together. Rejections are easy to cover: the model sees a weak candidate and confidently says 'no'. These correct rejections accumulate to 90%, while false promotions pile up in another corner. And promotion is our only action without human involvement.
A natural attempt to fix this is Mondrian by class: a separate quantile for non-seniors to guarantee coverage specifically for y = 0. On calibration, this yields ≥ 0.9 by construction. On new skills, non-senior coverage is 0.847 (S1), 0.808 (S3), 0.800 (S4). The class-specific guarantee relies on the same exchangeability assumption, and a skill shift breaks it.
Weighted CP (Tibshirani et al., 2019) reweights the calibration set to match the test set based on density ratios. Where there was under-coverage, it fixes it: on S4 0.859 → 0.888, on S3 0.893 → 0.960 at the cost of escalation 5% → 20%. However, the 'calibration or test' classifier perfectly separates them, AUC 1.00 in all shifts, meaning the density ratio estimate is degenerate. Weights remain reasonable only due to logistic regression regularization. I would not call this a working method.
The safety of the automatic part depends more on model bias than on the conformal wrapper. TF-IDF+LR with the same split CP on S1: coverage 0.906, autonomy 22.8%, error among promoted 2.9% (1 out of 34). This model on new skills underestimates seniors, promotes rarely, and with almost no errors. full with the same wrapper promotes twice as often and has a 14.9% error rate.
The implementation was verified separately: on 200 random halves of the combined pool, where exchangeability holds, split CP coverage is 0.897-0.902 for all four models.
We need to directly control the metric we are responsible for: P(not senior | promoted). For this, there are conformal risk control and Learn-then-Test methods, which provide guarantees for arbitrary risk, not just coverage. The next experiment (E23-shift) aims for P(not senior | promoted) ≤ 0.07 with a shift adjustment; the baseline is the same conservative 0.9 threshold, which already yields 7.8% on S1.
Weighted CP only makes sense with density estimation based on features that do not directly reveal the skill (length, style), otherwise AUC 1.00 nullifies its effectiveness.
full on only 112 interviews: no model predictions on the training set, and if there were, they would be in-sample.The conformal guarantee was honestly fulfilled on this data. The mistake was that I expected it to protect promotions, but it only promised coverage.