We had a result ready to publish: the orchestrator knows when to stay silent, and our decision rule was throwing that knowledge away. Then we ran the same measurement on the model that actually ships, and the diagnosis inverted. The difference between them is five rows out of fifteen — not distinguishable from noise — so what needs fixing is the evaluation, not the training.
We had a result we were ready to publish. We tested it on a second model first, because the test was cheap. It did not survive, and the reason it did not survive is more useful than the result would have been.
The orchestrator has three moves: stay SILENT, DISPATCH a research arm, or NOTE a correction into the worker's stream. Silence is the default and the one we have never got right. Our episode ids encode which scenario generated them, and one scenario is entirely gold-SILENCE — cases where saying nothing is correct — while three others are entirely gold-not-silent. Those are the two poles of restraint, and until this week nobody had scored them separately.
On the 8B merged model the answer looked clean
At argmax it gets 13 of 15 restraint cases right, well above guessing. It pays for that everywhere else: on the never-silent scenario it scores 45%.
Sweeping the decision threshold showed the two scenarios are not simply traded against each other. If there were no real signal and we were only moving a global "how often do I stay quiet" dial, the two scores would sum to roughly a constant. They do not. The sum peaks in the middle.
silence offset restraint never-silent sum 0.0 (argmax) 87% 45% 132 -3.0 80% 73% 153 -4.0 73% 82% 155 -6.0 27% 82% 108
A peak in the middle means information the decision rule was discarding. The reading: the model knew when to hold back and argmax threw it away. Recalibrating — three numbers, swept offline, no retraining — takes the never-silent scenario from 45% to 91% and overall accuracy from 76.6% to 84.8%.
That was the entry we were ready to publish.
Then we ran it on the model that actually ships
A 7B with a dedicated adapter. Thirty-three minutes on two T4s, identical method, identical held-out rows.
at argmax 8B merged 7B shipping overall 76.6% 85.6% restraint scenario 87% (13/15) 53% (8/15) never-silent scenario 45% 73% predictions on the 15 restraint SILENT 13 DISPATCH 7 rows DISPATCH 2 SILENT 8
The 7B is the better model overall and already close to its own ceiling — 85.6% at argmax against 88.9% at the best threshold anywhere. But it does not appear to recognise restraint cases at all: it dispatches on nearly half of them. We swept the entire two-dimensional threshold space and no operating point lifts its restraint score above 60%, and reaching even that collapses overall accuracy to 67%.
So the diagnosis inverts. On the 8B, restraint is known and mis-read. On the 7B, the threshold is close to right and the signal is weak. The conclusion did not generalise one model sideways.
The part that matters more: we cannot actually tell
That comparison is 13 of 15 against 8 of 15. Five rows.
8B restraint above chance p = 3.1e-05 clearly yes
7B restraint above chance p = 0.088 not clearly
8B vs 7B difference p = 0.109 not significant
(Fisher exact, two-tailed)
The difference the entire generalisation claim rests on is not statistically distinguishable from noise. Both readings survive: a real difference between the models, or a fifteen-row coin landing differently twice.
We could have published the first result on its own. It was clean, it was surprising, it argued against our own prior work, and nothing in it was wrong — it is still a correct statement about that model. What makes it unpublishable as a general claim is a test we only ran because it was cheap.
The failure mode is worth naming, because it is not the one we were guarding against. The withdrawn draft stated its sample size correctly. It said n=15, it said one row moves the number by 6.7 points, and it said so in its own section rather than a footnote. And it still carried a general claim on top of that. Stating a caveat is not the same as respecting it.
What we are changing
Not the training. The evaluation. The restraint scenario is 15 rows of a 389-row held-out slice. Every restraint experiment we run against it will keep returning p of about 0.1 and licensing whichever conclusion the reader brought with them. Enlarging that slice is now a precondition for any further claim in this area, ours included.
What still stands
On the 8B the restraint signal is real and argmax discards it. The threshold sweep's failure to conserve the sum is far outside what a global prior shift could produce, and that remains true whatever the 7B does.
The 7B is the better model overall and is already well calibrated. And recalibration is cheap: three numbers swept offline against logits we had already recorded, against 95 minutes of GPU per data point back when we were doing this by editing the training corpus.
What we are not claiming. The recalibration figures are offline projections, re-scored from recorded logits under a candidate threshold chosen by a rule fixed in advance. Nothing has run live with that threshold applied. The projection was also fitted against our own corpus's gold labels, and corpus accuracy improving is not the same claim as live task performance improving. Only a live run settles that.
Reproduce with scenario_slice.py in the lab directory. Inputs are the archived deploy reports and two sets of recorded action logits.