1 SummaryQuestion accuracy
82.0%
95% CI [79.3, 84.6], 811 questions
Items fully correct
72.5%
443 items
Right when the kit acts
415 of 420
answers past the act-on-it thresholds
Wrong actions, default policy
37 of 200
policy decisions replayed
Misses to review
146
see Error analysis
Latency per request
251 ms
p95 275 ms
Headline over coding recipes; vision recipes in the capability panel (they counted toward the headline before the 2026-10-10 benchmark foundations, so earlier published headlines are not comparable).
Key findings Note Overall accuracy
82.0% of 811 questions were answered correctly (95% CI 79.3% to 84.6%), and 72.5% of 443 items had every question right.
Watch Answers past the kit's thresholds
415 of 420 answers that cleared the kit's act-on-it thresholds (choice confidence >= 0.85; noul P >= 0.80 or <= 0.20) were correct (98.8%).
Watch Where the choice misses fell
3 of 28 choice misses cleared the 0.85 trust line; the most confident miss (effort_072, approach) had confidence 0.93.
Watch Clarify vague requests
Following 'Ask a clarifying question when specific_enough <= 0.20.' on 108 items, an agent would have acted on a vague request 20 times (intent_020, intent_021, intent_027, intent_047, intent_048, intent_049, intent_054, intent_067, intent_078, intent_079, intent_080, intent_081, intent_082, intent_091, intent_092, intent_093, intent_094, intent_106, intent_107, intent_108) and asked an unnecessary question 0 times.
Watch Stop risky changes for review
Following 'Stop and request review when needs_review >= 0.80 or risk >= 2.0 of 3.' on 92 items, an agent would have shipped a risky change unreviewed 17 times (risk_015, risk_016, risk_022, risk_038, risk_045, risk_047, risk_048, risk_072, risk_073, risk_076, risk_081, risk_082, risk_083, risk_084, risk_087, risk_088, risk_089) and requested an unnecessary review 0 times.
Note choice calibration
choice answers are under-confident: mean top-label confidence 0.84 against 92.2% accuracy (ECE 0.083).
Note noul calibration
noul answers are under-confident: mean top-label confidence 0.82 against 90.3% accuracy (ECE 0.095).
Note risk answers shrink toward the middle
On the 0 to 3 scale, gold 'Negligible' items average 0.61 and gold 'High' items average 1.98. Thresholds on the expected score work better than rounding it to a level.
Note effort answers shrink toward the middle
On the 0 to 3 scale, gold 'Trivial (minutes)' items average 0.76 and gold 'Large (a day or more)' items average 1.82. Thresholds on the expected score work better than rounding it to a level.
Note Strongest and weakest questions
colour_changed is strongest at 100.0%; effort is weakest at 42.1%.
