1 SummaryQuestion accuracy
81.3%
95% CI [79.5, 83.3], 1535 questions
Items fully correct
64.0%
508 items
Right when the kit acts
902 of 963
answers past the act-on-it thresholds
Wrong actions, default policy
37 of 200
policy decisions replayed
Misses to review
287
see Error analysis
Latency per request
2617 ms
p95 11785 ms
Headline over coding recipes; vision recipes in the capability panel (they counted toward the headline before the 2026-10-10 benchmark foundations, so earlier published headlines are not comparable).
Key findings Note Overall accuracy
81.3% of 1535 questions were answered correctly (95% CI 79.5% to 83.3%), and 64.0% of 508 items had every question right.
Watch Answers past the kit's thresholds
902 of 963 answers that cleared the kit's act-on-it thresholds (choice confidence >= 0.85; noul P >= 0.80 or <= 0.20) were correct (93.7%).
Watch Where the choice misses fell
3 of 28 choice misses cleared the 0.85 trust line; the most confident miss (effort_072, approach) had confidence 0.93.
Watch Clarify vague requests
Following 'Ask a clarifying question when specific_enough <= 0.20.' on 108 items, an agent would have acted on a vague request 20 times (intent_020, intent_021, intent_027, intent_047, intent_048, intent_049, intent_054, intent_067, intent_078, intent_079, intent_080, intent_081, intent_082, intent_091, intent_092, intent_093, intent_094, intent_106, intent_107, intent_108) and asked an unnecessary question 0 times.
Watch Stop risky changes for review
Following 'Stop and request review when needs_review >= 0.80 or risk >= 2.0 of 3.' on 92 items, an agent would have shipped a risky change unreviewed 17 times (risk_015, risk_016, risk_022, risk_038, risk_045, risk_047, risk_048, risk_072, risk_073, risk_076, risk_081, risk_082, risk_083, risk_084, risk_087, risk_088, risk_089) and requested an unnecessary review 0 times.
Note choice calibration
choice answers are under-confident: mean top-label confidence 0.84 against 92.2% accuracy (ECE 0.083).
Note noul calibration
noul answers are calibrated on average: mean top-label confidence 0.86 against 83.4% accuracy (ECE 0.028).
Note risk answers shrink toward the middle
On the 0 to 3 scale, gold 'Negligible' items average 0.61 and gold 'High' items average 1.98. Thresholds on the expected score work better than rounding it to a level.
Note effort answers shrink toward the middle
On the 0 to 3 scale, gold 'Trivial (minutes)' items average 0.76 and gold 'Large (a day or more)' items average 1.82. Thresholds on the expected score work better than rounding it to a level.
Note Strongest and weakest questions
layer/rust-core is strongest at 100.0%; status/draft is weakest at 0.0%.
