1 SummaryQuestion accuracy
80.8%
95% CI [78.8, 82.8], 1565 questions
Items fully correct
63.6%
538 items
Right when the kit acts
904 of 967
answers past the act-on-it thresholds
Wrong actions, default policy
37 of 200
policy decisions replayed
Misses to review
300
see Error analysis
Latency per request
2971 ms
p95 12387 ms
Headline over coding recipes; vision recipes in the capability panel (they counted toward the headline before the 2026-10-10 benchmark foundations, so earlier published headlines are not comparable).
Key findings Note Overall accuracy
80.8% of 1565 questions were answered correctly (95% CI 78.8% to 82.8%), and 63.6% of 538 items had every question right.
Watch Answers past the kit's thresholds
904 of 967 answers that cleared the kit's act-on-it thresholds (choice confidence >= 0.85; noul P >= 0.80 or <= 0.20) were correct (93.5%).
Watch Where the choice misses fell
5 of 41 choice misses cleared the 0.85 trust line; the most confident miss (effort_072, approach) had confidence 0.93.
Watch Clarify vague requests
Following 'Ask a clarifying question when specific_enough <= 0.20.' on 108 items, an agent would have acted on a vague request 20 times (intent_020, intent_021, intent_027, intent_047, intent_048, intent_049, intent_054, intent_067, intent_078, intent_079, intent_080, intent_081, intent_082, intent_091, intent_092, intent_093, intent_094, intent_106, intent_107, intent_108) and asked an unnecessary question 0 times.
Watch Stop risky changes for review
Following 'Stop and request review when needs_review >= 0.80 or risk >= 2.0 of 3.' on 92 items, an agent would have shipped a risky change unreviewed 17 times (risk_015, risk_016, risk_022, risk_038, risk_045, risk_047, risk_048, risk_072, risk_073, risk_076, risk_081, risk_082, risk_083, risk_084, risk_087, risk_088, risk_089) and requested an unnecessary review 0 times.
Note choice calibration
choice answers are under-confident: mean top-label confidence 0.82 against 89.5% accuracy (ECE 0.073).
Note noul calibration
noul answers are calibrated on average: mean top-label confidence 0.86 against 83.4% accuracy (ECE 0.028).
Note risk answers shrink toward the middle
On the 0 to 3 scale, gold 'Negligible' items average 0.61 and gold 'High' items average 1.98. Thresholds on the expected score work better than rounding it to a level.
Note effort answers shrink toward the middle
On the 0 to 3 scale, gold 'Trivial (minutes)' items average 0.76 and gold 'Large (a day or more)' items average 1.82. Thresholds on the expected score work better than rounding it to a level.
Note Strongest and weakest questions
layer/rust-core is strongest at 100.0%; status/draft is weakest at 0.0%.
