1 SummaryQuestion accuracy
83.2%
95% CI [79.5, 86.8], 424 questions
Items fully correct
74.6%
232 items
Right when the kit acts
234 of 234
answers past the act-on-it thresholds
Wrong actions, default policy
14 of 104
policy decisions replayed
Misses to review
71
see Error analysis
Latency per request
933 ms
p95 1091 ms
Headline over coding recipes; vision recipes in the capability panel (they counted toward the headline before the 2026-10-10 benchmark foundations, so earlier published headlines are not comparable).
Key findings Note Overall accuracy
83.2% of 424 questions were answered correctly (95% CI 79.5% to 86.8%), and 74.6% of 232 items had every question right.
Holds Answers past the kit's thresholds
234 of 234 answers that cleared the kit's act-on-it thresholds (choice confidence >= 0.85; noul P >= 0.80 or <= 0.20) were correct (100.0%).
Holds Where the choice misses fell
0 of 12 choice misses cleared the 0.85 trust line; the most confident miss (effort_036, approach) had confidence 0.81.
Watch Clarify vague requests
Following 'Ask a clarifying question when specific_enough <= 0.20.' on 56 items, an agent would have acted on a vague request 7 times (intent_020, intent_021, intent_027, intent_047, intent_048, intent_049, intent_054) and asked an unnecessary question 0 times.
Watch Stop risky changes for review
Following 'Stop and request review when needs_review >= 0.80 or risk >= 2.0 of 3.' on 48 items, an agent would have shipped a risky change unreviewed 7 times (risk_015, risk_016, risk_022, risk_038, risk_045, risk_047, risk_048) and requested an unnecessary review 0 times.
Note choice calibration
choice answers are under-confident: mean top-label confidence 0.84 against 93.8% accuracy (ECE 0.100).
Note noul calibration
noul answers are under-confident: mean top-label confidence 0.83 against 91.7% accuracy (ECE 0.100).
Note risk answers shrink toward the middle
On the 0 to 3 scale, gold 'Negligible' items average 0.60 and gold 'High' items average 2.07. Thresholds on the expected score work better than rounding it to a level.
Note effort answers shrink toward the middle
On the 0 to 3 scale, gold 'Trivial (minutes)' items average 0.75 and gold 'Large (a day or more)' items average 1.74. Thresholds on the expected score work better than rounding it to a level.
Note Strongest and weakest questions
colour_changed is strongest at 100.0%; effort is weakest at 42.5%.
