Benchmark design

4Benchmark design

The dataset has 475 items and 811 scored questions: 359 choice, 308 noul, and 176 score. Each line is self-contained: the state, the recipe's fixed question set, a gold label for every question, its source, and a one-line rationale. Other implementations can score it without this harness.

9 items reproduce the worked examples in the agent kit's playbook and sit in dev; the other 466 were written for this benchmark. Items are split into dev (236 items, for tuning) and test (239 items, for reporting), stratified so that every label of every question appears in both. The dataset card below answers the datasheet questions of [12].

Figure 2 Items per recipe, split into dev and test.

Label balance

Question	Gold label	Meaning	dev	test
`intent`	`question`	Explain or answer something; no code changes requested	14	13
	`implement`	Make a change: add, modify, or configure something	13	14
	`investigate`	Look into a problem and report findings	14	13
	`fix`	Repair a specific broken behavior	13	14
`specific_enough`	`True`	The target and desired outcome are both identifiable	32	35
	`False`	The target or desired outcome is unclear	22	19
`failure_kind`	`logic_bug`	The code under test returns a wrong result	12	11
	`environment`	Missing service, dependency, port, file, or configuration	11	12
	`flaky`	Timing, ordering, or nondeterminism; may pass on retry	12	11
	`test_bug`	The test itself is wrong or outdated	11	12
`retry`	`True`	yes	12	11
	`False`	no	34	35
`risk`	`0`	Negligible	12	11
	`1`	Low	11	12
	`2`	Medium	12	11
	`3`	High	11	12
`needs_review`	`True`	yes	23	23
	`False`	no	23	23
`owner`	`api`	HTTP handlers and request validation	8	7
	`storage`	Database, migrations, and persistence	7	8
	`ui`	Frontend rendering and client state	8	7
	`infra`	Build, CI, deployment, and configuration	7	8
	`unclear`	Not enough information to tell	8	7
`effort`	`0`	Trivial (minutes)	7	11
	`1`	Small (under an hour)	15	15
	`2`	Medium (a few hours)	10	8
	`3`	Large (a day or more)	4	6
`approach`	`minimal_patch`	Smallest change that fixes the symptom	9	10
	`refactor`	Restructure the code so the fix is natural	8	11
	`new_component`	Add a new module or service	10	9
	`ask_user`	The trade-off needs a human decision	9	10
`dominant_red`	`True`	Red is the dominant colour	2	2
	`False`	Red is not dominant	2	2
`dominant_colour`	`red`	The image is mostly red or warm-red tones	1	1
	`green`	The image is mostly green tones	1	1
	`blue`	The image is mostly blue tones	1	1
	`mixed`	No single colour clearly dominates	1	1
`alert_level`	`0`	No alert (calm green)	1	1
	`1`	Low (yellow)	1	1
	`2`	Medium (orange)	1	1
	`3`	High (red)	1	1
`colour_changed`	`True`	The dominant colour is different in the two frames	2	2
	`False`	The dominant colour is the same in both frames	2	2

Recipes

Each recipe asks the same fixed question set for every item, because ember weighs the questions in one call jointly.

Intent and readiness

intent_readiness

When: Before changing files in response to a conversational request.
State: The user's latest message, verbatim.
Decides: What the user is asking for, and whether it is specific enough to act on without a clarifying question.
Family: coding

Question	Type	Asks	Options
`intent`	`choice`	What is the user asking the coding agent to do?	question: Explain or answer something; no code changes requested implement: Make a change: add, modify, or configure something investigate: Look into a problem and report findings fix: Repair a specific broken behavior
`specific_enough`	`noul`	Is the request specific enough to act on without asking a clarifying question?	true: The target and desired outcome are both identifiable false: The target or desired outcome is unclear

Failure triage

failure_triage

When: Before retrying or fixing a failing test or command.
State: The test or command, and the relevant error lines.
Decides: The most likely cause, and whether a plain re-run would pass.
Family: coding

Question	Type	Asks	Options
`failure_kind`	`choice`	What most likely caused this failure?	logic_bug: The code under test returns a wrong result environment: Missing service, dependency, port, file, or configuration flaky: Timing, ordering, or nondeterminism; may pass on retry test_bug: The test itself is wrong or outdated
`retry`	`noul`	Is simply re-running likely to make it pass?	yes or no

Change risk

change_risk

When: Before committing, pushing, or merging.
State: A diff summary, the files touched, and the lines changed.
Decides: How risky the change is to ship unreviewed, and whether a person should review it.
Family: coding

Question	Type	Asks	Options
`risk`	`score`	How risky is shipping this change without human review?	0: Negligible 1: Low 2: Medium 3: High
`needs_review`	`noul`	Should a human review this change before it is merged?	yes or no

Routing and ownership

routing

When: When an error needs an owner.
State: The error message and the file or module it came from.
Decides: Which part of the system should handle it. The option set is a template that projects replace with their own.
Family: coding

Question	Type	Asks	Options
`owner`	`choice`	Which part of the system should handle this?	api: HTTP handlers and request validation storage: Database, migrations, and persistence ui: Frontend rendering and client state infra: Build, CI, deployment, and configuration unclear: Not enough information to tell

Effort and approach

effort_approach

When: When planning a task.
State: The task and its constraints.
Decides: How much work the task is, and which approach fits. The option sets are templates.
Family: coding

Question	Type	Asks	Options
`effort`	`score`	How much work is this task?	0: Trivial (minutes) 1: Small (under an hour) 2: Medium (a few hours) 3: Large (a day or more)
`approach`	`choice`	Which approach best fits the constraints?	minimal_patch: Smallest change that fixes the symptom refactor: Restructure the code so the fix is natural new_component: Add a new module or service ask_user: The trade-off needs a human decision

Vision: dominant colour, yes/no

vision_noul

When: Smoke check of the vision path.
State: One synthetic colour swatch.
Decides: Whether one colour dominates the image.
Family: vision

Question	Type	Asks	Options
`dominant_red`	`noul`	Is the image predominantly red?	true: Red is the dominant colour false: Red is not dominant

Vision: dominant colour, choice

vision_choice

When: Smoke check of the vision path.
State: One synthetic colour swatch.
Decides: Which colour dominates the image.
Family: vision

Question	Type	Asks	Options
`dominant_colour`	`choice`	What is the dominant colour in the image?	red: The image is mostly red or warm-red tones green: The image is mostly green tones blue: The image is mostly blue tones mixed: No single colour clearly dominates

Vision: alert level, score

vision_score

When: Smoke check of the vision path.
State: One synthetic status indicator.
Decides: How severe the alert the indicator signals is.
Family: vision

Question	Type	Asks	Options
`alert_level`	`score`	The image shows a status indicator. How severe is the alert it is signalling?	0: No alert (calm green) 1: Low (yellow) 2: Medium (orange) 3: High (red)

Vision: colour change across frames

vision_video

When: Smoke check of the video path.
State: Two synthetic frames.
Decides: Whether the dominant colour changed between the frames.
Family: vision

Question	Type	Asks	Options
`colour_changed`	`noul`	The frames are from a two-frame video clip. Did the dominant colour change between frame 1 and frame 2?	true: The dominant colour is different in the two frames false: The dominant colour is the same in both frames

Labelling protocol

Labels were judged from state and the option descriptions alone, and were fixed before any model run. *(since dataset version 1e1bb88d47f0)*
needs_review is true exactly when risk is Medium or High, and retry is true only for flaky failures. *(since dataset version 1e1bb88d47f0)*
Drafts whose evidence fit two options were rewritten or dropped while authoring, before any run. *(since dataset version 1e1bb88d47f0)*
A label changes only when a reviewer finds it wrong on its merits, never because the model disagreed. A change to any item's state, media, questions, gold, or split alters the dataset's content hash, which each run records, so runs on different label sets cannot be compared by accident; annotations (kind, tier, source, rationale) do not. *(since dataset version 1e1bb88d47f0)*
Every score scale runs from least to most of the quantity its instruction names; index 0 is always the least. The expected score is read in those units. A scale is never reversed to make a label read naturally. *(since dataset version 1e1bb88d47f0)*
For a noul question, P(true) is the probability of the condition the instruction names; the condition is phrased so that true is the pole that calls for caution. *(since dataset version 1e1bb88d47f0)*

Policy in force: no policy slot was filled for this run, so every recipe applied its default criteria.

Dataset card

Why was it built?: To measure whether ember's probabilities are accurate and calibrated enough for agents to act on the agent kit's thresholds, and to compare model revisions and setups.
What does it contain?: 475 items across five recipes, 811 scored questions, a gold label per question, a rationale per item.
How was it collected?: 9 items reproduce the agent kit's worked examples; 466 were written by the ember authors. No user data.
How was it labelled?: From state and the option descriptions alone, fixed before any run, under the rules in the labelling protocol.
What is it for?: Tune on dev, report on test. It is not a measure of agent behaviour, and not a training set.
How is it maintained?: Relabelling happens only on review, changes the dataset's SHA-256, and invalidates comparisons with earlier runs.

Generated card

Generated from the dataset this run scored (ember eval dataset card), version 1e1bb88d47f0; the per-label counts per split for every question are in evals/dataset/CARD.md. Label stability and inter-source agreement are unmeasured until Phase 1.

Recipe	Items (dev / test)	Kinds	Sources	Easy	Rater count	External sources
`change_risk`	92 (46 / 46)	untagged 92	authored 92	48	1	none
`effort_approach`	76 (36 / 40)	untagged 76	authored 76	25	1	none
`failure_triage`	92 (46 / 46)	untagged 92	authored 92	87	1	none
`intent_readiness`	108 (54 / 54)	untagged 108	authored 108	88	1	none
`routing`	75 (38 / 37)	untagged 75	authored 75	73	1	none
`vision_choice`	8 (4 / 4)	untagged 8	authored 8	8	1	none
`vision_noul`	8 (4 / 4)	untagged 8	authored 8	8	1	none
`vision_score`	8 (4 / 4)	untagged 8	authored 8	8	1	none
`vision_video`	8 (4 / 4)	untagged 8	authored 8	8	1	none