Benchmark design

4Benchmark design

The dataset has 570 items and 1565 scored questions: 389 choice, 1032 noul, and 176 score. Each line is self-contained: the state, the recipe's fixed question set, a gold label for every question, its source, and a one-line rationale. Other implementations can score it without this harness.

9 items reproduce the worked examples in the agent kit's playbook and sit in dev; the other 561 were written for this benchmark. Items are split into dev (274 items, for tuning) and test (296 items, for reporting), stratified so that every label of every question appears in both. The dataset card below answers the datasheet questions of [12].

Figure 2 Items per recipe, split into dev and test.

Label balance

Question	Gold label	Meaning	dev	test
`intent`	`question`	Explain or answer something; no code changes requested	14	13
	`implement`	Make a change: add, modify, or configure something	13	14
	`investigate`	Look into a problem and report findings	14	13
	`fix`	Repair a specific broken behavior	13	14
`specific_enough`	`True`	The target and desired outcome are both identifiable	32	35
	`False`	The target or desired outcome is unclear	22	19
`failure_kind`	`logic_bug`	The code under test returns a wrong result	12	11
	`environment`	Missing service, dependency, port, file, or configuration	11	12
	`flaky`	Timing, ordering, or nondeterminism; may pass on retry	12	11
	`test_bug`	The test itself is wrong or outdated	11	12
`retry`	`True`	yes	12	11
	`False`	no	34	35
`risk`	`0`	Negligible	12	11
	`1`	Low	11	12
	`2`	Medium	12	11
	`3`	High	11	12
`needs_review`	`True`	yes	23	23
	`False`	no	23	23
`owner`	`api`	HTTP handlers and request validation	8	7
	`storage`	Database, migrations, and persistence	7	8
	`ui`	Frontend rendering and client state	8	7
	`infra`	Build, CI, deployment, and configuration	7	8
	`unclear`	Not enough information to tell	8	7
`effort`	`0`	Trivial (minutes)	7	11
	`1`	Small (under an hour)	15	15
	`2`	Medium (a few hours)	10	8
	`3`	Large (a day or more)	4	6
`approach`	`minimal_patch`	Smallest change that fixes the symptom	9	10
	`refactor`	Restructure the code so the fix is natural	8	11
	`new_component`	Add a new module or service	10	9
	`ask_user`	The trade-off needs a human decision	9	10
`dominant_red`	`True`	Red is the dominant colour	2	2
	`False`	Red is not dominant	2	2
`dominant_colour`	`red`	The image is mostly red or warm-red tones	1	1
	`green`	The image is mostly green tones	1	1
	`blue`	The image is mostly blue tones	1	1
	`mixed`	No single colour clearly dominates	1	1
`alert_level`	`0`	No alert (calm green)	1	1
	`1`	Low (yellow)	1	1
	`2`	Medium (orange)	1	1
	`3`	High (red)	1	1
`colour_changed`	`True`	The dominant colour is different in the two frames	2	2
	`False`	The dominant colour is the same in both frames	2	2
`domain/tooling`	`True`	The note calls for `domain/tooling`: Build, CI, observer, TTS pipeline, dev-time tools	11	16
`layer/infra`	`True`	The note calls for `layer/infra`: Build, CI, deployment infrastructure	3	3
`type/code`	`True`	The note calls for `type/code`: Code architecture, autoloads, conventions, tooling. Carries `code-refs:`. (e.g. `Code/Autoloads — Disturbance.md`)	2	7
	`False`	The note has no reason to carry `type/code`	21	25
`type/decision`	`True`	The note calls for `type/decision`: Architectural or design decision made by an agent during a session. Written to `docs/vault/Decisions/`. (Agent notes only — spec 080.)	5	4
	`False`	The note has no reason to carry `type/decision`	20	30
`type/design`	`True`	The note calls for `type/design`: Design intent, player-facing concepts, world lore. (e.g. `Design/World Model.md`)	4	5
	`False`	The note has no reason to carry `type/design`	12	28
`type/discovery`	`True`	The note calls for `type/discovery`: Non-obvious constraint, gap, or conflict found by an agent during a session. Written to `docs/vault/Discoveries/`. (Agent notes only — spec 080.)	4	5
	`False`	The note has no reason to carry `type/discovery`	20	25
`type/moc`	`False`	The note has no reason to carry `type/moc`	21	28
`type/reference`	`True`	The note calls for `type/reference`: Glossary, decision log, open questions, epics. (e.g. `Reference/Glossary.md`)	3	6
	`False`	The note has no reason to carry `type/reference`	21	22
`type/session-log`	`True`	The note calls for `type/session-log`: Session activity log; permanent audit trail. Written to `docs/vault/Sessions/`. Append-only, never pruned. (Agent notes only — spec 080.)	4	5
	`False`	The note has no reason to carry `type/session-log`	16	29
`type/spec`	`False`	The note has no reason to carry `type/spec`	23	32
`type/system`	`True`	The note calls for `type/system`: A bounded subsystem with implementation. Carries `code-refs`, `spec-refs`, `implements:` frontmatter. (e.g. `Systems/Audio — Car Radio.md`)	4	5
	`False`	The note has no reason to carry `type/system`	18	22
`concept/horror`	`True`	The note calls for `concept/horror`: Horror register and dread design	1	2
`domain/narrative`	`True`	The note calls for `domain/narrative`: Tone, horror register, forest transmissions narrative, Gary/KTLK	2	2
`type/principle`	`True`	The note calls for `type/principle`: Governance and constitutional notes. Non-negotiable rules. (e.g. `Governance/Constitution.md`)	1	1
	`False`	The note has no reason to carry `type/principle`	22	32
`domain/governance`	`True`	The note calls for `domain/governance`: Constitution, phased spec, policies	7	8
`type/vision`	`False`	The note has no reason to carry `type/vision`	22	31
`domain/world`	`True`	The note calls for `domain/world`: World model, biomes, terrain, procedural generation, weather	5	4
`layer/gdscript`	`True`	The note calls for `layer/gdscript`: GDScript (Godot scene-tier logic)	2	6
`status/reviewed`	`True`	The note calls for `status/reviewed`: Agent-created note verified against codebase within its creating session. Eligible for auto-injection in subsequent task dispatches. (Agent notes only — spec 080.)	8	8
`concept/disturbance`	`True`	The note calls for `concept/disturbance`: The disturbance mechanic thread (Article X)	1	1
`domain/player`	`True`	The note calls for `domain/player`: Player mode, input, controls, camera, exit/entry	2	3
`domain/audio`	`True`	The note calls for `domain/audio`: Audio systems: world ambient, radio, transmissions, buses, pipeline	4	5
`concept/ephemeral-world`	`True`	The note calls for `concept/ephemeral-world`: Run-reset, transient worlds (Article IV Rules 6–7)	1	1
`status/draft`	`True`	The note calls for `status/draft`: Authoring in progress, not yet verified	3	5
`layer/python`	`True`	The note calls for `layer/python`: Python development tooling (`tools/`)	0	1
`domain/vehicle`	`True`	The note calls for `domain/vehicle`: The Samba: vehicle physics, dashboard, damage, panels	2	8
`concept/procedural`	`True`	The note calls for `concept/procedural`: Procedural generation, seed determinism	1	1
`domain/anomalies`	`True`	The note calls for `domain/anomalies`: Anomaly taxonomy, individual anomaly systems, spawn, behavior	0	1
`concept/diegetic`	`True`	The note calls for `concept/diegetic`: Diegetic audio / UI / world-sourced design	1	2
`concept/governance`	`True`	The note calls for `concept/governance`: Constitutional and governance concepts	0	1
`concept/contamination`	`True`	The note calls for `concept/contamination`: Across-runs panel and garage degradation; the slow-horror mechanic that doesn't reset with the world	0	1
`domain/art`	`True`	The note calls for `domain/art`: Visual direction, palettes, references, asset pipeline	1	4
`layer/rust-core`	`True`	The note calls for `layer/rust-core`: Engine-agnostic Rust core (`rust/rust-core/`)	0	1
`placement`	`Art`	Place the note under Art/	2	2
	`Code`	Place the note under Code/	2	2
	`Decisions`	Place the note under Decisions/	1	2
	`Design`	Place the note under Design/	1	2
	`Discoveries`	Place the note under Discoveries/	1	2
	`Governance`	Place the note under Governance/	1	2
	`Narrative`	Place the note under Narrative/	0	1
	`Reference`	Place the note under Reference/	1	2
	`Sessions`	Place the note under Sessions/	1	2
	`Systems`	Place the note under Systems/	1	2

Recipes

Each recipe asks the same fixed question set for every item, because ember weighs the questions in one call jointly.

Intent and readiness

intent_readiness

When: Before changing files in response to a conversational request.
State: The user's latest message, verbatim.
Decides: What the user is asking for, and whether it is specific enough to act on without a clarifying question.
Family: coding

Question	Type	Asks	Options
`intent`	`choice`	What is the user asking the coding agent to do?	question: Explain or answer something; no code changes requested implement: Make a change: add, modify, or configure something investigate: Look into a problem and report findings fix: Repair a specific broken behavior
`specific_enough`	`noul`	Is the request specific enough to act on without asking a clarifying question?	true: The target and desired outcome are both identifiable false: The target or desired outcome is unclear

Failure triage

failure_triage

When: Before retrying or fixing a failing test or command.
State: The test or command, and the relevant error lines.
Decides: The most likely cause, and whether a plain re-run would pass.
Family: coding

Question	Type	Asks	Options
`failure_kind`	`choice`	What most likely caused this failure?	logic_bug: The code under test returns a wrong result environment: Missing service, dependency, port, file, or configuration flaky: Timing, ordering, or nondeterminism; may pass on retry test_bug: The test itself is wrong or outdated
`retry`	`noul`	Is simply re-running likely to make it pass?	yes or no

Change risk

change_risk

When: Before committing, pushing, or merging.
State: A diff summary, the files touched, and the lines changed.
Decides: How risky the change is to ship unreviewed, and whether a person should review it.
Family: coding

Question	Type	Asks	Options
`risk`	`score`	How risky is shipping this change without human review?	0: Negligible 1: Low 2: Medium 3: High
`needs_review`	`noul`	Should a human review this change before it is merged?	yes or no

Routing and ownership

routing

When: When an error needs an owner.
State: The error message and the file or module it came from.
Decides: Which part of the system should handle it. The option set is a template that projects replace with their own.
Family: coding

Question	Type	Asks	Options
`owner`	`choice`	Which part of the system should handle this?	api: HTTP handlers and request validation storage: Database, migrations, and persistence ui: Frontend rendering and client state infra: Build, CI, deployment, and configuration unclear: Not enough information to tell

Effort and approach

effort_approach

When: When planning a task.
State: The task and its constraints.
Decides: How much work the task is, and which approach fits. The option sets are templates.
Family: coding

Question	Type	Asks	Options
`effort`	`score`	How much work is this task?	0: Trivial (minutes) 1: Small (under an hour) 2: Medium (a few hours) 3: Large (a day or more)
`approach`	`choice`	Which approach best fits the constraints?	minimal_patch: Smallest change that fixes the symptom refactor: Restructure the code so the fix is natural new_component: Add a new module or service ask_user: The trade-off needs a human decision

Vision: dominant colour, yes/no

vision_noul

When: Smoke check of the vision path.
State: One synthetic colour swatch.
Decides: Whether one colour dominates the image.
Family: vision

Question	Type	Asks	Options
`dominant_red`	`noul`	Is the image predominantly red?	true: Red is the dominant colour false: Red is not dominant

Vision: dominant colour, choice

vision_choice

When: Smoke check of the vision path.
State: One synthetic colour swatch.
Decides: Which colour dominates the image.
Family: vision

Question	Type	Asks	Options
`dominant_colour`	`choice`	What is the dominant colour in the image?	red: The image is mostly red or warm-red tones green: The image is mostly green tones blue: The image is mostly blue tones mixed: No single colour clearly dominates

Vision: alert level, score

vision_score

When: Smoke check of the vision path.
State: One synthetic status indicator.
Decides: How severe the alert the indicator signals is.
Family: vision

Question	Type	Asks	Options
`alert_level`	`score`	The image shows a status indicator. How severe is the alert it is signalling?	0: No alert (calm green) 1: Low (yellow) 2: Medium (orange) 3: High (red)

Vision: colour change across frames

vision_video

When: Smoke check of the video path.
State: Two synthetic frames.
Decides: Whether the dominant colour changed between the frames.
Family: vision

Question	Type	Asks	Options
`colour_changed`	`noul`	The frames are from a two-frame video clip. Did the dominant colour change between frame 1 and frame 2?	true: The dominant colour is different in the two frames false: The dominant colour is the same in both frames

Tag assignment

tag_assignment

When: When deciding which controlled tags a vault note deserves.
State: A note's title and body plus the vault's controlled vocabulary (as read).
Decides: Which candidate tags the note deserves; the gold is the note's own vocabulary tags.
Family: mined_vault

Question	Type	Asks	Options
`domain/tooling`	`noul`	Does this note deserve the tag `domain/tooling`?	true: The note calls for `domain/tooling`: Build, CI, observer, TTS pipeline, dev-time tools false: The note has no reason to carry `domain/tooling`
`layer/infra`	`noul`	Does this note deserve the tag `layer/infra`?	true: The note calls for `layer/infra`: Build, CI, deployment infrastructure false: The note has no reason to carry `layer/infra`
`type/code`	`noul`	Does this note deserve the tag `type/code`?	true: The note calls for `type/code`: Code architecture, autoloads, conventions, tooling. Carries `code-refs:`. (e.g. `Code/Autoloads — Disturbance.md`) false: The note has no reason to carry `type/code`
`type/decision`	`noul`	Does this note deserve the tag `type/decision`?	true: The note calls for `type/decision`: Architectural or design decision made by an agent during a session. Written to `docs/vault/Decisions/`. (Agent notes only — spec 080.) false: The note has no reason to carry `type/decision`
`type/design`	`noul`	Does this note deserve the tag `type/design`?	true: The note calls for `type/design`: Design intent, player-facing concepts, world lore. (e.g. `Design/World Model.md`) false: The note has no reason to carry `type/design`
`type/discovery`	`noul`	Does this note deserve the tag `type/discovery`?	true: The note calls for `type/discovery`: Non-obvious constraint, gap, or conflict found by an agent during a session. Written to `docs/vault/Discoveries/`. (Agent notes only — spec 080.) false: The note has no reason to carry `type/discovery`
`type/moc`	`noul`	Does this note deserve the tag `type/moc`?	true: The note calls for `type/moc`: Map of Content at any of three tiers (folder / domain / concept). (e.g. `Audio.md`) false: The note has no reason to carry `type/moc`
`type/reference`	`noul`	Does this note deserve the tag `type/reference`?	true: The note calls for `type/reference`: Glossary, decision log, open questions, epics. (e.g. `Reference/Glossary.md`) false: The note has no reason to carry `type/reference`
`type/session-log`	`noul`	Does this note deserve the tag `type/session-log`?	true: The note calls for `type/session-log`: Session activity log; permanent audit trail. Written to `docs/vault/Sessions/`. Append-only, never pruned. (Agent notes only — spec 080.) false: The note has no reason to carry `type/session-log`
`type/spec`	`noul`	Does this note deserve the tag `type/spec`?	true: The note calls for `type/spec`: Spec document at `docs/vault/Specs/NNN Title/spec.md`. Carries `spec-refs:`. (e.g. `Specs/047 Car Radio/spec.md`) false: The note has no reason to carry `type/spec`
`type/system`	`noul`	Does this note deserve the tag `type/system`?	true: The note calls for `type/system`: A bounded subsystem with implementation. Carries `code-refs`, `spec-refs`, `implements:` frontmatter. (e.g. `Systems/Audio — Car Radio.md`) false: The note has no reason to carry `type/system`

Note placement

note_placement

When: When deciding where a vault note belongs.
State: A note's title and body.
Decides: Which folder the note belongs in; the gold is its actual placement.
Family: mined_vault

Question	Type	Asks	Options
`placement`	`choice`	Which folder does this vault note belong in?	Art: Place the note under Art/ Code: Place the note under Code/ Decisions: Place the note under Decisions/ Design: Place the note under Design/ Discoveries: Place the note under Discoveries/ Governance: Place the note under Governance/ Narrative: Place the note under Narrative/ Reference: Place the note under Reference/ Sessions: Place the note under Sessions/ Systems: Place the note under Systems/

Labelling protocol

Labels were judged from state and the option descriptions alone, and were fixed before any model run. *(since dataset version 1e1bb88d47f0)*
needs_review is true exactly when risk is Medium or High, and retry is true only for flaky failures. *(since dataset version 1e1bb88d47f0)*
Drafts whose evidence fit two options were rewritten or dropped while authoring, before any run. *(since dataset version 1e1bb88d47f0)*
A label changes only when a reviewer finds it wrong on its merits, never because the model disagreed. A change to any item's state, media, questions, gold, or split alters the dataset's content hash, which each run records, so runs on different label sets cannot be compared by accident; annotations (kind, tier, source, rationale) do not. *(since dataset version 1e1bb88d47f0)*
Every score scale runs from least to most of the quantity its instruction names; index 0 is always the least. The expected score is read in those units. A scale is never reversed to make a label read naturally. *(since dataset version 1e1bb88d47f0)*
For a noul question, P(true) is the probability of the condition the instruction names; the condition is phrased so that true is the pole that calls for caution. *(since dataset version 1e1bb88d47f0)*

Policy in force: no policy slot was filled for this run, so every recipe applied its default criteria.

Dataset card

Why was it built?: To measure whether ember's probabilities are accurate and calibrated enough for agents to act on the agent kit's thresholds, and to compare model revisions and setups.
What does it contain?: 570 items across five recipes, 1565 scored questions, a gold label per question, a rationale per item.
How was it collected?: 9 items reproduce the agent kit's worked examples; 561 were written by the ember authors. No user data.
How was it labelled?: From state and the option descriptions alone, fixed before any run, under the rules in the labelling protocol.
What is it for?: Tune on dev, report on test. It is not a measure of agent behaviour, and not a training set.
How is it maintained?: Relabelling happens only on review, changes the dataset's SHA-256, and invalidates comparisons with earlier runs.

Generated card

Generated from the dataset this run scored (ember eval dataset card), version e0bbb468e213; the per-label counts per split for every question are in evals/dataset/CARD.md. Label stability and inter-source agreement are unmeasured until Phase 1.

Recipe	Items (dev / test)	Kinds	Sources	Easy	Rater count	External sources
`change_risk`	92 (46 / 46)	untagged 92	authored 92	48	1	none
`effort_approach`	76 (36 / 40)	untagged 76	authored 76	25	1	none
`failure_triage`	92 (46 / 46)	untagged 92	authored 92	87	1	none
`intent_readiness`	108 (54 / 54)	untagged 108	authored 108	88	1	none
`note_placement`	30 (11 / 19)	untagged 30	mined 30	0	1	oldgrowth-vault (All rights reserved; written permission granted for the ember benchmark dataset)
`routing`	75 (38 / 37)	untagged 75	authored 75	73	1	none
`tag_assignment`	65 (27 / 38)	untagged 65	mined 65	4	1	oldgrowth-vault (All rights reserved; written permission granted for the ember benchmark dataset)
`vision_choice`	8 (4 / 4)	untagged 8	authored 8	8	1	none
`vision_noul`	8 (4 / 4)	untagged 8	authored 8	8	1	none
`vision_score`	8 (4 / 4)	untagged 8	authored 8	8	1	none
`vision_video`	8 (4 / 4)	untagged 8	authored 8	8	1	none