4Benchmark design

The dataset has 30 items and 30 scored questions: 30 choice, 0 noul, and 0 score. Each line is self-contained: the state, the recipe's fixed question set, a gold label for every question, its source, and a one-line rationale. Other implementations can score it without this harness.

0 items reproduce the worked examples in the agent kit's playbook and sit in dev; the other 30 were written for this benchmark. Items are split into dev (11 items, for tuning) and test (19 items, for reporting), stratified so that every label of every question appears in both. The dataset card below answers the datasheet questions of [12].

Label balance

Recipes

Each recipe asks the same fixed question set for every item, because ember weighs the questions in one call jointly.

Labelling protocol

• Labels were judged from state and the option descriptions alone, and were fixed before any model run. *(since dataset version 1e1bb88d47f0)*
• needs_review is true exactly when risk is Medium or High, and retry is true only for flaky failures. *(since dataset version 1e1bb88d47f0)*
• Drafts whose evidence fit two options were rewritten or dropped while authoring, before any run. *(since dataset version 1e1bb88d47f0)*
• A label changes only when a reviewer finds it wrong on its merits, never because the model disagreed. A change to any item's state, media, questions, gold, or split alters the dataset's content hash, which each run records, so runs on different label sets cannot be compared by accident; annotations (kind, tier, source, rationale) do not. *(since dataset version 1e1bb88d47f0)*
• Every score scale runs from least to most of the quantity its instruction names; index 0 is always the least. The expected score is read in those units. A scale is never reversed to make a label read naturally. *(since dataset version 1e1bb88d47f0)*
• For a noul question, P(true) is the probability of the condition the instruction names; the condition is phrased so that true is the pole that calls for caution. *(since dataset version 1e1bb88d47f0)*

Policy in force: no policy slot was filled for this run, so every recipe applied its default criteria.

Dataset card

Why was it built?

To measure whether ember's probabilities are accurate and calibrated enough for agents to act on the agent kit's thresholds, and to compare model revisions and setups.

What does it contain?

30 items across five recipes, 30 scored questions, a gold label per question, a rationale per item.

How was it collected?

0 items reproduce the agent kit's worked examples; 30 were written by the ember authors. No user data.

How was it labelled?

From state and the option descriptions alone, fixed before any run, under the rules in the labelling protocol.

What is it for?

Tune on dev, report on test. It is not a measure of agent behaviour, and not a training set.

How is it maintained?

Relabelling happens only on review, changes the dataset's SHA-256, and invalidates comparisons with earlier runs.

Generated card

Generated from the dataset this run scored (ember eval dataset card), version e0bbb468e213; the per-label counts per split for every question are in evals/dataset/CARD.md. Label stability and inter-source agreement are unmeasured until Phase 1.