Sample size. 1535 questions give a 95% interval about 4 points wide overall; per-question samples of 20 to 28 are wider still, and calibration bins hold a few items each.
Single rater. One maintainer labels every item from state and the option descriptions; no inter-rater figure exists and none is shown. The blinded re-labelling that publishes label stability (an optimistic bound on the human ceiling) is pending (Phase 1). Shared assumptions could make items easier or harder for ember than real traffic.
Curated, not sampled. Items are short, English, and written for the benchmark rather than drawn from real agent sessions.
Ordinal judgments. Risk and effort levels are judgments; neighbouring levels are often defensible, which is why MAE, RPS, and within-one are reported alongside exact accuracy.
One model, one machine. One pinned model revision in fp16 on MPS; numerics on CPU or CUDA can differ slightly. Repeat runs on the same server are deterministic for identical requests.
Fixed question sets. Questions are weighed jointly, so results hold for these question sets. The routing and effort option sets are templates; a project that replaces them should measure its own.
Model and server only. The benchmark calls the HTTP endpoint directly. It does not measure an agent's decision to consult ember or the quality of the questions it writes.
