ember

11Error analysis

The run has 146 misses. Each card shows the evidence ember saw, the gold label and its rationale, ember's full answer, and what an agent following the kit would have done with it. Reviewers can use this section as the label-review queue.

What the kit does with the missMisses
within one level of gold63
deferred: treated as unsure28
missed the exact level25
deferred: gather evidence or ask14
acted, flagged for verification11
acted on a wrong answer5

intent_001: specific_enough

Intent and readiness · dev
Gold
The target and desired outcome are both identifiable (True)
ember
The target or desired outcome is unclear (False)
Signal
P(true) 0.21 (unsure)
Kit outcome
deferred: treated as unsure
Why this gold label
Asks for an explanation of an identifiable mechanism; no change requested.
how does the llm know how to call the tool?
ember's answer for intent_001, specific_enoughP(true) placed on the no, unsure, and yes zones.nounsureyesgold: yesP = 0.21

intent_003: specific_enough

Intent and readiness · dev
Gold
The target and desired outcome are both identifiable (True)
ember
The target or desired outcome is unclear (False)
Signal
P(true) 0.45 (unsure)
Kit outcome
deferred: treated as unsure
Why this gold label
Asks what a named question type returns.
what does a noul question return, exactly?
ember's answer for intent_003, specific_enoughP(true) placed on the no, unsure, and yes zones.nounsureyesgold: yesP = 0.45

intent_004: specific_enough

Intent and readiness · test
Gold
The target and desired outcome are both identifiable (True)
ember
The target or desired outcome is unclear (False)
Signal
P(true) 0.48 (unsure)
Kit outcome
deferred: treated as unsure
Why this gold label
Asks the reason for a design choice, not about a malfunction.
why do we load the model on the CPU first and then move it to MPS?
ember's answer for intent_004, specific_enoughP(true) placed on the no, unsure, and yes zones.nounsureyesgold: yesP = 0.48

intent_005: specific_enough

Intent and readiness · dev
Gold
The target and desired outcome are both identifiable (True)
ember
The target or desired outcome is unclear (False)
Signal
P(true) 0.29 (unsure)
Kit outcome
deferred: treated as unsure
Why this gold label
Explanation request about a named behavior.
can you explain how the questions in one call are weighed jointly?
ember's answer for intent_005, specific_enoughP(true) placed on the no, unsure, and yes zones.nounsureyesgold: yesP = 0.29

intent_013: intent

Intent and readiness · dev
Gold
Make a change: add, modify, or configure something (implement)
ember
Repair a specific broken behavior (fix)
Signal
confidence 0.48 (defer band)
Kit outcome
deferred: gather evidence or ask
Why this gold label
Asks for a change to an unnamed target with no defined outcome.
make it better
ember's answer for intent_013, intentProbability ember gave each option.question0.08implement0.38 goldinvestigate0.07fix0.48 ember's pick

intent_021: specific_enough

Intent and readiness · dev
Gold
The target or desired outcome is unclear (False)
ember
The target and desired outcome are both identifiable (True)
Signal
P(true) 0.51 (unsure)
Kit outcome
deferred: treated as unsure
Why this gold label
No symptom or component identified.
users are complaining, check what's going on
ember's answer for intent_021, specific_enoughP(true) placed on the no, unsure, and yes zones.nounsureyesgold: noP = 0.51

failure_010: failure_kind

Failure triage · test
Gold
Missing service, dependency, port, file, or configuration (environment)
ember
Timing, ordering, or nondeterminism; may pass on retry (flaky)
Signal
confidence 0.45 (defer band)
Kit outcome
deferred: gather evidence or ask
Why this gold label
Port conflict that persists until the other process stops.
{
  "test": "make start",
  "output": "OSError: [Errno 48] Address already in use: ('127.0.0.1', 8765)\nanother long-running server owns the port"
}
ember's answer for failure_010, failure_kindProbability ember gave each option.logic_bug0.02environment0.38 goldflaky0.45 ember's picktest_bug0.15

failure_012: failure_kind

Failure triage · test
Gold
Missing service, dependency, port, file, or configuration (environment)
ember
Timing, ordering, or nondeterminism; may pass on retry (flaky)
Signal
confidence 0.40 (defer band)
Kit outcome
deferred: gather evidence or ask
Why this gold label
A file permission problem on the host.
{
  "test": "tests/test_cli.py::test_start_writes_pidfile",
  "output": "PermissionError: [Errno 13] Permission denied: '/Users/ci/Library/Application Support/ember/server.pid'"
}
ember's answer for failure_012, failure_kindProbability ember gave each option.logic_bug0.05environment0.33 goldflaky0.40 ember's picktest_bug0.23

failure_023: failure_kind

Failure triage · test
Gold
The test itself is wrong or outdated (test_bug)
ember
The code under test returns a wrong result (logic_bug)
Signal
confidence 0.46 (defer band)
Kit outcome
deferred: gather evidence or ask
Why this gold label
The fixture is not perfectly calibrated; the code is right and the assertion is wrong.
{
  "test": "tests/test_eval_metrics.py::test_ece_perfect_calibration",
  "output": "AssertionError: assert 0.1 == 0.0\nthe fixture uses ten predictions at confidence 0.8 with nine correct, so the expected ECE is 0.1, not 0.0"
}
ember's answer for failure_023, failure_kindProbability ember gave each option.logic_bug0.46 ember's pickenvironment0.01flaky0.12test_bug0.42 gold

risk_001: risk

Change risk · dev
Gold
Negligible (0)
ember
Low (1)
Signal
expected 0.68, 0.68 levels from gold
Kit outcome
within one level of gold
Why this gold label
Test-only cosmetic rename.
{
  "diff_summary": "Rename local variable `tmp` to `result` in one test helper",
  "files": [
    "tests/test_helpers.py"
  ],
  "lines_changed": 4
}
ember's answer for risk_001, riskProbability ember gave each option.Negligible0.38 goldLow0.58 ember's pickMedium0.03High0.02

risk_002: risk

Change risk · test
Gold
Negligible (0)
ember
Low (1)
Signal
expected 0.53, 0.53 levels from gold
Kit outcome
within one level of gold
Why this gold label
Documentation-only change.
{
  "diff_summary": "Fix a typo in the README",
  "files": [
    "README.md"
  ],
  "lines_changed": 1
}
ember's answer for risk_002, riskProbability ember gave each option.Negligible0.52 goldLow0.44 ember's pickMedium0.02High0.01

risk_003: risk

Change risk · dev
Gold
Negligible (0)
ember
Low (1)
Signal
expected 0.50, 0.50 levels from gold
Kit outcome
within one level of gold
Why this gold label
Docstring-only change.
{
  "diff_summary": "Fix the spelling of 'directory' in a docstring",
  "files": [
    "ember/paths.py"
  ],
  "lines_changed": 1
}
ember's answer for risk_003, riskProbability ember gave each option.Negligible0.54 goldLow0.42 ember's pickMedium0.02High0.01

risk_004: risk

Change risk · test
Gold
Negligible (0)
ember
Low (1)
Signal
expected 0.68, 0.68 levels from gold
Kit outcome
within one level of gold
Why this gold label
Mechanical whitespace change in tests.
{
  "diff_summary": "Reformat one test file with ruff format; whitespace only, no logic changes",
  "files": [
    "tests/test_media.py"
  ],
  "lines_changed": 12
}
ember's answer for risk_004, riskProbability ember gave each option.Negligible0.39 goldLow0.56 ember's pickMedium0.03High0.02

risk_005: risk

Change risk · dev
Gold
Negligible (0)
ember
Low (1)
Signal
expected 0.60, 0.60 levels from gold
Kit outcome
within one level of gold
Why this gold label
No code affected.
{
  "diff_summary": "Update the copyright year in the LICENSE file",
  "files": [
    "LICENSE"
  ],
  "lines_changed": 1
}
ember's answer for risk_005, riskProbability ember gave each option.Negligible0.45 goldLow0.51 ember's pickMedium0.03High0.02

risk_016: risk

Change risk · dev
Gold
Medium (2)
ember
Low (1)
Signal
expected 1.30, 0.70 levels from gold
Kit outcome
within one level of gold
Why this gold label
Runtime dependency upgrade under the HTTP server.
{
  "diff_summary": "Upgrade fastapi from 0.142 to 0.150",
  "files": [
    "pyproject.toml",
    "uv.lock"
  ],
  "lines_changed": 22
}
ember's answer for risk_016, riskProbability ember gave each option.Negligible0.11Low0.52 ember's pickMedium0.33 goldHigh0.04

risk_016: needs_review

Change risk · dev
Gold
yes (True)
ember
no (False)
Signal
P(true) 0.29 (unsure)
Kit outcome
deferred: treated as unsure
Why this gold label
Runtime dependency upgrade under the HTTP server.
{
  "diff_summary": "Upgrade fastapi from 0.142 to 0.150",
  "files": [
    "pyproject.toml",
    "uv.lock"
  ],
  "lines_changed": 22
}
ember's answer for risk_016, needs_reviewP(true) placed on the no, unsure, and yes zones.nounsureyesgold: yesP = 0.29

risk_018: risk

Change risk · dev
Gold
Medium (2)
ember
High (3)
Signal
expected 2.54, 0.54 levels from gold
Kit outcome
within one level of gold
Why this gold label
Changes files written into users' projects.
{
  "diff_summary": "Change the generated opencode plugin template and the format of the config it writes",
  "files": [
    "ember/opencode_plugin.py"
  ],
  "lines_changed": 150
}
ember's answer for risk_018, riskProbability ember gave each option.Negligible0.05Low0.06Medium0.19 goldHigh0.70 ember's pick

risk_019: risk

Change risk · dev
Gold
High (3)
ember
Medium (2)
Signal
expected 2.00, 1.00 levels from gold
Kit outcome
within one level of gold
Why this gold label
Weakens authentication.
{
  "diff_summary": "Skip auth token validation when the request originates from localhost to unblock local development",
  "files": [
    "ember/server.py",
    "ember/mcp_server.py"
  ],
  "lines_changed": 180
}
ember's answer for risk_019, riskProbability ember gave each option.Negligible0.09Low0.16Medium0.41 ember's pickHigh0.34 gold

risk_022: risk

Change risk · test
Gold
High (3)
ember
Medium (2)
Signal
expected 1.67, 1.33 levels from gold
Kit outcome
missed the exact level
Why this gold label
One line, but it exposes an unauthenticated server to the network.
{
  "diff_summary": "Bind the model server to 0.0.0.0 by default instead of 127.0.0.1",
  "files": [
    "ember/config.py"
  ],
  "lines_changed": 1
}
ember's answer for risk_022, riskProbability ember gave each option.Negligible0.13Low0.22Medium0.49 ember's pickHigh0.16 gold

risk_022: needs_review

Change risk · test
Gold
yes (True)
ember
no (False)
Signal
P(true) 0.45 (unsure)
Kit outcome
deferred: treated as unsure
Why this gold label
One line, but it exposes an unauthenticated server to the network.
{
  "diff_summary": "Bind the model server to 0.0.0.0 by default instead of 127.0.0.1",
  "files": [
    "ember/config.py"
  ],
  "lines_changed": 1
}
ember's answer for risk_022, needs_reviewP(true) placed on the no, unsure, and yes zones.nounsureyesgold: yesP = 0.45

risk_023: risk

Change risk · dev
Gold
High (3)
ember
Medium (2)
Signal
expected 1.97, 1.03 levels from gold
Kit outcome
missed the exact level
Why this gold label
Can terminate unrelated processes.
{
  "diff_summary": "Make `ember stop` kill any process listening on the configured port",
  "files": [
    "ember/process.py"
  ],
  "lines_changed": 25
}
ember's answer for risk_023, riskProbability ember gave each option.Negligible0.08Low0.13Medium0.54 ember's pickHigh0.26 gold

risk_024: risk

Change risk · test
Gold
High (3)
ember
Medium (2)
Signal
expected 2.41, 0.59 levels from gold
Kit outcome
within one level of gold
Why this gold label
Irreversible deletion of unrelated user data.
{
  "diff_summary": "Make `ember model rm` delete the whole Hugging Face cache directory instead of one model",
  "files": [
    "ember/models.py"
  ],
  "lines_changed": 10
}
ember's answer for risk_024, riskProbability ember gave each option.Negligible0.06Low0.08Medium0.26 ember's pickHigh0.61 gold

effort_002: effort

Effort and approach · dev
Gold
Trivial (minutes) (0)
ember
Small (under an hour) (1)
Signal
expected 0.51, 0.51 levels from gold
Kit outcome
within one level of gold
Why this gold label
A single default value.
{
  "task": "Change the default number of lines `ember logs` prints from 40 to 100",
  "constraints": [
    "one-line change"
  ]
}
ember's answer for effort_002, effortProbability ember gave each option.Trivial (minutes)0.60 goldSmall (under an hour)0.31 ember's pickMedium (a few hours)0.08Large (a day or more)0.01

effort_003: effort

Effort and approach · test
Gold
Trivial (minutes) (0)
ember
Small (under an hour) (1)
Signal
expected 0.56, 0.56 levels from gold
Kit outcome
within one level of gold
Why this gold label
A guard clause in one function.
{
  "task": "Guard the missing null check that crashes the order total calculation for empty carts",
  "constraints": [
    "hotfix",
    "touch only the failing function"
  ]
}
ember's answer for effort_003, effortProbability ember gave each option.Trivial (minutes)0.51 goldSmall (under an hour)0.44 ember's pickMedium (a few hours)0.04Large (a day or more)0.01

effort_006: effort

Effort and approach · test
Gold
Small (under an hour) (1)
ember
Medium (a few hours) (2)
Signal
expected 1.50, 0.50 levels from gold
Kit outcome
within one level of gold
Why this gold label
Mechanical extraction covered by existing tests.
{
  "task": "The same 15-line retry loop is copy-pasted in 6 HTTP client methods that already have unit tests; extract it into a shared helper",
  "constraints": [
    "no new dependencies",
    "keep existing API"
  ]
}
ember's answer for effort_006, effortProbability ember gave each option.Trivial (minutes)0.07Small (under an hour)0.39 goldMedium (a few hours)0.51 ember's pickLarge (a day or more)0.03

effort_009: effort

Effort and approach · test
Gold
Medium (a few hours) (2)
ember
Large (a day or more) (3)
Signal
expected 2.78, 0.78 levels from gold
Kit outcome
within one level of gold
Why this gold label
A module split with no behavior change.
{
  "task": "The 900-line server module mixes routing, validation, metrics, and model calls; split it by responsibility before the next feature lands",
  "constraints": [
    "no API changes",
    "keep the public import path"
  ]
}
ember's answer for effort_009, effortProbability ember gave each option.Trivial (minutes)0.02Small (under an hour)0.02Medium (a few hours)0.12 goldLarge (a day or more)0.84 ember's pick

effort_013: effort

Effort and approach · dev
Gold
Large (a day or more) (3)
ember
Medium (a few hours) (2)
Signal
expected 2.01, 0.99 levels from gold
Kit outcome
within one level of gold
Why this gold label
A persistent queue and worker are new subsystems.
{
  "task": "Add a background job queue so long-running eval runs execute asynchronously with progress reporting",
  "constraints": [
    "must survive server restarts",
    "no external broker"
  ]
}
ember's answer for effort_013, effortProbability ember gave each option.Trivial (minutes)0.04Small (under an hour)0.09Medium (a few hours)0.69 ember's pickLarge (a day or more)0.18 gold

effort_015: effort

Effort and approach · test
Gold
Large (a day or more) (3)
ember
Medium (a few hours) (2)
Signal
expected 1.85, 1.15 levels from gold
Kit outcome
missed the exact level
Why this gold label
A new package with its own packaging and tests.
{
  "task": "Build a plugin for a second editor that mirrors the existing opencode plugin",
  "constraints": [
    "reuse the existing MCP server",
    "ship it as a separate package"
  ]
}
ember's answer for effort_015, effortProbability ember gave each option.Trivial (minutes)0.04Small (under an hour)0.17Medium (a few hours)0.70 ember's pickLarge (a day or more)0.09 gold

effort_016: effort

Effort and approach · dev
Gold
Trivial (minutes) (0)
ember
Small (under an hour) (1)
Signal
expected 1.29, 1.29 levels from gold
Kit outcome
missed the exact level
Why this gold label
The edit is one default; the accuracy-versus-memory trade-off is a product decision.
{
  "task": "Switch the default model from flash (9B) to full (27B)",
  "constraints": [
    "full is more accurate but needs about 3x the memory and is about 3x slower",
    "many users have 32 GB machines"
  ]
}
ember's answer for effort_016, effortProbability ember gave each option.Trivial (minutes)0.21 goldSmall (under an hour)0.37 ember's pickMedium (a few hours)0.33Large (a day or more)0.09

effort_017: effort

Effort and approach · dev
Gold
Small (under an hour) (1)
ember
Medium (a few hours) (2)
Signal
expected 1.61, 0.61 levels from gold
Kit outcome
within one level of gold
Why this gold label
Dropping support for some users is a human call; the code change is small.
{
  "task": "Drop CPU fallback support to simplify the runtime",
  "constraints": [
    "some users run without MPS",
    "removes about 200 lines"
  ]
}
ember's answer for effort_017, effortProbability ember gave each option.Trivial (minutes)0.06Small (under an hour)0.33 goldMedium (a few hours)0.56 ember's pickLarge (a day or more)0.05

effort_018: effort

Effort and approach · test
Gold
Small (under an hour) (1)
ember
Medium (a few hours) (2)
Signal
expected 2.38, 1.38 levels from gold
Kit outcome
missed the exact level
Why this gold label
Breaking a public contract needs a human decision.
{
  "task": "Rename the public advise tool to consult across the MCP server, docs, and agent kit",
  "constraints": [
    "the tool name is public API used by external agents",
    "renaming breaks every existing integration"
  ]
}
ember's answer for effort_018, effortProbability ember gave each option.Trivial (minutes)0.06Small (under an hour)0.09 goldMedium (a few hours)0.25 ember's pickLarge (a day or more)0.60

effort_019: effort

Effort and approach · dev
Gold
Small (under an hour) (1)
ember
Medium (a few hours) (2)
Signal
expected 1.75, 0.75 levels from gold
Kit outcome
within one level of gold
Why this gold label
A breaking change with no deprecation policy.
{
  "task": "Rename the HTTP endpoint /v1/systemone to /v1/advise",
  "constraints": [
    "external clients call /v1/systemone directly",
    "no deprecation policy exists yet"
  ]
}
ember's answer for effort_019, effortProbability ember gave each option.Trivial (minutes)0.12Small (under an hour)0.21 goldMedium (a few hours)0.46 ember's pickLarge (a day or more)0.20

effort_020: effort

Effort and approach · dev
Gold
Large (a day or more) (3)
ember
Medium (a few hours) (2)
Signal
expected 1.58, 1.42 levels from gold
Kit outcome
missed the exact level
Why this gold label
Conflicts with a stated privacy promise; building it would take a day or more.
{
  "task": "Add anonymous usage telemetry that reports to a hosted endpoint",
  "constraints": [
    "the project promises that state never leaves the machine",
    "maintainers want usage data"
  ]
}
ember's answer for effort_020, effortProbability ember gave each option.Trivial (minutes)0.08Small (under an hour)0.31Medium (a few hours)0.56 ember's pickLarge (a day or more)0.05 gold

effort_020: approach

Effort and approach · dev
Gold
The trade-off needs a human decision (ask_user)
ember
Smallest change that fixes the symptom (minimal_patch)
Signal
confidence 0.38 (defer band)
Kit outcome
deferred: gather evidence or ask
Why this gold label
Conflicts with a stated privacy promise; building it would take a day or more.
{
  "task": "Add anonymous usage telemetry that reports to a hosted endpoint",
  "constraints": [
    "the project promises that state never leaves the machine",
    "maintainers want usage data"
  ]
}
ember's answer for effort_020, approachProbability ember gave each option.minimal_patch0.38 ember's pickrefactor0.03new_component0.28ask_user0.31 gold

intent_029: specific_enough

Intent and readiness · test
Gold
The target and desired outcome are both identifiable (True)
ember
The target or desired outcome is unclear (False)
Signal
P(true) 0.30 (unsure)
Kit outcome
deferred: treated as unsure
Why this gold label
Asks a factual question about a named setting.
what's the default port for the model server?
ember's answer for intent_029, specific_enoughP(true) placed on the no, unsure, and yes zones.nounsureyesgold: yesP = 0.30

intent_031: specific_enough

Intent and readiness · dev
Gold
The target and desired outcome are both identifiable (True)
ember
The target or desired outcome is unclear (False)
Signal
P(true) 0.23 (unsure)
Kit outcome
deferred: treated as unsure
Why this gold label
Asks to compare two named options.
what's the difference between the flash and full models?
ember's answer for intent_031, specific_enoughP(true) placed on the no, unsure, and yes zones.nounsureyesgold: yesP = 0.23

intent_032: specific_enough

Intent and readiness · test
Gold
The target and desired outcome are both identifiable (True)
ember
The target or desired outcome is unclear (False)
Signal
P(true) 0.49 (unsure)
Kit outcome
deferred: treated as unsure
Why this gold label
Explanation request about a named metric.
can you explain what ECE measures in the benchmark report?
ember's answer for intent_032, specific_enoughP(true) placed on the no, unsure, and yes zones.nounsureyesgold: yesP = 0.49

intent_048: specific_enough

Intent and readiness · test
Gold
The target or desired outcome is unclear (False)
ember
The target and desired outcome are both identifiable (True)
Signal
P(true) 0.50 (unsure)
Kit outcome
deferred: treated as unsure
Why this gold label
The problem is never described.
there's a problem somewhere in the build, investigate
ember's answer for intent_048, specific_enoughP(true) placed on the no, unsure, and yes zones.nounsureyesgold: noP = 0.50

risk_025: risk

Change risk · test
Gold
Negligible (0)
ember
Low (1)
Signal
expected 0.64, 0.64 levels from gold
Kit outcome
within one level of gold
Why this gold label
Documentation-only change.
{
  "diff_summary": "Fix a broken link in CONTRIBUTING.md",
  "files": [
    "CONTRIBUTING.md"
  ],
  "lines_changed": 1
}
ember's answer for risk_025, riskProbability ember gave each option.Negligible0.42 goldLow0.55 ember's pickMedium0.02High0.01

risk_026: risk

Change risk · dev
Gold
Negligible (0)
ember
Low (1)
Signal
expected 0.58, 0.58 levels from gold
Kit outcome
within one level of gold
Why this gold label
Comment-only change.
{
  "diff_summary": "Correct the spelling of 'calibration' in a code comment",
  "files": [
    "evals/metrics.py"
  ],
  "lines_changed": 1
}
ember's answer for risk_026, riskProbability ember gave each option.Negligible0.47 goldLow0.49 ember's pickMedium0.02High0.01

risk_027: risk

Change risk · test
Gold
Negligible (0)
ember
Low (1)
Signal
expected 0.65, 0.65 levels from gold
Kit outcome
within one level of gold
Why this gold label
Reordering with no effect.
{
  "diff_summary": "Sort the entries of .gitignore alphabetically",
  "files": [
    ".gitignore"
  ],
  "lines_changed": 6
}
ember's answer for risk_027, riskProbability ember gave each option.Negligible0.41 goldLow0.54 ember's pickMedium0.03High0.02

risk_028: risk

Change risk · dev
Gold
Negligible (0)
ember
Low (1)
Signal
expected 0.63, 0.63 levels from gold
Kit outcome
within one level of gold
Why this gold label
Test-only cosmetic rename.
{
  "diff_summary": "Rename a local loop variable from i to index in one test",
  "files": [
    "tests/test_media.py"
  ],
  "lines_changed": 3
}
ember's answer for risk_028, riskProbability ember gave each option.Negligible0.43 goldLow0.53 ember's pickMedium0.03High0.02

risk_030: risk

Change risk · dev
Gold
Negligible (0)
ember
Low (1)
Signal
expected 0.73, 0.73 levels from gold
Kit outcome
within one level of gold
Why this gold label
Whitespace only.
{
  "diff_summary": "Reflow a Markdown paragraph in a vault session log; no wording changes",
  "files": [
    "vault/sessions/2026-10-03-eval-benchmark-harness.md"
  ],
  "lines_changed": 8
}
ember's answer for risk_030, riskProbability ember gave each option.Negligible0.35 goldLow0.58 ember's pickMedium0.05High0.01

risk_043: risk

Change risk · test
Gold
High (3)
ember
Medium (2)
Signal
expected 2.01, 0.99 levels from gold
Kit outcome
within one level of gold
Why this gold label
Sends private code off the machine.
{
  "diff_summary": "Log the full advise request state, including user code, to a remote analytics endpoint",
  "files": [
    "ember/server.py"
  ],
  "lines_changed": 45
}
ember's answer for risk_043, riskProbability ember gave each option.Negligible0.07Low0.12Medium0.54 ember's pickHigh0.27 gold

risk_044: risk

Change risk · dev
Gold
High (3)
ember
Medium (2)
Signal
expected 2.19, 0.81 levels from gold
Kit outcome
within one level of gold
Why this gold label
Unprompted recursive deletion of a user-chosen path.
{
  "diff_summary": "Remove the confirmation prompt and rm -rf the state directory in `ember uninstall`, even when EMBER_STATE_DIR points elsewhere",
  "files": [
    "ember/cli.py"
  ],
  "lines_changed": 8
}
ember's answer for risk_044, riskProbability ember gave each option.Negligible0.07Low0.11Medium0.36 ember's pickHigh0.45 gold

risk_045: risk

Change risk · test
Gold
High (3)
ember
Low (1)
Signal
expected 1.37, 1.63 levels from gold
Kit outcome
missed the exact level
Why this gold label
Lets callers make the server read host files.
{
  "diff_summary": "Let `images` accept file:// URLs",
  "files": [
    "ember/media.py"
  ],
  "lines_changed": 6
}
ember's answer for risk_045, riskProbability ember gave each option.Negligible0.14Low0.42 ember's pickMedium0.37High0.07 gold

risk_045: needs_review

Change risk · test
Gold
yes (True)
ember
no (False)
Signal
P(true) 0.35 (unsure)
Kit outcome
deferred: treated as unsure
Why this gold label
Lets callers make the server read host files.
{
  "diff_summary": "Let `images` accept file:// URLs",
  "files": [
    "ember/media.py"
  ],
  "lines_changed": 6
}
ember's answer for risk_045, needs_reviewP(true) placed on the no, unsure, and yes zones.nounsureyesgold: yesP = 0.35

risk_046: risk

Change risk · dev
Gold
High (3)
ember
Medium (2)
Signal
expected 2.46, 0.54 levels from gold
Kit outcome
within one level of gold
Why this gold label
Kills unrelated processes.
{
  "diff_summary": "Make `ember stop` SIGKILL every process whose command line contains 'ember'",
  "files": [
    "ember/process.py"
  ],
  "lines_changed": 15
}
ember's answer for risk_046, riskProbability ember gave each option.Negligible0.06Low0.08Medium0.20 ember's pickHigh0.66 gold

risk_047: risk

Change risk · test
Gold
High (3)
ember
Low (1)
Signal
expected 1.46, 1.54 levels from gold
Kit outcome
missed the exact level
Why this gold label
Leaks a secret into shareable files.
{
  "diff_summary": "Store the OpenRouter API key in the agent eval results JSON for reproducibility",
  "files": [
    "scripts/run_agent_evals.py"
  ],
  "lines_changed": 3
}
ember's answer for risk_047, riskProbability ember gave each option.Negligible0.17Low0.38 ember's pickMedium0.26High0.18 gold

risk_047: needs_review

Change risk · test
Gold
yes (True)
ember
no (False)
Signal
P(true) 0.34 (unsure)
Kit outcome
deferred: treated as unsure
Why this gold label
Leaks a secret into shareable files.
{
  "diff_summary": "Store the OpenRouter API key in the agent eval results JSON for reproducibility",
  "files": [
    "scripts/run_agent_evals.py"
  ],
  "lines_changed": 3
}
ember's answer for risk_047, needs_reviewP(true) placed on the no, unsure, and yes zones.nounsureyesgold: yesP = 0.34

risk_048: risk

Change risk · dev
Gold
High (3)
ember
Medium (2)
Signal
expected 1.98, 1.02 levels from gold
Kit outcome
missed the exact level
Why this gold label
One flag, but agents may act outside the sandbox.
{
  "diff_summary": "Run agent eval sessions with --auto so every permission request is approved",
  "files": [
    "evals/agent/opencode.py"
  ],
  "lines_changed": 1
}
ember's answer for risk_048, riskProbability ember gave each option.Negligible0.12Low0.20Medium0.26 ember's pickHigh0.42 gold

risk_048: needs_review

Change risk · dev
Gold
yes (True)
ember
no (False)
Signal
P(true) 0.40 (unsure)
Kit outcome
deferred: treated as unsure
Why this gold label
One flag, but agents may act outside the sandbox.
{
  "diff_summary": "Run agent eval sessions with --auto so every permission request is approved",
  "files": [
    "evals/agent/opencode.py"
  ],
  "lines_changed": 1
}
ember's answer for risk_048, needs_reviewP(true) placed on the no, unsure, and yes zones.nounsureyesgold: yesP = 0.40

effort_022: effort

Effort and approach · test
Gold
Trivial (minutes) (0)
ember
Small (under an hour) (1)
Signal
expected 0.71, 0.71 levels from gold
Kit outcome
within one level of gold
Why this gold label
One constant.
{
  "task": "Change the `ember logs` follow interval from 0.5 s to 1 s",
  "constraints": [
    "one constant"
  ]
}
ember's answer for effort_022, effortProbability ember gave each option.Trivial (minutes)0.39 goldSmall (under an hour)0.51 ember's pickMedium (a few hours)0.08Large (a day or more)0.01

effort_023: effort

Effort and approach · dev
Gold
Trivial (minutes) (0)
ember
Small (under an hour) (1)
Signal
expected 1.03, 1.03 levels from gold
Kit outcome
missed the exact level
Why this gold label
One string.
{
  "task": "Make `ember status` print 'stopped' instead of 'not running'",
  "constraints": [
    "keep the exit code"
  ]
}
ember's answer for effort_023, effortProbability ember gave each option.Trivial (minutes)0.28 goldSmall (under an hour)0.44 ember's pickMedium (a few hours)0.23Large (a day or more)0.04

effort_027: effort

Effort and approach · test
Gold
Medium (a few hours) (2)
ember
Large (a day or more) (3)
Signal
expected 2.77, 0.77 levels from gold
Kit outcome
within one level of gold
Why this gold label
A module split with behavior preserved.
{
  "task": "The 800-line CLI module mixes server lifecycle, models, evals, and agents; split it by responsibility",
  "constraints": [
    "identical CLI behavior",
    "existing tests pass"
  ]
}
ember's answer for effort_027, effortProbability ember gave each option.Trivial (minutes)0.01Small (under an hour)0.02Medium (a few hours)0.16 goldLarge (a day or more)0.81 ember's pick

effort_029: effort

Effort and approach · test
Gold
Small (under an hour) (1)
ember
Medium (a few hours) (2)
Signal
expected 1.77, 0.77 levels from gold
Kit outcome
within one level of gold
Why this gold label
Consolidation across two call sites.
{
  "task": "The benchmark and agent runners resolve the server URL from flags, env, and config in different ways; unify them",
  "constraints": [
    "same defaults"
  ]
}
ember's answer for effort_029, effortProbability ember gave each option.Trivial (minutes)0.06Small (under an hour)0.19 goldMedium (a few hours)0.67 ember's pickLarge (a day or more)0.08

effort_031: effort

Effort and approach · test
Gold
Large (a day or more) (3)
ember
Small (under an hour) (1)
Signal
expected 1.35, 1.65 levels from gold
Kit outcome
missed the exact level
Why this gold label
A new adapter with its own parsing and isolation.
{
  "task": "Add a Claude Code adapter to the agent eval alongside opencode",
  "constraints": [
    "same scenarios and scoring",
    "isolated config"
  ]
}
ember's answer for effort_031, effortProbability ember gave each option.Trivial (minutes)0.10Small (under an hour)0.47 ember's pickMedium (a few hours)0.39Large (a day or more)0.04 gold

effort_032: effort

Effort and approach · dev
Gold
Large (a day or more) (3)
ember
Small (under an hour) (1)
Signal
expected 1.19, 1.81 levels from gold
Kit outcome
missed the exact level
Why this gold label
A new reporting component.
{
  "task": "Add a static HTML dashboard that tracks benchmark results across runs over time",
  "constraints": [
    "static files only",
    "no server"
  ]
}
ember's answer for effort_032, effortProbability ember gave each option.Trivial (minutes)0.18Small (under an hour)0.48 ember's pickMedium (a few hours)0.31Large (a day or more)0.03 gold

effort_032: approach

Effort and approach · dev
Gold
Add a new module or service (new_component)
ember
Smallest change that fixes the symptom (minimal_patch)
Signal
confidence 0.66 (verify band)
Kit outcome
acted, flagged for verification
Why this gold label
A new reporting component.
{
  "task": "Add a static HTML dashboard that tracks benchmark results across runs over time",
  "constraints": [
    "static files only",
    "no server"
  ]
}
ember's answer for effort_032, approachProbability ember gave each option.minimal_patch0.66 ember's pickrefactor0.08new_component0.07 goldask_user0.19

effort_033: effort

Effort and approach · dev
Gold
Medium (a few hours) (2)
ember
Small (under an hour) (1)
Signal
expected 1.50, 0.50 levels from gold
Kit outcome
within one level of gold
Why this gold label
A new endpoint.
{
  "task": "Add a /v1/batch endpoint that scores many advise requests in one call",
  "constraints": [
    "keep /v1/systemone unchanged"
  ]
}
ember's answer for effort_033, effortProbability ember gave each option.Trivial (minutes)0.10Small (under an hour)0.36 ember's pickMedium (a few hours)0.48 goldLarge (a day or more)0.06

effort_033: approach

Effort and approach · dev
Gold
Add a new module or service (new_component)
ember
Smallest change that fixes the symptom (minimal_patch)
Signal
confidence 0.52 (defer band)
Kit outcome
deferred: gather evidence or ask
Why this gold label
A new endpoint.
{
  "task": "Add a /v1/batch endpoint that scores many advise requests in one call",
  "constraints": [
    "keep /v1/systemone unchanged"
  ]
}
ember's answer for effort_033, approachProbability ember gave each option.minimal_patch0.52 ember's pickrefactor0.09new_component0.31 goldask_user0.08

effort_034: approach

Effort and approach · dev
Gold
Add a new module or service (new_component)
ember
Smallest change that fixes the symptom (minimal_patch)
Signal
confidence 0.51 (defer band)
Kit outcome
deferred: gather evidence or ask
Why this gold label
A small new module.
{
  "task": "Add a CSV export of the per-item benchmark trace",
  "constraints": [
    "one new module",
    "no new dependencies"
  ]
}
ember's answer for effort_034, approachProbability ember gave each option.minimal_patch0.51 ember's pickrefactor0.04new_component0.40 goldask_user0.05

effort_035: effort

Effort and approach · test
Gold
Medium (a few hours) (2)
ember
Small (under an hour) (1)
Signal
expected 1.23, 0.77 levels from gold
Kit outcome
within one level of gold
Why this gold label
A new module and metric.
{
  "task": "Add a Prometheus metric for MCP autostarts in a new metrics module for the MCP process",
  "constraints": [
    "the MCP process must stay torch-free"
  ]
}
ember's answer for effort_035, effortProbability ember gave each option.Trivial (minutes)0.12Small (under an hour)0.56 ember's pickMedium (a few hours)0.29 goldLarge (a day or more)0.03

effort_036: effort

Effort and approach · test
Gold
Trivial (minutes) (0)
ember
Small (under an hour) (1)
Signal
expected 0.75, 0.75 levels from gold
Kit outcome
within one level of gold
Why this gold label
One default; the cost trade-off is a human call.
{
  "task": "Make the agent eval default to Opus-class models",
  "constraints": [
    "about five times the cost per run",
    "may flatter ember compared with the cheaper models users run"
  ]
}
ember's answer for effort_036, effortProbability ember gave each option.Trivial (minutes)0.48 goldSmall (under an hour)0.35 ember's pickMedium (a few hours)0.13Large (a day or more)0.05

effort_036: approach

Effort and approach · test
Gold
The trade-off needs a human decision (ask_user)
ember
Smallest change that fixes the symptom (minimal_patch)
Signal
confidence 0.81 (verify band)
Kit outcome
acted, flagged for verification
Why this gold label
One default; the cost trade-off is a human call.
{
  "task": "Make the agent eval default to Opus-class models",
  "constraints": [
    "about five times the cost per run",
    "may flatter ember compared with the cheaper models users run"
  ]
}
ember's answer for effort_036, approachProbability ember gave each option.minimal_patch0.81 ember's pickrefactor0.07new_component0.03ask_user0.09 gold

effort_038: effort

Effort and approach · dev
Gold
Trivial (minutes) (0)
ember
Small (under an hour) (1)
Signal
expected 0.64, 0.64 levels from gold
Kit outcome
within one level of gold
Why this gold label
Changes every agent's behavior on thin evidence.
{
  "task": "Lower the kit's choice trust threshold from 0.85 to 0.60",
  "constraints": [
    "agents would act more often",
    "one benchmark run supports it"
  ]
}
ember's answer for effort_038, effortProbability ember gave each option.Trivial (minutes)0.48 goldSmall (under an hour)0.41 ember's pickMedium (a few hours)0.10Large (a day or more)0.01

effort_038: approach

Effort and approach · dev
Gold
The trade-off needs a human decision (ask_user)
ember
Smallest change that fixes the symptom (minimal_patch)
Signal
confidence 0.79 (verify band)
Kit outcome
acted, flagged for verification
Why this gold label
Changes every agent's behavior on thin evidence.
{
  "task": "Lower the kit's choice trust threshold from 0.85 to 0.60",
  "constraints": [
    "agents would act more often",
    "one benchmark run supports it"
  ]
}
ember's answer for effort_038, approachProbability ember gave each option.minimal_patch0.79 ember's pickrefactor0.11new_component0.02ask_user0.08 gold

effort_039: effort

Effort and approach · test
Gold
Large (a day or more) (3)
ember
Small (under an hour) (1)
Signal
expected 1.49, 1.51 levels from gold
Kit outcome
missed the exact level
Why this gold label
Privacy trade-off; building it is a day or more.
{
  "task": "Send anonymized benchmark results to a shared leaderboard by default",
  "constraints": [
    "users may run on private code",
    "maintainers want comparisons"
  ]
}
ember's answer for effort_039, effortProbability ember gave each option.Trivial (minutes)0.12Small (under an hour)0.34 ember's pickMedium (a few hours)0.47Large (a day or more)0.07 gold

effort_039: approach

Effort and approach · test
Gold
The trade-off needs a human decision (ask_user)
ember
Add a new module or service (new_component)
Signal
confidence 0.41 (defer band)
Kit outcome
deferred: gather evidence or ask
Why this gold label
Privacy trade-off; building it is a day or more.
{
  "task": "Send anonymized benchmark results to a shared leaderboard by default",
  "constraints": [
    "users may run on private code",
    "maintainers want comparisons"
  ]
}
ember's answer for effort_039, approachProbability ember gave each option.minimal_patch0.38refactor0.10new_component0.41 ember's pickask_user0.10 gold

effort_040: effort

Effort and approach · test
Gold
Trivial (minutes) (0)
ember
Small (under an hour) (1)
Signal
expected 1.07, 1.07 levels from gold
Kit outcome
missed the exact level
Why this gold label
A workflow trade-off for the user.
{
  "task": "Change the AGENTS.md policy so agents consult ember before every file edit",
  "constraints": [
    "raises consultation",
    "adds about a second to every edit"
  ]
}
ember's answer for effort_040, effortProbability ember gave each option.Trivial (minutes)0.19 goldSmall (under an hour)0.59 ember's pickMedium (a few hours)0.20Large (a day or more)0.03

effort_040: approach

Effort and approach · test
Gold
The trade-off needs a human decision (ask_user)
ember
Smallest change that fixes the symptom (minimal_patch)
Signal
confidence 0.67 (verify band)
Kit outcome
acted, flagged for verification
Why this gold label
A workflow trade-off for the user.
{
  "task": "Change the AGENTS.md policy so agents consult ember before every file edit",
  "constraints": [
    "raises consultation",
    "adds about a second to every edit"
  ]
}
ember's answer for effort_040, approachProbability ember gave each option.minimal_patch0.67 ember's pickrefactor0.13new_component0.11ask_user0.09 gold

intent_057: specific_enough

Intent and readiness · dev
Gold
The target and desired outcome are both identifiable (True)
ember
The target or desired outcome is unclear (False)
Signal
P(true) 0.36 (unsure)
Kit outcome
deferred: treated as unsure
Why this gold label
Asks how a named mechanism behaves; no change requested.
what does the model server do when more advise requests arrive than its admission limit allows?
ember's answer for intent_057, specific_enoughP(true) placed on the no, unsure, and yes zones.nounsureyesgold: yesP = 0.36

intent_059: specific_enough

Intent and readiness · dev
Gold
The target and desired outcome are both identifiable (True)
ember
The target or desired outcome is unclear (False)
Signal
P(true) 0.50 (unsure)
Kit outcome
deferred: treated as unsure
Why this gold label
Asks for the reason behind an identifiable design choice.
why does the MCP server write its logs to stderr instead of stdout?
ember's answer for intent_059, specific_enoughP(true) placed on the no, unsure, and yes zones.nounsureyesgold: yesP = 0.50

intent_061: specific_enough

Intent and readiness · dev
Gold
The target and desired outcome are both identifiable (True)
ember
The target or desired outcome is unclear (False)
Signal
P(true) 0.47 (unsure)
Kit outcome
deferred: treated as unsure
Why this gold label
Asks how a named metric is computed.
how is the Brier score in the benchmark report computed for choice questions?
ember's answer for intent_061, specific_enoughP(true) placed on the no, unsure, and yes zones.nounsureyesgold: yesP = 0.47

intent_063: specific_enough

Intent and readiness · dev
Gold
The target and desired outcome are both identifiable (True)
ember
The target or desired outcome is unclear (False)
Signal
P(true) 0.41 (unsure)
Kit outcome
deferred: treated as unsure
Why this gold label
Asks to explain a named flag.
what does `ember eval context --rescore` do differently from a normal run?
ember's answer for intent_063, specific_enoughP(true) placed on the no, unsure, and yes zones.nounsureyesgold: yesP = 0.41

intent_064: specific_enough

Intent and readiness · test
Gold
The target and desired outcome are both identifiable (True)
ember
The target or desired outcome is unclear (False)
Signal
P(true) 0.46 (unsure)
Kit outcome
deferred: treated as unsure
Why this gold label
Asks for the reason behind a named design choice.
why does the long-context probe pad items with public-domain text instead of random tokens?
ember's answer for intent_064, specific_enoughP(true) placed on the no, unsure, and yes zones.nounsureyesgold: yesP = 0.46

intent_085: specific_enough

Intent and readiness · dev
Gold
The target and desired outcome are both identifiable (True)
ember
The target or desired outcome is unclear (False)
Signal
P(true) 0.48 (unsure)
Kit outcome
deferred: treated as unsure
Why this gold label
Names the metric, the runs, and the report wanted.
routing accuracy dropped from 0.85 to 0.70 between the last two benchmark runs; compare the traces and report which items changed
ember's answer for intent_085, specific_enoughP(true) placed on the no, unsure, and yes zones.nounsureyesgold: yesP = 0.48

intent_086: specific_enough

Intent and readiness · test
Gold
The target and desired outcome are both identifiable (True)
ember
The target or desired outcome is unclear (False)
Signal
P(true) 0.40 (unsure)
Kit outcome
deferred: treated as unsure
Why this gold label
A specific discrepancy to explain.
the model server uses 2 GB more memory on my M3 than on the M4 with the same model; figure out where the difference comes from
ember's answer for intent_086, specific_enoughP(true) placed on the no, unsure, and yes zones.nounsureyesgold: yesP = 0.40

intent_089: specific_enough

Intent and readiness · dev
Gold
The target and desired outcome are both identifiable (True)
ember
The target or desired outcome is unclear (False)
Signal
P(true) 0.37 (unsure)
Kit outcome
deferred: treated as unsure
Why this gold label
A specific measurement to explain.
a 16K-token request takes 48 s while a 2K one takes 6 s; break down where the time goes and write it up
ember's answer for intent_089, specific_enoughP(true) placed on the no, unsure, and yes zones.nounsureyesgold: yesP = 0.37

intent_091: specific_enough

Intent and readiness · dev
Gold
The target or desired outcome is unclear (False)
ember
The target and desired outcome are both identifiable (True)
Signal
P(true) 0.53 (unsure)
Kit outcome
deferred: treated as unsure
Why this gold label
Asks for a look, but 'weird' names no symptom.
something's weird with the logs, take a look
ember's answer for intent_091, specific_enoughP(true) placed on the no, unsure, and yes zones.nounsureyesgold: noP = 0.53

intent_102: intent

Intent and readiness · test
Gold
Repair a specific broken behavior (fix)
ember
Make a change: add, modify, or configure something (implement)
Signal
confidence 0.58 (defer band)
Kit outcome
deferred: gather evidence or ask
Why this gold label
Named defect and where to repair it.
the calibration chart's x-axis title overlaps the tick labels in the report's print layout; fix the spacing
ember's answer for intent_102, intentProbability ember gave each option.question0.01implement0.58 ember's pickinvestigate0.02fix0.40 gold

failure_062: failure_kind

Failure triage · test
Gold
Missing service, dependency, port, file, or configuration (environment)
ember
The test itself is wrong or outdated (test_bug)
Signal
confidence 0.57 (defer band)
Kit outcome
deferred: gather evidence or ask
Why this gold label
The host lacks the required device.
{
  "test": "make eval-context-smoke",
  "output": "RuntimeError: the long-context probe needs MPS; torch.backends.mps.is_available() is False on this Linux runner"
}
ember's answer for failure_062, failure_kindProbability ember gave each option.logic_bug0.15environment0.25 goldflaky0.03test_bug0.57 ember's pick

failure_076: retry

Failure triage · test
Gold
yes (True)
ember
no (False)
Signal
P(true) 0.44 (unsure)
Kit outcome
deferred: treated as unsure
Why this gold label
Nondeterministic ordering of equal timestamps.
{
  "test": "tests/test_vault_audit.py::test_notes_sorted_by_date",
  "output": "AssertionError: ['b.md', 'a.md'] != ['a.md', 'b.md']\nboth notes were written in the same second and the sort falls back to mtime order"
}
ember's answer for failure_076, retryP(true) placed on the no, unsure, and yes zones.nounsureyesgold: yesP = 0.44

risk_049: risk

Change risk · dev
Gold
Negligible (0)
ember
Low (1)
Signal
expected 0.58, 0.58 levels from gold
Kit outcome
within one level of gold
Why this gold label
Help-text typo.
{
  "diff_summary": "Fix a typo in the `ember eval context --help` text",
  "files": [
    "ember/commands/parser.py"
  ],
  "lines_changed": 1
}
ember's answer for risk_049, riskProbability ember gave each option.Negligible0.47 goldLow0.49 ember's pickMedium0.02High0.02

risk_050: risk

Change risk · test
Gold
Negligible (0)
ember
Low (1)
Signal
expected 0.58, 0.58 levels from gold
Kit outcome
within one level of gold
Why this gold label
Whitespace only.
{
  "diff_summary": "Remove a duplicated blank line between two functions",
  "files": [
    "ember/serving/limits.py"
  ],
  "lines_changed": 1
}
ember's answer for risk_050, riskProbability ember gave each option.Negligible0.47 goldLow0.50 ember's pickMedium0.02High0.01

risk_051: risk

Change risk · dev
Gold
Negligible (0)
ember
Low (1)
Signal
expected 0.73, 0.73 levels from gold
Kit outcome
within one level of gold
Why this gold label
Documentation wording.
{
  "diff_summary": "Reword one sentence in CONTRIBUTING.md about running make pr-ready",
  "files": [
    "CONTRIBUTING.md"
  ],
  "lines_changed": 2
}
ember's answer for risk_051, riskProbability ember gave each option.Negligible0.34 goldLow0.62 ember's pickMedium0.03High0.02

risk_052: risk

Change risk · test
Gold
Negligible (0)
ember
Low (1)
Signal
expected 0.64, 0.64 levels from gold
Kit outcome
within one level of gold
Why this gold label
Markup-only documentation change.
{
  "diff_summary": "Add alt text to a decorative divider image on the site's About page",
  "files": [
    "site/about.md"
  ],
  "lines_changed": 1
}
ember's answer for risk_052, riskProbability ember gave each option.Negligible0.41 goldLow0.55 ember's pickMedium0.02High0.01

risk_053: risk

Change risk · dev
Gold
Negligible (0)
ember
Low (1)
Signal
expected 0.55, 0.55 levels from gold
Kit outcome
within one level of gold
Why this gold label
Whitespace only.
{
  "diff_summary": "Strip trailing whitespace from three Markdown files",
  "files": [
    "docs/stride-review.md",
    "COMPATIBILITY.md",
    "SUPPORT.md"
  ],
  "lines_changed": 3
}
ember's answer for risk_053, riskProbability ember gave each option.Negligible0.50 goldLow0.46 ember's pickMedium0.02High0.02

risk_054: risk

Change risk · test
Gold
Negligible (0)
ember
Low (1)
Signal
expected 0.70, 0.70 levels from gold
Kit outcome
within one level of gold
Why this gold label
Documentation spelling.
{
  "diff_summary": "Capitalize 'GitHub' consistently in the README",
  "files": [
    "README.md"
  ],
  "lines_changed": 4
}
ember's answer for risk_054, riskProbability ember gave each option.Negligible0.35 goldLow0.60 ember's pickMedium0.03High0.02

risk_055: risk

Change risk · dev
Gold
Negligible (0)
ember
Low (1)
Signal
expected 0.56, 0.56 levels from gold
Kit outcome
within one level of gold
Why this gold label
Test-only annotation.
{
  "diff_summary": "Add a missing type hint to a helper inside one test module",
  "files": [
    "tests/test_limits.py"
  ],
  "lines_changed": 1
}
ember's answer for risk_055, riskProbability ember gave each option.Negligible0.49 goldLow0.48 ember's pickMedium0.02High0.01

risk_056: risk

Change risk · test
Gold
Negligible (0)
ember
Low (1)
Signal
expected 0.57, 0.57 levels from gold
Kit outcome
within one level of gold
Why this gold label
A comment in a template.
{
  "diff_summary": "Fix the indentation of a comment in the bug-report issue template",
  "files": [
    ".github/ISSUE_TEMPLATE/bug_report.yml"
  ],
  "lines_changed": 1
}
ember's answer for risk_056, riskProbability ember gave each option.Negligible0.49 goldLow0.48 ember's pickMedium0.02High0.02

risk_057: risk

Change risk · dev
Gold
Negligible (0)
ember
Low (1)
Signal
expected 0.81, 0.81 levels from gold
Kit outcome
within one level of gold
Why this gold label
Project-notes fix.
{
  "diff_summary": "Correct a wrong line reference in a vault discovery note",
  "files": [
    "vault/discoveries/2026-10-09-request-size-checks-miss-what-the-model-sees.md"
  ],
  "lines_changed": 1
}
ember's answer for risk_057, riskProbability ember gave each option.Negligible0.29 goldLow0.64 ember's pickMedium0.05High0.02

risk_058: risk

Change risk · test
Gold
Negligible (0)
ember
Low (1)
Signal
expected 0.53, 0.53 levels from gold
Kit outcome
within one level of gold
Why this gold label
Test-only rename.
{
  "diff_summary": "Rename a test function to match the naming convention; body unchanged",
  "files": [
    "tests/test_media.py"
  ],
  "lines_changed": 1
}
ember's answer for risk_058, riskProbability ember gave each option.Negligible0.52 goldLow0.44 ember's pickMedium0.02High0.02

risk_059: risk

Change risk · dev
Gold
Negligible (0)
ember
Low (1)
Signal
expected 0.56, 0.56 levels from gold
Kit outcome
within one level of gold
Why this gold label
Documentation formatting.
{
  "diff_summary": "Add the bash language tag to a code fence in the README",
  "files": [
    "README.md"
  ],
  "lines_changed": 1
}
ember's answer for risk_059, riskProbability ember gave each option.Negligible0.50 goldLow0.46 ember's pickMedium0.02High0.02

risk_073: risk

Change risk · dev
Gold
Medium (2)
ember
Low (1)
Signal
expected 1.41, 0.59 levels from gold
Kit outcome
within one level of gold
Why this gold label
Runtime dependency upgrade under the model.
{
  "diff_summary": "Upgrade transformers from 5.18 to 5.20",
  "files": [
    "pyproject.toml",
    "uv.lock"
  ],
  "lines_changed": 30
}
ember's answer for risk_073, riskProbability ember gave each option.Negligible0.11Low0.43 ember's pickMedium0.42 goldHigh0.05

risk_073: needs_review

Change risk · dev
Gold
yes (True)
ember
no (False)
Signal
P(true) 0.35 (unsure)
Kit outcome
deferred: treated as unsure
Why this gold label
Runtime dependency upgrade under the model.
{
  "diff_summary": "Upgrade transformers from 5.18 to 5.20",
  "files": [
    "pyproject.toml",
    "uv.lock"
  ],
  "lines_changed": 30
}
ember's answer for risk_073, needs_reviewP(true) placed on the no, unsure, and yes zones.nounsureyesgold: yesP = 0.35

risk_076: risk

Change risk · test
Gold
Medium (2)
ember
Low (1)
Signal
expected 0.92, 1.08 levels from gold
Kit outcome
missed the exact level
Why this gold label
One line, but it widens memory exposure.
{
  "diff_summary": "Raise the maximum number of images per request from 32 to 128",
  "files": [
    "ember/serving/media.py"
  ],
  "lines_changed": 1
}
ember's answer for risk_076, riskProbability ember gave each option.Negligible0.25Low0.62 ember's pickMedium0.10 goldHigh0.04

risk_076: needs_review

Change risk · test
Gold
yes (True)
ember
no (False)
Signal
P(true) 0.17 (no)
Kit outcome
acted on a wrong answer
Why this gold label
One line, but it widens memory exposure.
{
  "diff_summary": "Raise the maximum number of images per request from 32 to 128",
  "files": [
    "ember/serving/media.py"
  ],
  "lines_changed": 1
}
ember's answer for risk_076, needs_reviewP(true) placed on the no, unsure, and yes zones.nounsureyesgold: yesP = 0.17

risk_081: risk

Change risk · dev
Gold
Medium (2)
ember
Low (1)
Signal
expected 1.42, 0.58 levels from gold
Kit outcome
within one level of gold
Why this gold label
Could miss peaks that set shipped caps.
{
  "diff_summary": "Poll the probe's memory sampler every 100 ms instead of every 20 ms",
  "files": [
    "evals/context/memory_sampler.py"
  ],
  "lines_changed": 1
}
ember's answer for risk_081, riskProbability ember gave each option.Negligible0.15Low0.36 ember's pickMedium0.41 goldHigh0.08

risk_081: needs_review

Change risk · dev
Gold
yes (True)
ember
no (False)
Signal
P(true) 0.32 (unsure)
Kit outcome
deferred: treated as unsure
Why this gold label
Could miss peaks that set shipped caps.
{
  "diff_summary": "Poll the probe's memory sampler every 100 ms instead of every 20 ms",
  "files": [
    "evals/context/memory_sampler.py"
  ],
  "lines_changed": 1
}
ember's answer for risk_081, needs_reviewP(true) placed on the no, unsure, and yes zones.nounsureyesgold: yesP = 0.32

risk_082: risk

Change risk · test
Gold
High (3)
ember
Medium (2)
Signal
expected 1.78, 1.22 levels from gold
Kit outcome
missed the exact level
Why this gold label
Silently answers on text the model never saw.
{
  "diff_summary": "Truncate oversized requests to the cap instead of refusing them with 413",
  "files": [
    "ember/serving/runtime.py"
  ],
  "lines_changed": 20
}
ember's answer for risk_082, riskProbability ember gave each option.Negligible0.08Low0.19Medium0.59 ember's pickHigh0.14 gold

risk_083: risk

Change risk · dev
Gold
High (3)
ember
Low (1)
Signal
expected 1.06, 1.94 levels from gold
Kit outcome
missed the exact level
Why this gold label
An environment switch disables a safety check.
{
  "diff_summary": "Skip the request size check entirely when EMBER_DEBUG=1 is set",
  "files": [
    "ember/serving/runtime.py"
  ],
  "lines_changed": 4
}
ember's answer for risk_083, riskProbability ember gave each option.Negligible0.22Low0.56 ember's pickMedium0.15High0.07 gold

risk_083: needs_review

Change risk · dev
Gold
yes (True)
ember
no (False)
Signal
P(true) 0.22 (unsure)
Kit outcome
deferred: treated as unsure
Why this gold label
An environment switch disables a safety check.
{
  "diff_summary": "Skip the request size check entirely when EMBER_DEBUG=1 is set",
  "files": [
    "ember/serving/runtime.py"
  ],
  "lines_changed": 4
}
ember's answer for risk_083, needs_reviewP(true) placed on the no, unsure, and yes zones.nounsureyesgold: yesP = 0.22

risk_084: risk

Change risk · test
Gold
High (3)
ember
Low (1)
Signal
expected 0.93, 2.07 levels from gold
Kit outcome
missed the exact level
Why this gold label
Writes a secret to the log.
{
  "diff_summary": "Log the bearer token at startup to help debug auth problems",
  "files": [
    "ember/serving/server.py"
  ],
  "lines_changed": 1
}
ember's answer for risk_084, riskProbability ember gave each option.Negligible0.24Low0.62 ember's pickMedium0.09High0.04 gold

risk_084: needs_review

Change risk · test
Gold
yes (True)
ember
no (False)
Signal
P(true) 0.13 (no)
Kit outcome
acted on a wrong answer
Why this gold label
Writes a secret to the log.
{
  "diff_summary": "Log the bearer token at startup to help debug auth problems",
  "files": [
    "ember/serving/server.py"
  ],
  "lines_changed": 1
}
ember's answer for risk_084, needs_reviewP(true) placed on the no, unsure, and yes zones.nounsureyesgold: yesP = 0.13

risk_087: risk

Change risk · dev
Gold
High (3)
ember
Medium (2)
Signal
expected 1.84, 1.16 levels from gold
Kit outcome
missed the exact level
Why this gold label
Ships unreviewed builds to every user.
{
  "diff_summary": "Publish to PyPI on every push to main instead of from release tags",
  "files": [
    ".github/workflows/release-ember.yml"
  ],
  "lines_changed": 15
}
ember's answer for risk_087, riskProbability ember gave each option.Negligible0.09Low0.18Medium0.54 ember's pickHigh0.20 gold

risk_088: risk

Change risk · test
Gold
High (3)
ember
Low (1)
Signal
expected 1.00, 2.00 levels from gold
Kit outcome
missed the exact level
Why this gold label
Exposes secrets to untrusted code.
{
  "diff_summary": "Run the CI workflow on pull_request_target so fork pull requests can use repository secrets",
  "files": [
    ".github/workflows/ci.yml"
  ],
  "lines_changed": 2
}
ember's answer for risk_088, riskProbability ember gave each option.Negligible0.21Low0.63 ember's pickMedium0.12High0.05 gold

risk_088: needs_review

Change risk · test
Gold
yes (True)
ember
no (False)
Signal
P(true) 0.22 (unsure)
Kit outcome
deferred: treated as unsure
Why this gold label
Exposes secrets to untrusted code.
{
  "diff_summary": "Run the CI workflow on pull_request_target so fork pull requests can use repository secrets",
  "files": [
    ".github/workflows/ci.yml"
  ],
  "lines_changed": 2
}
ember's answer for risk_088, needs_reviewP(true) placed on the no, unsure, and yes zones.nounsureyesgold: yesP = 0.22

risk_089: risk

Change risk · dev
Gold
High (3)
ember
Medium (2)
Signal
expected 1.93, 1.07 levels from gold
Kit outcome
missed the exact level
Why this gold label
Deletes users' other servers in every project.
{
  "diff_summary": "Make `ember init --global` replace the whole mcp table in the global opencode config",
  "files": [
    "ember/opencode/opencode_config.py"
  ],
  "lines_changed": 10
}
ember's answer for risk_089, riskProbability ember gave each option.Negligible0.08Low0.14Medium0.54 ember's pickHigh0.24 gold

risk_090: risk

Change risk · test
Gold
High (3)
ember
Medium (2)
Signal
expected 2.02, 0.98 levels from gold
Kit outcome
within one level of gold
Why this gold label
Invites out-of-memory crashes.
{
  "diff_summary": "Raise the admission limit to 64 concurrent requests and remove the per-request cap",
  "files": [
    "ember/serving/server.py",
    "ember/serving/limits.py"
  ],
  "lines_changed": 25
}
ember's answer for risk_090, riskProbability ember gave each option.Negligible0.08Low0.13Medium0.48 ember's pickHigh0.31 gold

risk_092: risk

Change risk · test
Gold
High (3)
ember
Medium (2)
Signal
expected 2.45, 0.55 levels from gold
Kit outcome
within one level of gold
Why this gold label
Puts a secret in a config file.
{
  "diff_summary": "Write the S3 secret access key into the generated kilo.json so remote setups work without env vars",
  "files": [
    "ember/kilocode/kilocode_config.py"
  ],
  "lines_changed": 6
}
ember's answer for risk_092, riskProbability ember gave each option.Negligible0.06Low0.08Medium0.21 ember's pickHigh0.65 gold

routing_072: owner

Routing and ownership · test
Gold
Not enough information to tell (unclear)
ember
Build, CI, deployment, and configuration (infra)
Signal
confidence 0.63 (verify band)
Kit outcome
acted, flagged for verification
Why this gold label
No indication of which part failed.
{
  "error": "exit status 1",
  "path": ""
}
ember's answer for routing_072, ownerProbability ember gave each option.api0.04storage0.02ui0.02infra0.63 ember's pickunclear0.29 gold

routing_074: owner

Routing and ownership · test
Gold
Not enough information to tell (unclear)
ember
Build, CI, deployment, and configuration (infra)
Signal
confidence 0.84 (verify band)
Kit outcome
acted, flagged for verification
Why this gold label
Could be any part of the system.
{
  "error": "the job didn't finish",
  "path": ""
}
ember's answer for routing_074, ownerProbability ember gave each option.api0.01storage0.01ui0.01infra0.84 ember's pickunclear0.13 gold

effort_041: effort

Effort and approach · dev
Gold
Trivial (minutes) (0)
ember
Small (under an hour) (1)
Signal
expected 0.78, 0.78 levels from gold
Kit outcome
within one level of gold
Why this gold label
A one-string edit.
{
  "task": "Show the default value in the --lengths help text of `ember eval context`",
  "constraints": [
    "help text only"
  ]
}
ember's answer for effort_041, effortProbability ember gave each option.Trivial (minutes)0.39 goldSmall (under an hour)0.45 ember's pickMedium (a few hours)0.15Large (a day or more)0.02

effort_044: effort

Effort and approach · test
Gold
Trivial (minutes) (0)
ember
Small (under an hour) (1)
Signal
expected 0.63, 0.63 levels from gold
Kit outcome
within one level of gold
Why this gold label
A one-line formatting change.
{
  "task": "Round the latency shown by `ember status` to whole milliseconds",
  "constraints": [
    "display only"
  ]
}
ember's answer for effort_044, effortProbability ember gave each option.Trivial (minutes)0.47 goldSmall (under an hour)0.43 ember's pickMedium (a few hours)0.09Large (a day or more)0.01

effort_046: effort

Effort and approach · test
Gold
Trivial (minutes) (0)
ember
Small (under an hour) (1)
Signal
expected 1.02, 1.02 levels from gold
Kit outcome
missed the exact level
Why this gold label
A single value.
{
  "task": "Return exit code 2 instead of 1 when `ember eval context` gets an unknown model name",
  "constraints": [
    "keep the error message"
  ]
}
ember's answer for effort_046, effortProbability ember gave each option.Trivial (minutes)0.23 goldSmall (under an hour)0.54 ember's pickMedium (a few hours)0.20Large (a day or more)0.03

effort_048: effort

Effort and approach · test
Gold
Trivial (minutes) (0)
ember
Small (under an hour) (1)
Signal
expected 0.77, 0.77 levels from gold
Kit outcome
within one level of gold
Why this gold label
A one-line output change.
{
  "task": "Remove the blank line `ember doctor` prints between harness sections",
  "constraints": [
    "output only"
  ]
}
ember's answer for effort_048, effortProbability ember gave each option.Trivial (minutes)0.39 goldSmall (under an hour)0.46 ember's pickMedium (a few hours)0.13Large (a day or more)0.01

effort_050: approach

Effort and approach · test
Gold
Restructure the code so the fix is natural (refactor)
ember
Add a new module or service (new_component)
Signal
confidence 0.43 (defer band)
Kit outcome
deferred: gather evidence or ask
Why this gold label
Extract one helper and switch three callers.
{
  "task": "The three eval runners parse the JSONL dataset with slightly different error handling; share one loader",
  "constraints": [
    "no output changes",
    "existing tests must pass"
  ]
}
ember's answer for effort_050, approachProbability ember gave each option.minimal_patch0.19refactor0.34 goldnew_component0.43 ember's pickask_user0.04

effort_052: effort

Effort and approach · test
Gold
Small (under an hour) (1)
ember
Medium (a few hours) (2)
Signal
expected 1.61, 0.61 levels from gold
Kit outcome
within one level of gold
Why this gold label
Move duplicated test code into one place.
{
  "task": "Four test modules each define their own fake joint model with copy-pasted encode functions; move it to one shared module",
  "constraints": [
    "tests only",
    "no production changes"
  ]
}
ember's answer for effort_052, effortProbability ember gave each option.Trivial (minutes)0.06Small (under an hour)0.31 goldMedium (a few hours)0.57 ember's pickLarge (a day or more)0.05

effort_053: effort

Effort and approach · dev
Gold
Small (under an hour) (1)
ember
Medium (a few hours) (2)
Signal
expected 1.75, 0.75 levels from gold
Kit outcome
within one level of gold
Why this gold label
Extract one shared helper.
{
  "task": "doctor, status, and the MCP server each describe the endpoint in their own format; give them one formatter",
  "constraints": [
    "keep each command's wording"
  ]
}
ember's answer for effort_053, effortProbability ember gave each option.Trivial (minutes)0.08Small (under an hour)0.21 goldMedium (a few hours)0.59 ember's pickLarge (a day or more)0.12

effort_054: effort

Effort and approach · test
Gold
Small (under an hour) (1)
ember
Medium (a few hours) (2)
Signal
expected 1.77, 0.77 levels from gold
Kit outcome
within one level of gold
Why this gold label
Gather duplicated strings behind one function.
{
  "task": "Messages for invalid settings are built in five places with different wording; centralize them",
  "constraints": [
    "same exit codes"
  ]
}
ember's answer for effort_054, effortProbability ember gave each option.Trivial (minutes)0.04Small (under an hour)0.23 goldMedium (a few hours)0.65 ember's pickLarge (a day or more)0.08

effort_055: effort

Effort and approach · dev
Gold
Small (under an hour) (1)
ember
Medium (a few hours) (2)
Signal
expected 2.07, 1.07 levels from gold
Kit outcome
missed the exact level
Why this gold label
A mechanical replacement behind an enum.
{
  "task": "Row status values are compared as bare strings in nine places; replace them with an enum",
  "constraints": [
    "no on-disk format change"
  ]
}
ember's answer for effort_055, effortProbability ember gave each option.Trivial (minutes)0.04Small (under an hour)0.09 goldMedium (a few hours)0.63 ember's pickLarge (a day or more)0.24

effort_055: approach

Effort and approach · dev
Gold
Restructure the code so the fix is natural (refactor)
ember
Smallest change that fixes the symptom (minimal_patch)
Signal
confidence 0.43 (defer band)
Kit outcome
deferred: gather evidence or ask
Why this gold label
A mechanical replacement behind an enum.
{
  "task": "Row status values are compared as bare strings in nine places; replace them with an enum",
  "constraints": [
    "no on-disk format change"
  ]
}
ember's answer for effort_055, approachProbability ember gave each option.minimal_patch0.43 ember's pickrefactor0.41 goldnew_component0.06ask_user0.10

effort_056: effort

Effort and approach · test
Gold
Medium (a few hours) (2)
ember
Small (under an hour) (1)
Signal
expected 1.47, 0.53 levels from gold
Kit outcome
within one level of gold
Why this gold label
Merge two code paths under strict output constraints.
{
  "task": "The probe worker and the rescore path compute the same per-cell statistics in two functions; make both use one",
  "constraints": [
    "byte-identical summaries",
    "resume must keep working"
  ]
}
ember's answer for effort_056, effortProbability ember gave each option.Trivial (minutes)0.11Small (under an hour)0.37 ember's pickMedium (a few hours)0.47 goldLarge (a day or more)0.05

effort_058: effort

Effort and approach · test
Gold
Large (a day or more) (3)
ember
Medium (a few hours) (2)
Signal
expected 1.97, 1.03 levels from gold
Kit outcome
missed the exact level
Why this gold label
A large restructuring across many modules.
{
  "task": "Move the eval harness out of evals/ and scripts/ into an installable package so `ember eval` works outside a clone",
  "constraints": [
    "keep every command and its output",
    "no new dependencies"
  ]
}
ember's answer for effort_058, effortProbability ember gave each option.Trivial (minutes)0.05Small (under an hour)0.16Medium (a few hours)0.56 ember's pickLarge (a day or more)0.23 gold

effort_059: effort

Effort and approach · dev
Gold
Medium (a few hours) (2)
ember
Small (under an hour) (1)
Signal
expected 1.39, 0.61 levels from gold
Kit outcome
within one level of gold
Why this gold label
A new mode with its own path through the probe.
{
  "task": "Add a memory-only mode to the long-context probe that measures peak memory without scoring quality",
  "constraints": [
    "reuse the worker",
    "no change to canonical runs"
  ]
}
ember's answer for effort_059, effortProbability ember gave each option.Trivial (minutes)0.13Small (under an hour)0.40 ember's pickMedium (a few hours)0.43 goldLarge (a day or more)0.05

effort_059: approach

Effort and approach · dev
Gold
Add a new module or service (new_component)
ember
Smallest change that fixes the symptom (minimal_patch)
Signal
confidence 0.68 (verify band)
Kit outcome
acted, flagged for verification
Why this gold label
A new mode with its own path through the probe.
{
  "task": "Add a memory-only mode to the long-context probe that measures peak memory without scoring quality",
  "constraints": [
    "reuse the worker",
    "no change to canonical runs"
  ]
}
ember's answer for effort_059, approachProbability ember gave each option.minimal_patch0.68 ember's pickrefactor0.10new_component0.17 goldask_user0.04

effort_060: approach

Effort and approach · test
Gold
Add a new module or service (new_component)
ember
Smallest change that fixes the symptom (minimal_patch)
Signal
confidence 0.60 (defer band)
Kit outcome
deferred: gather evidence or ask
Why this gold label
One new artifact.
{
  "task": "Add a Grafana dashboard JSON for the ember_advise_* metrics",
  "constraints": [
    "new file only",
    "no server changes"
  ]
}
ember's answer for effort_060, approachProbability ember gave each option.minimal_patch0.60 ember's pickrefactor0.01new_component0.30 goldask_user0.09

effort_062: effort

Effort and approach · test
Gold
Small (under an hour) (1)
ember
Medium (a few hours) (2)
Signal
expected 1.54, 0.54 levels from gold
Kit outcome
within one level of gold
Why this gold label
One new workflow.
{
  "task": "Add a nightly workflow that builds the site and checks every internal link",
  "constraints": [
    "never on pull requests",
    "pin every action"
  ]
}
ember's answer for effort_062, effortProbability ember gave each option.Trivial (minutes)0.07Small (under an hour)0.36 goldMedium (a few hours)0.53 ember's pickLarge (a day or more)0.04

effort_064: effort

Effort and approach · test
Gold
Large (a day or more) (3)
ember
Medium (a few hours) (2)
Signal
expected 1.96, 1.04 levels from gold
Kit outcome
missed the exact level
Why this gold label
A new deployment artifact with many parts.
{
  "task": "Add a Helm chart for running the model server on a CUDA cluster",
  "constraints": [
    "values for the GPU pool and auth token",
    "no code changes"
  ]
}
ember's answer for effort_064, effortProbability ember gave each option.Trivial (minutes)0.05Small (under an hour)0.14Medium (a few hours)0.61 ember's pickLarge (a day or more)0.20 gold

effort_065: effort

Effort and approach · dev
Gold
Large (a day or more) (3)
ember
Medium (a few hours) (2)
Signal
expected 2.10, 0.90 levels from gold
Kit outcome
within one level of gold
Why this gold label
A new backend path through the probe, plus a run.
{
  "task": "Add CUDA support to the long-context probe: a CUDA memory sampler, per-GPU memory budgets, and a documented run on a hosted GPU",
  "constraints": [
    "MPS results must stay comparable",
    "no change to the cap rule"
  ]
}
ember's answer for effort_065, effortProbability ember gave each option.Trivial (minutes)0.02Small (under an hour)0.08Medium (a few hours)0.67 ember's pickLarge (a day or more)0.23 gold

effort_066: effort

Effort and approach · test
Gold
Medium (a few hours) (2)
ember
Small (under an hour) (1)
Signal
expected 1.42, 0.58 levels from gold
Kit outcome
within one level of gold
Why this gold label
A new process.
{
  "task": "Add a health-check sidecar that restarts the model server when /health stops answering",
  "constraints": [
    "separate process",
    "never kill by port"
  ]
}
ember's answer for effort_066, effortProbability ember gave each option.Trivial (minutes)0.11Small (under an hour)0.42 ember's pickMedium (a few hours)0.42 goldLarge (a day or more)0.05

effort_067: approach

Effort and approach · dev
Gold
Add a new module or service (new_component)
ember
Smallest change that fixes the symptom (minimal_patch)
Signal
confidence 0.56 (defer band)
Kit outcome
deferred: gather evidence or ask
Why this gold label
One generated artifact.
{
  "task": "Add a JSON Schema file for the advise request and ship it with the package",
  "constraints": [
    "generated from the Pydantic models",
    "one new module"
  ]
}
ember's answer for effort_067, approachProbability ember gave each option.minimal_patch0.56 ember's pickrefactor0.04new_component0.26 goldask_user0.15

effort_068: effort

Effort and approach · test
Gold
Trivial (minutes) (0)
ember
Small (under an hour) (1)
Signal
expected 0.96, 0.96 levels from gold
Kit outcome
within one level of gold
Why this gold label
A one-constant change whose trade-off is a product decision.
{
  "task": "Lower the default per-request cap from 32,768 to 24,576 tokens",
  "constraints": [
    "requests that work today would get a 413",
    "one constant plus docs"
  ]
}
ember's answer for effort_068, effortProbability ember gave each option.Trivial (minutes)0.32 goldSmall (under an hour)0.44 ember's pickMedium (a few hours)0.19Large (a day or more)0.05

effort_068: approach

Effort and approach · test
Gold
The trade-off needs a human decision (ask_user)
ember
Smallest change that fixes the symptom (minimal_patch)
Signal
confidence 0.84 (verify band)
Kit outcome
acted, flagged for verification
Why this gold label
A one-constant change whose trade-off is a product decision.
{
  "task": "Lower the default per-request cap from 32,768 to 24,576 tokens",
  "constraints": [
    "requests that work today would get a 413",
    "one constant plus docs"
  ]
}
ember's answer for effort_068, approachProbability ember gave each option.minimal_patch0.84 ember's pickrefactor0.08new_component0.02ask_user0.06 gold

effort_069: effort

Effort and approach · dev
Gold
Trivial (minutes) (0)
ember
Small (under an hour) (1)
Signal
expected 0.94, 0.94 levels from gold
Kit outcome
within one level of gold
Why this gold label
Changes a pre-declared rule.
{
  "task": "Loosen the probe's accuracy tolerance from 2 to 3 points so the current items can resolve it",
  "constraints": [
    "the tolerance was declared before any run",
    "one constant"
  ]
}
ember's answer for effort_069, effortProbability ember gave each option.Trivial (minutes)0.33 goldSmall (under an hour)0.43 ember's pickMedium (a few hours)0.21Large (a day or more)0.03

effort_069: approach

Effort and approach · dev
Gold
The trade-off needs a human decision (ask_user)
ember
Smallest change that fixes the symptom (minimal_patch)
Signal
confidence 0.85 (trust band)
Kit outcome
acted on a wrong answer
Why this gold label
Changes a pre-declared rule.
{
  "task": "Loosen the probe's accuracy tolerance from 2 to 3 points so the current items can resolve it",
  "constraints": [
    "the tolerance was declared before any run",
    "one constant"
  ]
}
ember's answer for effort_069, approachProbability ember gave each option.minimal_patch0.85 ember's pickrefactor0.05new_component0.01ask_user0.09 gold

effort_070: approach

Effort and approach · test
Gold
The trade-off needs a human decision (ask_user)
ember
Smallest change that fixes the symptom (minimal_patch)
Signal
confidence 0.70 (verify band)
Kit outcome
acted, flagged for verification
Why this gold label
Trades correctness for convenience.
{
  "task": "Truncate requests over the cap instead of refusing them",
  "constraints": [
    "agents would get answers on text the model never saw",
    "removes the 413 path"
  ]
}
ember's answer for effort_070, approachProbability ember gave each option.minimal_patch0.70 ember's pickrefactor0.23new_component0.03ask_user0.04 gold

effort_071: effort

Effort and approach · dev
Gold
Small (under an hour) (1)
ember
Medium (a few hours) (2)
Signal
expected 1.51, 0.51 levels from gold
Kit outcome
within one level of gold
Why this gold label
Repository cost against reproducibility.
{
  "task": "Commit the raw probe rows (about 2 GB) next to the summary",
  "constraints": [
    "repository size",
    "the rows let others re-score runs"
  ]
}
ember's answer for effort_071, effortProbability ember gave each option.Trivial (minutes)0.15Small (under an hour)0.30 goldMedium (a few hours)0.44 ember's pickLarge (a day or more)0.11

effort_072: effort

Effort and approach · test
Gold
Trivial (minutes) (0)
ember
Small (under an hour) (1)
Signal
expected 0.64, 0.64 levels from gold
Kit outcome
within one level of gold
Why this gold label
Compute against coverage.
{
  "task": "Drop the full model from the default probe run",
  "constraints": [
    "full takes about ten days",
    "its cap would stay unmeasured"
  ]
}
ember's answer for effort_072, effortProbability ember gave each option.Trivial (minutes)0.55 goldSmall (under an hour)0.29 ember's pickMedium (a few hours)0.13Large (a day or more)0.03

effort_072: approach

Effort and approach · test
Gold
The trade-off needs a human decision (ask_user)
ember
Smallest change that fixes the symptom (minimal_patch)
Signal
confidence 0.93 (trust band)
Kit outcome
acted on a wrong answer
Why this gold label
Compute against coverage.
{
  "task": "Drop the full model from the default probe run",
  "constraints": [
    "full takes about ten days",
    "its cap would stay unmeasured"
  ]
}
ember's answer for effort_072, approachProbability ember gave each option.minimal_patch0.93 ember's pickrefactor0.03new_component0.01ask_user0.03 gold

effort_073: approach

Effort and approach · dev
Gold
The trade-off needs a human decision (ask_user)
ember
Smallest change that fixes the symptom (minimal_patch)
Signal
confidence 0.78 (verify band)
Kit outcome
acted, flagged for verification
Why this gold label
Security against operability.
{
  "task": "Require the bearer token on /health as well as on /v1/systemone",
  "constraints": [
    "monitoring probes call /health without credentials",
    "one dependency change"
  ]
}
ember's answer for effort_073, approachProbability ember gave each option.minimal_patch0.78 ember's pickrefactor0.09new_component0.01ask_user0.12 gold

effort_074: effort

Effort and approach · test
Gold
Trivial (minutes) (0)
ember
Small (under an hour) (1)
Signal
expected 0.61, 0.61 levels from gold
Kit outcome
within one level of gold
Why this gold label
A default with user-facing trade-offs.
{
  "task": "Turn EMBER_AUTOSTART off by default",
  "constraints": [
    "first run needs an extra step",
    "fewer surprise model loads"
  ]
}
ember's answer for effort_074, effortProbability ember gave each option.Trivial (minutes)0.50 goldSmall (under an hour)0.40 ember's pickMedium (a few hours)0.08Large (a day or more)0.02

effort_074: approach

Effort and approach · test
Gold
The trade-off needs a human decision (ask_user)
ember
Smallest change that fixes the symptom (minimal_patch)
Signal
confidence 0.86 (trust band)
Kit outcome
acted on a wrong answer
Why this gold label
A default with user-facing trade-offs.
{
  "task": "Turn EMBER_AUTOSTART off by default",
  "constraints": [
    "first run needs an extra step",
    "fewer surprise model loads"
  ]
}
ember's answer for effort_074, approachProbability ember gave each option.minimal_patch0.86 ember's pickrefactor0.09new_component0.03ask_user0.03 gold

effort_075: approach

Effort and approach · dev
Gold
The trade-off needs a human decision (ask_user)
ember
Smallest change that fixes the symptom (minimal_patch)
Signal
confidence 0.73 (verify band)
Kit outcome
acted, flagged for verification
Why this gold label
Conflicts with a project principle.
{
  "task": "Make a paid hosted endpoint the default server_url for new installs",
  "constraints": [
    "local-first is a constitution principle",
    "state would leave the machine"
  ]
}
ember's answer for effort_075, approachProbability ember gave each option.minimal_patch0.73 ember's pickrefactor0.09new_component0.06ask_user0.11 gold

effort_076: effort

Effort and approach · test
Gold
Small (under an hour) (1)
ember
Medium (a few hours) (2)
Signal
expected 1.72, 0.72 levels from gold
Kit outcome
within one level of gold
Why this gold label
Touches the benchmark's integrity.
{
  "task": "Replace gold labels with the model's predictions wherever a reviewer agrees",
  "constraints": [
    "labels are frozen before runs",
    "would raise reported accuracy"
  ]
}
ember's answer for effort_076, effortProbability ember gave each option.Trivial (minutes)0.08Small (under an hour)0.21 goldMedium (a few hours)0.60 ember's pickLarge (a day or more)0.10