ember

14Reproducibility

Run these from a clone of the repository on Apple Silicon. Identical requests to the same server return identical answers, so a re-run on the pins below should reproduce these numbers.

make bootstrap               # dependencies, pinned weights, readiness check
ember start                  # warm model server on 127.0.0.1:8765
ember eval run               # every item; or --split test, --category <recipe>
ember eval export            # this report: Markdown, HTML, figures, raw data
ember eval report --compare results/<a>_results.json results/<b>_results.json
PinValue
Commit6a34e29
Dataset SHA-2561581134e791f35ba4c6f886b9431dba50ecc188a45917b077f7be22971ee8a47
Model revision17f0b0ad64efb65d273590632833508766b2aae6
Results file17f0b0ad64efb65d273590632833508766b2aae6_20261012T012758Z_results.json
Trace file17f0b0ad64efb65d273590632833508766b2aae6_20261012T012758Z_trace.jsonl
Bootstrap1,000 resamples, seed 0, percentile interval
Package ember-advise0.12.0
Package torch2.14.1
Package transformers5.18.0
Package mcp2.3.0