14Reproducibility
Run these from a clone of the repository on Apple Silicon. Identical requests to the same server return identical answers, so a re-run on the pins below should reproduce these numbers.
make bootstrap # dependencies, pinned weights, readiness check
ember start # warm model server on 127.0.0.1:8765
ember eval run # every item; or --split test, --category <recipe>
ember eval export # this report: Markdown, HTML, figures, raw data
ember eval report --compare results/<a>_results.json results/<b>_results.json| Pin | Value |
|---|---|
| Commit | f3187ba |
| Dataset SHA-256 | 1ca360b3af432c36767d4870e47c40f39e2c768bc692de9ef70b9d17c0b8e54e |
| Model revision | not recorded |
| Results file | flash_20261012T220610Z_results.json |
| Trace file | flash_20261012T220610Z_trace.jsonl |
| Bootstrap | 1,000 resamples, seed 0, percentile interval |
| Package ember-advise | 0.14.0 |
| Package torch | 2.14.1 |
| Package transformers | 5.18.0 |
| Package mcp | 2.3.0 |