ember

3System under test

Two runners exercise the same server. The benchmark runner posts every item to the model server's POST /v1/systemone, the endpoint the MCP tool forwards to, so it measures the model and server end to end. The agent eval runner drives a real coding agent through opencode run --pure, so it also measures whether the agent calls ember at the right moments, asks well, and acts on the answer.

ember call pathBoxes for the agent, ember-mcp, the model server, and Clef-Flash, joined left to right; the agent eval runner drives the agent and the benchmark runner calls the model server directly.tools/call adviseHTTPEngine.adviseopencode run --pureHTTP, every itemCoding agentopencode, Claude Code,Codexember-mcpMCP stdio: advise tool,ember://guideModel serverFastAPI: POST/v1/systemoneClef-FlashQwen3.5 + joint schemahead, fp16 on MPSAgent eval runnerevals/eval/run_agent_evals.pyBenchmark runnerevals/eval/run_evals.py
Figure 1 The call path of an agent's advise call. The dashed routes are the two runners: the agent eval drives a coding agent, and the benchmark posts straight to the model server.
Text version
agent eval runner (evals/eval/run_agent_evals.py)
   │ opencode run --pure, one sandbox per session
   ▼
coding agent ──tools/call advise──► ember-mcp (stdio, ember/mcp/mcp_server.py)
                                       │ HTTP POST /v1/systemone
                                       ▼
model benchmark runner ──HTTP──► model server (ember/serving/server.py)
(evals/eval/run_evals.py)              │ Engine.advise
                                       ▼
                                Clef-Flash, fp16 on MPS
FactValue
Modelflash, ? parameters
Pinned revisionnot recorded
Device and precisionmps, float16
Context window262,144 tokens
HostApple M4 Max, macOS-26.6.2-arm64-arm-64bit
SoftwarePython 3.12.11, ember-advise 0.12.0, torch 2.14.1, transformers 5.18.0, mcp 2.3.0
DeploymentLocal (MPS)
Commit51d55b3
Run at2026-10-12 02:18 UTC
Report generated2026-10-12 02:26 UTC
Datasetclef-flash.jsonl
Dataset SHA-256b008a7714ffd785513986905079d87629ffdde0563b2ab1e3686ace917d693c6
Dataset unchanged since the runyes
Request errors0