ember

3System under test

Two runners exercise the same server. The benchmark runner posts every item to the model server's POST /v1/systemone, the endpoint the MCP tool forwards to, so it measures the model and server end to end. The agent eval runner drives a real coding agent through opencode run --pure, so it also measures whether the agent calls ember at the right moments, asks well, and acts on the answer.

ember call pathBoxes for the agent, ember-mcp, the model server, and Clef-Flash, joined left to right; the agent eval runner drives the agent and the benchmark runner calls the model server directly.tools/call adviseHTTPEngine.adviseopencode run --pureHTTP, every itemCoding agentopencode, Claude Code,Codexember-mcpMCP stdio: advise tool,ember://guideModel serverFastAPI: POST/v1/systemoneClef-FlashQwen3.5 + joint schemahead, fp16 on MPSAgent eval runnerevals/eval/run_agent_evals.pyBenchmark runnerevals/eval/run_evals.py
Figure 1 The call path of an agent's advise call. The dashed routes are the two runners: the agent eval drives a coding agent, and the benchmark posts straight to the model server.
Text version
agent eval runner (evals/eval/run_agent_evals.py)
   │ opencode run --pure, one sandbox per session
   ▼
coding agent ──tools/call advise──► ember-mcp (stdio, ember/mcp/mcp_server.py)
                                       │ HTTP POST /v1/systemone
                                       ▼
model benchmark runner ──HTTP──► model server (ember/serving/server.py)
(evals/eval/run_evals.py)              │ Engine.advise
                                       ▼
                                Clef-Flash, fp16 on MPS
FactValue
Modelflash, ? parameters
Pinned revisionnot recorded
Device and precisionmps, float16
Context window262,144 tokens
HostApple M4 Max, macOS-26.6.2-arm64-arm-64bit
SoftwarePython 3.12.11, ember-advise 0.14.0, torch 2.14.1, transformers 5.18.0, mcp 2.3.0
DeploymentLocal (Apple Silicon, MPS)
Commite79dd60
Run at2026-10-12 23:16 UTC
Report generated2026-10-12 23:44 UTC
Datasetclef-flash.jsonl
Dataset SHA-2564389bc9100e8526427a0ef8a8459fd420b80c8b9f14d57121ca8ce1edaad0353
Dataset unchanged since the runyes
Request errors0