ember

3System under test

Two runners exercise the same server. The benchmark runner posts every item to the model server's POST /v1/systemone, the endpoint the MCP tool forwards to, so it measures the model and server end to end. The agent eval runner drives a real coding agent through opencode run --pure, so it also measures whether the agent calls ember at the right moments, asks well, and acts on the answer.

ember call pathBoxes for the agent, ember-mcp, the model server, and Clef-Flash, joined left to right; the agent eval runner drives the agent and the benchmark runner calls the model server directly.tools/call adviseHTTPEngine.adviseopencode run --pureHTTP, every itemCoding agentopencode, Claude Code,Codexember-mcpMCP stdio: advise tool,ember://guideModel serverFastAPI: POST/v1/systemoneClef-FlashQwen3.5 + joint schemahead, fp16 on MPSAgent eval runnerscripts/run_agent_evals.pyBenchmark runnerscripts/run_evals.py
Figure 1 The call path of an agent's advise call. The dashed routes are the two runners: the agent eval drives a coding agent, and the benchmark posts straight to the model server.
Text version
agent eval runner (scripts/run_agent_evals.py)
   │ opencode run --pure, one sandbox per session
   ▼
coding agent ──tools/call advise──► ember-mcp (stdio, mcp_server.py)
                                       │ HTTP POST /v1/systemone
                                       ▼
model benchmark runner ──HTTP──► model server (server.py)
(scripts/run_evals.py)                 │ Engine.advise
                                       ▼
                                Clef-Flash, fp16 on MPS
FactValue
ModelCloudflare/clef-flash, 9B parameters
Pinned revision17f0b0ad64efb65d273590632833508766b2aae6
Device and precisionmps, float16
Context window262,144 tokens
HostApple M4 Max, macOS-26.6.2-arm64-arm-64bit
SoftwarePython 3.12.11, gut 0.1.0, torch 2.14.1, transformers 5.18.0, mcp 2.3.0
Serverhttp://127.0.0.1:8765
Commitcadc1ec
Run at2026-10-03 20:38 UTC
Report generated2026-10-03 22:55 UTC
Datasetclef-flash.jsonl
Dataset SHA-2566a457f6badb7dd53b50a9ce04857d4d01e885d4718f3eb6db9db58658d9f923c
Dataset unchanged since the runyes
Request errors0