ember

3System under test

Two runners exercise the same server. The benchmark runner posts every item to the model server's POST /v1/systemone, the endpoint the MCP tool forwards to, so it measures the model and server end to end. The agent eval runner drives a real coding agent through opencode run --pure, so it also measures whether the agent calls ember at the right moments, asks well, and acts on the answer.

ember call pathBoxes for the agent, ember-mcp, the model server, and Clef-Flash, joined left to right; the agent eval runner drives the agent and the benchmark runner calls the model server directly.tools/call adviseHTTPEngine.adviseopencode run --pureHTTP, every itemCoding agentopencode, Claude Code,Codexember-mcpMCP stdio: advise tool,ember://guideModel serverFastAPI: POST/v1/systemoneClef-FlashQwen3.5 + joint schemahead, fp16 on MPSAgent eval runnerevals/eval/run_agent_evals.pyBenchmark runnerevals/eval/run_evals.py
Figure 1 The call path of an agent's advise call. The dashed routes are the two runners: the agent eval drives a coding agent, and the benchmark posts straight to the model server.
Text version
agent eval runner (evals/eval/run_agent_evals.py)
   │ opencode run --pure, one sandbox per session
   ▼
coding agent ──tools/call advise──► ember-mcp (stdio, ember/mcp/mcp_server.py)
                                       │ HTTP POST /v1/systemone
                                       ▼
model benchmark runner ──HTTP──► model server (ember/serving/server.py)
(evals/eval/run_evals.py)              │ Engine.advise
                                       ▼
                                Clef-Flash, fp16 on MPS
FactValue
ModelCloudflare/clef-flash, 9B parameters
Pinned revision17f0b0ad64efb65d273590632833508766b2aae6
Device and precisioncuda, float16
Context window262,144 tokens
HostApple M5 Pro, macOS-26.5.2-arm64-arm-64bit
SoftwarePython 3.12.13, ember-advise 0.10.2, torch 2.14.1, transformers 5.18.0, mcp 2.3.0
DeploymentRemote hosted (GPU, CUDA)
Commit6ff5120
Run at2026-10-10 19:27 UTC
Report generated2026-10-11 02:20 UTC
Datasetclef-flash.jsonl
Dataset SHA-256fbb98c3516153987634b11d1ac4c9307e56c43a7131e7de598de557e1c48a050
Dataset unchanged since the runyes
Request errors0