One table per dataset version, newest first; results on different versions are not compared. One row per model, deployment, and engine settings. The headline covers the five coding recipes and appears once every one has been scored; each recipe cell links to the run that produced it, and a dash marks a recipe not yet scored on this version. Vision is reported separately as capability.
Dataset e0bbb468e213 · 538 coding items (current)
| Model | Deployment | Device | Headline | Items correct | Non-easy | Intent and readiness | Failure triage | Change risk | Routing and ownership | Effort and approach | Vision: dominant colour, yes/no | Vision: dominant colour, choice | Vision: alert level, score | Vision: colour change across frames | Tag assignment | Note placement | Capability | Dangerous recall | Long context | Latency p50 | ember version | Run at |
|---|
| flash | Local (Apple Silicon, MPS) | mps | 81% [79, 83] | 64% | 69% | 91% | 97% | 70% | 97% | 58% | 100% | 100% | 100% | 100% | 81% | 57% | 100% (32 q) | — | not measured | 917 ms | 0.14.0 | 2026-10-12 23:16 UTC |
| Cloudflare/clef-flash (9B) | Remote hosted (GPU, CUDA) | cuda | — | — | 0% | — | — | — | — | — | — | — | — | — | 81% | 57% | — | — | not measured | 346 ms | 0.15.0 | 2026-10-13 02:14 UTC |
Dataset 4049ac55a1fa
| Model | Deployment | Device | Headline | Items correct | Non-easy | Intent and readiness | Failure triage | Change risk | Routing and ownership | Effort and approach | Vision: dominant colour, yes/no | Vision: dominant colour, choice | Vision: alert level, score | Vision: colour change across frames | Tag assignment | Capability | Dangerous recall | Long context | Latency p50 | ember version | Run at |
|---|
| flash | Local (Apple Silicon, MPS) | mps | — | — | 69% | 91% | 97% | 70% | 97% | 58% | 100% | 100% | 100% | 100% | 81% | 100% (32 q) | — | not measured | 915 ms | 0.14.0 | 2026-10-12 22:06 UTC |
| Cloudflare/clef-flash (9B) | Remote hosted (GPU, CUDA) | cuda | — | — | 79% | — | — | — | — | — | — | — | — | — | 80% | — | — | not measured | 1687 ms | server 0.12.0, client 0.14.0 | 2026-10-12 21:58 UTC |
Dataset 1e1bb88d47f0
| Model | Deployment | Device | Headline | Items correct | Non-easy | Intent and readiness | Failure triage | Change risk | Routing and ownership | Effort and approach | Vision: dominant colour, yes/no | Vision: dominant colour, choice | Vision: alert level, score | Vision: colour change across frames | Capability | Dangerous recall | Long context | Latency p50 | ember version | Run at |
|---|
| flash | Local (MPS) | mps | — | — | 40% | 91% | 97% | 70% | 97% | 58% | 100% | 100% | 100% | 100% | 100% (32 q) | — | not measured | 909 ms | 0.12.0 | 2026-10-12 02:18 UTC |
| Cloudflare/clef-flash (9B) | Remote hosted — AWS g5.2xlarge, NVIDIA A10G, 24 GB VRAM (us-west-2) | cuda | — | — | — | 91% | 97% | 70% | 97% | 58% | 100% | 100% | 100% | 100% | 100% (32 q) | — | not measured | 252 ms | 0.12.0 | 2026-10-12 01:27 UTC |
| Cloudflare/clef-flash (9B) | Remote hosted (GPU, CUDA) | cuda | — | — | — | 91% | 97% | 70% | 97% | 58% | 100% | 100% | 100% | 100% | 100% (32 q) | — | not measured | 504 ms | 0.10.2 | 2026-10-10 19:27 UTC |
Dataset 913e1b65cf5a
| Model | Deployment | Device | Headline | Items correct | Non-easy | Intent and readiness | Failure triage | Change risk | Routing and ownership | Effort and approach | Vision: dominant colour, yes/no | Vision: dominant colour, choice | Vision: alert level, score | Vision: colour change across frames | Capability | Dangerous recall | Long context | Latency p50 | ember version | Run at |
|---|
| Cloudflare/clef-flash (9B) | Local (Apple Silicon, MPS) | mps | — | — | — | 91% | 97% | 72% | 100% | 61% | 100% | 100% | 100% | 100% | 100% (32 q) | — | not measured | 922 ms | not recorded | 2026-10-03 20:38 UTC |
