ember

Compatibility

This document records what ember has been tested against, and the known limits. Update it when testing a new model revision, dependency series, or device.

Last verified: 2026-10-03 on macOS on Apple Silicon (M4 Max, 128 GB).

Tested models

Model Registry key Source Parameters On disk Modality License Revision Verified
Clef-Flash flash Cloudflare/clef-flash (Hugging Face Hub) 9B about 18 GiB Text and vision Apache-2.0 17f0b0ad... Yes, locally
Clef full Cloudflare/clef (Hugging Face Hub) 27B about 55 GiB Text and vision Apache-2.0 2f3de3dd... Not locally

Registry keys live in ember/models.py (REGISTRY). Select a model with EMBER_MODEL, or pull the weights with ember model pull. flash is the default registry entry. ember does not verify downloaded weights against a hash (constitution Article V, “Model Loading”) — it supports any model that fits its loader contract, not a hand-maintained allowlist of individually pinned, hash-verified weights.

A hosted deployment (e.g. Outerbounds) that supplies the model’s location directly at start time via EMBER_MODEL_S3_URI doesn’t use this registry at all — see README.md “Hosted deployment: a model location supplied at start time.” AWS credentials (EMBER_S3_ACCESS_KEY_ID/_SECRET_ACCESS_KEY, see README.md) for that path are optional — boto3’s default credential chain (an attached IAM role, e.g. on Outerbounds; env vars; ~/.aws/credentials) applies when they are unset.

Warning: full (27B) is not verified on this hardware. Expect a longer load and much larger memory use. The 8 GiB self-hosted CI runners cannot hold it, and the ~19 GB fp16 flash model does not fit the hosted runners either, so model-backed tests run locally only.

Hardware requirements

ember runs decision models (currently Cloudflare’s Clef) on Apple Silicon. The model has to fit in unified memory, and that is the binding constraint — not CPU speed.

Model Parameters On disk (fp16) Weight in memory Unified memory Verified
flash 9B about 18 GiB float16 32 GB or more 128 GB (M4 Max)
full 27B about 55 GiB float16 96 GB or more not yet

Running in the cloud

ember is Apple-Silicon-first locally and supports NVIDIA GPUs for hosted deployment (constitution Article VI, “Apple Silicon and CUDA”). Cloud means one of:

Two cloud caveats:

Tested stack

Component Version Notes
macOS Apple Silicon (arm64) Local use (MPS); Intel is out of scope
Linux x86_64 / aarch64 Hosted deployment (CUDA); torch’s default PyPI wheel already bundles CUDA support here, no special index needed
Python 3.12 managed by uv (.python-version)
torch 2.14.1 MPS/CUDA backend, float16; float32 on CPU
torchvision 0.29.1 required by the Clef image processor
transformers 5.18.0 model and processor
mcp 2.3.0 MCP stdio server
fastapi 0.142.2 model server HTTP API
uvicorn 0.54.0 ASGI server

torch and torchvision are pinned to the tested minor series because MPS behavior is version-sensitive. Widening the range is a constitution-governed change. CUDA support has not yet been exercised against real NVIDIA hardware — the code path is implemented and unit tested (ember/serving/runtime.py::pick_device/pick_dtype), but no model-backed run on a CUDA host has been recorded here yet. Update this note once one has.

Capability matrix

Capability MPS (default) CUDA (EMBER_DEVICE=cuda) CPU (EMBER_DEVICE=cpu)
Text advice Verified, float16 Not yet verified, float16 Verified, float32
Image inputs Verified Expected, not yet verified Expected, slower
Video frame inputs Verified Expected, not yet verified Expected, slower

Observed on the tested hardware: model load in about 5 seconds, and a warm request in about 0.9 to 1.3 seconds for 220 to 360 input tokens. Counting the full request before inference adds about 0.4 ms to a warm 153-token request (about 0.65 seconds end to end), under 0.1%.

Since ember 0.12.0 every POST /v1/systemone response splits usage.input_tokens into state_tokens, fixed_tokens, and media_tokens (the same measurement the request-size check makes). The benchmark harness (ember eval run) requires the split and refuses to score a server that does not return it, so a hosted deployment must run 0.12.0 or later before it is benchmarked. Older clients ignore the extra fields.

Known issues

See also: README.md for install steps, and RESPONSIBLE_USE.md for what the numbers mean.