ExpertCache

ExpertCache is an experimental, page-aware Metal runtime and reproducibility harness for running oversized sparse mixture-of-experts checkpoints on Apple Silicon without binding or copying the complete expert tensor into the active Metal working set.

The current artifact targets the official GPT-OSS 120B MXFP4 GGUF and a pinned llama.cpp revision. It is research software, not a production inference runtime and not an AMOS Desktop dependency.

Results at a glance

Both hardware results use the same pinned 63,387,346,208-byte GPT-OSS 120B MXFP4 checkpoint. The 16 GiB path does not substitute a smaller model, remote inference, or a more aggressively quantized derivative.

Physical host	What completed	Functional result	Measured speed	Evidence boundary
64 GiB M1 Max	Full checkpoint, direct selected-expert views, grouped dispatch, and routed-union prefetch	Targeted 3/3 coding gate and a bit-exact 1,128-token trajectory in the cited arms	Real-prompt prefill 5.75 → 9.80 tok/s (+70%); final decode near 3 tok/s	One-host decision-grade A/B, not a counterbalanced production estimate
16 GiB M1 Pro	Full checkpoint, protected natural completion, and a later seven-scenario 8K qualification	14/16 functional points; the comparable local and hosted 120B diagnostics each scored 12/16	Clean completion: 0.75 prompt / 0.72 decode tok/s; warm qualification: 8,249 s total	One clean-boot feasibility session plus one warm engineering qualification; not speed parity or a multi-host reproduction

The 16 GiB result is evidence of functional capability retention, not a claim of strict quality superiority. The 14/16 functional score includes an audited evaluator correction for Sign-ups versus signup; the original raw score remains 11/16. The local and hosted controls were single diagnostics, not a counterbalanced comparison, and their response bytes were not identical. The practical conclusion is narrower and still significant: the oversized checkpoint remained useful when executed through the 16 GiB path, but it was far too slow for interactive production use.

64 GiB performance result

On one 64 GiB Apple M1 Max system, the pinned artifact:

• executed the complete 63.4 GB GPT-OSS 120B MXFP4 checkpoint while avoiding catastrophic swap;
• preserved a targeted 3/3 coding gate that was held out during the original run and is now disclosed with the artifact;
• exposed only selected expert ranges to Metal through page-aligned direct host-memory views;
• produced a bit-exact 1,128-token trajectory with grouped dispatch and async routed-union prefetch; and
• improved measured real-prompt prefill from 5.75 to 9.80 tokens/second (+70%) in the cited decision-grade A/B.

These are bounded results from one machine and one model. They do not show frontier parity, general production readiness, universal model quality, or a counterbalanced publication-grade performance result. Decode remained near three tokens/second in the final gate. The full qualification suite has been rerun on the final runtime at low and medium reasoning effort, but those single warm-state runs are diagnostic evidence rather than a publication-grade quality or performance estimate; see the governed qualification boundary.

Read the direct-view result, latency ceilings and prefetch result, and the failed mapped-page design before quoting the work.

16 GiB capability result

On one physical 16 GiB M1 Pro, an experimental explicit-placement configuration:

• executed the complete pinned 63.4 GB checkpoint and stopped naturally after the requested 50-token completion;
• kept peak swap to 256 KiB during the clean-session feasibility sequence;
• later completed the seven-scenario, 16-point qualification contract at an 8,192-token context;
• earned 14/16 functional points, with one genuine miss on the parked-approval outcome; and
• stayed below 4.81 MiB peak swap with at least 38 percent free memory during that later warm qualification.

The qualification took 8,249 seconds, including a 5,339.7-second distractor-heavy long-context scenario. This is not a stock-runtime result, a power-normalized speed comparison, or a reproducibility claim. Automatic fit remained a protected no-go; the successful path disabled it, explicitly selected all GPU layers, and disabled whole-file mmap prefetch. A second clean-boot repeat remains required before calling the result reproducible.

Read the complete 16 GiB result and qualification table and the protected runbook before repeating or quoting the experiment.

Repository map

• runtime/ — pinned runtime manifest, llama.cpp patch, and native Metal probes.
• harness/ — privacy-safe route capture, GGUF layout inspection, and mapped page replay.
• src/ — architecture-neutral trace simulation and route-window analysis.
• scripts/ — runtime preparation, benchmarks, ablations, and reports.
• test/ — deterministic unit and replay tests.
• evidence/ — committed, machine-readable decision-grade result bundles.
• docs/REPRODUCIBILITY.md — end-to-end experimental workflow.
• docs/GOVERNED_MODEL_HARNESS.md — raw-model, evaluator-audit, and product-harness result boundaries.
• docs/HOSTED_CONTROL.md — matched Bedrock GPT-OSS qualification control.
• docs/SECOND_CHECKPOINT_20B_RESULTS.md — partial same-family portability result and unresolved stock/direct boundary.
• paper/ — optional manuscript assets, claim ledger, registered protocol, and result macros retained for rigor and provenance.
• artifact/ — machine-readable experiment matrix and publication manifest.
• ROADMAP.md — research gates, production architecture, and retained negative results.

Reproducibility and ongoing experiments

The repository is the primary way this work is shared. Development is not blocked on formal paper submission. The optional manuscript assets and registered experiment matrix remain in the tree because they keep claims tied to evidence, preserve negative results, and make later replication easier. Existing measurements remain labeled decision-grade until counterbalanced clean-host blocks, the full quality gate, telemetry gate, and second-checkpoint portability study close.

npm run paper:check
npm run artifact:validate
npm run experiment:publication
npm run experiment:second-checkpoint

The experiment-matrix command prints the registered blocks without starting inference. The second-checkpoint command prints the pinned official GPT-OSS 20B MXFP4 artifact, download command, four correctness arms, and explicit claim boundary without downloading or running anything. A live block requires explicit execution and clean-boot confirmations; see paper/EXPERIMENT_PROTOCOL.md. A 32 GiB Mac is an optional physical-hardware replication, not something this 64 GiB host can legitimately emulate for a hardware claim.

The 2026-07-31 20B gate completed all four arms. Direct, grouped, and prefetch-6 are canonical-response equivalent, but stock differs from the custom path; the gate therefore remains active. See the 20B result boundary.

For a physical 16 GiB Apple Silicon host, experiment:low-memory provides a separate, protected one-token feasibility gate. It aborts on bounded swap, memory-pressure, or wall-time limits and must not be used as a 32 GiB claim. Follow the physical 16 GiB M1 Pro runbook without skipping directly to quality or throughput testing.

The 2026-07-31 single-host experimental arm passed protected one-token, eight-token, and natural 50-token completion gates after disabling automatic fit and selecting all GPU layers explicitly. Peak session swap was 256 KiB. The later warm 8K qualification earned 14/16 functional points while taking 8,249 seconds. This is a combined experimental-configuration result, not a stock-runtime, speed-parity, or multi-boot reproducibility claim; see the 16 GB results and evidence boundary.

Quick validation

Requirements: Node.js 22 or newer and Python 3.11. The deterministic Node tests have no package dependencies. The mapped-page replay tests require NumPy.

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r harness/requirements-replay.txt
npm test
npm run check

Preparing the live runtime clones the pinned llama.cpp revision, verifies the patch digest, applies it, and builds the Metal binaries in the requested checkout (the publication workflow uses .cache/runtime/llama.cpp):

npm run runtime:prepare

Model weights are not redistributed. Obtain the exact pinned model artifact listed in runtime/runtime-manifest.json, verify its size and revision, and follow docs/REPRODUCIBILITY.md.

Relationship to AMOS

This work originated as an isolated experiment in amos-labs/amos-agent. The portable runtime, harness, evidence, and tests now live here. AMOS keeps its product routing, local-model qualification, company data, policy, approvals, receipts, and deployment controls in the product repository. A future AMOS integration should consume a versioned ExpertCache runtime as an optional backend; the research repository must not depend on AMOS Desktop.

Historical machine-readable schemas retain their amos.* namespace so the published evidence remains compatible with the original artifact.

License and attribution

ExpertCache-authored code is Apache-2.0. The manuscript source and compiled paper are separately licensed CC BY 4.0; see paper/LICENSE.md. The runtime patch applies to the MIT-licensed llama.cpp; its license is preserved in third_party/llama.cpp-LICENSE.txt. See NOTICE and PROVENANCE.md. The AI-assisted research and writing boundary is documented in AI_USAGE.md; AI systems are acknowledged as tools, not listed as authors.

Citation

Use CITATION.cff to cite the software artifact. Optional manuscript and arXiv metadata remain under paper/; no paper submission is required to use, reproduce, challenge, or contribute to the repository.