long-context-serving: Kimi long-context FP8 benchmark release

This repo reproduces the Kimi-Linear long-context FP8 benchmark sweep on one 8-GPU AMD Instinct MI355X node. The public path is intentionally one command: build the image if needed, enter the runtime, run the sweep, and write the report and charts back to the host.

The Docker image contains the vLLM v0.25.1 + upstream AITER (v0.1.19.post2) stack, this repo's benchmark/report code, and the locked prompt substrate corpus. Model weights stay in your Hugging Face cache and run artifacts stay on the host.

Hugging Face Token

The benchmark downloads a gated model from Hugging Face. Provide a token in either of two ways:

• Export HF_TOKEN in your shell.
• Write the token to ~/hf_token (one line). make run reads this file when HF_TOKEN is not set; override the path with HF_TOKEN_FILE=/path/to/token.

The token is passed into the container with value-less docker run -e HF_TOKEN, so it is not written into the image or placed directly on the command line.

Quick Start

# Full 1Ki..64Mi sweep:
make run

# Quick ranges:
make run FROM=1Ki TO=16Ki
make run FROM=4Mi TO=8Mi REPEATS=3
make run FROM=32Mi TO=64Mi STAMP=20260515T035829Z

FROM/TO are inclusive context-length bounds expanded to powers of two (Ki/Mi suffixes or raw token counts). REPEATS sets measured runs per point. STAMP resumes an existing campaign timestamp in place.

Campaign outputs land on the host under:

experiments/hf_long_context/runs/<run_id>_<UTC_stamp>_campaign/

Keep report.md in that directory: it is the primary campaign report. The directory also contains consolidated_results.jsonl and charts generated by the report builder.

What Gets Baked

• Stock vllm/vllm-openai-rocm:v0.25.1 base with AITER reinstalled from the pinned upstream release tag v0.1.19.post2 (native long-context MLA kernels). See docker/Dockerfile.
• Repo code under /workspace/long-context-serving inside the image.
• The locked source-code prompt substrate from data/metadata/substrate_repos_manifest.json.

Common Overrides

IMAGE_TAG=my-image:tag make run
RUNS_DIR=/data/long-context-runs make run
HF_CACHE=/data/hf-cache HF_TOKEN="$HF_TOKEN" make run

Build the image without running a sweep:

make image

Detailed Docs

• experiments/hf_long_context/README.md explains hardware assumptions, runtime knobs, artifacts, diagnostics, and report regeneration.
• docker/README.md inventories the image build and Docker build arguments.