Not started. This repo is scaffolded — CI, test harness and the results pipeline are wired, but no code has been written and there are no measurements. The page below is the plan, not a report. It will be rewritten around the result when there is one.
One model, many configs, real measured TTFT/TPOT/goodput under load, with a written tradeoff analysis and a config recommendation per SLO.
The one question this repo will answer:
Which serving config meets a stated SLO at the lowest cost per million tokens?
Scope constraint. Ship TensorRT-LLM config directory labelled UNEXECUTED with an explicit statement it was never run.
ADR: goodput not throughput, plus an admission-control postmortem where p99 exploded from queueing rather than compute.
Constraints this repo inherits from the portfolio:
- No GPU. 24-core CPU, 32 GB RAM,
torch.cuda.is_available()is False. Anything specced for an accelerator is re-scoped to a CPU-measurable question or shipped with the untested path explicitly labelled. - No live model calls in CI. Recorded fixtures, so the suite is free and deterministic.
- Every committed number is generated by
scripts/generate_results.py, carrying its date, hardware, model revision, seed, reproduce command and raw artifact path. Nothing is typed by hand. - Committed results must be machine-independent — ratios, orderings and invariants. Absolute timings go to a gitignored raw file, because CI regenerates results and fails on any diff.
- 3A benchmark harness with written tradeoff analysis (DO-6)
- 3A TTFT/TPOT/throughput/goodput
- 3A p50/p95/p99 under load, queueing delay vs compute time
- 3A vLLM internals: PagedAttention, continuous batching, prefix caching, chunked prefill, CUDA graphs
- 3A SGLang, TensorRT-LLM, TGI, llama.cpp and when each wins
- 3A scheduling: max batch size, preemption, priority, admission control
- 3A load testing methodology
| scaffold, CI, test harness | done |
| implementation | not started |
| measurements | none |
MIT