feat(projection): add InferaSim, a workload-driven serving simulator and projector- #137 - #138
Open
araina-amd wants to merge 89 commits into
Open
feat(projection): add InferaSim, a workload-driven serving simulator and projector- #137#138araina-amd wants to merge 89 commits into
araina-amd wants to merge 89 commits into
Conversation
Add infera/projection: analytical + GPU-calibrated inference/serving projection (TTFT / ITL / throughput / KV-cache) and the LLM tuning agent for recipe search. Scope is the serving path. Performance is anchored by an in-process vLLM harness that supplies sub-scale depth override, cudagraph capture buckets, and pure decode-step timing; the anchor JSON is engine-neutral so other engine harvesters (SGLang / ATOM) can be added later. - inferasim CLI (inference suite + anchor-harvest shim) - inferasim-tune CLI (DSPy planner / RLM; projector-scored, no GPU by default) - confidence ladder for regime-aware anchor GPU selection - pyproject: [projection] / [projection-tuning] extras, console scripts, package-data for the model / preset YAMLs
- add a cli/main.py shim so the tuning-agent evaluator can spawn the projector via 'projection inference ...' -> infera.projection.cli - evaluator: put the Infera repo root on the subprocess PYTHONPATH so the shim can import infera.projection.cli - workload: resolve the model / preset configs from the packaged configs tree so tuning runs with zero env setup - ship example tuning assets (target_cluster_*.yaml + MI355X workload YAMLs)
Add a --prefix-cache-hit-rate knob (alias --prefix-hit-fraction), in [0,1), that models automatic prefix caching / shared-prefix reuse: the cached prefix (rate * input_len tokens) skips prefill compute and only the non-cached suffix is run through the network, still attending over the full context. TTFT and the prefill share of continuous-batching pollution therefore scale with (1 - rate); decode and KV sizing are unchanged. - request config: prefix_cache_hit_rate field + resolved_* (clamped to <1 so at least one token is always prefilled) - prefill_latency_ms: discount effective prefill tokens (measured + analytical + chunked paths); continuous-batching pollution sizes chunks off the suffix - CLI flag + launcher arg mapping + feature summary line - settable per workload via a YAML inference: block (tuning agent inherits it) rate=0 (default) is a cold cache and reproduces prior output exactly.
…l in the DES Extend the discrete-event simulator to a fleet of N engine replicas behind a router sharing a prefix pool, so prefix-cache hits are DERIVED from the routing policy + per-instance locality instead of the static --prefix-cache-hit-rate. Each request draws one of P shared prefixes (a system-prompt / template of L tokens); each instance keeps its own resident-prefix LRU; a request routed to an instance already holding its prefix is a hit and only its suffix is prefilled (seeded into the scheduler's num_computed, so the existing token-step packer and cost kernel model it exactly). - routing policies: prefix_aware (KV-aware: co-locate a prefix's requests on one home instance -> misses ~ P, independent of N), round_robin / random (scatter -> misses ~ P*N) - instances are independent except through the router, so each instance's sub-stream is simulated and the raw latency samples are pooled into one fleet-level DESResult (throughput sums; makespan is the slowest instance) - reports fleet size, routing, hit rate, avg cached tokens, per-instance hit spread; surfaces the cache-locality vs load-balance trade-off - CLI: --des-instances / --des-routing / --des-num-prefixes / --des-prefix-len / --des-prefix-zipf / --des-cache-slots; single-engine path unchanged when instances=1 and no prefix pool
In benchmark-calibrated mode the analytical TTFT discount was discontinuous at the first prefix-cache hit: hit=0 used the measured full-prefill anchor while hit>0 switched to the per-token rate path, and those two differ by the batch factor (a single-batch anchor holds full-prefill flat). Result: TTFT cliffed at the first hit instead of scaling smoothly with the hit rate. Discount the SAME chosen baseline proportionally to the non-cached suffix instead of switching cost methods, so benchmark-mode TTFT scales continuously with the hit rate. Analytical path unchanged (models suffix attention over full context); DES path was already correct (per-token measured prefill throughout).
Rename the inference-projection + tuning-agent package to projection_core and update its dotted imports, the tuning-agent path literals (model resolver, shim path, PYTHONPATH marker), the entry point, the configs package-data glob, and README paths. Pure rename; no behaviour change.
…a.projection.* Lay out the engine as first-class infera.projection.* packages (core, agents, configs, modules, platforms). The tuning-agent spawn shim lives in _tuning_shim/. inferasim / inferasim-tune are the console entry points (with infera-projection / infera-tuning kept as back-compat aliases). Path/depth logic in the tuning agent (configs/models resolution, shim path, PYTHONPATH repo-root discovery) and the package-data globs are wired for this layout. No behaviour change.
Standardise runtime log prefixes to [inferasim:...] and configuration env-var names to INFERASIM_* across the projection engine, DES, tuning agent, CLI help, example configs, and README. An import-time alias shim keeps legacy env-var names resolvable so existing environments keep working.
… DES Replace the prefix-id LRU with a content-addressed, paged KV block cache per engine instance (as real serving engines like vLLM / SGLang do). A prompt is an ordered sequence of block-hash ids; a hit is the longest contiguous leading run of resident blocks. The cache is finite (capacity in blocks) and LRU-evicts under pressure, so the prefix-cache hit rate is emergent from workload content + capacity + routing rather than a static number. Matched blocks seed num_computed so only the uncached suffix prefills (decode / KV unchanged). Add overlap-scored KV-aware routing (--des-routing kv): route each request to the instance holding the most of its leading blocks (ties -> least loaded), via a global prefix index. Surfaces the real cache-locality vs load-balance trade-off (hot prefixes overconcentrate, lowering fleet throughput). Add Mooncake trace replay (--des-mooncake-trace): JSONL/JSON with timestamp, input_length, output_length, hash_ids; the hash_ids drive real content-addressed matching and enable the DES without --request-rate. Block sequences otherwise synthesised from the shared-prefix pool, blockified at --des-block-size. New knobs: --des-block-size, --des-kv-blocks (per-instance block capacity; --des-cache-slots kept as legacy alias), --des-mooncake-trace, routing 'kv'. Fleet report now shows block size, capacity, evictions, and block-reuse rate.
…t models
Add three inference operation models to the performance projector:
- sampling / logits post-processing (module_profilers/sampling.py): a
memory-bound reduction over the vocab for greedy / temperature / top-k /
top-p, folded into every step; knobs sampling_enabled / sampling_top_k /
sampling_top_p / sampling_temperature (+ CLI). Enabled by default at the
greedy (cheapest) path; the top-k / top-p / temperature knobs are no-op at
their defaults and only add streaming passes when set.
- runtime activation quantization / cast (module_profilers/quantization.py):
fp8 / mxfp4 activation cast cost per dense and MoE layer, auto-detected from
the weight dtype; knob act_quant_dtype (+ CLI). Auto-enabled for fp8 / mxfp4
serving; a bf16 path resolves to no cast (there is nothing to charge).
- small-tensor kernel-launch floor: a depth-scaled lower bound on the
pure-simulate decode / mixed step so small-batch decode does not underflow
launch dispatch; disabled under CUDA-graph capture; knobs
kernel_launch_latency_us / kernels_per_layer (+ CLI). Off by default
(kernel_launch_latency_us=0) until calibrated.
The default projection stays equal to the pure kernel-compute roofline:
sampling runs at greedy, activation-quant charges only for low-precision
serving, and the launch floor stays off until calibrated.
Co-authored-by: Cursor <cursoragent@cursor.com>
…I examples - name the experiment-config class InferaSimConfig / InferaSimParser and the loaders load_config / convert_config_to_inference_config; the global accessor get_config; log tags InferaSimMaster; tuning-agent config_root paths. - drop the legacy env-var alias shim; INFERASIM_* is the only prefix. - correct module-path examples to infera.projection.agents.tuning_agent and point docstrings at the inferasim projector CLI. No behaviour change; projection output is unchanged. Co-authored-by: Cursor <cursoragent@cursor.com>
Cap the attention span and KV-cache footprint at a sliding window, so models
that use local attention (gpt-oss, Mistral, Gemma-2/3, Qwen2.5) are projected
against the window rather than the full context.
- model config: sink_sliding_window / sink_window_even_layers_only, picked up
straight from the model YAML.
- request config: sliding_window override (0 forces full attention) and
sliding_window_layer_fraction for models that interleave local and global
layers; resolvers blend the windowed and full layers into one representative
KV length (effective_attn_kv).
- performance: attention (prefill and decode) reads the blended window-capped
KV length; the sparse-attention path keeps the true context.
- kv_cache: per-sequence footprint caps at the window, blended by the
windowed-layer fraction.
- CLI: --sliding-window / --sliding-window-layer-fraction, plus a feature
summary line.
Full attention is unchanged: a window at or above the context, or no window at
all, reproduces the previous projection exactly.
Co-authored-by: Cursor <cursoragent@cursor.com>
Each ladder rung costs a real GPU benchmark run. A flat per-GPU pair at rung g
certifies targets up to 2*g, so rungs 1/2/4 already certify an 8-GPU target --
climbing past 4 buys little for the calibration time it spends.
- LADDER_MAX_GPUS = 4, resolved by ladder_max_gpus() from an explicit
max_gpus argument, INFERASIM_LADDER_MAX_GPUS, then the default.
- confidence_ladder takes the cap, returns next_gpus=None plus a new capped
flag once the top rung sits at the cap, so callers stop asking for a rung
they cannot run; larger targets are reported as extrapolated from the top
rung instead of silently certified.
- climb_anchor_ladder defaults its ceiling to the cap.
- launcher advisory points at the cap knob instead of printing a null rung.
Targets that a 4-GPU rung can certify keep HIGH confidence and are unaffected.
Co-authored-by: Cursor <cursoragent@cursor.com>
…ctives The intra-node collective floor was keyed on a message-size crossover: below it the training-scale fixed RCCL overhead was stripped in favour of the measured latency floor, at or above it the constant snapped straight back. That made a marginally larger message cost sharply more and then stay flat, because the constant -- not bandwidth -- was setting the price. On a TP=8 model the decode all-reduce jumped abruptly between two adjacent batch sizes. A graph-captured intra-node serving collective does not pay the training-scale constant at any size, so strip it throughout and price the collective as max(measured floor, bandwidth transfer). The floor governs small decode messages, the bandwidth term takes over smoothly as the message grows, and prefill-scale messages are bandwidth-dominated either way. Multi-node keeps the base model, whose NIC overheads are real. Also add a DeepSeek-R1 serving workload (MLA + MoE, TP=8) matching the measured sweep in bench/pd_mori_1p1d/results, for fidelity comparisons. Configurations whose decode messages already sat below the crossover are unchanged. Co-authored-by: Cursor <cursoragent@cursor.com>
TTFT was priced at a single prompt's prefill with no contention, but one engine prefills every concurrent request. A serving benchmark runs closed loop (--max-concurrency N --request-rate inf), so all N stay outstanding, the scheduler sweeps them FIFO within its token budget, and the average request sits half way down that sweep. The old model was concurrency-independent: DeepSeek-R1 reported the same TTFT at the narrowest and the widest concurrency alike. Against the measured gpt-oss-120b vLLM run (TP=8, MI355X) the TTFT error drops substantially; the residual is uncalibrated roofline optimism, which an anchor closes. The measured ratio of tail to median TTFT matches a sweep of several scheduling groups, which is what this models. Disaggregated serving gets the same treatment on its prefill pool, spread over the prefill replicas. Co-authored-by: Cursor <cursoragent@cursor.com>
Three defects made the inference memory report unusable for capacity planning, which is what drives sustainable concurrency and every memory-feasibility decision the tuning agent makes. * Weights were not tensor-sharded. `estimated_num_params(rank=...)` counts a rank's pipeline and expert share but not its TP slice, so every rank was charged the whole model. Its only caller is the inference memory path, so dividing there is safe and leaves training untouched. * `mxfp4` was missing from the dtype table and silently fell back to bf16, i.e. several times the real width. MXFP4 is block-scaled (4-bit elements plus a shared exponent per block), so its effective width is slightly wider than the nominal one rather than exactly it. * The activation working set multiplied the whole concurrent batch by the full prompt length. A scheduler step is capped by the engine's prefill budget across the batch, not per sequence. gpt-oss-120b at mxfp4/TP=8 was reported at orders more memory per rank than vLLM actually loads. DeepSeek-R1 at fp8/TP=8 now lands where its checkpoint divided across its ranks says it should. Co-authored-by: Cursor <cursoragent@cursor.com>
…antom shared expert Sliding-window attention was modelled but never reachable: the window lived in the primus_turbo kernel module config, and module keys shadow model keys when the two are merged, so a model preset could not declare its own window and every gpt-oss projection ran full attention. The window is a property of the architecture, not of the kernel, so it moves to the gpt-oss presets, which interleave windowed layers with full-attention layers. The kernel toggle (use_sink_attention) stays where it was. gpt-oss also inherited a shared expert from the DeepSeek-V2 base it extends. It has none, and the phantom expert added to per-token MoE FLOPs and weight traffic. At long context the KV cache per sequence halves and sustainable concurrency roughly doubles at every context length. Co-authored-by: Cursor <cursoragent@cursor.com>
A DeepSeek-R1 anchor projected a served TPOT well off the reference. The anchor was not wrong: re-measuring it on a newer vLLM reproduced the original numbers closely. It described a different deployment than the one it was scored against -- one without speculative decoding. Speculation was invisible to the whole pipeline. It was not on the regime axes, so an anchor measured without it sat at zero regime distance from a target using it and was reused silently; and benchmark_vllm.py could not enable it or record it, so no artifact could even state which case it belonged to. Measured attribution on MI355X, same anchor config with only speculation changed: MTP-1 accounts for most of the gap. The remainder is static-batch benchmarking versus continuous-batching serving, independently corroborated by an in-house disaggregated run. Expert parallelism was ruled out by measurement -- its speedup grows with batch while the residual shrinks. Speculation is regime-defining rather than transportable because per-token latency becomes step_cost(batch * (k+1)) / (1 + a + ... + a**k) and the acceptance rate a cannot be derived analytically from a non-speculative anchor. An artifact with no recorded setting is treated as unknown, and unknown is counted as a mismatch only against a speculating target: everything measured before this tracking ran without speculation, so legacy anchors keep working for non-speculative targets while the dangerous direction is closed. Since this adds a regime axis, the anchor store now fingerprints the axis set it indexed under and re-derives a stale index instead of comparing signatures built from two different definitions.
…r knobs TTFT was charged as one uncontended prompt, which ignores that a request also waits behind the prompts queued ahead of it. Replacing that with a full FIFO sweep over the resident set is the other extreme and overshoots badly whenever generation is long enough to leave the prefill stage idle (MiniMax at long OSL ran far above measured). Both limits are wrong for some Hyperloom workload, so price the wait with exact mean-value analysis of the closed network the serving harness actually is: it returns the bare prefill under light load and the sweep under saturation, and interpolates between them. That still left DeepSeek-R1 1P1D far under measured, because its TTFT is barely prefill at all. Two engine flags sit on the first-token path and cost no throughput, which is exactly why production configs turn them up: --stream-interval buffers N tokens per flush, so the client's first token arrives with the flush carrying it, and the decode scheduler polls for new work only every num-continuous-decode-steps x scheduler-recv-interval steps. Model both, with buffering moving time out of TPOT rather than adding to end-to-end latency. Measured error on the 1P1D sweep improves substantially, and now tracks the measurement's near-flat shape in concurrency rather than growing with it. The residual is decode-step optimism: with a measured anchor, leave-one-out TPOT lands close and TTFT within a usable band. Also returns the merged config from the launcher, so a caller can see what the model preset, module defaults and CLI overrides resolved to, and adds the first regression tests for the projector. Co-authored-by: Cursor <cursoragent@cursor.com>
The tuning agent optimized a single scalar, so "max_throughput" always walked to the largest batch that fits in HBM -- the config nobody would deploy for an interactive workload, and the one that looks best when latency is not a constraint. Serving is the constrained problem instead: maximize throughput subject to a latency promise. Add optimization.slo (ttft_ms / tpot_ms / request_latency_ms) and enforce it where the HBM cap is already enforced, so a miss is rejected through the same path and shows up in the trial log. The rejection names the budget and the overshoot because that string is what the planner reads back: it is the signal to trade concurrency away rather than to try another dtype. Unset keys stay unconstrained, and a metric the projection never reported cannot fail a budget. This is worth doing only now that TTFT is modelled: against a projector that reported milliseconds for a measurement in seconds, a TTFT budget would have ranked on noise. Co-authored-by: Cursor <cursoragent@cursor.com>
… runs There was no way to answer "did that change make the projector better", so every fix to it was argued from first principles. This scores it two ways that are deliberately kept apart: fidelity against the serving results Hyperloom has already measured, and a full-workload diff between two checkouts. The measured side carries the engine flags each run was launched under, because a projection that ignores --stream-interval and the decode scheduler's polling granularity is answering a different question than the benchmark asked. It also turns those measured results into a calibration anchor, so a session can calibrate from results it already has instead of booking GPUs, and leave-one-out checks that the anchor transports to concurrencies it was not fitted on. Adds a regression test for the collective discontinuity, which is the case that motivates reading these numbers carefully: the step made aggregate error look better while making the curve useless to search. Co-authored-by: Cursor <cursoragent@cursor.com>
…loors Three defects made the projector systematically optimistic about MoE decode, and each was worst exactly where the measured error was worst. 1. Tokens were divided by tensor-parallel size when sizing the expert GEMM. TP shards the hidden and FFN dimensions; every TP rank holds the whole token axis. Only context parallelism shards tokens, and expert parallelism splits routings and experts by the same factor. The per-expert row count was therefore wrong by ep/tp -- correct at EP==TP, and far too small at the EP=1,TP=8 configurations that DeepSeek-R1 and gpt-oss actually run. It was wrong in the same direction for prefill, where it also understated TTFT. 2. The expert dtype was applied as a hand-picked speedup multiplier instead of as operand bytes. Narrower weights are not a tuning constant: they are fewer bytes to stream and a wider matrix instruction. The GEMM roofline now takes the real width (mxfp4 is block-scaled, so a little wider than nominal) and the multiplier is gone. 3. Restoring a measured anchor to another TP/EP scaled the whole step by the simulator's ratio. A decode step is a fixed per-step cost plus work that shards, and only the second part responds to parallelism; re-scaling the fixed part inflated a less-sharded target severalfold. Decode now moves the anchor by the simulator's difference. Prefill, which has no such floor, keeps the ratio. Against Infera's own 1P1D DeepSeek-R1 concurrency sweep, uncalibrated TPOT and aggregate throughput error both fall sharply, and leave-one-out calibrated TPOT lands close throughout. Against our own static DeepSeek microbenchmarks the decode step's MAPE roughly halves, and anchor reuse across a TP+EP move improves by a large multiple. What remains is a batch-independent per-step cost the analytical path still does not model, which is why the low-batch end stays optimistic. The scheduler test now asserts against the decode step rather than TPOT: fixing (1) makes prefill genuinely more expensive at EP=1, so TPOT carries real mixed-step pollution and is no longer a stand-in for the step the scheduler actually waits on. Co-authored-by: Cursor <cursoragent@cursor.com>
…sing The measured TP ladder (gpt-oss-120b, real weights, TP=1,2,4,8, across batch) over-determines step(tp) = floor + compute(1)/tp and shows a floor that is TP-invariant across the low-batch range with small residual. Four terms were wrong against it. Attention sharded the token axis instead of the head axis, so every rank was charged for streaming the whole Q/K/V/O weight matrices; modelled attention barely moved from TP=1 to TP=8 while the measurement shards strongly. MLA had the same error, with the down-projections correctly replicated. SDPA had no HBM roofline. The tile model prices per-workgroup GEMMs on one CU and scales by wave count, which never bounds the result by the bandwidth of the device the KV cache lives in. At decode that is the whole cost, and the model implied a KV read rate many times the part's HBM bandwidth. The collective floor was an RCCL number applied to a vLLM deployment that runs its own one-shot all-reduce, putting a flat cost into every gpt-oss decode step at TP>1 regardless of batch or TP. The ladder bounds the real cost far below that, because TP=1 carries no collective at all. Per-kernel GPU occupancy was modelled only as host launch latency, disabled under graph capture and applied as a max. Graph replay removes the dispatch, not the execution, and the small latency-bound kernels run alongside the large data-bound ones rather than instead of them. Ladder MAPE over the measured points falls from unusable (the tp1->tp2 delta had the wrong sign at low batch) to a usable band. The occupancy constant is the ladder floor divided by an assumed kernel count, and the collective floor is bounded by measurement rather than measured. Both are marked interim; measure_kernel_floor.py measures the first directly.
…cy on every path Two defects in the previous commit, both caught by scoring the measured serving runs before and after. The SDPA HBM roofline derived KV bytes from head counts, which is right for MHA/GQA but wrong for MLA: DeepSeek caches one compressed latent (kv_lora_rank + rope dims) that every head shares and that TP replicates rather than shards. Charging it per head overstated DeepSeek's KV by a large factor and pushed TPOT from close to measurement to far above it across the concurrency sweep. The attention profiler now supplies the cache's real per-token footprint, including its own dtype, since fp8 KV with bf16 activations is common. Per-kernel occupancy was charged only in ``_decode_step_latency_ms``, which the continuous-batching path does not call -- so every vLLM workload was missing it while the disaggregated path had it. A mixed step runs the same decode kernels with a prefill chunk added, so both steps pay it. With both corrected, the occupancy constant taken from the gpt-oss ladder also holds on DeepSeek without refitting, tracking measurement across the concurrency sweep. That is the cross-model check the constant needed to be more than one model's tuning. TPOT MAPE over the serving runs falls, and minimax bf16 moves from under to near measurement on both TPOT and throughput. Still open: gpt-oss TPOT low and TTFT low everywhere, both of which look like prefill/admission modelling rather than the decode step.
…ed values The MLA regression got in because nothing asserted that a modelled byte stream has to fit through the memory system. These tests are written as inequalities against MI355X's limits, so they survive the model getting more accurate and fail as soon as a term goes unbounded again: SDPA may not imply more KV read bandwidth than the part has, MLA's latent must cost less than a per-head footprint, attention must shard with TP, occupancy must survive graph capture while the launch floor must not, and the intra-node all-reduce floor must stay inside what the ladder allows. Also adds a prefill check against the ladder, which mainly documents why that data cannot be tuned against: at high batch it reads the same at TP=1 and TP=8, so no speedup from eight times the hardware, and those points measure admission rather than FLOPs. It does establish direction -- modelled prefill sits above measurement -- which places the low serving TTFT in the queueing model rather than the prefill cost.
The high-batch TP under-prediction looked like it could come from the MoE GEMM: the roofline gives sharding near-ideal 1/etp relief, and narrow slices might stream less efficiently than wide ones. Measured on MI355X, that is not where it comes from. The grouped GEMM the decode step actually issues holds most of peak bandwidth across etp=1..8 and delivers close to the ideal relief, so the roofline is right here and an efficiency knob tuned in at this spot would only be covering for a term that lives somewhere else. Timing experts one at a time says the opposite, but that is an artifact: a lone [1 x k] x [k x n] call is latency-bound and reads a small fraction of peak. The benchmark now batches over the experts a step touches. Pins the measured relief as a bound so it is not "fixed" the wrong way, and repoints the router hook at its vLLM 0.25 path.
The TP ladder put the decode model's error in a specific place: fitting step = floor + compute/tp on tp=2,4,8 and using tp=1 -- the only rung that runs no collective -- to pin floor+compute left an implied all-reduce far larger than what was being charged for gpt-oss at high batch. Measuring vLLM's custom all-reduce directly confirms it. That is the kernel a decode step runs -- everything under its size threshold goes to it, not to RCCL -- and its cost rises gently with batch on 8 ranks, fitting floor + bytes/bw very closely. Two things were wrong: the composition is additive, not max(floor, bandwidth), which at decode message sizes drops a term of comparable size to the one it keeps; and the achieved bandwidth is far below the node link, with no clean ring or one-shot factor mapping one onto the other, so it has to be measured. Charging comm properly then exposed that the occupancy floor had been absorbing it. The per-kernel occupancy constant came from a floor fit that left comm in, so the same milliseconds were about to be billed twice. Refitting with measured comm removed gives a floor that is flat where it should be, across the low-batch range on the most floor-dominated rung, and a correspondingly smaller per-kernel constant. Ladder MAPE over all points improves, as does the static decode case. Also records, without wiring in, that a grouped GEMM only reaches peak bandwidth once it is large enough to saturate memory: measured from a small fraction of peak on one expert climbing to a high plateau, both models collapsing onto one curve against bytes moved. It belongs to the step, not to each call within it -- a decode step streams multi-GB back-to-back under one graph replay -- and applying it per call costs a large amount of MAPE.
MoE decode cost tracks how many *distinct* experts a step reads, which the model estimated from a uniform router. Real routers are not uniform, so this generalises the coupon-collector to a Zipf popularity law, reducing exactly to the old closed form at skew 0. Existing gpt-oss vLLM sweeps under forced Zipf routing turn out to be a controlled experiment on this -- same model and batch, only the distribution changes -- and they confirm the mechanism: decode time is linear in the distinct count across the swept skew range, at a near-constant cost per expert. The default stays uniform, because the same data says realistic imbalance does not move the count. At a mild skew -- already a heavier max/mean load than a balance-loss router shows -- the distinct count barely differs from uniform's. Recorded because fitting the skew to the ladder is tempting: it has a clean optimum at a much higher skew, and taking it would improve MAPE. But that skew implies an extreme max/mean expert load, so the fit is an unrelated error wearing a plausible-looking knob, and the mid-batch residual is not routing skew. Pinned as a test so it does not get adopted later.
Hyperloom spawns one process per config, so every projected config paid the import cost before doing any work. That cost dwarfed the projection itself -- nearly all of the wall time was overhead -- and most of it was torch. Nothing in an analytical projection needs torch. It arrived through module_profilers.utils, which is entirely real-GPU benchmarking helpers (benchmark_layer, CUDA-graph timing, routing patches), imported at module scope by six profilers that only call it in their measure-on-hardware branch. Moving those imports into the branches that use them cuts the projection stack's startup by roughly an order of magnitude, with the GPU path unchanged. Also adds speed_benchmark.py to quantify the trade Hyperloom is making, with the real-measurement baseline taken from this machine's ladder campaign rather than assumed: projecting a config is orders of magnitude cheaper than measuring one on 8 GPUs, and the config grid it sweeps costs a fraction of a second instead of minutes of GPU time.
A deployment search spends most of its GPU time discovering that candidates are slow or do not fit, and it only needs the ranking to decide what to measure. This projects a whole TP x EP x concurrency space in one process and returns it ranked, so real GPU time goes to the finalists. Measured on gpt-oss-120b, the whole space projects in seconds against the GPU time it would take to measure at this machine's own campaign rate. Doing the sweep in-process matters as much as the per-config cost, since the fixed setup -- now almost entirely the Origami import -- is paid once instead of per config. Feasibility is the part that needs no accuracy argument at all: a config whose weights and KV cache do not fit is unrunnable for a reason the memory model settles exactly. Infeasible points are kept and marked rather than dropped, so a caller can tell "does not fit" from "was never tried", and one bad point costs one point instead of the sweep.
araina-amd
requested review from
JohnQinAMD,
jiejingzhangamd,
limou102 and
xiaobochen-amd
as code owners
August 25, 2026 23:54
Serving stacks ship this by default -- TRT-LLM's native "dram" offload, SGLang's HiCache -- so modelling only HBM understates both the concurrency and the prefix reuse those deployments get. The tier holds KV for idle sessions and for prefix blocks evicted from HBM, and a hit it holds is staged back over the host link rather than recomputed. Prefill now charges that fetch.
Quantities belonging to a single replica, or a single request, were being sized at the whole system. The decode step was sized at system-wide concurrency while every replica was then credited with that throughput, counting the same sequences once per replica; the KV handoff was sized at the resident batch, so a per-request first-token latency grew with concurrency; attention-DP rounded the per-rank batch down, charging fewer sequences than memory had provisioned; and nothing capped the system at what prefill could feed, so an under-provisioned prefill pool reported the decode pool's unfed ceiling. Attention DP also becomes per-pool, since deployments run it on prefill and plain TP attention on decode, which one global degree cannot express. Throughput per GPU is now reported over billed tokens and the whole fleet, because a decode-only figure ranks every prefill:decode ratio identically.
A decode step read each expert's full weights on every rank, as though TP had never sharded them, so the projection barely scaled with tensor parallelism. The factor is tp // ep, so only configs with TP above EP were affected -- which is why the pure-TP ladder was wrong while every EP-sharded config looked healthy. That cap was the older explanation for pure-TP MoE decode missing the weight roofline. The grouped-GEMM efficiency model accounts for the same shortfall by group size, so having both live cancelled the relief twice. Anchor selection follows. Whether expert parallelism shards at all is a different kernel set rather than a further degree of one, so it is now a sort key ahead of transport distance rather than a weight -- which would have to exceed every distance a store can produce. The batch-1 expert-parallelism bound returns to its original range, now that the gpt-oss EP ladders have been re-measured without a forced router skew and can arbitrate it.
It was priced at the full FFN width on every rank, so each rank paid for work the whole TP group splits between them. Compute-bound prefill carried the error in full; models without a shared expert are unaffected.
Dispatch and combine were charged at a few percent of peak HBM as random access. Measured on MI355X, the unit gathered is a whole hidden-size row, which is contiguous and streams at most of peak; the penalty only appears once the row shrinks past a few hundred bytes. The two halves are charged separately now, combine being slower for reducing topk rows with no reuse.
vLLM caches by default and the sweep replays one set of prompts, so on artifacts predating the harness disabling it every timed prefill is a block lookup -- consumed as ground truth, that let a fleet claim prefill supply it does not have. The cache state is now recorded even when off, and an artifact that does not assert it falls back to simulated prefill. Decode still calibrates: it is differenced between two runs that both hit.
Every flash-attention workgroup was charged a full 256-row query tile sweeping the whole KV, but a decode step brings one or two rows. A flash-decode kernel splits the KV across the CUs that short query axis leaves idle rather than padding, so the step stays bounded by the KV stream. The pad put the compute term above the bandwidth roofline and let it set the answer.
The group was strided by the full TP width, so a TP4/EP4 worker was sized as a 16-GPU communicator and its intra-node transfer priced at cluster bandwidth; it now strides by expert_tp = tp // ep. The measured intra-node fit was also applied at any size, though it was taken on decode-scale dispatches of a few MB -- extrapolated to a 1.6 GB prefill dispatch it charged seconds. The all-reduce already guards its own fit this way.
It compared a value parsed back out of the report against full-precision arithmetic at a relative tolerance, so a figure printed to three decimals could miss by a third of its last place and still fail. Half that last place is the tightest a parsed value can be held to, which is what the sibling prompt-cost check already does.
The per-kernel occupancy minimum was added on top of the forward pass for every kernel a layer issues, including the GEMMs and attention the layer profilers already time. Occupancy is a floor, not an addend: a kernel streaming 0.87 GB of KV takes 155 us and its 5 us minimum is inside that. It now covers only the norms, residual adds, RoPE, activations and top-k.
The breakdown called the collective model directly, printing each collective's standalone cost while the step charged the overlapped one -- on a DeepEP decode that read as 7.20 ms of expert all-to-all beside a step that had charged none of it. Measured mode keeps the standalone figure, having no separable comm to report.
Attention's projections and the dense MLP fell back to fp8, or to bf16 with no fp8 flag, while the memory model sized those same weights from weight_dtype -- so a 4-bit checkpoint was provisioned at 4 bits and then streamed at twice that width. A new field rather than an inference, because gpt-oss puts mxfp4 experts behind bf16 attention where an NVFP4 checkpoint quantizes the projections too.
Co-authored-by: Cursor <cursoragent@cursor.com>
The note pointed at measure_kernel_floor.py, which is no longer in the tree, so it asked for a tool a reader cannot run. Describe the sweep itself instead, and say it belongs on every architecture served rather than once: the constant is a property of the silicon, so one measurement does not cover the next vendor. Co-authored-by: Cursor <cursoragent@cursor.com>
The dispatch is performed by ``ep`` ranks and that group size is already passed to the collective. The parallelism argument alongside it only says which domain the transfer crosses: the collective multiplies it by the group size and compares against the node. Passing ``tp // ep`` there priced an on-node dispatch at pod bandwidth, inflating a collective that the expert ranks alone perform.
An artifact carrying no prefix_caching marker has its prefill discarded: a prefill timed against a warm cache measures a block lookup rather than prompt processing, so it cannot be calibrated against. The warning said as much but read as a note, and it never said that the fallback moves TTFT onto the simulator -- usually the number the projection was run for.
Keep why the domain argument is 1 -- the group size is already passed, so the parallelism alongside it only picks the domain, and anything above 1 inflates a collective that the expert ranks alone perform. Drop the incident narrative and the per-layer figures, which dated the comment to the bug rather than describing the code.
Four MoE architectures the measured deployments run and the projector had none for. Kimi-K3's linear-attention layers are recorded but not yet read, so every layer is charged as full latent attention; the header says so rather than letting the preset look like it models them.
The memory side of the ceiling read a constant that was never defined, so an architecture without a hardware profile raised NameError from the one branch that exists to cover it. Paired with the throughput the peak getter already falls back to, so both halves describe the same part.
A wide expert group that genuinely straddles two nodes crossed a round-number bound the hardware ratio supports. Widen it, and assert what it stood in for directly: sweep the message over a range of batch and require the per-byte price to stay flat, which a fixed per-layer cost cannot do.
One global per-kernel occupancy, solved on one part, priced another's decode step at more fixed cost than the whole step measures. It now resolves from the GPU that was named, and the same resolution hands the collective model its interconnect. Also stops the argument derivation discarding a declared KV-head count, which charged every query head against the one head actually cached.
Selecting a thousand keys from a hundred thousand is a prefill saving, where attention is quadratic and compute-bound. A decode step is a streaming read whose indexer still scores every token before it can choose any, and crediting it there halved the measured step on two models at long context. The indexer is now counted among the kernels a layer issues.
The memory model knew a rank holds the cache for a subset of the requests; the time model charged every rank the whole batch. That reverses a serving decision rather than mis-scaling one -- measured latent attention decodes well over half again faster with DP attention on, and this had it slower.
Hiding the All-to-All whenever expert compute merely exceeded it gave every deployment an asynchronous dispatch/combine, which made expert parallelism free and sent a mix search wide everywhere. The overlap now comes from the DeepEP/SyncFree efficiency, which is zero unless the engine has it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
infera/projection/— InferaSim, a workload-driven simulator and projector for the serving stack. It answers serving-configuration questions without occupying a node to ask each one: time to first token, inter-token latency, throughput, KV and weight memory, feasibility, and cost. One honest experiment on a large model costs a full node for minutes, and these questions are asked in the thousands, so the search happens in simulation and hardware is spent only on the shortlist.It is deliberately two coupled models over one measured foundation:
The governing idea is measure sparsely, transport analytically: a small number of cheap sub-scale benchmarks are harvested into anchors, and every other configuration is projected from the nearest applicable anchor rather than measured.
The change is additive. 273 new files under
infera/projection/andtests/unit/projection/, with no existing platform code modified apart frompyproject.toml, which gains aprojectionextra and theinferasim/inferasim-tuneentry points.Type of change
Changes
README.md(task-oriented) andARCHITECTURE.md(how it works, and an explicit statement of what it does not model).Checklist:
152 unit tests under
tests/unit/projection/, all of which run without a GPU.