Background
Accurate request profiling is required for admission, scheduling, and memory residency decisions.
Scope
- Build a cost estimator from prompt length, output length, history, and live runtime signals.
- Estimate both latency risk and memory residency cost.
- Support calibration against experimental traces.
Deliverables
- Profiling module.
- Estimation formula documentation.
- Calibration and offline analysis scripts.
Acceptance Criteria
- Output: expected_prefill_ms, expected_decode_ms, expected_kv_growth_bytes, expected_weight_residency_cost, queue_slack_ms, deadline_miss_risk.
- Profiling outputs are consumable by scheduler and admission logic.
- Calibration workflow is reproducible on saved traces.
Dependencies
Notes
Supports informed admission control under dynamic multi-model workloads.
Background
Accurate request profiling is required for admission, scheduling, and memory residency decisions.
Scope
Deliverables
Acceptance Criteria
Dependencies
Notes
Supports informed admission control under dynamic multi-model workloads.