From 896918e95cb0f6f9adf2cbebdb78be595aab1f24 Mon Sep 17 00:00:00 2001 From: ADou Date: Wed, 9 Sep 2026 11:06:19 +0000 Subject: [PATCH] [None][docs] drop deleted accuracy_eval/gen_server_config from disagg benchmark README Signed-off-by: ADou --- .../disaggregated/slurm/benchmark/README.md | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/examples/disaggregated/slurm/benchmark/README.md b/examples/disaggregated/slurm/benchmark/README.md index b211ba3e088e..6d68c391ccd3 100644 --- a/examples/disaggregated/slurm/benchmark/README.md +++ b/examples/disaggregated/slurm/benchmark/README.md @@ -13,9 +13,8 @@ The benchmarking process is orchestrated through a combination of Python scripts - `start_worker.sh`: Initializes context and generation workers - `start_server.sh`: Starts the disaggregated serving coordinator - `wait_server.sh`: Waits for server readiness before benchmarking - - `run_benchmark.sh` / `run_benchmark_nv_sa.sh`: Execute benchmark workloads - - `accuracy_eval.sh`: Runs accuracy evaluation using lm_eval - - `gen_server_config.py`: Generates server configuration from worker settings + - `run_benchmark.sh` / `run_benchmark_nv_sa.sh` / `run_benchmark_aiperf.sh`: Execute benchmark workloads + - `submit.py` invokes `lm_eval` directly when accuracy evaluation is enabled (no separate `accuracy_eval.sh`) ## Configuration (config.yaml) @@ -220,12 +219,15 @@ Enable accuracy evaluation using the lm_eval framework: ```yaml accuracy: enable_accuracy_test: true - model: "local-completions" - tasks: "gsm8k,hellaswag,mmlu" # Comma-separated task list - model_args_extra: "num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096" + tasks: + gsm8k: + model: "local-completions" + model_args_extra: "num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=7200,max_gen_toks=16384" + extra_kwargs: + trust_remote_code: true ``` -Accuracy results will be saved in `/accuracy_eval/` after benchmark completion. +`submit.py` launches one `lm_eval` command per task. Accuracy results are saved under `/accuracy_eval_/` after benchmark completion. #### 2. NVIDIA Nsight Systems Profiling @@ -272,4 +274,4 @@ environment: trtllm_wheel_path: "/path/to/tensorrt_llm-*.whl" trtllm_repo: "" build_wheel: false -``` +``` \ No newline at end of file