diff --git a/docs/cli-options.md b/docs/cli-options.md index 3af9300901..3b126886fe 100644 --- a/docs/cli-options.md +++ b/docs/cli-options.md @@ -356,6 +356,11 @@ The API endpoint type to benchmark. Determines request/response format and suppo Enable streaming responses. When enabled, the server streams tokens incrementally as they are generated. Automatically disabled if the selected endpoint type does not support streaming. Enables measurement of time-to-first-token (TTFT) and inter-token latency (ITL) metrics.
_Flag (no value required)_ +#### `--allow-empty-content` + +Retain explicitly empty content or reasoning strings as timed responses for streaming OpenAI Chat Completions. This changes TTFT and all metrics using the shared response timeline, but TTFO still requires non-empty output and empty strings add no tokens. +
_Flag (no value required)_ + #### `-u`, `--url` `` Base URL(s) of the API server(s) to benchmark. Multiple URLs can be specified for load balancing across multiple instances (e.g., `--url http://server1:8000 --url http://server2:8000`). The endpoint path is automatically appended based on `--endpoint-type` (e.g., `/v1/chat/completions` for `chat`). URLs that do not include a scheme (no `://`) have `http://` prepended automatically. @@ -1891,6 +1896,11 @@ The API endpoint type to benchmark. Determines request/response format and suppo Enable streaming responses. When enabled, the server streams tokens incrementally as they are generated. Automatically disabled if the selected endpoint type does not support streaming. Enables measurement of time-to-first-token (TTFT) and inter-token latency (ITL) metrics.
_Flag (no value required)_ +#### `--allow-empty-content` + +Retain explicitly empty content or reasoning strings as timed responses for streaming OpenAI Chat Completions. This changes TTFT and all metrics using the shared response timeline, but TTFO still requires non-empty output and empty strings add no tokens. +
_Flag (no value required)_ + #### `-u`, `--url` `` Base URL(s) of the API server(s) to benchmark. Multiple URLs can be specified for load balancing across multiple instances (e.g., `--url http://server1:8000 --url http://server2:8000`). The endpoint path is automatically appended based on `--endpoint-type` (e.g., `/v1/chat/completions` for `chat`). URLs that do not include a scheme (no `://`) have `http://` prepended automatically. diff --git a/docs/metrics-reference.md b/docs/metrics-reference.md index 4ac5fe79b8..e916ad1ca1 100644 --- a/docs/metrics-reference.md +++ b/docs/metrics-reference.md @@ -208,7 +208,9 @@ any knowledge of the individual request/response data. ## Streaming Metrics > [!NOTE] -> All metrics in this section require the `--streaming` flag with a token-producing endpoint and at least one non-empty response chunk. +> All metrics in this section require `--streaming` with a token-producing endpoint and one or more qualifying parsed responses. By default, a qualifying response has non-empty parsed content. With `--allow-empty-content`, an OpenAI Chat Completions `chat.completion.chunk` with an explicitly empty (`""`) `content`, `reasoning_content`, or `reasoning` string also qualifies. Missing fields, `null`, role/handshake metadata, and other non-content chunks do not qualify. + +Qualifying responses form one shared response timeline. Enabling `--allow-empty-content` can therefore change TTFT, TTST, Request Latency, Decode Duration, ICL, ITL, and the per-user output-throughput metrics together. It does not change token accounting: token normalization remains Output Sequence Length (OSL)-based, and an empty decoded string adds no client output or reasoning tokens. When server usage is available, its reported token counts remain authoritative with `--allow-empty-content` enabled. ### Time to First Token (TTFT) @@ -231,7 +233,7 @@ ttft_seconds = ttft_ns / 1e9 **Notes:** - Includes network latency, queuing time, prompt processing, and generation of the first token (or chunk of tokens). - Raw timestamps are in nanoseconds; converted to milliseconds for display and seconds for rate calculations. -- Response chunks refer to individual messages with non-empty content received during streaming. +- Uses the first qualifying parsed response on the shared response timeline. By default this is non-empty parsed content; with `--allow-empty-content`, an explicit empty Chat Completions content or reasoning string can be first. --- @@ -251,7 +253,7 @@ ttst_ms = ttst_ns / 1e6 ``` **Notes:** -- Requires at least 2 non-empty response chunks to compute the time between first and second tokens. +- Requires at least 2 qualifying parsed responses on the shared response timeline. With `--allow-empty-content`, either response can be an explicit empty Chat Completions content or reasoning string. - Raw timestamps are in nanoseconds; converted to milliseconds for display. --- @@ -276,7 +278,7 @@ ttfo_ms = ttfo_ns / 1e6 **Notes:** - TTFO vs TTFT: Time to First Output (TTFO) measures time to the first non-reasoning token, while Time to First Token (TTFT) measures time to any first token including reasoning tokens. For models without reasoning, TTFO and TTFT are equivalent. - Non-reasoning tokens include TextResponseData with non-empty text, or ReasoningResponseData with non-empty content field (regardless of reasoning field). -- Requires at least one non-empty non-reasoning response chunk. +- Requires at least one non-empty non-reasoning response chunk. `--allow-empty-content` never makes an empty content or reasoning string satisfy TTFO. --- @@ -284,7 +286,7 @@ ttfo_ms = ttfo_ns / 1e6 **Type:** [Record Metric](#record-metrics) -Measures the client-observed wall-clock interval between the first and final non-empty streamed content responses. Unlike ITL, this metric is not normalized by the number of generated tokens. +Measures the client-observed wall-clock interval between the first and final qualifying parsed responses on the shared response timeline. Unlike ITL, this metric is not normalized by the number of generated tokens. **Formula:** ```python @@ -295,6 +297,7 @@ decode_duration_ms = decode_duration_ns / 1e6 **Notes:** - This is an end-to-end client observation. It includes response delivery and client-observed gaps after TTFT; it does not isolate server kernel execution time. +- With `--allow-empty-content`, explicit empty Chat Completions content or reasoning strings participate in both ends of this interval. For example, a trailing Kimi K3 empty-content chunk extends Decode Duration. - Compare Decode Duration with Output Sequence Length when early stopping or a generation-length change is possible. - A run can have similar Decode Duration but a much higher ITL if it produces fewer output tokens. - Requires valid `time_to_first_token` and `request_latency` metrics. @@ -305,20 +308,20 @@ decode_duration_ms = decode_duration_ns / 1e6 **Type:** [Record Metric](#record-metrics) -Measures the client-observed wall-clock interval from the first non-empty parsed -content response until the HTTP response is fully consumed. Unlike Decode +Measures the client-observed wall-clock interval from the first qualifying parsed +response until the HTTP response is fully consumed. Unlike Decode Duration, it includes time after the last parsed content chunk, such as generation that a structured-output parser suppresses before the terminal usage response. **Formula:** ```python -full_decode_duration_ns = request.end_perf_ns - first_content_response.perf_ns +full_decode_duration_ns = request.end_perf_ns - first_qualifying_response.perf_ns ``` **Notes:** - This is a client-observed full-response duration, not server kernel execution time. - It includes terminal serialization, transport, and client-processing overhead. -- It requires an explicit request-end timestamp and at least one non-empty content response. +- It requires an explicit request-end timestamp and at least one qualifying parsed response. Its end is always `request.end_perf_ns`, even when `--allow-empty-content` changes the shared response timeline. - Existing profile exports can reconstruct it as `(request_end_ns - request_start_ns) - time_to_first_token_ns`. --- @@ -343,7 +346,7 @@ inter_token_latency_ms = inter_token_latency_ns / 1e6 **Notes:** - Requires an output sequence length of at least 2 tokens and valid `time_to_first_token`, `request_latency`, and `output_sequence_length` metrics. -- ITL is generated-token-normalized decode duration. It is not the total wall-clock decode duration. +- ITL is OSL-normalized Decode Duration on the shared response timeline. It is not the total wall-clock decode duration, and empty decoded strings do not add client tokens to OSL. - Compare runs at equivalent output lengths, or inspect Decode Duration and Output Sequence Length alongside ITL. - Streaming chunks can contain multiple tokens. ITL uses token count, while ICL uses chunk arrival timestamps. - Result is in seconds when used for throughput calculations (Output Token Throughput Per User). @@ -356,7 +359,7 @@ inter_token_latency_ms = inter_token_latency_ns / 1e6 **Type:** [Record Metric](#record-metrics) Measures the average token interval over the full client-observed decode window, -from the first non-empty parsed content response through explicit HTTP request +from the first qualifying parsed response through explicit HTTP request completion. **Formula:** @@ -368,7 +371,7 @@ full_response_inter_token_latency_ns = ( **Notes:** - Requires an output sequence length of at least 2 tokens and a valid Full Decode Duration. -- Uses the same token normalization as Inter Token Latency while extending the interval through HTTP response completion. +- Uses the same OSL token normalization as Inter Token Latency while extending the interval through `request.end_perf_ns`; empty decoded strings do not add client tokens. - With server-reported token counting, it keeps the duration and token count aligned when a response parser suppresses generated tokens. - This remains a client-observed average, not a distribution of raw engine token-to-token timestamps. - Streaming chunks can contain multiple tokens. A response delivered entirely in one content chunk may not expose a meaningful post-first-content decode interval. @@ -387,7 +390,7 @@ inter_chunk_latency = [request.content_responses[i].perf_ns - request.content_re ``` **Notes:** -- Requires at least 2 response chunks. +- Requires at least 2 qualifying parsed responses on the shared response timeline. With `--allow-empty-content`, explicit empty Chat Completions content or reasoning strings are included. - Unlike ITL (which produces a single average), ICL provides the full distribution of inter-chunk times. - Useful for detecting variability, jitter, or issues in streaming delivery. - Analyzing ICL distributions can reveal batching behavior, scheduling issues, or network variability. @@ -410,6 +413,7 @@ output_token_throughput_per_user = 1.0 / inter_token_latency_seconds **Notes:** - Computes the inverse of ITL to show tokens per second from an individual user's perspective. +- Uses the same shared response timeline and OSL normalization as ITL; empty decoded strings do not increase the client token total. - Differs from Output Token Throughput (aggregate across all concurrent requests) by focusing on single-request experience. - Useful for understanding the user experience independent of concurrency effects. - Assumes the output token count describes the content-delivery interval. If a server reports tokens that its response parser suppresses, compare it with the full-response metric pair. @@ -432,7 +436,8 @@ full_response_output_token_throughput_per_user = ( **Notes:** - Computes the inverse of Full-Response Inter Token Latency, mirroring the relationship between Output Token Throughput Per User and Inter Token Latency. -- Excludes TTFT but measures through full HTTP response completion. +- Excludes TTFT but measures through `request.end_perf_ns`, regardless of where qualifying responses appear on the shared response timeline. +- Uses OSL normalization; explicit empty decoded strings do not add client tokens. - With server-reported token counting, this approximates raw engine decode TPS when the response parser suppresses generated tokens. - It remains a client-observed approximation because AIPerf does not have raw engine first/last-token timestamps. - Aggregate latency percentiles and throughput percentiles are not interchangeable: `1 / p75(latency)` describes the slow tail, while `p75(throughput)` describes the fast side of the reciprocal distribution. @@ -1753,8 +1758,8 @@ total_error_isl = sum(r.error_isl for r in records if not r.valid) **Type:** [Record Metric](#record-metrics) -Measures the time from request start until the final non-empty parsed content -response. Usage-only and terminal responses are excluded. +Measures the time from request start until the final qualifying parsed response +on the shared response timeline. Usage-only and terminal responses are excluded. **Formula:** ```python @@ -1762,9 +1767,9 @@ request_latency_ns = request.content_responses[-1].perf_ns - request.start_perf_ ``` **Notes:** -- Includes network time, queuing, prompt processing, and content delivery through the final parsed content chunk. +- Includes network time, queuing, prompt processing, and content delivery through the final qualifying parsed response. With `--allow-empty-content`, a trailing explicit empty Kimi K3 content chunk extends Request Latency (and therefore Decode Duration). - It can be shorter than the HTTP lifecycle when a response parser suppresses generated content or terminal usage arrives later. -- Use `http_req_duration` for the complete HTTP exchange and Full Decode Duration for the post-TTFT interval through request completion. +- Use `http_req_duration` for the complete HTTP exchange and Full Decode Duration, Full-Response ITL, or Full-Response Output Token Throughput Per User for the post-TTFT interval through `request.end_perf_ns`. --- diff --git a/docs/tutorials/openai-text-endpoints.md b/docs/tutorials/openai-text-endpoints.md index 899dac46aa..e0a5d6bf91 100644 --- a/docs/tutorials/openai-text-endpoints.md +++ b/docs/tutorials/openai-text-endpoints.md @@ -45,6 +45,30 @@ aiperf profile \ ``` +### Include explicit empty streaming responses + +Some OpenAI-compatible servers, including Kimi K3 deployments, can emit explicit +empty content or reasoning strings as part of a Chat Completions stream. Enable +their inclusion in the response timeline with: + +```bash +aiperf profile \ + --model Kimi-K3 \ + --endpoint-type chat \ + --endpoint /v1/chat/completions \ + --streaming \ + --allow-empty-content \ + --url localhost:8000 \ + --request-count 20 +``` + +The flag is off by default because OpenAI-compatible servers can emit +handshake or role metadata before generation. It admits only explicit empty +strings in Chat Completions content or reasoning fields, not missing fields, +`null`, or metadata. All shared response-timeline metrics change together when +these strings are included. TTFO still waits for non-empty output, and an empty +decoded string does not increment output-token totals. + **Sample Output (Successful Run):** ``` INFO Starting AIPerf System @@ -162,4 +186,4 @@ aiperf profile \ --request-count 10 ``` - \ No newline at end of file + diff --git a/src/aiperf/common/models/model_endpoint_info.py b/src/aiperf/common/models/model_endpoint_info.py index a9ba57a5f2..46f9e0e0c6 100644 --- a/src/aiperf/common/models/model_endpoint_info.py +++ b/src/aiperf/common/models/model_endpoint_info.py @@ -77,6 +77,11 @@ class EndpointInfo(AIPerfBaseModel): default=EndpointDefaults.STREAMING, description="Whether the endpoint supports streaming.", ) + allow_empty_content: bool = Field( + default=EndpointDefaults.ALLOW_EMPTY_CONTENT, + description="Whether explicitly empty OpenAI chat content or reasoning " + "strings participate in the parsed response timeline.", + ) headers: list[tuple[str, str]] = Field( default=[], description="Custom URL headers to use for the endpoint.", @@ -205,6 +210,7 @@ def from_run(cls, run: BenchmarkRun) -> ModelEndpointInfo: type=ep.type, custom_endpoint=getattr(ep, "path", None), streaming=ep.streaming, + allow_empty_content=ep.allow_empty_content, base_urls=list(ep.urls), headers=list((getattr(ep, "headers", {}) or {}).items()), extra=list((getattr(ep, "extra", {}) or {}).items()), diff --git a/src/aiperf/common/models/record_models.py b/src/aiperf/common/models/record_models.py index 5dcebb5347..a2389f76d3 100644 --- a/src/aiperf/common/models/record_models.py +++ b/src/aiperf/common/models/record_models.py @@ -1590,10 +1590,11 @@ def end_perf_ns(self) -> int: @cached_property def content_responses(self) -> list[ParsedResponse]: - """Get only responses with actual content (data is not None or empty). + """Get parsed content responses on the shared metric timeline. - This excludes usage-only or [DONE] responses that may appear at the end of streaming responses. - Useful for timing metrics that should measure content delivery. + Explicit empty chat strings may be present when endpoint extraction is + configured to retain them. Usage-only, metadata-only, null-data, and + [DONE] responses are excluded. """ return [response for response in self.responses if response.data] @@ -1631,7 +1632,7 @@ def create_error_from_invalid(self) -> None: err = InvalidInferenceResultError("Invalid inference result") if len(self.responses) == 0 or len(self.content_responses) == 0: err.add_note( - "No responses with actual content were received from the server (only usage/metadata, null/empty data, or [DONE] markers)" + "No parsed content responses were received from the server (only usage/metadata, null data, or [DONE] markers)" ) if self.start_perf_ns <= 0 or self.start_perf_ns >= sys.maxsize: err.add_note( diff --git a/src/aiperf/config/endpoint.py b/src/aiperf/config/endpoint.py index 5ea37ea782..6ad090f189 100644 --- a/src/aiperf/config/endpoint.py +++ b/src/aiperf/config/endpoint.py @@ -47,6 +47,7 @@ class EndpointDefaults: CUSTOM_ENDPOINT = None TYPE = EndpointType.CHAT STREAMING = False + ALLOW_EMPTY_CONTENT = False URL = "http://localhost:8000" URL_STRATEGY = URLSelectionStrategy.ROUND_ROBIN TIMEOUT = 6 * 60 * 60 # 6 hours, match vLLM benchmark default @@ -200,6 +201,16 @@ def _redact_api_key(self, value: str | None) -> str | None: ), ] + allow_empty_content: Annotated[ + bool, + Field( + default=EndpointDefaults.ALLOW_EMPTY_CONTENT, + description="Retain explicitly empty content or reasoning strings as " + "timed responses for streaming OpenAI Chat Completions. This changes " + "the shared response timeline without adding output tokens.", + ), + ] + _streaming_explicitly_set: bool = False transport: Annotated[ diff --git a/src/aiperf/config/flags/_converter_endpoint.py b/src/aiperf/config/flags/_converter_endpoint.py index cee2871b23..b2b8c2d0b7 100644 --- a/src/aiperf/config/flags/_converter_endpoint.py +++ b/src/aiperf/config/flags/_converter_endpoint.py @@ -59,6 +59,7 @@ def _endpoint_template_fallback(endpoint: dict[str, Any]) -> None: "url_selection_strategy": "url_strategy", "endpoint_type": "type", "streaming": "streaming", + "allow_empty_content": "allow_empty_content", "custom_endpoint": "path", "api_key": "api_key", "timeout_seconds": "timeout", diff --git a/src/aiperf/config/flags/_section_fields.py b/src/aiperf/config/flags/_section_fields.py index 0e939b38f7..3b344d844d 100644 --- a/src/aiperf/config/flags/_section_fields.py +++ b/src/aiperf/config/flags/_section_fields.py @@ -17,6 +17,7 @@ ENDPOINT_FIELDS: frozenset[str] = frozenset( { "api_key", + "allow_empty_content", "connection_reuse_strategy", "custom_endpoint", "download_video_content", diff --git a/src/aiperf/config/flags/cli_config.py b/src/aiperf/config/flags/cli_config.py index 75a8afc3b6..a98b73d09a 100644 --- a/src/aiperf/config/flags/cli_config.py +++ b/src/aiperf/config/flags/cli_config.py @@ -178,6 +178,20 @@ class CLIConfig(BaseConfig): ), ] = EndpointDefaults.STREAMING + allow_empty_content: Annotated[ + bool, + Field( + description="Retain explicitly empty content or reasoning strings as " + "timed responses for streaming OpenAI Chat Completions. This changes " + "TTFT and all metrics using the shared response timeline, but TTFO " + "still requires non-empty output and empty strings add no tokens.", + ), + CLIParameter( + name=("--allow-empty-content",), + group=Groups.ENDPOINT, + ), + ] = EndpointDefaults.ALLOW_EMPTY_CONTENT + urls: Annotated[ list[str], Field( diff --git a/src/aiperf/config/schema/aiperf-config.schema.json b/src/aiperf/config/schema/aiperf-config.schema.json index 122cd9ac39..68cf5059ee 100644 --- a/src/aiperf/config/schema/aiperf-config.schema.json +++ b/src/aiperf/config/schema/aiperf-config.schema.json @@ -13013,6 +13013,12 @@ "title": "Streaming", "type": "boolean" }, + "allowEmptyContent": { + "default": false, + "description": "Retain explicitly empty content or reasoning strings as timed responses for streaming OpenAI Chat Completions. This changes the shared response timeline without adding output tokens.", + "title": "Allowemptycontent", + "type": "boolean" + }, "transport": { "anyOf": [ { diff --git a/src/aiperf/endpoints/openai_chat.py b/src/aiperf/endpoints/openai_chat.py index c2c8c099f5..1e44a69d54 100644 --- a/src/aiperf/endpoints/openai_chat.py +++ b/src/aiperf/endpoints/openai_chat.py @@ -14,6 +14,7 @@ ReasoningResponseData, RequestInfo, RequestRecord, + TextResponseData, ToolCallResponseData, Turn, ) @@ -243,6 +244,30 @@ def process_responses( } return parsed_responses, Turn(role="assistant", raw_messages=[assistant_msg]) + def _allowed_empty_channels(self, json_obj: JsonObject) -> tuple[bool, bool]: + """Return opted-in explicit empty channels for one chat stream chunk.""" + if ( + not self.model_endpoint.endpoint.allow_empty_content + or json_obj.get("object") != "chat.completion.chunk" + ): + return False, False + + choices = json_obj.get("choices") + if not isinstance(choices, list) or not choices: + return False, False + choice = choices[0] + if not isinstance(choice, dict): + return False, False + delta = choice.get("delta") + if not isinstance(delta, dict): + return False, False + + empty_output = delta.get("content") == "" + empty_reasoning = ( + delta.get("reasoning_content") == "" or delta.get("reasoning") == "" + ) + return empty_output, empty_reasoning + def extract_chat_response_data( self, json_obj: JsonObject ) -> BaseResponseData | None: @@ -255,6 +280,9 @@ def extract_chat_response_data( chunks/messages so client-side TTFT and OSL include the tokens the model generated for the dispatch (function name + arguments). Precedence is ``reasoning > content+tool_calls > tool_calls > content``. + With ``allow_empty_content`` enabled, explicit empty streaming strings + are parsed as responses. Empty text bypasses ``make_text_response_data`` + because that generic helper intentionally excludes empty strings. A chunk that carries both prose ``content`` and a ``tool_calls`` delta returns a ``ToolCallResponseData`` with both fields set (~18% of agentic turns) so client-side OSL counts both portions @@ -266,6 +294,8 @@ def extract_chat_response_data( Returns: Extracted response data or None if no content """ + empty_output, empty_reasoning = self._allowed_empty_channels(json_obj) + match json_obj.get("object"): case "chat.completion": data_key = "message" @@ -279,12 +309,17 @@ def extract_chat_response_data( return None choices = json_obj.get("choices") - if not choices: + if not isinstance(choices, list) or not choices: self.debug(lambda: f"No choices found in response: {json_obj}") return None - data = choices[0].get(data_key) - if not data: + choice = choices[0] + if not isinstance(choice, dict): + self.debug(lambda: f"Malformed choice found in response: {json_obj}") + return None + + data = choice.get(data_key) + if not isinstance(data, dict) or not data: self.debug(lambda: f"No data found in response: {json_obj}") return None @@ -322,6 +357,15 @@ def extract_chat_response_data( if content: return self.make_text_response_data(content) + if empty_reasoning: + return ReasoningResponseData( + content="" if empty_output else None, + reasoning="", + ) + + if empty_output: + return TextResponseData(text="") + return None def build_assistant_turn(self, record: RequestRecord) -> Turn | None: diff --git a/src/aiperf/metrics/types/decode_duration_metric.py b/src/aiperf/metrics/types/decode_duration_metric.py index 119fc579a9..2f96eecaa1 100644 --- a/src/aiperf/metrics/types/decode_duration_metric.py +++ b/src/aiperf/metrics/types/decode_duration_metric.py @@ -63,7 +63,7 @@ def _parse_record( ) -> int: if not record.content_responses: raise NoMetricValue( - "Full decode duration requires at least 1 non-empty content response." + "Full decode duration requires at least 1 qualifying content response." ) if record.request.end_perf_ns is None: raise NoMetricValue( diff --git a/src/aiperf/metrics/types/request_latency_metric.py b/src/aiperf/metrics/types/request_latency_metric.py index a2f2923d8d..c8db0d28c8 100644 --- a/src/aiperf/metrics/types/request_latency_metric.py +++ b/src/aiperf/metrics/types/request_latency_metric.py @@ -33,14 +33,14 @@ def _parse_record( """ This method extracts the request and last response timestamps, and calculates the differences in time. - Note: Uses the last content response (with actual data), not usage-only chunks. + Note: Uses the last qualifying content response, not usage-only chunks. """ request_ts: int = record.start_perf_ns - # Use content_responses to get last response with actual content + # Use content_responses to get the last qualifying content response. if not record.content_responses: raise NoMetricValue( - "Request latency requires at least 1 non-empty content response." + "Request latency requires at least 1 qualifying content response." ) final_response_ts = record.content_responses[-1].perf_ns diff --git a/tests/aiperf_mock_server/README.md b/tests/aiperf_mock_server/README.md index 767c90af57..f6046f80dd 100644 --- a/tests/aiperf_mock_server/README.md +++ b/tests/aiperf_mock_server/README.md @@ -111,6 +111,7 @@ Configuration via CLI arguments or environment variables (`MOCK_SERVER_` prefix) | `--host` | | `127.0.0.1` | Bind address | | `--workers` | `-w` | `1` | Uvicorn worker count | | `--fast` | `-f` | `false` | Zero latency mode | +| `--emit-empty-chat-content` | | `false` | Emit one explicit empty content chunk before streaming Chat Completions output | | `--log-level` | | `INFO` | Logging level (DEBUG/INFO/WARNING/ERROR/CRITICAL) | | `--verbose` | `-v` | `false` | Debug logging (overrides log-level) | | `--access-logs` | | `false` | HTTP access logs | diff --git a/tests/aiperf_mock_server/config.py b/tests/aiperf_mock_server/config.py index a7cd3c627f..101906bc5d 100644 --- a/tests/aiperf_mock_server/config.py +++ b/tests/aiperf_mock_server/config.py @@ -373,6 +373,15 @@ def apply_flags(self) -> Self: Parameter(name=("--fast", "-f")), ] = False + emit_empty_chat_content: Annotated[ + bool, + Field( + description="Emit one explicit empty content chunk before streaming " + "Chat Completions output." + ), + Parameter(name="--emit-empty-chat-content"), + ] = False + anthropic_split_usage: Annotated[ bool, Field( diff --git a/tests/aiperf_mock_server/utils.py b/tests/aiperf_mock_server/utils.py index 7bf3f3db51..2a27660348 100644 --- a/tests/aiperf_mock_server/utils.py +++ b/tests/aiperf_mock_server/utils.py @@ -497,6 +497,22 @@ async def stream_chat_completion( has_reasoning = bool(ctx.reasoning_content_tokens) try: + if server_config.emit_empty_chat_content: + yield _sse( + { + "id": ctx.request_id, + "object": "chat.completion.chunk", + "created": int(time.time()), + "model": ctx.model, + "choices": [ + { + "index": 0, + "delta": {"role": "assistant", "content": ""}, + } + ], + } + ) + # Stream reasoning tokens first (if any) for token in ctx.reasoning_content_tokens: await ctx.latency_sim.wait_for_next_token() diff --git a/tests/unit/config/test_allow_empty_content.py b/tests/unit/config/test_allow_empty_content.py new file mode 100644 index 0000000000..1eb4bd8bce --- /dev/null +++ b/tests/unit/config/test_allow_empty_content.py @@ -0,0 +1,95 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import textwrap + +from cyclopts import App + +from aiperf.common.models.model_endpoint_info import ModelEndpointInfo +from aiperf.config import BenchmarkRun +from aiperf.config.flags._converter_endpoint import build_endpoint +from aiperf.config.flags.cli_config import CLIConfig +from aiperf.config.loader import build_benchmark_plan, load_config_from_string + + +def _parse_cli_args(argv: list[str]) -> CLIConfig: + captured: dict[str, CLIConfig] = {} + app = App(name="test_profile") + + @app.default + def _runner(*, cli_config: CLIConfig) -> None: + captured["config"] = cli_config + + try: + app( + [ + "--url", + "http://localhost:8000", + "--model", + "test-model", + *argv, + ], + exit_on_error=False, + ) + except SystemExit as exc: + if exc.code not in (0, None): + raise + return captured["config"] + + +def test_allow_empty_content_cli_defaults_false_and_unset() -> None: + config = _parse_cli_args([]) + + assert config.allow_empty_content is False + assert "allow_empty_content" not in config.model_fields_set + + +def test_allow_empty_content_cli_maps_to_endpoint() -> None: + config = _parse_cli_args(["--allow-empty-content"]) + + assert config.allow_empty_content is True + assert build_endpoint(config)["allow_empty_content"] is True + + +def test_allow_empty_content_unset_is_not_emitted_by_converter() -> None: + assert "allow_empty_content" not in build_endpoint(CLIConfig()) + + +def test_allow_empty_content_yaml_reaches_runtime_endpoint() -> None: + config = load_config_from_string( + textwrap.dedent( + """\ + benchmark: + models: [test-model] + endpoint: + urls: [http://localhost:8000/v1/chat/completions] + type: chat + streaming: true + allow_empty_content: true + datasets: + - name: default + type: synthetic + phases: + - name: profiling + type: concurrency + concurrency: 1 + requests: 1 + """ + ), + substitute_env=True, + ) + plan = build_benchmark_plan(config) + cfg = plan.configs[0] + run = BenchmarkRun( + benchmark_id="test-run", + sweep_id=plan.sweep_id, + cfg=cfg, + artifact_dir=cfg.artifacts.dir, + random_seed=None, + variables={}, + ) + + assert run.cfg.endpoint.allow_empty_content is True + assert ModelEndpointInfo.from_run(run).endpoint.allow_empty_content is True diff --git a/tests/unit/endpoints/test_openai_chat_empty_content.py b/tests/unit/endpoints/test_openai_chat_empty_content.py new file mode 100644 index 0000000000..9eb731e841 --- /dev/null +++ b/tests/unit/endpoints/test_openai_chat_empty_content.py @@ -0,0 +1,176 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +from __future__ import annotations + +import orjson +import pytest +from pytest import param + +from aiperf.common.models import ( + BaseResponseData, + ReasoningResponseData, + SSEMessage, + TextResponseData, +) +from aiperf.endpoints.openai_chat import ChatEndpoint +from aiperf.plugin.enums import EndpointType +from tests.unit.endpoints.conftest import create_model_endpoint + + +def _endpoint(*, enabled: bool) -> ChatEndpoint: + model_endpoint = create_model_endpoint(EndpointType.CHAT, streaming=True) + model_endpoint.endpoint.allow_empty_content = enabled + return ChatEndpoint(model_endpoint) + + +def _chunk(delta: dict[str, object], *, finish_reason: str | None = None) -> SSEMessage: + payload = { + "object": "chat.completion.chunk", + "choices": [{"delta": delta, "finish_reason": finish_reason}], + } + return SSEMessage.parse( + b"data: " + orjson.dumps(payload) + b"\n\n", + perf_ns=100, + ) + + +@pytest.mark.parametrize( + ("delta", "expected_type"), + [ + param({"content": ""}, TextResponseData, id="empty-content"), + param( + {"reasoning_content": ""}, + ReasoningResponseData, + id="empty-reasoning-content", + ), + param({"reasoning": ""}, ReasoningResponseData, id="empty-reasoning-alias"), + param( + {"content": "", "reasoning_content": ""}, + ReasoningResponseData, + id="both-empty", + ), + ], +) # fmt: skip +def test_enabled_retains_explicit_empty_strings( + delta: dict[str, object], expected_type: type[BaseResponseData] +) -> None: + parsed = _endpoint(enabled=True).parse_response(_chunk(delta)) + + assert parsed is not None + assert isinstance(parsed.data, expected_type) + + +@pytest.mark.parametrize( + "delta", + [ + param({"content": ""}, id="empty-content"), + param({"reasoning_content": ""}, id="empty-reasoning-content"), + param({"reasoning": ""}, id="empty-reasoning-alias"), + ], +) # fmt: skip +def test_disabled_discards_explicit_empty_strings(delta: dict[str, object]) -> None: + assert _endpoint(enabled=False).parse_response(_chunk(delta)) is None + + +@pytest.mark.parametrize( + "delta", + [ + param({"role": "assistant"}, id="role-only"), + param({"content": None}, id="null-content"), + param({}, id="empty-delta"), + param({"tool_calls": []}, id="empty-tool-list"), + ], +) # fmt: skip +def test_enabled_still_discards_non_generation_deltas( + delta: dict[str, object], +) -> None: + assert _endpoint(enabled=True).parse_response(_chunk(delta)) is None + + +def test_enabled_retains_terminal_explicit_empty_content() -> None: + parsed = _endpoint(enabled=True).parse_response( + _chunk({"content": ""}, finish_reason="stop") + ) + + assert parsed is not None + assert parsed.data == TextResponseData(text="") + + +def test_non_streaming_empty_content_remains_excluded() -> None: + endpoint = _endpoint(enabled=True) + payload = { + "object": "chat.completion", + "choices": [{"message": {"role": "assistant", "content": ""}}], + } + response = SSEMessage.parse( + b"data: " + orjson.dumps(payload) + b"\n\n", perf_ns=100 + ) + + assert endpoint.parse_response(response) is None + + +def test_non_empty_reasoning_alias_wins_over_empty_reasoning_content() -> None: + parsed = _endpoint(enabled=True).parse_response( + _chunk({"content": "", "reasoning_content": "", "reasoning": "thinking"}) + ) + + assert parsed is not None + assert parsed.data == ReasoningResponseData(content="", reasoning="thinking") + + +def test_enabled_finish_only_chunk_remains_excluded() -> None: + assert ( + _endpoint(enabled=True).parse_response(_chunk({}, finish_reason="stop")) is None + ) + + +def test_enabled_usage_only_chunk_has_no_content_data() -> None: + payload = { + "object": "chat.completion.chunk", + "choices": [], + "usage": {"completion_tokens": 7}, + } + response = SSEMessage.parse( + b"data: " + orjson.dumps(payload) + b"\n\n", perf_ns=100 + ) + + parsed = _endpoint(enabled=True).parse_response(response) + + assert parsed is not None + assert parsed.data is None + assert parsed.usage is not None + + +def test_enabled_done_marker_remains_excluded() -> None: + assert ( + _endpoint(enabled=True).parse_response( + SSEMessage.parse(b"data: [DONE]\n\n", perf_ns=100) + ) + is None + ) + + +@pytest.mark.parametrize( + "payload", + [ + param( + {"object": "chat.completion.chunk", "choices": ["bad"]}, + id="non-object-choice", + ), + param( + { + "object": "chat.completion.chunk", + "choices": [{"delta": "bad"}], + }, + id="non-object-delta", + ), + ], +) # fmt: skip +def test_enabled_discards_malformed_stream_events( + payload: dict[str, object], +) -> None: + response = SSEMessage.parse( + b"data: " + orjson.dumps(payload) + b"\n\n", perf_ns=100 + ) + + assert _endpoint(enabled=True).parse_response(response) is None diff --git a/tests/unit/metrics/test_empty_content_timeline.py b/tests/unit/metrics/test_empty_content_timeline.py new file mode 100644 index 0000000000..49d8acbff6 --- /dev/null +++ b/tests/unit/metrics/test_empty_content_timeline.py @@ -0,0 +1,186 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import orjson +import pytest + +from aiperf.common.models import ( + ParsedResponseRecord, + RequestRecord, + SSEMessage, + TokenCounts, +) +from aiperf.endpoints.openai_chat import ChatEndpoint +from aiperf.metrics.types.decode_duration_metric import ( + DecodeDurationMetric, + FullDecodeDurationMetric, +) +from aiperf.metrics.types.inter_chunk_latency_metric import InterChunkLatencyMetric +from aiperf.metrics.types.inter_token_latency_metric import ( + FullResponseInterTokenLatencyMetric, + InterTokenLatencyMetric, +) +from aiperf.metrics.types.output_token_throughput_metrics import ( + FullResponseOutputTokenThroughputPerUserMetric, + OutputTokenThroughputPerUserMetric, +) +from aiperf.metrics.types.prefill_throughput_per_user import ( + PrefillThroughputPerUserMetric, +) +from aiperf.metrics.types.request_latency_metric import RequestLatencyMetric +from aiperf.metrics.types.time_to_first_output_token_metric import ( + TimeToFirstOutputTokenMetric, +) +from aiperf.metrics.types.ttft_metric import TTFTMetric +from aiperf.metrics.types.ttst_metric import TTSTMetric +from aiperf.plugin.enums import EndpointType +from tests.unit.endpoints.conftest import create_model_endpoint +from tests.unit.metrics.conftest import run_simple_metrics_pipeline + +START_NS = 1_000_000_000 + + +def _event(perf_ns: int, payload: dict[str, object]) -> SSEMessage: + return SSEMessage.parse( + b"data: " + orjson.dumps(payload) + b"\n\n", + perf_ns=perf_ns, + ) + + +def _chat_chunk( + delta: dict[str, object], finish_reason: str | None = None +) -> dict[str, object]: + return { + "object": "chat.completion.chunk", + "choices": [{"delta": delta, "finish_reason": finish_reason}], + } + + +def _record(*, enabled: bool, output_tokens: int = 4) -> ParsedResponseRecord: + model_endpoint = create_model_endpoint(EndpointType.CHAT, streaming=True) + model_endpoint.endpoint.allow_empty_content = enabled + endpoint = ChatEndpoint(model_endpoint) + request = RequestRecord( + model_name="test-model", + start_perf_ns=START_NS, + end_perf_ns=1_600_000_000, + responses=[ + _event(1_050_000_000, _chat_chunk({"role": "assistant"})), + _event(1_100_000_000, _chat_chunk({"content": ""})), + _event(1_200_000_000, _chat_chunk({"content": "hello"})), + _event(1_300_000_000, _chat_chunk({"reasoning_content": ""})), + _event( + 1_400_000_000, + _chat_chunk({"content": ""}, finish_reason="stop"), + ), + _event(1_500_000_000, {"usage": {"completion_tokens": output_tokens}}), + ], + ) + return ParsedResponseRecord( + request=request, + responses=endpoint.extract_response_data(request), + token_counts=TokenCounts(input=100, output=output_tokens), + ) + + +def _empty_only_record() -> ParsedResponseRecord: + model_endpoint = create_model_endpoint(EndpointType.CHAT, streaming=True) + model_endpoint.endpoint.allow_empty_content = True + endpoint = ChatEndpoint(model_endpoint) + request = RequestRecord( + model_name="test-model", + start_perf_ns=START_NS, + end_perf_ns=1_200_000_000, + responses=[ + _event(1_100_000_000, _chat_chunk({"content": ""})), + ], + ) + return ParsedResponseRecord( + request=request, + responses=endpoint.extract_response_data(request), + token_counts=TokenCounts(input=100, output=0), + ) + + +def test_enabled_empty_responses_drive_one_metric_timeline() -> None: + record = _record(enabled=True) + results = run_simple_metrics_pipeline( + [record], + TTFTMetric.tag, + TTSTMetric.tag, + TimeToFirstOutputTokenMetric.tag, + RequestLatencyMetric.tag, + DecodeDurationMetric.tag, + FullDecodeDurationMetric.tag, + InterChunkLatencyMetric.tag, + InterTokenLatencyMetric.tag, + FullResponseInterTokenLatencyMetric.tag, + OutputTokenThroughputPerUserMetric.tag, + FullResponseOutputTokenThroughputPerUserMetric.tag, + PrefillThroughputPerUserMetric.tag, + ) + + assert [response.perf_ns for response in record.content_responses] == [ + 1_100_000_000, + 1_200_000_000, + 1_300_000_000, + 1_400_000_000, + ] + assert results[TTFTMetric.tag] == [100_000_000] + assert results[TTSTMetric.tag] == [100_000_000] + assert results[TimeToFirstOutputTokenMetric.tag] == [200_000_000] + assert results[RequestLatencyMetric.tag] == [400_000_000] + assert results[DecodeDurationMetric.tag] == [300_000_000] + assert results[FullDecodeDurationMetric.tag] == [500_000_000] + assert results[InterChunkLatencyMetric.tag] == [ + [100_000_000, 100_000_000, 100_000_000] + ] + assert results[InterTokenLatencyMetric.tag] == pytest.approx([100_000_000]) + assert results[FullResponseInterTokenLatencyMetric.tag] == pytest.approx( + [500_000_000 / 3] + ) + assert results[OutputTokenThroughputPerUserMetric.tag] == pytest.approx([10.0]) + assert results[FullResponseOutputTokenThroughputPerUserMetric.tag] == pytest.approx( + [6.0] + ) + assert results[PrefillThroughputPerUserMetric.tag] == pytest.approx([1_000.0]) + + +def test_disabled_empty_responses_preserve_non_empty_timeline() -> None: + record = _record(enabled=False) + results = run_simple_metrics_pipeline( + [record], + TTFTMetric.tag, + TTSTMetric.tag, + TimeToFirstOutputTokenMetric.tag, + RequestLatencyMetric.tag, + DecodeDurationMetric.tag, + FullDecodeDurationMetric.tag, + InterChunkLatencyMetric.tag, + ) + + assert [response.perf_ns for response in record.content_responses] == [ + 1_200_000_000 + ] + assert results[TTFTMetric.tag] == [200_000_000] + assert results[TimeToFirstOutputTokenMetric.tag] == [200_000_000] + assert results[RequestLatencyMetric.tag] == [200_000_000] + assert results[DecodeDurationMetric.tag] == [0] + assert results[FullDecodeDurationMetric.tag] == [400_000_000] + assert TTSTMetric.tag not in results + assert InterChunkLatencyMetric.tag not in results + + +def test_enabled_empty_only_response_is_valid_but_has_no_token_metrics() -> None: + record = _empty_only_record() + + assert record.valid + assert record.token_counts is not None + assert record.token_counts.output == 0 + results = run_simple_metrics_pipeline( + [record], TTFTMetric.tag, InterTokenLatencyMetric.tag + ) + assert results[TTFTMetric.tag] == [100_000_000] + assert InterTokenLatencyMetric.tag not in results diff --git a/tests/unit/records/test_inference_result_parser.py b/tests/unit/records/test_inference_result_parser.py index ba6fe5ea3a..eb25321270 100644 --- a/tests/unit/records/test_inference_result_parser.py +++ b/tests/unit/records/test_inference_result_parser.py @@ -10,12 +10,14 @@ from aiperf.common.models import ( ErrorDetails, ParsedResponse, + ReasoningResponseData, RequestRecord, TextResponse, TextResponseData, Usage, ) from aiperf.endpoints.openai_chat import ChatEndpoint +from aiperf.records.inference_result_parser import InferenceResultParser from tests.unit.records.conftest import ( create_invalid_record, create_test_request_info, @@ -147,7 +149,7 @@ async def test_no_content_responses_converted_to_error( assert record.has_error assert record.error.type == "InvalidInferenceResultError" - assert "No responses with actual content" in record.error.message + assert "No parsed content responses" in record.error.message assert result.token_counts.input == 8 assert result.responses == [] @@ -285,11 +287,57 @@ async def test_client_side_token_counts_uses_async( assert result.output == 3 assert spy_tokenizer.encode.called + async def test_empty_strings_do_not_change_client_output_token_count( + self, setup_inference_parser: InferenceResultParser, spy_tokenizer: MagicMock + ) -> None: + setup_inference_parser.get_tokenizer = AsyncMock(return_value=spy_tokenizer) + baseline = [ParsedResponse(perf_ns=200, data=TextResponseData(text="hello"))] + with_empty = [ + ParsedResponse(perf_ns=100, data=TextResponseData(text="")), + baseline[0], + ParsedResponse( + perf_ns=300, + data=ReasoningResponseData(reasoning="", content=None), + ), + ParsedResponse(perf_ns=400, data=TextResponseData(text="")), + ] + + baseline_counts = ( + await setup_inference_parser._compute_client_side_token_counts( + RequestRecord(model_name="test-model"), baseline + ) + ) + empty_counts = await setup_inference_parser._compute_client_side_token_counts( + RequestRecord(model_name="test-model"), with_empty + ) + + assert baseline_counts.output == empty_counts.output == 1 + assert baseline_counts.reasoning is None + assert empty_counts.reasoning is None + @pytest.mark.asyncio class TestServerTokenCount: """Tests for --use-server-token-count flag functionality.""" + async def test_empty_responses_do_not_change_server_reported_token_count( + self, server_token_parser: InferenceResultParser + ) -> None: + responses = [ + ParsedResponse(perf_ns=100, data=TextResponseData(text="")), + ParsedResponse( + perf_ns=200, + data=TextResponseData(text="hello"), + usage={"prompt_tokens": 10, "completion_tokens": 7}, + ), + ParsedResponse(perf_ns=300, data=TextResponseData(text="")), + ] + + counts = await server_token_parser._compute_server_token_counts(responses) + + assert counts.input == 10 + assert counts.output == 7 + async def test_uses_server_values( self, server_token_parser, request_record, spy_tokenizer ): @@ -640,7 +688,7 @@ async def test_malformed_response_recorded_as_failure_not_parser_crash( assert record.has_error # Honest cause: "no content from server", NOT a parser-internal ValueError. assert record.error.type == "InvalidInferenceResultError" - assert "No responses with actual content" in str(record.error) + assert "No parsed content responses" in str(record.error) assert "Unsupported OpenAI object type" not in str(record.error) assert result.responses == [] diff --git a/tests/unit/server/test_config.py b/tests/unit/server/test_config.py index 45f8c9fe72..6c090bf915 100644 --- a/tests/unit/server/test_config.py +++ b/tests/unit/server/test_config.py @@ -25,6 +25,7 @@ def test_default_config(self): assert config.workers == 1 assert config.ttft == 20.0 assert config.itl == 5.0 + assert config.emit_empty_chat_content is False assert config.log_level == "INFO" assert config.verbose is False assert config.error_rate == 0.0 diff --git a/tests/unit/server/test_utils.py b/tests/unit/server/test_utils.py index 5e3e72029c..f6919fb2be 100644 --- a/tests/unit/server/test_utils.py +++ b/tests/unit/server/test_utils.py @@ -4,7 +4,9 @@ import time +import orjson import pytest +from aiperf_mock_server.config import MockServerConfig from aiperf_mock_server.models import ( ChatCompletionRequest, CompletionRequest, @@ -140,6 +142,33 @@ async def test_stream_chat_completion_basic(self): assert chunks[-1] == b"data: [DONE]\n\n" assert any(b"data:" in chunk for chunk in chunks) + @pytest.mark.asyncio + async def test_stream_chat_completion_emits_opted_in_empty_content_first( + self, monkeypatch + ): + """Emit an explicit empty content delta before generated tokens when enabled.""" + monkeypatch.setattr( + "aiperf_mock_server.utils.server_config", + MockServerConfig(emit_empty_chat_content=True, fast=True), + ) + req = ChatCompletionRequest( + model="test", messages=[Message(role="user", content="Hi")] + ) + ctx = make_ctx(req, "/v1/chat/completions", time.perf_counter()) + + chunks = [ + chunk + async for chunk in stream_chat_completion( + ctx, "/v1/chat/completions", False + ) + ] + + first_payload = orjson.loads(chunks[0].removeprefix(b"data: ").strip()) + assert first_payload["object"] == "chat.completion.chunk" + assert first_payload["choices"] == [ + {"index": 0, "delta": {"role": "assistant", "content": ""}} + ] + @pytest.mark.asyncio async def test_stream_chat_completion_with_reasoning(self): req = ChatCompletionRequest(