feat(endpoints): support explicit empty streaming chunks

docs/cli-options.md

@@ -356,6 +356,11 @@ The API endpoint type to benchmark. Determines request/response format and suppo
 Enable streaming responses. When enabled, the server streams tokens incrementally as they are generated. Automatically disabled if the selected endpoint type does not support streaming. Enables measurement of time-to-first-token (TTFT) and inter-token latency (ITL) metrics.
 <br/>_Flag (no value required)_
 
+#### `--allow-empty-content`
+
+Retain explicitly empty content or reasoning strings as timed responses for streaming OpenAI Chat Completions. This changes TTFT and all metrics using the shared response timeline, but TTFO still requires non-empty output and empty strings add no tokens.
+<br/>_Flag (no value required)_
+
 #### `-u`, `--url` `<list>`
 
 Base URL(s) of the API server(s) to benchmark. Multiple URLs can be specified for load balancing across multiple instances (e.g., `--url http://server1:8000 --url http://server2:8000`). The endpoint path is automatically appended based on `--endpoint-type` (e.g., `/v1/chat/completions` for `chat`). URLs that do not include a scheme (no `://`) have `http://` prepended automatically.
@@ -1891,6 +1896,11 @@ The API endpoint type to benchmark. Determines request/response format and suppo
 Enable streaming responses. When enabled, the server streams tokens incrementally as they are generated. Automatically disabled if the selected endpoint type does not support streaming. Enables measurement of time-to-first-token (TTFT) and inter-token latency (ITL) metrics.
 <br/>_Flag (no value required)_
 
+#### `--allow-empty-content`
+
+Retain explicitly empty content or reasoning strings as timed responses for streaming OpenAI Chat Completions. This changes TTFT and all metrics using the shared response timeline, but TTFO still requires non-empty output and empty strings add no tokens.
+<br/>_Flag (no value required)_
+
 #### `-u`, `--url` `<list>`
 
 Base URL(s) of the API server(s) to benchmark. Multiple URLs can be specified for load balancing across multiple instances (e.g., `--url http://server1:8000 --url http://server2:8000`). The endpoint path is automatically appended based on `--endpoint-type` (e.g., `/v1/chat/completions` for `chat`). URLs that do not include a scheme (no `://`) have `http://` prepended automatically.

docs/metrics-reference.md

@@ -208,7 +208,9 @@ any knowledge of the individual request/response data.
 ## Streaming Metrics
 
 > [!NOTE]
-> All metrics in this section require the `--streaming` flag with a token-producing endpoint and at least one non-empty response chunk.
+> All metrics in this section require `--streaming` with a token-producing endpoint and one or more qualifying parsed responses. By default, a qualifying response has non-empty parsed content. With `--allow-empty-content`, an OpenAI Chat Completions `chat.completion.chunk` with an explicitly empty (`""`) `content`, `reasoning_content`, or `reasoning` string also qualifies. Missing fields, `null`, role/handshake metadata, and other non-content chunks do not qualify.
+
+Qualifying responses form one shared response timeline. Enabling `--allow-empty-content` can therefore change TTFT, TTST, Request Latency, Decode Duration, ICL, ITL, and the per-user output-throughput metrics together. It does not change token accounting: token normalization remains Output Sequence Length (OSL)-based, and an empty decoded string adds no client output or reasoning tokens. When server usage is available, its reported token counts remain authoritative with `--allow-empty-content` enabled.
 
 ### Time to First Token (TTFT)
 
@@ -231,7 +233,7 @@ ttft_seconds = ttft_ns / 1e9
 **Notes:**
 - Includes network latency, queuing time, prompt processing, and generation of the first token (or chunk of tokens).
 - Raw timestamps are in nanoseconds; converted to milliseconds for display and seconds for rate calculations.
-- Response chunks refer to individual messages with non-empty content received during streaming.
+- Uses the first qualifying parsed response on the shared response timeline. By default this is non-empty parsed content; with `--allow-empty-content`, an explicit empty Chat Completions content or reasoning string can be first.
 
 ---
 
@@ -251,7 +253,7 @@ ttst_ms = ttst_ns / 1e6
 ```
 
 **Notes:**
-- Requires at least 2 non-empty response chunks to compute the time between first and second tokens.
+- Requires at least 2 qualifying parsed responses on the shared response timeline. With `--allow-empty-content`, either response can be an explicit empty Chat Completions content or reasoning string.
 - Raw timestamps are in nanoseconds; converted to milliseconds for display.
 
 ---
@@ -276,15 +278,15 @@ ttfo_ms = ttfo_ns / 1e6
 **Notes:**
 - TTFO vs TTFT: Time to First Output (TTFO) measures time to the first non-reasoning token, while Time to First Token (TTFT) measures time to any first token including reasoning tokens. For models without reasoning, TTFO and TTFT are equivalent.
 - Non-reasoning tokens include TextResponseData with non-empty text, or ReasoningResponseData with non-empty content field (regardless of reasoning field).
-- Requires at least one non-empty non-reasoning response chunk.
+- Requires at least one non-empty non-reasoning response chunk. `--allow-empty-content` never makes an empty content or reasoning string satisfy TTFO.
 
 ---
 
 ### Decode Duration
 
 **Type:** [Record Metric](#record-metrics)
 
-Measures the client-observed wall-clock interval between the first and final non-empty streamed content responses. Unlike ITL, this metric is not normalized by the number of generated tokens.
+Measures the client-observed wall-clock interval between the first and final qualifying parsed responses on the shared response timeline. Unlike ITL, this metric is not normalized by the number of generated tokens.
 
 **Formula:**
 ```python
@@ -295,6 +297,7 @@ decode_duration_ms = decode_duration_ns / 1e6
 
 **Notes:**
 - This is an end-to-end client observation. It includes response delivery and client-observed gaps after TTFT; it does not isolate server kernel execution time.
+- With `--allow-empty-content`, explicit empty Chat Completions content or reasoning strings participate in both ends of this interval. For example, a trailing Kimi K3 empty-content chunk extends Decode Duration.
 - Compare Decode Duration with Output Sequence Length when early stopping or a generation-length change is possible.
 - A run can have similar Decode Duration but a much higher ITL if it produces fewer output tokens.
 - Requires valid `time_to_first_token` and `request_latency` metrics.
@@ -305,20 +308,20 @@ decode_duration_ms = decode_duration_ns / 1e6
 
 **Type:** [Record Metric](#record-metrics)
 
-Measures the client-observed wall-clock interval from the first non-empty parsed
-content response until the HTTP response is fully consumed. Unlike Decode
+Measures the client-observed wall-clock interval from the first qualifying parsed
+response until the HTTP response is fully consumed. Unlike Decode
 Duration, it includes time after the last parsed content chunk, such as generation
 that a structured-output parser suppresses before the terminal usage response.
 
 **Formula:**
 ```python
-full_decode_duration_ns = request.end_perf_ns - first_content_response.perf_ns
+full_decode_duration_ns = request.end_perf_ns - first_qualifying_response.perf_ns
 ```
 
 **Notes:**
 - This is a client-observed full-response duration, not server kernel execution time.
 - It includes terminal serialization, transport, and client-processing overhead.
-- It requires an explicit request-end timestamp and at least one non-empty content response.
+- It requires an explicit request-end timestamp and at least one qualifying parsed response. Its end is always `request.end_perf_ns`, even when `--allow-empty-content` changes the shared response timeline.
 - Existing profile exports can reconstruct it as `(request_end_ns - request_start_ns) - time_to_first_token_ns`.
 
 ---
@@ -343,7 +346,7 @@ inter_token_latency_ms = inter_token_latency_ns / 1e6
 
 **Notes:**
 - Requires an output sequence length of at least 2 tokens and valid `time_to_first_token`, `request_latency`, and `output_sequence_length` metrics.
-- ITL is generated-token-normalized decode duration. It is not the total wall-clock decode duration.
+- ITL is OSL-normalized Decode Duration on the shared response timeline. It is not the total wall-clock decode duration, and empty decoded strings do not add client tokens to OSL.
 - Compare runs at equivalent output lengths, or inspect Decode Duration and Output Sequence Length alongside ITL.
 - Streaming chunks can contain multiple tokens. ITL uses token count, while ICL uses chunk arrival timestamps.
 - Result is in seconds when used for throughput calculations (Output Token Throughput Per User).
@@ -356,7 +359,7 @@ inter_token_latency_ms = inter_token_latency_ns / 1e6
 **Type:** [Record Metric](#record-metrics)
 
 Measures the average token interval over the full client-observed decode window,
-from the first non-empty parsed content response through explicit HTTP request
+from the first qualifying parsed response through explicit HTTP request
 completion.
 
 **Formula:**
@@ -368,7 +371,7 @@ full_response_inter_token_latency_ns = (
 
 **Notes:**
 - Requires an output sequence length of at least 2 tokens and a valid Full Decode Duration.
-- Uses the same token normalization as Inter Token Latency while extending the interval through HTTP response completion.
+- Uses the same OSL token normalization as Inter Token Latency while extending the interval through `request.end_perf_ns`; empty decoded strings do not add client tokens.
 - With server-reported token counting, it keeps the duration and token count aligned when a response parser suppresses generated tokens.
 - This remains a client-observed average, not a distribution of raw engine token-to-token timestamps.
 - Streaming chunks can contain multiple tokens. A response delivered entirely in one content chunk may not expose a meaningful post-first-content decode interval.
@@ -387,7 +390,7 @@ inter_chunk_latency = [request.content_responses[i].perf_ns - request.content_re
 ```
 
 **Notes:**
-- Requires at least 2 response chunks.
+- Requires at least 2 qualifying parsed responses on the shared response timeline. With `--allow-empty-content`, explicit empty Chat Completions content or reasoning strings are included.
 - Unlike ITL (which produces a single average), ICL provides the full distribution of inter-chunk times.
 - Useful for detecting variability, jitter, or issues in streaming delivery.
 - Analyzing ICL distributions can reveal batching behavior, scheduling issues, or network variability.
@@ -410,6 +413,7 @@ output_token_throughput_per_user = 1.0 / inter_token_latency_seconds
 
 **Notes:**
 - Computes the inverse of ITL to show tokens per second from an individual user's perspective.
+- Uses the same shared response timeline and OSL normalization as ITL; empty decoded strings do not increase the client token total.
 - Differs from Output Token Throughput (aggregate across all concurrent requests) by focusing on single-request experience.
 - Useful for understanding the user experience independent of concurrency effects.
 - Assumes the output token count describes the content-delivery interval. If a server reports tokens that its response parser suppresses, compare it with the full-response metric pair.
@@ -432,7 +436,8 @@ full_response_output_token_throughput_per_user = (
 
 **Notes:**
 - Computes the inverse of Full-Response Inter Token Latency, mirroring the relationship between Output Token Throughput Per User and Inter Token Latency.
-- Excludes TTFT but measures through full HTTP response completion.
+- Excludes TTFT but measures through `request.end_perf_ns`, regardless of where qualifying responses appear on the shared response timeline.
+- Uses OSL normalization; explicit empty decoded strings do not add client tokens.
 - With server-reported token counting, this approximates raw engine decode TPS when the response parser suppresses generated tokens.
 - It remains a client-observed approximation because AIPerf does not have raw engine first/last-token timestamps.
 - Aggregate latency percentiles and throughput percentiles are not interchangeable: `1 / p75(latency)` describes the slow tail, while `p75(throughput)` describes the fast side of the reciprocal distribution.
@@ -1753,18 +1758,18 @@ total_error_isl = sum(r.error_isl for r in records if not r.valid)
 
 **Type:** [Record Metric](#record-metrics)
 
-Measures the time from request start until the final non-empty parsed content
-response. Usage-only and terminal responses are excluded.
+Measures the time from request start until the final qualifying parsed response
+on the shared response timeline. Usage-only and terminal responses are excluded.
 
 **Formula:**
 ```python
 request_latency_ns = request.content_responses[-1].perf_ns - request.start_perf_ns
 ```
 
 **Notes:**
-- Includes network time, queuing, prompt processing, and content delivery through the final parsed content chunk.
+- Includes network time, queuing, prompt processing, and content delivery through the final qualifying parsed response. With `--allow-empty-content`, a trailing explicit empty Kimi K3 content chunk extends Request Latency (and therefore Decode Duration).
 - It can be shorter than the HTTP lifecycle when a response parser suppresses generated content or terminal usage arrives later.
-- Use `http_req_duration` for the complete HTTP exchange and Full Decode Duration for the post-TTFT interval through request completion.
+- Use `http_req_duration` for the complete HTTP exchange and Full Decode Duration, Full-Response ITL, or Full-Response Output Token Throughput Per User for the post-TTFT interval through `request.end_perf_ns`.
 
 ---
 

docs/tutorials/openai-text-endpoints.md

@@ -45,6 +45,30 @@ aiperf profile \
 ```
 <!-- /aiperf-run-vllm-default-openai-endpoint-server -->
 
+### Include explicit empty streaming responses
+
+Some OpenAI-compatible servers, including Kimi K3 deployments, can emit explicit
+empty content or reasoning strings as part of a Chat Completions stream. Enable
+their inclusion in the response timeline with:
+
+```bash
+aiperf profile \
+    --model Kimi-K3 \
+    --endpoint-type chat \
+    --endpoint /v1/chat/completions \
+    --streaming \
+    --allow-empty-content \
+    --url localhost:8000 \
+    --request-count 20
+```
+
+The flag is off by default because OpenAI-compatible servers can emit
+handshake or role metadata before generation. It admits only explicit empty
+strings in Chat Completions content or reasoning fields, not missing fields,
+`null`, or metadata. All shared response-timeline metrics change together when
+these strings are included. TTFO still waits for non-empty output, and an empty
+decoded string does not increment output-token totals.
+
 **Sample Output (Successful Run):**
 ```
 INFO     Starting AIPerf System
@@ -162,4 +186,4 @@ aiperf profile \
     --request-count 10
 
 ```
-<!-- /aiperf-run-vllm-default-openai-endpoint-server -->
+<!-- /aiperf-run-vllm-default-openai-endpoint-server -->

src/aiperf/common/models/model_endpoint_info.py

@@ -77,6 +77,11 @@ class EndpointInfo(AIPerfBaseModel):
         default=EndpointDefaults.STREAMING,
         description="Whether the endpoint supports streaming.",
     )
+    allow_empty_content: bool = Field(
+        default=EndpointDefaults.ALLOW_EMPTY_CONTENT,
+        description="Whether explicitly empty OpenAI chat content or reasoning "
+        "strings participate in the parsed response timeline.",
+    )
     headers: list[tuple[str, str]] = Field(
         default=[],
         description="Custom URL headers to use for the endpoint.",
@@ -205,6 +210,7 @@ def from_run(cls, run: BenchmarkRun) -> ModelEndpointInfo:
                 type=ep.type,
                 custom_endpoint=getattr(ep, "path", None),
                 streaming=ep.streaming,
+                allow_empty_content=ep.allow_empty_content,
                 base_urls=list(ep.urls),
                 headers=list((getattr(ep, "headers", {}) or {}).items()),
                 extra=list((getattr(ep, "extra", {}) or {}).items()),

src/aiperf/common/models/record_models.py

@@ -1590,10 +1590,11 @@ def end_perf_ns(self) -> int:
 
     @cached_property
     def content_responses(self) -> list[ParsedResponse]:
-        """Get only responses with actual content (data is not None or empty).
+        """Get parsed content responses on the shared metric timeline.
 
-        This excludes usage-only or [DONE] responses that may appear at the end of streaming responses.
-        Useful for timing metrics that should measure content delivery.
+        Explicit empty chat strings may be present when endpoint extraction is
+        configured to retain them. Usage-only, metadata-only, null-data, and
+        [DONE] responses are excluded.
         """
         return [response for response in self.responses if response.data]
 
@@ -1631,7 +1632,7 @@ def create_error_from_invalid(self) -> None:
             err = InvalidInferenceResultError("Invalid inference result")
             if len(self.responses) == 0 or len(self.content_responses) == 0:
                 err.add_note(
-                    "No responses with actual content were received from the server (only usage/metadata, null/empty data, or [DONE] markers)"
+                    "No parsed content responses were received from the server (only usage/metadata, null data, or [DONE] markers)"
                 )
             if self.start_perf_ns <= 0 or self.start_perf_ns >= sys.maxsize:
                 err.add_note(

src/aiperf/config/endpoint.py

@@ -47,6 +47,7 @@ class EndpointDefaults:
     CUSTOM_ENDPOINT = None
     TYPE = EndpointType.CHAT
     STREAMING = False
+    ALLOW_EMPTY_CONTENT = False
     URL = "http://localhost:8000"
     URL_STRATEGY = URLSelectionStrategy.ROUND_ROBIN
     TIMEOUT = 6 * 60 * 60  # 6 hours, match vLLM benchmark default
@@ -200,6 +201,16 @@ def _redact_api_key(self, value: str | None) -> str | None:
         ),
     ]
 
+    allow_empty_content: Annotated[
+        bool,
+        Field(
+            default=EndpointDefaults.ALLOW_EMPTY_CONTENT,
+            description="Retain explicitly empty content or reasoning strings as "
+            "timed responses for streaming OpenAI Chat Completions. This changes "
+            "the shared response timeline without adding output tokens.",
+        ),
+    ]
+
     _streaming_explicitly_set: bool = False
 
     transport: Annotated[

src/aiperf/config/flags/_converter_endpoint.py

@@ -59,6 +59,7 @@ def _endpoint_template_fallback(endpoint: dict[str, Any]) -> None:
     "url_selection_strategy": "url_strategy",
     "endpoint_type": "type",
     "streaming": "streaming",
+    "allow_empty_content": "allow_empty_content",
     "custom_endpoint": "path",
     "api_key": "api_key",
     "timeout_seconds": "timeout",

src/aiperf/config/flags/_section_fields.py

@@ -17,6 +17,7 @@
 ENDPOINT_FIELDS: frozenset[str] = frozenset(
     {
         "api_key",
+        "allow_empty_content",
         "connection_reuse_strategy",
         "custom_endpoint",
         "download_video_content",

src/aiperf/config/flags/cli_config.py

@@ -178,6 +178,20 @@ class CLIConfig(BaseConfig):
         ),
     ] = EndpointDefaults.STREAMING
 
+    allow_empty_content: Annotated[
+        bool,
+        Field(
+            description="Retain explicitly empty content or reasoning strings as "
+            "timed responses for streaming OpenAI Chat Completions. This changes "
+            "TTFT and all metrics using the shared response timeline, but TTFO "
+            "still requires non-empty output and empty strings add no tokens.",
+        ),
+        CLIParameter(
+            name=("--allow-empty-content",),
+            group=Groups.ENDPOINT,
+        ),
+    ] = EndpointDefaults.ALLOW_EMPTY_CONTENT
+
     urls: Annotated[
         list[str],
         Field(

src/aiperf/config/schema/aiperf-config.schema.json

@@ -13013,6 +13013,12 @@
           "title": "Streaming",
           "type": "boolean"
         },
+        "allowEmptyContent": {
+          "default": false,
+          "description": "Retain explicitly empty content or reasoning strings as timed responses for streaming OpenAI Chat Completions. This changes the shared response timeline without adding output tokens.",
+          "title": "Allowemptycontent",
+          "type": "boolean"
+        },
         "transport": {
           "anyOf": [
             {