Evidence case: self-hosted OpenAI-compatible server, client vs server surface (Discussion #26)

conformance/evidence/selfhosted-openai-compatible-reconcile/README.md

@@ -0,0 +1,93 @@
+# Self-hosted OpenAI-compatible server: client surface vs server surface
+
+## Purpose
+
+Pins [standard/known-limits-unprovable-by-surface.md](../../../standard/known-limits-unprovable-by-surface.md)
+section 6 with one live capture, so the boundary can be re-checked by running a file
+instead of by reading a description. From Discussion #26.
+
+On a self-hosted OpenAI-compatible server, the client surface can be missing two things:
+
+1. **A call's usage.** A stream sent without `stream_options.include_usage` comes back
+   with no usage object at all. That is absent, not zero.
+2. **The cache split.** vLLM sends `prompt_tokens_details` only when the server is started
+   with `--enable-prompt-tokens-details` (off by default), so the client is not told how
+   many prompt tokens came from the prefix cache, even when most of them did.
+
+The server surface (`/metrics` counters, two snapshots) recovers the period totals and the
+cached count. It cannot split a gap between two or more absent calls.
+
+## Run
+
+From the repository root:
+
+```bash
+python3 conformance/evidence/selfhosted-openai-compatible-reconcile/run_case.py
+```
+
+Offline, stdlib only.
+
+## Fixture
+
+Live capture, not synthetic. Colab Tesla T4 (driver 580.82.07), vLLM 0.27.1,
+torch 2.13.0+cu130, `Qwen/Qwen2.5-0.5B-Instruct --dtype half`, every other server flag
+default (prefix caching on, prompt token details off). The same request three times, one
+after another ("Name three rivers in Europe.", `max_tokens` 16, `temperature` 0,
+`ignore_eos` true), 2026-10-14T17:40Z:
+
+| call | request | usage the client got |
+|---|---|---|
+| A | plain | 35 prompt / 16 output, `prompt_tokens_details: null` |
+| B | stream, no `stream_options` | none |
+| C | stream, `include_usage: true` | 35 / 16, `prompt_tokens_details: null` |
+
+Server counters, `/metrics` before A and after C, fresh server:
+
+| counter | before | after |
+|---|---|---|
+| `vllm:request_success_total` | 0 | 3 |
+| `vllm:request_prompt_tokens_sum` | 0 | 105 |
+| `vllm:request_generation_tokens_sum` | 0 | 48 |
+| `vllm:prompt_tokens_cached_total` | 0 | 64 |
+
+`fixture.jsonl` holds exactly these values. The raw files they were taken from (full
+`/metrics` snapshots, the client's usage per call, the server log) are published unedited
+at [qaisermehdi3-coder/qvunex@1094fd5](https://github.com/qaisermehdi3-coder/qvunex/tree/1094fd5/benchmarks/data/selfhosted-t4-2026-10-14),
+with their sha256 in `expected.json`.
+
+## Required outcomes
+
+| | outcome |
+|---|---|
+| client | 3 calls, 2 with usage (70 / 32), 1 absent; cache split `UNREPORTED`, cached count `null` |
+| server | 3 requests, 105 / 48, 64 cached prompt tokens |
+| period total | `PROVABLE`: gap +35 / +16 |
+| the absent call's usage | `PROVABLE_BY_ELIMINATION`: with exactly one absent call the whole gap is that call's |
+| same fixture, two calls absent | `UNPROVABLE`: the gap (+70 / +32) is a total only |
+| cache split per call | `UNPROVABLE` from either surface |
+
+Forbidden: absent usage read as zero; the client total presented as the period total;
+a missing `prompt_tokens_details` read as zero cached tokens; per-call attribution of the
+gap with two or more calls absent. A missing counter or one that went down (restart) is
+unknown, never a zero delta.
+
+## What the runner proves
+
+The runner computes both surfaces from the fixture, reconciles them, and checks the result
+against `expected.json`. It then injects five faults (absent usage summed as zero, a missing
+cache split read as zero, per-call attribution with two calls absent, a period total claimed
+while request counts differ, missing or reset counters) and confirms each is rejected. Each
+check was also confirmed to fail when the rule it guards is removed from the runner.
+
+The note on elimination is a small refinement to section 6: per-request attribution is
+unprovable from server sums in general, but with exactly one absent call it is forced.
+
+## Claim boundary
+
+One live run of three identical requests on one small model and one card. It pins what each
+surface reported in that run. It says nothing about speed or cost, or about other servers'
+defaults; SGLang and other engines are not covered by this fixture.
+
+AI-assisted: yes. The runner, fixture extraction and this README were written with an AI
+assistant. The run itself was done live, and every fixture value was checked by script
+against the raw files.

conformance/evidence/selfhosted-openai-compatible-reconcile/expected.json

@@ -0,0 +1,65 @@
+{
+  "case": "selfhosted-openai-compatible-reconcile",
+  "schema": "agentmeasure.evidence/selfhosted-openai-compatible-reconcile/0.1",
+  "fixture_class": "live-capture",
+  "standard_ref": "standard/known-limits-unprovable-by-surface.md section 6",
+  "expected": {
+    "client": {
+      "calls": 3,
+      "usage_present": 2,
+      "usage_absent": 1,
+      "prompt_tokens_reported": 70,
+      "output_tokens_reported": 32,
+      "cache_split": "UNREPORTED",
+      "cached_prompt_tokens": null
+    },
+    "server": {
+      "requests": 3,
+      "prompt_tokens": 105,
+      "output_tokens": 48,
+      "cached_prompt_tokens": 64
+    },
+    "reconciliation": {
+      "requests_gap": 0,
+      "prompt_tokens_gap": 35,
+      "output_tokens_gap": 16,
+      "period_total": "PROVABLE",
+      "absent_call_usage": "PROVABLE_BY_ELIMINATION",
+      "cache_split_per_call": "UNPROVABLE"
+    }
+  },
+  "two_absent_variant": {
+    "description": "same fixture with call A's usage also removed",
+    "absent_call_usage": "UNPROVABLE"
+  },
+  "forbidden": [
+    "absent usage read as zero tokens",
+    "client-reported total presented as the period total",
+    "absent prompt_tokens_details read as zero cached tokens",
+    "per-call attribution of the gap when two or more calls are absent"
+  ],
+  "provenance": {
+    "run": "Colab Tesla T4 (driver 580.82.07), vLLM 0.27.1, torch 2.13.0+cu130, Qwen/Qwen2.5-0.5B-Instruct --dtype half, server flags otherwise default; 2026-10-14T17:40Z",
+    "raw_files_commit": "qaisermehdi3-coder/qvunex@1094fd5",
+    "raw_files": {
+      "before.txt": {
+        "sha256": "4cbdb62b20a8a685a561f6b5483d9b1e14cc7c42591b9ac6b8ec393c89ebe99f",
+        "url": "https://github.com/qaisermehdi3-coder/qvunex/blob/1094fd5/benchmarks/data/selfhosted-t4-2026-10-14/before.txt"
+      },
+      "after.txt": {
+        "sha256": "fd8bd2f56681713ebaba7f7a4ff2a781d25aa6ae2dc8c5c04237c955c3dcfd61",
+        "url": "https://github.com/qaisermehdi3-coder/qvunex/blob/1094fd5/benchmarks/data/selfhosted-t4-2026-10-14/after.txt"
+      },
+      "conditions.json": {
+        "sha256": "1fc8c99eb19e9a92ee858d5128c89b10806d02d5955eddae85bf3d8a119627ae",
+        "url": "https://github.com/qaisermehdi3-coder/qvunex/blob/1094fd5/benchmarks/data/selfhosted-t4-2026-10-14/conditions.json"
+      },
+      "server.log": {
+        "sha256": "9cf9174bdbed7b205fcf910013febca4008466c71fccd1284c8c7d05e128ff2b",
+        "url": "https://github.com/qaisermehdi3-coder/qvunex/blob/1094fd5/benchmarks/data/selfhosted-t4-2026-10-14/server.log"
+      }
+    },
+    "discussion": "https://github.com/roy-tong/AgentMeasure/discussions/26"
+  },
+  "claim_boundary": "One live run of three identical requests on one small model and one card. It pins what each surface reported; it says nothing about speed, cost or any other server's defaults."
+}

conformance/evidence/selfhosted-openai-compatible-reconcile/fixture.jsonl

@@ -0,0 +1,7 @@
+{"surface":"client","call_id":"A","request":"plain","usage":{"prompt_tokens":35,"completion_tokens":16,"total_tokens":51,"completion_tokens_details":null,"prompt_tokens_details":null}}
+{"surface":"client","call_id":"B","request":"stream, no stream_options","usage":null}
+{"surface":"client","call_id":"C","request":"stream, stream_options.include_usage=true","usage":{"prompt_tokens":35,"completion_tokens":16,"total_tokens":51,"completion_tokens_details":null,"prompt_tokens_details":null}}
+{"surface":"server","metric":"vllm:request_success_total","before":0.0,"after":3.0}
+{"surface":"server","metric":"vllm:request_prompt_tokens_sum","before":0.0,"after":105.0}
+{"surface":"server","metric":"vllm:request_generation_tokens_sum","before":0.0,"after":48.0}
+{"surface":"server","metric":"vllm:prompt_tokens_cached_total","before":0.0,"after":64.0}

conformance/evidence/selfhosted-openai-compatible-reconcile/run_case.py

@@ -0,0 +1,184 @@
+#!/usr/bin/env python3
+"""Executable evidence case: self-hosted OpenAI-compatible server, two surfaces.
+
+Run from the repository root:
+    python3 conformance/evidence/selfhosted-openai-compatible-reconcile/run_case.py
+
+Pins standard/known-limits-unprovable-by-surface.md section 6 with one live
+capture (vLLM 0.27.1 on a T4). The client surface is what three responses
+carried; the server surface is four /metrics counters before and after.
+Offline, stdlib only. Includes fault injections that a conformant reader
+must reject.
+"""
+from __future__ import annotations
+
+import copy
+import json
+import sys
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+FIXTURE = HERE / "fixture.jsonl"
+EXPECTED = HERE / "expected.json"
+
+SERVER_METRICS = {
+    "requests": "vllm:request_success_total",
+    "prompt_tokens": "vllm:request_prompt_tokens_sum",
+    "output_tokens": "vllm:request_generation_tokens_sum",
+    "cached_prompt_tokens": "vllm:prompt_tokens_cached_total",
+}
+
+
+def load_jsonl(path: Path) -> list[dict]:
+    rows = []
+    for n, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
+        if not line.strip():
+            continue
+        row = json.loads(line)
+        if not isinstance(row, dict):
+            raise ValueError(f"{path.name}:{n}: each row must be an object")
+        rows.append(row)
+    return rows
+
+
+def client_surface(rows: list[dict]) -> dict:
+    """What the responses said. Absent usage stays absent: it is counted, never summed as 0."""
+    calls = [r for r in rows if r.get("surface") == "client"]
+    present = [r for r in calls if isinstance(r.get("usage"), dict)]
+    split_reported = [r for r in present if isinstance(r["usage"].get("prompt_tokens_details"), dict)]
+    cached = None
+    if present and len(split_reported) == len(present):
+        cached = sum(r["usage"]["prompt_tokens_details"].get("cached_tokens") or 0 for r in present)
+    return {
+        "calls": len(calls),
+        "usage_present": len(present),
+        "usage_absent": len(calls) - len(present),
+        "prompt_tokens_reported": sum(r["usage"]["prompt_tokens"] for r in present),
+        "output_tokens_reported": sum(r["usage"]["completion_tokens"] for r in present),
+        "cache_split": "REPORTED" if cached is not None else "UNREPORTED",
+        "cached_prompt_tokens": cached,
+    }
+
+
+def server_surface(rows: list[dict]) -> dict:
+    """Counter deltas. A missing counter or one that went down is unknown (None), never 0."""
+    by_metric = {r["metric"]: r for r in rows if r.get("surface") == "server"}
+    out = {}
+    for key, metric in SERVER_METRICS.items():
+        r = by_metric.get(metric)
+        if r is None or r.get("before") is None or r.get("after") is None or r["after"] < r["before"]:
+            out[key] = None
+        else:
+            out[key] = int(round(r["after"] - r["before"]))
+    return out
+
+
+def reconcile(client: dict, server: dict) -> dict:
+    gap = lambda s, c: None if s is None else s - c
+    req_gap = gap(server["requests"], client["calls"])
+    out = {
+        "requests_gap": req_gap,
+        "prompt_tokens_gap": gap(server["prompt_tokens"], client["prompt_tokens_reported"]),
+        "output_tokens_gap": gap(server["output_tokens"], client["output_tokens_reported"]),
+    }
+    complete = req_gap == 0 and None not in (server["prompt_tokens"], server["output_tokens"])
+    out["period_total"] = "PROVABLE" if complete else "UNPROVABLE"
+    if not complete or client["usage_absent"] == 0:
+        out["absent_call_usage"] = "NOT_APPLICABLE" if complete else "UNPROVABLE"
+    elif client["usage_absent"] == 1:
+        out["absent_call_usage"] = "PROVABLE_BY_ELIMINATION"   # the whole gap is that one call
+    else:
+        out["absent_call_usage"] = "UNPROVABLE"                # sums cannot split it between calls
+    out["cache_split_per_call"] = "REPORTED" if client["cache_split"] == "REPORTED" else "UNPROVABLE"
+    return out
+
+
+def violations(client: dict, recon: dict, server: dict) -> list[str]:
+    """The forbidden outcomes from expected.json, as checks on a reader's output."""
+    errs = []
+    if client["usage_absent"] and recon.get("prompt_tokens_gap") in (0, None) and server["prompt_tokens"]:
+        errs.append("absent usage read as zero tokens (no gap left for the absent call)")
+    if recon.get("period_total") == "PROVABLE" and recon.get("requests_gap") != 0:
+        errs.append("client-reported total presented as the period total")
+    if client["cache_split"] == "UNREPORTED" and client.get("cached_prompt_tokens") is not None:
+        errs.append("absent prompt_tokens_details read as zero cached tokens")
+    if client["usage_absent"] >= 2 and recon.get("absent_call_usage") != "UNPROVABLE":
+        errs.append("per-call attribution of the gap with two or more calls absent")
+    return errs
+
+
+def main() -> int:
+    rows = load_jsonl(FIXTURE)
+    exp = json.loads(EXPECTED.read_text(encoding="utf-8"))["expected"]
+    ok = True
+
+    def check(name: str, cond: bool, detail: str = "") -> None:
+        nonlocal ok
+        ok = ok and cond
+        print(("PASS " if cond else "FAIL ") + name + ("" if cond else f"  [{detail}]"))
+
+    client, server = client_surface(rows), server_surface(rows)
+    recon = reconcile(client, server)
+    check("client surface: 2 of 3 calls carry usage, 70 / 32, cache split unreported",
+          client == exp["client"], f"got {client}")
+    check("server surface: 3 requests, 105 / 48, 64 prompt tokens from the prefix cache",
+          server == exp["server"], f"got {server}")
+    check("reconciliation: gap 35 / 16 is the one absent call; period total provable",
+          recon == exp["reconciliation"], f"got {recon}")
+    check("no forbidden outcome in the reference reading", not violations(client, recon, server))
+
+    # Fault 1: a reader that sums absent usage as 0 and reports the client total
+    # as if it were the period total.
+    bad = copy.deepcopy(recon)
+    bad.update(prompt_tokens_gap=0, output_tokens_gap=0)
+    check("fault rejected: absent usage read as zero",
+          bool(violations(client, bad, server)))
+
+    # Fault 2: a reader that turns the missing prompt_tokens_details into a cache of 0.
+    bad_client = dict(client, cached_prompt_tokens=0)
+    check("fault rejected: missing cache split read as 0 cached tokens",
+          bool(violations(bad_client, recon, server)))
+
+    # Fault 3: with two calls absent, attributing the gap per call is not allowed.
+    two = copy.deepcopy(rows)
+    next(r for r in two if r.get("call_id") == "A")["usage"] = None
+    c2 = client_surface(two)
+    r2 = reconcile(c2, server)
+    check("two absent calls: per-call usage is UNPROVABLE (gap 70 / 32 is a total only)",
+          r2["absent_call_usage"] == "UNPROVABLE" and r2["prompt_tokens_gap"] == 70
+          and r2["output_tokens_gap"] == 32 and not violations(c2, r2, server), f"got {r2}")
+    check("fault rejected: per-call attribution claimed with two calls absent",
+          bool(violations(c2, dict(r2, absent_call_usage="PROVABLE_BY_ELIMINATION"), server)))
+
+    # Fault 4: another client used the server in the window (4 requests counted
+    # for 3 calls); claiming the period total is then not allowed.
+    busy = copy.deepcopy(rows)
+    next(r for r in busy if r.get("metric") == "vllm:request_success_total")["after"] = 4.0
+    s5 = server_surface(busy)
+    r5 = reconcile(client, s5)
+    check("a request the client did not make leaves the period total UNPROVABLE",
+          r5["requests_gap"] == 1 and r5["period_total"] == "UNPROVABLE", f"got {r5}")
+    check("fault rejected: period total claimed while requests do not match",
+          bool(violations(client, dict(r5, period_total="PROVABLE"), s5)))
+
+    # A counter missing from a snapshot is unknown, and so is the period total.
+    gone = [r for r in rows if r.get("metric") not in
+            ("vllm:request_success_total", "vllm:prompt_tokens_cached_total")]
+    s3 = server_surface(gone)
+    r3 = reconcile(client, s3)
+    check("missing counters stay unknown: no request count means no provable period total",
+          s3["requests"] is None and s3["cached_prompt_tokens"] is None
+          and r3["period_total"] == "UNPROVABLE", f"got {s3} {r3}")
+    restarted = copy.deepcopy(rows)
+    next(r for r in restarted if r.get("metric") == "vllm:request_prompt_tokens_sum")["before"] = 500.0
+    s4 = server_surface(restarted)
+    check("a counter that went down (server restart) is unknown, not a negative or zero delta",
+          s4["prompt_tokens"] is None and reconcile(client, s4)["period_total"] == "UNPROVABLE",
+          f"got {s4}")
+
+    print("\nCONFORMANT" if ok else "\nNON-CONFORMANT")
+    return 0 if ok else 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())