diff --git a/conformance/evidence/selfhosted-openai-compatible-reconcile/README.md b/conformance/evidence/selfhosted-openai-compatible-reconcile/README.md new file mode 100644 index 0000000..9004f0b --- /dev/null +++ b/conformance/evidence/selfhosted-openai-compatible-reconcile/README.md @@ -0,0 +1,93 @@ +# Self-hosted OpenAI-compatible server: client surface vs server surface + +## Purpose + +Pins [standard/known-limits-unprovable-by-surface.md](../../../standard/known-limits-unprovable-by-surface.md) +section 6 with one live capture, so the boundary can be re-checked by running a file +instead of by reading a description. From Discussion #26. + +On a self-hosted OpenAI-compatible server, the client surface can be missing two things: + +1. **A call's usage.** A stream sent without `stream_options.include_usage` comes back + with no usage object at all. That is absent, not zero. +2. **The cache split.** vLLM sends `prompt_tokens_details` only when the server is started + with `--enable-prompt-tokens-details` (off by default), so the client is not told how + many prompt tokens came from the prefix cache, even when most of them did. + +The server surface (`/metrics` counters, two snapshots) recovers the period totals and the +cached count. It cannot split a gap between two or more absent calls. + +## Run + +From the repository root: + +```bash +python3 conformance/evidence/selfhosted-openai-compatible-reconcile/run_case.py +``` + +Offline, stdlib only. + +## Fixture + +Live capture, not synthetic. Colab Tesla T4 (driver 580.82.07), vLLM 0.27.1, +torch 2.13.0+cu130, `Qwen/Qwen2.5-0.5B-Instruct --dtype half`, every other server flag +default (prefix caching on, prompt token details off). The same request three times, one +after another ("Name three rivers in Europe.", `max_tokens` 16, `temperature` 0, +`ignore_eos` true), 2026-10-14T17:40Z: + +| call | request | usage the client got | +|---|---|---| +| A | plain | 35 prompt / 16 output, `prompt_tokens_details: null` | +| B | stream, no `stream_options` | none | +| C | stream, `include_usage: true` | 35 / 16, `prompt_tokens_details: null` | + +Server counters, `/metrics` before A and after C, fresh server: + +| counter | before | after | +|---|---|---| +| `vllm:request_success_total` | 0 | 3 | +| `vllm:request_prompt_tokens_sum` | 0 | 105 | +| `vllm:request_generation_tokens_sum` | 0 | 48 | +| `vllm:prompt_tokens_cached_total` | 0 | 64 | + +`fixture.jsonl` holds exactly these values. The raw files they were taken from (full +`/metrics` snapshots, the client's usage per call, the server log) are published unedited +at [qaisermehdi3-coder/qvunex@1094fd5](https://github.com/qaisermehdi3-coder/qvunex/tree/1094fd5/benchmarks/data/selfhosted-t4-2026-10-14), +with their sha256 in `expected.json`. + +## Required outcomes + +| | outcome | +|---|---| +| client | 3 calls, 2 with usage (70 / 32), 1 absent; cache split `UNREPORTED`, cached count `null` | +| server | 3 requests, 105 / 48, 64 cached prompt tokens | +| period total | `PROVABLE`: gap +35 / +16 | +| the absent call's usage | `PROVABLE_BY_ELIMINATION`: with exactly one absent call the whole gap is that call's | +| same fixture, two calls absent | `UNPROVABLE`: the gap (+70 / +32) is a total only | +| cache split per call | `UNPROVABLE` from either surface | + +Forbidden: absent usage read as zero; the client total presented as the period total; +a missing `prompt_tokens_details` read as zero cached tokens; per-call attribution of the +gap with two or more calls absent. A missing counter or one that went down (restart) is +unknown, never a zero delta. + +## What the runner proves + +The runner computes both surfaces from the fixture, reconciles them, and checks the result +against `expected.json`. It then injects five faults (absent usage summed as zero, a missing +cache split read as zero, per-call attribution with two calls absent, a period total claimed +while request counts differ, missing or reset counters) and confirms each is rejected. Each +check was also confirmed to fail when the rule it guards is removed from the runner. + +The note on elimination is a small refinement to section 6: per-request attribution is +unprovable from server sums in general, but with exactly one absent call it is forced. + +## Claim boundary + +One live run of three identical requests on one small model and one card. It pins what each +surface reported in that run. It says nothing about speed or cost, or about other servers' +defaults; SGLang and other engines are not covered by this fixture. + +AI-assisted: yes. The runner, fixture extraction and this README were written with an AI +assistant. The run itself was done live, and every fixture value was checked by script +against the raw files. diff --git a/conformance/evidence/selfhosted-openai-compatible-reconcile/expected.json b/conformance/evidence/selfhosted-openai-compatible-reconcile/expected.json new file mode 100644 index 0000000..abb9cdc --- /dev/null +++ b/conformance/evidence/selfhosted-openai-compatible-reconcile/expected.json @@ -0,0 +1,65 @@ +{ + "case": "selfhosted-openai-compatible-reconcile", + "schema": "agentmeasure.evidence/selfhosted-openai-compatible-reconcile/0.1", + "fixture_class": "live-capture", + "standard_ref": "standard/known-limits-unprovable-by-surface.md section 6", + "expected": { + "client": { + "calls": 3, + "usage_present": 2, + "usage_absent": 1, + "prompt_tokens_reported": 70, + "output_tokens_reported": 32, + "cache_split": "UNREPORTED", + "cached_prompt_tokens": null + }, + "server": { + "requests": 3, + "prompt_tokens": 105, + "output_tokens": 48, + "cached_prompt_tokens": 64 + }, + "reconciliation": { + "requests_gap": 0, + "prompt_tokens_gap": 35, + "output_tokens_gap": 16, + "period_total": "PROVABLE", + "absent_call_usage": "PROVABLE_BY_ELIMINATION", + "cache_split_per_call": "UNPROVABLE" + } + }, + "two_absent_variant": { + "description": "same fixture with call A's usage also removed", + "absent_call_usage": "UNPROVABLE" + }, + "forbidden": [ + "absent usage read as zero tokens", + "client-reported total presented as the period total", + "absent prompt_tokens_details read as zero cached tokens", + "per-call attribution of the gap when two or more calls are absent" + ], + "provenance": { + "run": "Colab Tesla T4 (driver 580.82.07), vLLM 0.27.1, torch 2.13.0+cu130, Qwen/Qwen2.5-0.5B-Instruct --dtype half, server flags otherwise default; 2026-10-14T17:40Z", + "raw_files_commit": "qaisermehdi3-coder/qvunex@1094fd5", + "raw_files": { + "before.txt": { + "sha256": "4cbdb62b20a8a685a561f6b5483d9b1e14cc7c42591b9ac6b8ec393c89ebe99f", + "url": "https://github.com/qaisermehdi3-coder/qvunex/blob/1094fd5/benchmarks/data/selfhosted-t4-2026-10-14/before.txt" + }, + "after.txt": { + "sha256": "fd8bd2f56681713ebaba7f7a4ff2a781d25aa6ae2dc8c5c04237c955c3dcfd61", + "url": "https://github.com/qaisermehdi3-coder/qvunex/blob/1094fd5/benchmarks/data/selfhosted-t4-2026-10-14/after.txt" + }, + "conditions.json": { + "sha256": "1fc8c99eb19e9a92ee858d5128c89b10806d02d5955eddae85bf3d8a119627ae", + "url": "https://github.com/qaisermehdi3-coder/qvunex/blob/1094fd5/benchmarks/data/selfhosted-t4-2026-10-14/conditions.json" + }, + "server.log": { + "sha256": "9cf9174bdbed7b205fcf910013febca4008466c71fccd1284c8c7d05e128ff2b", + "url": "https://github.com/qaisermehdi3-coder/qvunex/blob/1094fd5/benchmarks/data/selfhosted-t4-2026-10-14/server.log" + } + }, + "discussion": "https://github.com/roy-tong/AgentMeasure/discussions/26" + }, + "claim_boundary": "One live run of three identical requests on one small model and one card. It pins what each surface reported; it says nothing about speed, cost or any other server's defaults." +} diff --git a/conformance/evidence/selfhosted-openai-compatible-reconcile/fixture.jsonl b/conformance/evidence/selfhosted-openai-compatible-reconcile/fixture.jsonl new file mode 100644 index 0000000..a2bb3d9 --- /dev/null +++ b/conformance/evidence/selfhosted-openai-compatible-reconcile/fixture.jsonl @@ -0,0 +1,7 @@ +{"surface":"client","call_id":"A","request":"plain","usage":{"prompt_tokens":35,"completion_tokens":16,"total_tokens":51,"completion_tokens_details":null,"prompt_tokens_details":null}} +{"surface":"client","call_id":"B","request":"stream, no stream_options","usage":null} +{"surface":"client","call_id":"C","request":"stream, stream_options.include_usage=true","usage":{"prompt_tokens":35,"completion_tokens":16,"total_tokens":51,"completion_tokens_details":null,"prompt_tokens_details":null}} +{"surface":"server","metric":"vllm:request_success_total","before":0.0,"after":3.0} +{"surface":"server","metric":"vllm:request_prompt_tokens_sum","before":0.0,"after":105.0} +{"surface":"server","metric":"vllm:request_generation_tokens_sum","before":0.0,"after":48.0} +{"surface":"server","metric":"vllm:prompt_tokens_cached_total","before":0.0,"after":64.0} diff --git a/conformance/evidence/selfhosted-openai-compatible-reconcile/run_case.py b/conformance/evidence/selfhosted-openai-compatible-reconcile/run_case.py new file mode 100644 index 0000000..4401001 --- /dev/null +++ b/conformance/evidence/selfhosted-openai-compatible-reconcile/run_case.py @@ -0,0 +1,184 @@ +#!/usr/bin/env python3 +"""Executable evidence case: self-hosted OpenAI-compatible server, two surfaces. + +Run from the repository root: + python3 conformance/evidence/selfhosted-openai-compatible-reconcile/run_case.py + +Pins standard/known-limits-unprovable-by-surface.md section 6 with one live +capture (vLLM 0.27.1 on a T4). The client surface is what three responses +carried; the server surface is four /metrics counters before and after. +Offline, stdlib only. Includes fault injections that a conformant reader +must reject. +""" +from __future__ import annotations + +import copy +import json +import sys +from pathlib import Path + +HERE = Path(__file__).resolve().parent +FIXTURE = HERE / "fixture.jsonl" +EXPECTED = HERE / "expected.json" + +SERVER_METRICS = { + "requests": "vllm:request_success_total", + "prompt_tokens": "vllm:request_prompt_tokens_sum", + "output_tokens": "vllm:request_generation_tokens_sum", + "cached_prompt_tokens": "vllm:prompt_tokens_cached_total", +} + + +def load_jsonl(path: Path) -> list[dict]: + rows = [] + for n, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1): + if not line.strip(): + continue + row = json.loads(line) + if not isinstance(row, dict): + raise ValueError(f"{path.name}:{n}: each row must be an object") + rows.append(row) + return rows + + +def client_surface(rows: list[dict]) -> dict: + """What the responses said. Absent usage stays absent: it is counted, never summed as 0.""" + calls = [r for r in rows if r.get("surface") == "client"] + present = [r for r in calls if isinstance(r.get("usage"), dict)] + split_reported = [r for r in present if isinstance(r["usage"].get("prompt_tokens_details"), dict)] + cached = None + if present and len(split_reported) == len(present): + cached = sum(r["usage"]["prompt_tokens_details"].get("cached_tokens") or 0 for r in present) + return { + "calls": len(calls), + "usage_present": len(present), + "usage_absent": len(calls) - len(present), + "prompt_tokens_reported": sum(r["usage"]["prompt_tokens"] for r in present), + "output_tokens_reported": sum(r["usage"]["completion_tokens"] for r in present), + "cache_split": "REPORTED" if cached is not None else "UNREPORTED", + "cached_prompt_tokens": cached, + } + + +def server_surface(rows: list[dict]) -> dict: + """Counter deltas. A missing counter or one that went down is unknown (None), never 0.""" + by_metric = {r["metric"]: r for r in rows if r.get("surface") == "server"} + out = {} + for key, metric in SERVER_METRICS.items(): + r = by_metric.get(metric) + if r is None or r.get("before") is None or r.get("after") is None or r["after"] < r["before"]: + out[key] = None + else: + out[key] = int(round(r["after"] - r["before"])) + return out + + +def reconcile(client: dict, server: dict) -> dict: + gap = lambda s, c: None if s is None else s - c + req_gap = gap(server["requests"], client["calls"]) + out = { + "requests_gap": req_gap, + "prompt_tokens_gap": gap(server["prompt_tokens"], client["prompt_tokens_reported"]), + "output_tokens_gap": gap(server["output_tokens"], client["output_tokens_reported"]), + } + complete = req_gap == 0 and None not in (server["prompt_tokens"], server["output_tokens"]) + out["period_total"] = "PROVABLE" if complete else "UNPROVABLE" + if not complete or client["usage_absent"] == 0: + out["absent_call_usage"] = "NOT_APPLICABLE" if complete else "UNPROVABLE" + elif client["usage_absent"] == 1: + out["absent_call_usage"] = "PROVABLE_BY_ELIMINATION" # the whole gap is that one call + else: + out["absent_call_usage"] = "UNPROVABLE" # sums cannot split it between calls + out["cache_split_per_call"] = "REPORTED" if client["cache_split"] == "REPORTED" else "UNPROVABLE" + return out + + +def violations(client: dict, recon: dict, server: dict) -> list[str]: + """The forbidden outcomes from expected.json, as checks on a reader's output.""" + errs = [] + if client["usage_absent"] and recon.get("prompt_tokens_gap") in (0, None) and server["prompt_tokens"]: + errs.append("absent usage read as zero tokens (no gap left for the absent call)") + if recon.get("period_total") == "PROVABLE" and recon.get("requests_gap") != 0: + errs.append("client-reported total presented as the period total") + if client["cache_split"] == "UNREPORTED" and client.get("cached_prompt_tokens") is not None: + errs.append("absent prompt_tokens_details read as zero cached tokens") + if client["usage_absent"] >= 2 and recon.get("absent_call_usage") != "UNPROVABLE": + errs.append("per-call attribution of the gap with two or more calls absent") + return errs + + +def main() -> int: + rows = load_jsonl(FIXTURE) + exp = json.loads(EXPECTED.read_text(encoding="utf-8"))["expected"] + ok = True + + def check(name: str, cond: bool, detail: str = "") -> None: + nonlocal ok + ok = ok and cond + print(("PASS " if cond else "FAIL ") + name + ("" if cond else f" [{detail}]")) + + client, server = client_surface(rows), server_surface(rows) + recon = reconcile(client, server) + check("client surface: 2 of 3 calls carry usage, 70 / 32, cache split unreported", + client == exp["client"], f"got {client}") + check("server surface: 3 requests, 105 / 48, 64 prompt tokens from the prefix cache", + server == exp["server"], f"got {server}") + check("reconciliation: gap 35 / 16 is the one absent call; period total provable", + recon == exp["reconciliation"], f"got {recon}") + check("no forbidden outcome in the reference reading", not violations(client, recon, server)) + + # Fault 1: a reader that sums absent usage as 0 and reports the client total + # as if it were the period total. + bad = copy.deepcopy(recon) + bad.update(prompt_tokens_gap=0, output_tokens_gap=0) + check("fault rejected: absent usage read as zero", + bool(violations(client, bad, server))) + + # Fault 2: a reader that turns the missing prompt_tokens_details into a cache of 0. + bad_client = dict(client, cached_prompt_tokens=0) + check("fault rejected: missing cache split read as 0 cached tokens", + bool(violations(bad_client, recon, server))) + + # Fault 3: with two calls absent, attributing the gap per call is not allowed. + two = copy.deepcopy(rows) + next(r for r in two if r.get("call_id") == "A")["usage"] = None + c2 = client_surface(two) + r2 = reconcile(c2, server) + check("two absent calls: per-call usage is UNPROVABLE (gap 70 / 32 is a total only)", + r2["absent_call_usage"] == "UNPROVABLE" and r2["prompt_tokens_gap"] == 70 + and r2["output_tokens_gap"] == 32 and not violations(c2, r2, server), f"got {r2}") + check("fault rejected: per-call attribution claimed with two calls absent", + bool(violations(c2, dict(r2, absent_call_usage="PROVABLE_BY_ELIMINATION"), server))) + + # Fault 4: another client used the server in the window (4 requests counted + # for 3 calls); claiming the period total is then not allowed. + busy = copy.deepcopy(rows) + next(r for r in busy if r.get("metric") == "vllm:request_success_total")["after"] = 4.0 + s5 = server_surface(busy) + r5 = reconcile(client, s5) + check("a request the client did not make leaves the period total UNPROVABLE", + r5["requests_gap"] == 1 and r5["period_total"] == "UNPROVABLE", f"got {r5}") + check("fault rejected: period total claimed while requests do not match", + bool(violations(client, dict(r5, period_total="PROVABLE"), s5))) + + # A counter missing from a snapshot is unknown, and so is the period total. + gone = [r for r in rows if r.get("metric") not in + ("vllm:request_success_total", "vllm:prompt_tokens_cached_total")] + s3 = server_surface(gone) + r3 = reconcile(client, s3) + check("missing counters stay unknown: no request count means no provable period total", + s3["requests"] is None and s3["cached_prompt_tokens"] is None + and r3["period_total"] == "UNPROVABLE", f"got {s3} {r3}") + restarted = copy.deepcopy(rows) + next(r for r in restarted if r.get("metric") == "vllm:request_prompt_tokens_sum")["before"] = 500.0 + s4 = server_surface(restarted) + check("a counter that went down (server restart) is unknown, not a negative or zero delta", + s4["prompt_tokens"] is None and reconcile(client, s4)["period_total"] == "UNPROVABLE", + f"got {s4}") + + print("\nCONFORMANT" if ok else "\nNON-CONFORMANT") + return 0 if ok else 1 + + +if __name__ == "__main__": + sys.exit(main())