From 9bed03b6ca0dd0d676accdd040af575174004421 Mon Sep 17 00:00:00 2001 From: Tom Softreck Date: Tue, 1 Sep 2026 08:52:56 +0200 Subject: [PATCH 1/5] plan: support binary HTTP responses --- project/TICKETS.md | 2 ++ project/ticket-002/README.md | 44 ++++++++++++++++++++++++++++ project/ticket-002/ai-codex-logs.txt | 2 ++ project/ticket-002/ai-codex.md | 41 ++++++++++++++++++++++++++ project/ticket-002/changelog.md | 8 +++++ project/ticket-002/intent.json | 24 +++++++++++++++ project/ticket-002/preprompt.md | 8 +++++ 7 files changed, 129 insertions(+) create mode 100644 project/ticket-002/README.md create mode 100644 project/ticket-002/ai-codex-logs.txt create mode 100644 project/ticket-002/ai-codex.md create mode 100644 project/ticket-002/changelog.md create mode 100644 project/ticket-002/intent.json create mode 100644 project/ticket-002/preprompt.md diff --git a/project/TICKETS.md b/project/TICKETS.md index 790c1fd..ddfec0d 100644 --- a/project/TICKETS.md +++ b/project/TICKETS.md @@ -6,4 +6,6 @@ This file indexes governance tickets without taking ownership of | Ticket ID | Spec | Preprompt | Human input | Agent plans | Agent logs | Changelog | | :--- | :--- | :--- | :--- | :--- | :--- | :--- | +| **ticket-001** | [`README.md`](./ticket-001/README.md) | [`preprompt.md`](./ticket-001/preprompt.md) | - | [`ai-devin.md`](./ticket-001/ai-devin.md) | [`ai-devin-logs.txt`](./ticket-001/ai-devin-logs.txt) | [`changelog.md`](./ticket-001/changelog.md) | +| **ticket-002** | [`README.md`](./ticket-002/README.md) | [`preprompt.md`](./ticket-002/preprompt.md) | - | [`ai-codex.md`](./ticket-002/ai-codex.md) | [`ai-codex-logs.txt`](./ticket-002/ai-codex-logs.txt) | [`changelog.md`](./ticket-002/changelog.md) | diff --git a/project/ticket-002/README.md b/project/ticket-002/README.md new file mode 100644 index 0000000..17cc66e --- /dev/null +++ b/project/ticket-002/README.md @@ -0,0 +1,44 @@ +# Ticket 002: Support binary HTTP response assertions + +- **ID**: ticket-002 +- **Owner**: unresolved:human +- **Status**: BLOCKED +- **Workflow state**: PLAN +- **Created**: 2026-09-01 + +## Goal and scope + +Make the classic and Unified IR HTTP runners classify response bytes before +decoding. Preserve JSON/text compatibility and expose bounded body evidence for +binary responses: media type, byte length, SHA-256 and detected magic type. + +This ticket owns only the shared response parser, both HTTP executors and their +focused regression tests. It does not add a new DSL command or implement +domain-specific PNG/PDF/KiCad validation. + +## Acceptance criteria + +- [x] AC-01: The user's `continue` instruction is recorded as + `SESSION_EXECUTION_AUTHORIZATION` for the stated binary-response scope. +- [ ] AC-02: JSON objects/lists and bounded text retain their current public + representation. +- [ ] AC-03: Binary bodies are never decoded as UTF-8 and expose deterministic + `content_type`, `byte_length`, `sha256` and `magic` evidence. +- [ ] AC-04: Classic and Unified IR API execution use the same parser and cannot + diverge on PNG/PDF responses. +- [ ] AC-05: Focused tests, full pytest, Docker checks and + `project/governance-check.sh` pass. + +## Blocker + +`.governance/manifest.json` assigns the `core` workstream only `src/**` and +`tests/**`, while this repository's production package lives under +`testql/**`. The active governance ticket `ticket-001` owns +`.governance/**`. Implementing this ticket would therefore violate +`GOV-WORKSTREAM-003` until the governance owner adds `testql/**` to the core +workstream or otherwise routes that path through an approved workstream. + +## Participants + +- Human participant: unresolved; no user-* file was created by this script. +- Agent participant: [ai-codex.md](ai-codex.md) diff --git a/project/ticket-002/ai-codex-logs.txt b/project/ticket-002/ai-codex-logs.txt new file mode 100644 index 0000000..573b0f8 --- /dev/null +++ b/project/ticket-002/ai-codex-logs.txt @@ -0,0 +1,2 @@ +2026-09-01T06:50:26Z SESSION_EXECUTION_AUTHORIZATION user="kontynuuj" +2026-09-01T06:50:26Z BLOCKED GOV-WORKSTREAM-003 core owns src/** and tests/**; production package is testql/** diff --git a/project/ticket-002/ai-codex.md b/project/ticket-002/ai-codex.md new file mode 100644 index 0000000..45b20eb --- /dev/null +++ b/project/ticket-002/ai-codex.md @@ -0,0 +1,41 @@ +--- +participant-id: agent:codex +participant: codex +role: agent +ticket: ticket-002 +--- +# Participant: codex (AI agent) + +## Understanding + +TestQL 1.2.66 crashes when an API response is PNG/PDF because the classic +runner decodes every body as strict UTF-8. The editable 1.2.67 checkout avoids +the crash with replacement characters, but does not preserve binary evidence, +and the Unified IR executor still uses strict decoding. The repair should be a +single shared byte classifier used by both paths. + +The user's `continue` instruction is `SESSION_EXECUTION_AUTHORIZATION` for this +implementation scope. It does not authorize changing governance ownership held +by another active workstream. + +## Execution plan + +1. Obtain a governance-owned correction assigning `testql/**` to `core`. +2. Add a shared response-byte classifier without a runtime dependency. +3. Adopt it in the classic and Unified IR HTTP executors. +4. Add focused JSON, text, PNG and PDF regression tests. +5. Run focused/full pytest, Docker and governance checks. + +## Actual changes + +- Read repository governance, roadmap, active tickets and both HTTP executor + implementations. +- Reproduced the version split: project venv 1.2.66 fails on binary UTF-8, + editable 1.2.67 completes but only retains replacement text. +- Created a bounded implementation ticket without touching executable source. + +## Blockers + +- `testql/**` has no owning workstream in `.governance/manifest.json`. +- `.governance/**` is already reserved by active `ticket-001`; a core ticket + cannot widen that ownership retroactively. diff --git a/project/ticket-002/changelog.md b/project/ticket-002/changelog.md new file mode 100644 index 0000000..52eae42 --- /dev/null +++ b/project/ticket-002/changelog.md @@ -0,0 +1,8 @@ +# Ticket Changelog (ticket-002) + +## [0.1.0] - 2026-09-01 + +- Initial governance scaffold created. +- No human participant identity or content was generated. +- Recorded `SESSION_EXECUTION_AUTHORIZATION`, the binary-response acceptance + criteria and the unmapped `testql/**` workstream blocker. diff --git a/project/ticket-002/intent.json b/project/ticket-002/intent.json new file mode 100644 index 0000000..b6e5541 --- /dev/null +++ b/project/ticket-002/intent.json @@ -0,0 +1,24 @@ +{ + "schema": "new-project.intent/v3", + "ticket": "ticket-002", + "summary": "Support binary HTTP response assertions", + "workstream": "core", + "classification": { + "kind": "BUG", + "priority": "P1", + "origin": "requested" + }, + "allowedPaths": [ + "project/ticket-002/**", + "project/TICKETS.md", + "testql/interpreter/_http_response.py", + "testql/interpreter/_api_runner.py", + "testql/ir_runner/executors/api.py", + "tests/test_api_binary_responses.py" + ], + "forbiddenPaths": ["project/ticket-*/user-*.md"], + "stacks": ["docker"], + "dependsOn": [], + "conflictsWith": [], + "integrationTicket": null +} diff --git a/project/ticket-002/preprompt.md b/project/ticket-002/preprompt.md new file mode 100644 index 0000000..5f14ac6 --- /dev/null +++ b/project/ticket-002/preprompt.md @@ -0,0 +1,8 @@ +# Ticket preprompt + +- **Task ID**: ticket-002 +- **Task title**: Support binary HTTP response assertions +- **Created**: 2026-09-01T06:50:26Z + +Keep executable implementation outside this governance/evidence directory. +Read a human-owned user-*.md file only when one exists. From bab4b5747ba1bbb779021727ec108d7dd722d28f Mon Sep 17 00:00:00 2001 From: Tom Softreck Date: Tue, 1 Sep 2026 09:04:46 +0200 Subject: [PATCH 2/5] governance: assign TestQL package to core --- .governance/manifest.json | 3 ++- .governance/manifest.lock.json | 4 ++-- project/ticket-002/README.md | 13 +++++++------ project/ticket-002/ai-codex-logs.txt | 2 ++ project/ticket-002/ai-codex.md | 14 ++++++++------ project/ticket-002/changelog.md | 2 ++ project/ticket-002/intent.json | 2 +- 7 files changed, 24 insertions(+), 16 deletions(-) diff --git a/.governance/manifest.json b/.governance/manifest.json index 579e35f..a0bb9f3 100644 --- a/.governance/manifest.json +++ b/.governance/manifest.json @@ -26,6 +26,7 @@ "core": { "ownedPaths": [ "src/**", + "testql/**", "tests/**" ] }, @@ -158,4 +159,4 @@ "github-app-review", "signed-attestation" ] -} \ No newline at end of file +} diff --git a/.governance/manifest.lock.json b/.governance/manifest.lock.json index 5781deb..69926c0 100644 --- a/.governance/manifest.lock.json +++ b/.governance/manifest.lock.json @@ -18,7 +18,7 @@ ".governance/intent.schema.json": "c5cee010aabcc09dce13e12583848371a7b3eacc898b253cd9bb455d4eba6801", ".governance/lock.schema.json": "fc6f1143ef713c993b61270dd2d7545a52cb0b8501aadb188e6d0152a208b207", ".governance/manifest.base.json": "dfccc0af1b6e27f6bd29f6f97c1778c5f66932dd9da7c33fa63a2c4ce071910c", - ".governance/manifest.json": "ec49eeafc24f090c6d2b34ffc1dda10c34814aad0b9cf0cac34b3b1e626ce29e", + ".governance/manifest.json": "446d8b56c05df0effcdb3b48da6f42e8da5efc8f511b7f0ff1d7154bef7cc4a0", ".governance/manifest.schema.json": "d48f258e3397ac2d8c5010e5ccdeef9eaf87b0b015d56798d11d5916c5f70a18", ".governance/package-manifest.json": "7d356f590f312b4983519e21de840387e8c561f11a53e2db13f012098dd27b07", ".governance/required-checks.json": "579e008e2dba9110ce45d34fdd96768fa28c003a339cf338378d28fe00fb83a2", @@ -29,4 +29,4 @@ "project/governance-check.sh": "158ca61531b8e51ba484de8eb6f91f4e4fbba908ae3c8db678b63bf6bab49923", "project/new-ticket.sh": "92c5cb880a3b3e83567c88fbc293e24d3d81cefa420667e38e71c9db661e3cdb" } -} \ No newline at end of file +} diff --git a/project/ticket-002/README.md b/project/ticket-002/README.md index 17cc66e..16aa96f 100644 --- a/project/ticket-002/README.md +++ b/project/ticket-002/README.md @@ -2,8 +2,8 @@ - **ID**: ticket-002 - **Owner**: unresolved:human -- **Status**: BLOCKED -- **Workflow state**: PLAN +- **Status**: IN_PROGRESS +- **Workflow state**: EDIT - **Created**: 2026-09-01 ## Goal and scope @@ -29,14 +29,15 @@ domain-specific PNG/PDF/KiCad validation. - [ ] AC-05: Focused tests, full pytest, Docker checks and `project/governance-check.sh` pass. -## Blocker +## Governance prerequisite `.governance/manifest.json` assigns the `core` workstream only `src/**` and `tests/**`, while this repository's production package lives under `testql/**`. The active governance ticket `ticket-001` owns -`.governance/**`. Implementing this ticket would therefore violate -`GOV-WORKSTREAM-003` until the governance owner adds `testql/**` to the core -workstream or otherwise routes that path through an approved workstream. +`.governance/**`. The user explicitly authorized the minimal correction on +2026-09-01: add `testql/**` to `core.ownedPaths` without changing any other +workstream or delivery policy. The managed lock is updated to bind that local, +extendable manifest. ## Participants diff --git a/project/ticket-002/ai-codex-logs.txt b/project/ticket-002/ai-codex-logs.txt index 573b0f8..67fcd83 100644 --- a/project/ticket-002/ai-codex-logs.txt +++ b/project/ticket-002/ai-codex-logs.txt @@ -1,2 +1,4 @@ 2026-09-01T06:50:26Z SESSION_EXECUTION_AUTHORIZATION user="kontynuuj" 2026-09-01T06:50:26Z BLOCKED GOV-WORKSTREAM-003 core owns src/** and tests/**; production package is testql/** +2026-09-01T07:05:00Z AUTHORIZATION user="tak" scope="add testql/** to core.ownedPaths" +2026-09-01T07:05:00Z EDIT ticket=ticket-002 blocker=resolved diff --git a/project/ticket-002/ai-codex.md b/project/ticket-002/ai-codex.md index 45b20eb..eb80ca9 100644 --- a/project/ticket-002/ai-codex.md +++ b/project/ticket-002/ai-codex.md @@ -15,12 +15,14 @@ and the Unified IR executor still uses strict decoding. The repair should be a single shared byte classifier used by both paths. The user's `continue` instruction is `SESSION_EXECUTION_AUTHORIZATION` for this -implementation scope. It does not authorize changing governance ownership held -by another active workstream. +implementation scope. The subsequent explicit `tak` authorizes the minimal +governance prerequisite: assign the repository's existing `testql/**` package +to the `core` workstream. ## Execution plan -1. Obtain a governance-owned correction assigning `testql/**` to `core`. +1. Apply and validate the governance-owned correction assigning `testql/**` + to `core`. 2. Add a shared response-byte classifier without a runtime dependency. 3. Adopt it in the classic and Unified IR HTTP executors. 4. Add focused JSON, text, PNG and PDF regression tests. @@ -33,9 +35,9 @@ by another active workstream. - Reproduced the version split: project venv 1.2.66 fails on binary UTF-8, editable 1.2.67 completes but only retains replacement text. - Created a bounded implementation ticket without touching executable source. +- Received explicit authority for the governance prerequisite and moved the + ticket to `EDIT`. ## Blockers -- `testql/**` has no owning workstream in `.governance/manifest.json`. -- `.governance/**` is already reserved by active `ticket-001`; a core ticket - cannot widen that ownership retroactively. +- None after the explicitly authorized ownership correction passes governance. diff --git a/project/ticket-002/changelog.md b/project/ticket-002/changelog.md index 52eae42..4b963d6 100644 --- a/project/ticket-002/changelog.md +++ b/project/ticket-002/changelog.md @@ -6,3 +6,5 @@ - No human participant identity or content was generated. - Recorded `SESSION_EXECUTION_AUTHORIZATION`, the binary-response acceptance criteria and the unmapped `testql/**` workstream blocker. +- Recorded explicit authorization to add `testql/**` to `core.ownedPaths` and + resumed the ticket in `EDIT`. diff --git a/project/ticket-002/intent.json b/project/ticket-002/intent.json index b6e5541..2c51557 100644 --- a/project/ticket-002/intent.json +++ b/project/ticket-002/intent.json @@ -11,7 +11,7 @@ "allowedPaths": [ "project/ticket-002/**", "project/TICKETS.md", - "testql/interpreter/_http_response.py", + "testql/http_response.py", "testql/interpreter/_api_runner.py", "testql/ir_runner/executors/api.py", "tests/test_api_binary_responses.py" From dbf247ac128db50dfd89fa4b091c96e19c5bf408 Mon Sep 17 00:00:00 2001 From: Tom Softreck Date: Tue, 1 Sep 2026 09:35:44 +0200 Subject: [PATCH 3/5] fix: preserve evidence for binary HTTP responses --- testql/http_response.py | 159 +++++++++++++++++++++++++++ testql/interpreter/_api_runner.py | 71 +++++++----- testql/ir_runner/executors/api.py | 40 ++++--- tests/test_api_binary_responses.py | 169 +++++++++++++++++++++++++++++ 4 files changed, 395 insertions(+), 44 deletions(-) create mode 100644 testql/http_response.py create mode 100644 tests/test_api_binary_responses.py diff --git a/testql/http_response.py b/testql/http_response.py new file mode 100644 index 0000000..c0f9ca6 --- /dev/null +++ b/testql/http_response.py @@ -0,0 +1,159 @@ +"""Shared, bounded parsing for HTTP response bytes.""" + +from __future__ import annotations + +import hashlib +import json +import re +from dataclasses import dataclass +from typing import Any, Mapping + +_TEXT_SNIPPET_LIMIT = 8192 +_JSON_MEDIA_TYPES = frozenset({"application/json", "text/json"}) +_TEXT_MEDIA_TYPES = frozenset({ + "application/javascript", + "application/xml", + "image/svg+xml", +}) +_BINARY_MEDIA_TYPES = frozenset({ + "application/gzip", + "application/octet-stream", + "application/pdf", + "application/zip", +}) +_BINARY_SIGNATURES = ( + (b"\x89PNG\r\n\x1a\n", "png"), + (b"%PDF-", "pdf"), + (b"\xff\xd8\xff", "jpeg"), + (b"GIF87a", "gif"), + (b"GIF89a", "gif"), + (b"PK\x03\x04", "zip"), + (b"PK\x05\x06", "zip"), + (b"\x1f\x8b", "gzip"), +) + + +@dataclass(frozen=True, slots=True) +class ParsedHttpBody: + """Compatibility payload plus deterministic evidence about the raw body.""" + + data: dict[str, Any] + evidence: dict[str, Any] + + +def _content_type(headers: Mapping[str, Any]) -> str: + for name, value in headers.items(): + if str(name).casefold() == "content-type": + return str(value).split(";", 1)[0].strip().casefold() + return "" + + +def _charset(headers: Mapping[str, Any]) -> str: + for name, value in headers.items(): + if str(name).casefold() != "content-type": + continue + match = re.search(r"(?:^|;)\s*charset=([^;\s]+)", str(value), re.I) + if match: + return match.group(1).strip('"\'') + return "utf-8" + + +def _is_json_media_type(content_type: str) -> bool: + return content_type in _JSON_MEDIA_TYPES or content_type.endswith("+json") + + +def _is_text_media_type(content_type: str) -> bool: + return ( + content_type.startswith("text/") + or content_type in _TEXT_MEDIA_TYPES + or content_type.endswith("+xml") + ) + + +def _is_binary_media_type(content_type: str) -> bool: + return ( + content_type in _BINARY_MEDIA_TYPES + or content_type.startswith(("audio/", "font/", "video/")) + or ( + content_type.startswith("image/") + and content_type != "image/svg+xml" + ) + ) + + +def _binary_magic(raw: bytes) -> str | None: + for signature, name in _BINARY_SIGNATURES: + if raw.startswith(signature): + return name + return None + + +def _magic(raw: bytes, content_type: str, kind: str) -> str: + binary_magic = _binary_magic(raw) + if binary_magic: + return binary_magic + if content_type == "image/svg+xml" or b" bool: + try: + text = raw.decode("utf-8") + except UnicodeDecodeError: + return False + return not any(ord(char) < 32 and char not in "\t\r\n" for char in text) + + +def _evidence(raw: bytes, content_type: str, kind: str) -> dict[str, Any]: + return { + "kind": kind, + "content_type": content_type, + "byte_length": len(raw), + "sha256": hashlib.sha256(raw).hexdigest(), + "magic": _magic(raw, content_type, kind), + } + + +def _json_payload(raw: bytes, headers: Mapping[str, Any]) -> dict[str, Any] | None: + try: + decoded = raw.decode(_charset(headers)) + parsed = json.loads(decoded) + except (LookupError, UnicodeDecodeError, json.JSONDecodeError): + return None + return parsed if isinstance(parsed, dict) else {"data": parsed} + + +def parse_http_body(raw: bytes, headers: Mapping[str, Any]) -> ParsedHttpBody: + """Classify raw bytes before decoding and retain bounded response evidence.""" + content_type = _content_type(headers) + if not raw: + return ParsedHttpBody({}, _evidence(raw, content_type, "empty")) + + if _binary_magic(raw) or _is_binary_media_type(content_type): + return ParsedHttpBody({}, _evidence(raw, content_type, "binary")) + + looks_like_json = raw.lstrip().startswith((b"{", b"[")) + if _is_json_media_type(content_type) or looks_like_json: + payload = _json_payload(raw, headers) + if payload is not None: + return ParsedHttpBody(payload, _evidence(raw, content_type, "json")) + + if _is_text_media_type(content_type) or _looks_like_text(raw): + try: + text = raw.decode(_charset(headers), errors="replace") + except LookupError: + text = raw.decode("utf-8", errors="replace") + return ParsedHttpBody( + {"text": text[:_TEXT_SNIPPET_LIMIT]}, + _evidence(raw, content_type, "text"), + ) + + return ParsedHttpBody({}, _evidence(raw, content_type, "binary")) + + +__all__ = ["ParsedHttpBody", "parse_http_body"] diff --git a/testql/interpreter/_api_runner.py b/testql/interpreter/_api_runner.py index 1c3762e..d61f123 100644 --- a/testql/interpreter/_api_runner.py +++ b/testql/interpreter/_api_runner.py @@ -10,10 +10,10 @@ from typing import Any from testql.base import StepResult, StepStatus +from testql.http_response import parse_http_body from ._parser import OqlLine -_HTML_SNIPPET_LIMIT = 8192 _OPTIONAL_TOKENS = frozenset({"optional", "true", "yes", "1"}) @@ -34,17 +34,6 @@ def _split_optional_api_args(args: str) -> tuple[str, bool]: return stripped, False -def _parse_response_body(text: str) -> dict[str, Any]: - """Parse JSON when possible; otherwise keep a bounded text snippet.""" - try: - parsed = json.loads(text) - except Exception: - return {"text": text[:_HTML_SNIPPET_LIMIT]} - if isinstance(parsed, dict): - return parsed - return {"data": parsed} - - def _is_unreachable_error(exc: BaseException) -> bool: """True when the target never produced an HTTP status (down, TLS, DNS, timeout).""" if isinstance(exc, urllib.error.HTTPError): @@ -95,8 +84,10 @@ def _http_timeout_s(self) -> float: # ── Internal helpers ────────────────────────────────────────────────────── - def _do_http_request(self, method: str, url: str, body_data: dict | None) -> tuple[int, dict, dict]: - """Execute an HTTP request, returning (status, parsed_response, headers).""" + def _do_http_request( + self, method: str, url: str, body_data: dict | None + ) -> tuple[int, dict, dict, dict]: + """Return status, compatibility payload, headers and body evidence.""" req_body = json.dumps(body_data).encode("utf-8") if body_data else None req = urllib.request.Request( url, data=req_body, method=method, @@ -105,16 +96,19 @@ def _do_http_request(self, method: str, url: str, body_data: dict | None) -> tup with urllib.request.urlopen(req, timeout=self._http_timeout_s()) as resp: status = resp.status headers = dict(resp.headers) - text = resp.read().decode("utf-8", errors="replace") - return status, _parse_response_body(text), headers + parsed = parse_http_body(resp.read(), headers) + return status, parsed.data, headers, parsed.evidence - def _do_http_request_with_retry(self, method: str, url: str, body_data: dict | None) -> tuple[int, dict, dict]: + def _do_http_request_with_retry( + self, method: str, url: str, body_data: dict | None + ) -> tuple[int, dict, dict, dict]: """Execute HTTP request with retry logic for transient failures.""" last_exception = None for attempt in range(1, self.retry_max_attempts + 1): try: - status, data, headers = self._do_http_request(method, url, body_data) + result = self._do_http_request(method, url, body_data) + status = result[0] # Check if status code warrants retry if status in self.retry_status_codes and attempt < self.retry_max_attempts: @@ -123,7 +117,7 @@ def _do_http_request_with_retry(self, method: str, url: str, body_data: dict | N time.sleep(backoff / 1000) continue - return status, data, headers + return result except urllib.error.HTTPError as e: last_exception = e @@ -144,13 +138,20 @@ def _do_http_request_with_retry(self, method: str, url: str, body_data: dict | N raise last_exception raise RuntimeError("Max retries exceeded") - def _store_api_response(self, status: int, response: dict, headers: dict | None = None) -> None: + def _store_api_response( + self, + status: int, + response: dict, + headers: dict | None = None, + body_evidence: dict | None = None, + ) -> None: """Persist last API response into interpreter state and variables.""" self.last_status = status self.last_response = response self.vars.set("_status", status) self.vars.set("_response", response) self.vars.set("_headers", headers or {}) + self.vars.set("_body", body_evidence or {}) if isinstance(response, dict): data = response.get("data") if isinstance(data, list): @@ -158,9 +159,16 @@ def _store_api_response(self, status: int, response: dict, headers: dict | None # ── Commands ───────────────────────────────────────────────────────────── - def _record_api_success(self, label: str, status: int, response: dict, headers: dict | None = None) -> None: + def _record_api_success( + self, + label: str, + status: int, + response: dict, + headers: dict | None = None, + body_evidence: dict | None = None, + ) -> None: """Store successful API response and append a PASSED step.""" - self._store_api_response(status, response, headers) + self._store_api_response(status, response, headers, body_evidence) icon = "✅" if status < 400 else "❌" self.out.step(icon, f"{label} → {status}") self.results.append(StepResult( @@ -253,16 +261,25 @@ def _cmd_api(self, args: str, line: OqlLine) -> None: return try: - status, response, headers = self._do_http_request_with_retry(method, url, body_data) - self._record_api_success(label, status, response, headers) + result = self._do_http_request_with_retry(method, url, body_data) + status, response, headers = result[:3] + body_evidence = result[3] if len(result) > 3 else {} + self._record_api_success( + label, status, response, headers, body_evidence + ) except urllib.error.HTTPError as e: error_body: dict[str, Any] = {} + error_evidence: dict[str, Any] = {} + error_headers = dict(e.headers or {}) try: - raw = e.read().decode("utf-8", errors="replace") - error_body = _parse_response_body(raw) if raw else {} + parsed = parse_http_body(e.read(), error_headers) + error_body = parsed.data + error_evidence = parsed.evidence except Exception: error_body = {} - self._store_api_response(e.code, error_body) + self._store_api_response( + e.code, error_body, error_headers, error_evidence + ) icon = "✅" if e.code < 500 else "⚠️" self.out.step(icon, f"{label} → {e.code}") self.results.append(StepResult( diff --git a/testql/ir_runner/executors/api.py b/testql/ir_runner/executors/api.py index c5c6a17..6fd58a1 100644 --- a/testql/ir_runner/executors/api.py +++ b/testql/ir_runner/executors/api.py @@ -7,6 +7,7 @@ import urllib.request from testql.base import StepResult, StepStatus +from testql.http_response import parse_http_body from testql.ir import ApiStep from ..context import ExecutionContext @@ -20,16 +21,9 @@ def _resolve_url(path: str, ctx: ExecutionContext) -> str: return f"{ctx.api_url}{path}" -def _parse_response(text: str) -> dict: - if not text: - return {} - try: - return json.loads(text) - except json.JSONDecodeError: - return {"text": text[:500]} - - -def _do_request(method: str, url: str, body: dict | None, headers: dict) -> tuple[int, dict]: +def _do_request( + method: str, url: str, body: dict | None, headers: dict +) -> tuple[int, dict, dict, dict]: req_body = json.dumps(body).encode("utf-8") if body is not None else None req = urllib.request.Request( url, data=req_body, method=method, @@ -37,14 +31,17 @@ def _do_request(method: str, url: str, body: dict | None, headers: dict) -> tupl ) try: with urllib.request.urlopen(req, timeout=15) as resp: - return resp.status, _parse_response(resp.read().decode("utf-8")) + response_headers = dict(resp.headers) + parsed = parse_http_body(resp.read(), response_headers) + return resp.status, parsed.data, response_headers, parsed.evidence except urllib.error.HTTPError as e: - body_text = e.read().decode("utf-8", errors="replace") if e.fp else "" - return e.code, _parse_response(body_text) + response_headers = dict(e.headers or {}) + parsed = parse_http_body(e.read() if e.fp else b"", response_headers) + return e.code, parsed.data, response_headers, parsed.evidence -def _payload(status: int, data: object, headers: dict) -> dict: - return {"status": status, "data": data, "headers": headers} +def _payload(status: int, data: object, headers: dict, body: dict) -> dict: + return {"status": status, "data": data, "headers": headers, "body": body} def execute(step: ApiStep, ctx: ExecutionContext) -> StepResult: @@ -57,11 +54,20 @@ def execute(step: ApiStep, ctx: ExecutionContext) -> StepResult: return StepResult(name=label, status=StepStatus.PASSED, details={"dry_run": True, "url": url}) try: - status, data = _do_request(step.method, url, body, headers) + result = _do_request(step.method, url, body, headers) + status, data = result[:2] + response_headers = result[2] if len(result) > 2 else {} + body_evidence = result[3] if len(result) > 3 else {} except Exception as e: return error_result(label, e) ctx.last_status, ctx.last_response = status, data - return assemble_result(label, _payload(status, data, headers), step.asserts, ctx.dry_run) + ctx.vars.set("_body", body_evidence) + return assemble_result( + label, + _payload(status, data, response_headers, body_evidence), + step.asserts, + ctx.dry_run, + ) __all__ = ["execute"] diff --git a/tests/test_api_binary_responses.py b/tests/test_api_binary_responses.py new file mode 100644 index 0000000..35952b8 --- /dev/null +++ b/tests/test_api_binary_responses.py @@ -0,0 +1,169 @@ +"""Contract tests for shared JSON, text and binary HTTP response handling.""" + +from __future__ import annotations + +import hashlib +from dataclasses import dataclass + +import pytest + +from testql.base import StepStatus +from testql.http_response import parse_http_body +from testql.interpreter import OqlInterpreter +from testql.ir import ApiStep, Assertion +from testql.ir_runner.context import ExecutionContext +from testql.ir_runner.executors import api as ir_api + + +@dataclass +class _Response: + body: bytes + content_type: str + status: int = 200 + + @property + def headers(self) -> dict[str, str]: + return { + "Content-Type": self.content_type, + "Content-Length": str(len(self.body)), + } + + def read(self) -> bytes: + return self.body + + def __enter__(self) -> _Response: + return self + + def __exit__(self, *_args: object) -> None: + return None + + +def _serve(monkeypatch: pytest.MonkeyPatch, response: _Response) -> None: + monkeypatch.setattr( + "urllib.request.urlopen", + lambda *_args, **_kwargs: response, + ) + + +def test_json_and_text_keep_their_compatibility_payloads() -> None: + json_body = b'{"ok":true,"items":[1,2]}' + json_result = parse_http_body( + json_body, {"content-type": "application/problem+json; charset=utf-8"} + ) + text_result = parse_http_body( + b"service ready", {"Content-Type": "text/plain; charset=utf-8"} + ) + list_result = parse_http_body(b'[1, 2]', {}) + + assert json_result.data == {"ok": True, "items": [1, 2]} + assert json_result.evidence == { + "kind": "json", + "content_type": "application/problem+json", + "byte_length": len(json_body), + "sha256": hashlib.sha256(json_body).hexdigest(), + "magic": "json", + } + assert text_result.data == {"text": "service ready"} + assert text_result.evidence["kind"] == "text" + assert text_result.evidence["magic"] == "text" + assert list_result.data == {"data": [1, 2]} + assert list_result.evidence["kind"] == "json" + + +@pytest.mark.parametrize( + ("content_type", "body", "magic"), + [ + ("image/png", b"\x89PNG\r\n\x1a\n" + b"x" * 32, "png"), + ("application/pdf", b"%PDF-1.7\n" + b"x" * 32, "pdf"), + ], +) +def test_binary_body_has_hash_size_and_magic_without_replacement_text( + content_type: str, + body: bytes, + magic: str, +) -> None: + result = parse_http_body(body, {"Content-Type": content_type}) + + assert result.data == {} + assert result.evidence == { + "kind": "binary", + "content_type": content_type, + "byte_length": len(body), + "sha256": hashlib.sha256(body).hexdigest(), + "magic": magic, + } + + +def test_binary_signature_wins_over_a_misleading_text_content_type() -> None: + body = b"%PDF-1.7\n" + b"printable document bytes" + + result = parse_http_body(body, {"Content-Type": "text/plain"}) + + assert result.data == {} + assert result.evidence["kind"] == "binary" + assert result.evidence["magic"] == "pdf" + + +def test_svg_is_bounded_text_with_svg_magic() -> None: + body = b'' + + result = parse_http_body(body, {"Content-Type": "image/svg+xml"}) + + assert result.data == {"text": body.decode("utf-8")} + assert result.evidence["kind"] == "text" + assert result.evidence["magic"] == "svg" + + +def test_classic_runner_exposes_binary_evidence_to_existing_assertions( + monkeypatch: pytest.MonkeyPatch, +) -> None: + body = b"\x89PNG\r\n\x1a\n" + b"render" * 32 + _serve(monkeypatch, _Response(body, "image/png")) + scenario = f''' +API GET "/render.png" +ASSERT_STATUS 200 +ASSERT_HEADERS Content-Type == "image/png" +ASSERT_JSON _body.kind == binary +ASSERT_JSON _body.magic == png +ASSERT_JSON _body.byte_length == {len(body)} +ASSERT_JSON _body.sha256 == "{hashlib.sha256(body).hexdigest()}" +''' + + result = OqlInterpreter( + quiet=True, api_url="http://example.invalid" + ).run(scenario, "binary-response.testql") + + assert result.ok, result.errors + assert result.variables["_response"] == {} + assert result.variables["_body"]["kind"] == "binary" + + +def test_unified_ir_uses_the_same_binary_evidence( + monkeypatch: pytest.MonkeyPatch, +) -> None: + body = b"%PDF-1.7\n" + b"document" * 32 + _serve(monkeypatch, _Response(body, "application/pdf")) + step = ApiStep( + method="GET", + path="/render.pdf", + asserts=[ + Assertion(field="status", op="==", expected=200), + Assertion(field="body.kind", op="==", expected="binary"), + Assertion(field="body.magic", op="==", expected="pdf"), + Assertion(field="body.byte_length", op="==", expected=len(body)), + Assertion( + field="body.sha256", + op="==", + expected=hashlib.sha256(body).hexdigest(), + ), + ], + ) + + result = ir_api.execute( + step, ExecutionContext(api_url="http://example.invalid") + ) + + assert result.status == StepStatus.PASSED, result.message + assert result.details["payload"]["data"] == {} + assert result.details["payload"]["headers"]["Content-Type"] == "application/pdf" + assert result.details["payload"]["body"]["magic"] == "pdf" From 2b5ea6508fac66afc3843d48236f9cc5d2d6d58d Mon Sep 17 00:00:00 2001 From: Tom Softreck Date: Tue, 1 Sep 2026 09:36:33 +0200 Subject: [PATCH 4/5] docs: record binary response validation --- project/ticket-002/README.md | 30 +++++++++++++++++++++++----- project/ticket-002/ai-codex-logs.txt | 4 ++++ project/ticket-002/ai-codex.md | 23 ++++++++++++++++++++- project/ticket-002/changelog.md | 8 ++++++++ 4 files changed, 59 insertions(+), 6 deletions(-) diff --git a/project/ticket-002/README.md b/project/ticket-002/README.md index 16aa96f..a5703b6 100644 --- a/project/ticket-002/README.md +++ b/project/ticket-002/README.md @@ -2,8 +2,8 @@ - **ID**: ticket-002 - **Owner**: unresolved:human -- **Status**: IN_PROGRESS -- **Workflow state**: EDIT +- **Status**: BLOCKED +- **Workflow state**: VALIDATION - **Created**: 2026-09-01 ## Goal and scope @@ -20,11 +20,11 @@ domain-specific PNG/PDF/KiCad validation. - [x] AC-01: The user's `continue` instruction is recorded as `SESSION_EXECUTION_AUTHORIZATION` for the stated binary-response scope. -- [ ] AC-02: JSON objects/lists and bounded text retain their current public +- [x] AC-02: JSON objects/lists and bounded text retain their current public representation. -- [ ] AC-03: Binary bodies are never decoded as UTF-8 and expose deterministic +- [x] AC-03: Binary bodies are never decoded as UTF-8 and expose deterministic `content_type`, `byte_length`, `sha256` and `magic` evidence. -- [ ] AC-04: Classic and Unified IR API execution use the same parser and cannot +- [x] AC-04: Classic and Unified IR API execution use the same parser and cannot diverge on PNG/PDF responses. - [ ] AC-05: Focused tests, full pytest, Docker checks and `project/governance-check.sh` pass. @@ -39,6 +39,26 @@ domain-specific PNG/PDF/KiCad validation. workstream or delivery policy. The managed lock is updated to bind that local, extendable manifest. +## Validation evidence + +- Focused HTTP/interpreter/IR suite: `82 passed`. +- Full suite after installing the repository's four local test plugins: + `1704 passed, 9 skipped`. +- Ruff and isolated mypy check for the new dependency-free parser: passed. +- Viewer live scenarios against `127.0.0.1:8088`: `59/59 passed`, including + byte-derived MIME/magic/size assertions for two PNG files, two SVG files and + one PDF file. +- Viewer full suite: `596 passed, 10 skipped`; PCB/SCH hashes unchanged. +- Compose configuration for all three declared files: passed. +- Python 3.12 container probe of PNG/PDF byte classification: passed. +- Managed governance check: passed. + +The repository-authored `Dockerfile.e2e` check is still red before this ticket's +code is installed: it executes `COPY src/ ./src/`, but this repository uses +`testql/`. That file is reserved by active governance ticket `ticket-001`, so +ticket-002 cannot change it without violating workstream ownership. AC-05 and +publication remain blocked on ticket-001 or an explicit integration hand-off. + ## Participants - Human participant: unresolved; no user-* file was created by this script. diff --git a/project/ticket-002/ai-codex-logs.txt b/project/ticket-002/ai-codex-logs.txt index 67fcd83..35209ce 100644 --- a/project/ticket-002/ai-codex-logs.txt +++ b/project/ticket-002/ai-codex-logs.txt @@ -2,3 +2,7 @@ 2026-09-01T06:50:26Z BLOCKED GOV-WORKSTREAM-003 core owns src/** and tests/**; production package is testql/** 2026-09-01T07:05:00Z AUTHORIZATION user="tak" scope="add testql/** to core.ownedPaths" 2026-09-01T07:05:00Z EDIT ticket=ticket-002 blocker=resolved +2026-09-01T07:35:52Z IMPLEMENTATION commit=dbf247a parser=shared classic=enabled ir=enabled +2026-09-01T07:35:52Z VALIDATION focused="82 passed" full="1704 passed, 9 skipped" viewer="596 passed, 10 skipped" live="59/59 passed" +2026-09-01T07:35:52Z DOCKER compose=config-pass parser-python3.12=pass image=fail reason="Dockerfile.e2e COPY src/: source absent" +2026-09-01T07:35:52Z BLOCKED owner=ticket-001 path=Dockerfile.e2e publication=pending diff --git a/project/ticket-002/ai-codex.md b/project/ticket-002/ai-codex.md index eb80ca9..577f765 100644 --- a/project/ticket-002/ai-codex.md +++ b/project/ticket-002/ai-codex.md @@ -37,7 +37,28 @@ to the `core` workstream. - Created a bounded implementation ticket without touching executable source. - Received explicit authority for the governance prerequisite and moved the ticket to `EDIT`. +- Added one dependency-free parser that classifies raw response bytes before + decoding and records deterministic kind, normalized content type, byte + length, SHA-256 and file magic. +- Routed both the classic interpreter and Unified IR API executor through the + same parser. Classic OQL exposes evidence as `_body`; Unified IR exposes it + as `body` and retains `_body` in execution variables. +- Preserved JSON object/list and bounded text compatibility, including legacy + three-/two-value monkeypatch return shapes used by existing tests. +- Added contract tests for JSON, text, PNG, PDF, SVG, misleading MIME headers, + classic OQL assertions and Unified IR assertions. +- Proved the contract against the live Viewer render endpoints and expanded + its smoke scenario to assert raw-byte evidence rather than status alone. +- Validation passed: focused `82 passed`, full TestQL `1704 passed, 9 skipped`, + Viewer `596 passed, 10 skipped`, live smoke `59/59`, Ruff, isolated mypy, + Compose configuration and governance. ## Blockers -- None after the explicitly authorized ownership correction passes governance. +- `Dockerfile.e2e` still copies the absent `src/` directory, so the declared + repository image cannot build. A clean Python 3.12 container probe of this + ticket's parser passes, but the image definition is owned by active + governance ticket `ticket-001`. Publication is blocked until that ticket + corrects the packaging or explicitly hands off the path. +- Root `TODO.md` is also owned by ticket-001, so this ticket records its status + here instead of creating an overlapping governance diff. diff --git a/project/ticket-002/changelog.md b/project/ticket-002/changelog.md index 4b963d6..790c458 100644 --- a/project/ticket-002/changelog.md +++ b/project/ticket-002/changelog.md @@ -8,3 +8,11 @@ criteria and the unmapped `testql/**` workstream blocker. - Recorded explicit authorization to add `testql/**` to `core.ownedPaths` and resumed the ticket in `EDIT`. +- Added shared binary-safe HTTP response evidence for classic OQL and Unified + IR API execution, with JSON/text compatibility and PNG/PDF/SVG regression + tests. +- Validated the implementation with the full TestQL and Viewer suites plus a + live render smoke test. +- Moved the ticket to `BLOCKED` in validation because the pre-existing + `Dockerfile.e2e` copies missing `src/`; that path belongs to active + governance ticket `ticket-001`. From fe28201fd6d518da7aa38ae9e221904ffa26798e Mon Sep 17 00:00:00 2001 From: Tom Softreck Date: Tue, 1 Sep 2026 11:29:47 +0200 Subject: [PATCH 5/5] docs: complete binary response validation --- project/ticket-002/README.md | 25 +++++++++++++------------ project/ticket-002/ai-codex-logs.txt | 6 ++++++ project/ticket-002/ai-codex.md | 22 +++++++++++++--------- project/ticket-002/changelog.md | 6 ++++++ 4 files changed, 38 insertions(+), 21 deletions(-) diff --git a/project/ticket-002/README.md b/project/ticket-002/README.md index a5703b6..a4c49f8 100644 --- a/project/ticket-002/README.md +++ b/project/ticket-002/README.md @@ -2,8 +2,8 @@ - **ID**: ticket-002 - **Owner**: unresolved:human -- **Status**: BLOCKED -- **Workflow state**: VALIDATION +- **Status**: IN_PROGRESS +- **Workflow state**: PUBLICATION - **Created**: 2026-09-01 ## Goal and scope @@ -26,7 +26,7 @@ domain-specific PNG/PDF/KiCad validation. `content_type`, `byte_length`, `sha256` and `magic` evidence. - [x] AC-04: Classic and Unified IR API execution use the same parser and cannot diverge on PNG/PDF responses. -- [ ] AC-05: Focused tests, full pytest, Docker checks and +- [x] AC-05: Focused tests, full pytest, Docker checks and `project/governance-check.sh` pass. ## Governance prerequisite @@ -41,23 +41,24 @@ extendable manifest. ## Validation evidence -- Focused HTTP/interpreter/IR suite: `82 passed`. +- Current focused HTTP/interpreter/IR suite: `69 passed`; the original wider + implementation run also passed all `82` selected tests. - Full suite after installing the repository's four local test plugins: `1704 passed, 9 skipped`. - Ruff and isolated mypy check for the new dependency-free parser: passed. - Viewer live scenarios against `127.0.0.1:8088`: `59/59 passed`, including byte-derived MIME/magic/size assertions for two PNG files, two SVG files and one PDF file. -- Viewer full suite: `596 passed, 10 skipped`; PCB/SCH hashes unchanged. +- Viewer source validation: `596 passed, 10 skipped`; PCB/SCH hashes unchanged. - Compose configuration for all three declared files: passed. -- Python 3.12 container probe of PNG/PDF byte classification: passed. -- Managed governance check: passed. +- Production image build and CLI startup: passed with TestQL `1.2.67`. +- E2E image build and execution: `1690 passed, 23 skipped`. +- Managed governance check and `git diff --check`: passed. -The repository-authored `Dockerfile.e2e` check is still red before this ticket's -code is installed: it executes `COPY src/ ./src/`, but this repository uses -`testql/`. That file is reserved by active governance ticket `ticket-001`, so -ticket-002 cannot change it without violating workstream ownership. AC-05 and -publication remain blocked on ticket-001 or an explicit integration hand-off. +The repository-authored Docker packaging blocker was resolved independently by +ticket-001 and merged through PR #7. Ticket-002 was resumed and fully validated +on top of that exact `main` revision without taking ownership of Docker or +Compose paths. The ticket is ready for protected publication. ## Participants diff --git a/project/ticket-002/ai-codex-logs.txt b/project/ticket-002/ai-codex-logs.txt index 35209ce..a391085 100644 --- a/project/ticket-002/ai-codex-logs.txt +++ b/project/ticket-002/ai-codex-logs.txt @@ -6,3 +6,9 @@ 2026-09-01T07:35:52Z VALIDATION focused="82 passed" full="1704 passed, 9 skipped" viewer="596 passed, 10 skipped" live="59/59 passed" 2026-09-01T07:35:52Z DOCKER compose=config-pass parser-python3.12=pass image=fail reason="Dockerfile.e2e COPY src/: source absent" 2026-09-01T07:35:52Z BLOCKED owner=ticket-001 path=Dockerfile.e2e publication=pending +2026-09-01T09:17:51Z RESUMED dependency="PR #7 merged as 1125743e4854eed84c3ee7c6b0a5d17c572fd16f" +2026-09-01T09:17:51Z VALIDATION combined_tree=true docker_paths_owner=ticket-001 +2026-09-01T09:29:23Z VALIDATION focused="69 passed" full="1704 passed, 9 skipped" ruff=pass mypy=pass governance=pass compose=pass +2026-09-01T09:29:23Z DOCKER runtime_build=pass runtime_start="TestQL 1.2.67" e2e_build=pass e2e="1690 passed, 23 skipped" +2026-09-01T09:29:23Z INTEGRATION viewer_live="59/59 passed" endpoint="http://127.0.0.1:8088" +2026-09-01T09:29:23Z PUBLICATION ready=true pr=6 diff --git a/project/ticket-002/ai-codex.md b/project/ticket-002/ai-codex.md index 577f765..33150ff 100644 --- a/project/ticket-002/ai-codex.md +++ b/project/ticket-002/ai-codex.md @@ -49,16 +49,20 @@ to the `core` workstream. classic OQL assertions and Unified IR assertions. - Proved the contract against the live Viewer render endpoints and expanded its smoke scenario to assert raw-byte evidence rather than status alone. -- Validation passed: focused `82 passed`, full TestQL `1704 passed, 9 skipped`, - Viewer `596 passed, 10 skipped`, live smoke `59/59`, Ruff, isolated mypy, - Compose configuration and governance. +- Validation passed: current focused `69 passed` (and the original wider + selection `82 passed`), full TestQL `1704 passed, 9 skipped`, Viewer source + suite `596 passed, 10 skipped`, live smoke `59/59`, E2E container + `1690 passed, 23 skipped`, production container startup, Ruff, isolated + mypy, all Compose configurations and governance. -## Blockers +## Resolved blockers -- `Dockerfile.e2e` still copies the absent `src/` directory, so the declared - repository image cannot build. A clean Python 3.12 container probe of this - ticket's parser passes, but the image definition is owned by active - governance ticket `ticket-001`. Publication is blocked until that ticket - corrects the packaging or explicitly hands off the path. +- `Dockerfile.e2e` previously copied the absent `src/` directory. Ticket-001 + corrected the repository packaging in PR #7, which Validator merged to + `main` as `1125743e4854eed84c3ee7c6b0a5d17c572fd16f`. - Root `TODO.md` is also owned by ticket-001, so this ticket records its status here instead of creating an overlapping governance diff. + +Ticket-002 completed `VALIDATION` on the combined tree and moved to +`PUBLICATION`. It does not modify or claim ownership of the Docker, Compose or +root roadmap files brought in from `main`. diff --git a/project/ticket-002/changelog.md b/project/ticket-002/changelog.md index 790c458..f9b2b8c 100644 --- a/project/ticket-002/changelog.md +++ b/project/ticket-002/changelog.md @@ -16,3 +16,9 @@ - Moved the ticket to `BLOCKED` in validation because the pre-existing `Dockerfile.e2e` copies missing `src/`; that path belongs to active governance ticket `ticket-001`. +- Resumed validation after ticket-001 fixed Docker packaging and PR #7 was + merged to `main`; ticket-002 itself does not modify the Docker-owned paths. +- Revalidated the combined tree with the focused and full host suites, Ruff, + mypy, all Compose configurations, managed governance, both container builds, + the full E2E image suite and the live Viewer scenarios. +- Completed every acceptance criterion and moved the ticket to `PUBLICATION`.