[Product Gap] Deliver schema-bound extraction with durable WARC/PROV replay

[seonghobae]

Buyer-visible gap

OriginWeave protected main contains strong typed evidence, destination, transport, TLS, policy, and resource primitives, but a buyer still cannot preserve one completed browser task as a durable, independently replayable evidence package. Current provenance values are construction contracts; they are not yet a tenant-scoped capture store, WARC archive, PROV graph, retention system, or offline replay product.

A production buyer must be able to answer:

What exact source bytes and browser observations produced this field, under which policy and software versions, and can the result be reproduced without revisiting the live site?

This issue owns the first complete evidence-native extraction and replay vertical slice. It does not duplicate issue #28, which owns the first live Chromium observation/action chain, or issue #9, which owns bounded HTTP/1.1 semantics.

Required product slice

Implement independently reusable Rust capture and replay boundaries, with provider-neutral storage ports, that compose the exact shipped authorities from #9, #10, and #28 when those prerequisites are available.

bounded browser/network observation
→ schema-bound field extraction
→ exact source-node / source-response evidence
→ immutable WARC record(s)
→ PROV-JSON-LD provenance bundle
→ structured result manifest
→ encrypted tenant-scoped persistence
→ offline verification and replay

Schema-bound extraction

• Define a versioned extraction schema with bounded field identifiers, value types, cardinality, required/optional status, normalization rules, and source-channel requirements.
• Permit only typed extraction from approved channels such as semantic node, JSON-LD, table cell, bounded network response, or a separately approved model interpretation.
• Bind every extracted value digest to the exact OriginWeave browser session, browsing context, canonical origin, document epoch, node/response identity, capture time, adapter version, and extraction-schema version.
• Preserve the raw value only inside the authorized result store; model-visible and audit summaries use field identifiers, digests, classifications, and opaque references unless policy explicitly permits disclosure.
• Reject ambiguous multi-source claims, stale node/response identities, cross-origin evidence, malformed locators, unsupported types, overlong values, and unverified provenance.
• LLM-derived values must retain model/provider/prompt/schema/reasoning metadata and supporting evidence identifiers; model output never becomes authority by itself.

WARC capture

• Emit WARC 1.1 / ISO 28500-compatible records through a versioned adapter.
• Preserve bounded request/response/resource metadata and payload bytes only when the applicable credential, personal-data, retention, and export policies allow it.
• Never persist raw cookies, authorization fields, client secrets, secret-broker values, or hidden long-lived credentials in generic WARC records.
• Record canonical origin, content type, payload length, cryptographic digest, capture timestamp, truncation/completeness state, and the exact OriginWeave network/TLS/HTTP evidence references.
• Treat active content as inert bytes. Replay must not execute JavaScript, contact external origins, resolve DNS, load remote resources, or regain browser/Agent authority.

PROV and result manifest

• Emit a W3C PROV-compatible JSON-LD graph connecting source entities, OriginWeave activities, software/adapter revisions, policy decisions, human approvals, generated artifacts, and extracted values.
• Preserve truth classes: observed evidence, policy decision, model inference, human approval, and final exported value must remain distinct.
• Produce one signed/hash-addressed capture_manifest that references every WARC record, PROV entity/activity, structured value, policy receipt, software revision, schema revision, and retention rule.
• Support deterministic verification without requiring the original live website.

Persistence and lifecycle

Provide ports for PostgreSQL metadata and S3-compatible object storage without binding the core crates to a single vendor.

Required lifecycle states:

capture_started
→ capture_completed
→ verified
→ retained | legal_hold
→ deletion_requested
→ deleted

• Encrypt tenant artifacts with versioned key references and support customer/region policy.
• Enforce tenant, task, subject, purpose, and retention boundaries at metadata, object, cache, index, export, and replay layers.
• Make capture creation idempotent and detect byte- or manifest-level duplicates.
• Support integrity verification, corruption detection, partial-capture classification, retention expiry, legal hold, key rotation, cryptographic erasure evidence, and backup expiry.
• Deletion receipts must not retain the deleted protected values.

Data model

Every database object must contain at least two words and use snake_case. Suggested objects:

capture_session
capture_manifest
capture_artifact
warc_record
provenance_entity
provenance_activity
provenance_relation
extraction_schema
extraction_field
extracted_value
source_evidence_binding
retention_policy_record
legal_hold_record
deletion_receipt_record
replay_verification
object_storage_reference

Normalize authority, lifecycle, retention, and relation data to 3NF. Raw WARC/PROV/provider payloads may remain immutable artifacts; billing-, policy-, or authorization-relevant fields must not be hidden only in JSON blobs.

Realistic tests and benchmark evidence

The first acceptance corpus must include a deterministic local fixture and a versioned set of public, legally reviewable pages.

Tests must prove:

1. one real Chromium task extracts a typed field and binds it to exact semantic-node and/or bounded network evidence;
2. the same accepted value is represented in the result manifest, WARC record set, and PROV graph without identity drift;
3. offline replay verifies hashes and reproduces the structured result without network access;
4. corruption, missing records, reordered relations, wrong schema, wrong tenant, stale node, cross-origin source, partial HTTP body, and unsupported MIME fail closed;
5. replay never executes active content or follows external references;
6. duplicate capture submission produces one canonical result and deterministic receipts;
7. retention expiry, legal hold, deletion, key rotation, and backup-expiry workflows produce the expected available/unavailable states;
8. protected-value scanners find no unapproved value in logs, traces, metrics, exceptions, CI output, support bundles, model history, URLs, or generic provenance summaries;
9. extraction precision and recall are measured against a gold corpus rather than inferred from successful parsing;
10. provenance completeness is exactly 100% for every released field;
11. OriginWeave-owned production function, line, region, and branch coverage remain exactly 100%; and
12. all public Rust APIs have complete rustdoc.

No skipped required browser, replay, retention, or corruption test may be represented as passing release evidence.

Documentation and architecture

• Add an ADR separating evidence construction, WARC persistence, PROV representation, structured extraction, retention, replay, and authorization.
• Add Mermaid data-flow and sequence diagrams for capture, model-assisted extraction, replay, legal hold, and deletion.
• Update PRD, TRD, ARCHITECTURE, DATA_GOVERNANCE, THREAT_MODEL, TEST_STRATEGY, OPERABILITY, API contract, ERD/UML, traceability, roadmap, README, AGENTS.md, CLAUDE.md, and CHANGELOG only where the implementation changes protected-main truth.
• Document standalone use and use as a naruon/CWL evidence module without direct database coupling.

Dependencies and non-goals

Dependencies:

• [Product Gap] Complete the first real Chromium agent vertical slice #28 supplies the first live Chromium observation/action boundary;
• [Product Gap] Bound HTTP/1.1 semantics over the authenticated TLS stream #9 supplies bounded HTTP/1.1 response semantics;
• [Enterprise Gap] Preserve operational PII through policy-bound disclosure instead of blanket masking #10 supplies purpose-bound protected-data disclosure and retention policy where sensitive values are involved.

Out of scope for this issue:

• automatic legal entitlement to archive third-party content;
• unrestricted web crawling;
• arbitrary Cypher/SQL writes;
• treating PROV or WARC presence as authorization or factual correctness;
• content execution during replay;
• replacing customer records-management, e-discovery, or legal-hold systems without separately scoped requirements.

Commercial proof

A buyer can select any released structured field, trace it to exact captured source evidence and policy decisions, verify artifact integrity, reproduce the result offline, and delete or retain it under policy—without exposing ambient credentials or promoting model output into authority.

Standards and research references — APA 7th

International Organization for Standardization. (2017). Information and documentation—WARC file format (ISO Standard No. 28500:2017). https://www.iso.org/standard/68004.html

World Wide Web Consortium. (2013). PROV-O: The PROV ontology. https://www.w3.org/TR/prov-o/

World Wide Web Consortium. (2013). PROV-DM: The PROV data model. https://www.w3.org/TR/prov-dm/

Internet Engineering Task Force. (2022). HTTP semantics (RFC 9110). https://doi.org/10.17487/RFC9110

Internet Engineering Task Force. (2022). HTTP/1.1 (RFC 9112). https://doi.org/10.17487/RFC9112