diff --git a/.agent-loop/REVIEW_LOG.md b/.agent-loop/REVIEW_LOG.md index 37b8596b..69ebbb06 100644 --- a/.agent-loop/REVIEW_LOG.md +++ b/.agent-loop/REVIEW_LOG.md @@ -2844,7 +2844,6 @@ inline finding. Its unexplained 37.61 percent docstring warning is non-actionable because the unchanged repository gate independently passes the same head at 87.6 percent against the 80 percent floor. Human review and explicit approval of PR #178 remain. - ## 2026-07-23 - WS-AUTH-001-10C Internal Implementation Review AUTH-10C exact implementation SHA @@ -2883,3 +2882,60 @@ retryability. Full GitHub shards then exposed and closed a stale linked-event matrix classification plus legacy-fixture trigger/constraint cleanup residue; the exact isolated audit case and sequential upgrade/downgrade matrices pass. fresh GitHub Backend, Agent Gates, CodeRabbit, and human review remain. + +# WS-ART-001-PLAN2 Submission Bundle Reconciliation + +- Signed automation run `30100940860` cancelled the rejected combined + `WS-ART-001-03` implementation before runtime edits. +- Human intent locks one outer ZIP, bounded process-local scratch through every + pre-submit gate, no candidate retention/provider namespace, conservative + existing limits, existing recovery reuse, and full integrity recomputation on + downstream byte streams. +- Initial planning review failed on stale ART history, incomplete guide chunks, + duplicate upload-session/artifact-set contracts, partial scratch surfaces, + pre-submit evidence privacy, nested-archive ambiguity, AUTH sequencing, and + failing governance/coverage/e2e gates. +- The repaired candidate splits 03A-C and 04A-C/05, reconciles canonical specs + and templates, defines the exact AUTH handoff, preserves existing + scratch/admission/recovery abstractions, and passes all deterministic gates. +- Final senior, architecture, QA/test, security/auth, product/ops, reuse/dedup, + CI integrity, test-delta, and docs reviews pass with no open finding. + +## 2026-07-24 - WS-ART-001-PLAN2 External Review Reconciliation + +CodeRabbit found seven valid inconsistencies in phase ownership, hidden route +composition, canonical ZIP paths, AUTH retirement custody, successor wording, +put-attempt state vocabulary, and server-owned manifest input. Hosted CI also +found noncanonical chunk headings and a successor-title mismatch. The repair +resolves all findings without runtime behavior or AUTH activation; refreshed +internal review, hosted checks, and human approval remain required. + +## 2026-07-25 - WS-ART-001-PLAN2 Admission And Authorization Amendment + +The final planning candidate now closes verified-but-unbound admission +lifecycle, canonical executable intent, and fresh authorization at durable put +intent and Submission consumption. Review repair replaced shared PermissionIds +misnamed as service actions with the canonical phase-specific AUTH ActionIds +and strengthened the governance test to require exact no-expiry, no-release, +no-deletion, no-cleanup, and no-retention-process clauses. Exact material SHA +`3185af57055f8e7b2411a2090671a9707a26dc58` is rebased on trusted main +`bba4ba5f171a4438b072740707a5cf8bde49d9af`; all nine internal reviewer tracks +pass with no open finding. Hosted external checks and human merge approval +remain required. + +GitHub Agent Gates then exercised the full pytest collection that the direct +script entry point does not run and exposed two line-wrap-sensitive assertions. +The repair makes the 05 typed-capability/no-AUTH-repository boundary explicit +and normalizes handoff whitespace before semantic ActionId/PermissionId checks. +The exact hosted-equivalent command passes 300 tests with 90.46 percent branch +coverage for the loop-memory updater. + +The later trusted-main reconciliation incorporates PR #198's exact CI custody +and Ruff/orchestration repairs without changing the 22-commit PLAN2 patch +series; range-diff equivalence and fresh exact-base integration review are the +required evidence before republishing PR #197. + +The exact Ruff 0.15.22 integration review found one branch-owned `F841`: a +historical ART contract read remained after its assertion was retired. The +repair removes only that dead read; focused Ruff passes, all 104 collected +agent-gate pytest cases pass, and the 100 direct regression cases pass. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/AUTH_HANDOFF.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/AUTH_HANDOFF.md new file mode 100644 index 00000000..f7b48574 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/AUTH_HANDOFF.md @@ -0,0 +1,91 @@ +# WS-ART-001 Authorization Handoff + +ART owns hidden artifact behavior, canonical product resource facts, lifecycle +guards, surface manifests, and feature tests. AUTH owns ActionId/PermissionId +catalogues, service identities, fixed matrices, evaluator integration, grants, +activation custody, and availability. + +## Guide Source Sequence + +1. Existing guide-source actions remain planned and unavailable. +2. ART-03A implements hidden `artifact.guide_source.ingest` behavior and its + exact resource/guard/surface manifest. +3. AUTH activates only that exact action through a separately reviewed AUTH + contract after consuming ART evidence. +4. ART-03B implements hidden `artifact.guide_source.read` and + `artifact.guide_source.binding.create` behavior; binding maps to fixed + permission `artifact.binding.create`. +5. AUTH activates only those exact actions after consuming 03B evidence. +6. ART-03C performs the legacy clean cut. No ART chunk writes availability. + +## Submission Bundle Sequence + +Before ART-04A starts, AUTH must merge a separately reviewed registration +contract, provisionally named +`WS-AUTH-001-ART-SUBMISSION-BUNDLE-REGISTRATION`, that: + +- registers planned ActionId `artifact.submission_bundle.prepare`; +- maps it only to existing human PermissionId `submission.create`; +- limits candidates to the assigned contributor for the exact task/project; +- names ART-owned canonical facts for actor, identity link, project, task, + active assignment, locked policy context, and operation generation; +- keeps the action unavailable and adds no grant or evaluator activation; +- records parity evidence in AUTH's closed catalogue/constraint/owner manifests; +- explicitly retires the unused planned multi-step upload-session ActionIds or + proves they are unavailable and have no route/command manifest entry. + +Until that AUTH contract merges, current agent-gate catalogue assertions retain +those strings only as an exact planned/unavailable discovery baseline. Their +presence in the closed catalogue is not an active design, grant, route, or +permission to implement a second intake path, and PLAN2 does not edit AUTH-owned +catalogue or parity assertions. + +ART-04A through 04C then implement one hidden continuous surface and publish +its exact route/resource/guard manifest. After 04C, a separate reviewed AUTH +activation contract may integrate the evaluator and change only +`artifact.submission_bundle.prepare` to active. ART-05 cannot start until that +activation merges. + +The preparation surface authorizes before scratch intake, but the initial +decision cannot authorize the later durable mutation. Immediately before +capacity reservation and `ArtifactPutAttempt` creation, 04C must consume an +AUTH-owned transaction-local prepared capability whose canonical facts cover +the current actor, exact identity link, project authority, assignment, task, +predecessor, locked task/guide/policy context, action availability, and +operation generation. AUTH and the owning product services reload/lock their +own facts; ART receives only the typed capability and never imports AUTH-owned +repositories. Authorization evidence, capacity reservation, and durable put +intent commit atomically before provider I/O. + +ART-05 requires a new human authorization decision for `submission.create` and +a separately prepared fixed-service capability for +ActionId `artifact.submission.binding.create`, mapped to PermissionId +`artifact.binding.create`. Both are consumed in the one transaction that locks +the ready admission and TASK-owned context, creates Submission and binding, +and marks the admission consumed. Human authority implies no service authority. +Revocation after durable put intent does not cancel verification/recovery, but +the resulting admission remains unbound until a fresh 05 decision succeeds. +Authorization denial precedes admission-detail errors so unrelated actors +cannot distinguish missing, ready, stale, or consumed admissions. + +The continuous contributor action never implies the fixed service actions: + +- `artifact.pre_submit.checker_input.materialize` and + `artifact.post_submit.checker_input.materialize`, both mapped to PermissionId + `artifact.checker_input.materialize`; +- `artifact.verification.execute`; +- `artifact.pending_work.scan`; +- `artifact.put_attempt.resolve`; +- `artifact.submission.binding.create`, mapped to PermissionId + `artifact.binding.create`. + +Each fixed service action retains its canonical provisioned service identity, +matrix row, resource facts, terminal reauthorization, and separate activation +evidence. No human grant supplies fixed service authority. + +## Fail-Closed Rule + +An ART implementation contract stops if its required AUTH registration or +activation contract is absent, unmerged, inactive, differently mapped, or +targets a different resource fact shape. Planned catalogue presence, a local +action string, or hidden feature code is never executable authority. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md index b90c0b63..8af1f995 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md @@ -12,14 +12,19 @@ Each chunk is one PR. No later chunk starts automatically. | `WS-ART-001-02A3` | Replace ArtifactStore v1 with byte-only v2, activate API-startup and Celery Beat scratch cleanup, migrate schema/callers/factory, and remove `flow_node` in one atomic clean cut. | L1 | Merged through PR #141 as `a10d901` on 2026-07-18 | | `WS-ART-001-02B1` | Implement the S3-compatible adapter, MinIO integration, and AWS S3 production profile. | L1 | Merged through PR #151 as `1b5422fc` on 2026-07-19 | | `WS-ART-001-02C1` | Add the generic durable-byte admission ledger and durable put-attempt state foundation without provider execution. | L1 | Merged through PR #154 as `44f2467c` on 2026-07-19 | -| `WS-ART-001-02C2` | Add put resolution, verification publication, complete-object observation, immutable receipts, and PostgreSQL execution fencing without recovery attempts or routes. | L1 | Active after PR #154 and explicit user start on 2026-07-19 | -| `WS-ART-001-02C3` | Add the recovery-attempt model and exact idempotent source-job to retry-job chain without public or Operator routes. | L1 | Proposed after 02C2 | -| `WS-ART-001-02D` | Add hidden Operator content/job/retry/recovery/audit APIs, canonical resource composition, and production-readiness checks while actions and provider profiles remain inactive. | L1 | Proposed after 02C3, AUTH-09E, and `WS-AUTH-001-ART-CUSTODY` | -| `WS-ART-001-03` | Store and bind guide-source bytes; add same-snapshot setup recovery through the authorized artifact reader. | L1 | Proposed after 02D | -| `WS-ART-001-04A` | Add task-scoped upload sessions/items, trusted archive inspection, independent verification, immutable sealing, and artifact-set manifests. | L1 | Proposed after 03 | -| `WS-ART-001-04B` | Execute authoritative pre-submit against sealed artifact sets and persist exact admissions with bounded infrastructure continuation. | L1 | Proposed after 04A | -| `WS-ART-001-05` | Atomically bind admitted artifact sets to submissions and remove legacy URI/hash/finalization contracts. | L1 | Proposed after 04B | -| `WS-ART-001-06A` | Persist checker input snapshots and materialize authorized immutable bytes into bounded checker workspaces. | L1 | Proposed after 05 | +| `WS-ART-001-02C2` | Add put resolution, verification publication, complete-object observation, immutable receipts, and PostgreSQL execution fencing without recovery attempts or routes. | L1 | Merged through PR #159 as `bc5e6a42` | +| `WS-ART-001-02C3` | Add the recovery-attempt model and exact idempotent source-job to retry-job chain without public or Operator routes. | L1 | Merged through PR #174 as `92b8a7aa` | +| `WS-ART-001-02D` | Add hidden Operator content/job/retry/recovery/audit APIs, canonical resource composition, and production-readiness checks while actions and provider profiles remain inactive. | L1 | Merged through PR #177 as `93c14181` | +| `WS-ART-001-03` | Original combined guide-source cutover. | L1 | Cancelled before implementation; no runtime changes | +| `WS-ART-001-PLAN2` | Reconcile guide and one-ZIP submission planning with bounded scratch, existing immutable admission/recovery, exact AUTH sequencing, and downstream ownership. | L1 | Planning-only successor proposed after cancellation | +| `WS-ART-001-03A` | Add hidden guide-source byte ingest through existing preparation, admission, verification, and publication. | L1 | Proposed after PLAN2 | +| `WS-ART-001-03B` | Bind verified guide-source content and provide authorized integrity-checking setup materialization. | L1 | Proposed after 03A and exact AUTH activation | +| `WS-ART-001-03C` | Remove legacy guide-source identity and add exact same-generation setup continuation. | L1 | Proposed after 03B and exact AUTH activation | +| `WS-ART-001-04A` | Accept one outer ZIP in bounded scratch, safely inspect its tree, normalize executable intent, produce canonical identities, and reject unchanged work before provider I/O. | L1 | Proposed after 03C and AUTH planned action registration | +| `WS-ART-001-04B` | Run mandatory platform and locked Project Guide pre-submit checks against the same scratch-bound tree and executable semantics without durable storage. | L1 | Proposed after 04A | +| `WS-ART-001-04C` | Reauthorize at durable intent, admit/verify the passing ZIP once, and publish one capacity-charged `ready` admission that may remain unbound. | L1 | Proposed after 04B; AUTH activation follows hidden completion | +| `WS-ART-001-05` | Freshly authorize and atomically consume one ready admission into one immutable Submission/binding, or terminally stale it on proven context drift. | L1 | Proposed after 04C and exact AUTH activation | +| `WS-ART-001-06A` | Persist checker input snapshots and materialize authorized immutable bytes with identical normalized executable semantics. | L1 | Proposed after 05 | | `WS-ART-001-06B` | Ingest checker logs/outputs as artifacts, persist checker completion facts, and preserve existing checker-owned routing without creating review aggregates. | L1 | Proposed after 06A | | `WS-ART-001-07` | Prove Local/MinIO plus AWS S3 readiness, Operator recovery, and exact-byte guide/pre/post-submit behavior through real APIs. | L1 | Proposed after 06B | @@ -35,9 +40,17 @@ OBJECT-STORAGE-AMENDMENT -> 02C2 put resolution, verification publication, and fencing -> 02C3 recovery attempt and idempotency chain -> 02D Operator and production readiness --> 03 guide source cutover --> 04A upload/inspection/sealing --> 04B pre-submit admission and outage continuation +-> PLAN2 planning reconciliation +-> 03A guide-source byte ingest +-> AUTH activation for exact 03A actions +-> 03B guide-source binding/materialization +-> AUTH activation for exact 03B actions +-> 03C guide-source clean cut/continuation +-> AUTH planned registration of `artifact.submission_bundle.prepare` +-> 04A one-ZIP scratch intake/inspection/manifest/change gate +-> 04B scratch-bound platform/project pre-submit checks +-> 04C one-time immutable admission/verification +-> AUTH activation of exact complete contributor surface -> 05 submission cutover -> 06A checker input/materialization -> 06B checker output/post-submit routing @@ -48,10 +61,14 @@ OBJECT-STORAGE-AMENDMENT deferred. It has no active chunk, runtime profile, credential service, or configuration value in v0.1. `ReviewPacketManifest` and `ReviewEvidenceArtifact` remain owned by WS-REV. -Physical deletion and semantic search require separate approved initiatives. +Physical deletion, temporary provider retention, candidate object storage, and +semantic search require separate approved initiatives. ## Cross-Initiative Handoffs +The exact authorization sequence and stop conditions are recorded in +`AUTH_HANDOFF.md`. + - Artifact actions follow AUTH planned registration -> hidden ART behavior and canonical resource composition -> AUTH evaluator integration and activation. `WS-AUTH-001-ART-CUSTODY` first transfers the 25 current ART actions to eight @@ -64,7 +81,10 @@ Physical deletion and semantic search require separate approved initiatives. - WS-REV owns `ReviewPacketManifest` and `ReviewEvidenceArtifact`. Review code receives verified Workstream `ArtifactBinding` IDs through a narrow review-facing capability; it must not receive provider references, scratch - paths, or concrete adapters. + paths, or concrete adapters. REV also owns reviewer decisions and + note/findings for + the exact `Submission`; `needs_revision` authorizes a later contributor ZIP + but contains no reviewer-uploaded artifact. - A future optional contribution-evidence projection requires separately approved ART-owned read/write capabilities and AUTH action activation. Core ContributionRecord creation makes no ART capability/provider call and is not @@ -73,8 +93,17 @@ Physical deletion and semantic search require separate approved initiatives. - Cross-initiative terminology must use ART's canonical `resource_type`, `resource_id`, and `logical_role`, or define an explicit integration mapping; product initiatives must not create a second binding vocabulary implicitly. +- The existing immutable `Submission` row is the version aggregate. TASK/REV + jointly own the exact `needs_revision` response relation and indexed + latest/current/accepted access; no initiative creates a competing + `SubmissionVersion` table. +- Reviewer and delivery streams consume an ART-owned integrity-checking read + capability that recomputes full SHA-256 and byte count. ART does not own the + review decision, ContributionRecord, compensation, reputation, or delivery + lifecycle that consumes that capability. ## Checkpoint Before Checker Expansion Do not resume checker feature expansion until `WS-ART-001-06B` proves pre-submit -and post-submit consume the same immutable artifact-set commitment. +evidence and post-submit execution name the same archive identity, +semantic-manifest hash, verified admission, and exact binding. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md index d36712f2..68b83c57 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md @@ -149,23 +149,25 @@ Operators receive bounded read-only admission-usage visibility and alerts. Quota expansion is a reviewed configuration/runbook operation; it never releases or edits an authoritative charge. -## D19 - Upload-Session Slot Expiry +## D19 - Submission-Bundle Scratch Capacity -Contributor open-session slots are separate from retained-byte admission. -Chunk 04A owns PostgreSQL-clock periodic Celery expiry and lazy expiry before -new-session admission. Both paths use the fixed system permission -`artifact.upload_session.expire` and one atomic terminal transition that -releases the slot exactly once. Cancellation or expiry never releases a -completed durable-byte charge. +The earlier durable multi-step upload-session slot design is superseded before +activation. One continuous submission-bundle preparation reserves bounded +scratch capacity through the existing `ArtifactScratchManager`. No durable +upload session/item is a candidate store, no scratch path crosses a process or +Celery boundary, and process loss requires reupload. ART-04A must remove or make +unreachable every unused upload-session/item model and action path; no alias or +parallel contributor intake remains. ## D20 - Closed Materialization Sources -The provider-neutral materializer accepts exactly one sealed upload artifact -set whose items are all `ready` before submission creation, or immutable -`ArtifactBinding` IDs after submission creation. Staging never creates a -premature product binding. Both forms resolve through phase-specific -authorization, stream exact provider bytes, and recompute SHA-256 and byte -count before a checker receives a read-only workspace. +Before durable admission, the checker receives only one internal +`PreparedArtifact`/`ArtifactScratchManager`-owned submission-bundle workspace in +the same process-local orchestration. After admission, the provider-neutral +materializer accepts only immutable `ArtifactBinding` IDs. Neither form creates +a second handle, lease, ledger, quota, cleanup path, or premature product +binding. Both recompute the relevant SHA-256 and byte counts before a checker +receives a read-only workspace. ## D21 - Configured Storage Namespace Fence @@ -254,3 +256,109 @@ defined canonically in `WS-AUTH-001-ART-CUSTODY` chunk contract. Operator verification retry remains an independently authorized human action. Service identity, Celery executor identity, and execution-generation fencing never substitute for one another. + +## D29 - One Outer ZIP Per Submission + +Every contributor Submission version accepts exactly one outer ZIP. Workstream +recursively inspects the directory/file tree represented by that archive so a +locked Project Guide checker can require entries such as `task.toml`. An archive +entry that is itself a ZIP remains one opaque ordinary file. Nested archive +unpacking is outside v0.1 and requires a separate reviewed capability with +cumulative safety limits before a Project Guide may request it. + +## D30 - Scratch Before Durable Admission + +Unchecked contributor bytes remain only in bounded private scratch through +format/safety inspection, canonical manifest generation, unchanged-work +comparison, mandatory platform gates, and locked project pre-submit checks. +Failed, unsafe, unchanged, abandoned, or checker-failing attempts never enter +object storage. Process or scratch loss before durable intent requires reupload. + +No candidate provider namespace, retention window, promotion/copy, or physical +cleanup API is added. A passing ZIP enters the existing immutable +`ArtifactStore` path once. Ambiguous durable effects reuse existing put attempt, +observation, verification, receipt, scanner, and recovery abstractions. + +## D31 - Canonical Submission Bundle Identity + +Workstream records both exact archive identity (server-computed SHA-256 and byte +count) and semantic tree identity. The semantic manifest commits to normalized +file/directory path, entry type, each file's SHA-256 and byte count, and the +normalized executable flag for regular files. It does +not commit to archive timestamps, compression method/level, comments, ownership, +or ownership, group, read/write/special bits, or other platform permission +metadata. Valid Unix execute bits are normalized into the sole semantic +permission flag; non-Unix/invalid mode metadata defaults it to false. Symlinks +and unsupported special entries are +rejected. Exact archive equality or semantic-manifest equality with the +immediate prior immutable Submission rejects before provider I/O. + +## D32 - Existing Submission Is The Version Aggregate + +The current immutable `Submission` row plus its `version` and +`supersedes_submission_id` is the canonical version aggregate. There is no +competing `SubmissionVersion` table. A reviewer attaches only one of `accept`, +`needs_revision`, or `reject` plus note/findings to the exact Submission. A +contributor response to `needs_revision` uploads a new complete ZIP, creates the +next Submission, and records the exact prior-Submission and review relationship +through the TASK/REV joint contract. + +Indexed latest/current/accepted queries or projections serve normal reads; +immutable relationships retain complete history and audit reconstruction. + +## D33 - Integrity Is Rechecked On Egress + +Historical verification is necessary but not sufficient for a later full-byte +read. Checker materialization, reviewer download, and downstream delivery use an +ART-owned provider-neutral read capability that recomputes complete SHA-256 and +byte count while streaming and fails closed on missing or mismatched bytes. +Consumer initiatives own their lifecycle decisions and reference the exact +Submission/binding; ART owns only byte identity, integrity, manifests, binding, +and access capability. + +## D34 - Conservative Limits Remain + +The existing configured upload, scratch, duration, and provider limits remain +unchanged. Any larger ZIP, extracted-byte, per-file, entry-count, depth, +compression-ratio, scratch, or time limit requires a separate reviewed change +with concurrency, capacity, runtime, cost, and abuse evidence. + +## D35 - Verified Admissions May Remain Unbound + +`SubmissionBundleAdmission` has the closed lifecycle `ready -> consumed|stale`. +Client abandonment is valid and remains charged to existing completed-byte +capacity. Consumption and Submission/binding creation are one transaction with +database uniqueness. Proven task/predecessor/locked-context drift makes a ready +admission stale; authority loss alone does not. No expiry, release, deletion, +retention process, or cleanup lifecycle exists in v0.1. + +## D36 - Authorization Is Fresh At Both Durable Mutations + +Initial route authorization is insufficient. 04C consumes an AUTH-owned +transaction-local prepared capability in the durable-intent transaction before +provider I/O. 05 obtains and consumes new human submission and fixed-service +binding capabilities in the atomic Submission/binding/admission transaction. +ART never reads or locks AUTH-owned persistence directly. + +## D37 - Executable Intent Is Canonical, Not Archive Permission Preservation + +Regular-file executable intent participates in semantic identity. Materializers +project fixed read-only or read-and-execute modes and fixed read-and-traverse +directory modes; they never preserve arbitrary ZIP permissions or automatically +execute a file. The Project Guide/checker policy remains the execution decision +owner. Shared scratch startup rejects inconsistent canonical mode configuration, +and secure cleanup remains owned by `ArtifactScratchManager`. + +## D38 - One Contributor Preparation Action + +The prior multi-step upload-session action proposal does not govern the new +continuous scratch-bound flow. AUTH first registers planned ActionId +`artifact.submission_bundle.prepare` mapped to existing PermissionId +`submission.create`. ART-04A through 04C then provide the complete hidden route, +canonical task/assignment/actor resource facts, guards, and manifest. AUTH alone +integrates and activates it afterward. + +Contributor authority never implies fixed service authority. Checker +materialization, verification execution, pending-work scanning, ambiguous-put +resolution, and binding retain their distinct fixed actions. No compatibility +alias exposes the obsolete multi-step session actions. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/INTENT.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/INTENT.md index 28fc9331..374bd689 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/INTENT.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/INTENT.md @@ -36,13 +36,35 @@ authorized bytes -> content-addressed S3-compatible object key -> independent complete-object verification -> immutable ArtifactContent and ArtifactReplica facts --> sealed artifact-set commitment --> ArtifactBinding to guide, submission, checker input, log, or output +-> ArtifactBinding to guide, checker input, log, or output + +submission outer ZIP only +-> verified archive identity and canonical semantic-manifest commitment +-> exact Submission binding ``` PostgreSQL stores metadata, bindings, operation receipts, lifecycle state, audit, and recovery coordination. The object provider stores bytes only. +For contributor work, the mandatory v0.1 invariant is: + +```text +one immutable Submission row/version +-> one uploaded outer ZIP +-> one safely inspected internal file/directory tree +-> one canonical semantic manifest +-> mandatory platform and locked Project Guide prechecks +-> one ArtifactStore admission and complete read-back verification +-> one capacity-charged ready admission, which may remain unbound +-> one exact ArtifactBinding +-> the same bytes checked, reviewed, accepted, recorded, and delivered +``` + +The outer ZIP may contain one file, a codebase, evidence, datasets, nested +directories, or any other content allowed by the locked Project Guide. A ZIP +entry inside the outer archive remains an ordinary file. Nested archive +unpacking is outside v0.1 even when a guide would otherwise request it. + ## First-Principle Constraints - Workstream computes and verifies canonical SHA-256 and byte count. @@ -59,6 +81,20 @@ audit, and recovery coordination. The object provider stores bytes only. retention references, or object metadata. - v0.1 performs no physical object deletion. Release and garbage collection require a later approved deletion-policy initiative. +- v0.1 has no candidate/quarantine object store or temporary provider + retention. Unchecked contributor bytes remain only in bounded private scratch + and enter the existing immutable store once, after every pre-submit check + passes. +- The existing `Submission` row is the immutable version aggregate. Reviewers + attach only a decision and note/findings to that exact version; a contributor + response to `needs_revision` is another complete ZIP and immutable Submission. +- A verified admission may remain unbound through client abandonment. It stays + capacity-charged and creates no product lifecycle effect; consumption is + atomic with Submission/binding creation and no admission expires or deletes + bytes in v0.1. +- Regular-file executable intent is normalized into semantic identity and + materialized consistently, without preserving arbitrary archive permissions + or granting execution authority. - Local storage is forbidden in staging and production. - No compatibility alias or dual provider-construction path is retained. @@ -83,6 +119,9 @@ v0.1 contribution lifecycle. - no provider-side legal hold, pin, retain, or release API; - no physical deletion or garbage collection; - no semantic search; +- no candidate storage namespace, promotion copy, or temporary provider + retention window; +- no second artifact recovery aggregate; - no review packet or reviewer evidence implementation, which remains WS-REV; - no payment, reputation, blockchain, or marketplace expansion. @@ -101,6 +140,10 @@ v0.1 contribution lifecycle. Celery under PostgreSQL generation fencing; - guide, pre-submit, post-submit, and reviewer-facing records resolve the same immutable content commitment; +- exact-archive and semantic-manifest equality reject unchanged resubmissions + before provider I/O; +- checker, reviewer, and delivery reads recompute SHA-256 and byte count while + streaming the exact bound bytes; - the final real API drill runs without direct database inspection. ## Human Decisions diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md index 60db8577..8dddb785 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md @@ -29,7 +29,7 @@ Those product capabilities are closed and operation-specific: ```text GuideArtifactIngestPort -ContributorArtifactUploadPort +SubmissionBundlePreparationPort ArtifactBindingPort ArtifactMaterializationPort CheckerArtifactOutputPort @@ -184,15 +184,16 @@ durable acknowledgement-unknown attempt rather than committing a terminal fact. computes digest and size, and rejects a mismatched client commitment before provider I/O. 3. PostgreSQL atomically reserves unique-byte charges at every applicable task, - producer, project, and deployment scope. Contributor upload-session slots - are a separate 04A reservation. Any exceeded scope fails before provider - I/O. Provisional and completed byte charges count; exact replay and + producer, project, and deployment scope. For contributor submissions this + durable reservation occurs only after all scratch-bound pre-submit gates + pass. Any exceeded scope fails before provider I/O. Provisional and + completed byte charges count; exact replay and concurrent same-content reservations cannot double-charge or oversubscribe. Product callers do not assemble this set; artifact orchestration derives it - from authoritative actor/service, project, task, upload-session or + from authoritative actor/service, project, task, submission-bundle operation or checker-run, and deployment records. A missing required relationship fails before provider I/O. -4. Transaction A reserves the upload item and commits the server-computed +4. Transaction A reserves the producer-specific admission and commits the server-computed digest, size, media type, operation identity, request digest, and CAS values. 5. Workstream passes the sealed `CommittedArtifactSource` to the injected adapter outside the transaction. @@ -201,9 +202,10 @@ durable acknowledgement-unknown attempt rather than committing a terminal fact. the adapter conditionally stores under the content-addressed key or resolves an exact replay candidate. 7. Transaction B records provider acknowledgement, completes the provisional - admission charges, sets the item to `stored_pending_verification`, and + admission charges, sets the `ArtifactPutAttempt` to `object_confirmed`, and creates the replica with pending verification and unknown - availability/integrity; no binding exists. + availability/integrity; while a legacy contributor upload item exists, it + alone moves to `stored_pending_verification`. No binding exists. 8. A durable verification job is committed in PostgreSQL and published to Celery after commit. A periodic scanner republishes pending work within the configured SLA. @@ -215,14 +217,15 @@ Provider acknowledgement loss keeps the durable put attempt and admission charges provisional. A PostgreSQL scanner publishes ambiguous and expired in-flight attempts; a fixed service principal runs read-only `observe_put_result` plus a complete hash. Matching bytes complete Transaction B -once, authoritative absence releases charges and makes the item -`replay_required`, and mismatched bytes quarantine the key. No background +once, authoritative absence releases charges and moves the put attempt to +`absent_replay_required`; while a legacy contributor upload item exists, it +alone moves to `replay_required`. Mismatched bytes quarantine the key. No background resolver repeats a provider write. Exact replay after absence must atomically reacquire capacity before another provider call. Workstream never stores upload bytes in Postgres, Redis, or Celery payloads. -Before binding, an object confirmed missing returns its reserved upload item to -`replay_required`, and only the original authorized uploader may replay the +Before binding, an object confirmed missing returns its reserved put attempt to +`absent_replay_required`, and only the original authorized producer may replay the same bytes under the same operation identity. That pre-binding replay resets the same replica from `missing/unavailable/unknown` to `pending/unknown/unknown`, appends a receipt, and creates a new verification @@ -233,9 +236,9 @@ replaces or rebinds the bytes. A digest/size mismatch on an existing key is also unrecoverable in v0.1 and becomes a security incident; Workstream never overwrites that key. -Contributor uploads, authorized caller-supplied guide bytes, and generated -checker logs/outputs -all use the same bounded two-pass `PreparedArtifact` scratch boundary. The first +Contributor submission ZIPs, authorized caller-supplied guide bytes, and +generated checker logs/outputs all use the same bounded two-pass +`PreparedArtifact` scratch boundary. The first pass uses private ephemeral local scratch to hash/count; the second pass exposes a sealed `CommittedArtifactSource` to ArtifactStore. No caller can pass an arbitrary expected digest beside an arbitrary stream. Scratch is never @@ -274,30 +277,104 @@ Physical deletion, garbage collection, legal hold, and retention windows are a later explicit initiative. This removes destructive storage behavior from the first production cutover. -## Server-Generated Artifact Set - -The server seals an upload session before authoritative pre-submit execution. -Trusted inspection derives normalized display names, logical roles, media -types, archive members, path safety, individual sizes, total size, and -Workstream SHA-256 values. Canonical ordering and JSON produce -`artifact_set_hash`. - -The pre-submit admission record binds actor, task, sealed session, artifact-set -hash, locked guide/policy/checker context, result, expiry, and canonical -submission input hash. Submission creation row-locks and consumes that exact -admission and session. No checked-versus-submitted byte swap is possible. - -Pre-submit artifact reads use a bounded Workstream-owned transient retry budget. -If it is exhausted, the API returns HTTP 503 with -`pre_submission_infrastructure_unavailable`; this is neither a checker failure -nor a product review decision. The exact precheck attempt, sealed upload -session, and artifact set remain unconsumed and reusable until normal expiry. -Only infrastructure-attempt and audit facts are written. Idempotency binds -actor, task, session/artifact set, locked context, client key, and canonical -request digest: exact replay continues or returns the same attempt, changed -replay conflicts, and concurrent replay creates no duplicate. After retry-after -or infrastructure recovery, the contributor continues the same exact attempt -without Project Manager or Operator approval. +## Server-Generated Submission Bundle Identity + +Every Submission version has exactly one outer ZIP. Workstream keeps that ZIP +only in bounded private scratch while it walks the complete normalized +file/directory tree, enforces archive-safety/resource limits, hashes each file, +and derives two identities: + +```text +archive identity = SHA-256 and byte count of the exact outer ZIP +semantic identity = canonical hash of normalized path, entry type, + file SHA-256, file byte count, and normalized executable flag +``` + +ZIP timestamps, compression settings, comments, ownership, group, read/write +bits, special bits, and other platform permission metadata do not change +semantic identity. Executable intent is the sole permission-derived semantic: +for a regular entry created with valid Unix mode metadata it is true when any +execute bit is present; otherwise it is false. Directories have no executable +field. Explicit empty directories +and synthetic parents use one documented canonical representation. Symlinks and +special entries are rejected. Canonical paths use `/` separators, contain only +relative non-empty segments, and normalize Unicode to NFC. The identity remains +case-sensitive, but Workstream rejects exact duplicates, NFC collisions, and +Unicode case-fold collisions so the checked tree cannot vary by filesystem. +Collision detection completes before an entry enters the manifest, any project +precheck or materializer observes the tree, or provider I/O starts. A ZIP entry +within the outer archive is opaque in v0.1; recursive inspection means walking +the outer archive tree, not opening nested archives. A later capability requires +separate cumulative safety proof. + +Both identities are compared with the immediate prior immutable `Submission`. +Exact archive equality or semantic equality rejects before checker and provider +I/O. Mandatory Workstream gates and the task's locked Project Guide checker then +consume the same read-only scratch tree. A project may narrow platform limits +but cannot disable gates or raise limits. Checker failure creates findings only +and destroys scratch without durable artifact, Submission, or review state. + +A passing result stays bound to the same process-local scratch generation and is +consumed immediately by the normal durable admission path. It is never a +provider object or reusable candidate. Process/scratch loss requires reupload. +The durable path writes the outer ZIP once, independently reads it completely, +and publishes a bindable admission only after the observed archive identity +matches. Existing put-attempt, observation, verification, receipt, scanner, and +recovery behavior resolves ambiguity. No candidate namespace, promotion copy, +retention window, physical deletion, or second recovery aggregate exists. + +### Verified Submission-Bundle Admission + +Verification publishes an immutable, capacity-charged +`SubmissionBundleAdmission` in `ready`. It binds the preparing actor and +identity-link provenance, project, task, assignment, immediate predecessor, +exact locked task/guide/policy context, verified `ArtifactContent`, semantic +manifest, and immutable pre-submit evidence set. Client abandonment is an +accepted quota-bounded v0.1 outcome: a `ready` admission may remain unbound +without creating a Submission, review, contribution, compensation, or +reputation effect. + +The only transitions are `ready -> consumed` and `ready -> stale`. +Consumption occurs in the same transaction that creates the immutable +Submission and binding, and database uniqueness permits one consuming +Submission. Proven task closure, predecessor advancement, or locked-context +replacement makes a still-ready admission `stale` during a consumption attempt. +Authority loss alone does not make it stale; every attempt obtains fresh AUTH +authority, and restored authority may consume a still-compatible admission. +`consumed` and `stale` are terminal. No state expires, releases capacity, or +authorizes provider deletion. Ready, stale, and consumed admissions continue to +count against existing completed-byte scopes. Existing Operator admission-usage +projections add bounded unbound-ready and stale counts/bytes without exposing +content or provider identities. + +Exact preparation replay returns the original admission without another put, +charge, evidence set, or admission. Uniqueness and status/terminal-field check +constraints enforce the lifecycle; the Submission owns a unique admission +reference as the second database fence. + +### Durable Authorization Boundaries + +The contributor request is authorized before scratch intake. Immediately before +durable capacity reservation and `ArtifactPutAttempt` creation, 04C opens the +owning transaction and consumes AUTH's transaction-local prepared capability. +AUTH—not ART—reloads and validates the current ActorProfile, exact identity link, +project authority, assignment, action availability, and canonical resource +facts. TASK/PROJECT owners lock task, predecessor, and locked context through +their typed capabilities. Authorization evidence, capacity reservation, and put +intent commit atomically; failure means no provider I/O and scratch cleanup. + +Durable put intent creates a technical recovery obligation that later human +revocation does not cancel. Verification may finish, but binding remains +impossible until 05 obtains fresh human `submission.create` authority and fresh +fixed-service ActionId `artifact.submission.binding.create` authority, mapped +to PermissionId `artifact.binding.create`. 05 consumes both prepared +capabilities in the single task transaction that locks the admission/context, +creates Submission and binding, and marks the admission consumed. Denial, +cancellation, stale execution, or persistence failure rolls back every protected +effect. After authority succeeds, proven task closure, predecessor advancement, +or locked-context replacement may instead commit only the terminal stale +transition and bounded evidence, with no Submission or binding. ART/TASK never +imports or locks AUTH-owned tables. ## Durable Verification And Recovery @@ -394,20 +471,29 @@ exact resource/content/job/attempt filters. ## Product Cutover -1. Guide source capture stores exact bytes and binds the immutable source - snapshot to content. -2. Task-scoped upload sessions accept contributor artifacts and seal one exact - server artifact set. -3. Authoritative pre-submit executes against sealed verified bytes. - One authorized artifact materializer allocates through the canonical scratch - manager, recomputes every digest and byte count from the provider read, seals - the verified workspace read-only, and is reused unchanged by post-submit. -4. Submission creation consumes the same admission and creates immutable - bindings; contributor finalization is not a manager action. -5. Post-submit dispatch reads the same content commitment and stores checker - inputs, logs, and outputs as artifacts. -6. Review packet/evidence integration remains WS-REV, but it consumes the same - `ArtifactBinding` model. +1. Guide-source delivery is split into hidden byte ingest, AUTH activation, + verified snapshot binding/materialization, AUTH activation, and the legacy + identity/continuation clean cut. +2. Contributor intake accepts one outer ZIP, inspects and manifests its complete + tree in bounded private scratch, and rejects exact or semantic unchanged work. +3. Mandatory platform gates and locked Project Guide pre-submit checks execute + against that same scratch tree. Failure produces no durable bytes. +4. Passing bytes enter the existing immutable store once. Complete read-back + verification publishes one bindable admission; ambiguity uses existing ART + recovery. +5. Submission creation consumes that exact admission and creates one immutable + `Submission` version and binding; contributor finalization is not a manager + action and no competing `SubmissionVersion` table is introduced. +6. Post-submit dispatch materializes the same binding, recomputes integrity, and + stores checker logs/outputs as separately bound artifacts. +7. Review packet/evidence integration remains WS-REV. REV attaches a decision + (`accept`, `needs_revision`, or `reject`) and note/findings to the exact + Submission; the reviewer uploads no revision artifact. +8. A contributor response to `needs_revision` is another complete ZIP and + immutable Submission linked to the prior Submission and exact Review. +9. CON and future delivery own their lifecycle records but reference the same + accepted Submission/binding. Full-byte streams recompute SHA-256 and byte + count and fail closed on mismatch. The existing misleading `/submissions/{id}/finalize` endpoint is not retained as a normal handoff. Its genuine exceptional behavior moves to diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/RISKS.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/RISKS.md index fa254d12..c64b8b41 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/RISKS.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/RISKS.md @@ -27,11 +27,11 @@ | Concurrent prepared artifacts exhaust worker disk | High | Full-max pre-reservation, dedicated quota, cross-process ledger, bounded concurrency, free-space floor, and deadline-before-TTL cleanup. | | Any guide, contributor, or checker producer creates unbounded retained bytes | Critical | One generic admission service atomically caps cumulative unique provisional/completed bytes at every applicable task, producer, project, and deployment scope before provider I/O; only authoritative absence releases a provisional charge. | | Operators cannot diagnose or expand exhausted storage capacity | High | Bounded admission-usage API, scope metrics/alerts, and a configuration-driven expansion/rollback runbook; no charge deletion or database editing. | -| Expired upload sessions retain open-session capacity forever | High | PostgreSQL-clock periodic Celery expiry plus lazy expiry before admission perform one atomic terminal transition and release the session slot exactly once. | +| Legacy planned upload-session actions accidentally become a second intake path | Critical | ART-04A removes only ART-side reachability. The AUTH registration contract must retire, or prove unavailable and route-less, the AUTH-owned ActionIds and routes before AUTH may activate the single bundle action. | | Recovery replay or ancestor reuse creates multiple jobs | High | Canonical request digest plus a lifetime unique source-job constraint return original IDs on exact replay, reject different-key/concurrent reuse, and require chain progression through the immediate retry job. | | Operator cannot observe retry completion | High | Authorized recovery-attempt GET includes immutable source-job and current retry-job status plus audit IDs. | | Storage failure becomes contributor failure or review decision | Critical | Stable infrastructure errors; no accept/needs_revision/reject, contribution, payment, or reputation effect. | -| Pre-submit outage strands or duplicates a contributor attempt | High | Bounded Workstream retry, stable 503, sealed unconsumed artifact set, exact request-digest idempotency, and contributor continuation without manager/operator approval. | +| Pre-submit process or scratch loss strands an attempt | High | Return a stable infrastructure outcome, clean scratch, and require reupload without manager/operator approval; only post-intent ambiguity uses existing durable ART recovery. | | Pre-existing lifecycle rule deletes evidence | Critical | Read-only AWS lifecycle inspection blocks activation for any deletion/expiration rule intersecting the completed-object prefix. | | Local adapter reaches production | High | Closed backend enum and production startup rejection. | | MinIO proof is mistaken for AWS readiness | High | Shared conformance runs on LocalStorage/MinIO, while AWS has a separate private-bucket, workload-identity, lifecycle, and anonymous-read live proof. | @@ -39,3 +39,16 @@ | R2 remains a hidden v0.1 dependency | High | No active R2 chunk/configuration/runtime path; stale-contract gate rejects reintroduction outside a separately approved initiative. | | Route tests overclaim fleet atomicity | Medium | Per-image proof only; deployment barrier documented separately. | | Large chunks become unreviewable | High | Preparation, v2 cutover, provider profiles, verification, recovery, upload/sealing, pre-submit admission, checker input, and checker output/routing are separate checkpoints with explicit allowed files and stop conditions. | +| Temporary candidate storage consumes capacity or creates a second lifecycle | Critical | Candidate storage is rejected for v0.1. Unchecked bytes remain only in bounded scratch and passing bytes enter the existing immutable store once. | +| Outer ZIP tree cannot be checked against the Project Guide | Critical | Safely walk every normalized file/directory entry in the outer ZIP, hash files during bounded materialization, and give the locked checker the exact read-only tree. Nested ZIP entries stay opaque throughout v0.1. | +| Repackaging unchanged work wastes reviewer capacity | High | Compare both exact archive identity and canonical semantic-manifest identity with the immediate prior immutable Submission before checker/provider I/O. | +| Scratch path is sent to another process or survives process loss | Critical | Keep 04A-04C one process-local orchestrated operation; never serialize scratch paths. Process/scratch loss invalidates the attempt and requires reupload. | +| Verified bytes become unbound after process/transaction loss | High | Persist exact durable intent and use idempotent retry plus existing put/verification recovery; create no Submission until verified content and admission are bindable. | +| Contributor abandons a verified admission | High | Treat unbound `ready` as a valid capacity-charged state, expose bounded Operator counts/bytes, enforce existing quotas, and add no expiry, release, deletion, or retention process. | +| One admission creates multiple Submissions | Critical | Atomic `ready -> consumed` transition, row lock/CAS, unique Submission admission reference, terminal-state constraints, and concurrent-consumption tests. | +| ZIP executable intent is lost or arbitrary permissions escape | High | Normalize only valid Unix execute bits for regular files into the semantic manifest; default other entries false, reject special entries, project fixed sealed modes, and never preserve arbitrary archive permissions. | +| Authority changes during long preparation or before binding | Critical | Consume fresh AUTH-owned prepared capabilities in both durable transactions; commit authorization evidence with the protected mutation, perform no provider I/O before the first commit, and never let human authority substitute for fixed binding authority. | +| Reviewer note is confused with a revision artifact | High | REV attaches decision and note/findings to the exact Submission. Only a contributor uploads the next complete ZIP after `needs_revision`. | +| Full history requires expensive recursive traversal | Medium | Preserve immutable links but add indexed latest/current/accepted repository queries or projections maintained under lifecycle transactions. | +| Stored content changes after initial verification | Critical | Recompute full SHA-256 and byte count during checker materialization, reviewer full download, and downstream delivery; fail closed and record an artifact incident on mismatch. | +| Speculative larger limits exhaust disk or processing time | High | Preserve current configured limits until separately reviewed capacity, concurrency, runtime, cost, and abuse evidence justifies a change. | diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md index 3bb7ed80..3bb2d701 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md @@ -1,72 +1,66 @@ # Status: WS-ART-001 Immutable Artifact Storage -## Current State +## Completed Foundation -Original planning merged through PR #97, artifact/LocalStorage foundation -merged through PR #101, the AWS-first object-storage amendment merged through -PR #120 as `4408256`, the external-service adapter foundation merged through -PR #127 as `f64a8e5`, committed-source preparation merged through PR #129 as -`9a04434`, the ArtifactStore v2 Local clean cut merged through PR #141 as -`a10d901`, and S3-compatible MinIO/AWS preparation merged through PR #151 as -`1b5422fc` on 2026-07-19. ART admission and the put-attempt foundation then -merged through PR #154 as `44f2467c`, and the user explicitly started -`WS-ART-001-02C2` on 2026-07-19. +Planning and the artifact foundation merged through PR #97 and PR #101. The +AWS-first object-storage amendment and typed adapter clean cut merged through +PRs #120, #127, #129, #141, and #151. Durable admission, put attempts, +verification/publication, recovery idempotency, and hidden Operator operations +merged through PRs #154, #159, #174, and #177 (`WS-ART-001-02D`). -The planning-only cross-initiative boundary reconciliation merged through -PR #139 as `5d353b6`, and AUTH's owner reconciliation merged through PR #140 as -`d541521`. ART now consumes AUTH's canonical activation-custody and prepared -mutation contracts without editing or activating AUTH runtime behavior. -AUTH-09D-A merged through PR #148 as `99ae4c9`, AUTH-09D-B merged through PR -#152 as `93dd392`, the contributor foundation merged through PR #153 as -`8d5eb15b`, AUTH-09E merged through PR #157 as `42a89b2d`, and the -availability-neutral ART custody transfer merged through PR #158 as -`be2a79a2`, and the unrelated availability-neutral REV custody transfer merged -through PR #160 as `fe0e4492`. Backend CI sharding and timeout repair merged -through PRs #163 and #164 as `b0f9ad64` and `61bc0390`; signed-start loop-memory -planning merged through PR #165 as `58d0514a`; and AUTH-PREP merged through PR -#162 as `c559d556`. All are integrated into the ART candidate. AUTH-PREP adds no -ART consumer or activation. The three ART internal feature actions remain -assigned to future -`WS-AUTH-001-ART-02D-INTERNAL` activation custody, but remain planned and -inactive. +The current v0.1 provider direction remains AWS S3 in production, MinIO for +local/CI protocol proof, and LocalStorage for development/focused tests. Flow +Node and R2 remain deferred. Completed objects have no physical deletion path. +AUTH's owner reconciliation merged through PR #140 as +`d541521`; PLAN2 preserves AUTH ownership and proposes no availability edit. -The Flow Node-focused amendment candidate `6cc422d` passed deterministic checks -but failed internal review on recovery/API completeness. Before repair, the user -approved a first-principle change on 2026-07-14: +## Cancelled Work + +`WS-ART-001-03` received a signed implementation start on current history, but +mandatory preimplementation review rejected the combined contract before any +runtime edit. The user authorized cancellation, and signed automation run +`30100940860` recorded `stopped_after_cancel` on 2026-07-24. The rejected +contract combined guide byte ingest, binding, materialization, setup recovery, +migration, and inactive AUTH dependencies without a safe executable boundary. + +## Current Planning Reconciliation + +`WS-ART-001-PLAN2` is planning-only. It incorporates the human-approved +submission invariant: ```text -v0.1 production bytes -> S3CompatibleArtifactStore -> AWS S3 -local/CI proof bytes -> S3CompatibleArtifactStore -> MinIO -development bytes -> LocalStorageAdapter -future optional bytes -> Flow Node adapter initiative +one outer ZIP +-> bounded private scratch inspection and canonical manifest +-> exact/semantic unchanged rejection +-> mandatory platform and locked Project Guide prechecks +-> one existing ArtifactStore admission and complete read-back verification +-> one immutable Submission binding +-> the same bytes checked, reviewed, accepted, recorded, and delivered ``` -The failed Flow Node candidate and every reviewer session are closed. It is not -approval or reusable evidence. Its source remains on branch -`codex/ws-art-001-fn01-isolation-amendment` for the deferred Flow Node plan. - -## Current Work +There is no candidate store, temporary provider retention, promotion copy, +physical deletion, second recovery aggregate, speculative capacity increase, or +competing `SubmissionVersion` table. Reviewers attach a decision plus +note/findings to the +exact `Submission`; contributors answer `needs_revision` with another complete +ZIP and immutable Submission. -`WS-ART-001-02C2` is active after the user's explicit start on 2026-07-19. It -adds caller-only committed put execution, read-only ambiguous-put resolution, -durable verification jobs and typed receipts, PostgreSQL executor/generation -fencing, bounded publication scanning, and complete-read deadlines. Production -composition remains deny-only: the three internal artifact actions stay -planned, no 02C2 Beat schedule is active, and AUTH retains sole activation -custody. Recovery attempts, Operator routes, product cutovers, deletion, and -background write replay remain out of scope. +PLAN2 also treats client-abandoned verified admissions as valid, bounded, +capacity-charged `ready` facts with terminal `consumed|stale` outcomes; it +normalizes regular-file executable intent into the semantic manifest; and it +requires fresh AUTH prepared capabilities at durable put intent and atomic +Submission/binding consumption. ## Next Proposed Chunk -`02C3` may add the recovery-attempt and idempotency chain only after 02C2 merges -and receives a separate explicit start. No deferred provider has a v0.1 chunk. +After this planning package merges, `WS-ART-001-03A` is the only immediate ART +successor. It adds hidden guide-source byte ingest through the existing artifact +preparation/admission/verification path. It requires a separate signed start and +does not activate its own AUTH action. ## Gate -The candidate integrates trusted main `c559d556`, the sharded Backend workflow, -and merged AUTH-PREP without activating ART. Exact-SHA internal evidence is -complete for reviewed code `59fbab56`; hosted checks must rerun on the published -evidence head. The current gate is external CI/review followed by explicit human -merge approval. -Production activation and recovery remain in later owning chunks. No later -artifact chunk starts automatically, and only the user may approve merge. +Planning evidence and all required internal reviewer tracks must pass before a +PR. The planning merge starts no successor. Every implementation chunk retains +its separate signed start, exact AUTH activation sequence, internal review, CI, +CodeRabbit, human checkpoint, and automated merge-memory stop. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03-guide-source-cutover.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03-guide-source-cutover.md index 05324f14..6e61a7a1 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03-guide-source-cutover.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03-guide-source-cutover.md @@ -1,6 +1,12 @@ # Chunk Contract: WS-ART-001-03 - Guide Source Artifact Cutover -Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 02D +Initiative: `WS-ART-001` | Risk: L1 | Status: Cancelled before implementation; superseded by 03A-03C + +This contract was signed and then cancelled after mandatory preimplementation +review found that it combined guide byte intake, binding, setup-agent reads, +recovery continuation, migration, and inactive AUTH dependencies without an +implementable byte-transport or activation sequence. It authorizes no work and +is retained only as history. Artifact contract phase: `guide_source_cutover` diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03A-guide-source-byte-ingest.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03A-guide-source-byte-ingest.md new file mode 100644 index 00000000..ce9e7f13 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03A-guide-source-byte-ingest.md @@ -0,0 +1,73 @@ +# Chunk Contract: WS-ART-001-03A - Guide Source Byte Ingest + +Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after PLAN2 + +Artifact contract phase: `guide_source_cutover` + +## Goal + +Add hidden, authorized guide-source byte ingest through the existing preparation, +admission, put-attempt, verification, and content-publication path without guide +snapshot binding, setup-agent reads, or public activation. + +## Allowed Files + +- project guide-source ingest models, one migration, repository, schemas, + service, and router; +- ART guide ingest capability, canonical resource facts/guards, and composition; +- existing prepared-source/admission call sites only for this producer; +- focused migration, authorization, ingest, ambiguity, and API tests; +- backend workflow/agent-gate files only to add and preserve exact coverage; +- related docs and chunk memory. + +## Not Allowed + +- snapshot binding/activation, setup materialization/continuation, or legacy + schema removal; +- task/submission/checker/review changes; +- AUTH-owned catalogue/evaluator/grant/identity/matrix/availability edits; +- provider/factory changes, compatibility aliases, or network URL fetching. + +## Acceptance Criteria + +- authorized caller-supplied/import-pipeline bytes cross `PreparedArtifact` and + generic admission once; caller hashes and provider references are never truth; +- `ArtifactContent` is published only after complete read-back verification; +- exact retry and ambiguity use existing put/recovery state; +- legacy hash-only/provider-object guide ingest is unavailable on the hidden + path, but destructive schema removal waits for 03C; +- `artifact.guide_source.ingest` behavior remains fail-closed until AUTH + activates the exact action after this hidden implementation merges; +- changed subsystem coverage is at least 90 percent and repository coverage + remains at least 78 percent. + +## Exact CI Coverage Gates + +```bash +coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifact_operations.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 +coverage report --include='app/interfaces/external_services.py' --precision=2 --fail-under=90 +coverage report --include='app/core/config.py' --precision=2 --fail-under=90 +coverage report --include='app/workers/*' --precision=2 --fail-under=90 +coverage report --include='app/main.py' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 +coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 +coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 +coverage report --include='app/api/router.py' --precision=2 --fail-under=90 +coverage report --include='app/modules/projects/*' --precision=2 --fail-under=90 +``` + +## Verification + +```bash +docker compose up -d --wait postgres redis minio +(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_alembic.py tests/test_guide_artifacts.py tests/test_artifact_admission.py tests/test_artifact_put_resolution.py tests/test_artifact_verification.py -q --cov=app.modules.projects --cov=app.modules.artifacts --cov-report=term-missing --cov-fail-under=90) +(metadata_dir="$(mktemp -d)" && trap 'rm -rf "$metadata_dir"' EXIT && (cd backend && WORKSTREAM_TEST_ADMIN_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/postgres .venv/bin/python scripts/run_isolated_tests.py --metadata-json "$metadata_dir/result.json" --timeout-seconds 12600 -- .venv/bin/python -m pytest -q --ignore=tests/test_isolated_database_runner.py --cov=app --cov-report=term-missing --cov-fail-under=78)) +(cd backend && .venv/bin/ruff check app tests) +python3 scripts/check_stale_artifact_contracts.py +python3 scripts/test_agent_gates.py +``` + +## Required Reviewers + +Senior engineering, architecture, QA/test, security/auth, product/ops, +reuse/dedup, CI integrity, test delta, and docs. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B-guide-source-binding-materialization.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B-guide-source-binding-materialization.md new file mode 100644 index 00000000..7a177641 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B-guide-source-binding-materialization.md @@ -0,0 +1,72 @@ +# Chunk Contract: WS-ART-001-03B - Guide Source Binding And Materialization + +Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 03A and exact AUTH activation + +Artifact contract phase: `guide_source_cutover` + +## Goal + +Bind verified guide-source content to snapshot items and give setup agents one +authorized, provider-neutral, integrity-checking read/materialization capability +without performing the legacy schema cutover. + +## Allowed Files + +- guide snapshot/item binding models, one migration, repository, schemas, and service; +- ART guide read/binding capability and canonical resource facts/guards; +- existing artifact scratch manager/materializer integration for setup input; +- setup-agent input assembly consuming the typed ART capability; +- focused migration, authorization, binding, materialization, and integrity tests; +- backend workflow/agent-gate files only to add and preserve exact coverage; +- related docs and chunk memory. + +## Not Allowed + +- contributor submission/checker/review behavior; +- direct provider access, URL fetching, a second materializer/scratch manager, + setup policy decisions, or destructive legacy-field removal; +- AUTH-owned catalogue/evaluator/grant/identity/matrix/availability edits. + +## Acceptance Criteria + +- snapshot items bind exact verified `ArtifactContent` without circularly making + the item both the source and authority for that content; +- snapshot/bundle identity commits to ordered content identities and descriptors; +- setup agents receive bounded bytes/manifests only through ART and every full + read recomputes SHA-256 and byte count; +- missing/mismatched bound bytes are artifact incidents, not guide insufficiency; +- fixed guide read/binding actions remain fail-closed until AUTH activation; +- changed subsystem coverage is at least 90 percent and repository coverage + remains at least 78 percent. + +## Exact CI Coverage Gates + +```bash +coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifact_operations.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 +coverage report --include='app/interfaces/external_services.py' --precision=2 --fail-under=90 +coverage report --include='app/core/config.py' --precision=2 --fail-under=90 +coverage report --include='app/workers/*' --precision=2 --fail-under=90 +coverage report --include='app/main.py' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 +coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 +coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 +coverage report --include='app/api/router.py' --precision=2 --fail-under=90 +coverage report --include='app/modules/projects/*' --precision=2 --fail-under=90 +coverage report --include='app/adapters/project_agents/*,app/interfaces/project_agents.py' --precision=2 --fail-under=90 +``` + +## Verification + +```bash +docker compose up -d --wait postgres redis minio +(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_alembic.py tests/test_guide_artifacts.py tests/test_project_setup.py tests/test_checker_materialization.py tests/test_artifact_scratch_manager.py -q --cov=app.modules.projects --cov=app.modules.artifacts --cov-report=term-missing --cov-fail-under=90) +(metadata_dir="$(mktemp -d)" && trap 'rm -rf "$metadata_dir"' EXIT && (cd backend && WORKSTREAM_TEST_ADMIN_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/postgres .venv/bin/python scripts/run_isolated_tests.py --metadata-json "$metadata_dir/result.json" --timeout-seconds 12600 -- .venv/bin/python -m pytest -q --ignore=tests/test_isolated_database_runner.py --cov=app --cov-report=term-missing --cov-fail-under=78)) +(cd backend && .venv/bin/ruff check app tests) +python3 scripts/check_stale_artifact_contracts.py +python3 scripts/test_agent_gates.py +``` + +## Required Reviewers + +Senior engineering, architecture, QA/test, security/auth, product/ops, +reuse/dedup, CI integrity, test delta, and docs. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03C-guide-source-cutover-continuation.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03C-guide-source-cutover-continuation.md new file mode 100644 index 00000000..4a8380f6 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03C-guide-source-cutover-continuation.md @@ -0,0 +1,75 @@ +# Chunk Contract: WS-ART-001-03C - Guide Source Cutover And Continuation + +Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 03B and exact AUTH activation + +Artifact contract phase: `guide_source_cutover` + +## Goal + +Remove legacy guide-source identity, make verified ART bindings authoritative, +and resume the exact same setup generation after artifact recovery without +adding a Project Manager resume command. + +## Allowed Files + +- guide-source/snapshot models, one clean-cut migration, repository, schemas, + service, and router; +- exact same-generation setup continuation record/service and affected setup tasks; +- `backend/scripts/api_contract_e2e.py` only for the guide clean cut; +- focused refusal, continuation, recovery, API, and real-flow tests; +- stale-contract/agent-gate/backend workflow files only for the exact phase and + preserved coverage gates; +- related docs and chunk memory. + +## Not Allowed + +- task/submission/checker/review cutover, synthetic backfill, URL fetching, + provider/factory changes, or Project Manager resume command; +- AUTH-owned catalogue/evaluator/grant/identity/matrix/availability edits. + +## Acceptance Criteria + +- migration refuses unsafe populated legacy rows rather than fabricating bytes; +- caller `content_cid`, provider-object schemes, and hash-only identity are + removed without aliases after verified replacement is available; +- recovery continuation references a persisted setup run/generation and resumes + only when project, guide, snapshot, policy, and generation still match; +- Operator recovery authorization and automatic setup continuation remain + distinct audit facts; neither approves guide sufficiency; +- changed guide bytes create a new snapshot/setup generation, while locked tasks + retain their prior context unless explicitly rebased; +- changed subsystem coverage is at least 90 percent and repository coverage + remains at least 78 percent. + +## Exact CI Coverage Gates + +```bash +coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifact_operations.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 +coverage report --include='app/interfaces/external_services.py' --precision=2 --fail-under=90 +coverage report --include='app/core/config.py' --precision=2 --fail-under=90 +coverage report --include='app/workers/*' --precision=2 --fail-under=90 +coverage report --include='app/main.py' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 +coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 +coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 +coverage report --include='app/api/router.py' --precision=2 --fail-under=90 +coverage report --include='app/modules/projects/*' --precision=2 --fail-under=90 +coverage report --include='app/adapters/project_agents/*,app/interfaces/project_agents.py' --precision=2 --fail-under=90 +``` + +## Verification + +```bash +docker compose up -d --wait postgres redis minio +(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_alembic.py tests/test_projects.py tests/test_project_setup.py tests/test_guide_artifacts.py tests/test_artifact_recovery.py -q --cov=app.modules.projects --cov=app.modules.artifacts --cov=app.workers --cov-report=term-missing --cov-fail-under=90) +(metadata_dir="$(mktemp -d)" && trap 'rm -rf "$metadata_dir"' EXIT && (cd backend && WORKSTREAM_TEST_ADMIN_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/postgres .venv/bin/python scripts/run_isolated_tests.py --metadata-json "$metadata_dir/result.json" --timeout-seconds 12600 -- .venv/bin/python -m pytest -q --ignore=tests/test_isolated_database_runner.py --cov=app --cov-report=term-missing --cov-fail-under=78)) +(cd backend && WORKSTREAM_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/python scripts/api_contract_e2e.py) +(cd backend && .venv/bin/ruff check app tests) +python3 scripts/check_stale_artifact_contracts.py +python3 scripts/test_agent_gates.py +``` + +## Required Reviewers + +Senior engineering, architecture, QA/test, security/auth, product/ops, +reuse/dedup, CI integrity, test delta, and docs. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04A-upload-inspection-sealing.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04A-upload-inspection-sealing.md index e07b2d90..efe56046 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04A-upload-inspection-sealing.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04A-upload-inspection-sealing.md @@ -1,85 +1,96 @@ -# Chunk Contract: WS-ART-001-04A Upload, Inspection, And Sealing +# Chunk Contract: WS-ART-001-04A - One-ZIP Scratch Intake And Manifest -Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 03 +Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 03C and AUTH planned registration -Artifact contract phase: `guide_source_cutover` +Artifact contract phase: `upload_admission` ## Goal -Add contributor-authorized task upload sessions/items, trusted archive -inspection, independent verification, immutable sealing, and artifact-set -manifests without running pre-submit or creating submissions. +Accept exactly one contributor outer ZIP into bounded private scratch, safely +inspect its complete internal tree, and produce authoritative archive and +semantic-manifest commitments without provider I/O, project checker execution, +or Submission creation. ## Allowed Files -- artifact upload-session/item/set models and one migration; -- artifact repository/service/router/schemas for upload, inspect, seal, and - logical cancellation APIs and internal upload-session expiry; -- trusted archive/media inspection helpers; -- artifact call sites consuming exact decisions already delivered by the - approved WS-AUTH dependency; no Authorization Service owner files; -- focused migration, concurrency, upload, inspection, sealing, and real-API tests; -- `.github/workflows/backend.yml` only to expand the exact 90 percent scoped gate; -- `scripts/test_agent_gates.py` only to assert that backend CI retains this - chunk's exact scoped coverage sources and fail-closed 90 percent threshold; -- related docs and chunk memory. +- ART-owned scratch-intake, ZIP inspection, normalization, and manifest code; +- existing `ArtifactUploadSession`/`ArtifactUploadItem` models, migration, + repositories, schemas, and routes only to remove them or make them unreachable + as contributor intake; no retained provider/content-result staging semantics; +- task-scoped hidden upload surface and ART canonical resource facts/guards; +- one migration for the bounded intake/manifest control plane if required; +- focused archive, quota, concurrency, cleanup, fuzz/regression, API, coverage, + docs, and chunk-memory changes; +- CI files only to add and preserve the exact scoped 90 percent gate. ## Not Allowed -- pre-submit checker execution or admission records; -- submission creation/finalization changes; -- post-submit checker or review changes; -- presigned/direct provider upload or provider identifiers in responses; -- free-form client policy/checker overrides; -- physical provider deletion on cancellation. +- `ArtifactStore.put`, `ArtifactContent`, replica, binding, or provider I/O; +- project pre-submit checker execution or admission; +- Submission, Review, Contribution, compensation, reputation, or delivery work; +- multi-item submission sets, direct/presigned uploads, candidate storage, + retention windows, or physical deletion; +- AUTH-owned catalogue, evaluator, grant, identity, matrix, or activation edits; +- raising current configured limits. ## Acceptance Criteria -- only the assigned contributor with exact task authorization can create or - mutate a live task upload session; -- create, read, item write, seal, and cancel call the distinct task-scoped - permissions `artifact.upload_session.create`, - `artifact.upload_session.read`, `artifact.upload_item.write`, - `artifact.upload_session.seal`, and `artifact.upload_session.cancel` - through the central AUTH kernel with ART-owned canonical resource facts and - guards; this chunk first supplies hidden behavior while actions remain planned, - and AUTH alone activates them afterward; none implies another; -- every ID-addressed read/mutation uses concealed deny/not-found behavior and - is tested across actors, projects, revocation, terminal states, and random IDs; -- upload requires a server-approved logical role, bounded display name, client - SHA-256 commitment, expected byte count, and streamed file; -- Workstream prepares and hashes the complete stream, rejects mismatch before - provider I/O, and derives the key only from the server commitment; -- PostgreSQL atomically enforces configured open-session slots at task, actor, - project, and deployment scope; contributor item bytes use the generic 02C1 - admission service at task, authenticated actor, project, and deployment - scope after canonical hashing and before provider I/O; -- PostgreSQL-clock expiry is owned by a bounded periodic Celery scanner and is - also applied lazily before new-session admission. Both paths require the - fixed internal permission `artifact.upload_session.expire`; one atomic - terminal transition releases each open-session slot exactly once, and stale - scanner execution cannot release it again; -- cancellation and expiry release only the open-session slot. Provisional and - completed byte charges follow the generic admission state machine; - cancelled, expired, and unbound completed bytes remain charged in v0.1, - while exact deduplicated content is charged once per applicable scope; -- every item is independently verified before it can be sealed; -- limits cover item count, aggregate/per-file bytes, archive expansion, entry - count, path safety, nesting, and compression ratio; -- sealing row-locks the session/items, requires verified replicas, creates a - deterministic manifest/hash, and is idempotent under exact replay; -- cancellation is logical and cannot delete shared completed bytes; -- race tests cover upload/seal/cancel/expiry conflicts, repeated-session quota - exhaustion, concurrent quota admission without oversubscription, terminal - session-slot release by periodic and lazy expiry, retained charges for - cancelled/expired/unbound content, - and first-writer and cross-project digest-key poisoning attempts; -- changed subsystem coverage is at least 90 percent and repository coverage - remains at least 78 percent; -- backend CI installs this chunk's exact focused 90 percent gate, preserves - every earlier scoped 90 percent gate, and retains the exact 78 percent - repository command below; `scripts/test_agent_gates.py` fails on workflow - command, source-set, threshold, or cumulative-retention drift. +- a contributor attempt accepts exactly one outer ZIP and rejects every other + archive form or additional item; +- Workstream streams, sizes, and hashes the exact ZIP while reserving bounded + aggregate scratch through the canonical `ArtifactScratchManager`; +- the outer ZIP tree is walked recursively without `extractall`; ZIP entries + contained inside that tree remain opaque ordinary files by default; +- path normalization rejects absolute/drive/UNC paths, `.`/`..`, NUL/control + characters, backslashes, symlinks/special entries, duplicate normalized paths, + Unicode-normalization collisions, and configured case-fold collisions; +- bounded streaming enforces current upload, extracted-byte, per-file, entry, + path-depth/path-length, compression-ratio, deadline, concurrency, and free-space + limits without allocating from untrusted declared sizes; +- the canonical semantic manifest records normalized file and directory paths, + entry type, file SHA-256/byte count, and `executable` for regular files; + valid Unix mode metadata normalizes any execute bit to true, while non-Unix + or invalid mode metadata defaults false and directories record null; +- timestamps, compression, comments, ownership, group, read/write/special bits, + and other platform permission metadata do not affect the manifest hash; +- same bytes with a changed normalized executable flag produce different + semantic hashes, while timestamp/compression-only repackaging does not; +- explicit empty directories and deterministic synthetic parents have one + documented canonical representation; +- exact archive hash and semantic manifest hash are compared with the immediate + prior immutable Submission under task/version locking; equality returns a + stable unchanged error before checker or provider I/O; +- the server-computed ZIP commitment is authoritative; an optional client + commitment can only cause early mismatch rejection; +- successful output is a process-local, lease-bound prepared bundle handle that + contains no path/provider reference in an API response and is unusable after + cleanup, expiry, process loss, or generation mismatch; +- that handle is metadata around the existing `ArtifactScratchManager`, + `PreparedArtifact`, and `CommittedArtifactSource` lifecycle, not a second + handle/lease/ledger/quota/cleanup abstraction; +- 04A exposes only internal hidden library seams and tests. It composes no + contributor route until 04C completes the same-process orchestration; +- every old multi-step upload-session/item route and action path is removed or + statically unreachable; no multi-item or provider-result compatibility path + remains; +- process loss requires reupload; normal, error, cancellation, timeout, and + crash cleanup release scratch quota without deleting durable artifacts; +- ZIP inspection never blindly applies archive modes; the scratch manager seals + regular files to fixed read-only or read-and-execute modes and directories to + a fixed read-and-traverse mode, with startup parity across every process that + shares the scratch root and secure cleanup owned by the manager; cleanup may + restore only manager-owned directory write access under an exclusive valid + lease after no-follow ownership/type checks; +- executable intent alone never causes execution; the locked Project Guide and + checker policy remain the execution decision owners; +- focused tests prove false-to-true execute-bit changes alter semantic hash, + timestamp/compression-only changes do not, non-Unix entries default false, + and symlink/special entries are rejected; +- 04A declares no contributor route or routable action manifest entry; the + planned `artifact.submission_bundle.prepare` action and AUTH mapping are + composed only by 04C after the complete internal operation exists; +- focused subsystem coverage is at least 90 percent and repository coverage + remains at least 78 percent. ## Exact CI Coverage Gates @@ -100,8 +111,8 @@ coverage report --include='app/adapters/project_agents/*,app/interfaces/project_ ## Verification ```bash -docker compose up -d --wait postgres redis minio -(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_alembic.py tests/test_artifact_upload_api.py -q --cov=app.modules.artifacts --cov-report=term-missing --cov-fail-under=90) +docker compose up -d --wait postgres redis +(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_alembic.py tests/test_submission_bundle_intake.py tests/test_submission_zip_safety.py tests/test_submission_bundle_manifest.py tests/test_artifact_scratch_manager.py -q --cov=app.modules.artifacts --cov-report=term-missing --cov-fail-under=90) (metadata_dir="$(mktemp -d)" && trap 'rm -rf "$metadata_dir"' EXIT && (cd backend && WORKSTREAM_TEST_ADMIN_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/postgres .venv/bin/python scripts/run_isolated_tests.py --metadata-json "$metadata_dir/result.json" --timeout-seconds 12600 -- .venv/bin/python -m pytest -q --ignore=tests/test_isolated_database_runner.py --cov=app --cov-report=term-missing --cov-fail-under=78)) (cd backend && .venv/bin/ruff check app tests) python3 scripts/check_stale_artifact_contracts.py @@ -115,6 +126,6 @@ reuse/dedup, CI integrity, test delta, and docs. ## Human Review Focus -- Can a client choose a provider key or bypass independent verification? -- Is the sealed artifact-set commitment deterministic and immutable? -- Are archive and resource-exhaustion boundaries explicit and tested? +- Can more than one outer ZIP or unchecked bytes escape scratch? +- Is recursive inspection clearly the outer archive tree, not nested unpacking? +- Does semantic equality ignore packaging-only changes without ignoring files? diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04B-pre-submit-admission.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04B-pre-submit-admission.md index 3da38010..8adc3cdb 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04B-pre-submit-admission.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04B-pre-submit-admission.md @@ -1,4 +1,4 @@ -# Chunk Contract: WS-ART-001-04B Pre-Submit Admission +# Chunk Contract: WS-ART-001-04B - Scratch-Bound Pre-Submission Checks Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 04A @@ -6,81 +6,79 @@ Artifact contract phase: `upload_admission` ## Goal -Execute the task's locked project pre-submit checker against one sealed -artifact set and persist an exact admission, including bounded continuation -when artifact infrastructure is temporarily unavailable. +Run mandatory Workstream archive gates and the task's locked Project Guide +pre-submission checker against the exact prepared ZIP tree while it remains in +bounded private scratch. Persist checker evidence, but create no durable +artifact content, provider object, or Submission. ## Allowed Files -- pre-submit attempt/admission models and one migration; -- task/checker/artifact input assembly, execution, repository, service, router, - schemas, and stable infrastructure error mapping; -- the canonical artifact scratch manager and one authorized, bounded, - digest-verifying checker materializer shared by both checker phases; -- artifact/task/checker call sites consuming exact decisions already delivered - by the approved WS-AUTH dependency; no Authorization Service owner files; -- focused migration, idempotency, concurrency, outage, checker, and real-API tests; -- `.github/workflows/backend.yml` only to expand the exact 90 percent scoped gate; -- `scripts/test_agent_gates.py` only to assert that backend CI retains this - chunk's exact scoped coverage sources and fail-closed 90 percent threshold; -- `scripts/check_stale_artifact_contracts.py` and its focused tests only to - advance the marker to `upload_admission`; -- related docs and chunk memory. +- ART/checker/task input assembly for one prepared bundle; +- the shared bounded, read-only scratch workspace/materialization capability; +- pre-submit attempt/result control-plane models and one migration; +- hidden task/checker surfaces with ART-owned canonical resource facts/guards; +- focused checker, timeout, cleanup, concurrency, API, coverage, docs, and + chunk-memory changes; +- CI files only to add and preserve the exact scoped 90 percent gate. ## Not Allowed -- submission creation or consumption of an admission; -- post-submit execution or review routing; -- any storage outage translated into checker failure, `needs_revision`, reject, - payment, contribution, or reputation state; -- Project Manager or Operator approval to continue a contributor precheck; -- client policy/checker/provider overrides. +- provider I/O, candidate storage, `ArtifactContent`, replica, or binding; +- passing admission publication or Submission creation; +- post-submit routing or Review/Contribution/delivery changes; +- checker interpretation as `accept`, `needs_revision`, or `reject`; +- manager/operator approval for an ordinary contributor retry; +- AUTH-owned activation changes or larger configured limits. ## Acceptance Criteria -- authoritative pre-submit executes the task's locked project checker against - the exact sealed artifact-set hash; -- checker workspace creation declares - `artifact.pre_submit.checker_input.materialize`, mapped to the fixed service - permission `artifact.checker_input.materialize`; contributor upload - permissions do not authorize materialization; -- pre-submit passes a provider-neutral materialization request for the sealed - upload artifact set and resolves only its verified `ready` items; it does not - require or create a product binding before submission creation. Post-submit - later uses the same materializer with immutable binding IDs; -- the shared materializer reserves its complete checker workspace in the - canonical scratch ledger and recomputes every SHA-256 and byte count while - reading the exact provider bytes; -- a mismatch fails closed as an artifact incident before checker execution; - verified files are sealed read-only before the checker receives the workspace - handle, and the handle cannot expose provider references or credentials; -- the canonical manager owns aggregate quota, private/no-follow paths, normal - completion and cancellation cleanup, crash/stale cleanup, and fail-closed - exhaustion for both source preparation and checker workspace allocations; -- a passing admission binds actor, task, session, artifact set, locked guide/ - policy/checker context, canonical request digest, checker result, and expiry; -- checker failure returns structured results and creates no admission, - submission, product review decision, or contributor outcome; -- Workstream owns a bounded transient retry budget for artifact reads; -- exhausted transient storage failure returns HTTP 503 with stable code - `pre_submission_infrastructure_unavailable`, not checker failure; -- on infrastructure unavailability the sealed session and artifact set remain - sealed, unconsumed, and reusable until normal expiry; only infrastructure - attempt and audit facts are persisted; -- idempotency scope includes actor, task, sealed session/artifact set, locked - context, client key, and canonical request digest; exact replay continues or - returns the same attempt, changed replay conflicts, and concurrent replay - creates no duplicate admission or attempt; -- after retry-after or infrastructure recovery, the contributor can continue - the same exact attempt without manager/operator approval; -- changed subsystem coverage is at least 90 percent and repository coverage - remains at least 78 percent; -- backend CI installs this chunk's exact focused 90 percent gate, preserves - every earlier scoped 90 percent gate, and retains the exact 78 percent - repository command below; `scripts/test_agent_gates.py` fails on workflow - command, source-set, threshold, or cumulative-retention drift; -- the stale-contract phase advances to `upload_admission` only after the sealed - artifact admission path is active and the phase scan passes. +- platform-owned format, safety, manifest, change, and scratch-integrity gates + cannot be disabled or weakened by a Project Guide; +- the project checker uses the exact task-locked guide/policy/checker context and + may require content such as `task.toml`, directories, evidence, configuration, + or tests from the safely materialized outer-ZIP tree; +- project policy may narrow platform limits but cannot raise them; +- checker input records the archive commitment, manifest ID/hash, task, actor, + locked policy context, and scratch generation used; +- durable attempt/evidence records are actor/task/attempt scoped, contain no raw + bytes, filenames, scratch/provider references, credentials, or unbounded + checker output, and follow the existing audit retention policy; +- v0.1 exposes no ID-addressed pre-submit-evidence read route. The authorized + contributor receives only the bounded redacted result of the same continuous + request; existing authorized audit/Operator surfaces may diagnose metadata, + while reviewers, CON, and delivery cannot consume it before Submission binding; +- the checker receives a read-only private workspace and no provider reference, + credential, mutable intake path, or arbitrary host path; +- the workspace projects the manifest's normalized executable flag using the + same fixed sealed modes as post-submit materialization; it never preserves + arbitrary ZIP permission metadata or executes a file merely because the flag + is true; +- checker workspace creation uses fixed-service ActionId + `artifact.pre_submit.checker_input.materialize`, mapped to PermissionId + `artifact.checker_input.materialize`; contributor preparation authority never + grants or substitutes for that service authority; +- a checker failure creates structured findings only: no durable artifact, + admission, Submission, Review decision, contribution, payment, reputation, or + provider charge; +- infrastructure failure remains a stable retryable infrastructure outcome and + never becomes contributor blame or product review state; +- successful checks create a short-lived, single-use passing result bound to the + same scratch handle and exact predecessor observed in 04A; they do not make + bytes authoritative or durable; +- 04B remains an internal hidden library seam with no separately invocable + contributor route; 04C composes the one continuous same-process endpoint; +- the complete operation remains process-local through 04C; process loss or + scratch loss invalidates the result and requires reupload rather than routing + a later distributed service to a local path; +- completion, failure, cancellation, deadline, and abandoned-work cleanup are + bounded and idempotent; +- focused tests prove pre-submit and post-submit views expose identical + executable semantics for Unix, non-Unix, symlink/special, and permission-only + revision cases; +- the continuous contributor surface remains hidden; this chunk does not + activate the planned contributor action or change fixed service grants; +- focused subsystem coverage is at least 90 percent and repository coverage + remains at least 78 percent. ## Exact CI Coverage Gates @@ -103,8 +101,8 @@ coverage report --include='app/modules/checkers/*' --precision=2 --fail-under=90 ## Verification ```bash -docker compose up -d --wait postgres redis minio -(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_alembic.py tests/test_submission_precheck.py tests/test_artifact_precheck_outage.py tests/test_checker_materialization.py tests/test_artifact_scratch_manager.py -q --cov=app.modules.artifacts --cov=app.modules.tasks --cov=app.modules.checkers --cov-report=term-missing --cov-fail-under=90) +docker compose up -d --wait postgres redis +(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_alembic.py tests/test_submission_precheck.py tests/test_submission_precheck_scratch.py tests/test_checker_materialization.py tests/test_artifact_scratch_manager.py -q --cov=app.modules.artifacts --cov=app.modules.checkers --cov-report=term-missing --cov-fail-under=90) (metadata_dir="$(mktemp -d)" && trap 'rm -rf "$metadata_dir"' EXIT && (cd backend && WORKSTREAM_TEST_ADMIN_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/postgres .venv/bin/python scripts/run_isolated_tests.py --metadata-json "$metadata_dir/result.json" --timeout-seconds 12600 -- .venv/bin/python -m pytest -q --ignore=tests/test_isolated_database_runner.py --cov=app --cov-report=term-missing --cov-fail-under=78)) (cd backend && .venv/bin/ruff check app tests) python3 scripts/check_stale_artifact_contracts.py @@ -118,8 +116,6 @@ reuse/dedup, CI integrity, test delta, and docs. ## Human Review Focus -- Is the exact staged payload bound to one admission? -- Can infrastructure failure ever become contributor blame or product state? -- Are retry and idempotency semantics exact under concurrency and replay? -- Does pre-submit execute only the read-only bytes whose digest and size it - recomputed through the shared materializer? +- Can a failed or unchecked bundle reach object storage? +- Can an async/distributed service receive an unusable local scratch path? +- Are checker findings separate from reviewer decisions? diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04C-verified-submission-bundle-admission.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04C-verified-submission-bundle-admission.md new file mode 100644 index 00000000..31ca953e --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04C-verified-submission-bundle-admission.md @@ -0,0 +1,138 @@ +# Chunk Contract: WS-ART-001-04C - Verified Submission Bundle Admission + +Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 04B + +Artifact contract phase: `upload_admission` + +## Goal + +Consume one passing scratch-bound pre-submit result, write the exact outer ZIP +once through the existing immutable `ArtifactStore` admission path, independently +verify it, and publish one bindable submission-bundle admission. + +## Allowed Files + +- ART admission orchestration and one submission-bundle admission model/migration + with the closed `ready|consumed|stale` lifecycle and database constraints; +- existing put-attempt, replica, receipt, verification-job, recovery, scanner, + and scratch-source integration needed for this producer; +- hidden submission-bundle admission surfaces and ART resource facts/guards; +- focused ambiguity, idempotency, verification, cleanup, API, coverage, docs, + and chunk-memory changes; +- CI files only to add and preserve the exact scoped 90 percent gate. + +## Not Allowed + +- a second provider namespace, candidate store, promotion/copy, retention + window, physical deletion, or new recovery aggregate; +- Submission creation or binding consumption; +- Review, Contribution, compensation, reputation, or delivery behavior; +- AUTH-owned activation changes or larger configured limits. + +## Acceptance Criteria + +- admission requires the exact unexpired passing result, prepared ZIP + commitment, semantic manifest, locked task context, actor, and predecessor; +- the predecessor is rechecked under task/version locking before provider I/O; + drift invalidates the attempt and requires a fresh upload/check; +- initial request authorization occurs before scratch intake; immediately before + durable capacity/put intent, the owning transaction consumes AUTH's fresh + prepared capability and typed TASK/PROJECT context capabilities without ART + importing or locking AUTH-owned tables; +- current actor/identity, project authority, assignment, task, predecessor, + locked context, canonical resource facts, authorization evidence, capacity, + and put intent validate/commit atomically; denial or drift causes no provider + I/O, durable admission, or Submission and cleans scratch; +- the existing generic admission service derives and reserves authoritative + byte scopes before one conditional `ArtifactStore.put`; +- Workstream persists the put attempt before I/O and uses existing observation, + receipts, verification jobs, scanners, generation fencing, and recovery; +- background verification, pending-work publication, and ambiguous-put + resolution retain their distinct fixed service actions + `artifact.verification.execute`, `artifact.pending_work.scan`, and + `artifact.put_attempt.resolve`; contributor preparation authority implies none; +- provider acknowledgement alone creates no bindable admission; +- complete provider read-back recomputes ZIP SHA-256 and byte count; only an + exact match publishes `ArtifactContent`, a ready replica, and one immutable + `ready` admission bound to preparation actor/identity provenance, project, + task, assignment, predecessor, exact locked context, manifest, and immutable + checker-evidence set; +- the admission lifecycle is exactly `ready -> consumed|stale`; only 05 may + perform either terminal transition and both terminal states are irreversible; +- provider absence, mismatch, ambiguity, or unavailability creates no + Submission and never becomes checker failure or review state; +- exact replay is idempotent and concurrent replay creates one provider/business + effect; no `ArtifactOutboxRecoveryAttempt` or receipt-lookup port is added; +- scratch is cleaned after safe source handoff or terminal outcome; if process + loss occurs before durable put intent, reupload is required; after durable + intent, existing ART recovery resolves the operation; +- client abandonment may leave a valid unbound `ready` admission; it creates no + product lifecycle effect and remains charged to existing completed-byte + scopes alongside terminal `stale`/`consumed` admissions; +- exact preparation replay returns the original admission with no new provider + write, charge, evidence set, or admission; v0.1 adds no expiry, release, + deletion, retention process, or cleanup route; +- Operator capacity projections expose bounded counts/bytes for unbound ready + and stale admissions through existing admission-usage authority; +- after 04A-04C merge their complete hidden manifest, AUTH alone may activate + `artifact.submission_bundle.prepare`; ART changes no action availability; +- 04C alone declares the hidden contributor route and routable + `artifact.submission_bundle.prepare` action, mapped by AUTH to + `submission.create`, with ART-owned task/assignment/actor facts and guards; + old upload-session actions are never aliases or alternate routes; +- this chunk alone composes the hidden contributor endpoint across the 04A + intake, 04B checks, and 04C durable handoff in one process-local operation; + no scratch handle/path is serialized into a later request or Celery payload; +- v0.1 exposes no separate preparation-status GET route; an exact idempotent + POST retry may return only the same bounded durable operation/admission state + after a fresh `artifact.submission_bundle.prepare` decision; it never relies + on the original human authorization evidence; +- revocation after committed put intent does not cancel fixed-service technical + verification/recovery, but no later Submission consumption inherits the old + human decision; +- crossed-state tests revoke authority during upload and checker execution and + prove no durable put/provider I/O; revocation after durable intent permits + verification only; exact replay and abandoned-ready capacity projections are + single-effect and bounded; +- focused subsystem coverage is at least 90 percent and repository coverage + remains at least 78 percent. + +## Exact CI Coverage Gates + +```bash +coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifact_operations.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 +coverage report --include='app/interfaces/external_services.py' --precision=2 --fail-under=90 +coverage report --include='app/core/config.py' --precision=2 --fail-under=90 +coverage report --include='app/workers/*' --precision=2 --fail-under=90 +coverage report --include='app/main.py' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 +coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 +coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 +coverage report --include='app/api/router.py' --precision=2 --fail-under=90 +coverage report --include='app/modules/projects/*' --precision=2 --fail-under=90 +coverage report --include='app/adapters/project_agents/*,app/interfaces/project_agents.py' --precision=2 --fail-under=90 +coverage report --include='app/modules/tasks/*' --precision=2 --fail-under=90 +coverage report --include='app/modules/checkers/*' --precision=2 --fail-under=90 +``` + +## Verification + +```bash +docker compose up -d --wait postgres redis minio +(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_alembic.py tests/test_submission_bundle_admission.py tests/test_artifact_put_resolution.py tests/test_artifact_verification.py tests/test_artifact_recovery.py -q --cov=app.modules.artifacts --cov-report=term-missing --cov-fail-under=90) +(metadata_dir="$(mktemp -d)" && trap 'rm -rf "$metadata_dir"' EXIT && (cd backend && WORKSTREAM_TEST_ADMIN_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/postgres .venv/bin/python scripts/run_isolated_tests.py --metadata-json "$metadata_dir/result.json" --timeout-seconds 12600 -- .venv/bin/python -m pytest -q --ignore=tests/test_isolated_database_runner.py --cov=app --cov-report=term-missing --cov-fail-under=78)) +(cd backend && .venv/bin/ruff check app tests) +python3 scripts/check_stale_artifact_contracts.py +python3 scripts/test_agent_gates.py +``` + +## Required Reviewers + +Senior engineering, architecture, QA/test, security/auth, product/ops, +reuse/dedup, CI integrity, test delta, and docs. + +## Human Review Focus + +- Is the exact checked ZIP written once and independently verified? +- Does every ambiguous effect reuse existing ART recovery? +- Can a Submission be created before bindable verification? diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-05-submission-artifact-cutover.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-05-submission-artifact-cutover.md index c66fc262..da548d96 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-05-submission-artifact-cutover.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-05-submission-artifact-cutover.md @@ -1,91 +1,94 @@ -# Chunk Contract: WS-ART-001-05 Submission Artifact Cutover +# Chunk Contract: WS-ART-001-05 - Submission Bundle Binding Cutover -Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 04B +Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 04C and exact AUTH activation Artifact contract phase: `submission_cutover` ## Goal -Make submission creation atomically consume the exact passing admission and -sealed artifact set, create immutable submission bindings, and automatically -enter `evaluation_pending` without manager finalization. +Atomically consume one verified submission-bundle admission, create the next +immutable `Submission` row and exact artifact binding, and remove caller-owned +package transport/hash/manifest authority. ## Allowed Files -- submission/task models, migration, repository, schemas, service, and router -- artifact admission consumption and binding service -- project policy/compiler and checker contract files only to remove the legacy - caller storage-scheme, manifest/hash, and compiler-primitive fields owned by - this clean cut -- Celery post-submit dispatch handoff -- task/artifact call sites consuming exact decisions already delivered by the - approved WS-AUTH dependency; no Authorization Service owner files -- focused migration, race, API, and real-flow tests -- `.github/workflows/backend.yml` only to expand the exact 90 percent scoped gate -- `scripts/test_agent_gates.py` only to assert the exact workflow command, - source set, threshold, and cumulative retention -- `backend/scripts/week2_api_e2e.py` to migrate its complete upload, - pre-submit, and submission flow from legacy transport fields and the removed - `/finalize` route -- `backend/scripts/api_contract_e2e.py` only to migrate project-policy, - submission request, and response assertions from legacy storage fields to - upload-session/admission semantics -- `examples/terminal_benchmark/terminal_benchmark_api_e2e.py` only to migrate - its private local fixture and manual API example from removed transport and - storage-policy fields; this sanitized local example is not CI or acceptance - proof, and Chunk 07 owns standalone executable proof -- `scripts/check_stale_artifact_contracts.py` and its focused tests only to - advance the marker to `submission_cutover` -- related docs/chunk memory +- task/submission models, migration, repository, schemas, service, and router; +- ART admission consumption and binding capability; +- project/checker contracts only for the exact legacy transport-field cutover; +- post-submit dispatch handoff and affected real-API examples; +- `backend/scripts/api_contract_e2e.py` only to migrate the exact submission + request/response flow to verified submission-bundle admission; +- focused migration, concurrency, history, API, coverage, docs, and chunk memory; +- CI files only to add and preserve the exact scoped 90 percent gate. ## Not Allowed -- reviewer decision/revision implementation; -- manager/admin finalization of ordinary contributor submissions; -- caller package URI, package hash, artifact manifest, evidence IDs, provider - reference, or artifact-set hash; +- a competing `SubmissionVersion` table or alias; +- reviewer decision/note implementation, reviewer-uploaded revision files, or + Review/Contribution/delivery ownership; +- caller package URI/hash/manifest/evidence IDs, provider references, content + IDs, or artifact-set hashes; - direct provider calls from task/submission modules; -- checker-result interpretation as a human review decision. -- Authorization Service implementation, permission registration, or policy - changes. +- AUTH-owned activation changes or checker results interpreted as review. ## Acceptance Criteria -- immutable submission binding creation declares - `artifact.submission.binding.create`, mapped to the fixed service permission - `artifact.binding.create`; submission creation authority does not imply it; - -- request body contains only summary, contributor attestation, and upload - session ID. -- assigned contributor can create their own submission after exact pre-submit - admission passes; no manager action is required. -- one transaction locks and consumes admission/session/items once, allocates - server submission version, creates immutable bindings, stamps exact locked - policy/checker/artifact-set context, and transitions to - `evaluation_pending`. -- the exact payload checked is the payload submitted; changed summary, - attestation, task, actor, policy, checker, session, or artifact set fails. -- automatic post-submit dispatch is durable and duplicate-safe. -- concurrent/replayed creation yields one submission/business effect. -- the misleading `/submissions/{id}/finalize` route is replaced by the exact - Operator-only `/operator/submissions/{id}/pre-review-gate-repair` command; - existing API-only drill consumers move in the same clean cut. -- pre-review-gate repair requires exact authorization, reason, audit, and a - failed/stale automatic dispatch; it cannot re-finalize healthy submissions or - create another checker run. -- storage failure leaves infrastructure state and never blames contributor. -- no live project/task/checker contract retains the removed caller transport, - storage-policy, or compiler-primitive fields; immutable Alembic history and - test fixture literals are not rewritten merely to satisfy wording scans. -- changed subsystem coverage is at least 90 percent and repository coverage - does not decrease. -- backend CI installs this chunk's exact focused 90 percent gate, preserves - every earlier scoped 90 percent gate, and retains the exact 78 percent - repository command below; `scripts/test_agent_gates.py` fails on workflow - command, source-set, threshold, or cumulative-retention drift. -- the stale-contract phase advances to `submission_cutover`; active docs and - project/task/checker contracts contain no caller-owned provider URI - references and the phase scan passes. +- one `Submission` row is one immutable version and binds exactly one verified + outer ZIP plus its canonical semantic manifest and checker evidence; +- request body contains only contributor-authored product fields and the + opaque verified admission identifier; all artifact identity is server-owned; +- before revealing admission state/context, the route obtains a fresh human + authorization decision; AUTH denial remains concealed from unrelated actors; +- one transaction consumes fresh prepared human `submission.create` and fixed + service ActionId `artifact.submission.binding.create` capabilities (mapped to + PermissionId `artifact.binding.create`), locks task, assignment, admission and + predecessor, and recomposes final resource facts through typed capabilities; + ART and TASK never import AUTH-owned repositories; +- the transaction requires `ready`, matches actor profile, task, project, + assignment, predecessor and exact locked context, allocates the next version, + creates the immutable Submission and binding, and changes admission to + `consumed` before one commit; +- the admission records preparation identity-link provenance, while consumption + records its new decision/identity evidence and requires the same canonical + actor rather than an obsolete identity-link row; +- database status/terminal-field checks and a unique Submission admission + reference guarantee at most one consumer; `consumed` and `stale` are terminal; +- proven task closure, predecessor advancement, or locked-context replacement + changes a still-ready admission to `stale`; authority loss alone does not, so + restored authority may later consume a still-compatible ready admission; +- binding creation uses fixed-service ActionId + `artifact.submission.binding.create`, mapped to PermissionId + `artifact.binding.create`; the contributor's active + `artifact.submission_bundle.prepare` ActionId and `submission.create` + permission do not imply internal binding authority; +- version `N+1` records `supersedes_submission_id = N`; the exact relationship + to a `needs_revision` Review is added only through the reviewed REV/TASK joint + contract and cannot be fabricated from a note or client ID; +- exact replay/concurrency creates one Submission and one business effect; +- concurrent consumption yields one success and one exact replay or stable + conflict; cancellation, denial, service-action unavailability, context drift, + stale execution, or persistence failure creates no Submission/binding; +- after successful authorization, proven task closure, predecessor advancement, + or locked-context replacement may commit only the terminal stale transition + and bounded evidence; every other failed mutation rolls back authorization + evidence and admission transition together; +- stable authorized outcomes distinguish already-consumed, context-changed, and + stale admissions, but unauthorized callers learn no admission existence/state; +- crossed-state tests cover authority restoration for a compatible ready + admission, predecessor advancement to stale, disabled binding service action, + token-valid but revoked identity/grant/assignment, stale execution, and two + concurrent requests producing one business effect; +- latest/current/accepted access uses indexed repository queries or projections, + while the immutable version/review graph remains fully queryable; +- no manager finalization is required for an ordinary passing contributor + submission, and infrastructure failure never blames the contributor; +- all active callers and docs lose legacy package URI, caller hash, and caller + artifact-manifest authority without fabricated backfill for historical rows; +- migration explicitly preserves readable historical rows, refuses any unsafe + authoritative backfill, and documents rollback/cutover ordering; +- hidden behavior remains unavailable until AUTH activates its exact actions; +- focused subsystem coverage is at least 90 percent and repository coverage + remains at least 78 percent. ## Exact CI Coverage Gates @@ -109,11 +112,9 @@ coverage report --include='app/modules/checkers/*' --precision=2 --fail-under=90 ```bash docker compose up -d --wait postgres redis minio -(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_alembic.py tests/test_submission_api.py tests/test_submission_concurrency.py tests/test_projects.py tests/test_checkers.py -q --cov=app.modules.tasks --cov=app.modules.artifacts --cov=app.modules.projects --cov=app.modules.checkers --cov=app.adapters.project_agents --cov=app.interfaces.project_agents --cov=app.workers --cov-report=term-missing --cov-fail-under=90) +(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_alembic.py tests/test_submission_api.py tests/test_submission_concurrency.py tests/test_submission_history.py tests/test_projects.py tests/test_checkers.py -q --cov=app.modules.tasks --cov=app.modules.artifacts --cov=app.modules.projects --cov=app.modules.checkers --cov-report=term-missing --cov-fail-under=90) (metadata_dir="$(mktemp -d)" && trap 'rm -rf "$metadata_dir"' EXIT && (cd backend && WORKSTREAM_TEST_ADMIN_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/postgres .venv/bin/python scripts/run_isolated_tests.py --metadata-json "$metadata_dir/result.json" --timeout-seconds 12600 -- .venv/bin/python -m pytest -q --ignore=tests/test_isolated_database_runner.py --cov=app --cov-report=term-missing --cov-fail-under=78)) -(cd backend && WORKSTREAM_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/python scripts/week2_api_e2e.py) (cd backend && WORKSTREAM_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/python scripts/api_contract_e2e.py) -(cd backend && .venv/bin/python -m py_compile ../examples/terminal_benchmark/terminal_benchmark_api_e2e.py) (cd backend && .venv/bin/ruff check app tests) python3 scripts/check_stale_artifact_contracts.py python3 scripts/test_agent_gates.py @@ -126,8 +127,6 @@ reuse/dedup, CI integrity, test delta, and docs. ## Human Review Focus -- Does contributor intent produce and lock exactly one submission version? -- Is all manager finalization removed from the normal path? -- Is pre-submit payload binding authoritative and race-safe? -- Are all active API callers migrated without treating the private local - Terminal Benchmark example as reproducible proof? +- Does exactly one checked and verified ZIP become exactly one Submission? +- Is the current `Submission` aggregate preserved without a duplicate model? +- Is historical migration honest and fail closed? diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06A-checker-input-materialization.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06A-checker-input-materialization.md index 85147cf3..3ba3ab3d 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06A-checker-input-materialization.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06A-checker-input-materialization.md @@ -1,4 +1,4 @@ -# Chunk Contract: WS-ART-001-06A Checker Input And Materialization +# Chunk Contract: WS-ART-001-06A - Checker Input And Materialization Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 05 @@ -37,14 +37,15 @@ bounded isolated checker workspaces. ## Acceptance Criteria -- `CheckerInputSnapshot` commits to submission version, artifact set, exact - binding/content IDs, hashes/sizes, locked policy/checker versions, and checker +- `CheckerInputSnapshot` commits to submission version, submission-bundle + manifest, exact binding/content IDs, hashes/sizes, locked policy/checker versions, and checker implementation identity; -- pre-submit and post-submit input prove the same artifact-set hash; +- pre-submit evidence and post-submit input prove the same archive commitment, + semantic-manifest hash, and exact binding; - the post-submit runner receives only authorized immutable Workstream binding - IDs and reuses the canonical materializer that pre-submit invoked with a - sealed-ready upload-set source; both are closed forms of the same typed - materialization request; + IDs and reuses the same scratch-manager/materialization primitives without + accepting a pre-submit scratch handle; pre-submit was process-local and + post-submit is binding-only; - the fixed checker service principal declares `artifact.post_submit.checker_input.materialize`, mapped to `artifact.checker_input.materialize`; it does not authorize any binding-create @@ -57,6 +58,12 @@ bounded isolated checker workspaces. - files are sealed read-only after digest/size verification and before checker access; success, failure, cancellation, and crash/stale paths cannot orphan confidential bytes; +- materialization reproduces each manifest entry's normalized executable flag + with the same fixed read-only/read-and-execute/read-and-traverse modes used by + pre-submit scratch; arbitrary archive modes are never restored and executable + intent alone never authorizes execution; +- parity tests prove identical pre/post materialization semantics and reject any + mode configuration mismatch across processes sharing a scratch root; - transient unavailability leaves the post-submit task in `evaluation_pending`; missing/integrity mismatch blocks execution as an artifact incident and never becomes a review decision; diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-07-recovery-live-proof.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-07-recovery-live-proof.md index adeb5ec1..c8151334 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-07-recovery-live-proof.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-07-recovery-live-proof.md @@ -1,4 +1,4 @@ -# Chunk Contract: WS-ART-001-07 Real API And Provider Proof +# Chunk Contract: WS-ART-001-07 - Real API And Provider Proof Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 06B @@ -39,9 +39,11 @@ without direct database inspection or Terminal Benchmark product coupling. - a standalone project guide is ingested, verified, snapshotted, and processed through setup using real APIs and visible async status. -- a contributor uploads exact artifacts, seals them, runs pre-submit, creates a - submission, and automatically enters `evaluation_pending`. -- post-submit reads the same immutable artifact-set commitment and stores +- a contributor uploads one outer ZIP; Workstream inspects and checks it in + bounded scratch, admits and verifies it once, creates a submission, and + automatically enters `evaluation_pending`. +- post-submit reads the same immutable binding/archive/semantic-manifest + commitment and stores checker logs/outputs as verified artifacts. - post-submit remains `evaluation_pending` while execution or infrastructure retry is active. Durable completion preserves the existing checker-owned @@ -50,9 +52,9 @@ without direct database inspection or Terminal Benchmark product coupling. assignment, or decision. - provider outage produces infrastructure pending/retry, not contributor or review outcomes; restart recovers through the periodic scanner. -- pre-submit outage exhausts to `pre_submission_infrastructure_unavailable`, - preserves the exact sealed unconsumed attempt, and continues on contributor - retry without manager/operator approval; post-submit outage separately keeps +- pre-submit process/scratch loss requires reupload without manager/operator + approval; durable-admission ambiguity is resolved by existing ART recovery; + post-submit outage separately keeps `evaluation_pending` and uses checker retry infrastructure. - changed/truncated/missing bytes and stale/duplicate Celery executions are proven safely. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-PLAN2-submission-bundle-reconciliation.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-PLAN2-submission-bundle-reconciliation.md new file mode 100644 index 00000000..e92cf33b --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-PLAN2-submission-bundle-reconciliation.md @@ -0,0 +1,138 @@ +# Chunk Contract: WS-ART-001-PLAN2 - Submission Bundle Reconciliation + +Initiative: `WS-ART-001` | Risk: L1 | Status: Planning only + +## Goal + +Reconcile the remaining artifact initiative with current `main` and lock the +smallest v0.1 path that proves one contributor ZIP is safely inspected in +bounded private scratch, checked under the task's locked Project Guide, admitted +once through the existing immutable `ArtifactStore`, and referenced unchanged +by the resulting Submission and every downstream consumer. + +This chunk changes planning and contracts only. It does not activate ART, +AUTH, task, checker, review, contribution, or delivery behavior. + +## Allowed Files + +- WS-ART-001 intent, discovery, plan, decisions, risks, status, chunk map, and + remaining unimplemented chunk contracts; +- the WS-ART-001 authorization handoff naming exact registration, hidden + behavior, activation, fixed-service separation, and stop conditions; +- the canonical artifact-storage specification, shared glossary, and one + availability-neutral authorization-spec clarification needed to remove stale + multi-step upload guidance without changing AUTH runtime ownership; +- submission artifact-policy and submission-packet templates needed to replace + caller hash/manifest and multi-step session guidance; +- ART planning review evidence and PR trust bundle; +- `scripts/test_agent_gates.py` only to replace stale ART status/chunk-map + assertions with the exact merged/cancelled planning projection, bind new + successor phases/coverage contracts, and preserve the signed-automation-only + authority warning instead of treating copied root projections as live state, + plus focused regression assertions required by an in-scope documentation + gate repair; +- `scripts/check_stale_authorization_docs.py` only for the narrow technical + background-service module/path recognition repair required to scan the changed + contracts without re-admitting deprecated human product-role vocabulary; +- cross-initiative handoff prose that names dependencies without modifying + another initiative's owned runtime contract; +- exactly one merge intent for this planning chunk. + +## Not Allowed + +- backend, frontend, migration, workflow, provider, or deployment changes; +- Authorization Service catalogue, evaluator, grant, identity, or activation + changes; +- Review, Contribution, compensation, reputation, or delivery implementation; +- candidate/quarantine object storage, temporary provider namespaces, physical + deletion, retention windows, or a second artifact recovery aggregate; +- larger upload, expansion, scratch, entry-count, or duration limits without + separately reviewed capacity evidence; +- starting any successor automatically. + +## Acceptance Criteria + +- every Submission version accepts exactly one outer ZIP; its contents are + governed by the locked Project Guide; +- Workstream recursively walks the normalized directory/file tree represented + by the outer ZIP, while a ZIP entry inside that tree remains an ordinary file; + nested archive unpacking is explicitly outside v0.1; +- bounded private scratch is the only pre-admission custody; failed, unsafe, + unchanged, abandoned, or checker-failing attempts never enter object storage; +- the canonical semantic manifest commits to normalized file/directory paths, + entry type, each file's SHA-256/byte count, and normalized regular-file + executable intent while excluding packaging, timestamps, ownership, group, + read/write/special bits, compression, and other permission metadata; +- exact archive equality and manifest equality are compared with the immediate + prior immutable Submission and both reject before provider I/O; +- all mandatory platform gates and locked project pre-submit checks pass before + the existing `ArtifactStore` admission path is called; +- passing bytes are written once, independently read back, verified, published + as `ArtifactContent`, and bound to the Submission through existing ART + identity, put-attempt, verification-job, receipt, and recovery abstractions; +- the plan creates no candidate store, promotion, retention policy, duplicate + provider write, `ArtifactOutboxRecoveryAttempt`, or provider delete path; +- process loss before durable admission requires reupload; an ambiguous durable + put uses existing observation/recovery and never creates a Submission until + verified content is bindable; +- a verified admission may remain unbound through client abandonment; its + closed lifecycle is `ready -> consumed|stale`, every state stays charged to + existing completed-byte scopes, and v0.1 adds no expiry/release/delete process; +- fresh AUTH-owned prepared capabilities guard durable put intent and later + Submission/binding consumption in their respective transactions; ART never + reads AUTH persistence directly and old human authority never implies fixed + service binding authority; +- pre-submit and post-submit materializers project identical fixed executable + semantics without preserving arbitrary archive modes or automatically + executing files; +- current conservative configured limits remain unchanged; +- the current `Submission` row remains the immutable version aggregate; a new + version links through `supersedes_submission_id` and, when responding to + `needs_revision`, through an exact review relationship owned jointly with + REV rather than a competing `SubmissionVersion` table; +- reviewer decision values remain exactly `accept`, `needs_revision`, and + `reject`; reviewer note/findings attach to the exact Submission and contain + no reviewer-uploaded revision artifact; +- normal reads use indexed latest/current/accepted projections while immutable + Submission/Review relationships preserve complete history; +- checker materialization and downstream reviewer/delivery streaming resolve + the exact binding and recompute full SHA-256 and byte count while streaming; +- ART owns identity, integrity, manifests, binding, and byte access; AUTH, REV, + CON, and delivery consume explicit capabilities through their own reviewed + contracts and activation sequence; +- the obsolete multi-step upload-session action plan is not reused blindly: + AUTH first registers one planned contributor action + `artifact.submission_bundle.prepare` mapped to `submission.create`, ART-04A-C + then implement one hidden continuous orchestration, and AUTH alone activates + that action after exact manifest/guard proof; existing fixed service actions + remain distinct for verification, pending-work scanning, put resolution, + checker materialization, and binding; +- every implementation successor is PR-sized, names exact allowed/forbidden + files, and requires a separate signed start. + +## Verification + +```bash +git diff --check +python3 scripts/check_stale_artifact_contracts.py +python3 scripts/test_agent_gates.py +python3 scripts/check_markdown_links.py +rg -n "candidate storage|seven-day|7 days|ArtifactOutboxRecoveryAttempt|SubmissionVersion table" .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage docs +``` + +The wording scan may match an explicit rejection in this planning contract or +decision log; it must not find an active design that implements those concepts. + +## Required Reviewers + +Senior engineering, architecture, QA/test, security/auth, product/ops, +reuse/dedup, CI integrity, test delta, and docs. + +## Human Review Focus + +- Is one outer ZIP the only contributor submission boundary? +- Can any failed or unchecked payload reach immutable object storage? +- Is scratch clearly ephemeral rather than a second artifact store? +- Does the plan preserve exact bytes across checks, review, acceptance, + contribution, and delivery without moving ownership into ART? +- Are AUTH activation and cross-initiative handoffs explicit and fail closed? diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-PLAN2-external-review-response.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-PLAN2-external-review-response.md new file mode 100644 index 00000000..0191c8b1 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-PLAN2-external-review-response.md @@ -0,0 +1,67 @@ +# External Review Response: WS-ART-001-PLAN2 + +## Comments addressed + +- moved 04A to `upload_admission`, kept it internal and non-routable, and made + 04C the sole owner of the hidden contributor route/action declaration; +- defined relative `/`-separated NFC paths and fail-closed exact, NFC, and + Unicode case-fold collision rejection before checks, materialization, or + provider I/O; +- clarified that ART removes only ART-side legacy reachability while the AUTH + registration contract owns ActionId retirement/unavailable proof; +- described 03A as the only immediate successor; +- reconciled `ArtifactPutAttempt` states with the implemented closed enum and + separated them from legacy producer-item and replica states; +- removed the contributor-supplied hash-manifest field and documented the + manifest as server-generated; +- scoped archive and semantic-manifest identities to Submission outer ZIPs; +- canonicalized every changed chunk heading and aligned the PLAN2 successor + title so repository governance gates can parse the contracts; +- expanded the PR description to the canonical trust-bundle structure. +- defined abandoned verified admissions as capacity-charged `ready` records + with only terminal `consumed` or `stale` outcomes and no expiry, release, + deletion, cleanup, or retention process; +- made normalized regular-file executable intent part of semantic identity and + fixed read-only materialization without preserving arbitrary ZIP permissions; +- required fresh AUTH-owned prepared capabilities at durable put intent and at + atomic Submission/admission consumption; +- used canonical phase-specific fixed-service ActionIds and kept + `artifact.binding.create` and `artifact.checker_input.materialize` as their + shared PermissionIds; +- strengthened agent-gate assertions so lifecycle and ActionId/PermissionId + regressions fail closed. +- repaired the hosted pytest-only failure by asserting the explicit typed + capability/no-AUTH-repository boundary and normalizing Markdown whitespace + before semantic ActionId/PermissionId checks. +- rebased without conflict onto `bba4ba5f`, including PR #198's exact CI/Ruff + custody repairs; range-diff confirms the complete PLAN2 series is unchanged. +- removed one obsolete `artifact_contract` read exposed by Ruff `F841`; no live + assertion or coverage obligation was removed. + +## Comments deferred + +None. + +## Human decisions needed + +None beyond the normal human review and explicit merge approval for PR #197. + +## Commands rerun + +- `git diff --check` +- `python3 scripts/check_stale_artifact_contracts.py` +- `python3 scripts/check_stale_authorization_docs.py` +- `python3 scripts/check_markdown_links.py` +- `python3 scripts/test_agent_gates.py` — 100 tests +- hosted-equivalent loop-memory branch-coverage command — 300 tests, 90.46 + percent +- `ruff 0.15.22 check app tests scripts` — PASS on the reconciled tree +- focused agent-gate pytest collection — 104 PASS +- `python3 scripts/check_internal_review_evidence.py` +- `python3 scripts/update_post_merge_memory.py validate-merge-intent --base-ref origin/main` + +## Remaining risks + +Hosted GitHub checks and refreshed CodeRabbit review must validate the pushed +repair. No backend runtime behavior changed, so slow backend suites remain +hosted rather than local. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-PLAN2-internal-review-evidence.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-PLAN2-internal-review-evidence.md new file mode 100644 index 00000000..5a443f78 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-PLAN2-internal-review-evidence.md @@ -0,0 +1,102 @@ +# Internal Review Evidence: WS-ART-001-PLAN2 + +Reviewed code SHA: `1dbc0d4e52c262844106ef9434c69139e38b33e9` + +Reviewed against trusted main: `bba4ba5f171a4438b072740707a5cf8bde49d9af` + +Reviewed at: `2026-07-25T16:41:40Z` + +Reviewer run IDs: `art_plan2_ext_senior`, `art_plan2_amend_arch`, +`art_plan2_ext_qa`, `art_plan2_amend_security`, `art_plan2_ext_product`, +`art_plan2_amend_docs`, `art_plan2_final_reuse`, `art_plan2_final_ci`, +`art_plan2_final_test_delta` + +## Candidate + +Planning-only reconciliation after signed cancellation of `WS-ART-001-03`. +No backend, frontend, migration, workflow, provider, or deployment behavior is +implemented or activated. + +## Deterministic Evidence + +- `git diff --check` — PASS +- `python3 scripts/check_stale_artifact_contracts.py` — PASS at + `artifact_store_cutover` +- `python3 scripts/check_stale_authorization_docs.py` — PASS +- `python3 scripts/check_markdown_links.py` — PASS +- `python3 scripts/update_post_merge_memory.py validate-merge-intent + --base-ref origin/main` — PASS +- `python3 scripts/test_agent_gates.py` — PASS, 100 tests +- hosted-equivalent loop-memory coverage command — PASS, 300 tests and 90.46 + percent branch coverage +- targeted stale wording scan — only explicit rejected/legacy/unavailable + references remain + +## Reviewer Results + +| Reviewer | Result | Blocking findings | Notes | +|---|---|---|---| +| senior engineering | PASS AFTER FIXES | none | The only final finding was this expected evidence refresh; admission, executable, and reauthorization contracts are maintainable and operationally bounded. | +| architecture | PASS | none | Preserved ART/AUTH/TASK/REV ownership, reused existing scratch/admission/recovery, and resolved latest-main review-history integration. | +| QA/test | PASS AFTER FIXES | none | Closed executable-template and hosted-collection gaps; proved lifecycle, authorization crossed states, concurrency, and governance gates. | +| security/auth | PASS | none | Preserved prepared-capability ordering, fixed-service separation, concealment, and ready/consumed/stale integrity fencing. | +| product/ops | PASS | none | Preserved one-ZIP contributor flow, reviewer decision vocabulary, and zero downstream effect before immutable Submission consumption. | +| reuse/dedup | PASS | none | Reused `ArtifactScratchManager`, `PreparedArtifact`, `CommittedArtifactSource`, and existing admission/put/verification/recovery paths. | +| CI integrity | PASS AFTER FIXES | none | Canonical headings/merge intent parse; stale-auth scanner remains fail-closed; hosted 300-test collection and coverage floor pass. | +| test delta | PASS AFTER FIXES | none | No removed/skipped tests or weakened assertions; semantic guards are whitespace-robust and inspect the owning contracts. | +| docs | PASS AFTER FIXES | none | Reconciled storage/auth specs and templates, including canonical executable-intent semantics in the submission packet. | + +## Material Repairs + +- removed candidate storage, retention, promotion, duplicate provider writes, + physical deletion, and a second recovery aggregate; +- locked one outer ZIP, safe outer-tree inspection, opaque nested archives, + exact archive identity, and canonical semantic-manifest identity; +- kept failed/unchecked bytes only in bounded process-local scratch; +- split guide cutover into 03A/03B/03C and submission work into + 04A/04B/04C/05; +- retired the future multi-step session design behind an exact AUTH-owned + registration/activation handoff; +- preserved existing immutable `Submission` as the version aggregate and REV's + decision/note ownership; +- required integrity recomputation for checker, reviewer, and delivery streams; +- defined verified-but-unbound admissions as capacity-charged + `ready -> consumed|stale` records with no expiry, deletion, release, retention + worker, or downstream product effect before atomic consumption; +- included normalized regular-file executable intent in semantic identity while + excluding arbitrary permission preservation and execution authority; +- required fresh transaction-local AUTH capabilities immediately before durable + put intent and again during atomic Submission/admission consumption; +- corrected fixed-service contracts to use canonical phase-specific ActionIds, + with shared `artifact.binding.create` and + `artifact.checker_input.materialize` named only as PermissionIds; +- strengthened the governance regression to require exact no-expiry, no-release, + no-deletion, no-cleanup, and no-retention-process admission clauses; +- corrected the materializer regression assertion to inspect its owning 04B + contract rather than the adjacent 04A contract; +- made 05's typed-capability boundary and prohibition on ART/TASK imports of + AUTH-owned repositories explicit, then proved the exact hosted pytest + collection rather than relying only on the direct script entry point; +- rebased onto trusted main `bba4ba5f` and preserved AUTH-10C, AUTH-002, CI, + and PLAN2 review history; +- recorded the final amendment review outcome in the canonical review log. +- proved all 22 PLAN2 commits patch-equivalent across the latest-main rebase and + confirmed the merged exact Ruff/CI custody changes introduce no ART conflict; + Ruff 0.15.22, the 300-test hosted-equivalent suite, and all deterministic + planning gates pass; +- removed the obsolete ART contract read exposed by the new Ruff `F841` rule; + focused Ruff, 104 collected pytest cases, and the 100 direct regression cases + pass without weakening an assertion. + +## Accepted Risks + +- Current AUTH catalogue/gate rows for upload-session actions remain as exact + trusted-main planned/unavailable discovery state until the separately reviewed + AUTH registration contract retires them. They have no route or activation and + block ART-04A until reconciled. +- Process loss before durable artifact intent requires contributor reupload in + v0.1; no temporary provider custody is introduced to mask that tradeoff. + +Valid findings addressed: yes + +Open sub-agent sessions: none diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-PLAN2-pr-trust-bundle.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-PLAN2-pr-trust-bundle.md new file mode 100644 index 00000000..305bdb8d --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-PLAN2-pr-trust-bundle.md @@ -0,0 +1,74 @@ +# PR Trust Bundle: WS-ART-001-PLAN2 + +## Intent + +Replace the rejected combined ART-03/future upload design with a bounded plan +that proves the same contributor ZIP is safely checked, immutably stored, +reviewed, accepted, recorded, and delivered. + +## Design + +```text +one outer ZIP +-> bounded process-local scratch +-> safe outer-tree inspection and canonical manifest +-> exact/semantic unchanged rejection +-> mandatory platform and locked-guide checks +-> one existing ArtifactStore admission +-> complete read-back verification +-> one capacity-charged ready SubmissionBundleAdmission +-> fresh human and fixed-service authorization +-> one immutable Submission binding +``` + +Nested archives remain opaque in v0.1. Failed or unchecked bytes never enter +object storage. Existing admission, put-attempt, verification, receipt, scanner, +and recovery abstractions are reused. + +## Scope + +Planning, successor contracts, canonical artifact documentation/templates, and +the exact agent-gate assertions required by the new phase map. No runtime code, +migration, provider, workflow, or authorization availability changes. + +Exactly one merge intent is added: +`.agent-loop/merge-intents/WS-ART-001-PLAN2.json`, naming only same-initiative +successor `WS-ART-001-03A` with a separate explicit start. + +## Evidence + +- diff, stale-contract, markdown-link, and 100 agent-gate tests pass; +- all nine required reviewer tracks pass after repair; +- exact reviewed candidate `1dbc0d4e` is based on trusted main `bba4ba5f`; +- the stale-authorization documentation gate and its technical-module + regression repair pass without reopening deprecated product-role vocabulary; +- detailed findings and resolutions are recorded in + `WS-ART-001-PLAN2-internal-review-evidence.md`. + +## Human Review Focus + +- Confirm one outer ZIP and no candidate retention are the intended v0.1 UX. +- Confirm process loss before durable intent requiring reupload is acceptable. +- Confirm AUTH must retire planned session actions and register/activate the + exact bundle action before ART-04A/05 can proceed. +- Confirm ART owns bytes/identity/access capabilities while REV, CON, and + delivery retain their lifecycle decisions. +- Confirm abandoned ready admissions remain quota-bounded and capacity-charged + in v0.1, with no expiry, deletion, release, or retention worker. +- Confirm executable intent is semantic manifest data, not permission + preservation or authority to execute contributor content. +- Confirm authority is freshly revalidated at durable put intent and again at + atomic Submission consumption through AUTH-owned prepared capabilities. + +## Gate + +This planning merge starts no implementation. `WS-ART-001-03A` requires a +fresh signed explicit start after human merge approval and automated merge +memory reconciliation. + +## External Review + +CodeRabbit reported seven valid planning/specification inconsistencies. All +were addressed in the smallest planning-only repair, alongside the two +governance failures exposed by hosted CI. The detailed disposition is recorded +in `WS-ART-001-PLAN2-external-review-response.md`. diff --git a/.agent-loop/merge-intents/WS-ART-001-PLAN2.json b/.agent-loop/merge-intents/WS-ART-001-PLAN2.json new file mode 100644 index 00000000..679f1f70 --- /dev/null +++ b/.agent-loop/merge-intents/WS-ART-001-PLAN2.json @@ -0,0 +1,9 @@ +{ + "chunk_id": "WS-ART-001-PLAN2", + "chunk_title": "Submission Bundle Reconciliation", + "initiative_id": "WS-ART-001", + "next_chunk_id": "WS-ART-001-03A", + "next_chunk_title": "Guide Source Byte Ingest", + "next_requires_explicit_start": true, + "schema_version": 2 +} diff --git a/docs/glossary.md b/docs/glossary.md index 63c74a01..d33dc5b5 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -68,11 +68,13 @@ actor IDs or grants. The active-or-revoked link between one canonical issuer/opaque subject and one ActorProfile. Raw tokens, provider credentials, and full claims are not stored. -## ArtifactUploadSession +## SubmissionBundlePreparation -The task- or guide-scoped operational staging record that authorizes bounded -artifact upload, sealing, expiry, and single-use consumption. It is not a -submission and grants no review authority. +The bounded process-local operation that receives one contributor outer ZIP, +inspects and checks its internal tree in private scratch, and hands passing bytes +once to immutable artifact admission. It is not durable candidate storage, a +Submission, or review authority; process loss before durable intent requires +reupload. ## ArtifactContent @@ -80,12 +82,20 @@ Workstream's provider-neutral immutable content identity: server-computed SHA-256, byte count, and bounded media metadata. Opaque provider identifiers and protocol observations are replica details, not this record's identity. +## SubmissionBundleAdmission + +The capacity-charged verified result of one passing submission-bundle +preparation. Its closed lifecycle is `ready -> consumed|stale`. A ready admission +may remain unbound through client abandonment; consumption is atomic with one +immutable Submission and binding, while proven task/predecessor/locked-context +drift may make it stale. No state expires, releases storage capacity, or +authorizes deletion in v0.1. + ## ArtifactUploadItem -The per-item mutable upload-operation ledger inside an -`ArtifactUploadSession`. It owns byte reservation, logical role, scoped -idempotency, request digest, CAS state, opaque `provider_object_ref`, and the -resulting `ArtifactContent`. It is not an `ArtifactBinding`. +Legacy unavailable multi-item staging metadata. ART-04A must remove it or make +it statically unreachable before submission-bundle preparation activates. It is +not the v0.1 contributor intake, `ArtifactContent`, or an `ArtifactBinding`. ## ArtifactBinding @@ -105,20 +115,20 @@ states. Logical Workstream references are represented only by ## ArtifactOperationReceipt Append-only Workstream evidence for one immutable put acknowledgement. It links -the exact upload item and replica and records operation, idempotency key, +the exact producer admission/attempt and replica and records operation, idempotency key, `request_digest`, opaque `provider_object_ref`, replay observation, bounded outcome/details, attempt number, correlation ID, and creation time. It contains no response digest or provider receipt. -## ArtifactSetManifest +## SubmissionBundleManifest -The server-generated canonical description of a sealed upload session's content -IDs, logical roles, trusted file facts, SHA-256 values, and byte counts. Every -entry has a server-derived identity over all semantic fields, exact duplicate -entries are rejected, and a total ordering makes the hash deterministic. Its -hash commits only to that exact set. A separate pre-submit admission record -binds the hash to actor, task, policy, checker, exact submission input, result, -and expiry. +The server-generated canonical semantic description of the file/directory tree +inside one contributor outer ZIP. It commits to normalized paths, entry types, +each file's SHA-256/byte count, and normalized regular-file executable intent +while excluding other ZIP packaging and permission metadata. +It is distinct from the exact outer-ZIP SHA-256/byte count. Together they bind +pre-submit evidence, verified admission, the immutable Submission, and its exact +artifact binding. ## ReviewPacketManifest diff --git a/docs/operations_queue_policy.md b/docs/operations_queue_policy.md index 84ec28eb..b1706fb1 100644 --- a/docs/operations_queue_policy.md +++ b/docs/operations_queue_policy.md @@ -254,7 +254,7 @@ Every operating day starts with: | `DRAFT -> SCREENING` | project id, locked guide candidate, task source/description fields, acceptance and rejection criteria | | `SCREENING -> READY` | screening decision, guide version lock, guide source snapshot id/hash lock, acceptance criteria, effective project submission artifact policy hash lock, project `PreSubmitCheckerPolicy` compiled bundle hash lock, approved generated project `PostSubmitCheckerPolicy` with matching provenance, review policy, revision policy | | `READY -> CLAIMED` | active published ContributionPolicyVersion whose `accepted_submission` rule is explicit; TaskAssignment freezes that version | -| `IN_PROGRESS -> SUBMITTED` | blocking pre-submit checks passed, submission packet, artifact hash manifest, evidence references, contributor attestation | +| `IN_PROGRESS -> SUBMITTED` | blocking pre-submit checks passed against the exact outer ZIP, submission packet, server-generated submission-bundle manifest and verified admission binding, evidence references, contributor attestation | | `SUBMITTED -> EVALUATION_PENDING` | immutable submission version, locked post-submit checker policy id/version/hash/body copied from the task context | | `EVALUATION_PENDING -> REVIEW_PENDING` | durable, final, current CheckerRun for the exact Submission with outcome exactly `allow_review`, verified artifact bindings, no blocking failures | | `EVALUATION_PENDING -> NEEDS_REVISION` | durable, final, current CheckerRun for the exact Submission with outcome exactly `needs_revision`, verified artifact bindings, outcome source `auto_checker`, contributor-visible checker failures with severity, message, suggested fix | diff --git a/docs/spec_artifact_storage_service.md b/docs/spec_artifact_storage_service.md index 0603bdc4..8cb26a62 100644 --- a/docs/spec_artifact_storage_service.md +++ b/docs/spec_artifact_storage_service.md @@ -53,22 +53,20 @@ product role, product authorization, or proof of stored content. ## Canonical Records -### ArtifactUploadSession +### SubmissionBundlePreparation -Fields include id, actor, project, optional task/guide, permitted logical roles, -state, item/byte limits and totals, sealed artifact-set hash, expiry, consumed -timestamp, database timestamps, and CAS version. +One contributor operation accepts exactly one outer ZIP. The ZIP remains in the +existing bounded `ArtifactScratchManager` through safe tree inspection, +canonical archive/semantic-manifest identity, unchanged-work comparison, and +mandatory platform/locked-guide checks. It is not candidate storage, carries no +provider reference, cannot cross a process boundary, and is discarded on +failure or process loss. -States are `open`, `sealed`, `consumed`, `expired`, and `cancelled`. - -### ArtifactUploadItem - -One per staged object. Fields include session, logical role, normalized display -name, expected SHA-256, expected byte count, canonical request digest, scoped -idempotency key, CAS version, content ID, failure code, and timestamps. - -States are `reserved`, `uploading`, `replay_required`, -`stored_pending_verification`, `ready`, `failed`, and `cancelled`. +Only a passing preparation enters generic durable admission. Existing +`ArtifactUploadSession`/`ArtifactUploadItem` rows and planned actions are legacy +unavailable staging input and must be removed or made statically unreachable by +ART-04A plus the separately reviewed AUTH registration handoff before the new +surface activates. ### ArtifactStorageAdmissionLedger @@ -107,7 +105,7 @@ configuration for task, producer, project, and deployment scopes; a missing, malformed, or nonpositive required limit fails startup rather than creating an unbounded scope. Product callers never supply scope collections. They submit one of the closed producer requests defined below; artifact orchestration loads -the authoritative project, task, upload-session, checker-run, actor/service, +the authoritative project, task, submission-bundle operation, checker-run, actor/service, and deployment facts, derives the complete `AdmissionScopeSet` internally, and commits all reservations or none. A missing canonical relationship fails before provider I/O rather than silently omitting a quota scope. @@ -121,7 +119,7 @@ manufacture capacity. ### ArtifactPutAttempt One durable attempt owns the gap between Transaction A and Transaction B. -Fields include producer request type/reference, upload item or system output +Fields include producer request type/reference, producer admission or system output reference, content commitment, deterministic object identity, admission-charge references, operation/request digests, status, next-run time, executor UUID, database-clock lease expiry, execution generation, bounded terminal result, @@ -140,19 +138,17 @@ A bounded PostgreSQL scanner publishes `prepared`, `acknowledgement_unknown`, and expired `put_in_flight` attempts. The resolver uses read-only `observe_put_result`, then opens and hashes any observed object. Matching bytes complete the original charges and Transaction B facts exactly -once. Authoritative absence releases charges and moves the item to -`replay_required`. Mismatched bytes remain charged, quarantine the key, and +once. Authoritative absence releases charges and moves the producer attempt to +`absent_replay_required`; a legacy contributor upload item, while it exists, +moves to `replay_required`. Mismatched bytes remain charged, quarantine the key, and create an incident. Terminal writes require matching executor and generation. No background resolver performs another provider write. -Contributor upload sessions also reserve open-session slots at task, actor, -project, and deployment scope. PostgreSQL-clock expiry is owned by one bounded -periodic scanner and is also applied lazily before new session admission. The -scanner and lazy path use the fixed system permission -`artifact.upload_session.expire`; the atomic terminal transition releases the -slot once. Completed byte charges remain after cancellation, expiry, or absence -of a product binding because v0.1 has no physical deletion. Provider I/O never -starts when any applicable scope lacks capacity. +Submission-bundle preparation reserves bounded scratch through the existing +scratch ledger, not a durable open-session slot. Failed or lost preparations +release scratch without altering completed durable-byte charges. Provider I/O +never starts until every pre-submit gate passes and every applicable durable +scope has capacity. ### ArtifactContent @@ -175,20 +171,22 @@ availability_state = unknown | available | unavailable integrity_state = unknown | valid | invalid ``` -Transaction B sets the upload item to `stored_pending_verification` and creates -the replica as `pending/unknown/unknown`. A matching complete read sets the item -to `ready` and replica to `verified/available/valid`. A provider-unavailable or +Transaction B sets the `ArtifactPutAttempt` to `object_confirmed`, sets a legacy +contributor upload item, while it exists, to `stored_pending_verification`, and +creates the replica as `pending/unknown/unknown`. A matching complete read makes the +admission bindable and replica `verified/available/valid`. A provider-unavailable or conflict job result does not fabricate a replica observation. A confirmed absent object sets the replica to `missing/unavailable/unknown`. Before any binding exists, and only while the -original upload session/item remains eligible, an exact replay by the original -authorized uploader may move that same replica to `pending/unknown/unknown`, -set the item back to `stored_pending_verification`, append a new operation -receipt, and create a new verification job. After a binding exists, the missing +original producer attempt remains eligible, an exact replay by the original +authorized producer may move that same replica to `pending/unknown/unknown`, +set the attempt back to `object_confirmed`, set a legacy contributor upload item, +while it exists, to `stored_pending_verification`, append a new operation receipt, +and create a new verification job. After a binding exists, the missing replica is terminal and cannot return to pending in v0.1. A digest/size mismatch sets the replica to `integrity_mismatch/available/invalid`, fails the pre-binding -item when applicable, and quarantines access; it never returns to pending. +attempt when applicable, and quarantines access; it never returns to pending. Provider references are internal. They never appear in contributor, reviewer, project-owner, or public API responses. @@ -270,7 +268,7 @@ preserves history. ### ArtifactOperationReceipt Append-only Workstream evidence for one immutable put acknowledgement. It links -the exact upload item and replica and records operation, idempotency key, +the exact producer admission/attempt and replica and records operation, idempotency key, `request_digest`, opaque `provider_object_ref`, replay observation, bounded outcome/details, attempt number, correlation ID, and `created_at`. Adapter and namespace identity resolve through the linked replica. A response digest, @@ -356,9 +354,9 @@ Product modules receive only these narrow asynchronous capabilities: ```text GuideArtifactIngestPort.ingest(GuideArtifactIngestRequest) -ContributorArtifactUploadPort.create/read/write/seal/cancel(...) +SubmissionBundlePreparationPort.prepare(SubmissionBundlePreparationRequest) ArtifactBindingPort.bind_verified(ArtifactBindingCreateRequest) -ArtifactMaterializationPort.materialize_ready_upload_set(ReadyUploadSetRequest) +ArtifactMaterializationPort.materialize_prepared_bundle(PreparedBundleMaterializationRequest) ArtifactMaterializationPort.materialize_bindings(BindingMaterializationRequest) CheckerArtifactOutputPort.store(CheckerOutputArtifactRequest) ArtifactOperatorReadPort.list_bindings/list_replicas/list_receipts/ @@ -368,8 +366,10 @@ ArtifactOperatorRecoveryPort.retry_verification(ArtifactRecoveryRequest) `GuideArtifactIngestRequest` contains the authenticated actor context, project and guide-source snapshot item IDs, logical role, authorized byte source, and -optional client commitment. `ReadyUploadSetRequest` contains task ID, sealed -upload-session ID, locked policy/checker context, and authorization context. +optional client commitment. `SubmissionBundlePreparationRequest` contains the +authorized contributor/task/assignment context and one outer ZIP byte source. +`PreparedBundleMaterializationRequest` is internal and process-local; it wraps +only the current `PreparedArtifact` generation and locked policy/checker context. `BindingMaterializationRequest` contains task/submission/checker-run context and immutable binding IDs. `CheckerOutputArtifactRequest` contains the fixed service actor, task/submission/checker-run IDs, logical role, and generated byte @@ -735,8 +735,9 @@ in the v2 clean cut. No compatibility adapter or dual format remains. 5. The claimed attempt is set to `put_in_flight`; the adapter conditionally stores or resolves an exact replay candidate. 6. Transaction B validates the reservation/CAS, completes every provisional - charge, records content, replica, and operation receipt, sets the item to - `stored_pending_verification`, and sets the replica to + charge, records content, replica, and operation receipt, sets the + `ArtifactPutAttempt` to `object_confirmed`, sets a legacy contributor upload item, while it exists, + to `stored_pending_verification`, and sets the replica to `pending/unknown/unknown`. 7. The transaction creates an outbox/publication obligation for one verification job. @@ -759,10 +760,11 @@ Acknowledgement loss leaves the durable put attempt and charges provisional. The attempt scanner publishes resolution after an ambiguous outcome or expired execution lease. Resolution performs a fresh read-only observation and full hash. A confirmed object completes the same charges and Transaction B facts. -Fresh authoritative absence releases them and moves the item to exact replay; -replay must atomically reacquire every applicable charge before another provider -call. Integrity-mismatched or quarantined existing bytes remain completed and -charged. +Fresh authoritative absence releases them and moves the `ArtifactPutAttempt` to +`absent_replay_required`; while a legacy contributor upload item exists, it +alone moves to `replay_required`. Replay must atomically reacquire every +applicable charge before another provider call. Integrity-mismatched or +quarantined existing bytes remain completed and charged. ### Prepared Byte Sources @@ -884,11 +886,11 @@ The same canonical `ArtifactScratchManager` also owns checker-workspace allocations. Authoritative pre-submit introduces one authorized artifact materializer, and post-submit reuses it without a second workspace manager. The materializer exposes two separate methods, not one caller-selected source -union: `materialize_ready_upload_set` accepts a sealed upload artifact set whose -items are all `ready` before a submission binding exists, and -`materialize_bindings` accepts immutable `ArtifactBinding` IDs after submission -creation. Source resolution is phase-specific and authorized; staging never -creates a premature product binding. The materializer reserves +union: an internal pre-submit method accepts only the current process-local +`PreparedArtifact` submission-bundle workspace, and `materialize_bindings` +accepts immutable `ArtifactBinding` IDs after submission creation. No scratch +path or handle crosses an API/Celery boundary, and staging never creates a +premature product binding. The materializer reserves the complete workspace against the same aggregate ledger and quotas, streams exact provider bytes into private no-follow paths, and recomputes SHA-256 and byte count for every file. Any mismatch becomes an artifact incident before @@ -1021,20 +1023,32 @@ those stable Workstream IDs. No route returns provider object references, bucket/key, endpoint, credentials, signed URLs, or raw provider responses. Pagination is bounded and stable. -## Exact Artifact-Set Admission - -Sealing generates `ArtifactSetManifest` from trusted server facts. Entries are -deterministically ordered and commit to logical role, normalized display name, -content ID, SHA-256, and byte count. Exact duplicates are rejected. - -The pre-submit admission binds the artifact-set hash to actor, task, effective -project policy, project pre-submit checker, summary, contributor attestation, -expiry, and upload session. Submission creation locks and consumes that exact -admission and session in one transaction. A changed field or changed artifact -requires a new precheck. - -Artifact reads during authoritative pre-submit have a bounded Workstream-owned -transient retry budget. Exhaustion returns: +## Exact Submission-Bundle Admission + +One contributor operation receives one outer ZIP into bounded private scratch. +Workstream computes its exact SHA-256/byte count, walks the complete safe outer +archive tree, and generates a deterministic `SubmissionBundleManifest` from +normalized file/directory paths, entry type, file SHA-256/byte count, and the +normalized executable flag for each regular file. Nested +archives remain opaque in v0.1. Exact archive or semantic-manifest equality with +the immediate prior immutable Submission rejects before checker/provider I/O. + +For a regular Unix-created ZIP entry with valid mode metadata, `executable` is +true when any execute bit is present. It defaults false for non-Unix or invalid +mode metadata and is null for directories. Ownership, group, read/write bits, +setuid/setgid/sticky bits, timestamps, compression, and comments are excluded. +Symlinks/special entries remain rejected. Materialization never restores raw +archive modes: fixed startup-validated modes produce sealed read-only files, +read-and-execute files, and read-and-traverse directories. The flag never +authorizes execution by itself. Pre-submit and post-submit materialization use +the same projection and `ArtifactScratchManager` retains secure cleanup custody. +The exact numeric modes are one startup configuration shared by every process +using the scratch root. Cleanup under an exclusive generation/lease may restore +only the manager-owned directory write bit after no-follow ownership/type +validation, then remove the tree; archive-supplied modes are never applied. + +Mandatory platform and locked Project Guide checks consume that same read-only +scratch tree. Infrastructure exhaustion returns: ```text HTTP 503 @@ -1042,26 +1056,96 @@ code = pre_submission_infrastructure_unavailable ``` This is infrastructure state, not a checker result, review decision, or -contributor outcome. It creates no admission, submission, checker finding, -compensation, contribution, or reputation effect. The sealed upload session and -artifact set remain sealed, unconsumed, and reusable until normal expiry; only -the infrastructure attempt and audit are persisted. Idempotency scope is actor, -task, sealed session/artifact set, locked context, client key, and canonical -request digest. Exact replay continues or returns the same attempt, changed -replay conflicts, and concurrent replay cannot create duplicate attempts or -admissions. After retry-after or storage recovery, the contributor continues -that same exact attempt without Project Manager or Operator approval. - -Before authoritative pre-submit invokes the checker, the shared authorized -materializer resolves every `ready` item in the sealed upload artifact set, -reads its content through the referenced verified replica into a canonical -scratch-manager workspace, recomputes SHA-256 and byte count, and compares them -with the upload-item/content commitment. No `ArtifactBinding` exists at this -stage. The checker executes only that verified read-only workspace. Post-submit -materialization instead resolves immutable binding IDs and compares bytes with -their bound content commitments. A mismatch is an artifact incident; quota -exhaustion or transient read failure is infrastructure state and creates no -checker result or admission. +contributor outcome. It creates no durable artifact, admission, Submission, +compensation, contribution, or reputation effect. Scratch is cleaned and +process loss requires reupload without manager/operator approval. + +Only a passing result is handed immediately to generic durable admission in the +same process. Workstream writes the outer ZIP once, independently reads it back, +and publishes a `ready` bindable admission only after exact verification. Existing put +attempt, observation, receipt, verification, scanner, and recovery records own +durable ambiguity. Submission creation locks and consumes the exact admission; +post-submit materialization resolves its immutable binding and recomputes +integrity before checker execution. + +### SubmissionBundleAdmission + +One immutable verified preparation with closed status: + +```text +status = ready | consumed | stale +actor_profile_id = UUID +preparation_identity_link_id = UUID +project_id = UUID +task_id = UUID +assignment_id = UUID +predecessor_submission_id = UUID | null +locked_task_context_id/hash = immutable canonical context reference +artifact_content_id = UUID +submission_bundle_manifest_id = UUID +pre_submit_evidence_set_id = UUID +ready_at = timestamp +consumed_at = timestamp | null +consumed_by_submission_id = UUID | null +stale_at = timestamp | null +stale_reason = bounded token | null +``` + +It records preparation actor/profile and identity-link provenance, project, +task, assignment, immediate predecessor, exact locked task/guide/policy context, +`ArtifactContent`, `SubmissionBundleManifest`, immutable pre-submit evidence-set +identity, timestamps, terminal reason, and optional consuming Submission. +Status-field constraints require terminal facts only in their matching state. +A unique Submission admission reference plus locked/CAS transition permits at +most one consumer. + +`ready -> consumed` occurs only in the transaction creating Submission and +binding. `ready -> stale` occurs only when consumption proves task closure, +predecessor advancement, or locked-context replacement. Authority loss alone +does not stale the admission. Both terminal states are irreversible. Client +abandonment may leave `ready` unbound indefinitely; it has no Submission, +review, contribution, compensation, or reputation effect. + +All states count against existing completed-byte task/actor/project/deployment +capacity. Existing Operator admission-usage projections expose bounded unbound +ready and stale counts/bytes. No admission expiry, release, deletion, provider +cleanup, retention process, or new recovery aggregate exists. Exact preparation +replay returns the original admission without another provider write, charge, +evidence set, or admission. + +### Authorization At Durable Boundaries + +The preparation route authorizes `artifact.submission_bundle.prepare` before +accepting bytes. Immediately before capacity reservation and put-attempt +creation, the owning transaction consumes AUTH's transaction-local prepared +capability. AUTH and typed TASK/PROJECT capabilities reload/lock their own actor, +identity-link, project authority, assignment, task, predecessor, locked context, +action availability, and canonical resource facts. ART does not access +AUTH-owned persistence. Authorization evidence, capacity, and durable put intent +commit atomically; provider I/O starts only afterward. + +Revocation after durable put intent does not cancel fixed-service verification +or recovery. It does prevent later binding until Submission creation receives a +new human decision. That transaction consumes both fresh human +`submission.create` and fixed-service ActionId +`artifact.submission.binding.create` capabilities, mapped to PermissionId +`artifact.binding.create`, locks and matches +admission/task/assignment/predecessor/context, creates the +immutable Submission and binding, and marks admission consumed. Any denial, +cancellation, stale execution, or persistence failure rolls back all effects. +After authority succeeds, proven task closure, predecessor advancement, or +locked-context replacement may commit only `ready -> stale` plus bounded +evidence, never a Submission/binding. Authorization/concealment precedes +detailed admission status errors. + +Stable outcomes for an already authorized matching resource include: + +```text +409 submission_bundle_admission_already_consumed +409 submission_bundle_admission_context_changed +409 submission_bundle_admission_stale +403 +``` After cutover, the public request is: @@ -1069,33 +1153,27 @@ After cutover, the public request is: { "summary": "...", "contributor_attestation": "...", - "upload_session_id": "..." + "submission_bundle_admission_id": "..." } ``` Clients do not provide package URI, provider reference, canonical digest -manifest, artifact-set hash, or server content IDs. +manifest, semantic-manifest hash, or server content IDs. -### Product Upload APIs +### Product Submission-Bundle APIs ```text -POST /api/v1/tasks/{task_id}/artifact-upload-sessions -GET /api/v1/artifact-upload-sessions/{session_id} -POST /api/v1/artifact-upload-sessions/{session_id}/artifacts -POST /api/v1/artifact-upload-sessions/{session_id}/seal -DELETE /api/v1/artifact-upload-sessions/{session_id} -POST /api/v1/tasks/{task_id}/submission-precheck +POST /api/v1/tasks/{task_id}/submission-bundle-preparations POST /api/v1/tasks/{task_id}/submissions ``` -The upload APIs return Workstream IDs, canonical SHA-256, byte count, detected -media type, verification/readiness, and bounded artifact summaries. They never -return provider internals. Session cancellation is logical staging cleanup; it -does not physically delete a completed content-addressed object. - -Every ID-addressed session read or mutation uses concealed deny/not-found -behavior. Cross-actor, cross-project, revoked, expired, cancelled, consumed, -and random IDs cannot be used as an existence oracle. +The preparation POST performs scratch intake, inspection, checks, and durable +handoff as one process-local orchestration. It returns only a Workstream +operation/admission identity and bounded redacted status. An exact idempotent +POST retry may observe the same durable operation after durable intent; v0.1 +exposes no separate preparation-status GET route. No retry resolves a scratch +path, every retry obtains fresh preparation authority, and APIs never return +provider internals. ## Guide And Checker Binding @@ -1114,9 +1192,10 @@ policy. Recovery authorization and automatic setup continuation write separate audit events. A changed snapshot creates a new setup run and cannot resume the old one. -`CheckerInputSnapshot` references binding/content IDs, digest, byte count, -artifact-set hash, locked policy/checker identities, and checker implementation -identity. Pre-submit and post-submit consume the same sealed artifact-set hash. +`CheckerInputSnapshot` references binding/content IDs, outer-ZIP digest/byte +count, semantic-manifest hash, locked policy/checker identities, and checker +implementation identity. Pre-submit evidence and post-submit execution name the +same verified admission and exact binding. Checker logs and generated outputs become artifact bindings. Transient post-submit storage unavailability leaves the task in @@ -1130,10 +1209,13 @@ mismatch on an existing content-addressed key is likewise unrecoverable in v0.1: the replica remains quarantined, evaluation stays blocked, and a security incident is required. Recovery never overwrites the poisoned key. -Pre-submit and post-submit outage continuation are distinct contracts. The -former preserves a sealed, unconsumed attempt and returns the stable 503 above; -the latter preserves `evaluation_pending` and uses checker retry -infrastructure. Neither path fabricates `accept`, `needs_revision`, or `reject`. +Pre-submit and post-submit outage continuation are distinct contracts. Before +durable artifact intent, the former cleans scratch, preserves no reusable +attempt, returns the stable 503 above, and requires reupload after scratch or +process loss. After durable handoff, existing ART put/verification recovery +owns ambiguity. Post-submit checker outage preserves `evaluation_pending` and +uses checker retry infrastructure. No path fabricates `accept`, +`needs_revision`, or `reject`. ## Authorization Dependencies diff --git a/docs/spec_authorization_service.md b/docs/spec_authorization_service.md index f1252a0a..7b95f154 100644 --- a/docs/spec_authorization_service.md +++ b/docs/spec_authorization_service.md @@ -362,12 +362,21 @@ The paired artifact hidden-behavior matrix is closed: |---|---| | `WS-ART-001-02D` | Operator binding/replica/receipt/verification-job/recovery-attempt/audit reads; the operations-domain `operations.artifact_storage_admission.read` action mapped to `operations.status.read`; verification retry; `artifact.verification.execute`; `artifact.pending_work.scan`; and `artifact.put_attempt.resolve` | | `WS-ART-001-03` | `artifact.guide_source.ingest`, `artifact.guide_source.read`, and `artifact.guide_source.binding.create` mapped to `artifact.binding.create` | -| `WS-ART-001-04A` | upload-session create/read/seal/cancel/expire and upload-item write | +| `WS-ART-001-04A` legacy unavailable baseline | planned upload-session create/read/seal/cancel/expire and upload-item write have no route/command and must be retired by the separate AUTH registration contract before 04A starts | +| `WS-ART-001-04A` through `04C` after AUTH registration | one hidden `artifact.submission_bundle.prepare` surface mapped to `submission.create`; remains unavailable until 04C evidence and a later AUTH activation contract | | `WS-ART-001-04B` | `artifact.pre_submit.checker_input.materialize` mapped to `artifact.checker_input.materialize` | | `WS-ART-001-05` | `artifact.submission.binding.create` mapped to `artifact.binding.create` | | `WS-ART-001-06A` | `artifact.post_submit.checker_input.materialize` mapped to `artifact.checker_input.materialize` | | `WS-ART-001-06B` | `artifact.checker_output.write` and `artifact.checker_output.binding.create` mapped to `artifact.binding.create` using the checker-run resource | +The `WS-ART-001-04A` row records the current planned/unavailable catalogue +baseline only. ART PLAN2 proposes a separate AUTH-owned registration contract +that retires those unused multi-step actions and registers planned +`artifact.submission_bundle.prepare -> submission.create`. No ART implementation +may use that new ActionId before the AUTH contract merges; no action activates +until ART-04A-C publish the complete hidden surface and a later AUTH activation +contract consumes its exact evidence. + Every row requires AUTH-07A's registry and AUTH-07B's kernel first. A row with an Operator principal also requires its AUTH-08 grant definition; a row with a fixed service principal also requires AUTH-09A's static matrix, AUTH-09B's provisioned service @@ -435,7 +444,7 @@ and unavailable, and add no migration. | `artifact.verification.execute` | `artifact.verification.execute` | fixed verifier service | verification job | `02D` | | `artifact.pending_work.scan` | `artifact.pending_work.scan` | fixed scheduler service | system pending-work scope | `02D` | | `artifact.put_attempt.resolve` | `artifact.put_attempt.resolve` | fixed put-resolver service | put attempt | `02D` | -| `artifact.pre_submit.checker_input.materialize` | `artifact.checker_input.materialize` | fixed materializer service | sealed upload session and task | `04B` | +| `artifact.pre_submit.checker_input.materialize` | `artifact.checker_input.materialize` | fixed materializer service | task plus current process-local prepared-bundle generation; no scratch path/handle is serialized | `04B` | | `artifact.post_submit.checker_input.materialize` | `artifact.checker_input.materialize` | fixed materializer service | checker run and immutable bindings | `06A` | | `artifact.checker_output.write` | `artifact.checker_output.write` | fixed checker-output service | checker run | `06B` | diff --git a/docs/template_preflight_evidence.md b/docs/template_preflight_evidence.md index c3af85f7..8e91cc27 100644 --- a/docs/template_preflight_evidence.md +++ b/docs/template_preflight_evidence.md @@ -32,14 +32,15 @@ State what was checked before review. | --- | --- | --- | --- | --- | | `` | `` | `` | `` | `` | -## Package Evidence +## Submission Bundle Evidence -- package path or URI: -- package hash: -- artifact count: +- server-generated outer-ZIP SHA-256: +- server-generated outer-ZIP byte count: +- inspected entry count: - required files present: -- artifact hash manifest: -- evidence ids bound to artifact hashes: +- server-generated submission-bundle manifest reference/hash: +- verified admission/binding IDs: +- evidence IDs bound to the verified admission and manifest: ## Review Readiness diff --git a/docs/template_project_guide.md b/docs/template_project_guide.md index 6ff1aed9..9b80e28d 100644 --- a/docs/template_project_guide.md +++ b/docs/template_project_guide.md @@ -80,13 +80,15 @@ Define prohibited behavior, tools, copied material, generated artifacts, confide ## Required Submission Fields - summary -- output files or package +- exactly one outer ZIP containing every required output/evidence file - evidence - revision replay when applicable - contributor attestation -- artifact hash manifest -Workstream assigns submission version server-side after blocking pre-submit checks pass. The contributor does not provide a submission version or any guide/policy version. +Workstream generates the archive commitment and semantic submission-bundle +manifest, then assigns the submission version server-side after blocking +pre-submit checks pass. The contributor does not provide a manifest, submission +version, or guide/policy version. ## Submission Expectations Summary diff --git a/docs/template_submission_artifact_policy.md b/docs/template_submission_artifact_policy.md index 7de999d9..85d50485 100644 --- a/docs/template_submission_artifact_policy.md +++ b/docs/template_submission_artifact_policy.md @@ -86,21 +86,31 @@ Every project inherits Workstream default submission artifact rules. Project pol Default required packet fields: - summary -- artifact hash manifest - contributor attestation +Workstream generates the archive commitment and semantic artifact manifest +after safely inspecting the submitted outer ZIP. Clients do not supply an +artifact hash manifest as packet input; later APIs may expose an immutable +server-generated manifest reference. + Default artifact rules: - artifact paths must be relative - artifact paths must not contain empty, `.`, or `..` segments - uploaded artifacts and storage-backed evidence require `sha256:<64 lowercase hex>` hashes in production - test fixtures may use deterministic placeholder hash tokens only in explicit local test paths +- Workstream normalizes regular-file executable intent from valid Unix ZIP mode + metadata; non-Unix/invalid metadata defaults to non-executable +- executable intent participates in semantic identity, but never by itself + authorizes a checker or service to execute the file Default storage rules: -- clients submit bytes through Workstream upload sessions and receive only - Workstream artifact IDs +- clients submit exactly one outer ZIP through Workstream submission-bundle + preparation and receive only Workstream operation/admission IDs - persisted product references are immutable Workstream artifact bindings +- verified admissions may remain unbound and capacity-charged in `ready` or + terminal `stale`; clients cannot expire, release, or delete them - signed URLs, raw local filesystem paths, provider references, credentials, query strings, bucket secrets, and token-bearing references are rejected before persistence diff --git a/docs/template_submission_packet.md b/docs/template_submission_packet.md index bd481175..95a83def 100644 --- a/docs/template_submission_packet.md +++ b/docs/template_submission_packet.md @@ -16,16 +16,27 @@ Assigned by Workstream after blocking pre-submit checks pass. The contributor do Briefly describe what was completed. -## Output +## Submission Bundle -List files, links, packages, or deliverables. +Upload exactly one outer ZIP. Its internal files and directories must satisfy +the locked Project Guide. Workstream safely inspects that tree and computes all +artifact identity; the contributor does not submit links or storage references. ## Provenance -- package hash: -- artifact hash manifest: - generated at: +Workstream records the server-computed outer-ZIP SHA-256/byte count, canonical +semantic manifest including normalized regular-file executable intent, verified +admission, and immutable binding. Those values are +server output, not contributor input. + +A successful preparation may return a verified `ready` admission before a +Submission exists. Submission creation obtains fresh authority and atomically +consumes that admission with the immutable Submission/binding. An abandoned +ready admission has no review, contribution, compensation, or reputation +effect, remains capacity-charged, and has no client expiry/release/delete path. + Workstream derives the locked project guide version, locked guide-source snapshot id/hash, effective project submission artifact policy id/hash, generated project pre-submit checker policy id/bundle hash, post-submit @@ -46,11 +57,13 @@ submission row, no submission version, and no submission-created audit event, and do not return review decision values: `accept`, `needs_revision`, or `reject`. -## Artifact Hash Manifest +## Submission Bundle Manifest -| Artifact | Hash | Size | Notes | -| --- | --- | ---: | --- | -| `` | `sha256:<64 lowercase hex>` | `` | `` | +Workstream generates this manifest from normalized directory/file paths, entry +type, each file's SHA-256/byte count, and normalized executable intent for +regular files. Valid Unix execute bits normalize to true; non-Unix or invalid +mode metadata defaults false, and directories have no executable value. Other +archive permission metadata is excluded. Nested archives remain opaque in v0.1. ## Evidence diff --git a/scripts/check_stale_authorization_docs.py b/scripts/check_stale_authorization_docs.py index 69297841..14153c6d 100644 --- a/scripts/check_stale_authorization_docs.py +++ b/scripts/check_stale_authorization_docs.py @@ -377,6 +377,7 @@ def technical_worker_match(text: str, match: re.Match[str]) -> bool: prefix = text[max(0, worker_offset - 120) : worker_offset] token_end = worker_offset + len(token.group(0)) suffix = text[token_end : token_end + 20] + authority_suffix = text[token_end : token_end + 120] exact_code_path = token.group(0).lower() == "workers" and ( ( re.search(r"(?:^|[^A-Za-z0-9_.-])(?:backend/)?app/$", prefix) @@ -388,6 +389,30 @@ def technical_worker_match(text: str, match: re.Match[str]) -> bool: and suffix.startswith(".") ) ) + exact_coverage_module = ( + token.group(0).lower() == "workers" + and re.search(r"(?:^|\s)--cov=app\.$", prefix) is not None + and (not suffix or suffix[0].isspace()) + ) + after_code_reference = re.sub( + r"^(?:[/.][A-Za-z0-9_.-]+)+", "", authority_suffix + ) + exact_coverage_path = bool( + exact_code_path + and re.search(r"coverage\s+report\s+--include=['\"]app/$", prefix) + and re.match(r"/\*['\"](?:\s|$)", suffix) + ) + exact_celery_module_path = bool( + exact_code_path + and re.search(r"(?:^|\n)[^;\n]*\bcelery\s+-A\s+app\.$", prefix) + and re.match(r"\s+worker\s+--", after_code_reference) + ) + code_path_has_prose = bool( + exact_code_path + and not exact_coverage_path + and not exact_celery_module_path + and re.search(r"[A-Za-z]", after_code_reference) + ) exact_technical_cli_flag = ( token.group(0).lower() == "worker" and prefix.endswith("--start-api-") @@ -407,7 +432,8 @@ def technical_worker_match(text: str, match: re.Match[str]) -> bool: ) return bool( TECHNICAL_WORKER_PREFIX.search(prefix) - or exact_code_path + or (exact_code_path and not code_path_has_prose) + or exact_coverage_module or exact_celery_cli or exact_technical_cli_flag ) diff --git a/scripts/test_agent_gates.py b/scripts/test_agent_gates.py index 68819fc5..e9a657df 100644 --- a/scripts/test_agent_gates.py +++ b/scripts/test_agent_gates.py @@ -36,9 +36,12 @@ "02C2", "02C3", "02D", - "03", + "03A", + "03B", + "03C", "04A", "04B", + "04C", "05", "06A", "06B", @@ -77,19 +80,23 @@ "02D": ( "coverage report --include='app/api/router.py' --precision=2 --fail-under=90", ), - "03": ( + "03A": ( "coverage report --include='app/modules/projects/*' " "--precision=2 --fail-under=90", + ), + "03B": ( "coverage report " "--include='app/adapters/project_agents/*,app/interfaces/project_agents.py' " "--precision=2 --fail-under=90", ), + "03C": (), "04A": (), "04B": ( "coverage report --include='app/modules/tasks/*' --precision=2 --fail-under=90", "coverage report --include='app/modules/checkers/*' " "--precision=2 --fail-under=90", ), + "04C": (), "05": (), "06A": (), "06B": (), @@ -163,9 +170,9 @@ def artifact_contract_phase_for(coverage_phase: str) -> str: phase = "foundation" if active_index >= ARTIFACT_COVERAGE_ORDER.index("02A3"): phase = "artifact_store_cutover" - if active_index >= ARTIFACT_COVERAGE_ORDER.index("03"): + if active_index >= ARTIFACT_COVERAGE_ORDER.index("03A"): phase = "guide_source_cutover" - if active_index >= ARTIFACT_COVERAGE_ORDER.index("04B"): + if active_index >= ARTIFACT_COVERAGE_ORDER.index("04A"): phase = "upload_admission" if active_index >= ARTIFACT_COVERAGE_ORDER.index("05"): phase = "submission_cutover" @@ -5445,6 +5452,7 @@ def test_stale_authorization_rule_examples_are_rejected() -> None: "See app/workers/tasks.py.", "coverage report --include='app/workers/*' --precision=2 --fail-under=90", "ruff check app/workers/reviews.py", + "pytest --cov=app.workers --cov-report=term-missing", "review_lifecycle_live_drill.py --start-api-worker-beat --require-workers", ) for sample in technical_worker_statements: @@ -5488,6 +5496,20 @@ def test_stale_authorization_rule_examples_are_rejected() -> None: "review_lifecycle_live_drill.py --require-workers-extra", "maliciousapp/workers/reviews.py", "maliciousapp.workers.reviews", + "app.workers submit task packets.", + "app.workers approves project work.", + "The app.workers module submits contributor packets.", + "app.workers may review a contributor submission.", + "app.workers approves project guides.", + "app.workers manages contributor grants.", + "backend/app/workers/reviews.py may review a contributor submission.", + "backend/app/workers/reviews.py claims a task.", + "backend/app/workers/tasks.py can claim a task.", + "backend/app/workers/tasks.py creates a project.", + "backend/app/workers/tasks.py can create a project.", + "app/workers/tasks.py creates a project.", + "backend/app/workers/tasks.py uses submitter authority.", + "backend/app/workers/tasks.py is a human product role.", ) for sample in human_worker_statements: assert gate.scan_text("docs/new_active_doc.md", sample), sample @@ -5658,7 +5680,7 @@ def test_auth_spec_orders_service_admission_before_project_roles() -> None: def test_parallel_initiative_status_matches_trusted_main() -> None: - """Auth prerequisites and the current ART gate remain internally consistent.""" + """Auth history and authored ART planning remain internally consistent.""" auth_map = Path( ".agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/" "CHUNK_MAP.md" @@ -5672,12 +5694,20 @@ def test_parallel_initiative_status_matches_trusted_main() -> None: artifact_status = Path( ".agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md" ).read_text(encoding="utf-8") - artifact_contract = Path( - ".agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/" - "WS-ART-001-02C2-verification-publication-fencing.md" - ).read_text(encoding="utf-8") work_queue = Path(".agent-loop/WORK_QUEUE.md").read_text(encoding="utf-8") loop_state = Path(".agent-loop/LOOP_STATE.md").read_text(encoding="utf-8") + contribution_guide = Path("CONTRIBUTING.md").read_text(encoding="utf-8") + loop_guide = Path(".agent-loop/README.md").read_text(encoding="utf-8") + + # Authored main snapshots below remain historical discovery input. Live + # active/cancel state is read only from independently verified signed + # automation, so ART assertions use its initiative status/map rather than + # pretending copied root projections are current authority. + canonical_warning = ( + "Only independently verified signed automation state is canonical authority" + ) + assert canonical_warning in " ".join(contribution_guide.split()) + assert canonical_warning in " ".join(loop_guide.split()) assert "Merged through PR #131 as `aa0fdcd`" in auth_map assert "Merged through PR #143 as `053242b`" in auth_map @@ -5711,7 +5741,6 @@ def test_parallel_initiative_status_matches_trusted_main() -> None: "as `8d5eb15` on 2026-07-19" in work_queue ) assert "Active implementation chunk: `WS-AUTH-001-CONTRIBUTOR-FOUNDATION`" not in loop_state - assert "Active ART implementation chunk: `WS-ART-001-02C2`" in loop_state assert "Merged through PR #157 as `42a89b2d`" in work_queue assert "ActionIds, with 17 active actions" in loop_state assert "candidate total of 17" not in loop_state @@ -5742,28 +5771,20 @@ def test_parallel_initiative_status_matches_trusted_main() -> None: assert "Merged through PR #141 as `a10d901`" in artifact_map assert "Merged through PR #151 as `1b5422fc`" in artifact_map assert "Merged through PR #154 as `44f2467c`" in artifact_map - assert "Active after PR #154 and explicit user start" in artifact_map - assert "Status: Active after explicit start on 2026-07-19" in artifact_contract + assert "Merged through PR #159 as `bc5e6a42`" in artifact_map + assert "Merged through PR #174 as `92b8a7aa`" in artifact_map + assert "Merged through PR #177 as `93c14181`" in artifact_map assert ( "AUTH's owner reconciliation merged through PR #140 as\n" "`d541521`" in artifact_status ) - assert "`WS-ART-001-02C2` is active" in artifact_status - assert "The current gate is external CI/review followed by explicit human" in ( - artifact_status.replace("\n", " ") - ) - assert "No later artifact chunk starts automatically" in artifact_status.replace( - "\n", " " - ) + assert "`WS-ART-001-03` received a signed implementation start" in artifact_status + assert "recorded `stopped_after_cancel`" in artifact_status + assert "The planning merge starts no successor" in artifact_status assert ( "| `WS-AUTH-001-09C` | Actor And Identity-Link Administration Reads | L1 | " "Merged through PR #146 as `0ffdabf`" in work_queue ) - assert ( - "| `WS-ART-001-02C2` | Verification Publication And Fencing | L1 | " - "Latest-main, sharded-CI, AUTH-PREP, and outbox-repair reconciliation complete" - in work_queue - ) assert ( "| `WS-AUTH-001-ART-CUSTODY` | ART Activation Custody Transfer | L1 | " "Merged through PR #158 as `be2a79a2`" in work_queue @@ -5779,12 +5800,72 @@ def test_parallel_initiative_status_matches_trusted_main() -> None: "Merged through PR #162 as `c559d556`" in work_queue ) assert "no feature consumer or activation" in work_queue - assert "Current ART gate: latest-main integration" in loop_state - assert "hosted checks, and explicit human merge approval remain" in " ".join( - loop_state.split() + assert "| `WS-ART-001-PLAN2` |" in artifact_map + assert "Planning-only successor proposed after cancellation" in artifact_map + assert "`WS-ART-001-03A` is the only immediate ART" in artifact_status + + +def test_artifact_plan2_closes_submission_bundle_lifecycle_gaps() -> None: + """PLAN2 locks admission, executable, and durable-auth semantics.""" + initiative = ROOT / ".agent-loop/initiatives/WS-ART-001-immutable-artifact-storage" + plan = (initiative / "PLAN.md").read_text(encoding="utf-8") + decisions = (initiative / "DECISIONS.md").read_text(encoding="utf-8") + auth_handoff = (initiative / "AUTH_HANDOFF.md").read_text(encoding="utf-8") + contract_04a = (initiative / "chunks/WS-ART-001-04A-upload-inspection-sealing.md").read_text(encoding="utf-8") + contract_04b = (initiative / "chunks/WS-ART-001-04B-pre-submit-admission.md").read_text(encoding="utf-8") + contract_04c = (initiative / "chunks/WS-ART-001-04C-verified-submission-bundle-admission.md").read_text(encoding="utf-8") + contract_05 = (initiative / "chunks/WS-ART-001-05-submission-artifact-cutover.md").read_text(encoding="utf-8") + contract_06a = (initiative / "chunks/WS-ART-001-06A-checker-input-materialization.md").read_text(encoding="utf-8") + storage_spec = (ROOT / "docs/spec_artifact_storage_service.md").read_text(encoding="utf-8") + normalized_auth_handoff = " ".join(auth_handoff.split()) + + for text in (plan, decisions, contract_04c, storage_spec): + assert "ready -> consumed" in text + assert "stale" in text + no_lifecycle_process = { + "plan": "No state expires, releases capacity, or authorizes provider deletion.", + "decisions": ( + "No expiry, release, deletion, retention process, or cleanup lifecycle " + "exists in v0.1." + ), + "04c": ( + "v0.1 adds no expiry, release, deletion, retention process, or cleanup " + "route;" + ), + "storage_spec": ( + "No admission expiry, release, deletion, provider cleanup, retention " + "process, or new recovery aggregate exists." + ), + } + for name, text in { + "plan": plan, + "decisions": decisions, + "04c": contract_04c, + "storage_spec": storage_spec, + }.items(): + normalized = " ".join(text.split()) + clause = no_lifecycle_process[name] + assert clause in normalized + assert clause not in normalized.replace(clause, "Admission expiry is enabled.") + assert "Client abandonment" in plan + assert "existing completed-byte scopes" in plan + assert "transaction-local prepared capability" in normalized_auth_handoff + assert "ART and TASK never import AUTH-owned repositories" in contract_05 + assert "authorization evidence" in contract_04c + assert "ActionId `artifact.submission.binding.create`" in normalized_auth_handoff + assert "PermissionId `artifact.binding.create`" in normalized_auth_handoff + assert "`artifact.pre_submit.checker_input.materialize`" in normalized_auth_handoff + assert "`artifact.post_submit.checker_input.materialize`" in normalized_auth_handoff + assert ( + "mapped to PermissionId `artifact.checker_input.materialize`" + in normalized_auth_handoff ) - assert "hosted reruns, and explicit human merge approval remain" in work_queue - assert "PR #154 then merged `WS-ART-001-02C1`" in loop_state + assert "fixed service action\n `artifact.checker_input.materialize`" not in contract_04b + assert "fixed service action `artifact.binding.create`" not in contract_05 + assert "normalized executable flag" in contract_04a + assert "same fixed" in contract_06a + assert "non-Unix" in storage_spec + assert "submission_bundle_admission_already_consumed" in storage_spec def test_stale_authorization_discovery_includes_new_untracked_docs() -> None: @@ -7161,7 +7242,7 @@ def test_artifact_chunk_verification_commands_are_isolated_and_rerunnable() -> N assert "run_isolated_tests.py" in metadata_command assert metadata_command.endswith("))") assert "alembic upgrade head" not in verification - if phase in {"03", "05"}: + if phase in {"03C", "05"}: assert "backend/scripts/api_contract_e2e.py" in contract assert "scripts/api_contract_e2e.py" in verification expected_contract_phase = artifact_contract_phase_for(phase)