ops: capture worker cgroup memory evidence

docs/adr/0247-worker-cgroup-memory-evidence.md

@@ -0,0 +1,72 @@
+# ADR 0247: Worker cgroup memory evidence before capacity limits
+
+- Status: Accepted
+- Date: 2026-08-27
+
+## Context
+
+The canonical worker was once observed with exit code 137 and was later
+recreated healthy. Exit 137 establishes a `SIGKILL`, not its cause. Recreation
+also discards the prior container's Docker state and cgroup counters, so the
+new container's `OOMKilled=false` cannot disprove a historical OOM.
+
+The base Compose service has no worker-specific memory limit or reservation.
+Docker therefore exposes the Docker Desktop VM capacity, not an accepted
+worker capacity envelope. Setting `mem_limit` from the current idle footprint,
+an arbitrary percentage, or an undocumented headroom multiplier would turn an
+unrepresentative observation into a production failure boundary.
+
+## Decision
+
+`scripts/capture_worker_memory_evidence.py` is the canonical worker-memory
+measurement procedure. It captures two snapshots around an explicitly chosen
+representative workload window from the unchanged `lineageweave` worker:
+
+- Docker status, exit code, `OOMKilled`, restart count, and configured memory
+  limit/reservation;
+- cgroup v2 `memory.current`, `memory.peak`, `memory.max`, and the keyed local
+  event counters in `memory.events.local`.
+
+The procedure rejects a container replacement, unavailable cgroup v2
+evidence, decreasing counters, and non-positive windows. It classifies OOM as
+confirmed only when Docker records `OOMKilled` or the kernel's local
+`oom_kill` counter increases. Exit 137 without either signal remains
+`sigkill_unattributed`. `high`, `max`, or `oom` deltas establish memory
+pressure without inventing an OOM kill.
+
+The core `low`, `high`, `max`, `oom`, and `oom_kill` counters are required.
+`oom_group_kill` is recorded when the host exposes it, but its absence remains
+an explicit `null` delta because neither OOM confirmation nor pressure
+classification depends on that optional group counter.
+
+If the unchanged worker exits during the window, Compose discovery includes
+stopped containers and Docker inspection preserves its terminal state. The
+terminated cgroup is no longer readable, so ending current usage and event
+deltas remain `null`; the output retains only the peak captured before exit
+and labels that limited scope. Docker `OOMKilled` may still confirm OOM and an
+otherwise unattributed exit 137 remains distinguishable. Every other terminal
+state without ending cgroup evidence is rejected rather than classified.
+
+No observation emits a memory-limit proposal. `memory.peak` is a measured
+maximum for that cgroup lifetime, but neither Docker nor the kernel defines a
+universal safety margin that turns it into a safe hard limit. A future limit
+requires an accepted representative workload/capacity envelope and a separate
+decision that names the workload, concurrency, host capacity, observation
+window, zero-OOM acceptance, and rollback procedure. Disabling the OOM killer
+is prohibited.
+
+## Consequences
+
+- Operators must capture evidence before recreating a failed worker.
+- A healthy idle sample proves only that the sampled window had no new local
+  pressure events; it is not capacity acceptance.
+- Canonical Compose remains unchanged until representative workload evidence
+  supports a bounded configuration.
+
+## References
+
+Docker, Inc. (2026a). *Define services in Docker Compose*. https://docs.docker.com/reference/compose-file/services/
+
+Docker, Inc. (2026b). *Resource constraints*. https://docs.docker.com/engine/containers/resource_constraints/
+
+The Linux Kernel Organization. (2026). *Control group v2*. https://docs.kernel.org/admin-guide/cgroup-v2.html

docs/adr/README.md

@@ -30,6 +30,8 @@ decision from them.
 | Product semantic catalog and typed evidence relations | [0228](0228-evidence-bound-product-semantic-catalog.md) |
 | Source-preserving voice semantic taxonomy | [0244](0244-source-preserving-voice-semantic-taxonomy.md) |
 | Expanded Voice-of-X post lookup and ontology | [0246](0246-expanded-voice-of-x-post-taxonomy.md) |
+| Worker cgroup memory evidence | [0247](0247-worker-cgroup-memory-evidence.md) |
+| [`WORKER_CGROUP_MEMORY_REFERENCES.md`](../doctoring/WORKER_CGROUP_MEMORY_REFERENCES.md) | [0247](0247-worker-cgroup-memory-evidence.md) |
 
 [0011](0011-prov-o-standard-relations.md) and [0065](0065-prov-o-provenance-boundary.md) cite the dated W3C PROV-O and PROV-DM Recommendations (https://www.w3.org/TR/2013/REC-prov-o-20130430/ and https://www.w3.org/TR/2013/REC-prov-dm-20130430/).
 

docs/doctoring/WORKER_CGROUP_MEMORY_REFERENCES.md

@@ -0,0 +1,22 @@
+# Worker cgroup memory references
+
+This supporting register documents the evidence boundary adopted by ADR 0247.
+Docker Compose defines `mem_limit` as a hard allocation limit and
+`mem_reservation` as a reservation. Docker Engine documents that the kernel
+kills container processes on OOM by default and warns against disabling that
+behavior without a hard memory limit. Linux cgroup v2 defines `memory.peak` as
+the maximum observed usage and `memory.events.local` as the non-hierarchical
+counter source; `oom_kill` counts processes killed by an OOM killer.
+
+These contracts do not specify a universal multiplier or percentage for
+turning one observed peak into a safe service limit. LineageWeave therefore
+records measured evidence and leaves the limit unset until a representative
+capacity acceptance is approved.
+
+## References — APA 7th
+
+Docker, Inc. (2026a). *Define services in Docker Compose*. https://docs.docker.com/reference/compose-file/services/
+
+Docker, Inc. (2026b). *Resource constraints*. https://docs.docker.com/engine/containers/resource_constraints/
+
+The Linux Kernel Organization. (2026). *Control group v2*. https://docs.kernel.org/admin-guide/cgroup-v2.html

docs/operability/postgresql-observed-tuning.md

@@ -46,3 +46,20 @@ uv run python scripts/plan_postgres_tuning.py rollback \
 
 The base `docker-compose.yml` contains no tuned command. Removing the tuning
 overlay and recreating PostgreSQL is the secondary rollback path.
+
+## Non-identifying canonical observation — 2026-08-27
+
+Since the 2026-08-24 statistics reset, the canonical PostgreSQL 16 instance
+reported 25,308 requested checkpoints versus 382 timed checkpoints, 336.7 GB
+of WAL, 7,598,680 `wal_buffers_full` events, 81,194,401 backend buffer writes,
+and no lock waiter at capture. The running configuration retained
+`wal_level=replica`, `max_wal_size=1GB`, and `shared_buffers=128MB` under read
+committed isolation.
+
+This snapshot confirms severe cumulative pressure, not an apply value.
+PostgreSQL documents that `max_wal_size` pressure can start a checkpoint before
+`checkpoint_timeout`, that high WAL output can require more WAL buffers, and
+that its own WAL recycling estimate adapts to prior checkpoint cycles. Run the
+aligned planner across the representative write workload before applying its
+segment-aligned proposal. The snapshot supplies no evidence for changing
+`shared_buffers`, durability, isolation, or storage concurrency.

docs/operability/worker-memory-evidence.md

@@ -0,0 +1,23 @@
+# Worker memory evidence procedure
+
+Run this before restarting or recreating a worker, over a declared window that
+contains the workload and concurrency being accepted:
+
+```bash
+uv run python scripts/capture_worker_memory_evidence.py \
+  --sample-seconds "$OBSERVATION_SECONDS" \
+  --output /tmp/lineageweave-worker-memory-evidence.json
+```
+
+The output contains aggregates and no container identifier or record content.
+Preserve it outside git with the workload definition and host capacity. An
+`oom_confirmed` result requires Docker `OOMKilled` or a local kernel
+`oom_kill` delta. `sigkill_unattributed` requires further host/runtime logs;
+do not relabel it OOM. A container change invalidates the window. If the same
+container exits, ending cgroup values remain unavailable and the retained
+pre-exit peak is labeled as such; it is not a whole-window maximum.
+
+Acceptance requires the declared representative workload to finish on one
+unchanged container with zero `high`, `max`, `oom`, and `oom_kill` deltas.
+The observed peak is evidence, not a proposed Compose limit. Any future
+`mem_limit`/`mem_reservation` change needs a separate ADR and rollback test.

docs/product-technical-gap-baseline.md

@@ -1,17 +1,23 @@
 # Product & Technical Gap Baseline
 
-> Dashboard delivery snapshot: 2026-08-27 01:44 KST. Protected `main` was
+> Dashboard delivery snapshot: 2026-08-27. Protected `main` was
 > `ff7431bd1851c03e737808d22c6a2d43968582f9`. Dashboard PR #640 exact
-> observed head was `5594029c801263a7f629c287ce41580ecf4e0739`; this branch is not
-> protected-main release evidence. The queue contained 29 open PRs (22
-> `BLOCKED`, five `UNSTABLE`, two `CLEAN`) and no exact-head approval. PR #715
+> observed head is `b3befa8bec8dd2807994444299df7eafdd1c7781`; this branch is not
+> protected-main release evidence. The queue contained 35 open PRs (23
+> `BLOCKED`, six `UNSTABLE`, four `CLEAN`, and two `UNKNOWN`) and no exact-head
+> approval. PR #715
 > merged normally into #640 and repaired the four stale HTTP transport test
 > doubles plus one Python-before-3.7 Semgrep false positive that contradicted
-> the repository's Python >=3.12 contract. Stacked PR #722 pre-documentation
-> head `eed7cabd` restores semantic-query and opt-in public-verification
-> factories in the dedicated Ask worker and the production-equivalent
-> concurrent-migration fixture path; its focused evidence is 46 unit tests and
-> one live Keycloak/PostgreSQL public-verification integration test.
+> the repository's Python >=3.12 contract. PR #722 also merged normally into
+> #640, restoring semantic-query and opt-in public-verification factories in
+> the dedicated Ask worker and the production-equivalent concurrent-migration
+> fixture path. PR #727 merged normally as `353dfd01`, replacing general-reader
+> implementation wording with evidence actions. Worker evidence PR #725's
+> implementation exact is `40283c9b`; subsequent evidence-only refreshes do not
+> change that code and its fresh hosted gates are pending. Storybook follow-up
+> #730 merged normally at `b3befa8b` from exact head `c21cac5d`.
+> Current #640 required checks are queued and independent approval remains
+> absent, so the candidate stays blocked.
 
 ## Operations Dashboard PRD/TRD traceability
 
@@ -25,8 +31,8 @@ the current #640 head. The canonical containers currently return HTTP 200 from
 backend `/healthz` and the frontend root, but their Compose labels do not prove
 the source commit; therefore neither the running stack nor the historical k6
 run is exact-head authenticated acceptance. Exact-head desktop/mobile
-screenshots and k6 remain required after #722 is incorporated and #640 is
-rebuilt. Historical test projects are retired only by their exact Compose
+screenshots and k6 remain required after #640 is rebuilt from its current
+exact head. Historical test projects are retired only by their exact Compose
 project label and without named-volume deletion. PR #678 implementation head
 `da98de07` fixes the default project name; its follow-up exact-label audit also
 removed the remaining identifiable isolated test containers while preserving
@@ -45,17 +51,20 @@ named volumes.
 | Similar VOC, customer cohort, prior action | Persisted repeat-issue candidate semantics plus orchestrator pair adjudication and extractive evidence | Candidate live post endpoint and post-detail UI implemented; authenticated runtime acceptance pending |
 | TEPP independent Event Lineage anchor | Accepted, persisted TEPP criterion bound to exact snapshot/cutoff before fast-mlsirm activation | Consumer PR #606 is on protected main; TEPP producer PR #237 remains open, so no end-to-end accepted artifact is release evidence yet |
 | Temporal Lineage topics and multilevel important posts | ADR 0210; TEPP posterior topic/plausible-value contract followed by fast-mlsirm observed-information case-deletion influence | This stacked candidate adds normalized persistence, exact run/snapshot/cutoff binding, pre-aggregation scope authorization, API diagnostics, and populated/unavailable Storybook surfaces. TEPP PR #247 remains open at `063f10f3`; stacked #251–#254 provide fail-closed input validation, full joint precision, deterministic joint plausible-value draws, and the canonical research register, while complete provenance assembly remains gated. fast-mlsirm PR #1418 validates the Rust consumer envelope but intentionally returns `EstimatorUnavailable` until the scientific estimator lands. Runtime therefore remains honestly unavailable with no local Python or fallback score. |
-| PostgreSQL WAL/checkpoint pressure | ADR 0227; aligned two-snapshot `pg_stat_wal`/checkpoint deltas, PostgreSQL WAL-segment and checkpoint constraints, cgroup memory, and data-volume space | Candidate procedure emits a content-authenticated plan and Compose environment, retains unmeasured memory/I/O/compression settings, preserves durability, validates the overlay without mutation, rejects stale preconditions, and requires an approved service recreation for apply or rollback. The observed CPU-bound GIN scan remains distinct from historical checkpoint pressure; canonical runtime application waits for the active migration to complete. |
+| PostgreSQL WAL/checkpoint and worker memory pressure | ADR 0227 and ADR 0247; aligned PostgreSQL counter deltas plus unchanged-container Docker/cgroup v2 evidence | Candidate PostgreSQL procedure emits a content-authenticated restart/rollback plan while retaining unmeasured settings and durability. The current cumulative counters establish sustained historical WAL/checkpoint pressure but do not replace a representative aligned apply window. A prior worker exit 137 is not attributable after container recreation: the current healthy worker has no configured service memory limit/reservation, and a one-second non-identifying observation showed an approximately 109 MiB cgroup lifetime peak with no new local pressure/OOM event. That idle window is not capacity acceptance. Capture the declared representative workload before recreation; do not add a limit or headroom multiplier until that evidence supports a separately accepted capacity boundary. |
 
-Customer-copy audit at #640 exact `c142c4ea` retained the ADR-required
+Customer-copy audit began at #640 exact `c142c4ea` and was delivered by #727
+at merge `353dfd01`; it retained the ADR-required
 measurement-administrator terms and explicit ontology/provenance inspection
 labels. Two general-reader gaps were isolated: Customer Master explained the
 ontology/semantic implementation boundary instead of the evidence action, and
 Global Ask called authorized workspace evidence "internal" posts. The stacked
 copy repair tells the reader to compare the source identifier with related
 posts and organization evidence, and reuses the authorized-citation action.
 Five-locale consistency, rendered component tests, and desktop/narrow
-Storybook scenes cover the repair. `프로젝트별 관측 Event` and ADR 0210's exact
+Storybook scenes cover the Customer Master repair. Follow-up #730 renders the
+Global Ask no-public-claim result and its next action in desktop and narrow
+scenes delivered at merge `b3befa8b`. `프로젝트별 관측 Event` and ADR 0210's exact
 `model influence` estimand name remain unchanged.
 
 ### Technical contract and flow