coverage-evidence sandbox: stale requirements hash-lock (missing asyncpg/psycopg2) + invalid pnpm --trust-lockfile flag block LineageWeave approvals

[seonghobae]

Problem

The central coverage-evidence sandbox (OpenCode Review Dispatch workflow, coverage-evidence job, "Measure test and docstring evidence" step) blocks OpenCode approval for LineageWeave PRs with a spurious FAIL that has nothing to do with the PR's actual code.

Evidence

Run 32628753453 against ContextualWisdomLab/LineageWeave#417 at head 91b7e1cd914442e61e322620f194d1ba9da48567 (a clean, unrelated docs/backend PR — TEPP topic-lineage ADR + analysis-run scaffolding). Result: Coverage Decision: FAIL, Failure count: 4. All 4 are environment/tooling failures in the sandbox itself, not the PR:

1. Python test collection: 39/39 test files fail to import

ModuleNotFoundError: No module named 'asyncpg'
ModuleNotFoundError: No module named 'psycopg2'

Every test file that imports LineageWeave's backend app modules fails collection (1 skipped, 39 errors in 1.57s). asyncpg>=0.29.0 and psycopg2-binary>=2.9.12 are genuine, declared dependencies in LineageWeave's pyproject.toml — they're not missing from the PR, they're missing from the sandbox's pip install --require-hashes -r /tmp/requirements-opencode-review-ci-hashes.txt step. That hash-pinned requirements file appears stale or generated from an incomplete source relative to LineageWeave's actual pyproject.toml.

2. JS/TS dependency install: unsupported pnpm flag

$ corepack pnpm install --offline --frozen-lockfile --trust-lockfile --ignore-scripts --store-dir ...
ERROR  Unknown option: 'trust-lockfile'
Did you mean 'frozen-lockfile'? Use "--config.unknown=value" to force an unknown option.

--trust-lockfile isn't a valid pnpm flag on the version installed in the sandbox (pnpm 11.17.0 per LineageWeave's own CI). This looks like either a typo for --frozen-lockfile (already present separately in the same command) or a flag from a different pnpm major version.

3 & 4. Cascading failures from #2
JS/TS test coverage (vitest: not found — node_modules never got installed because of #2) and JS/TS coverage threshold (no coverage summary files were produced) both fail as a direct downstream consequence.

Impact

This is a hard block, not a slow-down: coverage-evidence failing means OpenCode explicitly posts REQUEST_CHANGES ("OpenCode cannot approve yet because required coverage evidence did not pass") regardless of the PR's actual content. Since asyncpg/psycopg2 are core LineageWeave backend dependencies, this should reproduce on essentially any LineageWeave PR that touches backend/tests — i.e., this is very likely blocking a large fraction of the current org-wide review backlog independently of .github#624's model-pool issue.

Not attempting a fix

The install pipeline here is deliberately hardened (hash-pinned pip install --require-hashes --only-binary=:all:, VCS-locked base dependencies with import-root/symlink/compiled-extension validation, trusted-source materialization without a repository token) — consistent with the same kind of anti-tampering design already documented as under active maintenance elsewhere in this org's CI. I don't have visibility into how requirements-opencode-review-ci-hashes.txt or the pnpm invocation are generated/maintained, so I'm not guessing at a fix here; recording the evidence for whoever owns that generation step.

[seonghobae]

Related: same bug class as .github#1234

A peer session (working on RankWeave) flagged that this is structurally the same failure mode as .github#1234 — the central coverage-evidence runner's pinned/hash-locked tooling desyncing from an individual repo's actual manifest. There, it's uv's version lagging RankWeave's pyproject.toml pin; here, it's the requirements-opencode-review-ci-hashes.txt hash-lock missing asyncpg/psycopg2 that LineageWeave's pyproject.toml actually declares. Different symptom, same root shape: the central sandbox's per-language dependency snapshot isn't kept in sync with each repository's own manifest/lockfile.

Whoever picks up either issue may want to look at both together — the fix is likely the same class of change (regenerate/re-derive the sandbox's pinned dependency set from each repo's current manifest rather than a separately-maintained snapshot), just applied to two different ecosystems (Python hash-lock here, uv version there).

Not attempting a fix myself, same reasoning as before.

[seonghobae]

Root cause is now repaired on the existing OpenCode owner PR #1052 at exact head 8f106a2539d4f2e156f5ee79693fb300a1312e19.

Fresh run evidence from 32628753453/97169032536 shows the central tool lock installed successfully, and the trusted consumer base uv.lock was also materialized, preflighted, and installed successfully. The contradiction occurred because uv export omitted project optional extras by default, so LineageWeave's dev/backend imports (asyncpg, psycopg2-binary, redis, and the exact VCS-pinned fast-mlsirm) were absent despite the successful lock-install receipt. The repair adds --all-extras to the frozen/offline trusted base export; it does not expand the central fixed CI-tool lock with consumer-specific packages.

The same run proved the independent JavaScript cause: the validated base/head pnpm lock was accepted, but pnpm rejected the unsupported --trust-lockfile option before installing dependencies. The repair removes only that unsupported option while retaining the exact base/head lock-blob comparison, offline store, --frozen-lockfile, and disabled lifecycle scripts.

Verification: RED 4 failures; focused GREEN 73 tests; full local suite 1,423 passed with 16 subtests after updating the intentional workflow-blob receipt. Keep this issue open until exact-head hosted checks pass and a fresh unchanged LineageWeave #417 canary completes coverage coherently.

[seonghobae]

Hosted exact-head evidence is now complete on owner #1052 at 4aa738ac400f05756b6b3b250673d1e0133a6874 against main@9ad0ad50409561292b424d6f35a95d670a277e77.

• all 15 substantive hosted workflows succeeded (Strix, CodeQL, SAST, Security Scan, Secret Scan, Python Security, OSV, SBOM, Scorecard, exact artifact/SBOM, organization readiness, Rust toolchain, trusted uv materializer, agent router, NIM repair); the unrelated CodeRabbit repair workflow is skipped;
• the hosted predecessor Strix run reproduced the stale --trust-lockfile assertion only after 1,497 tests, 1 skipped, and 16 subtests passed; the exact-head contract now rejects that unsupported option;
• the same exact head also repairs and tests multi-page coverage check-run receipts with gh api --paginate --slurp (focused 48 passed), and all review threads are resolved.

This is source/hosted owner evidence, not downstream acceptance. Keep #1250 open until protected-main integration and a fresh unchanged/current LineageWeave #417 coverage canary prove the optional dev/backend dependencies install and pnpm no longer fails. #1052 still has no exact-current-head formal Reviews API verdict, so it was not merged.

[seonghobae]

Project #1 pickup: Codex is following owner PR #1052 through protected-main integration and a fresh unchanged LineageWeave canary. Status: In Progress; phase: Ops; component: dot-github. No duplicate implementation branch was created because the minimal root-cause repair already exists on the active owner PR.

[seonghobae]

Fresh pnpm HEAD-lock trust asymmetry (Inkspan canary)

Fresh unchanged consumer evidence now exposes the next central boundary after the Storybook runtime-classifier repair:

• Inkspan #362 exact head 11d5cfecdcc0949ec98e6ca110d482124bff00c4, base 3b38ead2d00f44eb578d0689087b9293b3dabe1e.
• Central OpenCode run 32702575402, coverage-evidence job 97411185557, fails before measurement with:
  Current pnpm lock pnpm-lock.yaml differs from the validated base; refusing --trust-lockfile for PR-controlled dependency resolution.
• Protected main@613a33e0cb1c6db9790fae99f6253445712ac37a already excludes .storybook/** and *.stories.* from product-runtime coverage classification. The canary's changed package.json/lock add the Storybook scripts and exact Storybook dev dependencies; this is no longer the historical Storybook path-classifier defect.

Root cause is an npm/pnpm trust asymmetry in the central sandbox. materialize_base_javascript_packages.py --head-sha validates and materializes a changed current-head npm lock into the trusted offline cache and manifest, but materializes only the base pnpm lock. Runtime trusted_pnpm_lock_matches_base() consequently rejects every legitimate changed pnpm lock before the already-preserved --offline --frozen-lockfile --ignore-scripts install can run.

The narrow safe repair belongs on the existing OpenCode owner path #1052, currently exact head 8ea117c06de1e4274044a0636a91f0b9ff82dd58; no competing branch was created. Acceptance should add RED fixtures for a validated changed pnpm lock and unsafe/unverifiable resolution controls, then give pnpm the same exact revision + lock-blob + worktree-blob + trusted-manifest binding already used for npm. The image build may populate only an integrity/provenance-validated HEAD lock into the tokenless offline store; the consumer sandbox must retain offline, frozen-lockfile, disabled lifecycle scripts, symlink/path escape rejection, and fail closed for unbounded registry/VCS/file/workspace provenance.

GREEN still requires focused/full central tests, exact-head hosted evidence, protected-main integration, and a fresh Inkspan #362 canary. This comment records a causal owner boundary, not passing coverage evidence.

[seonghobae]

Fresh Inkspan canary: validated changed pnpm lock still lacks HEAD archive closure

A second unchanged Inkspan canary exposes a distinct downstream boundary in the same central coverage-evidence owner lane:

• consumer: ContextualWisdomLab/inkspan#373
• exact head: c2db417df2c8f041d31ac5880c4898f00692c04e
• exact base: protected inkspan/main@3b38ead2d00f44eb578d0689087b9293b3dabe1e
• central required-workflow run/job: 32702585538 / 97411969364
• trusted central workflow source for that historical run: 33b69e160945390a6fb2e7dc6e32e1bcf25460c8
• current central protected main: 613a33e0cb1c6db9790fae99f6253445712ac37a
• existing OpenCode owner PR: fix(opencode): split review surfaces, give NIM two hours, and remove GitHub Models #1052, current exact head f3e43ef71ded66adf15d6ad2e03148bfa92e24d2

The run correctly resolved the exact consumer base/head pair and changed dependency files, then failed before measurement during its single deterministic pnpm install. pnpm install --frozen-lockfile --offline could not retrieve @sindresorhus/merge-stream@4.0.0: pnpm reported ERR_PNPM_META_FETCH_FAIL because cache mode was only-if-cached and the archive/metadata was absent from the trusted offline store.

This is not an Inkspan source/test failure. It is also narrower and later than the #362 changed-lock trust asymmetry already recorded above: accepting/provenance-binding a legitimate HEAD pnpm lock is insufficient unless the trusted build phase materializes the complete integrity-bound package closure named by that exact HEAD lock into the tokenless offline store.

Falsifiable RCA: with the exact validated HEAD lock unchanged, the coverage image/store is constructed without at least one package archive required only by HEAD; therefore the later offline/frozen consumer install cannot execute even though the lock itself is authoritative.

Owner acceptance should add a RED fixture where base lacks a dependency and the validated HEAD lock adds it, proving the current build/store path cannot satisfy the offline install. The narrow GREEN should materialize only the exact integrity/provenance-validated HEAD lock closure into the trusted build cache, while preserving --offline, --frozen-lockfile, disabled lifecycle scripts, path/symlink escape rejection, and fail-closed handling for registry/VCS/file/workspace provenance that cannot be authenticated. Do not substitute registry access in the consumer sandbox or treat a missing archive as source failure.

After owner integration, rerun this same unchanged inkspan#373@c2db417… canary and require the coverage job to reach real measurement. Historical run/job output remains non-passing and must not be transferred.

[seonghobae]

Orgmetra canary confirms the central coverage-evidence owner boundary is still active on protected-main .github@613a33e0cb1c6db9790fae99f6253445712ac37a.

Target: ContextualWisdomLab/Orgmetra#54@cc6784ec33b1145c342bbbb99ebece1d37aeec80 (unchanged target head). Orgmetra-native exact-head evidence is terminal GREEN on the same SHA: Workforce Intelligence 32446632080, Foundation 32446632202, People 32446632154, Job-Analysis 32446632216, Recovery 32446632120, SAST 32446632060, Security 32446632029; the earlier target-local OpenCode coverage-evidence check on this same SHA also succeeded (96744426819).

Fresh central repository-dispatch run 32790319079 nevertheless failed in .github, with coverage-evidence job 97638916455 failing specifically at Measure test and docstring evidence; OpenCode then posted REQUEST_CHANGES on Orgmetra #54 citing 12 coverage-evidence failures. The dispatch run's source is central main@613a33e0..., not the Orgmetra branch. No Orgmetra source/head change explains the contradiction.

Current existing owner PR #1052 is still open/unmerged at f3e43ef71ded66adf15d6ad2e03148bfa92e24d2 against main@613a33e0...; do not create a competing repair from Orgmetra. Acceptance for this canary: after the existing central owner repair integrates, redispatch unchanged/current Orgmetra #54 and require coverage-evidence + OpenCode outcome to pass without changing Orgmetra solely to satisfy central sandbox/tooling drift.

[seonghobae]

Orgmetra canary: exact-head coverage is GREEN locally, central hash-binding rejects it

Fresh downstream evidence isolates a new/current central coverage-evidence boundary for ContextualWisdomLab/Orgmetra#40.

• target PR/head: ContextualWisdomLab/Orgmetra#40@8d8896b14db10a5a4981f0b9e209ea00ee3be64c
• target-local Structured Interview Plan Quality run/job: 32476007995 / 96752378435
• the target-local job explicitly checked out and asserted that exact SHA, ran 110 tests, and produced exact owned production coverage 221 statements / 64 branches = 100%, then passed clean-checkout
• all five target-local exact-head workflows for this SHA are terminal GREEN: Structured Interview Plan Quality, Foundation CI, Recovery Rehearsal, SAST Semgrep, Security Scan
• central repository-dispatch OpenCode run/job: 32898120719 / 97966384105, central source ContextualWisdomLab/.github@613a33e0cb1c6db9790fae99f6253445712ac37a
• first failing step: Create hash-bound coverage evidence; review/model/fallback steps before it succeeded
• the central step detected a coverage.py JSON artifact at 100.00% with fingerprint 6376819fc7be6f4957c186450a441308712028868dc9dfa401bd59e8c0b93583, but rejected it because the analyzer/evidence provenance was not accepted as bound to the exact target SHA, ending with coverage-evidence-step-failed: could not verify target coverage evidence
• consequence: OpenCode posted CHANGES_REQUESTED on ⚡ Bolt: JSON 파싱 성능 최적화 #40 despite the unchanged exact target head having complete target-local GREEN coverage/security/recovery evidence.

Owner boundary: no Orgmetra source/test/gate failure is reproduced here. The consumer job already proves exact checkout + exact 100% statement/branch coverage. The first failing boundary is the central review sandbox's construction/acceptance of hash-bound coverage provenance, so this belongs on the existing OpenCode owner path rather than an Orgmetra workaround. No .github source/ref/workflow/PR state was modified from the Orgmetra loop.

RED acceptance: an unchanged consumer SHA whose target-local run explicitly proves exact checkout and 100% coverage must not receive CHANGES_REQUESTED merely because central evidence generation loses/fails to bind that already-known target SHA. Unsafe/stale/predecessor evidence must still fail closed.

Smallest remedy: at the central evidence-construction boundary, bind the coverage artifact/analyzer receipt to TARGET_REPO + TARGET_SHA from the authenticated dispatch and verify that binding against the exact checkout/run being measured; do not relax the 100% threshold or accept summary-only/predecessor evidence.

Required GREEN proof: focused central regression for exact-SHA binding + full current owner-PR checks + protected-main integration + fresh unchanged Orgmetra #40 canary where the coverage-evidence step is terminal GREEN and the review no longer requests changes for this cause.

Current existing owner PR #1052 remains open; fresh API state observed its branch at abf47ce275fd8c1efa8306d30f1d6afbadd989ab. This comment only routes the causal evidence into the existing owner path.

[seonghobae]

Orgmetra target-side canary update (read-only foreign-owner handoff; no .github source/ref/workflow mutation):

• ContextualWisdomLab/Orgmetra#40 exact head 8d8896b14db10a5a4981f0b9e209ea00ee3be64c has Orgmetra-native exact-head GREEN and 100% owned coverage, but central repository-dispatch run 32898120719 failed coverage-evidence job 97966384105 at Measure test and docstring evidence; downstream OpenCode therefore REQUEST_CHANGES.
• The target repository has now opened Orgmetra PR Document html4tree governance onboarding gap #122, fix/central-sandbox-sibling-imports, exact head 60ad07a02c766b54673d7cbbdd7e7bdcd08d129e. It is intentionally target-owned: service-local test bootstrap adds workspace packages/*/src and services/*/src only during pytest collection, and repository-file tests resolve paths from Path(__file__) rather than process CWD. Production runtime imports are not changed.
• Document html4tree governance onboarding gap #122 exact-head Foundation 32906466684, People 32906466790, Job-Analysis 32906466945, Recovery 32906466656, SAST 32906466774, Security 32906466865 are still queued, so this is not yet claimed as GREEN.

Acceptance/canary after #122 itself is GREEN and integrated through normal Orgmetra governance: redispatch the unchanged/then-current #40 (and #54) target SHA through the central coverage sandbox. The owner repair is complete only when plain service-directory pytest collection and hash-bound coverage evidence succeed against the exact target SHA without broadening sandbox trust, weakening coverage/docstring gates, or requiring target production-runtime path mutation.

[seonghobae]

Orgmetra canary refinement: central nested-coverage invocation is independently reproducible

Fresh read-only analysis of the exact merge-tree artifact produced by central run 32898120719 (coverage-source-tree artifact SHA-256 4d3605b39436918d2436ffe3d8ace580d75578b2dc4fa637cd5bef34f1b0b0a8) isolates an additional/earlier central measurement defect that target PR #122 cannot correctly repair.

Historical target remains ContextualWisdomLab/Orgmetra#40@8d8896b14db10a5a4981f0b9e209ea00ee3be64c; central failing job is 97966384105, step Measure test and docstring evidence. The trusted central workflow's generic Python fallback runs each tracked project as:

cd "$project_dir" && PYTHONPATH="$([ -d src ] && printf src:. || printf .)" python3 -m coverage run -m pytest tests && python3 -m coverage report --show-missing

On the downloaded exact #40 merge tree, packages/interview-plan/pyproject.toml already configures pytest-cov through pytest addopts (--cov=orgmetra_interview_plan --cov-branch --cov-fail-under=100). Replaying the central command produces a falsifiable two-layer result:

• python3 -m coverage run -m pytest tests exits 0; pytest-cov runs 110 tests, reports 221 statements / 64 branches = 100.00%, and satisfies its 100% threshold;
• the outer python3 -m coverage report --show-missing then exits 1 with CoverageWarning: No data was collected / No data to report, because the nested pytest-cov session owns/resets the coverage data used during pytest.

The same pattern reproduces on other Orgmetra Python packages that configure pytest-cov in pyproject.toml: functional tests and the package's own exact coverage gate pass, then the central wrapper's second coverage report fails. This failure exists even before service-local sibling-import/CWD issues addressed by Orgmetra #122, so removing pytest-cov configuration from every target package would be an incorrect target-side workaround that weakens/duplicates owned coverage contracts.

Owner boundary: central .github coverage measurement orchestration. Orgmetra #122 should remain only the target-owned fix for service-local workspace imports and CWD-independent repository-file tests; it cannot make this nested-coverage design correct.

RED acceptance: a project whose pytest configuration already enforces exact coverage with pytest-cov must not be wrapped in a second coverage run and then rejected because the outer collector has no data; actual test/coverage failures must still fail closed.

Material remedies to evaluate at the existing owner boundary: (1) detect trusted pytest-cov configuration and execute pytest directly, accepting its exact threshold/report evidence through the existing hash/provenance binding; or (2) if central coverage must own collection, invoke pytest with repository addopts coverage instrumentation disabled in a narrowly controlled way and let the outer collector own measurement while preserving the repository's source/branch/threshold semantics. Do not simply ignore a nonzero coverage-report exit or accept summary-only/stale evidence.

Required GREEN proof: focused central regression for a pytest-cov-configured project + full owner PR checks + protected-main integration + fresh unchanged/current Orgmetra #40/#54 canary where Measure test and docstring evidence and downstream OpenCode review both pass on the exact target SHA.

No .github source/ref/workflow/PR state was modified from the Orgmetra loop.

[seonghobae]

Fresh Orgmetra owner-boundary evidence (2026-08-26): .github owner PR #1052 has advanced to actual head abf47ce275fd8c1efa8306d30f1d6afbadd989ab (its PR body still describes predecessor heads). Read-only inspection of .github/workflows/opencode-review-dispatch.yml at that exact head shows the generic Python fallback remains:

python3 -m coverage run -m pytest tests && python3 -m coverage report --show-missing

inside run_python_test_coverage() when no configured CI command is discovered. That is the same first causal boundary reproduced by Orgmetra #40 central run 32898120719 / coverage-evidence job 97966384105: pytest-cov itself completes 110 tests with 221 statements / 64 branches at 100%, then the outer coverage collector reports No data was collected / No data to report.

Acceptance remains: detect/execute the target's trusted configured pytest-cov command without wrapping it in a second coverage collector, or otherwise ensure only one authoritative coverage collector owns the run; preserve exact branch+statement evidence and sandbox isolation. After owner repair integrates, require fresh repository-dispatch canaries for unchanged/current Orgmetra #40 and #54 heads before clearing their OpenCode CHANGES_REQUESTED. This comment is handoff evidence only; no .github source/ref/workflow/PR state was mutated by the Orgmetra loop.

[seonghobae]

Fresh Orgmetra canary recheck (2026-08-26): the foreign-owner defect remains present on current .github PR #1052 head abf47ce275fd8c1efa8306d30f1d6afbadd989ab (live base main@d2c554dbbc04854db6215970fabb70cef1ceb690). Read-only inspection of .github/workflows/opencode-review-dispatch.yml at that exact head still contains the generic fallback:

python3 -m coverage run -m pytest tests && python3 -m coverage report --show-missing

for each discovered Python project when no configured command is selected. That is the same first causal boundary previously reproduced on Orgmetra #40: packages/interview-plan already runs its own pytest-cov contract to 100%, then the outer generic coverage process reports No data was collected / No data to report, causing central coverage-evidence run 32898120719 to fail and OpenCode to post CHANGES_REQUESTED on exact Orgmetra head 8d8896b14db10a5a4981f0b9e209ea00ee3be64c. Orgmetra #54 remains the second affected canary.

Acceptance remains unchanged: the owner fix must avoid double-wrapping a project that already supplies authoritative pytest-cov/coverage evidence, preserve exact-head binding and fail-closed 100% thresholds, and then produce fresh central GREEN/OpenCode canaries for Orgmetra #40 and #54. No .github source/ref/workflow/PR state was modified from the Orgmetra loop.