Skip to content

fix(test): repair the fail-closed enrichment assertion that is red on main - #264

Merged
cryptoxdog merged 3 commits into
mainfrom
claude/cognitive-engine-graphs-bootstrap-grag6c
Sep 19, 2026
Merged

cryptoxdog merged 3 commits into
mainfrom
claude/cognitive-engine-graphs-bootstrap-grag6c

Conversation

@cryptoxdog

Copy link
Copy Markdown
Collaborator

Problem

tests/unit/test_gate_egress.py::test_request_enrichment_fails_closed_without_gate_url has failed on main since #259, which introduced the test and the implementation together. The Test Suite check on main HEAD (9c8e3cf) is red on this one assertion and nothing else:

tests/unit/test_gate_egress.py:85: in test_request_enrichment_fails_closed_without_gate_url
    assert result == {"status": "failed", "error": "gate_not_configured", "action": "enrich"}
E   AssertionError: assert {'action': 'e...us': 'failed'} == {'action': 'e...us': 'failed'}
E     Left contains 1 more item:
E     {'idempotency_key': 'ceg:enrich:acme:ent-1:aefbbc7455fffdbd'}

===== 1 failed, 1971 passed, 37 skipped, 56 xfailed, 7 warnings in 18.42s ======

Closes #

Fix

request_enrichment mints the idempotency key before the GATE_URL check and returns it on all three of its return paths (engine/gate_egress.py:99, :127, :136), so a caller holds a replayable key whether or not anything reached Gate. The fail-closed test asserted an exact dict that omitted the key — it encoded a contract the code never had.

Classified as a test defect, not a product defect: the success-path test in the same file already asserts result["idempotency_key"], so the engine's behaviour is the established one. No production behaviour changed; nothing was skipped, weakened, or mocked away.

Two changes:

  1. The fail-closed assertion now names the key via enrichment_idempotency_key(...) rather than a hardcoded digest, so the test cannot drift from the key function. The exact-dict form is deliberately kept — a loosened assertion would have hidden the original defect and would hide the next one.
  2. The SDK-error return asserted status and error but not idempotency_key, leaving the "always replayable" contract unasserted on that path. Added.

Overlap with #262 — read this before merging. Open PR #262 (feat/idea-portfolio-domain) already carries change (1), byte-for-byte identical to this branch's version. That is deliberate: this branch's hunk was aligned to #262's exactly so the two merge cleanly in either order, and main gets the fix whichever lands first. Change (2) is unique to this branch. Rejected alternative: dropping the duplicated hunk and waiting for #262 to merge — that leaves main red on a fix that is ready now, and makes this branch's value contingent on another PR's timing.

Risk

  • Low — additive, reversible, no data or contract change
  • Medium — touches shared code, config, or a public interface
  • High — breaking change, migration, IAM/network, or irreversible

Blast radius: one unit-test file. No engine/, chassis/, schema, or domain-spec surface is touched.
Rollback: revert the commits; main returns to its current red Test Suite.

Evidence

Run on Python 3.12.3 with requirements-dev.txt installed:

$ ruff check .
All checks passed!

$ ruff format --check .
363 files already formatted

$ mypy engine/
Success: no issues found in 133 source files

$ pytest tests/unit/
1421 passed, 7 skipped, 7 warnings in 9.31s

$ PYTHONPATH=. pytest tests/ -n auto --timeout=300     # CI's invocation shape
2006 passed, 10 skipped, 56 xfailed, 10 warnings, 25 errors in 15.75s

The 25 errors are tests/integration/ and tests/performance/ fixture failures: no Docker daemon in this container, so testcontainers-neo4j cannot start. Not a code failure and not suppressed — CI runs those against a live service.

Before the fix, the same unit run reported 1 failed, 1420 passed; the one failure is the assertion above.

Gates

  • Regression test added that fails without this fix — the changed assertions are the regression coverage; both fail against the pre-fix expectations
  • No secrets, tokens, or customer data in code, tests, fixtures, or logs
  • semgrep clean, or findings triaged below — not run locally; CodeQL and the repo's scanners run in CI on this PR
  • New IAM / workflow permissions are least privilege and enumerated — none added
  • Third-party actions pinned to a full commit SHA — no workflow files touched
  • Public interface change is documented and versioned — no public interface changed
  • Observability exists for the new path (metric, log, trace, or alert) — no new code path; the fail-closed branch already logs a warning

Reviewer focus

Confirm the direction of the fix: the test was wrong, not request_enrichment. The deciding evidence is that all three return paths in engine/gate_egress.py carry idempotency_key and the success-path test already asserted it.

Second, confirm the #262 overlap handling. The hunks are identical by construction; if #262 later edits that region, the two must be re-aligned or one merged first.

Changes by intent

Modified

  • tests/unit/test_gate_egress.py — fail-closed assertion realigned with the implemented contract (the mainline-red fix), plus the missing idempotency_key assertion on the SDK-error return

Files touched

pending — the bot fills this in on push

🤖 Generated with Claude Code

https://claude.ai/code/session_01BXqHneK2x7xKy1WkbBMumX


Generated by Claude Code

`request_enrichment` mints the idempotency key before the GATE_URL check
and returns it on every path, including the fail-closed
`gate_not_configured` branch. The test for that branch asserted an exact
dict without the key, so it has failed since the seam-audit commit that
introduced both (#259) — CI's Test Suite is red on main HEAD for this one
assertion and nothing else.

Classified as a test defect, not a product defect: the success-path test
in the same file already asserts `result["idempotency_key"]`, so the
contract is that the key is always returned. The assertion now names the
key via `enrichment_idempotency_key(...)`, keeping the exact-dict form so
no extra field can leak in unnoticed.

Verified: tests/unit 1421 passed, 7 skipped; full suite (no Docker)
2006 passed, 56 xfailed; ruff check + ruff format + mypy engine/ clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BXqHneK2x7xKy1WkbBMumX
Validate & Repair surfaced that the "always replayable" contract was
asserted on the success and fail-closed returns but not on the SDK-error
one, though request_enrichment returns the key there too. Adds the
missing assertion so every failing return of request_enrichment is
covered.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BXqHneK2x7xKy1WkbBMumX
Open PR #262 (feat/idea-portfolio-domain) already carries the identical
fail-closed assertion fix. The only difference was an explanatory comment
on this branch, which made the same region conflict textually and blocked
the overlap gate. Dropping the comment makes both sides of the hunk
identical, so whichever PR lands first the other merges clean. The
rationale lives in the PR body and in this branch's first commit message.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BXqHneK2x7xKy1WkbBMumX
Copilot AI lite review requested due to automatic review settings September 18, 2026 13:39
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@github-actions

Copy link
Copy Markdown

PR reviewable size is within recommended limits

@github-actions

Copy link
Copy Markdown

L9 Audit Harness Report

  • Generated: 2026-09-18T13:39:47.769996+00:00
  • Repo root: /home/runner/work/Cognitive.Engine.Graphs/Cognitive.Engine.Graphs
  • Overall result: ✅ PASSED
  • Exit code: 0

Step Results

Step Status Exit Code Notes
Architecture Audit ✅ Passed 0
Spec Coverage ✅ Passed 0
Contract Wiring ✅ Passed 0

Architecture Audit Findings

Severity Count
🔴 CRITICAL 0
🟠 HIGH 0
🟡 MEDIUM 17
🔵 LOW 0

See artifacts/audit_report.md for full details.

Spec Coverage

  • ✅ Implemented: 37
  • ⚠️ Partial: 9
  • ❌ Missing: 0
  • Total features: 46
Category Implemented Partial Missing Total
gates 10 0 0 10
scoring 7 0 0 7
v1.1_node 2 0 0 2
v1.1_edge 2 0 0 2
v1.1_action 0 2 0 2
v1.1_scoring 1 1 0 2
action_handler 0 6 0 6
gds_algorithm 5 0 0 5
research_pattern 10 0 0 10

See artifacts/coverage_report.md for full details.

Next Steps

All checks passed. Safe to merge.

@sonarqubecloud

Copy link
Copy Markdown

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The changes are limited to unit-test expectations/coverage and align them with existing engine behavior without altering production code paths.

Pull request overview

Repairs a unit test assertion in tests/unit/test_gate_egress.py so it matches the already-implemented request_enrichment() contract of always returning an idempotency_key, including on fail-closed and SDK-error paths, restoring green tests on main.

Changes:

  • Update the fail-closed test to assert an exact result dict that includes idempotency_key computed via enrichment_idempotency_key(...).
  • Add an idempotency_key assertion to the SDK-error failure-path test to enforce the “always replayable” contract consistently.
File summaries
File Description
tests/unit/test_gate_egress.py Fixes the fail-closed expected payload to include idempotency_key and extends SDK-error assertions to cover it too.
Review details
  • Files reviewed: 1/1 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copy link
Copy Markdown
Collaborator Author

OpenSSF Scorecard is red here, and it is not this PR's failure

What is failing: the OpenSSF Scorecard check, on head 9c69038.

Why: the job dies before any analysis runs — it cannot pull the action image.

/usr/bin/docker pull gcr.io/openssf/scorecard-action:v2.4.0
Error response from daemon: Head "https://gcr.io/v2/openssf/scorecard-action/manifests/v2.4.0":
denied: This API method requires billing to be enabled. Please enable billing on project #367732848534
##[error]Docker pull failed with exit code 1

That is billing on upstream OpenSSF's own GCR project, not anything in this repository. The job already retried the pull three times with backoff and got the identical response each time.

Why it is not attributable to this PR:

  1. This PR's entire diff is one unit-test file, tests/unit/test_gate_egress.py. It cannot influence a third-party action's container registry.
  2. The check is red on the base branch too — OpenSSF Scorecard has failed on main HEAD 9c8e3cf on 2026-09-05, 2026-09-07 and 2026-09-14, all predating this branch.
  3. The failure is in the install/pull phase, before any Scorecard rule is evaluated.

No fix ported, because none exists to port. A real repair means moving the workflow off the unbillable gcr.io image (for example to the GHCR-served ossf/scorecard-action release) — a change to an owner-managed CI surface, and no open PR in this repository carries it. Widening a one-test-file PR into .github/workflows/ to chase an upstream registry outage is the wrong shape, and deleting the analysis workflow to remove the check is explicitly not acceptable here.

Re-run: not spent. The job's own three retries, six seconds apart, plus three separate base-branch failures over two weeks, already establish that this reproduces deterministically; a fourth pull of the same unbillable manifest is not new evidence.

Everything else on this head is green, including the check this PR exists to fix:

Check Result
Test Suite ✅ success (×4)
Lint + Type Check ✅ success
Scan for Contract Violations ✅ success
L9 Audit Harness ✅ passed — 0 critical, 0 high
SonarCloud Quality Gate ✅ passed — 0 new issues
Secrets Scan (GitGuardian) ✅ success
OpenSSF Scorecard ❌ upstream image pull, above

Test Suite was failing on main before this branch; it passes here, which is the whole point of the change.

I am leaving this PR watched until it merges or closes.


Generated by Claude Code

cryptoxdog pushed a commit that referenced this pull request Sep 19, 2026
The enrichment assertion is red on main and takes seven checks down with it on
this PR: Test Suite, Pre-commit Hooks (via the pytest hook), Coverage (Codecov),
Quality Gate, Baseline Ratchet (Required Tests + Verdict, which records it as an
unledgered finding), and the CI Gate rollup that blocks merge.

The fix exists in open PRs #264 and #262, which make byte-identical changes to
this hunk. Porting it rather than waiting: it no-ops once main carries it.

This is the exact hunk both PRs carry, deliberately with no edits of my own.
An earlier attempt here rewrote the same assertion with an explanatory comment
inside the dict, which made it textually different and so a genuine add/add
conflict — that is what the overlap gate caught, and it was right to. Identical
text on both sides merges cleanly in a three-way merge, so this port conflicts
with neither PR.

engine/gate_egress.py documents the contract this restores: "retry is the
caller's decision and requires the idempotency key returned in the result".
All three return paths of request_enrichment honour it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nwd3DgnqkpEuaBMPKYLhF3
@cryptoxdog
cryptoxdog merged commit 06706c4 into main Sep 19, 2026
54 of 56 checks passed
@cryptoxdog
cryptoxdog deleted the claude/cognitive-engine-graphs-bootstrap-grag6c branch September 19, 2026 14:12
cryptoxdog added a commit that referenced this pull request Sep 19, 2026
…entifiers (#271)

* fix(tests): assert the full fail-closed enrich envelope in gate_egress test

test_request_enrichment_fails_closed_without_gate_url compared the whole
result dict against a three-key literal that omitted idempotency_key, so it
failed against the implementation it was introduced alongside (#259).

engine/gate_egress.py documents the authoritative contract: "one attempt per
call; retry is the caller's decision and requires the idempotency key returned
in the result". All three return paths of request_enrichment honour that, the
sibling success-path test already asserts result["idempotency_key"], and
engine/health/enrichment_trigger.py forwards the whole envelope to its caller,
so the key is load-bearing on the fail-closed path too.

The test encoded the wrong envelope, so the test is corrected. Strict whole-dict
equality is kept — the envelope stays exactly pinned, with the expected key
derived from the module's public enrichment_idempotency_key helper rather than a
hardcoded digest.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nwd3DgnqkpEuaBMPKYLhF3

* fix(compliance): stop the phone pattern matching inside opaque identifiers

handle_match builds query_id as `q_` + uuid4().hex[:12]. The PHONE value
pattern was the only one in _PII_PATTERNS without boundary guards, so any
10-digit run inside that token matched it. ComplianceEngine.redact_response
then removed the field outright (PIIHandler.redact pops the key), so the match
response silently lost query_id.

Measured on 200,000 generated ids: 1.572% were classified as a phone number.
That is a live response-shape defect, and it is what made
tests/test_handlers.py::test_match_returns_structure fail intermittently — the
assertion is correct, the engine was dropping the key.

`\b` is not usable here: the pattern can start with `+` or `(`, which are not
word characters, so a leading `\b` would break `+1 (555) 123-4567`. SSN and
IP_ADDRESS can use `\b` because they start with a digit. Lookarounds for
[0-9A-Za-z_] give the same protection without that constraint.

Verified: 0/200,000 false positives after the change, while `555-123-4567`,
`+1 (555) 123-4567`, `5551234567` and `call 555.123.4567 now` are all still
detected, as is detection by field name. Both directions are pinned by new
regression tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nwd3DgnqkpEuaBMPKYLhF3

* revert(tests): drop the gate_egress fix already carried by open PR #264

The overlap gate blocked publication: tests/unit/test_gate_egress.py
textually conflicts with PR #264 ("fix(test): repair the fail-closed
enrichment assertion that is red on main") and PR #262.

PR #264 already contains the same correction, derived independently and
byte-identical in the block it touches, and additionally asserts the
idempotency key on the SDK-error path. That PR owns this file, so carrying a
duplicate here would only add an add/add conflict for whichever landed second.

This branch keeps only the compliance repair, which neither #264 nor #262
touches. This PR therefore stacks on #264's head so the enrichment assertion
is inherited rather than duplicated.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nwd3DgnqkpEuaBMPKYLhF3

* fix(tests): port the fail-closed enrich assertion fix from #264

The enrichment assertion is red on main and takes seven checks down with it on
this PR: Test Suite, Pre-commit Hooks (via the pytest hook), Coverage (Codecov),
Quality Gate, Baseline Ratchet (Required Tests + Verdict, which records it as an
unledgered finding), and the CI Gate rollup that blocks merge.

The fix exists in open PRs #264 and #262, which make byte-identical changes to
this hunk. Porting it rather than waiting: it no-ops once main carries it.

This is the exact hunk both PRs carry, deliberately with no edits of my own.
An earlier attempt here rewrote the same assertion with an explanatory comment
inside the dict, which made it textually different and so a genuine add/add
conflict — that is what the overlap gate caught, and it was right to. Identical
text on both sides merges cleanly in a three-way merge, so this port conflicts
with neither PR.

engine/gate_egress.py documents the contract this restores: "retry is the
caller's decision and requires the idempotency key returned in the result".
All three return paths of request_enrichment honour it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nwd3DgnqkpEuaBMPKYLhF3

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants