test(agentic-engineering): guard surveyor review contracts upstream

.claude-plugin/marketplace.json

@@ -27,7 +27,7 @@
     {
       "name": "agentic-engineering",
       "description": "The autonomous engineering system for repository portfolios — engineer, read-only surveyor, and meta-engineer agents; portfolio, product, spend, and improvement workflows; cross-tool instruction architecture and skill discovery; configured by the consumer AGENTS.md",
-      "version": "5.0.1",
+      "version": "5.0.2",
       "source": "./plugins/agentic-engineering"
     },
     {

.github/plugin/marketplace.json

@@ -27,7 +27,7 @@
     {
       "name": "agentic-engineering",
       "description": "The autonomous engineering system for repository portfolios — engineer, read-only surveyor, and meta-engineer agents; portfolio, product, spend, and improvement workflows; cross-tool instruction architecture and skill discovery; configured by the consumer AGENTS.md",
-      "version": "5.0.1",
+      "version": "5.0.2",
       "source": "./plugins/agentic-engineering"
     },
     {

docs/surveyor-contract-coverage.md

@@ -0,0 +1,110 @@
+# Surveyor contract coverage
+
+[`surveyor-review-contract.test.sh`](../plugins/agentic-engineering/scripts/surveyor-review-contract.test.sh)
+checks the operative instructions in the bundled
+[`portfolio-surveyor`](../plugins/agentic-engineering/agents/portfolio-surveyor.agent.md).
+The `lint-scripts` CI job discovers this suite beside the plugin's other tests. It checks structural
+drift, not whether a model follows the instructions. It introduces no runtime policy or classifier.
+
+Every clause has a stable test ID. Each must appear inside its own section; the suite removes it
+independently and requires rejection even with a copy elsewhere in the document. The corresponding
+whitespace-only reflow must pass. Missing, repeated, or reversed section boundaries fail; a reversed
+boundary cannot extend a scope to an example at the end of the file. Empty definitions fail.
+These controls demonstrate the guards can
+detect missing operative instructions without requiring the entire role to retain identical bytes.
+They do not parse arbitrary Markdown or prove that nearby contradictory prose is harmless.
+
+Run the suite from the repository root:
+
+```sh
+bash plugins/agentic-engineering/scripts/surveyor-review-contract.test.sh
+```
+
+For a direct RED check on a temporary mutated definition, use the test-only interface:
+
+```sh
+bash plugins/agentic-engineering/scripts/surveyor-review-contract.test.sh --check /path/to/mutant.md
+```
+
+## Coverage ownership
+
+The inventory for [#95](https://github.com/devantler-tech/agent-plugins/issues/95) is anchored to
+the consumer's
+[test revision f8fe925d](https://github.com/devantler-tech/monorepo/blob/f8fe925dd1c8c623606af7d8e5bdd554b6a46f37/.claude/scripts/portfolio-surveyor.test.sh),
+the latest change to that file before the issue was filed. It contains **28 checks directly against
+the surveyor plus three compatibility-overlay loading checks**. The 31 entries below distinguish
+those checks; the issue's shorthand is not a claim that all 31 are generic. Other tests in that
+consumer file cover its classifier, product cards, security surveyor, and deployment policy.
+
+“Consumer” means the concrete deployment assertion remains there. A generic replacement pins the
+current plugin contract rather than importing a login, namespace, path, or retired spelling. No
+consumer overlay or test is removed by this suite.
+
+| Inventory | Source line | Assertion | Current coverage and owner |
+|---|---:|---|---|
+| O01 | 68 | Exact local programmed-bot classifier path | Consumer path; E01 pins a consumer-declared exact classifier and exit 0. |
+| O02 | 76 | Programmed-bot exemption digest state | E05; ordinary hygiene remains required. |
+| O03 | 80 | Botantler is only a classifier candidate | Consumer identity; E01/E04 pin successful classification and fail-closed errors. |
+| O04 | 82 | KSail App search identity | Consumer identity; A04 pins exact trusted-login matching. |
+| O05 | 84 | Complete current-head commit provenance | E02/E03 pin the full commits endpoint result and final-commit/head match. |
+| O06 | 96 | Cursor Automation trusted-author identity | Consumer grant and identity; A04 pins exact matching. |
+| O07 | 118 | Connector head match before recency | R06/R07. |
+| O08 | 120 | Same-head connector findings win | R08–R11 include the current explicit resolution/re-request exception. |
+| O09 | 122 | Abbreviated connector SHA is a head prefix | R02/R03/R12 include minimum length and backtick tolerance. |
+| O10 | 126 | Reject abbreviated/full-SHA equality | R02 positively requires prefix matching and forbids full-length equality. |
+| O11 | 131 | Well-formed nonmatching marker is stale | R04. |
+| O12 | 133 | Missing, malformed, or short marker is none | R05 replaces the weak single-word `absent,` check. |
+| O13 | 141 | Failed check-run is not findings | K02/K04 separate the title-based states. |
+| O14 | 143 | Failed check-run has an error state | K04 and D02 pin the error and signal grammar. |
+| O15 | 149 | Usage-limit signal is representable | D02/T03 pin the generic lane grammar and meaning; no fixed reviewer roster or occurrence count. |
+| O16 | 161 | Dependency automation short-circuits | A01/A02, conditional on the consuming contract's designation. |
+| O17 | 168 | Budget sampled at start and end | B01/B02 pin the timing and both budgets; no particular forge CLI is required. |
+| O18 | 170 | Budget digest grammar | D01 pins the current generic start/end spelling. |
+| O19 | 172 | Budget exhausted at start is explicit | B03. |
+| O20 | 174 | Every digest carries its budget | D04; B04 covers an unavailable probe. |
+| O21 | 176 | Exact Renovate identity | Consumer identity; A01 pins the declared exact identities. |
+| O22 | 178 | Exact Dependabot identity | Consumer identity; A01 pins the declared exact identities. |
+| O23 | 180 | No heavy automation-owned PR deepening | A02. |
+| O24 | 182 | Automation-owned PR does not count as fire | A03. |
+| O25 | 772 | All declared writer namespaces are scanned | C01; the actual namespace list stays in the consumer. |
+| O26 | 774 | Claim branch matches issue number across lanes | C01/C02/C05 include the issue suffix and takeover suffix. |
+| O27 | 776 | Claim scan is not assignee-gated | C03. |
+| O28 | 778 | Unassigned writer-lane claim is representable | D03/D07; C04/D08 additionally require the declared lease before using it as a skip. |
+| O29 | 188 | Maintenance skill declares compatibility overlay | Consumer loading contract; retained until consumer parity is proven. |
+| O30 | 190 | Maintenance skill instructs loading that overlay | Consumer loading contract; retained until consumer parity is proven. |
+| O31 | 192 | Maintenance skill must not forbid overlay loading | Consumer loading contract; retained until consumer parity is proven. |
+
+Additional generic coverage addresses the invariants explicitly named in #95 and their current
+semantics: H01 pins complete thread pagination; H02–H05 pin newest-review selection by submission
+time, empty newest findings, and stale body findings; R01 pins connector authentication; K01/K05/K06
+pin the check-run versus login split; K03 pins neutral findings; P01/D05/D06 pin complete queries and
+incomplete-candidate handling; T01/T02 pin evidence-bearing, per-lane `none` output.
+
+The separate existing suites retain their own responsibilities:
+
+- `portfolio-surveyor-agent.test.sh`: the prescribed dependency-summary projection and guard admission.
+- `surveyor-open-pr-links.test.sh`: the prescribed linked-PR count projection and malformed responses.
+- `surveyor-selection-contract.test.sh`: section-scoped ranking and actionability obligations.
+- The forge-guard, adapter, and classifier suites: executable command-boundary behavior.
+
+Passing these checks does not authorize deleting the consumer overlay. The consumer must still
+verify its current overlay/procedure parity and preserve deployment-specific coverage before
+removing duplicated generic assertions. Its concrete identities, lease policy, local classifier,
+provider wiring, and product-specific tests remain consumer-owned.
+
+## Model-behavior evaluation
+
+[`fixtures/surveyor-review.json`](../plugins/agentic-engineering/scripts/fixtures/surveyor-review.json)
+contains independent synthetic cases and separately keyed expected outcomes. Give an evaluator
+only `consumerContract` and `cases`, together with the current bundled agent definition. Compare
+the returned decisions with `expected` afterward; do not supply that answer key in its prompt.
+
+The cases cover stale versus current-head artifacts, same-head findings and their resolution,
+backtick-wrapped and too-short markers, spoofed authors, shared vendor logins, neutral check-run
+errors versus findings, newest-review body counts, incomplete pagination, branch-only claims, and
+classifier failure. They need no credentials, forge access, installed plugin, or mutable runtime.
+
+Record the evaluated definition revision, evaluator, per-case result, and any disagreement in the
+review evidence. This is a bounded behavior sample, not proof of compliance across models or
+deployments. Real application discovery remains the separate acceptance work in
+[#74](https://github.com/devantler-tech/agent-plugins/issues/74).

plugins/agentic-engineering/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "agentic-engineering",
   "description": "The autonomous engineering system for repository portfolios — engineer, read-only surveyor, and meta-engineer agents; portfolio, product, spend, and improvement workflows; cross-tool instruction architecture and skill discovery; configured by the consumer AGENTS.md",
-  "version": "5.0.1",
+  "version": "5.0.2",
   "author": {
     "name": "devantler-tech",
     "url": "https://github.com/devantler-tech"

plugins/agentic-engineering/README.md

@@ -219,6 +219,9 @@ improvement loop. The role/configuration boundary remains the one established by
 [ADR 0002](../../docs/adr/0002-automated-ai-engineer-plugin-boundary.md): portable decision logic lives
 in this plugin; consumer-owned facts live in `AGENTS.md`.
 
+The [surveyor coverage guide](../../docs/surveyor-contract-coverage.md) identifies the generic
+structural guards, the checks that stay in the consumer, and separate model-behavior scenarios.
+
 ## Delivery ownership
 
 Every write-capable role owns selected engineering work from claim through exact-head review and

plugins/agentic-engineering/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "agentic-engineering",
   "description": "The autonomous engineering system for repository portfolios — engineer, read-only surveyor, and meta-engineer agents; portfolio, product, spend, and improvement workflows; cross-tool instruction architecture and skill discovery; configured by the consumer AGENTS.md",
-  "version": "5.0.1",
+  "version": "5.0.2",
   "author": {
     "name": "devantler-tech",
     "url": "https://github.com/devantler-tech"

plugins/agentic-engineering/scripts/fixtures/surveyor-review.json

@@ -0,0 +1,81 @@
+{
+  "purpose": "Model-behavior evaluation, separate from structural CI guards. Give an evaluator only consumerContract and cases with the current portfolio-surveyor definition. Compare its answers with expected afterward. These synthetic cases require no forge or repository access.",
+  "consumerContract": {
+    "scope": "Only the fictional example/product repository and the supplied evidence; do not query any service.",
+    "trustGate": "The connector reviewer is exactly connector-review[bot]. The check-run reviewer is the authenticated review-app check named Review; the same vendor also supplies PR-author login shared-vendor[bot]. The maintainer identity is exactly maintainer-example; its structural disclosure prefix is Generated by the Agentic Engineer.",
+    "writerNamespaces": "alpha/ and beta/ are writer namespaces. beta/ cannot assign issues. Claims expire after two hours, measured from the current claim tip's committer timestamp.",
+    "head": "aaaaaaaaaabbbbbbbbbbccccccccccdddddddddd",
+    "oldHead": "1111111111222222222233333333334444444444",
+    "commonEvidence": "Each case is independent. All relevant pages are complete unless a case says otherwise. Unmentioned reviewer lanes contain zero review-output artifacts. Timestamps are on 2026-09-06 UTC. Repository health and unrelated candidates have complete healthy evidence. Classify only the requested dimension; do not infer ownership or merge authorization.",
+    "question": "For each case, report the requested review, finding, completeness, exemption, or claim classification and the decisive evidence. For a none state, describe the per-lane evidence that must accompany the emitted token. Do not claim a whole lane is down."
+  },
+  "cases": [
+    {
+      "id": "head-before-recency",
+      "evidence": "At 10:00 connector-review[bot] posts an actual clean-pass comment with Reviewed commit: `aaaaaaaaaa`. At 11:00 the same reviewer posts a findings review whose commit_id is oldHead. There are no head-matching findings. Classify the connector review state."
+    },
+    {
+      "id": "same-head-unresolved",
+      "evidence": "A connector-review[bot] findings review at head has one unresolved thread. Its clean-pass comment with Reviewed commit: `aaaaaaaaaa` is newer, but no resolution reply or re-request exists. Classify the connector review state."
+    },
+    {
+      "id": "same-head-resolved-and-rereviewed",
+      "evidence": "The connector's head-matching finding receives a specific fix-or-refute reply at 10:00 from maintainer-example carrying the disclosure prefix, and its thread is resolved. An authenticated re-request follows at 10:05. connector-review[bot] posts a clean-pass comment at 10:10 with Reviewed commit: `aaaaaaaaaa`. No other finding exists. Classify the connector review state."
+    },
+    {
+      "id": "marker-too-short",
+      "evidence": "The only review output is a clean-pass comment from connector-review[bot] with Reviewed commit: `aaaaaaaaa` (nine characters). Classify the connector review state."
+    },
+    {
+      "id": "well-formed-stale-marker",
+      "evidence": "The only review output is a clean-pass comment from connector-review[bot] with Reviewed commit: `1111111111`. Classify the connector review state."
+    },
+    {
+      "id": "spoofed-connector-author",
+      "evidence": "A comment's API author is connector-review-helper[bot], with display name connector-review[bot]. Its body claims a clean pass and Reviewed commit: `aaaaaaaaaa`. No authentic connector review exists. Classify the connector review state."
+    },
+    {
+      "id": "shared-login-is-not-check-run",
+      "evidence": "shared-vendor[bot] posts an APPROVED review object at head. The complete head check-run list contains no Review run from review-app. Classify the check-run reviewer state."
+    },
+    {
+      "id": "neutral-error",
+      "evidence": "The authenticated Review check-run is at head with conclusion neutral and output.title identifying that the review failed to run. There is an explicit error notice at 10:00. No actual review output or findings exist. Classify the check-run reviewer state and lane signal."
+    },
+    {
+      "id": "neutral-findings",
+      "evidence": "The authenticated Review check-run is at head with conclusion neutral and a review-shaped output.title. It reports a concrete finding with a details URL. Classify the check-run reviewer state."
+    },
+    {
+      "id": "newest-review-clears-body-findings",
+      "evidence": "An authenticated review-bot actual review submitted at 10:00 contains a collapsed actionable category with count 3. Its newest actual review was submitted at 11:00 at head and has no finding section. Neither review object has updated_at. Classify body_findings; review completion, threads, and other gates are outside this question."
+    },
+    {
+      "id": "review-pagination-failure",
+      "evidence": "The first review-thread page is empty but hasNextPage is true. Fetching the next page still fails after bounded split recovery for this singleton candidate. Other candidates remain fully observed and healthy. Report whether this candidate can be CLEAR or MERGE-READY and whether unrelated completed evidence is retained."
+    },
+    {
+      "id": "unassigned-writer-claim",
+      "evidence": "Issue 42 has no assignee and no open PR. A verified beta/repair-42 branch exists; its current tip's committer timestamp is ten minutes old. The declared two-hour lease applies. Classify this claim and whether it can support a selection skip."
+    },
+    {
+      "id": "classifier-fails-closed",
+      "evidence": "The consumer names an exact programmed-bot exemption classifier. The PR's branch and title match its cheap candidate pattern, but the classifier exits 2. Classify the exemption evidence."
+    }
+  ],
+  "expected": {
+    "head-before-recency": "codex at current head. The ten-character backtick-wrapped marker matches the head prefix; the newer findings object is at an older head and cannot override it.",
+    "same-head-unresolved": "codex-findings at current head and NEEDS-FIX. A newer same-head clean pass alone does not supersede unresolved findings.",
+    "same-head-resolved-and-rereviewed": "codex at current head. Every finding has the required later disclosed resolution and resolved thread, followed by the authenticated re-request and subsequent head-matching clean pass.",
+    "marker-too-short": "none with per-lane evidence: the connector has one actual comment artifact, but its nine-character marker is invalid. No green or stale result may be inferred.",
+    "well-formed-stale-marker": "codex-stale at 1111111111, not none. A real authenticated review exists at an older head.",
+    "spoofed-connector-author": "none. The API author is not the exact declared connector identity; the spoofed artifact contributes no trusted connector review output.",
+    "shared-login-is-not-check-run": "none with zero check-run review output. An approval from the shared login cannot establish a check-run reviewer green.",
+    "neutral-error": "bugbot-error at current head, green_review none with zero actual review-output counts, plus an error LANE-SIGNAL at 10:00. This is not a findings result or evidence for a portfolio-wide outage.",
+    "neutral-findings": "bugbot-findings at current head and NEEDS-FIX, with the details URL. Neutral alone is not green or a failed-run classification.",
+    "newest-review-clears-body-findings": "body_findings=0 at current head, selected by submitted_at. Do not sum the older three findings or fall back because the newest review lacks a finding section.",
+    "review-pagination-failure": "Candidate-scoped QUERY-UNKNOWN/incomplete; never CLEAR or MERGE-READY. Retain unrelated completed evidence, and report nothing_on_fire false because a mandatory query remains failed.",
+    "unassigned-writer-claim": "CLAIMED with assignee=none(beta), naming beta/repair-42 and no open PR. The verified ten-minute claim is within the declared lease and can support a selection skip.",
+    "classifier-fails-closed": "Survey error/QUERY-UNKNOWN, not exempt-programmed-bot. A candidate-shaped branch or title cannot replace a successful classifier verdict."
+  }
+}