test(agentic-engineering): guard surveyor review contracts upstream - #206
Conversation
Validation at 45cc31f:
The evaluated surveyor Git blob is |
@coderabbitai full review Please review the complete change at 45cc31f. All required CI checks pass. Focus on whether each section-scoped removal control detects its intended missing rule, whether reflow controls remain valid, and whether the 31-check ownership mapping preserves consumer-specific obligations. Independent behavior evaluation and validation are recorded here. |
|
I will check the section-scoped removal controls, whitespace-reflow controls, and the 31-check ownership mapping. I will verify that consumer-specific obligations remain consumer-owned. ✅ Action performedFull review finished. |
|
Warning Review limit reachedNext included review available in 43 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Team Run ID: 📒 Files selected for processing (8)
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Team Run ID: 📒 Files selected for processing (8)
Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review. 📜 Recent review details🧰 Additional context used📓 Path-based instructions (2)Any change to the plugin set updates both manifests in the same PR Edit both together.📄 CodeRabbit inference engine (AGENTS.md) Files:
README and manifests stay in lockstep.📄 CodeRabbit inference engine (AGENTS.md) Files:
🔇 Additional comments (8)
📝 WalkthroughWalkthroughThe change adds structural contract tests for the bundled Merge Risk: ⚪ Minimal · up to This release adds surveyor contract coverage, evaluation fixtures, and ownership documentation while preserving the shipped surveyor behavior. The version metadata is synchronized, and no merge-blocking current-head risk remains. Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 error)
✅ Passed checks (4 passed)
Full details: Docstring CoverageExplanation Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 6 functions across 1 files. (7 skipped: 7 unsupported.) Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Fixed at 1520b5b:
ShellCheck and the complete new suite pass. The evaluated role and 13 scenario inputs retain their previous bytes, so the separate model-evaluation evidence remains bound to the same definition. Fresh CI and review are required at this new head. |
@coderabbitai full review Please review the complete PR at 1520b5b. Required CI is green. The resolution addresses helper documentation and the independently reproduced reversed-section escape. All 53 clause controls, four malformed-boundary cases and the independent reproduction pass. |
🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '=== changed files against PR base ==='
git diff --name-status "$(git merge-base HEAD origin/main)" HEAD 2>/dev/null || git diff --name-status HEAD~1 HEAD
echo '=== complete diff for review ==='
git diff --find-renames "$(git merge-base HEAD origin/main)" HEAD 2>/dev/null || git diff --find-renames HEAD~1 HEAD
echo '=== script outline ==='
ast-grep outline plugins/agentic-engineering/scripts/surveyor-review-contract.test.sh --items all || true
echo '=== documentation headings and tables ==='
rg -n -C 2 '^(#|##|###)|\| O[0-9]+|R[0-9]+|ownership|consumer|generic' docs/surveyor-contract-coverage.md
echo '=== target agent section headings and requirement IDs ==='
rg -n -C 2 '^(#|##|###)|R[0-9]{2}|portfolio-surveyor|review' plugins/agentic-engineering/agents/portfolio-surveyor.agent.md
echo '=== fixture structure ==='
python3 - <<'PY'
import json
p='plugins/agentic-engineering/scripts/fixtures/surveyor-review.json'
data=json.load(open(p))
print('top-level:', ', '.join(data))
for key, value in data.items():
if isinstance(value, list):
print(f'{key}: {len(value)} entries')
PYLength of output: 71514 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '=== shell test discovery and lint workflow references ==='
rg -n -C 4 --glob '*.yml' --glob '*.yaml' --glob '*.sh' \
'(lint-scripts|scripts/.*test\.sh|find .*test\.sh|fd .*test\.sh|ShellCheck|shellcheck)' \
.github plugins README.md 2>/dev/null || true
echo '=== contract script: extraction, boundary mutations, and clause loop ==='
nl -ba plugins/agentic-engineering/scripts/surveyor-review-contract.test.sh | sed -n '15,210p'
echo '=== exact section boundary occurrences in shipped definition ==='
for marker in \
'### 0. Budget sample' \
'### 1. Open PRs' \
'### 2. Claim branches' \
'### 3. Short-circuit' \
'### 3a. Maintainer-login' \
'### 3b. Hygiene pentad' \
'### 3c. (e) Green-review' \
'**Connector lane.**' \
'**Check-run lane.**' \
'**`self@<sha>`**' \
'### 3d. Programmed-bot' \
'### 3e. Review coordination' \
'### 6. Reconcile the repo set' \
'## Return —' \
'## Survey digest —' \
'### Operate' \
'### Advance' \
'### Digest rules'
do
printf '%s\t' "$marker"
grep -Fxc "$marker" plugins/agentic-engineering/agents/portfolio-surveyor.agent.md || true
done
echo '=== duplicates of requirement literals within their assigned normalized source sections ==='
python3 - <<'PY'
import re
from pathlib import Path
script = Path("plugins/agentic-engineering/scripts/surveyor-review-contract.test.sh").read_text()
agent = Path("plugins/agentic-engineering/agents/portfolio-surveyor.agent.md").read_text().splitlines()
clauses = re.search(r"cat <<'CLAUSES'\n(.*?)\nCLAUSES", script, re.S).group(1).splitlines()
scopes = {
"budget": ("### 0. Budget sample","### 1. Open PRs"),
"claims": ("### 2. Claim branches","### 3. Short-circuit"),
"automation": ("### 3. Short-circuit","### 3a. Maintainer-login"),
"hygiene": ("### 3b. Hygiene pentad","### 3c. (e) Green-review"),
"connector": ("**Connector lane.**","**Check-run lane.**"),
"check_run": ("**Check-run lane.**","**`self@<sha>`**"),
"exemption": ("### 3d. Programmed-bot","### 3e. Review coordination"),
"pagination": ("### 6. Reconcile the repo set","## Return —"),
"reporting": ("## Return —","## Survey digest —"),
"digest_budget": ("## Survey digest —","### Operate"),
"digest_operate": ("### Operate","### Advance"),
"digest_advance": ("### Advance","### Digest rules"),
"digest_rules": ("### Digest rules",""),
}
for row in clauses:
ident, group, literal = row.split("|", 2)
start, end = scopes[group]
s = next(i for i,x in enumerate(agent) if x.startswith(start))
e = next((i for i,x in enumerate(agent[s+1:],s+1) if x.startswith(end)), len(agent)) if end else len(agent)
normalized = re.sub(r"\s+", " ", " ".join(agent[s:e]))
count = normalized.count(literal)
if count != 1:
print(f"{ident}\t{group}\tcount={count}")
PY
echo '=== consumer test lines for O01–O31 at documented revision ==='
gh api repos/devantler-tech/monorepo/contents/.claude/scripts/portfolio-surveyor.test.sh \
-f ref=f8fe925dd1c8c623606af7d8e5bdd554b6a46f37 \
--jq .content \
| base64 -d \
| nl -ba \
| sed -n '60,200p;760,790p'Length of output: 22296 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '=== lint-scripts bundled test execution ==='
sed -n '115,155p' .github/workflows/ci.yaml
echo '=== contract script source with line numbers ==='
awk '{ printf "%5d %s\n", NR, $0 }' \
plugins/agentic-engineering/scripts/surveyor-review-contract.test.sh \
| sed -n '15,210p'
echo '=== exact section boundary occurrence counts ==='
for marker in \
'### 0. Budget sample' '### 1. Open PRs' \
'### 2. Claim branches' '### 3. Short-circuit' \
'### 3a. Maintainer-login' '### 3b. Hygiene pentad' \
'### 3c. (e) Green-review' '**Connector lane.**' \
'**Check-run lane.**' '**`self@<sha>`**' \
'### 3d. Programmed-bot' '### 3e. Review coordination' \
'### 6. Reconcile the repo set' '## Return —' \
'## Survey digest —' '### Operate' '### Advance' '### Digest rules'
do
count=$(grep -Fxc "$marker" plugins/agentic-engineering/agents/portfolio-surveyor.agent.md || :)
printf '%s => %s\n' "$marker" "${count:-0}"
done
echo '=== requirement literals whose assigned source section does not contain exactly one copy ==='
python3 - <<'PY'
import re
from pathlib import Path
script = Path("plugins/agentic-engineering/scripts/surveyor-review-contract.test.sh").read_text()
agent = Path("plugins/agentic-engineering/agents/portfolio-surveyor.agent.md").read_text().splitlines()
clauses = re.search(r"cat <<'CLAUSES'\n(.*?)\nCLAUSES", script, re.S).group(1).splitlines()
scopes = {
"budget": ("### 0. Budget sample","### 1. Open PRs"), "claims": ("### 2. Claim branches","### 3. Short-circuit"),
"automation": ("### 3. Short-circuit","### 3a. Maintainer-login"), "hygiene": ("### 3b. Hygiene pentad","### 3c. (e) Green-review"),
"connector": ("**Connector lane.**","**Check-run lane.**"), "check_run": ("**Check-run lane.**","**`self@<sha>`**"),
"exemption": ("### 3d. Programmed-bot","### 3e. Review coordination"), "pagination": ("### 6. Reconcile the repo set","## Return —"),
"reporting": ("## Return —","## Survey digest —"), "digest_budget": ("## Survey digest —","### Operate"),
"digest_operate": ("### Operate","### Advance"), "digest_advance": ("### Advance","### Digest rules"),
"digest_rules": ("### Digest rules",""),
}
problems = []
for row in clauses:
ident, group, literal = row.split("|", 2)
start, end = scopes[group]
start_at = [i for i,line in enumerate(agent) if line.startswith(start)]
end_at = [i for i,line in enumerate(agent) if end and line.startswith(end)]
if len(start_at) != 1 or (end and len(end_at) != 1) or (end and end_at[0] <= start_at[0]):
problems.append(f"{ident}: boundary count/order invalid")
continue
scoped = agent[start_at[0]:(end_at[0] if end else len(agent))]
text = re.sub(r"\s+", " ", " ".join(scoped))
if text.count(literal) != 1:
problems.append(f"{ident}: literal count={text.count(literal)}")
print("\n".join(problems) if problems else "all 53 requirement literals occur exactly once in their assigned section")
PY
echo '=== consumer test lines from cited revision ==='
tmp=$(mktemp)
trap 'rm -f "$tmp"' EXIT
gh api 'repos/devantler-tech/monorepo/contents/.claude/scripts/portfolio-surveyor.test.sh?ref=f8fe925dd1c8c623606af7d8e5bdd554b6a46f37' \
--jq .content | base64 -d > "$tmp"
awk 'NR >= 60 && NR <= 200 { printf "%5d %s\n", NR, $0 }
NR >= 760 && NR <= 790 { printf "%5d %s\n", NR, $0 }' "$tmp"Length of output: 31257
I found no blocking issues. The revised extractor requires unique, ordered boundaries. The malformed-boundary controls reject the previous reversed-boundary scope escape and its out-of-scope R03 rescue case. Each of the 53 requirement literals occurs exactly once in its assigned operative section. Each removal control checks the expected missing ID. Each reflow control accepts normalized whitespace-only content. The The O01–O31 mapping preserves consumer ownership for deployment-specific paths, identities, writer namespaces, lease policy, compatibility-overlay loading, provider wiring, and product-specific tests.
|
@codex review Please review the complete current diff at All 41 checks have passed. The final boundary repair, its RED-to-GREEN evidence, and the helper documentation resolution are recorded at #206 (comment). The 53 structural clauses and independent 13-case evaluation are covered by this PR; the role definition itself is unchanged. |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex Review: Didn't find any major issues. Breezy! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Readiness verified at I exercised the documented suite as its CI user: all 53 clause removals were rejected, all 53 whitespace reflows were accepted, and all four malformed-boundary cases plus empty input were rejected. Manifest validation, the base-to-head version gate, desired-state digest verification, ShellCheck, and whitespace checks pass. The historical O01–O31 mapping was checked against its cited consumer revision. An independent full-diff review found no actionable defect or weakened control; all four manifests consistently select 5.0.2. The authenticated Codex verdict names this head and reports no major issues. All 40 hosted checks are terminal (38 successful, two intentional skips), the additional status is successful, and there are no unresolved threads, current-head findings, maintainer prohibitions, or active foreign work. The latest foreign push was at 18:10:10 UTC and its activity window has expired. The suite measures structural drift; the separately recorded 13-case model evaluation remains a bounded sample. This change does not remove the consumer overlay or claim runtime enforcement. Promoting this reviewed head under the normal merge gate. |
1 participant
The shared surveyor carries review and pagination rules that were guarded only in its consuming monorepo. This adds an upstream CI suite for 53 operative clauses, including head matching before recency, abbreviated markers, same-head findings, check-run identity, newest-review counting, and incomplete queries.
Every clause has a removal control and a whitespace-reflow control. A copy elsewhere in the document cannot rescue a missing operative rule, and missing, repeated or reversed section boundaries are rejected. The coverage guide maps the original 31 checks and identifies the concrete identities, paths, policies, and three overlay-loading checks that remain consumer-owned. It also provides 13 independent scenarios for separate model evaluation.
Agentic-engineering moves to 5.0.2 because the bundle gains test resources. Agent definitions, synced skills, and desired-state resources retain their existing bytes. These are structural drift guards, not proof of model compliance; consumer overlay removal still requires its own parity review.
Validation: all existing helper, manifest, version and digest suites pass. The new suite proves all 53 removal/reflow pairs, four malformed-boundary cases and empty input; an independent evaluator matched all 13 scenario outcomes without receiving the answer key. Exact evidence and limits, boundary-fix evidence.
Fixes #95