Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1,434 changes: 1,293 additions & 141 deletions .github/workflows/bedrock-pr-review.yml

Large diffs are not rendered by default.

43 changes: 27 additions & 16 deletions .github/workflows/check-prompt-embed.yml
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
# Guards against prompt drift. The review prompt lives in one file
# (.github/codex/prompts/codex-pr-review.md) but is embedded as base64
# (PROMPT_B64) in codex-pr-review.yml so consumers on internal repos can run
# the reviewer without a cross-repo checkout. This job fails if the embedded
# copy no longer matches the file, so the two can never silently diverge.
# (PROMPT_B64) in every reviewer workflow that needs it, so consumers on
# internal repos can run the reviewer without a cross-repo checkout. This job
# fails if any embedded copy no longer matches the file, so they can never
# silently diverge.
#
# To fix a failure, regenerate the blob and paste it as PROMPT_B64:
# base64 < .github/codex/prompts/codex-pr-review.md | tr -d '\n'
Expand All @@ -14,12 +15,14 @@ on:
paths:
- .github/codex/prompts/codex-pr-review.md
- .github/workflows/codex-pr-review.yml
- .github/workflows/bedrock-pr-review.yml
- .github/workflows/check-prompt-embed.yml
push:
branches: [main]
paths:
- .github/codex/prompts/codex-pr-review.md
- .github/workflows/codex-pr-review.yml
- .github/workflows/bedrock-pr-review.yml

permissions:
contents: read
Expand All @@ -35,18 +38,26 @@ jobs:
run: |
set -euo pipefail
prompt=.github/codex/prompts/codex-pr-review.md
workflow=.github/workflows/codex-pr-review.yml

expected="$(base64 < "$prompt" | tr -d '\n')"
embedded="$(grep -oE 'PROMPT_B64: "[^"]+"' "$workflow" | sed -E 's/PROMPT_B64: "(.*)"/\1/')"
status=0

for workflow in \
.github/workflows/codex-pr-review.yml \
.github/workflows/bedrock-pr-review.yml
do
embedded="$(grep -oE 'PROMPT_B64: "[^"]+"' "$workflow" | sed -E 's/PROMPT_B64: "(.*)"/\1/')"
if [ -z "$embedded" ]; then
echo "::error::PROMPT_B64 not found in $workflow." >&2
status=1
continue
fi
if [ "$expected" != "$embedded" ]; then
echo "::error::PROMPT_B64 in $workflow is out of sync with $prompt." >&2
echo "Regenerate: base64 < $prompt | tr -d '\\n'" >&2
status=1
continue
fi
echo "$workflow: embedded prompt is in sync with $prompt."
done

if [ -z "$embedded" ]; then
echo "::error::PROMPT_B64 not found in $workflow." >&2
exit 1
fi
if [ "$expected" != "$embedded" ]; then
echo "::error::PROMPT_B64 in $workflow is out of sync with $prompt." >&2
echo "Regenerate: base64 < $prompt | tr -d '\\n'" >&2
exit 1
fi
echo "Embedded prompt is in sync with $prompt."
exit "$status"
35 changes: 35 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
name: Tests

# Validates this repo's own reusable workflows. The reconcile logic that decides
# which review comments to post, move, keep, or delete lives inside
# bedrock-pr-review.yml; tests/reconcile.test.mjs extracts that region from the
# YAML and exercises it, so the workflow file is the only copy.

on:
pull_request:
push:
branches: [main]

permissions:
contents: read

jobs:
test:
name: Workflow tests
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- uses: actions/setup-node@v4
with:
node-version: "20"

- name: Reconcile logic tests
run: node --test tests/

# Every workflow here must be loadable as YAML and every embedded script
# must parse. A raw control character or an unbalanced brace inside a
# block scalar otherwise fails at dispatch time in a consumer repo, where
# it is far more expensive to notice.
- name: Validate embedded scripts
run: python3 scripts/validate-workflows.py
131 changes: 131 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -243,6 +243,137 @@ action, and no review.

---

## `bedrock-pr-review`: AI code review through your own AWS account

The same review shape as `codex-pr-review`, but the model call goes to AWS Bedrock and
authentication is GitHub OIDC into an IAM role, so there is no API token to store or
rotate and no code leaves your AWS boundary. Advisory and non-blocking, like the Codex
reviewer.

Its review lifecycle follows the FinalWalk standard
([`digital-analytics-len/FinalWalk`](https://github.com/digital-analytics-len/FinalWalk),
ADR-004), because positional comment identity is what makes an AI reviewer noisy:

- **Findings survive edits that move them.** A comment's identity is the literal source
text of the line it sits on, hashed, not the line number. An unrelated change earlier
in the file no longer deletes and re-posts every finding below it, so an
acknowledgement or a resolution survives the next push.
- **Resolving a thread dismisses that finding for good.** Resolution is the dismissal
signal. The exact finding is suppressed on every later run, and so is a nearby
reworded restatement of it (same file, within 10 lines, claim-token similarity
≥ 0.35). Each suppression prints a `::notice::` in the run log, so nothing is dropped
silently.
- **Re-reviews converge instead of treadmilling.** A finding the model does not re-emit
is deleted as fixed only when the code it anchors on actually changed since the last
reviewed commit. On unchanged code, non-reemission is sampling noise: the code cannot
have been fixed by not changing. A restatement of a claim already open on a nearby
thread reconciles with that thread rather than posting a duplicate.
- **A thread a human replied to is never destroyed.** Not on a move, not on a rescan.
- **One summary comment, updated in place.** Instead of a new review per push. It stamps
the commit it reviewed, which is also how the next run knows what to skip.
- **Every finding is rechecked before it posts.** An adjudication pass sends each
candidate back with the surrounding source and drops the ones that source refutes
(a "missing check" whose excerpt contains the check), plus duplicates, speculation
with no named mechanism, and anything contradicting your review guide. It is
fail-open: an adjudicator error keeps the full candidate set rather than losing
findings.

### Caller

Copy [`examples/pr-review-bedrock.yml`](examples/pr-review-bedrock.yml) to
`.github/workflows/pr-review.yml` and set `aws_role_arn`. The role must allow
`bedrock:InvokeModel` / `bedrock:Converse` on the model and trust your repo via OIDC.
No secrets are needed.

Keeping the `issue_comment` and `pull_request_review_comment` triggers enables:

| Comment | Effect |
|---|---|
| `@bedrock-review` | Re-review the current HEAD, even if it was already reviewed or the PR is a draft. |
| `@bedrock-review dismiss` | Delete every finding this reviewer posted. Resolved threads are left as the permanent record. |

**Both commands require write access.** Anyone can comment on a PR, and these are
privileged: dismiss deletes findings, and on-demand spends against your Bedrock role while
deliberately bypassing the debounce and the already-reviewed gate. The reviewer checks the
comment's `author_association` and acts only for `OWNER`, `MEMBER`, or `COLLABORATOR`.
Anything else is ignored with a `::notice::` naming the association, so a denied request is
visible in the run log rather than silent.

### Make it sharper per repo

Drop a `.github/review-guide.md` with your project's actual blocker and architecture
rules. It is fed to both passes, and the adjudicator drops any finding that contradicts
it. Concrete rules that point at real defects work; style preferences do not.

```markdown
# Review guide

## Blocker rules
- No user-controlled input may reach shell execution or dynamic eval.
- Auth checks must fail closed when user, role, tenant, or scope is missing.

## Architecture rules
- Falcon aggregates and reshapes; business rules belong in TIM or ItemLib.
- API handlers call services, not repositories directly.
```

### Inputs

| Input | Required | Default | Description |
|---|---|---|---|
| `aws_role_arn` | **Yes** | n/a | IAM role assumed via GitHub OIDC. |
| `aws_region` | No | `us-east-1` | Bedrock region. |
| `model_id` | No | `openai.gpt-5.6-terra` | Finding model. `openai.*` tiers use the Responses API on bedrock-mantle; everything else uses Converse. |
| `adjudicator_model_id` | No | `us.anthropic.claude-sonnet-4-6-v1:0` | Model for the verification pass. Precision is model-insensitive once verification is on, but the judgment's recall is not: a weak judge silently drops real hedged findings, so this stays strong regardless of the finding model. |
| `adjudicate` | No | `true` | Run the verification pass. |
| `min_confidence` | No | `medium` | Drop findings below this self-reported confidence (`low`/`medium`/`high`). A finding with no confidence field counts as `medium`, so an omitted field never silently drops it. |
| `inline_min_severity` | No | `suggestion` | Minimum severity that opens an inline thread. Below the floor goes under "Minor notes". |
| `max_inline_comments` | No | `10` | Cap on inline threads per run; overflow is listed in the body. |
| `max_files` | No | `50` | Skip PRs touching more files than this; `0` disables. A PR far over the cap blows the diff budget and yields shallow findings. Counted from the PR's exact `changed_files`, so the cap holds above 100. |
| `review_guide_path` | No | `.github/review-guide.md` | Project rules, fed to both passes. A missing file is not an error. |
| `trigger_phrase` | No | `@bedrock-review` | On-demand review phrase. Must match the phrase in the caller's `if`. |
| `pr_number` | No | auto | Resolved from the triggering event; pass it only for `workflow_dispatch`. |
| `max_tokens` | No | `4096` | Max tokens per model response. |
| `debounce_seconds` | No | `30` | Collapse a burst of pushes into one review. Never applied to an on-demand request. |

Dependabot PRs are skipped: they run without access to repo or org secrets, so the role
assumption cannot succeed, and a no-op beats a red check.

**The advisory check never goes red.** Every step from the OIDC role assumption through
posting is fail-soft, so a misconfiguration degrades to "no review" instead of a red X on
the PR. The run summary names what failed. An OIDC `AccessDenied` almost always means the
repo is not yet in the reviewer role's trust policy.

### Outputs

| Output | Description |
|---|---|
| `review_url` | URL of the persistent review summary comment. |

### Tests

The logic that decides what gets posted is unit-tested. Each suite extracts the code under
test out of the workflow YAML at test time, so the workflow file is the only copy and there
is nothing to drift out of sync:

| Suite | Covers |
|---|---|
| `tests/reconcile.test.mjs` | The `PURE LOGIC` region: comment identity, dismissal and variant matching, the confidence and severity floors, sticky threads, marker ownership, thread preservation. Plus workflow-shape guards (concurrency, fail-soft, exact file count). |
| `tests/trigger.test.mjs` | The `Classify trigger` bash, including the author-association gate and that a hostile comment body cannot reach the shell. |
| `tests/adjudication.test.mjs` | The verdict applier: keep, drop, rewrite, re-grade, and fail-open on unparseable output. |

```bash
node --test tests/
python3 scripts/validate-workflows.py
```

`validate-workflows.py` loads every workflow here as YAML and parses each embedded
bash, Python, and JavaScript block. A reusable workflow only fails when a consumer
dispatches it, so a syntax error in a block scalar is otherwise invisible until it
breaks somebody else's PR.

---

## `secret-scan`: block secrets and flag PII on every PR

A CI-side gate that stops hardcoded secrets (API keys, tokens, private keys,
Expand Down
42 changes: 36 additions & 6 deletions examples/pr-review-bedrock.yml
Original file line number Diff line number Diff line change
@@ -1,17 +1,33 @@
# .github/workflows/pr-review.yml
#
# Bedrock PR review (modsy standard). Authenticates to AWS via GitHub OIDC,
# no API token to manage. Advisory, non-blocking. Uses a non-Claude model
# (Amazon Nova) by default for reviewer independence.
# Bedrock PR review (modsy standard). Authenticates to AWS via GitHub OIDC, no
# API token to manage. Advisory, non-blocking. The finding pass defaults to a
# non-Claude model (GPT-5.6 Terra) so the reviewer is independent of the Claude
# Code that authors the change; the adjudicator that rechecks each finding
# defaults to Sonnet 4.6.
#
# Copy to .github/workflows/pr-review.yml in your repo and set aws_role_arn to
# a role that allows bedrock:Converse (trusted for your repo via OIDC).
#
# The issue_comment and pull_request_review_comment triggers are what make
# on-demand review work:
#
# @bedrock-review re-review the current HEAD
# @bedrock-review dismiss delete every finding this reviewer posted
#
# Drop those two triggers if you only want automatic review on push. Keep the
# phrase in the `if` below identical to the reviewer's trigger_phrase input,
# or comments will spin up a runner only to be ignored.

name: PR Review

on:
pull_request:
types: [opened, synchronize, reopened, ready_for_review]
issue_comment:
types: [created]
pull_request_review_comment:
types: [created]

permissions:
contents: read
Expand All @@ -21,9 +37,23 @@ permissions:

jobs:
review:
if: github.event.pull_request.draft == false
# Draft PRs are skipped by the reusable workflow itself, so do not filter
# them here: `github.event.pull_request` is absent on issue_comment events,
# and a draft check at this level would swallow every on-demand request.
if: >-
github.event_name == 'pull_request' ||
(github.event_name == 'issue_comment' &&
github.event.issue.pull_request &&
contains(github.event.comment.body, '@bedrock-review')) ||
(github.event_name == 'pull_request_review_comment' &&
contains(github.event.comment.body, '@bedrock-review'))
uses: modsy/ci-workflows/.github/workflows/bedrock-pr-review.yml@v1.0.0
with:
pr_number: ${{ github.event.pull_request.number }}
aws_role_arn: arn:aws:iam::052457202381:role/github-bedrock-pr-review
# model_id: us.amazon.nova-pro-v1:0 # override if needed
# pr_number is resolved from the triggering event; pass it only for
# workflow_dispatch.
#
# model_id: us.amazon.nova-pro-v1:0 # override the finding model
# min_confidence: high # post only high-confidence findings
# adjudicate: false # skip the verification pass
# review_guide_path: .github/review-guide.md
Loading
Loading