Add PR 674 review reliability observability

[hxy91819]

Summary

Adds the review-reliability observation layer that must land before #674.

This PR intentionally starts from main and does not copy PR 674's generation-bound queue protocol or producer implementation. It establishes the backward-compatible storage, observer, aggregate API, dashboard card, health rules, and rollout contract that PR 674 can produce into after rebasing.

What ships

• Extends durable per-item review telemetry with optional generation/operation identity, trigger attribution, terminal reasons/times, indexed outcomes, and phase durations while retaining v1 compatibility and full JSON records.
• Adds immutable run-level workflow telemetry plus a read-only workflow_run observer for completed ClawSweeper waves, including plan/item/publication job evidence and conservative terminal reconciliation.
• Adds a lease-aware watchdog for stale refreshing attempts. It lists orphans and rolls them into health; it only terminalizes when the workflow is confirmed terminal and no valid lease remains.
• Adds GET /api/review-observability?range=6h|24h|7d&repo=all|owner/repo with bounded outcome, coverage, percentile, lane, recovery, and anomaly aggregation.
• Adds the Review reliability dashboard card between Work execution and Exact Review, with hero roll-up through review_execution_degraded / review_execution_critical.
• Documents green/amber/red thresholds, passive rollout, canary checks, PR 674 integration, and non-disruptive rollback in docs/pr-674-observation-runbook.md.

Passive rollout and PR 674 integration

Before v2 producers are enabled, the card deliberately displays Awaiting v2 producers; it does not claim green health from absent data.

After this PR lands, #674 should:

1. Rebase onto the resulting main.
2. Connect its exact-event and per-item matrix jobs to the durable producer endpoints at authoritative claim, queue/claim/review/publication transitions, and one immutable terminal outcome.
3. Supply generation and operation IDs plus the appropriate trigger_lane, trigger_origin, and terminal reason (superseded/generation_superseded versus unexpected cancelled/workflow_cancelled).
4. Keep telemetry writes warning-only so observability cannot block review execution.
5. Enable required enforcement with a 30-minute warm-up, then run the two-generation proposal-only canary and observe at least one hot-intake and normal-backfill cycle.

The observer remains independent of producer finalizers so plan failures, whole-workflow cancellation, and missing item telemetry remain visible. Normal backfill is evaluated separately and cannot be masked by exact-event or hot-intake success.

Validation

• Node v24.15.0
• pnpm run check
• Focused dashboard/worker/telemetry/workflow tests (180 passing before the final full check)
• Autoreview via /data/code/openclaw/openclaw/.agents/skills/autoreview/scripts/autoreview --mode local --stream-engine-output
  • fixed the exact-1000-job pagination boundary
  • pinned privileged workflow_run checkout to the trusted default branch with persisted credentials disabled
  • rejected the stale actions/checkout@v7 availability claim because this repository explicitly requires v7 and its checkout guard test passes

Safety

This change does not pause the live sweep, run apply/close, mutate foreign pull requests, or enable PR 674's producer enforcement.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed July 19, 2026, 3:20 AM ET / 07:20 UTC.

Summary
The branch adds passive durable review telemetry, a completed-workflow observer, observability aggregation API, dashboard reliability card, rollout documentation, and focused tests for later integration with per-item review sharding.

Reproducibility: not applicable. This PR proposes new observability and rollout capability rather than fixing a reported current-main bug. The supplied tests and validation describe expected behavior, but no bug reproduction is the decision basis.

Review metrics: 2 noteworthy metrics.

• Patch surface: 16 files affected; 2,540 added, 30 removed. The change spans durable storage, Worker routing, dashboard rendering, a GitHub Actions workflow, documentation, and tests, so it needs cross-surface operational review.
• Automation surface: 1 new workflow_run workflow. The observer executes after completed ClawSweeper runs and becomes part of production review telemetry.

Merge readiness
Overall: 🦐 gold shrimp
Proof: 🌊 off-meta tidepool
Patch quality: 🦐 gold shrimp
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

• Obtain explicit maintainer sponsorship for the durable telemetry and observer direction.
• [P2] Rebase onto current main and run the documented isolated two-generation canary before enabling required enforcement.

Risk before merge

• [P1] Merging creates a new workflow_run-triggered telemetry path; incorrect run classification or terminal reconciliation could distort reliability health and operator decisions even though it is passive initially.
• [P1] The durable schema and observer are designed around later producer integration in Review each selected PR in its own item shard #674, so maintainers need to own the cross-PR rollout, warm-up, and canary contract before enabling enforcement.

Maintainer options:

1. Approve a passive canary first (recommended)
   Retain warning-only telemetry, rebase on current main, and verify observer records and dashboard health through an isolated canary before authorizing producer enforcement.
2. Pause for rollout ownership
   Hold the PR until a maintainer explicitly approves the durable telemetry and PR-674 integration contract rather than merging an independently evolving foundation.

Next step before merge

• [P2] A maintainer must sponsor the durable telemetry, workflow_run observer, and coupled rollout direction; there is no narrow mechanical repair to dispatch.

Maintainer decision needed

• Question: Should ClawSweeper adopt this durable review-observability schema and workflow_run observer as the supported rollout foundation for the per-item review-sharding work in Review each selected PR in its own item shard #674?
• Rationale: The patch is a new operational contract rather than a narrow established-behavior repair; the correct retention, rollout, and enforcement relationship to the coupled sharding work depends on maintainer ownership of the product and operational direction.
• Likely owner: hxy91819 — They hold the clearest available implementation context across this PR and the explicitly coupled review-sharding proposal.
• Options:
  • Sponsor passive foundation (recommended): Merge only after maintainer approval of the passive schema and require a fresh-main canary before any producer enforcement is enabled.
  • Keep direction together: Pause this PR until the maintainers choose the final producer and enforcement contract for the related sharding work, then land a narrower coordinated change.

Security
Cleared: The hydrated workflow diff uses a default-branch checkout with persisted credentials disabled and read-only declared permissions; no concrete security or supply-chain regression is evident in the proposed change.

Review details

Best possible solution:

Adopt the passive telemetry and observer only if maintainers sponsor it as the supported foundation for the item-shard rollout, then validate it against a fresh main rebase and an isolated two-generation canary before enabling required enforcement.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this PR proposes new observability and rollout capability rather than fixing a reported current-main bug. The supplied tests and validation describe expected behavior, but no bug reproduction is the decision basis.

Is this the best way to solve the issue?

Unclear: the passive, warning-only rollout is a thoughtful safety direction, but whether this durable schema and observer should become the platform contract for the related sharding work requires maintainer approval before merge.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 8bc5e3a0de48.

Label changes

Label changes:

• add P2: This is a substantial but passive operational improvement whose rollout affects review reliability rather than an active user-facing outage.
• add merge-risk: 🚨 automation: The PR adds a production workflow_run observer and telemetry path whose classification or reconciliation behavior can affect automation health reporting after merge.
• add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🌊 off-meta tidepool and patch quality is 🦐 gold shrimp.
• add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: This repository-member PR is exempt from the external-contributor real-behavior-proof gate; the provided validation is supplemental, while a maintainer-owned isolated workflow canary remains the relevant operational proof before enforcement.

Label justifications:

• P2: This is a substantial but passive operational improvement whose rollout affects review reliability rather than an active user-facing outage.
• merge-risk: 🚨 automation: The PR adds a production workflow_run observer and telemetry path whose classification or reconciliation behavior can affect automation health reporting after merge.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🌊 off-meta tidepool and patch quality is 🦐 gold shrimp.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: This repository-member PR is exempt from the external-contributor real-behavior-proof gate; the provided validation is supplemental, while a maintainer-owned isolated workflow canary remains the relevant operational proof before enforcement.

Evidence reviewed

What I checked:

• Passive observability contract: The PR adds run-level telemetry ingestion and a read-only observability endpoint, while the dashboard configuration keeps required enforcement disabled during the initial rollout. (dashboard/exact-review-queue.ts:860, 775fe49b2b51)
• Workflow safety boundary: The new workflow reacts to completed ClawSweeper runs, uses read-only declared permissions, checks out the repository default branch, and disables persisted checkout credentials before invoking the observer. (.github/workflows/review-reliability-observer.yml:1, 775fe49b2b51)
• Coupled rollout remains open: The PR body and related-item context describe this as the observability foundation for Review each selected PR in its own item shard #674, which remains an open draft and is not a safe replacement or completed implementation path. (docs/pr-674-observation-runbook.md:17, 775fe49b2b51)
• Branch provenance: The current proposed implementation is a single commit, “Add review reliability observability,” at the reviewed head; it has not been established as present on current main or in release v0.3.0. (775fe49b2b51)

Likely related people:

• hxy91819: Authored this observability proposal and the coupled per-item sharding proposal, so has the strongest available context for the intended producer/observer rollout contract. (role: related reliability-work contributor; confidence: medium; commits: 775fe49b2b51; files: dashboard/exact-review-queue.ts, dashboard/review-observability.ts, scripts/review-run-observer.mjs)

What the crustacean ranks mean

• 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
• 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
• 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
• 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
• 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
• 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
• 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (1 earlier review cycle)

• reviewed 2026-07-19T07:13:39.845Z sha 775fe49 :: needs maintainer review before merge. :: none