Skip to content

feat(dashboard): expose apply lane health telemetry#732

Open
brokemac79 wants to merge 8 commits into
mainfrom
codex/csw-053-apply-lane-telemetry
Open

feat(dashboard): expose apply lane health telemetry#732
brokemac79 wants to merge 8 commits into
mainfrom
codex/csw-053-apply-lane-telemetry

Conversation

@brokemac79

@brokemac79 brokemac79 commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a compact, independently sourced Apply / close health card to the Overview. It is updated for the current bounded publication and State-writer architecture; it does not change review admission or result-publication behaviour.

Problem

After the recent ClawSweeper and clawsweeper-state recovery work, operators can see admission and publication telemetry but cannot reliably tell the difference between a currently running apply, a terminal apply, a proof-only failure before applying begins, and unavailable telemetry.

Implementation

  • Stores signed, bounded-retention apply-observability v1 records in the dashboard StatusStore and serves GET /api/apply-observability?range=6h|24h|7d.
  • Adds the Overview card with range controls, current apply state, completed throughput, failure signals, and explicit unknown values where the producer has not measured data.
  • Publishes one non-blocking in_progress observation when apply-existing actually begins, then a terminal observation after the current durable apply/state-publication flow completes.
  • Carries the lifecycle start timestamp through the artifact handoff, so an older terminal or proof-only failure cannot replace a newer active apply.
  • Limits an active observation to the six-hour apply job budget plus a 45-minute terminal-publication margin; a missing terminal then becomes unknown rather than looking active indefinitely.
  • Preserves retained pre-marker terminal records and their measured throughput, while proof-only events remain visible as failures but do not blank completed-throughput totals.
  • Includes an optional target repository only when it has a fresh observation under the same liveness rule as the API summary.

Current architecture and API load

This is dashboard-local telemetry: the Overview reads its Durable Object StatusStore and receives signed workflow posts. It adds no GitHub REST or GraphQL reads, no repository scan, and no workflow dispatch. It complements the current bounded batch publisher and State-writer model; #733 maps that changed publishing process into OpenClaw Bay's visual lanes.

Current-main review and CI repair

This branch was already based on current main 3cb93c6a16bf1209835942083ae9dd0ee8753184; the recovery PRs merged since the original work do not require a telemetry semantic change. The previously failed pnpm check was solely oxfmt on these six changed files. Commit 2794703e4aa3a2849bfdbd697dbe15c9aada61d7 applies that formatter output without changing behaviour.

Validation

  • node --test --test-concurrency=1 test/apply-observability.test.ts - 13 passing.
  • node --test --test-concurrency=1 test/dashboard-worker.test.ts - 189 passing.
  • node --test --test-concurrency=1 --test-name-pattern="^review and apply primary boundaries ignore ledger-only failures$|^apply workflow bounds checkpoints and requeues with a fresh token$" test/sweep-workflow.test.ts - 2 passing structural workflow checks.
  • pnpm exec oxfmt --check dashboard/apply-observability.ts dashboard/worker.ts scripts/publish-apply-observability.mjs test/apply-observability.test.ts test/dashboard-worker.test.ts test/sweep-workflow.test.ts - passing.
  • pnpm run build:dashboard
  • pnpm run lint:dashboard
  • pnpm run lint:scripts
  • node --check scripts/publish-apply-observability.mjs
  • git diff --check

The repository-wide pnpm run format:check reports pre-existing checkout-wide CRLF/format noise on this Windows host. The focused formatter command above covers every PR file and is the same surface that caused the hosted pnpm check failure.

The wider workflow-helper test path remains a pre-existing Windows-host limitation: its WSL relay does not provide /bin/bash. This PR does not change platform policy or work around that unrelated Linux workflow test surface.

Review closeout

  • Dirty-patch Codex review: codex review --uncommitted - clean; formatting-only, no actionable findings.
  • Branch Codex review: git fetch origin && codex review --base origin/main - clean; no actionable correctness defects relative to current main.
  • Local ClawSweeper review: pnpm run review -- --local-range --target-repo openclaw/clawsweeper --base origin/main - read-only, no GitHub mutation, keep_open / medium confidence, no discrete correctness or security finding. Reviewed current main through the prior semantic head; this final commit is formatter-only.

Risks and rollout

The card remains Awaiting producer until fresh signed telemetry arrives. Some producer fields intentionally stay unknown: the apply lane does not have a durable arrival/retry/dead-letter ledger, and terminal artifacts do not truthfully measure the live Actions queue. The workflow posts are continue-on-error, so telemetry cannot block apply/close work. Normal CI should provide one representative apply lifecycle proving both the start and terminal signed observations reach the dashboard API/health card.

@brokemac79
brokemac79 requested a review from a team as a code owner July 21, 2026 02:33
@brokemac79

Copy link
Copy Markdown
Contributor Author

@clawsweeper review

@clawsweeper

clawsweeper Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal priority bug or improvement with limited blast radius. merge-risk: 🚨 automation 🚨 Merging this PR could break CI, automerge, proof capture, label sync, or automation. labels Jul 21, 2026
@clawsweeper

clawsweeper Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed July 24, 2026, 6:45 PM ET / 22:45 UTC.

ClawSweeper review

What this changes

This PR adds signed apply-lane lifecycle telemetry, persists it in the dashboard StatusStore, exposes an apply-observability API and Overview card, and publishes non-blocking workflow observations with tests.

Merge readiness

Blocked until real behavior proof is added - 5 items remain

Keep open. The prior producer-failure finding appears to have been addressed by the follow-up commits and focused workflow coverage, but this external dashboard/workflow feature still lacks inspectable after-fix proof that a real apply lifecycle posts both start and terminal observations and that the resulting API/card state is correct. Likely related people: brokemac79 has adjacent merged apply-publishing history; steipete introduced the related apply-lane lease work. Both are medium-confidence routing candidates.

Priority: P2
Reviewed head: efb8de3c753784bdfc8324d594062a5fe91b0c41
Owner decision: Required. See Decision needed.

Review scores

Measure Result What it means
Overall readiness 🧂 unranked krab (1/6) The patch has focused validation and no confirmed current blocking code defect, but missing real lifecycle proof keeps it below merge-ready quality.
Proof confidence 🧂 unranked krab (1/6) Needs real behavior proof before merge: The PR reports tests and static validation but the supplied context contains no inspectable after-fix live apply run showing the signed start and terminal observations reaching the dashboard API or card. Please attach redacted terminal/live output, logs, recording, or dashboard evidence; redact IP addresses, tokens, and non-public endpoints. Updating the PR body should trigger re-review, or a maintainer can comment @clawsweeper re-review. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Patch quality 🦐 gold shrimp (3/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Needs proof Needs real behavior proof before merge: The PR reports tests and static validation but the supplied context contains no inspectable after-fix live apply run showing the signed start and terminal observations reaching the dashboard API or card. Please attach redacted terminal/live output, logs, recording, or dashboard evidence; redact IP addresses, tokens, and non-public endpoints. Updating the PR body should trigger re-review, or a maintainer can comment @clawsweeper re-review. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed 5 items Workflow lifecycle wiring: The PR adds apply-job outputs, an early non-blocking telemetry start publication, and terminal publication wiring so the dashboard can distinguish active from terminal apply lifecycles.
Dashboard storage and summarization: The new apply-observability module defines bounded retention, normalization, freshness rules, failure categories, and health aggregation for signed apply events.
Focused regression coverage: The branch adds dedicated observability tests and dashboard-worker coverage, including signed ingestion and API summary behavior; this is useful supplemental validation but not real after-fix workflow proof.
Findings None None.
Security None None.

How this fits together

The apply lane turns eligible review proposals into GitHub actions and durable state updates. This change adds a separate signed telemetry feed from that workflow into the dashboard’s Durable Object, where the Overview summarizes current apply health without new GitHub polling.

flowchart LR
  A[Apply workflow] --> B[Telemetry start event]
  A --> C[Terminal apply result]
  B --> D[Signed dashboard webhook]
  C --> D
  D --> E[Durable status store]
  E --> F[Apply observability API]
  F --> G[Overview health card]
Loading

Decision needed

Question Recommendation
Should ClawSweeper add this bounded apply-lane health surface to the production Overview after the contributor supplies live lifecycle proof? Sponsor bounded telemetry: Accept the dashboard-local apply health card once redacted proof confirms a real start and terminal observation are correctly represented.

Why: The patch adds a new operator-facing dashboard capability and workflow producer, so passing tests alone cannot establish whether this operational surface and its retained semantics are the desired long-term product boundary.

Before merge

  • Add real behavior proof - Needs real behavior proof before merge: The PR reports tests and static validation but the supplied context contains no inspectable after-fix live apply run showing the signed start and terminal observations reaching the dashboard API or card. Please attach redacted terminal/live output, logs, recording, or dashboard evidence; redact IP addresses, tokens, and non-public endpoints. Updating the PR body should trigger re-review, or a maintainer can comment @clawsweeper re-review. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
  • Resolve merge risk (P1) - Without a redacted real lifecycle trace, the workflow-to-dashboard handoff could appear healthy in isolated tests while a live apply run omits, rejects, or misorders its start or terminal observation.
  • Resolve merge risk (P1) - This 1,633-line, eight-file change couples dashboard interpretation to apply workflow timing and artifact handoff; the remaining uncertainty is operational semantics rather than a confirmed source defect.
  • Complete next step (P2) - A human must decide whether to adopt the new production dashboard contract, and the contributor must supply real behavior proof; there is no narrow mechanical repair to dispatch.
Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Changed surface 8 files affected; 1,633 additions The feature spans workflow publication, a signing script, durable dashboard storage, UI behavior, and regression tests.
Focused validation 3 test files changed; 13 new apply-observability tests reported The branch includes targeted unit and structural workflow coverage, though it does not replace live lifecycle proof.

Merge-risk options

Maintainer options:

  1. Prove the live lifecycle before merge (recommended)
    Attach redacted live evidence showing both the non-blocking start post and terminal post reaching the dashboard API or Overview, including the resulting active-to-terminal transition.
  2. Pause the dashboard addition
    Keep the branch open or close it if maintainers do not want to own a new production telemetry contract for the Overview.

Technical review

Best possible solution:

Keep the telemetry feed bounded and non-blocking, then merge only after a representative real apply run demonstrates signed start and terminal events reaching the API and Overview with correct freshness and throughput semantics.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this PR adds a new apply-lane observability capability rather than fixing a reported current-main failure. The relevant verification is a real apply lifecycle that emits and displays the new telemetry.

Is this the best way to solve the issue?

Unclear: the bounded, signed dashboard-local design is plausibly maintainable and avoids new GitHub reads, but it needs maintainer acceptance of the new operational surface and live proof of the complete producer-to-card path.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 3cb93c6a16bf.

Labels

Label justifications:

  • P2: This is a bounded operator-facing telemetry feature whose failure would reduce dashboard accuracy but does not alter review admission or apply results.
  • merge-risk: 🚨 other: Merging introduces a new workflow-to-dashboard lifecycle contract whose live ordering, freshness, and terminal delivery require operational proof beyond tests.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🦐 gold shrimp.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR reports tests and static validation but the supplied context contains no inspectable after-fix live apply run showing the signed start and terminal observations reaching the dashboard API or card. Please attach redacted terminal/live output, logs, recording, or dashboard evidence; redact IP addresses, tokens, and non-public endpoints. Updating the PR body should trigger re-review, or a maintainer can comment @clawsweeper re-review. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Evidence

What I checked:

Likely related people:

  • brokemac79: Authored this telemetry branch and the adjacent merged state-publication timeout change, giving them concrete recent history in the apply publishing path. (role: recent adjacent contributor; confidence: medium; commits: 279935333c52, efb8de3c7537; files: .github/workflows/sweep.yml, scripts/publish-apply-observability.mjs, dashboard/apply-observability.ts)
  • steipete: Authored the merged apply-lane lease-contention work that this PR observes, making them a relevant routing candidate for apply lifecycle semantics. (role: adjacent feature contributor; confidence: medium; commits: daa70adfb324, 3c45975e5e85; files: .github/workflows/sweep.yml, src/repair/git-publish.ts)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

  • Attach redacted evidence from one real apply lifecycle showing the start observation, terminal observation, and resulting API/card transition.
  • Obtain maintainer confirmation that this new Overview telemetry surface is the desired long-term operational boundary.

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (2 earlier review cycles)
  • reviewed 2026-07-21T02:36:10.974Z sha b306079 :: needs real behavior proof before merge. :: [P1] Emit failure kinds only for failed producer inputs
  • reviewed 2026-07-21T03:17:52.499Z sha b306079 :: needs real behavior proof before merge. :: [P1] Emit failures only when the observed step failed

@clawsweeper clawsweeper Bot added merge-risk: 🚨 other 🚨 Merging this PR has meaningful risk outside the owned taxonomy. and removed merge-risk: 🚨 automation 🚨 Merging this PR could break CI, automerge, proof capture, label sync, or automation. labels Jul 22, 2026
@brokemac79
brokemac79 force-pushed the codex/csw-053-apply-lane-telemetry branch from b306079 to d532da0 Compare July 24, 2026 22:29
@brokemac79
brokemac79 force-pushed the codex/csw-053-apply-lane-telemetry branch from d532da0 to efb8de3 Compare July 24, 2026 22:42
@clawsweeper

clawsweeper Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper status: review started.

I am starting a fresh review of this pull request: feat(dashboard): expose apply lane health telemetry This is item 1/1 in the current shard. Shard 0/1.

This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 other 🚨 Merging this PR has meaningful risk outside the owned taxonomy. P2 Normal priority bug or improvement with limited blast radius. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant