ops: daily orchestrator summary 2026-07-28-run2 - #2149
Merged
Conversation
The 'Bundle budget into daily summary' step in .github/workflows/orchestrator.yml only bundles a run's budget JSON when an OPEN ops/daily-<date>* PR exists. When a run writes no summary PR of its own -- because the orchestrator exited before Step 8, or because the prior run's summary PR had already merged -- the step falls back to pushing a standalone ops/budget-<date>-<run_id> branch and explicitly does NOT open a PR for it. Nothing has ever merged those branches. 31 such branches have accumulated since 2026-07-14, holding $283.31 of measured spend that never reached main. Every daily-total computed from dev/budget/*.json since then has therefore been understated -- including the [high] budget escalation in dev/daily/2026-07-28.md, whose 2026-07-27 figure of $405.30 is actually $463.74 (232% of the $200 cap, not 203%). This commit lands all 31 records verbatim from their fallback branches. No record already present on main was modified.
…port Two defects in .github/workflows/orchestrator.yml found while reconciling today's runs, filed against the orchestrator-automation track (their correct home -- they are workflow defects, not harness tooling, and harness.md is being edited concurrently by PR #2148). A-BUDGET-ORPHAN is the mechanism behind the 31-record backfill in the previous commit. A-SUMMARY-STALE-FALLBACK explains why two runs today exited green having written no summary at all.
…C audit record
dayfine
force-pushed
the
ops/daily-2026-07-28-run2
branch
from
July 28, 2026 17:18
8eaece7 to
1d66809
Compare
Measured rather than assumed: both silent runs exited voluntarily (is_error false, 33 and 51 turns of a 200 cap), so the wall-clock hypothesis was wrong. Both did log ShellError exit 128 ~10s in -- git's dubious-ownership refusal, which this run also hit on its first git call and which leaves Step 0.5/1b/5.5/8 inoperable until an agent fixes it by hand.
dayfine
added a commit
that referenced
this pull request
Jul 28, 2026
Automated daily orchestrator run 3 for 2026-07-28 (GHA run [30393663268](https://github.com/dayfine/trading/actions/runs/30393663268)). Full summary: `dev/daily/2026-07-28-run3.md`. **Three PRs merged this run:** #2149 (`f44e5bb2`), #2152 (`14270b57`), #2148 (`751de67d`). **The finding worth reading:** run 2 (`30380136239`) is the **first orchestrator run observed with `conclusion: failure`**, and nothing surfaced it. Its Step 9 bundling step hit `fatal: 'ops/daily-2026-07-28-run2' is already checked out at '/tmp/ops-run2'` (exit 128) — because run 2's own Step 8 worktree was still holding the branch. Critically, the step's `git checkout` had **already removed** the generated budget record before the fatal, so run 2's `$36.82` reached neither its PR nor the `ops/budget-*` orphan fallback: it was destroyed, not stranded. Reconstructed here from the job log. Step 10 (`Fail on escalations`) is `skipped` when step 9 fails, so the escalation gate never ran either. This is systemic rather than a slip: Step 4's GHA guidance (`git checkout -b` in the shared tree) is wrong, so every recent run overrides it with `git worktree add` — while the bundling step was written against the checkout-based assumption it replaced. Filed as `A-WORKTREE-BLOCKS-BUNDLE` in `dev/status/orchestrator-automation.md` with three fix options. Interim mitigation applied by this run: no worktree left on the summary branch. Contents: the run-3 summary, two audit records, two `dev/reviews/*` combined results (written by the orchestrator, not the QC agents, per the PR-D'c migration), the fast health report, the reconstructed budget record, and the Step 5.5 index reconcile. 🤖 Generated by the GHA orchestrator. --------- Co-authored-by: claude-orchestrator <noreply@github.com> Co-authored-by: GitHub Actions Bot <bot@github-actions>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated daily orchestrator run — 2026-07-28 run 2 (GHA run 30380136239).
Full summary:
dev/daily/2026-07-28-run2.md.What happened
790d23a0) — re-QC after rework iteration 1. Both gatesAPPROVED q4 at the exact tip; CI re-verified immediately before merge.
consecutive_rework_count2 → 0.dev/status/orchestrator-automation.md.The budget records were not a bookkeeping problem
.github/workflows/orchestrator.yml:423-453bundles a run's budget JSON onto theops/daily-*PR only if such a PR is currently open. Otherwise it pushes to astandalone
ops/budget-<date>-<run_id>branch and, by design (comment line 384),opens no PR. Nothing ever merged them.
Consequence: every daily total computed from
dev/budget/*.jsonon main has beenunderstated — including run 1's
[high]budget escalation, which was raisedthis morning off the incomplete set.
Records are copied verbatim from their fallback branches; nothing already on main
was modified.
Three defects filed
A-BUDGET-ORPHAN— the mechanism above. Cheapest fix is onecurl -X POST /pullsbeside the existing fallback push.A-SUMMARY-STALE-FALLBACK— two runs today (30353290609,30366079322)exited
successhaving written no summary.publish-summarythen silentlyselected run 1's already-merged file, and the
Fail on escalationsgategraded them against it. The gate's guarantee is void in exactly the
circumstance it exists for.
A-GIT-SAFE-DIRECTORY—gitis unusable at the start of every run(
--user 0+HOME: /home/opam⇒detected dubious ownership, exit 128 onevery command) until an agent fixes it by hand. Both silent runs logged
ShellError: Failed with exit code 128~10s after start.Correction I made to my own reasoning
My first root cause for the silent runs was wall-clock exhaustion in Step 6.5's
CI poll. The logs disagree: both reported
"is_error": falsewithnum_turns33 and 51 against a
--max-turns 200cap. They ended voluntarily. Thesummary records the wrong hypothesis alongside the corrected one.
🤖 Dispatched by GHA orchestrator run 30380136239