ci: bound tend job runtime with timeout-minutes

[prql-bot]

Every tend job currently runs with no timeout-minutes, so the only bound is GitHub's 360-minute default job cap. This sets a per-job cap in .config/tend.yaml at roughly 2x the slowest run each workflow has actually had, and regenerates the eight tend-*.yaml files at the pinned 0.1.18.

This repo has already paid the full six hours once: tend-nightly run 26438780303 started 2026-05-26 07:32:14Z and was killed at 13:32:38Z. It hung inside the tend action step, reports as cancelled rather than failure, and filed no tend-outage row — so nothing surfaced it at the time.

Sizing

Measured over the last 100 completed runs of each workflow (updated_at - run_started_at), excluding the wedged nightly above:

workflow	job	slowest observed	cap
tend-review	review	56m	120
tend-mention	handle	47m	120
tend-triage	triage	36m	120
tend-review-runs	review-runs	55m	120
tend-nightly	nightly	101m	180
tend-ci-fix	fix-ci	6m	60
tend-weekly	weekly	6m	60
tend-notifications	notifications	4m	60
tend-mention	relay	6s	10
tend-mention	verify	12s	10

relay and verify are the two one-step script jobs in tend-mention, measured over its last 40 runs rather than 100 (started_at → completed_at per job). They're capped because handle declares needs: verify: a wedged verify holds a mention unanswered for the full six hours without the capped job ever being reached. Neither sits behind an environment approval gate — the tend environment carries only a branch_policy rule — so no approval wait competes with the cap.

The trade-off, stated plainly

These caps sit below the harness supervisor's timeout_seconds (350m), so the job timeout — not the supervisor — becomes the operative session budget. That is the point rather than a side effect: tend-review and tend-mention's handle both use cancel-in-progress: false, so an unbounded run holds that PR or thread's queue behind it for up to six hours.

The cost is that a job-timeout kill is hard — no graceful drain, and a session killed mid-work loses anything not yet pushed or posted. The margin above is the mitigation: 10x or more for the three fast workflows, and 2x or more everywhere else except tend-nightly, whose 180 is 1.8x its 101m maximum — that one run is the sole outlier in 96, only one other exceeded 90m, and the rest sit under 65m. If a legitimate session ever gets clipped, raising a number here is a one-line change.

What this does not fix

A timeout kill remains invisible. timeout-minutes is the same knob whose 360 default produced the wedge above, and that job's conclusion was cancelled, not failure — lowering the value bounds the cost but doesn't change the conclusion, so a tend-review clipped at 120m reports exactly like a routine concurrency cancel, and the bundled skills tell sessions to do nothing on cancelled. This PR turns a six-hour silent wedge into a two-hour silent wedge; it does not make one visible. The distinguishing signal exists (a timeout kill leaves The job running on runner ... has exceeded the maximum execution time in the job log, and the job's elapsed time equals the cap exactly), but acting on it means teaching a skill to read job logs on cancelled — a separate change, not this one.

Why config rather than editing the workflows

.github/workflows/tend-*.yaml are generated and carry "Do not edit this file directly"; the nightly tend/update-workflows regen would revert a hand-edit. workflows.<name>.jobs.<job>.timeout-minutes is the supported override (docs), so the config change survives regen and the generated files here are just its output.

Regenerated with uvx tend@0.1.18 init — pinned to the version already in the header rather than @latest, so this PR carries no version bump; that stays with the nightly regen. The generator also reflowed some unrelated long expression strings in tend-mention.yaml (a YAML-dumper difference, not a config effect); I reverted those and appended only the timeout-minutes line, so the diff is 8 files × 1 line, plus the two tend-mention job lines added in the follow-up commit (uvx isn't available on the runner that added them, so those two lines were written to match the generator's placement rather than regenerated; both files parse and each cap lands on its intended job).

Verified: all eight files parse, and each cap lands on the intended job — in tend-mention on handle, verify and relay respectively, with nothing else in the file touched.

Upstream context

Raised as max-sixty/tend#962, which argues for a timeout-minutes default in the generator templates. That is a policy call for the tend maintainer and is still open; three prior PRs proposing it (max-sixty/tend#257, max-sixty/tend#474, max-sixty/tend#736) closed unmerged. This PR does not depend on that outcome — it uses the per-job override that already exists for adopters. If a generator default later lands with different numbers, these entries can be dropped.

The sibling adopter max-sixty/worktrunk had five 360-minute wedges in the 48 hours to 2026-08-20, one of which cost a PR its only review. Those hung in the pre-agent setup region, which is a shape this repo's setup composite does not share (no apt-get); the case for capping here rests on our own nightly run above, not on theirs.

[prql-bot]

Reviewing my own PR, so this is a COMMENT rather than a verdict. The config keys check out against tend's documented override schema (workflows.<name>.jobs.<job>.timeout-minutes, including the mention → handle targeting), all eight files parse, and each cap lands on the intended job. I re-measured the run durations independently and they match the table. Two things the PR leaves open:

A timeout kill is still invisible. timeout-minutes is the same knob whose 360 default produced the wedge this PR opens with — and that job's conclusion was cancelled, not failure (gh api repos/PRQL/prql/actions/runs/26438780303/jobs → nightly, cancelled, 07:32:17Z → 13:32:33Z). Setting the value lower doesn't change the conclusion, so a tend-review clipped at 120m reports exactly like a routine concurrency cancel — and the bundled skills instruct sessions to do nothing on cancelled. So this bounds the cost of a wedge from six hours to two but leaves it just as unsurfaced, which is worth being explicit about rather than reading the PR as closing that gap. The distinguishing signal exists (a timeout kill leaves The job running on runner ... has exceeded the maximum execution time in the job log, and the job's elapsed time equals the cap exactly); acting on it belongs in a follow-up, not here.

relay and verify in tend-mention keep the 360 default. They're one-step script jobs, so a hang is unlikely — but handle declares needs: verify, which means a wedged verify holds a mention unanswered for six hours without ever reaching the capped job. That's the same failure mode the PR targets, and it's two more config entries. Reasonable to argue it's out of scope for a PR aimed at the agent jobs; flagging it so the decision is deliberate rather than an oversight.

No inline suggestions: both fixes would have to touch .config/tend.yaml and the matching generated file together, and uvx isn't available on this runner, so I can't regenerate — a one-click apply on the config alone would leave the two out of sync.

I also corrected the description's "every cap is at least 2x the slowest run observed" line, which didn't hold for tend-nightly (180 against a 101m maximum is 1.8x). The number still looks right to me — that run is the sole outlier in 96 successful runs, only one other exceeded 90m — but the claim as written was wrong.

[prql-bot]

Both points actioned in 9fc8fd5d, taking the second as a fix and the first as a description change.

relay and verify are now capped at 10 minutes each. The review is right that this is the same failure mode: handle needs verify, so a wedged verify never reaches the capped job and the mention sits unanswered for the full six hours. Measured started_at → completed_at per job over tend-mention's last 40 runs, relay maxes at 6s and verify at 12s, so 10m is ~50x the observed worst case while still bounding a wedge at ten minutes. I checked the one thing that could make that cap bite a healthy run — an environment approval gate on verify, whose wait would be indistinguishable from a hang — and the tend environment carries only a branch_policy rule, no required_reviewers, so nothing waits there.

uvx still isn't on this runner, so those two lines were written to match the generator's placement rather than regenerated (the same hand-append the eight original lines got). Both files parse, and tend-mention's three jobs now read relay: 10, verify: 10, handle: 120 with nothing else in the file changed.

The invisibility gap is now stated in the PR body under "What this does not fix", in the review's own terms: this turns a six-hour silent wedge into a two-hour silent wedge, it does not make one visible. Agreed that acting on the distinguishing signal belongs elsewhere — it means teaching a skill to read job logs on cancelled, which is a change to the bundled skills rather than to this repo's config, so it doesn't ride along here.

Measurement

40 most recent completed tend-mention runs, per-job started_at → completed_at:

relay   n=40  max=0.1m  (all 40 at 0.1m)
verify  n=40  max=0.2m  (0.2, 0.1, 0.1, 0.1, 0.1, ...)

Environment rules, for the approval-gate check:

$ gh api repos/PRQL/prql/environments --jq '.environments[] | {name, protection_rules: [.protection_rules[]?.type]}'
{"name":"tend","protection_rules":["branch_policy"]}