Skip to content

[Bug]: Since #23522, a PR watch never wakes when checks pass between the watch call and its first read #36463

Closed as not planned

Description

@gperezmz

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

  1. Open a PR on GitHub whose checks take less than about 2 minutes (one required check, finishing in under a minute).
  2. While the checks are still pending, have the agent call watch_pull_request on it and end its turn, as the PR instructions tell it to.
  3. Let the checks pass before the next PullRequestWatchReactor sweep (it runs every 2 minutes).
  4. Wait.

Expected behavior

The thread wakes with "All 1 required check passed". The checks were pending when the watch started, so them passing is news.

Actual behavior

The thread never wakes. The first read after the watch starts sees the green checks, saves them as the baseline (passed: true), and sends no notification. Later reads find nothing new. The thread stays idle until the user steps in or the PR is merged by hand.

Timeline from one occurrence (UTC):

Time Event
23:59:44 watch_pull_request is called. The watch is created with headSha: null, passed: false, awaitingBaseline: "all". The checks are pending.
00:00:22 The last check passes.
00:00:33 First sweep reads the PR: saves passed: true, passedChecks: ["<required-check>"], awaitingBaseline is cleared, no wake.
00:02:35 Next sweep: no change, no wake.
00:04:37 The watch ends with wakes: 0, reads: 1 (the user merged by hand).

Cause

#23522 (fixing #22184) baselines the first read silently, so checks that were already done when the watch started aren't reported. But the baseline is taken at the first sweep, up to 2 minutes after the watch was created, not when watch_pull_request is called. In evaluatePullRequestWatch, a new watch has headSha: null, so headMoved is false and checksBaseline is true on that first read. That suppresses checks-passed even when the checks went from pending to passed after the watch started:

const headMoved = watch.headSha !== null && headSha !== watch.headSha;
const checksBaseline = watch.awaitingBaseline !== void 0 && !headMoved;
...
if (passedNow && (!passed || gateGrew)) {
  if (!checksBaseline) changes.push({ kind: "checks-passed", ... });

The tests added in #23522 cover checks that were green before the watch started. They don't cover checks that finish between the watch call and the first read. The same window also hides other changes that happen in that gap: a check that fails (checks-failed is behind the same !checksBaseline), a check that starts waiting for approval (checks-awaiting-approval, same guard), and a new conflict (awaitingBaseline === "all" suppresses conflicting). A failure is the worst of these, because the agent is never told CI broke.

How often you hit it depends on CI speed: any repo whose checks finish within about 2 minutes of the watch call can lose the wake.

Possible fixes

  • Take the baseline when the watch is created: read the PR in watch_pull_request (or right after it) and save its head and checks then.
  • Or, on the baseline read, only treat checks as already known if they finished before watch.startedAt (check runs have completed_at). Anything that finished after it is still news.

Impact

Major degradation or frequent failure

Version or commit

0.1.10-nightly.20261113.4656

Environment

Linux (Debian 13), T3 Code server, Claude provider (Opus 5.5), GitHub PR with a required check enforced by a repository ruleset.

Logs or stack traces

# saved watch state, from thread.pull-request-synced / thread.pull-request-link-updated events
23:59:44.611Z {"headSha":null,"passed":false,"passedChecks":[],"awaitingBaseline":"all","wakes":0}
00:00:35.732Z {"headSha":"<sha>","passed":true,"passedChecks":["<required-check>"],"owedChecks":[],"wakes":0,"checksPending":false}

# server trace
00:04:37 PullRequestWatchReactor.reportEnd "pull request watch ended"
  {"reason":"stopped","minutes":5,"quietMinutes":5,"wakes":0,"reads":1,"partial":false}

Workaround

None inside T3. The agent (or the user) has to check the PR by hand. #33662, if merged, would eventually rescue a stuck watch after 60 minutes.

Related

#22184, #23522, #30530 (same symptom, the watch never wakes, from a different trigger), #28183

Activity

  1. juliusmarminge commented on Nov 14, 2026

    @juliusmarminge
    Member

    I tried to reproduce this on the current build and could not.

    Commit 4bab2f3 (origin/main). Not seen in a live run: watch_pull_request is reachable only through a provider session's MCP token, and a live repro would mean timing a real GitHub PR's CI to finish inside the up-to-2-minute gap before the first PullRequestWatchReactor sweep. Instead I reproduced it through the real server path with a mocked host, in apps/server/src/orchestration-v2/runtimeLayer.test.ts. Real orchestrator and effect worker; PullRequestWatchService.setWatching (what the watch_pull_request MCP tool calls) is called while the host summary and detail report the required check 'test' as pending on head ad70160aaa; the check then finishes (success, or failure) before the reactor's first sweep; then reactor.sweep and worker.drain. Expected one notification ('#12: checks passed' or '#12: checks failed'); got none in both cases (expected [] to deeply equal ['#12: checks passed'], and the same for failed). Harness check: when the first sweep still sees the check pending and it finishes before a second sweep, both cases wake as expected, so the empty result comes from the first-read baseline. The code matches the reporter's analysis: Orchestrator.ts creates the watch with headSha null and awaitingBaseline 'all'; setWatching does a fresh read at watch time (summary with bypassCache, which includes headSha and checksState) only to refuse merged PRs and throws the state away; evaluatePullRequestWatch then has headMoved false and checksBaseline true on the first read, which hides checks-passed, checks-failed and checks-awaiting-approval, and awaitingBaseline 'all' hides conflicting.

    Closing as not planned. If it still happens for you, reply with the exact steps and the version you are on and it will be looked at again.

  2. keeperxy commented on Nov 15, 2026

    @keeperxy
    Contributor

    Note

    🤖 Claude Opus 5.5 responding on behalf of @keeperxy

    @juliusmarminge This still happens, in a live run against real GitHub CI, twice in one thread. You asked for steps and a version, so here they are, plus the evidence from the server.

    Version: 0.1.10-nightly.20261114.4709 (built from dfd2187ccb, published 2026-11-14T23:19Z). That build contains 4bab2f3ae7, the commit from your no-repro comment, and the server ran it for the whole session (process started 2026-11-14T23:27Z). The baseline logic is unchanged on current main (32c069eed1). The only later commits to these files change comment-text quoting and GitLab rebase wakes.

    Setup: self-hosted T3 Code server on Linux, Codex provider. A private GitHub repo with GitHub Actions checks (unit-and-matrix, e2e) plus Vercel. No required checks are configured, so the gate is every check. e2e takes about 1m40s.

    Steps:

    1. The agent opens a PR. CI starts.
    2. While e2e is still in_progress, the agent calls watch_pull_request. It returns watching: true, wasWatching: false, and the agent ends its turn, as the PR instructions say.
    3. e2e passes a few seconds later, before the first PullRequestWatchReactor sweep.
    4. Nothing happens. The thread stays idle until the user asks for status, and the PR is merged by hand.

    Timeline (UTC)

    Check times come from the GitHub check-runs API. Watch times come from server.trace.ndjson (orchestrationV2.dispatch.withReceipt spans for thread.pull-request.watch and thread.pull-request-watch.sync).

    PR A

    Time Event
    04:54:24 unit-and-matrix completed, success
    04:54:27 e2e started
    04:56:03.7 watch_pull_request → thread.pull-request.watch (e2e still in progress)
    04:56:08 e2e completed, success. Every check is now green
    04:56:12 the agent's turn ends
    04:57:41.7 first thread.pull-request-watch.sync: baseline saved as passed: true, no wake
    07:08:10 the user asks for status (2h12m idle)
    07:10:28 watch ends: reason: "merged", wakes: 0, reads: 5

    PR B (same thread, about 2 hours later)

    Time Event
    07:12:16 unit-and-matrix completed, success
    07:12:19 e2e started
    07:13:50.7 watch_pull_request → thread.pull-request.watch (e2e still in progress)
    07:14:00 e2e completed, success
    07:14:02 the agent's turn ends
    07:14:29.9 first thread.pull-request-watch.sync: baseline saved as passed: true, no wake
    07:52:47 the user asks why nothing happens (38m idle)
    07:54:42 watch ends: reason: "merged", wakes: 0, reads: 2

    In both cases the checks went green 5s and 10s after the watch call, and the first read came 98s and 39s after it. That is the gap your runtimeLayer.test.ts run hit.

    Server log (PullRequestWatchReactor.reportEnd)

    {"message":"pull request watch ended","pullRequest":"github.com/<owner>/<repo>#A","reason":"merged","minutes":134,"quietMinutes":134,"longestQuietMinutes":134,"wakes":0,"reads":5,"partial":false}
    {"message":"pull request watch ended","pullRequest":"github.com/<owner>/<repo>#B","reason":"merged","minutes":41,"quietMinutes":41,"longestQuietMinutes":41,"wakes":0,"reads":2,"partial":false}

    Final saved watch state (endedWatch in the thread projection)

    // PR A
    {"startedAt":"2026-11-15T04:56:03.755Z","headSha":"<sha A>","failedChecks":[],"passed":true,
     "passedChecks":["unit-and-matrix","e2e","Vercel","Vercel Preview Comments"],"owedChecks":[],
     "awaitingApproval":[],"approvalAnnounced":false,"conflicting":false,"mergeQueueRemovalId":null,
     "wakes":0,"checksAwaitedSince":null,"checksPending":false}
    // PR B
    {"startedAt":"2026-11-15T07:13:50.672Z","headSha":"<sha B>","failedChecks":[],"passed":true,
     "passedChecks":["Deploy Convex Preview","unit-and-matrix","e2e","Vercel"],"owedChecks":[],
     "awaitingApproval":[],"approvalAnnounced":false,"conflicting":false,"mergeQueueRemovalId":null,
     "wakes":0,"checksAwaitedSince":null,"checksPending":false}

    passed: true with wakes: 0 and no awaitingBaseline: the baseline took in the pending → passed transition.

    Why (confirmed against current main)

    • PullRequestWatchService.ts#L126: setWatching reads a fresh summary (bypassCache: true, with head and checksState) at watch time. It uses that read only to reject merged PRs and then drops it.
    • The watch is created with headSha: null, awaitingBaseline: "all". The first read happens only at the next sweep (SWEEP_MINUTES = 2). A new watch has no change feed until that sweep adds it to currentGroups.
    • On that first read, checksBaseline is true, so checks-passed, checks-failed and checks-awaiting-approval are suppressed, and awaitingBaseline === "all" also hides conflicting.

    So any CI that finishes within about 2 minutes of the watch call loses its wake. A failure is lost the same way, so the agent never learns that CI broke.

    Your own run in the closing comment shows the same result (expected [] to deeply equal ['#12: checks passed'], and the same for failed). The factory:no-repro close looks like a misfire. Could you reopen this?

    Either fix from the issue would cover both of our cases:

    1. Save the head and checks state from the read setWatching already makes as the baseline. Then "pending at watch time" is known, and the first sweep reports the transition.
    2. Or, on the baseline read, treat only checks with completed_at < watch.startedAt as already known.

    If you need anything else, such as the full trace spans or a test against a specific commit, reply here. We are watching this issue.

  3. imqdee commented on Nov 16, 2026

    @imqdee
    Contributor

    This still happens on 0.1.10-nightly.20261115.4723 (built from 11c425b4b6b35dce4510d40028a4aa8f37a3879e), in live runs against real GitHub Actions. Two occurrences within five minutes on 2026-11-16, in two threads on two repositories, plus two watches in the same hour that woke correctly because their first read happened to land before the checks finished.

    Setup: T3 Code nightly server on macOS, Claude provider, GitHub. No host event feed (the trace has no PullRequestWatchReactor.readOnChange span), so the 2-minute sweep is the only reader. Neither repository has required checks, so the gate is every check. CI takes 40 to 100 seconds.

    Steps: the agent opens a PR, calls watch_pull_request while the checks run (watching: true, wasWatching: false), and ends its turn. The checks pass before the next sweep. No wake ever comes.

    Timelines (UTC)

    Check times come from the GitHub API. The trace file had rotated past the first reads, so the sweep times marked ≈ are computed back from the sweep cadence the surviving trace shows (20 sweeps, 120.9 s apart on average).

    Missed, PR A Missed, PR B Woke, PR C Woke, PR D
    watch_pull_request 15:35:04 15:39:58 15:13:22 16:08:14
    Last check passes 15:36:22 15:40:28 15:14:48 16:09:37
    First sweep after the watch ≈15:36:52 ≈15:40:54 ≈15:14:46 16:09:08
    First read sees green green pending pending
    Wake none none 15:16:48 16:11:09
    User steps in by hand 15:47:47 15:50:49

    PR C woke only because its first read came about two seconds before its last check finished.

    Server log (PullRequestWatchReactor.reportEnd)

    {"message":"pull request watch ended","pullRequest":"github.com/<owner>/<repo-1>#A","reason":"stopped","minutes":14,"quietMinutes":14,"longestQuietMinutes":14,"wakes":0,"reads":1,"partial":false}
    {"message":"pull request watch ended","pullRequest":"github.com/<owner>/<repo-2>#B","reason":"stopped","minutes":13,"quietMinutes":13,"longestQuietMinutes":13,"wakes":0,"reads":1,"partial":false}
    {"message":"pull request watch ended","pullRequest":"github.com/<owner>/<repo-1>#D","reason":"stopped","minutes":5,"quietMinutes":5,"longestQuietMinutes":5,"wakes":1,"reads":2,"partial":false}

    wakes: 0, reads: 1 over 13 and 14 minutes: the single read is the baseline, taken after the checks went green, and every later sweep finds the fingerprint unchanged.

    How often

    The first read lands uniformly within one sweep interval after the watch call. With r seconds of CI left when the agent calls the watch, the chance of a lost wake is about 1 - r/121 for r < 121, and zero above. The two repositories here had 30 to 86 seconds left, so between one watch in three and three in four. Any repository whose CI ends within two minutes of the watch call is exposed on every PR.

    Open pull requests

    #37494 and #37492 shorten the delay for watches whose last read saw pending checks. A watch that has just been created has no last read and is not in the last sweep's groups, so its first read still waits for the sweep and still baselines whatever it finds. The cause described in this issue is unchanged on main as of 12b9845d1c5aec2bfede36b313adfc5098a2fb93: none of the commits since the build above touch pullRequestWatch.ts, PullRequestWatchReactor.ts or PullRequestWatchService.ts.

    Taking the baseline at registration, from the summary setWatching already reads and discards, would close the window.

  4. SwiftlyDaniel commented on Nov 17, 2026

    @SwiftlyDaniel

    This comment was prepared by GPT-6.1 Sol (Codex), reporting an occurrence observed in Dani’s T3 Code session.

    We observed a missed wakeup on self-hosted GitLab that appears consistent with this issue.

    Environment: T3 Code Nightly 0.1.11-nightly.20261117.4825, macOS 27.2.1 arm64, Codex provider (gpt-6.1-sol), glab 1.130.0.

    Timeline on 17 November 2026, UTC:

    • 15:56:42.299: the agent’s GitLab API read showed the MR’s head pipeline still running.
    • 15:56:47.253: watch_pull_request registered successfully, returning watching: true, wasWatching: false.
    • 15:56:48.779: GitLab records successful pipeline completion, 1.526 seconds after watch registration.
    • 15:56:49.155: the server trace records thread.pull-request-watch.sync.
    • 15:58:46.203: the user prompted the thread again. No automatic continuation had occurred.
    • 16:00:13.307: the trace records the watch ending with reason: "stopped", wakes: 0, reads: 2, partial: false.

    The agent ended its preceding turn at 15:56:51.001. The retained run history shows no watcher-triggered continuation before the user’s next message.

    This timing fits the registration-to-first-read window described here. However, we have not captured the saved evaluator baseline or the exact checks returned by its first read, so the internal cause remains unconfirmed. A GitLab-specific read or refresh failure has not been ruled out.

    A preceding MR also received no wake, but its checks had already completed before registration. We are keeping that separate because suppressing already-completed checks may be intentional.

    Could this issue be reopened to investigate this additional GitLab occurrence?

    Private repository URLs and thread identifiers have been omitted.

  5. mertcanekiz commented on Nov 18, 2026

    @mertcanekiz

    Note

    🤖 Claude Opus 5.5 responding on behalf of @mertcanekiz

    Another live occurrence, in a variant the open fixes don't cover: watching a PR again on a head the last watch already reported passing. A new watch inherits that passed: true. CI then re-runs on the same head (here because the PR was reopened) and passes before the first sweep, so nothing ever changes from the watch's point of view.

    Version: 0.1.8-nightly.20261109.4433, macOS, Claude provider, GitHub, no host event feed. The inheritance is unchanged on main (8cf97b5f06).

    Steps:

    1. Watch PR X. Its checks pass and the agent is woken with "All 5 checks passed on dff7660". The agent calls unwatch_pull_request.
    2. Without a push, CI runs again on dff7660. Here the base branch was deleted, GitHub closed the PR, and reopening it started a new run. A re-run, or a retarget that triggers CI, would do the same.
    3. While that run is in progress, the agent calls watch_pull_request (watching: true, wasWatching: false) and ends its turn.
    4. The run passes before the next sweep. No wake ever comes.

    Timeline (UTC, 2026-11-18)

    Time Event
    14:56:50 new CI run on dff7660 starts (PR reopened)
    14:57:02.98 watch_pull_request, while lint and test are in progress
    14:57:11 / 14:57:14 lint, test pass
    ≈14:58:3x first sweep (later sweeps at 15:00:33, 15:02:34, …): passed → passed, no change
    15:51 user asks for status, then merges by hand
    {"message":"pull request watch ended","pullRequest":"github.com/<owner>/<repo>#X","reason":"stopped","minutes":56,"quietMinutes":56,"longestQuietMinutes":56,"wakes":0,"reads":2,"partial":false}

    Final endedWatch: {"startedAt":"2026-11-18T14:57:02.983Z","headSha":"dff7660…","failedChecks":[],"passed":true,"passedChecks":["lint","test","bundle","production","isler"],"owedChecks":[],"remarksThrough":"2026-11-18T14:57:02.983Z","remarkIds":[],"conflicting":false,"wakes":0,"checksAwaitedSince":null}

    Why

    Orchestrator.ts#L5079-L5087 seeds a new watch from endedWatch: same headSha, passed, passedChecks ("Watching again must not repeat check or conflict news the last watch told"). The head didn't move, so the first read finds passed already true and passedNow && !passed never fires. A first read that lands while the re-run is pending would clear passed, and the pass would wake. So this is the same up-to-2-minute gap as this issue, plus one more way in.

    This matters for the fix proposed above. A baseline taken from the read setWatching already makes would cover this case only if a pending read there replaces the inherited passed. And #32276 proposes reporting a pass once per head, which would make this case silent by design. The agent re-watched specifically to wait for the new run, so to it the pass is news. Which behavior do you want for a same-head re-run after a re-watch?

  6. Bishi commented on Nov 22, 2026

    @Bishi

    🤖 Prepared by Claude Opus 5.5 (Claude Code, via t3 triage) on behalf of the reporter.

    This still happens on 0.1.11 (server 0.1.11, Windows 10, Codex provider, GitHub, web client). It also shows that #39116 fixed the failure side of this window but not the pass side.

    Timeline (UTC, one PR, no required checks, one check "Doc size"):

    Time Event
    15:31:14 Doc size starts on the new head
    15:31:33.9–15:31:35.0 Agent's pull_request_read returns Doc size pending
    15:31:34 GitHub check run completed_at, conclusion success
    15:31:48 Agent calls watch_pull_request(until: "merged") and ends its turn
    15:34:13 No wake; the user steps in, and the agent re-reads and merges

    Final saved watch: {"passed":true,"passedChecks":["Doc size"],"wakes":0,"checksPending":false,"until":"merged"}.

    Earlier in the same thread, a second PR's watch woke correctly ("checks passed") because its check finished about 25 minutes after the watch started. So the miss happens only when checks finish before the first read.

    Cause (v0.1.11, pullRequestWatch.ts): #39116 seeds checksPending: true from the agent's last read and exempts failures from the baseline (newlyFailed.length > 0 && (!checksBaseline || watch.checksPending === true)). The passed wake is still behind a bare if (!checksBaseline). A watch seeded with checksPending: true whose first read finds the checks green is a pending→passed change, and it should wake just as a pending→failed one does. main has the same guard today.

    Suggested fix: mirror the failure exemption, so the checks-passed change is reported when !checksBaseline || watch.checksPending === true.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions