Detect Claude auth expiry before it silently stops every agent

.claude/commands/gm-doctor.md

@@ -431,6 +431,134 @@ ssh -o ConnectTimeout=5 "$SERVER_HOST" '$HOME/.local/bin/audit.sh' 2>/dev/null
 
 ---
 
+### Check 9: Claude Auth Health
+
+The server's Claude Code login expires roughly every 30 days. When it lapses,
+every agent task fails or stalls and the only symptom is that no work happens.
+This check reports how long is left, proves the login actually works, and
+installs the monitor that will catch the next expiry automatically.
+
+**Do NOT use `claude auth status` (or `claude doctor`, or `claude mcp list`) as
+the health check.** They read cached local config and never contact Anthropic.
+On credentials that had been dead for 95 days, `claude auth status` still
+returned, with exit 0:
+
+```json
+{"loggedIn": true, "authMethod": "claude.ai", "subscriptionType": "max"}
+```
+
+**Steps:**
+
+1. **Report days remaining** — free, offline. The access token lasts ~8h and
+   self-refreshes; the refresh token is the real clock (~30 days, sliding):
+```bash
+ssh -o ConnectTimeout=5 "$SERVER_HOST" 'python3 -c "
+import json, time, os
+p = os.path.expanduser(\"~/.claude/.credentials.json\")
+try:
+    d = json.load(open(p))
+except Exception:
+    print(\"NO_CREDENTIALS_FILE\"); raise SystemExit
+o = d.get(\"claudeAiOauth\")
+if not o:
+    print(\"NO_OAUTH_BLOCK\"); raise SystemExit
+e = o.get(\"refreshTokenExpiresAt\")
+if e is None:
+    print(\"PRE_2_1_X_FORMAT_DEAD\"); raise SystemExit
+print(\"DAYS_LEFT=%d\" % int((e/1000 - time.time()) // 86400))
+"' 2>/dev/null
+```
+   - `DAYS_LEFT=N` — report it.
+   - `NO_OAUTH_BLOCK` / `NO_CREDENTIALS_FILE` is **not** proof of failure — the
+     credential may live in an OS keyring or be a long-lived setup-token. Just
+     note that the expiry date is unknown and let the probe in step 2 decide.
+   - `PRE_2_1_X_FORMAT_DEAD` means the credential predates Claude Code 2.1.x,
+     has no sliding refresh token, and is certainly dead.
+
+2. **Run the live probe** — the only truthful check. About 40 tokens, a few
+   seconds. Run it exactly once:
+```bash
+ssh -o ConnectTimeout=15 "$SERVER_HOST" 'export PATH="$HOME/.local/bin:$HOME/.npm-global/bin:$PATH" && claude -p "hi" --model haiku --max-turns 1 </dev/null >/dev/null 2>&1 && echo AUTH_OK || echo AUTH_FAILED' 2>/dev/null
+```
+
+3. **Rectify existing installs.** GMs provisioned before the monitor existed have
+   no expiry detection at all. Check whether it's there:
+```bash
+ssh -o ConnectTimeout=5 "$SERVER_HOST" 'test -x $HOME/.local/bin/claude-auth-monitor.sh && echo MONITOR_INSTALLED || echo MONITOR_MISSING; crontab -l 2>/dev/null | grep -q claude-auth-monitor.sh && echo CRON_INSTALLED || echo CRON_MISSING' 2>/dev/null
+```
+
+   **If either is missing, install it — do not just report it.**
+
+   a. Locate the template (repo checkout first, then the installed plugin):
+   ```bash
+   TASKYOU_OS_DIR=""
+   if [ -f "./templates/claude-auth-monitor.tmpl" ]; then
+     TASKYOU_OS_DIR="."
+   else
+     PLUGIN_DIR=$(python3 -c "import json; d=json.load(open('$HOME/.claude/plugins/installed_plugins.json')); entries=d.get('plugins',{}).get('taskyou-os@taskyou-os',[]); print(entries[0]['installPath'] if entries else '')" 2>/dev/null)
+     if [ -n "$PLUGIN_DIR" ] && [ -f "$PLUGIN_DIR/templates/claude-auth-monitor.tmpl" ]; then
+       TASKYOU_OS_DIR="$PLUGIN_DIR"
+     fi
+   fi
+   ```
+
+   b. Read `$TASKYOU_OS_DIR/templates/claude-auth-monitor.tmpl`, substitute
+      `{{SERVER_HOME}}` and `{{PROJECT_NAME}}` with the values from `config.env`,
+      write it to a temp file, then deploy:
+   ```bash
+   scp -q /tmp/claude-auth-monitor.rendered "$SERVER_HOST:$SERVER_HOME/.local/bin/claude-auth-monitor.sh"
+   ssh "$SERVER_HOST" "mkdir -p $SERVER_HOME/scripts && chmod +x $SERVER_HOME/.local/bin/claude-auth-monitor.sh"
+   rm -f /tmp/claude-auth-monitor.rendered
+   ```
+
+   c. Install the cron entry (every 30 minutes). **Every path it writes — flag,
+      state, log — must live under this user's own `$HOME`.** Several GMs can
+      share one box; a log or flag in a shared `/tmp` owned by another user is
+      exactly how a daemon start got broken in production. Never point this at
+      `/tmp`:
+   ```bash
+   CRON_LINE="*/30 * * * * export PATH=$SERVER_HOME/.local/bin:$SERVER_HOME/bin:$SERVER_HOME/.npm-global/bin:\$PATH && $SERVER_HOME/.local/bin/claude-auth-monitor.sh >> $SERVER_HOME/scripts/claude-auth-monitor.log 2>&1"
+   ssh "$SERVER_HOST" "crontab -l 2>/dev/null | grep -q claude-auth-monitor.sh || (crontab -l 2>/dev/null; echo '$CRON_LINE') | crontab -"
+   ```
+
+   d. Confirm the cron entry landed:
+   ```bash
+   ssh "$SERVER_HOST" 'crontab -l 2>/dev/null | grep claude-auth-monitor.sh'
+   ```
+
+4. **Clear a stale failure flag.** If `~/scripts/.auth-failed` exists but the
+   probe in step 2 returned `AUTH_OK`, the login was fixed but the flag was never
+   cleared — `linear-poll.mjs` is still creating tasks without executing them:
+```bash
+ssh "$SERVER_HOST" 'test -f $HOME/scripts/.auth-failed && rm -f $HOME/scripts/.auth-failed && echo CLEARED_STALE_FLAG'
+```
+   (The monitor clears this itself on its next healthy run; doing it here means
+   the operator isn't blocked for up to 30 minutes.)
+
+**Results:**
+
+- **Probe `AUTH_OK`, monitor + cron present, more than 5 days left:** PASS —
+  "Claude auth healthy, N days left; monitor checks every 30 min."
+- **Probe `AUTH_OK` but 5 days or fewer remain:** WARN — tell the user to run
+  `ssh <SERVER_HOST>` then `claude /login` before it lapses.
+- **Monitor or cron was missing and you installed it:** WARN — "Installed the
+  Claude auth monitor and its cron entry. This install had no expiry detection
+  at all before now."
+- **Stale `.auth-failed` cleared:** WARN — "Cleared a stale auth-failure flag
+  that was stopping the Linear poller from executing tasks."
+- **Probe `AUTH_FAILED`:** FAIL — the agents cannot run. Give the user the fix
+  verbatim:
+  ```
+  ssh <SERVER_HOST>
+  claude /login
+  ```
+  You may mention `claude setup-token`, which issues a 1-year token, as an option
+  for boxes that don't need claude.ai MCP connectors or Remote Control — it
+  disables both, so it is not a universal fix and must not be the default.
+- **SSH connection fails:** FAIL with "Could not connect to server."
+
+---
+
 ## Summary
 
 After all checks, present a summary table:
@@ -446,6 +574,7 @@ TaskYou-OS Doctor
   GM templates          PASS/WARN/FAIL
   Task event channel    PASS/WARN/FAIL
   Security audit        PASS/WARN/FAIL
+  Claude auth health    PASS/WARN/FAIL
 ─────────────────────────────────
 ```
 

.claude/commands/launch.md

@@ -210,30 +210,44 @@ ssh <HOST> 'mkdir -p ~/.claude && test -f ~/.claude/settings.json || echo "{\"sk
 
 ### Connecting your Claude account
 
-Check if Claude is already authenticated on the server:
+**Never trust `claude auth status` as proof of working auth.** It reads cached
+local config and never contacts Anthropic, so on credentials that have been dead
+for months it still prints `{"loggedIn": true, "authMethod": "claude.ai",
+"subscriptionType": "max"}` and exits 0. `claude doctor`, `claude mcp list` and
+`claude auth status --text` are equally offline and equally wrong. The only
+truthful check is a real model request.
+
+Check whether Claude actually works on the server:
 ```bash
-ssh <HOST> 'claude auth status 2>&1'
+ssh <HOST> 'claude -p "hi" --model haiku --max-turns 1 </dev/null >/dev/null 2>&1 && echo AUTH_OK || echo AUTH_FAILED'
 ```
+This costs about 40 tokens and takes a few seconds. Exit 0 (`AUTH_OK`) means the
+login genuinely works; on an expired login it exits 1 with "Failed to
+authenticate: OAuth session expired and could not be refreshed".
 
-If Claude isn't logged in, copy the user's local credentials to the server automatically:
+If it prints `AUTH_FAILED`, copy the user's local credentials to the server:
 ```bash
 ssh <HOST> 'mkdir -p ~/.claude'
 scp ~/.claude/.credentials.json <HOST>:~/.claude/.credentials.json
 ```
 
-Then verify it worked:
+Then re-run the same probe to verify it actually worked:
 ```bash
-ssh <HOST> 'claude auth status 2>&1'
+ssh <HOST> 'claude -p "hi" --model haiku --max-turns 1 </dev/null >/dev/null 2>&1 && echo AUTH_OK || echo AUTH_FAILED'
 ```
 
-If the transfer worked (shows `loggedIn: true`), tell the user: "I've connected your agents to your Claude account — they'll use the same subscription you use on your Mac."
+If it now prints `AUTH_OK`, tell the user: "I've connected your agents to your Claude account — they'll use the same subscription you use on your Mac."
 
-If the local credentials file doesn't exist (`~/.claude/.credentials.json`), the user isn't logged in locally either. In that case, explain: "I need you to log into your Claude account on the server. This connects the agents to your subscription so they can do their work." Give them:
+If the local credentials file doesn't exist (`~/.claude/.credentials.json`), the user isn't logged in locally either — or, on a Mac, their credentials live in the Keychain rather than that file, so there is nothing to copy. In that case, explain: "I need you to log into your Claude account on the server. This connects the agents to your subscription so they can do their work." Give them:
 ```
 ssh <SERVER_HOSTNAME>
-claude login
+claude /login
 ```
 
+Once auth works, mention that this login expires roughly every 30 days, that
+`setup.sh` installs a monitor which checks it every 30 minutes and alerts before
+it lapses, and that `/doctor` will report exactly how many days are left.
+
 ### Connecting GitHub (only if GitHub was chosen)
 ```bash
 ssh <HOST> 'gh auth status 2>&1'

README.md

@@ -155,6 +155,61 @@ These are configured via flags in `config.env` during setup. They're part of the
 - **Cloudflare R2** (`R2_ENABLED=true`) — Public URLs for files and assets agents generate
 - **GitHub** (`GITHUB_REPOS=workspace:org/repo`) — Push agent work to your repositories
 
+## Claude Auth Expiry
+
+Agent servers authenticate Claude Code per-Unix-user with `claude /login`
+(claude.ai OAuth against a Max subscription). **That login expires roughly every
+30 days.** When it lapses, every agent task fails or stalls, and the only symptom
+is that no work happens.
+
+`setup.sh` installs `claude-auth-monitor.sh` plus a cron entry that runs it every
+30 minutes. On failure it writes `~/scripts/.auth-failed` — the flag
+`modules/linear/linear-poll.mjs` already checks before executing tasks — and
+appends an `auth_failed` event to `~/notifications.jsonl`, which the Slack bridge
+already tails. It also warns with `auth_expiring` when fewer than 5 days remain.
+Alerts are rate-limited (failures re-alert every 6h, warnings every 24h) so a
+30-minute cron can't storm Slack. Every path it writes lives under the user's own
+`$HOME`, never a shared `/tmp` — several GMs can share one box.
+
+**Do not use `claude auth status` to check this.** It reads cached local config
+and never contacts Anthropic. On credentials that had been dead for 95 days it
+still returned, with exit 0:
+
+```json
+{"loggedIn": true, "authMethod": "claude.ai", "subscriptionType": "max"}
+```
+
+`claude doctor`, `claude mcp list` and `claude auth status --text` are equally
+offline and equally wrong. The only truthful check is a real model request:
+
+```bash
+claude -p "hi" --model haiku --max-turns 1 </dev/null >/dev/null 2>&1
+```
+
+Exit 0 means healthy; exit 1 on an expired login, with "Failed to authenticate:
+OAuth session expired and could not be refreshed". It costs about 40 tokens, so
+the monitor gates it behind a free offline read of
+`~/.claude/.credentials.json` → `claudeAiOauth.refreshTokenExpiresAt`. The access
+token lasts ~8h and self-refreshes; the refresh token is the real ~30-day sliding
+clock. A `claudeAiOauth` block with no `refreshTokenExpiresAt` predates Claude
+Code 2.1.x and is certainly dead. No `claudeAiOauth` block at all is *unknown*,
+not dead — the credential may be in an OS keyring — so the probe decides.
+
+Run `/doctor` at any time to see days remaining, probe the login for real, and
+install the monitor on a GM that predates it.
+
+### Fixing an expired login
+
+```bash
+ssh <SERVER_HOST>
+claude /login
+```
+
+**Alternative for boxes that don't use connectors:** `claude setup-token` issues
+a token valid for a year, which removes the monthly chore. It disables claude.ai
+MCP connectors and Remote Control for that user, so it is not a universal fix and
+is not the default — use it only where those features aren't needed.
+
 ## Manual Setup
 
 If you prefer to skip the interactive `/launch` wizard:

modules/slack/slack-bridge.mjs

@@ -224,6 +224,25 @@ export function formatNotification(event) {
   const title = event.title || "";
   const project = event.project ? ` _(${event.project})_` : "";
   switch (event.event) {
+    // ── Claude auth health ───────────────────────────────────────────────────
+    // Emitted by claude-auth-monitor.sh, not by a task hook, so these carry no
+    // task_id — they describe the server's own Claude login. With no task_id
+    // they always land in SLACK_NOTIFY_CHANNEL rather than a task thread.
+    case "auth_failed":
+      return (
+        `:rotating_light: *Claude login expired — agents cannot run*${project}\n` +
+        `${title || "The Claude Code login on the agent server has expired."}\n` +
+        `Tasks will still be created, but nothing will execute until it's renewed.`
+      );
+    case "auth_expiring": {
+      const days = Number.isFinite(event.days_remaining)
+        ? ` (${event.days_remaining} day${event.days_remaining === 1 ? "" : "s"} left)`
+        : "";
+      return (
+        `:hourglass: *Claude login expires soon*${days}${project}\n` +
+        `${title || "Renew the Claude Code login on the agent server."}`
+      );
+    }
     case "completed":
       return `:white_check_mark: *Task #${id} completed*: ${title}${project}`;
     case "blocked":

modules/slack/slack-bridge.test.mjs

@@ -115,6 +115,39 @@ test("formatNotification renders each event type", () => {
   assert.match(formatNotification({ event: "weird", task_id: "4", title: "Huh" }), /Task #4/);
 });
 
+test("formatNotification renders Claude auth events without a task_id", () => {
+  // These come from claude-auth-monitor.sh, which has no task to reference.
+  // The old default branch would have rendered them as "Task #? (auth_failed)".
+  const failed = formatNotification({
+    event: "auth_failed",
+    title: "Claude login on agents-1 has expired — run 'claude /login' as exedev.",
+    project: "engineering",
+  });
+  assert.match(failed, /Claude login expired/);
+  assert.match(failed, /claude \/login/);
+  assert.doesNotMatch(failed, /Task #/);
+
+  const expiring = formatNotification({
+    event: "auth_expiring",
+    title: "Claude login on agents-1 expires in 3 day(s).",
+    days_remaining: 3,
+    project: "engineering",
+  });
+  assert.match(expiring, /expires soon/);
+  assert.match(expiring, /3 days left/);
+  assert.doesNotMatch(expiring, /Task #/);
+
+  // Singular day, and a missing day count must not print "NaN".
+  assert.match(
+    formatNotification({ event: "auth_expiring", title: "x", days_remaining: 1 }),
+    /1 day left/
+  );
+  assert.doesNotMatch(
+    formatNotification({ event: "auth_expiring", title: "x" }),
+    /NaN|undefined/
+  );
+});
+
 test("buildClassifierContext includes thread + open tasks + message", () => {
   const ctx = buildClassifierContext("do the thing", {
     threadTaskId: "12",

setup.sh

@@ -625,6 +625,46 @@ install_daemon_service() {
   fi
 }
 
+# Install the Claude auth expiry monitor + its cron entry.
+# $1 = ssh target, $2 = that host's home directory.
+#
+# Claude Code logins on agent servers expire about every 30 days, and until this
+# existed nothing noticed: tasks simply stopped working. `claude auth status`
+# cannot be used as the check — it reads cached config and never contacts
+# Anthropic, so it keeps reporting {"loggedIn": true, ...} with exit 0 on
+# credentials that have been dead for months. The monitor does a free offline
+# read of the refresh-token expiry and then one real ~40-token model request,
+# which is the only truthful signal. See templates/claude-auth-monitor.tmpl.
+install_auth_monitor() {
+  local ssh_target="$1"
+  local remote_home="$2"
+
+  log "Installing Claude auth monitor"
+
+  # Render against THIS host's home (exe and server modes pass different homes).
+  local rendered="/tmp/taskyou-claude-auth-monitor.$$"
+  ( export SERVER_HOME="$remote_home"
+    render_file "$TEMPLATES_DIR/claude-auth-monitor.tmpl" "$rendered" )
+
+  ssh "$ssh_target" "mkdir -p $remote_home/.local/bin $remote_home/scripts"
+  scp -q "$rendered" "$ssh_target:$remote_home/.local/bin/claude-auth-monitor.sh"
+  ssh "$ssh_target" "chmod +x $remote_home/.local/bin/claude-auth-monitor.sh"
+  rm -f "$rendered"
+  ok "claude-auth-monitor.sh"
+
+  # Cron every 30 minutes. Every path the script writes (flag, state, log) lives
+  # under this user's own $HOME — several GMs can share one box, and a log or
+  # flag in a shared /tmp owned by another user is exactly how a daemon start
+  # got broken in production.
+  local cron_line="*/30 * * * * export PATH=$remote_home/.local/bin:$remote_home/bin:$remote_home/.npm-global/bin:\$PATH && $remote_home/.local/bin/claude-auth-monitor.sh >> $remote_home/scripts/claude-auth-monitor.log 2>&1"
+  if ssh "$ssh_target" "crontab -l 2>/dev/null" | grep -q "claude-auth-monitor.sh"; then
+    ok "Auth monitor cron job already exists"
+  else
+    ssh "$ssh_target" "(crontab -l 2>/dev/null; echo '$cron_line') | crontab -"
+    ok "Auth monitor cron job installed (every 30 minutes)"
+  fi
+}
+
 # ── Local-server setup (server = this machine) ───────────────────────────────
 
 # True when the "server" is the same box the GM runs on (no SSH, no systemd).
@@ -908,6 +948,10 @@ EOF
   # Install daemon as a systemd user service (auto-starts on boot, restarts on crash)
   install_daemon_service "$SERVER_HOST" "$SERVER_HOME"
 
+  # Claude auth expiry monitor (detects the ~30-day login lapse before it
+  # silently stops every agent).
+  install_auth_monitor "$SERVER_HOST" "$SERVER_HOME"
+
   log "Server setup complete"
 }
 
@@ -1234,6 +1278,10 @@ EOF
   # Install daemon as a systemd user service (auto-starts on boot, restarts on crash)
   install_daemon_service "$EXE_HOST" "$EXE_HOME"
 
+  # Claude auth expiry monitor (detects the ~30-day login lapse before it
+  # silently stops every agent).
+  install_auth_monitor "$EXE_HOST" "$EXE_HOME"
+
   log "exe.dev deployment complete!"
   echo ""
   echo "  Task board:    https://${EXE_DEV_VM_NAME}.exe.xyz"