fix(worker): size the re-index ceiling from a rebuild that finished

CHANGELOG.md

@@ -6,6 +6,45 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ## [Unreleased]
 
+### Fixed — a full re-index could not finish, and the ceiling was the symptom
+
+Three ceilings cut three runs off on 2026-08-27 — 1800.02 s inside `code_symbol_embed`,
+3600.00 s inside `generate_docs` at document 80 of 758, and a deploy — before the real
+obstacle was found. `_run_steps` reads the completed-step set once
+(`pipeline_runner.py:168`) and gated four steps on it. **`code_symbol_embed` wrote
+`complete_step` and nothing read it**, so every resume ran it again:
+
+    11:37:23 → 12:15:43   code_symbol_embed   2300 s
+    12:44:57 → 13:20:33   code_symbol_embed   2136 s   ← the same work, redone
+
+Attempt N+1 therefore reached no further than attempt N, and no ceiling could have
+fixed that. With the gate in place, measured live: enqueue → `generate_docs` in **96 s**
+against ~36 minutes before. `ast_parse` and `graph_build` stay ungated deliberately —
+the first rebuilds in-memory state nothing else supplies, the second merges into the
+stored graph — and a test fails if either joins the gated set. The pattern was already
+one step later in the same file: `generate_docs` resumes per document
+(`processed_doc_paths`), confirmed live at 80 → 300 → 758.
+
+**Two corrections to this release's own earlier entry.** It claimed the cron and the
+button run "the same work under two ceilings". They do not:
+`daily_knowledge_sync_service._run_repo_index` passes `force_full=False,
+chain_sync=False`, so the cron's 42.4-minute `completed` run was an *incremental*
+index with the chain off — never a full rebuild. And a test asserting `repo < daily`
+on the strength of that reading actively capped the manual path below what it needs;
+it is replaced by checks that each ceiling is sized against the work it actually
+carries, plus a guard that fails if the cron stops being incremental.
+
+A full rebuild of that repository — 9 981 files, 758 documents — measures **12 039 s
+(3.34 h)**, summed from the segments of the run that reached `pipeline_end` at
+15:51:39: `generate_docs` ~9 375 s at ~4.8 docs/min, `code_symbol_embed` ~2 300 s,
+everything else ~364 s. `repo_index_job_timeout_seconds` now defaults to **16200**,
+35 % over the measurement, and the number is argued in
+`tests/unit/services/test_repo_index_ceiling.py` rather than chosen.
+
+The chained code↔DB sync ran on its own at the end — `code_db_sync` 15:50:26 →
+15:51:39 — which is the first live confirmation of `auto_sync_after_index` defaulting
+on.
+
 ### Fixed — the nightly cron could rebuild a repository the "Re-index repository" button never could
 
 `run_repo_index_task` is the repo-index pipeline, and two ARQ jobs call it with two

CLAUDE.md

@@ -215,7 +215,8 @@ Worker functions (`backend/app/worker.py`):
 
 - `run_db_index` — schema indexing for a connection
 - `run_code_db_sync` — code↔DB cross-reference
-- `run_repo_index` — Git repo knowledge pipeline (per-function timeout `repo_index_job_timeout_seconds`, **3600 s since 2026-08-27**). The same pipeline also runs *inside* `run_daily_project_knowledge_sync`, which carries its own 7200 s ceiling — so until the raise, the nightly cron rebuilt a 9 981-file repository in 42.4 min while every press of "Re-index repository" died at exactly 1800 s inside `code_symbol_embed`. Keep this knob **below** `daily_knowledge_sync_job_timeout_seconds`, which contains it plus a DB index plus a code↔DB sync; the ordering is asserted in `tests/unit/services/test_repo_index_ceiling.py`, not just commented.
+- `run_repo_index` — Git repo knowledge pipeline (per-function timeout `repo_index_job_timeout_seconds`, **16200 s since 2026-08-27**). This job carries the **full** rebuild — `force_full=True` plus the chained code↔DB sync. The nightly cron runs the same pipeline with `force_full=False, chain_sync=False` under its own 7200 s ceiling, so **the two ceilings cover different work and must not be tied together**. Reading the cron's 42.4-minute incremental run as a full rebuild is what sized this knob at 1800 and then 3600, and each cut a real run off: 1800.02 s inside `code_symbol_embed`, 3600.00 s inside `generate_docs` at document 80 of 758. A full rebuild of that 9 981-file repository measures **12 039 s (3.34 h)** — `generate_docs` ~9 375 s at ~4.8 docs/min, `code_symbol_embed` ~2 300 s — summed from the segments of the run that reached `pipeline_end`. Asserted in `tests/unit/services/test_repo_index_ceiling.py`, which also fails if the cron stops being incremental.
+- **A resume no longer repays `code_symbol_embed`.** `_run_steps` reads the completed-step set once (`pipeline_runner.py:168`) and gated only four steps on it; `code_symbol_embed` recorded completion that nothing read, so every resume spent its 38 minutes again and attempt N+1 reached no further than attempt N — no ceiling could fix that. Measured after the gate: enqueue → `generate_docs` in **96 s**, against ~36 min before. `ast_parse` and `graph_build` stay ungated deliberately (in-memory state; graph merge), and a test fails if either joins the gated set.
 - `run_batch` — batch query execution
 - `run_analytics_collect` — collect one analytics connection's reports into its fact tables (per-function timeout `analytics_collect_job_timeout_seconds`)
 

backend/.env.example

@@ -401,16 +401,17 @@ CORS_ORIGINS=["http://localhost:3000","http://localhost:3100","https://checkmyda
 # Wall-clock budget for one connection's collection job (ARQ per-function
 # timeout; also passed to the in-process fallback).
 # ANALYTICS_COLLECT_JOB_TIMEOUT_SECONDS=1800
-# Ceiling for one repo-index job, in seconds. The repo index is the longest job in
-# the system; before 2026-08-19 it was the only long one with no knob and inherited
-# ARQ's hardcoded 1800 s. Measured on a 9 981-file repository: a full rebuild takes
-# 42.4 min, so 1800 s could never finish one and 3600 s finishes it with headroom.
-# Keep this BELOW DAILY_KNOWLEDGE_SYNC_JOB_TIMEOUT_SECONDS, which grants the same
-# pipeline its budget plus a DB index and a code↔DB sync — mismatching the two is
-# how the nightly cron rebuilt a repository the manual button never could.
-# Raising it only helps a worker that is NOT swapping: on an over-quota dyno it buys
-# a longer crawl, not a finished index (AUD-0819-20).
-# REPO_INDEX_JOB_TIMEOUT_SECONDS=3600
+# Ceiling for one repo-index job, in seconds. This job carries the FULL rebuild —
+# `force_full=True` plus the chained code↔DB sync — while the nightly cron runs the
+# same pipeline with `force_full=False, chain_sync=False`. Different work, so do NOT
+# tie this to DAILY_KNOWLEDGE_SYNC_JOB_TIMEOUT_SECONDS; reading the cron's 42-minute
+# incremental run as a full rebuild is what sized this at 1800 and then 3600, both
+# of which cut real runs off. Measured end to end on a 9 981-file / 758-document
+# repository: 12 039 s (3.34 h), dominated by generate_docs (~9 375 s at ~4.8
+# docs/min) and code_symbol_embed (~2 300 s). 16200 leaves 35 % over that; both
+# steps scale with repository size, so raise it for a bigger repo rather than
+# hoping. Raising it only helps a worker that is NOT swapping (AUD-0819-20).
+# REPO_INDEX_JOB_TIMEOUT_SECONDS=16200
 # How long a `running` batch may sit before another attempt may take its claim
 # (F-SCHED-07). `run_batch` inherits ARQ's class-level job_timeout of 1800 s, so past
 # that the previous attempt is provably dead — arq cancelled it. Matching the two is the

backend/app/config.py

@@ -520,18 +520,36 @@ def _fix_database_url(self) -> "Settings":
     # 8,552 files. It got a knob and the default was left at 1800 — the value that
     # had just been measured as too small.
     #
-    # Raised to 3600 on 2026-08-27, because leaving it produced the same failure
-    # eight days later and the reason to withhold the raise had gone. Two
-    # measurements, both from `indexing_runs` on the one real customer repository:
+    # Raised to 3600 on 2026-08-27 and that was still too small; corrected to 16200
+    # the same day, after a full rebuild was measured to the end for the first time.
     #
-    #     08-25 22:00  completed      42.4 min   nightly cron, ceiling 7200 s
-    #     08-27 09:30  TimeoutError   30.0 min   manual re-index, ceiling 1800 s
+    # The first raise rested on a mis-reading worth recording, because it is easy to
+    # repeat: the cron's 42.4-minute `completed` run was read as a full rebuild. It
+    # is not. `daily_knowledge_sync_service._run_repo_index` passes
+    # `force_full=False, chain_sync=False` — an incremental index with the code↔DB
+    # chain off, because the cron runs that sync itself as a separate step. The two
+    # ceilings therefore cover different work, and "the same work under two
+    # ceilings" was wrong.
     #
-    # The same pipeline, the same 2 544 s of work, two ceilings — because the cron
-    # runs it *inside* `run_daily_project_knowledge_sync`, which carries
-    # `daily_knowledge_sync_job_timeout_seconds`. So the repository rebuilt
-    # unattended at 3 a.m. and could never rebuild when a person pressed
-    # "Re-index repository", which is the path an operator reaches for after a fix.
+    # What a full rebuild costs, summed from measured segments of the run that
+    # reached `pipeline_end` at 15:51:39 (9 981 files, 758 documents):
+    #
+    #       151 s  setup + ast_parse + graph_build
+    #      2300 s  code_symbol_embed
+    #       144 s  analyze_files + cross_file_analysis + graph_db_bridge
+    #      9375 s  generate_docs, 758 docs at ~4.8/min
+    #        69 s  embed_and_store + bm25 + the chained sync
+    #     12039 s  = 3.34 h
+    #
+    # 16200 s leaves 35 % over that. Both dominant steps scale with repository size
+    # rather than with a clock, so the headroom is not decoration.
+    #
+    # It cut off three runs before it was sized: 1800.02 s inside `code_symbol_embed`,
+    # 3600.00 s inside `generate_docs` at document 80 of 758, and once more at a
+    # deploy. Raising the ceiling was never the whole fix — until 2026-08-27 a resume
+    # re-ran `code_symbol_embed` (38 min) because nothing read its recorded
+    # completion, so attempt N+1 reached no further than attempt N. That gate is in
+    # `pipeline_runner.py`; this number only decides whether one attempt suffices.
     #
     # The old note said raising this only helps when the worker is not swapping.
     # That precondition is now met and measured: the dyno is Standard-2X and a full
@@ -543,7 +561,7 @@ def _fix_database_url(self) -> "Settings":
     # Invariant, asserted in `tests/unit/services/test_repo_index_ceiling.py`: this
     # stays below `daily_knowledge_sync_job_timeout_seconds`, which contains it plus
     # a DB index plus a code↔DB sync.
-    repo_index_job_timeout_seconds: int = 3600
+    repo_index_job_timeout_seconds: int = 16200
 
     # F-SCHED-07: how long a `running` batch may sit before another attempt may take
     # its claim. `run_batch` inherits ARQ's class-level `job_timeout` (1800 s), so past