fix(knowledge): a resume repaid the most expensive step it had already finished

[sshlg]

The defect

_run_steps reads the completed-step set once (pipeline_runner.py:168) and gates four steps on it — detect_changes (:265), cleanup_deleted (:481), project_profile (:510), cross_file_analysis (:648). code_symbol_embed writes complete_step and nothing reads it, so every resume runs it again.

Measured twice in production on 2026-08-27, from the tracker:

11:37:23 → 12:15:43   code_symbol_embed   2300 s   (38.3 min)
12:44:57 → 13:20:33   code_symbol_embed   2136 s   (35.6 min)  ← the same work, redone

It is the most expensive step in the pipeline, and expensive on purpose: EMBEDDING_UPSERT_BATCH_SIZE is 8 to hold the worker inside its memory quota, trading ~17 % wall clock for ~552 MiB.

Why this matters more than the waste

A full rebuild of that repository needs about 3.4 hours — code_symbol_embed at ~38 min plus generate_docs at ~4.8 documents/minute over 758 documents. The manual job's ceiling cut it off inside generate_docs; resuming re-entered code_symbol_embed and spent the 38 minutes over, so attempt N+1 reached no further than attempt N and no number of attempts could finish. Raising the ceiling alone does not fix that — it only moves where the loop stalls.

The pattern copied is already one step later in the same file: generate_docs resumes at document granularity through processed_doc_paths (:970), confirmed live — the run resumed at document 80 and is now past 110. This is the coarse version of that.

ast_parse and graph_build stay ungated deliberately — the first rebuilds in-memory state nothing else supplies, the second merges into the stored graph so unchanged files survive — and a test fails if either joins the gated set.

Board, both against my own earlier claims

CB-OPS1 RE-OPENED. I struck it on "a full re-index logged zero R14/R15". Every run behind that claim died in or before code_symbol_embed — nothing had reached the late steps where memory actually peaks. The first run that did: 12:32–12:34 inside generate_docs, mem=1135M(100.1%), 4 × R14 on Standard-2X. What the resize genuinely bought is now stated precisely: before it, 170 × R14 and 2 × R15 with a 1 143 MiB peak against a 512 MiB quota — fatal. Now: over quota, no R15, the process survives. Marginal, not clear.

CB-OPS3 added (supersedes #238, which I am closing): a worker restart during a repo index loses the run and nothing retries it — measured on v279's own deploy.

Counts and the severity table are re-derived from the rows by script, not edited by hand.

Still open, deliberately not in this PR

repo_index_job_timeout_seconds is 3600 and that is known to be too small — a run died at exactly 3600 s inside generate_docs. The corrected ceiling needs a completed full rebuild to size it, and that measurement is running now under a temporary REPO_INDEX_JOB_TIMEOUT_SECONDS=10800. Encoding a number before measuring it is the mistake this branch exists to correct, so the constant follows in its own change.

Verification

• pytest tests/unit/knowledge/ tests/integration/ — 1014 passed, 2 skipped.
• ruff format --check, ruff check, mypy app/ — clean.
• pytest tests/unit/docs/ — 50 passed (the ratchets over this board's own arithmetic).