Skip to content

fix(worker): size the re-index ceiling from a rebuild that finished - #240

Merged
sshlg merged 1 commit into
mainfrom
fix/repo-index-ceiling-measured-2
Aug 27, 2026
Merged

fix(worker): size the re-index ceiling from a rebuild that finished#240
sshlg merged 1 commit into
mainfrom
fix/repo-index-ceiling-measured-2

Conversation

@sshlg

@sshlg sshlg commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

The measurement

Three ceilings cut three runs off on 2026-08-27 before one was measured to the end — 1800.02 s inside code_symbol_embed, 3600.00 s inside generate_docs at document 80 of 758, and a deploy. The run that reached pipeline_end at 15:51:39 gives the number.

A full rebuild of that repository — 9 981 files, 758 documents — costs 12 039 s (3.34 h), summed from measured segments rather than one wall clock, because the run was interrupted twice and a span across an idle gap would overstate the work:

   151 s  setup + ast_parse + graph_build
  2300 s  code_symbol_embed
   144 s  analyze_files + cross_file_analysis + graph_db_bridge
  9375 s  generate_docs, 758 docs at ~4.8/min across three segments
    69 s  embed_and_store + bm25 + the chained code↔DB sync
 12039 s

repo_index_job_timeout_seconds defaults to 16200 — 35 % over that. Both dominant steps scale with repository size rather than with a clock, so the headroom is not decoration.

Two corrections to this release's own earlier entry, both mine

  • It claimed the cron and the button run "the same work under two ceilings". They do not. daily_knowledge_sync_service._run_repo_index passes force_full=False, chain_sync=False, so the cron's 42.4-minute completed run was an incremental index with the chain off — never a full rebuild. That reading is what sized the knob at 1800 and then 3600.
  • The test asserting repo < daily rested on that reading and actively capped the manual path below what it needs. Replaced by checks that each ceiling is sized against the work it actually carries, plus a guard that fails if the cron stops being incremental.

Board, from the same run

CB-OPS1 now carries a real number. Peak 1 246 MB (112.7 %), 45 × R14, 0 × R15 across 3.34 h on Standard-2X — and every R14 falls inside generate_docs, not graph_build. The earlier "zero R14/R15" measured a rebuild that never reached the late steps where memory peaks. What the resize genuinely bought: before it, 170 × R14 and 2 × R15 with a 1 143 MiB peak against a 512 MiB quota — the process was killed. Now it runs 12 % over quota for hours and survives.

CB-KNOW1 added. The shape guard dropped 48 of 280 cached names and took mismatch 6 → 0, but clientlogses, mobile_identifierses, twiliocallses, twiliophoneses remain — an already-plural class name (ClientLogs) pluralised a second time. They pass is_plausible_table_name legitimately, so a different rule is needed, not a longer list.

What the run produced

sync_status before after
db_only 252 96
matched 31 137
code_only 42 23
mismatch 6 0

Cached table names 280 → 232. All six mismatch rows had been noise (tables any, ases, whatever; columns e, now).

The chained sync ran unprompted — code_db_sync 15:50:26 → 15:51:39 — the first live confirmation of auto_sync_after_index defaulting on.

Verification

  • pytest tests/6958 passed, 4 skipped, 1 xfailed.
  • ruff format --check, ruff check, mypy app/ — clean.
  • The temporary REPO_INDEX_JOB_TIMEOUT_SECONDS=10800 used as a measuring instrument is unset (v282); production is back on code defaults.

Three ceilings cut three runs off on 2026-08-27 before one was measured to the
end: 1800.02 s inside `code_symbol_embed`, 3600.00 s inside `generate_docs` at
document 80 of 758, and a deploy. The run that reached `pipeline_end` at 15:51:39
gives the number.

A full rebuild of that repository — 9 981 files, 758 documents — costs 12 039 s
(3.34 h), summed from measured segments rather than one wall clock, because the
run was interrupted twice and a span across an idle gap would overstate the work:

      151 s  setup + ast_parse + graph_build
     2300 s  code_symbol_embed
      144 s  analyze_files + cross_file_analysis + graph_db_bridge
     9375 s  generate_docs, 758 docs at ~4.8/min across three segments
       69 s  embed_and_store + bm25 + the chained code<->DB sync
    12039 s

`repo_index_job_timeout_seconds` defaults to 16200 — 35% over that. Both dominant
steps scale with repository size rather than with a clock, so the headroom is not
decoration.

Two corrections to the earlier entry in this same release, both mine:

- It claimed the cron and the button run "the same work under two ceilings". They
  do not. `daily_knowledge_sync_service._run_repo_index` passes `force_full=False,
  chain_sync=False`, so the cron's 42.4-minute `completed` run was an incremental
  index with the chain off — never a full rebuild, and the reading that sized the
  knob at 1800 and then 3600.
- The test asserting `repo < daily` rested on that reading and actively capped the
  manual path below what it needs. Replaced by checks that each ceiling is sized
  against the work it carries, plus a guard that fails if the cron stops being
  incremental.

Board, from the same run:

- CB-OPS1 now carries a real number. Peak 1246 MB (112.7%), 45 x R14, 0 x R15
  across 3.34 h on Standard-2X, every R14 inside `generate_docs`. The earlier
  "zero R14/R15" measured a rebuild that never reached the late steps.
- CB-KNOW1 added. The shape guard dropped 48 of 280 cached names and took
  `mismatch` 6 -> 0, but `clientlogses`, `mobile_identifierses`, `twiliocallses`
  and `twiliophoneses` remain: an already-plural class name pluralised a second
  time. They pass the shape rule legitimately, so a different rule is needed.

The chained sync ran unprompted at the end — `code_db_sync` 15:50:26 -> 15:51:39 —
the first live confirmation of `auto_sync_after_index` defaulting on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@sshlg
sshlg merged commit 58b5c04 into main Aug 27, 2026
2 checks passed
@sshlg
sshlg deleted the fix/repo-index-ceiling-measured-2 branch August 27, 2026 16:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant