fix(agent): summary-rewrite falls back to v1 instead of retrying with a smaller prompt on context-window overflow

[sebastianbraun25]

Problem

_compile_concepts() (openkb/agent/compiler.py) has two LLM calls that grow with the size of
the knowledge base: concepts-plan (uncapped concept/entity briefs, see #226) and
summary-rewrite (the whitelist of valid [[wikilink]] targets, known_targets_msg, which
carries every existing concept/entity page name).

The concepts-plan call already has a fix (from #235's branch): on
litellm.ContextWindowExceededError it retries once with the full source document (doc_msg)
dropped — the plan prompt works "based on the summary above" anyway — before giving up.

summary-rewrite has no equivalent retry. Its prompt is
[system_msg, doc_msg, summary_msg, known_targets_msg, ...]. As the KB grows, known_targets_msg
grows right along with it, and combined with a large source document this can push the call over
the context window even when concepts-plan (a smaller prompt, no whitelist) still fits fine for
the same document. Today any exception from this call — including
ContextWindowExceededError — is caught by a blanket except Exception that silently falls back
to writing the unmodified v1 summary. This avoids a crash, but permanently forfeits the improved
cross-linking summary-rewrite exists to produce, for every document from that point on as the KB
keeps growing.

Reproduction

• Command: openkb add <document> against a KB with a large number of existing concept/entity
  pages (large known_targets_msg) and/or a sizeable source document.
• Observed (log excerpt, doc/whitelist details redacted):

  summary-rewrite. failed
  openkb.agent.compiler WARNING: summary-rewrite failed for <doc>: litellm.ContextWindowExceededError: litellm.BadRequestError: ... prompt is too long: 200400 tokens > 200000 maximum. Falling back to v1.
  

• Expected: same behavior as concepts-plan — retry once with doc_msg dropped before falling
  back to the v1 summary, since summary-rewrite's prompt only asks the model to reconcile the
  already-generated summary (summary_msg) against the whitelist (known_targets_msg), not the
  original document.

Context

• openkb/agent/compiler.py: _compile_concepts(), the if rewrite_summary: block (Step 3a).
• Depends on _NON_RETRYABLE_LLM_ERRORS and the concepts-plan retry pattern introduced on the
  branch for fix(agent): stream LLM completions to avoid gateway idle-timeout on long compiles #235.

Related

• concepts-plan prompt embeds an uncapped, uncached listing of every concept and entity page (measured 3.5x per-document input growth over 28 docs) #226 — the underlying growth of the concept/entity whitelist with KB size.
• fix(agent): stream LLM completions to avoid gateway idle-timeout on long compiles #235 — introduces _NON_RETRYABLE_LLM_ERRORS and the analogous concepts-plan retry that this
  issue mirrors for summary-rewrite.

Suggested fix

Give summary-rewrite the same one-shot "retry without the full document" treatment as
concepts-plan: on _NON_RETRYABLE_LLM_ERRORS, retry once with [system_msg, summary_msg, known_targets_msg, user] (no doc_msg); if that also fails, fall through to the existing v1
fallback unchanged.

---

_{This issue was drafted with the assistance of an AI assistant.}