Skip to content

fix(markdown): resolve optional token threshold defaults - #543

Open
rudycelekli wants to merge 1 commit into
VectifyAI:mainfrom
rudycelekli:fix/markdown-optional-threshold-defaults-20261001
Open

rudycelekli wants to merge 1 commit into
VectifyAI:mainfrom
rudycelekli:fix/markdown-optional-threshold-defaults-20261001

Conversation

@rudycelekli

Copy link
Copy Markdown

Problem

The public pageindex.md_to_tree() API defaults both optional token thresholds to None. Enabling summaries or thinning without explicitly supplying a threshold compares an integer token count against None and raises TypeError, before summaries can be generated.

Change

Resolve omitted or explicit None thresholds to the CLI's existing defaults: 200 summary tokens and 5,000 thinning tokens. Preserve explicit zero and positive thresholds.

Validation

  • Reproduced both failures through the exported async API using real Markdown parsing and token counting before applying the fix (four failing subtests).
  • Added coverage in the existing Markdown test file for omitted/None thresholds, preserved merged child text, and explicit zero/positive thresholds.
  • Focused tests: 7 passed, 6 subtests passed. A falsy-default mutation fails both explicit-zero controls.
  • Full local repository gate: python -m pytest -q — 589 passed, 218 skipped, 6 subtests passed on Python 3.12.14/PDFium 5.13.0, without optional agent frameworks.
  • No live model/provider calls were made. Only the summary-generation boundary is stubbed where a model response is required.

The hosted Python 3.10/3.13 and optional-framework/PDFium matrix remains for upstream CI; the local result does not claim that matrix passed.

Prepared with AI assistance; source, reproductions and exact patch were independently reviewed.

Signed-off-by: Rudy Celekli <47457359+rudycelekli@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant