Review view: unified document tree, #541 as landed (never merge)

[rejojer]

Update (Oct 4): this branch now has a second commit, 50caf15, with fixes from reviewing this view. Open it from the Commits tab to review it on its own. The rest of this description covers 6d23caf as shipped. 50caf15 changes:

• Flash expand prices a candidate level with the intro it will get. The next merge can no longer fold back a node whose summary is already final, which raised "dropped or changed after it was marked final" on PRML and the 2023 annual report.
• heading_at_page_start no longer takes a running header that repeats the heading as the heading opening its page. A heading now needs a Latin letter to be placed; digits alone no longer count.
• Flash expand drops a proposed child whose page and title, leading number aside, match a node already in the tree.
• The local get_document_structure tool reads client.get_tree, with its end < start clamp. LocalAPI.raw_tree is removed.
• A test runs the standard tree_parser path end to end.

---

Frozen-base review view of #541 as squash-merged to main (6d23caf), released in v0.2.21. Base review/base-f0d67c1 is main right before that merge, so this diff is exactly what #541 shipped. Against #541's own PR diff, the added and removed lines are identical; one context line differs, from #542.

Never merge. Fixes found here go to main through their own PRs.

What shipped

Read side. get_tree returns one node shape in local and cloud mode: {title, node_id, start_index, end_index, summary, text, nodes}. page_index and prefix_summary no longer appear, which is breaking for code that reads them.

• utils.unify_tree only renames fields (page_index → start_index, prefix_summary → summary). It never adds or moves a node.
• When the cloud API returns a tree without end_index, each node's end is filled from the next node's start. This costs one extra metadata request.
• A document indexed before this change keeps its own ranges and summaries, so every node's summary still matches its range.

Index side, new local indexes in standard and flash:

• Intro nodes. A parent whose first child starts on a later page gets a first child "<parent title> (intro)" holding those pages, or "Intro" when the parent has no title. This is decided by page.
• Covering ranges. A parent's range covers its whole subtree, and its summary is written from its children's summaries.
• Summaries. Standard mode now summarizes with summarize_tree, as flash does: per node, deepest first, under the concurrency cap. A parent waits only for its own children.
• Fallback. A node the model leaves unsummarized falls back to its subsection titles (parent) or its opening text (leaf), via fallback_summary. A run in which no call is answered still raises.
• Split. The standard large-node split acts on leaves only, after intro nodes are added. It no longer replaces a parent's existing subsections.

Page boundaries share, never drop. When a cut can't tell where a heading sits on a page, the page goes to both sides.

• A parent's text runs onto the page its first child starts on (own_pages). It is empty when the parent's intro holds those pages (is_intro).
• The public add_node_text and add_node_text_with_labels cut by the same rule.
• The flash Preface takes the first section's page unless that section's heading opens the page.
• A heading with no Latin letter or digit counts as not at the top of its page (heading_at_page_start).

Known issues, already tracked (no need to re-report)

1. Flash expand gives a non-Latin node no children. tree_optimize.normalize keeps only [a-z0-9], so a CJK, Arabic or Hindi title normalizes to "".
   • In propose_children and children_from_cache, such a node title equals every non-Latin proposal, so the node gets zero children.
   • Under a Latin parent, only the first non-Latin proposal survives; the rest collide in seen.
   • The check that a proposal is printed on its page passes trivially.
2. Local flash text sources differ. get_tree text comes from PyPDF2 (pages.json), while flash builds the tree and writes summaries from pdfium text. The two can extract a page differently.
3. Unknown heading position. _heading_appears_at_page_top (flash/outline_assembly/assembly.py) returns True ("drop the page") when a heading has no group slot or page. This contradicts share-never-drop in theory. A probe hit it 0 times in 514 headings over 12 PDFs.
4. Dead code. flash/outline/tree.py build_tree is exported but never called. It ends each section at next.start - 1.
5. Unused after this change. utils.get_intro_text and SUMMARY_INTRO_MAX_PAGES always yield "" on a tree built with intro nodes, since the intro holds every opening. Cleanup candidate.

Review focus

• The get_tree breaking change, and unify_tree on old cloud trees (no end_index, prefix_summary) vs new ones.
• feat(sdk): node navigation helpers get_node, get_node_parent, get_node_path, get_node_map #542's node helpers (get_node, get_node_parent, get_node_path, get_node_map, create_node_mapping) on the new shape. This covers intro nodes and the "<id>.0" ids flash expand's attach_children gives a new parent's intro.
• The invariants on every new tree: each parent's range covers its subtree, every node has a summary, and each summary matches its node's range.
• summarize_tree scheduling and the fallback, and the leaf-only split.
• Text cutting at shared page boundaries (own_pages, is_intro, add_node_text*).

Tests (from #541)

• tests/test_tree_format.py: 10 tests, each failing on main before the merge.
• 591 passed with agent frameworks, excluding test_local_chat, whose failures come from an httpx environment issue and also fail on main. 601 passed and 218 skipped without frameworks.
• 7 example PDFs indexed through flash with the model stubbed. Each stored tree meets the invariants, and get_tree returns it unchanged.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
🔒 Security Review	✅ Completed 2026-10-03T19:13:57.887020Z	6d23caf	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.