Skip to content

The schema document is searched, not extracted - #561

Open
WaylandYang wants to merge 1 commit into
devfrom
fix/the-schema-document-leaves-extraction
Open

The schema document is searched, not extracted#561
WaylandYang wants to merge 1 commit into
devfrom
fix/the-schema-document-leaves-extraction

Conversation

@WaylandYang

Copy link
Copy Markdown
Contributor

Closes #553. Implements decision 7 of 0035 (the record itself arrives with the bench branch; its status line gets this PR's number there).

Mounting a data source ingests the schema as a markdown document so chat can search_chunks for a table. That document then went through extraction like prose, and the extractor filed every column name as an entity — on the wide bench corpus, 28 of 40 concept entities were column names.

What changes

  • sources.config.extract = false marks a source whose documents are searched and never extracted. Source::extracts() reads it (default true; a non-boolean counts as unset, so a typo cannot silently stop a source). sync_schema_doc creates the "Data schemas" folder with it; migration 0045 backfills existing folders and adds skipped to graph_status.
  • pipeline::process_document stops after indexing and embedding for such a document and marks it skippednone would read as "not yet queued" in the Library, and it never will be.
  • documents::queue_extraction filters these documents in SQL, so full rebuild, per-source re-extract and any future caller all skip them in one place. The two manual entry points (POST /documents/{id}/extract, POST /kbs/{id}/sources/{sid}/re-extract) refuse with source_not_extracted so a person hears why rather than "queued 0".
  • Library: a Not extracted / 不抽取 label for the status; the per-source re-extract button is hidden when the source does not extract (the per-document button's status whitelist already excluded it).

Not a name rule ("folders called Data schemas never extract"): the flag is on the source, which is the thing that has the property.

Verified

  • Unit: Source::extracts over {}, {path}, true, "no", 0, false.
  • DB test a_schema_document_is_searched_not_extracted: full-base and per-source queue_extraction leave the schema document out.
  • End to end on the bench base: fresh KB, mount widewide-schema.md is ready / skipped, 0 entities, 0 extract jobs; both manual endpoints return 422 source_not_extracted and queue nothing.
  • Migration applied on the bench base: both existing "Data schemas" folders backfilled, CHECK widened.

Touches the ingest path #526 also wants to change; separable — this decides whether a document is extracted, #526 decides when. Does not touch Mappings.tsx (#552) or chat.rs (#548).

🤖 Generated with Claude Code

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: WaylandYang <wayland0916@gmail.com>
@WaylandYang

Copy link
Copy Markdown
Contributor Author

Bench after this change, wide corpus, same scripts and truth as #501 / #520, fresh base each time.

before (#501 run, base 01a08375) after #553 (base 01a08642)
concept entities on the graph 93 (mostly column names, accumulated over runs) 12, all from exploration
schema document extracted skipped
exploration: metrics right / wrong 0 / 5 0 / 6
exploration: definitions covered 0/18 0/18
chat, no confirmed definitions 1/18 (6%) 2/18 (11%)

The graph number is the one this PR is for, and it moved from 93 to 12. The accuracy numbers did not move, and were not expected to: chat writes SQL from the schema document, which is unchanged, and exploration still cannot see a convention that is not in the schema. The one extra right answer (test_orders) is inside single-run noise. The lever for those two rows is #554#556; the seeded upper bound on the same corpus is 17/18 (#520).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The schema document is extracted as if it were prose

1 participant