The schema document is searched, not extracted - #561
Open
WaylandYang wants to merge 1 commit into
Open
Conversation
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: WaylandYang <wayland0916@gmail.com>
Contributor
Author
|
Bench after this change, wide corpus, same scripts and truth as #501 / #520, fresh base each time.
The graph number is the one this PR is for, and it moved from 93 to 12. The accuracy numbers did not move, and were not expected to: chat writes SQL from the schema document, which is unchanged, and exploration still cannot see a convention that is not in the schema. The one extra right answer ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #553. Implements decision 7 of 0035 (the record itself arrives with the bench branch; its status line gets this PR's number there).
Mounting a data source ingests the schema as a markdown document so chat can
search_chunksfor a table. That document then went through extraction like prose, and the extractor filed every column name as an entity — on the wide bench corpus, 28 of 40 concept entities were column names.What changes
sources.config.extract = falsemarks a source whose documents are searched and never extracted.Source::extracts()reads it (default true; a non-boolean counts as unset, so a typo cannot silently stop a source).sync_schema_doccreates the "Data schemas" folder with it; migration0045backfills existing folders and addsskippedtograph_status.pipeline::process_documentstops after indexing and embedding for such a document and marks itskipped—nonewould read as "not yet queued" in the Library, and it never will be.documents::queue_extractionfilters these documents in SQL, so full rebuild, per-source re-extract and any future caller all skip them in one place. The two manual entry points (POST /documents/{id}/extract,POST /kbs/{id}/sources/{sid}/re-extract) refuse withsource_not_extractedso a person hears why rather than "queued 0".Not extracted/不抽取label for the status; the per-source re-extract button is hidden when the source does not extract (the per-document button's status whitelist already excluded it).Not a name rule ("folders called Data schemas never extract"): the flag is on the source, which is the thing that has the property.
Verified
Source::extractsover{},{path},true,"no",0,false.a_schema_document_is_searched_not_extracted: full-base and per-sourcequeue_extractionleave the schema document out.wide→wide-schema.mdisready / skipped, 0 entities, 0 extract jobs; both manual endpoints return 422source_not_extractedand queue nothing.Touches the ingest path #526 also wants to change; separable — this decides whether a document is extracted, #526 decides when. Does not touch
Mappings.tsx(#552) orchat.rs(#548).🤖 Generated with Claude Code