OpenMind v2 Phase 3: enterprise document ingestion and appendable knowledge#9
Merged
Merged
Conversation
…v0004 Adds the deterministic, model-free document parsing plane that later steps persist and index. - docs/v2/phase-3-document-ingestion.md: the required design document. - openmind/documents/: parser SPI (probe -> registry -> normalized ParsedDocument), a shared block builder that enforces every limit in one place, and parsers for Markdown, plain text/RST/AsciiDoc, HTML, CSV, DOCX, PDF, XLSX, OpenAPI, JSON Schema and SQL DDL. - Selection never uses the extension alone: OOXML formats must show package structure (word/document.xml, xl/workbook.xml) and PDF must show %PDF-. Exactly one parser is selected; a tie at equal priority fails loudly. - Honest statuses throughout: partial on a limit (with the exact warning), needs-ocr for an image-only PDF (no OCR is performed), encrypted, unsupported and failed - never an empty successful parse. - Security: ZIP member/size/ratio/traversal caps before any OOXML parse, defusedxml.defuse_stdlib() required for OOXML, formulas read as text and never evaluated, HTML scripts/styles/hidden elements dropped, remote $ref never fetched. - Migration v0004: segments.content_blob_hash, jobs.payload_json, document_parses and document_index. Additive and idempotent. - Fixtures: 11 invented NameCheck documents; the 5 binary ones regenerate byte-identically via scripts/build_document_fixtures.py. Parser imports stay lazy, so importing openmind.documents pulls in no document dependency and the dependency-free artifact export is unaffected.
…earch - db: document_parses / document_index reads and writes; the parse record and the index row are written INSIDE commit_revision's transaction, so a rolled back revision can never leave a parse record or a chunk list behind. - commit_revision accepts pre-minted asset/revision/segment/evidence ids. The document pipeline needs them: its vector chunks carry those ids in their metadata and are upserted before the transaction runs. - documents/pipeline.py: parse -> block blobs -> chunks -> upsert -> single transaction -> replace the predecessor's chunks. Chunk ids are derived from (logical_key, content_hash, block_key), so a retry after a failed commit overwrites the same ids instead of stranding new ones. - AssetService.get_evidence gains a second resolver: a document segment's exact block text comes from its own content blob, with NO parser rerun, so a newer parser version cannot rewrite history. An attachment reports current-source status 'not-applicable' rather than a false 'missing'. - document_rag.py: vector + token-boundary lexical + exact-identifier retrieval fused with RRF. An exact identifier is PROMOTED above semantic-only hits, and API paths get a path-aware boundary rule so /name-check does not match /name-checker. search_knowledge returns code and documents separately and states that adjacency is not a relationship. - vectorstore: documents_<id> collection, recognized by the orphan sweep and dropped by both terminate and delete via one shared prefix list. - verify_migrations updated for the v0004 head (76 checks, was 64).
…tion - documents/intake.py: the import DECISION - duplicate, revision, new_asset, possible_revision, unsupported. A filename collision with different content writes nothing and returns the exact commands that resolve it, because merging two teams' requirements.docx is not recoverable. - documents/candidates.py: deterministic candidate association. Exact file paths, code symbols, requirement ids, tickets, API paths, config keys, database objects and glossary terms first; embedding similarity only fills the remaining budget and is labelled low. Nothing is persisted and no result claims implements/refines/verifies/contradicts. - services/document_service.py: plan_import, add_document, get_document, list_documents, get_outline, search, search_knowledge, find_related_candidates. The absolute origin path is used to read bytes and goes no further - the job payload carries a staged blob hash and a filename. - jobs.py: document_ingest job type (payload keys allow-listed, a path in the payload is rejected) plus a document sweep inside the full ingest. Document discovery has its own extension set and size limit, so a .pdf can never reach walker.read_text and one file never becomes two Assets. A filtered ingest never prunes documents; an attachment is never pruned at all. - Fixed: a removed-then-reappeared document was reactivated but never re-indexed, leaving it active in the database and invisible to search. commit_document now rebuilds the projection when the index row is missing. - document_rag: a bare-identifier query returns only token matches, matching the code RAG's exact-token contract instead of padding with similar text. Full core acceptance suite: 28 passed, 0 failed, 0 skipped.
- CLI: `document add|list|show|outline|search|related` and `knowledge search`.
--asset / --logical-key / --new-asset are mutually exclusive in the parser
AND in meaning. A possible_revision exits non-zero with the resolving
commands, so a script cannot read "nothing was written" as success.
- REST: additive routes under /projects/{id}/documents plus
/projects/{id}/knowledge/search. `search` is registered before {asset_id} so
it cannot be captured as an id; conflicting targets are a 400; a
cross-workspace id is a 404. Import takes a server-side path rather than an
upload, so no 25 MB multipart body is buffered just to hash it.
- MCP: six additive READ-ONLY document tools. There is deliberately no
document-write tool - importing reads a local file, and exposing that over
MCP would let a client make the server read a path it chose. Claude Code
drives `openmind document add` through its shell instead.
Tool set is now 19: the nine core tools and the four Phase 2 asset tools are
unchanged; the six document tools are added beside them.
Seven new suites, all registered in scripts/run_acceptance.py (an unregistered tests/verify_*.py already fails the runner, so a missing document test fails the manifest rather than going quietly unrun): verify_document_registry 42 checks verify_document_parsers 190 checks verify_document_security 74 checks verify_document_ingest 102 checks verify_document_search 79 checks verify_document_cli 90 checks verify_document_adapters 93 checks Three real bugs the suites caught and that are fixed here: - DocumentBuilder.full was a silent early-exit signal: a parser that broke out of its loop produced a truncated document still labelled `parsed`. Reading it now records the truncation, because that read IS the moment content is dropped. - decode_text tried UTF-16 before the single-byte fallbacks, and a UTF-16 decode of arbitrary text SUCCEEDS whenever the length is even - turning "caf\xe9 latte" into CJK mojibake and reporting no problem. UTF-16 is now attempted only behind a BOM. - cmd_document_add mutated ok/error AFTER emitting, so a possible_revision printed `"ok": true` on stdout while exiting non-zero. Two more found while writing the checks: extract_terms fed identifier COMPONENTS back as lexical terms (so REQ-NC-017 also matched REQ-NC-018), and document segment symbols used the block key instead of the parser's explicit name (so SQL tables were unfindable and the word "sql" became a database object). CI: seven document steps in the Ubuntu full gate, a fixture-reproducibility check, the MCP smoke asserts 19 tools and no write tool, and the cross-platform smoke now covers migration to v0004 plus Markdown/DOCX/PDF/XLSX ingestion, list, outline and search. Version constant advanced to 1.3.0-dev. Artifact schema stays 1.1.0. Full core acceptance suite: 35 passed, 0 failed, 0 skipped (was 28).
- README: an "Enterprise Documents (v2 Phase 3)" section, the document row in "What It Builds", the six additive MCP tools with the reason there is no document-write tool, document commands in the quick start, the new test scripts, and a roadmap that now lists OCR and Requirement extraction as explicitly NOT implemented rather than leaving it implied. - docs/cli.md: the `document` and `knowledge` command groups, the five import decisions in a table, how Evidence survives a parser upgrade, why related results are only candidates, why OCR is absent, why Requirement extraction is Phase 4, why .openmind stays at 1.1.0, and the full table of parser resource limits with their environment variables. - docs/database-migrations.md: v0004 in the migration table, why segments.content_blob_hash and jobs.payload_json exist, why v0004 declares upgrade() instead of STATEMENTS, and the Phase 2 -> Phase 3 upgrade path. - docs/v2/phase-2-asset-model.md: a status note on its deferred list, so a reader is not told document parsing is missing when Phase 3 has shipped it. The original text is left intact as a record of what Phase 2 itself did. Verified out of band: - with docx/pypdf/openpyxl/defusedxml all blocked, the artifact export still runs, the registry still loads, Markdown/HTML/CSV still parse, and DOCX/PDF/ XLSX report dependency_unavailable for the RIGHT parser instead of falling through to one that would mangle them; - a database populated by the Phase 2 build and opened by this one migrates 3 -> 4 with zero differences across projects, jobs, file index, assets, revisions, segments, evidence, Ask history, kv and project meta; legacy code segments keep content_blob_hash='' (not backfilled), and the upgraded database then accepts and searches a document.
Adds a Result section: the full-tier acceptance numbers (36 passed, 0 failed, 1,568 individual checks), the per-suite check counts, the schema head and tool set, the five bugs the suites caught during implementation, and the two out-of-band verifications (dependency-free behaviour and the populated Phase 2 -> Phase 3 upgrade).
Smoke (all three OS) — `KeyError: 'file'`
------------------------------------------
The asset smoke step took assets[0] and read locator["file"]. Since Phase 3 the
fixture repo's README.md and docs/GLOSSARY.md are also ingested (as DOCUMENT
assets) and they sort first, so assets[0] became a document whose locator names
its target in `document`, not `file`.
That is correct new behaviour — document discovery is supposed to find .md
files — so the fix is in the check, not the code. The step now:
* selects a source-code asset explicitly, because it exists to prove the CODE
asset chain and taking assets[0] would silently retarget it;
* asserts the locator kind is source-range, so a future retarget fails loudly
instead of passing on the wrong thing;
* additionally walks EVERY asset and asserts the portable-key rule holds
whichever locator kind it uses (`file` for code, `document` for documents,
never absolute). Locally this now covers 10 source-range and 2 text-range
locators, so both kinds are exercised.
Full gate — verify_document_security, windows-traversal member name
-------------------------------------------------------------------
ZipInfo.__init__ rewrites os.sep to "/", so a member written as `..\..\x` is
stored with forward slashes on Windows and verbatim with backslashes on Linux.
The assertion compared against the REQUESTED name, so it passed on Windows and
failed on Linux.
The refusal itself was always correct on both. Fixed by comparing against what
the archive actually stored, plus seven direct checks on the _is_traversal
predicate (including the literal-backslash form Linux keeps) so the detector is
pinned independently of whatever zipfile did to the name.
verify_document_security: 74 -> 81 checks.
Full suite, every tier: 36 passed, 0 failed, 0 skipped.
The "regenerate byte-identically" gate failed on Ubuntu for
sample-requirements.docx and sample-tests.xlsx. It was not version drift — CI
resolved exactly the versions used to generate them (python-docx 1.2.0,
openpyxl 3.1.5, lxml 6.1.1).
A DOCX/XLSX is a DEFLATE archive, and the compressed bytes depend on the
platform's zlib build. Same parts in, different container bytes out. So the
check was asserting something that cannot hold across operating systems, and
the property it actually needed - "regenerating yields the same document" -
was never being tested.
The check now compares by format, because the achievable guarantee differs:
PDFs written byte by byte by the generator, no compressor involved
-> compared as exact bytes, unchanged
OOXML unpacked and compared member by member, which is the property that
matters and still catches real generator or library drift
Verified both directions locally: it passes on the committed fixtures, and
perturbing the generator (changing the DOCX core author) is still caught, now
naming the exact differing part - "sample-requirements.docx: changed=
['docProps/core.xml']" - which the byte comparison could never have told us.
Storing the archives uncompressed WOULD give true byte-identity everywhere, but
it inflates sample-requirements.docx from 37 KB to 832 KB. Not a trade worth
making for a test fixture.
Also corrects the claim in fixtures/documents/README.md that regenerating and
committing "should produce an empty diff" - true on one machine, false across
platforms - and the matching claim in the generator's docstring. Nothing depends
on the exact committed bytes: the tests hash the fixture at runtime and no
expected hash is written down anywhere.
Full suite, every tier: 36 passed, 0 failed, 0 skipped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes enterprise documents first-class OpenMind Assets: a deterministic,
model-free document-ingestion plane on top of the Phase 2
Asset/Revision/Segment/Evidence foundation.
Design document:
docs/v2/phase-3-document-ingestion.mdWhat changed
Parser plane (
openmind/documents/) — a parser SPI:DocumentProbe(magic bytes, ZIP package structure, size) → a deterministic priority-ordered
registry → one normalized
ParsedDocumentofDocumentBlocks. Ten parsers:Markdown, plain text/RST/AsciiDoc, HTML, CSV/TSV, DOCX, text-based PDF, XLSX,
OpenAPI (JSON+YAML), JSON Schema, SQL DDL.
Persistence — migration
v0004:segments.content_blob_hash,jobs.payload_json,document_parses,document_index. The parse record andthe index row are written inside
commit_revision's transaction, so a rolledback revision can never strand either.
Append workflow —
DocumentService(runtime.documents) withplan_import,add_document,get_document,list_documents,get_outline,search,search_knowledge,find_related_candidates, plus adocument_ingestjob type and document discovery inside the full ingest.Retrieval —
document_rag.py: vector + token-boundary lexical +exact-identifier, RRF-fused, in a separate
documents_<ws>collection.Adapters — 7 CLI commands (
document add|list|show|outline|search|related,knowledge search), 8 additive REST routes, 6 additive read-only MCP tools.Design decisions worth reviewing
.docxsuffix does not containword/document.xmland is not claimed. Twoparsers claiming one probe at equal priority raises rather than picking
arbitrarily.
with different content returns
possible_revision, writes nothing, andhands back the exact commands that resolve it. Merging two teams'
requirements.docxis not recoverable, so it is surfaced rather than guessed.exact text is snapshotted as its own content-addressed blob, so a newer parser
version cannot rewrite history. Existing code segments keep
content_blob_hash = ''and resolve the Phase 2 way — deliberately notbackfilled.
code_<ws>would silently change what the existingsearchtool returns, andthat contract is depended on by external MCP clients.
implements,refines,verifiesandcontradictsare absent from the vocabulary; every resultcarries
status: "candidate"with a confidence, and nothing is persisted.pypdf(BSD-3), not PyMuPDF (AGPL). The cost is real — no columnreading-order reconstruction, no table recovery — and it is paid honestly
rather than by taking on copyleft for this MIT project.
over MCP would let a client make the server read a path it chose. Claude Code
drives
openmind document addthrough its shell, where the command is visible.payload carries a staged blob hash and a filename; the payload keys are
allow-listed and a path in the payload is rejected at enqueue.
Honesty guarantees (each has a test)
needs-ocr. No OCR isperformed and no result claims it was.
partialplus a warning naming that exact limitand the observed value. Nothing is silently truncated.
hidden sheet, an unparseable vendor SQL statement — is recorded with a count.
header: value) isderived, neververbatim.not-applicable, not a falsemissing.Security
ZIP member/size/ratio/traversal caps before any OOXML parse ·
defusedxml.defuse_stdlib()required (the OOXML parsers decline without it) ·formulas read as text and never evaluated · HTML scripts, styles and hidden
elements dropped · remote
$refnever fetched · YAML viasafe_loadonly ·a parser failure fails its own document, never the worker.
Schema
Head 3 → 4. Two columns with defaults, two new tables, five indexes.
v0004declaresupgrade(conn)rather thanSTATEMENTSbecause SQLite has noADD COLUMN IF NOT EXISTS.Verified with zero data loss: a database populated by the Phase 2 build and
opened by this one migrates with no differences across projects, jobs, file
index, assets, revisions, segments, evidence, Ask history, kv and project meta.
Compatibility
/projects, never/workspaces.openmindartifactVerification
1,568 individual checks. New suites: registry 42 · parsers 190 · security 74
· ingest 102 · search 79 · CLI 90 · adapters 93.
verify_migrationsgrew 64 → 76.Two out-of-band checks: with
python-docx/pypdf/openpyxl/defusedxmlallblocked, the artifact export still runs and DOCX/PDF/XLSX report
dependency_unavailablefor the right parser; and the populated Phase 2 →Phase 3 upgrade above.
Bugs the suites caught during implementation
DocumentBuilder.fullwas a silent early-exit — a parser breaking out of itsloop produced a truncated document still labelled
parsed.decode_texttried UTF-16 before the single-byte fallbacks; a UTF-16 decodesucceeds on any even-length input, turning
caf\xe9 latteinto mojibake withno warning.
cmd_document_addmutatedok/errorafter emitting, so apossible_revisionprinted"ok": truewhile exiting non-zero.extract_termsfed identifier components back as lexical terms, soREQ-NC-017also matchedREQ-NC-018.active in the database and invisible to search.
Three Phase 2 tests asserted "the Asset tools are the only additions"; they
now assert "every registered tool is an accounted-for addition", which still
catches a stray or renamed tool without failing on every future phase.
CI
Seven document steps in the Ubuntu full gate, a fixture-reproducibility check
(the generated binary fixtures must regenerate byte-identically, or the
"unchanged document creates no revision" guarantee would be untestable), an MCP
smoke asserting 19 tools and no write tool, and cross-platform smoke covering
migration to v0004 plus Markdown/DOCX/PDF/XLSX ingestion, list, outline and
search.
Dependencies
python-docx(MIT) ·pypdf(BSD-3) ·openpyxl(MIT) ·defusedxml(PSF-2.0).All permissive, none performs telemetry or network calls, all imported lazily —
a missing one fails only its own format.
Deliberately not in this phase
Requirement / Business Rule / Design Decision / Acceptance Criterion extraction ·
semantic document classification · document-authority inference · Claim and
Relation tables · Knowledge-Graph edges · requirement-to-code traceability ·
conflict detection · OCR · COBOL/JCL/PPTX/email parsing · Jira and Confluence
connectors · branch overlays · webhooks · Bundle 2.0 · Neo4j · worker-pool
rewrite · new UI.
Known limitations
recovery (the
pypdflicence trade above).certainty —
ABC-1234andNC-100are the same lexical shape. The rule isstated in the emitted reason rather than asserted as fact.
implemented.
partial current revision). A crash between them leaves unreferenced chunks;
they are overwritten by the same content-derived ids on retry, but nothing
sweeps them proactively.
those caps are not surfaced in the response.