fix(docling): stringify binary_hash to prevent OpenSearch long overflow - #3698
fix(docling): stringify binary_hash to prevent OpenSearch long overflow#3698SyedShahmeerAli12 wants to merge 21 commits into
Conversation
Implements MariaDBDocumentStore backed by MariaDB 11.7+ native VECTOR support with MHNSW indexing and full-text keyword search. - Full DocumentStore protocol: write_documents (FAIL/OVERWRITE/SKIP), filter_documents, delete_documents, count_documents - Vector similarity via VEC_DISTANCE_COSINE / VEC_DISTANCE_EUCLIDEAN - Full-text keyword search via MATCH ... AGAINST (NATURAL LANGUAGE MODE) - Haystack metadata filtering converted to JSON_EXTRACT SQL expressions - MariaDBEmbeddingRetriever and MariaDBKeywordRetriever with FilterPolicy - 80 tests: 68 unit + 12 integration (all verified against MariaDB 11.7) - GitHub Actions workflow with MariaDB 11.7 service container Closes deepset-ai#2340
- Make mariadb C extension import lazy so API reference builds without requiring libmariadb-dev in the docs environment - Fix Docker health check to use --su-mysql flag required by MariaDB 11.7 - Add mariadb (LGPL-2.1) to license compliance exclusion list
- Add skip-install=true to default hatch env so docs build does not attempt to compile the mariadb C extension (libmariadb-dev not available in the API reference runner). The pydoc search_path already points to src/ so modules are importable without installation. - Switch service container health check from healthcheck.sh (unreliable in some MariaDB 11.7 images) to mysqladmin ping which is more robust.
…THCHECK The mariadb:11.7 Docker image ships with a HEALTHCHECK instruction. GitHub Actions automatically waits for it when no custom options override it. Custom health-cmd variants (healthcheck.sh, mysqladmin) were failing because the slim runner environment handles them differently.
…enSearch long overflow
|
Hi @SyedShahmeerAli12, thanks for your interest in contributing to Haystack! 🙏 This is an automated message to help us keep the review queue healthy. |
github-actions
Bot
added
topic:CI
integration:docling
type:documentation
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
|
Closing reopening with a clean branch that doesn't carry unrelated MariaDB commits. |
1 participant
Related Issues
Proposed Changes
MetaExtractorpreviously stored Docling'sorigin.binary_hashas an integer in document metadata. Sincebinary_hashis an unsigned 64-bit integer, values greater than2^63 - 1break document stores that map numeric metadata to signed 64-bit types (e.g. OpenSearchlong), causing:Following @julian-risch's guidance on #3604,
binary_hashis now always stored as a string via a new_stringify_binary_hash()recursive helper applied in bothextract_dl_doc_metaandextract_chunk_meta.Reproduction
Before fix —
binary_hashreturned asint, overflows OpenSearchlong:# model_dump() returns: {binary_hash: 9768961288489567249} � int, rejected by OpenSearchAfter fix — stored as
str, no overflow:# _stringify_binary_hash() returns: {binary_hash: 9768961288489567249} � safe keywordTesting
TestMetaExtractortests passruff checkcleanNotes
binary_hashwasint— aligns with maintainer suggestion for a major release