Add MariaDB Vector DocumentStore integration by SyedShahmeerAli12 · Pull Request #3565 · deepset-ai/haystack-core-integrations

SyedShahmeerAli12 · 2026-07-08T13:59:38Z

Summary

Part of #2340

Implements a complete MariaDB document store integration using MariaDB 11.7+ native VECTOR support.

MariaDBDocumentStore full DocumentStore protocol (write_documents, filter_documents, delete_documents, count_documents)
Vector similarity search using VEC_DISTANCE_COSINE / VEC_DISTANCE_EUCLIDEAN with MHNSW indexing
Full-text keyword search via MATCH ... AGAINST (IN NATURAL LANGUAGE MODE) on a FULLTEXT index
Haystack metadata filtering converted to JSON_UNQUOTE(JSON_EXTRACT(...)) SQL expressions with parameterized queries
MariaDBEmbeddingRetriever and MariaDBKeywordRetriever with FilterPolicy support
DuplicatePolicy support: FAIL (INSERT), OVERWRITE (upsert via ON DUPLICATE KEY UPDATE), SKIP (INSERT IGNORE)
Lazy connection with reconnect on ping failure

Tests

80 tests total all passing:

68 unit tests (mocked DB, no external dependency)
12 integration tests verified against a real MariaDB 11.7 Docker container

CI

Added .github/workflows/mariadb.yml with a MariaDB 11.7 service container, matching the pattern used by pgvector.

How to run locally

docker run -d --name mariadb-haystack \
  -e MARIADB_ROOT_PASSWORD=password \
  -e MARIADB_DATABASE=haystack \
  -p 3306:3306 mariadb:11.7

cd integrations/mariadb
pip install -e .
MARIADB_USER=root MARIADB_PASSWORD=password pytest tests/ -m integration

socket-security · 2026-07-08T14:01:37Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	pypi/mariadb@1.1.14

View full report

github-actions · 2026-07-08T14:02:41Z

Coverage report (tavily)

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
integrations/tavily/src/haystack_integrations/components/fetchers/tavily
tavily_fetcher.py
Project Total

_{This report was generated by python-coverage-comment-action}

Implements MariaDBDocumentStore backed by MariaDB 11.7+ native VECTOR support with MHNSW indexing and full-text keyword search. - Full DocumentStore protocol: write_documents (FAIL/OVERWRITE/SKIP), filter_documents, delete_documents, count_documents - Vector similarity via VEC_DISTANCE_COSINE / VEC_DISTANCE_EUCLIDEAN - Full-text keyword search via MATCH ... AGAINST (NATURAL LANGUAGE MODE) - Haystack metadata filtering converted to JSON_EXTRACT SQL expressions - MariaDBEmbeddingRetriever and MariaDBKeywordRetriever with FilterPolicy - 80 tests: 68 unit + 12 integration (all verified against MariaDB 11.7) - GitHub Actions workflow with MariaDB 11.7 service container Closes deepset-ai#2340

- Make mariadb C extension import lazy so API reference builds without requiring libmariadb-dev in the docs environment - Fix Docker health check to use --su-mysql flag required by MariaDB 11.7 - Add mariadb (LGPL-2.1) to license compliance exclusion list

- Add skip-install=true to default hatch env so docs build does not attempt to compile the mariadb C extension (libmariadb-dev not available in the API reference runner). The pydoc search_path already points to src/ so modules are importable without installation. - Switch service container health check from healthcheck.sh (unreliable in some MariaDB 11.7 images) to mysqladmin ping which is more robust.

…THCHECK The mariadb:11.7 Docker image ships with a HEALTHCHECK instruction. GitHub Actions automatically waits for it when no custom options override it. Custom health-cmd variants (healthcheck.sh, mysqladmin) were failing because the slim runner environment handles them differently.

anakin87

I took a first look and found some points to address and some others to discuss.

anakin87 · 2026-07-21T08:13:55Z

+# skip-install prevents hatch from installing the project (and thus the mariadb C extension)
+# in the docs environment. The pydoc search_path points to src/ so modules are found directly.


what's the problem here?

mariadb==1.1.14 uses distutils internally, which is removed in Python 3.14 setuptools provides the shim to keep the test env working.

This seems like a workaround. Can we use the solution suggested in https://mariadb.com/docs/connectors/mariadb-connector-python/faq#modulenotfounderror-no-module-named-packaging?

Since users will need packaging to use mariadb, I'd add this package to main dependencies (not test). Does it make sense?

anakin87 · 2026-07-21T08:57:19Z

+    blob_mime_type VARCHAR(255),
+    meta JSON,
+    FULLTEXT KEY content_ft_idx (content)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4


We should also set the distance function at table creation.

Also, let's make distance not changeable after table creation:

Declare DISTANCE explicitly. The default is euclidean, and a query using a different distance function than the one the index was built for cannot use the index, it falls back to a full table scan.

See https://mariadb.com/docs/server/reference/sql-structure/vectors/create-table-with-vectors

anakin87 · 2026-07-21T09:10:48Z

+{where_clause}
+ORDER BY score DESC
+LIMIT ?
+"""


this might also return irrelevant documents if relevant ones are less than top_k
we should also apply matching on WHERE or filtering by score

SyedShahmeerAli12 · 2026-07-21T11:01:00Z

Addressed all the straightforward review comments. The HNSW index design (create_vector_index flag) and the lazy import approach are still open for discussion

anakin87

There are still some comments to address.

Please request my review when they are fixed. In the meantime, ask questions if needed.

SyedShahmeerAli12 · 2026-07-23T11:31:49Z

ruff auto-detects mariadb and haystack as separate import sections, so import mariadb must go in its own block after all from haystack imports.
Every attempt I made grouped them together, which ruff kept rejecting until I let --fix show me the exact layout it expected.

SyedShahmeerAli12 · 2026-08-03T10:11:53Z

I mistakently Referenced some PRs here , Well I have cleaned the mess the Code is opened for review

anakin87 · 2026-08-03T15:30:04Z

+"""
+
+
+class MariaDBDocumentStore(DocumentStore):


Suggested change

class MariaDBDocumentStore(DocumentStore):

class MariaDBDocumentStore:

DocumentStore is a Python protocol. We generally don't inherit from it.
To stick to the protocol, only implementing the necessary methods is required (which you're already doing).

anakin87 · 2026-08-03T15:38:10Z

+    meta JSON,
+    FULLTEXT KEY content_ft_idx (content),
+    VECTOR INDEX vec_idx (embedding) COMMENT 'MHNSW(distance={distance})'
+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4


this does not seem to match the syntax in https://mariadb.com/docs/server/reference/sql-structure/vectors/create-table-with-vectors

anakin87 · 2026-08-03T16:04:37Z

+        filters: dict[str, Any] | None = None,
+        top_k: int = 10,
+        score_threshold: float | None = None,
+        vector_function: str | None = None,


Searches using a different distance function will not be able to use a vector index

I'd simply not allow users to change the distance function at runtime, in other words by removing this parameter

anakin87 · 2026-08-03T16:09:17Z

+        user: Secret | str = Secret.from_env_var("MARIADB_USER"),
+        password: Secret | str = Secret.from_env_var("MARIADB_PASSWORD"),


Suggested change

user: Secret | str = Secret.from_env_var("MARIADB_USER"),

password: Secret | str = Secret.from_env_var("MARIADB_PASSWORD"),

user: Secret = Secret.from_env_var("MARIADB_USER"),

password: Secret = Secret.from_env_var("MARIADB_PASSWORD"),

We don't want to support strings

anakin87 · 2026-08-03T16:10:11Z

+# ---------------------------------------------------------------------------
+# Helper: build a store with a mocked DB connection
+# ---------------------------------------------------------------------------


Suggested change

# ---------------------------------------------------------------------------

# Helper: build a store with a mocked DB connection

# ---------------------------------------------------------------------------

Remove all these separation comments

anakin87

There are still a few rough edges to address in this PR, including some points that were left unresolved from the previous review.

Since this is a new integration, I'd like to see a more thorough pass: try it yourself, refine the implementation, and make sure to address the existing feedback.
Otherwise, I may need to deprioritize further review.

SyedShahmeerAli12 · 2026-08-04T11:01:16Z

Hi @anakin87 , I've addressed all of your review comments:

Removed DocumentStore protocol inheritance
Fixed the import order
Updated the vector index SQL to use VECTOR INDEX (embedding) (removed COMMENT 'MHNSW(...)')
Added HAVING score > 0 to the keyword search to filter out zero-score results
Changed user/password to use Secret only (removed the plain string fallback)
Removed the vector_function parameter from _embedding_retrieval (and the retriever)
Moved packaging>=21.3 to the main dependencies
Removed all section separator comments from the tests
Updated all test fixtures to use Secret.from_token()

I also verified everything against a real MariaDB 11.7 Docker instance all 120 tests are passing (65 unit + 55 integration).

SyedShahmeerAli12 requested a review from a team as a code owner July 8, 2026 13:59

SyedShahmeerAli12 requested review from anakin87 and removed request for a team July 8, 2026 13:59

github-actions Bot added topic:CI integration:tavily type:documentation Improvements or additions to documentation labels Jul 8, 2026

SyedShahmeerAli12 added 2 commits July 8, 2026 19:07

SyedShahmeerAli12 force-pushed the feat/mariadb-document-store-2340 branch from f71d1cd to ff12b6a Compare July 8, 2026 14:09

SyedShahmeerAli12 added 6 commits July 8, 2026 19:14

fix(mariadb): lint errors and test env setup

8afc968

fix(mariadb): remaining lint errors and add libmariadb-dev install

7bc8914

fix(mariadb): fix Python 3.14 distutils, S608/EM102/RUF100 lint

90aea75

fix(mariadb): use Any for cursor/connection to fix mypy union-attr

fd1ae83

anakin87 requested changes Jul 21, 2026

View reviewed changes

SyedShahmeerAli12 added 3 commits July 21, 2026 15:29

fix

b0a7f31

fix

b013ead

fix

0eca31c

SyedShahmeerAli12 requested a review from anakin87 July 21, 2026 11:01

add create_vector_index flag

269eeca

anakin87 requested changes Jul 23, 2026

View reviewed changes

fix imports and tests

8f91d92

SyedShahmeerAli12 force-pushed the feat/mariadb-document-store-2340 branch from 93e2728 to 8f91d92 Compare July 23, 2026 10:10

SyedShahmeerAli12 added 3 commits July 23, 2026 15:16

fix lint and MRO

f864931

fix lint and test ordering

4c0ece1

fix lint and embedding comparison

a8dc6bc

SyedShahmeerAli12 added 4 commits July 23, 2026 15:55

fix import order, forward ref, and != None filter

1f6003a

fix

97f394e

fix

21aa3ea

fix

c4457ac

SyedShahmeerAli12 requested a review from anakin87 July 23, 2026 11:31

Merge branch 'main' into feat/mariadb-document-store-2340

3fe0a05

This was referenced Jul 30, 2026

fix(docling): stringify binary_hash to prevent OpenSearch long overflow #3698

Closed

fix(docling)!: store binary_hash as a string to avoid 64-bit overflow #3699

Merged

feat(qdrant): expose RRF k and weights parameters in QdrantHybridRetriever #3702

Merged

github-actions Bot mentioned this pull request Aug 3, 2026

feat/microsoft-sharepoint: add region parameter to MSSharePointRetriever #3722

Merged

8 tasks

anakin87 reviewed Aug 3, 2026

View reviewed changes

anakin87 requested changes Aug 3, 2026

View reviewed changes

SyedShahmeerAli12 added 3 commits August 4, 2026 15:47

fix

76b729d

fix

074813a

fix

d7a7ca2

SyedShahmeerAli12 requested a review from anakin87 August 4, 2026 11:01

github-actions Bot mentioned this pull request Aug 6, 2026

feat: add Python 3.14 support for WatsonX integration #3744

Merged

		# skip-install prevents hatch from installing the project (and thus the mariadb C extension)
		# in the docs environment. The pydoc search_path points to src/ so modules are found directly.

	class MariaDBDocumentStore(DocumentStore):
	class MariaDBDocumentStore:

		user: Secret \| str = Secret.from_env_var("MARIADB_USER"),
		password: Secret \| str = Secret.from_env_var("MARIADB_PASSWORD"),

	# ---------------------------------------------------------------------------
	# Helper: build a store with a mocked DB connection
	# ---------------------------------------------------------------------------

Uh oh!

Conversation

SyedShahmeerAli12 commented Jul 8, 2026 • edited by anakin87 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Tests

CI

How to run locally

Uh oh!

socket-security Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 8, 2026

Coverage report (tavily)

Uh oh!

anakin87 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SyedShahmeerAli12 Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

SyedShahmeerAli12 commented Jul 21, 2026

Uh oh!

anakin87 left a comment

Choose a reason for hiding this comment

Uh oh!

SyedShahmeerAli12 commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SyedShahmeerAli12 commented Aug 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anakin87 left a comment

Choose a reason for hiding this comment

Uh oh!

SyedShahmeerAli12 commented Aug 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

SyedShahmeerAli12 commented Jul 8, 2026 •

edited by anakin87

Loading

socket-security Bot commented Jul 8, 2026 •

edited

Loading

SyedShahmeerAli12 Jul 21, 2026 •

edited

Loading

SyedShahmeerAli12 commented Jul 23, 2026 •

edited

Loading