Skip to content

feat: ElasticsearchDocumentStore with offset pagination - #3691

Open
davidsbatista wants to merge 12 commits into
mainfrom
feat/elasticsearch-offset-pagination
Open

feat: ElasticsearchDocumentStore with offset pagination#3691
davidsbatista wants to merge 12 commits into
mainfrom
feat/elasticsearch-offset-pagination

Conversation

@davidsbatista

@davidsbatista davidsbatista commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Related Issues

Proposed Changes:

  • added _skip_unique_value() and an async version to emulate offset with cursor

How did you test it?

  • integrations tests

Checklist

@github-actions github-actions Bot added integration:elasticsearch type:documentation Improvements or additions to documentation labels Jul 29, 2026
@github-actions

github-actions Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Coverage report (elasticsearch)

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  integrations/elasticsearch/src/haystack_integrations/document_stores/elasticsearch
  document_store.py 1842, 1853-1876, 1887-1900, 1904-1908, 1919-1932, 1940-1953, 1990-1999, 2025-2034
Project Total  

This report was generated by python-coverage-comment-action

@davidsbatista
davidsbatista requested a review from sjrl July 29, 2026 15:47
@davidsbatista
davidsbatista marked this pull request as ready for review July 29, 2026 15:47
@davidsbatista
davidsbatista requested a review from a team as a code owner July 29, 2026 15:47
Comment on lines -1860 to -1861
of the metadata field's own value (not the document content). NOTE: The matching is done with a server-side
script to accomplish the substring matching on the value of the field and this operation is quite expensive

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we bring back this note to the method?

metadata_field: str,
search_term: str | None = None,
from_: int = 0,
size: int = 10,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you plan on adding the filter param in a separate PR?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since this one is not approved yet I can add it here already, if you're OK with it?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah go for it

@davidsbatista
davidsbatista requested a review from sjrl August 10, 2026 10:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

integration:elasticsearch type:documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants