[FEATURE] Add OpenSearch (opensearch://) vector store support alongside AOSS - #397
Open
noel-improv wants to merge 11 commits into
Open
[FEATURE] Add OpenSearch (opensearch://) vector store support alongside AOSS#397noel-improv wants to merge 11 commits into
noel-improv wants to merge 11 commits into
Conversation
Add an opensearch:// connection string for self-managed (non-AWS) OpenSearch instances, alongside the existing aoss:// one. The local path skips AWS SigV4 entirely: it authenticates with HTTP basic auth from OPENSEARCH_USERNAME/OPENSEARCH_PASSWORD (env vars or GraphRAGConfig), or with no auth if neither is set. index_exists() now threads client_kwargs through to the OpenSearch client, so callers can override use_ssl/verify_certs for a plain-HTTP or self-signed local endpoint. The existing Classic knn_vector mapping is reused unchanged.
noel-improv
marked this pull request as ready for review
July 15, 2026 22:45
…gs last Docs and docstrings previously called the new non-AOSS path "self-managed" or "local" OpenSearch, which understates it -- OpenSearch runs anywhere (local, EC2, on-prem), and "local" only covers one case. Switch to plain "OpenSearch" for the open source path and "Amazon OpenSearch Serverless (AOSS)" for the managed one, matching the existing env var naming convention. Also make client_kwargs consistently the last parameter across create_opensearch_vector_client, OpenSearchIndex.for_index, and the OpenSearchIndex model fields -- it was sometimes before is_local, sometimes after.
noel-improv
changed the title
mykola-pereyma
left a comment
mykola-pereyma
left a comment
Collaborator
There was a problem hiding this comment.
Few suggestions:
-
Could we rename
is_localtois_serverless(inverted, default True)? The flag means "use AOSS SigV4 auth" vs "use basic auth or no auth" — not whether the instance is local. An EC2-hosted OpenSearch cluster is not local but would use this path.is_serverlessmatches AWS terminology and reads naturally at call sites. -
The docs page and README should mention how to connect over plain HTTP or with self-signed certs — local Docker users will hit SSL errors with the https default. Something like: "For plain HTTP, use
opensearch://http://localhost:9200. For self-signed TLS, passclient_kwargs={"verify_certs": False}to the factory." -
Worth adding a note that
opensearch://is for instances without AWS authentication (basic auth or no auth). A user with a managed OpenSearch Service domain (non-serverless but still on AWS) might try this prefix and get auth failures — a sentence clarifying that managed OpenSearch Service with IAM auth is not yet supported would save confusion.
…SS detection The flag distinguishes Amazon OpenSearch Serverless (AOSS, SigV4 auth) from an OpenSearch endpoint that isn't AOSS (basic auth or no auth), which is what the auth path actually keys off -- not whether the instance is local. An EC2-hosted OpenSearch cluster isn't local but takes the non-serverless path. is_serverless matches AWS terminology and reads at call sites; it defaults to True (AOSS), so aoss:// stays the zero-config default and opensearch:// sets it False. - Invert is_local -> is_serverless across create_os_client/async, index_exists, create_opensearch_vector_client, OpenSearchIndex, and the factory. - Rename the OPENSEARCH_LOCAL constant to OPENSEARCH and _local_http_auth to _basic_http_auth. - Addresses review feedback on awslabs#397.
…S-auth scope for opensearch://
Note that opensearch:// targets instances without AWS authentication (basic auth
or no auth) and that a managed Amazon OpenSearch Service domain using IAM auth is
not supported through this prefix yet. Document plain-HTTP endpoints
(opensearch://http://...) and disabling certificate verification for self-signed
TLS via client_kwargs={'verify_certs': False}. Addresses review feedback on awslabs#397.
acarbonetto
reviewed
Jul 16, 2026
acarbonetto
left a comment
acarbonetto
left a comment
Collaborator
There was a problem hiding this comment.
looks good - I agree that we need to move away from Open Source == Local. OpenSearch can be deployed in more ways than just a localhost. IF we only want to support localhost (for now) and raise a follow-up ticket to support more options later, that's fine. But the language in the code should be updated now.
Follows review on awslabs#397: the flag selects AWS SigV4 auth vs basic/no auth, so is_sigv4_auth names the mechanism directly at call sites. Same truth values as before (True for AOSS), no behavior change.
…view Rename the section to "Connecting to an OpenSearch vector store", link the opensearch-py client, note OpenSearch's SSL-on default, and mark managed Amazon OpenSearch Service (IAM/SigV4) as not yet supported, tracked in awslabs#407. Addresses review on awslabs#397.
…-vector-store-AN-3253 # Conflicts: # lexical-graph/src/graphrag_toolkit/lexical_graph/config.py # lexical-graph/src/graphrag_toolkit/lexical_graph/storage/vector/opensearch_vector_indexes.py # lexical-graph/tests/unit/test_config.py
acarbonetto
reviewed
Jul 17, 2026
Pass opensearch:// endpoints through without forcing an https:// scheme. Derive the clients' use_ssl default from the endpoint scheme, since opensearch-py only honors the scheme for https:// and a plain-HTTP endpoint would otherwise still attempt TLS; an explicit use_ssl in client_kwargs still wins. Reject an empty endpoint instead of silently falling through to the dummy vector store. Trim docs that duplicated opensearch-py documentation, link it directly, and add a limitations section referencing awslabs#407. Add TODO(awslabs#407) markers at the auth branches and extract does_index_exist for readability.
…handling Address correctness issues in the non-AOSS OpenSearch client path: - Always connect over TLS when signing with SigV4. use_ssl was derived from the endpoint scheme, so an aoss://http://... endpoint would send the signed request (AWS key id and signature) over plaintext. SigV4 now forces TLS regardless of scheme. - Match the http:// scheme case-insensitively. opensearch-py lowercases the scheme, so HTTP://host previously kept use_ssl=True and failed the handshake against a plaintext port. - Warn when credentials would be sent over plain HTTP. - Drop a caller-supplied is_sigv4_auth in the factory before forwarding kwargs, which otherwise collided with the derived keyword and raised TypeError. - Guard the aoss:// branch against an empty endpoint, mirroring the opensearch:// branch, instead of building a client against the bogus host 'https://'. - Make is_sigv4_auth keyword-only in for_index, create_opensearch_vector_client, and index_exists so a positional client_kwargs call keeps forwarding client tuning and the flag can never be passed positionally. - Close the async client when the index is not yet available and the vector client is recreated, so the retry loop no longer leaks AsyncOpenSearch connection pools while the index finishes provisioning. - Document that a default local OpenSearch uses a self-signed certificate, so the quickstart needs verify_certs=False or a trusted CA.
…opensearch:// Recognize a self-managed OpenSearch endpoint by its http:// or https:// scheme instead of requiring an opensearch:// prefix. aoss:// (and bare AOSS domains) still route to SigV4; anything else is treated as a non-AOSS cluster. The factory stays gated on the http(s):// scheme rather than claiming every non-aoss:// string, because it runs first in the vector-store dispatch and a blind catch-all would swallow postgres://, neptune-graph://, and s3vectors://. A stale opensearch:// string now falls through to a clear 'Unrecognized vector store info' error rather than mis-routing. Docs and README updated to the scheme-based form; tests cover the http(s):// endpoints, guard that a bare AOSS domain keeps SigV4, and that opensearch:// and schemeless endpoints no longer match.
oussamahansal
approved these changes
Jul 24, 2026
acarbonetto
requested changes
Jul 27, 2026
acarbonetto
left a comment
acarbonetto
left a comment
Collaborator
There was a problem hiding this comment.
Want to verify this manually
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds an
opensearch://connection string for an OpenSearch endpoint that isn't Amazon OpenSearch Serverless (AOSS) — a local Docker or Finch container, an EC2 instance, or any other OpenSearch deployment — alongside the existingaoss://one. Today the vector store factory only recognizes AOSS endpoints, and the client creation code hardcodes AWS SigV4 signing, so there's no way to point the toolkit at a plain OpenSearch instance without AWS credentials.Changes
OpenSearchVectorIndexFactoryrecognizes anopensearch://<endpoint>prefix and routes it through a newis_localflag onOpenSearchIndex, instead of AOSS detection.create_os_client/create_os_async_clientskip AWS SigV4 on this path. They authenticate with HTTP basic auth fromGraphRAGConfig.opensearch_username/opensearch_password(backed byOPENSEARCH_USERNAME/OPENSEARCH_PASSWORD), or with no auth if neither is set, and log a warning if only one of the two is set.index_exists()now acceptsclient_kwargs, threaded fromOpenSearchIndex.client_kwargs, so a plain-HTTP or self-signed endpoint can overrideuse_ssl/verify_certs(this path previously always forceduse_ssl=True, which the AOSS-only code never needed to override).is_localalso selectsis_aoss=Falseon the underlying vector client, so bulk ingest callsrefresh()after writes, matching OpenSearch's own behavior instead of AOSS's.engine: faiss/nmslib) unchanged — no new mapping logic needed.Problem
VectorStoreFactory.for_vector_store()has no way to target an OpenSearch endpoint that isn't Amazon OpenSearch Serverless; every code path assumes AWS SigV4 and an AOSS endpoint.Related issue (if any): #
noel-improv#18 stacks on top of this branch (deterministic AOSS Classic/NextGen generation detection). It's approved and will be rebased onto
mainand opened here once this merges.Testing
pytest)Unit tests cover the factory's
opensearch://detection, the basic-auth/no-auth/partial-credential client paths, andclient_kwargsthreading throughindex_exists(). Also ran a live smoke test against two Finch-hosted OpenSearch containers: an unsecured plain-HTTP instance (index creation, bulk ingest, query, confirmingrefresh()fires) and a basic-auth instance with a real admin password (confirming both successful auth and rejection of a wrong password).Checklist
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.