Skip to content

[dependency] Bump flink to 1.20.5 and 2.1.3#842

Merged
wenjin272 merged 2 commits into
apache:mainfrom
wenjin272:update-flink
Jun 13, 2026
Merged

[dependency] Bump flink to 1.20.5 and 2.1.3#842
wenjin272 merged 2 commits into
apache:mainfrom
wenjin272:update-flink

Conversation

@wenjin272

@wenjin272 wenjin272 commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Linked issue: #844 #692

Purpose of change


Commit 1 — [fix] Keep numpy off the async pool for cross-language RAG queries

Problem

A cross-language RAG query against a Python vector store (VectorStoreCrossLanguageTest) could hang intermittently in CI while passing locally. A Python store's query runs numpy (e.g. ChromaDB normalizes the query embedding via np.array). numpy releases and re-acquires the GIL during that copy, and pemja keeps a single PyThreadState; on the async executor's worker thread the re-acquire can stall, surfacing as a hang on few-core CI runners (machines with spare cores avoid the window). See #844.

Fix

Split the async RAG path so only the numpy step stays on the mailbox thread, leaving the rest async:

  • ContextRetrievalAction — for an async Python store, run embed async → normalize synchronously on the mailbox thread → query async with the pre-normalized vector, instead of running the whole query on the worker.
  • PythonVectorStore — add embedQuery / normalizeEmbedding / queryNormalized hooks; resolve the embedding model in open(); forward pre-computed vectors to the Python _embedding methods.
  • BaseVectorStore — expose getEmbeddingModel() to subclasses.
  • ChromaVectorStore + base vector_store.py — add _normalize_embeddings; _query_embedding accepts a pre-normalized ndarray (skips internal numpy) or a raw list.
  • Test — update PythonCollectionManageableVectorStoreTest for the Java-side embedding path (pre-computed embeddings + _add_embedding/_update_embedding assertions).

Commit 2 — [dependency] Bump flink to 1.20.5 and 2.1.3

Bumps pinned Flink versions: flink.1.20.version 1.20.4 → 1.20.5, flink.2.1.version 2.1.2 → 2.1.3, and aligns flink.2.0.version (2.0.2) and the e2e-integration matrix
accordingly across pom.xml, dist/pom.xml, and the integration e2e pom.


Tests

it & e2e

API

no

Documentation

  • doc-needed
  • doc-not-needed
  • doc-included

@github-actions github-actions Bot added doc-not-needed Your PR changes do not impact docs fixVersion/0.3.0 The feature or bug should be implemented/fixed in the 0.3.0 version. priority/major Default priority of the PR or issue. and removed doc-not-needed Your PR changes do not impact docs labels Jun 12, 2026
wenjin272 and others added 2 commits June 13, 2026 11:20
A Python vector store's query path runs numpy (e.g. chroma's embedding
normalization). numpy releases and re-acquires the GIL during the conversion,
which deadlocks on an async pemja worker thread since pemja keeps a single
PyThreadState. Split the async RAG query so only the numpy normalization runs
on the operator thread: embed and query stay async, normalize is sync.

- PythonVectorStore: resolve the embedding model in open(); add embedQuery /
  normalizeEmbedding / queryNormalized hooks; forward pre-computed vectors.
- BaseVectorStore: expose getEmbeddingModel to subclasses.
- ContextRetrievalAction: async embed -> sync normalize -> async query.
- ChromaVectorStore: add _normalize_embeddings; query accepts pre-normalized.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added doc-not-needed Your PR changes do not impact docs and removed doc-not-needed Your PR changes do not impact docs labels Jun 13, 2026
@wenjin272 wenjin272 merged commit 1c829cb into apache:main Jun 13, 2026
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

doc-not-needed Your PR changes do not impact docs fixVersion/0.3.0 The feature or bug should be implemented/fixed in the 0.3.0 version. priority/major Default priority of the PR or issue.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant