chore(release): Upgrade self-hosted Weaviate to 1.39

[dudanogueira]

Summary

Upgrades the bundled self-hosted Weaviate server from 1.27.0 to 1.39.0 (the current stable release). Deployment config + a telemetry header; no behavior change to the VDB code path.

• Bump the image to cr.weaviate.io/semitechnologies/weaviate:1.39.0 in docker/docker-compose.yaml, docker/docker-compose.middleware.yaml, and docker/docker-compose-template.yaml. This also moves the pull from Docker Hub to cr.weaviate.io, the registry Weaviate publishes to directly (the tag Weaviate's own docs point at).
• Add the missing WEAVIATE_GRPC_ENDPOINT / EXPOSE_WEAVIATE_GRPC_PORT entries to api/.env.example, the integration-test env, and middleware.env.example. gRPC (50051) has been required since 1.27; the examples never documented it.
• Send an X-Weaviate-Client-Integration: dify/<version> header so Weaviate can attribute traffic to Dify in its server-side telemetry, matching what other official Weaviate integrations emit. Best-effort — any failure (including client versions without the integrations API) is swallowed and never breaks client init.

Rebased twice, and the scope shrank both times — worth a skim if you reviewed an earlier revision:

• weaviate-client (4.20.5 → 4.22.0) was dropped: chore(deps): bump the vdb group with 9 updates #39911 landed that bump on main in the meantime, so this PR no longer touches pyproject.toml / uv.lock. What remains is the server upgrade; the client on main is already the version this targets.
• An in-repo migration guide under docs/weaviate/ was dropped: chore: remove obsolete development docs #40098 removed that directory as obsolete development docs, and re-adding it would walk that decision back. The upgrade path belongs in dify-docs instead — see below.

Why 1.39.0

1.39.0 ships no breaking changes (release notes) and is the newest minor, which is where Weaviate's bug and security backports land — deployments more than ~3 minors behind the newest release fall out of that window, and 1.27 is 12 minors behind. The storage format and Dify's collection layout (Vector_index_<dataset_id>_Node, self-provided named vector default, cosine) are unchanged, so existing data is forward compatible with no re-indexing.

Upgrade path for existing deployments

Fresh deployments are unaffected. Existing self-hosted 1.27 users cannot jump straight to 1.39 — skipping minors is untested upstream. The supported path is one minor at a time, landing on the latest patch of each, with a graceful stop between rungs (a hard kill leaves the HNSW commit log truncated: objects survive, but near_vector recall silently drops until the index rebuilds).

That procedure is user-facing documentation, not repo documentation, so it belongs in dify-docs alongside the existing Weaviate migration guide — whose compatibility matrix currently stops at "tested up to 1.33.1". I'm happy to open that PR (en + zh + ja) and link it here; say the word if you'd rather it land before or alongside this one.

Testing

• weaviate-client 4.22.0 (the version on main) verified against a live 1.39.0 server with a Dify-shaped collection: batch insert, near_vector, BM25, hybrid, byte-exact vector round-trip and delete all pass, over both HTTP and gRPC.
• providers/vdb/vdb-weaviate unit tests: 47 passed.
• docker-compose.yaml regenerated from docker-compose-template.yaml and verified in sync.
• The upgrade ladder itself was validated end-to-end on a seeded 1.27.0 volume through 1.38.2 — object count, byte-exact vectors, near_vector recall and BM25 checked at every rung, plus a hard-kill test that reproduces the truncated-HNSW failure described above. That run predates 1.39.0.

Screenshots

Before	After
N/A — backend / deployment config only (no UI)	N/A

Checklist

• This change requires a documentation update, included: Dify Document
• I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
• I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
• I ran make lint && make type-check (backend) — both pass. Frontend pnpm exec vp staged N/A (no frontend changes).

Fixes #38211

[41tair]

Hi, thanks for your contribution! Would you like to add some migration docs to docs ? Since this upgrade spans multiple versions, I think having a migration guide is necessary.

[dudanogueira]

hi there! Sure thing. I will still run some tests, but I do not expect and breaking change. We advise to no jump versions. For 1.27 -> 1.27.latest -> 1.28.latest -> etc etc

I will mark it as draft while I work on those tests.

Also, I am working on exposing some other options, such as compression, different index options and multi tenant support. This can help reducing resource consumption and large scale scenarios. Also, an integration for memory with our recently launched memory layer.

Thanks!

[dudanogueira]

@41tair thanks for the nudge — migration guide added: docs/weaviate/WEAVIATE_MIGRATION_GUIDE/1.27-to-1.38.md (the existing 1.19 → 1.27 README now links to it).

I validated the upgrade end-to-end rather than just documenting it. Seeded Dify-shaped data (a Vector_index_*_Node collection, self-provided named vector default, cosine, BYO vectors) on 1.27.0, then stepped the server one minor at a time on the same volume, verifying at every rung:

1.27.0 → 1.27.27 → 1.28.16 → 1.29.11 → 1.30.23 → 1.31.22 → 1.32.27 → 1.33.18 → 1.34.20 → 1.35.23 → 1.36.21 → 1.37.12 → 1.38.2 ✅

At each step the object count, exact properties, exact vectors, near_vector search and BM25 all survived. Existing data is forward-compatible — no re-indexing.

Key things the guide calls out (all surfaced during testing):

• One minor at a time, latest patch each — matches Weaviate's official guidance; skipping minors is unsupported. (A direct 1.27→1.38 jump happened to work in my controlled test, but it's not the recommended path.)
• Always run the latest patch and stay near the newest minor — patch releases get the most relevant fixes, and Weaviate backports them to the latest patch of the newest minor and ~3 minors before it; drifting further behind falls out of that window.
• Stop Weaviate gracefully before upgrading — this was the biggest finding. A hard kill (SIGKILL) between versions leaves the HNSW commitlog incomplete on reload: objects/vectors survive in the durable store, but near_vector recall drops until the index rebuilds. A graceful SIGTERM stop (docker compose stop/down, not kill/rm -f) preserves the vector index fully.
• Post-restart readiness gap (~1.33+) — the class index mounts a moment after /v1/meta reports ready; a query in that window fails with non-existing index. Wait/retry before relying on search.

Also bumped the bundled pin to the latest 1.38 patch (1.38.2) across the three compose files to match the guide.

[dudanogueira]

hi @41tair !! Can we merge this? Thanks!

[dudanogueira]

Docs follow-up is up: langgenius/dify-docs#940.

It adds self-host/deploy/troubleshooting/weaviate-server-migration-path (en + zh + ja, nav mirrored) with the one-minor-at-a-time ladder from 1.27 to 1.39, graceful-shutdown guidance, and per-rung verification — and cross-links with the existing Weaviate v4 migration guide, which stops at server 1.27.

That covers the upgrade path for existing self-hosted deployments, so this PR stays a pure deployment-config change. Fresh deployments are unaffected either way.