fix(api): defer vector index task dispatch until dataset update commits

[zl86790]

Important

1. Make sure you have read our contribution guidelines
2. Ensure there is an associated issue and you have been assigned to it
3. Use the correct syntax to link this PR: Fixes #<issue number>.

Summary

Fixes #40961.

When you change a high-quality knowledge base's embedding model, the API updates the dataset row and immediately dispatches the Celery re-indexing task (deal_dataset_vector_index_task.delay(...), and regenerate_summary_index_task.delay(...) for summary vectors). The catch is that the service only calls session.flush() here, not session.commit() — the actual commit happens later, in the controller's with_session decorator, after some more DB work runs (permission checks, member list updates, etc).

Since the Celery task opens its own independent DB session and re-reads the Dataset row from scratch (it's only ever given dataset_id, never the new model), there's a real race: the worker can dequeue and start executing before the web request's transaction actually commits. When that happens, it reads the old embedding model and re-embeds every chunk with it — so the dataset ends up permanently mismatched: metadata says the new model, but the stored vectors are still from the old one. Retrieval quality silently degrades, there's no error, and simply retrying doesn't fix it since the stored vectors already "match" whatever model was current when the task last ran.

This is a genuine race condition (Postgres default isolation means the worker's session can't see the flushed-but-uncommitted update), and it got wider recently: a prior fix (#39223) changed this code from commit() to flush() to solve a different bug (InvalidRequestError: Can't operate on closed transaction, #39191), which pushed the actual commit further away from the .delay() calls than it used to be.

The fix

Instead of dispatching the tasks right after flush(), defer them until the session's after_commit event actually fires — same pattern already used in snippet_service.py for a similar "don't fire a side effect until the transaction is durable" case. This way the worker can never see a stale row: by the time it starts, the update is guaranteed to be committed. No changes to the transaction handling itself (so #39191 stays fixed), no changes to the task signature — just moving when the dispatch happens.

Files touched:

• api/services/dataset_service.py — wrap the two .delay() calls in an after_commit listener instead of firing them inline
• api/tests/unit_tests/services/test_dataset_service_dataset.py — updated the existing test to assert the tasks are NOT dispatched before commit, and only fire once the after_commit callback runs

Screenshots

Not applicable — backend-only fix, no UI change.

Checklist

• This change requires a documentation update, included: Dify Document
• I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
• I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
• I've updated the documentation accordingly.
• I ran make lint && make type-check (backend) and cd web && pnpm exec vp staged (frontend) to appease the lint gods