fix: re-index after an embedding model switch runs with the previous model

[goingforstudying-ctrl]

Important

1. Make sure you have read our contribution guidelines
2. Ensure there is an associated issue and you have been assigned to it
3. Use the correct syntax to link this PR: Fixes #<issue number>.

Summary

Switching a knowledge base between two embedding models looks fine in the console, but every chunk ends up re-embedded with the model that was declared before the PATCH. The declaration and the stored vectors silently land in different vector spaces, and retrieval quality drops with no error anywhere. Fixes #40961.

The ordering in _update_internal_dataset is the problem: it flushes the new embedding_model onto the datasets row and then calls deal_dataset_vector_index_task.delay(...) while the request transaction is still open (the commit only happens later in the with_session decorator; the flush() in between is deliberate, see #39191). The worker opens its own session, reads the pre-update row, and re-embeds with the old model. The task publish pretty much always wins that race against the request commit.

The issue itself suggests deferring the dispatch until after commit, so that's what I went with, it's also the smaller change. Both celery calls now run inside an after_commit hook on the session (same idiom as register_new_agent_beta_publish_after_commit), with an after_rollback guard so a rolled-back request can't leave a stale listener that fires on some later commit. regenerate_summary_index_task is dispatched from the same spot and reads the same row, so it goes through the hook too.

For RAG-pipeline knowledge bases nothing really changes: _update_pipeline_knowledge_base_node_data already commits mid-request, which is why they were mostly spared. The hook just fires at the decorator's commit instead.

One thing I didn't do: the issue also floats a way to force a re-index once declaration and vectors have already diverged. There's no dispatch path to reuse for that today, so it felt like a separate feature and I left it out.

Tests: extended the SQLite-backed update tests in test_dataset_service_dataset.py. They now assert the tasks are not published during _update_internal_dataset itself, that they fire on the following commit() with the right arguments for add/remove/update, that a rollback suppresses them even if the session commits again later, and that no action means no dispatch. Ran the update tests, ruff, pyrefly and mypy locally.

Not 100% sure the hook belongs as a static method on DatasetService vs a small helper in the tasks module, happy to move it if there's a house preference.

Screenshots

Before	After
N/A (backend-only change)	N/A

Checklist

• This change requires a documentation update, included: Dify Document
• I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
• I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
• I've updated the documentation accordingly.
• I ran make lint && make type-check (backend) and vp staged (frontend) to appease the lint gods

[github-actions]

Pyrefly Diff

base → PR

--- /tmp/pyrefly_base.txt	2026-08-19 12:52:52.035989962 +0000
+++ /tmp/pyrefly_pr.txt	2026-08-19 12:52:44.427993925 +0000
@@ -8617,7 +8617,7 @@
 ERROR Object of class `object` has no attribute `kw` [missing-attribute]
    --> tests/unit_tests/services/test_clear_free_plan_tenant_expired_logs.py:757:12
 ERROR Object of class `NoneType` has no attribute `keyword_number` [missing-attribute]
-   --> tests/unit_tests/services/test_dataset_service_dataset.py:976:16
+    --> tests/unit_tests/services/test_dataset_service_dataset.py:1091:16
 ERROR Object of class `FakeAccount` has no attribute `id` [missing-attribute]
    --> tests/unit_tests/services/test_dataset_service_document.py:144:9
 ERROR Object of class `FakeAccount` has no attribute `current_tenant_id` [missing-attribute]

[github-actions]

Pyrefly Type Coverage

Metric	Base	PR	Delta
Type coverage	59.72%	59.73%	+0.01%
Strict coverage	59.30%	59.31%	+0.01%
Typed symbols	39,770	39,780	+10
Untyped symbols	27,011	27,011	0
Modules	3175	3175	0

[ddddajie]

I think there may be a RAG_PIPELINE edge case with the current listener registration order.

_update_pipeline_knowledge_base_node_data() can call session.commit() for a RAG pipeline, but _register_index_tasks_after_commit() is currently registered after that helper returns.

This means the dataset/model update may already be committed before the listener exists.

If something later in the controller fails after that internal commit, the outer with_session path rolls back. The after_rollback guard then prevents the re-index task from ever being dispatched, even though the new embedding model was already committed.

The resulting sequence would be:

dataset update → RAG helper commit → register listener → later controller failure → rollback → no re-index

leaving the committed dataset configuration potentially inconsistent with the existing vectors again.

Would it be safer to register the after_commit listener before _update_pipeline_knowledge_base_node_data()?

Then:

• normal datasets still dispatch on the outer with_session commit;
• RAG_PIPELINE datasets dispatch on the helper's existing internal commit;
• once=True prevents the later outer commit from dispatching twice.

A regression test where the RAG helper commits and then a later operation fails would help lock down this ordering.

[goingforstudying-ctrl]

Good catch, that ordering was wrong. In e3bbfb5 the listener is now registered before _update_pipeline_knowledge_base_node_data() runs, so a RAG pipeline dispatches on the helper's internal commit and normal datasets still dispatch on the outer with_session commit. once=True keeps the later outer commit from firing a second dispatch.

Added a regression test (test_update_internal_dataset_dispatches_on_rag_pipeline_internal_commit) that commits inside the helper and asserts the task fires exactly once across both commits.

Also fixed the three integration tests that asserted inline dispatch (they now commit before asserting) and retitled the PR for the title check.