feat(chat-history): server-side chat history with tombstones (follow-up to #9902)

[walcz-de]

Follow-up to #9902 by @TLoE419, continuing on top of their branch with the maintainer's agreement. The eight commits from #9902 are kept unchanged in authorship and rebased onto current master; four commits on top address the open review points.

Closes #9432.

What

Server-side persistence of the web UI chat conversations, per user, in the auth database (SQLite by default, PostgreSQL when configured). The Chat page syncs to /api/conversations and keeps localStorage as a cache.

Why the extra commits

The review on #9902 found that the server could not be authoritative across devices: a conversation deleted on device A came back when a stale device B reconnected, a message sent while the first list request was in flight could be lost, and Save after a soft delete resurrected the row. This PR fixes those with a small, explicit design and adds the missing docs, swagger and tests.

Design: tombstones and versions

Server (authoritative over deletions)

• DELETE turns the row into a tombstone: content dropped, deleted_at set, row kept. It is idempotent, and deleting an id the server has never seen still records a tombstone. That matters because the UI deletes immediately while a debounced PUT of the same chat may still be in flight; whichever request lands second, the delete wins.
• Any later write of a tombstoned id returns 410 Gone and writes nothing. GET /api/conversations/{id} returns 410 for a deleted conversation, 404 for one that never existed.
• GET /api/conversations returns {"conversations": [...], "deleted": [{"id", "version", "deletedAt"}]}.
• Every row has a server-assigned version: 1 on the first save, +1 on every write including the delete. It is returned on every save and in the list, and ignored on input.
• Save runs in a transaction. The version bump is a single UPDATE ... SET version = version + 1 with deleted_at IS NULL in the WHERE clause, so concurrent writers each count and a delete that commits between the lookup and the write still wins. Two concurrent first writes of the same id use ON CONFLICT DO NOTHING and retry once into the update path. This also removes the read-modify-write race the review flagged in Save.
• DeleteAll and PUT /bulk leave tombstones too. ReplaceAll no longer hard-deletes first: it upserts the payload, skips ids that were deleted (returned as skipped), and tombstones the user's other live conversations, so a replacement also reaches other devices as a deletion.
• createdAt and updatedAt are stored as the client sends them; only zero values are stamped. The UI compares updatedAt across copies, so the server's clock must not replace the client's.

Client (useChat.js)

mergeRemoteAndLocal is replaced by a reconcile step that can tell a never-synced draft from a server-side deletion:

Local	Server	Result
any	listed in deleted	dropped locally
present	present	the copy with the newer updatedAt wins (local keeps the server's version and is uploaded)
present, no version	absent	never-synced draft: kept and uploaded
present, has version	absent	the server has seen it and no longer has it: dropped, not resurrected (this also keeps one account's cached chats out of another account on a shared browser)
absent	present	added

• The version is kept in the localStorage cache, which is how the next load knows a chat was synced.
• Saves record the returned version. A 410 on save drops the chat locally instead of retrying it forever.
• Deletes made while the first list request is in flight are queued and sent when it arrives.
• Chats received from the server count as already sent, so adopting a server copy is not echoed straight back.
• The client no longer calls PUT /bulk for the localStorage migration. It replaced the whole server set from a possibly stale browser; the per-chat saves already upload never-synced drafts, and each goes through the tombstone check. The endpoint stays for API clients.

Not in scope: concurrent edits of the same live conversation on two devices are last-writer-wins at conversation granularity, as in #9902. The version is exposed, so conditional writes (If-Match-style, 409 on a stale base version) could be added later without a schema change. I left them out because they need a client-side conflict UI.

Review points and how each was addressed

1. Cross-device resurrection. Tombstones plus versions, as described above. Server: writes of a deleted id return 410; the list reports tombstones. Client: reconcile drops deleted and synced-but-missing chats and only uploads never-synced drafts. Regressions: API spec "a stale client reconnecting after a remote deletion" (4 cases, core/http/routes/chat_history_auth_test.go) and UI specs "a stale client drops a chat deleted on another device instead of uploading it again" and "a synced chat the server no longer has is dropped, not resurrected" (core/http/react-ui/e2e/chat-history-sync.spec.js).
2. First-load overwrite of in-flight edits. The reconcile compares updatedAt instead of letting the server always win, and the server now stores the client's updatedAt. UI spec "a message sent while the list is loading survives the older server copy" holds the list request open, sends a message, then releases the list. The opposite direction is covered by "a newer copy from the server replaces the older local one".
3. Save after delete. Save refuses a tombstoned id with ErrDeleted → 410 Gone, inside a transaction. That is neither a 500 nor a silent resurrection. Store specs "rejects a save after delete instead of resurrecting the row", "records a delete that arrives before the conversation's first save", and the ReplaceAll/DeleteAll variants.
4. Docs and swagger. New page docs/content/features/chat-history.md (permission, tombstone semantics, sync behaviour, API table, example), a pointer from features/chat.md, and a chat-history row in the instructions table of features/api-discovery.md. The /api/instructions/chat-history intro now describes the tombstone contract, and api_instructions_test.go lists chat-history. Swagger was regenerated with swag init -g core/http/app.go --output swagger (swag v1.16.6, the version in go.mod); the diff is limited to the chat-history paths and schema.Conversation* definitions. json.RawMessage fields carry swaggertype:"array,object" because swag cannot resolve the type otherwise.
5. Tests. HTTP handler specs for all seven endpoints, a useChat Playwright spec, and the store suite runs on both PostgreSQL and SQLite (see below). To make the handlers testable they now take the *chathistory.Store, and the route table moved into routes.RegisterChatHistoryRoutes, which RegisterLocalAIRoutes calls. The specs drive that function with the real auth.Middleware and the chat_history feature gate.
6. Field drift. serializeChat sends localaiAssistant, and schema.Conversation has the field. Since May, master's createNewChat added only localaiAssistant (mcpResources was already covered). A handler spec round-trips every per-chat setting the UI stores, and a UI spec checks the toggle on upload and in the local cache.
7. Default no-op when auth is disabled. Not redesigned; see the open question below. The docs now state it explicitly: without LOCALAI_AUTH=true (or with legacy API keys only, or with the web UI disabled) the endpoints are not registered, and the UI keeps using localStorage.

Also from the earlier review: API surface checklist items are unchanged from #9902 (swagger tags, instructionDefs, RouteFeatureRegistry, FeatureChatHistory in APIFeatures + APIFeatureMetas()). No MCP admin tool: these are per-user endpoints, not RequireAdmin, so .agents/localai-assistant-mcp.md does not apply.

Behaviour changes compared to #9902

• DELETE /api/conversations/{id} is idempotent: 200 for an unknown or already deleted id (it was 404). Reason: the in-flight save race above.
• GET of a deleted conversation is 410 (it was 404).
• PUT /bulk tombstones instead of hard-deleting, skips deleted ids, and returns skipped. count is the number stored.
• Server keeps the client's createdAt/updatedAt (previously updatedAt was overwritten with server time).
• Deleted conversations keep no content. The soft-delete in feat: persist chat conversations server-side (#9432) #9902 kept the content; I dropped it because the user asked for a delete and nothing prunes or audits it today.

Rebase notes

The branch was 1129 commits behind. Conflicts and their resolutions:

• core/http/auth/features.go, permissions.go: master added FeatureDecisions in the same spots. I kept both.
• api_instructions_test.go: master's count was 21, so with chat-history it is now 22.
• core/application/application.go: master now uses cmp and path/filepath (router corpus), which TLoE419's GORM refactor commit removed as unused. The imports are restored inside that commit, so every commit still builds.

Tests run

Go, with the tags the coverage gate uses (debug auth):

go run github.com/onsi/ginkgo/v2/ginkgo -tags "debug auth" \
  ./core/services/chathistory/ ./core/http/routes/ ./core/http/endpoints/localai/ ./core/http/auth/ ./core/schema/

Suite	Result
core/services/chathistory (PostgreSQL via testcontainers + SQLite via auth.InitDB)	44/44 passed
core/http/routes (incl. 22 chat history endpoint specs)	211/211 passed
core/http/endpoints/localai	388/388 passed
core/http/auth	213/213 passed
core/schema	83/83 passed

golangci-lint run --new-from-merge-base=upstream/master (v2.11.4, the CI version) on the touched packages, with and without --build-tags auth: 0 issues.

UI (Playwright against tests/e2e-ui/ui-test-server with the rebuilt bundle):

cd core/http/react-ui && npx playwright test e2e/chat-history-sync.spec.js   # 7 passed
npx playwright test e2e/chat e2e/forking-chat.spec.js e2e/home e2e/talk     # 212 passed (incl. the new spec)

Each fix was also checked to fail without it:

• Store/API: with saveInTx put back to feat: persist chat conversations server-side (#9432) #9902's write path (no tombstone check, plain gorm.Save upsert), 14 store specs fail (7 per dialect: the tombstone and version specs), and 4 API specs fail (the whole "stale client reconnecting after a remote deletion" context).
• UI: with feat: persist chat conversations server-side (#9432) #9902's useChat.js, all 6 sync specs that existed at that point fail. For "a newer copy from the server replaces the older local one" the red comes from the echo assertion, because the old code re-uploads the adopted server copy.
• Field drift: before the field was added, the handler round-trip spec failed with field "localaiAssistant" was dropped and the UI spec with Received: undefined.

The UI spec stubs /api/conversations with a small fake that follows the server contract (tombstones in the list, 410 for deleted ids, versions). The UI test server runs with auth disabled, where the real endpoints are not registered. The server side of the contract is covered by the Go specs above.

Open question for the maintainer

Chat history is a no-op when auth is disabled, which is LocalAI's default. The store needs Application.authDB, and that database only exists with LOCALAI_AUTH=true. #9902 made this trade-off deliberately (consistent with AgentStore/JobStore). This PR does not change it; it only documents it. Options if you want it on by default:

• (a) keep it auth-only (current);
• (b) open the same database without enabling auth and store everything under the anonymous user "". The store already partitions on it, but every browser on that server would then share one history;
• (c) something else you prefer.

Which way should it go?

A smaller follow-up I'd also suggest: tombstones are tiny (id, version, timestamp) but are never pruned. If you want a bound, a retention sweep on deleted_at is a one-line query. The trade-off: a client offline for longer than the retention period could then re-upload a chat it had synced before. Even then the client drops it rather than resurrecting it, because it carries a version the server no longer has.

Checklist

• Docs updated in the same change (docs/content/features/chat-history.md, chat.md, api-discovery.md)
• Swagger regenerated
• Tests: Go (store on PostgreSQL + SQLite, HTTP handlers with auth), Playwright (useChat sync)
• AI assistance disclosed via Assisted-by: trailers

🤖 Generated with Claude Code

[localai-org-maint-bot]

Changes requested on 2cfca0c383680e2fe3a48dfcffe31af85ea058cf. The server tombstones address the earlier successful-delete resurrection case, but two client paths still need correction:

1. [P1] Associate the local cache and unsynced drafts with the authenticated account. useChat.js still reads/writes the shared browser chat key, and reconcileWithServer keeps every unversioned local conversation missing from the server. Logout does not clear or partition that key. If account A has an unsynced conversation (for example a failed first save), signs out, and account B signs in, B's empty list retains A's conversation and the save effect uploads A's private history into B's account. The version check only protects already-synced rows. Partition the cache/pending operations by account and make legacy anonymous-cache migration explicit; add a two-account regression covering an unsuccessful first save.

2. [P2] Preserve failed deletions until the server acknowledges them. useChat.js:371-402 removes conversations locally and swallows DELETE failures. pendingDeletesRef is populated only while the initial probe is pending, and is not persisted. After a successful probe, disconnect the network and delete a synced conversation: it disappears from the UI, no pending deletion remains, and reopening Chat after reconnect adds the unchanged server copy back. The same problem affects delete-all and failures while flushing pending deletes. Persist deletion intent and retry it, or visibly restore/report a failed delete instead of silently treating it as complete.

I reproduced both paths with a Node harness executing the PR's actual reconciliation and delete callback: the failed delete left zero pending operations and the next reconciliation restored the row; an unversioned previous-account conversation survived reconciliation against an empty new account. This was a focused logic check, not the full browser/Go suite.

These affect privacy and the deletion contract, so they require coordinated state/sync changes rather than a minor review cleanup. The auth-only default remains a separate maintainer decision as described in the PR.

[walcz-de]

Thanks, both reproduced and fixed in ef0ae8f.

P1 (cache bound to the account): the browser cache now lives under localai_chats_data:<user id>, together with that account's pending deletes, and the Chat page remounts its state when the account changes. Without a signed-in account nothing is synced. The old shared key stays in use only without auth; when signed in it is neither read nor uploaded, because nothing tells which account it belongs to. Signing out does not clear the cache, so a draft whose first save failed survives and is uploaded when the same account signs in again, never into another one.

P2 (deletes until confirmed): single deletes, delete-all and deletes from the Home page are kept as pending deletes in the account's cache until the server confirms them. They override the server list in the reconcile and are re-sent on the next change, on online, and on the next load. While a delete-all is pending nothing is uploaded, so a chat started after it is not caught by it.

e2e/chat-history-sync.spec.js replays both scenarios (account switch after a failed first save; delete and delete-all while offline). Both fail on 2cfca0c and pass now. The counter-checks: the same account keeps and uploads its own draft, and a confirmed delete is neither re-sent nor resurrected. Docs updated in features/chat-history.md.

Open question: chats stored under the shared key before chat history was enabled are not adopted by a signed-in account. Is that the behaviour you want, or would you prefer an explicit import path?

[localai-org-maint-bot]

@mudler Incremental source-review sign-off at 03cd9c115, subject to conflict resolution, CI, and human DCO certification of the fix commit. The account-scoped cache and persistent deletion intents address my earlier privacy and failed-delete findings.

I found and fixed one remaining clear-all race: navigating away before the 500 ms cache debounce left an old unsynced draft on disk, allowing it to be uploaded after the deletion was acknowledged. Clear-all now persists the cleared list and deletion intent together. Added a Playwright regression and documentation. An isolated Node harness executing the actual PR functions reproduced the failure before the fix and passed afterward; JS syntax and whitespace checks also pass. I did not run the browser suite.

No further confirmed source blocker in this incremental review. The branch still conflicts with master 48f358dd9 in core/application/application.go, core/http/endpoints/localai/api_instructions_test.go, and core/http/react-ui/src/hooks/useChat.js; it is not merge-ready until those conflicts and the normal checks are resolved. The fix commit has Assisted-by attribution and needs human DCO certification.

[walcz-de]

Thanks for the review and the clear-all fix. Merged master 48f358dd9 (3e5d17803) and resolved the three conflicts:

• application.go: chatHistoryStore sits next to the new WebSocket ticket fields.
• api_instructions_test.go: both sides added an instruction (chat-history, motion), so the list has 23 entries.
• useChat.js: kept the account-scoped cache and sync; reasoningEffort (feat(ui): add reasoning effort and reasoning off controls to chat #12700, [ui] Fold the No reasoning chip into the effort picker #12716) is now part of the synced chat, and the browser cache still converts the old reasoningOff flag.

Running the browser suite locally turned up two problems in chat-history-sync.spec.js, fixed in a separate test-only commit (a5b50c19e):

• "clear all persists before navigation …" opened a single chat, and the menu offers no Clear all for one chat, so it timed out on every run. It now opens two unsynced drafts. Reverting the saveChats call from 03cd9c115 turns it red (["draft","other"] left in the cache), so it covers the race you found.
• "a delete the server confirmed is not sent again …" (mine) reopened the page as soon as the server had the DELETE, before the page had processed the response, so the still-pending delete was correctly re-sent (red in 4 of 5 runs). It now waits until the cache has no pending delete.

Full UI e2e suite: 1772 passed, 4 skipped; Go: core/application, core/http/endpoints/localai, core/services/chathistory pass. All commits carry my DCO sign-off; 03cd9c115 is now 8816a7200, same content, signed off.

The question from my last comment is still open: should chats stored under the shared key before chat history was enabled be adopted by the first account that signs in, or stay unadopted as now?

[mudler-agent]

One remaining persistence issue on this head: the server drops the conversation's reasoning-effort setting. Please preserve it and add a round-trip regression before merging.

Validation: a focused SQLite reproduction using unchanged copies of this head's schema and store confirmed the loss under the race detector. Focused Node checks for account-scoped caches and persisted deletion intent passed. The repository-level Go race tests and vet could not start because generated protobuf packages were absent; full browser tests were not run. These results do not establish full CI coverage.

[mudler-agent]

[P2] Preserve reasoningEffort in the server-side conversation schema. serializeChat in core/http/react-ui/src/hooks/useChat.js sends this property, but Conversation does not declare it. JSON binding therefore discards it before persistence, so loading a synced conversation loses the selected reasoning behavior.

I reproduced this with an actual schema/store round-trip: decode a conversation containing "reasoningEffort":"none", call Store.Save, then Store.Get and JSON-encode the result. The response omits reasoningEffort. The focused SQLite reproduction passed under the race detector.

Please add the field, regenerate the Swagger artifacts, and add a regression asserting that a non-empty reasoning-effort setting survives save/get/list.

[walcz-de]

Right — my merge claimed reasoningEffort was part of the synced chat, but schema.Conversation had no field for it, so the server dropped it. Fixed in 5f42128f1:

• schema.Conversation gets ReasoningEffort (json:"reasoningEffort,omitempty"); swagger and features/chat-history.md updated.
• The handler spec that round-trips every chat field (chat_history_auth_test.go) now includes reasoningEffort and checks the value.
• New core/schema/chat_conversation_ui_test.go: it reads serializeChat in useChat.js and fails for any key the UI sends that schema.Conversation cannot hold. On the previous head it reports exactly reasoningEffort; this is the second time a field drifted this way (localaiAssistant before), so the check should catch the next one.

go test ./core/schema/ ./core/http/routes/ ./core/services/chathistory/ pass locally (with generated protobufs).

[mudler-agent]

The reasoningEffort persistence fix in 5f42128 resolves my original finding. I verified the UI/schema guard goes red with the previous schema, and the updated authenticated HTTP/SQLite history tests pass under the race detector.

One test-only blocker remains: core/schema/chat_conversation_ui_test.go uses t.Fatalf (twice), t.Fatal, and t.Errorf. These four new calls fail the repository forbidigo rule. Reproduced with Go 1.26.8 and golangci-lint against the PR diff.

Please keep the guard but convert it to the existing schema package Ginkgo/Gomega style: Describe/It, Expect(err).NotTo(HaveOccurred()), Expect(body).NotTo(BeNil()), a >= 10 key-count assertion, and HaveKey for the reflected JSON fields. I tested that conversion: the missing-field case still fails, the restored schema passes with -race, and changed-package lint reports 0 issues. No production change is needed.

CI is queued; this review does not claim a full CI or PostgreSQL pass.

[mudler-agent]

The reasoningEffort persistence fix and the follow-up Ginkgo conversion resolve my findings. The conversion is pushed as 6781979; it preserves the drift guard, fails with the missing field, and passes schema race tests and changed-package lint with Go 1.26.8. Authenticated HTTP and SQLite history tests also passed under the race detector. Approval is source/local-test review, not a substitute for required CI. PostgreSQL coverage was not available locally. The bot follow-up intentionally leaves human DCO certification to the contributor, per repository policy.