fix: stop a single streamed token invalidating the whole conversation by lodystage[bot] · Pull Request #751 · LodyAI/Lody

lodystage · 2026-09-16T08:27:40Z

Related issue

Problem / pressure

Users report the renderer becoming unresponsive on long conversations in 0.94.0,
with 0.93.3 fine. The two releases differ by the windowed conversation read
(#376).

The window is not the problem. A content notification reported the span between
the lowest and highest position it touched and then named every turn id
inside it. One synced batch routinely carries an early turn's status write
alongside the streaming tail — auto-seen, a permission response,
operation-progress, the preceding user turn's endedAt, or simply several
remote commits arriving in one doc.import. So a single token delta told the
display cache that the whole conversation had changed.

The cache amplified that a second time. rowChanged compared itemCount /
planCount — which a container-backed directory row deliberately never carries
— against the real counts a hydrated row holds, so the comparison was
42 !== undefined on every refresh and every hydrated row was re-read.

Measured on a 1,000-turn synthetic conversation, for one token delta in such a
batch: 996 directory rows and 997 turn bodies re-read, ~1.2 s of main-thread
CPU, and React told that 996 turns changed. Bumping those content epochs also
cancelled in-flight reads for turns nothing had touched: taking a 300-turn
window lease while such batches arrived every 5 ms took 1,890 ms and did 3.5
body reads per turn — during which that 5 ms interval fired 4 times. That is
the unresponsiveness in the report.

Summary

Change notifications now carry what actually changed, and four
whole-conversation costs that ran per frame or per open became incremental or
deferred.

changeScopeOf reports exact positions (packages/shared/src/session-data/loro.ts).
Structural batches keep the shifted-suffix range they need, because later
positions really did move.
The view tracks content targets as reported ids and refreshes them in
contiguous runs, resolving positions at flush time. A content refresh that
finds the length changed escalates to a structural re-key rather than splicing
rows from a sparse read.
Body invalidation follows the reported ids, never the directory diff — a
directory row cannot tell whether a body changed, since a grown text item
moves no scalar. carryBodyFacts fills the counts a refresh cannot supply
before rowChanged compares them, and an unchanged row keeps its object:
placeholder items and Virtua rows key on that identity.
Send configuration projects on first read. It is a schema parse per user
turn, while resolveSessionConversationConfig reads the newest source and
walks older ones only until it finds an explicit Role. Deferred and memoized
at each of the four hops that used to force it.
The shared fact table is held, not disposed, on the last release. Deriving
a goal / scheduled task / proposed plan / file diff needs the body, so
discarding the facts re-materialized every turn the next time the tab opened.
It keeps its view subscription, stops its background pass, resumes on
re-acquire, and is still collected with the view. That pass now yields to real
idle time instead of back-to-back macrotasks.
use-session-diff-summary stops serializing the conversation every frame —
identical entries settle by reference, only a replaced entry is serialized.
normalizeTexMathDelimiters and the Mermaid fence test settle the common
case with a substring test instead of rescanning the whole accumulated answer
on every delta (quadratic in answer length).
Array reuse for the index-row list, chat stream items, conversation config
sources and the ordered fact list, plus a new ConversationView.structureVersion
so the accepted-history projection stops rebuilding a whole-conversation slot
array at token rate.

Two scoped AGENTS.md files record the new invalidation and lifetime contracts,
and a bilingual Agent Note records the investigation.

Visual explanation

One daemon batch touching turn 4 and turn 999, on a 1,000-turn conversation:

flowchart TD
    D["Daemon commit batch<br/>touches turn 4 and turn 999"] --> IMP["renderer doc.import<br/>one event batch"]
    IMP --> SCOPE["changeScopeOf"]

    subgraph before["Before — ~1.2 s main thread"]
      direction TB
      B1["reduce to range 4..1000"] --> B2["observe names 996 ids"]
      B2 --> B3["mergeDirty span<br/>readDirectory 996 rows"]
      B3 --> B4["rowChanged: 42 vs undefined<br/>true for every hydrated row"]
      B4 --> B5["re-read 140 bodies<br/>bump 140 content epochs"]
      B5 --> B6["emit changed 996 ids"]
      B6 --> B7["fact table restarts<br/>997 bodies re-materialized"]
      B5 --> B8["in-flight window reads cancelled<br/>3.5 reads per turn"]
    end

    subgraph after["After — ~60 ms"]
      direction TB
      A1["keep positions 4 and 999"] --> A2["observe names 2 ids"]
      A2 --> A3["dirtyIds holds those 2<br/>positions resolved at flush"]
      A3 --> A4["readDirectory per contiguous run<br/>2 rows"]
      A4 --> A5["carryBodyFacts, then rowChanged<br/>unchanged rows keep their object"]
      A5 --> A6["re-read 2 bodies<br/>bump 2 content epochs"]
      A6 --> A7["emit changed 2 ids"]
    end

    SCOPE -.->|previous| B1
    SCOPE ==>|this PR| A1

Before / after

Synthetic Loro fixtures, Node, one machine. Library measurements, useful as
ratios rather than budgets — real turns carry far more content than the fixture.

Before	After
Amplified delta (1,000 turns): 996 directory rows, 997 bodies, 996 changed ids, 1021–1569 ms CPU	2 rows, 2 bodies, 2 changed ids, 51–91 ms
Tail-only delta: 7–9 ms, 1 row, 1 body	unchanged
300-turn window lease under 5 ms churn: 1,890 ms, 3.5 body reads per turn	247 ms, 1.1
Full directory read: 1,000 turns 195.8 ms / 4,000 turns 728.4 ms	60.1 ms / 264.1 ms
Reopening a session tab re-derives every fact: 4,020 turn reads, ~2.8 s CPU	facts survive the release
Markdown rescan, 120 KiB math-free answer: 2.49 ms per delta, ~2.55 s per turn	0.06 ms, ~60 ms
Diff-summary fingerprint: 144 KiB serialized per frame at 4,000 turns	only a replaced entry is serialized

Test plan

packages/components: 481 files / 3,716 tests pass (NODE_ENV=development).
packages/shared: 107 files / 1,227 tests pass.
apps/electron: 118 tests pass.
apps/cli: 264 files pass; tests/gh-shim-script.test.ts fails on this
machine for a pre-existing environment reason unrelated to this change.
tsgo --noEmit clean in both changed packages. pnpm lint 0 errors,
pnpm run docs check no errors, check:public-boundary, lint:i18n,
check:code-collab-imports, check:platform-boundaries all pass. Oxfmt clean
on every file this PR touches.
Two behavioural regressions added to their owning suites:
- conversation-view-from-reader.test.ts — a batch carrying an early edit and
  the streaming tail reads 2 directory rows and 2 bodies. Verified to fail at
  23 rows against the pre-change implementation.
- conversation-derivation.test.ts — facts survive the last release, and the
  background pass stays held until someone re-acquires.
Not run: device-scale cold open, real streaming frame time, and long-session
memory. Those remain the acceptance work the windowed-reader note deferred.

Context handoff

Instructions for reviewing agents

Review focus: changeScopeOf in packages/shared/src/session-data/loro.ts
now returns sparse positions — check its structural and whole-directory
branches still over-report rather than under-report. Then flushContent and
applyChange, where the reported-id set is the sole authority for body
invalidation.
Decisions to challenge: deferring pickDirectoryInputConfig behind an
enumerable getter (a consumer that spreads a row still forces it); holding the
fact table after the last release, trading memory bounded by the session-store
cache for the reopen cost; keeping the row object when rowChanged is false,
which several render caches now depend on.
Plausible failures / evidence gaps: a content batch that silently changed
membership used to be absorbed by the dense span; the new code escalates on a
length mismatch, and that is the least exercised branch. All numbers are
synthetic fixtures in Node — no device or Electron measurement.

Authoring context

User goal / directives: investigate why the renderer freezes on the latest
release when 0.93.3 did not, enumerate every optimisation the scan found, then
fix all of them.
Constraints / non-goals: no change to what the UI shows, to send-config
resolution semantics, or to the analytics funnel's reporting. Facts were not
moved into write-time metadata; the directory was not reduced to a skeleton.
Risk-bearing decisions: the change-notification contract is now sparse,
which every SessionHistoryReader consumer inherits; body invalidation is
driven by reported ids rather than re-derived from the directory; the fact
table's lifetime changed from refcount to view lifetime.
Destructive or irreversible behavior: none. No storage format, wire
format, migration, or user data is touched; all changes are read-path and
in-memory.
Deliberately not done or tested: snapshot import still decodes on the
renderer thread (moving it means moving the document off the main thread); the
fact table still materializes every turn once per session (needs facts written
alongside history); the permission-funnel analytics still share the fact table,
because decoupling it would change which requests it reports; the LRU touch
in bulk scans is left alone as measured-negligible and bounded by pins and the
retained tail.
Unknowns / confidence: high confidence in the invalidation fix — it is
pinned by a regression that fails on the old code and the ratios are large and
reproducible. Lower confidence that it is the whole story for every report,
since first-open cost on very long conversations is improved but not solved.

A content notification reported `[min(position), max(position)+1)` and named every turn id inside it. One synced batch routinely carries an early turn's status write alongside the streaming tail, so a single token delta told the display cache that the whole conversation had changed. The cache then re-read the whole directory and, because `rowChanged` compared `itemCount`/`planCount` that a container-backed directory row never carries, reported a change for every hydrated row and re-materialized its body. Measured on a 1000-turn synthetic conversation with a window lease and the shared fact table open, for one such batch: 996 directory rows and 997 turn bodies re-read at ~1.2 s of main-thread CPU, now 2 rows and 2 bodies. A tail-only delta was already 1 and 1; it is unchanged. - `changeScopeOf` carries the exact touched positions for a content batch; structural batches keep the shifted-suffix range they need. - The view tracks content targets as the reported ids and refreshes them in contiguous runs, escalating to a structural re-key if the length moved. - `carryBodyFacts` keeps the counts a refresh cannot supply, so `rowChanged` compares like with like and an unchanged row keeps its object identity — which is what the placeholder and Virtua row caches key on. Model: claude-opus-5

… open Opening a conversation reads the whole directory, and every user turn's row carried an eagerly projected send configuration. That projection runs a schema parse (~0.17 ms per turn, measured), while its only consumers resolve sticky Role/model/mode from the newest turn or two — `resolveSessionConversationConfig` reads the latest source and walks older ones only until it finds an explicit Role. The projection is now deferred and memoized at each hop that used to force it: the directory row, the view's index row, and both source collections. Container crossings stay eager, so the raw record is captured without retaining a Loro handle past the read. Full directory read of a synthetic conversation: 1,000 turns 195.8 ms -> 60.1 ms, 4,000 turns 728.4 ms -> 264.1 ms. Model: claude-opus-5

…rame Three costs that all scale with turn count and all run while a turn streams: - The shared fact table discarded its facts when the last consumer released, so closing and reopening a session tab re-materialized every turn body to rebuild them. Measured on a synthetic conversation: 4,000 turns, 4,020 turn reads, ~2.8 s of CPU, repeated on every reopen. The table is now held rather than disposed — it keeps its facts and its view subscription, stops its background pass, and resumes where it left off. It is still collected with the view, so an evicted session frees it. - That background pass yielded with back-to-back macrotasks, which kept it at the head of the queue for its whole run. It now yields to real idle time with a timeout so a busy tab still makes progress. - `use-session-diff-summary` serialized every turn's file diffs each frame to decide whether anything changed — a 144 KiB string per frame at 4,000 turns. Identical entries are now settled by reference and only a replaced entry is serialized. The index-row list, the chat stream items and the conversation config sources now hand back their previous array when nothing changed, which is what the derivations keyed on those arrays recompute on. Config sources reuse the index row itself instead of allocating a wrapper per historical user turn. Model: claude-opus-5

…eamed token - `normalizeTexMathDelimiters` walks its input character by character, and a streaming turn re-ran it over the whole accumulated answer on each delta. Only an opening `\[` can produce a replacement, so text without one returns immediately: a 120 KiB math-free answer went from 2.49 ms to 0.06 ms per delta (~2.5 s to ~60 ms across the turn). The Mermaid fence test gets the same substring guard. - The markdown search-highlight effect queried its rendered subtree on every delta to unwrap marks that were never inserted. It now tracks whether this block holds any. - `createProjectedConversationView` rebuilt a slot array over the whole conversation whenever the base view's version moved — at token rate, for as long as an unconfirmed entry existed, which is exactly while a turn streams. Slots follow membership, so `ConversationView` now exposes `structureVersion` and the wrapper keys on it. - `useSessionTurnFacts` hands back its previous ordered array when the facts are unchanged, and the session-meta equality gate memoizes the serialization of the retained value instead of recomputing it on every emission. Measured on a 800-turn synthetic conversation, taking a 300-turn window lease while an early edit and the streaming tail arrive every 5 ms: 1,890 ms and 3.5 body reads per turn, now 247 ms and 1.1 — the invalidation fixes earlier in this branch stopped cancelling in-flight reads for turns nothing touched. Model: claude-opus-5

Model: claude-opus-5

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 53671a585b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-09-16T08:32:01Z

+        if (indexChanged) {
+          // Row identity is the renderer's change signal (placeholder items and
+          // Virtua rows are keyed by it), so an unchanged row keeps its object.
+          rows[pos] = row;


Replace evicted rows after body-only changes

When a reported ID belongs to an evicted turn and the edit changes only body data such as items, fileDiff, or inputConfig, indexChanged remains false. The freshly read row is therefore discarded—even though its summary was cleared—leaving the existing row's stale summary and memoized send configuration visible indefinitely; outline/placeholder caches can keep stale content, and sticky Role/MCP resolution can use obsolete configuration without hydrating the turn. Replace the row for reported evicted IDs, while preserving row identity only for turns that were not actually edited.

AGENTS.md reference: packages/components/src/lib/conversation-view/AGENTS.md:L15-L18

Useful? React with 👍 / 👎.

…dex rows Two regressions from this branch, each found in review and pinned by a test that fails on the previous commit. Holding the fact table instead of disposing it stopped its background pass but kept its view subscription. That is right when the table is keyed on the conversation's own view and wrong when it is keyed on a projection wrapper: the wrapper is rebuilt whenever an optimistic entry appears or resolves, so the base view's listener set accumulated one released wrapper — and one live table — per message sent, each still deriving on every token, and none collectable. `ConversationView` now exposes `factSource`; the wrapper points at the view it wraps and `acquireConversationDerivation` keys and subscribes there. That also collapses a duplicate older than this branch: the diff summary acquired on the base view while the turn-fact readers acquired on the wrapper, so an unconfirmed entry meant two full fact tables for one conversation. Keeping the index row when `rowChanged` reported no change assumed `rowChanged` sees everything the row carries. It does not see `inputConfig`, which is a deferred projection that cannot be diffed without forcing it. A user turn whose send configuration changed while outside every hydrated window kept its old model, Role and MCP selection in the index — which is what the sticky-config resolver reads. A reported turn now always takes the fresh row; identity is preserved only for turns the notification did not name, which is where the churn that optimization targets came from. The amplified-delta measurement is unchanged: 2 directory rows, 2 bodies and 2 changed ids for a batch carrying an early edit plus the streaming tail. Model: claude-opus-5

zxch3n added 5 commits September 16, 2026 07:21

docs: record why streaming cost scaled with conversation length

53671a5

Model: claude-opus-5

github-actions Bot added scope: shared scope: components labels Sep 16, 2026

chatgpt-codex-connector Bot reviewed Sep 16, 2026

View reviewed changes

lodystage Bot merged commit 4de83a5 into main Sep 16, 2026
6 checks passed

Navigation Menu

fix: stop a single streamed token invalidating the whole conversation - #751

fix: stop a single streamed token invalidating the whole conversation#751
lodystage[bot] merged 6 commits into
mainfrom
fix/conversation-view-streaming-cost

lodystage Bot commented Sep 16, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Sep 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

lodystage Bot commented Sep 16, 2026

Related issue

Problem / pressure

Summary

Visual explanation

Before / after

Test plan

Context handoff

Instructions for reviewing agents

Authoring context

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant