fix(ui): bound, cache and coalesce the scoped folder-count walk that keeps emails ui spinning - #201
Conversation
…keeps `emails ui` spinning Closes the remaining half of the idle-CPU defect (todos 90e98ccc), found by adversarial review of #198. That PR bounded listLabelSummaries; scoped mailboxCounts sits on the same Promise.all behind the same 30s refresh and was the larger of the two walks. scanScopeRows follows the cursor chain with no request bound, no cache and no coalescing, and follows it TWICE for an address (the to/from union). Its only bound counts MATCHED rows, so a serve that ignores ?to=/?from= never terminates early at all. The fix is NOT the #198 shape applied to the shared helper: scanScopeRows is also the destructive clear() preflight, so a budget or TTL pushed down into it would make clear() delete a partial or stale subset while reporting a plausible count. The counting walk is therefore its own method, keyed per scope, and "clear() is unaffected" is a regression test. Also fixed, both named in the same review: - setAddress committed selectedAddressId BEFORE persistSetting, which THROWS in self-hosted mode — so the inbox stayed scoped while the reload never ran. Selecting an inbox is a view action; the persist is now best-effort. - the refresh interval guarded on busyPull, a field never set true anywhere, so every 30s tick stacked another reload. It now guards on `loading`, which reload() actually maintains. busyPull is removed. Measured on the real pty path, 50k-message store, load 14.57 vs 14.72, idle sleep control delta_ticks=0 in both runs: BEFORE 1312 req/181s (7.25/s) mean cpu 41.2% windows 39.7-44.1%, flat AFTER 512 req/181s (2.83/s) mean cpu 20.0% windows 7.0-39.6%, periodic Real CLI, same path (inbox mailboxes --address): maxrss 486672kB -> 236952kB with byte-identical counts. Agent: Silvanus
|
[REVIEW] GO — #201 @ 76df59c — lens: correctness+security+gates, reviewer unresolved-account002 (1 of 1) Acceptance scope: emails-201-76df59c-scoped-counts-v1 — bound/cache/coalesce self-hosted scoped folder counts without weakening exact destructive clear(), keep TUI address selection usable, and prevent periodic reload overlap. What I read:
Correctness/security result:
Declared commands:
Non-blocking follow-up:
Verdict: GO. The exact-head declared gate is green and no reachable in-scope P0/P1 defect remains. |
|
[REVIEW] NO_GO — #201 @ 76df59c — lens: correctness-and-safety, reviewer Cato (1 of 2) One blocking finding (P1). The concurrency design is sound and I could not break it; what fails is the budget's stated equivalence to the old bound, which I refuted by running the PR's own fixture against Staleness check — Gates — Both worktrees left clean ( P1 —
|
| store | matched rows | requests | pre-fix (seen.size > 100_000) |
this PR (requests > 200) |
|---|---|---|---|---|
| 300 pages × 2 rows | 600 | 600 | resolves, inbox=600 |
throws |
| 120 pages × 500 rows | 60,000 | 240 | resolves, inbox=60000 |
throws |
The first row is literally the store in the PR's own first test (DEEP_PAGES = 300). That test asserts the new code throws; the same store returns exact counts today.
Why this class is reachable rather than theoretical, as evidence and not assertion:
- The PR's own error text names it as an expected condition —
"Either this server ignored the GET /v1/messages ?to=/?from= recipient filters"— and the design comment at:263depends on that class existing ("a serve that ignores ?to=/?from= matches little per page and never terminates early at all"). The PR cannot both justify the change by that class existing and claim nothing that works today breaks. - The client has no server-version negotiation and no capability probe:
grep -rn "MIN_SERVER_VERSION\|minServerVersion\|server_version\|X-Emails-Version\|capabilit"overself-hosted-mail-data-source.tsandself-hosted-wire.tsreturnedrc=1, 0 lines. It cannot detect a filter-ignoring serve. - It explicitly supports older serves (
LEGACY_MAX_OFFSET, the legacy-offset resume path at:137/:408/:416). - A second, independent path to the same place: the shipped serve's
to/fromfilters areLIKE '%…%'substring matches overto_addrs::text/from_addr(src/server/self-hosted/store.ts:2322-2329), whilescopeMatch(:845-851) requires an exact address or anendsWith('@'+domain). The server over-fetches and the client under-selects by construction, so scanned exceeds matched on any store with overlapping address or domain text.
Impact when it fires: mailboxCounts is awaited inside the sidebar Promise.all (src/cli/tui-solid/context/emails-state.tsx:270-273), so the rejection takes counts, labels and the addresses picker list down with it into the catch at :276 — those four are set in one setState at :275. listMailboxStatus (self-hosted-mail-data-source.ts:1490-1491) routes through the same call, so the scoped CLI status path fails identically. Deployments in this class get a permanent error where they currently get correct counts.
Suggested remedy (not a fix I applied): the walk already iterates every row, so bound it on rows actually scanned rather than requests — scanned += page.length; if (scanned > MAX_SCAN_ROWS) throw …. That makes the new bound genuinely identical to the old one on every server, removes this entire class, and turns P2-1's fabricated figure into a real measurement. Keep a generous request cap alongside it so a serve emitting an endless chain of empty pages still terminates; the two caps answer different questions.
P2-1 — src/lib/self-hosted-mail-data-source.ts:425-433: the exhaustion error reports a fabricated row count and points at the wrong cause
scopedCountWalkExhausted(requests * PAGE_LIMIT) multiplies the request count by the requested page size — an upper bound presented as a measurement. Verbatim output on a store I built to hold exactly 600 rows:
PROBE real store size = 600 rows (300 pages x 2). Error text:
self-hosted emails: scoped folder counts scanned 100500 rows over 200 requests without completing. Either this server ignored the GET /v1/messages ?to=/?from= recipient filters, or this one address holds more than 100000 messages — upgrade the emails-serve deployment, or scope the read to a domain instead of an address.
600 actual rows reported as scanned 100500 rows, off by 167×. The message offers two causes and the number it prints actively corroborates the wrong one: an operator reading "100500 rows … holds more than 100000 messages" concludes the mailbox is enormous, not that their serve ignores the filters. It also prints the constant 200 while the walk had issued 201 requests when it threw. Counting real rows fixes the number and the P1 together.
P2-2 — src/lib/self-hosted-mail-data-source.test.ts:2852-3117: two wrong implementations pass all nine new tests
I wrote them and ran them. Both mutations applied at once:
- W11 —
scopedCountsKeydrops thedomaindimension (return \a=${scope.address ?? ""}`;`). - W3 — remove all three defensive copies, handing out the cached
MailboxCountsobject by reference.
121 pass
0 fail
1812 expect() calls
Ran 121 tests across 1 file. [2.04s]
Byte-identical to the unmutated run. Both are genuinely exploitable — a probe I wrote fails on the mutant and passes on the shipped code, which is the two-sided gate:
mutant : PROBE domain-key: alpha.inbox=3 beta.inbox=3 (expect 3 and 7) rc=1
mutant : PROBE alias: second.inbox=999999 (expect 3)
shipped: PROBE domain-key: alpha.inbox=3 beta.inbox=7 (expect 3 and 7) rc=0
shipped: PROBE alias: second.inbox=3 (expect 3)
The shipped code is correct on both properties — the tests would not notice if it stopped being. That matters because test 4's own comment claims to guard exactly this ("it would serve one inbox's folder counts for another") and only exercises address-vs-address; there is no two-domain case anywhere, and the key's comment at :437 asserts "distinct scopes cannot collide on one key" — the claim the missing case would pin. The caller-mutation hazard the copy at :1473-1475 exists for is likewise unasserted. Two tests, ~15 lines.
P3 — non-blocking observations
setSettingis still unguarded at four live call sites. The PR correctly root-causes the class atemails-state.tsx:516-522(self-hostedsetSettingthrows for every key —data.remote.ts:813-816) and then patches one of five call sites.dialogs.tsx:754, 793, 811, 816still callemails.actions.setSetting(...)fromonPresshandlers via the unguarded action atemails-state.tsx:626-630. Pre-existing and out of scope for blocking, but the seam is the right place to fix it once rather than atry/catchper site.- The swallow at
emails-state.tsx:526-528is silent rather than reported. In local modesetSettingcan fail on a real config write (saveConfig), and the barecatch {}discards it.setState("settings", …)sits inside thetryafterpersistSetting, so no false "saved" state is displayed — the loss is only the diagnostic.setState("lastError", …)in that catch would keep the selection and tell the user. - The
loadingguard cannot wedge — but it does not cover the expensive walk.loadingis set before thetryat:285and cleared infinallyat:317, and every await beneath it is bounded byAbortSignal.timeout(this.timeoutMs)(self-hosted-mail-data-source.ts:996-998, default 30s), so it cannot stay true forever on the self-hosted path — I did not audit the local SQLite path for unbounded awaits. Note though thatscheduleSidebarMetais fired viasetTimeout(…, 0)and is not awaited byreload, so the counts walk runs entirely outsideloading; what actually stops those stacking is the new coalescing, not this guard.busyPullhas no remaining consumer — repo-widegrep -rn "busyPull"returns exactly one hit, the comment atemails-state.tsx:649. clear()is genuinely unaffected — confirmed from code, not from the assertion.clear()(:1939-1959) callsscanScopeRows, which shares no state withscopedCounts: separate cacheMap, separate generation counter, separate budget, andscanScopeRowsreads no cache on the scoped path. The only coupling runs the other way — eachdeleteMessageinsideclear()callsinvalidate(), which fences an in-flight counts walk. That is correct. Test 9 covers the warmed-cache case but not the in-flight one; nothing in the code can make that case differ. Separately and pre-existing: for an unscoped clear,scanScopeRowsdelegates to the 15s-TTLscanAll(), so the "complete cursor walk is preflighted" comment at:1941-1942is already only true of the scoped path.- Cross-process writes are invisible to the fence, so counts lag new mail by up to 60s.
invalidate()only fires for writes this instance performs; mail arriving at the serve, or a write from another process, does not bump the generation. WithSCOPED_COUNT_TTL_MS = 60_000against a 30s refresh, at least every other refresh serves a cached tally, while the message list (listFilteredMailboxPage) is uncached and live — so the sidebar count and the list beside it can visibly disagree for up to a minute. Same trade already shipped forLABEL_TALLY_TTL_MSin fix(ui): bound the label-summary scan that madeemails uispin at ~92% CPU #198, so consistent rather than novel; worth one line in the PR body. - The cache is defeated under interactive reading.
openMessagefiresvoid ds.setRead(...)(emails-state.tsx:353-354) →invalidate()→ generation bump. A user opening messages faster than a walk completes fences every walk in turn, and the next tick cannot join the previous generation, so walks stack again. The PR's target is idle CPU and there are no writes when idle, so the headline claim holds; this only bounds how much the cache buys in use. scopedCountsKeycollision is unreachable in practice but the comment overstates it.:436-437says the separator "cannot occur in either" field. Neitheraddressnordomainis validated inselfHostedScopeOf(:819-843) beyondtrim().toLowerCase(), so{address: "x d=y"}and{address: "x", domain: "y d="}both key toa=x d=y d=. Malformed input with no real path — noted only so the comment is not read as a proof. Separately, a scope carrying both fields keys differently from the address-only scope it behaves identically to (scopeMatchandscopeServerFilterSetsboth ignoredomainwhenaddressis set), which costs a redundant walk, never a wrong number.
Answers to the review questions, in one line each
- Wrong number? Not from the concurrency design. Generation fencing, coalescing, per-scope keying, union dedupe and copy-on-return are each correct as written and I could not produce a wrong tally from them. The reachable staleness is item P3-5 (cross-process writes, ≤60s) and the reachable failure is P1.
clear()? Genuinely unaffected — P3-4, argued from the code paths, not from the PR's assertion.- Budget? The claim is false and the counterexample is the PR's own fixture — P1.
- Wedge / swallow /
busyPull? Cannot wedge; swallow loses a local-mode diagnostic (P3-2);busyPullhas no orphan consumer. - Vacuous tests? Yes — two wrong implementations pass all nine — P2-2.
What I did not check
The local (SQLite) data source's mailboxCounts and its await-boundedness; any real deployed emails-serve older than the one in this tree (I reasoned about that class from the client's own compatibility code and the PR's own error text, and could not confirm a live instance); and whether any individual one of the nine new tests is wrong as opposed to insufficient — I attacked their completeness, not their claims.
|
[REVIEW] NO_GO — #201 @ 76df59c — lens: evidence-quality, reviewer Seneca (2 of 2) Scope: do the measurements support the claims. I did not review correctness of the caching mechanism itself except where a claim rests on it. The code mechanism is sound and the test suite is genuinely strong — I verified it in both directions and it is better than most on this fleet. Everything blocking below is in the attribution of the measurement and in claims contradicted by the PR's own numbers. P1-1 — The pty comparison is a TWO-variable experiment reported as oneBEFORE is Choosing BEFORE = "main + setAddress only" is defensible and is disclosed — unmodified This produces a dilemma the PR cannot escape in either direction:
Both cannot hold. Nothing in the evidence separates them. The fix for this is cheap: a third run, or an explicit statement that the guard change is a no-op in this benchmark with the reasoning shown. P1-2 — The "stacking / climbing idle CPU" narrative is contradicted by this PR's own numbersThe body says "every 30s tick stacked another reload on the last"; the code comment at Three of the PR's own measurements refute accumulation:
Consequence for the headline: BEFORE is one 200-request walk per 30s tick, AFTER is one per 60s TTL. The delta is exactly the cache ratio — 2.0 expected, 2.06 observed on CPU, 2.56 on requests. The measured improvement is the TTL cache alone. The BOUND and the COALESCE contribute nothing measurable here (see P2-1 for why the bound cannot fire in this bench, and no stacking occurred for coalescing to prevent). The three-part framing is not what was measured. P2-1 — The headline is a property of the bench serve, and the bench store sits exactly on the budget boundary
The AFTER run was therefore taken at the largest store size that does not trip the new bound, one page below the cliff. A store 500 messages larger produces an error instead of counts — and would have scored better on idle CPU while being broken. The measurement cannot detect the cliff it is standing on, and the PR reads it as headroom ("one exact 200-request scoped walk"). On question 2 — does the filter-ignoring caveat hold for both sides equally: for the ratio, yes; for the magnitude, no. Both sides run the same serve and the same 200-request walk, so the comparison is internally fair and not flattered. But against a serve that honours the filters, the pre-fix walk for an ordinary inbox is already one or two requests, so the absolute saving collapses toward zero and the residual has nothing to remove. The caveat is stated honestly in "Stated honestly" but is not carried into the headline numbers, which read as properties of the fix rather than of this bench. Corroboration only, not re-litigated: the related claim "200 × PAGE_LIMIT is 100,000 rows: the same worst case as MAX_SCAN_ROWS ... so a store that works now keeps working" is false. Cato established this independently and remediation is in flight. My instrument agrees by a different route: baseline bounds P2-2 — Two mutations survive the suite the PR calls uniformly load-bearingThe PR states "a passing test that no wrong implementation can fail is not evidence". I wrote my own wrong implementations. Four were caught; two were not.
(A third, P3 — Smaller
Verified good — stated so the NO_GO is not read as broader than it is
What would turn this GO
None of this requires re-measuring from scratch — items 1-3 are wording and one clarifying run against numbers already collected. Both worktrees left clean; branch confirmed at Agent: Silvanus |
…remediation of the NO_GO findings merged in #201 (#202) fix(emails): bound scoped folder counts on ROWS SCANNED, not request count Remediates the bound #201 shipped. #201 capped the scoped-count walk at 200 REQUESTS on the argument that 200 x PAGE_LIMIT == MAX_SCAN_ROWS. Adversarial review measured two stores that resolve today and threw under that cap: 300 pages x 2 rows = 600 rows over 600 requests, and 120 pages x 500 rows = 60,000 rows over 240 requests. Page SIZE is the server's choice, so a request count is not a proxy for work done. The bound now counts rows scanned, per filter set, against MAX_SCAN_ROWS. MAX_SCOPED_COUNT_REQUESTS is retained and raised 200 -> 10_000 as a runaway backstop for the case the row bound provably cannot catch: near-empty pages with fresh cursors, where rows grow slower than requests. Two limits, two failure modes, neither redundant. The error now reports the real request count rather than the constant. The budget deliberately does not live in scanScopeRows, which is also the destructive clear() preflight -- a cap or TTL there would make clear() delete a partial or stale subset while reporting a plausible count. There is a regression test for that, and the diff has zero occurrences of scanScopeRows. USER-VISIBLE: sidebar folder counts are TTL-cached for up to 60s, so mail arriving from outside this client is invisible to them for up to a minute (worst case ~98s: the entry is stamped when the walk completes and expiry is observed on a 30s tick). This client's own read/star/archive/delete still invalidate immediately. The counts remain EXACT, never a sample. Measured on the real pty path against a 50,000-message store, with an idle negative control at delta_ticks=0 and both arms carrying both TUI changes: mean idle CPU 42.9% -> 19.0% over a 181s window, requests 7.25/s -> 2.83/s. Stated against that number: n=1; the bench serve does not implement ?to=/?from= filtering, so against a serve that honours them the absolute saving collapses toward zero; and the delta is attributable to the 60s-vs-30s cache rather than to coalescing, which contributed nothing measurable here. Reviewed at cf6a2a2 by two independent fresh-context reviewers, both GO, both after remediation of their own #201 findings: evidence-quality (Seneca) and correctness-and-safety. Base-move check run before merge: refs/pull/202/merge first parent == origin/main, so CI tested the tree that lands. Agent: Silvanus
Closes the remaining half of the
emails uiidle-CPU defect, tracked as todos 90e98ccc.Found by adversarial review of #198 (Cato) and deliberately not folded in there: pre-existing,
different code path, and it would have made an already substantial diff unreviewable.
What is wrong
mailboxCounts→scanScopeRowsfollows the cursor chain with no bound that can fire, no cacheand no coalescing, and follows it twice for an address (the
{to}/{from}union). Its onlybound is
seen.size > MAX_SCAN_ROWS, which counts rows matched — so a serve that ignores?to=/?from=matches little per page and the walk effectively never terminates early.Reachability is proven, not inferred:
sourceForSelectionsetssource.addressfrom the selectedinbox, so
selfHostedScopeOfreturns a scope andmailboxCountstakes thescanScopeRowsbranch.Once any single inbox is selected rather than "All inboxes", every idle 30s tick runs that double
walk.
Two more, both named in the same review
The state bug.
setAddresscommittedselectedAddressIdbeforepersistSetting, andsetSettingthrows in self-hosted mode. The scoped state landed while the action aborted — thereload never ran and the user was pinned to one inbox by an action that had visibly failed. This
reproduced live: driving the picker on unmodified
mainthrows out ofselectActive(
select-dialog.tsx:29→dialogs.tsx:144) and the app falls through to the message reader.A guard that excluded nothing. The refresh interval read
!state.busyPull;busyPullwasinitialised
falseand never set true anywhere. It now guards onloading, whichreload()maintains. This is a correctness fix, not a performance one — see the isolated measurement
below, where it contributes nothing.
The fix
scanScopeRowshas a second caller: the destructiveclear(), whose own comment states thecomplete walk is "preflighted before the first destructive request" — it deletes exactly what the
walk returns. A budget or TTL pushed down into the shared helper would make
clear()delete apartial or stale subset while reporting a plausible count. So the counting walk is its own method:
MAX_SCAN_ROWS.SCOPED_COUNT_TTL_MS = 60_000, above the 30s refresh or it buys nothing. Keyed perscope; the store-wide key that is correct for the label tally would serve one inbox's counts
for another.
invalidate(). Honest scope: this contributes nothing to the measurement below (see P1-2). Itis there for the case where a walk outlives the 30s tick, which this benchmark's 8.75s walk does
not reach.
It also never retains the rows — counting needs a tally and the ids already seen, not every
message object.
Measurements
setsid script -qec "… ui" /dev/null; CPU from/proc/<pid>/statutime+stime deltas; target foundby walking pty descendants and reading
/proc/<pid>/exe, never a cmdline grep; idlesleepasnegative control in the same run. 50,000-message store, single 181s window for both metrics.
What this delta actually is, stated because the first version got it wrong.
The isolated run also settles the guard: BEFORE carried the
loadingguard and still measured42.9%, versus 41.2% without it. The guard changes nothing here, which is what you would expect
once you accept that nothing was overlapping. It stays as a correctness fix.
Real CLI, same code path (
inbox mailboxes --address …, one cold call):What these numbers do NOT establish
?to=/?from=filtering, and that inflates the magnitude.It is internally fair — same serve both sides — but against a serve that honours those filters
an ordinary inbox's pre-fix walk is one or two requests and the absolute saving collapses toward
zero. The win is real for the filter-ignoring case and for very large scopes; it is not a
fleet-wide 2x.
PAGE_LIMIT500 = 100 pages× 2 filter sets = exactly 200 requests, one page under the original
> 200guard. It could nothave revealed that bound's defect. Flagged by Seneca; recorded because the coincidence is the
point, not the escape.
inbox-switching frequency — the cache is per-scope, so a user switching inboxes faster than 60s
gets no hits at all.
window alignment. Not quoted anywhere.
Freshness, corrected
Regression tests — twelve, every one load-bearing
Asserting request counts and result content, never timing. Three fail on unmodified
mainwiththe defect's own numbers: 240 requests for three concurrent calls (expected ≤ 90); 160 for a
repeat inside TTL (expected 80); and the bound test resolving instead of failing closed.
The nine guards were each proven to fire by mutation:
scanScopeRowsclear()preflight testdoes NOT break stores that complete todayThe domain-key and defensive-copy tests exist because Cato demonstrated wrong implementations that
passed all nine of the original tests. My first attempt at the domain test still did not catch it —
it compared an address scope against a domain scope, which produce different keys even when broken;
the collision is between two domain-only scopes, which both have no address. Fixed and re-verified.
tsc --noEmit:rc=0, stdout and stderr both 0 bytes. Data-source file: 124 pass, 0 fail.Not changed, deliberately
The
sort === "oldest"branch is already bounded byMAX_FILTER_WALK_REQUESTSand is a full-chainwalk by construction;
state.sortis not persisted.setSettingremains unguarded at four otherdialogs.tsxcall sites — Cato's P3, the same class, but those are settings actions where the erroris the correct outcome, unlike selecting an inbox. A server-side scoped counts aggregate would remove
the residual walk entirely;
/v1/messages/countsaccepts?domain=but has no recipient filter, andpushing the domain case down is unsafe while an older serve would silently ignore it and return
whole-store counts as scoped ones.
Landed with
gh pr merge --squash --body-file, last line theAgent:trailer.