Skip to content

Improve TUI search, deletion, and export workflows - #723

Open
salmonumbrella wants to merge 7 commits into
kenn-io:mainfrom
salmonumbrella:fix/issues-40-41-44-72-88
Open

Improve TUI search, deletion, and export workflows#723
salmonumbrella wants to merge 7 commits into
kenn-io:mainfrom
salmonumbrella:fix/issues-40-41-44-72-88

Conversation

@salmonumbrella

Copy link
Copy Markdown
Contributor

What changed

  • Added session-only inline search history in the TUI, including draft restore and a 100-query cap.
  • Split deletion staging so d uses the selection/current row and D resolves every message-list filter or search match in the background before confirmation.
  • Added [data].export_dir and routed TUI attachment ZIPs, downloads, and opened files through it.
  • Corrected export-eml usage and default-filename guidance.
  • Documented how incremental Gmail sync and expired-history recovery reconcile messages deleted at the source.

Why

Search-heavy cleanup should not require retyping queries or selecting one loaded page at a time. Attachment exports should also have one predictable destination, while the CLI and sync docs should describe the behavior users actually get.

The all-match deletion path keeps source boundaries intact, rejects bounded semantic results, paginates fast and deep searches, stays responsive during resolution, and always presents the final count before a batch is staged.

Closes #40
Closes #41
Closes #44
Closes #72
Closes #88

Usage

  • In inline search, press Up or Down to recall earlier queries and return to the draft you were typing.
  • In a message list, press d for the current selection or D for every filter/search match.
  • Set export_dir under [data] to choose where TUI attachment exports land. Relative paths resolve from the config directory; the default is <data_dir>/exports.
  • Run msgvault export-eml <message-id> [source-message-id], optionally with -o - for stdout.

@roborev-ci

roborev-ci Bot commented Aug 30, 2026

Copy link
Copy Markdown

roborev: Combined Review (247e03c)

Verdict: Medium-severity issues found in the bulk-deletion path.

Medium

  • internal/tui/actions.go:218-254 — Bulk deletion with an active search converts all matching emails into deletion targets without restricting source types. Imported mbox/PST messages may produce manifests that delete-staged rejects as unsupported. Resolve matches through a source-aware deletion path or filter unsupported sources before creating the manifest.

  • internal/tui/actions.go:197-205 — Without an active search, bulk deletion does not apply all active list filters. In particular, WithAttachmentsOnly and empty-value bucket filters are ignored, potentially staging unrelated Gmail messages for deletion. Apply every active filter or resolve IDs through the normal filtered list path before creating the manifest.


Reviewers: 2 done | Synthesis: codex, 7s | Total: 14m31s

@salmonumbrella
salmonumbrella force-pushed the fix/issues-40-41-44-72-88 branch from 247e03c to c8f1c6f Compare August 30, 2026 12:06
@roborev-ci

roborev-ci Bot commented Aug 30, 2026

Copy link
Copy Markdown

roborev: Combined Review (c8f1c6f)

Verdict: High-risk scope and deletion-target issues remain in bulk deletion.

High

  • internal/tui/actions.go:232-234 — Bulk deletion can escape the active MessageFilter scope. Deep search and some local fast-search paths drop filters such as sender/recipient name, conversation ID, and empty-value targets, allowing D to stage messages outside the visible list. Resolve matches using the complete filter, or fail closed for unsupported filter combinations.

Medium

  • internal/tui/actions.go:269-296 — The daemon-backed engine lacks the optional message-ID resolver, so the fallback includes every Gmail result without checking MessageSummary.DeletedAt. Retained source-deleted messages may therefore enter the deletion manifest. Add a daemon-backed resolver or skip messages with non-nil DeletedAt.

Reviewers: 2 done | Synthesis: codex, 7s | Total: 14m16s

@salmonumbrella
salmonumbrella force-pushed the fix/issues-40-41-44-72-88 branch from c8f1c6f to 4fa0b70 Compare August 30, 2026 12:18
@roborev-ci

roborev-ci Bot commented Aug 30, 2026

Copy link
Copy Markdown

roborev: Combined Review (4fa0b70)

Verdict: Two medium-severity issues require attention; no critical or high-severity findings were reported.

Medium

  • internal/tui/actions.go:223-230 — Bulk deletion with a nonempty query passes unsupported filters to remote fast search, causing deletion staging to fail in filtered views. Use a compatible remote query path or apply client-side scope intersection.

  • internal/tui/actions.go:205-224 — Bulk deletion materializes all matching messages as full summaries, potentially causing excessive memory usage on large archives. Stream paginated IDs and process targets in bounded batches.


Reviewers: 2 done | Synthesis: codex, 7s | Total: 8m51s

@salmonumbrella
salmonumbrella force-pushed the fix/issues-40-41-44-72-88 branch from 4fa0b70 to 9af74a9 Compare August 30, 2026 12:29
@roborev-ci

roborev-ci Bot commented Aug 30, 2026

Copy link
Copy Markdown

roborev: Combined Review (9af74a9)

Verdict: One medium-severity issue requires attention; no critical or high-severity findings were reported.

Medium

  • Malformed search queries can widen bulk-deletion scopeinternal/tui/actions.go:205-210
    The bulk D workflow passes parsed search queries to the search function without checking parsed.Err(). Invalid filters such as before:not-a-date may leave constraints unset, causing an unfiltered deletion batch to include all live Gmail messages. Reject parser errors before staging IDs, and apply equivalent validation to the existing TUI search path where appropriate.

Reviewers: 2 done | Synthesis: codex, 7s | Total: 19m37s

@salmonumbrella
salmonumbrella force-pushed the fix/issues-40-41-44-72-88 branch 2 times, most recently from 3026d35 to 3aa9b3c Compare August 30, 2026 13:24
@roborev-ci

roborev-ci Bot commented Aug 30, 2026

Copy link
Copy Markdown

roborev: Combined Review (3aa9b3c)

Verdict: Medium-severity issue identified; no security vulnerabilities found.

Medium

  • internal/tui/actions.go:326-343 — Bulk deletion silently skips all messages when asynchronously loaded account metadata is unavailable or failed. In the normal daemon-backed TUI, pressing D may incorrectly report “no messages selected” despite matching messages existing. Gate deletion until account metadata loads successfully, resolve source metadata through the daemon, or return an explicit metadata error.

Reviewers: 2 done | Synthesis: codex, 6s | Total: 10m3s

Add session search history, all-match deletion staging, and configurable attachment exports. Correct export-eml guidance and document Gmail deletion reconciliation.
@salmonumbrella
salmonumbrella force-pushed the fix/issues-40-41-44-72-88 branch from 3aa9b3c to 5c791a3 Compare August 30, 2026 13:37
@roborev-ci

roborev-ci Bot commented Aug 30, 2026

Copy link
Copy Markdown

roborev: Combined Review (5c791a3)

No issues found.


Reviewers: 2 done | Synthesis: codex | Total: 17m50s

@wesm wesm self-assigned this Sep 1, 2026
@wesm

wesm commented Sep 1, 2026

Copy link
Copy Markdown
Member

reviewing

Bulk deletion could outlive the TUI request, drift while paging a live archive,
and record stale selection details in its manifest. Invalid partial search input
could also interrupt typing, and semantic queries were incorrectly parsed as
structured searches.

Resolve each all-match set in one backend query, preserve its exact filter
provenance, and cancel work when the user leaves it. Keep structured validation
inline and pass semantic text through unchanged.

Generated with Codex
Co-authored-by: Codex <noreply@openai.com>
@wesm

wesm commented Sep 1, 2026

Copy link
Copy Markdown
Member

I pushed a follow-up in d117726c that fixes the review issues.

Here is what changed:

  • Invalid structured search text now stays as inline feedback while you type. It no longer opens a modal or consumes the next key. Semantic search now sends the natural-language query through unchanged.
  • Uppercase D now has a clear scope: in a message list it stages every match for the current filter or search; in an aggregate view it stages the current row. Lowercase d still stages the Space selection, or the current item when nothing is selected.
  • All-match resolution can now be canceled. Esc stops it, and q stops it before showing the quit prompt.
  • Filtered and searched deletion targets are resolved in one backend query, including daemon mode. This removes the live LIMIT/OFFSET paging that could skip or duplicate messages, and avoids the extra archive-wide passes.
  • All-match manifests now describe the actual filter and search scope. A stale Space selection can no longer change the description or audit fields.
  • A filter or search with no deletable matches now reports that directly instead of saying that no messages were selected.
  • The repeated loading-state check is now one helper.

The generated API client was updated for the daemon path. The full tagged Go test suite, go vet, lint, and the generated API check all pass.

@roborev-ci

roborev-ci Bot commented Sep 1, 2026

Copy link
Copy Markdown

roborev: Combined Review (d117726)

Verdict: 3 actionable findings remain—1 High and 2 Medium severity.

High

  • internal/tui/actions.go:275-276, internal/tui/keys.go:444-452 — All-match deletion using an empty sender/domain/etc. bucket drops EmptyValueTargets, causing unrelated active Gmail messages to be returned instead of only empty-bucket matches. Preserve empty aggregate predicates and handle them in the deletion-target resolver before staging the manifest.

Medium

  • internal/daemonclient/engine_adapter.go:83-95, internal/api/handlers.go:752-755 — Semantic search forwards raw operator-shaped queries, but the daemon still parses and rejects invalid structured operators before hybrid search. Queries such as before:not-a-date explain... fail remotely. Align the client and daemon contracts by validating/stripping operators client-side or allowing raw semantic text through the daemon parser.

  • internal/daemonclient/engine_adapter.go:83-95 — Forwarding both raw query operators and the view’s message_type can widen scope. For example, message_type:sms invoice in Email mode may send both SMS and email scopes, which the server can treat as an OR. Intersect query and view message-type constraints, returning no matches for disjoint scopes.


Reviewers: 2 done | Synthesis: codex, 10s | Total: 10m1s

An empty aggregate bucket previously became an unconstrained deletion filter.
This could stage unrelated Gmail messages instead of the selected empty bucket.

Daemon-backed semantic search could also reject natural-language text that
looked like an invalid operator. A query-level message type could conflict with
the active TUI view and widen the results.

Keep empty-bucket predicates through deletion resolution. Send semantic text in
a parser-safe form, and intersect message-type constraints before the request.

Generated with Codex
Co-authored-by: Codex <noreply@openai.com>
@wesm

wesm commented Sep 1, 2026

Copy link
Copy Markdown
Member

Fixed the three follow-up review findings:

  • Deleting an “(empty)” sender, recipient, display-name, domain, or label bucket now keeps that empty-bucket filter. It cannot fall back to every active Gmail message.
  • Semantic searches can include natural-language text that looks like a broken search operator. The client now sends that text in a form the daemon treats as text instead of rejecting it.
  • A message_type: in the query is now combined safely with the current TUI view. Matching types are sent once, and conflicting types return no results instead of widening the search.

I added regression tests for each case. The full test suite, go vet ./..., and the repository lint checks all pass.

@roborev-ci

roborev-ci Bot commented Sep 1, 2026

Copy link
Copy Markdown

roborev: Combined Review (a402d94)

Verdict: Two medium-severity issues require attention.

Medium

  • internal/tui/keys.go:547-553 — During bulk deletion, q and Ctrl+C are consumed for cancellation and do not reach normal quit handling, requiring users to press them again. Cancel the resolver, then forward these keys through the standard global-key handler.

  • internal/daemonclient/engine_adapter.go:103-108 — A semantic query containing only a matching message_type: operator becomes empty after moving the operator to MessageTypes. The HTTP endpoint rejects it with missing_query. Reject operator-only queries with inline feedback or preserve a non-empty query when sending the request.


Reviewers: 2 done | Synthesis: codex, 7s | Total: 12m22s

Semantic search needs free text to rank messages. A query with only a matching
message type became empty after the client moved that filter into the HTTP
parameter, so the daemon returned a missing-query error.

Reject filter-only semantic input before it replaces the current results. Keep
the same guard in the daemon adapter so non-TUI callers cannot send an empty
search request.

Generated with Codex
Co-authored-by: Codex <noreply@openai.com>
@wesm

wesm commented Sep 1, 2026

Copy link
Copy Markdown
Member

Reviewed both follow-up findings:

  • Fixed the operator-only semantic search case. A query such as message_type:email now shows “semantic search requires free text” beside the search box and keeps the current results. The daemon client also rejects it locally instead of sending an empty query.
  • I did not change the bulk-deletion quit keys because they already work on the first keypress. Both q and Ctrl+C cancel the resolver and then continue into the normal quit handler. The existing focused test confirms that q opens the quit confirmation immediately; Ctrl+C follows the same path and quits immediately.

I added focused coverage for the semantic case. The full test suite, go vet ./..., and the repository lint checks pass.

@roborev-ci

roborev-ci Bot commented Sep 1, 2026

Copy link
Copy Markdown

roborev: Combined Review (aadc5b5)

Verdict: One medium-severity issue requires correction; no material security issues were identified.

Medium

  • Empty queries are incorrectly treated as searchesinternal/tui/actions.go:275-297, internal/daemonclient/engine_adapter.go:1022-1026
    Semantically empty queries such as subject: or whitespace are canonicalized to an empty q but still sent with search_mode, causing daemon rejection instead of staging all current matches. Treat trimmed/canonical empty queries as no-search and omit q/search_mode.

Reviewers: 2 done | Synthesis: codex, 6s | Total: 18m32s

A search box can contain whitespace or an operator with no value even though
the parser finds no search criteria. Bulk deletion treated that raw input as an
active search, so a remote daemon received an empty query with a search mode
and rejected the request.

Treat successfully parsed empty input the same as no search. Preserve malformed
query errors, and keep the deletion manifest aligned with the filter-only scope
that selected the messages.

Generated with Codex
Co-authored-by: Codex <noreply@openai.com>
@wesm

wesm commented Sep 1, 2026

Copy link
Copy Markdown
Member

Fixed this follow-up issue.

Whitespace and empty operators such as subject: now use the same filter-only deletion path as an empty search box. The deletion manifest records “all matches” without a search query or search mode, and daemon requests omit both q and search_mode. Malformed queries still return their validation error instead of falling back to a wider filter.

I added focused tests for both the TUI staging record and the exact daemon request. The full test suite, go vet ./..., and the repository lint checks pass.

@roborev-ci

roborev-ci Bot commented Sep 1, 2026

Copy link
Copy Markdown

roborev: Combined Review (66fa809)

Verdict: Two medium-severity issues require attention.

Medium

  • internal/query/sqlite.go:1507-1510 — Bulk deletion combines search predicates and the current filter with AND, unlike normal TUI searches where sender, recipient, and domain scopes use OR. This can cause deletion to affect only the intersection rather than the messages displayed. Reuse the normal query/filter composition semantics while preserving separate predicates for unsupported filter fields.

  • internal/tui/model.go:1635-1649 — Invalid or changed inline search input does not invalidate an in-flight search request. A stale response can still match searchRequestID and overwrite results after the UI reports the query as invalid. Invalidate relevant request IDs or verify responses against the current input before applying them.


Reviewers: 2 done | Synthesis: codex, 8s | Total: 32m23s

Bulk deletion could resolve a smaller set than the messages shown when a
search and a sender, recipient, or domain view filter were both active.
Inline search results could also arrive after the input changed or became
invalid and replace the current list.

Use the same filter composition for displayed and staged matches. Invalidate
older search and message-list requests as soon as the inline input or search
mode makes them obsolete.

Generated with Codex
Co-authored-by: Codex <noreply@openai.com>
@wesm

wesm commented Sep 1, 2026

Copy link
Copy Markdown
Member

Fixed both remaining review findings in 7383f92.

  • Bulk “D” deletion now uses the same sender, recipient, and domain matching rules as the normal TUI search. This keeps the staged set aligned with the messages on screen. Filters that search cannot represent directly—such as names, conversations, and empty buckets—still narrow the staged set.
  • Changing the inline search text, or switching to a mode where the text is invalid, now cancels the authority of older search and message-list requests. A late response can no longer replace the current results or clear the inline error.

I added regression coverage for fast and deep deletion searches, empty-bucket filters, changed input, and invalid input.

@roborev-ci

roborev-ci Bot commented Sep 1, 2026

Copy link
Copy Markdown

roborev: Combined Review (7383f92)

Review Verdict: Changes require fixes before merge due to one high-severity and one medium-severity issue.

High

  • Aggregate deletion can stage messages outside the displayed scopeinternal/tui/keys.go:458-465
    Aggregate D ignores the active search query and top-level attachment-only filter, so users may delete unrelated messages in the same aggregate. Resolve aggregate deletion through the same complete, search-aware scope used by message-list bulk deletion, and add an integration test covering matching and non-matching messages.

Medium

  • All-match deletion provenance is lost in remote TUI deletionsinternal/tui/actions.go:180-196; internal/daemonclient/convert.go:127
    Manifest.RawFilter is recorded but omitted during daemon-client manifest conversion, causing remote deletions to lose their search/filter scope. Preserve RawFilter in the generated API request and add an HTTP round-trip test.

Reviewers: 2 done | Synthesis: codex, 7s | Total: 16m17s

Aggregate deletion could stage messages hidden by the active search or
attachment filter. Remote TUI staging also dropped the complete match
criteria before saving through a daemon.

Use the displayed aggregate scope for resolution and carry the provenance
JSON across the daemon request so staged batches remain accurate and
auditable.

Generated with Codex
Co-authored-by: Codex <noreply@openai.com>
@wesm

wesm commented Sep 2, 2026

Copy link
Copy Markdown
Member

Fixed the two latest review findings in 8eef0b37.

  • Pressing D on a grouped row now stages only messages in the scope currently shown: the active search, attachment-only setting, account and drill-down filters, deleted-message setting, and the current row. The background lookup remains cancellable.
  • When the TUI connects through the daemon, the saved deletion record now keeps the exact search and filter details instead of dropping them.

I added a real SQLite regression test for the grouped-row scope and an HTTP round-trip test for the saved filter details. The full test suite, vet, lint, and generated-client consistency check pass.

@roborev-ci

roborev-ci Bot commented Sep 2, 2026

Copy link
Copy Markdown

roborev: Combined Review (8eef0b3)

Verdict: Two high-severity correctness issues remain in bulk deletion resolution.

High

  • internal/tui/actions.go:296-303, internal/query/sqlite.go:1278-1469: All-match deletion for empty sender, recipient, name, domain, or label buckets ignores EmptyValueTargets, potentially staging every active Gmail message. Apply empty-value predicates in the authoritative filter resolver and add an integration test.

  • internal/query/sqlite.go:1491-1514: Bulk search resolution does not preserve Fast-search filter semantics. Converting filter.Label into SQLite q.Labels changes exact-label matching into substring matching, so a Work view may stage Homework messages. Share predicate logic with displayed Fast results or retain view filters as independent predicates.


Reviewers: 2 done | Synthesis: codex, 7s | Total: 15m5s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants