Semantic Output: adaptive overview and detail, raw-preserving folding, providers and copy modes

Decision / product contract — adaptive progressive disclosure

NMSh remains a real-shell/PTY frontend, not a shell replacement, terminal emulator or universal parser. Semantic Output is optional, reversible, local, post-completion presentation. Original recorded output is authoritative and always directly accessible.

Revised recommendation: semantic overviews first when useful, but NOT an entire semantic section unconditionally above raw. A concise outcome, richer diagnostic details, and an enhanced full-output view have different roles. The presentation must adapt to those roles, actual information value, trust/confidence, terminal width and user activity.

1. Compact factual overview (when useful) near the existing command result. Reuse the existing deterministic timeline summaries (Deterministic summaries for common CLI tools #6) and lifecycle row; no duplicate success/test-count lines. Keep essential outcome/severity visible even when details fold.
2. Semantic detail (when useful) as an independently foldable secondary section for error groups, test failures or structured changes.
3. Enhanced primary view (when appropriate) for large diffs, process tables and similar transformations. This should be a primary view of the command's output, not hundreds of semantic rows forced above the raw section.
4. Raw output always available, with a persistent, explicitly labeled Raw Output · N lines disclosure in the same command block. It may start fully collapsed when a sufficiently faithful/useful semantic view safely represents the outcome, but never solely because a provider produced any text.
5. Raw only whenever output is already clear, a provider is unavailable/uncertain, or the transformation could conceal an important error. Raw-only mode preserves existing Smart/Always/Off folding, including its head/tail behavior.
6. Running commands remain untouched. Stream original output normally; semantic processing runs only after completion and must not rewrite PTY data, change pipes/redirects, unexpectedly auto-fold what is being read, steal selection or jump the viewport when a provider result arrives later.
7. Copy raw by default, regardless of displayed/folded views. /copy --raw, --semantic (raw fallback), and --both remain distinct. Parse is excluded as a provider/foundation.

Illustrative outcomes, not fixed mockups or literal formatting mandates:

$ npm test
  ✓ 1,797 passed · 0 failed                [concise useful overview]
  ▸ Test details                            [optional, independently foldable]
  ▸ Raw output · 2,846 lines                [fully collapsed when safe]
  ✦ Worked for 8.4s · done 19:14           [lifecycle once, not twice]

$ git diff
  ▾ Enhanced diff · delta                   [rich primary output view]
    [rendered diff lines]
  ▸ Raw output · 420 lines                   [immediately accessible]

$ git status
  On branch main
  nothing to commit, working tree clean    [raw only: no redundant semantic view]

Why this change? CI job summaries/test explorers place outcomes ahead of detailed evidence; terminal/block interfaces (iTerm2 shell integration, VS Code terminal, Warp) generally preserve original output as the central artifact while adding status/navigation; GitLab log sections fold original details. These are useful patterns, not an industry mandate for semantic-on-top. Progressive disclosure says to foreground essentials, not to foreground every transformation. Test NMSh's live-to-complete behavior, failure diagnostics and long diffs rather than judging only static screenshots.

References: GitHub Actions summaries, workflow logs, VS Code testing, VS Code shell integration, iTerm2 shell integration, Warp blocks, GitLab log sections, progressive disclosure.

Guarantee: Raw is accessible, not necessarily expanded. Do not discard it, replace it in storage, or exclude it from raw search/copy/export. Implement in reviewable phases without duplicating existing features.

Why / boundary with existing features

Distinct jobs:

Observed current repository foundations to reuse/reconcile:

• src/output/OutputBuffer.ts: CompletedCommand stores command, output, exitCode, lifecycleText, line ownership, activities and one existing expanded; serializeCopyPayload() and completed output storage exist.
• src/output/FoldPolicy.ts: never | smart | always and deterministic fold heuristics; an optional advisory hint affects folding only, not content.
• src/output/TranscriptPresenter.ts: raw folding uses head/tail preview. Preserve it for raw-only mode; suppress head/tail only when a trustworthy semantic interpretation and safe presentation policy explicitly choose a fully collapsed raw section.
• src/output/Classifier.ts: protect LIVE/PASSTHROUGH. No live semantic rewriting.
• src/ui/BlockActions.ts: Copy command, Copy output, and Copy command + output already exist. Don't redefine the last as raw+semantic!
• src/commands/slashCommands.ts: current branch parser accepts /copy [N]; pending Copy: selections, picker, sticky Copy; Smart Folding folds long output #358 specifies far richer selectors, copy settings and picker, and Smart Folding changes. Integrate with Copy: selections, picker, sticky Copy; Smart Folding folds long output #358 rather than building a parallel copy parser/picker.
• Output folding: preserve raw PTY output with per-command presentation state #2 raw-preserving folding, Deterministic summaries for common CLI tools #6 deterministic tool-effect summaries, Theme Bridge and semantic terminal integration #304 Theme Bridge, Smart Folding leaves large diagnostic output expanded; copying must never change folding #350/Copy: selections, picker, sticky Copy; Smart Folding folds long output #358 folding and copy workflow. Theme Bridge controls colors for external tools, not the provenance/meaning of transcript output; do not conflate it with Semantic Output.

The public docs may describe an unreleased copy workflow (#358) not present in the currently fetched default-branch slash parser. Audit the integration branch/merged state at implementation time.

UX specification — adaptive overview, enhanced view and raw disclosure

One command record, one authoritative raw output, one lifecycle status. An overview is short outcome-oriented information; semantic details are expandable evidence; enhanced views present substantial output differently. Avoid creating three copies of the same fact or three fake command blocks.

[real command header / historical prompt]
[optional SHORT semantic overview; use existing summary/status when possible]
[optional independently foldable diagnostic/details section]
    OR
[optional enhanced primary view, e.g. delta diff or formatted table]
[Raw Output · N lines] [independent disclosure; ALWAYS directly accessible]
[original captured text when raw is expanded]
[existing lifecycle row, displayed once]

Display modes and layout selection

• Adaptive / Both (recommended): automatically choose useful compact overview, detail or primary enhanced view, with the same-block Raw Output disclosure always visible. Not a fixed ordering of two big sections.
• Raw only: no semantic work/display, preserve existing raw folding and transcript behavior; Semantic Output Off uses this mode.
• On demand: display existing raw normally and offer Show semantic view for supported commands, without auto-transforming historical content.
• Semantic focus (optional later): prioritize a rich view but never remove or bury raw output; raw must remain one action away. No semantic-only storage or export that discards raw.
• Consider optional Prefer raw position / Prefer semantic position only after user validation, without overriding error safety or requiring pointless dual panels.

Semantic usefulness gate — give the user a new view only when it improves understanding, not because a parser merely recognizes a command:

• Long successful tests/builds: 1–3 line factual overview first, optional detailed results, labeled raw collapsed fully when safe.
• Failed tests/builds: show concrete actionable failure/diagnostic context and warnings. A count such as 3 failed alone is insufficient grounds to hide original errors. If diagnostic coverage cannot be established, favor visible raw.
• Large diff: enhanced diff as primary output rendering if genuinely useful, adjacent separately expandable original raw; do not stack a 400-line formatted diff as an enormous preface above raw.
• Process table/logs: structured view or diagnostic synopsis only if it clarifies information and preserves visibility of important rows/fields.
• Short git status, cat, simple rg/text: ordinary raw if it already reads well. No redundant semantic area.
• Weak/ambiguous/partial result, tool unavailable: ordinary raw plus existing folding.
• Alt-screen/fullscreen, REPL, interactive, live/long-streaming commands: preserve original PTY behavior; only consider safe post-completion interpretation.

Folding and independent user state

The existing raw folding policy in FoldPolicy.ts is still based on original command/output, exit code, warnings, stream facts and line count. New semantic detail folding has its own Off / Smart / Always policy based on meaning/structure, not blindly on raw length. Preserve independent raw/semantic manual states; never reuse the same expanded boolean.

Fully collapsed raw means no head/tail rows in that particular semantic-assisted view; the labeled raw disclosure remains. In raw-only mode, preserve the existing head/tail preview. For failures and warnings, a raw default override must not silently hide essential diagnostics. Let the user manually expand/collapse either.

Live-to-complete transition (must be deliberately designed)

Raw output is streamed and rendered normally during execution. Provider results arrive only after completion, potentially asynchronously. Never reflow/insert a semantic view above the current reading position, auto-collapse selected material, or change scroll anchors simply because a formatter finished. Detect active reading/selection/viewport state; defer presentation promotion to a safe interaction boundary or show a quiet optional Enhanced view available affordance.

No semantic output is inserted into authoritative SerializedLine[], PTY stream, shell pipes or command history. A late semantic result changes only presentation if safe. On resume, regenerate derived views lazily from the recorded output without re-running commands and respecting per-block manual interaction.

Discoverability and details

• Use clear distinct labels for Raw Output, Details, Enhanced view; a short overview can be inline with an existing result without its own redundant heading.
• Section click/Enter/Space folds that section only. Preserve existing Ctrl+O and focused-control behavior until audited; never take keyboard input from a real shell/TUI.
• Block Actions: Show/hide semantic, expand/collapse raw, expand/collapse details, copy raw, copy semantic, copy raw+semantic. Contextually hide unavailable actions.
• Raw search always includes all stored raw lines, even when collapsed. Semantic search is an optional separately labeled scope, never a replacement; don't double-count duplicated summary facts. Preserve match reveal, selection and copy semantics.
• Narrow widths, Chat mode, mouse and keyboard access, screen readers, tabular reflow, sticky headers, 16/256/truecolor, NO_COLOR, scroll/resize/selection and historical reattach must remain usable.

/copy CLI and settings — preserve raw as default

Grammar

/copy [existing-selector] [--raw | --semantic | --both] [existing copy flags]

• /copy defaults to RAW output, whatever is displayed or folded; this is the recommended saved default.
• /copy --raw explicitly uses full original recorded output.
• /copy --semantic copies only the semantic textual export, irrespective of its fold state. If unavailable, fallback to raw (no clipboard failure), with concise truthful feedback Semantic unavailable · copied raw.
• /copy --both copies RAW AND SEMANTIC (raw first, semantic second), using stable plainly labeled separators; if semantic is unavailable, copy raw once and explain fallback; never duplicate raw.
• --both is the preferred spelling. Do not add --raw-and-semantic as the primary spelling: verbose, redundant. No -2 alias for a content mode: Copy: selections, picker, sticky Copy; Smart Folding folds long output #358 already reserves -N to mean “latest N blocks.”
• Flags are mutually exclusive, regardless of ordering. Conflicting/unknown flags yield helpful usage errors; no silent fallback to a different requested mode.
• /copy --raw 2, /copy 2 --semantic, /copy 1,3-5 --both, and (if Copy: selections, picker, sticky Copy; Smart Folding folds long output #358 is integrated) /copy -2 --both should work. Here -2 selects the latest two outputs; it does not mean raw/semantic/both.
• Empty output, missing record, missing/failed semantic provider, stale cached view and mixed eligible/ineligible multi-select have specified non-crashing outcomes.
• Semantic-only export includes the complete semantic overview plus available details/enhanced-view text, regardless of what is folded on screen, with readable headings, warnings, provenance and omission markers where needed. Avoid duplicate lifecycle facts, ANSI decorations and ambiguous unlabelled dumps.
• For --both, stable per-block boundaries and explicit RAW OUTPUT / SEMANTIC OUTPUT labels; never make separator lines appear to be original raw data. Honor Copy: selections, picker, sticky Copy; Smart Folding folds long output #358's output ordering, selection semantics and --status / --no-status exactly once per record.
• Existing Copy command, Copy output, Copy command + output block actions retain their meanings; adding Copy raw + semantic is a separate action. No mixing the command string into --both unless user separately requests the pre-existing command+output action.
• Interactive picker, quick copy, sticky copy, Ctrl shortcuts, selection copy, clipboard feedback, and Auto-expand copied output must reconcile with Copy: selections, picker, sticky Copy; Smart Folding folds long output #358. Copy should not unfold anything unless that independent opt-in setting is enabled; if enabled, define which selected section expands, never expand both by surprise.
• Saved copy preference choices: Raw (default), Semantic (fallback raw), Both. --raw|--semantic|--both override the saved preference for one invocation only. Clearly label a non-default global choice in settings/help, so people cannot unknowingly lose the raw copy default.
• Plain text export should use NMSh's existing clipboard sanitization contract (strip NMSh chrome/terminal controls). “Raw” means the currently stored faithful textual command output, not necessarily a byte-perfect PTY capture: audit snapshotPlain, ANSI/control handling and session retention; don't promise unattainable byte identity.

Examples

/copy                      # latest recorded command, raw (unless user changed saved default)
/copy 3 --raw              # third previous output, raw
/copy --semantic           # semantic text, raw fallback with notice
/copy 2 --both             # record 2: labeled RAW + SEMANTIC
/copy -2 --both            # pending #358 selector: latest two records, oldest first
/copy 1,3-5 --semantic     # pending #358 multi-selector

Settings / discoverability

One coherent transcript/output settings area, not a second configuration maze. Provider means how data is interpreted; presentation means how it is displayed; copy mode means what is exported.

Detect providers read-only (PATH/config, with sensible cache). Status: Not installed, Available, Enabled, Unsupported, Disabled, Timed out, Faulted, Conflicting. No automatic install or persistent shell/Git config edits. Explain skipped/unavailable semantic formatting unobtrusively, not with persistent transcript noise.

Integrate block Actions, slash autocomplete/help and #358 copy settings/picker, avoiding a competing copy preference mechanism.

Provider architecture — capabilities, not a flat list of interchangeable binaries

Do NOT implement or depend on Parse / atif-1402/parse. It is relatively new and not the chosen foundation. We want NMSh's own constrained representation boundary and proven, optional integrations, not an all-purpose external output proxy.

Prioritize an NMSh-native minimal adapter/representation when the output facts are already available. Providers can contribute parsing, syntax-aware formatting, or an external inspector action; these are different capabilities.

Potential built-in formats: Git status/diff headers, known test summaries, build results, package-manager effects from #6, reliably parsed tabular output, JSON/YAML formatting if already structured. Do not promise all formats, all commands or arbitrary localization/version variants in v1.

A provider result should be a typed read-only interpretation with a presentation role (compact overview, structured diagnostic detail, or enhanced primary view), not an arbitrary string shoved into the PTY:

• provenance native|external, provider id/version, command/output fingerprint, format/kind, role and confidence/compatibility;
• structured sections/rows/tables/diff hunks/diagnostic severities with safe textual export;
• explicit warnings, source-line/link references where supported, omitted-field marker, truncation status, fallback reason;
• renderers adapted to NMSh width/theme/accessibility, with safe characters/control sanitization;
• not-applicable, success, partial and failed outcomes; explicit completeness/critical-diagnostic coverage to decide whether raw may safely start folded. Partial output must not silently be treated as complete.

Providers must prove that an interpretation is meaningfully useful, not merely dump the same output under a second heading. Set a conservative usefulness threshold and deduplicate existing lifecycle summaries.

Provider selection rules

1. Read exact command metadata, stdin/PTY evidence, recognized format; do not infer provenance solely from superficial headers if ambiguous.
2. Native adapter first when it yields a sufficiently useful representation; otherwise selected eligible external adapter (or capability-specific user preference).
3. One chosen provider per representation capability; avoid running seven tools on one command. Explicit alternate view switch can be researched later.
4. Semantic parsing happens after completion, off the typing/render path, with bounded inputs, timeouts and cancellation. Never intercept user pipelines/redirects and never re-run arbitrary commands.
5. Prefer true machine-readable output only if it is already in the captured result; never change the user's executed command behind their back just to get JSON. Some external tools can parse captured human text; the native runner remains authoritative.
6. If a provider is missing, incompatible, slow or errors, cancel/fail closed to raw; preserve current interactive/folding state; surface nonblocking diagnostic in provider settings (not noisy transcript errors).
7. Zero additional agents/LLMs, model downloads or remote services. Optional future assisted explanation is out of scope and must not silently replace factual parsing.

Security, resource and compatibility rules

• Treat all command output and external formatter results as untrusted. Strip/validate terminal control sequences, OSC, hyperlinks, bidi tricks, SGR and unsafe screen movement according to existing transcript/host handling. Do not treat generated output as terminal instructions.
• External executables: no sh -c, no interpolated commands, constrained explicit argv and stdin, no implicit network or installs, avoid temp files, bounded stdout/stderr/memory/output size, timeouts, failure isolation, cancellation and cache invalidation. Review symlinks/PATH resolution and process lifecycle. Check license/packaging before any bundling.
• Captured output may contain secrets (tokens, URLs, logs, environment values). Local-only by default, external providers are explicitly opt-in, no telemetry, no network; do not copy provider error logs containing secrets into ordinary UI.
• Preserve shell stdin/stdout/stderr, exit code, original command identity, lifecycle timing, prompt/context and chronology. A semantic failure must never become a command failure.
• Passthrough and fullscreen TUI, interactive agents, live streams, SSH, tmux/screen, raw terminal modes, partial/rewritten progress, shell pipelines and redirected output keep original behavior. Automatic static semantics are post-completion only where safe; otherwise on-demand/reject.
• Handling large logs: process asynchronously, cache bounded derived views keyed by original-record stable identity/content hash, source revision, provider version/options and theme-independent structured data; don't duplicate huge strings unnecessarily. Evict derived caches without touching raw transcript.
• Respect Zsh/Bash/Fish integration and macOS/Linux provider availability. Work when optional programs are absent. Test Ghostty, Terminal.app, VS Code/Zed terminal and NO_COLOR.
• Reflow width/resize, selection, copy, mouse focus, historical resume, transcript cut/clear, session handoff, agent nested activity and detached sessions must not corrupt line ownership. Do not create synthetic “commands” for a semantic child section.
• Avoid treating an accidental header match as semantic provenance; ambiguous git status -s vs porcelain and localized command outputs should fail closed. Never auto-transform machine-readable intent when that cannot be established.
• No raw data loss: do not overwrite CompletedCommand.output, SerializedLine[], transcript storage or any /copy --raw path.

Implementation sketch (evaluate against actual architecture before coding)

real shell / PTY
    ├─ authoritative original output + existing transcript records
    │     ├─ existing raw folding, search, clipboard, replay/export
    │     └─ stable source for derived view
    └─ completion + existing deterministic result facts
             └─ bounded, optional semantic adapter
                   └─ typed overview / detail / enhanced-primary-view
                        └─ usefulness, fidelity and diagnostic safety gate
                              └─ adaptive non-disruptive presenter:
                                    brief factual overview where useful
                                    + rich detail or enhanced output where useful
                                    + ALWAYS-accessible independent raw disclosure

Avoid reusing one CompletedCommand.expanded for two controls or inserting semantic rows into SerializedLine[]. Model distinct raw/semantic detail states and overrides, stable ownership, view type, provider provenance, accept/reject reason and a deferred-promotion state while the reader interacts. Reuse existing #6 fact extraction instead of building a competing timeline or inventing effects. Preserve command/exit/stderr semantics; do not re-execute commands to generate JSON.

Audit TranscriptPresenter, OutputBuffer, FoldPolicy, CommandClassifier, BlockActions, config, slash parser, session serialization, selection/search/copy and #358 before choosing module boundaries. Cached semantic views are derived, bounded and invalidateable, never the record. Historical transcripts without semantics still render normally.

Delivery plan (separate reviewable PRs)

• Phase 0 — real UX/architecture audit: reconcile Output folding: preserve raw PTY output with per-command presentation state #2, Deterministic summaries for common CLI tools #6, Theme Bridge and semantic terminal integration #304, Smart Folding leaves large diagnostic output expanded; copying must never change folding #350 and Copy: selections, picker, sticky Copy; Smart Folding folds long output #358; prototype test summary, failed diagnostics, long diff, process table and raw-only. Assess scroll/selection during live-to-complete and late provider results, and whether a “semantic” panel duplicates current summaries.
• Phase 1 — adaptive native MVP: typed overview/detail/enhanced-view, usefulness and safety gate, existing-summary reuse, independent raw/detail folding, immediate raw disclosure, original-output fallback and no disruptive promotion. No mandatory external tools.
• Phase 2 — copy/settings: --raw|--semantic|--both, selectors, raw default, semantic fallback, non-duplicative export, Copy: selections, picker, sticky Copy; Smart Folding folds long output #358 picker/status/sticky integration and adaptive/user-preference controls.
• Phase 3 — providers: evaluate opt-in local jc for known structured formats and delta for safe noninteractive captured diffs; per-capability status/selection; bounded subprocess/cancellation. Defer others.
• Phase 4 — QA/polish: external inspectors ov/lnav, evaluate Difftastic, narrow-width fallbacks, session restore, performance, accessibility and docs, if useful.

Each phase is reviewable and testable, with physical QA where needed. No forced release milestone.

Acceptance criteria and regression scenarios

• Long successful test/build shows a concise useful overview before optional detail and a fully collapsed Raw Output disclosure when safe, without duplicate lifecycle summary.
• Large Git diff/table uses an enhanced primary view, not a giant mandatory semantic preface. Raw is still immediately accessible in the same block.
• Short, readable git status/plain-text command can remain entirely raw, with no cosmetic semantic duplicate.
• Failed tests/build expose useful concrete errors; provider uncertainty/incompleteness cannot hide critical evidence behind two opaque collapsed headers.
• No semantic/Off/unknown/timeout renders exact pre-feature raw-only display and folding.
• Semantic overview remains visible while separately configurable semantic detail Smart/Always/Off folds; raw has independent control and existing raw-only Smart/Always/Off policy.
• Manual raw and semantic fold/expand states are independent, survive reflow/scroll/history/resume and are not overridden by delayed formatters.
• Raw continues streaming during command execution. A late semantic result cannot insert/reorder above a scrolled/selected reader, fold content under them, steal selection or jump the scroll position; deferred enhancement works.
• Raw search/copy/export always refers to original stored output, whether folded or not; semantic search reveals its own results without double counting.
• /copy default raw; --raw, --semantic (including full hidden detail; raw fallback) and --both labeled raw+semantic work with selectors in any supported order and never double-copy raw.
• /copy -2 remains a selection for two latest outputs as specified in Copy: selections, picker, sticky Copy; Smart Folding folds long output #358, not a semantic mode shorthand; support compatible --status, picker, block actions and copy settings.
• No provider changes exit/stderr, shell PTY, pipelines, history or original output. Interactive/fullscreen/live programs remain unmodified.
• Optional provider failures, hostile ANSI, timeouts, missing dependency and secret-bearing logs fail closed to raw, don't block input and don't exfiltrate data.
• Cover build, test, Git status/diff, brew cleanup, ps, logs, narrow terminal, CJK/wide glyphs, NO_COLOR, macOS/Linux, SSH/tmux, nested activities, detached sessions, large output and resumed history.
• Benchmark overhead, avoid duplicated derived caches/lifecycle facts, update docs/help/settings/compatibility migrations and physical QA.

Open implementation questions (defaults above remain proposed)

1. Exact settings location under Transcript/Output, and whether “Prefer raw placement”/“Prefer semantic placement” is warranted after prototype testing.
2. Which existing Deterministic summaries for common CLI tools #6 deterministic summaries are already the compact overview, and how to avoid duplicate status lines.
3. Minimum typed schema for overview, detail, and enhanced primary output, including provenance, omissions and diagnostic completeness.
4. Conservative criteria for when raw can safely fully collapse; especially failures, warnings, partial outputs and localized formats.
5. How to preserve scroll anchor, selection and manual folding across asynchronous completion, deferred enhancement, resize and session resume.
6. Whether per-block raw state and semantic details state need separate persistence, and how to avoid stale derived-view serialization.
7. Noninteractive delta captured-output integration and jc format recognition without re-running commands.
8. Whether simple recoloring should count as semantic at all (recommend no, it is styling).
9. Copy export format when there are overview + detail + enhanced view without duplicating content/status.

Related: #2, #6, #304, #350, #358. This new issue is specifically the adaptive semantic overview/details/enhanced-view UX + always-accessible raw output, providers and copy contract, not a replacement for their work.

External references (optional integrations, no Parse): jc, delta, Difftastic, GRC, ov, lnav.