diff --git a/docs/design/background-tasks.md b/docs/design/background-tasks.md new file mode 100644 index 00000000..dd9266c6 --- /dev/null +++ b/docs/design/background-tasks.md @@ -0,0 +1,179 @@ +# Background tasks + +Issue: [#382](https://github.com/raiseCatError/notMyShell/issues/382). A core terminal workflow feature: run +long-running commands (`adb logcat`, `tail -f`, `journalctl -f`, `docker logs -f`, dev servers, watchers, `ping`, +`scrcpy`) beside the normal shell instead of in front of it. + +Status: **core modules and tests** (`src/backgroundTasks/`: detector, preferences, output, supervisor). The task bar +and `/background` manager are a stacked PR; wiring into the session service and the frontend is the integration step +described below. Nothing here changes the running app yet. + +## Why a separate, supervised process + +The real shell runs one foreground job at a time; NMSh's composer, queue (#357) and input awareness (#348) all depend +on that. Its job control (`&`, `bg`, `fg`) belongs to the person and stays untouched. A background task therefore +cannot be "the shell, but in the background". It is an **NMSh-owned process in its own pseudo-terminal**: + +- a PTY, because programs behave as in a terminal (line buffering, colors, `scrcpy`/`adb` checking `isatty`), exactly + like the foreground would show them; +- its own session and process group (the PTY makes the child a session leader), so a stop reaches the task's + processes, including pipeline stages, and nothing else. + +Alternatives rejected: sending `&` to the live shell (the shell's job then shares the foreground terminal, its output +interleaves with every command, and NMSh cannot stop it truthfully); pipes instead of a PTY (changes program +behavior); moving a running foreground command (impossible without changing its controlling terminal and session; +NMSh never does it, never kills and re-runs it). + +### What a task inherits, and what it does not + +The command text is passed **as one argument** to the session's own shell: ` -c `. NMSh never +re-quotes, concatenates or re-parses it, so quoting means exactly what it means in the shell (tested with mixed +quotes, `$` in single quotes and escaped double quotes). + +A task runs in the live shell's reported working directory, with the session's launch environment plus the live +`PATH`. It does **not** see the live shell's aliases, functions, unexported variables, options, or variables exported +after the session started. The suggestion says so, and the detector never recognizes commands that obviously depend +on live state (expansions, substitutions). If a command needs that state, the person runs it in the foreground. + +## Detection (`detector.ts`) + +Conservative and explainable; nothing is executed or expanded, and classification of 2,000 realistic commands takes +well under 500 ms in tests (it runs only on submission, never per keystroke). + +- A command is recognized only when its words say it keeps running: `adb logcat` (but not `adb logcat -d/-c/-t N`, + `adb devices`), `tail -f/-F/--follow` (not plain `tail`), `journalctl -f`, `docker|podman|kubectl logs -f`, + `kubectl get --watch`, `ping` without a count, `scrcpy` (not `--version`/`--list-*`), `tsc --watch`, + `webpack serve|--watch`, `cargo watch`, `python -m http.server`, `php -S`, `rails server`, `flask run`, + `hugo server`, `jekyll serve`, `ng serve`, `vite`, `next|nuxt|astro dev|start`, `nodemon`. +- Package scripts `dev`, `start`, `serve`, `watch`, `preview`, `storybook` are only **likely** (it depends on the + project); `npm test` and builds are not recognized. +- adb's global `-d` (USB device) is told apart from logcat's `-d` (dump and exit). +- Only plain pipelines are classified (the first stage decides: `adb logcat | grep X`). Substitutions, expansions in + double quotes, redirections, heredocs, lists, background jobs, subshells, multi-line input, `sudo`/`doas`/`ssh`, + and full-screen or interactive programs (`top`, `less`, editors, `watch`, `tmux`, `fzf`, …) are never recognized. +- Families are specific (`adb logcat`, `npm run dev`), so a remembered decision never covers a whole program. + +## Preferences (`preferences.ts`) + +`background-tasks.json` in NMSh's config directory, written atomically with mode 0600. It stores families and +settings, never commands, arguments or output. + +- Per family: **Always** (run in the background without asking), **Never** (do not ask), or unset (**Ask**). +- Suggestions on or off; remembered Always rules still apply when suggestions are off. +- No global "always background" mode. An invocation can always be forced to the foreground. +- Retention per task (lines and bytes, clamped to 100–100,000 lines and 64 KiB–32 MiB), finished tasks kept, + notifications (never / on failure / always), bar collapsed or expanded, whether Up enters the bar. + +`decide()` turns a classification and the preferences into foreground / suggest / background. + +## Supervisor (`supervisor.ts`) + +- `start` validates the command (non-empty, no NUL, bounded), an absolute shell and an existing directory, and a + running-task limit (16 by default), then spawns the PTY. The environment is used for the spawn only; task records + never carry it. +- States come from evidence: `running`; `stopping` after a stop request; `stopped` (it exited after a request), + `exited` (status 0) or `failed` (non-zero or an unrequested signal) **only after the exit is observed**. +- **Stop** is SIGINT to the task's process group, what Ctrl+C does in a terminal. It is a signal, not the byte 0x03 + written to the task's input, which a program in raw mode reads as data. `nextEscalation` offers SIGTERM, then + SIGKILL; each is an explicit request. NMSh's interactive shell, other tasks, and processes that left the group + (an adb server started by `adb logcat` detaches with `setsid`) are never signalled (tested). +- After a stop request the task is the whole process group: a shell that does not `exec` its command (dash, Ubuntu's + `/bin/sh`) dies of SIGINT while a program that ignores SIGINT keeps running, so the task stays `stopping`, and + escalation still reaches the group, until the group is gone (found by Linux CI; reproduced on macOS in tests). +- When the leader exits on its own while members of its group still run (they ignored the hangup), the task says so + (`leftovers`) and they are killed only on an explicit request. The group is probed every 250 ms until it is empty, + however long that takes, and then forgotten, so `leftovers` never goes stale and a later kill request finds nothing + to signal. +- **Remove** deletes a finished task and its output; a running task is never removed, and neither is a finished task + whose processes still run (nothing could stop them afterwards). **Dismissing** a view is a presentation matter and + touches no task. These are three different operations. +- Each stop request records when it was made (`stopRequestedAt`), so a view can pace escalation: a repeated or held + key does not turn an interrupt into SIGKILL before the program had a moment to exit. +- `shutdown` ends every task (interrupt, terminate, kill, with a grace period) for an explicit session exit. +- Output notifications are coalesced to at most one per 50 ms per task; state changes are immediate. +- Group ids and reuse: a process group's id cannot be reused while any member exists. Signals go to the group while + its leader is unreaped, or after the leader exited while the supervisor's probe (signal 0, at most 250 ms old) says + the group still has members; a group seen empty is forgotten. The residual window is a group that empties and whose + id is reused within one probe interval, just as a kill is requested; NMSh cannot close it without a process + handle for groups (none is portable). + +## Output (`output.ts`) + +PTY output is one stream (stdout and stderr merged exactly as a terminal shows them; separating them would need +pipes and change program behavior). Each completed line is sanitized with NMSh's `stripTerminalControls`: colors, +OSC titles and hyperlinks, cursor movement, C0/C1 controls and bidi overrides are removed, so a log line cannot +restyle NMSh, move its cursor or pose as another task. `\r\r\n` (a program's `\r\n` through a PTY) ends a line; a +lone `\r` overwrites it, so progress lines keep their last state. Lines are numbered (`seq`), so a view knows what is +new; the stream is bounded by lines and bytes, counts what it dropped and what it received, and splits endless lines. +An 8 MiB hostile burst stays within a 64 KiB bound in tests, and line handling is linear: a 64 Ki run of carriage +returns, which a trailing-`\r` regular expression took seconds over, is a test. + +## Lifecycle and ownership (integration) + +The supervisor belongs to the **session service** (`nmshd`), one per session, beside the session's shell PTY: + +| Event | Behavior | +| --- | --- | +| Frontend closes or detaches | tasks keep running (they are the service's, like the shell) | +| Reattach | the frontend receives the task list and each task's bounded recent output; nothing is started again | +| Session exited or deleted explicitly | `shutdown`: interrupt, then terminate, then kill what remains; the result is reported | +| Service restart or crash | tasks of the previous service are **lost**: listed as such, never claimed as controlled, never restarted | +| Session restore from the journal | finished task records may be shown; nothing is re-run | + +Journal: task records (id, command, family, cwd, times, state, exit status) may be journaled; output follows the +existing journal budget and privacy rules, and the environment is never stored. + +### Service protocol (to add with the integration) + +Client → service: `task-start {command, family?, cwd}` (the service supplies shell, environment and PATH), +`task-stop {taskId, level}`, `task-remove {taskId}`, `task-output {taskId, after: seq}`. +Service → client: `tasks {tasks}` on attach and on change, `task-output {taskId, lines, partial, dropped}`. +A service without the feature does not list `background-tasks` in `SERVICE_FEATURES`; the frontend then offers no +background execution and says why. + +### Frontend contract (TerminalApp, after #359's checkpoint) + +1. **Submission.** Before sending a submitted command to the shell, call `classifyCommand` and `decide`. `foreground` + → unchanged path. `suggest` → show the suggestion strip (UI PR) and wait for the choice; Enter keeps the normal + foreground path. `background` → `task-start`. +2. **Never while foreground input is owned.** No classification or task-bar key handling while a foreground command + is running or waiting for input (#348), while passthrough owns the terminal, or for queued commands (#357's queue + keeps its foreground semantics). +3. **Transcript.** Each task gets an NMSh-owned block (`Background: adb logcat · t3`) with its live output, folded and + bounded; foreground command blocks never receive task output, and task output never enters the shell's PTY. +4. **Keys and the bar** are in the UI PR (`background-task-ui.md`). + +Conflict-prone paths: `src/app/TerminalApp.ts`, `src/session/SessionService.ts`, `src/session/SessionProtocol.ts`, +`src/commands/slashCommands.ts`. + +## Security review + +| Concern | Handling | +| --- | --- | +| Command meaning | one argv element to the session's shell; no NMSh parsing or quoting | +| Automatic execution | only for a family the person set to Always; never from the heuristic alone | +| Environment and secrets | used for the spawn only; not on records, not in preferences; journals keep no environment | +| Signals | only to the task's own process group, only while its leader is unreaped; escalation explicit | +| Foreign processes | never signalled; detached helpers survive (tested) | +| Hostile output | sanitized per line; bounded; cannot reach the foreground PTY | +| Resource exhaustion | per-task line/byte bounds, running-task limit, coalesced notifications | +| Multiple NMSh instances | tasks belong to one session in one service; another frontend sees them only by attaching to that session | +| Restart/crash | unknown tasks are lost, not adopted | + +## Tests + +`tests/backgroundSupervisor.test.ts` (real PTYs and controlled programs): a logcat-like stream stopped by SIGINT and +reported only after exit; a program ignoring SIGINT stays `stopping` until an explicit SIGTERM, one ignoring both +until SIGKILL; a pipeline stopped as a whole while a detached adb-server-like helper and another task keep running; +leftover group members reported and killed only on request, a leftover group that ends later noticed and forgotten +and never removed while it runs; exact quoting; exit status; environment never on records; +refusals; an 8 MiB hostile burst. `tests/backgroundDetector.test.ts`: recognition and refusal tables, timing, +decisions, preference normalization and private storage, output line handling. + +## Physical QA (not performed) + +- Real `adb logcat` with a device or emulator: stream, stop with `x`, confirm the adb server and other adb commands + keep working. +- Real `scrcpy`: the window closes on stop; escalation if it does not. +- A dev server (`npm run dev`) on macOS and Linux; WSL2 where available. +- Detach the frontend while tasks run, reattach, confirm state and output. diff --git a/src/backgroundTasks/detector.ts b/src/backgroundTasks/detector.ts new file mode 100644 index 00000000..6e7067ee --- /dev/null +++ b/src/backgroundTasks/detector.ts @@ -0,0 +1,164 @@ +/** + * Which submitted commands may keep running (#382). Conservative and + * explainable: a command is recognized only when its words alone say it + * streams or serves (`adb logcat`, `tail -f`, `npm run dev`), never from the + * program name alone (`adb devices`, `tail`, `npm test` return). Anything + * whose meaning depends on live shell state or would change if run elsewhere + * is not recognized: substitutions, redirections, heredocs, lists, subshells, + * background jobs, `sudo`/`ssh`, and full-screen or interactive programs. + * + * Classification reads only the command text. Nothing is executed, expanded + * or looked up, so it costs microseconds and can run on submission. + */ + +export type TaskKind = 'logging' | 'following' | 'monitoring' | 'server' | 'watcher' | 'gui'; + +export interface Classification { + /** Stable family for remembered decisions: specific enough that `adb logcat` never covers all of `adb`. */ + family: string; + kind: TaskKind; + /** `streams`: runs until interrupted. `likely`: usually keeps running, depending on the project. */ + likelihood: 'streams' | 'likely'; + /** One sentence for the suggestion. */ + reason: string; +} + +export type Unrecognized = {family?: undefined; why: string}; + +export const MAX_CLASSIFIED_LENGTH = 2048; + +/** + * Shell words of a simple pipeline: quotes and backslashes are honored; anything that needs a real shell parser or + * live state to understand makes the command unclassifiable (it then runs as typed, in the foreground, unless the + * person chooses otherwise). + */ +export function simpleStages(command: string): string[][] | {why: string} { + if (command.length > MAX_CLASSIFIED_LENGTH) return {why: 'too long to classify'}; + const stages: string[][] = [[]]; + let word = '', inWord = false, quote: '"' | "'" | undefined; + const end = () => { if (inWord) stages.at(-1)!.push(word); word = ''; inWord = false; }; + for (let index = 0; index < command.length; index += 1) { + const char = command[index]!; + if (quote === "'") { if (char === "'") quote = undefined; else word += char; continue; } + if (quote === '"') { + if (char === '"') { quote = undefined; continue; } + if (char === '$' || char === '`') return {why: 'uses expansion inside quotes'}; + if (char === '\\' && index + 1 < command.length) { word += command[++index]!; continue; } + word += char; continue; + } + if (char === "'" || char === '"') { quote = char; inWord = true; continue; } + if (char === '\\') { if (index + 1 >= command.length) return {why: 'ends with a backslash'}; word += command[++index]!; inWord = true; continue; } + if (char === ' ' || char === '\t') { end(); continue; } + if (char === '|') { + if (command[index + 1] === '|' || command[index + 1] === '&') return {why: 'uses a command list'}; + end(); + if (!stages.at(-1)!.length) return {why: 'has an empty pipeline stage'}; + stages.push([]); + continue; + } + if (char === '\n' || char === '\r') return {why: 'spans several lines'}; + if ('$`;&<>(){}'.includes(char)) return {why: char === '$' || char === '`' ? 'uses expansion or substitution' : char === ';' || char === '&' ? 'uses a command list or a background job' + : char === '<' || char === '>' ? 'uses redirection' : 'uses a subshell or group'}; + if (char === '#' && !inWord) break; + word += char; inWord = true; + } + if (quote) return {why: 'has an unclosed quote'}; + end(); + if (stages.some(stage => !stage.length)) return {why: 'has an empty pipeline stage'}; + return stages; +} + +/** Programs that need the terminal itself or the person's live input: never moved away from the foreground. */ +const FOREGROUND_ONLY = new Set(['sudo', 'doas', 'su', 'ssh', 'mosh', 'sftp', 'ftp', 'telnet', 'top', 'htop', 'btop', 'atop', 'glances', 'less', 'more', 'most', 'man', + 'vi', 'vim', 'nvim', 'nano', 'emacs', 'hx', 'micro', 'tmux', 'screen', 'zellij', 'fzf', 'sk', 'lazygit', 'tig', 'k9s', 'watch', 'exec', 'nohup', 'source', '.', 'cd', 'export']); + +const ENV_ASSIGNMENT = /^[A-Za-z_][A-Za-z0-9_]*=/u; +const has = (args: readonly string[], ...flags: string[]) => args.some(arg => flags.includes(arg) || flags.some(flag => flag.startsWith('--') && arg.startsWith(`${flag}=`))); +/** A short flag cluster containing a letter (`-fn100` has f): only for clusters, not long options. */ +const clusterHas = (args: readonly string[], letter: string) => args.some(arg => /^-[A-Za-z]+\d*$/u.test(arg) && arg.slice(1).replace(/\d+$/u, '').includes(letter)); + +const result = (family: string, kind: TaskKind, likelihood: Classification['likelihood'], reason: string): Classification => ({family, kind, likelihood, reason}); + +/** adb's global options before the subcommand (`adb -s SERIAL logcat`); `-d` there means "USB device", not "dump". */ +function adbSubcommand(args: readonly string[]): {name?: string; rest: string[]} { + let index = 0; + while (index < args.length && args[index]!.startsWith('-')) index += ['-s', '-t', '-H', '-P', '-L'].includes(args[index]!) ? 2 : 1; + return {name: args[index], rest: args.slice(index + 1)}; +} + +const SCRIPT_SERVERS = new Set(['dev', 'start', 'serve', 'watch', 'preview', 'storybook']); + +function classifyStage(words: readonly string[]): Classification | undefined { + const [program, ...args] = words; + if (!program) return undefined; + const name = program.split('/').at(-1)!; + switch (name) { + case 'adb': { + const sub = adbSubcommand(args); + if (sub.name !== 'logcat') return undefined; + // Dump, clear, statistics, buffer-size queries and counted reads return on their own. + if (has(sub.rest, '-d', '-c', '--clear', '-g', '--buffer-size', '-S', '--statistics', '-m', '--max-count', '-t', '-h', '--help') || clusterHas(sub.rest, 'd')) return undefined; + return result('adb logcat', 'logging', 'streams', 'adb logcat keeps streaming device logs until it is interrupted.'); + } + case 'tail': return has(args, '-f', '-F', '--follow') || clusterHas(args, 'f') || clusterHas(args, 'F') + ? result('tail -f', 'following', 'streams', 'tail -f keeps following the file until it is interrupted.') : undefined; + case 'journalctl': return has(args, '-f', '--follow') || clusterHas(args, 'f') + ? result('journalctl -f', 'following', 'streams', 'journalctl -f keeps following the journal until it is interrupted.') : undefined; + case 'docker': case 'podman': case 'kubectl': case 'docker-compose': { + const logs = args.indexOf('logs'); + const following = logs !== -1 && (has(args.slice(logs + 1), '-f', '--follow') || clusterHas(args.slice(logs + 1), 'f')); + if (following) return result(`${name}${args[0] === 'compose' ? ' compose' : ''} logs -f`, 'following', 'streams', `${name} logs -f keeps following the logs until it is interrupted.`); + if (name === 'kubectl' && args[0] === 'get' && has(args, '-w', '--watch')) return result('kubectl get --watch', 'monitoring', 'streams', 'kubectl get --watch keeps reporting changes until it is interrupted.'); + return undefined; + } + case 'ping': case 'ping6': + return has(args, '-c', '-n', '--count') || clusterHas(args, 'c') ? undefined + : result(name, 'monitoring', 'streams', `${name} without a count keeps running until it is interrupted.`); + case 'scrcpy': return has(args, '--help', '-h', '--version', '-v', '--list-devices', '--list-displays', '--list-encoders', '--list-cameras') ? undefined + : result('scrcpy', 'gui', 'streams', 'scrcpy keeps running while its window is open.'); + case 'npm': case 'pnpm': case 'yarn': case 'bun': { + const script = args[0] === 'run' || args[0] === 'run-script' ? args[1] : args[0]; + return script && SCRIPT_SERVERS.has(script) + ? result(`${name} run ${script}`, 'server', 'likely', `${name} run ${script} usually starts a server or watcher that keeps running, depending on this project.`) : undefined; + } + case 'vite': return !args[0] || ['dev', 'serve', 'preview'].includes(args[0]) ? result('vite', 'server', 'likely', 'vite serves the project until it is interrupted.') : undefined; + case 'next': case 'nuxt': case 'astro': return ['dev', 'start', 'preview'].includes(args[0] ?? '') ? result(`${name} ${args[0]}`, 'server', 'likely', `${name} ${args[0]} keeps serving until it is interrupted.`) : undefined; + case 'nodemon': return result('nodemon', 'watcher', 'likely', 'nodemon keeps restarting the program on changes until it is interrupted.'); + case 'tsc': return has(args, '-w', '--watch') ? result('tsc --watch', 'watcher', 'streams', 'tsc --watch keeps compiling on changes until it is interrupted.') : undefined; + case 'webpack': return args[0] === 'serve' || has(args, '--watch', '-w') ? result(args[0] === 'serve' ? 'webpack serve' : 'webpack --watch', args[0] === 'serve' ? 'server' : 'watcher', 'streams', 'webpack keeps running until it is interrupted.') : undefined; + case 'cargo': return args[0] === 'watch' ? result('cargo watch', 'watcher', 'streams', 'cargo watch keeps rebuilding on changes until it is interrupted.') : undefined; + case 'python': case 'python3': return args[0] === '-m' && args[1] === 'http.server' ? result(`${name} -m http.server`, 'server', 'streams', 'http.server keeps serving until it is interrupted.') : undefined; + case 'php': return has(args, '-S') ? result('php -S', 'server', 'streams', 'php -S keeps serving until it is interrupted.') : undefined; + case 'rails': return ['server', 's'].includes(args[0] ?? '') ? result('rails server', 'server', 'streams', 'rails server keeps serving until it is interrupted.') : undefined; + case 'flask': return args[0] === 'run' ? result('flask run', 'server', 'streams', 'flask run keeps serving until it is interrupted.') : undefined; + case 'hugo': return args[0] === 'server' || args[0] === 'serve' ? result('hugo server', 'server', 'streams', 'hugo server keeps serving until it is interrupted.') : undefined; + case 'jekyll': return args[0] === 'serve' || args[0] === 's' ? result('jekyll serve', 'server', 'streams', 'jekyll serve keeps serving until it is interrupted.') : undefined; + case 'ng': return args[0] === 'serve' ? result('ng serve', 'server', 'streams', 'ng serve keeps serving until it is interrupted.') : undefined; + default: return undefined; + } +} + +/** + * The classification of a submitted command, or why it is not recognized. Only the first stage of a pipeline decides + * (`adb logcat | grep Foo` streams because adb logcat does); every stage must be a plain program, and none may be a + * foreground-only program. + */ +export function classifyCommand(command: string): Classification | Unrecognized { + const trimmed = command.trim(); + if (!trimmed) return {why: 'empty'}; + if (trimmed.startsWith('/')) { + // A slash command is NMSh's; an absolute path to a program is fine. + if (!/^\/[^\s]*\//u.test(trimmed)) return {why: 'an NMSh command'}; + } + const stages = simpleStages(trimmed); + if (!Array.isArray(stages)) return {why: stages.why}; + // Leading `NAME=value` words set the program's environment (the shell applies them under `-c` too); the program follows. + const commands = stages.map(stage => { const start = stage.findIndex(word => !ENV_ASSIGNMENT.test(word)); return start === -1 ? [] : stage.slice(start); }); + if (commands.some(stage => !stage.length)) return {why: 'only sets variables'}; + for (const stage of commands) if (FOREGROUND_ONLY.has(stage[0]!.split('/').at(-1)!)) return {why: `${stage[0]} needs the terminal or the shell itself`}; + return classifyStage(commands[0]!) ?? {why: 'not a command NMSh knows to keep running'}; +} + +export function isClassified(value: Classification | Unrecognized): value is Classification { + return value.family !== undefined; +} diff --git a/src/backgroundTasks/output.ts b/src/backgroundTasks/output.ts new file mode 100644 index 00000000..5e081f08 --- /dev/null +++ b/src/backgroundTasks/output.ts @@ -0,0 +1,109 @@ +import {stripTerminalControls} from '../util/terminalControls.js'; + +/** + * One background task's output (#382), as NMSh presents it: plain text + * lines, bounded in count and bytes, each numbered so a view can tell what + * is new. A PTY merges stdout and stderr exactly as a terminal shows them, + * so the stream is one stream; separating them would need pipes, which + * change how programs buffer and color their output. + * + * Output is hostile text: escape sequences (colors, OSC titles and links, + * cursor movement), C0/C1 controls and bidi overrides are removed when a + * line completes, so nothing a program prints can restyle NMSh, move its + * cursor or pose as another task. Carriage-return progress lines keep only + * their last state. Invalid UTF-8 arrives already replaced (U+FFFD) by the + * PTY decoder. + */ + +export interface OutputLine { + /** 1-based, increasing for the task's lifetime (also across dropped lines). */ + seq: number; + text: string; +} + +export interface OutputLimits { + lines: number; + bytes: number; + /** Longest line kept, in characters; longer lines are split. */ + lineLength?: number; +} + +const DEFAULT_LINE_LENGTH = 4096; + +export class TaskOutput { + private readonly kept: OutputLine[] = []; + private keptBytes = 0; + private partialRaw = ''; + private nextSeq = 1; + /** Lines removed to stay within the limits (oldest first). */ + dropped = 0; + /** Bytes received in total, before any bound. */ + received = 0; + + constructor(private readonly limits: OutputLimits) {} + + /** Append raw PTY text. Complete lines are sanitized and kept; a trailing partial line waits for its end. */ + append(chunk: string): void { + this.received += Buffer.byteLength(chunk); + let data = this.partialRaw + chunk; + const maxRaw = (this.limits.lineLength ?? DEFAULT_LINE_LENGTH) * 4; + let newline: number; + while ((newline = data.indexOf('\n')) >= 0) { + this.push(data.slice(0, newline)); + data = data.slice(newline + 1); + } + // A line that never ends (a stream without newlines) is split rather than held without bound. + while (data.length > maxRaw) { this.push(data.slice(0, maxRaw)); data = data.slice(maxRaw); } + this.partialRaw = data; + } + + /** The unfinished last line as it would be shown now (a progress line shows its latest state). */ + partial(): string { + return clean(this.partialRaw, this.limits.lineLength ?? DEFAULT_LINE_LENGTH); + } + + /** Flush the unfinished line, when the task ends. */ + finish(): void { + if (this.partialRaw) this.push(this.partialRaw); + this.partialRaw = ''; + } + + lines(): readonly OutputLine[] { return this.kept; } + + /** Lines after `seq` (what a view has not shown yet). */ + since(seq: number): OutputLine[] { + const first = this.kept.findIndex(line => line.seq > seq); + return first === -1 ? [] : this.kept.slice(first); + } + + get lastSeq(): number { return this.nextSeq - 1; } + get bytes(): number { return this.keptBytes; } + + private push(raw: string): void { + const length = this.limits.lineLength ?? DEFAULT_LINE_LENGTH; + const text = clean(raw, length * 4); + for (let start = 0; start === 0 || start < text.length; start += length) { + const piece = text.slice(start, start + length); + this.kept.push({seq: this.nextSeq++, text: piece}); + this.keptBytes += Buffer.byteLength(piece) + 1; + if (text.length <= length) break; + } + while (this.kept.length > this.limits.lines || (this.keptBytes > this.limits.bytes && this.kept.length > 1)) { + const removed = this.kept.shift()!; + this.keptBytes -= Buffer.byteLength(removed.text) + 1; + this.dropped += 1; + } + } +} + +/** One raw line → display text: `\r\n` ends a line, a lone `\r` overwrites it (progress output), escapes and controls go. */ +export function clean(raw: string, maxLength: number): string { + // A PTY turns a program's "\r\n" into "\r\r\n": every trailing \r belongs to the line ending, not to a progress overwrite. + // Trimmed by hand: `/\r+$/` retries from every \r of a long run, which is quadratic on hostile output. + let end = raw.length; + while (end > 0 && raw.charCodeAt(end - 1) === 13) end -= 1; + const line = raw.slice(0, end); + const last = line.lastIndexOf('\r'); + const visible = last === -1 ? line : line.slice(last + 1); + return stripTerminalControls(visible.replace(/\t/gu, ' '), maxLength); +} diff --git a/src/backgroundTasks/preferences.ts b/src/backgroundTasks/preferences.ts new file mode 100644 index 00000000..04786a59 --- /dev/null +++ b/src/backgroundTasks/preferences.ts @@ -0,0 +1,114 @@ +import {mkdirSync, readFileSync, renameSync, writeFileSync} from 'node:fs'; +import {dirname, join} from 'node:path'; +import {nmshConfigDirectory} from '../configuration/paths.js'; +import type {Classification} from './detector.js'; + +/** + * Background task preferences (#382). Remembered decisions are per command + * family (`adb logcat`, `npm run dev`), never per program, so "always" for + * `adb logcat` never covers `adb devices`. There is no global "always + * background" mode: automatic background execution happens only for a family + * the person explicitly set to Always, and an invocation can still be forced + * to the foreground. + */ + +export type FamilyDecision = 'always' | 'never'; + +export interface BackgroundPreferences { + version: 1; + /** Ask about recognized families, or never suggest (remembered Always decisions still apply). */ + suggestions: 'ask' | 'off'; + families: Record; + retention: { + /** Output kept per task, in lines and bytes (whichever is reached first). */ + lines: number; + bytes: number; + /** Finished tasks kept in the list before the oldest are removed. */ + finishedKept: number; + }; + /** A notice when a task ends: never, only when it fails, or always. */ + notify: 'never' | 'failure' | 'always'; + /** Show the task bar collapsed (one summary row) or expanded. */ + bar: 'collapsed' | 'expanded'; + /** Up from an idle composer focuses the task bar (like the agent shelf); off leaves Shift+Up and /background. */ + upArrow: 'idle' | 'off'; +} + +export const DEFAULT_PREFERENCES = (): BackgroundPreferences => ({ + version: 1, suggestions: 'ask', families: {}, retention: {lines: 5000, bytes: 2 * 1024 * 1024, finishedKept: 20}, notify: 'failure', bar: 'collapsed', upArrow: 'idle', +}); + +const LIMITS = {lines: [100, 100_000], bytes: [64 * 1024, 32 * 1024 * 1024], finishedKept: [0, 200]} as const; +const FAMILY = /^[a-z0-9][\w .:@/+-]{0,63}$/iu; + +const clamp = (value: unknown, [low, high]: readonly [number, number], fallback: number) => + typeof value === 'number' && Number.isFinite(value) ? Math.max(low, Math.min(high, Math.round(value))) : fallback; + +/** Any stored value to valid preferences: unknown fields dropped, numbers clamped, at most 256 family decisions. */ +export function normalizePreferences(value: unknown): BackgroundPreferences { + const defaults = DEFAULT_PREFERENCES(); + if (typeof value !== 'object' || value === null || Array.isArray(value)) return defaults; + const data = value as Record; + const families: Record = {}; + if (typeof data.families === 'object' && data.families !== null && !Array.isArray(data.families)) { + for (const [family, decision] of Object.entries(data.families).slice(0, 256)) { + if (FAMILY.test(family) && (decision === 'always' || decision === 'never')) families[family] = decision; + } + } + const retention = typeof data.retention === 'object' && data.retention !== null ? data.retention as Record : {}; + return { + version: 1, + suggestions: data.suggestions === 'off' ? 'off' : 'ask', + families, + retention: { + lines: clamp(retention.lines, LIMITS.lines, defaults.retention.lines), + bytes: clamp(retention.bytes, LIMITS.bytes, defaults.retention.bytes), + finishedKept: clamp(retention.finishedKept, LIMITS.finishedKept, defaults.retention.finishedKept), + }, + notify: data.notify === 'never' || data.notify === 'always' ? data.notify : 'failure', + bar: data.bar === 'expanded' ? 'expanded' : 'collapsed', + upArrow: data.upArrow === 'off' ? 'off' : 'idle', + }; +} + +export type Decision = + | {action: 'foreground'; reason: string} + | {action: 'suggest'; classification: Classification} + | {action: 'background'; classification: Classification; reason: string}; + +/** + * What happens when a command is submitted. Unrecognized commands, forced-foreground invocations and Never families + * run normally; a recognized family is suggested when suggestions are on; only an explicit Always runs it in the + * background without asking. + */ +export function decide(classification: Classification | undefined, preferences: BackgroundPreferences, override?: 'foreground'): Decision { + if (override === 'foreground') return {action: 'foreground', reason: 'run in the foreground for this invocation'}; + if (!classification) return {action: 'foreground', reason: 'not recognized as long-running'}; + const remembered = preferences.families[classification.family]; + if (remembered === 'always') return {action: 'background', classification, reason: `you chose to always run ${classification.family} in the background`}; + if (remembered === 'never') return {action: 'foreground', reason: `you chose never to be asked about ${classification.family}`}; + return preferences.suggestions === 'ask' ? {action: 'suggest', classification} : {action: 'foreground', reason: 'suggestions are off'}; +} + +/** A remembered decision for one family, or undefined to forget it (back to Ask). */ +export function rememberFamily(preferences: BackgroundPreferences, family: string, decision: FamilyDecision | undefined): BackgroundPreferences { + if (!FAMILY.test(family)) return preferences; + const families = {...preferences.families}; + if (decision) families[family] = decision; else delete families[family]; + return {...preferences, families}; +} + +export const preferencesPath = (env: NodeJS.ProcessEnv = process.env) => join(nmshConfigDirectory(env), 'background-tasks.json'); + +export function loadPreferences(env: NodeJS.ProcessEnv = process.env): BackgroundPreferences { + try { return normalizePreferences(JSON.parse(readFileSync(preferencesPath(env), 'utf8'))); } catch { return DEFAULT_PREFERENCES(); } +} + +/** Written atomically, private to the user. Command families are stored; commands, arguments and output are not. */ +export function savePreferences(preferences: BackgroundPreferences, env: NodeJS.ProcessEnv = process.env): void { + const path = preferencesPath(env); + mkdirSync(dirname(path), {recursive: true, mode: 0o700}); + const staged = `${path}.${process.pid}.tmp`; + writeFileSync(staged, `${JSON.stringify(normalizePreferences(preferences), null, 2)}\n`, {encoding: 'utf8', mode: 0o600}); + renameSync(staged, path); +} diff --git a/src/backgroundTasks/supervisor.ts b/src/backgroundTasks/supervisor.ts new file mode 100644 index 00000000..6a0b53fe --- /dev/null +++ b/src/backgroundTasks/supervisor.ts @@ -0,0 +1,286 @@ +import {statSync} from 'node:fs'; +import {isAbsolute} from 'node:path'; +import {spawn as spawnPty, type IPty} from 'node-pty'; +import {TaskOutput, type OutputLimits} from './output.js'; + +/** + * The background task supervisor (#382). Each task is an NMSh-owned process + * in its own pseudo-terminal: the PTY makes the child a session and + * process-group leader, so every process the task's shell starts without + * leaving that group (pipelines included) can be signalled together, and + * nothing else. NMSh's interactive shell, other tasks and processes that + * deliberately detach (an adb server started by `adb logcat`) are never + * signalled. + * + * The exact command text is passed as one argument to the session's own + * shell (` -c `): never re-quoted, concatenated or parsed by + * NMSh. It runs with the environment and directory the caller supplies (the + * session's launch environment, the live PATH and the live working + * directory); it does not see the live shell's aliases, functions, + * unexported variables or options. + * + * States are reported only from evidence: `stopping` after a stop request, + * `stopped` / `exited` / `failed` only once the process has exited. A task + * whose leader exited while other processes of its group still run says so. + */ + +export type TaskState = 'running' | 'stopping' | 'exited' | 'failed' | 'stopped'; +export type StopLevel = 'interrupt' | 'terminate' | 'kill'; + +export interface TaskRecord { + /** NMSh's id: short, unique within the supervisor, never reused. */ + id: string; + command: string; + /** The detector's family, when the task came from a recognized command. */ + family?: string; + cwd: string; + shell: string; + startedAt: number; + endedAt?: number; + state: TaskState; + exitCode?: number; + signal?: number; + /** The strongest stop request made so far, and when the latest one was made (a view paces escalation by it). */ + stopRequested?: StopLevel; + stopRequestedAt?: number; + /** Processes of the task's group still running after its leader exited. */ + leftovers?: boolean; +} + +export interface StartRequest { + command: string; + family?: string; + cwd: string; + /** The session's shell executable (absolute path). */ + shell: string; + /** The environment to run with; held for the spawn only, never stored on the record. */ + env: Record; + columns?: number; + rows?: number; +} + +export type PtySpawner = (file: string, args: string[], options: {cwd: string; env: Record; cols: number; rows: number; name: string}) => IPty; + +export interface SupervisorOptions { + output: OutputLimits; + /** Concurrent running tasks allowed. */ + maxRunning?: number; + spawn?: PtySpawner; + now?: () => number; + /** Called (coalesced) when a task changes state or receives output. */ + onChange?: (task: TaskRecord) => void; +} + +interface LiveTask { + record: TaskRecord; + output: TaskOutput; + pty?: IPty; + /** The process group the PTY made: the leader's pid. Forgotten once the group is observed empty. */ + group?: number; + pending?: ReturnType; + /** Watches a finished task's leftover group until it is gone. */ + watch?: ReturnType; +} + +const SIGNALS: Record = {interrupt: 'SIGINT', terminate: 'SIGTERM', kill: 'SIGKILL'}; +const ORDER: readonly StopLevel[] = ['interrupt', 'terminate', 'kill']; +/** How often a finished task's leftover group is probed: also the longest a dead group's id is remembered. */ +const LEFTOVER_PROBE_MS = 250; + +export class BackgroundTaskSupervisor { + private readonly tasks = new Map(); + private counter = 0; + private readonly spawn: PtySpawner; + private readonly now: () => number; + + constructor(private readonly options: SupervisorOptions) { + this.spawn = options.spawn ?? ((file, args, opts) => spawnPty(file, args, opts)); + this.now = options.now ?? Date.now; + } + + list(): TaskRecord[] { return [...this.tasks.values()].map(task => ({...task.record})); } + get(id: string): TaskRecord | undefined { const task = this.tasks.get(id); return task ? {...task.record} : undefined; } + output(id: string): TaskOutput | undefined { return this.tasks.get(id)?.output; } + + start(request: StartRequest): {ok: true; task: TaskRecord} | {ok: false; reason: string} { + if (!request.command.trim() || request.command.length > 64 * 1024) return {ok: false, reason: 'the command is empty or too long'}; + if (request.command.includes('\u0000')) return {ok: false, reason: 'the command contains a NUL character'}; + if (!isAbsolute(request.shell)) return {ok: false, reason: 'the shell must be an absolute path'}; + try { if (!isAbsolute(request.cwd) || !statSync(request.cwd).isDirectory()) return {ok: false, reason: 'the working directory is not a directory'}; } + catch { return {ok: false, reason: 'the working directory does not exist'}; } + const running = [...this.tasks.values()].filter(task => task.record.state === 'running' || task.record.state === 'stopping').length; + if (running >= (this.options.maxRunning ?? 16)) return {ok: false, reason: `${running} background tasks are already running`}; + const id = `t${++this.counter}`; + const record: TaskRecord = {id, command: request.command, ...(request.family ? {family: request.family} : {}), cwd: request.cwd, shell: request.shell, startedAt: this.now(), state: 'running'}; + const task: LiveTask = {record, output: new TaskOutput(this.options.output)}; + let pty: IPty; + try { + // One argv element for the command: the shell parses it exactly as typed. + pty = this.spawn(request.shell, ['-c', request.command], {cwd: request.cwd, env: request.env, cols: Math.max(20, request.columns ?? 120), rows: Math.max(5, request.rows ?? 40), name: request.env.TERM || 'xterm-256color'}); + } catch (error) { + return {ok: false, reason: `could not start: ${error instanceof Error ? error.message : String(error)}`.slice(0, 200)}; + } + task.pty = pty; + task.group = pty.pid; + this.tasks.set(id, task); + pty.onData(data => { task.output.append(data); this.changed(task); }); + pty.onExit(({exitCode, signal}) => this.exited(task, exitCode, signal)); + this.changed(task, true); + return {ok: true, task: {...record}}; + } + + /** + * Ask a running task to stop. `interrupt` is what Ctrl+C would do in a terminal (SIGINT to the task's process group, + * not a byte written to its input, which a program in raw mode would read as data); `terminate` and `kill` are the + * explicit escalations. The task is reported stopped only when it has exited. + */ + stop(id: string, level: StopLevel = 'interrupt'): {ok: true} | {ok: false; reason: string} { + const task = this.tasks.get(id); + if (!task) return {ok: false, reason: 'no such task'}; + if (task.record.state !== 'running' && task.record.state !== 'stopping') { + return task.record.leftovers && level === 'kill' ? this.killLeftovers(task) : {ok: false, reason: 'the task is not running'}; + } + if (!task.group) return {ok: false, reason: 'the task has no process group'}; + try { process.kill(-task.group, SIGNALS[level]); } + catch (error) { return {ok: false, reason: (error as NodeJS.ErrnoException).code === 'ESRCH' ? 'the task already ended' : 'the signal could not be delivered'}; } + const previous = task.record.stopRequested; + task.record.stopRequested = previous && ORDER.indexOf(previous) > ORDER.indexOf(level) ? previous : level; + task.record.stopRequestedAt = this.now(); + task.record.state = 'stopping'; + this.changed(task, true); + return {ok: true}; + } + + /** The next escalation for a task that has not stopped yet, if any. */ + nextEscalation(id: string): StopLevel | undefined { + const requested = this.tasks.get(id)?.record.stopRequested; + if (!requested) return 'interrupt'; + return ORDER[ORDER.indexOf(requested) + 1]; + } + + /** Resize a task's PTY (its programs see the new size as in a terminal). */ + resize(id: string, columns: number, rows: number): void { + const task = this.tasks.get(id); + if (task?.pty && (task.record.state === 'running' || task.record.state === 'stopping')) { + try { task.pty.resize(Math.max(20, columns), Math.max(5, rows)); } catch { /* exited meanwhile */ } + } + } + + /** + * Remove a finished task from the list (its output goes with it). Running tasks are never removed; stop them first. + * Neither is a finished task whose processes are still running: removing it would leave them with nothing that can + * stop them. + */ + remove(id: string): boolean { + const task = this.tasks.get(id); + if (!task || task.record.state === 'running' || task.record.state === 'stopping' || task.record.leftovers) return false; + clearTimeout(task.pending); + clearInterval(task.watch); + this.tasks.delete(id); + return true; + } + + /** Remove the oldest finished tasks beyond `keep`. */ + prune(keep: number): number { + const finished = [...this.tasks.values()].filter(task => task.record.endedAt !== undefined).sort((a, b) => a.record.endedAt! - b.record.endedAt!); + let removed = 0; + for (const task of finished.slice(0, Math.max(0, finished.length - keep))) if (this.remove(task.record.id)) removed += 1; + return removed; + } + + /** + * End every task this supervisor started, for an explicit session exit: interrupt, then terminate and kill what + * remains after the grace period. Resolves when all have exited. + */ + async shutdown(graceMs = 2000): Promise { + const live = () => [...this.tasks.values()].filter(task => task.record.state === 'running' || task.record.state === 'stopping'); + for (const level of ORDER) { + for (const task of live()) this.stop(task.record.id, level); + const deadline = Date.now() + graceMs; + while (live().length && Date.now() < deadline) await new Promise(resolve => setTimeout(resolve, 25)); + if (!live().length) break; + } + for (const task of [...this.tasks.values()].filter(item => item.record.leftovers)) this.killLeftovers(task); + } + + private exited(task: LiveTask, exitCode: number, signal?: number): void { + const record = task.record; + if (signal) record.signal = signal; else record.exitCode = exitCode; + task.pty = undefined; + // After a stop request the task is the whole group, not just its leader: a shell that does not exec its command + // (dash) dies of SIGINT while a program that ignores SIGINT keeps running. Until the group is gone the task is + // still stopping, and escalation still reaches it. + if (record.stopRequested && groupAlive(task.group)) { + record.leftovers = true; + this.changed(task, true); + const poll = setInterval(() => { + if (groupAlive(task.group)) return; + clearInterval(poll); + this.finish(task); + }, 50); + poll.unref?.(); + return; + } + this.finish(task); + } + + private finish(task: LiveTask): void { + task.output.finish(); + const record = task.record; + record.endedAt = this.now(); + // Stopped: it ended after the person asked. Otherwise its own exit status decides. + record.state = record.stopRequested ? 'stopped' : record.signal || record.exitCode !== 0 ? 'failed' : 'exited'; + record.leftovers = groupAlive(task.group); + this.changed(task, true); + if (record.leftovers) this.watchLeftovers(task); else task.group = undefined; + } + + /** + * The kernel hangs up the group when its leader exits; members that obey it are gone a moment later, others may run + * for hours. The group is probed until it is empty, then forgotten: its id is free for reuse once its last member + * exits, so a later kill request must find nothing to signal rather than a recycled group. + */ + private watchLeftovers(task: LiveTask): void { + clearInterval(task.watch); + task.watch = setInterval(() => { + if (groupAlive(task.group)) return; + clearInterval(task.watch); + task.watch = undefined; + task.group = undefined; + task.record.leftovers = false; + this.changed(task, true); + }, LEFTOVER_PROBE_MS); + task.watch.unref?.(); + } + + private killLeftovers(task: LiveTask): {ok: true} | {ok: false; reason: string} { + if (!task.group || !groupAlive(task.group)) { + clearInterval(task.watch); + task.group = undefined; + task.record.leftovers = false; + this.changed(task, true); + return {ok: false, reason: 'nothing is left running'}; + } + try { process.kill(-task.group, 'SIGKILL'); } catch { /* raced with their exit */ } + task.record.stopRequested = 'kill'; + task.record.stopRequestedAt = this.now(); + this.changed(task, true); + return {ok: true}; + } + + /** Output notifications are coalesced (at most every 50 ms per task); state changes are immediate. */ + private changed(task: LiveTask, immediate = false): void { + if (!this.options.onChange) return; + if (immediate) { clearTimeout(task.pending); task.pending = undefined; this.options.onChange({...task.record}); return; } + if (task.pending) return; + task.pending = setTimeout(() => { task.pending = undefined; this.options.onChange?.({...task.record}); }, 50); + task.pending.unref?.(); + } +} + +/** Whether any process of a group still exists (signal 0 checks without signalling). */ +export function groupAlive(group: number | undefined): boolean { + if (!group || group <= 1) return false; + try { process.kill(-group, 0); return true; } + catch (error) { return (error as NodeJS.ErrnoException).code === 'EPERM'; } +} diff --git a/tests/backgroundDetector.test.ts b/tests/backgroundDetector.test.ts new file mode 100644 index 00000000..d9106a02 --- /dev/null +++ b/tests/backgroundDetector.test.ts @@ -0,0 +1,121 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import {mkdtemp, rm, stat} from 'node:fs/promises'; +import {tmpdir} from 'node:os'; +import {join} from 'node:path'; +import {classifyCommand, isClassified, simpleStages} from '../src/backgroundTasks/detector.js'; +import {decide, DEFAULT_PREFERENCES, loadPreferences, normalizePreferences, preferencesPath, rememberFamily, savePreferences} from '../src/backgroundTasks/preferences.js'; +import {clean, TaskOutput} from '../src/backgroundTasks/output.js'; + +const family = (command: string) => { const result = classifyCommand(command); return isClassified(result) ? result.family : undefined; }; + +test('streaming invocations are recognized by what their words say, not by the program name', () => { + const cases: Array<[string, string | undefined]> = [ + ['adb logcat', 'adb logcat'], ['adb -s emulator-5554 logcat -v time', 'adb logcat'], ['adb -d logcat', 'adb logcat'], + ['adb logcat -d', undefined], ['adb logcat -c', undefined], ['adb logcat -t 100', undefined], ['adb devices', undefined], ['adb shell', undefined], + ['tail -f app.log', 'tail -f'], ['tail -fn 100 app.log', 'tail -f'], ['tail -F app.log', 'tail -f'], ['tail --follow=name app.log', 'tail -f'], ['tail -n 100 app.log', undefined], ['tail app.log', undefined], + ['journalctl -f -u nginx', 'journalctl -f'], ['journalctl -u nginx', undefined], + ['docker logs -f web', 'docker logs -f'], ['docker compose logs -f', 'docker compose logs -f'], ['docker logs web', undefined], ['kubectl logs -f pod/x', 'kubectl logs -f'], + ['kubectl get pods -w', 'kubectl get --watch'], ['ping example.com', 'ping'], ['ping -c 3 example.com', undefined], + ['scrcpy', 'scrcpy'], ['scrcpy --max-size 1024', 'scrcpy'], ['scrcpy --version', undefined], + ['npm run dev', 'npm run dev'], ['pnpm dev', 'pnpm run dev'], ['yarn start', 'yarn run start'], ['bun run watch', 'bun run watch'], ['npm test', undefined], ['npm run build', undefined], ['npm install', undefined], + ['vite', 'vite'], ['vite build', undefined], ['tsc --watch', 'tsc --watch'], ['tsc', undefined], ['python3 -m http.server 8000', 'python3 -m http.server'], ['python3 script.py', undefined], + ['/usr/bin/tail -f /var/log/system.log', 'tail -f'], ['ANDROID_SERIAL=abc adb logcat', 'adb logcat'], ['adb logcat | grep ActivityManager', 'adb logcat'], + ]; + for (const [command, expected] of cases) assert.equal(family(command), expected, command); + const logcat = classifyCommand('adb logcat'); + assert.deepEqual(logcat, {family: 'adb logcat', kind: 'logging', likelihood: 'streams', reason: 'adb logcat keeps streaming device logs until it is interrupted.'}); + assert.equal((classifyCommand('npm run dev') as {likelihood: string}).likelihood, 'likely', 'a project script is only "likely"'); +}); + +test('anything that needs a shell parser, live state, the terminal or elevated access is never recognized', () => { + const refused: Array<[string, RegExp]> = [ + ['tail -f $(ls *.log | head -1)', /expansion or substitution/u], ['tail -f "$LOG"', /expansion inside quotes/u], ['tail -f a.log > copy.log', /redirection/u], + ['tail -f a.log; echo done', /command list/u], ['tail -f a.log &', /background job/u], ['tail -f a.log && x', /command list/u], ['tail -f a.log || x', /command list/u], + ['(tail -f a.log)', /subshell/u], ['cat < { + const logcat = classifyCommand('adb logcat'); + const classification = isClassified(logcat) ? logcat : undefined; + let preferences = DEFAULT_PREFERENCES(); + assert.equal(decide(classification, preferences).action, 'suggest'); + assert.equal(decide(undefined, preferences).action, 'foreground'); + preferences = rememberFamily(preferences, 'adb logcat', 'always'); + assert.equal(decide(classification, preferences).action, 'background'); + assert.equal(decide(classification, preferences, 'foreground').action, 'foreground', 'one invocation can still run in the foreground'); + const devices = classifyCommand('adb devices'); + assert.equal(decide(isClassified(devices) ? devices : undefined, preferences).action, 'foreground', 'Always for adb logcat is not Always for adb'); + preferences = rememberFamily(preferences, 'adb logcat', 'never'); + assert.equal(decide(classification, preferences).action, 'foreground'); + preferences = rememberFamily(rememberFamily(preferences, 'adb logcat', undefined), 'adb logcat', undefined); + assert.deepEqual(preferences.families, {}); + assert.equal(decide(classification, {...preferences, suggestions: 'off'}).action, 'foreground', 'suggestions off asks nothing'); + assert.equal(decide(classification, {...rememberFamily(preferences, 'adb logcat', 'always'), suggestions: 'off'}).action, 'background', 'an explicit rule still applies'); +}); + +test('preferences are normalized, bounded and saved privately; commands and output are never stored', async () => { + const normalized = normalizePreferences({suggestions: 'sometimes', families: {'adb logcat': 'always', 'rm -rf': 'maybe', '\u001b[31m': 'never', ...Object.fromEntries(Array.from({length: 400}, (_, i) => [`f${i}`, 'never']))}, + retention: {lines: 10, bytes: 1e12, finishedKept: -5}, notify: 'loud', bar: 'expanded', upArrow: 'off', extra: true}); + assert.equal(normalized.suggestions, 'ask'); + assert.equal(normalized.families['adb logcat'], 'always'); + assert.equal(normalized.families['rm -rf'], undefined); + assert.ok(Object.keys(normalized.families).length <= 256); + assert.deepEqual(normalized.retention, {lines: 100, bytes: 32 * 1024 * 1024, finishedKept: 0}); + assert.equal(normalized.notify, 'failure'); + assert.deepEqual([normalized.bar, normalized.upArrow], ['expanded', 'off']); + assert.ok(!('extra' in normalized)); + const directory = await mkdtemp(join(tmpdir(), 'nmsh-bgprefs-')); + try { + const env = {XDG_CONFIG_HOME: directory}; + assert.deepEqual(loadPreferences(env), DEFAULT_PREFERENCES(), 'missing file: defaults'); + savePreferences(rememberFamily(DEFAULT_PREFERENCES(), 'tail -f', 'always'), env); + assert.equal(loadPreferences(env).families['tail -f'], 'always'); + assert.equal((await stat(preferencesPath(env))).mode & 0o777, 0o600); + } finally { await rm(directory, {recursive: true, force: true}); } +}); + +test('task output: line endings, progress lines, splitting and bounds', () => { + const output = new TaskOutput({lines: 3, bytes: 1024, lineLength: 8}); + output.append('one\r\r\ntwo\r\npar'); + assert.deepEqual(output.lines().map(line => line.text), ['one', 'two']); + assert.equal(output.partial(), 'par'); + output.append('tial\r10%\r50%'); + assert.equal(output.partial(), '50%', 'an unfinished progress line shows its latest state'); + output.append('\n'); + assert.deepEqual(output.lines().map(line => [line.seq, line.text]), [[1, 'one'], [2, 'two'], [3, '50%']]); + output.append('0123456789ABCDEFG\n'); + assert.deepEqual(output.lines().map(line => line.text), ['01234567', '89ABCDEF', 'G'], 'long lines split at the line length; the oldest go'); + assert.equal(output.dropped, 3); + assert.deepEqual(output.since(5).map(line => line.text), ['G']); + output.append('no newline at all'); + output.finish(); + assert.deepEqual(output.lines().map(line => line.text), ['no newli', 'ne at al', 'l'], 'finish flushes the unfinished line'); + assert.equal(clean('\u001b]0;title\u0007\u001b[1mbold\u001b[0m\t‮end', 100), 'bold end'); +}); + +test('task output: a hostile run of carriage returns is handled in linear time', () => { + // `/\r+$/` retried from every \r of the run: 64 Ki of them took seconds and froze NMSh while a task printed them. + const output = new TaskOutput({lines: 100, bytes: 1024 * 1024}); + const hostile = `${'\r'.repeat(64 * 1024)}x\n`; + const started = performance.now(); + output.append(hostile); + output.append(hostile); + const elapsed = performance.now() - started; + assert.ok(elapsed < 250, `took ${elapsed.toFixed(0)} ms`); + assert.deepEqual(output.lines().map(line => line.text), ['x', 'x'], 'the progress overwrite still keeps the last state'); + assert.equal(clean(`done${'\r'.repeat(5000)}`, 100), 'done', 'trailing carriage returns end the line'); +}); diff --git a/tests/backgroundSupervisor.test.ts b/tests/backgroundSupervisor.test.ts new file mode 100644 index 00000000..faab2706 --- /dev/null +++ b/tests/backgroundSupervisor.test.ts @@ -0,0 +1,198 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import {mkdtemp, readFile, realpath, rm} from 'node:fs/promises'; +import {tmpdir} from 'node:os'; +import {join} from 'node:path'; +import {BackgroundTaskSupervisor, groupAlive, type TaskRecord} from '../src/backgroundTasks/supervisor.js'; + +const STREAM = join(import.meta.dirname, 'fixtures', 'background', 'stream.mjs'); +const BURST = join(import.meta.dirname, 'fixtures', 'background', 'burst.mjs'); +const SLEEPER = join(import.meta.dirname, 'fixtures', 'background', 'sleeper.mjs'); +const SHELL = '/bin/sh'; +const node = (script: string, ...args: string[]) => [process.execPath, script, ...args].map(part => `'${part}'`).join(' '); +const env = (): Record => ({PATH: process.env.PATH ?? '/usr/bin:/bin', HOME: process.env.HOME ?? '/', TERM: 'xterm-256color'}); +const alive = (pid: number) => { try { process.kill(pid, 0); return true; } catch { return false; } }; + +async function waitFor(check: () => boolean, what: string, timeout = 10_000): Promise { + const deadline = Date.now() + timeout; + while (!check()) { + if (Date.now() > deadline) throw new Error(`timed out waiting for ${what}`); + await new Promise(resolve => setTimeout(resolve, 20)); + } +} + +async function withSupervisor(run: (supervisor: BackgroundTaskSupervisor, cwd: string, changes: TaskRecord[]) => Promise, limits = {lines: 2000, bytes: 1024 * 1024}) { + const cwd = await realpath(await mkdtemp(join(tmpdir(), 'nmsh-bg-'))); + const changes: TaskRecord[] = []; + const supervisor = new BackgroundTaskSupervisor({output: limits, onChange: task => changes.push(task)}); + try { await run(supervisor, cwd, changes); } + finally { await supervisor.shutdown(500); await rm(cwd, {recursive: true, force: true}); } +} + +const text = (supervisor: BackgroundTaskSupervisor, id: string) => supervisor.output(id)!.lines().map(line => line.text).join('\n'); + +test('a logcat-like stream: output arrives live; stop is SIGINT to its own group and is reported only after exit', async () => { + await withSupervisor(async (supervisor, cwd) => { + const started = supervisor.start({command: node(STREAM), family: 'adb logcat', cwd, shell: SHELL, env: env()}); + assert.ok(started.ok); + const id = started.task.id; + await waitFor(() => supervisor.output(id)!.lines().length >= 5, 'streamed lines'); + assert.match(text(supervisor, id), /I\/ActivityManager\(\s*\d+\): line 1/u); + assert.deepEqual(supervisor.stop(id), {ok: true}); + assert.equal(typeof supervisor.get(id)!.stopRequestedAt, 'number', 'the request time paces escalation in views'); + assert.equal(supervisor.get(id)!.state === 'stopping' || supervisor.get(id)!.state === 'stopped', true, 'never "stopped" before the exit is observed'); + await waitFor(() => supervisor.get(id)!.state === 'stopped', 'stopped'); + const task = supervisor.get(id)!; + // The leader is the program itself when the shell execs it (bash, zsh) and the shell when it does not (dash). + assert.ok(task.exitCode === 130 || task.signal === 2, JSON.stringify(task)); + assert.equal(task.stopRequested, 'interrupt'); + assert.match(text(supervisor, id), /interrupted \(SIGINT\)/u, 'the program saw SIGINT, as with Ctrl+C'); + assert.equal(task.leftovers, false); + assert.deepEqual(supervisor.stop(id), {ok: false, reason: 'the task is not running'}); + }); +}); + +test('a task that ignores SIGINT stays "stopping"; escalation is explicit, one level at a time', async () => { + await withSupervisor(async (supervisor, cwd) => { + const {task} = supervisor.start({command: node(STREAM, '--ignore-int'), cwd, shell: SHELL, env: env()}) as {ok: true; task: TaskRecord}; + await waitFor(() => supervisor.output(task.id)!.lines().length >= 2, 'output'); + supervisor.stop(task.id); + await waitFor(() => /ignoring SIGINT/u.test(text(supervisor, task.id)), 'the program refusing'); + await new Promise(resolve => setTimeout(resolve, 300)); + assert.equal(supervisor.get(task.id)!.state, 'stopping', 'still running: not reported as stopped'); + assert.equal(supervisor.nextEscalation(task.id), 'terminate'); + supervisor.stop(task.id, 'terminate'); + await waitFor(() => supervisor.get(task.id)!.state === 'stopped', 'stopped after SIGTERM'); + assert.equal(supervisor.get(task.id)!.stopRequested, 'terminate'); + // It also survives the hangup: where /bin/sh does not exec the program (dash), SIGTERM ends the shell, the kernel + // hangs up the terminal (SIGHUP, then EIO on writes), and a program that died of that would be stopped by it rather + // than by SIGKILL. That stop is real and reported as such; this test is about the explicit escalation. + const stubborn = supervisor.start({command: node(STREAM, '--ignore-int', '--ignore-term', '--ignore-hup'), cwd, shell: SHELL, env: env()}) as {ok: true; task: TaskRecord}; + await waitFor(() => supervisor.output(stubborn.task.id)!.lines().length >= 1, 'output'); + supervisor.stop(stubborn.task.id); supervisor.stop(stubborn.task.id, 'terminate'); + await new Promise(resolve => setTimeout(resolve, 300)); + assert.equal(supervisor.get(stubborn.task.id)!.state, 'stopping', JSON.stringify({task: supervisor.get(stubborn.task.id), output: text(supervisor, stubborn.task.id).split('\n').slice(-4)})); + assert.equal(supervisor.nextEscalation(stubborn.task.id), 'kill'); + supervisor.stop(stubborn.task.id, 'kill'); + await waitFor(() => supervisor.get(stubborn.task.id)!.state === 'stopped', 'killed'); + assert.equal(supervisor.get(stubborn.task.id)!.stopRequested, 'kill'); + assert.equal(supervisor.get(stubborn.task.id)!.leftovers, false, 'the whole group is gone'); + assert.equal(supervisor.nextEscalation(stubborn.task.id), undefined, 'nothing beyond kill'); + }); +}); + +test('a pipeline stops as a whole; a detached helper it started (like the adb server) and other tasks are untouched', async () => { + await withSupervisor(async (supervisor, cwd) => { + const pidfile = join(cwd, 'daemon.pid'); + const piped = supervisor.start({command: `${node(STREAM, `--daemon=${pidfile}`)} | cat`, cwd, shell: SHELL, env: env()}) as {ok: true; task: TaskRecord}; + const other = supervisor.start({command: node(STREAM), cwd, shell: SHELL, env: env()}) as {ok: true; task: TaskRecord}; + await waitFor(() => supervisor.output(piped.task.id)!.lines().length >= 3 && supervisor.output(other.task.id)!.lines().length >= 3, 'both streaming'); + const daemon = Number(await readFile(pidfile, 'utf8')); + try { + supervisor.stop(piped.task.id); + await waitFor(() => ['stopped', 'exited', 'failed'].includes(supervisor.get(piped.task.id)!.state), 'pipeline ended'); + await waitFor(() => supervisor.get(piped.task.id)!.leftovers === false, 'no leftovers in the group', 3000); + assert.equal(alive(daemon), true, 'a process that left the group (setsid) is not NMSh\'s to stop'); + assert.equal(supervisor.get(other.task.id)!.state, 'running', 'stopping one task leaves the others running'); + const before = supervisor.output(other.task.id)!.lastSeq; + await waitFor(() => supervisor.output(other.task.id)!.lastSeq > before + 3, 'the other task still streams'); + } finally { try { process.kill(daemon, 'SIGKILL'); } catch { /* gone */ } } + }); +}); + +test('processes left in the group after the leader exits are reported, and killed only on request', async () => { + await withSupervisor(async (supervisor, cwd) => { + const {task} = supervisor.start({command: `${node(SLEEPER)} & sleep 1; exit 0`, cwd, shell: SHELL, env: env()}) as {ok: true; task: TaskRecord}; + await waitFor(() => supervisor.get(task.id)!.state !== 'running', 'leader exited'); + await new Promise(resolve => setTimeout(resolve, 300)); + const record = supervisor.get(task.id)!; + assert.equal(record.state, 'exited'); + assert.equal(record.leftovers, true, 'a member that ignored the hangup is still running, and the task says so'); + assert.deepEqual(supervisor.stop(task.id), {ok: false, reason: 'the task is not running'}, 'a plain stop does not reach leftovers'); + assert.deepEqual(supervisor.stop(task.id, 'kill'), {ok: true}); + await waitFor(() => supervisor.get(task.id)!.leftovers === false, 'leftovers gone'); + }); +}); + +test('a leftover group that ends later is noticed and forgotten: no stale "processes left", no kill aimed at a reused group id', async () => { + await withSupervisor(async (supervisor, cwd) => { + // The member outlives the leader by more than a second, past any one-off check after the exit. + const {task} = supervisor.start({command: `${node(SLEEPER, '1600')} & sleep 0.2; exit 0`, cwd, shell: SHELL, env: env()}) as {ok: true; task: TaskRecord}; + await waitFor(() => supervisor.get(task.id)!.state === 'exited', 'leader exited'); + await waitFor(() => supervisor.get(task.id)!.leftovers === true, 'leftovers reported'); + assert.equal(supervisor.remove(task.id), false, 'a task whose processes still run is not removed: nothing could stop them afterwards'); + await waitFor(() => supervisor.get(task.id)!.leftovers === false, 'the late exit noticed', 5000); + assert.deepEqual(supervisor.stop(task.id, 'kill'), {ok: false, reason: 'the task is not running'}, 'nothing is signalled once the group is gone'); + assert.equal(supervisor.remove(task.id), true); + }); +}); + +test('short commands: exit status decides exited/failed; the exact command text reaches the shell unchanged', async () => { + await withSupervisor(async (supervisor, cwd) => { + const quoting = supervisor.start({command: `printf '%s|\\n' "a b" 'c$HOME' "d\\"e" && pwd`, cwd, shell: SHELL, env: env()}) as {ok: true; task: TaskRecord}; + const failing = supervisor.start({command: 'echo going; exit 3', cwd, shell: SHELL, env: env()}) as {ok: true; task: TaskRecord}; + await waitFor(() => supervisor.get(quoting.task.id)!.state === 'exited' && supervisor.get(failing.task.id)!.state === 'failed', 'both ended'); + assert.deepEqual(supervisor.output(quoting.task.id)!.lines().map(line => line.text), ['a b|', 'c$HOME|', 'd"e|', cwd]); + assert.equal(supervisor.get(failing.task.id)!.exitCode, 3); + assert.equal(supervisor.get(quoting.task.id)!.endedAt! >= supervisor.get(quoting.task.id)!.startedAt, true); + }); +}); + +test('environment is used for the spawn and never kept on the task record', async () => { + await withSupervisor(async (supervisor, cwd, changes) => { + const {task} = supervisor.start({command: 'echo "$NMSH_TEST_TOKEN"', cwd, shell: SHELL, env: {...env(), NMSH_TEST_TOKEN: 'secret-value-123'}}) as {ok: true; task: TaskRecord}; + await waitFor(() => supervisor.get(task.id)!.state === 'exited', 'exit'); + assert.equal(text(supervisor, task.id), 'secret-value-123', 'the program saw its environment'); + assert.ok(!JSON.stringify([supervisor.list(), changes]).includes('secret-value-123'), 'records carry no environment'); + }); +}); + +test('refusals: bad directory, relative shell, NUL bytes, the running-task limit', async () => { + const supervisor = new BackgroundTaskSupervisor({output: {lines: 10, bytes: 1024}, maxRunning: 1}); + assert.deepEqual(supervisor.start({command: 'true', cwd: '/definitely/not/here', shell: SHELL, env: env()}), {ok: false, reason: 'the working directory does not exist'}); + assert.deepEqual(supervisor.start({command: 'true', cwd: '/', shell: 'sh', env: env()}), {ok: false, reason: 'the shell must be an absolute path'}); + assert.deepEqual(supervisor.start({command: 'true\u0000rm', cwd: '/', shell: SHELL, env: env()}), {ok: false, reason: 'the command contains a NUL character'}); + const first = supervisor.start({command: node(STREAM), cwd: '/', shell: SHELL, env: env()}); + assert.ok(first.ok); + assert.deepEqual(supervisor.start({command: 'true', cwd: '/', shell: SHELL, env: env()}), {ok: false, reason: '1 background tasks are already running'}); + await supervisor.shutdown(500); + assert.ok(supervisor.list().every(task => task.state === 'stopped')); + assert.equal(supervisor.remove(supervisor.list()[0]!.id), true, 'a finished task can be removed'); +}); + +test('high-volume hostile output is bounded, inert and does not stall', async () => { + await withSupervisor(async (supervisor, cwd) => { + const started = Date.now(); + const {task} = supervisor.start({command: node(BURST, '8'), cwd, shell: SHELL, env: env()}) as {ok: true; task: TaskRecord}; + await waitFor(() => supervisor.get(task.id)!.state === 'exited', 'burst finished', 60_000); + const output = supervisor.output(task.id)!; + assert.ok(output.lines().length <= 500 && output.bytes <= 64 * 1024, `bounded: ${output.lines().length} lines, ${output.bytes} bytes`); + assert.ok(output.dropped > 1000, 'old lines were dropped, and counted'); + assert.ok(output.received > 8 * 1024 * 1024); + for (const line of output.lines()) assert.doesNotMatch(line.text, /[\u0000-\u0008\u000b-\u001f\u007f-\u009f‮]/u); + assert.deepEqual(output.lines().slice(-2).map(line => line.text), ['progress 100%', 'done'], 'a progress line keeps its last state'); + assert.ok(output.lines().some(line => line.text.includes('�')), 'invalid UTF-8 shows as replacement characters'); + assert.ok(Date.now() - started < 60_000); + }, {lines: 500, bytes: 64 * 1024}); +}); + +test('groupAlive never claims a group that is gone, and never checks group 0 or 1', () => { + assert.equal(groupAlive(undefined), false); + assert.equal(groupAlive(1), false); + assert.equal(groupAlive(2 ** 22 + 12345), false); +}); + +test('a wrapper shell that dies of SIGINT does not end the task while its program still runs', async () => { + // dash-style: the shell does not exec the program, so SIGINT ends the shell first; the program ignores it. + await withSupervisor(async (supervisor, cwd) => { + const {task} = supervisor.start({command: `${node(STREAM, '--ignore-int')}; true`, cwd, shell: SHELL, env: env()}) as {ok: true; task: TaskRecord}; + await waitFor(() => supervisor.output(task.id)!.lines().length >= 2, 'output'); + supervisor.stop(task.id); + await new Promise(resolve => setTimeout(resolve, 400)); + assert.equal(supervisor.get(task.id)!.state, 'stopping', 'the program still runs, so the task is not stopped'); + assert.equal(supervisor.nextEscalation(task.id), 'terminate'); + assert.deepEqual(supervisor.stop(task.id, 'terminate'), {ok: true}); + await waitFor(() => supervisor.get(task.id)!.state === 'stopped', 'stopped once the group is gone'); + assert.equal(supervisor.get(task.id)!.leftovers, false); + }); +}); diff --git a/tests/fixtures/background/burst.mjs b/tests/fixtures/background/burst.mjs new file mode 100644 index 00000000..af76d42a --- /dev/null +++ b/tests/fixtures/background/burst.mjs @@ -0,0 +1,15 @@ +// High-volume hostile output: escapes, OSC titles/links, bidi controls, invalid UTF-8 and a progress line, then exit 0. +const total = Number(process.argv[2] ?? 20) * 1024 * 1024; +let written = 0, n = 0; +const chunk = () => Buffer.concat([ + Buffer.from(`\u001b[31mred ${n}\u001b[0m \u001b]0;PWNED\u0007\u001b]8;;https://evil.example\u0007link\u001b]8;;\u0007 ‮evil\t`), + Buffer.from([0xff, 0xfe, 0xc3]), Buffer.from(`\r\n`), +]); +function write() { + while (written < total) { + const data = chunk(); n += 1; written += data.length; + if (!process.stdout.write(data)) return process.stdout.once('drain', write); + } + process.stdout.write('progress 10%\rprogress 50%\rprogress 100%\r\ndone\r\n', () => process.exit(0)); +} +write(); diff --git a/tests/fixtures/background/sleeper.mjs b/tests/fixtures/background/sleeper.mjs new file mode 100644 index 00000000..21030555 --- /dev/null +++ b/tests/fixtures/background/sleeper.mjs @@ -0,0 +1,4 @@ +// A quiet group member that ignores the hangup its group gets when the task's leader exits. +// An optional argument is how long it stays, in milliseconds (default a minute). +process.on('SIGHUP', () => {}); +setTimeout(() => {}, Number(process.argv[2] ?? 60_000)); diff --git a/tests/fixtures/background/stream.mjs b/tests/fixtures/background/stream.mjs new file mode 100644 index 00000000..9474777b --- /dev/null +++ b/tests/fixtures/background/stream.mjs @@ -0,0 +1,21 @@ +// A stand-in for `adb logcat` / `tail -f` / `scrcpy`: streams lines until stopped. +// Flags: --ignore-int, --ignore-term, --ignore-hup, --gui (announces a window and closes it on stop), --daemon= (starts a detached helper first, like adb's server). +import {spawn} from 'node:child_process'; +import {writeFileSync} from 'node:fs'; +const flags = new Set(process.argv.slice(2)); +const daemon = process.argv.slice(2).find(arg => arg.startsWith('--daemon='))?.slice(9); +if (daemon) { + const child = spawn(process.execPath, ['-e', 'process.on("SIGHUP",()=>{});setTimeout(()=>{},60000)'], {detached: true, stdio: 'ignore'}); + writeFileSync(daemon, String(child.pid)); + child.unref(); +} +if (flags.has('--gui')) console.log('window opened'); +const stop = name => () => { console.log(flags.has('--gui') ? `closing window (${name})` : `interrupted (${name})`); process.exit(name === 'SIGINT' ? 130 : 143); }; +process.on('SIGINT', flags.has('--ignore-int') ? () => console.log('ignoring SIGINT') : stop('SIGINT')); +process.on('SIGTERM', flags.has('--ignore-term') ? () => console.log('ignoring SIGTERM') : stop('SIGTERM')); +// Ignoring the hangup also means surviving the terminal going away: on Linux, once the controlling process exits the +// terminal is hung up and writes fail with EIO, which would otherwise end the program through an unhandled error. +if (flags.has('--ignore-hup')) { process.on('SIGHUP', () => {}); process.stdout.on('error', () => {}); } +const print = line => { try { console.log(line); } catch { /* the terminal is gone */ } }; +let n = 0; +setInterval(() => print(`I/ActivityManager( ${process.pid}): line ${++n}`), 20);