Skip to content

feat(process): retain native raw spills with owned cleanup - #917

Draft
Haoming9527 wants to merge 3 commits into
work/872-resident-task-hostfrom
work/872-native-raw-spill
Draft

Haoming9527 wants to merge 3 commits into
work/872-resident-task-hostfrom
work/872-native-raw-spill

Conversation

@Haoming9527

@Haoming9527 Haoming9527 commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Refs #872. Stacked on #916; this is the shared raw-output storage foundation, not public task/session activation or feature closure.

What was wrong

Resident process ranges retained bounded memory only. Fresh tasks and persistent sessions need the existing complete raw-output recovery and retention contract, but filesystem writes must not hold capture/control locks or falsely claim cleanup after an unacknowledged cancellation. A late writer completion must also never restore a previously poisoned stream's path, EOF or clean status.

The change

Add one lazily initialized, cloneable native spill store, used by the existing output-reader threads. Startup allocation performs no filesystem I/O. Streams retain their bounded tail; after memory overflow, an exclusive private file records every observed byte from zero up to 64 MiB. Live metadata describes a complete observed prefix. Cap/write/seal failure withdraws the full path while tail draining continues. Unsealed files use delete-on-close; joined EOF permits checked sealing.

Bounded Windows writes retain their operation buffers until completion is observed. Cancellation addresses the owned writer/file. Expired or failed reader joins retain the poisoned owner and require independent control shutdown followed by public-host kernel teardown. Poison is irreversible across late appends, EOF and repeated joins. A seal already in flight can finish after poisoning; metadata stays withdrawn and cleanup stays explicitly unconfirmed, rather than claiming every file was removed.

Extend the existing spill ledger and bounded retention pass with ops-native-v1 origin and the actual kernel PID. Preserve legacy entries, exact file/directory shape, no-follow and ownership checks, byte budgets and individual-file deletion. No second store service or retention scheduler is added. Bilingual docs, changelog and real Windows CI probes cover the boundary. Persistent private-channel reuse and production task/session wire integration remain subsequent slices; this commit does not advertise task-spill-v1.

Validation

The integrated stack before the final poison-latching regression passes 723 native tests, zero failures, six ignored cases. Five are explicitly invoked child fixtures (including the fatal-I/O public-host probe); the existing UNC fixture is separately invoked by Windows CI. The final process crate passes 34 tests, zero failures, four explicitly invoked child fixtures. Final workspace/all-target Clippy with -D warnings, TypeScript build, all 117 workflow checks, formatting/diff checks and all 911 tracked code files pass.

The shared-ledger/retention suites pass 23 checks across three suites; two pre-existing POSIX-specific fixtures explicitly report unavailable on Windows. Real public-host spill checks pass 6/6: exact 64,000-byte memory boundary; live/sealed full recovery through the existing file bridge; refusal to modify the spill under workspace-write; exact 64 MiB cap plus one-byte withdrawal; cancellation sealing; existing retention cleanup; and hard-kernel-death cleanup. The fatal-I/O probe passes 3/3, proving retained uncertain ownership, public-host termination with no observed survivors/raw file, and a usable independent host-owned process afterward.

Compatibility checks pass 10/10 task-capture cases and 18/18 candidate direct-driver cases. The candidate runtime remains unpublished and stock runtime remains ineligible for complete persistent sessions. The poison regression first fails because late completion restores EOF; the guarded implementation passes, with the red and green logs retained. Final head 691ec977dc832df9d182524ebbd0109866f94734 passed its own pull-request CI required: https://github.com/T-Auto/dsh-ops/actions/runs/43488785466 .

Environment

Windows x64, Node 26.12.1, Rust 1.99.0 locally; CI uses the pinned toolchain. Base: 8c0f06c19953d56dbad43ddb2cce2d7796ecfe74 (includes the deterministic pre-spawn deadline fixture correction). No dependency or runtime-pin change, PR merge, release or deployment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant