Repository navigation
feat(process): retain native raw spills with owned cleanup - #917
Draft
Haoming9527 wants to merge 3 commits into
Draft
Haoming9527 wants to merge 3 commits into
Haoming9527 wants to merge 3 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #872. Stacked on #916; this is the shared raw-output storage foundation, not public task/session activation or feature closure.
What was wrong
Resident process ranges retained bounded memory only. Fresh tasks and persistent sessions need the existing complete raw-output recovery and retention contract, but filesystem writes must not hold capture/control locks or falsely claim cleanup after an unacknowledged cancellation. A late writer completion must also never restore a previously poisoned stream's path, EOF or clean status.
The change
Add one lazily initialized, cloneable native spill store, used by the existing output-reader threads. Startup allocation performs no filesystem I/O. Streams retain their bounded tail; after memory overflow, an exclusive private file records every observed byte from zero up to 64 MiB. Live metadata describes a complete observed prefix. Cap/write/seal failure withdraws the full path while tail draining continues. Unsealed files use delete-on-close; joined EOF permits checked sealing.
Bounded Windows writes retain their operation buffers until completion is observed. Cancellation addresses the owned writer/file. Expired or failed reader joins retain the poisoned owner and require independent control shutdown followed by public-host kernel teardown. Poison is irreversible across late appends, EOF and repeated joins. A seal already in flight can finish after poisoning; metadata stays withdrawn and cleanup stays explicitly unconfirmed, rather than claiming every file was removed.
Extend the existing spill ledger and bounded retention pass with
ops-native-v1origin and the actual kernel PID. Preserve legacy entries, exact file/directory shape, no-follow and ownership checks, byte budgets and individual-file deletion. No second store service or retention scheduler is added. Bilingual docs, changelog and real Windows CI probes cover the boundary. Persistent private-channel reuse and production task/session wire integration remain subsequent slices; this commit does not advertisetask-spill-v1.Validation
The integrated stack before the final poison-latching regression passes 723 native tests, zero failures, six ignored cases. Five are explicitly invoked child fixtures (including the fatal-I/O public-host probe); the existing UNC fixture is separately invoked by Windows CI. The final process crate passes 34 tests, zero failures, four explicitly invoked child fixtures. Final workspace/all-target Clippy with
-D warnings, TypeScript build, all 117 workflow checks, formatting/diff checks and all 911 tracked code files pass.The shared-ledger/retention suites pass 23 checks across three suites; two pre-existing POSIX-specific fixtures explicitly report unavailable on Windows. Real public-host spill checks pass 6/6: exact 64,000-byte memory boundary; live/sealed full recovery through the existing file bridge; refusal to modify the spill under workspace-write; exact 64 MiB cap plus one-byte withdrawal; cancellation sealing; existing retention cleanup; and hard-kernel-death cleanup. The fatal-I/O probe passes 3/3, proving retained uncertain ownership, public-host termination with no observed survivors/raw file, and a usable independent host-owned process afterward.
Compatibility checks pass 10/10 task-capture cases and 18/18 candidate direct-driver cases. The candidate runtime remains unpublished and stock runtime remains ineligible for complete persistent sessions. The poison regression first fails because late completion restores EOF; the guarded implementation passes, with the red and green logs retained. Final head
691ec977dc832df9d182524ebbd0109866f94734passed its own pull-requestCI required: https://github.com/T-Auto/dsh-ops/actions/runs/43488785466 .Environment
Windows x64, Node 26.12.1, Rust 1.99.0 locally; CI uses the pinned toolchain. Base:
8c0f06c19953d56dbad43ddb2cce2d7796ecfe74(includes the deterministic pre-spawn deadline fixture correction). No dependency or runtime-pin change, PR merge, release or deployment.