Skip to content

feat(runtime): configurable garbage collector, memory leak fix, regression test#1248

Open
zaccharyfavere wants to merge 3 commits into
SkipLabs:mainfrom
zaccharyfavere:gc-config-and-leak-fix
Open

feat(runtime): configurable garbage collector, memory leak fix, regression test#1248
zaccharyfavere wants to merge 3 commits into
SkipLabs:mainfrom
zaccharyfavere:gc-config-and-leak-fix

Conversation

@zaccharyfavere
Copy link
Copy Markdown

@zaccharyfavere zaccharyfavere commented May 27, 2026

Adds a configurable GC sweep mechanism for orphan reactive resources, exposes Skip persistent memory (from palloc.c) usage as a metric, fixes a separate memory leak in the reactive read tracking, and ships a regression test that detects future memory leaks.

GC configuration:

  • New GCConfig type and SkipService.gcConfig?: () => GCConfig in the public API
  • The config lives in user TypeScript as a lambda; the runtime invokes it via FFI each time it is needed, with no caching
  • New SkipRuntime_callGCConfigProvider FFI bridge (Skiplang -> JS via C++) following the same pattern as Mapper__mapEntry
  • checkGarbage() sweeps expired resources from the garbage queue and evicts the oldest entries when the queue exceeds maxGarbageSize
  • getGarbageQueueSize() returns the current queue depth

Memory metric:

  • New SKIP_get_persistent_size() in palloc.c (64-bit) aligned with the existing 32-bit version in runtime32_specific.c
  • Exposed through the full FFI chain (Skiplang prelude binding, Runtime.sk wrapper, C++/WASM bindings, public TS method getSkipPersistentSize())

Reads leak fix:

  • Context.reads (SortedSet) accumulated read paths from root-level operations (createReactiveResource, closeReactiveResource) that were never released, unlike the per-mapper save/restore mechanism
  • The set is now cleared at the end of each successful sweep in checkGarbage()

Regression test (skipruntime-ts/examples/stress-tests/gc-stress.ts):

  • Default mode (~30s) runs two tests on the two leak scenarios
  • Full mode (-f) runs longer, exhaustive versions
  • It can display its trace and what it tries to do with the --trace option
  • Computes a linear regression slope on memory measurements taken during a stable phase (after skipping the warmup window)
  • Fails if the slope exceeds 100 bytes/cycle threshold
  • Uses getSkipPersistentSize() for byte-precise measurement
  • Wrapped by a Makefile target: make test-mem-stress, with args pass-through via ARGS="--trace --full"

Notes:

  • The DeletedDir leak fix is in a separate PR
  • The "instantiate-close-with-map" sub-test currently FAILs on this branch alone; it passes once the DeletedDir PR is also merged

@zaccharyfavere zaccharyfavere force-pushed the gc-config-and-leak-fix branch 2 times, most recently from 4d54c4d to 87ae2cf Compare May 28, 2026 14:34
@mbouaziz mbouaziz requested a review from skiplabsdaniel May 28, 2026 14:36
@zaccharyfavere zaccharyfavere force-pushed the gc-config-and-leak-fix branch from 87ae2cf to 0e64d43 Compare May 28, 2026 14:45
…, regression test

Adds a GC configuration mechanism for orphan reactive resources,
exposes Skip persistent memory usage as a metric, fixes a separate
memory leak in the reactive read tracking, and ships a regression
test that detects future memory leaks.

GC configuration:
- New GCConfig type and SkipService.gcConfig?: () => GCConfig in
  the public API. The config lives in user TypeScript and is
  invoked by the runtime each time it is needed, with no caching.
- SkipRuntime_callGCConfigProvider FFI bridge (Skiplang -> JS via
  C++) following the same pattern as Mapper__mapEntry.
- getGCConfig in Runtime.sk calls the lambda and parses the
  returned CJObject into a typed GCConfig, with fallback to
  defaults on error.
- checkGarbage() sweeps expired resources from the garbage queue
  and evicts the oldest entries when the queue exceeds
  maxGarbageSize.
- getGarbageQueueSize() returns the current queue depth.

Memory metric:
- New SKIP_get_persistent_size() in palloc.c (64-bit) aligned with
  the existing 32-bit version in runtime32_specific.c.
- Exposed through the full FFI chain (Skiplang prelude binding,
  Runtime.sk wrapper, C++/WASM bindings, public TS method
  getSkipPersistentSize()).

Reads leak fix:
- Context.reads (SortedSet<Path>) accumulated read paths from root-
  level operations (createReactiveResource, closeReactiveResource)
  that were never released, unlike the per-mapper save/restore
  mechanism.
- The set is now cleared at the end of each successful sweep in
  checkGarbage().

Regression test (skipruntime-ts/examples/stress-tests/gc-stress.ts):
- Default mode (~30s) runs two tests on the two leak scenarios.
- Full mode (-f) runs longer, exhaustive versions.
- Computes a linear regression slope on memory measurements taken
  during a stable phase (after skipping the warmup window).
- Fails if the slope exceeds 100 bytes/cycle threshold.
- Uses getSkipPersistentSize() for byte-precise measurement.
- Wrapped by a `test-mem-stress` Makefile target that handles
  LD_LIBRARY_PATH and SKIP_PLATFORM exports. Args pass-through:
  make test-mem-stress ARGS="--trace --full".

Notes:
- The "instantiate-close-with-map" sub-test currently FAILs on this
  branch alone because it exercises DeletedDir cleanup, which is in
  the separate fix-deleteddir-leak PR. The test passes once both PRs
  are merged together.
@zaccharyfavere zaccharyfavere force-pushed the gc-config-and-leak-fix branch from 0e64d43 to 551acfc Compare June 3, 2026 18:24
zaccharyfavere and others added 2 commits June 3, 2026 20:27
…, regression test

Adds a GC configuration mechanism for orphan reactive resources,
exposes Skip persistent memory usage as a metric, fixes a separate
memory leak in the reactive read tracking, and ships a regression
test that detects future memory leaks.

GC configuration:
- New GCConfig type and SkipService.gcConfig?: () => GCConfig in
  the public API. The config lives in user TypeScript and is
  invoked by the runtime each time it is needed, with no caching.
- SkipRuntime_callGCConfigProvider FFI bridge (Skiplang -> JS via
  C++) following the same pattern as Mapper__mapEntry.
- getGCConfig in Runtime.sk calls the lambda and parses the
  returned CJObject into a typed GCConfig, with fallback to
  defaults on error.
- checkGarbage() sweeps expired resources from the garbage queue
  and evicts the oldest entries when the queue exceeds
  maxGarbageSize.
- getGarbageQueueSize() returns the current queue depth.

Memory metric:
- New SKIP_get_persistent_size() in palloc.c (64-bit) aligned with
  the existing 32-bit version in runtime32_specific.c.
- Exposed through the full FFI chain (Skiplang prelude binding,
  Runtime.sk wrapper, C++/WASM bindings, public TS method
  getSkipPersistentSize()).

Reads leak fix:
- Context.reads (SortedSet<Path>) accumulated read paths from root-
  level operations (createReactiveResource, closeReactiveResource)
  that were never released, unlike the per-mapper save/restore
  mechanism.
- The set is now cleared at the end of each successful sweep in
  checkGarbage().

Regression test (skipruntime-ts/examples/stress-tests/gc-stress.ts):
- Default mode (~30s) runs two tests on the two leak scenarios.
- Full mode (-f) runs longer, exhaustive versions.
- Computes a linear regression slope on memory measurements taken
  during a stable phase (after skipping the warmup window).
- Fails if the slope exceeds 100 bytes/cycle threshold.
- Uses getSkipPersistentSize() for byte-precise measurement.
- Wrapped by a `test-mem-stress` Makefile target that handles
  LD_LIBRARY_PATH and SKIP_PLATFORM exports. Args pass-through:
  make test-mem-stress ARGS="--trace --full".

Notes:
- The "instantiate-close-with-map" sub-test currently FAILs on this
  branch alone because it exercises DeletedDir cleanup, which is in
  the separate fix-deleteddir-leak PR. The test passes once both PRs
  are merged together.
@zaccharyfavere zaccharyfavere force-pushed the gc-config-and-leak-fix branch from bd1c802 to f6071d4 Compare June 4, 2026 10:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant