build(webkit): rebase fork patches onto upstream idle memory release - #4
Merged
Merged
Conversation
Only the owning thread can collect its mimalloc thread-local heap. An AutomaticThread that finishes its work and waits on its condition keeps every page it freed: retired pages, blocks other threads freed into its pages, and the free runs inside pages it still uses. Nothing touches them until the thread times out after 10 s and exits, or until it works again. For the wasm compiler threads this is about 10 MB each after one compile of a 4 MB module. With numberOfWasmCompilerThreads at cores - 1, a process that compiles wasm modules holds cores x 10 MB of RSS that no live object accounts for (oven-sh/bun#41438). The JIT worklist threads hold their B3 and Air temporaries the same way. When a thread has waited 100 ms without a notify, it now calls releaseFastMallocFreeMemoryForIdleThread() with the worklist lock dropped, then waits out the rest of its timeout. A thread that is notified within 100 ms, the usual case between tasks, pays nothing. The flag resets on each unit of work. On the consumer's mimalloc (USE_EXTERNAL_MIMALLOC) the release is mi_on_thread_idle(), which also discards the free runs inside still-used pages and hands the arena purge to mimalloc's scavenger thread. The vendored mimalloc does not have that entry point, so it falls back to a forced mi_theap_collect. The jsc shell and testFFI link the archives against the vendored mimalloc-obj, so they define mi_on_thread_idle in terms of the vendored API (shell/ExternalMimallocShims.cpp). libpas and system malloc are unchanged: the hook is compiled only under USE(MIMALLOC).
- notifyOne prefers a thread that is asleep over one that is releasing its free memory, which only sees the notification when it is done with that - the 100 ms before the release count towards the timeout of the thread - the fallback for the vendored mimalloc does not force the collect: that also purges what every other thread just freed
[JSC] Return free memory sooner: idle worker threads, Atomics.wait
Port oven-sh#753 (3a983c8), authored by Peter Steinberger (@steipete). Provenance: native W113 port 22908502aae3dea32fb377bc65cb15ecebfc4e0c, based on 1600131. Restore the end boundary before reading word status; preserve the regression corpus.
Port oven-sh#626 (f1e23f9), authored by robobun (@robobun). Provenance: native W113 port 6ecc6245dcab7418d05441535815a6388c82177d. Keep the default unchanged. The embedding Bun runtime must opt in and rebuild against the matching engine layout.
Integrate native W137 patch 7f54367f20a026648fcfc8d3790d303e1d656056 by Peter Steinberger. Related upstream work: oven-sh#274 and oven-sh/bun#37933 by robobun (@robobun). Capture the registration context and restore it for full loader reaction dispatch, preserving the result slot and disambiguating deferred-import tuples. The input patch passed 14 loader variants, retention and continuous-GC proof.
Integrate native W120 heap patch 70ac649212825168704d1ba8297a74397732bbbc and stack patch 013095fce722cd26795e15760b40562003ecc38c by Peter Steinberger. The matching Bun adapter builds on oven-sh/bun#32896 by robobun (@robobun). Measure the managed live set after full collection and exclude ArrayBuffer backing storage. The embedder owns worker-only termination. Process defaults stay unchanged. Native input proof passed seven engine counters, fourteen worker regressions and both custody cases.
Build the nine variants consumed by the OpenClaw Bun workflows with unchanged upstream recipes. Gate publication on matched native Linux regression proof, checksum verification, protected manual authorization and immutable GitHub releases. Remove inherited publishing workflows and document patching and rollback.
Bun prioritizes CI revision variables over its own checkout. Clear those variables only for nested Bun builds, preserving the outer WebKit workflow identity, and require both runtime revision fields to match the pinned sources.
Permit the exact openclaw/batch-1 branch through the existing source and protected-publication guards. Keep batch-one source independent while later engine work lands on openclaw/main; no recipe, engine or regression gate changes.
Bun's inherited `__esModule` accessor makes tsx unwrap an ordinary ESM namespace and lose explicit exports such as `zod.z`. This adds a native `require(esm)` facade with an own enumerable marker, null prototype, and live export bindings. Imported namespaces stay unmarked; explicit marker exports and normal namespace mutation rules are preserved. Adapts the own-property work in [oven-sh#279](oven-sh#279) by @robobun, together with [oven-sh/bun#33894](oven-sh/bun#33894). The paired Bun adapter removes the prototype override, covers normal require and require-cache materialization, preserves explicit CJS markers, and keeps native namespaces forwarded through CJS intact. Bun's existing ESM cache invalidation policy is unchanged. The read-only PR workflow builds the unchanged Linux engine lane, checks JSC results, and qualifies the matching Bun adapter. Publication remains in the separate manual artifact workflow. Engine and Bun must be rebuilt together. Validation on one AWS Linux c7a.24xlarge: - Node 24 marker-reflection matrix matches exactly across 10 ESM/CJS shapes; import-first, require-first, cache-first, explicit markers, live bindings, and CJS forwarding controls pass. - Both minimal Zod reproductions pass, including import-then-require. - 138 JSC configurations, 159 focused/plugin tests, and four continuous-GC controls pass. - Fork selection: all 46 result rows pass across 44 files and two dependency checks, with zero regressions. - Real production UI Vite/tsx builds, grouped and individual modules: Node 2/2, baseline Bun 0/2 with missing `zod.z`, patched Bun 2/2. - Broader module/VM run: 1,545 pass, eight skips, 64 existing TODOs. Two unrelated baseline failures and a release-only internal-test-module import error remain; all reproduce before this change. - Isolated Codex review is scoped-clean through P2. ABBA performance uses eight measured samples per arm. Warm namespace reads are unchanged (-0.03%); cached require/import are +0.13%/+0.69%. Fresh CJS imports are +2.09%. First ESM require costs +5.87% (47.69 → 50.49 ms per 1,024 modules), and reflection costs +14.42% (52.11 → 59.62 ms per 300,000 iterations) while exposing the additional own marker. The extra facade allocation and enumerable property have a measurable cost on these paths.
OpenClaw's catalog performance consumers require byte-based allocation profiles that survive garbage collection. JSC's existing snapshots and CPU profiler cannot recover the allocation sites of objects already collected. Add a VM-owned allocation sampler to `HeapProfiler`. It limits the existing free-list range to route selected C++ and generated allocations through the slow path, preserving the disabled fast-path instructions. Exponential byte intervals and inverse inclusion-probability weights produce allocation-site trees and samples. Precise allocations, auxiliary growth and relocation, and deduplicated JS string backing allocations participate; sampled addresses are reconciled before sweeping without retaining objects. Stack capture is limited to 128 frames. Collection inclusion follows the actual major/minor collection type. The sampling contract follows [Node/V8's implementation](https://github.com/nodejs/node/blob/v24.21.0/deps/v8/src/profiler/sampling-heap-profiler.cc). Measurements cover JSC cells, newly reported string backing storage, and GC-managed auxiliary storage. Other native allocations and external buffers are outside this profile. This changes engine layouts and requires a joint Bun rebuild and the matching in-process inspector adapter. Validation on one direct AWS Linux c7a.24xlarge using the unchanged WebKit lane recipe: - JSC regression passes with default tiers, interpreter, baseline JIT, and eager FTL. It covers warmed allocation paths, all major/minor inclusion combinations, collection/reuse, live profiles, sample IDs/ordinals, weighted totals, growth, an 8 MB auxiliary allocation, and short/long string payloads (including reports below the GC accounting threshold). The old engine fails because the sampler is absent. - Node 24.21.0 and matching Bun pass the shared callback/promise protocol and independent-worker fixtures. The surrounding inspector suite passes 78 tests. - The fork's 44-file Linux selection and two dependency checks pass on both baseline and candidate: 46/46, zero regressions. - Complete scoped P2 review is clean. ABBA measurements use eight samples per arm, pinned to one CPU, at the default 32 KiB interval. Ratios compare allocation-loop medians; setup, final profile retrieval/GC, and serialization are outside the timed loops. | Hot path | Candidate/baseline, sampling disabled | Enabled/disabled on candidate | | --- | ---: | ---: | | Objects | 1.002× | 1.279× | | Arrays | 0.998× | 1.345× | | JSON parsing | 1.004× | 1.013× | | Escaped-string parsing | 1.0004× | 1.007× | Both unmodified OpenClaw catalog consumers pass on snapshot `e9edda349faee43c13244cf3f4547fcbcb9c37fe`; baseline Bun fails both at missing `HeapProfiler.startSampling`, and Node 24 passes both. The gateway observes about 1.17 MB/list and 524 CPU samples with both profilers active. The native ordinary/catalog walks observe about 403 MB/5.82 MB, each over 3,000 rows and 47 pages. Node's corresponding allocation estimates are 209 MB/10.20 MB; runtime allocation strategies and layouts differ, so these values are not expected to match numerically. The existing read-only engine PR workflow now runs the sampler regression across those tiers using the same Linux artifact as namespace qualification, before its existing paired Bun/fork gate. Artifact publication remains the separate coordinator-controlled workflow; this PR does not dispatch it. The matching Bun change stays draft until the batch-2 engine is published and pinned.
steipete
changed the base branch from
openclaw/main
to
openclaw/release-5718a6ec
October 4, 2026 21:53
Retain the landed OpenClaw patch ancestry on the new upstream base, resolving only source/provenance metadata. Preserve the engine patches and lane recipes. Gate the protected release on the exact prepared upstream-sync Bun tree, unchanged memory-release tests, native patch regressions and the original paired fork selection. Record all adapter and source identities in provenance. Keep batch 1 immutable and isolate this publication on openclaw/release-5718a6ec. No Bun branch or consumer manifest is published by this change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebase the existing OpenClaw engine patches onto upstream
5718a6ec579b98362ea7276a426deedcc6281ef5(oven-sh#768), which releases free memory in idle compiler threads and long Atomics waits. Retain the Segmenter, opt-in VM Proxy, module-loader ALS, worker heap/stack, namespace and allocation-sampling patches as focused commits with their existing contributor credit.All nine original fork commits compare equal after replay, and the upstream delta has no direct file overlap with the six engine patches. The original main ancestry is retained; the engine and upstream lane/Docker recipes stay unchanged. This candidate is isolated on
openclaw/release-5718a6ecand publication remains behind the protected environment and exact-source green gates.Qualification now includes JSC promise/module/namespace/Segmenter regressions, sampling across four tiers, the original W113 paired Bun selection, and a separate build of W150's exact prepared upstream-sync Bun tree. That build retains the production checksum resolver, applies recorded embedding adapters, and runs the full sync/engine selection, inspector, ALS and worker regressions, seven native heap counters, startup checks, and the unchanged Atomics purge/notification and idle-Wasm memory tests. The old-source W113 candidate explicitly takes the newer mimalloc required by the engine's new idle hook. No Bun branch is pushed.
Static validation: 18 release-integrity tests, actionlint, shell syntax, exact W150 tree reproduction, and clean adapter application. Native run 37238008297 passed: 1,640 JSC stress and 1,639 module configurations, four sampler tiers, W113 46/46 in both arms, upstream-synced Bun 85/85 CI rows from 83 selected files, three unchanged memory-release tests with no skips, seven heap counters, four Proxy cases, fourteen worker cases, 41 namespace tests, 78 inspector tests and 223 ALS/resolver tests. Startup and ALS hook/continuous-GC/realm/retention checks passed. The tested merge tree is identical to the reviewed head. The nine-lane protected release remains pending; this PR does not itself publish.
The independent engine review raised the preexisting limited-worker full-GC scheduling policy. W120 explicitly documents and qualifies this policy; it is preserved here. A generational scheduling redesign and worker performance qualification remain separate work. No worker-performance improvement is claimed.
Thanks @Jarred-Sumner for the upstream allocator release behavior, and @steipete and @robobun for the retained engine work. Prior immutable releases and the Bun main manifest remain available for separately sequenced integration.