fix(npm): Windows cache-lock contention aborts a concurrent first-run install with EPERM

Summary

@fullstack-ai-infra/mem-mcp's installer can abort a concurrent first run on
Windows with a hard EPERM instead of waiting for the process that already
holds the cache lock.

The per-asset lock is taken with mkdir, which is atomic on every platform. The
retry loop in acquireAssetLock recognizes only one error code as "someone else
has it":

} catch (err) {
  if (err.code !== "EEXIST") throw err;   // npm/install.js:309 on main
}

On Windows a contended mkdir does not reliably surface EEXIST. Node maps
the sharing/permission violation to EPERM (or EACCES), so ordinary contention
escapes as a fatal error, install() fails, and the user sees a failed install
even though nothing is wrong. POSIX hosts never see this because mkdir returns
EEXIST there.

Evidence (e2-source, upgraded to e3-reproduced by the linked PR)

Deterministic failure captured on the repository's own Windows gate,
npm wrapper compatibility (node24-windows):

✖ independent processes contend on one lock and download one binary (417.5845ms)
  Error: worker 7700 failed: code=1 signal=null
  Error: EPERM: operation not permitted, mkdir
    'C:\Users\RUNNER~1\AppData\Local\Temp\mem-mcp-install-test-GY8wZU\cache\.mem-mcp-windows-amd64.exe.lock'
      at mkdirSync (node:fs:1411:26)
      at acquireAssetLock (D:\a\mem\mem\npm\install.js:296:7)
      at async install (D:\a\mem\mem\npm\install.js:627:10)

Run: https://github.com/bytefolk/mem/actions/runs/33359259126/job/99387306881
This is the only occurrence recorded by the repository's Windows job so far.
That is expected for a timing-dependent race and is not evidence against the
diagnosis: the failing source line and the contention test are both on main
today (verified at 731a468), so no branch introduces the defect.
Reproduced deterministically off-Windows by injecting a single simulated
Windows EPERM at the first lock mkdir: the reproduction is red on the
pre-fix npm/install.js and green on the fixed one, which isolates the error
classification as the cause rather than the runner.
The failing test is test("independent processes contend on one lock and download one binary"), which is exactly the concurrent-first-run scenario a
real user hits when two mem-mcp processes start on a cold cache.

Impact

A cold-cache parallel start (two agents or shells launching mem-mcp at once)
can fail one of them on Windows with an unhelpful EPERM.
Because npm wrapper compatibility (node24-windows) is a required check that
every branch runs against this same installer, the race can redden any PR in
the repository and makes "green CI" a weaker signal.
No data corruption: the failure happens before any binary is published, and the
temp/lock cleanup paths are not implicated.

Requirements

REQ-001: A lock acquisition that fails with the error codes Windows uses for a
contended mkdir waits for the holder and then proceeds, on Windows.
REQ-002: A permission failure that is not contention still fails closed and
immediately. It must not be swallowed into the 120-second wait path, so a
genuinely unwritable cache directory remains a clear error rather than a hang.
REQ-003: Non-Windows behavior is unchanged: EEXIST remains contention, and
EPERM/EACCES remain fatal there.
REQ-004: The win32 branch must be provable on the host that runs the suite, so
the classification is not gated on having a Windows machine available.

Acceptance criteria

AC-001: isLockContention classifies EEXIST on any platform and
EPERM/EACCES only on win32; unrelated codes and a code-less Error are
not contention. Unit-tested as a table, so it runs on every job.
AC-002: A regression test drives the contention path through the real
acquireAssetLock and is red on the pre-fix source and green after it. It
injects one simulated Windows EPERM at the first lock mkdir and asserts the
call retries, takes the lock, writes its own owner.json, and releases cleanly.
AC-003: A fail-closed test proves a permission error that is not contention
rejects instead of waiting out the deadline.
AC-004: npm test passes in npm/, and the previously flaking
npm wrapper compatibility (node24-windows) job is green on the exact head.
AC-005: [Unreleased] in CHANGELOG.md records the user-visible fix.

Non-goals and forbidden shortcuts

No change to lock semantics: still one lock per asset, still mkdir-atomic,
same stale/orphan reclaim thresholds, same 120-second deadline.
No graceful-fs-style retry shim, no new dependency, no widening of the
contention set beyond EPERM/EACCES on win32.
Not switching the lock primitive (for example to open(..., 'wx')). The
mkdir design is sound; only the error classification is wrong.
No try/except: pass around lock failures anywhere.

Lifecycle, status, priority

priority: P2 — correctness bug on a supported platform plus a repository-wide
CI flake, with a retry workaround available to users.
Blockers: none.
Open decisions: none. The classification set is fixed by Node's documented
error mapping, and REQ-002 keeps the fail-closed property.
Labels applied by the author: type:bug, area:infra, severity:s2-medium,
evidence:e3-reproduced, status:ready. A maintainer should confirm triage.

Plan

Implement isLockContention(err, currentPlatform) in npm/install.js, resolve
osPlatform in acquireAssetLock the same way install() already does, thread
it into reclaimStaleLock, add the three tests above, and keep the diff to
npm/install.js, npm/install.test.js, and CHANGELOG.md.

Refs #122, #120, #113. The v0.1.1 npm publication in #122 ships this installer,
so landing it before the v0.1.1 tag is cut would put the fix in the first
public npm artifact.

schemaVersion: requirement-decision.v1
canonicalIssue: "https://github.com/fullstack-ai-infra/mem/issues/133"
previousRevision: R0
resultingRevision: R1
decisionType: clarification
oldDecision: "The issue had stable REQ-001..REQ-004 and AC-001..AC-005, but no explicit revision marker; its historical wording also referred to a v0.1.1 tag that has since been published."
newDecision: "R1 freezes the existing narrow Windows cache-lock classification and fail-closed requirements without changing their scope. The implementation may be evaluated against R1. v0.1.1 remains immutable and this work can only be considered for a later, separately authorized release."
reason: "A clean replacement for PR #134 needs a canonical consumed revision and an accurate release boundary. The requirement and acceptance text are unchanged; this records their reviewable identity and corrects only the already-observed release chronology."
impact: "A replacement PR may use Refs #133 and declare consumed revision R1. It still requires exact-head CI, independent CODEOWNER approval, normal merge authorization, post-merge verification, and separate product acceptance. This decision does not authorize a tag, npm publication, release, merge, or Issue closure."
approver: "@PeterGuy326"
decidedAt: "2026-08-31T18:35:02Z"

Owner lifecycle clarification: the canonical Issue is R1 / in-progress. Its existing REQ/AC and non-goals are unchanged. The published v0.1.1 release is not moved or rewritten.

Implementation candidate evidence (not a merge ledger)

Canonical consumed revision: R1.
Clean successor: PR #137, exact head 6f9b55cd9f091164297a5f44ad27708ec7914ad7, base cc727db0bc72655f299166de1f60756f5c686cc7.
Exact-head CI: all required checks succeeded, including npm wrapper compatibility (node24-windows); run family: https://github.com/fullstack-ai-infra/mem/actions/runs/33425962843.
Independent code preflight: PASS; local Node 24 full suite 41 passed, 0 failed, 1 expected Windows-only skip.
Human gate: @Bindy-lbb was requested once for a current-head CODEOWNER review. The author and last pusher will not self-approve.
Old PR #134 is closed as audit-only because of public non-noreply commits and the P1 bounded-retry defect. Its branch was not rewritten.

This records a review-ready candidate only. It does not authorize merge, tag, npm publication, release, or Issue closure.

fix(npm): bound Windows cache-lock contention retries (#137)

Closed by the assignee after AC-by-AC execution against `main`

Verified on 7a194f1eba4167d54bd46cf84cdbe86e00532319 (the squash of #137, current main tip), Node 24.19.0, Linux. Every number below was produced by running the suite on that tree, not by reading the diff.

AC	Verdict	Evidence as executed
AC-001	met	`isLockContention(err, currentPlatform = platform())` at `npm/install.js:247-250` returns `true` for `EEXIST` on any platform and for `EPERM`/`EACCES` only under `win32`. The table at `npm/install.test.js:770` is a plain synchronous test with 9 assertions covering `EEXIST`×{linux,darwin,win32}, `EPERM`/`EACCES` on `win32`, `EPERM` on linux, `EACCES` on darwin, `ENOENT` on win32, and a code-less `Error`. It lives in `install.test.js`, which every `npm wrapper` job runs.
AC-002	met, including the "red on the pre-fix source" clause re-proved today	`npm/install.test.js:783` drives the real `acquireAssetLock` in a child process, injects exactly one simulated Windows `EPERM` at the first lock `mkdir` (an `armed` latch), and asserts the retry succeeds, the lock is taken, the `owner.json` nonce/pid belong to this process, and the lock directory is gone afterward. Baseline on `main`: `install.test.js` 30 tests / 30 pass / 0 fail. Reverting `install.js:325` to the pre-fix line this issue quotes (`if (err.code !== "EEXIST") throw err;`) turns 2 red — AC-002's own case plus `a Windows lock inspection permission error fails promptly` (30 / 28 pass / 2 fail).
AC-003	met	Two fail-closed cases: `install.test.js:948` (persistent `EPERM` with no lock present) and `install.test.js:981` (`EACCES` while inspecting the lock). Each asserts the propagated `error.code` and that the wall time stayed under 1000 ms against a 10 s `waitTimeoutMs`, so a real permission failure cannot be swallowed into the wait path. `install.test.js:1024` covers the unrelated-code case propagating immediately. Second mutation control: forcing the `win32` arm of `isLockContention` to `return false` turns exactly 3 red — the AC-001 table, AC-002, and `:981` — the same three recorded before the merge.
AC-004	met	`npm test` on `main`: 42 tests / 41 pass / 0 fail / 1 skip (the skip is the pre-existing Windows-only `.cmd` shim case, not this fix). `npm wrapper compatibility (node24-windows)` is `success` on the exact landing head `ab8279a9c`, on `main@7a194f1`, and on 5 of 5 CI runs since the merge, all at `run_attempt=1` with no reds — so the timing-dependent contention test that this issue captured failing has had five consecutive real-Windows passes, not one lucky pass.
AC-005	met	`CHANGELOG.md` → `[Unreleased]` → `### Fixed` records the user-visible change in full, including the `EPERM`/`EACCES` classification and the preserved prompt-failure property.

Non-goals and forbidden shortcuts, checked rather than assumed. The lock semantics are untouched: LOCK_WAIT_TIMEOUT_MS = 120 * 1000, LOCK_STALE_MS = 10 * 60 * 1000, LOCK_POLL_MS = 50 (install.js:39-42), the primitive is still a single mkdirSync(lockPath, …) call site, npm/package.json declares zero dependencies so no graceful-fs-style shim arrived, and #137's diff is confined to the three files this issue's Plan named (CHANGELOG.md, npm/install.js, npm/install.test.js).

What I did not do. No concurrent install was run by hand on a Windows desktop; the Windows-side evidence is the repository's own windows-latest job plus the injected-EPERM cases. The two CI runs since the merge that were not all-green failed on Web → Audit dependencies and on Go — the npm advisory endpoint timeout/503 shape seen repeatedly in this repo, unrelated to the installer — and I did not chase either one here.

What closing this does not mean. It records that the defect is fixed on main and that the acceptance criteria above hold. It is not a release action: the published 0.1.1 artifact predates this fix and npm/package.json still carries 0.1.1, so the installer users actually get from npm today still has the old classification. That belongs to a separately authorized release.

The "this Windows box never throws EPERM from `mkdir`" explanation is contradicted by this repo's own CI log — and the real-Windows contention slot is still open

Comment, not a review, not a vote, and not an argument for reopening #133. waterbro-8, assignee and author of the closure comment above.

Where this text comes from

I drafted it on 2026-09-06 as a reply to a four-round Windows report about #168's acceptance, circulated in chat. Two of that draft's four sections are now moot and are not reproduced below: the web/audit-retry.mjs fail-open finding was fixed by its author in 65da42bc272 before I sent anything, and the "PR body has no tracking record" item resolved itself. What survives is the part that belongs on this issue rather than in chat, and I re-measured all of it today against main@2986fe38175f.

1. What the report got right, now re-measured today on `main`

npm/install.js is unchanged between 7a194f1eba4 and today's main (blob 55dcf41ca997df8e1ade15e2d265e5f9a436a11d, md5 7b8da775be88e66507580f385de402c5), so I re-ran the mutation there rather than quoting an old number. On install.test.js alone (blob 49aa4e54de266b22e4848152e8c512103a48d4ab), Linux, Node 24.19.0:

state	`tests / pass / fail`
as shipped	30 / 30 / 0
win32 arm of `isLockContention` (`install.js:247-250`) reverted to `return false`, nothing else touched	30 / 27 / 3

The three red are EEXIST is contention everywhere while permission errors are contention only on win32 (the all-platform table at :770), a contended Windows lock reported as EPERM waits for a proven lock (:783) and a Windows lock inspection permission error fails promptly (:981). That is the same three-set the report produced on 7a194f1eba4 and the same three I measured on 09-04 on the #137 merge tree ab8279a9c3a, so AC-001..AC-003's load-bearing claim now has three runs on three trees behind it. install.js was restored and its md5 re-checked after the experiment.

:948 (persistent EPERM with no lock present) staying green under that mutation is correct, not a coverage gap: forcing the arm to return false makes the immediate throw that case asserts true.

2. The root cause offered for "the mutation changed nothing in my runs" is false

The claim was that on that machine — NTFS under a WSL2 host — fs.mkdirSync on an existing directory always yields EEXIST and never EPERM/EACCES, so the win32 arm cannot be reached there, possibly needing a specific Windows version, an ACL change, or an antivirus to trigger. This repo's own CI holds the counter-example, and I re-retrieved it today:

field	value
job	`99387306881`, `npm wrapper compatibility (node24-windows)`
run / attempt	`33359259126` / attempt 1
conclusion	`failure`, completed `2026-08-31T05:05:39Z`
labels	`windows-2025` (a GitHub-hosted image)
red step	`Test cache, concurrency, and wrapper lifecycle`
failing test	`independent processes contend on one lock and download one binary`

Line 215, with the runner's 2026-08-31T05:05:34.98xxxxxxZ timestamp prefix dropped from each line (every other byte is as logged; line 214 immediately above it is Error: worker 7700 failed: code=1 signal=null):

Error: EPERM: operation not permitted, mkdir 'C:\Users\RUNNER~1\AppData\Local\Temp\mem-mcp-install-test-GY8wZU\cache\.mem-mcp-windows-amd64.exe.lock'
    at mkdirSync (node:fs:1411:26)
    at acquireAssetLock (D:\a\mem\mem\npm\install.js:296:7)
    at async install (D:\a\mem\mem\npm\install.js:627:10)

The stack frame is self-consistent: at head 46499b2ada, line 296 of npm/install.js is mkdirSync(lockPath, { mode: 0o700 });. So this is native Windows on a clean GitHub-hosted image, path in AppData\Local\Temp on NTFS — no special version, no modified ACL, no third-party lock holder.

A caveat for anyone who goes looking: GET /actions/runs/33359259126/jobs today lists nine jobs, all success, because that run was re-run. The failure lives in attempts/1. Read actions/jobs/<job_id> (it returns run_attempt and conclusion directly) rather than the run's current job list, or a real piece of evidence reads as a fabrication.

3. The narrowing that the evidence actually supports

"Not reachable by my method", not "not reachable". Two processes calling mkdir at the same instant get EEXIST; the EPERM above needs the lock directory to be delete-pending — someone has just removed it while a handle inside it is still open. #133's own body described this as a timing-dependent race with one recorded occurrence, which is consistent with both readings.

4. The slot #133's closure left open is still open, and here is a way to fill it that needs no race

acquireAssetLock (:297-339) prints nothing while it is contending — every log line in that flow comes from the caller after the lock is held. So "proc 1 logged installed verified binary at … and proc 2 logged verified existing binary at the same path" is produced identically by (a) proc 2 colliding with the lock and being retried, and (b) proc 2 simply arriving after proc 1 finished, never colliding. stderr cannot separate them, which is what the report correctly observed.

You do not need to win a race to exercise that arm. Pre-hold the lock instead, on the same MEM_MCP_CACHE_DIR the installer will use:

create <cache>/.mem-mcp-windows-amd64.exe.lock/ containing an owner.json whose hostname is not the test machine's;
set its mtime an hour in the future;
run the install once and time it.

Expected on the shipped code: the first mkdir is a contention error, the loop polls, and at the LOCK_WAIT_TIMEOUT_MS = 120 s default it throws Timed out waiting for mem-mcp cache lock: <path> (:334). Expected on the return false mutant: it aborts immediately with the raw EPERM/EACCES. "Waits the full budget" versus "aborts at once" is the measurement, and it does not depend on luck.

I ran that shape today on Linux against the exact shipped blob: node /tmp/lock133.cjs <path to npm/install.js>, where the harness require()s the installer's exported acquireAssetLock (module.exports at :782) and calls it against npm/install.js fetched at 2986fe38175f, with waitTimeoutMs shortened to 5000 ms and pollMs to 100 ms — the defaults are 120 s and 50 ms. So these are the shipped code path's numbers, not a reimplementation's; the harness itself is session-local scratch, and the four steps above are the repeatable part.

pre-held lock	result
foreign hostname in `owner.json`, mtime +1 h	threw `Timed out waiting for mem-mcp cache lock` after 5030 ms
foreign hostname, mtime −11 min	acquired after 101 ms (the reclaim path does work)
no `owner.json` at all, mtime +1 h	threw the same timeout after 5024 ms
same hostname, dead pid, mtime +1 h	threw the same timeout after 5029 ms

That table is a Linux run, so it exercises the wait/reclaim machinery and not the currentPlatform === "win32" arm — EEXIST is contention everywhere. But it does settle one loose end from the report: the observation "I pre-created .lock with a future mtime and the install still succeeded" cannot be produced by this code. age is clamped at :270 (Math.max(0, Date.now() - info.mtimeMs)), so a future mtime makes age zero and no reclaim branch can fire: a hostname mismatch returns null from ownerProcessIsAlive (:221) and lands on age >= staleMs, a same-host dead pid lands on age >= orphanGraceMs. Both are false at age 0. If an install completed in seconds with a lock supposedly in place, then the lock it looked at is not the lock that was created — most likely a different MEM_MCP_CACHE_DIR between the two steps, or an asset-name mismatch in the .<asset>.lock filename. Worth re-running with the lockPath from the thrown message pinned against the directory you created, before either of us calls it a defect. If it does reproduce with the paths equal, that is a separate bug and deserves its own issue rather than a comment on this closed one.

5. What I am not claiming

AC-002 and AC-003 remain injected-error evidence in the unit suite. The 08-31 log above is genuinely first-party Windows evidence that real contention has happened in this repo's CI — it cuts both ways, since it is also the proof that a stock Windows runner can produce the EPERM. What the closure comment recorded as not-done is a hand-run pair on a real Windows box by the person doing the acceptance, and that slot is still empty. §4 is an offer of how to fill it without depending on a race going your way.
Today's numbers are install.test.js only (node --test install.test.js), not npm test, and they are a Linux run. Neither reaches the win32 arm natively; that is the whole point of §2.
Nothing here is a request to reopen fix(npm): Windows cache-lock contention aborts a concurrent first-run install with EPERM #133, change its labels, or hold up a release. The fix is on main; what is open is one evidence slot and one root-cause sentence.

Navigation Menu

Repository navigation

fix(npm): Windows cache-lock contention aborts a concurrent first-run install with EPERM #133

Summary

Evidence (e2-source, upgraded to e3-reproduced by the linked PR)

Impact

Requirements

Acceptance criteria

Non-goals and forbidden shortcuts

Lifecycle, status, priority

Plan

Implementation candidate evidence (not a merge ledger)

Closed by the assignee after AC-by-AC execution against `main`

The "this Windows box never throws EPERM from `mkdir`" explanation is contradicted by this repo's own CI log — and the real-Windows contention slot is still open

Where this text comes from

1. What the report got right, now re-measured today on `main`

2. The root cause offered for "the mutation changed nothing in my runs" is false

3. The narrowing that the evidence actually supports

4. The slot #133's closure left open is still open, and here is a way to fill it that needs no race

5. What I am not claiming

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

fix(npm): Windows cache-lock contention aborts a concurrent first-run install with EPERM #133

Description

Summary

Evidence (e2-source, upgraded to e3-reproduced by the linked PR)

Impact

Requirements

Acceptance criteria

Non-goals and forbidden shortcuts

Lifecycle, status, priority

Plan

Activity

PeterGuy326 commented on Aug 31, 2026

PeterGuy326 commented on Aug 31, 2026

Implementation candidate evidence (not a merge ledger)

waterbro-8 commented on Sep 6, 2026

Closed by the assignee after AC-by-AC execution against main

waterbro-8 commented on Sep 8, 2026

The "this Windows box never throws EPERM from mkdir" explanation is contradicted by this repo's own CI log — and the real-Windows contention slot is still open

Where this text comes from

1. What the report got right, now re-measured today on main

2. The root cause offered for "the mutation changed nothing in my runs" is false

3. The narrowing that the evidence actually supports

4. The slot #133's closure left open is still open, and here is a way to fill it that needs no race

5. What I am not claiming

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Closed by the assignee after AC-by-AC execution against `main`

The "this Windows box never throws EPERM from `mkdir`" explanation is contradicted by this repo's own CI log — and the real-Windows contention slot is still open

1. What the report got right, now re-measured today on `main`