test(mq,app): fit the unit budget; fail fast on an uncreatable store

[EricAndrechek]

Fixes #617. Part of #613.

internal/app and internal/mq each took 11–12 s of the 15 s unit-test budget on main under a full parallel make test-unit, and timed out when the machine was loaded. After this PR they take under 5 s each. This PR also makes NewEmbedded fail at once on a store directory it cannot create, instead of waiting out the 5 s readiness check.

What changes

Commit	Change
2a494bd	NewEmbedded runs MkdirAll on the store first and returns nats store: <mkdir error>. Before this, a regular file at <data_dir>/nats failed JetStream in the background, and boot reported only nats server not ready after 5 s. New test: TestNewEmbedded_AStoreItCannotCreateFailsAtOnce.
d46e273, f897394	The directory is created at 0700, the same mode nats-server uses (defaultDirPerms). CHANGELOG entry under Fixed, limited to what mkdir catches: an existing directory that cannot be written to still fails the old way.
98acc11	The mq tests call t.Parallel. A new mq.EmbeddedSyncAlways is true in production and is set to false only in TestMain: on macOS the fsync on every JetStream write was more than half the run. Each store lives in storeDir, which retries its removal (#442).
e441f57	internal/app's new TestMain also turns the fsync off.
668c2c4	Removes the "occupied"-directory hunk that #612's squash added to TestEmbeddedNATS_SetMaxBytes_AQueueThatCannotOpen (see below).

These are 2a19d7e, 96e2112, bd059f2, 3d959b9 and 3c8a93b from perf/unit-test-budget, cherry-picked onto main. Two hunks from the perf branch are left out because they belong to stacks that are not on main: the t.Parallel lines for embedded_failed_test.go and for TestEmbeddedNATS_Publish_IdempotencyKeyDropsARepeat. Neither test exists on main; they come from the integration tree (#645). The CHANGELOG entry was re-applied under main's ### Fixed.

The AQueueThatCannotOpen conflict: measured

#612 (f5d8f48) and the perf change fix the same race in different ways. The race: after a failed open, the server removes the empty streams/ and $G directories on a goroutine of its own, and that removal overlaps the next open. f5d8f48 has three hunks:

• AQ-occ: an ignored occupied directory keeps streams/ non-empty during acme's failed open in SetMaxBytes_AQueueThatCannotOpen. This was added to main's squash.
• P-globex: PacesTheRetriesOfAQueueThatCannotOpen opens globex first, so its streams keep the directory occupied.
• App-globex: internal/app's TestNew_QueueOpenFailure "nested" subtest blocks globex rather than acme.

The perf change's version of the fix is AQ-wait: require.Eventually until the server has removed $G, and only then open globex.

These runs were on this branch, with -race and GOTOOLCHAIN=go1.26.6, on 2026-09-25 on an otherwise idle machine. "Isolated" means -run '^Name$' -count=20. "pkg" means the whole parallel internal/mq package with -count=5. "Mutation" means JetStreamMaxStore: math.MaxInt64 instead of / 2, which is the refusal the test exists to catch, at -count=5.

P-globex	AQ variant	AQueue isolated	Paces isolated	AQueue in pkg	Paces in pkg	other pkg fails	Mutation caught
kept	wait only (this PR)	20/20 pass	20/20	5/5	5/5	0	5/5 fail (caught)
kept	occupied only	20/20	20/20	5/5	5/5	0	5/5 fail (caught)
kept	both	0/20	20/20	0/5	5/5	—	—
dropped	wait only	20/20	20/20	5/5	4/5	1	5/5 caught
dropped	occupied only	20/20	20/20	5/5	5/5	0	5/5 caught
dropped	both	0/20	20/20	0/5	3/5	2	—

App-globex	TestNew_QueueOpenFailure isolated ×20	in the internal/app package ×3
kept (main)	20/20	3/3
reverted to acme	20/20	3/3

What the runs show:

• Both AQ fixes together always fail, 20/20. The occupied directory keeps $G from ever being removed, so the wait for its removal times out. The finding from test(e2e): api, ingest and sweeper in separate processes #645 reproduces.
• P-globex is what PacesTheRetries needs. Without it, the pacing test raced in the full parallel package (1/5 and 2/5 failures), even though it passed 20/20 when run alone. This is the finding from feat(mq): coord leases on a NATS KV bucket #646. It is a different test from AQueueThatCannotOpen, so the two findings do not conflict. P-globex is on main and this PR keeps it.
• For AQueueThatCannotOpen, AQ-wait alone and AQ-occ alone both passed every run, and both caught the mutation in this matrix. The earlier finding, that the occupied variant still passes with MaxInt64, did not reproduce here. On main as merged (occupied only), the mutation also fails the test 5/5 (measured). I kept AQ-wait because it is the version the perf change was written and measured against. It is also what the test's comment describes: no queue is kept open, and no directory is kept around to hold the reservation count up. Dropping AQ-occ instead of AQ-wait would work equally well by these numbers.
• App-globex made no difference in either direction in 20 isolated runs and 3 package runs. It is left as main has it.

Timings: full parallel make test-unit

Four runs, alternating between main at 2a2b886 and this branch, with -count=1 -race -cover, on an otherwise idle machine for every run.

	internal/app	internal/mq	whole run (DONE … in)
main (2a2b886)	11.30 / 11.37 / 11.41 / 11.45 s	12.05 / 12.18 / 12.30 / 12.38 s	12.07–12.39 s
this PR	4.71 / 4.72 / 4.73 / 4.81 s	3.42 / 3.64 / 3.64 / 3.94 s	5.07–5.27 s

An earlier baseline of main alone, under heavy load, gave internal/app 12.2–13.5 s and internal/mq 12.8–15.1 s. The 15.1 s run was already over the budget.

Verification

• make ci passed through the shared queue (GOTOOLCHAIN=go1.26.6, the known golangci-lint toolchain workaround), including all coverage gates.
• Pre-push reviewers were both run on HEAD 668c2c4 (opus, fresh context):
  • pre-push-reviewer: ship_it, with 0 MUST, 0 SHOULD and 0 MAY findings.
  • docs-reviewer: ship_it, with 0 findings. The CHANGELOG entry was checked against nats-server's defaultDirPerms and wireMQ's error path. No docs page needed a change.
  • Because of fix(agents): pre-push review gate can attest to unreviewed code (marker inherited across commits; wrong worktree checked) #454, the hook wrote the markers against the main checkout's HEAD, not this branch's HEAD. No marker was written by hand.

Left for later

• internal/testutil.NewEmbeddedMQ, used by the internal/ingest and internal/api tests, still fsyncs on every write. Those packages are not near the budget today. The reviewer noted it as the next place to get the same speed-up.
• 8776b4d (feat(app): choose the DynamoDB dedupe backend at boot #635) and 09d5c14 (feat(app): mq.backend selects embedded or external NATS #639) from the perf branch belong to their own stacks and are not included here.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EJr5tY4WQUy2sc4MbW67vL

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: b1aa60d5-1b67-4dd9-99ad-41805ee44375

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-code-quality]

Code Coverage Overview

Languages: Go

Go

The overall line coverage in commit 668c2c4 in the perf/unit-test-budge... branch remains at 94%, unchanged from commit 2a2b886 in the main branch.

Show a line coverage summary of the most impacted files.

File	main 2a2b886	perf/unit-test-budge... 668c2c4	+/-
internal/mq/embedded.go	89%	89%	0%
internal/ingest/worker.go	97%	98%	+1%

---

_{Updated September 25, 2026 22:07 UTC}