Skip to content

DX-3053: Stop one integration file deleting every other file's snapshots - #248

Merged
alitariksahin merged 1 commit into
mainfrom
DX-3053
Sep 24, 2026
Merged

alitariksahin merged 1 commit into
mainfrom
DX-3053

Conversation

@alitariksahin

@alitariksahin alitariksahin commented Sep 21, 2026 •

Copy link
Copy Markdown
Collaborator

The integration job has been red on main since at least 9 September, and which tests fail moves around between runs. Two separate causes. This handles the one that lives in this repo.

The suite was sabotaging itself

Vitest runs integration files in parallel, and delete-snapshots.integration.test.ts called the account-wide delete twice: once as a test, and once in afterAll. Nine other files depend on their own snapshot surviving long enough to restore from it. Whichever of them happened to be mid-run when that fired lost its snapshot and failed with Snapshot is not ready.

That explains the drift. One run it was the directory tests, the next a snapshot naming test, and on a third the directory file failed at a different case. Nothing was flaky. The timing of the wipe was what moved.

It is worse than intra-run interference: two CI runs on two pull requests share one key, so a wipe on one can break the other.

What changed

The by-id tests now record what they created and delete only that. Cleanup deletes the same list rather than everything.

The account-wide case is kept, not deleted, but gated behind UPSTASH_BOX_ALLOW_ACCOUNT_WIDE=1. It cannot safely share an account with anything else, so CI leaves it unset and it can be run deliberately against a scratch account. Its request shape is already asserted in box-delete-snapshots.test.ts, which pins the ?all=true query, so the guarantee CI cares about is unchanged.

One case added along the way: an empty id list must be rejected rather than read as "everything". That is the bug CLOUD-4771 fixed, and nothing in the integration suite held it in place.

Verified

Ran the three files that previously collided together against a live API, in parallel, as CI does:

delete-snapshots + cd + name
Test Files  3 passed (3)
Tests      37 passed | 2 skipped (39)

The two skips are the account-wide describe, correctly opted out.

The other cause is not in this repo

The two browser.integration.test.ts failures are a product bug: closing the last tab in a box makes Chromium exit, and the close then reports that teardown as an error although the tab is gone. Reproduced against production, three times out of three, with closing a non-last tab succeeding every time.

That is fixed in upstash/box-backend#272, which is open. Until it merges those two tests stay red here, and this pull request does not paper over them.

A third failure, uncovered by this change

With the snapshot wipe gone, cd / cwd > agent.run respects cwd after cd runs its assertions for the first time in a while, and fails on its own merits:

expected 'The README.md file does not exist in /workspace/home.' to contain '# Project A'

The snapshot bug was masking it: the test used to die in beforeEach before reaching the assertion.

It is not a flaky model answer. The agent names its directory explicitly, and reproducing against production gives the same result twice:

A. cd BEFORE any agent run, then ask for pwd  -> /workspace/home
B. agent run, then cd, then ask for pwd       -> /workspace/home

So this is not session pinning. The reason is narrower than "the folder is ignored", and my first reading of it was wrong.

runner-claude.ts does receive and use the folder. It deliberately pins Claude's process directory to /workspace/home so that session transcripts stay in one place across different folder values, and communicates the selected folder two other ways: it adds it to additionalDirectories, and it tells Claude in the system prompt that this is the working directory and to prefix shell commands with a cd. A prompt instruction is not a process working directory, so relative paths still resolve against the real one, which is what the failure shows.

runner-codex.ts has no such workaround and passes workingDirectory: WORK_DIR straight through, so the inconsistency is specific to the Claude runner.

That is a backend gap rather than anything this pull request can fix, and I have deliberately not skipped the test to make the job green.

It needs its own ticket: Claude agent runs must honour the selected working directory without breaking session continuation. Documenting agents as exempt from cd() would paper over an inconsistency rather than fix it, so the test stays as written. The fix is not simply swapping the cwd value either, since the pinned directory exists to keep sessions resumable, and that has to keep working.

@linear-code

linear-code Bot commented Sep 21, 2026

Copy link
Copy Markdown

DX-3053

Vitest runs integration files in parallel, and delete-snapshots called the
account-wide delete twice, once as a test and once in cleanup. Nine other files
depend on their own snapshot surviving, so whichever of them happened to be
mid-run lost it and failed with "Snapshot is not ready". That is why the red
tests moved around between runs: the directory tests one time, a snapshot name
test the next.

The by-id tests now track what they created and delete only that. The
account-wide case is kept but gated behind UPSTASH_BOX_ALLOW_ACCOUNT_WIDE,
because it cannot share a key with anything else, including a second CI run on
another pull request. Its request shape is already asserted in
box-delete-snapshots.test.ts, so CI keeps that guarantee without the hazard.

Also adds a case pinning that an empty id list is rejected rather than treated
as "everything", which is the bug CLOUD-4771 fixed.

Verified by running delete-snapshots, cd and name together against a live API:
37 passed. Those three collided before.
@alitariksahin
alitariksahin merged commit f5646dc into main Sep 24, 2026
2 of 5 checks passed
@alitariksahin
alitariksahin deleted the DX-3053 branch September 24, 2026 14:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants