diff --git a/docs/TRAPS.md b/docs/TRAPS.md index 80e4de0a..79621928 100644 --- a/docs/TRAPS.md +++ b/docs/TRAPS.md @@ -144,6 +144,8 @@ Every number here comes from our vocabulary measurements: a synthetic TTS set an - **Token file readable by others** [T4.45] (M1). Given a fresh install, the token file must be mode 0600 (or a user-only ACL on Windows) and created atomically. Test: stat after `akou doctor`. - **A 413 the client never reads** [CI flake] (M1). Given a body over 64 KB, akou must answer 413 with its JSON error, and the client must read it whole. Answering while the body is still arriving and then closing the socket leaves unread bytes on it; the kernel resets the connection instead of closing it, and on macOS and Windows a reset drops the answer already in flight, so the client sees `ECONNRESET`. Bun 1.4 closes at once when `maxRequestBodySize` is exceeded (Bun 1.3 did not), which made 5 % of 70 KB requests and two thirds of 512 KB requests fail. The 64 KB limit is the guard's and `readBody`'s; the server reads the rest of the body out (up to 1 MB, and for at most 2 s, since Bun's idle timeout is reset by every chunk and a trickled body could otherwise hold a refusal open) before any answer that did not consume it, whether a 413, a 401 or a 404. Test: a 512 KB body ten times over, a chunked body with no `Content-Length`, a bad token with a large body, each with the answer's JSON read; the limit exactly at 64 KB with the byte under it accepted as the positive control; the drain's two bounds on a fake reader, with a body that ends inside them read to its end. - **A stalled upload holds its socket** [SV-D3] (M1). Given an upload that stops sending mid-body, or a peer that drops off the network without closing, the server must cut it and delete the spool file. The upload routes turn Bun's idle cut off before reading the body, so each such request held a socket, a file descriptor and an open `.upload` until the process ended. The idle cut cannot simply stay on: in Bun 1.4 it counts from the start of the request, not from the last body chunk (measured: a body sending a byte a second is cut at the idle limit), so it would cut every upload longer than 60 s. The body has its own clock instead: no chunk for 60 s and the read fails with a 400 `bad_multipart` that closes the connection (`Connection: close`; without it the kept-alive socket stays open, since the idle cut is off for that request). Test: [upload-idle.e2e.test.ts](../tests/upload-idle.e2e.test.ts), a silent upload is answered 400, its socket closed and its file deleted. Positive control: under a 2 s idle timeout, an upload that sends a chunk a second for 5 s completes. +- **An upload fills the server's disk** [DSK-F1] (M1). Given a `jobs` key that keeps sending uploads (in parallel, or with `keep_audio`, which no retention sweep removes), the data volume filled: nothing checked free space before or while an upload was written, and the queue's room check cannot see uploads still arriving. Every multipart upload must keep the 1 GB a model download keeps (SV-M2): under it, 507 `insufficient_storage` before the first byte, and a cut, with its spool file deleted, when the volume falls under it while the body arrives (a look every 16 MB). A per-key cap on kept audio is a separate decision. Test: [multipart.test.ts](../tests/multipart.test.ts) and [upload-floor.e2e.test.ts](../tests/upload-floor.e2e.test.ts), with the same upload accepted when there is room as the positive control. +- **A 507 read as a server that is down** [DSK-F2] (M1). Given a server that refuses an upload with 507 because its disk is under the floor, its clients must read a refusal with its reason. The forwarding primary read any 5xx as a remote that is down: it marked it down, probed it and sent the job again in a loop. The dictation remote counted it toward the down-probe and told the user the remote was down. A 507 is now a rejection like a 422 on the primary: the remote stays up but gets no job for one probe period, so the job goes to another remote that offers it, else runs here or fails. It never counts toward the dictation probe, says `remote disk full`, and `akou transcribe` and `akou_transcribe` print the server's reason, never exit 69. Test: [remote-dispatch.e2e.test.ts](../tests/remote-dispatch.e2e.test.ts), [dictation-remote.e2e.test.ts](../tests/dictation-remote.e2e.test.ts) and [upload-floor.e2e.test.ts](../tests/upload-floor.e2e.test.ts), each with a 503 or an upload with room as the positive control; on the primary, two remotes where the first answers 507 and the job lands on the second, with the first alone as the control, where the job fails with `insufficient_storage`. - **A WAV header's channel count sizes the read buffer** [SV-P6] (M1). Given a 16 kHz PCM WAV whose header claims up to 65535 channels, the no-ffmpeg reader must read it with a buffer of a few MB. It sized the buffer as 65536 frames, so one crafted header asked for 8.6 GB whatever the file held. The buffer is about 4 MB of whole frames, at least one. Test: [server-audio.test.ts](../tests/server-audio.test.ts), a 65535-channel WAV read with no allocation over 8 MB. Positive control: a stereo WAV longer than one buffer averages every frame. - **The window's RPC socket taken over** [ElectroBun #518] (M1). Given a local process that opens the window's `ws://127.0.0.1:/socket?webviewId=` (the upgrade carries no token), the window loses its socket: the frames stay unreadable and unforgeable under the per-webview AES-GCM key, but every push and answer goes to the other socket. The page must not show a frozen transcript as if it were live: it must say it is reconnecting and, once that outlasts a follow request's timeout, tell the user to close and reopen the window. The recording is unaffected. Test: the state label with the reopen hint, and `suggestReopen` only for the window past `REOPEN_AFTER_MS`. diff --git a/docs/api/openapi.json b/docs/api/openapi.json index cd7fbb83..9b9a4757 100644 --- a/docs/api/openapi.json +++ b/docs/api/openapi.json @@ -5224,6 +5224,9 @@ "503": { "$ref": "#/components/responses/503.models_missing" }, + "507": { + "$ref": "#/components/responses/507.insufficient_storage" + }, "default": { "$ref": "#/components/responses/Error" } @@ -6940,6 +6943,9 @@ "503": { "$ref": "#/components/responses/503.no_engine" }, + "507": { + "$ref": "#/components/responses/507.insufficient_storage" + }, "default": { "$ref": "#/components/responses/Error" } @@ -7901,6 +7907,9 @@ "503": { "$ref": "#/components/responses/503.no_engine" }, + "507": { + "$ref": "#/components/responses/507.insufficient_storage" + }, "default": { "$ref": "#/components/responses/Error" } @@ -8002,6 +8011,9 @@ "503": { "$ref": "#/components/responses/503.no_engine" }, + "507": { + "$ref": "#/components/responses/507.insufficient_storage" + }, "default": { "$ref": "#/components/responses/Error" } @@ -8185,6 +8197,9 @@ "500": { "$ref": "#/components/responses/500.diarize_unavailable.engine_unavailable.internal.interrupted.model_download_failed.models_missing.remote_refused.transcription_failed" }, + "507": { + "$ref": "#/components/responses/507.insufficient_storage" + }, "default": { "$ref": "#/components/responses/Error" } @@ -8307,6 +8322,9 @@ "500": { "$ref": "#/components/responses/500.diarize_unavailable.engine_unavailable.internal.interrupted.model_download_failed.models_missing.remote_refused.transcription_failed" }, + "507": { + "$ref": "#/components/responses/507.insufficient_storage" + }, "default": { "$ref": "#/components/responses/Error" } @@ -11028,6 +11046,29 @@ } } } + }, + "507.insufficient_storage": { + "description": "Insufficient storage", + "content": { + "application/json": { + "schema": { + "allOf": [ + { + "$ref": "#/components/schemas/Error" + }, + { + "properties": { + "error": { + "enum": [ + "insufficient_storage" + ] + } + } + } + ] + } + } + } } }, "schemas": { @@ -11228,6 +11269,10 @@ "const": "idempotency_conflict", "description": "The Idempotency-Key was used with another file or other options." }, + { + "const": "insufficient_storage", + "description": "The server's disk is under the free space it keeps, so it takes no upload now; retrying does not help until space is freed." + }, { "const": "internal", "description": "An unexpected failure inside akou." diff --git a/src/main/api/errors.ts b/src/main/api/errors.ts index ebccff24..a363a12d 100644 --- a/src/main/api/errors.ts +++ b/src/main/api/errors.ts @@ -56,6 +56,8 @@ export const ERROR_CODES = { forbidden: "The key's scope does not reach this route.", gone: "The job was deleted, or passed the `retain_days` akou keeps one from its creation.", idempotency_conflict: "The Idempotency-Key was used with another file or other options.", + insufficient_storage: + "The server's disk is under the free space it keeps, so it takes no upload now; retrying does not help until space is freed.", internal: "An unexpected failure inside akou.", interrupted: "The job was running when the server stopped.", json_required: "A request that changes something needs `Content-Type: application/json`.", diff --git a/src/main/api/multipart.ts b/src/main/api/multipart.ts index a2730a2c..aa32752d 100644 --- a/src/main/api/multipart.ts +++ b/src/main/api/multipart.ts @@ -11,6 +11,11 @@ * it. Each is written to its own `.upload` in the folder the caller names, which is the job * queue's upload folder, whose orphans are deleted when the server starts. The caller owns every * spooled file: it keeps the one it needs and calls `discard` for the rest, on every path. + * + * The volume keeps a floor of free space (`UploadRoom`): an upload is refused with 507 + * `insufficient_storage` before its first byte when the volume is under the floor, and cut, its + * files deleted, when it falls under the floor while the body arrives. The check before the body + * cannot see the uploads already in flight; the one every `SPACE_CHECK_BYTES` can. */ import { createHash, randomUUID } from "node:crypto"; @@ -22,6 +27,29 @@ import { HttpError } from "./http.ts"; export const MAX_TEXT_BYTES = 64 * 1024; /** One part's header block. */ const MAX_HEADER_BYTES = 16 * 1024; +/** The file bytes written between two looks at the volume's free space. */ +export const SPACE_CHECK_BYTES = 16 * 1024 * 1024; + +/** The free space an upload must leave on the volume of its folder. */ +export interface UploadRoom { + /** The bytes free on the volume now. */ + free(): number; + /** Under this many bytes free, the upload is refused or cut. */ + floor: number; + /** The file bytes between two looks; `SPACE_CHECK_BYTES` by default. */ + every?: number; +} + +function checkRoom(room: UploadRoom): void { + const free = room.free(); + if (free >= room.floor) return; + throw new HttpError( + 507, + "insufficient_storage", + `the server has ${free} bytes free and keeps ${room.floor} free, so it takes no upload now`, + { free_bytes: free, floor_bytes: room.floor }, + ); +} /** A file part, on disk. */ export interface SpooledFile { @@ -109,12 +137,14 @@ function parseHead(block: string): PartHead { /** * Reads a `multipart/form-data` body, writing each file part into `dir` as it arrives. On any - * error the files spooled so far are deleted before it throws. + * error the files spooled so far are deleted before it throws. With `room`, the volume's free + * space is checked before the first byte and every `room.every` file bytes (507). */ export async function readMultipart( body: ReadableStream | null, contentType: string | null, dir: string, + room?: UploadRoom, ): Promise { const boundary = boundaryOf(contentType); if (!boundary || !body) throw badBody("the body is not multipart/form-data with a boundary"); @@ -126,6 +156,8 @@ export async function readMultipart( let buf: Buffer = Buffer.alloc(0); let ended = false; let text = 0; + /** The file bytes written since the last look at the free space. */ + let unchecked = 0; /** Reads more of the body into `buf`; false once the body has ended. */ const more = async (): Promise => { @@ -163,6 +195,8 @@ export async function readMultipart( }; try { + // Before the first byte: a volume already under the floor takes nothing. + if (room) checkRoom(room); // The preamble, up to the first boundary. for (;;) { const at = buf.indexOf(first); @@ -218,6 +252,12 @@ export async function readMultipart( form.add(head.name, spooled); try { await partBody(async (chunk) => { + // Uploads in flight together pass the first check together; this one sees them. + unchecked += chunk.length; + if (room && unchecked >= (room.every ?? SPACE_CHECK_BYTES)) { + unchecked = 0; + checkRoom(room); + } hash.update(chunk); size += chunk.length; await file.write(chunk); diff --git a/src/main/api/openapi.ts b/src/main/api/openapi.ts index c4cb4aba..cf517c25 100644 --- a/src/main/api/openapi.ts +++ b/src/main/api/openapi.ts @@ -93,6 +93,7 @@ const ERROR_STATUS_TEXT: Record = { 500: "Internal error", 501: "Not implemented", 503: "Unavailable", + 507: "Insufficient storage", }; /** @@ -117,8 +118,10 @@ export function sharedErrors(method: string, d: RouteDoc): RouteErrors { const multipart = d.body !== undefined && isMultipart(d.body); if (changes) add(415, multipart ? "multipart_required" : "json_required"); if (changes && d.body) { - if (multipart) add(400, "bad_multipart"); - else { + if (multipart) { + add(400, "bad_multipart"); + add(507, "insufficient_storage"); + } else { const fields = Object.keys(d.body); add(400, "bad_json"); if (d.body !== OPEN_BODY) add(400, "unknown_field"); diff --git a/src/main/api/routes/jobs.ts b/src/main/api/routes/jobs.ts index 6ec141ae..3a998685 100644 --- a/src/main/api/routes/jobs.ts +++ b/src/main/api/routes/jobs.ts @@ -31,7 +31,12 @@ import { QWEN_LANGUAGE_CODES } from "../../asr/llama-catalog.ts"; import { CANARY_1B_V2, WHISPER_LARGE_V3 } from "../../asr/models.ts"; import { translateEngine } from "../../asr/transcribe-cpp.ts"; import { eventView, type JobService, type QueueFull } from "../../server/jobs.ts"; -import { type ModelChoice, ModelRefused } from "../../server/model-store.ts"; +import { + FREE_MARGIN_BYTES, + freeOf, + type ModelChoice, + ModelRefused, +} from "../../server/model-store.ts"; import { PRESET_NAMES } from "../../server/presets.ts"; import { FORWARDED_HEADER } from "../../server/remotes.ts"; import { @@ -218,7 +223,8 @@ export function taskOf(form: Form): JobTask { /** * The multipart body, its files written into `dir` as they arrive (SV-D3), or 400 when it is not - * one. The caller deletes the files it does not keep with `form.discard`. + * one. The caller deletes the files it does not keep with `form.discard`. The volume of `dir` keeps + * the same 1 GB as a model download (SV-M2): under it the upload is refused or cut, 507. */ export async function formOf( c: RouteContext, @@ -229,7 +235,11 @@ export async function formOf( // so it would cut any upload that takes longer than it. It is off while the body arrives, and // the body has its own clock instead: an upload that stops sending is cut (`stalling`). c.timeout?.(0); - return readMultipart(stalling(c.req.body, stallMs), c.req.headers.get("content-type"), dir); + const free = () => (c.app.freeBytes ? c.app.freeBytes(dir) : freeOf(dir)); + return readMultipart(stalling(c.req.body, stallMs), c.req.headers.get("content-type"), dir, { + free, + floor: FREE_MARGIN_BYTES, + }); } /** How long an upload may send nothing before it is cut (SV-D3): the server's idle limit. */ diff --git a/src/main/api/server.ts b/src/main/api/server.ts index 79661704..aaf7ba4d 100644 --- a/src/main/api/server.ts +++ b/src/main/api/server.ts @@ -245,6 +245,8 @@ export interface ApiApp { recognizerFailed?(): boolean; /** Jobs waiting or running, for `/healthz`. */ queueDepth?(): number; + /** The bytes free on the volume that holds `dir`: an upload keeps a floor of it (507). */ + freeBytes?(dir: string): number; /** * The file jobs (docs/ux/SERVER.md section 5), in both modes: in app mode the one token owns * every job, as the key id `app`. diff --git a/src/main/dictation/remote.ts b/src/main/dictation/remote.ts index 8778235c..2da67a51 100644 --- a/src/main/dictation/remote.ts +++ b/src/main/dictation/remote.ts @@ -715,12 +715,15 @@ export interface RemoteHold { /** Dictations in a row that fail before the remote counts as down and is probed. */ export const REMOTE_DOWN_AFTER = 3; -/** A failure that says the remote is down, not misconfigured: no answer, or a 5xx. */ +/** + * A failure that says the remote is down, not misconfigured: no answer, or a 5xx. A 507 is a + * remote that is up with its disk under its floor: a refusal, never a reason to probe it. + */ function isDown(e: RemoteDictationError): boolean { return ( e.kind === "unreachable" || e.kind === "timeout" || - (e.kind === "status" && (e.status ?? 0) >= 500) + (e.kind === "status" && (e.status ?? 0) >= 500 && e.status !== 507) ); } @@ -809,7 +812,10 @@ export class RemoteEngine { ...l, engine: local.name, fallback_from: "remote", - notice: `remote down, used ${local.name}`, + notice: + err.kind === "status" && err.status === 507 + ? `remote disk full, used ${local.name}` + : `remote down, used ${local.name}`, remote_error: err.message, }; } diff --git a/src/main/index.ts b/src/main/index.ts index 5ba0244d..33c5869f 100644 --- a/src/main/index.ts +++ b/src/main/index.ts @@ -1280,6 +1280,10 @@ export class AkouApp implements ApiApp { * (`asr/models.ts`), and answers at once; `models()` reports the progress. When it finishes the * recognizer starts on the new files. */ + freeBytes(dir: string): number { + return (this.o.jobs?.modelStore?.freeBytes ?? freeOf)(dir); + } + pullModels(): ModelsStatus { const now = this.models(); if (now.state === "ready" || now.state === "downloading") return now; diff --git a/src/main/server/jobs.ts b/src/main/server/jobs.ts index cee63bd3..9d988d69 100644 --- a/src/main/server/jobs.ts +++ b/src/main/server/jobs.ts @@ -1759,6 +1759,16 @@ export class JobService { return route === "remote" ? { where: "wait" } : { where: "local" }; } + /** + * Marks a remote full after its 507 and says whether another remote may take the job: one is up + * and offers it, or would once it has room or is probed, as `routeOf` picks it. + */ + private otherRemote(j: Job, url: string): boolean { + this.remotes.full(url); + const route = this.noEngine() !== null ? "remote" : j.route; + return this.remotes.pick(this.namesOf(j), route !== "remote") !== null; + } + /** * With no engine here (#436), ends every queued job that can never run: one stored to run here * (a dictation, or a job a primary forwarded), every job while no remote is set, and, once every @@ -1887,6 +1897,8 @@ export class JobService { // The remote no longer offers it: asked again after its next probe. this.heldUntil.set(job.id, performance.now() + (this.o.remoteProbeMs ?? 30_000)); void this.remotes.probeAll(); + } else if (e.status === 507 && this.otherRemote(job, url)) { + // Its disk is full (DSK-F2): another remote that offers the job takes it. } else if (job.route === "remote" || this.noEngine()) { // With no engine here (#436) a refused job has nowhere else to run either. this.conclude(job, { diff --git a/src/main/server/remotes.ts b/src/main/server/remotes.ts index a71cc4bf..6c23d2b9 100644 --- a/src/main/server/remotes.ts +++ b/src/main/server/remotes.ts @@ -125,6 +125,8 @@ interface Remote { probing: boolean; /** The jobs it reported stalled at its last probe (`queue.stalled`, #407): none is sent to it. */ stalled: number; + /** Until when it is skipped: it answered 507, its disk under its floor, to an upload. */ + fullUntil: number; } export interface RemotesOptions { @@ -196,6 +198,7 @@ export class Remotes { inFlight: 0, probing: false, stalled: 0, + fullUntil: 0, }; this.remotes.set(e.url, r); } @@ -355,6 +358,15 @@ export class Remotes { if (was !== r.state) this.o.log("warn", `remote.${r.state} ${url}: ${err.message}`); } + /** + * A remote that answered 507 to an upload: up, with its disk under its floor. No job is sent to + * it for one probe period, so another remote that offers the job takes it (DSK-F2). + */ + full(url: string): void { + const r = this.remotes.get(url); + if (r) r.fullUntil = this.now() + (this.o.probeMs ?? REMOTE_PROBE_MS); + } + /** What `GET /v1/server` shows. */ view(): RemoteView[] { return this.current().map((r) => ({ @@ -400,6 +412,8 @@ export class Remotes { if (r.state !== "up" || !names.some((n) => r.offers.has(n))) continue; // A remote with a job stuck at running (#407) would hold a new one the same way. if (r.stalled > 0) continue; + // A remote whose disk is full (DSK-F2) would refuse it the same way. + if (r.fullUntil > this.now()) continue; if (r.inFlight >= REMOTE_IN_FLIGHT) { busy = true; continue; @@ -474,6 +488,10 @@ export class Remotes { if (res.status === 404 || res.status === 410) { return new RemoteError("lost", message, res.status, code); } + // 507: the remote is up and its disk is under its floor. Like a 422 it refuses the job; the + // remote is not marked down, it is skipped for a probe period (`full`), and the job goes to + // another remote that offers it, or runs here, or fails. + if (res.status === 507) return new RemoteError("rejected", message, res.status, code); if (res.status === 408 || res.status === 429 || res.status >= 500) { return new RemoteError("down", message, res.status, code); } diff --git a/tests/dictation-remote.e2e.test.ts b/tests/dictation-remote.e2e.test.ts index 70c00dec..490b1d35 100644 --- a/tests/dictation-remote.e2e.test.ts +++ b/tests/dictation-remote.e2e.test.ts @@ -659,6 +659,23 @@ describe("DC-R3: a remote down three dictations in a row is probed", () => { } }); + test("[DSK-F2] a remote whose disk is under its floor (507) is up: it never starts the probe", async () => { + const fake = await fakeRemote({ status: 507 }); + const e = engine({ url: `http://127.0.0.1:${fake.port}` }, { probeMs: 20 }); + try { + for (let i = 0; i < REMOTE_DOWN_AFTER + 1; i++) { + // Each dictation still falls back with the remote's reason, and never says it is down. + const r = await e.decode(HELLO); + expect(r.remote_error).toContain("507"); + expect(r.notice).toBe("remote disk full, used fast"); + } + expect(e.health()).toMatchObject({ down: false, failures: 0, probing: false }); + } finally { + e.close(); + fake.stop(); + } + }); + test("a dictation the remote answers resets the count", async () => { const f = flaky(); const e = engine({ url: f.url }, { probeMs: 60_000 }); diff --git a/tests/multipart.test.ts b/tests/multipart.test.ts index c6749f2a..c7deb708 100644 --- a/tests/multipart.test.ts +++ b/tests/multipart.test.ts @@ -204,3 +204,71 @@ describe("SV-D3: a multipart body read as it arrives", () => { expect(readdirSync(dir)).toEqual([]); }); }); + +describe("[DSK-F1] an upload keeps the volume's free-space floor", () => { + const FLOOR = 1_000_000; + + test("a volume under the floor refuses the upload with 507 before reading a byte", async () => { + const form = new FormData(); + form.append("file", new Blob([new Uint8Array(10_000)]), "a.wav"); + const body = await encoded(form); + const { dir } = tempDir("multipart-"); + let pulled = 0; + const stream = new ReadableStream({ + pull(ctl) { + pulled++; + ctl.enqueue(body.bytes); + ctl.close(); + }, + }); + const err = (await readMultipart(stream, body.type, dir, { + free: () => FLOOR - 1, + floor: FLOOR, + }).catch((e) => e)) as HttpError; + expect([err.status, err.code]).toEqual([507, "insufficient_storage"]); + expect(err.extra).toMatchObject({ free_bytes: FLOOR - 1, floor_bytes: FLOOR }); + expect(readdirSync(dir)).toEqual([]); + // Nothing was asked of the body beyond what a stream starts with. + expect(pulled).toBeLessThanOrEqual(1); + }); + + test("a volume that falls under the floor mid-body cuts the upload and deletes its file", async () => { + const form = new FormData(); + form.append("file", new Blob([new Uint8Array(400_000)]), "a.wav"); + const body = await encoded(form); + const { dir } = tempDir("multipart-"); + let looks = 0; + // Room at the start and at the first look inside the file, then none: another upload filled it. + const free = () => (++looks <= 2 ? FLOOR * 10 : FLOOR - 1); + const err = (await readMultipart(chunked(body.bytes, 16_384), body.type, dir, { + free, + floor: FLOOR, + every: 100_000, + }).catch((e) => e)) as HttpError; + expect([err.status, err.code]).toEqual([507, "insufficient_storage"]); + expect(looks).toBe(3); + expect(readdirSync(dir)).toEqual([]); + }); + + test("positive control: with room all along, the same upload is read whole", async () => { + const form = new FormData(); + const file = new Uint8Array(400_000).map((_, i) => i % 251); + form.append("file", new Blob([file]), "a.wav"); + const body = await encoded(form); + const f = folder(); + let looks = 0; + const got = await readMultipart(chunked(body.bytes, 16_384), body.type, f.dir, { + free: () => { + looks++; + return FLOOR; + }, + floor: FLOOR, + every: 100_000, + }); + expect(got.files()[0]?.sha256).toBe(sha(file)); + // One look before the body, then one after each 100,000 file bytes or a little more. + expect(looks).toBeGreaterThanOrEqual(4); + await got.discard(); + f.cleanup(); + }); +}); diff --git a/tests/remote-dispatch.e2e.test.ts b/tests/remote-dispatch.e2e.test.ts index 2baae89b..38fe8b4a 100644 --- a/tests/remote-dispatch.e2e.test.ts +++ b/tests/remote-dispatch.e2e.test.ts @@ -959,6 +959,133 @@ describe("a remote that forgot a job", () => { }); }); +describe("[DSK-F2] a remote whose disk is under its floor", () => { + test("its 507 to the upload is a refusal: the job runs here or fails, and the remote is not down", async () => { + const dir = tempDir("akou-remote-full-").dir; + const audio = join(dir, "a.wav"); + writeFileSync(audio, NOTE); + const url = "http://remote.example"; + const upload = async (status: number) => { + const r = new Remotes({ + entries: () => [`${url} ${keyFile(dir, "ak_x")}`], + fetch: (async () => + Response.json( + { error: status === 507 ? "insufficient_storage" : "internal", message: "no room" }, + { status }, + )) as unknown as typeof fetch, + log: () => {}, + }); + expect(r.configured()).toBe(true); + const job = { id: "job_1", audio, language: "auto", keywords: [], languages: [] }; + return (await r + .submit(url, { ...job, diarize: false }) + .catch((e: unknown) => e)) as RemoteError; + }; + const full = await upload(507); + expect(full).toBeInstanceOf(RemoteError); + expect([full.kind, full.status, full.code]).toEqual(["rejected", 507, "insufficient_storage"]); + expect(full.message).toContain("no room"); + // Positive control: another 5xx is still a remote that is down. + expect((await upload(503)).kind).toBe("down"); + }); +}); + +describe("[DSK-F2] a remote whose disk is full is skipped for another that offers the job", () => { + let posts = 0; + let full: ReturnType; + + beforeAll(() => { + // Up and offering fast, with its disk under its floor: every upload is refused with 507. + full = Bun.serve({ + hostname: "127.0.0.1", + port: 0, + fetch: (req) => { + const u = new URL(req.url); + if (u.pathname === "/v1/server") { + return Response.json({ + name: "akou", + version: "9.9.9", + mode: "server", + presets: [{ name: "fast", available: true }], + engines: [], + capabilities: { jobs: true, events: true, webhooks: true }, + }); + } + if (u.pathname === "/v1/keys/me") return Response.json({ id: "k1", scopes: ["jobs"] }); + if (u.pathname === "/v1/jobs" && req.method === "POST") { + posts++; + return Response.json( + { error: "insufficient_storage", message: "under 1 GB free" }, + { status: 507 }, + ); + } + return Response.json({ error: "not_found", message: "no" }, { status: 404 }); + }, + }); + }); + afterAll(() => full.stop(true)); + + test("a job only a remote can run lands on the second remote, and the full one stays up", async () => { + const worker = await server({ installed: true, keyName: "primary" }); + const pdir = tempDir("akou-primary-").dir; + const kf = keyFile(pdir, worker.key); + const primary = await server({ + home: pdir, + installed: false, + keyName: "archive", + settings: { + "server.remotes": [ + `http://127.0.0.1:${full.port} ${kf}`, + `http://127.0.0.1:${worker.port} ${kf}`, + ], + }, + }); + try { + // Both up first, so the job is offered to the full one before the other. + await until( + async () => + ((await get(primary, "/v1/server")).body?.remotes ?? []).every( + (r: Body) => r.state === "up", + ), + 10_000, + "both remotes up", + ); + posts = 0; + const r = await submit(primary, { preset: "fast" }); + expect(r.status).toBe(202); + const done = (await get(primary, `/v1/jobs/${r.body.id}?wait=30`)).body; + expect([done.status, done.error?.code]).toEqual(["done", undefined]); + expect(posts).toBe(1); + expect((await get(worker, "/v1/jobs", worker.token)).body.jobs).toHaveLength(1); + expect((await get(primary, "/v1/server")).body.remotes[0].state).toBe("up"); + } finally { + await primary.close(); + await worker.close(); + } + }); + + test("positive control: with no other remote, the job fails with the remote's reason", async () => { + const pdir = tempDir("akou-primary-").dir; + const primary = await server({ + home: pdir, + installed: false, + keyName: "archive", + settings: { "server.remotes": [`http://127.0.0.1:${full.port} ${keyFile(pdir, "ak_x")}`] }, + }); + try { + await remoteState(primary, "up"); + posts = 0; + const r = await submit(primary, { preset: "fast" }); + expect(r.status).toBe(202); + const end = (await get(primary, `/v1/jobs/${r.body.id}?wait=30`)).body; + expect([end.status, end.error?.code]).toEqual(["failed", "insufficient_storage"]); + expect(posts).toBe(1); + } finally { + await primary.close(); + } + }); +}); + describe("[SV-X5] a remote that went down while a probe was in flight is seen coming back", () => { // The stall of 2026-10-07: every remote's long-poll timed out while the periodic probe was in // flight. The probe ended `up`, as it began, so it logged nothing and told the queue nothing, diff --git a/tests/upload-floor.e2e.test.ts b/tests/upload-floor.e2e.test.ts new file mode 100644 index 00000000..3d61e91e --- /dev/null +++ b/tests/upload-floor.e2e.test.ts @@ -0,0 +1,163 @@ +/** + * The free-space floor of uploads, end to end through a real server: an upload is refused with + * 507 `insufficient_storage` when the volume of the upload folder is under 1 GB free, before the + * body is read and while it arrives, and leaves no file behind. The clients that upload (`akou + * transcribe`, the MCP server's `akou_transcribe`) read the 507 as a refusal with its reason, never + * as a server that is down. The forwarding primary and the dictation remote are in + * remote-dispatch.e2e.test.ts and dictation-remote.e2e.test.ts. + */ + +import { afterAll, beforeAll, describe, expect, setDefaultTimeout, test } from "bun:test"; +import { existsSync, readdirSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { ApiClient, EXIT } from "../src/main/cli/client.ts"; +import { FREE_MARGIN_BYTES } from "../src/main/server/model-store.ts"; +import { type AppRig, appRig } from "./api-helpers.ts"; +import { cli } from "./cli-helpers.ts"; +import { concat, silence, speak } from "./fixtures/asr-fake.ts"; +import { monoWav } from "./fixtures/audio.ts"; +import { tempDir } from "./helpers.ts"; +import { mcpClient } from "./mcp-helpers.ts"; + +setDefaultTimeout(30_000); + +const NOTE = monoWav(concat(silence(0.4), speak(["hello", "world"]), silence(1))); + +let rig: AppRig; +let key: string; +/** What the free-space probe reports; each test sets it. */ +let free: (dir: string) => number = () => 1e12; +const box = tempDir("akou-upload-floor-"); + +beforeAll(async () => { + rig = await appRig({ + settings: { "server.enabled": true, "api.bind": "127.0.0.1" }, + jobs: { modelStore: { freeBytes: (dir) => free(dir) } }, + }); + const r = await cli({ ...process.env, ...rig.env }, [ + "keys", + "create", + "--name", + "floor", + "--scope", + "jobs", + "--json", + ]); + expect(r.code).toBe(0); + key = r.json.key; +}); + +afterAll(async () => { + await rig?.close(); + box.cleanup(); +}); + +/** The spool files left in the job queue's upload folder. */ +function spooled(): string[] { + const dir = join(rig.app.configDir, "jobs", "audio"); + return existsSync(dir) ? readdirSync(dir).filter((f) => f.endsWith(".upload")) : []; +} + +async function post(path: string, file: Uint8Array, fields: Record = {}) { + const form = new FormData(); + form.append("file", new Blob([file], { type: "audio/wav" }), "note.wav"); + for (const [k, v] of Object.entries(fields)) form.append(k, v); + const res = await fetch(`http://127.0.0.1:${rig.port}/v1${path}`, { + method: "POST", + headers: { authorization: `Bearer ${key}` }, + body: form, + }); + const text = await res.text(); + // biome-ignore lint/suspicious/noExplicitAny: bodies are inspected field by field. + let body: any = null; + try { + body = JSON.parse(text); + } catch {} + return { status: res.status, body }; +} + +describe("[DSK-F1] an upload keeps 1 GB free on the server's volume", () => { + test("under the floor, a job upload is refused with 507 and leaves no file", async () => { + free = () => FREE_MARGIN_BYTES - 1; + const r = await post("/jobs", NOTE); + expect(r.status).toBe(507); + expect(r.body).toMatchObject({ + error: "insufficient_storage", + free_bytes: FREE_MARGIN_BYTES - 1, + floor_bytes: FREE_MARGIN_BYTES, + }); + expect(r.body.message).toContain("bytes free"); + expect(spooled()).toEqual([]); + // The OpenAI-compatible door reads its upload the same way. + const o = await post("/audio/transcriptions", NOTE); + expect([o.status, o.body?.error]).toEqual([507, "insufficient_storage"]); + expect(spooled()).toEqual([]); + }); + + test("a volume that falls under the floor mid-upload cuts it, and the client reads the 507", async () => { + let looks = 0; + // Room at the first look, before the body; none at the next, 16 MB into the file. + free = () => (++looks <= 1 ? 1e12 : FREE_MARGIN_BYTES - 1); + const big = new Uint8Array(40 * 1024 * 1024); + big.set(NOTE); + const r = await post("/jobs", big); + expect([r.status, r.body?.error]).toEqual([507, "insufficient_storage"]); + expect(looks).toBe(2); + expect(spooled()).toEqual([]); + }); + + test("positive control: with room on the volume, the same upload is accepted", async () => { + free = () => FREE_MARGIN_BYTES; + const r = await post("/jobs", NOTE); + expect(r.status).toBe(202); + expect(typeof r.body?.id).toBe("string"); + }); +}); + +describe("[DSK-F2] the clients read a 507 as a refusal, never as a server that is down", () => { + const note = join(box.dir, "note.wav"); + const env = () => ({ + ...process.env, + AKOU_HOME: box.dir, + AKOU_URL: `http://127.0.0.1:${rig.port}`, + AKOU_API_KEY: key, + }); + + test("akou transcribe prints the server's reason and exits 70, not 69 unavailable", async () => { + writeFileSync(note, NOTE); + free = () => FREE_MARGIN_BYTES - 1; + const r = await cli(env(), ["transcribe", note]); + expect(r.code).toBe(EXIT.software); + expect(r.code).not.toBe(EXIT.unavailable); + expect(r.out).toBe(""); + expect(r.err).toContain("bytes free"); + const j = await cli(env(), ["transcribe", note, "--json"]); + expect(j.json).toMatchObject({ error: "insufficient_storage" }); + // Positive control: with room, the same command prints the transcript. + free = () => 1e12; + const ok = await cli(env(), ["transcribe", note]); + expect(ok.code).toBe(EXIT.ok); + expect(ok.out).toContain("hello"); + }); + + test("akou_transcribe answers a tool error naming insufficient_storage", async () => { + writeFileSync(note, NOTE); + free = () => FREE_MARGIN_BYTES - 1; + const c = await mcpClient( + new ApiClient({ env: env(), client: "mcp", launch: null }), + "claude-code", + "server", + ); + try { + const r = await c.call("akou_transcribe", { path: note, wait: 0 }); + expect(r.isError).toBe(true); + expect(r.text).toStartWith("insufficient_storage: "); + expect(r.text).toContain("bytes free"); + // Positive control: with room, the same call submits the job. + free = () => 1e12; + expect((await c.call("akou_transcribe", { path: note, wait: 0 })).isError).toBe(false); + } finally { + await c.close(); + } + }); +});