diff --git a/docs/TRAPS.md b/docs/TRAPS.md index 7896dd7f..5d44baee 100644 --- a/docs/TRAPS.md +++ b/docs/TRAPS.md @@ -147,6 +147,7 @@ Every number here comes from our vocabulary measurements: a synthetic TTS set an - **Token file readable by others** [T4.45] (M1). Given a fresh install, the token file must be mode 0600 (or a user-only ACL on Windows) and created atomically. Test: stat after `akou doctor`. - **A 413 the client never reads** [CI flake] (M1). Given a body over 64 KB, akou must answer 413 with its JSON error, and the client must read it whole. Answering while the body is still arriving and then closing the socket leaves unread bytes on it; the kernel resets the connection instead of closing it, and on macOS and Windows a reset drops the answer already in flight, so the client sees `ECONNRESET`. Bun 1.4 closes at once when `maxRequestBodySize` is exceeded (Bun 1.3 did not), which made 5 % of 70 KB requests and two thirds of 512 KB requests fail. The 64 KB limit is the guard's and `readBody`'s; the server reads the rest of the body out (up to 1 MB, and for at most 2 s, since Bun's idle timeout is reset by every chunk and a trickled body could otherwise hold a refusal open) before any answer that did not consume it, whether a 413, a 401 or a 404. Test: a 512 KB body ten times over, a chunked body with no `Content-Length`, a bad token with a large body, each with the answer's JSON read; the limit exactly at 64 KB with the byte under it accepted as the positive control; the drain's two bounds on a fake reader, with a body that ends inside them read to its end. - **A stalled upload holds its socket** [SV-D3] (M1). Given an upload that stops sending mid-body, or a peer that drops off the network without closing, the server must cut it and delete the spool file. The upload routes turn Bun's idle cut off before reading the body, so each such request held a socket, a file descriptor and an open `.upload` until the process ended. The idle cut cannot simply stay on: in Bun 1.4 it counts from the start of the request, not from the last body chunk (measured: a body sending a byte a second is cut at the idle limit), so it would cut every upload longer than 60 s. The body has its own clock instead: no chunk for 60 s and the read fails with a 400 `bad_multipart` that closes the connection (`Connection: close`; without it the kept-alive socket stays open, since the idle cut is off for that request). Test: [upload-idle.e2e.test.ts](../tests/upload-idle.e2e.test.ts), a silent upload is answered 400, its socket closed and its file deleted. Positive control: under a 2 s idle timeout, an upload that sends a chunk a second for 5 s completes. +- **An upload held in memory whole** [UPL-1] (M1). Given `akou transcribe`, `akou dictate FILE` or `akou_transcribe` with a long recording, the client must read the file from disk as it sends it. `fetch` with a `FormData` body builds the whole multipart body in memory before it sends a byte, even when the file part is a `Bun.file` (measured on Bun 1.4.2: a 256 MiB upload grew the CLI by about 535 MB with either a read file or a `Bun.file`). The client sends every form as its own streamed body instead (`uploadBody`), with the Content-Length it computes, so a server still refuses an upload over `server.max_upload_mb` before reading it; a file that changes size while it is sent fails the request. Test: [upload-stream.e2e.test.ts](../tests/upload-stream.e2e.test.ts), the CLI as a child sends a 256 MiB file to a fake server and its peak memory stays within 96 MiB of a 1 KiB upload; the streamed body goes through the server's multipart reader intact with and without a Content-Length. Positive control: a child that sends the same `FormData` with plain `fetch` grows past the limit. - **An upload fills the server's disk** [DSK-F1] (M1). Given a `jobs` key that keeps sending uploads (in parallel, or with `keep_audio`, which no retention sweep removes), the data volume filled: nothing checked free space before or while an upload was written, and the queue's room check cannot see uploads still arriving. Every multipart upload must keep the 1 GB a model download keeps (SV-M2): under it, 507 `insufficient_storage` before the first byte, and a cut, with its spool file deleted, when the volume falls under it while the body arrives (a look every 16 MB). A per-key cap on kept audio is a separate decision. Test: [multipart.test.ts](../tests/multipart.test.ts) and [upload-floor.e2e.test.ts](../tests/upload-floor.e2e.test.ts), with the same upload accepted when there is room as the positive control. - **A 507 read as a server that is down** [DSK-F2] (M1). Given a server that refuses an upload with 507 because its disk is under the floor, its clients must read a refusal with its reason. The forwarding primary read any 5xx as a remote that is down: it marked it down, probed it and sent the job again in a loop. The dictation remote counted it toward the down-probe and told the user the remote was down. A 507 is now a rejection like a 422 on the primary: the remote stays up but gets no job for one probe period, so the job goes to another remote that offers it, else runs here or fails. It never counts toward the dictation probe, says `remote disk full`, and `akou transcribe` and `akou_transcribe` print the server's reason, never exit 69. Test: [remote-dispatch.e2e.test.ts](../tests/remote-dispatch.e2e.test.ts), [dictation-remote.e2e.test.ts](../tests/dictation-remote.e2e.test.ts) and [upload-floor.e2e.test.ts](../tests/upload-floor.e2e.test.ts), each with a 503 or an upload with room as the positive control; on the primary, two remotes where the first answers 507 and the job lands on the second, with the first alone as the control, where the job fails with `insufficient_storage`. - **A WAV header's channel count sizes the read buffer** [SV-P6] (M1). Given a 16 kHz PCM WAV whose header claims up to 65535 channels, the no-ffmpeg reader must read it with a buffer of a few MB. It sized the buffer as 65536 frames, so one crafted header asked for 8.6 GB whatever the file held. The buffer is about 4 MB of whole frames, at least one. Test: [server-audio.test.ts](../tests/server-audio.test.ts), a 65535-channel WAV read with no allocation over 8 MB. Positive control: a stereo WAV longer than one buffer averages every frame. diff --git a/src/main/cli/client.ts b/src/main/cli/client.ts index d1bd2695..a1588db9 100644 --- a/src/main/cli/client.ts +++ b/src/main/cli/client.ts @@ -50,6 +50,7 @@ import { stopList, stopStrayLaunchers, } from "./heal.ts"; +import { uploadBody } from "./upload-body.ts"; /** Exit codes (DESIGN 6.1). */ export const EXIT = { @@ -242,7 +243,10 @@ export interface ClientOptions { export interface RequestOptions { body?: unknown; - /** A multipart body instead of JSON: an upload route (`akou transcribe`). */ + /** + * A multipart body instead of JSON: an upload route (`akou transcribe`). It is sent as a stream + * (`uploadBody`), so a file part is read from disk as it goes, never held in memory whole. + */ form?: FormData; query?: Record; /** Launch the app if it is not running. Default true. */ @@ -418,14 +422,20 @@ export class ApiClient { if (v !== undefined) url.searchParams.set(k, String(v)); } const hasBody = method !== "GET" && method !== "HEAD"; + // A form is streamed, built again for each send: a retry after a launch reads the file anew. + const upload = o.form ? uploadBody(o.form) : null; return fetch(url, { method, headers: { authorization: `Bearer ${remote ? to.key : this.token()}`, "x-akou-client": o.client ?? this.o.client, - ...(hasBody && !o.form ? { "content-type": "application/json" } : {}), + ...(upload + ? { "content-type": upload.contentType, "content-length": String(upload.length) } + : hasBody + ? { "content-type": "application/json" } + : {}), }, - body: o.form ?? (hasBody ? JSON.stringify(o.body ?? {}) : undefined), + body: upload ? upload.body : hasBody ? JSON.stringify(o.body ?? {}) : undefined, signal: o.signal ? AbortSignal.any([o.signal, AbortSignal.timeout(o.timeoutMs ?? 60_000)]) : AbortSignal.timeout(o.timeoutMs ?? 60_000), diff --git a/src/main/cli/commands/dictate.ts b/src/main/cli/commands/dictate.ts index c5b525d7..bd68faf8 100644 --- a/src/main/cli/commands/dictate.ts +++ b/src/main/cli/commands/dictate.ts @@ -15,11 +15,11 @@ * prints the new text, leaving the dictation as it was. */ -import { readFileSync, statSync } from "node:fs"; import { basename } from "node:path"; import { bool, duration, int, str } from "../args.ts"; import { EXIT, Hung, Unreachable } from "../client.ts"; import { api, type Body, type Command, type Ctx, finish, wall } from "../context.ts"; +import { uploadFile } from "../upload-body.ts"; import { usage } from "./calls.ts"; const ENGINES = ["auto", "fast", "best", "live", "remote"]; @@ -110,15 +110,11 @@ export const dictateCommand: Command = { if (engine !== undefined && !ENGINES.includes(engine)) { return usage(ctx, `--engine is one of ${ENGINES.join(", ")}`); } - let bytes: Uint8Array; - try { - if (!statSync(file).isFile()) return usage(ctx, `${file} is not a file`); - bytes = readFileSync(file); - } catch (err) { - return usage(ctx, `cannot read ${file}: ${(err as Error).message}`); - } + // Read only as it is sent (`uploadBody`), never held in memory whole. + const media = uploadFile(file); + if (typeof media === "string") return usage(ctx, media); const form = new FormData(); - form.append("file", new Blob([bytes]), basename(file)); + form.append("file", media, basename(file)); if (engine) form.append("engine", engine); const language = str(p, "language"); if (language) form.append("language", language); diff --git a/src/main/cli/commands/transcribe.ts b/src/main/cli/commands/transcribe.ts index ffc8d1ee..100cd308 100644 --- a/src/main/cli/commands/transcribe.ts +++ b/src/main/cli/commands/transcribe.ts @@ -12,11 +12,12 @@ * here: the server's own refusal is printed, so its limits live in one place (docs/ux/CLI.md rule 1). */ -import { readFileSync, statSync } from "node:fs"; +import { readFileSync } from "node:fs"; import { basename } from "node:path"; import { bool, repeated, str } from "../args.ts"; import { EXIT } from "../client.ts"; import { api, type Body, type Command, finish } from "../context.ts"; +import { uploadFile } from "../upload-body.ts"; import { usage } from "./calls.ts"; const PRESETS = ["lite", "fast", "best", "fusion", "auto"]; @@ -74,15 +75,11 @@ export const transcribeCommand: Command = { return usage(ctx, `cannot read ${keywordsFile}: ${(err as Error).message}`); } } - let bytes: Uint8Array; - try { - if (!statSync(file).isFile()) return usage(ctx, `${file} is not a file`); - bytes = readFileSync(file); - } catch (err) { - return usage(ctx, `cannot read ${file}: ${(err as Error).message}`); - } + // Read only as it is sent (`uploadBody`): a long recording is never held in memory whole. + const media = uploadFile(file); + if (typeof media === "string") return usage(ctx, media); const form = new FormData(); - form.append("file", new Blob([bytes]), basename(file)); + form.append("file", media, basename(file)); if (preset) form.append("preset", preset); const language = str(p, "language"); if (language) form.append("language", language); diff --git a/src/main/cli/upload-body.ts b/src/main/cli/upload-body.ts new file mode 100644 index 00000000..7f6e995e --- /dev/null +++ b/src/main/cli/upload-body.ts @@ -0,0 +1,124 @@ +/** + * An upload's multipart body, read from disk as it is sent (docs/TRAPS.md, "An upload held in + * memory whole"). `fetch` with a `FormData` body builds the whole body in memory before it sends a + * byte, even when the file part is a `Bun.file`: a 2 GB recording made the CLI hold 2 GB or more. + * Here every file part is its own `Blob`'s stream, so the client holds one chunk at a time. + * + * The body declares its length, as `fetch` does for a `FormData`, so a server refuses an upload + * over its `server.max_upload_mb` before a byte of the file is read. A file that changes size + * while it is sent fails the request instead of sending a body of another length. + */ + +import { closeSync, openSync, statSync } from "node:fs"; +import { basename } from "node:path"; + +/** A multipart body to send, with its headers. */ +export interface UploadBody { + body: ReadableStream; + contentType: string; + length: number; +} + +/** A name or filename inside a quoted header parameter, escaped as browsers do (WHATWG). */ +function quoted(s: string): string { + return s.replace(/\r/g, "%0D").replace(/\n/g, "%0A").replace(/"/g, "%22"); +} + +/** + * `form` as a streamed `multipart/form-data` body. A fresh call reads every file again. A file + * part's filename is its `File`'s own name: Bun's `FormData.entries()` drops the filename given to + * `append`, so name the `File` itself (`mediaFile`). + */ +export function uploadBody(form: FormData): UploadBody { + const enc = new TextEncoder(); + const boundary = `akou-${crypto.randomUUID()}`; + // Each file with its size now: the length declared is what the body must hold. + const parts: (Uint8Array | { blob: Blob; size: number })[] = []; + for (const [name, value] of form.entries()) { + if (typeof value === "string") { + parts.push( + enc.encode( + `--${boundary}\r\nContent-Disposition: form-data; name="${quoted(name)}"\r\n\r\n${value}\r\n`, + ), + ); + continue; + } + const file = value as File; + // A `Bun.file` appended as it is keeps its whole path as its name, whatever filename + // `append` was given: only the last segment ever goes out. + parts.push( + enc.encode( + `--${boundary}\r\nContent-Disposition: form-data; name="${quoted(name)}"; filename="${quoted(basename(file.name || "blob"))}"\r\n` + + `Content-Type: ${file.type || "application/octet-stream"}\r\n\r\n`, + ), + { blob: file, size: file.size }, + enc.encode("\r\n"), + ); + } + parts.push(enc.encode(`--${boundary}--\r\n`)); + const length = parts.reduce((n, p) => n + (p instanceof Uint8Array ? p.byteLength : p.size), 0); + + let next = 0; + /** The file part being sent: its reader, and the bytes it still owes. */ + let reading: { + reader: { + read(): Promise<{ done: boolean; value?: Uint8Array }>; + cancel(r?: unknown): Promise; + }; + left: number; + } | null = null; + const body = new ReadableStream({ + async pull(ctl) { + for (;;) { + if (reading) { + const r = await reading.reader.read(); + if (r.value) { + reading.left -= r.value.byteLength; + if (reading.left < 0) throw new Error("a file grew while it was being uploaded"); + ctl.enqueue(r.value); + return; + } + if (reading.left !== 0) throw new Error("a file shrank while it was being uploaded"); + reading = null; + } + const part = parts[next++]; + if (part === undefined) { + ctl.close(); + return; + } + if (part instanceof Uint8Array) { + ctl.enqueue(part); + return; + } + reading = { reader: part.blob.stream().getReader(), left: part.size }; + } + }, + async cancel(reason) { + await reading?.reader.cancel(reason).catch(() => {}); + }, + }); + return { body, contentType: `multipart/form-data; boundary=${boundary}`, length }; +} + +/** + * The file at `path` to upload, read only as it is sent, or why it cannot be: it is opened once + * here, so a missing or unreadable file is a usage error before anything is sent. + */ +export function uploadFile(path: string): File | string { + try { + if (!statSync(path).isFile()) return `${path} is not a file`; + closeSync(openSync(path, "r")); + } catch (err) { + return `cannot read ${path}: ${(err as Error).message}`; + } + return mediaFile(path); +} + +/** + * `path` as a `File` named by its last segment, still read from disk only as it is sent. A + * `Bun.file` in a `FormData` keeps its whole path as its name. + */ +export function mediaFile(path: string): File { + const disk = Bun.file(path); + return new File([disk], basename(path), { type: disk.type }); +} diff --git a/src/main/mcp/server.ts b/src/main/mcp/server.ts index 03fcfea2..88ac7d73 100644 --- a/src/main/mcp/server.ts +++ b/src/main/mcp/server.ts @@ -49,6 +49,7 @@ import { APP_VERSION } from "../app-info.ts"; import { callModeWords } from "../capture/call-mode.ts"; import type { ApiClient, ApiResponse, RequestOptions } from "../cli/client.ts"; import { type Body, describeError, wall } from "../cli/context.ts"; +import { mediaFile } from "../cli/upload-body.ts"; import { listPresets, usesSpeaker } from "../notes/presets.ts"; import { estimateTokens, type PackState, packState, quoteCallText } from "../query/render.ts"; import { capAnswer, type ToolResult } from "./bound.ts"; @@ -665,7 +666,8 @@ export const MAX_ANSWER_BYTES = 16 * 1024 * 1024; /** * A file to upload, or why not: missing, not a file, over `max` bytes, or not audio or video. The - * file goes out as a `Bun.file`, read as it is sent, never held in memory whole. + * file goes out in the client's streamed body (`uploadBody`), read from disk as it is sent: a + * `FormData` handed to `fetch` as it is would be built whole in memory first, `Bun.file` or not. */ export function readMedia( path: string, @@ -699,7 +701,7 @@ export function readMedia( if (!isMediaHead(head)) { return { error: `${path} is not an audio or video file; nothing was sent` }; } - return { file: Bun.file(path) }; + return { file: mediaFile(path) }; } /** A loopback host: a plain http request to it never leaves this machine. */ diff --git a/tests/fixtures/peak-rss.ts b/tests/fixtures/peak-rss.ts new file mode 100644 index 00000000..cb62e0a9 --- /dev/null +++ b/tests/fixtures/peak-rss.ts @@ -0,0 +1,26 @@ +/** + * Preloaded into a spawned bun (`--preload`): when the process exits, it writes its own peak + * memory, in bytes, to the file `AKOU_TEST_PEAK_FILE` names (tests/upload-stream.e2e.test.ts). + * + * On Linux `maxRSS` is not the child's own: `execve` keeps the larger of the spawning process's + * resident size and the child's, so a child spawned by a 1 GB test runner reports 1 GB whatever it + * does. `VmHWM` in `/proc/self/status` is the peak of the child's own memory map. Elsewhere + * `maxRSS` is the child's, in bytes on macOS and KiB on others. + */ + +import { readFileSync, writeFileSync } from "node:fs"; + +const MiB = 1024 * 1024; + +function peakBytes(): number { + if (process.platform === "linux") { + const hwm = /^VmHWM:\s+(\d+) kB$/m.exec(readFileSync("/proc/self/status", "utf8")); + if (!hwm?.[1]) throw new Error("no VmHWM in /proc/self/status"); + return Number(hwm[1]) * 1024; + } + const maxRSS = process.resourceUsage().maxRSS; + return maxRSS > 4 * MiB ? maxRSS : maxRSS * 1024; +} + +const out = process.env.AKOU_TEST_PEAK_FILE; +if (out) process.on("exit", () => writeFileSync(out, String(peakBytes()))); diff --git a/tests/upload-stream.e2e.test.ts b/tests/upload-stream.e2e.test.ts new file mode 100644 index 00000000..dbf55d47 --- /dev/null +++ b/tests/upload-stream.e2e.test.ts @@ -0,0 +1,254 @@ +/** + * An upload is read from disk as it is sent (docs/TRAPS.md, "An upload held in memory whole"). + * `fetch` with a `FormData` builds the whole multipart body in memory first, even for a + * `Bun.file` part, so `akou transcribe` of a long recording held the whole file, and more, before + * a byte went out. + * + * The memory cases run the real `akou` entry point as a child against a fake `AKOU_URL` server and + * compare the child's peak memory sending a 256 MiB file with sending a 1 KiB one. The parser + * cases send the streamed body to the server's own multipart reader, with and without a + * Content-Length. + */ + +import { afterEach, describe, expect, setDefaultTimeout, test } from "bun:test"; +import { createHash } from "node:crypto"; +import { readFileSync, truncateSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { readMultipart } from "../src/main/api/multipart.ts"; +import { uploadBody, uploadFile } from "../src/main/cli/upload-body.ts"; +import { CLI } from "./cli-helpers.ts"; +import { tempDir } from "./helpers.ts"; + +// Each memory case sends 256 MiB over loopback twice; a loaded CI box is slow. +setDefaultTimeout(120_000); + +const cleanups: (() => void)[] = []; +afterEach(() => { + for (const c of cleanups.splice(0).reverse()) c(); +}); + +const MiB = 1024 * 1024; +const BIG = 256 * MiB; + +interface Seen { + path: string; + length: string | null; + received: number; +} + +/** A fake akou server that takes jobs and dictations, reads every body to its end and refuses it. */ +function fakeServer(): { base: string; seen: Seen[] } { + const seen: Seen[] = []; + const server = Bun.serve({ + hostname: "127.0.0.1", + port: 0, + maxRequestBodySize: 2 * BIG, + fetch: async (req) => { + const path = new URL(req.url).pathname; + if (req.method === "GET" && path === "/v1/server") { + return Response.json({ mode: "server", capabilities: { jobs: true } }); + } + let received = 0; + if (req.body) for await (const chunk of req.body) received += chunk.byteLength; + seen.push({ path, length: req.headers.get("content-length"), received }); + return Response.json({ error: "test_refused", message: "read and refused" }, { status: 422 }); + }, + }); + cleanups.push(() => server.stop(true)); + return { base: `http://127.0.0.1:${server.port}`, seen }; +} + +/** A file of `bytes` zero bytes, sparse where the file system allows it. */ +function zeros(dir: string, name: string, bytes: number): string { + const path = join(dir, name); + writeFileSync(path, ""); + truncateSync(path, bytes); + return path; +} + +/** Writes the child's own peak memory, in bytes, to a file when it exits. */ +const PEAK_PRELOAD = join(import.meta.dir, "fixtures", "peak-rss.ts"); +/** Below this, a bun process's peak is not a real measurement. */ +const MIN_PEAK = 8 * MiB; +let peaks = 0; + +/** + * The child's peak memory, in bytes, as the child reports it on exit. Neither the parent's + * `Subprocess.resourceUsage()` nor the child's own `maxRSS` will do on Linux: both count the test + * runner's resident size at the spawn, so in the full suite a 1 KiB and a 256 MiB upload read the + * same peak (fixtures/peak-rss.ts). + */ +async function peak(argv: string[], env: Record, dir: string): Promise { + const out = join(dir, `peak-${++peaks}.txt`); + const proc = Bun.spawn([process.execPath, "--preload", PEAK_PRELOAD, ...argv], { + env: { ...env, AKOU_TEST_PEAK_FILE: out }, + stdin: "ignore", + stdout: "ignore", + stderr: "ignore", + }); + await proc.exited; + const bytes = Number(readFileSync(out, "utf8")); + expect(bytes).toBeGreaterThan(MIN_PEAK); + return bytes; +} + +function env(home: string, base: string): Record { + const e: Record = {}; + for (const [k, v] of Object.entries(process.env)) if (v !== undefined) e[k] = v; + for (const k of ["AKOU_API_KEY_FILE", "HTTP_PROXY", "http_proxy", "ALL_PROXY", "all_proxy"]) + delete e[k]; + return { ...e, AKOU_HOME: home, AKOU_URL: base, AKOU_API_KEY: "ak_test" }; +} + +/** + * Peak growth allowed for a 256 MiB upload: a body built whole grows by more than the file (about + * 535 MB measured on Bun 1.4.2), a streamed one by a few MB of buffers. + */ +const MAX_GROWTH = 96 * MiB; + +describe("[UPL-1] An upload held in memory whole", () => { + for (const [command, route] of [ + ["transcribe", "/v1/jobs"], + ["dictate", "/v1/dictations"], + ] as const) { + test(`akou ${command} of a 256 MiB file sends it whole without holding it in memory`, async () => { + const t = tempDir("akou-upl-"); + cleanups.push(t.cleanup); + const { base, seen } = fakeServer(); + const e = env(t.dir, base); + const small = await peak([CLI, command, zeros(t.dir, "small.wav", 1024)], e, t.dir); + const big = await peak([CLI, command, zeros(t.dir, "big.wav", BIG)], e, t.dir); + const uploads = seen.filter((s) => s.path === route); + expect(uploads).toHaveLength(2); + // The whole file went out, with the length the body declared. + expect(uploads[1]?.received).toBeGreaterThan(BIG); + expect(Number(uploads[1]?.length)).toBe(uploads[1]?.received as number); + // Compared as peaks, so a failure prints both. + expect(big).toBeLessThan(small + MAX_GROWTH); + }); + } + + test("positive control: fetch with the FormData as it is grows past the limit", async () => { + const t = tempDir("akou-upl-"); + cleanups.push(t.cleanup); + const { base, seen } = fakeServer(); + const e = env(t.dir, base); + const send = (path: string) => [ + "-e", + `const f = new FormData(); f.append("file", Bun.file(${JSON.stringify(path)}), "a.wav");` + + `await fetch(${JSON.stringify(`${base}/v1/jobs`)}, { method: "POST", body: f });`, + ]; + const small = await peak(send(zeros(t.dir, "small.wav", 1024)), e, t.dir); + const big = await peak(send(zeros(t.dir, "big.wav", BIG)), e, t.dir); + expect(seen.filter((s) => s.path === "/v1/jobs")).toHaveLength(2); + expect(big).toBeGreaterThan(small + MAX_GROWTH); + }); +}); + +describe("[UPL-1] The streamed body through the server's multipart reader", () => { + /** A server whose one route reads the body with `readMultipart`, as the upload routes do. */ + function parser(dir: string): string { + const server = Bun.serve({ + hostname: "127.0.0.1", + port: 0, + fetch: async (req) => { + const form = await readMultipart(req.body, req.headers.get("content-type"), dir); + const out = { + length: req.headers.get("content-length"), + fields: Object.fromEntries( + form.keys().map((k) => [k, form.getAll(k).filter((v) => typeof v === "string")]), + ), + files: form.files().map((f) => ({ + name: f.name, + type: f.type, + size: f.size, + sha256: f.sha256, + })), + }; + await form.discard(); + return Response.json(out); + }, + }); + cleanups.push(() => server.stop(true)); + return `http://127.0.0.1:${server.port}/`; + } + + // Three MiB of varied bytes, so the file crosses many chunks and holds the delimiter's bytes. + const bytes = new Uint8Array(3 * MiB + 17); + for (let i = 0; i < bytes.length; i++) bytes[i] = (i * 7919 + (i >> 9)) & 0xff; + const sha = createHash("sha256").update(bytes).digest("hex"); + + for (const declared of [true, false]) { + test(`fields and file arrive intact ${declared ? "with" : "without"} a Content-Length`, async () => { + const t = tempDir("akou-upl-"); + cleanups.push(t.cleanup); + const path = join(t.dir, "note.wav"); + writeFileSync(path, bytes); + const disk = Bun.file(path); + const form = new FormData(); + // The part's filename is the File's own: Bun's `FormData.entries()` drops the one `append` + // is given, so every caller names its File (`mediaFile`). + form.append("file", new File([disk], 'ñote "1".wav', { type: disk.type })); + form.append("preset", "fast"); + form.append("keywords[]", "Ñandú"); + form.append("keywords[]", "akou"); + const up = uploadBody(form); + const res = await fetch(parser(t.dir), { + method: "POST", + headers: { + "content-type": up.contentType, + ...(declared ? { "content-length": String(up.length) } : {}), + }, + body: up.body, + }); + expect(res.status).toBe(200); + const out = (await res.json()) as { + length: string | null; + fields: Record; + files: { name: string; type: string; size: number; sha256: string }[]; + }; + if (declared) expect(Number(out.length)).toBe(up.length); + else expect(out.length).toBeNull(); + expect(out.fields.preset).toEqual(["fast"]); + expect(out.fields["keywords[]"]).toEqual(["Ñandú", "akou"]); + expect(out.files).toHaveLength(1); + expect(out.files[0]).toMatchObject({ + name: "ñote %221%22.wav", + size: bytes.length, + sha256: sha, + }); + expect(out.files[0]?.type).toStartWith("audio/"); + }); + } + + test("a file that shrinks while it is sent fails the body instead of sending a short one", async () => { + const t = tempDir("akou-upl-"); + cleanups.push(t.cleanup); + const path = join(t.dir, "note.wav"); + writeFileSync(path, bytes); + const form = new FormData(); + form.append("file", Bun.file(path), "note.wav"); + const up = uploadBody(form); + truncateSync(path, 1024); + await expect(new Response(up.body).arrayBuffer()).rejects.toThrow("shrank"); + }); + + test("a Bun.file appended as it is sends its file name, never its folder", async () => { + const t = tempDir("akou-upl-"); + cleanups.push(t.cleanup); + const path = join(t.dir, "note.wav"); + writeFileSync(path, "RIFF"); + const form = new FormData(); + form.append("file", Bun.file(path), "note.wav"); + const sent = await new Response(uploadBody(form).body).text(); + expect(sent).toContain('filename="note.wav"'); + expect(sent).not.toContain(t.dir); + }); + + test("a missing or unreadable file is refused before anything is sent", () => { + const t = tempDir("akou-upl-"); + cleanups.push(t.cleanup); + expect(uploadFile(join(t.dir, "none.wav"))).toStartWith("cannot read "); + expect(uploadFile(t.dir)).toBe(`${t.dir} is not a file`); + }); +});