Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/TRAPS.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,8 @@ Every number here comes from our vocabulary measurements: a synthetic TTS set an
- **Token file readable by others** [T4.45] (M1). Given a fresh install, the token file must be mode 0600 (or a user-only ACL on Windows) and created atomically. Test: stat after `akou doctor`.
- **A 413 the client never reads** [CI flake] (M1). Given a body over 64 KB, akou must answer 413 with its JSON error, and the client must read it whole. Answering while the body is still arriving and then closing the socket leaves unread bytes on it; the kernel resets the connection instead of closing it, and on macOS and Windows a reset drops the answer already in flight, so the client sees `ECONNRESET`. Bun 1.4 closes at once when `maxRequestBodySize` is exceeded (Bun 1.3 did not), which made 5 % of 70 KB requests and two thirds of 512 KB requests fail. The 64 KB limit is the guard's and `readBody`'s; the server reads the rest of the body out (up to 1 MB, and for at most 2 s, since Bun's idle timeout is reset by every chunk and a trickled body could otherwise hold a refusal open) before any answer that did not consume it, whether a 413, a 401 or a 404. Test: a 512 KB body ten times over, a chunked body with no `Content-Length`, a bad token with a large body, each with the answer's JSON read; the limit exactly at 64 KB with the byte under it accepted as the positive control; the drain's two bounds on a fake reader, with a body that ends inside them read to its end.
- **A stalled upload holds its socket** [SV-D3] (M1). Given an upload that stops sending mid-body, or a peer that drops off the network without closing, the server must cut it and delete the spool file. The upload routes turn Bun's idle cut off before reading the body, so each such request held a socket, a file descriptor and an open `<uuid>.upload` until the process ended. The idle cut cannot simply stay on: in Bun 1.4 it counts from the start of the request, not from the last body chunk (measured: a body sending a byte a second is cut at the idle limit), so it would cut every upload longer than 60 s. The body has its own clock instead: no chunk for 60 s and the read fails with a 400 `bad_multipart` that closes the connection (`Connection: close`; without it the kept-alive socket stays open, since the idle cut is off for that request). Test: [upload-idle.e2e.test.ts](../tests/upload-idle.e2e.test.ts), a silent upload is answered 400, its socket closed and its file deleted. Positive control: under a 2 s idle timeout, an upload that sends a chunk a second for 5 s completes.
- **An upload fills the server's disk** [DSK-F1] (M1). Given a `jobs` key that keeps sending uploads (in parallel, or with `keep_audio`, which no retention sweep removes), the data volume filled: nothing checked free space before or while an upload was written, and the queue's room check cannot see uploads still arriving. Every multipart upload must keep the 1 GB a model download keeps (SV-M2): under it, 507 `insufficient_storage` before the first byte, and a cut, with its spool file deleted, when the volume falls under it while the body arrives (a look every 16 MB). A per-key cap on kept audio is a separate decision. Test: [multipart.test.ts](../tests/multipart.test.ts) and [upload-floor.e2e.test.ts](../tests/upload-floor.e2e.test.ts), with the same upload accepted when there is room as the positive control.
- **A 507 read as a server that is down** [DSK-F2] (M1). Given a server that refuses an upload with 507 because its disk is under the floor, its clients must read a refusal with its reason. The forwarding primary read any 5xx as a remote that is down: it marked it down, probed it and sent the job again in a loop. The dictation remote counted it toward the down-probe and told the user the remote was down. A 507 is now a rejection like a 422 on the primary: the remote stays up but gets no job for one probe period, so the job goes to another remote that offers it, else runs here or fails. It never counts toward the dictation probe, says `remote disk full`, and `akou transcribe` and `akou_transcribe` print the server's reason, never exit 69. Test: [remote-dispatch.e2e.test.ts](../tests/remote-dispatch.e2e.test.ts), [dictation-remote.e2e.test.ts](../tests/dictation-remote.e2e.test.ts) and [upload-floor.e2e.test.ts](../tests/upload-floor.e2e.test.ts), each with a 503 or an upload with room as the positive control; on the primary, two remotes where the first answers 507 and the job lands on the second, with the first alone as the control, where the job fails with `insufficient_storage`.
- **A WAV header's channel count sizes the read buffer** [SV-P6] (M1). Given a 16 kHz PCM WAV whose header claims up to 65535 channels, the no-ffmpeg reader must read it with a buffer of a few MB. It sized the buffer as 65536 frames, so one crafted header asked for 8.6 GB whatever the file held. The buffer is about 4 MB of whole frames, at least one. Test: [server-audio.test.ts](../tests/server-audio.test.ts), a 65535-channel WAV read with no allocation over 8 MB. Positive control: a stereo WAV longer than one buffer averages every frame.

- **The window's RPC socket taken over** [ElectroBun #518] (M1). Given a local process that opens the window's `ws://127.0.0.1:<port>/socket?webviewId=<n>` (the upgrade carries no token), the window loses its socket: the frames stay unreadable and unforgeable under the per-webview AES-GCM key, but every push and answer goes to the other socket. The page must not show a frozen transcript as if it were live: it must say it is reconnecting and, once that outlasts a follow request's timeout, tell the user to close and reopen the window. The recording is unaffected. Test: the state label with the reopen hint, and `suggestReopen` only for the window past `REOPEN_AFTER_MS`.
Expand Down
45 changes: 45 additions & 0 deletions docs/api/openapi.json
Original file line number Diff line number Diff line change
Expand Up @@ -5224,6 +5224,9 @@
"503": {
"$ref": "#/components/responses/503.models_missing"
},
"507": {
"$ref": "#/components/responses/507.insufficient_storage"
},
"default": {
"$ref": "#/components/responses/Error"
}
Expand Down Expand Up @@ -6940,6 +6943,9 @@
"503": {
"$ref": "#/components/responses/503.no_engine"
},
"507": {
"$ref": "#/components/responses/507.insufficient_storage"
},
"default": {
"$ref": "#/components/responses/Error"
}
Expand Down Expand Up @@ -7901,6 +7907,9 @@
"503": {
"$ref": "#/components/responses/503.no_engine"
},
"507": {
"$ref": "#/components/responses/507.insufficient_storage"
},
"default": {
"$ref": "#/components/responses/Error"
}
Expand Down Expand Up @@ -8002,6 +8011,9 @@
"503": {
"$ref": "#/components/responses/503.no_engine"
},
"507": {
"$ref": "#/components/responses/507.insufficient_storage"
},
"default": {
"$ref": "#/components/responses/Error"
}
Expand Down Expand Up @@ -8185,6 +8197,9 @@
"500": {
"$ref": "#/components/responses/500.diarize_unavailable.engine_unavailable.internal.interrupted.model_download_failed.models_missing.remote_refused.transcription_failed"
},
"507": {
"$ref": "#/components/responses/507.insufficient_storage"
},
"default": {
"$ref": "#/components/responses/Error"
}
Expand Down Expand Up @@ -8307,6 +8322,9 @@
"500": {
"$ref": "#/components/responses/500.diarize_unavailable.engine_unavailable.internal.interrupted.model_download_failed.models_missing.remote_refused.transcription_failed"
},
"507": {
"$ref": "#/components/responses/507.insufficient_storage"
},
"default": {
"$ref": "#/components/responses/Error"
}
Expand Down Expand Up @@ -11028,6 +11046,29 @@
}
}
}
},
"507.insufficient_storage": {
"description": "Insufficient storage",
"content": {
"application/json": {
"schema": {
"allOf": [
{
"$ref": "#/components/schemas/Error"
},
{
"properties": {
"error": {
"enum": [
"insufficient_storage"
]
}
}
}
]
}
}
}
}
},
"schemas": {
Expand Down Expand Up @@ -11228,6 +11269,10 @@
"const": "idempotency_conflict",
"description": "The Idempotency-Key was used with another file or other options."
},
{
"const": "insufficient_storage",
"description": "The server's disk is under the free space it keeps, so it takes no upload now; retrying does not help until space is freed."
},
{
"const": "internal",
"description": "An unexpected failure inside akou."
Expand Down
2 changes: 2 additions & 0 deletions src/main/api/errors.ts
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,8 @@ export const ERROR_CODES = {
forbidden: "The key's scope does not reach this route.",
gone: "The job was deleted, or passed the `retain_days` akou keeps one from its creation.",
idempotency_conflict: "The Idempotency-Key was used with another file or other options.",
insufficient_storage:
"The server's disk is under the free space it keeps, so it takes no upload now; retrying does not help until space is freed.",
internal: "An unexpected failure inside akou.",
interrupted: "The job was running when the server stopped.",
json_required: "A request that changes something needs `Content-Type: application/json`.",
Expand Down
42 changes: 41 additions & 1 deletion src/main/api/multipart.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,11 @@
* it. Each is written to its own `<uuid>.upload` in the folder the caller names, which is the job
* queue's upload folder, whose orphans are deleted when the server starts. The caller owns every
* spooled file: it keeps the one it needs and calls `discard` for the rest, on every path.
*
* The volume keeps a floor of free space (`UploadRoom`): an upload is refused with 507
* `insufficient_storage` before its first byte when the volume is under the floor, and cut, its
* files deleted, when it falls under the floor while the body arrives. The check before the body
* cannot see the uploads already in flight; the one every `SPACE_CHECK_BYTES` can.
*/

import { createHash, randomUUID } from "node:crypto";
Expand All @@ -22,6 +27,29 @@ import { HttpError } from "./http.ts";
export const MAX_TEXT_BYTES = 64 * 1024;
/** One part's header block. */
const MAX_HEADER_BYTES = 16 * 1024;
/** The file bytes written between two looks at the volume's free space. */
export const SPACE_CHECK_BYTES = 16 * 1024 * 1024;

/** The free space an upload must leave on the volume of its folder. */
export interface UploadRoom {
/** The bytes free on the volume now. */
free(): number;
/** Under this many bytes free, the upload is refused or cut. */
floor: number;
/** The file bytes between two looks; `SPACE_CHECK_BYTES` by default. */
every?: number;
}

function checkRoom(room: UploadRoom): void {
const free = room.free();
if (free >= room.floor) return;
throw new HttpError(
507,
"insufficient_storage",
`the server has ${free} bytes free and keeps ${room.floor} free, so it takes no upload now`,
{ free_bytes: free, floor_bytes: room.floor },
);
}

/** A file part, on disk. */
export interface SpooledFile {
Expand Down Expand Up @@ -109,12 +137,14 @@ function parseHead(block: string): PartHead {

/**
* Reads a `multipart/form-data` body, writing each file part into `dir` as it arrives. On any
* error the files spooled so far are deleted before it throws.
* error the files spooled so far are deleted before it throws. With `room`, the volume's free
* space is checked before the first byte and every `room.every` file bytes (507).
*/
export async function readMultipart(
body: ReadableStream<Uint8Array> | null,
contentType: string | null,
dir: string,
room?: UploadRoom,
): Promise<StreamedForm> {
const boundary = boundaryOf(contentType);
if (!boundary || !body) throw badBody("the body is not multipart/form-data with a boundary");
Expand All @@ -126,6 +156,8 @@ export async function readMultipart(
let buf: Buffer = Buffer.alloc(0);
let ended = false;
let text = 0;
/** The file bytes written since the last look at the free space. */
let unchecked = 0;

/** Reads more of the body into `buf`; false once the body has ended. */
const more = async (): Promise<boolean> => {
Expand Down Expand Up @@ -163,6 +195,8 @@ export async function readMultipart(
};

try {
// Before the first byte: a volume already under the floor takes nothing.
if (room) checkRoom(room);
// The preamble, up to the first boundary.
for (;;) {
const at = buf.indexOf(first);
Expand Down Expand Up @@ -218,6 +252,12 @@ export async function readMultipart(
form.add(head.name, spooled);
try {
await partBody(async (chunk) => {
// Uploads in flight together pass the first check together; this one sees them.
unchecked += chunk.length;
if (room && unchecked >= (room.every ?? SPACE_CHECK_BYTES)) {
unchecked = 0;
checkRoom(room);
}
hash.update(chunk);
size += chunk.length;
await file.write(chunk);
Expand Down
7 changes: 5 additions & 2 deletions src/main/api/openapi.ts
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,7 @@ const ERROR_STATUS_TEXT: Record<number, string> = {
500: "Internal error",
501: "Not implemented",
503: "Unavailable",
507: "Insufficient storage",
};

/**
Expand All @@ -117,8 +118,10 @@ export function sharedErrors(method: string, d: RouteDoc): RouteErrors {
const multipart = d.body !== undefined && isMultipart(d.body);
if (changes) add(415, multipart ? "multipart_required" : "json_required");
if (changes && d.body) {
if (multipart) add(400, "bad_multipart");
else {
if (multipart) {
add(400, "bad_multipart");
add(507, "insufficient_storage");
} else {
const fields = Object.keys(d.body);
add(400, "bad_json");
if (d.body !== OPEN_BODY) add(400, "unknown_field");
Expand Down
16 changes: 13 additions & 3 deletions src/main/api/routes/jobs.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,12 @@ import { QWEN_LANGUAGE_CODES } from "../../asr/llama-catalog.ts";
import { CANARY_1B_V2, WHISPER_LARGE_V3 } from "../../asr/models.ts";
import { translateEngine } from "../../asr/transcribe-cpp.ts";
import { eventView, type JobService, type QueueFull } from "../../server/jobs.ts";
import { type ModelChoice, ModelRefused } from "../../server/model-store.ts";
import {
FREE_MARGIN_BYTES,
freeOf,
type ModelChoice,
ModelRefused,
} from "../../server/model-store.ts";
import { PRESET_NAMES } from "../../server/presets.ts";
import { FORWARDED_HEADER } from "../../server/remotes.ts";
import {
Expand Down Expand Up @@ -218,7 +223,8 @@ export function taskOf(form: Form): JobTask {

/**
* The multipart body, its files written into `dir` as they arrive (SV-D3), or 400 when it is not
* one. The caller deletes the files it does not keep with `form.discard`.
* one. The caller deletes the files it does not keep with `form.discard`. The volume of `dir` keeps
* the same 1 GB as a model download (SV-M2): under it the upload is refused or cut, 507.
*/
export async function formOf(
c: RouteContext<ApiApp>,
Expand All @@ -229,7 +235,11 @@ export async function formOf(
// so it would cut any upload that takes longer than it. It is off while the body arrives, and
// the body has its own clock instead: an upload that stops sending is cut (`stalling`).
c.timeout?.(0);
return readMultipart(stalling(c.req.body, stallMs), c.req.headers.get("content-type"), dir);
const free = () => (c.app.freeBytes ? c.app.freeBytes(dir) : freeOf(dir));
return readMultipart(stalling(c.req.body, stallMs), c.req.headers.get("content-type"), dir, {
free,
floor: FREE_MARGIN_BYTES,
});
}

/** How long an upload may send nothing before it is cut (SV-D3): the server's idle limit. */
Expand Down
2 changes: 2 additions & 0 deletions src/main/api/server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -245,6 +245,8 @@ export interface ApiApp {
recognizerFailed?(): boolean;
/** Jobs waiting or running, for `/healthz`. */
queueDepth?(): number;
/** The bytes free on the volume that holds `dir`: an upload keeps a floor of it (507). */
freeBytes?(dir: string): number;
/**
* The file jobs (docs/ux/SERVER.md section 5), in both modes: in app mode the one token owns
* every job, as the key id `app`.
Expand Down
12 changes: 9 additions & 3 deletions src/main/dictation/remote.ts
Original file line number Diff line number Diff line change
Expand Up @@ -715,12 +715,15 @@ export interface RemoteHold {
/** Dictations in a row that fail before the remote counts as down and is probed. */
export const REMOTE_DOWN_AFTER = 3;

/** A failure that says the remote is down, not misconfigured: no answer, or a 5xx. */
/**
* A failure that says the remote is down, not misconfigured: no answer, or a 5xx. A 507 is a
* remote that is up with its disk under its floor: a refusal, never a reason to probe it.
*/
function isDown(e: RemoteDictationError): boolean {
return (
e.kind === "unreachable" ||
e.kind === "timeout" ||
(e.kind === "status" && (e.status ?? 0) >= 500)
(e.kind === "status" && (e.status ?? 0) >= 500 && e.status !== 507)
);
}

Expand Down Expand Up @@ -809,7 +812,10 @@ export class RemoteEngine {
...l,
engine: local.name,
fallback_from: "remote",
notice: `remote down, used ${local.name}`,
notice:
err.kind === "status" && err.status === 507
? `remote disk full, used ${local.name}`
: `remote down, used ${local.name}`,
remote_error: err.message,
};
}
Expand Down
4 changes: 4 additions & 0 deletions src/main/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1280,6 +1280,10 @@ export class AkouApp implements ApiApp {
* (`asr/models.ts`), and answers at once; `models()` reports the progress. When it finishes the
* recognizer starts on the new files.
*/
freeBytes(dir: string): number {
return (this.o.jobs?.modelStore?.freeBytes ?? freeOf)(dir);
}

pullModels(): ModelsStatus {
const now = this.models();
if (now.state === "ready" || now.state === "downloading") return now;
Expand Down
12 changes: 12 additions & 0 deletions src/main/server/jobs.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1759,6 +1759,16 @@ export class JobService {
return route === "remote" ? { where: "wait" } : { where: "local" };
}

/**
* Marks a remote full after its 507 and says whether another remote may take the job: one is up
* and offers it, or would once it has room or is probed, as `routeOf` picks it.
*/
private otherRemote(j: Job, url: string): boolean {
this.remotes.full(url);
const route = this.noEngine() !== null ? "remote" : j.route;
return this.remotes.pick(this.namesOf(j), route !== "remote") !== null;
}

/**
* With no engine here (#436), ends every queued job that can never run: one stored to run here
* (a dictation, or a job a primary forwarded), every job while no remote is set, and, once every
Expand Down Expand Up @@ -1887,6 +1897,8 @@ export class JobService {
// The remote no longer offers it: asked again after its next probe.
this.heldUntil.set(job.id, performance.now() + (this.o.remoteProbeMs ?? 30_000));
void this.remotes.probeAll();
} else if (e.status === 507 && this.otherRemote(job, url)) {
// Its disk is full (DSK-F2): another remote that offers the job takes it.
} else if (job.route === "remote" || this.noEngine()) {
// With no engine here (#436) a refused job has nowhere else to run either.
this.conclude(job, {
Expand Down
Loading
Loading