Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -255,7 +255,7 @@ A call's streaming model loads in the live Worker when the call starts, and its
4. **Pad** every span to at least 0.5 s before decoding, with the zeros added **after** the speech, in one function shared by every path (live, final, single file). Short words vanished under a 0.3 s engine floor in hark; that floor has not been measured on sherpa-onnx with Parakeet yet. The padding evidence from the streaming zipformer is mixed and does not transfer: a fresh stream of that model dropped the first words *until* it had 2 to 8 s of leading audio, so on that model leading audio restores words rather than losing them. The 0.2 s "Yes." fixture in [TRAPS.md](TRAPS.md) runs against Parakeet, with leading and trailing padding as separate conditions, before this rule counts as measured.
5. **Decode** with the offline recognizer, greedy by default. With `asr.parakeet.decoding` `beam`, decoding uses `modified_beam_search` and the call's current decode list is passed to `createStream(list)` when the model is a transducer. A word added mid-call is in the list from the next segment. Emit a `seg` to main, which assigns `seq`, computes wall time and appends.
6. **Provisional line.** While a segment is open, re-decode it every second (bounded by the 12 s window). It is published with a 3 s expiry and is never written to the log. On x64 the re-decode waits for new speech (`provisionalInPause` is off there): one that falls in the pause that closes the segment would hold the decoder while the committed line waits for it, and on four cores that cost 0.4 s a line.
7. **Echo.** Without headphones the mic hears the call. A mic segment whose text overlaps a call segment within ±1.5 s by at least 70 % of tokens is marked `echo: true` (kept in the log, hidden from views and packs). Real echo cancellation on the recognizer's copy of the mic (`webrtc-audio-processing`, BSD-3) is an M2 option.
7. **Echo.** Without headphones the mic hears the call. A mic segment whose text overlaps the call segments within ±1.5 s, taken together, by at least 70 % of tokens is marked `echo: true` (kept in the log, hidden from views and packs). So that a read-back of a number or a name stays the user's, the mic segment needs at least 3 tokens and the call segment must start first; the rule is one function (`src/core/log/echo.ts`) used live, whichever channel commits a line, and by the final pass. Real echo cancellation on the recognizer's copy of the mic (`webrtc-audio-processing`, BSD-3) is an M2 option.

Targets, measured in M0: committed line within 1.5 s of the end of an utterance on Apple silicon and within 2.5 s on 4-core x64, for lines after the first 10 s of a cold call (ROADMAP G6); first provisional words within 1.5 s of speech; both channels together under 25 % of real time on Apple Silicon and under 50 % on a 4-core x64 laptop.

Expand Down
2 changes: 1 addition & 1 deletion docs/TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ The suites that exist, one line each, so a reader knows where a new test goes.

- **Log and fold** (`events`, `log-io`, `fold`, `clock`): append-only writes, torn last lines, the one sort order, revisions win, wall-clock rendering. The fold is the only reader of raw events, so its tests cover every view.
- **Call machine** (`call-machine`, `capture-*`): start answers 201 only once audio is written, bounded teardown, restarts in the same folder, crash reconciliation, health verdicts.
- **Speech** (`asr-live`, `asr-final`, `asr-speakers`, `asr-models`, `first-run`): padding of short spans, the whole-timeline rule of the final pass, echo marking, cluster naming, checksummed resumable downloads.
- **Speech** (`asr-live`, `asr-final`, `echo`, `asr-speakers`, `asr-models`, `first-run`): padding of short spans, the whole-timeline rule of the final pass, echo marking (the rule's bounds in `echo`, its writers on the live path and in the final pass), cluster naming, checksummed resumable downloads.
- **Query** (`query`, `query-traps`, `bench-pack`, `eval/replay`): packs stay inside the budget, recency and BM25 find the answer, citations are wall-clock, pack p95 under 50 ms on a synthetic 3-hour call.
- **Notes and providers** (`notes`, `memo`, `reenhance`, `llm-harness`, `llm-http`, `session-reuse`): citations are checked, the harness is spawned with no tools, usage-limit exits fall back to excerpts, sessions are reused.
- **Vocabulary** (`vocab-*`, `bpe-vocab`): decode lists, read-time correction with raw text kept, approval before anything is learned.
Expand Down
1 change: 1 addition & 0 deletions docs/TRAPS.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ Traps that belong to the user's own downstream pipelines (archive formats, word-
- **A live line that starts inside the line before it** [LV-1] (M1). Given `asr.segmentPause` under 0.7 s on the Parakeet live path, a line closed `postRoll` (0.2 s) after its last speech, and the next one opened `preRoll` (0.5 s) before its first, so the two overlapped by 0.7 s minus the pause: 0.4 s at a 0.3 s pause, and the next line could start with the end of the last word of the line before. The next line must never start before the last closed line ended, on the same channel and part. Test: `tests/asr-live.test.ts` `[LV-1]`, a 0.3 s pause (the second line starts where the first ended) with the default pause as the control (its pre-roll is whole).
- **A live decode that throws shifts the rest of the part** [LV-2] (M1). Given a one-off native error in a Parakeet live decode, the error left the channel's unfinished VAD window where it was before the audio message, so every later line on that channel was cut from audio one message too early, and the line was lost with only an app-log error. A decode that throws must lose only its own line, with a warning that names its span (the final pass decodes that audio again), and the channel's timeline must stay with its audio whatever a decode throws, a provisional re-decode included. Test: `tests/asr-live.test.ts` `[LV-2]`, a recognizer that throws once, on a closing decode and on a provisional one: the other lines have the times of a run with no error.
- **A failing final pass that holds the start's catch-up** [FN-1] (M1). Given a final pass that fails by running out of its budget (Qwen on a machine with no GPU and a long call), the start's catch-up runs it again at every start, for its whole budget, and every call that ended later waited behind it. A call whose last pass failed must run after every call that has not had a pass, and must still run, so audio that becomes readable later still gets its final layer. Test: `tests/asr-final-catchup.e2e.test.ts` `[FN-1]`, a failed call that the start meets first and a call with no pass: the call with no pass starts first and both end `done`.
- **Mic hears the call** [ECHO-1] (M1). Given a call on laptop speakers, the mic hears the remote side, so each remote sentence became a `you` line as well as a call line: the user was credited with the other side's words in every view, pack, share and agent answer. The schema and the fold already hid a line marked `echo`, but nothing wrote the mark. A mic line must be marked `echo: true`, in a revision of its own on the live path (checked when either channel writes a line, since a call line can close after the mic line that heard it) and on revision 1 in the final pass, by one rule: at least 3 words, the call line started first, the mic line starts no later than 1.5 s after the call line ends, and at least 70 % of its words are in the call line. The call lines near a mic line count together, since the channels are not cut in the same places: the final pass cuts the call channel at every turn between two remote speakers, and a pause can close a call line where the mic's copy runs on; one call line at a time left such a mic line visible after the final pass while the live path had hidden it. The rule is narrow so a read-back of a number or a name stays the user's. The line stays in the log; only views leave it out. Tests: `tests/echo.test.ts` `[ECHO-1]`, each bound with the case just inside and just outside it; `tests/asr-live.test.ts` and `tests/asr-final.test.ts` `[ECHO-1]`, the same speech on both channels (the mic line marked and out of the view, with either line closing first, and with the call cut in two where the mic is not) and different speech as the positive control (no mark).

## Event log and clocks

Expand Down
67 changes: 67 additions & 0 deletions src/core/log/echo.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
/**
* Echo marking (DESIGN 3.1 step 7): without headphones the mic hears the call, so a remote
* sentence comes back as a `you` line too. A mic line that repeats a call line is marked
* `echo: true`: kept in the log, hidden from views and packs. One rule for the live path and the
* final pass.
*
* The rule must not hide the user's own speech, so it is narrow: a person often reads back a
* number or a name, which is short and said after the other side finished. A mic line is echo only
* when it has at least `ECHO_MIN_TOKENS` words, the call line started first (or at the same time),
* the mic line starts no later than `ECHO_WINDOW_MS` after the call line ends, and at least
* `ECHO_MIN_SHARE` of the mic line's words are in the call line.
*/

import { tokenize } from "../vocab/correct.ts";

/** How far after the call line ends a mic line may start and still be its echo. */
export const ECHO_WINDOW_MS = 1500;
/** The share of the mic line's words that must be in the call line, in tenths: 7 is 70 %. */
export const ECHO_MIN_SHARE_TENTHS = 7;
/** A shorter mic line is never echo: two words read back are the user's. */
export const ECHO_MIN_TOKENS = 3;

/** What the rule reads of a line: its wall times and its text (null when retracted). */
export interface EchoLine {
w0: number;
w1: number;
text: string | null;
}

/** Whether the mic line `mic` is the mic hearing the call line `call`. */
export function isEcho(mic: EchoLine, call: EchoLine): boolean {
if (call.w0 > mic.w0) return false;
if (mic.w0 > call.w1 + ECHO_WINDOW_MS) return false;
const said = tokenize(mic.text ?? "").map((t) => t.folded);
if (said.length < ECHO_MIN_TOKENS) return false;
const pool = new Map<string, number>();
for (const t of tokenize(call.text ?? "")) pool.set(t.folded, (pool.get(t.folded) ?? 0) + 1);
let found = 0;
for (const w of said) {
const n = pool.get(w) ?? 0;
if (n === 0) continue;
found++;
pool.set(w, n - 1);
}
return found * 10 >= said.length * ECHO_MIN_SHARE_TENTHS;
}

/**
* Whether the mic line `mic` is the mic hearing the call lines near it, taken together. The two
* channels are not cut in the same places (the final pass cuts the call channel at every turn
* between two remote speakers, and a pause can close a call line where the mic's copy runs on), so
* one mic line can hold the echo of several call lines. The call lines that may have been heard are
* the ones that start before the mic line ends and end at most `ECHO_WINDOW_MS` before it starts;
* they are read as one line, from the first one's start to the last end.
*/
export function isEchoOfCalls(mic: EchoLine, calls: readonly EchoLine[]): boolean {
const near = calls
.filter((c) => c.w1 + ECHO_WINDOW_MS >= mic.w0 && c.w0 <= mic.w1)
.sort((a, b) => a.w0 - b.w0);
const first = near[0];
if (!first) return false;
return isEcho(mic, {
w0: first.w0,
w1: Math.max(...near.map((c) => c.w1)),
text: near.map((c) => c.text ?? "").join(" "),
});
}
7 changes: 7 additions & 0 deletions src/main/asr/finalize-worker.ts
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@
import { closeSync, openSync, readSync } from "node:fs";
import { join } from "node:path";
import { workerData } from "node:worker_threads";
import { type EchoLine, isEchoOfCalls } from "../../core/log/echo.ts";
import type { Channel, EventDraft, LogEvent } from "../../core/log/events.ts";
import { type CallView, fold } from "../../core/log/fold.ts";
import { readLog } from "../../core/log/reader.ts";
Expand Down Expand Up @@ -717,6 +718,12 @@ export async function runFinalPass(
};
const closePart = (p: number, lines: Line[]) => {
lines.sort((a, b) => (a.w0 as number) - (b.w0 as number) || (a.ch === "mic" ? -1 : 1));
// Echo (DESIGN 3.1 step 7), by the live path's rule: a mic line that repeats the call lines
// near it. Taken together, since only the call channel is cut at diarized turns.
const calls = lines.filter((l) => l.ch === "call") as EchoLine[];
for (const l of lines) {
if (l.ch === "mic" && isEchoOfCalls(l as EchoLine, calls)) l.echo = true;
}
for (const l of lines) {
layer.push({ ...l, id: `f${String(nextFinal).padStart(6, "0")}` } as EventDraft);
nextFinal++;
Expand Down
30 changes: 30 additions & 0 deletions src/main/asr/live-worker.ts
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@
*/

import { workerData } from "node:worker_threads";
import { isEchoOfCalls } from "../../core/log/echo.ts";
import type { Channel, EventDraft, LogEvent } from "../../core/log/events.ts";
import type { CallView } from "../../core/log/fold.ts";
import { CHANNELS, type Clock, realClock, withDeadline } from "../capture/engine.ts";
Expand Down Expand Up @@ -1882,8 +1883,13 @@ interface HostCall {
live?: LiveChoice | null;
/** The call's second pass, asked once with the engine; null for none. */
upgrade?: HostUpgrade | null;
/** The ids of the lines written in the last `ECHO_RECENT_MS`, for the echo check. */
recent: string[];
}

/** How far back a written line is kept to be compared with the other channel's next lines. */
const ECHO_RECENT_MS = 10_000;

/** Worker answers that end a use of the models (`LiveAsr.releaseIdle`). */
const USE_ENDS: ReadonlySet<FromWorker["type"]> = new Set([
"decoded",
Expand Down Expand Up @@ -2430,6 +2436,7 @@ export class LiveAsr {
parts: new Map(),
lagLevel: 0,
superseded: false,
recent: [],
};
this.calls.set(callId, c);
this.current = c;
Expand Down Expand Up @@ -2718,6 +2725,29 @@ export class LiveAsr {
c.access.view.provisional.commit(m.ch, w1);
if (c.upgrade && m.key !== undefined) c.upgrade.keys.set(m.key, id);
if (m.held && m.key !== undefined) c.held.set(m.key, id);
this.markEcho(c, id);
}

/**
* Echo (DESIGN 3.1 step 7): a mic line that repeats the call lines near it (`isEchoOfCalls`)
* gets a revision carrying `echo: true` alone. Checked whichever channel just wrote, since a call
* line can close after the mic line that heard it. Never writes `echo: false`, and leaves a line
* a person touched as it is.
*/
private markEcho(c: HostCall, id: string): void {
const view = c.access.view;
const s = view.segment(id);
if (!s) return;
c.recent = c.recent.filter((r) => (view.segment(r)?.w1 ?? 0) >= s.w0 - ECHO_RECENT_MS);
c.recent.push(id);
const lines = c.recent.flatMap((r) => view.segment(r) ?? []);
const calls = lines.filter((l) => l.ch === "call");
const mics = s.ch === "mic" ? [s] : lines.filter((l) => l.ch === "mic");
for (const mic of mics) {
if (mic.echo || mic.revisions.some((r) => r.by !== undefined)) continue;
if (!isEchoOfCalls(mic, calls)) continue;
c.access.record({ type: "seg", id: mic.id, rev: mic.rev + 1, echo: true });
}
}

/**
Expand Down
49 changes: 49 additions & 0 deletions tests/asr-final.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,55 @@ describe("[T1.9, T4.19] Short spans lose their words: the final path pads too",
});
});

describe("[ECHO-1] Mic hears the call: the final pass marks it too", () => {
const remote = ["we", "should", "move", "the", "build", "to", "new", "box"];
const part = (micWords: string[]) => ({
1: {
mic: concat(silence(1.1), speak(micWords), silence(2)),
call: concat(silence(1), speak(remote, { voice: 3 }), silence(2.1)),
},
});

test("the mic's copy of a call line is written with echo and leaves the final view", async () => {
const r = await run(callLog([1]), part(remote));
const byCh = [...r.segs].sort((a, b) => (a.ch as string).localeCompare(b.ch as string));
expect(byCh.map((s) => [s.ch, s.text, s.echo])).toEqual([
["call", remote.join(" "), undefined],
["mic", remote.join(" "), true],
]);
const view = fold(
callLog([1], (b) => {
for (const s of r.segs) b.add(s as EventDraft);
}),
);
expect(view.lines("final").map((l) => l.ch)).toEqual(["call"]);
});

test("positive control: the user's own words at the same time are not echo", async () => {
const r = await run(callLog([1]), part(["hello", "world", "thanks", "meeting", "today"]));
expect(r.segs.map((s) => s.echo)).toEqual([undefined, undefined]);
});

test("two remote speakers: the call is cut at the turn, the mic is not, and the mic line is still echo", async () => {
const a = ["we", "should", "move", "the", "build"];
const b = ["to", "new", "box", "thanks", "meeting"];
const r = await run(callLog([1]), {
1: {
mic: concat(silence(1.1), speak([...a, ...b]), silence(2)),
call: concat(silence(1), speak(a, { voice: 1 }), speak(b, { voice: 4 }), silence(2.1)),
},
});
// The case the test is about: two call lines, one mic line.
expect(r.segs.filter((s) => s.ch === "call").map((s) => s.text)).toEqual([
a.join(" "),
b.join(" "),
]);
expect(r.segs.filter((s) => s.ch === "mic").map((s) => [s.text, s.echo])).toEqual([
[[...a, ...b].join(" "), true],
]);
});
});

describe("speakers across the call", () => {
test("diarization runs over all parts at once: one voice has one label in every part", async () => {
const part = (v: number, w: string[]) => ({
Expand Down
Loading
Loading