Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 28 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -416,16 +416,34 @@ correctly, and proved on a GPU only on real hardware.

### Adverts a host inserts as it serves

Some podcasts insert adverts into the audio as it is downloaded, so the file is
different every time. SelfPod can spot these by downloading one episode a second
time, a day later, and comparing: whatever changed cannot be the theme tune, so it
is an advert by construction.

This is done sparingly and on purpose. A second download counts again in the
publisher's listener figures, so SelfPod only does it for episodes whose audio shows
positive signs of having been stitched together, never sooner than a day, and only a
couple at a time. It will not disguise itself to defeat a host's caching — that
would be interfering with the publisher's measurement, which is not SelfPod's to do.
Some hosts — Simplecast, Acast and others — insert adverts into the audio as it is
downloaded, picked by the listener's country and by chance. A French show fetched from
the UK carries English adverts; from Germany, German ones. So SelfPod does not look for
a language or a country. It looks for what every inserted advert has in common: the
host splices in a file it already has, so **the same advert is the same bytes in every
episode it lands in** — while the programme, encoded afresh each time, never is.

- **Copied audio.** A stretch whose frames are byte for byte the same in two or more
episodes is cut. RMC's sponsor opener and closer are found this way in every episode.
- **Breaks between markers.** Hosts like Acast stitch a short clip at both edges of
every break. SelfPod finds those markers and cuts what lies between them — even
adverts it has never heard before. Shown as *Pre-roll*, *Mid-roll* and *Post-roll*
breaks on the Adverts page, each with *Restore here*.
- **A second download, a day later.** For a feed whose host inserts, every new episode
is downloaded once more and compared, to learn adverts that rotate. SelfPod turns
this on by itself once it has seen the host insert; the subscription form has the
switch too. Each second download counts again in the publisher's figures.

In automatic mode these are cut without asking, with two exceptions that are offered
instead: anything over two and a half minutes, and a stretch only two episodes share in
the middle (a "best of" could share real programme that way). SelfPod never disguises
itself to see different adverts — a proxy would only swap which adverts arrive.

To see what it finds in episodes you downloaded yourself:

```bash
node scripts/check-copied-audio.mjs path/to/folder-of-mp3s
```

Only for `.mp3` at present. Other formats are published untouched.

Expand Down
110 changes: 94 additions & 16 deletions doc/selfpod-spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -1052,22 +1052,86 @@ per request. It is the stronger signal, and the only one that identifies
an advert rather than something that merely repeats: a theme tune is in
both copies, so it cannot be what differs between them.

### 19.3 The cost of looking twice, and why it is rationed

A second download is a second IAB-countable listen, so it doubles the
publisher's figures for an episode taken once. Hosts also cache the
stitch per listener, keyed on requesting address and user agent, so two
requests from one container seconds apart are the same listener by
construction and would hit that cache. Defeating it would need a rotating
user agent or egress address; that is deliberate evasion of the
publisher's measurement and SelfPod will not do it.

So a second download happens only when the first file carries a positive
signal — a sample-rate or channel-mode change part-way through, or a Xing
header disagreeing with the frames present — never sooner than a day, a
couple per tick across all subscriptions, and charged to the same daily
byte budget. The second copy is deleted; the file on the share is the one
the owner has.
### 19.3 Audio the host inserts as it serves

Hosts such as Simplecast and Acast stitch adverts into the MP3 as each
request is served, chosen by the requester's country and by chance. A
French show fetched from the UK carries English adverts; fetched from
Germany, German ones; fetched from France, French ones. So nothing about
an inserted advert's language or content identifies it, and changing
where SelfPod asks from (a proxy, another address) would only change
which adverts arrive. The evidence has to be something every inserted
advert has, whatever it says.

**The same bytes.** A host inserts by concatenating MP3 frames from a
file it already has; it does not re-encode the episode around them. So
an advert served to two episodes is *byte for byte* the same frames in
both, while the programme — mastered and encoded once per episode —
never shares a frame with another episode, its theme tune included.
Frames copied between episodes are therefore audio the host spliced in
(`src/lib/copied-audio.js`). Measured on 2026-10-28 from the UK:

- RMC on Simplecast, 30 episodes (three of them best-ofs): a 13.9 s
opener and a 10.1 s closer, the same frames in 30 of 30; nothing else.
The opener is the sponsor tag *and* the "Vous écoutez RMC" ident, one
file; the owner chose to cut all of it.
- Le Rendez-vous Tech on Acast, 6 episodes: UK adverts in English in
pre-, mid- and post-rolls, and a 2.2 s marker stitched at both edges of
every break.
- Laurent Gerra (RTL) on Audiomeans: nothing copied, nothing inserted.

**Breaks between markers.** A short copied clip found more than once in
one episode is a host's slot marker. What lies between two markers, or
between one and the edge of the file, and is no longer than two and a
half minutes, is a break — cut whether or not any advert in it was ever
heard elsewhere. Never when the breaks would add up to half the episode.
Breaks are catalogued one row per slot (pre-roll, mid-roll *n*,
post-roll), so a row is in each episode once and *Restore here* restores
one break.

**What is cut without asking**, and why the guards differ from §19.5's:
a copied stretch carries none of the ambiguity of a merely repeated one,
so no position guard and no fifteen-second floor (the RMC opener sits at
0:00 and its closer is ten seconds). What remains: two and a half minutes
at most; and a copy in only two episodes, mid-file, is offered rather
than cut — a best-of assembled from an earlier episode's file without
re-encoding would share programme exactly so. An advert reaches a third
episode soon enough, or sits where hosts put them, at the very start or
end of the file.

**One row per stretch.** The acoustic search (§19.2) hears inserted audio
too; it leaves alone what an inserted row explains, but never takes the
occurrences of a row that is already cutting. Audio another approved row
already cuts is not offered again.

**Frame hashes are kept** beside each fingerprint (`{episode}.h1.frames`
in the fingerprint folder) so this costs no decoding. An episode
fingerprinted before they were kept is read once, without decoding, and
only if its bytes are still the ones its fingerprint describes.

**Looking twice.** Comparing two downloads of one episode finds what a
host changes from day to day — the only way to learn an advert served to
one episode only, outside any marked break. A second download is a
second IAB-countable listen, so it is rationed: made when the first file
carries a stitch signal (a sample-rate or channel-mode change part-way
through, a Xing header that disagrees with the frames present), or when
the subscription is known to insert — set by SelfPod the first time it
finds copied audio in the show or a second download comes back
different, or by the owner on the subscription form. Never sooner than a
day, a couple per tick across all subscriptions, charged to the same
daily byte budget; the second copy is deleted.

Hosts cache the stitch per listener, keyed on address and user agent, so
a second request seconds later is the same listener by construction
(three downloads of one RMC episode, minutes apart, two user agents:
byte-identical). Defeating that would need a rotating user agent or
egress address. That is deliberate evasion of the publisher's
measurement, and it would not help: it changes which adverts arrive,
not whether they do. SelfPod asks again, later, as itself.

**What none of this finds:** an advert inserted into one episode only,
outside any marked break, served identically to both downloads. The words
(§19.6) and a taught boundary remain for that.

### 19.4 Cutting

Expand Down Expand Up @@ -1297,6 +1361,20 @@ point at the jingle by hand. Confirming a jingle does not guess where a *differe
kind of ident sits partway into an episode; it only ever cuts from 0:00 to where the
sound was heard.

**Not confirmed by itself when it starts after the host's audio ends.** The proposal
is the earliest sound every recent episode shares, and that need not be a station
ident. On Le Rendez-vous Tech (Acast, 2026-10-28) one sponsored episode carried no
adverts and so no break marker, which left the show's own theme as that sound — after
the host's spoken intro. Confirmed automatically, 1.10 cut the intro from every
episode, up to 1:33. Now, in any recent episode where the host's inserted audio
(§19.3: copied frames, or a break between markers) starts before the jingle but ends
more than three seconds before it, the jingle is not confirmed automatically: what
lies between is the show's own. It stays a proposal the owner can still confirm, and
the breaks are cut regardless. A jingle SelfPod confirmed by itself before this rule is
taken back the same way, with its cut; one the owner confirmed, or a taught boundary
linked, is never touched. RMC is unaffected: its "Vous écoutez RMC" ident is inside the
inserted opener, so the jingle starts before the inserted audio ends.

### 19.8 What a cut is, putting one back, and when SelfPod looks

**Added in 1.9.0.**
Expand Down
60 changes: 60 additions & 0 deletions scripts/check-copied-audio.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
#!/usr/bin/env node
/**
* What the copied-audio search finds in a folder of real episodes.
*
* node scripts/check-copied-audio.mjs <folder-of-mp3s> [--json]
*
* Runs `findCopiedAudio` exactly as the detector does, on files you downloaded from the
* host yourself (not SelfPod's served copies, which are already cut). Prints every
* stretch found in two or more episodes, which of them are slot markers, and the breaks
* between markers — so a change to the search can be checked against real audio before
* it is trusted with anyone's feed.
*/
import { readdir, readFile } from 'node:fs/promises';
import { join } from 'node:path';
import { findCopiedAudio } from '../src/lib/copied-audio.js';
import { frameProfile } from '../src/lib/mp3-frames.js';

const args = process.argv.slice(2);
const folder = args.find((arg) => !arg.startsWith('--'));
if (!folder) {
console.error('usage: node scripts/check-copied-audio.mjs <folder-of-mp3s> [--json]');
process.exit(2);
}

const names = (await readdir(folder)).filter((name) => name.toLowerCase().endsWith('.mp3')).sort();
const episodes = [];
for (const name of names) {
const profile = frameProfile(await readFile(join(folder, name)));
if (!profile?.frameCount) {
console.warn(`${name}: no MP3 frames, skipped`);
continue;
}
episodes.push({ id: name, hashes: profile.hashes, frameMs: profile.durationMs / profile.frameCount, durationMs: profile.durationMs });
}

const started = Date.now();
const result = findCopiedAudio(episodes);
const elapsed = Date.now() - started;

if (args.includes('--json')) {
console.log(JSON.stringify(result, null, 2));
process.exit(0);
}

const seconds = (ms) => (ms / 1000).toFixed(1);
const durations = Object.fromEntries(episodes.map((episode) => [episode.id, episode.durationMs]));
const where = (o) => {
const tail = durations[o.episodeId] - o.endMs < 1000 ? ' (to the end)' : '';
return `${o.episodeId}@${seconds(o.startMs)}–${seconds(o.endMs)}${tail}`;
};

console.log(`${episodes.length} episodes, searched in ${elapsed} ms`);
console.log(`\nCopied stretches: ${result.copied.length}`);
for (const row of result.copied) {
const marker = result.markers.includes(row.signature) ? ' [marker]' : '';
console.log(` ${seconds(row.durationMs)} s in ${row.episodeCount} episodes, ${row.occurrenceCount} times${marker}`);
for (const o of row.occurrences) console.log(` ${where(o)}`);
}
console.log(`\nBreaks between markers: ${result.blocks.length}`);
for (const block of result.blocks) console.log(` ${where(block)} (${seconds(block.endMs - block.startMs)} s)`);
3 changes: 3 additions & 0 deletions src/constants.js
Original file line number Diff line number Diff line change
Expand Up @@ -509,6 +509,9 @@ export const SEGMENT_KINDS = Object.freeze({
REPEATED_AUDIO: 'repeated_audio',
DIFF: 'diff',
TAUGHT_RANGE: 'taught_range',
// Audio the host spliced in as it served the file: the same MP3 frames in two or
// more episodes, or a break between the slot markers a host stitches (copied-audio.js).
INSERTED: 'inserted',
});

/** The `source` 1.8 would have written for a kind — kept so a rollback can read rows. */
Expand Down
123 changes: 123 additions & 0 deletions src/db/migrations/014-inserted-audio.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
-- Audio a host inserts as it serves, found by its MP3 frames (spec §19.3).
--
-- A new kind of catalogue row, `inserted`: a stretch whose frames are byte for byte the
-- same in two or more episodes (a file the host spliced in), or a break between the slot
-- markers a host stitches around one. And a flag on a subscription saying its host is
-- known to insert, so every new episode of it is downloaded a second time.
--
-- Additive in effect, not in form, exactly as 009 was. `ad_segments.kind` is a CHECK
-- constraint and SQLite cannot widen one in place, so the catalogue and the two tables
-- that point at it are rebuilt with the same rows. The migration runner keeps foreign
-- keys ON inside its transaction, so the order matters: the replacements are built and
-- filled first, the old children are dropped while nothing references them, the old
-- parent only once it has no children left for an implicit DELETE to cascade into, and
-- the renames come last — renaming the parent is what rewrites the new children's
-- foreign keys to point at `ad_segments`.
--
-- An older image run against this schema reads an `inserted` row as one more row it
-- does not have a special case for, and cuts it like any other approved row. Rolling
-- back stays non-destructive, the property every migration since 007 preserves.

CREATE TABLE ad_segments_v3 (
id TEXT PRIMARY KEY,
show_id TEXT NOT NULL REFERENCES shows(id) ON DELETE CASCADE,
signature TEXT NOT NULL,
-- 'transcript' is found by what was said. For a repeated read it is the hash of the
-- words; for a boundary the owner taught it is `marker:{id}`.
source TEXT NOT NULL CHECK (source IN ('corpus', 'diff', 'transcript')),
status TEXT NOT NULL DEFAULT 'candidate'
CHECK (status IN ('candidate', 'approved', 'rejected')),
auto_approved INTEGER NOT NULL DEFAULT 0,
hold_reason TEXT,
duration_ms INTEGER NOT NULL,
episode_count INTEGER NOT NULL DEFAULT 0,
occurrence_count INTEGER NOT NULL DEFAULT 0,
exemplar_episode_id TEXT REFERENCES episodes(id) ON DELETE SET NULL,
exemplar_start_ms INTEGER,
exemplar_end_ms INTEGER,
first_seen_at TEXT NOT NULL,
decided_at TEXT,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL,

-- What was said, for segments found by — or later matched to — the words.
text TEXT,
raw_text TEXT,
cue_score REAL,
cues TEXT,
language TEXT,

-- What kind of thing the row is (011). `inserted` is new here: audio the host
-- spliced in, found by its frames rather than by how it sounds.
kind TEXT NOT NULL DEFAULT 'repeated_audio'
CHECK (kind IN ('jingle', 'boundary_words', 'remembered_words', 'repeated_words',
'repeated_audio', 'diff', 'taught_range', 'inserted')),
marker_id TEXT REFERENCES ad_markers(id) ON DELETE CASCADE,
anchor_id TEXT REFERENCES ad_anchors(id) ON DELETE CASCADE,
first_cut_at TEXT,

UNIQUE(show_id, signature)
);

CREATE TABLE ad_segment_occurrences_v3 (
id INTEGER PRIMARY KEY AUTOINCREMENT,
segment_id TEXT NOT NULL REFERENCES ad_segments_v3(id) ON DELETE CASCADE,
episode_id TEXT NOT NULL REFERENCES episodes(id) ON DELETE CASCADE,
start_frame INTEGER NOT NULL,
end_frame INTEGER NOT NULL,
start_ms INTEGER NOT NULL,
end_ms INTEGER NOT NULL,
UNIQUE(segment_id, episode_id, start_frame)
);

CREATE TABLE ad_cut_overrides_v3 (
id TEXT PRIMARY KEY,
episode_id TEXT NOT NULL REFERENCES episodes(id) ON DELETE CASCADE,
segment_id TEXT REFERENCES ad_segments_v3(id) ON DELETE SET NULL,
start_ms INTEGER NOT NULL,
end_ms INTEGER NOT NULL CHECK (end_ms > start_ms),
created_at TEXT NOT NULL
);

INSERT INTO ad_segments_v3
(id, show_id, signature, source, status, auto_approved, hold_reason, duration_ms,
episode_count, occurrence_count, exemplar_episode_id, exemplar_start_ms, exemplar_end_ms,
first_seen_at, decided_at, created_at, updated_at, text, raw_text, cue_score, cues, language,
kind, marker_id, anchor_id, first_cut_at)
SELECT
id, show_id, signature, source, status, auto_approved, hold_reason, duration_ms,
episode_count, occurrence_count, exemplar_episode_id, exemplar_start_ms, exemplar_end_ms,
first_seen_at, decided_at, created_at, updated_at, text, raw_text, cue_score, cues, language,
kind, marker_id, anchor_id, first_cut_at
FROM ad_segments;

INSERT INTO ad_segment_occurrences_v3
(id, segment_id, episode_id, start_frame, end_frame, start_ms, end_ms)
SELECT id, segment_id, episode_id, start_frame, end_frame, start_ms, end_ms
FROM ad_segment_occurrences;

INSERT INTO ad_cut_overrides_v3 (id, episode_id, segment_id, start_ms, end_ms, created_at)
SELECT id, episode_id, segment_id, start_ms, end_ms, created_at
FROM ad_cut_overrides;

DROP TABLE ad_cut_overrides;
DROP TABLE ad_segment_occurrences;
DROP TABLE ad_segments;

ALTER TABLE ad_segments_v3 RENAME TO ad_segments;
ALTER TABLE ad_segment_occurrences_v3 RENAME TO ad_segment_occurrences;
ALTER TABLE ad_cut_overrides_v3 RENAME TO ad_cut_overrides;

CREATE INDEX idx_ad_segments_show ON ad_segments(show_id, status);
CREATE INDEX idx_ad_segments_kind ON ad_segments(show_id, kind);
CREATE INDEX idx_ad_segments_marker ON ad_segments(marker_id);
CREATE INDEX idx_ad_segments_anchor ON ad_segments(anchor_id);
CREATE INDEX idx_ad_occurrences_episode ON ad_segment_occurrences(episode_id);
CREATE INDEX idx_ad_cut_overrides_episode ON ad_cut_overrides(episode_id);
CREATE INDEX idx_ad_cut_overrides_segment ON ad_cut_overrides(segment_id);

-- The host of this feed has been seen inserting audio — set by SelfPod the first time
-- it finds some, or by the owner — so every new episode is downloaded a second time a
-- day later to learn the adverts that rotate (spec §19.3). Never a disguise: the same
-- address and the same user agent as the first download.
ALTER TABLE feed_subscriptions ADD COLUMN inserts_audio INTEGER NOT NULL DEFAULT 0;
Loading
Loading