diff --git a/CHANGELOG.md b/CHANGELOG.md index cc7a85b..dc03f84 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,44 @@ Updating is changing the image tag and redeploying. The database migrates itself forward on start, and no release so far has needed anything else — where a release changes what your listeners see, it says so. +## Unreleased + +### Fixed — the statistics said more than had happened + +- **A HEAD request was counted as a download of the whole file.** Overcast, Apple's crawler + and every feed validator ask for an episode's headers before fetching it, and SelfPod + recorded each of those as a completed download carrying the file's full size as "sent" — + while nothing had left the server. Two header checks on a 29 MB episode put 56 MB on the + Audio served card. A HEAD is no longer recorded at all. +- **A 304 was counted as a download.** A browser or an iPhone revalidating a cached episode + is answered "not modified" with no body; that reply was a download in every figure. It is + now in the log as a 304 that sent nothing, and in no total. +- **A download the app gave up on was counted as one it received.** The log already knew the + app had disconnected — it said so in the row — but the row kept its green 200, the + Downloads card counted it, and the CSV called it "ok". It is now shown as *partial*, + exported as `partial`, and counted nowhere. A stream the player stopped part-way through + is still a stream: that is how players behave. +- **Apple Podcasts never registered a download.** It fetches every episode with + `Range: bytes=0-` and is answered with the whole file; the presence of the range made it a + stream. A range from byte zero that delivers the entire file is now a download, whichever + app asked. The two-byte probe an iPhone sends first stays a stream. +- **"AppleCoreMedia" is no longer credited to Apple Podcasts.** It is the system player on an + iPhone, iPad or Mac, and it speaks for whichever app is streaming through it — Overcast, + Castro and Safari included. The chart now calls it "Apple media player", and the page says + what that means. +- **The Audio served card compared bytes as a bare number** ("+58,963,151 vs the previous 30 + days"). It now says "+56.2 MB". +- A resumed download answered with the whole file (an address from an earlier cut) lost the + range it had asked for before the row was written, so the log could not show why. The range + is kept. + +### Changed + +- Dependencies brought up to patched releases: Fastify 5.12.5 (seven advisories, including an + authentication bypass through malformed URLs and X-Forwarded header spoofing), sharp 0.35.5 + (libheif and librsvg), music-metadata 11.16.1 (crafted MP4, EBML and DSF files could crash + the scanner — and the scanner reads whatever is dropped on the share). `npm audit` is clean. + ## 1.9.1 — 2026-09-16 ### Fixed diff --git a/README.md b/README.md index 2597315..353c88b 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,8 @@ regenerate button. and partial streams are counted separately, alongside a request-by-request log naming the app that asked (“Pocket Casts”, “Apple Podcasts”). Failed requests are shown first, so an episode that will not download in someone's podcast app - is visible here instead of only on their phone. No IP addresses are stored, and + is visible here instead of only on their phone, and a download the app gave up + on is listed as partial rather than counted. No IP addresses are stored, and your own visits to the admin interface are never counted. - **Tells you when something is wrong.** Every scan is logged in plain language — a permission problem names the exact path and the UID SelfPod runs as. You diff --git a/doc/audit-2026-10-09.md b/doc/audit-2026-10-09.md new file mode 100644 index 0000000..6a49f35 --- /dev/null +++ b/doc/audit-2026-10-09.md @@ -0,0 +1,184 @@ +# SelfPod 1.9.1 — full audit, 9 October 2026 + +Scope: statistics correctness, security, performance, UI/UX, dependencies. +Method: four independent code audits (one per area), plus live testing of the +dev server over a real socket (curl with Range/HEAD/conditional requests, +sqlite3 on the resulting database) and in a real browser at desktop and phone +widths. Every finding below was verified in code or reproduced live; items the +agents reported that did not hold up were dropped. `npm test`: 1304 pass, 0 fail. + +Severity key: **H** fix soon · **M** worth fixing · **L** tidy-up / optional. + +--- + +## 1. Statistics — are the numbers correct? + +The pipeline is well built (one row per response, own visits excluded on every +preview path, aggregates agree with each other, indexes cover every query plan, +365-day retention runs daily). But four things make the headline numbers wrong +in ordinary use. All reproduced against the dev server. + +| # | Sev | Finding | Evidence | +|---|-----|---------|----------| +| S1 | **H** | A `HEAD` request is counted as a completed **download** with the file's full size as bytes sent. Two HEADs on a 29 MB file made "Audio served" read 56 MB when ~30 MB left the server. Overcast, Apple's crawler and feed validators HEAD enclosures routinely. | Fastify exposes HEAD on every GET; `kind` is decided only by the `Range` header (`src/routes/public.js:619`) and `bytesSent` falls back to `Content-Length` (`public.js:70-72`). Nothing checks `request.method`. | +| S2 | **H** | A `304 Not Modified` on an episode is counted as a download. | Media route leaves `@fastify/send` validators on; every aggregate uses `status_code < 400` (`src/services/stats.js:247,272,302,348,418,441,508`). | +| S3 | **H** | A download the app **disconnected from mid-transfer** is logged as incomplete but still counted in Downloads, shown with a green 200 badge, and exported as `ok`. | `bytesSent` is `null` on abort (`public.js:70`), but `incomplete` requires `bytes_sent !== null` (`stats.js:637-642`); the existing test fakes `bytes_sent: 1000` so never exercises the real path. | +| S4 | **M** | Apple Podcasts always asks with `Range: bytes=0-`, so a complete Apple download is counted as a **stream**, never a download. Downloads are therefore undercounted per app, not uniformly. | `public.js:619`; the same route already classes `bytes=0-` as "from the start" at `:424`. IAB guidance treats `bytes=0-` as a download. | +| S5 | **M** | Day boundaries are an hour off on DST days in zones where the transition is within the UTC offset of midnight (Sydney, Auckland, Santiago…). Europe/London is unaffected. | `fromLocalInputValue` probes the zone offset at the wall-clock triple read as UTC (`src/lib/dates.js:128-131`). Tests only cover Europe/London and UTC. | +| S6 | **M** | Deleting an episode cascades away its whole request history, so show and instance totals shrink retroactively. | `media_access.episode_id … ON DELETE CASCADE` (migration 004). The `LEFT JOIN episodes` in `list()` suggests rows were meant to outlive the episode. | +| S7 | **M** | `AppleCoreMedia` is attributed to "Apple Podcasts", but it is the user agent of *any* iOS app streaming through AVFoundation (Overcast, Castro…). The "Which apps" chart over-credits Apple. | `stats.js:56-57`. | +| S8 | **L** | "vs the previous period" on the Audio served tile prints raw bytes (`+58,963,151`) instead of a size. | Stats page tile. | +| S9 | **L** | The comparison window is lopsided: current = N−1 full days + today's partial day, previous = N full days, so the delta is biased downward early in the day. | `src/lib/time-range.js:60-64`. | +| S10 | **L** | `/stats?showId=` filters only the access log; cards, chart and per-show table stay instance-wide. The failures card counts download/stream kinds only while "Failures only" counts every kind. | `src/web/routes/pages.js:741-772`, `stats.js:126,420`. | +| S11 | **L** | CSV is silently capped at 50 000 rows, newest first, while the page says it holds every row. | `pages.js:532`, `stats.js:96,619`. | +| S12 | **L** | A resume-after-re-cut request has its `Range` deleted before the kind/range are recorded, so the one case where the range explains a retry is logged as a plain download. | `public.js:583` runs before `:619`. | + +What is correct and should not be touched: own-visit exclusion (session cookie +on every path, including the editor player, cuts previews and the reachability +check); half-open day boundaries in the configured zone; instance = Σ shows = +Σ episodes; CSV uses the same query as the page; failures (≥ 400) excluded from +downloads/streams; missing-file 404s and post-stat 500s both recorded with a +plain-language reason; no integer division or NULL mistakes in the SQL. + +--- + +## 2. Security + +Threat model: the media folder is an SMB share others can write to; the NAS can +see the LAN; one admin account behind a tunnel. Accepted risks (plain HTTP on +the LAN, unrestricted egress) are not listed. + +| # | Sev | Finding | Evidence / exploit | +|---|-----|---------|--------------------| +| X1 | **M** | A decompression-bomb image on the share can OOM the container on every scan. No `sharp()` call sets `limitInputPixels`; the 40 Mpx constant exists but is never applied. A 16383×16383 solid WebP (a few hundred KB) dropped as `episode.webp`, or embedded in an MP3 tag under the 12 MB cap, decodes to ~1 GB on scan → OOM → restart → rescan → loop, with no banner. | `src/services/episode-art.js:58,77,82`, `src/services/covers.js:56,113,147,247`; `REMOTE_IMAGE_MAX_PIXELS` in `constants.js:356` unused. | +| X2 | **M** | The fingerprinter, transcriber and trimmer `readFile` the raw share path with no containment check and no size gate, while every *serving* route goes through `resolveContained()`. The scanner deliberately accepts symlinks. `ep.mp3 -> /dev/zero`, a FIFO, or a multi-GB file hangs or exhausts the heap on every scheduled pass once advert detection is on. | `src/services/ad-detect.js:114,147`, `transcriber.js:359-362`, `trimmer.js:185-188`; `scanner.js:354`. | +| X3 | **L** | Anonymous callers read full health-banner text via `/api/status` and the login page: show and episode titles, followed-feed hosts, data paths, UID/GID. Only the permission banners need to be visible before sign-in. | `src/routes/api/status.js:10-30`, `pages.js:89,109`. | +| X4 | **L** | Anyone holding the public URL can keep the admin locked out indefinitely: one wrong password every ~5 minutes resets the 300 s account-level backoff. (Per-source limiting is deliberately absent because of the tunnel; `MAX_ATTEMPTS_PER_WINDOW` is exported and unused.) Refusals return 401 rather than 429. | `src/plugins/auth.js:34,61-70,86-93`. Verified live: after 3 failures every attempt is refused until the window passes. | +| X5 | **L** | Changing the password in the UI does not invalidate other sessions (up to 30 days). The reset script does it right. | `auth.js:113-120` vs `scripts/reset-password.js:54`. | +| X6 | **L** | Supply chain: GitHub Actions by tag, base images without digests, whisper.cpp pinned by movable tag, `test` job has no `permissions:` block. | `.github/workflows/docker.yml`, `Dockerfile:10,45,52,82`. | +| X7 | **L** | Dependencies: `npm audit --omit=dev` reports 8 advisories. `sharp` ≤0.35.3 (libheif/librsvg, high), `music-metadata` 11.14 (crafted MP4/EBML/DSF crash the process — relevant because the scanner parses attacker-supplied tags), `js-yaml` via xmlbuilder2 (not reachable: no YAML is parsed), `ip-address` via @fastify/rate-limit (not used by the feed fetcher). | `npm audit` | +| X8 | info | `trustProxy: true` trusts every hop and compose publishes 8080 on all interfaces. Nothing security-relevant keys off forwarded headers today, so no exploit. | `src/app.js:33`, `docker-compose.yml:65`. | +| X9 | info | Session not regenerated on login. Verified not exploitable (`saveUninitialized: false`, no pre-login sid), but a one-liner for defence in depth. | `routes/api/auth.js:18`, `pages.js:115`. | + +Checked and sound: cookie flags, bcrypt 12 with constant-time dummy compare, +CSRF via `Sec-Fetch-Site` with Origin fallback on every mutating route and no +state-changing GET, `requireAdminApi` on every API route and one scoped +`requireAdminPage` hook on fragments, `resolveContained` on all four serving +routes, the guarded fetcher (scheme/credential/port rules, every DNS answer +classified, pinned lookup so no rebinding, per-hop redirect re-validation, +size/stall/throughput timers, closed error enum, host-only logging), the +reachability check (admin + CSRF + rate-limited, fixed target, HMAC-proved +body), XML escaping and `]]>` neutralisation, allow-listed dynamic SQL, +`execFile` with argument arrays only, CSP `script-src 'self'` enforced by a +test, no stack traces on 5xx, token redaction in logs, Docker drops to PUID +with `cap_drop: ALL` and `no-new-privileges`. + +--- + +## 3. Performance + +Measured by a benchmark importing the repo's own modules against a synthetic +1-hour MP3 on an M-series Mac (a NAS Celeron is 3–6× slower). + +| # | Sev | Finding | Evidence | +|---|-----|---------|----------| +| P1 | **H** | **Audio decoding and fingerprinting run on the HTTP thread.** 1 h of MP3 = 12.4 s of a fully blocked event loop (a 10 ms timer fired 2.1 s late). During that, every in-flight range request stalls, SSE heartbeats stop and `/health` does not answer; the Docker healthcheck times out at 4 s with 3 retries, so one long episode on a slow box can get the container restarted mid-pass. A fresh install fingerprints the whole library this way straight after the first scan. Only the two corpus *searches* were moved to the worker. | `src/lib/decode-audio.js:93-148` (no await), called from `ad-detect.js:182` and `transcriber.js:288`; `src/workers/audio-search.js` has only `repeatedAudio`/`headAnchors`. | +| P2 | **H** | The advert pass is not incremental and runs every rescan tick (5 min). `detectAnchors` loads **every** fingerprint into memory (~1.2 MB per hour of audio; 300 h ≈ 360 MB) and re-runs `locateAnchor` for every episode; `hearShow` parses every transcript and re-runs `locatePhrase` for every known read × every episode (1.7 s for 50×100). `episode_fingerprints.searched_at` and `ad_anchor_hits.checked_at` were added for exactly this and are written but never read. | `ad-detect.js:1596-1604,1678-1700,491-527,1319-1356`; `scheduler.js:132`; grep `searched_at` → only the upsert. | +| P3 | **M** | Whole-file buffering: trimmer holds ~2× file size (read + `Buffer.concat`); `sample.mp3` reads the **entire episode** off SMB and profiles it to serve a 10-second clip, on every play; the transcriber keeps the whole buffer alive across every whisper run. | `trimmer.js:188,201,208`; `routes/api/ad-segments.js:323-345`; `transcriber.js:362,395`. | +| P4 | **M** | No response compression for HTML/CSS/JS/JSON (only the feed is pre-compressed). First load ≈ 430 KB uncompressed; a 300-row episode table is hundreds of KB per render over the tunnel. | `src/app.js` registers no compress plugin. | +| P5 | **L** | Session `rolling: true` → one UPSERT per request including every `/assets/*` hit. | `src/plugins/session.js:84`. | +| P6 | **L** | SQLite hygiene: `PRAGMA optimize` never run; no `journal_size_limit`; hot statements re-prepared per call (`episodes.findByFilename`, `shows.isFolderRemoved`, `listSegments` N+1 inside `map`); four FK columns on the advert tables unindexed. | `src/db/index.js`, `episodes.js:98-110`, `shows.js:357`, `ad-detect.js:2139-2147`. | +| P7 | **L** | Boot: eager `sharp`/`xmlbuilder2`/`qrcode` imports (~0.8 s Mac), `chown -R node_modules` over 4 491 files every start, whisper probe loads the model at boot. In polling mode the watcher stats every file at ≤60 s *and* the scheduler rescans at the rescan interval — double the SMB stats on exactly the mount where it hurts. | `index.js:129,181,188`, `entrypoint.sh:81`, `watcher.js:102`. | +| P8 | **L** | Image ships both whisper models (250 MB) regardless of `WHISPER_MODEL`, plus `better-sqlite3/deps` (9.8 MB of SQLite source). | `Dockerfile:74-77`. | +| P9 | **L** | Dashboard: cover images are full 1400 px into a 250 px card (several MB per load over a tunnel); each show runs ~13 queries plus a segments N+1; "Rescan all" on N shows triggers N card fetches. | `show-card.eta:14`, `pages.js:230`. | + +Already done well: feed cached per show with TTL + event invalidation, ETag +and Last-Modified that never go backwards, pre-compressed gzip/brotli, weak +ETag comparison for Cloudflare; media streamed with Range/206/416 from +`@fastify/send`, `private` cache-control; scanner fast path on (name, size, +mtime), 2 MB identity hash, metadata only on changed bytes; WAL + NORMAL + +busy_timeout; scans serialised and coalesced; SSE capped at 24 with cleanup; +immutable assets in production; remote feeds with conditional GETs and a byte +budget. + +--- + +## 4. UI / UX + +Seen in a real browser at 1024 px and 375 px, and read in the templates. + +### Broken or misleading + +| # | Sev | Finding | Evidence | +|---|-----|---------|----------| +| U1 | **H** | **Remove / Delete file from the Episode page closes the modal and shows nothing**, while the work happens. The response retargets `#episode-table`, which only exists on the Show page; htmx throws "Invalid re-target", so the page keeps showing an episode that is gone and every later action 404s. | `fragments.js:1166`; `pages/episode-edit.eta` has no `#episode-table`. | +| U2 | **H** | **Subscription page: Check now / Pause / Resume / Stop following give no visible result** with JavaScript on. Buttons use `hx-swap="none"` but the handlers always 303 back; the XHR follows it, discards the page, and the flash waits for the next full load. "Stop following" deletes with no confirmation and no feedback. | `pages/subscription.eta:59-79`; `fragments.js:619-665` `redirectBack` unconditionally. | +| U3 | **H** | **No loading state, no double-submit guard, no network-error message anywhere.** Zero `hx-indicator` / `hx-disabled-elt` / `hx-sync` in any template. "Check now" and both Teach forms run the work inside the request (minutes on a phone); a second tap queues a second run. `htmx:sendError` (offline, tunnel dropped, blocked by an extension) is never handled, so those fail silently — the same shape as the 1.9.0 blank page. | `app.js:27-41`; `fragments.js:188,371,444,506`. | +| U4 | **H** | **Background SSE refreshes throw away what the user is typing** on the Adverts panel and the episode cuts card: the swapped region contains the boundary-words input, the settings `
` and the From/To range inputs. Any scheduled pass closes the details and wipes a half-typed form; there is also a redundant double render after every decision. | `ad-panel.eta:12-15`, `episode-adverts.eta:11-14`, `ad-pipeline.js:375,420`. | +| U5 | **H** | Password-changed toast renders as bare unstyled text with a × that throws: `modal-closed.eta` puts `hx-swap-oob` on the `.toast` element itself, so htmx inserts only its children. | `partials/modal-closed.eta:6` vs `partials/toast.eta:5`. | +| U6 | **H** | On a phone the Show page topbar overflows: the breadcrumb wraps to four lines and "Follow a feed" is cut off the right edge. `.topbar` has no wrap or mobile rule for the action cluster. **Seen live at 375 px.** | `pages.js:991-1006`, `app.css:1061`. | +| U7 | **H** | On a phone the Statistics access log and "Most fetched" tables are unusable: horizontal scroll, one row fills the screen. **Seen live at 375 px.** | `access-log.eta`, `app.css:1538`. | +| U8 | **M** | Primary buttons and accent links fail WCAG AA: `#D9602B` on white ≈ 3.7:1; `--ink-mute #8A8A86` on paper ≈ 3.1:1 is used for information-carrying text (the "· test" affordance, stat-card headings, timestamps, hints). | `app.css:88,279,378,1134,1152,1256,1419,1449`. | +| U9 | **M** | Episode cut pill links to `/adverts#ep-`, but the timeline pages 30 episodes, so for older episodes the anchor does not exist and the link lands at the top. | `adverts-view.js:624,28`. | +| U10 | **M** | `/shows/new` opens with "New show" pre-typed in the name field (the page title leaks into `value`), and a validation error discards what was typed. | `pages.js:250,269`; `show-new.eta:14`. | +| U11 | **M** | Upload success message is painted red (`succeed()` never adds the `ok` class). | `app.js:996-1003`, `app.css:797`. | +| U12 | **M** | Dashboard is never highlighted in the sidebar; the dashboard route never sets `active`. **Seen live.** | `pages.js:229-243`, `sidebar.eta:7`. | +| U13 | **M** | Ad-blocker exposure has moved from class names to URLs: `/ad-panel`, `/ad-work`, `/ad-detect`, `/adverts`, `/api/ad-segments/…/sample.mp3` are all XHR/`