From af5b5131ffc732e886eeadc7795ccc1d62711783 Mon Sep 17 00:00:00 2001 From: Emre Date: Mon, 14 Sep 2026 12:59:25 +0300 Subject: [PATCH 1/4] Correct the README where it had stopped being true Four things, one of which was actively misleading. measure-leakage was advertised as running "at split seed 42 and train seed 0". That is the Phase 5 regression, described as if it were the design: the command measures 15 seed pairs, and the single pair was the bug that this project has spent the last stretch removing. A README that documents the defect as the feature is worse than one that is merely out of date. --force was described without its limits. It applies to categories 2-5, which are inferences about what the recordings mean; categories 1 and 6 say no measurement can be constructed, and are refused with a reason. The command tour listed info, inspect, build, stats, audit and measure-leakage, but not train -- which the rest of the README and CONTRIBUTING both use. The roadmap checklist was missing --group-by (three schemes, including collection) and opt-in CUDA, both shipped, and did not list frequency-aware labelling even though the known-limitations section points at it as the fix. The status block keeps saying 0.4.0, because that is the released version: __version__, CITATION.cff and the newest tag all agree, and docs/release-notes/v0.5.0.md says on its first line that it is an unpublished draft. What it now adds is that main carries unreleased work, so a reader can tell which parts of this file are ahead of the wheel they installed. Co-Authored-By: Claude Opus 5 --- README.md | 31 +++++++++++++++++++++++-------- 1 file changed, 23 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index ebd8439..81e7b05 100644 --- a/README.md +++ b/README.md @@ -14,10 +14,14 @@ --- -> **Status: `0.4.0`.** On PyPI, tagged, CI green. The capture → dataset pipeline -> works end to end and is covered by tests. Interfaces may still change within -> `0.x` — see the [Roadmap](#roadmap) for what is planned and what is -> deliberately out of scope. +> **Status: latest release `0.4.0`.** On PyPI, tagged, CI green. The capture → +> dataset pipeline works end to end and is covered by tests. Interfaces may +> still change within `0.x` — see the [Roadmap](#roadmap) for what is planned +> and what is deliberately out of scope. +> +> `main` carries unreleased work on top of that release, listed under +> `[Unreleased]` in [CHANGELOG.md](CHANGELOG.md). This README describes `main`, +> so a few things below are ahead of what `pip install iqforge` gives you. > [!IMPORTANT] > **If you built a dataset with `--labels csv` or `--group-by csv:` over a @@ -79,6 +83,7 @@ iqforge inspect examples/bpsk_01.sigmf-meta # look at it, in your terminal iqforge build examples/ -o dataset/ --balance-by core:freq_lower_edge iqforge stats dataset/ # what did I just build? iqforge audit dataset/ # what could be wrong with it? +iqforge train dataset/ # is it actually trainable? iqforge measure-leakage recordings/ # preflight + paired measurement (if allowed) ``` @@ -317,12 +322,19 @@ unaltered; `--format json` gives the same content, `did_not_check` included. categories that eliminated four public datasets and let a fifth through ([methodology §6](docs/methodology.md)), then: - `REFUSED` exits non-zero -- `WOULD MEASURE` runs the paired cell (recording-level vs window-level) at - split seed 42 and train seed 0 +- `WOULD MEASURE` runs the paired cell (recording-level vs window-level) over + **15 seed pairs** — five split seeds by three training seeds, the same grid + every published table used. `--split-seeds` and `--train-seeds` make a + cheaper run a visible choice rather than a silent one, and the pair count is + printed with the result. `--force` overrides a refusal and keeps the overridden category in the header -so a pasted block cannot be mistaken for a clean run. `--sweep stride` runs the -fixed overlap ladder; there is intentionally no `--sweep snr`. +so a pasted block cannot be mistaken for a clean run. It applies to categories +2–5, which are inferences about what the recordings mean; categories 1 and 6 — +the reader cannot open the files, and `build` would refuse the split — say no +measurement can be constructed at all, and are refused with a reason instead. +`--sweep stride` runs the fixed overlap ladder; there is intentionally no +`--sweep snr`. ## Known limitations @@ -376,6 +388,9 @@ See [ROADMAP.md](ROADMAP.md) (Now / Next / Later). Short status: - [x] Real SigMF verification with public captures - [x] `iqforge audit` — leakage risk and measurability, without training - [x] `iqforge measure-leakage` — preflight + paired measurement (`--sweep stride` only) +- [x] `--group-by` — hold related recordings together (`path:`, `csv:`, `collection`) +- [x] Opt-in CUDA for new measurements; CPU stays the default and the published tables stay on it +- [ ] Frequency-aware labelling (the fix for the first known limitation below) - [ ] Verification with own hardware capture **Not planned for 0.x** — out of scope rather than pending: From baa2602677aeb4a36dfeb0d11aefaff97df6e2f6 Mon Sep 17 00:00:00 2001 From: Emre Date: Mon, 14 Sep 2026 12:59:44 +0300 Subject: [PATCH 2/4] Move the finished half of the real-capture item out of "Do next" "Repeat the leakage measurement on a real recording" sat unchecked in Now as though nothing had happened. Half of it has shipped: the stride sweep ran on DASH7 and LoRaIQ, both tables are in artifacts/, and LoRaIQ at 7/8 overlap is the first individually significant real-data result in this repository. A roadmap that does not show that makes the remaining work look larger than it is and the finished work invisible. It is now two entries. The overlap half is ticked, with the numbers and the limit that still applies -- the intermediate overlaps are unresolved at 15 seed pairs. The open item narrows to the SNR half, and points at docs/leakage-real-snr.md, where the seed count and SNR list are locked before the run so the result cannot pick its own stopping rule. The parity-gate re-measurement is added as a separate finished entry, since it is the thing that says the published tables still come out of the shipped command. The versioning footnote named 0.3.x / 0.4.0 as things to cut; 0.4.0 is out. It now says where unreleased work lives and that the release notes for the next version are marked drafts until a tag exists. Co-Authored-By: Claude Opus 5 --- ROADMAP.md | 31 ++++++++++++++++++++++++++----- 1 file changed, 26 insertions(+), 5 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index a7df76a..62976a9 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -32,11 +32,30 @@ Done recently (keep green): 31.8 dB above unannotated ones. Limits found on real data are in the README. +- [x] **The leakage measurement, repeated on real captures — the overlap half.** + The stride sweep ran on two public datasets. At zero overlap the + inflation is indistinguishable from zero three times over (+0.2 pp + synthetic, −3.7 pp DASH7, +1.5 pp LoRaIQ), and LoRaIQ at 7/8 overlap + reaches **+9.6 pp ± 2.7 (t = 3.5)** — the first individually significant + real-data result here. Tables in `artifacts/leakage_real_stride_table.md` + and `artifacts/leakage_loraiq_table.md`, reasoning in + [docs/methodology.md](docs/methodology.md) §3. What the intermediate + overlaps do is still not resolved: 15 seed pairs per point settle only + the largest effect. + +- [x] **The published grids re-measured through the shipped command.** + `scripts/parity_gate.py` re-ran three cells of each of the four tables + and compared run counts, seed pairs and every row's accuracies and window + counts against the recorded runs. All four passed. See SPEC §5.10.1 for + what that does and does not assert. + Do next, in this order: -1. **Repeat the leakage measurement on a real recording.** The current number is - synthetic BPSK/QPSK; the same curve on a real capture is what turns it from - an illustration into a result worth publishing. +1. **The SNR half of the real-capture measurement.** The overlap half is done + (above); the accuracy-against-SNR curve on a real capture is not. The plan + is locked in [docs/leakage-real-snr.md](docs/leakage-real-snr.md) — seed + count, SNR list and stem fixed before any run, so the result cannot choose + its own stopping rule. 2. **Verification with an own hardware capture.** One device end to end. Public files validated the reader; they cannot validate against the conventions of a radio nobody here has run. @@ -196,8 +215,10 @@ After Now is done — still reliability-first: Missing users is a product gap; more features will not close it. - [ ] Docs site (CLI + Python API reference) when the surface stops thrashing -Versioning: cut `0.3.x` / `0.4.0` when useful, on a schedule if needed — not -“only when hardware is done.” +Versioning: `0.4.0` is the latest release. Cut the next one when useful, on a +schedule if needed — not “only when hardware is done.” Unreleased work sits +under `[Unreleased]` in [CHANGELOG.md](CHANGELOG.md); `docs/release-notes/` +carries a draft for the next version, marked as a draft until it is tagged. --- From 5957658d153e1a0c40e9590bef59bc82e5da6445 Mon Sep 17 00:00:00 2001 From: Emre Date: Mon, 14 Sep 2026 13:00:13 +0300 Subject: [PATCH 3/4] Give methodology a map, so a reader can start from their question The document is over 1100 lines across nine numbered sections and had no contents at all -- the only way in was the top. Most readers arrive with one question, and the section that answers it is rarely the first. The table maps a question to a section rather than listing titles, because "Measurement 1 -- accuracy inflation against SNR" does not tell someone who wants to know how much a leaky split invents that this is their section. It also points out the two sections worth reading regardless of whether the measurements apply to your data: 7, the mistakes this project shipped, and 9, what the numbers do not support. Anchors verified against GitHub's slug rule rather than assumed. The first attempt used a checker that collapsed consecutive spaces, which flagged two correct anchors as dead -- the em dash in those two headings leaves a double space, and GitHub turns each space into its own hyphen without collapsing. Co-Authored-By: Claude Opus 5 --- docs/methodology.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/docs/methodology.md b/docs/methodology.md index 301db15..bd2646f 100644 --- a/docs/methodology.md +++ b/docs/methodology.md @@ -8,6 +8,29 @@ came from a run whose output is in [`artifacts/`](../artifacts/); where a statement has no number behind it, it is described as a design decision rather than a finding. +## Which section answers which question + +This document is long because the failures are the point. Start from the +question you have rather than from the top. + +| If you want to know | Read | +|---|---| +| Why splitting at the window level is a problem at all | [§1 The problem](#1-the-problem) | +| How much accuracy a leaky split invents, and at which SNR | [§2 Measurement 1 — inflation against SNR](#2-measurement-1--accuracy-inflation-against-snr) | +| Why overlap is the mechanism, and what happens at zero overlap | [§3 Measurement 2 — inflation against overlap](#3-measurement-2--accuracy-inflation-against-overlap) | +| Why the comparison is paired, and what is held fixed between the two arms | [§4 Experimental design](#4-experimental-design) | +| Whether the reader can be trusted on real captures — byte-level checks | [§5 Validation against real recordings](#5-validation-against-real-recordings) | +| Why four public datasets could not carry the measurement, and what the command's `category N` numbers mean | [§6 What it took to find a dataset](#6-what-it-took-to-find-a-dataset-that-could-carry-the-measurement) | +| What went wrong here and was only caught later | [§7 Silent failures found along the way](#7-silent-failures-found-along-the-way) | +| How claims were checked — mutation testing, controls, re-measuring published tables | [§8 Methods](#8-methods) | +| What none of this establishes | [§9 Limits](#9-limits) | +| The commands that regenerate every number above | [Reproducing](#reproducing) | + +Two sections are worth reading even if nothing here applies to your data. **§7** +is the list of mistakes this project made and shipped, including one that +published a wrong result; **§9** is what the numbers do not support, which is +more than they do. + --- ## 1. The problem From 24b0db35d71eb7225ade60163f7b8a15d26a5923 Mon Sep 17 00:00:00 2001 From: Emre Date: Mon, 14 Sep 2026 13:00:31 +0300 Subject: [PATCH 4/4] Mark the Phase 5/6 checklist superseded instead of editing its findings The file carries "Date checked: 2026-08-19" and reports readiness as of that date. One of its lines -- "no next-version draft file yet" -- is now false, because docs/release-notes/v0.5.0.md exists. Correcting the line in place would falsify a dated record, so the body is left exactly as written and a note at the top says the phase has since been executed and re-verified, names the line most likely to mislead, and points at CHANGELOG [Unreleased] and ROADMAP for current state. Co-Authored-By: Claude Opus 5 --- docs/phase5-phase6-checklist.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/phase5-phase6-checklist.md b/docs/phase5-phase6-checklist.md index dd9e155..eb793e0 100644 --- a/docs/phase5-phase6-checklist.md +++ b/docs/phase5-phase6-checklist.md @@ -5,6 +5,20 @@ Purpose: prep-only planning artifact for upcoming Phase 5 migration work and Pha Date checked: 2026-08-19 Branch checked: `cursor/phase-1-independent-gaps-38eb` +> **Superseded — kept as a record, not as a to-do.** This is a snapshot of what +> was true on the date above. Phase 5 has since been executed and re-verified: +> the migration runs through `iqforge measure-leakage`, and +> `scripts/parity_gate.py` has re-measured three cells of each of the four +> published tables against their recorded runs. +> +> Individual readiness lines below have expired and are deliberately left as +> they were written. The one most likely to mislead: the release-note item says +> no next-version draft exists, and `docs/release-notes/v0.5.0.md` has since been +> written — labelled an unpublished draft, because there is no `v0.5.0` tag. +> +> For current state read [CHANGELOG.md](../CHANGELOG.md) `[Unreleased]` and +> [ROADMAP.md](../ROADMAP.md). + --- ## Scope guardrails