Skip to content

RESULTS: canonical numbers with variant complex nodes (specs/046) #100

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
adamjohnwright merged 1 commit into from
Oct 10, 2026
Merged

RESULTS: canonical numbers with variant complex nodes (specs/046) #100

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -501,8 +501,8 @@ for the last feature that touched it rather than re-deriving from the code.
- `specs/046-variant-complex-nodes/` — a complex holding sets becomes one node
per member combination, and each reaction one copy per consistent choice
across all participants, catalysts and regulators included (generator
`LNG_VARIANT_NODES`, default on from LNG #109; `0` restores the old
networks). Arm vn7: held-out **+130** (p 1e-12), tuning −18, experimental
`LNG_VARIANT_NODES`, default on since LNG #109, canonical build
`20261009-2307_132d4b8`; `0` restores the old networks). Arm vn7: held-out **+130** (p 1e-12), tuning −18, experimental
−10 (ns), coverage identical, cut hand-offs 2,504 → 0. Two failed arms
(vn4, vn5) are recorded with every mechanism traced. The solver reads the
new `variant_pool` / `variant_split` edges as OR inputs (mean);
Expand Down
41 changes: 40 additions & 1 deletion docs/RESULTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# DeltaSignal: current results

**Last measured (canonical box below)**: 2026-10-05. Section 1 onwards: 2026-09-21, Reactome Release97, catalog build `cat_prod`
**Last measured (canonical box below)**: 2026-10-10. Section 1 onwards: 2026-09-21, Reactome Release97, catalog build `cat_prod`
(generator `f2842bc`), solver `5032771` with the current code defaults
(`hill_sat`, assembly-limiting on, `DS_INHIBITOR_EPS=1e-12`,
`DS_DEPLETION_H_MIN=0.1`).
Expand All @@ -19,6 +19,45 @@ commit.
---


> **2026-10-10: canonical numbers with variant complex nodes (specs/046).**
> Build `20261009-2307_132d4b8` (generator 132d4b8: `LNG_VARIANT_NODES` on by
> default, LNG #109), solver `0f0a11e` (`src/` unchanged since 5979e48),
> `results/0f0a11e`.
>
> | | cases | accuracy | macro-F1 |
> |---|---|---|---|
> | **curator held-out, in release** (70 pathways) | 18,573 | **88.63%** | **0.8524** |
> | curator held-out, every case (71 pathways) | 19,000 | 88.03% | 0.8440 |
> | curator, all pathways, in release (81) | 23,625 | 85.97% | 0.8274 |
> | curator, all pathways, every case | 24,100 | 85.42% | 0.8202 |
> | experimental, every case (10 pathways) | 849 | **68.79%** | **0.6097** |
>
> **What moved since the previous box:**
> - **Variant complex nodes.** A complex holding sets is one node per member
> combination, and a reaction is one copy per consistent choice across all its
> participants, catalysts and regulators included.
> - Cut hand-offs 2,504 → 0.
> - Against the previous box: held-out in release +0.70pp (macro-F1 +0.008),
> tuning −0.35pp, experimental −10 cases (594 → 584, p 0.22; macro-F1
> +0.009).
> - DSB repair carries +79 of the held-out gain; without it held-out is +51.
> - The rebuild reproduces the measured arm (vn7) exactly: experimental
> 584/849, the same convergence, the same pinned counts.
>
> Converged: 1,654 / 1,725 curator and 215 / 244 experimental solves.
>
> **Against MP-BioPath's published figures:**
> - Curator: 85.42% / 0.8202 against 83.34% / 0.8063, **+2.08pp**.
> - Experimental: 68.79% (584/849) against 75.74% (643/849), **−6.95pp**.
>
> **The experimental gap is network construction, not the propagator**
> (specs/047). Our propagator on MP-BioPath's own 2019 networks scores
> 610/849 and matches or beats MP-BioPath in 8 of 10 pathways. The shortfall
> is RAF, Mitotic G1, Cell Cycle Checkpoints and TP53, traced to five
> mechanism classes. That is the next work.
>
> **Previous box**, kept below for the record.

> **2026-10-05: canonical numbers on generator main with F7 set nodes and cap
> pools (specs/044, 045).** Build `20261004-1959_6990015` (generator 6990015:
> boundary layer split, F7 set nodes, `LNG_CAP_POOLS=1`), solver `5979e48`
Expand Down
Loading