Degree control that also matches the connectome's shortcut count - #3
Draft
aaravsinhaofficial wants to merge 94 commits into
Draft
aaravsinhaofficial wants to merge 94 commits into
aaravsinhaofficial wants to merge 94 commits into
Conversation
…tebook - Experiment 1: FlyWire MB connectome vs degree-matched controls on MQAR, spectral-radius matched (engine, plotter, 3 subruns, figures) — concluded. - Experiment 2: MB-core pruning vs the full 14k substrate + matched controls (engine reusing Exp 1's training loop, fleet launcher with --status, core-index build from the FlyWire annotation join, figures) — launched on the fleet. - scott/aws_fleet: validated spot-GPU training harness (stage/launch/watch/collect). - scott/labnotebook: dated entries + index for both experiments. - uv project setup (.python-version, uv.lock) and .gitignore/pyproject updates. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…t READMEs CX (cx_eigval_vs_eigvec): full CXBPU walkthrough — frozen N=7349 recurrent reservoir at rho=0.95, pool-gated 2-D self-motion input (sensory pool), K micro-steps with ReLU, pool-gated LINEAR readout to the polar-bump target (32-bin heading bump + 3-D home vector), 22,943 trained params identical across controls. Rationale for each choice (why frozen, why rho=0.95, why pool-gate, why micro-steps, why a linear probe = reservoir test). MB (mqar_mb_spectrum): MatrixEpisodicRNN — N=14025 trainable recurrent (sparse ~574k / dense ~196M), rho=0.95, ReLU, single step. Documents the two departures from CX and why: (1) I/O is NOT pool-gated (spans all N) because FlyWire MB has no cell types -> treat as a pure reservoir; (2) classification loss (masked CE) for the discrete MQAR recall task (vocab 32 + 3 role bits, 8 pairs / 8 queries, chance 0.031).
…CX + MB READMEs Controls: document the real Schur decomposition A = Z T Z^T (why Schur not eigendecomp -- the connectome is non-normal), and the two exact duals: spectrum_full = V T V^T (Haar-random directions, exact eigenvalues) vs eigvec_matched = Z T_rand Z^T (connectome directions + coupling, random eigenvalues), both dense, both rescaled to rho=0.95; plus spectrum_topk (Ginibre bulk), dense_random, and the sparse topology nulls, each with its own seed offset. Training: full recipe -- Adam, grad-clip 1.0, MSE (CX) vs masked cross-entropy (MQAR), the one-axis- at-a-time HP grid (LR/rho/wd/K) for CX and LR-only for MQAR, epochs/batch/seeds/patience, each model scored at its own best HP -- with the rationale for each choice.
pruned MB outperforms full control - slightly worse than complete 14.6k mushroom body
Exp 02 revealed bimodal distributions in some of the controls. The patience limitation was preventing full training. Those data are now excluded from the analysis, leading to a conservative analysis, but more sound. Results still hold that pruned connectome beats 14.6k control and does not perform much different than the full mushroom body
Added eigen vector controls to experiment 02. Only 10 runs for each control for time purposes. Due to low n, p values should not be taken so seriously, look for clear and obvious trends. Rerun with high N if using for publication
main aws script tells each worker to only download as much data as necessary for their shards --collect also changed to only collect final run data, no checkpoint bloat
… neurons on its native task A trainable-recurrent net whose recurrent matrix is the MB connectome, with a FREE (random-init) input projection, is trained on (a) MQAR (arbitrary) and (b) odor->valence reversal (MB native). Measure whether W_in converges onto the biological input cells (sensory pool / projection neurons), init->final, connectome vs random, FlyWire (proxy) + hemibrain (cell-type-grounded). Result: on the NATIVE task the connectome's input layer converges onto the projection neurons (hemibrain PN-AUC 0.46->0.59) AS it learns (tracks the learning curve), and it learns ~2x faster (epochs->0.9: 19-21 vs random never). On the arbitrary MQAR task there is no cell-type-grounded convergence (PN-AUC 0.50->0.50); random-init never converges on either task. So convergence-to-biology is TASK-DEPENDENT (connectome x matched-task interaction). Recurrent WEIGHTS scramble on both tasks (corr~0) -- only the input layer + weak functional activity find biology. Adds train+analysis scripts, two figures, and a precise README with methodology, TL;DR, significance, and caveats (modest AUC~0.6, 2 seeds, ER not degree-matched control, FlyWire proxy gives false positives).
…nificant)
Reran the native-task convergence at 20 seeds per condition (connectome vs random, FlyWire +
hemibrain). The input-layer convergence to biology is robust, not seed-luck:
- hemibrain (cell-type-grounded, PNs): connectome 0.599+/-0.066 vs random 0.499+/-0.026; connectome
beats random in 18/20 paired seeds; paired t p=1.1e-5, Wilcoxon p=3.6e-5; 18/20 clear chance.
- FlyWire: connectome 0.631+/-0.016 vs random 0.498+/-0.010; 20/20 paired seeds; paired t p=2.3e-18.
- random sits at chance (0.498-0.499) throughout; connectome reaches 0.9 reversal acc in ~20 epochs,
random never does in 30.
Updates the existing README (replaces the 2-seed caveat with the n=20 statistics + box plot), adds
the 20-seed box plot and the stats script. No degree-matched control / no new README.
…ine diagnostic Plot the underlying ||W_in|| distributions (biological vs non-biological cells, init->final, 20 seeds pooled) that the convergence AUC summarizes, plus a per-seed AUC diagnostic. Documents that AUC<0.5 only appears at the init baseline / random control (FlyWire baseline = chance p=0.62; hemibrain baseline 0.487, a fixed- positive-class artifact that cancels in the paired test), never in the connectome's trained value (0.60-0.63).
…fixed-class artifact The init W_in is drawn identically per neuron, so E[init AUC]=0.5 exactly (no per-neuron bias mechanism; 20-seed-mean norm spread 0.00710 matches iid 0.00708). The hemibrain 0.4866 at n=20 is a finite-sample low roll of those specific seeds; re-instantiating the identical init regresses it to chance (50->0.497, 100->0.498, 500->0.4996, CI 0.498-0.502). Replaces the earlier (incorrect) fixed-positive-class 'deterministic offset' wording.
Replaces the misleading 2-seed assoc_biology_convergence.png. CI bands make the init asymmetry legible as noise width, not bias: FlyWire's large 1089-cell class sits pinned at 0.50, hemibrain's small 168-PN class has ~2.5x wider bands and a 20-seed init of 0.487 (regresses to 0.4996 by 500 seeds). Updates stale 2-seed '0.46->0.59' text to the 20-seed 0.49->0.60, adds figure caption noting the class-size variance and the early KC/MBON transient.
…ayer (mirror of MB) Free-I/O CX RNN (connectome recurrent, trainable) on the native polar-bump path task: the readout spontaneously migrates onto the biological output pool (steering/PFL cells), AUC 0.50->0.65, connectome beats random-wired control 8/8 seeds (paired t p=3.9e-7), while the input layer stays null (~0.48). Mirror image of the MB (which converged on its INPUT/PN layer). n=8 of 16 seeds (run in progress); refresh on completion. FreeCXBPU = edge-passing+checkpointed SparseCXBPU, verified numerically identical (<1e-7 fwd, <1e-9 grad).
…, p=3.6e-13) Full 16-seed result. Readout migrates onto the CX biological output pool (steering/PFL): AUC 0.50->0.657, connectome beats paired random 16/16 seeds (paired t p=3.6e-13); input layer does NOT converge (0.50->0.478, null/slightly below). Refined cross-region framing: NOT a mirror image -- MB matches both interfaces (input-dominant +0.10, output +0.06), CX matches output only (+0.16), OL neither. Convergence strength tracks task dependence on each interface. Drops PRELIMINARY banner.
…egion/layer-specific One README unifying the two positive results with a cross-region summary figure: - MB (odor identity): INPUT converges onto PNs (Δ+0.10, 18/20, p=1e-5), input-dominant - CX (path integration): OUTPUT converges onto steering cells (Δ+0.16, 16/16, p=3.6e-13) Each circuit converges hardest on the interface its task depends on; matched random-wired control shows none of it. Includes methodology + the design choices (free I/O, trainable connectome recurrent, paired ER control, AUC metric) and interpretation.
…ched controls) Clean heading-error-vs-T figure over the cx_structure_polar per-seed metrics: connectome (cx_bpu) vs weight-shuffled, degree-shuffled, random, and no-recurrence controls, in the frozen reservoir and observed-edge trainable regimes, T=50/100/200, mean +/- SEM over 3 seeds. Connectome has the lowest heading error at every point in both regimes.
…ning curve Clean two-panel figure (MB native odor->valence task, hemibrain, 20 seeds, mean +/-95% CI): task accuracy and the ROC-AUC that the input weight ||W_in|| lands on the biological input cells (PNs), over training, for the connectome vs degree-matched random wiring. The connectome's input-port AUC rises from chance to ~0.60 as the task is learned; matched random stays at 0.5.
…; add economic-value argument - Regenerated assets/og.png (1200x630) from the new branded hero; bumped og/twitter image meta to ?v=2 to bust stale social caches. - Slimmed 'Where the prior does not help': dropped the control-hierarchy card and condensed the reproduction paragraph (kept the chart + limitations). - Added a conditional economic-value argument to the Scale section (connectome as a reusable, mapped-once prior for the parts of intelligence more scale hasn't solved). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… (negative result) Full left-OL connectome (48,749 neurons) floors on optic flow when wired through real FlyWire-783 cell-type ports (R1-6 photoreceptors -> HS/VS LPTCs), across ~12 training levers, while a degree-matched control learns. The deep biological readout (~4-5 synapses) starves the gradient (~30x weaker signal at init); the control wins only via manufactured shortcuts. Sharpens vis-01: the generic-I/O +12% does not survive biologically-faithful ports. Self-contained under outputs/results/optic_flow_biological_io/ (force-added past the outputs/ gitignore): README + figure, biological-port substrate builder + cell-type join, BioFlowRNN + fleet-ready runner, reproducible data + logs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…/results/ Move the biological-I/O optic-flow writeup alongside the other experiment folders in docs/results/; update README reproduce paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… matched controls) New region×task pairing: the FlyWire-783 antennal lobe (N=3499: 2282 ORN / 429 LN / 685 PN + 103 thermo/hygro, 53 olfactory glomeruli, signed by Dale's law) wired as a leaky-tanh RNN, detecting ethylene in the UCI-309 turbulent Me/CO mixture dataset under biological I/O (nonnegative sensor→glomerulus adapter → ORN input, projection-neuron readout). Train medium/high concentration, test held-out LOW concentration. Result (195 runs, 3 seeds, AWS spot fleet): the connectome is MORE sample-efficient and more robust to low concentration than every matched control — degree-preserving, edge-matched random, eigenvalue-spectrum-matched, and dense-Gaussian (all ρ=0.95). Low-conc recall @10% FA: connectome 0.682 vs degree 0.639 (d=2.8), random 0.648 (d=1.2), spectrum 0.115, dense 0.291; adapter-only floor 0.361. Biological I/O helps (0.682 vs 0.612 free I/O) — opposite of the optic-lobe stall. Connectome also detects the plume fastest right after onset (PN onset-emphasis). Honest caveat: connectome = 1 graph × 3 training seeds (pseudoreplication); degree-matched is the load-bearing null and is still beaten by d=2.8. Includes build scripts (substrate/task/operators), model, sharded fleet runner + driver, figures, and full metrics. Heavy substrate/operator artifacts (~156MB) git-ignored + regenerable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rewrite the writeup with a plain-language TL;DR, methodology, and technical detail. Add three figures with reproducible data: (1) the biological pathway + the two ports; (2) the signal deficit at the readout + training curves; (3) the mechanism — deficit robustness across rho, frozen-feature decodability, and the lever sweep that all floors. Replaces the single 2-panel figure and the make_writeup/plot_writeup scripts with generate_data.py + plot_figures.py. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…dation + deep README Scale-up of the AL×turbulent-ethylene experiment to 6 seeds (390 runs on a 20-GPU AWS spot fleet) + a UCI-270 long-term-drift external validation + a comprehensive technical README. Primary (6 seeds, biological I/O, low-conc recall @10% false-alarm): the connectome is the most sample-efficient and low-concentration-robust graph at every data fraction — 0.690 vs degree 0.652 (d=1.7), edge-random 0.651 (d=1.2); spectrum/dense far below; adapter-only floor 0.354. Advantage holds 5→100% data, connectome detects the plume fastest right after release, and biological I/O ties/beats free I/O. Robust to graded (non-spiking) local neurons. External validation (UCI-270 drift, 60 runs, chronological batches): HONEST NULL — connectome (0.599) does not beat the sparse controls (ER-random 0.654). The advantage is task-specific: the AL graph is a better substrate for the computation it evolved to do, not a generically better net. Adds: run_drift.py, make_drift_figure.py, make_overview_figure.py; drift metrics/analysis; 4 figures (summary, headline, substrate/task overview, drift); TL;DR + methodology + technical-depth README. Model gains an output_dim arg (multiclass drift) preserving the binary path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ls section - fig_training_curves.png: per-epoch train/val BCE (bio) at 100% and 10% data, connectome vs controls. Shows the sparse arms (connectome/degree/random) converge to ~0.11 train BCE while spectrum/dense cannot fit the data (~0.33, unstable) — explaining their metric collapse as a trainability failure through the narrow biological ports, not a capacity ceiling. - loss_history.csv: consolidated per-epoch training history (7,026 rows) committed. - Expanded the Controls section: construction algorithm, what each preserves/randomizes, node- identity, exact recurrent parameter counts (258,882 sparse vs 12.24M dense), and the specific null each isolates (degree = wiring specificity beyond degree; spectrum = dynamics-only; dense = density/capacity confound at ~47x params). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Remove fig_training_curves.png and its generator/README reference. On training/validation loss the three sparse arms (connectome/degree/random) are indistinguishable, so the curves did not add signal beyond the metrics; kept a short text note that the connectome's edge is in held-out low-conc generalisation, not in-distribution loss. loss_history.csv retained as raw data. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rewrite the narrative sections (circuit, task, model, controls, how-we-ran-it, interpretation) to be readable without jargon: shorter sentences, terms defined inline, notation and file-path clutter moved out of the prose, tables/figures/numbers kept. Framing corrected to match the honest picture: the strong effect is sparse >> dense; the connectome's edge over sparse controls is a small, top-ranked-of-7 margin (judged by rank, not the overstated Cohen's d). Updated the figure generator's auto-text to emit the same rank-based honest framing on future --collect. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…extension REGRESSION (Scott's Jul-13 ask; 72 runs, 6 seeds, AWS fleet). New continuous-tracking benchmark: an intermittent turbulent plume whose concentration c(t) changes every step, seen through 8 cross-reactive sensors with per-sensor first-order lag. The model must output c(t) at EVERY step; a collapsed network scores R^2 = 0 exactly. Result vs the "connectome collapses to a fixed point" hypothesis: NOT supported in its strong form — the connectome tracks (R^2 0.351 bio / 0.533 free, not 0). But a weaker version holds: every connectome-family net is under-responsive, emitting only ~57% of the target amplitude (GRU 0.83). And the connectome is not special here — it ties its controls under bio I/O (0.351 vs degree 0.344) and LOSES under free I/O (0.533 vs 0.597). Damning reference: a memoryless no-recurrence adapter scores 0.544, beating every bio-I/O recurrent arm; a GRU scores 0.678. On this task the recurrence destroys information rather than integrating it, and biological I/O hurts (opposite of gas classification). Flagged a readout-norm bug that would have faked a spectacular confirmation. 4x4 MATRIX (partial, deadline-bound): adds AL as a 4th region and gas as a 4th task on top of the existing 3x3. Size-matches every region to N=3499 (highest-degree induced subgraph, rho=0.95) since the prior grid showed unmatched sizes turn into a capacity confound. Gas column partial at 3 seeds and inconclusive; CX×gas and the AL row (MQAR/path/flow) not finished. Notes the ρ-rescale gotcha (raw AL adjacency has rho~2852 -> instant NaN in the shared harnesses). Model gains return_sequence for per-timestep regression readout (binary path unchanged). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Completes the grid by adding the antennal lobe as a 4th region and turbulent gas detection as a 4th task (7 new cells, 3 AWS fleets, ~180 runs), and adds the comparison that turns out to matter: every region running gas through ITS OWN biological interface rather than a generic one. MAIN FINDING — the interface, not the wiring, decides whether a connectome helps. On gas the AL connectome goes +0.4% (generic all-neuron I/O) -> +5.2% (its own ORN->PN interface): same graph, same task, same controls, only the interface changed. Under generic I/O nothing separates for any region, including the AL on its own native task. Column-wise, gas aligns perfectly: given each region its proper interface, ONLY the native region beats its controls — AL +5.2%, MB -0.5%, CX -0.7%, OL -100%. Row-wise, AL does NOT align: its biggest advantage is path (+11.7%) not gas (+5.2%). MB/CX/OL do peak natively. Caveat recorded in the README: the path column runs FROZEN recurrence, a regime that favours structure, so that row comparison is not apples-to-apples. OL fails outright on gas under its own biology — AUROC 0.500 (chance) — while its own degree/random controls on the SAME ports score 0.635/0.688. Not a pipeline bug: it is the documented optic-lobe biological-I/O stall (1399 R1-6 -> 22 HS/VS bottleneck starves the readout gradient) replicating in a completely different task. Method: all regions capped to a common N=3499 with PORT NEURONS PRESERVED (a plain top-degree cap would delete the neurons that make the interface biological), identical 61-channel adapter capacity across regions, rho=0.95 everywhere. Biological ports derived from FlyWire-783 cell types and independently cross-checked (MB's ALPN/MBON corroborated by ROI compartment profiles AND edge directionality). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Ran three causal tests on the AWS fleet (3 x 24 GPUs) to ask WHY the connectome helps.
ROBUST FACT: connectomes are contractive. Measured 4 ways across all 4 regions vs their own
degree/edge-matched controls: input->output transfer gain 0.10-0.46x the controls, readout operating
point 2.5-10x lower activity, reciprocity 2-20x higher, non-normality 2-3x higher. Rewiring
INCREASES throughput, so contraction is a property of the specific wiring, not of sparsity or the
degree sequence (degree heterogeneity is matched by construction and thus ruled out). Independently
reproduces Scott's dyn-01 ("normalization triples the contraction, dwarfing rho") and the regression
amplitude ratio 0.57.
TEST 1 PASSES (144 runs): the connectome's advantage is largest at the contractive operating point
and erodes ~5x when driven off it. corr(log input gain, advantage) = -0.64 vs degree, -0.67 vs
random. Mirrors vis-01 subrun 07, where leaving contraction unlocked optic flow but the connectome
then TIED its control.
TEST 2 FAILS (108 runs): it is NOT level-invariance. Under a controlled concentration nuisance
(g ~ LogNormal(0,sigma)) the advantage SHRINKS with sigma against the strong control
(corr = -0.69 vs degree). The "divisive normalisation buys invariance" story fails its own
dose-response test.
TEST 3 FAILS (54 runs): an untrained change-SNR measure reproduced the win/tie/loss ordering at n=3
regions but does not generalise -- within-region corr +0.60 (AL), +0.30 (MB), -0.36 (CX, wrong sign).
An n=3 ordering was not evidence for a law.
TWO CORRECTIONS TO PUBLISHED CLAIMS:
1. METRICS BUG: with tied scores a stable sort preserved input order, so a CONSTANT-OUTPUT model
scored AUPRC = 1.000 and F1 = 1.000 (verified in all 6 OL rows while AUROC correctly read 0.500).
Any AUPRC ranking would have scored the worst arm best. Fixed (tie-group aware) in
antennal_lobe_gas/common.py; a constant predictor now scores the base rate.
2. THE OL x GAS CELL IS A SIZE-MATCHING ARTIFACT, NOT BIOLOGY. In the full OL, R1-6 -> HS/VS is
reachable in 3 hops; the N=3499 degree-ranked cap deleted the low-degree retinotopic relays,
leaving the readout fully disconnected (0/22 reachable) and forcing AUROC = 0.500. It was also
wrong to call it a replication of the optic-lobe stall (that was gradient starvation at depth).
4x4 README corrected throughout; cell is now marked "not evaluable".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Standalone note answering the alignment question directly, with the honest scope. CONFIRMED, one column: with every region on its OWN biological interface, the gas column shows the diagonal — only the native region (AL) beats its controls (+5.0% vs degree-matched, +5.2% vs edge-random, p=0.019, rank 6/6), and it REPLICATES in an independent run (+10.4%). MB reliably ties (+0.3%/-0.5%); CX is unstable (-0.7% then +3.6%) and should be read as noise. NOT DEMONSTRATED, the matrix: only the gas column was run with proper biological I/O. The flow/mqar/path columns come from the earlier 3x3 grid under generic/other I/O regimes, so they can neither support nor refute a proper-I/O diagonal. The nine missing cells (esp. the native MB->mqar, CX->path, OL->flow under their own interfaces) are the decisive next run. Also records the interface effect that makes the question askable at all: under generic all-neuron I/O nothing separates anywhere, including AL on its own task (+0.4%); the same graphs under proper ports give +5.2%. Any earlier null that used generic I/O may have been testing the readout rather than the connectome. Carries forward the two corrections: OL x gas is an invalid size-matching artifact (readout disconnected by the cap; 3 hops in the full OL), and the AUPRC/F1 tie-handling bug. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…be insufficient Hunted for the mechanism behind the AL x gas connectome advantage. Proposed, tested and killed the best candidate, and in doing so separated two things that were being conflated. THE HYPOTHESIS. Trained recurrent weights travel 4-9x their initial norm (dW_rel 8.05 connectome vs 4.31 degree), so initial synapse VALUES wash out and only the edge SUPPORT matters. The one structural property surviving degree-matching is RECIPROCITY (mutual A<->B pairs, the shortest feedback loops): connectome 0.45 vs degree 0.21 vs edge-random 0.02, in all 4 regions. So we built a control matching degree AND reciprocity (degree corr 0.9997) -- a control nobody had run. FIRST RESULT LOOKED DECISIVE: recip-matched 0.663 vs degree 0.637 vs connectome 0.678, apparently recovering ~65% of the advantage. IT DIED ON ITS OWN DOSE-RESPONSE (42 runs, reciprocity ladder 0.02->0.50): flat and non-monotone, and the HIGHEST-reciprocity point scored LOWEST. An independent rebuild of the identical control landed at 0.635 instead of 0.663 -- the whole effect was build-to-build noise. POOLED OVER 4 INDEPENDENT RUNS: connectome 0.6910 +/- 0.028 (n=24) reciprocity-match 0.6493 +/- 0.030 (n=12) degree-matched 0.6487 +/- 0.025 (n=18) connectome vs degree d=+0.0423 t=+5.08 <- the effect is REAL and replicates recip-matched vs degree d=+0.0006 t=+0.06 <- reciprocity explains 0%, not 65% THE REAL RESULT THIS BOUGHT: CONTRACTION IS NOT SUFFICIENT. The reciprocity-matched graphs reproduce the connectome's contractive operating point almost exactly (readout activity 0.095 vs 0.097; degree 0.259, random 0.584) yet get NONE of the benefit. Contraction and the advantage are decoupled. Scott's contraction finding (dyn-01/vis-01) explains why connectomes are hard to TRAIN; it does not explain why the AL WINS. This is the experiment that separates them. Nine mechanisms are now ruled out: reciprocity, level-invariance, change-SNR, transient amplification, flat minima, discriminative-direction invariance, degree heterogeneity, capacity, contraction-alone. Whatever it is survives degree-, reciprocity-, spectrum- and density-matching, and lives entirely in the generalisation term (train losses identical 0.115/0.113/0.126) -> higher-order structure. METHOD LESSON: with effect ~0.04 and build-to-build sd ~0.03, six seeds of ONE graph build is not enough. Rebuild controls independently and pool before believing a structural claim. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… (216 runs) Ran the experiment DIAGONAL.md named as decisive: every region on every task through ITS OWN biological interface, 4 regions x 3 tasks x 3 arms x 6 seeds on the AWS fleet. RESULT: NO DIAGONAL. None of the three native cells beats its own matched controls. MB x mqar 0.1884: +4.2% vs degree (rank 5/6, p=0.046) but -3.1% vs random (rank 1/6) -> mixed CX x path 0.8889: -1.1% / -0.6%, rank 0/6 -> clean loss OL x flow: not evaluable (see below) Native cells average -1.9% vs their random controls, off-diagonal -4.1%: alignment does not predict the advantage. AND THE ONE CLEAN WIN IS OFF-DIAGONAL: AL x flow, +5.8% vs degree, rank 6/6, p=0.041 — the antennal lobe beating its controls on optic flow. So AL x gas was not a fluke, but it was not alignment either: the AL wins on two unrelated tasks while MB/CX/OL win on none, including their own. WHAT DOES PREDICT IT: interface width. Mean advantage vs degree per region is monotone in readout width — AL (685) +1.9%, CX (327) +0.4%, MB (96) -4.4%, OL (22) -65.8%; r=+0.53 across cells. The connectome helps when its biological readout is wide enough to carry the computation. OL NOT EVALUABLE (measured, not inferred): the pathway-preserving cap DID restore topology (22/22 outputs at median 3 hops, identical to the full lobe), but an adversarial audit measured actual signal DELIVERY: OL 1.0e-09 vs 2.3e-04 for its own degree control (200,000x) and 1e-3..1e-4 for AL/MB/CX. Any 3,499-node slice of a 96,816-node retinotopic lobe is not functionally an optic lobe. Consistent with Scott's vis-01 subrun 07, which found connectome ~ control at full scale (p=0.36-0.55). Method hardening: input pools now subsampled by out-degree + proximity to the readout (degree ranking left 52% of retained R1-6 as dead ends); the manifest records signal delivery because reachability is not function; train/val/test are independent draws with val-only model selection (the committed CSV carries val_score proving the no-leak code ran). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… fails"
A post-hoc adversarial audit of the operators found a systematic ANTI-CONNECTOME handicap that
invalidates every non-AL cell, so the headline conclusion of the previous commit is retracted.
Degree-preserving shuffling of a LAYERED graph manufactures direct input->output shortcuts the real
wiring forbids (direct in->out edges, connectome vs degree control):
AL 21382 vs 25428 (1.2x) CX 279 vs 6054 (21.7x) MB 32 vs 2680 (83.8x) OL 0 vs 51
mean hop to readout: AL 1.03/1.02, CX 1.81/1.00, MB 2.14/1.02, OL 3.00/1.18
The deeper a region's native pathway, the more the shuffle shortens it — and AL is the ONLY region
immune, because its pathway is already 1 hop. The design therefore yields "connectome wins only on
AL" EVEN IF WIRING WERE IRRELEVANT EVERYWHERE.
Consequences:
- "Alignment fails" is NOT established: MB x mqar and CX x path lost to controls handed an 84x /
22x shortcut advantage. Those losses may be entirely artifactual.
- "Interface width predicts advantage" is CONFOUNDED: readout width, pathway depth and shortcut
handicap co-vary perfectly across these 4 regions and cannot be separated.
- AL x gas and AL x flow remain VALID (handicap 1.2x).
- Secondary: random_control_matrix is not degree-matched (flattens fan-out).
Fix required: a PATH-MATCHED control preserving the input->output hop distribution. Until then the
non-AL cells are not evaluable, like the OL row.
This generalises a known one-off: the optic-lobe stall write-up already noted its "degree-matched
control wins via shortcuts" — the same artifact, now quantified in all four regions. Degree-preserving
rewiring is not a fair control for deep/layered circuits, which likely affects other cells in this
program.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A degree-preserving shuffle is the standard null here, but it is not a fair
one for a LAYERED circuit: rewiring at fixed degree invents direct
input->output edges the real wiring forbids. Measured on the operators these
experiments actually use:
MB 2,680 direct ALPN->MBON edges vs 32 real (83.8x)
CX 6,054 vs 279 (21.7x)
AL 25,428 vs 21,382 ( 1.19x)
An 84x express lane lets the MB control skip the Kenyon-cell layer, i.e. skip
the computation the circuit exists to do.
build_shortcut_matched.py adds a `degree_sm` arm: start from the same degree
shuffle, then repair it with degree-preserving double-edge swaps until the
direct input->output count matches the connectome's own. Degrees are preserved
exactly, so it stays a strict degree control that simply lost its shortcuts.
AL's 1.19x handicap is the built-in control for the control: the fix should
barely move AL while moving MB a lot.
Result (6 seeds, biological I/O, seed-paired):
MB x mqar connectome - degree +0.0103 (6/6)
connectome - degree_sm +0.0031 (5/6) -70%
AL x flow connectome - degree +0.0198 (6/6)
connectome - degree_sm +0.0189 (6/6) -5%
Dose-dependent as predicted: the control moved +0.0072 on MB vs +0.0010 on AL.
But the SIGN is opposite to my hypothesis -- removing the shortcuts made the
control BETTER, so those shortcuts were hurting the degree shuffle, not helping
it. The connectome's MB x mqar win was therefore partly an artifact of
comparing against a sabotaged opponent. MB x mqar was the one clean positive in
the earlier region x task grid; it is materially weaker than reported. AL x
flow is untouched.
Caveats recorded in the writeup: pseudoreplication (one connectome, six
TRAINING seeds), and an exact paired Wilcoxon with n=6 has a p floor of 0.031,
so "6/6 seeds" is the floor, not strong evidence.
Fleet fixes needed to run this:
* bootstrap.sh hardcoded CUDA_VISIBLE_DEVICES=0, so every worker on a
multi-GPU instance piled onto GPU 0 and left the rest idle. Now spreads
workers across all visible GPUs.
* config.env: widen the instance-type ladder for capacity resilience, kept
small-first so it never silently escalates cost.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
An adversarial audit recomputed every number in SHORTCUT_MATCHED.md. The
ARITHMETIC is clean -- direct-edge counts independently recounted from the raw
.npz operators reproduce 32/2680 (MB), 279/6054 (CX), 21382/25428 (AL); all 18
degree_sm operators hit their target exactly; degree vectors element-wise
identical; rho=0.9500 throughout. The INFERENCE was inflated. Corrections:
* MB vs the fair control is p=0.094 -- NOT significant. That number was in the
shipped summary CSV and the writeup never printed it.
* "removing the shortcuts made the control better (+0.0072)" was asserted from
a null: 95% CI [-0.0070,+0.0214], Wilcoxon p=0.31, 4/6 seeds, and 78% of the
mean comes from ONE run (MB degree seed 0 = 0.1593 vs 0.1811-0.1923 for the
other five, val-patience-stopped at 31 epochs vs 40 for its pair). Drop that
seed and the shift is +0.0019 and the "-70% shrink" becomes -32%.
* "dose-dependent, 7x more" is a ratio of two quantities that individually
cannot be told from zero (MB-vs-AL Welch p=0.32; p=0.71 without seed 0), over
TWO points whose middle dose (CX, 21.7x) was never trained.
* "AL never had shortcuts to remove" was FACTUALLY WRONG: AL had 4,046 surplus
shortcuts removed, MORE in absolute terms than MB's 2,648. Only the ratio is
small.
* OL was listed in the handicap table as part of the measured series, but no OL
degree_sm operator was built and no OL cell was trained. Removed.
* CX "~1.8 hops" had no shipped provenance (reach_audit.json covers only AL and
MB). Recomputed all four depths by BFS from the input ports ->
pathway_depth.json (AL 1.02, CX 1.81, MB 1.90, OL 3.00).
* Pseudoreplication was mis-stated in a way that let the analysis off the hook:
the CONTROL arms are six independent graph draws, so only the connectome side
is n=1 -- which makes the rank test the doc called "not run here" much closer
to hand than implied.
Surviving claim, which is still worth having: a shortcut-matched control shrinks
MB x mqar's margin from +0.0103 (p=0.031) to +0.0031 (p=0.094, n.s.) and leaves
AL x flow's +0.0189 (6/6, dz=3.3) intact. WHY it shrank is unresolved.
Figures (each adversarially checked, then fixed by its checker):
fig_per_seed_paired all 6 seeds as lines; CIs, exact p, (n.s.) tags,
leave-one-out, per-arm early-stop counts, off-scale note
fig_dose_response two points only -- no fit, no correlation; CX drawn as
a pending placeholder with no y value
fig_margin_shrink slope chart with per-seed scatter
fig_pathway_schematic explainer for why degree shuffles invent express lanes
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Collaborator
Author
|
| claim in the original PR body | status |
|---|---|
| MB margin −70% against a fair control | point estimate only — leave-one-out range is −32% to −80% |
| "removing shortcuts made the control better (+0.0072)" | NOT SUPPORTED — CI [−0.0070, +0.0214], p=0.31, 4/6 seeds |
| "dose-dependent… 7× more" | NOT SUPPORTED — MB-vs-AL Welch p=0.32 (p=0.71 without one seed) |
| "AL never had shortcuts to remove" | FACTUALLY WRONG — AL had 4,046 removed, more than MB's 2,648 |
| MB × mqar fair-control margin | p = 0.094, not significant — this was in my own summary CSV and never printed |
The single-run dependency
78% of the MB control shift comes from one run: degree seed 0 scored 0.1593 when the other five sit at 0.1811–0.1923, and it val-patience-stopped at 31 epochs vs 40 for its degree_sm partner. Drop it and the shift is +0.0019 and the shrink is −32%.
What survives
A shortcut-matched control shrinks MB × mqar from +0.0103 (p=0.031) to +0.0031 (p=0.094, n.s.), and leaves AL × flow's +0.0189 (6/6, dz=3.3) intact. Why it shrank is unresolved.
The shortcut confound remains a real structural fact (84× / 21.7× / 1.19×). Its training consequence is not yet measured.
Also fixed
- OL was listed in the handicap table as part of the measured series — no OL
degree_smwas ever built or trained. Removed. - CX "~1.8 hops" had no shipped provenance (
reach_audit.jsoncovers only AL/MB). Recomputed all four by BFS →pathway_depth.json(AL 1.02, CX 1.81, MB 1.90, OL 3.00). - Pseudoreplication was mis-stated in my favour: the control arms are six independent graph draws, so only the connectome side is n=1 — the rank test I called "not run here" is much closer to hand than I implied.
4 new figures
Each was generated, then handed to an independent verifier that opened the PNG and tried to prove it misleading; every one found real problems and fixed them. The dose-response figure fits no line and quotes no correlation — there are two points, and CX (21.7×), the middle dose, is drawn as a pending placeholder with no y value.
CX × path is the single most valuable missing run.
… directions The fleet finished six more cells, taking this from 2 to 8 complete cells (3 arms x 6 seeds each) across all three regions. The extra data changes the conclusion, and the new one is stronger. TWO VERDICTS FLIPPED SIGNIFICANCE, both in MB (84x), in OPPOSITE directions: MB x mqar +0.0103 (p=0.031) -> +0.0031 (p=0.094) a WIN evaporates MB x path -0.0267 (p=0.031) -> -0.0054 (p=0.312) a LOSS evaporates ~80% of MB x path's apparent deficit was the control's manufactured shortcuts. A control artefact that only inflated the connectome could not do both; a generic input->output shortcut can, because it helps the control on whichever side it lands. DIRECTION: mostly the shortcuts HELP the control -- the original hypothesis. 5/8 cells shift negative and BOTH significant ones do (MB x path -0.0213 p=0.031 0/6; CX x flow -0.0185 p=0.031 0/6). This RETRACTS the previous commit's reversal. With only MB x mqar and AL x flow in hand, MB x mqar's +0.0072 looked like evidence the shortcuts were HURTING the control. It is the only layered cell pointing that way, it is not significant (p=0.31, 4/6), and 78% of it comes from one anomalous run. I generalised from the single cell that disagreed with the other seven. LAYERED vs SHALLOW, not a graded dose: layered (MB/CX, 22-84x) n=5 cells mean |verdict move| 0.0101 shallow (AL, 1.19x) n=3 cells mean |verdict move| 0.0010 10x apart, but Mann-Whitney one-sided p=0.071 -- suggestive only. And NOT proportional to the ratio: MB (84x) 0.0103 ~= CX (22x) 0.0097. The "dose- response" framing stays dead; "layered or not" is what the data support. AL is unmoved in all three of its cells. Unit of analysis is the CELL, not the seed -- pooling seeds across cells would treat correlated runs as independent. Local (Blackwell) and fleet (L4) runs are never mixed WITHIN a cell, since a hardware difference inside a paired comparison would land in the difference; per-cell source is recorded. CX x path (the 9th cell) was still training at commit time. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Collaborator
Author
Update — 8 cells now, and the result is stronger (df3c718)The fleet finished six more cells. 2 → 8 complete cells (3 arms × 6 seeds each) across all three regions. This changes the conclusion. The headline: the confound flips two verdicts, in opposite directions
~80% of MB×path's apparent deficit was the control's manufactured shortcuts. This is the part that makes it convincing: a control artefact that only ever flattered the connectome could not flip a loss and a win. A generic input→output express lane can — it helps the control regardless of which side the truth is on. Direction: the shortcuts mostly HELP the control (the original hypothesis)5 of 8 cells shift negative, and both cells reaching significance do:
|
| group | cells | mean |verdict move| |
|---|---|---|
| layered (MB/CX, 22–84×) | 5 | 0.0101 |
| shallow (AL, 1.19×) | 3 | 0.0010 |
10× apart, but Mann-Whitney one-sided p = 0.071 — suggestive only. And not proportional to the ratio: MB (84×) 0.0103 ≈ CX (22×) 0.0097. The "dose-response"/"7×" framing stays dead; "layered or not" is what the data support. AL is unmoved in all three of its cells.
Method notes
- Unit of analysis is the CELL, not the seed — pooling seeds across cells would treat correlated runs as independent.
- Local (Blackwell) and fleet (L4) runs are never mixed within a cell, since a hardware difference inside a paired comparison would land in the difference. Per-cell source is recorded in
all_cells_summary.csv. - CX × path (the 9th cell) was still training at commit time.
The 9th and final cell (CX x path, run locally) completes the region x task grid: 9 cells, 162 runs. It strengthens every part of the result. CX x path -0.0206 (p=0.031) -> -0.0049 (p=0.562) another LOSS evaporates ~76% of its apparent deficit was the control's manufactured shortcuts. THREE verdicts now flip significance -- two losses and one win, spanning both layered regions and both signs: MB x mqar +0.0103 (p=0.031) -> +0.0031 (p=0.094) win evaporates MB x path -0.0267 (p=0.031) -> -0.0054 (p=0.312) loss evaporates CX x path -0.0206 (p=0.031) -> -0.0049 (p=0.562) loss evaporates A bias that merely flattered the connectome could not erase a loss AND a win. Each flip is a case where the standard comparison reports a significant result that a fair control does not support. DIRECTION is now unambiguous: 6/9 cells shift negative, and ALL THREE that reach significance do, each with 0/6 seeds dissenting (MB x path -0.0213, CX x flow -0.0185, CX x path -0.0157). The shortcuts were HELPING the control. LAYERED vs SHALLOW: 0.0110 (6 cells) vs 0.0010 (3 cells), Mann-Whitney one-sided p=0.048. Reported as MARGINAL -- 6 vs 3 units is the smallest n at which this test can clear 0.05, and it is uncorrected. STILL NOT GRADED: MB (84x) moves 0.0103 and CX (22x) moves 0.0117 -- CX moves slightly MORE despite a 4x smaller shortcut ratio. "Layered or not" is what the data support; a dose-response is not. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Collaborator
Author
Final — full 3×3 grid complete: three flipped verdicts (2250805)CX × path finished, completing the grid: 9 cells, 162 runs. It strengthens every part of the result. Three verdicts flip significance — two losses and one win
They span both layered regions and both signs. A bias that merely flattered the connectome could not erase a loss and a win — a generic input→output express lane can, because it helps whichever side the control is on. Each flip is a case where the standard comparison reports a significant result that a fair control does not support. Direction is now unambiguous6/9 cells shift negative, and all three that reach significance do — each with 0/6 seeds dissenting:
The shortcuts were helping the control — my original hypothesis, which I wrongly abandoned mid-way on the strength of the one cell that disagreed. Layered vs shallow — marginal, and still not graded0.0110 (6 cells) vs 0.0010 (3 cells), Mann-Whitney one-sided p = 0.048. I'm reporting that as marginal: 6 vs 3 units is the smallest n at which this test can clear 0.05, and it's uncorrected. Not graded: MB (84×) moves 0.0103, CX (22×) moves 0.0117 — CX moves slightly more despite a 4× smaller ratio. "Layered or not" is supported; a dose-response is not. Practical upshotFor any layered region, a degree-preserving shuffle is not a sufficient control — it should be reported alongside a shortcut-matched variant. Fleet terminated; all compute released. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A degree-preserving shuffle is the standard control in this repo. For a layered circuit it is not a fair one: rewiring at fixed degree invents direct input→output edges the real wiring forbids.
An 84× express lane lets the MB control skip the Kenyon-cell layer — skip the computation the circuit exists to perform.
This adds a
degree_smarm: the same degree shuffle, then repaired with degree-preserving double-edge swaps until its direct input→output count matches the connectome's own. Degrees are preserved exactly, so it remains a strict degree control that has simply lost its free shortcuts.AL is the built-in control for the control — its handicap is only 1.19×, so the fix should barely move AL while moving MB a lot.
Result (6 seeds, biological I/O, seed-paired)
Dose-dependent as predicted: the control moved +0.0072 on MB vs +0.0010 on AL.
But the sign is opposite to my hypothesis. I predicted the shortcuts helped the control; removing them made the control better. So those 2,680 shortcuts were hurting the degree shuffle. The connectome's MB×mqar win was partly an artifact of comparing against a sabotaged opponent — and MB×mqar was the one clean positive in the earlier region×task grid.
AL×flow is untouched.
Caveats (also in the writeup)
Fleet fixes required to run this
bootstrap.shhardcodedCUDA_VISIBLE_DEVICES=0— every worker on a multi-GPU instance piled onto GPU 0 and left the rest idle. Now spreads across all visible GPUs. (The 4-GPU instances finished this run roughly 2× faster than the 1-GPU ones.)config.env— widened the instance-type ladder for capacity resilience, kept small-first so it never silently escalates cost.🤖 Generated with Claude Code