Degree control that also matches the connectome's shortcut count

.claude/skills/README.md

@@ -0,0 +1,140 @@
+# Experiment skills — how to use them
+
+This project ships four **skills**: reusable instruction sets that tell Claude Code (or
+another agent) how to do a specific job the way this project wants it done. Together they
+cover the full life of an experiment — from first idea, to scaffolding, to running on GPUs,
+to writing up the result — so the work stays organized and the record stays trustworthy.
+
+You don't need to memorize them. Just know the four jobs and let the table below tell you
+which skill does which.
+
+## Where your work lives — your own folder
+
+Each person works inside **their own top-level folder** in the repo (for example `scott/`),
+which keeps everyone's experiments and notebooks cleanly separated. Your folder holds your
+experiments root and your lab notebook:
+
+```
+<yourname>/
+├── labnotebook/                     ← your lab notebook + its index (README.md)
+├── experiment_01_<slug>/            ← one folder per experiment
+└── experiment_02_<slug>/
+
+scott/aws_fleet/                     ← shared GPU-fleet harness, used by everyone (not per-user)
+```
+
+If you're new, create `<yourname>/` (mirror the layout in `scott/`) before your first
+experiment — or just tell `/build-experiment` and it will set it up in your folder. The
+skills work the same inside any user's folder; they key off the structure, not the name.
+The GPU-fleet harness in `scott/aws_fleet/` is shared infrastructure — your experiment's
+`run.py` drives it through a generated config without editing the shared files.
+
+## The skills
+
+| Skill | What it does | When to reach for it | How to invoke |
+|---|---|---|---|
+| **neuroresearch** | Acts as a rigorous, skeptical neuroscientist. Reviews a design or audits a result — checks the experiment actually tests its claim, finds confounds, proposes fixes. | Before building anything (pressure-test an idea), or after a run (audit whether the conclusion holds). Also fine for pure exploration before any structure exists. | `/neuroresearch` |
+| **build-experiment** | Sets up and then *guards* an experiment's folder + records. Creates the directory, the frozen `run.py`, the lab-notebook entry + index row, and enforces the organization rules for the rest of the chat. | When you're ready to start a new experiment — even if the design isn't final. | `/build-experiment` |
+| **aws-fleet** | Operates the spot-GPU training fleet: launch, monitor, collect results, tear down — with cost and teardown guardrails. | When you want to run an experiment on AWS, check a running fleet, collect results, or stop one. | `/aws-fleet` |
+| **labnotebook** | Writes a dated, structured entry (purpose / methods / results) in the lab notebook and keeps the index current. | At experiment kickoff (purpose + plan), along the way, and at the end (add results). | `/labnotebook` |
+
+## The expected workflow
+
+Experiments here follow one loop. Each arrow is a place you invoke a skill. It's **human-in-
+the-loop on purpose** — you drive each step; nothing runs unattended.
+
+```
+   idea / question
+        │
+        ▼
+ [1] /neuroresearch ──►  review the design: is it fair, rigorous, testing the real claim?
+        │
+        ▼
+ [2] /build-experiment ─►  scaffold the folder + frozen run.py, seed the lab-notebook entry
+        │
+        ▼
+ [3] /aws-fleet ───────►  launch on GPUs, monitor, collect results, tear down
+        │
+        ▼
+ [4] /labnotebook ─────►  write up the result; update the index
+        │
+        ▼
+ [1] /neuroresearch ──►  audit the result: does the conclusion actually hold?  → next experiment
+```
+
+Step by step, for a naive user:
+
+1. **Sharpen and review the idea — `/neuroresearch`.**
+   Describe what you want to test. Claude will pin down the question, the competing
+   hypotheses, and the controls you'd need, and flag confounds *before* you spend effort.
+   You can use this on a rough sketch with no code yet — that's encouraged.
+
+2. **Scaffold the experiment — `/build-experiment`.**
+   When the design is good enough to build (it doesn't have to be final), invoke this.
+   Claude creates `experiment_NN_<slug>/` with a `run.py` (all parameters pinned as
+   constants — this becomes the permanent record of what was run), a short README, and the
+   output/figure folders, and it seeds the lab-notebook entry + index row with whatever is
+   decided so far. From here on in the conversation, Claude will **enforce** the rules:
+   one frozen `run.py` per experiment, subruns inside it, data kept in the right place.
+
+3. **Run it — `/aws-fleet`.**
+   When you're ready to train, invoke this to launch on the spot-GPU fleet. Claude will
+   estimate the cost and ask you to confirm *before* spending, smoke-test small first,
+   monitor progress, pull results with `--collect`, and tear the fleet down so it stops
+   costing money. (First-time AWS account setup is a separate one-time thing — see
+   `scott/aws_fleet/SETUP.md`.)
+
+4. **Write it up — `/labnotebook`.**
+   Once results are in and sanity-checked, invoke this to add the results to the notebook
+   entry (with the key figures and headline numbers) and update the index. The entry and
+   the code point at each other, so anyone can trace a claim back to the run that produced it.
+
+5. **Audit, then iterate — `/neuroresearch` again.**
+   Before you trust a result, review it: does the lab-notebook claim match what `run.py`
+   actually launched and what the numbers actually show? The open questions it surfaces
+   become your next experiment, and the loop repeats.
+
+You don't have to do all four every time. Reviewing an old result? Just `/neuroresearch`.
+Logging a run someone else did? Just `/labnotebook`. The loop is the *full* path; use the
+piece you need.
+
+## How invoking a skill actually works
+
+- **In Claude Code:** type the slash command (e.g. `/build-experiment`) in the prompt, or
+  just describe the task in plain language ("set up a new experiment for X") — Claude
+  recognizes the intent and loads the matching skill. The slash command is the explicit
+  way; describing the task is the natural way. Both work.
+- **With another agent / tool:** each skill is just a Markdown file at
+  `.claude/skills/<name>/SKILL.md`. If your agent doesn't support slash-command skills,
+  point it at that file and tell it to follow it — the instructions are plain text.
+
+## Troubleshooting
+
+- **Claude can't see the skill (`/build-experiment` isn't recognized).**
+  These are **project skills** — they live in `.claude/skills/` inside this repository, and
+  Claude Code only loads them when it's launched from within the project. The usual cause is
+  **launching Claude from the wrong directory** (e.g. your home folder). Fix: quit, `cd` into
+  the repo, and start Claude there:
+  ```bash
+  cd /path/to/pathintegrationBPU
+  claude
+  ```
+  Then run `/help` (or start typing `/`) and confirm the four skills appear in the list.
+
+- **The skill name doesn't autocomplete.** Check the exact name — they are `neuroresearch`,
+  `build-experiment`, `aws-fleet`, `labnotebook` (hyphens, not spaces). Each has its own
+  folder under `.claude/skills/` with a `SKILL.md` inside; if a folder or its `SKILL.md` is
+  missing or misnamed, the skill won't register.
+
+- **Claude did the task but ignored the project's rules** (wrong folder, edited a frozen
+  `run.py`, skipped the notebook). It probably didn't load the skill. Invoke the skill
+  explicitly with its slash command rather than relying on intent detection, and confirm
+  you're in the repo (previous point).
+
+- **The fleet won't launch / no instances appear.** That's an AWS operations issue, not a
+  skill issue — invoke `/aws-fleet` and see its troubleshooting section, or
+  `scott/aws_fleet/SETUP.md` for first-time account setup.
+
+- **Want to change how a skill behaves.** Edit its `SKILL.md` directly — it's plain Markdown.
+  The top `description:` line controls *when* Claude reaches for the skill; the body is *what*
+  it does.

.claude/skills/aws-fleet/SKILL.md

@@ -0,0 +1,103 @@
+---
+name: aws-fleet
+description: Operate and troubleshoot the scott/aws_fleet spot-GPU training fleet — launch, monitor, collect results, and tear down — with cost, auth, and teardown guardrails. Use when the user wants to run an experiment on the AWS fleet, check on a running fleet, collect/analyze results, stop a fleet, or debug one that is stuck, preempted, or costing money. NOT for first-time AWS account setup (see scott/aws_fleet/SETUP.md) and NOT for per-experiment run parameters (those live in each experiment's run.py — see the build-experiment skill).
+---
+
+# AWS fleet — operate & troubleshoot
+
+`scott/aws_fleet/` is a scripted spot-GPU harness: the laptop orchestrates, training
+runs on cheap preemptible EC2 GPUs, results sync to S3. Your job here is to **drive a
+run safely and diagnose it when it misbehaves** — not to re-document the harness.
+
+**Scope boundaries — defer, don't duplicate:**
+- **First-time AWS setup** (region, AMI, IAM/keys, security group, filling `config.env`)
+  → `scott/aws_fleet/SETUP.md` and `README.md`. Point the user there; offer to walk a
+  step *with* them, but don't reinvent it.
+- **What a given experiment runs** (epochs, seeds, lr grid, fleet size, S3 prefix) →
+  that experiment's `run.py`, governed by the **`build-experiment`** skill. `run.py` is
+  the frozen record and the normal entry point; the fleet scripts are what it drives.
+
+## The two ways the fleet is driven
+
+1. **Through an experiment's `run.py`** (the normal path). `run.py` pins all params,
+   generates a run-specific `fleet_config.env`, and points the scripts at it via the
+   `FLEET_CONFIG` env var so the shared `config.env` is untouched. Flags:
+   `(bare)` stage+launch · `--status` · `--log` · `--collect` · `--stop`. Prefer this.
+2. **The raw scripts in `scott/aws_fleet/`** (lower level, for the shared default config
+   or debugging). Every script sources `${FLEET_CONFIG:-config.env}`, so export
+   `FLEET_CONFIG=/path/to/experiment/fleet_config.env` first to operate on a specific
+   run; otherwise they act on `config.env`.
+
+## The operational loop
+
+| Step | Via run.py | Raw script | What it does |
+|---|---|---|---|
+| Stage | `run.py` (bare) | `./stage_data.sh` | Tar the **current working tree** (tracked+untracked, respects `.gitignore`; captures uncommitted edits) + substrate + config → S3. **Re-run after any code/param change.** |
+| Launch | `run.py` (bare) | `./launch_fleet.sh` | Start `FLEET_SIZE` instances, each runs `plan[k::N]`. |
+| Monitor | `run.py --status` / `--log` | `./status.sh`, `./watch.sh [-f\|logs\|full]` | Live instances + S3 `result.json` count. `watch.sh -f` follows; `watch.sh logs` shows the per-worker frontier. No SSH. |
+| Collect | `run.py --collect` | `./collect.sh` | `s3 sync` outputs → experiment `outputs/`, then `run_experiment.py --analyze-only` → `metrics_by_run.csv` + `analysis.json`. Safe **anytime**, even mid-run, for a partial peek. |
+| Tear down | `run.py --stop` | `./stop.sh` | Terminate every `project=pathint` instance now. |
+
+Staging + launch are the only spend-incurring steps. `--status`/`--log`/`--collect`/
+`--stop` never launch anything. Launch is **idempotent + per-epoch checkpointed**, so
+re-launching after preemption tops up: finished runs skip, partial ones resume.
+
+## Guardrails — hold these whenever operating the fleet
+
+- **Confirm spend before launching.** State the rough cost first: `g6.xlarge` spot is
+  ~$0.30–0.50/GPU-hr; **total compute cost is ~flat in fleet size — a bigger fleet just
+  buys wall-clock**, not a bigger bill. Never launch a non-trivial fleet without the
+  user's explicit go-ahead. The bare `run.py` already prompts to confirm — don't pass
+  `--yes` unless the user has clearly approved the spend.
+- **Smoke-test first.** Validate the pipeline with `run_experiment.py --smoke` (seconds,
+  no GPU/download) and/or `FLEET_SIZE=1` before scaling up. Don't launch a big fleet as
+  the first test of new code.
+- **Always be able to stop the spend.** Workers self-terminate on shard completion
+  (`AUTO_SHUTDOWN=true`), so there's normally no idle bill — but **verify with
+  `--status`/`status.sh`** rather than assuming. If anything is wrong, the immediate move
+  is `--stop`/`stop.sh` (results in S3 are kept; relaunch resumes). Note `stop.sh`
+  terminates **all** `project=pathint` instances, not just one experiment's.
+- **Don't leave a fleet unattended without telling the user how to check/stop it** —
+  give them the `--status` and `--stop` commands.
+- **Re-stage after edits.** `stage_data.sh` snapshots the working tree; if code or params
+  changed and you didn't re-stage, the fleet runs the *old* code. Always stage before
+  relaunch when anything changed.
+- **Keys-on-box auth (default here).** With `IAM_INSTANCE_PROFILE` blank, `launch_fleet.sh`
+  injects your local AWS access keys into each instance's user-data — never written to
+  `config.env`, the tarball, or S3, only into your own short-lived instances' boot script.
+  If an IAM instance profile is set instead, keys are skipped automatically. Don't put
+  secrets in `config.env` (it's sourced on the workers).
+
+## Troubleshooting
+
+- **Spot capacity / "InsufficientInstanceCapacity".** Expected; `launch_fleet.sh` walks
+  `INSTANCE_TYPES` (g6→g5→g4dn), spot then on-demand, until one launches. If all fail,
+  retry later or widen `INSTANCE_TYPES`/region.
+- **Instances vanished early / preemption.** Spot reclaim is normal and safe. Just
+  relaunch (`run.py` bare, or `stage_data.sh && launch_fleet.sh`): finished runs skip,
+  partial ones resume from the last per-epoch S3 checkpoint (~≤1 epoch lost, synced every
+  `SYNC_INTERVAL_S`=60s).
+- **`result.json` count stuck.** Check `status.sh` for live instances. If instances are up
+  but the count isn't climbing, use `watch.sh logs` for each worker's latest line (boot
+  console only shows during early boot, before workers log). If no instances and count <
+  plan, relaunch to finish the remainder.
+- **No instances ever appear.** Usually staging/launch config: confirm `aws sts
+  get-caller-identity` works, `stage_data.sh` succeeded, and `AMI_ID`/region match. See
+  `SETUP.md` Part A.
+- **`--output table` hangs in a pager.** `AWS_PAGER=""` is set in `config.env`; if calling
+  the AWS CLI by hand, export it too.
+- **No SSH by default** (`SECURITY_GROUP_ID` blank = outbound only). Debug via `watch.sh`,
+  not SSH, unless the user set up a security group + keypair (SETUP.md A4/A5).
+
+## When invoked
+
+1. Identify which run: an experiment's `run.py` (preferred — operate through it) or the
+   raw scripts with `FLEET_CONFIG` exported. Read the relevant `run.py`/`config.env` to
+   get the actual params; don't guess fleet size or cost.
+2. For launching: smoke/`FLEET_SIZE=1` check → state the cost → get explicit approval →
+   stage → launch. For monitoring/collecting/stopping: just run the right read-only or
+   teardown command.
+3. After a run finishes and is collected, hand back to **`build-experiment`** /
+   **`labnotebook`**: results land in `outputs/`, then the notebook entry + index get the
+   numbers and key figures. Keep the spend stopped (`--status` to confirm nothing's left
+   running).