Skip to content

perf(training): hash snapshots in chunks instead of reading them whole - #31

Merged
ng merged 2 commits into
mainfrom
feat/streaming-snapshot-hash
Nov 16, 2026
Merged

ng merged 2 commits into
mainfrom
feat/streaming-snapshot-hash

Conversation

@ng

@ng ng commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Stacked on #28 (both touch movement_model.py). Every stage script checks the --snapshot SHA-256 against features.json. They did it with read_bytes(), which loads the whole database. The off-pod archive database never deletes rows; it is already 117 MB and grows with every pull.

Summary

  • New scripts/hashing.py with file_sha256(path): SHA-256 in 4 MiB chunks, same digest.
  • Snapshot checks now use it in feature-ablation.py, fusion.py, stage-backtest.py, vitals_reference.py and movement_model.py. movement_model.py also uses it for the raw export tarball and model.joblib, which are tens of MB each.
  • archive_index.py reuses the helper in place of its own copy.
  • export-runtime-features.ts writes the sources.snapshot hash those checks compare against. It now reads in 4 MiB chunks too (openSync/readSync) instead of readFileSync.
  • Small JSON and script hashes are unchanged.

Test plan

  • scripts/test_hashing.py: the chunked hash equals hashlib.sha256(read_bytes()) across chunk boundaries (empty, partial and exact multiples).
  • On a copy of the real 117 MB archive database, the sources.snapshot written by export-runtime-features.ts equals file_sha256 and the whole-file hash. Max RSS was about 100 MB.
  • The smoke run's feature-ablation.py --snapshot accepts the TS-written hash.
  • Every command in README "Validate without personal data" passed, with smoke outputs in a fresh data/tmp dir (214 unit tests).

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 54 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 36e41844-dd8d-41e9-9c85-f396b61cf9fc
Commits

Reviewing files that changed from the base of the PR and between 207ce8c and 8dbed62.

Files selected for processing: 9

  • scripts/archive_index.py
  • scripts/export-runtime-features.ts
  • scripts/feature-ablation.py
  • scripts/fusion.py
  • scripts/hashing.py
  • scripts/movement_model.py
  • scripts/stage-backtest.py
  • scripts/test_hashing.py
  • scripts/vitals_reference.py
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@ng
ng force-pushed the feat/archive-raw-inputs branch from d86a109 to 0f2e3de Compare November 16, 2026 00:12
@ng
ng force-pushed the feat/streaming-snapshot-hash branch from a89142b to a0a2250 Compare November 16, 2026 00:13
@ng
ng force-pushed the feat/archive-raw-inputs branch from 0f2e3de to 0b0d168 Compare November 16, 2026 00:17
@ng
ng force-pushed the feat/streaming-snapshot-hash branch from a0a2250 to 49346ec Compare November 16, 2026 00:20
@ng
ng changed the base branch from feat/archive-raw-inputs to main November 16, 2026 00:22
@ng
ng force-pushed the feat/streaming-snapshot-hash branch from 49346ec to daef3c9 Compare November 16, 2026 00:26
…e is never read into memory whole

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ng
ng force-pushed the feat/streaming-snapshot-hash branch from daef3c9 to 6c1d5af Compare November 16, 2026 00:31
@ng
ng merged commit 0eb0702 into main Nov 16, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant