Repository navigation
perf(training): hash snapshots in chunks instead of reading them whole - #31
Merged
Merged
Conversation
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 54 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
ng
force-pushed
the
feat/archive-raw-inputs
branch
from
November 16, 2026 00:12
d86a109 to
0f2e3de
Compare
ng
force-pushed
the
feat/streaming-snapshot-hash
branch
from
November 16, 2026 00:13
a89142b to
a0a2250
Compare
ng
force-pushed
the
feat/archive-raw-inputs
branch
from
November 16, 2026 00:17
0f2e3de to
0b0d168
Compare
ng
force-pushed
the
feat/streaming-snapshot-hash
branch
from
November 16, 2026 00:20
a0a2250 to
49346ec
Compare
ng
force-pushed
the
feat/streaming-snapshot-hash
branch
from
November 16, 2026 00:26
49346ec to
daef3c9
Compare
…e is never read into memory whole Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ng
force-pushed
the
feat/streaming-snapshot-hash
branch
from
November 16, 2026 00:31
daef3c9 to
6c1d5af
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #28 (both touch
movement_model.py). Every stage script checks the--snapshotSHA-256 againstfeatures.json. They did it withread_bytes(), which loads the whole database. The off-pod archive database never deletes rows; it is already 117 MB and grows with every pull.Summary
scripts/hashing.pywithfile_sha256(path): SHA-256 in 4 MiB chunks, same digest.feature-ablation.py,fusion.py,stage-backtest.py,vitals_reference.pyandmovement_model.py.movement_model.pyalso uses it for the raw export tarball andmodel.joblib, which are tens of MB each.archive_index.pyreuses the helper in place of its own copy.export-runtime-features.tswrites thesources.snapshothash those checks compare against. It now reads in 4 MiB chunks too (openSync/readSync) instead ofreadFileSync.Test plan
scripts/test_hashing.py: the chunked hash equalshashlib.sha256(read_bytes())across chunk boundaries (empty, partial and exact multiples).sources.snapshotwritten byexport-runtime-features.tsequalsfile_sha256and the whole-file hash. Max RSS was about 100 MB.feature-ablation.py --snapshotaccepts the TS-written hash.data/tmpdir (214 unit tests).🤖 Generated with Claude Code