fix: use chunked reading for SHA256 to prevent OOM crash on large model files#23
Open
amathxbt wants to merge 2 commits into
Open
fix: use chunked reading for SHA256 to prevent OOM crash on large model files#23amathxbt wants to merge 2 commits into
amathxbt wants to merge 2 commits into
Conversation
…models check_model_files() was loading entire model files into memory with f.read() before computing SHA256. Model files can be several gigabytes, causing out-of-memory errors on systems with limited RAM. Replaced with chunked 8KB iteration using hashlib's incremental update API, which computes the same hash while using constant memory.
check_model_files() was loading entire model files into memory before computing SHA256 checksums. Model files are often several gigabytes, causing out-of-memory crashes on systems with limited RAM. Replaced with chunked 8KB iteration using hashlib incremental update API, which produces the same hash while using O(1) memory.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug
check_model_files()indemo/nesa/download.pyreads entire model files into memory before computing their SHA256 checksum:Impact: Model files are routinely 4–70 GB. Loading one fully into RAM before hashing causes an out-of-memory crash on most consumer hardware, making integrity verification completely unusable.
Also note: the variable is named
bytes, shadowing the Python built-in.Fix
Replaced with hashlib's incremental update API using 8 KB chunks — identical hash output, O(1) memory usage: