(WIP) Modification for larges files and improve support across dtype#6
Open
frheault wants to merge 4 commits into
Open
(WIP) Modification for larges files and improve support across dtype#6frheault wants to merge 4 commits into
frheault wants to merge 4 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR updates the tractography IO utilities to better handle large/ZIP64 TRX archives and to make TRK parsing more memory-efficient, alongside a small repo hygiene change.
Changes:
- Reworked
readTRK()to parse viaDataViewand pre-size arrays (avoids over-provisioning large typed arrays). - Added ZIP central directory parsing to work around ZIP64 size issues when reading TRX; improved dtype handling/alignment and returns additional metadata (
groups,positions_dtype). - Updated
.gitignorewith Node-related ignores.
Reviewed changes
Copilot reviewed 1 out of 2 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| streamlineIO.mjs | Updates TRK/TRX readers for large-file handling, ZIP64 size correctness, and broader dtype support. |
| .gitignore | Adds Node-related ignore patterns. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+250
to
+251
| let dataView = new DataView(buffer, hdr_sz); | ||
| let totalWords = dataView.byteLength / 4; |
Comment on lines
+256
to
+261
| while (w < totalWords) { | ||
| let n_pts = dataView.getInt32(w * 4, true); | ||
| num_streamlines++; | ||
| num_points += n_pts; | ||
| w += 1 + n_pts * (3 + n_scalars) + n_properties; | ||
| } |
Comment on lines
+891
to
+898
| if (parts[0] === "dpg") { | ||
| dpg.push({ | ||
| id: parts[1] + "/" + tag, // e.g. "AF_R/volume" | ||
| fname: parts.slice(1).join("/"), // e.g. "AF_R/volume.uint32" | ||
| vals: vals.slice(), | ||
| }); | ||
| continue; | ||
| } |
Comment on lines
+786
to
+793
| function getAlignedArray(constructor, dataArray) { | ||
| const bytes = constructor.BYTES_PER_ELEMENT; | ||
| if (dataArray.byteOffset % bytes === 0) { | ||
| return new constructor(dataArray.buffer, dataArray.byteOffset, dataArray.byteLength / bytes); | ||
| } else { | ||
| return new constructor(dataArray.slice().buffer); | ||
| } | ||
| } |
Comment on lines
+800
to
+810
| const fd = fs.openSync(url, 'r'); | ||
| const chunkSize = 512 * 1024 * 1024; | ||
| let offset = 0; | ||
| while (offset < size) { | ||
| const bytesToRead = Math.min(chunkSize, size - offset); | ||
| const bytesRead = fs.readSync(fd, uint8Array, offset, bytesToRead, offset); | ||
| if (bytesRead === 0) break; | ||
| offset += bytesRead; | ||
| } | ||
| fs.closeSync(fd); | ||
| data = Buffer.from(arrayBuffer); |
…n to vertices during export
…e corrections for valid TRK export
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Disclaimer: The modifications to the code were generated using antigravity-cli using the gemini-3.5-pro model and were double-checked by me and thoroughly tested on various datasets. This is more of an draft exploration to allow benchmarking across languages [https://github.com/tee-ar-ex/trx-manuscript-2026-benchmark](see here).
When I tried to benchmark the JS implementation on large file OR with files from other languages (rust, cpp, etc.), I hit a few errors. For example, very large TRK were hitting V8's heap constraints so I tried to break the file into chunks to allow reading a full TRK, our trx rust implementation saved all internal files of the ZIP as ''large_file'' ZIP64 which was breaking the ZIP header metadata (leading to incorrect coordinates, offsets, headers, etc.).
So these are observations that lead to proposed fixes (but since I do not program in JS I believe an expert eye would help a lot). At least I can guarantee that single language round-trip (load/save) and between language compatibility are now working. Then I benchmarked on big files (not too big to break everything), and my modifications do not seem to be problematic in terms of resources.
@neurolabusc