Bundle Node entrypoint with ncc to slim install footprint - #10
Open
collin-rogers wants to merge 1 commit into
Open
Bundle Node entrypoint with ncc to slim install footprint#10collin-rogers wants to merge 1 commit into
collin-rogers wants to merge 1 commit into
Conversation
Installing cyberchef-node currently unpacks a very large node_modules tree (~16k files). On network-backed disks the per-file round-trips dominate install time (installs of consuming integrations have been observed at ~9 min). Bundle the pure-JS runtime dependencies into a single file with @vercel/ncc so the published package ships ~11 files instead of thousands, cutting the shipped file count by ~98% with no change in behavior. - Add scripts/bundle.js: runs ncc against the generated src/node/index.js, emitting dist/index.js (+ lazy chunks). WebAssembly packages (tesseract.js, tesseract.js-core, libyara-wasm, libbzip2-wasm, argon2-browser) cannot be inlined by ncc/webpack, so they are kept external and remain the package's only runtime dependencies. - package.json: point main at dist/index.js; split build into generate + bundle; add prepack so the bundle is produced automatically on npm pack/publish (release workflow needs no change). Move the now inlined runtime deps to devDependencies; only the WASM packages remain in dependencies. - package-lock.json: regenerated. The full dependency graph is preserved (378 packages before and after) so security scanning against the lockfile is unaffected; deps only move between dependencies and devDependencies. - .npmignore: publish dist/ instead of src/tests/scripts. - .gitignore: ignore generated dist/. Verified: the packed tarball with only the 4 WASM deps installed loads all 408 operations and runs both non-WASM and WASM-backed ops (YARA); bundle output matches source for sampled operations. Pre-existing crypto-api resolution failures (MD5/SHA2) are unchanged by this commit.
Author
|
This pr is a precursor to the work to be done for INT-2074 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Installing
cyberchef-nodeunpacks a very largenode_modulestree (~16k files, ~183M). On network-backed disks the per-file round-trips dominate install time — installs of consuming integrations have been observed at ~9 minutes. The file count is the lever, not total bytes.Change
Bundle the pure-JS runtime dependencies into a single file with
@vercel/nccat build/publish time. The published package now ships ~11 files instead of thousands.What's in the PR
scripts/bundle.js— runs ncc against the generatedsrc/node/index.js, emittingdist/index.js(+ ncc lazy chunks).tesseract.js,tesseract.js-core,libyara-wasm,libbzip2-wasm, andargon2-browsership WASM that ncc/webpack cannot inline. They remain the package's only runtimedependenciesand resolve fromnode_modulesat runtime.package.json—main→dist/index.js;buildsplit intogenerate+bundle; addedprepackso the bundle is produced automatically onnpm pack/npm publish(release workflow needs no change). Inlined runtime deps moved todevDependencies; only the 4 WASM packages remain independencies.package-lock.json— regenerated. Full dependency graph preserved (378 packages before and after) so security scanning against the lockfile is unaffected; deps only move betweendependencies/devDependencies..npmignore— publishdist/instead ofsrc/tests/scripts..gitignore— ignore generateddist/.Verification
crypto-apitest failures (analyzeHash,HMAC,Snefru) and the MD5/SHA2has160resolution issue exist identically onmaster— not introduced by this change.Notes / follow-ups
tesseract.jsOCR closure is the largest remaining external. If OCR isn't required downstream, dropping it would shrink the external closure further — separate change.