Repository navigation
feat(feed): compact json and holdings and history rows as arrays - #97
Merged
Merged
Conversation
The feed was 788 MiB of the 1 GB GitHub Pages site limit. Every file the updater writes is now compact JSON, and holdings and history pages carry rows as string arrays in headers order instead of objects that repeat every header. The app reads both row shapes.
One-off rewrite of the published files with the updater's feedJson and pageRows: the same data, 788 MiB -> 255 MiB.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
scripts/update-data.ts: every file the updater writes (index.json,meta.json, holdings and history pages,update-state.json) goes throughfeedJson(compact JSON plus a trailing newline); holdings and history pages carryrowsas string arrays inheadersorder (pageRows) instead of objects keyed by the headersrc/main.tsx:fetchPagereads array rows and still reads object rowsscripts/update-data.test.ts: the zero-diff pipeline test also checks that every written file is one line and that holdings and history pages carry array rows in header orderapi/ishares: all 5,523 files rewritten into the new format (second commit)Why
ETF-H3: the feed was 788 MiB of the 1 GB GitHub Pages site limit and grows weekly; at the limit the Pages deploy fails. Indentation was 28% of it, object rows repeating every header text another 40%
Sizes
api/(bytes of the 5,523 JSON files)api/on disk (du -sk)dist/(du)Data rewrite
A one-off Bun script (kept out of the repo, in the ignored
node_modules/.work/) importsfeedJsonandpageRowsfrom the updater and rewrites every JSON file ofapi/isharesthrough them (tmp file + rename). Before writing it checks that each file is exactlyJSON.stringify(_, null, 2) + "\n"output (so the compact form is a lossless round trip), that page headers are unique and that no row has a key outsideheaders(0 problems over 5,039 pages and 2,314,923 rows). A second script compared the old tree (git archive HEAD api) with the new one: same file list, non-page files deep-equal, page rows read by the new reader equal the old rows read by the old reader for all 5,523 filesConsumers (all checked)
src/main.tsxfetchPage): reads both shapes (this PR)fetchPage): reads both shapes since feat(sheets): read holdings and history rows published as arrays ETFs#86, merged and deployed to Pages before this PR/ETFs/<Brand>/api/**to files, does not parse themui-stdbrowser tests read the normalized rows of the apps, not the filesHow it was checked
bun install -E,bun test(36 pass),bun build --target=bun scripts/update-data.ts --outfile=/dev/null,bun build --target=bun src/main.tsx --outfile=/dev/null,bun run build,bun run build-github-pages,git diff --check.claude/tools/etfs/tc/check.sh: the same 53 pre-existing lines as on main, no new errorui-std/brand-uitest.tson the builtdist(holdings and history tabs load the array pages):44 passed, 0 failed; console errors: noneTICKERS=AAXJ,IVV,AGG REQUEST_SLEEP=0 bun scripts/update-data.tson the rewritten feed:3 unchanged, zero git diff (the writer produces the rewritten bytes exactly,index.jsonincluded)investments/scripts/serve.ts,/ETFs/?api=local, headless Chrome): AAXJ, AGG and IVV holdings and history cells equal the files,29 passed, 0 failed; console errors: none