Skip to content

feat(feed): compact json and holdings and history rows as arrays - #97

Merged
daggerok merged 2 commits into
mainfrom
feat/compact-array-pages
Oct 26, 2026
Merged

daggerok merged 2 commits into
mainfrom
feat/compact-array-pages

Conversation

@daggerok

Copy link
Copy Markdown
Owner

What

  • scripts/update-data.ts: every file the updater writes (index.json, meta.json, holdings and history pages, update-state.json) goes through feedJson (compact JSON plus a trailing newline); holdings and history pages carry rows as string arrays in headers order (pageRows) instead of objects keyed by the header
  • src/main.tsx: fetchPage reads array rows and still reads object rows
  • scripts/update-data.test.ts: the zero-diff pipeline test also checks that every written file is one line and that holdings and history pages carry array rows in header order
  • README feed layout describes the page shape
  • api/ishares: all 5,523 files rewritten into the new format (second commit)

Why

ETF-H3: the feed was 788 MiB of the 1 GB GitHub Pages site limit and grows weekly; at the limit the Pages deploy fails. Indentation was 28% of it, object rows repeating every header text another 40%

Sizes

before after
api/ (bytes of the 5,523 JSON files) 787.6 MiB 254.5 MiB (-67.7%)
api/ on disk (du -sk) 817,520 KB 271,772 KB
built dist/ (du) 799 MiB 266 MiB (271,952 KB)

Data rewrite

A one-off Bun script (kept out of the repo, in the ignored node_modules/.work/) imports feedJson and pageRows from the updater and rewrites every JSON file of api/ishares through them (tmp file + rename). Before writing it checks that each file is exactly JSON.stringify(_, null, 2) + "\n" output (so the compact form is a lossless round trip), that page headers are unique and that no row has a key outside headers (0 problems over 5,039 pages and 2,314,923 rows). A second script compared the old tree (git archive HEAD api) with the new one: same file list, non-page files deep-equal, page rows read by the new reader equal the old rows read by the old reader for all 5,523 files

Consumers (all checked)

  • iShares app (src/main.tsx fetchPage): reads both shapes (this PR)
  • ETFs hub (fetchPage): reads both shapes since feat(sheets): read holdings and history rows published as arrays ETFs#86, merged and deployed to Pages before this PR
  • investments shell: only maps /ETFs/<Brand>/api/** to files, does not parse them
  • ui-std browser tests read the normalized rows of the apps, not the files
  • the other 28 brands are untouched and keep object rows

How it was checked

  • bun install -E, bun test (36 pass), bun build --target=bun scripts/update-data.ts --outfile=/dev/null, bun build --target=bun src/main.tsx --outfile=/dev/null, bun run build, bun run build-github-pages, git diff --check
  • .claude/tools/etfs/tc/check.sh: the same 53 pre-existing lines as on main, no new error
  • ui-std/brand-uitest.ts on the built dist (holdings and history tabs load the array pages): 44 passed, 0 failed; console errors: none
  • live run TICKERS=AAXJ,IVV,AGG REQUEST_SLEEP=0 bun scripts/update-data.ts on the rewritten feed: 3 unchanged, zero git diff (the writer produces the rewritten bytes exactly, index.json included)
  • hub in local mode over this feed (investments/scripts/serve.ts, /ETFs/?api=local, headless Chrome): AAXJ, AGG and IVV holdings and history cells equal the files, 29 passed, 0 failed; console errors: none

The feed was 788 MiB of the 1 GB GitHub Pages site limit. Every file the
updater writes is now compact JSON, and holdings and history pages carry
rows as string arrays in headers order instead of objects that repeat
every header. The app reads both row shapes.
One-off rewrite of the published files with the updater's feedJson and
pageRows: the same data, 788 MiB -> 255 MiB.
@daggerok
daggerok merged commit 6eeb1df into main Oct 26, 2026
1 check passed
@daggerok
daggerok deleted the feat/compact-array-pages branch October 26, 2026 05:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant