Skip to content

feat(agg,cli,plugin): v0.1.12 — R-31–R-37 (xlsx, parquet, provenance, corr --by, category keys, agg budget) - #10

Merged
MonSp merged 3 commits into
mainfrom
feat/v0.1.12-requirements
Nov 9, 2026
Merged

MonSp merged 3 commits into
mainfrom
feat/v0.1.12-requirements

Conversation

@MonSp

@MonSp MonSp commented Nov 9, 2026 •

Copy link
Copy Markdown
Owner

Summary

v0.1.12 full requirements batch from the post-v0.1.11 lists (R-31–R-37):

  • R-36/R-37 (P0/P1 contract): agg / transform_agg always return up to 20 inline preview rows (min(20, rowCount)), with or without --out. Fixes the silent empty-rows hole when rowCount > 20 and no --out, and returns full rows for small results that also persist via --out. truncated is now simply rowCount > 20.
  • R-31: ingest --format xlsx via calamine — first sheet or --sheet <name|index> (exact sheet-name match wins, so all-digit names like 2024 stay selectable; digits fall back to the 0-based index); multi-sheet → sheet-skipped warning; corrupt/password → exit 2 actionable; .xlsm stays exit 4.
  • R-32: dataset storage moves to Parquet ({id}.parquet); legacy {id}.csv remains readable (lazy fallback); rm removes both. Internal Utf8 cells use an unambiguous encoding — strings stored as quoted JSON, scalars raw — so string values like "007" / "NA" survive save/load unchanged and meta dtypes stay consistent; exported parquet keeps raw strings (ingest round-trip unchanged).
  • R-33: ingest records sourceChecksum (sha256:…); derived sets record parentDatasetIds (join/union/diff = [left, right]; parentDatasetId stays first parent). New verify / dataset_verify command + capability token (--check-source), including fallback to parentDatasetId for legacy metas. dataset_list surfaces both fields.
  • R-34: corr --by g1[,g2…] — per-group pairwise pairs with by keys; caps 50 groups / 100 pairs + truncation warnings (both warning codes can co-occur); pairCount reports all candidate pairs (groupCount × column-pairs).
  • R-35: crosstab/pivot numeric category keys normalize whole-number floats so mixed 1 / 1.0 share one 1 label; 1.5 stays exact.

Frozen contract preserved: tool names, one-line stdout JSON, analysis/* codes, .dsh-analyze data root. Capability list appends verify.

Review rounds folded in: tool layer surfaces verify's exit-1 JSON (checks/mismatches), mock CLI kept in parity with Rust (capabilities, id validation, real row counts, source status), READMEs synced to v0.1.12.

Verification

  • cargo test -p dsh-analyze — 201/201 (64 unit + 39 agg + 35 periphery + 5 quality + 22 relation + 18 series + 18 transform)
  • tsc -p tsconfig.json — pass
  • node --test policy/tools/schema — 98/98
  • CI: CLI (ubuntu/macos/windows) + Plugin (node 20/22) all green
  • Release smoke: R-36 20-row preview; R-33 verify --check-source clean; R-32 .parquet written; R-31 xlsx round-trip 21 rows; version includes verify

Test plan

  • Rust integration tests for R-31–R-37 acceptance items
  • JS mock/tool-layer tests for R-34–R-37
  • Interface / CHANGELOG / security notes updated to v0.1.12
  • Review rounds: corr caps tests, verify mismatch/legacy-parent tests, xlsx/json string-fidelity tests, digit sheet-name test, README sync
  • Investigator e2e-v016-real-cli.mjs cases after CLI sync (follow-up, external repo)

MonSp added 3 commits November 9, 2026 15:17
…quet string fidelity, corr --by caps

- tool layer: runAnalyzeCli returns stdout JSON on non-zero exit when it parses with an ok field, so dataset_verify delivers checks/mismatches on mismatch instead of throwing a misclassified invalid-args
…back, mock/Rust parity, README sync to v0.1.12

- cmd_verify falls back to parentDatasetId when parentDatasetIds is absent (matches interface.md, load/list, and the mock); legacy-parent test covers present and missing parent
@MonSp
MonSp merged commit 468e7fb into main Nov 9, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant