v0.6.0 — web parity + analysis depth (Features A–I) - #2
Merged
Conversation
Add web/column_store.js (JS port of python store.ColumnStore) holding ~26 analysis channels as packed Float32Array columns + Float64Array timestamps instead of 590K record objects (~1.6 GB -> ~55 MB on the 7-day file). Add parser.js: - parseTrendColumnar: one-shot ArrayBuffer -> Transferable typed-array columns - parseTrendColumnarStream: chunked Blob.slice path so the full 438 MB ArrayBuffer is never resident; record-aligned 8 MB chunks - decodeColumnarSlice / allocColumns helpers Wire parser_worker.js to support parse-stream (Blob) and columnar parse, transferring the typed-array buffers back zero-copy. Legacy parseTrendBin path retained for small-file/CSV flows and existing tests. Tests: web/tests/columnar.test.js asserts record count + value parity with the legacy parse, streaming==one-shot, bounded per-chunk reads, and ColumnStore round-trips. Web suite 82 -> 87 green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
App now streams the dropped File through the worker (parse-stream), holds a resident ColumnStore (~62 MB) instead of 590 K record objects, and runs event / snapshot / insight detection + charts + range + tariff straight off the store. currentRecords stays null on the streaming path; exports materialise a transient records array via recordsForExport() and drop it. Make the analysis engines store-aware via web/column_source.js: - events.js, snapshots.js, insights.js accept a ColumnStore OR a records array - plots.js buildPlotData reads columns from a store (charts memory-bounded) - range_select.js renderRangeSelector + tariff.js computeCost read the store Fix a latent stack-overflow: ruleCurrentSpikeRatio / ruleBreakerMargin used Math.max(...valid) which blows the call stack on a ~590 K-element session; replaced with a loop-based arrayMax. This would have crashed the web app on the 7-day file. Validated headlessly on the real ES.004 (438 MB, 589,877 recs): streaming parse + full analysis peaks at ~278 MB RSS (vs the old ~1.6 GB), parses in ~0.5 s, analysis ~2 s, 18 events / 3 snapshots / 2 findings — all correct. Tests: store==records equivalence for events/snapshots/insights added to columnar.test.js. Web suite 87 -> 88 green; python 176 green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add web/analysis.js mirroring python analysis.py numerically: - RunningMoments (Welford), PercentileSketch (fixed-width histogram) - wholeSessionStats (per-channel count/min/p1/p5/median/mean/p95/p99/max/stdev + under-voltage / over-current second accounting) - classifyItic / eventItic (ITIC/CBEMA ride-through) - parsePeriod / parseTodWindow / timeOfDayProfile - correlateMarkers All accept a ColumnStore or records array via column_source. Surface in the web UI: a Statistics section (per-channel table + threshold note) and a time-of-day profile chart, computed off the resident store. Embed the whole-session stats table (and a narrative slot for Feature E) into the exported HTML report. Parity: python test_analysis_parity_golden.py emits a deterministic golden JSON; web/tests/analysis_parity.test.js recreates the identical session and asserts stats/ITIC/eventItic/ToD match within float tolerance (percentiles are exact since both use the same sketch). Web 88 -> 92 green; python 176 -> 177 green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A load wired with backwards iFlex CTs reads as a persistent generator (P_total < 0). Detect when real power is negative for >= 50% of non-outage time (default threshold; non-finite P samples skipped) and emit a loud notice. Python: - analysis.detect_ct_reversal + ct_reversal_notice - cli --auto-reverse-cts: when flagged and no explicit --reverse-cts, re-run the single-pass parse with reverse-CTs applied and report the corrected stats - always print the notice when flagged; write ct_reversal.json for reports/web Web (parity): - analysis.detectCtReversal / ctReversalNotice (identical algorithm) - a dismissable banner in the summary with an "Apply Reverse CTs" button that ticks all phase boxes and re-parses Validated on the real ES.004: flagged True, 52.05% of non-outage time negative, mean P = -37.0 kW (the raw load reads as export until the CTs are flipped). Tests: python test_ct_reversal.py (5) + golden CT entry; web parity test. Python 177 -> 182; web 92 -> 93 green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fluke-analyze stitch S1 S2 ... -o OUT concatenates consecutive sessions into one continuous timeline that beats the meter's 7-day cap, then runs the normal analysis over the stitched series. Python: - stitch.py: stitch_stores() sorts labelled ColumnStores by start time, concatenates retained columns + absolute ticks, records boundary gaps beyond a tolerance (no synthetic fill rows), and carries per-source provenance (label, record range, time span). - cli_stitch.py: the `stitch` subcommand (sibling to `compare`) — parses each input to a store, stitches, writes stitch.json provenance, events/insights/ stats over the stitched series, and a session.csv with a `source` column. - wired into cli.main dispatch. Web: - multi_session.js: stitchStores() (JS port) + MultiSession.buildStitched() for a stitched timeline alongside the existing compare overlay. Validated on the real consecutive captures ES.001 (5,024 recs) + ES.002 (74,873 recs): stitched to 79,897 continuous records with a correctly detected 801.8 s gap; full analysis ran clean over the stitched series. Tests: python test_stitch.py (8, incl. CLI e2e); web multi_session stitch parity (5). Python 182 -> 190; web 93 -> 98 green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rule-based, deterministic plain-English summary (no LLM) built from events + insights + whole-session stats + the CT-reversal check: scope, headline event (worst outage/dip/swell with leading-dip context), PF/imbalance, a CT-reversal warning when flagged, and a bottom line. Python: - narrative.py: build_narrative() + narrative_markdown() - cli writes narrative.md, prepends the summary to summary.txt, and threads it into the HTML report (Executive summary section) and the XLSX Summary sheet. Web (parity): - narrative.js (identical prose) + an Executive summary section in the UI, and the narrative embedded at the top of the exported HTML report. Parity: golden narrative built from fixed events/findings/stats/ct in the Python generator; web/tests asserts buildNarrative matches it character-for- character. Tests: python test_narrative.py (6); web narrative parity (2). Python 190 -> 196; web 98 -> 100 green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e F) Python (analysis.py): - ieee519_compliance: per-phase voltage-THD compliance assessed at p95 against the IEEE 519-2014 <=1 kV limits (8% total, 5% planning); current THD reported as p95 (informational, no hard limit without Isc/IL). all_voltage_compliant rolls up the three phases. - sarfi_indices: SARFI-90/80/70/50/10 counts of voltage events (dips+outages) below each residual-voltage threshold (IEEE 1159 / 1564). - cli writes pq_standards.json + a one-line [pq] summary. Added the six THD avg channels to the Python store STORE_COLUMNS (they were already in the JS store) so IEEE 519 can read them; no spec change. Web (parity): ieee519Compliance + sarfiIndices in analysis.js, surfaced in the Statistics panel and the exported HTML report. docs/PQ_STANDARDS.md explains the limits, the p95 assessment method, the SARFI bins, and the caveats. Parity: golden ieee519/sarfi entries; web parity asserts both match Python. Tests: python test_pq_standards.py (5); web parity (2). Python 196 -> 201; web 100 -> 102 green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sliding-window average real-power demand (default 15 min, --demand-window), reporting peak demand, the window it occurred in, mean demand, and an optional decimated demand series. Non-finite P samples treated as 0. Python: analysis.demand_analysis; cli --demand-window writes demand.json + a one-line [demand] summary and adds Peak demand rows to the XLSX Summary sheet. Web (parity): analysis.demandAnalysis, surfaced in the Statistics panel and the exported HTML report. Parity: golden demand entry on a deterministic ramp (P = i*100 W over 600 s, 120 s window); web parity asserts peak/mean/series match (timestamps compared as instants — JS emits Z, Python +00:00). Tests: python test_demand.py (5); web parity (1). Python 201 -> 206; web 102 -> 103 green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Timestamps stay UTC throughout; with --tz ZONE (IANA) reports additionally render local wall-clock alongside UTC. Default (no --tz) is unchanged: UTC only. Anchors already honoured explicit ISO offsets — --tz is the display complement. Python: tzutil.py (resolve_tz / to_utc / format_local_utc / tz_label using zoneinfo); cli --tz flag (fails fast on a bad zone) adds a local+UTC Time range block to summary.txt. Web (parity): tzutil.js (formatLocalUtc / isoInZone / isoUtc via Intl, matching Python isoformat incl. dropping zero-millisecond fractions); a report-timezone input under the summary (persisted) that renders the local+UTC range and is passed into the exported HTML report's Time range header. Parity: golden timezone entry (fixed instant in UTC + America/Chicago); web/tests/tzutil.test.js asserts both render identically. Tests: python test_timezone.py (9); web tzutil parity (5). Python 206 -> 215; web 103 -> 108 green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Override EventRules thresholds from a JSON/TOML file, keyed by asset_id/name so one file can hold a whole fleet's known trip points. Precedence: built-in defaults -> file [defaults] -> file [assets.<name>] (or [assets.default]). Unknown keys are rejected; min/gap keys coerced to int, the rest to float. Python: rules_file.py (load_rules / describe_overrides, JSON + TOML via tomllib); cli --rules-file resolves rules for the session's asset, prints a one-line [rules] diff, and threads them into detect_events. Web (parity): events.rulesFromObject (same precedence + validation); a rules-file picker in the summary that loads JSON, re-runs detection for the active asset, and a Clear button to revert to defaults. docs/RULES_FILE.md documents the format + every overridable key. Tests: python test_rules_file.py (8, incl. a CLI e2e where a 95% dip threshold flags a 93.9% dip the defaults miss); web events rules-file (6, incl. the same detection-change check). Python 215 -> 223; web 108 -> 114 green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bump version to 0.6.0 (pyproject + __init__). Add the v0.6.0 CHANGELOG entry covering Features A-I. Update ROADMAP: new "Shipped in v0.6" section, mark the deferred v0.5 web streaming/typed-array items DONE. README: new CLI flag rows (--demand-window, --tz, --auto-reverse-cts, --rules-file), stitch subcommand, and a power-quality/per-asset-rules section. New docs: STITCHING.md, DEMAND.md (PQ_STANDARDS.md and RULES_FILE.md landed with their features). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Round 2 — web parity + standards-grade analysis
Closes the web memory/parity gap from v0.5.0 and adds six analysis features. All built test-first with Python↔JS parity. Tests: Python 223 ✓ · Web 114 ✓.
Web streaming + parity (A, B)
column_store.js,parseTrendColumnarStream) — 7-day file (589,877 recs / 438 MB) parses + fully analyses at ~278 MB peak RSS vs old ~1.6 GB. Typed-array columns streamed via 8 MBBlob.sliceand Transferred from the worker.web/analysis.js— full JS port ofanalysis.py(Welford moments, percentile sketch, whole-session stats, ITIC, time-of-day, marker correlation). Stats panel + ToD chart in the UI; stats embedded in HTML export.Math.max(...arr)stack overflow that would crash on ~590K-element sessions.CT-reversal auto-detection (C)
--auto-reverse-ctsflags reversed-CT installs (real power negative ≥50% of non-outage time) and applies--reverse-ctswith a loud notice; web banner + one-click apply. Flags the real ES.004 (52% negative P, mean −37 kW). Python + JS.Multi-session stitching (D)
fluke-analyze stitch S1 S2 … -o OUT— gap-aware continuous timeline with per-source provenance (beats the meter's 7-day cap). Validated ES.001+ES.002 → 79,897 recs, 802 s gap detected.Executive summary (E)
narrative.md+ top of summary.txt/HTML/XLSX). Python + JS parity.Power-quality standards (F)
docs/PQ_STANDARDS.md.Demand, timezone, per-asset rules (G, H, I)
--demand-window); timezone-aware anchors/reports (--tz); per-asset threshold overrides (--rules-file).🤖 Generated with Claude Code