Large-session hardening: stream week-long sessions + new analysis features (v0.5.0) - #1
Merged
Merged
Conversation
One streaming pass builds ~20 array.array('f') columns + FILETIME ticks
instead of materializing 590K Record objects. Optional time_shift corrects
wrong-RTC sessions lazily without losing raw ticks.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces O(N*window) rolling pstdev (177M ops for a week) with prefix sums of P and P^2. pick_snapshots now accepts a ColumnStore or Records. Parity test confirms identical results to the old algorithm. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
detect_events accepts a ColumnStore or Records; columns are array.array views instead of 15 full-length parallel list comprehensions over a materialized list(records). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
All seven rules read array.array columns from the store; the N-entry time->index dict and records[-1]/[0] duration calc now come from the store. analyze() accepts a ColumnStore or Records. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
export_csv_multi walks trend.bin once, fanning records to full CSV, 1-min CSV, and the ColumnStore (full resolution). --max-csv-rows guard raises the full-CSV stride and logs exactly what was downsampled. iter_records_safe skips/logs bad magic (resync), truncated tails, and flags non-finite floats instead of aborting. time_shift + reverse_cts applied consistently to CSV and store. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CLI now parses trend.bin once via export_csv_multi, threading the ColumnStore through detect/snapshots/insights (no more list(iter_records) double materialization). Adds --anchor-start/--anchor-end clock correction (mutually exclusive), --max-csv-rows guard, and registers --split-by/--mark/--marks/--tod-profile/--no-stats flags. Window filter operates on the store. _run_extra_analyses hook stubbed for round-2. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New analysis.py with tested pure functions: whole_session_stats (streaming Welford mean/var + histogram percentiles), classify_itic / event_itic (ITIC/CBEMA ride-through), parse_period/bucket_key/ assign_buckets/slice_store (--split-by partitioning), correlate_markers (--mark/--marks nearest-event), time_of_day_profile (diurnal envelope), bucket_summary_row (per-bucket table). 31 new tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_run_extra_analyses now emits stats.json/csv, markers.json (--mark/ --marks), time_of_day_profile.csv (--tod-profile), augments events.json with ITIC, and on --split-by writes a full per-bucket report (CSV + events + summary + xlsx) plus buckets_summary.csv roll-up. Per-bucket events filed under t_start bucket; boundary-spanners flagged. 8 e2e tests incl. union-of-events == whole-session parity. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
write_xlsx accepts stats + tod_rows and adds a Statistics sheet (per-channel percentiles + threshold time) and a Time of Day sheet with an avg/min/max line chart. CLI threads both into the top-level report so the diurnal profile and whole-session stats are visible without gnuplot. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
uPlot chokes well before a week of per-second points; decimateSeries buckets the filtered series to ~plot-width and emits per-bucket min+max so dips/spikes/outages survive. CSV download keeps full resolution. A 'large session — chart decimated' notice appears on affected charts. 6 node tests. Typed-array record storage + chunked parse deferred to ROADMAP. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bump pyproject to 0.5.0. CHANGELOG entry for large-session hardening + round-2 features (257 tests). README CLI table documents the new flags. New docs/ITIC.md explains the dip classification. ROADMAP records the shipped work and the deferred web typed-array/chunked-parse items. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Real ES.004 data has a single NaN in P_total_avg_W (record 417073) that poisoned one day's kWh roll-up (nan). bucket_summary_row now skips non-finite V/I/P/PF samples in min/avg/max/kWh. Regression test added. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Large-session hardening + new analysis features (v0.5.0)
Motivated by a real ~6.8-day, 438 MB, 589,877-record session that the previous design could not process (it OOM'd partway — the pipeline materialized every record in memory multiple times). This PR re-architects parsing/analysis to stream, and adds the analysis features needed to make sense of multi-day captures.
Core hardening
store.ColumnStore— single streaming pass into ~21array.array('f')columns + FILETIME ticks, with optional lazytime_shift. Peak memory on the full week is ~59 MB (was >1 GB).events,snapshots,insightsto consume the store — removes the repeatedlist(iter_records())materializations and per-rule column copies.snapshotsrolling-stdev is now O(N) prefix-sum (parity-tested against the old O(N×window) implementation).parser.export_csv_multi— one pass writes the full + 1-min CSVs and builds the store;--max-csv-rowsguard auto-raises the stride and logs the downsample (no silent cap).iter_records_safe— tolerant parser: resyncs on bad magic, handles truncated tails, flags non-finite floats (real device dumps contain these).New CLI
--anchor-start/--anchor-end— clock correction for loggers with a wrong RTC (shifts the whole timeline to a known real start/end).--split-by hour|day|week|<duration>— partitions the single analysis into time buckets; each bucket gets a full report, plus a roll-up.--mark/--marks— overlay known external events (e.g. an equipment trip) and correlate them to detected events.--tod-profile [HH:MM-HH:MM]— time-of-day (diurnal) avg/min/max profile with per-day overlay.--no-statstoggle.New analysis (
analysis.py)buckets_summary.csv).Reporting / web
Docs
docs/ITIC.md.Tests
spec/field_map.jsonunchanged so the cross-language parity test still holds.Deferred to ROADMAP
Web typed-array record storage, chunked/streamed parse, and IndexedDB large-session verification.
🤖 Generated with Claude Code