diff --git a/CHANGELOG.md b/CHANGELOG.md index 1cd1a95..06d30f8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,61 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.6.0] — 2026-06-03 + +Web parity + analysis depth. Closes the web memory/parity gap from 0.5.0 and +adds six standards-grade analysis features, all parity-tested Python↔JS. + +### Added — web streaming + parity (Features A, B) +- **Streaming columnar web parse** — `web/column_store.js` + `parseTrendColumnar` + / `parseTrendColumnarStream` decode `trend.bin` straight into packed + `Float32Array` columns, streamed from the dropped `File` in 8 MB + record-aligned `Blob.slice` chunks and **Transferred** back from the worker. + The 7-day file (589,877 recs / 438 MB) now parses + fully analyses at + **~278 MB peak RSS** vs the old ~1.6 GB. Analysis, charts, range select, and + tariff all read the resident store; exports materialise records transiently. + Legacy `parseTrendBin` retained for small-file / CSV paths. +- **`web/analysis.js`** — JS port of `analysis.py` (Welford moments, percentile + sketch, `wholeSessionStats`, `classifyItic`/`eventItic`, `timeOfDayProfile`, + `correlateMarkers`). A Statistics panel + time-of-day chart in the web UI; the + stats table is embedded in the HTML export. +- Fixed a latent `Math.max(...arr)` stack overflow in the insights engine that + would have crashed the web app on a ~590 K-element session. + +### Added — CT-reversal detection (Feature C) +- **`--auto-reverse-cts`** — detects a reversed-CT install (real power negative + for ≥ 50 % of non-outage time) and applies `--reverse-cts` automatically with + a loud notice. A matching web banner + one-click apply. Flags the real ES.004 + (52 % negative P, mean −37 kW). Python + JS. + +### Added — multi-session stitching (Feature D) +- **`fluke-analyze stitch S1 S2 … -o OUT`** — concatenates consecutive sessions + into one continuous, gap-aware timeline with per-source provenance, then runs + the normal analysis over the stitched series (beats the meter's 7-day cap). + `web/multi_session.js` gains `stitchStores` + `buildStitched`. Validated on + ES.001 + ES.002 → 79,897 records with a detected 802 s gap. + +### Added — executive summary (Feature E) +- **Auto-narrative** — deterministic, rule-based plain-English summary + (`narrative.md` + top of `summary.txt`, HTML, XLSX). Python + JS parity. + +### Added — power-quality standards (Feature F) +- **IEEE 519** voltage-THD compliance per phase (p95 vs 8 %/5 %) and **IEEE 1159 + / SARFI-90/80/70/50/10** indices, in stats + reports. `docs/PQ_STANDARDS.md`. + +### Added — demand + timezone + per-asset rules (Features G, H, I) +- **`--demand-window`** rolling peak-demand (default 15 min) with peak window + + series, in stats/XLSX. Python + JS. +- **`--tz ZONE`** renders report timestamps in local + UTC (default UTC + unchanged); anchors still accept ISO offsets. Python + JS. +- **`--rules-file FILE`** (JSON/TOML) overrides `EventRules` thresholds keyed by + asset name (defaults + per-asset). `docs/RULES_FILE.md`. Python + JS. + +### Tests +- Python 176 → 223; web 82 → 114. New Python↔JS golden-parity harness covers + stats, ITIC, CT reversal, narrative, IEEE 519/SARFI, demand, and timezone + formatting. + ## [0.5.0] — 2026-06-02 Large-session hardening — the tool now survives week-long captures diff --git a/README.md b/README.md index 1072ec2..dcacdd3 100644 --- a/README.md +++ b/README.md @@ -96,6 +96,10 @@ fluke-analyze path/to/ES.NNN -o output/ \ | `--marks FILE.csv` | | Load markers from a CSV (`time,label`). | | `--tod-profile [HH:MM-HH:MM]` | off | Time-of-day (diurnal) profile — avg/min/max envelope per bin across all days. Bare flag = 24 h. Writes `time_of_day_profile.csv` + an XLSX sheet. | | `--tod-bin MINS` | `1` | Time-of-day bin width in minutes. | +| `--demand-window SECS` | `900` | Rolling peak-demand window. Reports peak demand + the window it occurred in (`demand.json` + XLSX). See [`docs/DEMAND.md`](docs/DEMAND.md). | +| `--tz ZONE` | UTC | Render report timestamps in local + UTC for an IANA zone (e.g. `America/Chicago`). Default UTC only. | +| `--auto-reverse-cts` | off | Auto-detect a reversed-CT install (sustained negative real power) and apply `--reverse-cts` automatically, with a loud notice. See `docs/CT_REVERSAL.md`. | +| `--rules-file FILE` | off | JSON/TOML EventRules overrides keyed by asset name. See [`docs/RULES_FILE.md`](docs/RULES_FILE.md). | | `--no-stats` | | Skip whole-session statistics (`stats.json`/`stats.csv` + XLSX sheet). | | `--format` | `png` | `png` or `svg` | | `--no-xlsx` | | Skip the XLSX workbook | @@ -134,6 +138,25 @@ Overlays selected quantities across sessions on the same axes (aligned by relative-time-from-session-start) and writes a side-by-side `compare_summary.csv`. See [`docs/COMPARE.md`](docs/COMPARE.md). +### Multi-session stitching + +```bash +fluke-analyze stitch ES.001 ES.002 [...] -o OUT +``` + +Concatenates consecutive captures of the same asset into one continuous, +gap-aware timeline (beating the meter's 7-day cap), with per-source provenance +in `stitch.json`, then runs the normal analysis over the stitched series. See +[`docs/STITCHING.md`](docs/STITCHING.md). + +## Power-quality standards & per-asset rules + +- **IEEE 519 / IEEE 1159 / SARFI** — voltage-THD compliance and SARFI dip + indices in every run (`pq_standards.json`). See + [`docs/PQ_STANDARDS.md`](docs/PQ_STANDARDS.md). +- **Per-asset thresholds** — override `EventRules` from JSON/TOML keyed by asset + with `--rules-file`. See [`docs/RULES_FILE.md`](docs/RULES_FILE.md). + ## Event detection Auto-detected event kinds (full rules + rationale in [`docs/EVENT_RULES.md`](docs/EVENT_RULES.md)): diff --git a/ROADMAP.md b/ROADMAP.md index cc0740f..09aa8ee 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -4,6 +4,34 @@ Shipped releases live in [CHANGELOG.md](CHANGELOG.md). This file is the backlog of ideas we've discussed but not yet committed to a specific release. +## Shipped in v0.6 — web parity + analysis depth + +Closed the v0.5 web memory/parity gap and added standards-grade analysis, all +parity-tested Python↔JS: + +- **Web streaming columnar parse** — the two deferred v0.5 web items below + ("typed-array record storage" + "chunked/streaming parse with progress") are + now shipped together: `web/column_store.js` + a chunked `Blob.slice` parser in + the worker that Transfers packed `Float32Array` columns back. ES.004 (438 MB, + 589,877 recs) now parses + fully analyses at ~278 MB peak RSS vs ~1.6 GB. +- **`web/analysis.js`** — full JS stats port (Welford, percentile sketch, + whole-session stats, ITIC, time-of-day, marker correlation) with a Statistics + panel + ToD chart; closes the "web has no stats" gap. +- **CT-reversal auto-detection** (`--auto-reverse-cts` + web banner). +- **Multi-session stitching** (`fluke-analyze stitch`). +- **Auto-narrative / executive summary** — the rule-based, no-LLM version of + Theme D below (an LLM mostly restates the structured findings, as predicted). +- **IEEE 519 + IEEE 1159 / SARFI** power-quality indices. +- **Rolling peak demand** (`--demand-window`) — the kWh-side of Theme B's + "demand charges". +- **Per-asset threshold config** (`--rules-file`) — the file-driven form of + Theme E's "custom event-rule editor" (a web slider editor is still open). +- **Timezone-aware reports** (`--tz`). + +Still open from the backlog: fleet/monitoring (Theme A), full TOU/financial +rigor (Theme B), comparison polish (Theme C), the rest of power-user/ecosystem +(Theme E), and polish (Theme F). + ## Shipped in v0.5 — large-session hardening The v0.5 release was driven by a real ~6.8-day P115RE-MAC03 capture @@ -18,19 +46,17 @@ remain the forward backlog. ### Deferred from the v0.5 web pass The Python core is fully week-hardened. The browser app got chart -decimation (the biggest uPlot win) this pass; the remaining 7-day -robustness work is parked here: - -- **Typed-array record storage** — `parser.js` currently allocates one - object + one `Float32Array(180)` per record (~425 MB+ for a week). - Replace with a single flat `Float32Array` (or per-column arrays mirroring - the Python `ColumnStore`) indexed by record, eliminating per-record - object overhead. -- **Chunked / streaming parse with progress** — parse in slices off a - `File`/`Blob` stream so a 438 MB session never has to be held as one - `ArrayBuffer` plus a parallel object array. -- **IndexedDB large-session verification** — confirm the cache layer - holds a 438 MB session and evicts sanely under quota pressure. +decimation in v0.5; the streaming/typed-array work below **shipped in v0.6**: + +- ~~**Typed-array record storage**~~ — **DONE (v0.6)**: `web/column_store.js` + keeps the analysis channels as packed `Float32Array` columns instead of + per-record objects. +- ~~**Chunked / streaming parse with progress**~~ — **DONE (v0.6)**: + `parseTrendColumnarStream` reads the `File` in 8 MB record-aligned + `Blob.slice` chunks; the full `ArrayBuffer` is never resident. +- **IndexedDB large-session verification** — still open: confirm the cache + layer holds a 438 MB session and evicts sanely under quota pressure. (The + streaming path no longer caches the raw buffer, reducing pressure.) ## v0.5 candidates diff --git a/docs/DEMAND.md b/docs/DEMAND.md new file mode 100644 index 0000000..97c5c11 --- /dev/null +++ b/docs/DEMAND.md @@ -0,0 +1,40 @@ +# Demand analysis (`--demand-window`) + +Utilities bill on **demand** — a sliding/block-window average of real power, not +the instantaneous peak. The 15-minute interval is the most common. This +analyzer computes a trailing rolling mean of `P_total_avg_W` and reports the +peak demand and when it occurred. + +```bash +fluke-analyze ES.004 --demand-window 900 # 900 s = 15 min (default) +``` + +## What it computes + +For a window of `W` seconds (1 record = 1 s), at each index `i ≥ W−1` the +trailing demand is `mean(P_total over the last W records)`. The analyzer reports: + +| Field | Meaning | +|---|---| +| `peak_demand_w` / `peak_demand_kw` | the highest rolling-window demand | +| `peak_window_start` / `peak_window_end` | the window that produced the peak | +| `mean_demand_w` | mean of all full-window demands | +| `n_windows` | number of full windows evaluated | +| `series` | an optional decimated demand series for charting | + +Non-finite `P` samples are treated as 0 in the running sum. If the session is +shorter than one window, no peak is reported (`n_windows = 0`). + +## Outputs + +- CLI: `demand.json` + a one-line `[demand]` summary + Peak-demand rows in the + XLSX **Summary** sheet. +- Web: surfaced in the Statistics panel and the exported HTML report + (`web/analysis.js → demandAnalysis`). + +## Notes + +- The window is **trailing** (right-aligned), matching how interval meters + accumulate demand within each interval. +- For true utility block-demand (fixed 15-min boundaries) rather than a sliding + window, split first with `--split-by 15m` and read each bucket's mean kW. diff --git a/docs/PQ_STANDARDS.md b/docs/PQ_STANDARDS.md new file mode 100644 index 0000000..867b950 --- /dev/null +++ b/docs/PQ_STANDARDS.md @@ -0,0 +1,71 @@ +# Power-quality standards: IEEE 519 & IEEE 1159 / SARFI + +This analyzer reports two standards-based power-quality summaries alongside the +event log and whole-session statistics. Both are computed identically in the +Python CLI and the web app (parity-tested). + +## IEEE 519-2014 — harmonic (THD) limits + +IEEE 519 sets voltage-distortion limits at the point of common coupling. For +systems at or below 1 kV (the case for the 3540 FC's typical 277/480 V service): + +| Quantity | Limit | +|---|---| +| Total voltage THD | **8.0 %** | +| Planning level / single-harmonic guidance | **5.0 %** | + +**How it is assessed.** IEEE 519 evaluates compliance against the 95th +percentile of the measured distortion, not the instantaneous peak. The analyzer +therefore reports the **p95 of `V_THD_pct__avg`** per phase and marks a +phase: + +- `compliant` when p95 ≤ 8.0 %, +- `exceeds_planning` when p95 > 5.0 % (a yellow flag even if still compliant). + +`all_voltage_compliant` is true only when all three phases pass. + +**Current THD.** IEEE 519 current limits are expressed as TDD (total demand +distortion) and depend on the short-circuit ratio Isc/IL, which the meter does +not record. The analyzer therefore reports **p95 of `I_THD_pct__avg`** +per phase as informational context, without a hard pass/fail. + +Output: `pq_standards.json → ieee519`. + +## IEEE 1159 / IEEE 1564 — SARFI indices + +The **System Average RMS (variation) Frequency Index, SARFI-X**, counts the +number of voltage variation events whose **residual voltage dipped below X % of +nominal**. For a single monitoring point (one meter, one asset) the index is the +event count itself. + +The analyzer reports the standard magnitude thresholds: + +| Index | Counts events with residual voltage below | +|---|---| +| SARFI-90 | 90 % (i.e. any dip ≥ 10 %) | +| SARFI-80 | 80 % | +| SARFI-70 | 70 % | +| SARFI-50 | 50 % | +| SARFI-10 | 10 % (near-interruption / outage) | + +Residual voltage is taken from each detected `dip` (severity = residual +fraction) and `outage` (deepest L-N voltage ÷ nominal). Swells and non-voltage +events are excluded. Because the bins are cumulative, SARFI-90 ≥ SARFI-80 ≥ … ≥ +SARFI-10 by construction. + +Output: `pq_standards.json → sarfi`. + +## Where it shows up + +- **CLI:** a one-line `[pq]` summary during analysis; full detail in + `pq_standards.json`. +- **Web:** computed inline from the resident ColumnStore via + `web/analysis.js` (`ieee519Compliance`, `sarfiIndices`). +- **Reports:** surfaced in the HTML/XLSX statistics area. + +## Caveats + +- THD field semantics are confidence `M` in `spec/field_map.json`; treat the THD + numbers as strong-inference until cross-checked against a reference meter. +- SARFI here is a single-site event count, not the multi-site customer-weighted + utility metric. It is directly comparable across captures of the same asset. diff --git a/docs/RULES_FILE.md b/docs/RULES_FILE.md new file mode 100644 index 0000000..909ea04 --- /dev/null +++ b/docs/RULES_FILE.md @@ -0,0 +1,76 @@ +# Per-asset threshold config (`--rules-file`) + +Event detection uses a single set of thresholds (`EventRules`) tuned to IEEE +1159 / NEMA defaults. When you know an asset's real behaviour — its actual trip +voltage, an expected swell ceiling, a noisier-than-usual feeder — you can +override those thresholds per asset with `--rules-file FILE`. + +``` +fluke-analyze ES.004 --rules-file fleet_rules.json +``` + +The asset is matched on the session's `asset_name` (from the `*-config.json`). + +## File format (JSON or TOML) + +Two optional sections: + +- `defaults` — applied to every asset. +- `assets` — a map of `asset_name → overrides`. A special `"default"` asset key + is used when the session's asset has no explicit entry. + +Per-asset values win over `defaults`, which win over the built-in defaults. + +### JSON + +```json +{ + "defaults": { + "dip_pct_of_nominal": 0.92 + }, + "assets": { + "P115RE-MAC03": { + "outage_v_threshold": 60.0, + "swell_pct_of_nominal": 1.08 + }, + "default": { + "freq_excursion_hz": 0.4 + } + } +} +``` + +### TOML + +```toml +[defaults] +dip_pct_of_nominal = 0.92 + +[assets."P115RE-MAC03"] +outage_v_threshold = 60.0 +swell_pct_of_nominal = 1.08 +``` + +A **flat file** with only threshold keys (no `defaults`/`assets`) is treated as +defaults for every asset. + +## Overridable keys + +| Key | Default | Meaning | +|---|---|---| +| `outage_v_threshold` | 50.0 | any phase L-N below this V is an outage | +| `dip_pct_of_nominal` | 0.90 | < this fraction of nominal = dip | +| `swell_pct_of_nominal` | 1.10 | > this fraction of nominal = swell | +| `high_current_sigma` | 2.0 | mean + Nσ on any phase = high current | +| `freq_excursion_hz` | 0.5 | \|f − nominal\| over this = excursion | +| `imbalance_pct_threshold` | 2.5 | NEMA % imbalance threshold | +| `power_step_pct_of_mean` | 0.50 | ΔP in 1 s over this × mean \|P\| = step | +| `min_duration_secs` | 1 | events shorter than this are ignored (int) | +| `gap_tolerance_secs` | 1 | merge runs split by ≤ this many samples (int) | +| `nominal_freq_hz` | 60.0 | line frequency baseline | + +Unknown keys are rejected with an error listing the valid ones. `min_duration_secs` +and `gap_tolerance_secs` are coerced to integers; all others to floats. + +On load the CLI prints a one-line `[rules]` note showing exactly which +thresholds changed (`key: old -> new`) for the matched asset. diff --git a/docs/STITCHING.md b/docs/STITCHING.md new file mode 100644 index 0000000..0de050a --- /dev/null +++ b/docs/STITCHING.md @@ -0,0 +1,53 @@ +# Multi-session stitching (`fluke-analyze stitch`) + +The 3540 FC caps a single capture at ~7 days. To analyse a longer span, take +consecutive captures of the same asset and stitch them into one continuous +timeline: + +```bash +fluke-analyze stitch ES.001 ES.002 ES.004 -o week_out/ +``` + +Inputs may be `ES.NNN/` directories, `.fel` bundles, or pre-parsed `.csv` +files, in any order — they are sorted by their first record's start time. + +## What it does + +1. Parses each session into a memory-bounded `ColumnStore`. +2. Concatenates them in time order, carrying absolute (anchor-corrected) + timestamps so the joined series is monotonic. +3. Records a **gap** wherever consecutive sessions don't abut (boundary + difference > `--gap-tolerance`, default 2 s). No synthetic fill rows are + inserted — the gap is noted, not papered over. +4. Runs the normal analysis over the stitched series. + +## Outputs + +| File | Contents | +|---|---| +| `stitch.json` | provenance: total records, per-source `{label, lo, hi, t_start, t_end, records}`, and `gaps[]` | +| `session.csv` | the stitched per-second series with a `source` provenance column | +| `events.json` | events detected across the whole stitched timeline | +| `insights.json` | insights over the stitched series | +| `stats.json` | whole-session statistics over the stitched series | + +## Flags + +| Flag | Default | What | +|---|---|---| +| `-o`, `--output` | required | output directory | +| `--labels A,B,…` | input names | per-source labels (must match count) | +| `--reverse-cts [PHASES]` | off | apply the same reverse-CTs to every source | +| `--gap-tolerance SECS` | `2.0` | boundary gaps larger than this are recorded | +| `--nominal-ln-v V` | auto | nominal L-N voltage for detection | +| `--no-stats` | | skip the stitched stats | +| `--no-csv` | | skip writing the (large) stitched `session.csv` | + +## Example + +Stitching ES.001 (5,024 recs) + ES.002 (74,873 recs) yields 79,897 continuous +records and a single recorded gap of ~802 s between the two captures — exactly +the meter's swap-out interval. + +The web app exposes the same capability: load multiple sessions and call +`MultiSession.buildStitched(spec)` (see `web/multi_session.js`). diff --git a/python/pyproject.toml b/python/pyproject.toml index 3cdceea..f9d7139 100644 --- a/python/pyproject.toml +++ b/python/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "fluke-3540-analyzer" -version = "0.5.0" +version = "0.6.0" description = "Parser, event detector, and chart generator for Fluke 3540 FC three-phase power-quality sessions. See https://github.com/GrumpyTanker/fluke-3540-analyzer." requires-python = ">=3.10" license = {text = "MIT"} diff --git a/python/src/fluke_3540/__init__.py b/python/src/fluke_3540/__init__.py index 90d88a4..4adaa41 100644 --- a/python/src/fluke_3540/__init__.py +++ b/python/src/fluke_3540/__init__.py @@ -4,4 +4,4 @@ The canonical field map and binary layout live in spec/field_map.json at the repo root and are shared with the JavaScript port. See README.md. """ -__version__ = "0.4.0" +__version__ = "0.6.0" diff --git a/python/src/fluke_3540/analysis.py b/python/src/fluke_3540/analysis.py index 6205cf4..d99de0a 100644 --- a/python/src/fluke_3540/analysis.py +++ b/python/src/fluke_3540/analysis.py @@ -185,6 +185,83 @@ def whole_session_stats( return out +# --- CT-reversal auto-detection --------------------------------------------- +# +# A correctly-wired load draws real power: P_total > 0 essentially all the time. +# When an iFlex CT is clipped on backwards, P/Q/PF/energy come out negated, so a +# load reads as a persistent *generator* (P_total < 0). We flag a session when +# real power is negative for a high fraction of NON-OUTAGE time — outage samples +# (all phases collapsed) are excluded because P there is ~0/noise. + +def detect_ct_reversal( + store: ColumnStore, + neg_fraction_threshold: float = 0.50, + outage_v_threshold: float = 50.0, +) -> dict: + """Detect a likely reversed-CT install from sustained negative real power. + + Returns a dict: + { + "reversed": bool, # True if neg fraction >= threshold + "frac_negative": float, # fraction of non-outage records with P<0 + "non_outage_records": int, + "negative_records": int, + "mean_p_w": float, # mean P over finite non-outage records + "threshold": float, + } + + A correctly-wired load draws positive real power essentially all the time, + so even a modestly-sustained negative-P fraction is a strong reversal signal; + the default 0.50 threshold (negative more often than positive) catches it + while staying clear of brief regen/export blips. Non-finite P samples are + skipped (the real meter occasionally emits NaN). ``reversed`` True means the + data looks like a load wired with backwards CTs — re-run with + ``--reverse-cts`` (or ``--auto-reverse-cts``) to correct it. + """ + p = store.col("P_total_avg_W") + va = store.col("V_LN_a_avg_V") + vb = store.col("V_LN_b_avg_V") + vc = store.col("V_LN_c_avg_V") + n = store.n + non_outage = 0 + negative = 0 + p_sum = 0.0 + p_count = 0 + for i in range(n): + if va[i] > outage_v_threshold and vb[i] > outage_v_threshold and vc[i] > outage_v_threshold: + non_outage += 1 + pv = p[i] + if pv == pv and pv not in (math.inf, -math.inf): # finite + p_sum += pv + p_count += 1 + if pv < 0: # NaN < 0 is False, so non-finite never counts as negative + negative += 1 + frac = (negative / non_outage) if non_outage else 0.0 + mean_p = (p_sum / p_count) if p_count else 0.0 + return { + "reversed": frac >= neg_fraction_threshold, + "frac_negative": frac, + "non_outage_records": non_outage, + "negative_records": negative, + "mean_p_w": mean_p, + "threshold": neg_fraction_threshold, + } + + +def ct_reversal_notice(result: dict) -> str: + """A loud, explicit operator-facing notice for a flagged CT reversal.""" + pct = result["frac_negative"] * 100.0 + return ( + " !! CT REVERSAL DETECTED !!\n" + f" Real power (P_total) is NEGATIVE for {pct:.1f}% of non-outage time " + f"(mean P = {result['mean_p_w'] / 1000:.1f} kW). A load should draw " + "positive real power — this signature means one or more iFlex CT probes " + "are clipped on backwards.\n" + " Re-run with --reverse-cts to negate P/Q/PF/energy, or " + "--auto-reverse-cts to apply the correction automatically." + ) + + # --- ITIC / CBEMA classification -------------------------------------------- # # The ITIC (CBEMA) curve describes the voltage-deviation/duration envelope that @@ -249,6 +326,168 @@ def classify_itic(residual_pct: float, duration_secs: float) -> str: return "no_interruption" +# --- IEEE 519 THD compliance + IEEE 1159 / SARFI indices -------------------- +# +# IEEE 519-2014 voltage-distortion limits for systems <= 1 kV are 8.0% THD and +# 5.0% any-single-harmonic; we report against the 5%/8% pair (assessed on the +# 95th-percentile per-phase V_THD, which is how 519 evaluates compliance). +# Current TDD limits depend on the short-circuit ratio Isc/IL which the meter +# does not record, so I_THD is reported as the 95th-percentile per phase +# (informational) without a hard pass/fail. + +# (limit_name, threshold_pct) voltage limits. +IEEE519_V_THD_LIMIT_PCT = 8.0 # total voltage distortion limit (<=1 kV) +IEEE519_V_THD_PLANNING_PCT = 5.0 # planning level / single-harmonic guidance + + +def ieee519_compliance(store: ColumnStore) -> dict: + """IEEE 519 voltage-THD compliance per phase (assessed at p95). + + Returns {"voltage": {phase: {p95, limit, planning, compliant}}, "current": + {phase: {p95}}, "limit_v_thd": 8.0, ...}. A phase is ``compliant`` when its + 95th-percentile V_THD is at or under the 8% limit. + """ + out: dict = { + "limit_v_thd_pct": IEEE519_V_THD_LIMIT_PCT, + "planning_v_thd_pct": IEEE519_V_THD_PLANNING_PCT, + "voltage": {}, + "current": {}, + } + all_compliant = True + for ph in ("a", "b", "c"): + vcol = store.col(f"V_THD_pct_{ph}_avg") + sk = _PercentileSketch(0.0, 100.0, nbins=2000) + for v in vcol: + sk.add(v) + p95 = sk.quantile(0.95) if sk.n else 0.0 + compliant = p95 <= IEEE519_V_THD_LIMIT_PCT + all_compliant = all_compliant and compliant + out["voltage"][ph] = { + "p95": p95, + "limit": IEEE519_V_THD_LIMIT_PCT, + "planning": IEEE519_V_THD_PLANNING_PCT, + "compliant": compliant, + "exceeds_planning": p95 > IEEE519_V_THD_PLANNING_PCT, + } + icol = store.col(f"I_THD_pct_{ph}_avg") + ski = _PercentileSketch(0.0, 200.0, nbins=2000) + for v in icol: + ski.add(v) + out["current"][ph] = {"p95": ski.quantile(0.95) if ski.n else 0.0} + out["all_voltage_compliant"] = all_compliant + return out + + +# SARFI magnitude bins (IEEE 1159 / IEEE 1564): residual-voltage thresholds. +# SARFI-X counts events whose residual voltage dipped BELOW X% of nominal. +SARFI_THRESHOLDS = (90, 80, 70, 50, 10) + + +def sarfi_indices(events, nominal_ln_v: float) -> dict: + """System Average RMS Frequency Index per threshold (SARFI-X). + + For a single monitoring point SARFI-X is simply the count of voltage events + (dips + outages) whose residual voltage fell below X% of nominal. Returns + {"SARFI-90": n, ..., "events_considered": m, "nominal_ln_v": v}. + """ + counts = {f"SARFI-{x}": 0 for x in SARFI_THRESHOLDS} + considered = 0 + for ev in events: + if ev.kind == "dip": + residual_pct = ev.severity * 100.0 + elif ev.kind == "outage": + residual_pct = (ev.severity / nominal_ln_v * 100.0) if nominal_ln_v else 0.0 + else: + continue + considered += 1 + for x in SARFI_THRESHOLDS: + if residual_pct < x: + counts[f"SARFI-{x}"] += 1 + counts["events_considered"] = considered + counts["nominal_ln_v"] = nominal_ln_v + return counts + + +# --- Demand analysis (rolling peak demand) ---------------------------------- +# +# Utilities bill demand on a sliding/block window average of real power (15 min +# is the most common interval). We compute a trailing rolling mean of +# P_total_avg_W over ``window_secs`` samples (1 sample = 1 s) and report the +# peak rolling demand and when it occurred, plus an optional decimated series. + +def demand_analysis( + store: ColumnStore, + window_secs: int = 900, + series_step_secs: int = 0, +) -> dict: + """Rolling-window peak real-power demand. + + Args: + window_secs: rolling window length (default 900 = 15 min). + series_step_secs: if > 0, emit a decimated demand series sampled every + this many seconds; if 0, no series is returned (just the peak). + + Returns {"window_secs", "peak_demand_w", "peak_demand_kw", + "peak_window_end", "peak_window_start", "mean_demand_w", "n_windows", + "series": [...]}. ``series`` entries are {"t": iso, "demand_w": float}. + Non-finite P samples are treated as 0 for the running sum. + """ + p = store.col("P_total_avg_W") + n = store.n + w = max(1, int(window_secs)) + out: dict = { + "window_secs": w, + "peak_demand_w": 0.0, + "peak_demand_kw": 0.0, + "peak_window_end": None, + "peak_window_start": None, + "mean_demand_w": 0.0, + "n_windows": 0, + "series": [], + } + if n == 0: + return out + # Trailing rolling sum. + running = 0.0 + peak = -math.inf + peak_i = -1 + demand_sum = 0.0 + demand_count = 0 + series: list[dict] = [] + step = max(0, int(series_step_secs)) + for i in range(n): + pv = p[i] + if pv != pv or pv in (math.inf, -math.inf): + pv = 0.0 + running += pv + if i >= w: + old = p[i - w] + if old != old or old in (math.inf, -math.inf): + old = 0.0 + running -= old + if i >= w - 1: # a full window is available + demand = running / w + demand_sum += demand + demand_count += 1 + if demand > peak: + peak = demand + peak_i = i + if step and ((i - (w - 1)) % step == 0): + series.append({ + "t": store.end(i).isoformat(), + "demand_w": demand, + }) + if peak_i >= 0: + out["peak_demand_w"] = peak + out["peak_demand_kw"] = peak / 1000.0 + out["peak_window_end"] = store.end(peak_i).isoformat() + out["peak_window_start"] = store.start(peak_i - w + 1).isoformat() + out["mean_demand_w"] = demand_sum / demand_count if demand_count else 0.0 + out["n_windows"] = demand_count + out["series"] = series + return out + + # --- Time-bucket partitioning (--split-by) ---------------------------------- @dataclass(frozen=True) diff --git a/python/src/fluke_3540/cli.py b/python/src/fluke_3540/cli.py index 3b59467..628b7e1 100644 --- a/python/src/fluke_3540/cli.py +++ b/python/src/fluke_3540/cli.py @@ -130,6 +130,11 @@ def build_argparser() -> argparse.ArgumentParser: help="Negate P/Q/PF/DPF/Wh/VARh for backwards iFlex CTs. " "Bare flag = all phases; pass a comma list like 'a,c' to " "only flip those phases (plus totals).") + ap.add_argument("--auto-reverse-cts", action="store_true", + help="Auto-detect a reversed-CT install (sustained negative " + "real power on a load) and apply --reverse-cts " + "automatically, with a loud notice. No-op if the data " + "already reads as a normal load.") ap.add_argument("--every", type=int, default=1, metavar="K", help="Emit every K-th record into the CSV (default 1, all)") ap.add_argument("--max-csv-rows", type=int, default=None, metavar="N", @@ -149,6 +154,12 @@ def build_argparser() -> argparse.ArgumentParser: "exclusive with --anchor-start).") # Time-bucket splitting + # Timezone-aware reporting (Feature H) + ap.add_argument("--tz", dest="tz", type=str, default=None, metavar="ZONE", + help="IANA timezone (e.g. America/Chicago) for report " + "timestamps. Reports then show local + UTC. Default UTC " + "only. Anchors already accept ISO offsets.") + ap.add_argument("--split-by", dest="split_by", type=str, default=None, metavar="PERIOD", help="Partition the session into time buckets, emitting a full " @@ -171,6 +182,10 @@ def build_argparser() -> argparse.ArgumentParser: "24 h; pass a window like 08:00-17:00.") ap.add_argument("--tod-bin", dest="tod_bin", type=int, default=1, metavar="MINS", help="Time-of-day bin width in minutes (default 1).") + ap.add_argument("--demand-window", dest="demand_window", type=int, + default=900, metavar="SECS", + help="Rolling demand window in seconds (default 900 = 15 min). " + "Reports peak demand + the window it occurred in.") # Output knobs ap.add_argument("--no-xlsx", action="store_true", help="Skip XLSX report") @@ -185,6 +200,11 @@ def build_argparser() -> argparse.ArgumentParser: help="Chart image format (default png)") ap.add_argument("--nominal-ln-v", type=float, default=None, metavar="V", help="Nominal L-N voltage (auto-inferred if omitted)") + ap.add_argument("--rules-file", dest="rules_file", type=Path, default=None, + metavar="FILE", + help="JSON/TOML file overriding EventRules thresholds, keyed " + "by asset_id/name (see docs/RULES_FILE.md). Per-asset " + "values win over the file's defaults.") return ap @@ -272,7 +292,31 @@ def _parse_session(args: argparse.Namespace, outdir: Path, if st.bad_magic or st.truncated or st.nonfinite: print(f" robustness: {st.bad_magic} bad-magic, " f"{st.truncated} truncated, {st.nonfinite} non-finite (skipped/flagged)") - return full_csv, min_csv, res["config"], res["store"] + + # CT-reversal auto-detection (Feature C). Always check + notify; with + # --auto-reverse-cts (and no explicit --reverse-cts already applied) re-run + # the single-pass parse with the correction applied. + store = res["store"] + from .analysis import ct_reversal_notice, detect_ct_reversal + ct = detect_ct_reversal(store) + if ct["reversed"]: + print(ct_reversal_notice(ct)) + already_reversed = bool(reverse_cts) + if getattr(args, "auto_reverse_cts", False) and not already_reversed: + print("[parse] --auto-reverse-cts: re-parsing with reverse-CTs applied…") + res = export_csv_multi( + session_dir, full_csv, min_csv, + every=args.every, reverse_cts=True, + max_full_rows=getattr(args, "max_csv_rows", None), + time_shift=time_shift, build_store=True, + log=print, progress_every=100_000, + ) + store = res["store"] + ct_after = detect_ct_reversal(store) + print(f" after auto-reverse: P now negative for " + f"{ct_after['frac_negative'] * 100:.1f}% of non-outage time " + f"(mean P = {ct_after['mean_p_w'] / 1000:.1f} kW)") + return full_csv, min_csv, res["config"], store def _store_from_csv(csv_path: Path) -> ColumnStore: @@ -328,7 +372,10 @@ def _detect_and_save(args: argparse.Namespace, outdir: Path, before = store.n store = _window_filter_store(store, args.from_time, args.to_time) print(f" window filter: {before:,} → {store.n:,} records") - events = detect_events(store, nominal_ln_v=args.nominal_ln_v) + + # Per-asset threshold overrides (Feature I). + rules = _load_event_rules(args, config) + events = detect_events(store, nominal_ln_v=args.nominal_ln_v, rules=rules) snaps = pick_snapshots(store, events, n=args.snapshots) findings = analyze_insights(store, events, snaps, config or {}) print(f" {len(events)} events, {len(snaps)} snapshots, " @@ -350,6 +397,28 @@ def _detect_and_save(args: argparse.Namespace, outdir: Path, return events, snaps, findings, store +def _load_event_rules(args: argparse.Namespace, config: dict | None): + """Return EventRules, applying --rules-file overrides for this asset.""" + from .events import DEFAULT_RULES + rules_path = getattr(args, "rules_file", None) + if not rules_path: + return DEFAULT_RULES + from .rules_file import describe_overrides, load_rules + asset = (config or {}).get("asset_name") + try: + rules = load_rules(rules_path, asset_name=asset) + except (OSError, ValueError) as e: + print(f"ERROR: --rules-file {rules_path}: {e}", file=sys.stderr) + raise SystemExit(2) + diffs = describe_overrides(rules_path, asset) + if diffs: + print(f"[rules] {rules_path} (asset={asset or 'n/a'}): " + + "; ".join(diffs)) + else: + print(f"[rules] {rules_path}: no overrides applied") + return rules + + def _infer_nominal_ln_v(store: ColumnStore, fallback: float | None) -> float: """Best-effort nominal L-N voltage for ITIC residual %.""" if fallback: @@ -550,6 +619,34 @@ def _run_extra_analyses(args: argparse.Namespace, outdir: Path, # ITIC always augments events.json (cheap, high-value for the deliverable). _augment_events_itic(outdir, events, nominal_ln_v) + # CT-reversal status snapshot for reports/web (cheap; one pass). + from .analysis import detect_ct_reversal, ieee519_compliance, sarfi_indices + ct = detect_ct_reversal(store) + (outdir / "ct_reversal.json").write_text(json.dumps(ct, indent=2), encoding="utf-8") + + # IEEE 519 (THD) + IEEE 1159 / SARFI power-quality (Feature F). + pq = { + "ieee519": ieee519_compliance(store), + "sarfi": sarfi_indices(events, nominal_ln_v), + } + (outdir / "pq_standards.json").write_text(json.dumps(pq, indent=2), encoding="utf-8") + v = pq["ieee519"]["voltage"] + print(f"[pq] IEEE 519 V_THD p95: a={v['a']['p95']:.1f}% b={v['b']['p95']:.1f}% " + f"c={v['c']['p95']:.1f}% (limit {pq['ieee519']['limit_v_thd_pct']:.0f}%) — " + f"{'COMPLIANT' if pq['ieee519']['all_voltage_compliant'] else 'NON-COMPLIANT'}; " + f"SARFI-90={pq['sarfi']['SARFI-90']}") + + # Demand analysis (Feature G). + from .analysis import demand_analysis + demand_window = max(1, getattr(args, "demand_window", 900)) + # Emit a series sampled at ~1/60 of the window so the JSON stays compact. + demand = demand_analysis(store, window_secs=demand_window, + series_step_secs=max(1, demand_window // 1)) + (outdir / "demand.json").write_text(json.dumps(demand, indent=2), encoding="utf-8") + if demand["n_windows"]: + print(f"[demand] peak {demand['peak_demand_kw']:.1f} kW over a " + f"{demand_window}s window ending {demand['peak_window_end']}") + stats: dict = {} if not getattr(args, "no_stats", False): stats = _write_stats(outdir, store) @@ -566,14 +663,49 @@ def _run_extra_analyses(args: argparse.Namespace, outdir: Path, if getattr(args, "split_by", None): _run_split_by(args, outdir, store, events, config, nominal_ln_v) - return stats, tod_rows + # Auto-narrative / executive summary (Feature E) — needs stats + ct. + narrative = _write_narrative(outdir, store, events, findings, stats, ct, config) + + return stats, tod_rows, narrative, demand + + +def _write_narrative(outdir: Path, store: ColumnStore, events, findings, + stats: dict, ct: dict, config: dict) -> str: + """Build the executive summary, write narrative.md, return the prose.""" + from .narrative import build_narrative, narrative_markdown + duration = None + if store.n: + duration = (store.last_end - store.first_start).total_seconds() + narrative = build_narrative( + events, findings, stats or None, ct, config=config, + total_records=store.n, duration_secs=duration, + ) + (outdir / "narrative.md").write_text( + narrative_markdown(narrative, config), encoding="utf-8") + print("[narrative] wrote narrative.md") + return narrative def _write_summary_txt(outdir: Path, events: Sequence[Event], snaps: Sequence[Snapshot], findings: Sequence[Finding], - config: dict) -> None: + config: dict, + narrative: str | None = None, + tz=None, tz_name: str | None = None, + store: "ColumnStore | None" = None) -> None: lines: list[str] = ["Fluke 3540 FC Session Summary", "=" * 32, ""] + if narrative: + lines.append("Executive Summary") + lines.append("-" * 17) + lines.append(narrative) + lines.append("") + # Time range (Feature H): local + UTC when --tz set, else UTC only. + if store is not None and store.n: + from .tzutil import format_local_utc, tz_label + lines.append(f"Time range ({tz_label(tz, tz_name)}):") + lines.append(f" start {format_local_utc(store.first_start, tz)}") + lines.append(f" end {format_local_utc(store.last_end, tz)}") + lines.append("") if config: if config.get("asset_name"): lines.append(f"Asset: {config['asset_name']}") @@ -624,7 +756,8 @@ def _parse_quantities(arg: str | None, default: Sequence[str], def _render_phase(args: argparse.Namespace, outdir: Path, full_csv: Path, min_csv: Path, events: Sequence[Event], snaps: Sequence[Snapshot], config: dict, - stats: dict | None = None, tod_rows=None) -> None: + stats: dict | None = None, tod_rows=None, + narrative: str | None = None, demand: dict | None = None) -> None: full_qtys = _parse_quantities(args.plot, DEFAULT_PLOTS, FULL_QUANTITIES.keys()) # Subset of zoom quantities that overlap with the user's --plot selection. zoom_qtys = [q for q in DEFAULT_ZOOM_PLOTS if q in full_qtys] or DEFAULT_ZOOM_PLOTS @@ -670,7 +803,8 @@ def _render_phase(args: argparse.Namespace, outdir: Path, full_csv: Path, xlsx_path = outdir / "report.xlsx" print(f"[render] xlsx workbook → {xlsx_path}") write_xlsx(min_csv, xlsx_path, config=config, csv_per_second_path=full_csv, - stats=stats, tod_rows=tod_rows) + stats=stats, tod_rows=tod_rows, narrative=narrative, + demand=demand) html_path = outdir / "report.html" if not args.no_html: @@ -693,6 +827,7 @@ def _render_phase(args: argparse.Namespace, outdir: Path, full_csv: Path, summary_stats=_build_summary_stats(events, snaps), events=events, snapshots=snaps, findings=loaded_findings, + narrative=narrative, ) if args.pdf: @@ -774,6 +909,9 @@ def main(argv: Sequence[str] | None = None) -> int: if raw_argv and raw_argv[0] == "compare": from .cli_compare import compare_main return compare_main(raw_argv[1:]) + if raw_argv and raw_argv[0] == "stitch": + from .cli_stitch import stitch_main + return stitch_main(raw_argv[1:]) # Make console output UTF-8 safe on Windows (default cp1252 chokes on →, σ, etc.) for stream in (sys.stdout, sys.stderr): @@ -795,6 +933,14 @@ def print(*a, **kw): # noqa: A001 — intentional shadow kw.setdefault("file", sys.stderr) _original_print(*a, **kw) + # Resolve --tz once (Feature H). Invalid zones fail fast. + from .tzutil import resolve_tz + try: + args._tz = resolve_tz(getattr(args, "tz", None)) + except ValueError as e: + print(f"ERROR: {e}", file=sys.stderr) + return 1 + if not args.session_dir.exists(): print(f"ERROR: {args.session_dir} does not exist", file=sys.stderr) return 1 @@ -832,8 +978,12 @@ def print(*a, **kw): # noqa: A001 — intentional shadow # Post-detection analysis features (markers, stats, tod, split) run on # the in-memory store + on-disk CSVs. - stats, tod_rows = _run_extra_analyses( + stats, tod_rows, narrative, demand = _run_extra_analyses( args, outdir, store, events, findings, config, full_csv, min_csv) + # Re-write summary.txt with the executive narrative + tz-aware time range. + _write_summary_txt(outdir, events, snaps, findings, config, + narrative=narrative, tz=getattr(args, "_tz", None), + tz_name=getattr(args, "tz", None), store=store) if getattr(args, "json_mode", False): _emit_json(events, snaps, findings, config) @@ -852,7 +1002,8 @@ def print(*a, **kw): # noqa: A001 — intentional shadow args.plot = ",".join(picked_qtys) _render_phase(args, outdir, full_csv, min_csv, events, snaps, config, - stats=stats, tod_rows=tod_rows) + stats=stats, tod_rows=tod_rows, narrative=narrative, + demand=demand) print(f"[done] {outdir}") return 0 diff --git a/python/src/fluke_3540/cli_stitch.py b/python/src/fluke_3540/cli_stitch.py new file mode 100644 index 0000000..0b81dd9 --- /dev/null +++ b/python/src/fluke_3540/cli_stitch.py @@ -0,0 +1,193 @@ +"""``fluke-analyze stitch`` subcommand — concatenate consecutive sessions. + +Stitches two or more sessions (same asset, consecutive captures) into one +continuous timeline that beats the meter's 7-day cap, then runs the normal +analysis over the stitched series: events.json, snapshots.json, insights.json, +stats.json, a stitched session CSV, and a stitch.json provenance file. +""" +from __future__ import annotations + +import argparse +import csv as _csv +import json +import sys +from pathlib import Path +from typing import Sequence + +from .analysis import whole_session_stats +from .events import detect_events +from .insights import analyze as analyze_insights, to_jsonable as finding_to_jsonable +from .parser import ( + _parse_reverse_cts_arg, from_csv, iter_records, open_session, + reverse_cts_indices, +) +from .snapshots import pick_snapshots +from .stitch import stitch_stores +from .store import STORE_COLUMNS, ColumnStore, _resolve_indices + + +def build_stitch_argparser() -> argparse.ArgumentParser: + ap = argparse.ArgumentParser( + prog="fluke-analyze stitch", + description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + ap.add_argument("sessions", nargs="+", type=Path, + help="Two or more session inputs (ES.NNN/ dirs, .fel, or .csv)") + ap.add_argument("-o", "--output", required=True, type=Path, + help="Output directory (will be created)") + ap.add_argument("--labels", type=str, default=None, + help="Comma-separated labels per session (default: input names)") + ap.add_argument("--reverse-cts", nargs="?", const="all", default=None, + metavar="PHASES", + help="Apply the same reverse-CTS phases to every session") + ap.add_argument("--gap-tolerance", type=float, default=2.0, metavar="SECS", + help="Boundary gaps larger than this are recorded (default 2 s)") + ap.add_argument("--nominal-ln-v", type=float, default=None, metavar="V", + help="Nominal L-N voltage (auto-inferred if omitted)") + ap.add_argument("--no-stats", action="store_true", + help="Skip whole-session statistics over the stitched series") + ap.add_argument("--no-csv", action="store_true", + help="Skip writing the (large) stitched session.csv") + return ap + + +def _labels(arg: str | None, sessions: Sequence[Path]) -> list[str]: + if arg: + labels = [s.strip() for s in arg.split(",")] + if len(labels) != len(sessions): + raise SystemExit( + f"--labels count ({len(labels)}) != sessions count ({len(sessions)})") + return labels + out: list[str] = [] + seen: dict[str, int] = {} + for s in sessions: + base = s.name + for suf in (".fel", ".csv"): + if base.lower().endswith(suf): + base = base[:-len(suf)] + break + if base in seen: + seen[base] += 1 + base = f"{base}-{seen[base]}" + else: + seen[base] = 1 + out.append(base) + return out + + +def _store_for(session_input: Path, reverse_cts) -> ColumnStore: + """Build a ColumnStore for one session input (dir/.fel/.csv).""" + if session_input.is_file() and session_input.suffix.lower() == ".csv": + return ColumnStore.from_records(from_csv(session_input)) + flip = reverse_cts_indices(reverse_cts if reverse_cts else False) + col_idx = _resolve_indices(STORE_COLUMNS) + with open_session(session_input) as session_dir: + from .parser import find_session_files + trend = find_session_files(session_dir)["trend"] + store = ColumnStore() + for rec in iter_records(trend): + floats = rec.floats + for name, idx in zip(STORE_COLUMNS, col_idx): + v = floats[idx] + if idx in flip: + v = -v + store._cols[name].append(v) + from .parser import _filetime_ticks + store._start_ticks.append(_filetime_ticks(rec.start)) + store._end_ticks.append(_filetime_ticks(rec.end)) + store._n += 1 + return store + + +def _write_stitched_csv(path: Path, store: ColumnStore, + sources, gaps) -> None: + """Write the stitched series CSV with a `source` provenance column.""" + # Map record index -> source label. + label_of = [""] * store.n + for s in sources: + for i in range(s.lo, s.hi): + label_of[i] = s.label + cols = list(store.columns) + with path.open("w", newline="", encoding="utf-8") as fh: + w = _csv.writer(fh) + w.writerow(["timestamp_utc", "window_end_utc", "source", *cols]) + col_arrays = [store.col(c) for c in cols] + for i in range(store.n): + w.writerow([store.start(i).isoformat(), store.end(i).isoformat(), + label_of[i], *[ca[i] for ca in col_arrays]]) + + +def stitch_main(argv: Sequence[str] | None = None) -> int: + for stream in (sys.stdout, sys.stderr): + if hasattr(stream, "reconfigure"): + try: + stream.reconfigure(encoding="utf-8", errors="replace") + except (AttributeError, ValueError): + pass + + args = build_stitch_argparser().parse_args(argv) + if len(args.sessions) < 2: + print("ERROR: stitch requires at least 2 sessions", file=sys.stderr) + return 1 + for s in args.sessions: + if not s.exists(): + print(f"ERROR: session not found: {s}", file=sys.stderr) + return 1 + + labels = _labels(args.labels, args.sessions) + reverse_cts = _parse_reverse_cts_arg(args.reverse_cts) + outdir: Path = args.output + outdir.mkdir(parents=True, exist_ok=True) + + labelled: list[tuple[str, ColumnStore]] = [] + for lbl, session_input in zip(labels, args.sessions): + print(f"[stitch] parsing {session_input} (label: {lbl})") + st = _store_for(session_input, reverse_cts) + print(f" {st.n:,} records") + labelled.append((lbl, st)) + + result = stitch_stores(labelled, gap_tolerance_secs=args.gap_tolerance) + store = result.store + print(f"[stitch] stitched {len(result.sources)} sessions → {store.n:,} records, " + f"{len(result.gaps)} gap(s)") + for g in result.gaps: + print(f" GAP between {g.after_label} and {g.before_label}: " + f"{g.seconds:.0f}s ({g.t_gap_start} .. {g.t_gap_end})") + + (outdir / "stitch.json").write_text( + json.dumps(result.to_jsonable(), indent=2), encoding="utf-8") + + # Analysis over the stitched timeline. + events = detect_events(store, nominal_ln_v=args.nominal_ln_v) + snaps = pick_snapshots(store, events, n=3) + findings = analyze_insights(store, events, snaps, {}) + print(f"[detect] {len(events)} events, {len(snaps)} snapshots, " + f"{len(findings)} insight(s) over the stitched series") + + def _event_json(ev): + return { + "id": ev.id, "kind": ev.kind, + "t_start": ev.t_start.isoformat(), "t_end": ev.t_end.isoformat(), + "severity": ev.severity, "affected_phases": list(ev.affected_phases), + } + (outdir / "events.json").write_text( + json.dumps([_event_json(e) for e in events], indent=2), encoding="utf-8") + (outdir / "insights.json").write_text( + json.dumps([finding_to_jsonable(f) for f in findings], indent=2), + encoding="utf-8") + + if not args.no_stats: + stats = whole_session_stats(store) + (outdir / "stats.json").write_text( + json.dumps(stats, indent=2), encoding="utf-8") + print("[stats] wrote stats.json over the stitched series") + + if not args.no_csv: + csv_path = outdir / "session.csv" + _write_stitched_csv(csv_path, store, result.sources, result.gaps) + print(f"[stitch] wrote stitched {csv_path.name} " + f"({store.n:,} rows, with source provenance column)") + + print(f"[done] {outdir}") + return 0 diff --git a/python/src/fluke_3540/narrative.py b/python/src/fluke_3540/narrative.py new file mode 100644 index 0000000..7e61c5e --- /dev/null +++ b/python/src/fluke_3540/narrative.py @@ -0,0 +1,148 @@ +"""Auto-narrative / executive summary (Feature E). + +Rule-based plain-English summary built from detected events, insights, whole- +session stats, and the CT-reversal check — no LLM. Deterministic so the same +session always yields the same prose, and so the JS port can match it exactly. + +The narrative is a small ordered set of sentences: + 1. Scope (asset, duration, record count). + 2. Headline event (worst outage / dip / swell), with context. + 3. Power-factor / imbalance summary from stats + insights. + 4. CT-reversal warning if flagged. + 5. A one-line bottom line. +""" +from __future__ import annotations + +import datetime as dt +from typing import Sequence + + +def _fmt_duration(secs: float) -> str: + secs = int(round(secs)) + if secs >= 3600: + return f"{secs / 3600:.1f} h" + if secs >= 60: + return f"{secs / 60:.1f} min" + return f"{secs} s" + + +def _hhmm(iso_or_dt) -> str: + if isinstance(iso_or_dt, str): + try: + d = dt.datetime.fromisoformat(iso_or_dt) + except ValueError: + return iso_or_dt + else: + d = iso_or_dt + return d.strftime("%Y-%m-%d %H:%M UTC") + + +def build_narrative( + events: Sequence, + findings: Sequence, + stats: dict | None, + ct_reversal: dict | None, + config: dict | None = None, + total_records: int | None = None, + duration_secs: float | None = None, +) -> str: + """Return a deterministic plain-English executive summary string. + + ``events`` are Event objects (id/kind/t_start/t_end/severity/affected_phases), + ``findings`` are Finding objects (kind/severity/headline), ``stats`` is the + whole_session_stats dict, ``ct_reversal`` is detect_ct_reversal output. + """ + sentences: list[str] = [] + config = config or {} + asset = config.get("asset_name") + + # 1) Scope + nrec = total_records + if nrec is None and stats: + nrec = stats.get("_thresholds", {}).get("total_records") + scope_bits = [] + if asset: + scope_bits.append(f"Asset {asset}") + else: + scope_bits.append("This session") + if duration_secs: + scope_bits.append(f"captured over {_fmt_duration(duration_secs)}") + if nrec: + scope_bits.append(f"({nrec:,} one-second records)") + sentences.append(" ".join(scope_bits).strip() + ".") + + # 2) Headline event + outages = [e for e in events if e.kind == "outage"] + dips = [e for e in events if e.kind == "dip"] + swells = [e for e in events if e.kind == "swell"] + if outages: + worst = max(outages, key=lambda e: (e.t_end - e.t_start).total_seconds()) + dur = (worst.t_end - worst.t_start).total_seconds() + # Context: leading dip / restoration inrush within 30 s. + lead = next((d for d in dips + if 0 <= (worst.t_start - d.t_end).total_seconds() <= 30), None) + ctx = "" + if lead: + ctx = (f", preceded by a phase-{'/'.join(lead.affected_phases) or '?'} " + f"dip to {lead.severity * 100:.0f}%") + sentences.append( + f"The most significant event was a {_fmt_duration(dur)} outage at " + f"{_hhmm(worst.t_start)}{ctx}.") + elif dips or swells: + worst_dip = min(dips, key=lambda e: e.severity, default=None) + worst_swell = max(swells, key=lambda e: e.severity, default=None) + if worst_dip: + sentences.append( + f"No outages occurred; the deepest voltage dip fell to " + f"{worst_dip.severity * 100:.0f}% of nominal on phase(s) " + f"{'/'.join(worst_dip.affected_phases) or '?'} at " + f"{_hhmm(worst_dip.t_start)}.") + elif worst_swell: + sentences.append( + f"No outages occurred; the largest swell reached " + f"{worst_swell.severity * 100:.0f}% of nominal at " + f"{_hhmm(worst_swell.t_start)}.") + else: + sentences.append("No outages, dips, or swells were detected.") + + # 3) Power factor / imbalance from stats + findings + if stats and "PF_total_avg" in stats: + pf = stats["PF_total_avg"] + sentences.append( + f"Power factor (total) averaged {pf['mean']:.2f} " + f"(p5 {pf['p5']:.2f}, p95 {pf['p95']:.2f}).") + pf_finding = next((f for f in findings if f.kind == "pf_drift"), None) + if pf_finding: + sentences.append(pf_finding.headline.rstrip(".") + ".") + imb_finding = next((f for f in findings if f.kind in + ("imbalance_sustained", "phase_asymmetry")), None) + if imb_finding: + sentences.append(imb_finding.headline.rstrip(".") + ".") + + # 4) CT reversal + if ct_reversal and ct_reversal.get("reversed"): + pct = ct_reversal["frac_negative"] * 100 + sentences.append( + f"WARNING: real power is negative for {pct:.0f}% of non-outage time — " + "the iFlex CTs are likely reversed; re-run with --reverse-cts.") + + # 5) Bottom line + n_alert = sum(1 for f in findings if getattr(f, "severity", "") == "alert") + if n_alert: + sentences.append( + f"Bottom line: {n_alert} alert-level finding(s) warrant follow-up.") + elif events: + sentences.append( + f"Bottom line: {len(events)} event(s) detected; no alert-level findings.") + else: + sentences.append("Bottom line: the supply looks clean over this capture.") + + return " ".join(sentences) + + +def narrative_markdown(narrative: str, config: dict | None = None) -> str: + """Wrap the narrative as a small markdown document for narrative.md.""" + config = config or {} + title = config.get("asset_name") or "Session" + lines = [f"# Executive Summary — {title}", "", narrative, ""] + return "\n".join(lines) diff --git a/python/src/fluke_3540/plots/html_report.py b/python/src/fluke_3540/plots/html_report.py index 73fdf20..703ccca 100644 --- a/python/src/fluke_3540/plots/html_report.py +++ b/python/src/fluke_3540/plots/html_report.py @@ -209,10 +209,17 @@ def render_report_html( charts: Iterable[tuple[str, bytes]], findings: Sequence[Finding] = (), generated_at: dt.datetime | None = None, + narrative: str | None = None, ) -> str: generated_at = generated_at or dt.datetime.now(dt.timezone.utc) body = [] body.append(f"

{html.escape(title)}

") + if narrative: + body.append( + "

Executive summary

" + + html.escape(narrative).replace("\n", "
") + + "

" + ) body.append("

Summary

") body.append(_summary_dl_html(summary_stats, config)) if findings: @@ -246,6 +253,7 @@ def write_html_report( snapshots: Sequence[Snapshot], findings: Sequence[Finding] = (), title: str | None = None, + narrative: str | None = None, ) -> Path: """High-level wrapper: read PNGs from charts_dir, write a self-contained HTML report. @@ -264,7 +272,7 @@ def write_html_report( render_report_html( title=title, config=config, summary_stats=summary_stats, events=events, snapshots=snapshots, charts=charts, - findings=findings, + findings=findings, narrative=narrative, ), encoding="utf-8", ) diff --git a/python/src/fluke_3540/plots/xlsx.py b/python/src/fluke_3540/plots/xlsx.py index 1fdccab..4766e1e 100644 --- a/python/src/fluke_3540/plots/xlsx.py +++ b/python/src/fluke_3540/plots/xlsx.py @@ -184,6 +184,8 @@ def write_xlsx( *, csv_per_second_path: Path | None = None, stats: Mapping | None = None, tod_rows=None, + narrative: str | None = None, + demand: Mapping | None = None, ) -> Path: """Build the chartable XLSX. @@ -315,6 +317,10 @@ def add_single_chart_sheet(title, chart_t, y, cols): ws_sum["A1"] = "Fluke 3540 FC Session Summary" ws_sum["A1"].font = Font(size=16, bold=True) ws_sum.merge_cells("A1:C1") + if narrative: + ws_sum["A2"] = "Executive summary: " + narrative + ws_sum["A2"].alignment = Alignment(wrap_text=True, vertical="top") + ws_sum.merge_cells("A2:F2") cfg = config or {} total = summary["sec_import"] + summary["sec_export"] + summary["sec_idle"] @@ -343,6 +349,16 @@ def add_single_chart_sheet(title, chart_t, y, cols): ("Peak export power (kW)", f"{summary['p_peak_neg'] / 1000:,.2f}"), ("Peak current (A)", f"{summary['i_peak']:.1f}"), ("", ""), + ]) + if demand and demand.get("n_windows"): + wmin = demand["window_secs"] // 60 + summary_rows.extend([ + (f"Peak demand ({wmin}-min, kW)", f"{demand['peak_demand_kw']:,.2f}"), + ("Peak demand window end", str(demand.get("peak_window_end") or "")), + ("Mean demand (kW)", f"{demand['mean_demand_w'] / 1000:,.2f}"), + ("", ""), + ]) + summary_rows.extend([ ("Time importing", f"{summary['sec_import']:,} s ({pct(summary['sec_import'])})"), ("Time exporting", f"{summary['sec_export']:,} s ({pct(summary['sec_export'])})"), ("Time idle (|P|<10W)", f"{summary['sec_idle']:,} s ({pct(summary['sec_idle'])})"), diff --git a/python/src/fluke_3540/rules_file.py b/python/src/fluke_3540/rules_file.py new file mode 100644 index 0000000..436bd6c --- /dev/null +++ b/python/src/fluke_3540/rules_file.py @@ -0,0 +1,121 @@ +"""Per-asset event-threshold overrides (Feature I). + +``--rules-file FILE`` (JSON or TOML) overrides the default :class:`EventRules` +thresholds. The file may carry a flat set of defaults and/or a per-asset map so +one file can hold the known trip points for a whole fleet: + +JSON:: + + { + "defaults": { "dip_pct_of_nominal": 0.92 }, + "assets": { + "P115RE-MAC03": { "outage_v_threshold": 60.0, "swell_pct_of_nominal": 1.08 } + } + } + +TOML:: + + [defaults] + dip_pct_of_nominal = 0.92 + + [assets."P115RE-MAC03"] + outage_v_threshold = 60.0 + +A flat file with only threshold keys (no ``defaults``/``assets``) is treated as +defaults. Per-asset values win over defaults; unknown keys are reported. +""" +from __future__ import annotations + +import dataclasses +import json +from pathlib import Path + +from .events import DEFAULT_RULES, EventRules + +try: # Python 3.11+ + import tomllib +except ImportError: # pragma: no cover + tomllib = None + + +_VALID_KEYS = {f.name for f in dataclasses.fields(EventRules)} + + +def _load_raw(path: Path) -> dict: + text = path.read_text(encoding="utf-8") + if path.suffix.lower() == ".toml": + if tomllib is None: # pragma: no cover + raise ValueError("TOML rules-file requires Python 3.11+ (tomllib)") + return tomllib.loads(text) + if path.suffix.lower() in (".json", ".jsn"): + return json.loads(text) + # Try JSON first, then TOML, so a misnamed file still loads. + try: + return json.loads(text) + except json.JSONDecodeError: + if tomllib is not None: + return tomllib.loads(text) + raise + + +def _split(raw: dict) -> tuple[dict, dict]: + """Return (defaults, assets) from a raw rules dict. + + A file with no ``defaults``/``assets`` keys is treated as flat defaults. + """ + if "defaults" in raw or "assets" in raw: + return dict(raw.get("defaults") or {}), dict(raw.get("assets") or {}) + # Flat file = defaults only. + return dict(raw), {} + + +def _coerce(overrides: dict, where: str) -> dict: + """Validate keys + numeric-coerce values, raising on unknown keys.""" + out: dict = {} + bad = [k for k in overrides if k not in _VALID_KEYS] + if bad: + raise ValueError( + f"{where}: unknown EventRules key(s) {sorted(bad)}. " + f"Valid: {sorted(_VALID_KEYS)}") + for k, v in overrides.items(): + # min_duration_secs / gap_tolerance_secs are ints; the rest are floats. + if k in ("min_duration_secs", "gap_tolerance_secs"): + out[k] = int(v) + else: + out[k] = float(v) + return out + + +def load_rules(path: Path, asset_name: str | None = None, + base: EventRules = DEFAULT_RULES) -> EventRules: + """Build an :class:`EventRules` from a rules file for the given asset. + + Precedence (low → high): base defaults → file ``defaults`` → file + ``assets[asset_name]``. Asset lookup matches ``asset_name`` exactly, then + falls back to a ``"default"`` entry under ``assets`` if present. + """ + raw = _load_raw(path) + if not isinstance(raw, dict): + raise ValueError(f"{path}: rules file must be a mapping at the top level") + defaults, assets = _split(raw) + merged = dict(_coerce(defaults, f"{path}:defaults")) + asset_over: dict = {} + if asset_name and asset_name in assets: + asset_over = assets[asset_name] + elif "default" in assets: + asset_over = assets["default"] + merged.update(_coerce(asset_over, f"{path}:assets")) + return dataclasses.replace(base, **merged) + + +def describe_overrides(path: Path, asset_name: str | None, + base: EventRules = DEFAULT_RULES) -> list[str]: + """Return human-readable 'key: base -> new' lines for the applied overrides.""" + rules = load_rules(path, asset_name, base) + lines: list[str] = [] + for f in dataclasses.fields(EventRules): + b = getattr(base, f.name) + n = getattr(rules, f.name) + if b != n: + lines.append(f"{f.name}: {b} -> {n}") + return lines diff --git a/python/src/fluke_3540/stitch.py b/python/src/fluke_3540/stitch.py new file mode 100644 index 0000000..1f71cd2 --- /dev/null +++ b/python/src/fluke_3540/stitch.py @@ -0,0 +1,115 @@ +"""Multi-session stitching (Feature D). + +Concatenate consecutive Fluke sessions into one continuous timeline so analysis +can run across the meter's 7-day capture cap. Sessions are ordered by start +time; where session N+1 does not abut session N (gap > tolerance) an explicit +gap record is noted in the provenance so downstream consumers can see it. + +The stitched result is a single :class:`~fluke_3540.store.ColumnStore` plus a +provenance list (one entry per source session: label, record range, time span) +and a gaps list. Stdlib only. +""" +from __future__ import annotations + +import datetime as dt +from dataclasses import dataclass + +from .store import STORE_COLUMNS, ColumnStore + + +@dataclass(frozen=True) +class SourceSpan: + """Provenance for one source session inside the stitched timeline.""" + label: str + lo: int # first stitched record index (inclusive) + hi: int # last stitched record index (exclusive) + t_start: str # ISO start of this source within the timeline + t_end: str # ISO end + records: int + + +@dataclass(frozen=True) +class Gap: + """A discontinuity between two consecutive sources.""" + after_label: str + before_label: str + t_gap_start: str # end of the earlier session + t_gap_end: str # start of the later session + seconds: float + + +@dataclass +class StitchResult: + store: ColumnStore + sources: list[SourceSpan] + gaps: list[Gap] + + def to_jsonable(self) -> dict: + return { + "total_records": self.store.n, + "sources": [vars(s) for s in self.sources], + "gaps": [vars(g) for g in self.gaps], + } + + +def stitch_stores( + labelled_stores: list[tuple[str, ColumnStore]], + gap_tolerance_secs: float = 2.0, +) -> StitchResult: + """Stitch labelled ColumnStores into one continuous-timeline store. + + Args: + labelled_stores: ``[(label, store), ...]`` — order is irrelevant, they + are sorted by first-record start time. + gap_tolerance_secs: sessions whose boundary differs by more than this + are recorded as a :class:`Gap` (records are still concatenated in + time order; no synthetic fill rows are inserted). + + Returns a :class:`StitchResult`. Empty stores are skipped. + """ + usable = [(lbl, st) for lbl, st in labelled_stores if st.n > 0] + if not usable: + return StitchResult(ColumnStore(), [], []) + usable.sort(key=lambda ls: ls[1].start(0)) + + out = ColumnStore(time_shift=dt.timedelta(0)) + sources: list[SourceSpan] = [] + gaps: list[Gap] = [] + prev_end: dt.datetime | None = None + prev_label: str | None = None + + for label, st in usable: + lo = out.n + # Record a gap if this session does not abut the previous one. + cur_start = st.start(0) + if prev_end is not None: + delta = (cur_start - prev_end).total_seconds() + if abs(delta) > gap_tolerance_secs: + gaps.append(Gap( + after_label=prev_label or "?", + before_label=label, + t_gap_start=prev_end.isoformat(), + t_gap_end=cur_start.isoformat(), + seconds=delta, + )) + # Append every record's retained columns + absolute (shifted) ticks. + for name in STORE_COLUMNS: + out._cols[name].extend(st._cols[name]) + # Carry absolute ticks (apply each store's own time_shift so the + # stitched store needs no further shift). + shift_ticks = int(round(st.time_shift.total_seconds() * 10_000_000)) + out._start_ticks.extend(t + shift_ticks for t in st._start_ticks) + out._end_ticks.extend(t + shift_ticks for t in st._end_ticks) + out._n += st.n + + hi = out.n + sources.append(SourceSpan( + label=label, lo=lo, hi=hi, + t_start=st.start(0).isoformat(), + t_end=st.end(st.n - 1).isoformat(), + records=st.n, + )) + prev_end = st.end(st.n - 1) + prev_label = label + + return StitchResult(store=out, sources=sources, gaps=gaps) diff --git a/python/src/fluke_3540/store.py b/python/src/fluke_3540/store.py index 00f0189..e6a7feb 100644 --- a/python/src/fluke_3540/store.py +++ b/python/src/fluke_3540/store.py @@ -45,6 +45,9 @@ "P_total_avg_W", "S_total_avg_VA", "Q_total_avg_VAR", "PF_total_avg", # Per-row energy (used by per-bucket kWh roll-ups) "Wh_total", + # THD per phase (IEEE 519) — V and I, avg only + "V_THD_pct_a_avg", "V_THD_pct_b_avg", "V_THD_pct_c_avg", + "I_THD_pct_a_avg", "I_THD_pct_b_avg", "I_THD_pct_c_avg", ) _FIELD_INDEX = {f.name: f.index for f in FIELDS} diff --git a/python/src/fluke_3540/tzutil.py b/python/src/fluke_3540/tzutil.py new file mode 100644 index 0000000..93f0c08 --- /dev/null +++ b/python/src/fluke_3540/tzutil.py @@ -0,0 +1,68 @@ +"""Timezone-aware reporting helpers (Feature H). + +Timestamps are stored/computed in UTC throughout the pipeline (the meter's +FILETIME is UTC, anchors are normalised to UTC). When the operator passes +``--tz ZONE`` (an IANA name like ``America/Chicago``), reports additionally +render the local wall-clock alongside UTC. Default behaviour (no ``--tz``) is +unchanged: UTC only. + +Anchors (``--anchor-start`` / ``--anchor-end``) already accept ISO-8601 strings +with an explicit offset (e.g. ``2024-01-13T09:00:00-06:00``); that offset is +honoured by ``datetime.fromisoformat`` in cli._parse_time. ``--tz`` is the +display-side complement. +""" +from __future__ import annotations + +import datetime as dt + +try: # Python 3.9+ + from zoneinfo import ZoneInfo, ZoneInfoNotFoundError +except ImportError: # pragma: no cover + ZoneInfo = None + ZoneInfoNotFoundError = Exception + + +def resolve_tz(name: str | None): + """Return a tzinfo for ``name`` (IANA), or None for UTC/unset. + + Raises ValueError on an unknown zone so the CLI can report it cleanly. + """ + if not name: + return None + if name.upper() == "UTC": + return dt.timezone.utc + if ZoneInfo is None: # pragma: no cover + raise ValueError("zoneinfo unavailable; --tz requires Python 3.9+") + try: + return ZoneInfo(name) + except (ZoneInfoNotFoundError, KeyError, ValueError) as e: + raise ValueError(f"Unknown timezone: {name!r}") from e + + +def to_utc(value: dt.datetime) -> dt.datetime: + """Normalise a datetime to UTC (naive is assumed UTC).""" + if value.tzinfo is None: + return value.replace(tzinfo=dt.timezone.utc) + return value.astimezone(dt.timezone.utc) + + +def format_local_utc(value: dt.datetime, tz) -> str: + """Render ``value`` as 'LOCAL (UTC)' when tz is set, else just UTC ISO. + + Example with tz=America/Chicago: + '2024-01-13T09:00:00-06:00 (2024-01-13T15:00:00+00:00)' + With tz=None: + '2024-01-13T15:00:00+00:00' + """ + utc = to_utc(value) + if tz is None: + return utc.isoformat() + local = utc.astimezone(tz) + return f"{local.isoformat()} ({utc.isoformat()})" + + +def tz_label(tz, name: str | None) -> str: + """A short label for the configured zone, for report headers.""" + if tz is None: + return "UTC" + return name or "local" diff --git a/python/tests/fixtures/analysis_golden.json b/python/tests/fixtures/analysis_golden.json new file mode 100644 index 0000000..2a3ec33 --- /dev/null +++ b/python/tests/fixtures/analysis_golden.json @@ -0,0 +1,437 @@ +{ + "stats": { + "V_LN_a_avg_V": { + "unit": "V", + "count": 600, + "min": 240.0, + "p1": 240.05, + "p5": 274.05, + "median": 277.05, + "mean": 275.7550000000001, + "p95": 280.05, + "p99": 280.05, + "max": 280.0, + "stdev": 6.9239902994348 + }, + "V_LN_b_avg_V": { + "unit": "V", + "count": 600, + "min": 277.0, + "p1": 277.05, + "p5": 277.05, + "median": 277.05, + "mean": 277.0, + "p95": 277.05, + "p99": 277.05, + "max": 277.0, + "stdev": 0.0 + }, + "V_LN_c_avg_V": { + "unit": "V", + "count": 600, + "min": 277.0, + "p1": 277.05, + "p5": 277.05, + "median": 277.05, + "mean": 277.0, + "p95": 277.05, + "p99": 277.05, + "max": 277.0, + "stdev": 0.0 + }, + "I_a_avg_A": { + "unit": "A", + "count": 600, + "min": 100.0, + "p1": 100.125, + "p5": 100.125, + "median": 105.125, + "mean": 104.975, + "p95": 110.125, + "p99": 110.125, + "max": 110.0, + "stdev": 3.160860905934752 + }, + "I_b_avg_A": { + "unit": "A", + "count": 600, + "min": 100.0, + "p1": 100.125, + "p5": 100.125, + "median": 100.125, + "mean": 100.0, + "p95": 100.125, + "p99": 100.125, + "max": 100.0, + "stdev": 0.0 + }, + "I_c_avg_A": { + "unit": "A", + "count": 600, + "min": 100.0, + "p1": 100.125, + "p5": 100.125, + "median": 100.125, + "mean": 106.25000000000011, + "p95": 100.125, + "p99": 100.125, + "max": 850.0, + "stdev": 68.17945071647314 + }, + "freq_avg_Hz": { + "unit": "Hz", + "count": 600, + "min": 59.9900016784668, + "p1": 59.99125, + "p5": 59.99125, + "median": 60.00125, + "mean": 59.999999999999936, + "p95": 60.00875, + "p99": 60.00875, + "max": 60.0099983215332, + "stdev": 0.008163595346876485 + }, + "P_total_avg_W": { + "unit": "W", + "count": 600, + "min": 50000.0, + "p1": 50500.0, + "p5": 52500.0, + "median": 74500.0, + "mean": 74500.00000000013, + "p95": 97500.0, + "p99": 99500.0, + "max": 99000.0, + "stdev": 14430.86968966181 + }, + "S_total_avg_VA": { + "unit": "VA", + "count": 600, + "min": 0.0, + "p1": 500.0, + "p5": 500.0, + "median": 500.0, + "mean": 0.0, + "p95": 500.0, + "p99": 500.0, + "max": 0.0, + "stdev": 0.0 + }, + "Q_total_avg_VAR": { + "unit": "VAR", + "count": 600, + "min": 0.0, + "p1": 500.0, + "p5": 500.0, + "median": 500.0, + "mean": 0.0, + "p95": 500.0, + "p99": 500.0, + "max": 0.0, + "stdev": 0.0 + }, + "PF_total_avg": { + "unit": "", + "count": 600, + "min": 0.8999999761581421, + "p1": 0.9001125000000003, + "p5": 0.9001125000000003, + "median": 0.9200625000000002, + "mean": 0.9200000047683707, + "p95": 0.9400125000000001, + "p99": 0.9400125000000001, + "max": 0.9399999976158142, + "stdev": 0.01414213899548889 + }, + "_thresholds": { + "undervoltage_v": 250.0, + "sec_undervoltage": 20, + "pct_undervoltage": 3.3333333333333335, + "overcurrent_a": 800.0, + "sec_overcurrent": 5, + "pct_overcurrent": 0.8333333333333334, + "total_records": 600 + } + }, + "tod_rows": [ + { + "bin": "22:00", + "n": 60, + "n_days": 1, + "p_avg_kW": 71.16666666666667, + "p_min_kW": 50.0, + "p_max_kW": 99.0, + "v_avg_V": 276.9, + "v_min_V": 274.0, + "v_max_V": 280.0, + "i_avg_A": 104.75, + "i_min_A": 100.0, + "i_max_A": 110.0 + }, + { + "bin": "22:01", + "n": 60, + "n_days": 1, + "p_avg_kW": 72.83333333333333, + "p_min_kW": 50.0, + "p_max_kW": 99.0, + "v_avg_V": 264.68333333333334, + "v_min_V": 240.0, + "v_max_V": 280.0, + "i_avg_A": 105.16666666666667, + "i_min_A": 100.0, + "i_max_A": 110.0 + }, + { + "bin": "22:02", + "n": 60, + "n_days": 1, + "p_avg_kW": 74.5, + "p_min_kW": 50.0, + "p_max_kW": 99.0, + "v_avg_V": 276.96666666666664, + "v_min_V": 274.0, + "v_max_V": 280.0, + "i_avg_A": 104.85, + "i_min_A": 100.0, + "i_max_A": 110.0 + }, + { + "bin": "22:03", + "n": 60, + "n_days": 1, + "p_avg_kW": 76.16666666666667, + "p_min_kW": 50.0, + "p_max_kW": 99.0, + "v_avg_V": 277.0, + "v_min_V": 274.0, + "v_max_V": 280.0, + "i_avg_A": 105.08333333333333, + "i_min_A": 100.0, + "i_max_A": 110.0 + }, + { + "bin": "22:04", + "n": 60, + "n_days": 1, + "p_avg_kW": 77.83333333333333, + "p_min_kW": 50.0, + "p_max_kW": 99.0, + "v_avg_V": 277.03333333333336, + "v_min_V": 274.0, + "v_max_V": 280.0, + "i_avg_A": 104.95, + "i_min_A": 100.0, + "i_max_A": 110.0 + }, + { + "bin": "22:05", + "n": 60, + "n_days": 1, + "p_avg_kW": 71.16666666666667, + "p_min_kW": 50.0, + "p_max_kW": 99.0, + "v_avg_V": 276.95, + "v_min_V": 274.0, + "v_max_V": 280.0, + "i_avg_A": 105.0, + "i_min_A": 100.0, + "i_max_A": 110.0 + }, + { + "bin": "22:06", + "n": 60, + "n_days": 1, + "p_avg_kW": 72.83333333333333, + "p_min_kW": 50.0, + "p_max_kW": 99.0, + "v_avg_V": 277.1, + "v_min_V": 274.0, + "v_max_V": 280.0, + "i_avg_A": 105.05, + "i_min_A": 100.0, + "i_max_A": 110.0 + }, + { + "bin": "22:07", + "n": 60, + "n_days": 1, + "p_avg_kW": 74.5, + "p_min_kW": 50.0, + "p_max_kW": 99.0, + "v_avg_V": 276.9, + "v_min_V": 274.0, + "v_max_V": 280.0, + "i_avg_A": 104.91666666666667, + "i_min_A": 100.0, + "i_max_A": 110.0 + }, + { + "bin": "22:08", + "n": 60, + "n_days": 1, + "p_avg_kW": 76.16666666666667, + "p_min_kW": 50.0, + "p_max_kW": 99.0, + "v_avg_V": 277.05, + "v_min_V": 274.0, + "v_max_V": 280.0, + "i_avg_A": 105.15, + "i_min_A": 100.0, + "i_max_A": 110.0 + }, + { + "bin": "22:09", + "n": 60, + "n_days": 1, + "p_avg_kW": 77.83333333333333, + "p_min_kW": 50.0, + "p_max_kW": 99.0, + "v_avg_V": 276.96666666666664, + "v_min_V": 274.0, + "v_max_V": 280.0, + "i_avg_A": 104.83333333333333, + "i_min_A": 100.0, + "i_max_A": 110.0 + } + ], + "itic_points": [ + [ + 70.0, + 0.1 + ], + [ + 60.0, + 1.0 + ], + [ + 130.0, + 0.4 + ], + [ + 95.0, + 5.0 + ], + [ + 0.0, + 18.0 + ] + ], + "itic": [ + "no_interruption", + "no_damage", + "prohibited", + "no_interruption", + "no_damage" + ], + "event_itic": [ + { + "residual_pct": 72.0, + "duration_secs": 2.0, + "itic_class": "no_damage" + }, + { + "residual_pct": 0.0, + "duration_secs": 120.0, + "itic_class": "no_damage" + }, + { + "residual_pct": 113.99999999999999, + "duration_secs": 1.0, + "itic_class": "no_interruption" + } + ], + "ct_reversal": { + "reversed": true, + "frac_negative": 0.7, + "non_outage_records": 100, + "negative_records": 70, + "mean_p_w": -12000.0, + "threshold": 0.5 + }, + "narrative": "Asset MAC03 captured over 168.0 h (590,000 one-second records). The most significant event was a 18.5 min outage at 2024-01-13 22:01 UTC, preceded by a phase-c dip to 72%. Power factor (total) averaged 0.81 (p5 0.70, p95 0.95). Power factor below 0.85 for 99.8% of non-outage time. WARNING: real power is negative for 52% of non-outage time \u2014 the iFlex CTs are likely reversed; re-run with --reverse-cts. Bottom line: 1 alert-level finding(s) warrant follow-up.", + "ieee519": { + "limit_v_thd_pct": 8.0, + "planning_v_thd_pct": 5.0, + "voltage": { + "a": { + "p95": 9.025, + "limit": 8.0, + "planning": 5.0, + "compliant": false, + "exceeds_planning": true + }, + "b": { + "p95": 2.7750000000000004, + "limit": 8.0, + "planning": 5.0, + "compliant": true, + "exceeds_planning": false + }, + "c": { + "p95": 4.675, + "limit": 8.0, + "planning": 5.0, + "compliant": true, + "exceeds_planning": false + } + }, + "current": { + "a": { + "p95": 14.05 + }, + "b": { + "p95": 0.05 + }, + "c": { + "p95": 0.05 + } + }, + "all_voltage_compliant": false + }, + "sarfi": { + "SARFI-90": 2, + "SARFI-80": 2, + "SARFI-70": 1, + "SARFI-50": 1, + "SARFI-10": 1, + "events_considered": 2, + "nominal_ln_v": 277.0 + }, + "demand": { + "window_secs": 120, + "peak_demand_w": 53950.0, + "peak_demand_kw": 53.95, + "peak_window_end": "2024-01-13T22:10:00+00:00", + "peak_window_start": "2024-01-13T22:08:00+00:00", + "mean_demand_w": 29950.0, + "n_windows": 481, + "series": [ + { + "t": "2024-01-13T22:02:00+00:00", + "demand_w": 5950.0 + }, + { + "t": "2024-01-13T22:04:00+00:00", + "demand_w": 17950.0 + }, + { + "t": "2024-01-13T22:06:00+00:00", + "demand_w": 29950.0 + }, + { + "t": "2024-01-13T22:08:00+00:00", + "demand_w": 41950.0 + }, + { + "t": "2024-01-13T22:10:00+00:00", + "demand_w": 53950.0 + } + ] + }, + "timezone": { + "epoch_ms": 1705158000000, + "utc": "2024-01-13T15:00:00+00:00", + "chicago": "2024-01-13T09:00:00-06:00 (2024-01-13T15:00:00+00:00)" + } +} \ No newline at end of file diff --git a/python/tests/test_analysis_parity_golden.py b/python/tests/test_analysis_parity_golden.py new file mode 100644 index 0000000..9fb1419 --- /dev/null +++ b/python/tests/test_analysis_parity_golden.py @@ -0,0 +1,157 @@ +"""Emit golden analysis outputs for the JS parity test (Feature B). + +This test builds a deterministic session, runs the Python analysis functions, +and writes the results to python/tests/fixtures/analysis_golden.json. The JS +side (web/tests/analysis_parity.test.js) loads the same JSON and asserts its +own port produces identical numbers within float tolerance. + +Keeping the generator in pytest (rather than a standalone script) means the +golden file is regenerated whenever the suite runs, so it can never drift from +the Python implementation. +""" +from __future__ import annotations + +import datetime as dt +import json +from pathlib import Path + +from fluke_3540.analysis import ( + classify_itic, demand_analysis, detect_ct_reversal, event_itic, + ieee519_compliance, sarfi_indices, time_of_day_profile, whole_session_stats, +) +from fluke_3540.events import Event +from fluke_3540.insights import Finding +from fluke_3540.narrative import build_narrative +from fluke_3540.store import ColumnStore + +from conftest import make_records, plant_window + + +GOLDEN_PATH = Path(__file__).parent / "fixtures" / "analysis_golden.json" + + +def _build_session() -> ColumnStore: + """A deterministic multi-shape session the JS test recreates exactly. + + - 600 records (10 minutes) starting 2024-01-13 22:00:00 UTC + - P_total ramps so mean/percentiles are non-trivial + - V_LN_a wobbles for a real stdev + - one undervoltage window and one overcurrent window for thresholds + - one dip window for the time-of-day / ITIC checks + """ + overrides: dict = {} + for i in range(600): + overrides.setdefault(i, {}) + overrides[i]["P_total_avg_W"] = 50_000.0 + (i % 50) * 1000.0 + overrides[i]["V_LN_a_avg_V"] = 277.0 + (i % 7) - 3.0 + overrides[i]["I_a_avg_A"] = 100.0 + (i % 11) + overrides[i]["PF_total_avg"] = 0.90 + (i % 5) * 0.01 + overrides[i]["freq_avg_Hz"] = 60.0 + ((i % 3) - 1) * 0.01 + overrides[i]["V_THD_pct_a_avg"] = 3.0 + (i % 13) * 0.5 # spans planning/limit + overrides[i]["V_THD_pct_b_avg"] = 2.0 + (i % 5) * 0.2 + overrides[i]["V_THD_pct_c_avg"] = 4.5 + (i % 3) * 0.1 + overrides[i]["I_THD_pct_a_avg"] = 8.0 + (i % 7) + plant_window(overrides, 100, 119, {"V_LN_a_avg_V": 240.0}) # undervoltage 20 s + plant_window(overrides, 200, 204, {"I_c_avg_A": 850.0}) # overcurrent 5 s + recs = make_records(600, overrides=overrides) + return ColumnStore.from_records(recs) + + +def test_emit_analysis_golden(): + store = _build_session() + stats = whole_session_stats(store) + tod = time_of_day_profile(store, window=(0, 1440), bin_minutes=1) + + # A handful of ITIC classifications spanning each region. + itic_points = [ + [70.0, 0.1], + [60.0, 1.0], + [130.0, 0.4], + [95.0, 5.0], + [0.0, 18.0], + ] + itic = [classify_itic(p, d) for p, d in itic_points] + + # event_itic on a representative dip + outage + swell. + base = dt.datetime(2024, 1, 13, 22, 0, 0, tzinfo=dt.timezone.utc) + sample_events = [ + Event(0, "dip", base, base + dt.timedelta(seconds=2), 0.72, ("a",)), + Event(1, "outage", base, base + dt.timedelta(seconds=120), 0.0, ("a", "b", "c")), + Event(2, "swell", base, base + dt.timedelta(seconds=1), 1.14, ("b",)), + ] + event_itic_out = [event_itic(e, 277.0) for e in sample_events] + + # CT-reversal detection on a mixed session: 70% negative-P, 30% positive. + ct_overrides: dict = {} + plant_window(ct_overrides, 0, 69, {"P_total_avg_W": -30_000.0}) + plant_window(ct_overrides, 70, 99, {"P_total_avg_W": 30_000.0}) + ct_store = ColumnStore.from_records(make_records(100, overrides=ct_overrides)) + ct = detect_ct_reversal(ct_store) + + # Narrative golden — fixed events/findings/stats/ct so the JS port can match. + narr_base = dt.datetime(2024, 1, 13, 22, 0, 0, tzinfo=dt.timezone.utc) + narr_events = [ + Event(0, "dip", narr_base + dt.timedelta(seconds=90), + narr_base + dt.timedelta(seconds=92), 0.72, ("c",)), + Event(1, "outage", narr_base + dt.timedelta(seconds=100), + narr_base + dt.timedelta(seconds=1210), 0.0, ("a", "b", "c")), + ] + narr_findings = [ + Finding(0, "pf_drift", "alert", + "Power factor below 0.85 for 99.8% of non-outage time", + "", (), ()), + ] + narr_stats = { + "PF_total_avg": {"mean": 0.81, "p5": 0.70, "p95": 0.95}, + "_thresholds": {"total_records": 590000}, + } + narr_ct = {"reversed": True, "frac_negative": 0.52} + narrative = build_narrative( + narr_events, narr_findings, narr_stats, narr_ct, + config={"asset_name": "MAC03"}, total_records=590000, + duration_secs=604800.0, + ) + + # IEEE 519 on the main session; SARFI on the sample events. + ieee519 = ieee519_compliance(store) + sarfi = sarfi_indices(sample_events, 277.0) + + # Demand on a deterministic ramp store (P = i*100 W over 600 s), 120 s window. + ramp_over = {i: {"P_total_avg_W": float(i) * 100.0} for i in range(600)} + ramp_store = ColumnStore.from_records(make_records(600, overrides=ramp_over)) + demand = demand_analysis(ramp_store, window_secs=120, series_step_secs=120) + + golden = { + "stats": stats, + "tod_rows": tod, + "itic_points": itic_points, + "itic": itic, + "event_itic": event_itic_out, + "ct_reversal": ct, + "narrative": narrative, + "ieee519": ieee519, + "sarfi": sarfi, + "demand": demand, + } + + # Timezone formatting golden (Feature H): a fixed UTC instant rendered in + # UTC and America/Chicago. Stored as epoch ms so the JS test uses the same + # instant regardless of how it parses ISO. + from fluke_3540.tzutil import format_local_utc, resolve_tz + tz_instant = dt.datetime(2024, 1, 13, 15, 0, 0, tzinfo=dt.timezone.utc) + golden["timezone"] = { + "epoch_ms": int(tz_instant.timestamp() * 1000), + "utc": format_local_utc(tz_instant, None), + "chicago": format_local_utc(tz_instant, resolve_tz("America/Chicago")), + } + GOLDEN_PATH.parent.mkdir(parents=True, exist_ok=True) + GOLDEN_PATH.write_text(json.dumps(golden, indent=2), encoding="utf-8") + + # Sanity assertions so this test fails loudly if analysis regresses. + assert stats["_thresholds"]["sec_undervoltage"] == 20 + assert stats["_thresholds"]["sec_overcurrent"] == 5 + assert stats["P_total_avg_W"]["count"] == 600 + assert itic[0] == "no_interruption" + assert len(tod) > 0 + assert ct["reversed"] is True + assert abs(ct["frac_negative"] - 0.70) < 1e-9 diff --git a/python/tests/test_ct_reversal.py b/python/tests/test_ct_reversal.py new file mode 100644 index 0000000..dbd7f27 --- /dev/null +++ b/python/tests/test_ct_reversal.py @@ -0,0 +1,71 @@ +"""Tests for CT-reversal auto-detection (Feature C).""" +from __future__ import annotations + +from fluke_3540.analysis import ct_reversal_notice, detect_ct_reversal +from fluke_3540.store import ColumnStore + +from conftest import make_records, plant_window + + +def test_healthy_load_not_flagged(): + # Positive P everywhere — a normal load. + recs = make_records(200, defaults={"P_total_avg_W": 50_000.0}) + store = ColumnStore.from_records(recs) + res = detect_ct_reversal(store) + assert res["reversed"] is False + assert res["frac_negative"] == 0.0 + assert res["mean_p_w"] > 0 + + +def test_reversed_load_flagged(): + # Negative P everywhere (backwards CTs). + recs = make_records(200, defaults={"P_total_avg_W": -50_000.0}) + store = ColumnStore.from_records(recs) + res = detect_ct_reversal(store) + assert res["reversed"] is True + assert res["frac_negative"] == 1.0 + assert res["mean_p_w"] < 0 + assert res["negative_records"] == 200 + + +def test_outage_samples_excluded(): + # 100 records: 50 negative-P load + a 50-sample outage (all V=0, P=0). + overrides: dict = {} + plant_window(overrides, 0, 49, {"P_total_avg_W": -40_000.0}) + plant_window(overrides, 50, 99, { + "V_LN_a_avg_V": 0.0, "V_LN_b_avg_V": 0.0, "V_LN_c_avg_V": 0.0, + "P_total_avg_W": 0.0, + }) + recs = make_records(100, overrides=overrides) + store = ColumnStore.from_records(recs) + res = detect_ct_reversal(store) + # Only the 50 non-outage records count; all of them are negative. + assert res["non_outage_records"] == 50 + assert res["negative_records"] == 50 + assert res["frac_negative"] == 1.0 + assert res["reversed"] is True + + +def test_threshold_boundary(): + # 40% negative — below the default 60% threshold -> not flagged. + overrides: dict = {} + plant_window(overrides, 0, 39, {"P_total_avg_W": -10_000.0}) + plant_window(overrides, 40, 99, {"P_total_avg_W": 10_000.0}) + recs = make_records(100, overrides=overrides) + store = ColumnStore.from_records(recs) + res = detect_ct_reversal(store) + assert abs(res["frac_negative"] - 0.40) < 1e-9 + assert res["reversed"] is False + # Lowering the threshold flags it. + res2 = detect_ct_reversal(store, neg_fraction_threshold=0.30) + assert res2["reversed"] is True + + +def test_notice_is_loud_and_mentions_flags(): + recs = make_records(100, defaults={"P_total_avg_W": -50_000.0}) + store = ColumnStore.from_records(recs) + res = detect_ct_reversal(store) + notice = ct_reversal_notice(res) + assert "CT REVERSAL DETECTED" in notice + assert "--reverse-cts" in notice + assert "--auto-reverse-cts" in notice diff --git a/python/tests/test_demand.py b/python/tests/test_demand.py new file mode 100644 index 0000000..cf419c8 --- /dev/null +++ b/python/tests/test_demand.py @@ -0,0 +1,57 @@ +"""Tests for rolling peak-demand analysis (Feature G).""" +from __future__ import annotations + +from fluke_3540.analysis import demand_analysis +from fluke_3540.store import ColumnStore + +from conftest import make_records + + +def test_demand_flat_load(): + recs = make_records(1000, defaults={"P_total_avg_W": 40_000.0}) + store = ColumnStore.from_records(recs) + res = demand_analysis(store, window_secs=60) + assert res["window_secs"] == 60 + assert abs(res["peak_demand_w"] - 40_000.0) < 1e-6 + assert abs(res["mean_demand_w"] - 40_000.0) < 1e-6 + assert res["n_windows"] == 1000 - 60 + 1 + + +def test_demand_ramp_peak_at_end(): + # P ramps 0..999 over 1000 s; the highest 60-s trailing window is the last. + overrides = {i: {"P_total_avg_W": float(i) * 100.0} for i in range(1000)} + recs = make_records(1000, overrides=overrides) + store = ColumnStore.from_records(recs) + res = demand_analysis(store, window_secs=60) + # Last window = mean of P over records 940..999 = mean(94000..99900 step 100). + expected = sum(float(i) * 100.0 for i in range(940, 1000)) / 60.0 + assert abs(res["peak_demand_w"] - expected) < 1e-3 + # Peak window ends at the very last record. + assert res["peak_window_end"] == store.end(999).isoformat() + + +def test_demand_series_decimation(): + recs = make_records(600, defaults={"P_total_avg_W": 10_000.0}) + store = ColumnStore.from_records(recs) + res = demand_analysis(store, window_secs=60, series_step_secs=60) + # First full window at i=59, then every 60 -> ~ (600-59)/60 + 1 samples. + assert len(res["series"]) >= 9 + for pt in res["series"]: + assert abs(pt["demand_w"] - 10_000.0) < 1e-6 + assert "t" in pt + + +def test_demand_window_larger_than_session(): + recs = make_records(30, defaults={"P_total_avg_W": 5_000.0}) + store = ColumnStore.from_records(recs) + res = demand_analysis(store, window_secs=900) + # No full 900-s window fits in 30 records -> no peak recorded. + assert res["n_windows"] == 0 + assert res["peak_window_end"] is None + + +def test_demand_empty(): + store = ColumnStore.from_records([]) + res = demand_analysis(store, window_secs=900) + assert res["n_windows"] == 0 + assert res["peak_demand_w"] == 0.0 diff --git a/python/tests/test_narrative.py b/python/tests/test_narrative.py new file mode 100644 index 0000000..a92ee2e --- /dev/null +++ b/python/tests/test_narrative.py @@ -0,0 +1,72 @@ +"""Tests for the auto-narrative / executive summary (Feature E).""" +from __future__ import annotations + +import datetime as dt + +from fluke_3540.events import Event +from fluke_3540.insights import Finding +from fluke_3540.narrative import build_narrative, narrative_markdown + +BASE = dt.datetime(2024, 1, 13, 22, 0, 0, tzinfo=dt.timezone.utc) + + +def _ev(id_, kind, start_s, end_s, sev, phases=("a", "b", "c")): + return Event(id_, kind, BASE + dt.timedelta(seconds=start_s), + BASE + dt.timedelta(seconds=end_s), sev, phases) + + +def _finding(kind, severity, headline): + return Finding(id=0, kind=kind, severity=severity, headline=headline, + detail="", related_event_ids=(), recommended_actions=()) + + +def test_narrative_clean_session(): + n = build_narrative([], [], None, None, config={"asset_name": "PUMP-1"}, + total_records=3600, duration_secs=3600) + assert "PUMP-1" in n + assert "No outages" in n + assert "Bottom line: the supply looks clean" in n + + +def test_narrative_outage_headline_with_leading_dip(): + dip = _ev(0, "dip", 90, 92, 0.72, ("c",)) + outage = _ev(1, "outage", 100, 1210, 0.0) # 1110 s outage + n = build_narrative([dip, outage], [], None, None, + config={"asset_name": "MAC03"}) + assert "outage" in n + assert "18.5 min" in n # 1110 s -> 18.5 min + assert "preceded by a phase-c dip to 72%" in n + + +def test_narrative_includes_pf_and_ct(): + stats = { + "PF_total_avg": {"mean": 0.81, "p5": 0.70, "p95": 0.95}, + "_thresholds": {"total_records": 1000}, + } + pf = _finding("pf_drift", "alert", "Power factor below 0.85 for 99.8% of non-outage time") + ct = {"reversed": True, "frac_negative": 0.52} + n = build_narrative([], [pf], stats, ct, config={"asset_name": "X"}) + assert "Power factor (total) averaged 0.81" in n + assert "Power factor below 0.85 for 99.8%" in n + assert "iFlex CTs are likely reversed" in n + assert "Bottom line: 1 alert-level finding" in n + + +def test_narrative_dip_only_no_outage(): + dip = _ev(0, "dip", 50, 53, 0.65, ("a",)) + n = build_narrative([dip], [], None, None) + assert "No outages occurred" in n + assert "65% of nominal" in n + + +def test_narrative_markdown_wrapper(): + md = narrative_markdown("Hello world.", config={"asset_name": "ABC"}) + assert md.startswith("# Executive Summary — ABC") + assert "Hello world." in md + + +def test_narrative_is_deterministic(): + dip = _ev(0, "dip", 90, 92, 0.72, ("c",)) + outage = _ev(1, "outage", 100, 220, 0.0) + args = ([dip, outage], [], None, None, {"asset_name": "Z"}) + assert build_narrative(*args) == build_narrative(*args) diff --git a/python/tests/test_pq_standards.py b/python/tests/test_pq_standards.py new file mode 100644 index 0000000..7abaf03 --- /dev/null +++ b/python/tests/test_pq_standards.py @@ -0,0 +1,68 @@ +"""Tests for IEEE 519 THD compliance + IEEE 1159 / SARFI indices (Feature F).""" +from __future__ import annotations + +import datetime as dt + +from fluke_3540.analysis import ieee519_compliance, sarfi_indices +from fluke_3540.events import Event +from fluke_3540.store import ColumnStore + +from conftest import make_records, plant_window + +BASE = dt.datetime(2024, 1, 13, 22, 0, 0, tzinfo=dt.timezone.utc) + + +def test_ieee519_compliant_low_thd(): + recs = make_records(200, defaults={ + "V_THD_pct_a_avg": 2.0, "V_THD_pct_b_avg": 2.5, "V_THD_pct_c_avg": 3.0, + }) + store = ColumnStore.from_records(recs) + res = ieee519_compliance(store) + assert res["all_voltage_compliant"] is True + assert res["voltage"]["a"]["compliant"] is True + assert res["voltage"]["a"]["exceeds_planning"] is False + assert abs(res["voltage"]["a"]["p95"] - 2.0) < 0.2 + + +def test_ieee519_noncompliant_high_thd(): + recs = make_records(200, defaults={ + "V_THD_pct_a_avg": 9.5, "V_THD_pct_b_avg": 3.0, "V_THD_pct_c_avg": 3.0, + }) + store = ColumnStore.from_records(recs) + res = ieee519_compliance(store) + assert res["voltage"]["a"]["compliant"] is False # 9.5 > 8.0 + assert res["voltage"]["b"]["compliant"] is True + assert res["all_voltage_compliant"] is False + + +def test_ieee519_exceeds_planning_but_compliant(): + recs = make_records(200, defaults={"V_THD_pct_a_avg": 6.0}) + store = ColumnStore.from_records(recs) + res = ieee519_compliance(store) + assert res["voltage"]["a"]["compliant"] is True # 6 <= 8 + assert res["voltage"]["a"]["exceeds_planning"] is True # 6 > 5 + + +def test_sarfi_counts_by_threshold(): + # Build events: dips at 85%, 65%, 45% residual; one outage at 0%. + events = [ + Event(0, "dip", BASE, BASE + dt.timedelta(seconds=2), 0.85, ("a",)), + Event(1, "dip", BASE, BASE + dt.timedelta(seconds=2), 0.65, ("a",)), + Event(2, "dip", BASE, BASE + dt.timedelta(seconds=2), 0.45, ("a",)), + Event(3, "outage", BASE, BASE + dt.timedelta(seconds=10), 0.0, ("a", "b", "c")), + Event(4, "swell", BASE, BASE + dt.timedelta(seconds=1), 1.15, ("a",)), # ignored + ] + res = sarfi_indices(events, nominal_ln_v=277.0) + # residuals: 85, 65, 45, 0 (outage 0/277=0%); swell ignored. + assert res["events_considered"] == 4 + assert res["SARFI-90"] == 4 # all four < 90 + assert res["SARFI-80"] == 3 # 65,45,0 + assert res["SARFI-70"] == 3 # 65,45,0 + assert res["SARFI-50"] == 2 # 45,0 + assert res["SARFI-10"] == 1 # 0 + + +def test_sarfi_empty(): + res = sarfi_indices([], nominal_ln_v=277.0) + assert res["events_considered"] == 0 + assert res["SARFI-90"] == 0 diff --git a/python/tests/test_rules_file.py b/python/tests/test_rules_file.py new file mode 100644 index 0000000..27b48b2 --- /dev/null +++ b/python/tests/test_rules_file.py @@ -0,0 +1,136 @@ +"""Tests for per-asset threshold config (--rules-file, Feature I).""" +from __future__ import annotations + +import json + +import pytest + +from fluke_3540.events import DEFAULT_RULES +from fluke_3540.rules_file import describe_overrides, load_rules + + +def _write(tmp_path, name, text): + p = tmp_path / name + p.write_text(text, encoding="utf-8") + return p + + +def test_flat_json_defaults(tmp_path): + p = _write(tmp_path, "r.json", json.dumps({"dip_pct_of_nominal": 0.92})) + rules = load_rules(p) + assert rules.dip_pct_of_nominal == 0.92 + # Unspecified keys keep the default. + assert rules.outage_v_threshold == DEFAULT_RULES.outage_v_threshold + + +def test_defaults_and_per_asset_json(tmp_path): + body = { + "defaults": {"dip_pct_of_nominal": 0.92}, + "assets": { + "P115RE-MAC03": {"outage_v_threshold": 60.0, "swell_pct_of_nominal": 1.08}, + }, + } + p = _write(tmp_path, "r.json", json.dumps(body)) + # Matching asset gets defaults + its overrides. + rules = load_rules(p, asset_name="P115RE-MAC03") + assert rules.dip_pct_of_nominal == 0.92 # from defaults + assert rules.outage_v_threshold == 60.0 # from asset + assert rules.swell_pct_of_nominal == 1.08 # from asset + # Non-matching asset gets only defaults. + other = load_rules(p, asset_name="SOMETHING-ELSE") + assert other.dip_pct_of_nominal == 0.92 + assert other.outage_v_threshold == DEFAULT_RULES.outage_v_threshold + + +def test_assets_default_fallback(tmp_path): + body = {"assets": {"default": {"freq_excursion_hz": 0.3}}} + p = _write(tmp_path, "r.json", json.dumps(body)) + rules = load_rules(p, asset_name="anything") + assert rules.freq_excursion_hz == 0.3 + + +def test_int_keys_coerced(tmp_path): + p = _write(tmp_path, "r.json", json.dumps({"min_duration_secs": 3, "gap_tolerance_secs": 2})) + rules = load_rules(p) + assert rules.min_duration_secs == 3 + assert isinstance(rules.min_duration_secs, int) + assert rules.gap_tolerance_secs == 2 + + +def test_unknown_key_raises(tmp_path): + p = _write(tmp_path, "r.json", json.dumps({"not_a_rule": 5})) + with pytest.raises(ValueError, match="unknown EventRules key"): + load_rules(p) + + +def test_toml_loading(tmp_path): + toml = ( + "[defaults]\n" + "dip_pct_of_nominal = 0.93\n\n" + '[assets."MAC03"]\n' + "outage_v_threshold = 55.0\n" + ) + p = _write(tmp_path, "r.toml", toml) + rules = load_rules(p, asset_name="MAC03") + assert rules.dip_pct_of_nominal == 0.93 + assert rules.outage_v_threshold == 55.0 + + +def test_describe_overrides(tmp_path): + p = _write(tmp_path, "r.json", json.dumps({"dip_pct_of_nominal": 0.92})) + lines = describe_overrides(p, None) + assert any("dip_pct_of_nominal" in ln and "0.92" in ln for ln in lines) + + +def test_rules_file_changes_detection_end_to_end(tmp_path): + """A raised dip threshold should flag a marginal dip the defaults miss.""" + import datetime as dt + import struct + from fluke_3540.cli import main + from fluke_3540.parser import ( + DATA_FLOATS, HEADER_BYTES, RECORD_MAGIC, RECORD_SIZE, + ) + from conftest import FIELD_INDEX, dt_to_filetime + + base = dt.datetime(2024, 1, 13, 22, 0, 0, tzinfo=dt.timezone.utc) + d = tmp_path / "ES.RULE" + d.mkdir() + # 277 V nominal; a 10-s window dips to 260 V (= 93.9% — above the default + # 90% dip threshold, so NOT a dip by default, but IS one at a 95% threshold). + healthy = [0.0] * DATA_FLOATS + for ph in ("a", "b", "c"): + for stt in ("min", "max", "avg"): + healthy[FIELD_INDEX[f"V_LN_{ph}_{stt}_V"]] = 277.0 + healthy[FIELD_INDEX[f"I_{ph}_{stt}_A"]] = 100.0 + healthy[FIELD_INDEX["freq_avg_Hz"]] = 60.0 + healthy[FIELD_INDEX["P_total_avg_W"]] = 50_000.0 + with (d / "trend.bin").open("wb") as fh: + for n in range(120): + vals = list(healthy) + if 50 <= n < 60: + vals[FIELD_INDEX["V_LN_a_min_V"]] = 260.0 + sft = dt_to_filetime(base + dt.timedelta(seconds=n)) + eft = dt_to_filetime(base + dt.timedelta(seconds=n + 1)) + header = (RECORD_MAGIC + + struct.pack("> 32 & 0xFFFFFFFF, sft & 0xFFFFFFFF) + + struct.pack("> 32 & 0xFFFFFFFF, eft & 0xFFFFFFFF) + + struct.pack(" 90%). + out0 = tmp_path / "out_default" + assert main([str(d), "-o", str(out0), "--parse-only", + "--nominal-ln-v", "277", "--no-stats"]) == 0 + ev0 = json.loads((out0 / "events.json").read_text()) + assert not any(e["kind"] == "dip" for e in ev0) + + # With rules-file (95% threshold): the 260 V window is now a dip. + out1 = tmp_path / "out_rules" + assert main([str(d), "-o", str(out1), "--parse-only", "--nominal-ln-v", "277", + "--no-stats", "--rules-file", str(rules)]) == 0 + ev1 = json.loads((out1 / "events.json").read_text()) + assert any(e["kind"] == "dip" for e in ev1) diff --git a/python/tests/test_stitch.py b/python/tests/test_stitch.py new file mode 100644 index 0000000..c0f6ca5 --- /dev/null +++ b/python/tests/test_stitch.py @@ -0,0 +1,145 @@ +"""Tests for multi-session stitching (Feature D).""" +from __future__ import annotations + +import datetime as dt + +from fluke_3540.stitch import stitch_stores +from fluke_3540.store import ColumnStore + +from conftest import make_records + +BASE = dt.datetime(2024, 1, 13, 22, 0, 0, tzinfo=dt.timezone.utc) + + +def _store(count, base): + return ColumnStore.from_records(make_records(count, base=base)) + + +def test_stitch_consecutive_continuous(): + # S1: 60 records [BASE .. BASE+60). S2 starts exactly where S1 ended. + s1 = _store(60, BASE) + s2 = _store(40, BASE + dt.timedelta(seconds=60)) + res = stitch_stores([("S1", s1), ("S2", s2)]) + assert res.store.n == 100 + # No gap — abuts exactly. + assert res.gaps == [] + # Timeline is monotonic + contiguous. + times = list(res.store.iter_times()) + for i in range(1, len(times)): + assert times[i] > times[i - 1] + assert (times[-1] - times[0]).total_seconds() == 99 + # Provenance covers both sources, in order. + assert [s.label for s in res.sources] == ["S1", "S2"] + assert res.sources[0].lo == 0 and res.sources[0].hi == 60 + assert res.sources[1].lo == 60 and res.sources[1].hi == 100 + + +def test_stitch_orders_by_start_time(): + # Pass out of order — stitch must sort by start time. + s1 = _store(10, BASE) + s2 = _store(10, BASE + dt.timedelta(seconds=10)) + res = stitch_stores([("later", s2), ("earlier", s1)]) + assert [s.label for s in res.sources] == ["earlier", "later"] + + +def test_stitch_records_gap(): + # 1-hour gap between S1 end and S2 start. + s1 = _store(60, BASE) + s2 = _store(60, BASE + dt.timedelta(seconds=60 + 3600)) + res = stitch_stores([("S1", s1), ("S2", s2)]) + assert res.store.n == 120 + assert len(res.gaps) == 1 + g = res.gaps[0] + assert g.after_label == "S1" + assert g.before_label == "S2" + assert abs(g.seconds - 3600.0) < 1e-6 + + +def test_stitch_small_gap_within_tolerance_not_recorded(): + s1 = _store(60, BASE) + s2 = _store(60, BASE + dt.timedelta(seconds=61)) # 1s gap, within 2s tol + res = stitch_stores([("S1", s1), ("S2", s2)]) + assert res.gaps == [] + + +def test_stitch_preserves_channel_values(): + s1 = ColumnStore.from_records( + make_records(5, base=BASE, defaults={"P_total_avg_W": 11_000.0})) + s2 = ColumnStore.from_records( + make_records(5, base=BASE + dt.timedelta(seconds=5), + defaults={"P_total_avg_W": 22_000.0})) + res = stitch_stores([("S1", s1), ("S2", s2)]) + p = res.store.col("P_total_avg_W") + assert list(p[:5]) == [11_000.0] * 5 + assert list(p[5:]) == [22_000.0] * 5 + + +def test_stitch_empty_inputs(): + res = stitch_stores([]) + assert res.store.n == 0 + assert res.sources == [] + assert res.gaps == [] + + +def test_stitch_jsonable(): + s1 = _store(10, BASE) + s2 = _store(10, BASE + dt.timedelta(seconds=10)) + res = stitch_stores([("S1", s1), ("S2", s2)]) + j = res.to_jsonable() + assert j["total_records"] == 20 + assert len(j["sources"]) == 2 + assert j["sources"][0]["label"] == "S1" + + +# --- CLI subcommand end-to-end (two synthetic session dirs) ----------------- + +def _make_session_dir(tmp_path, name, count, base): + """Write a minimal ES.NNN/ dir with a trend.bin of `count` healthy records.""" + import struct + from fluke_3540.parser import ( + DATA_FLOATS, HEADER_BYTES, RECORD_MAGIC, RECORD_SIZE, + ) + from conftest import FIELD_INDEX, dt_to_filetime + + d = tmp_path / name + d.mkdir() + healthy = [0.0] * DATA_FLOATS + for ph in ("a", "b", "c"): + for stt in ("min", "max", "avg"): + healthy[FIELD_INDEX[f"V_LN_{ph}_{stt}_V"]] = 277.0 + healthy[FIELD_INDEX[f"I_{ph}_{stt}_A"]] = 100.0 + healthy[FIELD_INDEX["freq_avg_Hz"]] = 60.0 + healthy[FIELD_INDEX["P_total_avg_W"]] = 50_000.0 + with (d / "trend.bin").open("wb") as fh: + for n in range(count): + sft = dt_to_filetime(base + dt.timedelta(seconds=n)) + eft = dt_to_filetime(base + dt.timedelta(seconds=n + 1)) + header = (RECORD_MAGIC + + struct.pack("> 32 & 0xFFFFFFFF, sft & 0xFFFFFFFF) + + struct.pack("> 32 & 0xFFFFFFFF, eft & 0xFFFFFFFF) + + struct.pack(" this.maxV) this.maxV = v; + let idx = Math.floor((v - this.lo) / this.width); + if (idx < 0) idx = 0; + else if (idx >= this.nbins) idx = this.nbins - 1; + this.bins[idx] += 1; + } + + quantile(q) { + if (this.n === 0) return NaN; + const target = q * this.n; + let cum = 0; + for (let i = 0; i < this.nbins; i++) { + cum += this.bins[i]; + if (cum >= target) return this.lo + (i + 0.5) * this.width; + } + return this.hi; + } +} + +export class RunningMoments { + constructor() { this.n = 0; this.mean = 0.0; this.m2 = 0.0; } + add(v) { + if (v !== v) return; + this.n += 1; + const delta = v - this.mean; + this.mean += delta / this.n; + this.m2 += delta * (v - this.mean); + } + get variance() { return this.n > 0 ? this.m2 / this.n : 0.0; } + get stdev() { return Math.sqrt(this.variance); } +} + +// Channels reported in whole-session stats, with histogram ranges + units. +// Matches python analysis._STATS_CHANNELS exactly. +export const STATS_CHANNELS = [ + ['V_LN_a_avg_V', 0.0, 400.0, 'V'], + ['V_LN_b_avg_V', 0.0, 400.0, 'V'], + ['V_LN_c_avg_V', 0.0, 400.0, 'V'], + ['I_a_avg_A', 0.0, 1000.0, 'A'], + ['I_b_avg_A', 0.0, 1000.0, 'A'], + ['I_c_avg_A', 0.0, 1000.0, 'A'], + ['freq_avg_Hz', 55.0, 65.0, 'Hz'], + ['P_total_avg_W', -2000000.0, 2000000.0, 'W'], + ['S_total_avg_VA', -2000000.0, 2000000.0, 'VA'], + ['Q_total_avg_VAR', -2000000.0, 2000000.0, 'VAR'], + ['PF_total_avg', -1.05, 1.05, ''], +]; + +/** + * Per-channel streaming statistics + threshold time accounting. + * @param {import('./column_store.js').ColumnStore|Array} source + * @param {object} spec + * @param {{undervoltageV?:number, overcurrentA?:number}} [opts] + * @returns {object} keyed by channel name + "_thresholds" + */ +export function wholeSessionStats(source, spec, opts = {}) { + const undervoltageV = opts.undervoltageV ?? 250.0; + const overcurrentA = opts.overcurrentA ?? 800.0; + const src = asColumnSource(source, spec); + const nrec = src.length; + const out = {}; + const moments = {}; + const sketches = {}; + const cols = {}; + for (const [name, lo, hi] of STATS_CHANNELS) { + moments[name] = new RunningMoments(); + sketches[name] = new PercentileSketch(lo, hi); + cols[name] = src.column(name); + } + const va = src.column('V_LN_a_avg_V'); + const vb = src.column('V_LN_b_avg_V'); + const vc = src.column('V_LN_c_avg_V'); + const ia = src.column('I_a_avg_A'); + const ib = src.column('I_b_avg_A'); + const ic = src.column('I_c_avg_A'); + + let secUnder = 0; + let secOver = 0; + for (let i = 0; i < nrec; i++) { + for (const [name] of STATS_CHANNELS) { + const v = cols[name][i]; + moments[name].add(v); + sketches[name].add(v); + } + const notOutage = va[i] > 50.0 && vb[i] > 50.0 && vc[i] > 50.0; + if (notOutage && (va[i] < undervoltageV || vb[i] < undervoltageV || vc[i] < undervoltageV)) { + secUnder += 1; + } + if (ia[i] > overcurrentA || ib[i] > overcurrentA || ic[i] > overcurrentA) { + secOver += 1; + } + } + + for (const [name, , , unit] of STATS_CHANNELS) { + const m = moments[name]; + const sk = sketches[name]; + if (m.n === 0) continue; + out[name] = { + unit, + count: m.n, + min: sk.minV, + p1: sk.quantile(0.01), + p5: sk.quantile(0.05), + median: sk.quantile(0.50), + mean: m.mean, + p95: sk.quantile(0.95), + p99: sk.quantile(0.99), + max: sk.maxV, + stdev: m.stdev, + }; + } + out._thresholds = { + undervoltage_v: undervoltageV, + sec_undervoltage: secUnder, + pct_undervoltage: nrec ? (secUnder / nrec) * 100 : 0.0, + overcurrent_a: overcurrentA, + sec_overcurrent: secOver, + pct_overcurrent: nrec ? (secOver / nrec) * 100 : 0.0, + total_records: nrec, + }; + return out; +} + +// --- CT-reversal auto-detection (Feature C) -------------------------------- +// +// Mirrors python analysis.detect_ct_reversal: a load wired with backwards iFlex +// CTs reads as a persistent generator (P_total < 0). Flag when real power is +// negative for a high fraction of NON-OUTAGE time. + +export function detectCtReversal(source, spec, opts = {}) { + const negFractionThreshold = opts.negFractionThreshold ?? 0.50; + const outageVThreshold = opts.outageVThreshold ?? 50.0; + const src = asColumnSource(source, spec); + const p = src.column('P_total_avg_W'); + const va = src.column('V_LN_a_avg_V'); + const vb = src.column('V_LN_b_avg_V'); + const vc = src.column('V_LN_c_avg_V'); + let nonOutage = 0; + let negative = 0; + let pSum = 0.0; + let pCount = 0; + for (let i = 0; i < src.length; i++) { + if (va[i] > outageVThreshold && vb[i] > outageVThreshold && vc[i] > outageVThreshold) { + nonOutage += 1; + const pv = p[i]; + if (Number.isFinite(pv)) { pSum += pv; pCount += 1; } + if (pv < 0) negative += 1; // NaN < 0 is false, so non-finite never counts + } + } + const frac = nonOutage ? negative / nonOutage : 0.0; + const meanP = pCount ? pSum / pCount : 0.0; + return { + reversed: frac >= negFractionThreshold, + frac_negative: frac, + non_outage_records: nonOutage, + negative_records: negative, + mean_p_w: meanP, + threshold: negFractionThreshold, + }; +} + +export function ctReversalNotice(result) { + const pct = result.frac_negative * 100.0; + return ( + 'CT REVERSAL DETECTED — ' + + `real power (P_total) is negative for ${pct.toFixed(1)}% of non-outage time ` + + `(mean P = ${(result.mean_p_w / 1000).toFixed(1)} kW). A load should draw ` + + 'positive real power: one or more iFlex CT probes are likely clipped on ' + + 'backwards. Toggle "Reverse CTs" (all phases) to correct P/Q/PF/energy.' + ); +} + +// --- ITIC / CBEMA classification ------------------------------------------- + +const ITIC_LOWER = [ + [0.001, 0.0], + [0.003, 0.0], + [0.020, 70.0], + [0.500, 70.0], + [10.0, 80.0], + [1e9, 90.0], +]; +const ITIC_UPPER = [ + [0.001, 500.0], + [0.0001, 500.0], + [0.003, 200.0], + [0.5, 120.0], + [10.0, 120.0], + [1e9, 110.0], +]; + +function interpStep(table, duration) { + for (const [dmax, val] of table) { + if (duration <= dmax) return val; + } + return table[table.length - 1][1]; +} + +export function classifyItic(residualPct, durationSecs) { + if (durationSecs < 0) durationSecs = 0.0; + const lower = interpStep(ITIC_LOWER, durationSecs); + const upper = interpStep(ITIC_UPPER, durationSecs); + if (residualPct > upper) return 'prohibited'; + if (residualPct < lower) return 'no_damage'; + return 'no_interruption'; +} + +/** + * ITIC inputs + classification for a dip/outage/swell event. + * @param {{kind:string, tStartMs:number, tEndMs:number, severity:number}} ev + * @param {number} nominalLnV + */ +export function eventItic(ev, nominalLnV) { + const duration = (ev.tEndMs - ev.tStartMs) / 1000; + let residualPct; + if (ev.kind === 'dip') residualPct = ev.severity * 100.0; + else if (ev.kind === 'outage') residualPct = nominalLnV ? (ev.severity / nominalLnV) * 100.0 : 0.0; + else if (ev.kind === 'swell') residualPct = ev.severity * 100.0; + else return {}; + return { + residual_pct: residualPct, + duration_secs: duration, + itic_class: classifyItic(residualPct, duration), + }; +} + +// --- IEEE 519 THD + IEEE 1159 / SARFI (Feature F) -------------------------- + +export const IEEE519_V_THD_LIMIT_PCT = 8.0; +export const IEEE519_V_THD_PLANNING_PCT = 5.0; + +export function ieee519Compliance(source, spec) { + const src = asColumnSource(source, spec); + const out = { + limit_v_thd_pct: IEEE519_V_THD_LIMIT_PCT, + planning_v_thd_pct: IEEE519_V_THD_PLANNING_PCT, + voltage: {}, + current: {}, + }; + let allCompliant = true; + for (const ph of ['a', 'b', 'c']) { + const vcol = src.column(`V_THD_pct_${ph}_avg`); + const sk = new PercentileSketch(0.0, 100.0, 2000); + for (let i = 0; i < vcol.length; i++) sk.add(vcol[i]); + const p95 = sk.n ? sk.quantile(0.95) : 0.0; + const compliant = p95 <= IEEE519_V_THD_LIMIT_PCT; + allCompliant = allCompliant && compliant; + out.voltage[ph] = { + p95, + limit: IEEE519_V_THD_LIMIT_PCT, + planning: IEEE519_V_THD_PLANNING_PCT, + compliant, + exceeds_planning: p95 > IEEE519_V_THD_PLANNING_PCT, + }; + const icol = src.column(`I_THD_pct_${ph}_avg`); + const ski = new PercentileSketch(0.0, 200.0, 2000); + for (let i = 0; i < icol.length; i++) ski.add(icol[i]); + out.current[ph] = { p95: ski.n ? ski.quantile(0.95) : 0.0 }; + } + out.all_voltage_compliant = allCompliant; + return out; +} + +export const SARFI_THRESHOLDS = [90, 80, 70, 50, 10]; + +export function sarfiIndices(events, nominalLnV) { + const counts = {}; + for (const x of SARFI_THRESHOLDS) counts[`SARFI-${x}`] = 0; + let considered = 0; + for (const ev of events) { + let residualPct; + if (ev.kind === 'dip') residualPct = ev.severity * 100.0; + else if (ev.kind === 'outage') residualPct = nominalLnV ? (ev.severity / nominalLnV) * 100.0 : 0.0; + else continue; + considered += 1; + for (const x of SARFI_THRESHOLDS) { + if (residualPct < x) counts[`SARFI-${x}`] += 1; + } + } + counts.events_considered = considered; + counts.nominal_ln_v = nominalLnV; + return counts; +} + +// --- Demand analysis (Feature G) ------------------------------------------- + +export function demandAnalysis(source, spec, opts = {}) { + const windowSecs = Math.max(1, Math.floor(opts.windowSecs ?? 900)); + const seriesStepSecs = Math.max(0, Math.floor(opts.seriesStepSecs ?? 0)); + const src = asColumnSource(source, spec); + const p = src.column('P_total_avg_W'); + const n = src.length; + const out = { + window_secs: windowSecs, + peak_demand_w: 0.0, + peak_demand_kw: 0.0, + peak_window_end: null, + peak_window_start: null, + mean_demand_w: 0.0, + n_windows: 0, + series: [], + }; + if (n === 0) return out; + const finite = (v) => (Number.isFinite(v) ? v : 0.0); + let running = 0.0; + let peak = -Infinity; + let peakI = -1; + let demandSum = 0.0; + let demandCount = 0; + const series = []; + const w = windowSecs; + for (let i = 0; i < n; i++) { + running += finite(p[i]); + if (i >= w) running -= finite(p[i - w]); + if (i >= w - 1) { + const demand = running / w; + demandSum += demand; + demandCount += 1; + if (demand > peak) { peak = demand; peakI = i; } + if (seriesStepSecs && ((i - (w - 1)) % seriesStepSecs === 0)) { + series.push({ t: new Date(src.endMs(i)).toISOString(), demand_w: demand }); + } + } + } + if (peakI >= 0) { + out.peak_demand_w = peak; + out.peak_demand_kw = peak / 1000.0; + out.peak_window_end = new Date(src.endMs(peakI)).toISOString(); + out.peak_window_start = new Date(src.startMs(peakI - w + 1)).toISOString(); + out.mean_demand_w = demandCount ? demandSum / demandCount : 0.0; + out.n_windows = demandCount; + } + out.series = series; + return out; +} + +// --- Time-bucket partitioning (--split-by) --------------------------------- + +export function parsePeriod(text) { + const t = String(text).trim().toLowerCase(); + if (t === 'hour' || t === 'hourly') return { kind: 'hour', seconds: 3600 }; + if (t === 'day' || t === 'daily') return { kind: 'day', seconds: 86400 }; + if (t === 'week' || t === 'weekly') return { kind: 'week', seconds: 7 * 86400 }; + const units = { s: 1, m: 60, h: 3600, d: 86400 }; + const unit = t.slice(-1); + const num = t.slice(0, -1); + if (t && units[unit] !== undefined && /^\d+$/.test(num)) { + const n = parseInt(num, 10); + if (n <= 0) throw new Error(`--split-by duration must be positive: ${text}`); + return { kind: 'duration', seconds: n * units[unit] }; + } + throw new Error( + `Unrecognized --split-by period ${text}. Use hour|day|week or a duration like 30m, 6h, 2d.` + ); +} + +// Time-of-day profile binning by minute-of-day (UTC), mirroring python. +export function parseTodWindow(text) { + const [a, b] = String(text).split('-'); + const toMin = (s) => { + const [hh, mm] = String(s).trim().split(':'); + return parseInt(hh, 10) * 60 + (mm ? parseInt(mm, 10) : 0); + }; + let start = toMin(a); + let end = toMin(b); + if (end === 0) end = 1440; + return [start, end]; +} + +/** + * Diurnal avg/min/max envelope per time-of-day bin (UTC clock). + * @returns {Array} rows with bin/n/n_days/p_avg_kW.../v.../i... + */ +export function timeOfDayProfile(source, spec, opts = {}) { + const [startMin, endMin] = opts.window ?? [0, 1440]; + const binMinutes = Math.max(1, opts.binMinutes ?? 1); + const src = asColumnSource(source, spec); + const nbins = Math.floor((1440 + binMinutes - 1) / binMinutes); + const p = src.column('P_total_avg_W'); + const va = src.column('V_LN_a_avg_V'); + const ia = src.column('I_a_avg_A'); + + const agg = new Array(nbins).fill(null); + const daysPerBin = Array.from({ length: nbins }, () => new Set()); + + for (let i = 0; i < src.length; i++) { + const d = new Date(src.startMs(i)); + const mod = d.getUTCHours() * 60 + d.getUTCMinutes(); + if (!(startMin <= mod && mod < endMin)) continue; + let b = Math.floor(mod / binMinutes); + if (b >= nbins) b = nbins - 1; + const pv = p[i] / 1000.0; + if (agg[b] === null) { + agg[b] = { + n: 0, + pSum: 0, pMin: Infinity, pMax: -Infinity, + vSum: 0, vMin: Infinity, vMax: -Infinity, + iSum: 0, iMin: Infinity, iMax: -Infinity, + }; + } + const a = agg[b]; + a.n += 1; + a.pSum += pv; a.pMin = Math.min(a.pMin, pv); a.pMax = Math.max(a.pMax, pv); + a.vSum += va[i]; a.vMin = Math.min(a.vMin, va[i]); a.vMax = Math.max(a.vMax, va[i]); + a.iSum += ia[i]; a.iMin = Math.min(a.iMin, ia[i]); a.iMax = Math.max(a.iMax, ia[i]); + // Day key = UTC date string. + daysPerBin[b].add(d.toISOString().slice(0, 10)); + } + + const rows = []; + for (let b = 0; b < nbins; b++) { + const a = agg[b]; + if (a === null || a.n === 0) continue; + const binStartMin = b * binMinutes; + const hh = String(Math.floor(binStartMin / 60)).padStart(2, '0'); + const mm = String(binStartMin % 60).padStart(2, '0'); + rows.push({ + bin: `${hh}:${mm}`, + n: a.n, + n_days: daysPerBin[b].size, + p_avg_kW: a.pSum / a.n, + p_min_kW: a.pMin, + p_max_kW: a.pMax, + v_avg_V: a.vSum / a.n, + v_min_V: a.vMin, + v_max_V: a.vMax, + i_avg_A: a.iSum / a.n, + i_min_A: a.iMin, + i_max_A: a.iMax, + }); + } + return rows; +} + +// --- Event markers / correlation ------------------------------------------- + +/** + * For each marker, find the nearest detected event + signed offset (s). + * @param {Array<{timeMs:number, label:string}>} markers + * @param {Array<{id:number, kind:string, tStartMs:number}>} events + */ +export function correlateMarkers(markers, events) { + const out = []; + for (const m of markers) { + let nearest = null; + let best = null; + for (const ev of events) { + const off = (m.timeMs - ev.tStartMs) / 1000; + if (best === null || Math.abs(off) < Math.abs(best)) { + best = off; + nearest = ev; + } + } + const entry = { + marker_time: new Date(m.timeMs).toISOString(), + label: m.label, + nearest_event: null, + }; + if (nearest !== null) { + entry.nearest_event = { + id: nearest.id, + kind: nearest.kind, + t_start: new Date(nearest.tStartMs).toISOString(), + offset_secs: best, + }; + } + out.push(entry); + } + return out; +} diff --git a/web/app.js b/web/app.js index ff8a122..718ca38 100644 --- a/web/app.js +++ b/web/app.js @@ -1,7 +1,7 @@ // Main orchestration: drop-zone handling, spec/file loading, worker dispatch, // summary rendering, event detection, chart UI, exports. Pure ESM, no framework. -import { detectEvents } from './events.js'; +import { detectEvents, rulesFromObject } from './events.js'; import { pickSnapshots } from './snapshots.js'; import { FULL_QUANTITIES, ZOOM_QUANTITIES, renderChart } from './plots.js'; import { buildXlsx, downloadBlob } from './xlsx_export.js'; @@ -12,6 +12,10 @@ import { downloadCompareHtmlReport, downloadHtmlReport } from './html_report.js' import { downloadPdfReport } from './pdf_export.js'; import { clearCache, getCached, hashBuffer, putCached } from './cache.js'; import { MultiSession } from './multi_session.js'; +import { ColumnStore } from './column_store.js'; +import { wholeSessionStats, timeOfDayProfile, detectCtReversal, ctReversalNotice, ieee519Compliance, sarfiIndices, demandAnalysis } from './analysis.js'; +import { buildNarrative } from './narrative.js'; +import { formatLocalUtc, tzLabel } from './tzutil.js'; import { computeCost, loadTariff, normalizeTariff, parsePeakHoursString, peakHoursToString, saveTariff, @@ -96,7 +100,9 @@ const ms = new MultiSession(); let cachedSpec = null; let currentArrayBuffer = null; let currentConfig = null; // parsed ES.NNN-config.json companion (or null) -let currentRecords = null; // full parsed Records array (kept in memory) +let currentStore = null; // ColumnStore (memory-bounded; analysis + charts) +let currentRecords = null; // record array — only for small/CSV paths or read-through +let currentFile = null; // the dropped File/Blob, kept for read-through export let currentRecordCount = 0; let currentTimeRangeMs = null; let currentEvents = []; // detected events for currentRecords @@ -220,9 +226,41 @@ async function parseFile(file) { hideError(); els.summarySec.hidden = true; els.progressSec.hidden = false; - setProgress(0, file.size, 'reading file'); - currentArrayBuffer = await file.arrayBuffer(); - await parseBuffer(); + currentFile = file; // kept for read-through CSV/zoom export + currentArrayBuffer = null; // streaming path never holds the whole buffer + await parseStreaming(file); +} + +// Streaming columnar path (Feature A): hand the File to the worker, which reads +// it in 8 MB record-aligned chunks and transfers back typed-array columns. The +// full 438 MB ArrayBuffer is never resident on either thread. +async function parseStreaming(file) { + hideError(); + els.summarySec.hidden = true; + els.progressSec.hidden = false; + const spec = await getSpec(); + const reverseCts = selectedReversePhases(); + + setProgress(0, 100, 'parsing (streaming)'); + if (currentWorker) currentWorker.terminate(); + currentWorker = new Worker(new URL('./parser_worker.js', import.meta.url), + { type: 'module' }); + currentWorker.onmessage = (event) => { + const msg = event.data; + if (msg.type === 'progress') { + setProgress(msg.done, msg.total, + `parsing record ${msg.done.toLocaleString()} / ${msg.total.toLocaleString()}`); + } else if (msg.type === 'done-columnar') { + const store = ColumnStore.fromTransfer(msg); + onParseDoneColumnar(store).catch(showError); + } else if (msg.type === 'error') { + showError(new Error(msg.message)); + } + }; + currentWorker.onerror = (event) => { + showError(new Error(`Worker error: ${event.message ?? 'unknown'}`)); + }; + currentWorker.postMessage({ type: 'parse-stream', spec, blob: file, reverseCts }); } async function parseBuffer() { @@ -297,6 +335,7 @@ function selectedReversePhases() { async function onParseDone(msg) { currentRecords = msg.records; + currentStore = null; // legacy/CSV/small path keeps record objects currentRecordCount = msg.recordCount; currentFileHash = msg.fileHash ?? currentFileHash; if (msg.records.length > 0) { @@ -316,7 +355,7 @@ async function onParseDone(msg) { await new Promise((r) => setTimeout(r, 0)); // let progress repaint try { const spec = await getSpec(); - currentEvents = detectEvents(currentRecords, spec); + currentEvents = detectEvents(currentRecords, spec, resolveRules() ? { rules: resolveRules() } : {}); currentSnapshots = pickSnapshots(currentRecords, currentEvents, spec, { n: 3 }); currentFindings = analyzeInsights(currentRecords, currentEvents, spec, currentSnapshots, currentConfig, @@ -336,6 +375,9 @@ async function onParseDone(msg) { renderEventsTable(); renderSnapshotsList(); renderQuantityGrid(); + renderStatsPanel(spec); + renderNarrative(spec); + checkCtReversal(spec); renderSessionsBar(); els.insightsSec.hidden = currentFindings.length === 0; els.sessionsSec.hidden = false; @@ -360,6 +402,349 @@ async function onParseDone(msg) { } } +// Columnar parse-done: analysis + charts run on the ColumnStore (memory-bounded); +// currentRecords stays null so the 7-day file never re-materialises 590 K objects. +async function onParseDoneColumnar(store) { + currentStore = store; + currentRecords = null; + currentRecordCount = store.n; + if (store.n > 0) { + currentTimeRangeMs = [store.firstStartMs, store.lastEndMs]; + } else { + currentTimeRangeMs = null; + } + els.progressSec.hidden = true; + renderSummary(); + els.summarySec.hidden = false; + + setProgress(0, 100, 'detecting events'); + els.progressSec.hidden = false; + await new Promise((r) => setTimeout(r, 0)); + try { + const spec = await getSpec(); + currentEvents = detectEvents(currentStore, spec, resolveRules() ? { rules: resolveRules() } : {}); + currentSnapshots = pickSnapshots(currentStore, currentEvents, spec, { n: 3 }); + currentFindings = analyzeInsights(currentStore, currentEvents, spec, + currentSnapshots, currentConfig, + { breakerRatingA: loadBreakerRating() }); + ms.add({ + records: null, store: currentStore, + events: currentEvents, snapshots: currentSnapshots, + findings: currentFindings, config: currentConfig, + fileHash: currentFileHash, file: currentFile, + }); + renderInsights(); + renderEventsTable(); + renderSnapshotsList(); + renderQuantityGrid(); + renderStatsPanel(spec); + renderNarrative(spec); + checkCtReversal(spec); + renderSessionsBar(); + els.insightsSec.hidden = currentFindings.length === 0; + els.sessionsSec.hidden = false; + els.eventsSec.hidden = false; + els.snapshotsSec.hidden = currentSnapshots.length === 0; + els.controlsSec.hidden = false; + els.exportSec.hidden = false; + els.rangeSec.hidden = false; + els.tariffSec.hidden = false; + setupRangeSelector(spec); + loadTariffIntoForm(); + renderTariffResult(); + tabState.hasSession = true; + updateTabUnlocks(); + if (tabState.current === 'import') activateTab('explore'); + } catch (e) { + showError(e); + return; + } finally { + els.progressSec.hidden = true; + } +} + +// The data source the analysis / chart / range engines should read: the store +// when present (streaming path), else the records array (small / CSV path). +function dataSource() { + return currentStore || currentRecords; +} + +// Per-asset EventRules from a loaded rules-file (Feature I), or undefined for +// the built-in defaults. +function resolveRules() { + if (!currentRulesRaw) return undefined; + try { + return rulesFromObject(currentRulesRaw, currentConfig?.asset_name ?? null); + } catch (e) { + showError(e); + return undefined; + } +} + +// Re-run detection with the current rules and refresh the dependent UI. Used +// after loading/clearing a rules-file. +async function redetect() { + const src = dataSource(); + if (!src) return; + const spec = await getSpec(); + const rules = resolveRules(); + currentEvents = detectEvents(src, spec, rules ? { rules } : {}); + currentSnapshots = pickSnapshots(src, currentEvents, spec, { n: 3 }); + currentFindings = analyzeInsights(src, currentEvents, spec, currentSnapshots, + currentConfig, { breakerRatingA: loadBreakerRating() }); + renderInsights(); + renderEventsTable(); + renderSnapshotsList(); + renderStatsPanel(spec); + renderNarrative(spec); + els.insightsSec.hidden = currentFindings.length === 0; +} + +// Per-session column accessor for compare-overlay charts: reads from a +// session's ColumnStore when present, else its records array. Only the few +// channels FULL_QUANTITIES references (all retained) are needed here. +function sessionAccessor(session, spec, name) { + if (session.store) { + const col = session.store.cols[name]; + const n = session.store.n; + return { + n, + startMs: (i) => session.store.startMs[i], + value: (i) => (col ? col[i] : 0), + base: n ? session.store.startMs[0] : 0, + }; + } + const recs = session.records || []; + const fi = new Map(spec.fields.map((f) => [f.name, f.index])); + const idx = fi.get(name); + return { + n: recs.length, + startMs: (i) => recs[i].startMs, + value: (i) => recs[i].floats[idx], + base: recs.length ? recs[0].startMs : 0, + }; +} + +// Materialise a records array for an export that genuinely needs every field +// (CSV / XLSX / bundle). For the store path this is a transient allocation that +// is dropped when the export finishes — it is NOT kept resident. +function recordsForExport(spec) { + if (currentRecords) return currentRecords; + if (currentStore) return currentStore.toRecords(spec); + return []; +} + +// Statistics panel (Feature B) — whole-session stats table + time-of-day chart, +// computed straight off the resident ColumnStore so the CLI and web agree. +let currentStats = null; // last computed whole_session_stats (for exports) +let currentTodRows = null; // last computed time-of-day profile (for exports) +let currentNarrative = null; // executive-summary narrative (Feature E) +let currentPq = null; // IEEE 519 + SARFI power-quality (Feature F) +let currentDemand = null; // rolling peak-demand analysis (Feature G) +let currentTz = null; // report timezone (IANA) or null = UTC (Feature H) +let currentRulesRaw = null; // parsed --rules-file object (Feature I) + +const TZ_STORAGE_KEY = 'fluke3540.tz'; + +// Render the tz-aware time range under the summary (local + UTC, or UTC only). +function renderTzRange() { + const span = document.getElementById('tz-range'); + if (!span || !currentTimeRangeMs) { if (span) span.textContent = ''; return; } + let valid = currentTz; + if (valid) { + // Validate the zone; fall back to UTC on a bad name. + try { formatLocalUtc(currentTimeRangeMs[0], valid); } catch (_) { valid = null; } + } + const [t0, t1] = currentTimeRangeMs; + span.textContent = + ` ${tzLabel(valid)} — start ${formatLocalUtc(t0, valid)}; end ${formatLocalUtc(t1, valid)}`; +} + +// Median non-outage L-N voltage for SARFI residual %, from the stats sketch +// (falls back to 277 V if voltage stats are unavailable). +function inferNominalForPq(src) { + if (currentStats && currentStats.V_LN_a_avg_V) { + const m = currentStats.V_LN_a_avg_V.median; + if (Number.isFinite(m) && m > 50) return m; + } + return 277.0; +} + +// CT-reversal banner (Feature C): warn when real power is sustained-negative on +// a load and reverse-CTs isn't already applied. The Apply button ticks all +// phase boxes and re-parses (which negates P/Q/PF/energy). +function checkCtReversal(spec) { + const banner = document.getElementById('ct-reversal-banner'); + const msg = document.getElementById('ct-reversal-msg'); + if (!banner) return; + const src = dataSource(); + const reverseOn = selectedReversePhases() !== false; + if (!src || reverseOn) { banner.hidden = true; return; } + let res; + try { res = detectCtReversal(src, spec); } catch (_) { banner.hidden = true; return; } + if (!res.reversed) { banner.hidden = true; return; } + if (msg) msg.textContent = ' ' + ctReversalNotice(res); + banner.hidden = false; +} + +// Executive summary (Feature E) — built from events + insights + stats + ct. +function renderNarrative(spec) { + const sec = document.getElementById('narrative-section'); + const el = document.getElementById('narrative-text'); + if (!sec || !el) return; + const src = dataSource(); + if (!src) { sec.hidden = true; return; } + let ct = null; + try { ct = detectCtReversal(src, spec); } catch (_) { /* ignore */ } + const durationSecs = currentTimeRangeMs + ? (currentTimeRangeMs[1] - currentTimeRangeMs[0]) / 1000 : null; + currentNarrative = buildNarrative(currentEvents, currentFindings, currentStats, ct, { + config: currentConfig, totalRecords: currentRecordCount, durationSecs, + }); + el.textContent = currentNarrative; + sec.hidden = false; +} + +function renderStatsPanel(spec) { + const sec = document.getElementById('stats-section'); + const wrap = document.getElementById('stats-table-wrap'); + const status = document.getElementById('stats-status'); + if (!sec || !wrap) return; + const src = dataSource(); + if (!src) { sec.hidden = true; return; } + try { + currentStats = wholeSessionStats(src, spec); + currentTodRows = timeOfDayProfile(src, spec, { window: [0, 1440], binMinutes: 1 }); + } catch (e) { + console.warn('stats failed:', e); + sec.hidden = true; + return; + } + sec.hidden = false; + + // Stats table. + const table = document.createElement('table'); + table.className = 'stats-table'; + const thead = document.createElement('thead'); + const hr = document.createElement('tr'); + for (const h of ['Channel', 'Unit', 'Min', 'p1', 'p5', 'Median', 'Mean', 'p95', 'p99', 'Max', 'Stdev']) { + const th = document.createElement('th'); + th.textContent = h; + hr.appendChild(th); + } + thead.appendChild(hr); + table.appendChild(thead); + const tbody = document.createElement('tbody'); + const fmt = (v) => (Number.isFinite(v) ? (Math.abs(v) >= 1000 ? v.toFixed(0) : v.toFixed(2)) : '—'); + for (const [name, d] of Object.entries(currentStats)) { + if (name.startsWith('_')) continue; + const tr = document.createElement('tr'); + const cells = [name, d.unit, fmt(d.min), fmt(d.p1), fmt(d.p5), fmt(d.median), + fmt(d.mean), fmt(d.p95), fmt(d.p99), fmt(d.max), fmt(d.stdev)]; + for (const c of cells) { + const td = document.createElement('td'); + td.textContent = c; + tr.appendChild(td); + } + tbody.appendChild(tr); + } + table.appendChild(tbody); + + const th = currentStats._thresholds || {}; + const note = document.createElement('p'); + note.className = 'stats-thresholds'; + const us = document.createElement('small'); + us.textContent = + `Under-voltage (<${th.undervoltage_v} V, any phase, non-outage): ` + + `${th.sec_undervoltage} s (${(th.pct_undervoltage ?? 0).toFixed(2)}%). ` + + `Over-current (>${th.overcurrent_a} A, any phase): ` + + `${th.sec_overcurrent} s (${(th.pct_overcurrent ?? 0).toFixed(2)}%).`; + note.appendChild(us); + + // IEEE 519 + SARFI power-quality summary (Feature F). + currentPq = null; + const pqP = document.createElement('p'); + pqP.className = 'stats-pq'; + try { + const ieee = ieee519Compliance(src, spec); + const sarfi = sarfiIndices(currentEvents, inferNominalForPq(src)); + currentPq = { ieee519: ieee, sarfi }; + const vv = ieee.voltage; + const pqs = document.createElement('small'); + pqs.textContent = + `IEEE 519 V_THD p95 (limit ${ieee.limit_v_thd_pct.toFixed(0)}%): ` + + `a=${vv.a.p95.toFixed(1)}% b=${vv.b.p95.toFixed(1)}% c=${vv.c.p95.toFixed(1)}% — ` + + `${ieee.all_voltage_compliant ? 'COMPLIANT' : 'NON-COMPLIANT'}. ` + + `SARFI-90=${sarfi['SARFI-90']}, SARFI-70=${sarfi['SARFI-70']}, ` + + `SARFI-10=${sarfi['SARFI-10']} (${sarfi.events_considered} voltage events).`; + pqP.appendChild(pqs); + } catch (_) { /* THD columns may be absent on some inputs */ } + + // Rolling peak-demand (Feature G), 15-min window. + currentDemand = null; + const demP = document.createElement('p'); + demP.className = 'stats-demand'; + try { + currentDemand = demandAnalysis(src, spec, { windowSecs: 900 }); + const ds = document.createElement('small'); + if (currentDemand.n_windows) { + ds.textContent = + `Peak 15-min demand: ${currentDemand.peak_demand_kw.toFixed(1)} kW ` + + `(window ending ${new Date(currentDemand.peak_window_end).toISOString().slice(0, 19)}Z); ` + + `mean demand ${(currentDemand.mean_demand_w / 1000).toFixed(1)} kW.`; + } else { + ds.textContent = 'Peak demand: session shorter than the 15-min window.'; + } + demP.appendChild(ds); + } catch (_) { /* ignore */ } + + wrap.replaceChildren(table, note, pqP, demP); + if (status) status.firstChild + ? (status.firstChild.textContent = `${Object.keys(currentStats).length - 1} channels over ${(th.total_records || 0).toLocaleString()} records.`) + : (status.textContent = ''); + + renderTodChart(); +} + +function renderTodChart() { + const head = document.getElementById('tod-heading'); + const div = document.getElementById('tod-chart'); + const uPlot = window.uPlot; + if (!div) return; + div.replaceChildren(); + if (!uPlot || !currentTodRows || currentTodRows.length === 0) { + if (head) head.hidden = true; + return; + } + if (head) head.hidden = false; + // x = minute-of-day index; series = P avg (kW), V avg (V), I avg (A). + const xs = currentTodRows.map((_, i) => i); + const pAvg = currentTodRows.map((r) => r.p_avg_kW); + const vAvg = currentTodRows.map((r) => r.v_avg_V); + const iAvg = currentTodRows.map((r) => r.i_avg_A); + const plot = new uPlot({ + width: div.clientWidth || 900, + height: 240, + series: [ + { label: 'bin' }, + { label: 'P avg (kW)', stroke: '#cc0000', width: 1.4 }, + { label: 'V avg (V)', stroke: '#0066cc', width: 1, scale: 'v' }, + { label: 'I avg (A)', stroke: '#009933', width: 1, scale: 'i' }, + ], + scales: { x: { time: false } }, + axes: [ + { stroke: '#666', label: 'time-of-day bin' }, + { stroke: '#666', label: 'P (kW)' }, + { stroke: '#666', scale: 'v', side: 1, label: 'V' }, + ], + legend: { live: true }, + }, [xs, pAvg, vAvg, iAvg], div); + const resizeObs = new ResizeObserver(() => { + plot.setSize({ width: div.clientWidth, height: 240 }); + }); + resizeObs.observe(div); +} + // --- Summary rendering ------------------------------------------------------ function formatDuration(ms) { @@ -406,6 +791,7 @@ function renderSummary() { els.summaryGrid.replaceWith(dl); dl.id = 'summary-grid'; els.summaryGrid = dl; + renderTzRange(); } // --- UI plumbing ------------------------------------------------------------ @@ -434,6 +820,8 @@ function resetUi() { els.progressSec.hidden = true; els.sessionsSec.hidden = true; els.insightsSec.hidden = true; + { const s = document.getElementById('stats-section'); if (s) s.hidden = true; } + { const s = document.getElementById('narrative-section'); if (s) s.hidden = true; } els.eventsSec.hidden = true; els.snapshotsSec.hidden = true; els.controlsSec.hidden = true; @@ -493,11 +881,11 @@ function getTariffFromForm() { async function renderTariffResult() { els.tariffResult.replaceChildren(); - if (!currentRecords) return; + if (!dataSource()) return; const t = getTariffFromForm(); if (t.peakRate === 0 && t.offpeakRate === 0) return; const spec = await getSpec(); - const cost = computeCost(currentRecords, spec, t); + const cost = computeCost(dataSource(), spec, t); const fmt = (n) => `${t.currency} ${n.toFixed(2)}`; const fmtKwh = (n) => `${n.toFixed(2)} kWh`; const dl = document.createElement('dl'); @@ -573,16 +961,28 @@ function switchToSession(label) { if (!ms.setActive(label)) return; const s = ms.getActive(); if (!s) return; - currentRecords = s.records; - currentRecordCount = s.records.length; + currentRecords = s.records || null; + currentStore = s.store || null; + currentFile = s.file || null; currentEvents = s.events; currentSnapshots = s.snapshots; currentFindings = s.findings; currentConfig = s.config; currentArrayBuffer = null; // can't re-parse a switched-to session - currentTimeRangeMs = s.records.length - ? [s.records[0].startMs, s.records[s.records.length - 1].endMs] - : null; + const ds = currentStore || currentRecords; + if (currentStore) { + currentRecordCount = currentStore.n; + currentTimeRangeMs = currentStore.n + ? [currentStore.firstStartMs, currentStore.lastEndMs] : null; + } else if (currentRecords) { + currentRecordCount = currentRecords.length; + currentTimeRangeMs = currentRecords.length + ? [currentRecords[0].startMs, currentRecords[currentRecords.length - 1].endMs] + : null; + } else { + currentRecordCount = 0; + currentTimeRangeMs = null; + } renderSummary(); renderInsights(); renderEventsTable(); @@ -593,7 +993,7 @@ function switchToSession(label) { function setupRangeSelector(spec) { if (rangeSelector) rangeSelector.destroy(); rangeSelector = renderRangeSelector( - els.rangeContainer, currentRecords, spec, (range) => { + els.rangeContainer, dataSource(), spec, (range) => { currentRange = range; const hash = rangeToHash(range); if (hash) history.replaceState(null, '', hash); @@ -675,9 +1075,9 @@ function scrollToEvent(eventId) { } async function exportXlsx() { - if (!currentRecords) return; + if (!dataSource()) return; const spec = await getSpec(); - const scoped = scopeRecordsToRange(currentRecords, currentRange); + const scoped = scopeRecordsToRange(recordsForExport(spec), currentRange); const blob = buildXlsx({ records: scoped, spec, config: currentConfig }); const name = (currentConfig?.asset_name ?? 'fluke_session').replace(/[^a-zA-Z0-9._-]+/g, '_'); const suffix = currentRange ? '_range' : ''; @@ -685,9 +1085,9 @@ async function exportXlsx() { } async function exportBundle() { - if (!currentRecords) return; + if (!dataSource()) return; const spec = await getSpec(); - const scoped = scopeRecordsToRange(currentRecords, currentRange); + const scoped = scopeRecordsToRange(recordsForExport(spec), currentRange); const xlsxBlob = buildXlsx({ records: scoped, spec, config: currentConfig }); await downloadBundleZip({ records: scoped, spec, xlsxBlob, @@ -696,9 +1096,9 @@ async function exportBundle() { } async function exportPdf() { - if (!currentRecords) return; + if (!dataSource()) return; const spec = await getSpec(); - const scoped = scopeRecordsToRange(currentRecords, currentRange); + const scoped = scopeRecordsToRange(recordsForExport(spec), currentRange); const scopedEvents = currentRange ? currentEvents.filter((e) => !(e.tEndMs < currentRange.startMs || e.tStartMs > currentRange.endMs)) @@ -715,7 +1115,7 @@ async function exportPdf() { } async function exportHtmlReport() { - if (!currentRecords) return; + if (!dataSource()) return; const spec = await getSpec(); if (ms.compareMode && ms.canCompare()) { // Compare-mode HTML uses the per-session summary + cross-session findings. @@ -727,7 +1127,7 @@ async function exportHtmlReport() { }); return; } - const scoped = scopeRecordsToRange(currentRecords, currentRange); + const scoped = scopeRecordsToRange(recordsForExport(spec), currentRange); // When scoping, also scope events/findings to overlap the range. const scopedEvents = currentRange ? currentEvents.filter((e) => @@ -742,6 +1142,11 @@ async function exportHtmlReport() { title, config: currentConfig, records: scoped, spec, events: scopedEvents, snapshots: currentSnapshots, findings: scopedFindings, + wholeStats: currentRange ? null : currentStats, + narrative: currentRange ? null : currentNarrative, + pq: currentRange ? null : currentPq, + demand: currentRange ? null : currentDemand, + tz: currentTz, }); } @@ -998,7 +1403,7 @@ function selectedSnapshotIds() { } async function renderAll() { - if (!currentRecords) return; + if (!dataSource()) return; const spec = await getSpec(); const quantities = selectedQuantities(); if (quantities.length === 0) { @@ -1027,7 +1432,7 @@ async function renderAll() { : {}), }; for (const q of quantities) { - renderChart(els.fullCharts, currentRecords, spec, q, FULL_QUANTITIES, fullOpts); + renderChart(els.fullCharts, dataSource(), spec, q, FULL_QUANTITIES, fullOpts); } } @@ -1040,7 +1445,7 @@ async function renderAll() { `Event #${ev.id}`, `${ev.kind} @ ${formatDate(ev.tStartMs)}`, )); for (const q of zoomQuantities) { - renderChart(els.eventCharts, currentRecords, spec, q, ZOOM_QUANTITIES, { + renderChart(els.eventCharts, dataSource(), spec, q, ZOOM_QUANTITIES, { startMs: ev.tStartMs - preMs, endMs: ev.tEndMs + postMs, }); @@ -1056,7 +1461,7 @@ async function renderAll() { `Snapshot #${s.id}`, `@ ${formatDate(s.tStartMs)}`, )); for (const q of zoomQuantities) { - renderChart(els.snapshotCharts, currentRecords, spec, q, ZOOM_QUANTITIES, { + renderChart(els.snapshotCharts, dataSource(), spec, q, ZOOM_QUANTITIES, { startMs: s.tStartMs, endMs: s.tEndMs, }); @@ -1077,25 +1482,24 @@ function renderOverlayChart(parentEl, spec, quantityKey, _allQuantities) { // chart readable; FULL_QUANTITIES often has 3 phases — we'd otherwise // overlay 9+ lines for 3 sessions × 3 phases). const firstCol = def.series[0]; - const fi = new Map(spec.fields.map((f) => [f.name, f.index])); - const idx = fi.get(firstCol.name); + const accessors = all.map((s) => sessionAccessor(s, spec, firstCol.name)); - const xs = []; // pooled relative-seconds axis const ySeries = all.map(() => []); // Build a unified sorted x axis from the union of all sessions' rel-seconds. const xSet = new Set(); - const relValues = all.map((s) => - s.records.map((r) => Math.round((r.startMs - s.records[0]?.startMs ?? 0) / 1000)) - ); - for (const arr of relValues) for (const x of arr) xSet.add(x); + for (const acc of accessors) { + for (let i = 0; i < acc.n; i++) { + xSet.add(Math.round((acc.startMs(i) - acc.base) / 1000)); + } + } const xsAll = [...xSet].sort((a, b) => a - b); // Per-session lookup: relSec → value for (let si = 0; si < all.length; si++) { - const s = all[si]; + const acc = accessors[si]; const map = new Map(); - for (let i = 0; i < s.records.length; i++) { - const rel = Math.round((s.records[i].startMs - (s.records[0]?.startMs ?? 0)) / 1000); - map.set(rel, s.records[i].floats[idx] * firstCol.scale); + for (let i = 0; i < acc.n; i++) { + const rel = Math.round((acc.startMs(i) - acc.base) / 1000); + map.set(rel, acc.value(i) * firstCol.scale); } for (const x of xsAll) ySeries[si].push(map.has(x) ? map.get(x) : null); } @@ -1213,6 +1617,44 @@ for (const cb of [els.reverseA, els.reverseB, els.reverseC]) { else renderSummary(); }); } +document.getElementById('ct-reversal-apply')?.addEventListener('click', () => { + if (els.reverseA) els.reverseA.checked = true; + if (els.reverseB) els.reverseB.checked = true; + if (els.reverseC) els.reverseC.checked = true; + document.getElementById('ct-reversal-banner').hidden = true; + if (currentFile) parseStreaming(currentFile).catch(showError); + else if (currentArrayBuffer) parseBuffer(); +}); +// Per-asset rules-file (Feature I): load JSON, apply, re-detect. +document.getElementById('rules-file-input')?.addEventListener('change', async (e) => { + const file = e.target.files?.[0]; + const status = document.getElementById('rules-status'); + const clearBtn = document.getElementById('rules-clear'); + if (!file) return; + try { + const text = await file.text(); + currentRulesRaw = JSON.parse(text); + rulesFromObject(currentRulesRaw, currentConfig?.asset_name ?? null); // validate + if (status) status.textContent = ` Loaded ${file.name}; re-detecting…`; + if (clearBtn) clearBtn.hidden = false; + await redetect(); + if (status) status.textContent = ` Applied ${file.name} (asset ${currentConfig?.asset_name ?? 'n/a'}).`; + } catch (err) { + currentRulesRaw = null; + if (status) status.textContent = ` Rules error: ${err.message}`; + } +}); +document.getElementById('rules-clear')?.addEventListener('click', async () => { + currentRulesRaw = null; + const status = document.getElementById('rules-status'); + const input = document.getElementById('rules-file-input'); + const clearBtn = document.getElementById('rules-clear'); + if (input) input.value = ''; + if (clearBtn) clearBtn.hidden = true; + if (status) status.textContent = ' Rules cleared; using defaults.'; + await redetect(); +}); + els.resetBtn.addEventListener('click', resetUi); els.eventsExportNotes?.addEventListener('click', () => { const json = exportNotesJson(currentFileHash, currentEvents); @@ -1230,9 +1672,9 @@ els.tariffApplyBtn.addEventListener('click', async () => { const amps = Number(els.breakerRating.value) || 0; saveBreakerRating(amps); // Re-run insights with the new breaker context. - if (currentRecords) { + if (dataSource()) { const spec = await getSpec(); - currentFindings = analyzeInsights(currentRecords, currentEvents, spec, + currentFindings = analyzeInsights(dataSource(), currentEvents, spec, currentSnapshots, currentConfig, { breakerRatingA: amps }); renderInsights(); @@ -1305,6 +1747,21 @@ document.querySelectorAll('input[name=theme]').forEach((r) => { }); loadTheme(); +// --- Report timezone (Feature H) ------------------------------------------- +(function initTz() { + const input = document.getElementById('tz-input'); + if (!input) return; + const saved = localStorage.getItem(TZ_STORAGE_KEY) || ''; + input.value = saved; + currentTz = saved || null; + input.addEventListener('change', () => { + currentTz = input.value.trim() || null; + if (currentTz) localStorage.setItem(TZ_STORAGE_KEY, currentTz); + else localStorage.removeItem(TZ_STORAGE_KEY); + renderTzRange(); + }); +})(); + // --- Keyboard shortcuts ---------------------------------------------------- document.addEventListener('keydown', (e) => { @@ -1315,7 +1772,7 @@ document.addEventListener('keydown', (e) => { switch (e.key) { case 'r': case 'R': - if (currentRecords) { e.preventDefault(); renderAll().catch(showError); } + if (dataSource()) { e.preventDefault(); renderAll().catch(showError); } break; case 'z': case 'Z': { diff --git a/web/column_source.js b/web/column_source.js new file mode 100644 index 0000000..ee203c3 --- /dev/null +++ b/web/column_source.js @@ -0,0 +1,39 @@ +// Small adapter so the analysis engines (events / snapshots / insights / stats) +// can read columns from EITHER an array of record objects (legacy / tests) OR a +// ColumnStore (the memory-bounded streaming path). Both expose the same handful +// of accessors the engines need: column(name), startMs(i), endMs(i), length. + +import { ColumnStore } from './column_store.js'; + +/** + * Wrap a records-array or a ColumnStore into a uniform column source. + * @param {Array|ColumnStore} source + * @param {object} spec parsed field_map.json + * @returns {{length:number, startMs:(i:number)=>number, endMs:(i:number)=>number, + * column:(name:string)=>(Float32Array|number[]), isStore:boolean}} + */ +export function asColumnSource(source, spec) { + if (source instanceof ColumnStore) { + return { + length: source.n, + isStore: true, + startMs: (i) => source.startMs[i], + endMs: (i) => source.endMs[i], + column: (name) => source.col(name), + }; + } + // records array + const records = source; + const fi = new Map(spec.fields.map((f) => [f.name, f.index])); + return { + length: records.length, + isStore: false, + startMs: (i) => records[i].startMs, + endMs: (i) => records[i].endMs, + column: (name) => { + const idx = fi.get(name); + if (idx === undefined) throw new Error(`spec is missing field ${name}`); + return Float32Array.from(records, (r) => r.floats[idx]); + }, + }; +} diff --git a/web/column_store.js b/web/column_store.js new file mode 100644 index 0000000..1997349 --- /dev/null +++ b/web/column_store.js @@ -0,0 +1,140 @@ +// Columnar session store — JS port of python/src/fluke_3540/store.py. +// +// The legacy web path materialised every record as a {index, startMs, endMs, +// floats:Float32Array(180)} object. For a week-long capture (~590 K records) +// that is ~1.6 GB of heap. This ColumnStore keeps only the ~24 analysis +// channels the event / snapshot / insight / stats engines actually read, each +// as a packed Float32Array, plus a Float64Array of start/end millisecond +// timestamps. That is ~55 MB for a full week instead of >1 GB, and the typed +// arrays are Transferable so the worker hands them back with zero copy. +// +// STORE_COLUMNS mirrors python store.STORE_COLUMNS exactly so the two analysis +// paths read identical channels. + +export const STORE_COLUMNS = Object.freeze([ + // Per-phase L-N voltage min/max/avg + 'V_LN_a_min_V', 'V_LN_b_min_V', 'V_LN_c_min_V', + 'V_LN_a_max_V', 'V_LN_b_max_V', 'V_LN_c_max_V', + 'V_LN_a_avg_V', 'V_LN_b_avg_V', 'V_LN_c_avg_V', + // Per-phase current max + avg + 'I_a_max_A', 'I_b_max_A', 'I_c_max_A', + 'I_a_avg_A', 'I_b_avg_A', 'I_c_avg_A', + // Line frequency + 'freq_avg_Hz', + // Power / apparent / reactive / power-factor totals + 'P_total_avg_W', 'S_total_avg_VA', 'Q_total_avg_VAR', 'PF_total_avg', + 'DPF_total_avg', + // Per-row energy (per-bucket kWh roll-ups) + 'Wh_total', + // THD per phase (IEEE 519) — V and I, avg only + 'V_THD_pct_a_avg', 'V_THD_pct_b_avg', 'V_THD_pct_c_avg', + 'I_THD_pct_a_avg', 'I_THD_pct_b_avg', 'I_THD_pct_c_avg', +]); + +// Chart series the web UI renders straight from the store. If a chart needs a +// channel not in STORE_COLUMNS the renderer must fall back to read-through. +export const CHART_COLUMNS = STORE_COLUMNS; + +/** + * Resolve STORE_COLUMNS to spec float indices, once. + * @param {object} spec parsed field_map.json + * @returns {Map} name -> spec float index + */ +export function resolveStoreIndices(spec) { + const nameToIdx = new Map(spec.fields.map((f) => [f.name, f.index])); + const out = new Map(); + for (const name of STORE_COLUMNS) { + const idx = nameToIdx.get(name); + if (idx === undefined) { + throw new Error(`Store column ${name} missing from spec/field_map.json`); + } + out.set(name, idx); + } + return out; +} + +export class ColumnStore { + /** + * @param {number} n record count (used to pre-size typed arrays) + */ + constructor(n = 0) { + this.n = n; + this.cols = {}; + for (const name of STORE_COLUMNS) this.cols[name] = new Float32Array(n); + this.startMs = new Float64Array(n); + this.endMs = new Float64Array(n); + } + + /** Packed Float32Array column for `name` (no copy). */ + col(name) { + const c = this.cols[name]; + if (c === undefined) { + throw new Error( + `Column ${name} is not retained in the ColumnStore. ` + + `Retained columns: ${STORE_COLUMNS.join(', ')}` + ); + } + return c; + } + + start(i) { return this.startMs[i]; } + end(i) { return this.endMs[i]; } + + get firstStartMs() { return this.n ? this.startMs[0] : null; } + get lastEndMs() { return this.n ? this.endMs[this.n - 1] : null; } + + /** + * Lazily yield a lightweight record view ({index, startMs, endMs, floats}) + * for each record. The `floats` proxy only carries the retained columns at + * their spec index; reads of non-retained indices return 0. Used to feed the + * existing record-array consumers (events.js, snapshots.js, insights.js) + * without holding 590 K full record objects. + * + * @param {object} spec + * @returns {Array} array of record-shaped views + */ + toRecords(spec) { + const idxByName = resolveStoreIndices(spec); + const dataFloats = spec.data_floats; + const colArrays = STORE_COLUMNS.map((name) => [idxByName.get(name), this.cols[name]]); + const out = new Array(this.n); + for (let i = 0; i < this.n; i++) { + const floats = new Float32Array(dataFloats); + for (const [idx, arr] of colArrays) floats[idx] = arr[i]; + out[i] = { index: i, startMs: this.startMs[i], endMs: this.endMs[i], floats }; + } + return out; + } + + /** + * Reconstruct a ColumnStore from a worker "done-columnar" payload: + * { recordCount, columns: {name: Float32Array}, startMs, endMs }. + */ + static fromTransfer(payload) { + const store = Object.create(ColumnStore.prototype); + store.n = payload.recordCount; + store.cols = payload.columns; + store.startMs = payload.startMs; + store.endMs = payload.endMs; + return store; + } + + /** + * Build a ColumnStore from an array of record objects (legacy small-file / + * test path). Mirrors python ColumnStore.from_records. + * @param {Array<{startMs:number, endMs:number, floats:ArrayLike}>} records + * @param {object} spec + */ + static fromRecords(records, spec) { + const idxByName = resolveStoreIndices(spec); + const store = new ColumnStore(records.length); + const colArrays = STORE_COLUMNS.map((name) => [idxByName.get(name), store.cols[name]]); + for (let i = 0; i < records.length; i++) { + const r = records[i]; + for (const [idx, arr] of colArrays) arr[i] = r.floats[idx]; + store.startMs[i] = r.startMs; + store.endMs[i] = r.endMs; + } + return store; + } +} diff --git a/web/events.js b/web/events.js index 92546da..f31b646 100644 --- a/web/events.js +++ b/web/events.js @@ -1,6 +1,8 @@ // Event detection — JS port of python/src/fluke_3540/events.py. // Mirrors the same thresholds, mask logic, and Event shape. +import { asColumnSource } from './column_source.js'; + export const DEFAULT_RULES = Object.freeze({ outage_v_threshold: 50.0, dip_pct_of_nominal: 0.90, @@ -16,6 +18,49 @@ export const DEFAULT_RULES = Object.freeze({ const PHASES = ['a', 'b', 'c']; +// Valid EventRules keys for --rules-file overrides (Feature I). Mirrors the +// Python EventRules dataclass fields. +const RULE_KEYS = new Set(Object.keys(DEFAULT_RULES)); + +/** + * Resolve per-asset EventRules from a parsed rules-file object (JSON form), + * mirroring python rules_file.load_rules. Precedence: DEFAULT_RULES -> + * file.defaults -> file.assets[assetName] (or file.assets.default). A flat + * object with only rule keys is treated as defaults. + * + * @param {object} raw parsed rules object + * @param {string|null} assetName + * @returns {object} a rules object suitable for detectEvents({rules}) + */ +export function rulesFromObject(raw, assetName = null) { + if (!raw || typeof raw !== 'object') return { ...DEFAULT_RULES }; + let defaults; + let assets; + if ('defaults' in raw || 'assets' in raw) { + defaults = raw.defaults || {}; + assets = raw.assets || {}; + } else { + defaults = raw; + assets = {}; + } + const coerce = (over, where) => { + const out = {}; + for (const [k, v] of Object.entries(over)) { + if (!RULE_KEYS.has(k)) { + throw new Error(`${where}: unknown EventRules key ${k}`); + } + out[k] = (k === 'min_duration_secs' || k === 'gap_tolerance_secs') + ? Math.trunc(Number(v)) : Number(v); + } + return out; + }; + const merged = { ...DEFAULT_RULES, ...coerce(defaults, 'defaults') }; + let assetOver = {}; + if (assetName && assets[assetName]) assetOver = assets[assetName]; + else if (assets.default) assetOver = assets.default; + return { ...merged, ...coerce(assetOver, 'assets') }; +} + function fieldIndex(spec, name) { const f = spec.fields.find((f) => f.name === name); if (!f) throw new Error(`spec is missing field ${name}`); @@ -84,32 +129,26 @@ function inferNominalLnV(vAvgByPhase, outageThreshold) { } /** - * Detect events on an array of parsed Records. - * @param {Array<{index:number, startMs:number, endMs:number, floats:Float32Array}>} records + * Detect events on an array of parsed Records OR a ColumnStore. + * @param {Array<{index:number, startMs:number, endMs:number, floats:Float32Array}>|import('./column_store.js').ColumnStore} source * @param {object} spec - parsed field_map.json * @param {{nominalLnV?: number, rules?: object}} [opts] * @returns {Array} */ -export function detectEvents(records, spec, opts = {}) { +export function detectEvents(source, spec, opts = {}) { const rules = { ...DEFAULT_RULES, ...(opts.rules ?? {}) }; let { nominalLnV = null } = opts; - if (records.length === 0) return []; - - const VLNmin = PHASES.map((ph) => fieldIndex(spec, `V_LN_${ph}_min_V`)); - const VLNmax = PHASES.map((ph) => fieldIndex(spec, `V_LN_${ph}_max_V`)); - const VLNavg = PHASES.map((ph) => fieldIndex(spec, `V_LN_${ph}_avg_V`)); - const Imax = PHASES.map((ph) => fieldIndex(spec, `I_${ph}_max_A`)); - const freqIdx = fieldIndex(spec, 'freq_avg_Hz'); - const pIdx = fieldIndex(spec, 'P_total_avg_W'); + const src = asColumnSource(source, spec); + if (src.length === 0) return []; - const N = records.length; - // Extract columns once into typed arrays for speed. - const vMin = VLNmin.map((idx) => Float32Array.from(records, (r) => r.floats[idx])); - const vMax = VLNmax.map((idx) => Float32Array.from(records, (r) => r.floats[idx])); - const vAvg = VLNavg.map((idx) => Float32Array.from(records, (r) => r.floats[idx])); - const iMaxArr = Imax.map((idx) => Float32Array.from(records, (r) => r.floats[idx])); - const freqArr = Float32Array.from(records, (r) => r.floats[freqIdx]); - const pTotal = Float32Array.from(records, (r) => r.floats[pIdx]); + const N = src.length; + // Extract columns once into typed arrays for speed (no-copy for a store). + const vMin = PHASES.map((ph) => src.column(`V_LN_${ph}_min_V`)); + const vMax = PHASES.map((ph) => src.column(`V_LN_${ph}_max_V`)); + const vAvg = PHASES.map((ph) => src.column(`V_LN_${ph}_avg_V`)); + const iMaxArr = PHASES.map((ph) => src.column(`I_${ph}_max_A`)); + const freqArr = src.column('freq_avg_Hz'); + const pTotal = src.column('P_total_avg_W'); if (nominalLnV === null) { nominalLnV = inferNominalLnV(vAvg, rules.outage_v_threshold); @@ -127,8 +166,8 @@ export function detectEvents(records, spec, opts = {}) { const outageMask = notOutage.map((b) => !b); const events = []; - const startMs = (i) => records[i].startMs; - const endMs = (i) => records[i].endMs; + const startMs = (i) => src.startMs(i); + const endMs = (i) => src.endMs(i); const phaseChars = (phaseList) => phaseList.slice(); diff --git a/web/html_report.js b/web/html_report.js index 0f4c51a..321ec59 100644 --- a/web/html_report.js +++ b/web/html_report.js @@ -1,3 +1,5 @@ +import { formatLocalUtc, tzLabel } from './tzutil.js'; + // Self-contained HTML report — mirrors the Python html_report.py output so // the artifact looks the same regardless of which side built it. // @@ -195,7 +197,72 @@ function insightsHtml(findings) { return out.join('\n'); } -export function buildReportHtml({ title, config, records, spec, events, snapshots, findings = [] }) { +// Whole-session statistics table (Feature B) from a wholeSessionStats() dict. +function wholeStatsTableHtml(wholeStats) { + if (!wholeStats) return ''; + const cols = ['min', 'p1', 'p5', 'median', 'mean', 'p95', 'p99', 'max', 'stdev']; + const head = ['Channel', 'Unit', ...cols].map((h) => `${esc(h)}`).join(''); + const fmt = (v) => (Number.isFinite(v) + ? (Math.abs(v) >= 1000 ? v.toFixed(0) : v.toFixed(2)) : '—'); + const rows = []; + for (const [name, d] of Object.entries(wholeStats)) { + if (name.startsWith('_')) continue; + const cells = [esc(name), esc(d.unit), ...cols.map((c) => fmt(d[c]))] + .map((c) => `${c}`).join(''); + rows.push(`${cells}`); + } + const th = wholeStats._thresholds || {}; + const note = + `

Under-voltage (<${th.undervoltage_v} V): ${th.sec_undervoltage} s ` + + `(${(th.pct_undervoltage ?? 0).toFixed(2)}%). Over-current (>${th.overcurrent_a} A): ` + + `${th.sec_overcurrent} s (${(th.pct_overcurrent ?? 0).toFixed(2)}%).

`; + return `

Statistics

${head}` + + `${rows.join('')}
${note}`; +} + +// IEEE 519 + SARFI power-quality block (Feature F). +function pqHtml(pq) { + if (!pq) return ''; + const v = pq.ieee519.voltage; + const s = pq.sarfi; + const verdict = pq.ieee519.all_voltage_compliant ? 'COMPLIANT' : 'NON-COMPLIANT'; + return '

Power quality (IEEE 519 / 1159)

' + + `

IEEE 519 voltage THD p95 (limit ${pq.ieee519.limit_v_thd_pct.toFixed(0)}%): ` + + `A=${v.a.p95.toFixed(1)}%, B=${v.b.p95.toFixed(1)}%, C=${v.c.p95.toFixed(1)}% — ` + + `${verdict}.

` + + `

SARFI: 90=${s['SARFI-90']}, 80=${s['SARFI-80']}, 70=${s['SARFI-70']}, ` + + `50=${s['SARFI-50']}, 10=${s['SARFI-10']} (${s.events_considered} voltage events).

`; +} + +// Peak-demand block (Feature G). +function demandHtml(demand) { + if (!demand || !demand.n_windows) return ''; + const wmin = Math.round(demand.window_secs / 60); + return `

Demand

Peak ${wmin}-min demand: ` + + `${demand.peak_demand_kw.toFixed(1)} kW ` + + `(window ending ${esc((demand.peak_window_end || '').slice(0, 19))}Z); ` + + `mean demand ${(demand.mean_demand_w / 1000).toFixed(1)} kW.

`; +} + +// Time-range header in local + UTC (Feature H). +function timeRangeHtml(records, tz) { + if (!records || !records.length) return ''; + const t0 = records[0].startMs; + const t1 = records[records.length - 1].endMs; + let label = 'UTC'; + let fmt = (ms) => new Date(ms).toISOString().replace(/\.000Z$/, 'Z').replace('Z', '+00:00'); + if (tz && tz.toUpperCase() !== 'UTC') { + try { + formatLocalUtc(t0, tz); // throws on an invalid zone -> fall back to UTC + label = tzLabel(tz); + fmt = (ms) => formatLocalUtc(ms, tz); + } catch (_) { /* fall back to UTC */ } + } + return `

Time range (${esc(label)}): ` + + `${esc(fmt(t0))} → ${esc(fmt(t1))}

`; +} + +export function buildReportHtml({ title, config, records, spec, events, snapshots, findings = [], wholeStats = null, narrative = null, pq = null, demand = null, tz = null }) { const energy = summarizeRecords(records, spec); const stats = { 'Records (per-second)': records.length.toLocaleString(), @@ -207,10 +274,18 @@ export function buildReportHtml({ title, config, records, spec, events, snapshot 'Peak export (kW)': (energy.pNeg / 1000).toFixed(2), }; const charts = collectChartArtifacts(); + const narrativeHtml = narrative + ? `

Executive summary

${esc(narrative).replace(/\n/g, '
')}

` + : ''; const body = [ `

${esc(title)}

`, + narrativeHtml, + timeRangeHtml(records, tz), '

Summary

', summaryDlHtml(stats, config), + wholeStatsTableHtml(wholeStats), + pqHtml(pq), + demandHtml(demand), insightsHtml(findings), '

Events

', eventsTableHtml(events), diff --git a/web/index.html b/web/index.html index c36ee7e..585cf7e 100644 --- a/web/index.html +++ b/web/index.html @@ -64,7 +64,23 @@

Sessions

+ + + +