Skip to content

Batch modernization phase 1: offline BatchPCA, dataset loaders, converter completion - #459

Open
kgdunn wants to merge 8 commits into
mainfrom
claude/process-improve-library-review-9h1fn2
Open

Batch modernization phase 1: offline BatchPCA, dataset loaders, converter completion#459
kgdunn wants to merge 8 commits into
mainfrom
claude/process-improve-library-review-9h1fn2

Conversation

@kgdunn

@kgdunn kgdunn commented Jul 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Summary

  • Adds BatchPCA, an offline batchwise-unfolded (BWU) PCA model for batch data: dict-of-batches in, batch-indexed scores/T2/SPE out, with an optional initial-conditions (Z) block joined onto the one-row-per-batch unfolded matrix. Built by composition over the existing multivariate.PCA + MCUVScaler, so it reuses the library's proven numerics. Loadings keep the (tag, sequence) index so they reshape to a variable-by-time grid.
  • Adds two batch model plots (time_varying_loading_plot, contribution_at_time_plot) and dataset loaders (load_nylon, load_dryer, load_batch_fake_data).
  • Completes the three unimplemented data_input converters (melted_to_wide, wide_to_dict, wide_to_melted) and sweeps small cleanups in batch/ (dead plotly import, resonsive typo, stdlib random -> check_random_state, print -> logger, stray flake8 noqa).

This is Phase 1 of the batch data processing modernization plan (offline model + loaders + cleanups). Online Nomikos-MacGregor monitoring, sklearn transformer facades, and BatchPLS-to-quality are planned follow-up PRs.

Test plan

  • New tests/batch/test_batch_pca.py: fit on aligned nylon data (shapes, monotone R2, loadings reshape to variable x time), transform round-trip, diagnose consistency, synthetic injected-fault batch flagged by SPE/T2, Z-block join and mismatch validation, wrong-length rejection
  • New tests/batch/test_batch_plots.py: both plots build; component/column-shape guards
  • New tests/batch/test_datasets.py: loaders return valid batch dicts
  • tests/batch/test_data_input.py: stub-pinning assertions replaced with round-trip identities (including a real-nylon round-trip)
  • uv run pytest full suite: 2026 passed, 2 pre-existing skips
  • uv run ruff check . and uv run mypy src/process_improve both pass
  • End-to-end smoke: load nylon -> resample-align -> BatchPCA fit -> diagnose -> score/SPE/loading/contribution plots render

Checklist

  • Version bumped in pyproject.toml (MINOR: 1.54.0 -> 1.55.0, new modules/API) and CITATION.cff synced
  • Tests added or updated where relevant
  • ruff check . passes
  • CHANGELOG.md updated

🤖 Generated with Claude Code

https://claude.ai/code/session_01SyEpexCyqHn1rwTMYQSiP6

claude added 2 commits July 16, 2026 20:58
@claude
Start batch modernization: offline batch PCA, dataset loaders, conver…
7f11dce 
…ter completion

Opens the Phase 1 work described in the batch data processing
modernization plan: an offline batchwise (BWU) PCA model with
batch-level diagnostics, dataset loader functions for the bundled
nylon/dryer/fake batch data, completion of the three unimplemented
data_input converters, and a sweep of small lint/dead-code cleanups.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SyEpexCyqHn1rwTMYQSiP6
@claude
Add loader functions for the bundled batch datasets
bbba739 
load_nylon(), load_dryer(), and load_batch_fake_data() return the
standard batch-data dictionary directly, following the loader-function
precedent in experiments/datasets.py. Until now every consumer had to
locate the CSVs by path and convert them by hand; the book's worked
examples need a one-line reproducible load.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SyEpexCyqHn1rwTMYQSiP6
@codecov

codecov Bot commented Jul 16, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.15385% with 8 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/process_improve/batch/_batch_pca.py 96.58% 3 Missing and 1 partial ⚠️
src/process_improve/batch/_batch_plots.py 92.30% 1 Missing and 3 partials ⚠️

📢 Thoughts on this report? Let us know!

claude added 6 commits July 16, 2026 21:07
@claude
Complete the melted/wide/dict converter stubs in batch data_input
1b64df6 
melted_to_wide returned an empty dict, wide_to_melted an empty frame,
and wide_to_dict None; the tests pinned that placeholder behaviour.
All three are now implemented by composing the existing, tested
conversions (melted_to_dict, dict_to_wide, dict_to_melted), and
wide_to_dict inverts the (tag, sequence) pivot directly. The tests now
assert round-trip identities instead of emptiness, including a
truncation-aligned round-trip on the real nylon dataset. wide_to_dict
and the new dataset loaders are exported from the batch namespace.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SyEpexCyqHn1rwTMYQSiP6
@claude
Sweep lint and dead-code debt in the batch subpackage
76dcdda 
- plotting.py: fix the silently-ignored 'resonsive' Plotly config typo;
  replace stdlib random.seed/shuffle with the package-wide
  check_random_state contract; replace the inert file-level
  'flake8: noqa: C901' directive with per-function noqa codes (it was
  blanket-suppressing C901 for the whole file under ruff).
- preprocessing.py: remove the dead guarded plotly import (go was never
  used in this module) and route the DTW progress print through the
  module logger.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SyEpexCyqHn1rwTMYQSiP6
@claude
Add BatchPCA: batchwise-unfolded (multiway) PCA for batch data
68479e0 
BatchPCA brings the Nomikos-MacGregor batch modelling approach to the
library. It unfolds an aligned batch-data dictionary batchwise (one row
per batch) via dict_to_wide, optionally joins an initial-conditions (Z)
block onto that row, mean-centres and scales every column with
MCUVScaler (which removes the average trajectory), and fits the existing
multivariate.PCA by composition. Batch-level scores, SPE and Hotelling's
T2 with control limits, plus contribution and score/SPE/loading plot
forwarders, are exposed as batch-indexed views. Loadings keep the
(tag, sequence) index so they reshape to a variable-by-time grid.

Until now the batch module stopped at feature extraction with no
multiway model; the unfold-then-PCA path existed only privately inside
find_reference_batch. This is the headline capability the batch case
study in the book depends on.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SyEpexCyqHn1rwTMYQSiP6
@claude
Add batch model plots: time-varying loadings and contribution-at-time
d8067bc 
time_varying_loading_plot draws one component's loadings as a function
of time (one trace per tag), reading the batchwise-unfolded model's
(tag, sequence) loading grid; initial-condition loadings are shown as a
marker group before time zero. contribution_at_time_plot renders the
per-tag SPE or T2 contribution at a chosen time sample for one batch, to
localize an abnormal event to the responsible variable. Both are plotly,
guarded behind the plotting extra, and use the package theme. The batch
score/SPE/T2 plots reuse the multivariate plot functions on the internal
model, so only these two time-resolved plots are new.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SyEpexCyqHn1rwTMYQSiP6
@claude
Bump to 1.55.0: batch modelling, loaders, converters, changelog and docs
2b40e03 
Version bump for the new batch capabilities (BatchPCA, batch model
plots, dataset loaders, completed data_input converters). CITATION.cff
version and date-released synced to 1.55.0 / 2026-07-16; CHANGELOG
section added with the compare-link footer; docs/api/batch.rst now
autodocuments BatchPCA, the dataset loaders, and the batch plots.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SyEpexCyqHn1rwTMYQSiP6
@claude
Cover BatchPCA and batch-plot error branches
945710d 
Adds tests for the validation and error paths flagged as uncovered:
initial-conditions type/numeric/missing-value guards, group_by_batch
with a Z block, fit_transform, diagnose with initial conditions,
ellipse_coordinates, and the contribution-plot unknown-batch and
out-of-range-time errors. Raises patch coverage of the new batch
modules (_batch_pca to 97%, _batch_plots to 94%).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SyEpexCyqHn1rwTMYQSiP6
This was referenced Jul 16, 2026
Open
Open
Open
This was referenced Aug 21, 2026
Merged
Open
kgdunn pushed a commit that referenced this pull request Aug 21, 2026
@claude
Complete the wide-format batch converters (from #459)
5dfd2f8 
melted_to_wide, wide_to_dict and wide_to_melted were stubs; they now
round-trip the standard batch dictionary, the melted frame and the
one-row-per-batch wide frame with the 2-level (tag, sequence) column
index. Ported from the stale #459 stack onto current main.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NZAyiPT8xURSdi24yFKCra
kgdunn pushed a commit that referenced this pull request Aug 21, 2026
@claude
Add the batch modelling substrate: BatchPCA, BatchPLS, BatchMonitor, …
d534669 
…plots, loaders

Consolidates the stale #459 / #460 / #462 stack onto current main, as
one commit series inside the mid-course-correction PR:

- BatchPCA: batchwise-unfolded (Nomikos-MacGregor) PCA with an optional
  initial-conditions (Z) join, composed over multivariate.PCA and
  MCUVScaler; batch-level scores, SPE, T2, contributions and the plot
  and limit conveniences.
- BatchPCA.predict_online: projection to the model plane for a
  partially-observed batch (upgraded to the shared missing-data
  projection API in a later commit series).
- BatchMonitor: time-varying online SPE and T2 limits learned from good
  batches, with per-sample alarms, plus online_monitoring_plot.
- BatchPLS: batchwise-unfolded [Z | X] -> Y regression to final
  quality. Refreshed beyond the original: the class now owns both
  scalers publicly (center_/scale_ for the unfolded block,
  y_center_/y_scale_ for the quality block; the inner PLS runs with
  scale=False), reports beta and RMSE in engineering units, and adds a
  prediction_interval wrapper. The mid-course corrector reads these
  public attributes instead of any PLS internals.
- time_varying_loading_plot and contribution_at_time_plot for the
  unfolded structure; dataset loaders load_nylon, load_dryer and
  load_batch_fake_data for the bundled batch CSVs.
- Integration test: BatchPLS on a simulator historical campaign (80
  batches, 4 components) reaches fit R2 0.83 and out-of-sample
  correlation 0.80 against fresh batches; thresholds pinned below with
  margin.

The #461 transformer facades are deliberately not re-landed; they can
be revived separately if wanted.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NZAyiPT8xURSdi24yFKCra
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kgdunn @claude