fix(loadmodel): anchor outlier rejection to the fuse, not to learned MAE - #742
Conversation
A rejected sample updates neither MAE nor Samples, so the band max(MAE × 10, 200) could never grow in response to being persistently wrong. With the filter arming after 50 samples — 51 minutes at the 60 s cadence — a model calibrated on one quiet hour rejected the real house permanently. Measured on a clean NewModel(10000): after one overnight hour at 400 W: samples=60 MAE=57.0 band=570.5 then four hours at 5 kW: accepted=0 rejected=240 prediction for that hour: 1794 W (truth 5000 W) after a full week at 5 kW: 1794 W — MAE still 57.0 100% rejection, permanently, because MAE only moves on accepted samples and nothing is accepted. The failure is silent and reads as a forecast problem from outside: a small reported error next to an arbitrarily large real one. Three changes: - Hard bound at 3 × PeakW, always on. Above the site's rated draw a reading is a fault, not a household. Absolute and derived from configured hardware, so it holds from the first sample and cannot be talked down by a model that has mislearned — this is what keeps the first day safe now that the soft filter arms later. - The soft filter arms after a day rather than an hour, so its band spans a night-and-day cycle instead of whichever hour followed a restart. - Ten consecutive same-direction rejections widen the band by exactly enough to admit the residual; the ordinary EMA takes over from there. Accepting the one sample and resetting was not enough — the band stayed narrow, the next nine were rejected too, and a day at a new level only reached 1650 of 3000 W. After: the 3-minute 6 kW spike is still rejected in full, a sustained shift is picked up within ten minutes, and every trained bucket tracks it exactly. Refs #739 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c74980a3ea
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if m.PeakW > 0 && actualLoadW > implausibleLoadFactor*m.PeakW { | ||
| return false |
There was a problem hiding this comment.
When an implausible reading arrives during cold weather for a bucket with at least MinTrustSamples, the heating-fit block above has already applied its residual to HeatingW_per_degC before this check returns false. Consequently, even one rejected meter fault can alter cold-weather forecasts, and repeated faults can drive the coefficient to HeatingCoefMaxW and predictions to the 3 * PeakW ceiling while Update reports that no update was applied. Perform this absolute fault check before the heating fit; only the adaptive soft filter needs to run afterward.
Useful? React with 👍 / 👎.
The MAE band rejected the upper half of the real load distribution, and could lock the model out of a level it had not seen before, permanently. A rejected sample updates neither MAE nor Samples, so the band max(MAE × 10, 200) never grew in response to being persistently wrong. Measured on a clean NewModel(10000): after one overnight hour at 400 W: samples=60 MAE=57.0 band=570.5 then four hours at 5 kW: accepted=0 rejected=240 prediction for that hour: 1794 W (truth 5000 W) after a full week at 5 kW: 1794 W — MAE still 57.0 The first fix attempt kept the band and let a run of same-direction rejections widen it. That worked, but it was propping up the wrong instrument. Household load is multimodal — a few hundred watts of baseline, then 11 kW when the sauna, oven and car overlap — and nothing about a residual's size separates "unusual but real" from "wrong". A band fitted to the quiet hours always excludes the busy ones, so the filter's normal operation, not just its failure mode, was discarding real consumption. It was also redundant. Telemetry already runs a Kalman filter per signal and this model reads the smoothed values, so short-term noise is handled one layer down, where it belongs. What remains is the rejection physics licenses: a house cannot draw more than its main fuse passes. The ceiling is fuse capacity × 1.25, threaded in from configuration rather than derived from loadPeakW — that one is a tunable proxy for "typical peak", this is physics, and deriving the second from the first would silently move a safety bound whenever somebody retuned the proxy. No fuse configured disables the check rather than inventing a limit. A sustained shift is now learned directly. A one-minute spike still trains, because it is real, and the bucket EMA damps it to a tenth of the gap. Refs #739 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
frahlg
changed the title
The physical bound ran after the online heating fit, so a cold-weather meter fault applied its residual to HeatingW_per_degC and only then got rejected — with Update still reporting that no update was applied. A run of faults could walk the coefficient toward its ceiling, inflating every cold-weather prediction from readings the model had ostensibly discarded. Measured with the bound in its old position: 40 rejected 50 kW samples at 0 °C moved the coefficient from 0 to 919 W/°C. At an 18 °C delta that is 16.5 kW of invented heating load. A fault must touch nothing on its way out — not the buckets, not MAE, not the heating coefficient — so the bound is now the first thing Update does after the negative-load guard. The comment about fitting heating before the outlier filter explained why a stale coefficient needs to be able to recover from the adaptive band; it never applied to an absolute physical limit, and that band is gone anyway. The regression test keeps every sample inside one hour-of-week bucket. The first version let the faults land in the next hour, where the fit is gated on bucket trust rather than on the bound — it passed against the buggy ordering and proved nothing. Codex P2 on PR #742. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Codex P2 addressed in 7389102 — it was a real one, thanks. The bound sat below the online heating fit, so a cold-weather fault applied its residual to A fault must touch nothing on its way out, so the bound is now the first thing Worth recording how close the regression test came to being useless: the first version let the faults land in the following hour, where the fit is gated on bucket trust rather than on the bound. It passed against the buggy ordering. It now asserts every sample stays inside one hour-of-week bucket, and fails with |
1 participant
Closes #739 — though not for the reason that issue originally gave. See the correction comment there.
The defect
A rejected sample updates neither
MAEnorSamples. So the bandmax(MAE × 10, 200)can never grow in response to being persistently wrong. Measured on a cleanNewModel(10000):100% rejection, permanently.
MAEonly moves on accepted samples and nothing is accepted.Why the band goes, rather than getting fixed
My first attempt kept the band and let a run of same-direction rejections widen it. It worked — spike still rejected, sustained shift learned within ten minutes. It was also propping up the wrong instrument, and review caught that.
Household load is multimodal. A few hundred watts of baseline for most of the day, then 11 kW when the sauna, oven and car overlap. Both are true readings. Nothing about a residual's size separates "unusual but real" from "wrong", so a band fitted to the quiet hours always excludes the busy ones. The lockout was the visible failure; discarding the top of the distribution was the normal operation.
It was also redundant. Telemetry already runs a Kalman filter per signal, and this model reads
SmoothedW. Short-term measurement noise is handled one layer down, where it belongs. The band was a second filter at the wrong altitude, doing damage the first one had already made unnecessary.What replaces it
The rejection physics licenses, and no other: a house cannot draw more than its main fuse passes.
MaxPlausibleW = fuse capacity × 1.25, threaded in from configuration. Deliberately not derived fromloadPeakW— that is a tunable proxy for "typical peak" (currentlyfuse × 0.5), while this is physics. Deriving one from the other would silently move a safety bound the next time somebody retunes the proxy. With no fuse configured the check disables itself rather than inventing a limit.Because the ceiling comes from hardware and never from what the model has learned, it holds from the first sample and a mislearned model cannot talk it down — which is exactly the property the old band lacked.
Behaviour after
Five tests cover those.
TestRejectsOutliersis kept and updated rather than deleted: same behaviour asserted, now with the physical ceiling set, plus a comment on why the band version also rejected genuine peaks.make verifyclean.Note on scope
This does not prove the reported site hit this defect;
/api/loadmodelfrom that box would settle it. It reproduces from a clean model in seconds and its signature matches what was reported.🤖 Generated with Claude Code