Port CT002/CT003 emulator to ESPHome as external component

.github/workflows/ci.yml

@@ -43,6 +43,112 @@ jobs:
       - name: Mypy
         run: uv run mypy src/
 
+  ct002-host-protocol:
+    needs: [ lint ]
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+        with:
+          enable-cache: true
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+
+      - name: Install build tooling
+        run: sudo apt-get update -qq && sudo apt-get install -y -qq cmake g++
+
+      - name: Install dependencies
+        run: uv sync --frozen --extra dev
+
+      - name: Run host-gcc protocol + wrappers + balancer parity tests
+        run: uv run pytest tests/components/ct002/test_host_protocol.py -v
+
+  # Discover the ct002 compile targets dynamically so adding a
+  # tests/components/ct002/test.*.yaml file is automatically compiled in CI
+  # without editing this workflow.
+  ct002-esphome-yaml:
+    needs: [ lint ]
+    runs-on: ubuntu-latest
+    outputs:
+      matrix: ${{ steps.find.outputs.matrix }}
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Collect test YAML files
+        id: find
+        run: |
+          matrix=$(ls tests/components/ct002/test.*.yaml | jq -R -s -c 'split("\n")[:-1]')
+          echo "matrix=$matrix" >> "$GITHUB_OUTPUT"
+
+  ct002-esphome-compile:
+    needs: [ ct002-esphome-yaml ]
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        yaml: ${{ fromJSON(needs.ct002-esphome-yaml.outputs.matrix) }}
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+
+      - name: Install ESPHome
+        run: pip install esphome
+
+      - name: Compile ${{ matrix.yaml }}
+        run: esphome compile ${{ matrix.yaml }}
+
+  ct002-host-e2e:
+    needs: [ lint ]
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+        with:
+          enable-cache: true
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+
+      - name: Install dependencies (astrameter + esphome)
+        run: |
+          uv sync --frozen --extra dev
+          uv tool install esphome
+
+      # Build both host binaries in their own steps so a compile failure
+      # shows up here (with full gcc output) rather than as an opaque pytest
+      # timeout inside the module fixtures. The fixtures find the binaries
+      # already built and skip recompiling.
+      - name: Build ct002 host binary
+        run: esphome compile tests/components/ct002/test.host.yaml
+
+      - name: Build ct002 test-hooks (e2e control) binary
+        run: esphome compile tests/components/ct002/test.e2e.host.yaml
+
+      # All cross-backend suites are tagged `@pytest.mark.esphome_e2e`
+      # (see pyproject.toml). Selecting by marker means new suites are
+      # picked up automatically without editing this workflow. Their
+      # `python` params also run in the `validate` job; the `esphome`
+      # params only run where the esphome CLI + test-hooks binary exist.
+      - name: Run host-platform E2E (cross-backend esphome suites)
+        run: uv run pytest -m esphome_e2e -v
+
   validate:
     needs: [ lint ]
     runs-on: ubuntu-latest

AGENTS.md

@@ -17,6 +17,10 @@ uv run pytest
 
 CI runs the same steps (see `.github/workflows/ci.yml`).
 
+## Python ↔ ESPHome parity (REQUIRED)
+
+`esphome/components/ct002/` is a C++ mirror of the Python CT002 stack. Any change to shared behavior must land on **both** sides in the same change. See `CONTRIBUTING.md` for the file mapping and what has no C++ counterpart. Verify with `uv run pytest tests/components/ct002/`.
+
 ## Changelog
 
 For user-facing work on a branch, keep **one bullet under `## Next`** that summarizes the **overall** outcome of that branch. **Add** it when you first document the change; on **later iterations** on the same branch, **edit that same bullet** if the scope or wording shifts—do **not** append extra bullets for each follow-up. Skip `CHANGELOG.md` entirely when nothing users would notice changes (refactors, tests-only, etc.).
@@ -25,6 +29,10 @@ Do **not** expand `CHANGELOG.md` with every internal or tooling-only follow-up.
 
 ## Adding a powermeter
 
+Powermeters are Python-only and have **no** ESPHome counterpart (the ESPHome
+component reads grid power from any native ESPHome sensor instead), so the
+parity rule above does not apply here.
+
 1. **Implementation** — Add `src/astrameter/powermeter/<module>.py` with a class subclassing `Powermeter`; implement `get_powermeter_watts()` (and `wait_for_message()` only if the base default is wrong for your source).
 2. **Exports** — Import and re-export the class from `src/astrameter/powermeter/__init__.py`.
 3. **Config** — In `src/astrameter/config/config_loader.py`: import the class, define a `*_SECTION` string, add a `section.startswith(...)` branch in `create_powermeter()`, and a `create_*_powermeter()` factory that reads options from the section. `POWER_OFFSET` / `POWER_MULTIPLIER`, `THROTTLE_INTERVAL`, and `NETMASK` are handled globally for any section that returns a powermeter — no extra wiring unless you need something custom.

CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## Next
 
+- **Added** experimental ESPHome external component `ct002` to run the CT002/CT003 emulator directly on an ESP32. See `esphome.example.yaml`.
 - **Added** Modbus UDP support via a `TRANSPORT = TCP|UDP` option in the `[MODBUS]` section (defaults to `TCP`).
 - **Fixed** Home Assistant powermeter timing out on startup with "Timeout waiting for Home Assistant state" when the `subscribe_entities` initial snapshot never arrives (e.g. entity not yet loaded when AstraMeter starts). The Home Assistant powermeter now also issues an explicit `get_states` fetch right after subscribing to seed the cache.
 - **Fixed** multi-Venus setups where a battery passing PV through to the grid (full SoC with "feed excess to grid" enabled) caused other batteries on different phases to stop charging. AstraMeter now populates the CT002 cross-talk `*_chrg_power` / `*_dchrg_power` fields from the per-battery instruction it sent rather than from the battery's reported AC output, so involuntary PV-passthrough no longer looks like a battery discharge to the rest of the fleet ([#376](https://github.com/tomquist/astrameter/issues/376)).

CONTRIBUTING.md

@@ -47,6 +47,32 @@ CI runs the same (ruff format check, ruff check, mypy on `src/`, pytest with cov
 
 Follow the checklist in [AGENTS.md](AGENTS.md) (**Adding a powermeter**), using paths under `src/astrameter/` (e.g. `src/astrameter/powermeter/<module>.py`, `src/astrameter/config/config_loader.py`).
 
+## ESPHome external component (parity rule)
+
+The `esphome/components/ct002/` directory is a C++ port of `src/astrameter/ct002/` and related modules. **Python is canonical, C++ is a mechanical mirror.** Filenames, class names, function names, and filter ordering all match Python so that a bug fix on one side maps to one file on the other.
+
+When you fix a bug in:
+
+- `src/astrameter/ct002/balancer.py` → also port the fix to `esphome/components/ct002/balancer.{h,cpp}` in the same PR.
+- `src/astrameter/ct002/ct002.py` → `ct002.{h,cpp}` (including the response-builder math, MAC validation, and `_compute_smooth_target` dispatch).
+- `src/astrameter/ct002/protocol.py` → `protocol.{h,cpp}`. Add a vector to `tests/components/ct002/fixtures/protocol_golden_vectors.json` if the behaviour change affects wire bytes; both the Python pytest and the host-gcc gtest will pick it up automatically once `_populate_wire_hex.py` is re-run.
+- `src/astrameter/powermeter/wrappers/{hampel,smoothing,pid}.py` → the matching `{hampel,smoothing,pid}.{h,cpp}` in the component directory.
+- `src/astrameter/mqtt_insights/marstek_mqtt.py` → `esphome/components/ct002/marstek_responder.{h,cpp}` (sub-block under `ct002:`). Wire-format changes (topic templates, `cd=1`/`cd=4` payload tokens, k=v ordering) must keep host-gcc `host_marstek_responder_test` green so the Marstek app and hm2mqtt-style parsers see identical bytes from both stacks.
+- `src/astrameter/mqtt_insights/discovery.py` → `esphome/components/ct002/ha_discovery.{h,cpp}`. Keep `node_id`/`unique_id`/`value_template` strings identical so HA dedupe across the Python and ESPHome paths works correctly when both happen to share a broker.
+- `src/astrameter/mqtt_insights/service.py` → `esphome/components/ct002/mqtt_insights.{h,cpp}`. The ESPHome port intentionally omits the asyncio queue, the reconnect loop, and the ARP lookup — see the header for the documented architectural diff. The whole file is gated by `#ifdef USE_MQTT` so it's a no-op on builds without `mqtt:` configured.
+- `src/astrameter/marstek_api.py` → `esphome/components/ct002/marstek_registration.{h,cpp}`. Keep the URL paths (`/app/Solar/v2_get_device.php`, `/ems/api/v1/getDeviceList`, `/app/Solar/v2_add_device.php`), the User-Agent (`Dart/2.19 (dart:io)`), the password MD5 hashing, and the `02b250` managed-MAC prefix in lockstep — the cloud API responses depend on a specific payload shape. The ESPHome port's only architectural change is running the Python helper's linear flow as a state machine in `loop()` so the watchdog stays fed between HTTPS calls. Gated by `#ifdef USE_CT002_MARSTEK_REGISTRATION` (defined from `_to_code_marstek_registration` in ct002/__init__.py).
+
+Fixes to `src/astrameter/powermeter/wrappers/{transform,throttling}.py` have **no** C++ counterpart — those wrappers are delegated to ESPHome's standard `sensor: filters:` (`offset:`, `multiply:`, `throttle:`) on the upstream sensor.
+
+The host-gcc gtest suite (`uv run pytest tests/components/ct002/test_host_protocol.py`) is the C++-side guard against translation drift. It builds via CMake with FetchContent-fetched googletest, so all you need locally is `cmake` and a C++17 compiler. Add a gtest case for any new C++ behavior that doesn't map 1:1 to a Python file.
+
+Two host e2e modules drive the compiled binary over real UDP:
+
+- `test_host_e2e.py` — the `BatterySimulator` round-trip against `test.host.yaml`, validating the real client path.
+- `test_shared_e2e.py` — **differential** scenarios written once and parametrized over two backends with a common `poll / set_grid / set_clock / advance_clock` interface: `python` (the canonical `CT002` driven in-process via `_handle_request` + a fake transport) and `esphome` (the host binary). Asserting the same wire facts on both is the cross-stack parity guard. The `python` parametrizations need no ESPHome toolchain; the `esphome` ones skip without it.
+
+The `esphome` backend uses a "test-hooks" binary (`test.e2e.host.yaml`) that compiles in a UDP control channel — enabled only by the test-only `test_control_port:` option, which adds the `USE_CT002_TEST_HOOKS` define (see `test_hooks.cpp`). The channel injects grid power and drives a mock clock so time-gated behaviour (dedup, saturation, eviction) is deterministic against the black-box binary. `test_control_port:` is **test-only** — never set it in a real config. When you add a shared scenario, write it against the `backend` fixture so it runs on both stacks.
+
 ## Branches and pull requests
 
 - Base feature work on **`develop`** and open PRs against **`develop`**.

README.md

@@ -149,6 +149,40 @@ image: ghcr.io/tomquist/astrameter:next
 
 All commands above work across Windows, macOS, and Linux. The only difference is how you open your terminal.
 
+### ESPHome External Component (run on an ESP32)
+
+AstraMeter also ships as an **ESPHome external component** that runs the CT002/CT003 emulator, balancer, and cross-phase filter pipeline directly on an ESP32 — no Python add-on, no Home Assistant required. Useful if you'd rather flash a dedicated board than run a server, and if your grid-power source is already addressable by ESPHome (Modbus, M-Bus, Tasmota, MQTT, Shelly, Envoy, etc.).
+
+Minimal YAML — point `power_sensor_l1` at any ESPHome sensor that reports grid power in watts:
+
+```yaml
+external_components:
+  - source: github://tomquist/astrameter@develop
+    components: [ct002]
+
+sensor:
+  - platform: homeassistant       # or modbus_controller / mqtt / template / …
+    id: grid_l1
+    entity_id: sensor.grid_power
+
+ct002:
+  id: ct002_main
+  power_sensor_l1: grid_l1
+```
+
+Everything else is optional. See **[`esphome.example.yaml`](esphome.example.yaml)** for the complete, annotated config — three-phase sensors, the cross-phase filter pipeline (Hampel / smoothing / deadband / PID), balancer and saturation tuning, and the two optional sub-blocks below — with every knob shown at its default.
+
+Two optional sub-blocks nest under the same `ct002:` key:
+
+- **`mqtt_insights:`** — publishes Home Assistant Device Discovery (one device per battery + a parent CT002 device with manual-target / active / auto-target controls and a force-rotation button) and answers Marstek-app polls on your MQTT broker, so the emulator shows up in the app without hame-relay. Requires an `mqtt:` block.
+- **`marstek_registration:`** — registers a managed CT002/CT003 with your Marstek cloud account on first boot (same flow as the Python `[MARSTEK]` section), persists the assigned MAC, and feeds it back into `ct002.ct_mac`. Requires an `http_request:` block. When combined with `mqtt_insights:`, the App-topic subscription picks up the MAC automatically — no reboot needed.
+
+**Status:** experimental — UDP emulator, balancer, filter pipeline, MQTT-insights, and Marstek cloud registration are all functional. Wider field testing welcome.
+
+**Requirements:** ESP32 with ≥4 MB flash (default for `esp32dev`, `esp32-s3-devkitc-1`, etc.). ESP8266 is not supported in v1 — RAM and flash budgets are too tight once HTTPS+TLS, MQTT, and the balancer are linked together. Pick a board with `flash_size: 4MB` or larger; for ESP-IDF builds you may need a custom partition table when you also add HTTPS+MQTT — there is no top-level `flash_size:` YAML key, set it via your `board:` choice and (for ESP-IDF) `esp32: framework: type: esp-idf` with appropriate `sdkconfig_options:` or a partition CSV.
+
+**One important divergence from the Python emulator:** per-phase transforms and throttling are *not* part of `ct002:` — they're delegated to ESPHome's standard `sensor: filters:` (`offset:`, `multiply:`, `throttle:`) on the upstream sensor. This matches the canonical order in Python (`Transform → Throttle → Hampel → Smoothed → Deadband → PID`). Put per-phase filters on the sensor itself, not after `ct002:` — they need to apply to the raw input, not the balancer's output.
+
 ## Additional Notes
 
 When the script is running, switch your Marstek battery to "Self-Adaptation" mode to enable the powermeter functionality.