fix: pin python deps, map encoder errors to 400 and refresh sink topics

[perezmlal]

Description

Follow-up to the dependency bumps merged today (#75, #76, #79, #81, #82). Testing them end to end, against the full Docker stack as well as in CI, turned up four problems that CI could not see. None of them was introduced by those PRs.

1. Event Publisher returns 500 for some malformed bodies. asn1tools reports most schema mismatches as EncodeError, but it raises KeyError for an unknown enumerated value (e.g. "altitudeConfidence": "nope") and TypeError for a string in an integer field. Only EncodeError was handled, so these two cases returned 500. Both are now caught and return 400, with the exception type added to the warning log.

2. On a fresh stack, the first message takes ~5 minutes to reach MQTT. The MQTT Sink connector subscribes to topics.regex about 100 s after start. v2x.denm.public only exists after the first POST, and a pattern subscription discovers new topics on its metadata refresh, which defaults to 5 minutes. The connector now sets consumer.override.metadata.max.age.ms: 10000. This is the "first end-to-end test still fails" row in the HTTP testing guide's troubleshooting table, which used to recommend restarting connect.

3. Two Docker images install unpinned Python dependencies. Dockerfile.nifi (pip3 install xmltodict asn1tools) and the Kafka speed exporter Dockerfile (pip install kafka-python prometheus_client) installed whatever was latest at build time. In practice, a local NiFi image still ran asn1tools 0.167.0 from Docker's build cache while a fresh build got 0.169.0. Both images now install pinned requirements.txt files, which Dependabot tracks. The NiFi scripts' Poetry project is also aligned to asn1tools 0.169.0, which removes 362 pyparsing deprecation warnings from the test run.

4. CI does not test the pinned versions. The test job ran pip install pytest pytest-cov flask asn1tools xmltodict prometheus_client, with no versions and without confluent_kafka at all. A green Dependabot PR therefore proved nothing about the version it bumped. Each suite now runs in its own virtualenv built from its component's requirements file. A new test also checks the reference DENM's exact UPER bytes, so an asn1tools bump that changes the encoding fails CI.

Type of change

• Bug fix
• New feature
• Documentation
• Refactor / chore

Checklist

• I have read the Contributing guidelines.
• Tests added or updated for the change (see the Testing Policy), or I have explained below why tests are not applicable.
• poetry run pytest passes locally.
• ruff check and bandit report no new issues (see the Development guide).
• Documentation updated where relevant (README, docs/, CHANGELOG.md).
• I am not including secrets, credentials, or sensitive data.
• Security-impacting changes have been considered; vulnerabilities are reported privately via SECURITY.md, not in this PR.

How this was verified

Check	Result
New regression tests (unknown-enum, string-for-integer)	Fail without the fix, pass with it
CI test job replayed locally (same run: blocks), Python 3.9 and 3.12	EP 21/21 (91.9%), exporter 3/3 (93.8%), NiFi 7/7 (98.5%)
poetry run pytest with the new lock, Python 3.9 and 3.12	7/7
ruff (locked 0.16.10 and latest), bandit, poetry check --lock, actionlint	Clean
Full stack docker compose up --build from this branch, fresh volumes	All 11 services up; NiFi flow bootstrapped; all Prometheus targets up
Images	NiFi: asn1tools 0.169.0, xmltodict 1.0.4; exporter: kafka-python 3.0.11, prometheus_client 0.26.0
Sink consumer config (from the Connect log)	metadata.max.age.ms = 10000 (the override was accepted)
First message on a fresh stack, POST → MQTT	10.7 s (was ~300 s; measured on main earlier today)
Later messages, POST → MQTT	0.9 s
Live API: unknown enum / string in integer field	400 (was 500)
NiFi encoder_decoder.py inside the container (DenmEtsi multi-part)	Byte-identical to the previous 0.167.0 output; no warnings

Notes for reviewers

• Existing stacks: the connector config is POSTed once, on first start, and Kafka has no volume. A running stack keeps the old config until it is recreated (docker-compose down && docker-compose up -d). This is noted in CHANGELOG.md and the troubleshooting table.
• Rebuilding the NiFi image now installs asn1tools 0.169.0 instead of whatever the build cache holds. The output was verified byte-identical.
• Duplicated NiFi dependencies: deployment/nifi/requirements.txt (what the image installs, and what CI tests) and nifi-scripts/pyproject.toml (the Poetry dev project) list the same libraries. Dependabot updates each directory separately. This is documented in docs/Development.md and in the AGENTS.md watchlist, the same way the duplicated ASN.1 schemas are.
• Not in this PR: out-of-range values (e.g. -5 for an unsigned station ID, or 10**40) still encode without error, because asn1tools does not check constraints by default. Turning that on is a behavior change for clients, so it deserves a separate PR.