Docs: the audit, the coverage, and the real-data checks behind them

README.md

@@ -321,7 +321,7 @@ that fits it and report the scale and offset.
 | Vector BLF | via python-can `BLFReader`. |
 | Vector ASC | via python-can `ASCReader`. |
 | MDF4 `.mf4` / `.mdf` | CANedge and similar. Needs `pip install canlab[mdf]`. |
-| openpilot `.rlog` / `.qlog` | Needs pycapnp and the cereal `log.capnp` schema. Raises a clear error if either is missing. |
+| openpilot `rlog` / `qlog` | Plain, `.bz2` or `.zst`, named the way openpilot names them. Needs `pip install canlab[openpilot]` (pycapnp); the cereal schema ships with CanLab. Frames the panda transmitted are kept apart from the car's traffic, and the log's GPS can be used as a calibration reference. |
 
 Every parser produces the same columns: `Timestamp, ID, Bus, DLC, Extended,
 B0..B7`, widened to `B63` when FD frames are present, plus a per-ID `Delta`. The
@@ -337,7 +337,7 @@ None of this sends anything anywhere.
 | Feature | Module | Notes |
 |---|---|---|
 | Checksum algorithms | `core/checksums.py` | Parametrised CRC-8 plus OEM variants (Hyundai, Toyota, Honda, Subaru, AUTOSAR), checked against published check values and against commaai/opendbc. |
-| J1939 and NMEA 2000 | `core/j1939.py` | Both protocols share the 29-bit frame and split the identifier the same way, so the data page decides which PGN table applies: J1939 PGNs and SPNs, or NMEA 2000's own range with radians, metres per second and kelvin. Multi-frame PGNs are reassembled first (see below) and decoded whole; one frame of one is never decoded alone, because that gives a confident wrong answer. |
+| J1939 and NMEA 2000 | `core/j1939.py` | Both protocols share the 29-bit frame and split the identifier the same way, so the data page decides which PGN table applies: J1939 PGNs and SPNs, or NMEA 2000's own range with radians, metres per second and kelvin. Multi-frame PGNs are reassembled first (see below) and decoded whole; one frame of one is never decoded alone, because that gives a confident wrong answer. J1939: 26 PGNs decoded to 152 parameters by their SAE J1939-71 bit positions, 30 more named, the preferred source-address table, and the J1939 error and not-available ranges (`core/j1939_db.py`). NMEA 2000: every standard PGN canboat defines, 216 of them, from a table distilled from [canboat](https://github.com/canboat/canboat) (`core/n2k_db.py`), with the hand-written decoders taking precedence. |
 | Counter and checksum detection | `core/counter_checksum_detector.py` | Sweeps every message. Counters are whole-byte or per-nibble, with the modulus read from the values seen and reported only when a roll-over was actually observed. |
 | Checksum algorithm guesser | `core/checksum_guesser.py` | Takes one message and one byte and scores all twelve algorithms, fitting on the first 70% of the capture and validating on the rest. Reports both numbers. |
 | Byte role classifier | `core/signal_classifier.py` | COUNTER, CHECKSUM, BOOLEAN, PHYSICAL or PADDING per byte. |
@@ -478,11 +478,11 @@ period, so keep the window under half of it.
 
 | Protocol | Module | Notes |
 |---|---|---|
-| ISO-TP (ISO 15765-2) | `core/isotp.py` | Single and multi-frame transmit with the flow-control handshake and STmin, reassembly, CAN FD escape frames, functional addressing. |
+| ISO-TP (ISO 15765-2) | `core/isotp.py` | Single and multi-frame transmit with the flow-control handshake and STmin, reassembly, CAN FD escape frames, functional addressing. Tested against can-isotp, an independent implementation. |
 | UDS (ISO 14229) | `core/uds.py` | Read DTCs, read ECU identification, service scan (read-only by default), NRC 0x78 response-pending handling, periodic TesterPresent during long scans. |
 | Security access | `core/security_access.py` | Seed and key algorithms, scripted key functions, rate-limited brute force that stops on the ECU's attempt-limit response. |
-| J1939 | `core/j1939.py` | PGN decoding plus DM1 active-DTC decode (SPN, FMI, CM, OC). |
-| OBD-II (SAE J1979) | `core/obd2_pids.py` | PID table and supported-PID discovery across continuation windows. |
+| J1939 | `core/j1939.py`, `core/j1939_db.py` | PGN and SPN decoding by the J1939-71 layouts, DM1 active-DTC decode (SPN, FMI, CM, OC), and transport-protocol sessions observed without taking part. |
+| OBD-II (SAE J1979) | `core/obd2_pids.py` | 78 mode 01 PIDs. A scan asks which PIDs the vehicle supports and reads only those. DTCs are read with modes 03 and 07, the way every OBD-II vehicle answers, with UDS 0x19 as the fallback; a vehicle that does not answer is reported as silent, not clean. |
 | XCP over CAN | `core/xcp.py` | Read-only client (CONNECT, UPLOAD, SHORT_UPLOAD) and a measurement poller. No memory-write or programming commands are implemented. |
 | DoIP (ISO 13400) | `core/doip.py` | Vehicle discovery, routing activation, UDS over IP, on stdlib sockets. |
 
@@ -717,12 +717,12 @@ The unit suite uses fixtures and a generated sample. Separately, the whole
 application is run end to end over real vehicle recordings, because synthetic
 data agrees with whatever the code assumes.
 
-Two corpora, 93 checks:
+Two corpora and two public car logs, 99 checks:
 
 | Corpus | What it is | Checks |
 |---|---|---|
 | SavvyCAN examples | 12,974 frames, 180 IDs, 11-bit, one bus | 36 |
-| CANedge recordings and python-can format files | 2 to 154,896 frames, native MDF4, 11-bit and 29-bit, dual-bus, CAN FD and error frames | 57 |
+| CANedge recordings, python-can format files and comma.ai logs | 2 to 154,896 frames, native MDF4, 11-bit and 29-bit, dual-bus, CAN FD and error frames, and two Toyota RAV4 drives from comma.ai | 63 |
 
 The second corpus is other people's hardware output, none of it produced here:
 five CANedge logger recordings in native MDF4 from
@@ -748,7 +748,7 @@ the identifier. All are fixed and pinned by tests.
 A third run pushes the size instead of the variety. It merges the 145,534-frame
 J1939 truck log and the 154,896-frame two-channel car log into one 300,430-frame
 capture carrying 11-bit and 29-bit identifiers on three bus tags, then times
-every stage against a budget: 30 checks, all passing.
+every stage against a budget: 33 checks, all passing.
 
 | | |
 |---|---|
@@ -765,6 +765,28 @@ agreeing with itself. The run found a real defect too. The PGN scan crashed on
 any log containing an active fault code, because those decode to lamps and a
 list rather than to a value and a unit, and this truck sends 196 of them.
 
+An audit against SAE J1939-71 then found that the J1939 table itself was
+wrong in places that produce plausible numbers: coolant temperature read from
+half of the crankcase pressure, three parameter groups filed under the wrong
+PGN, battery voltage read from the current bytes, every switch decoded as a
+whole byte. The corrected layouts are checked on the truck by readings that
+must agree: the brakes' front axle speed and the engine's wheel-based speed
+differ by 0.33 km/h over 1,957 pairs, absolute inlet pressure minus boost is
+the barometer, two distance counters of different resolution agree within the
+coarser one's 125 m step, and lifetime distance over lifetime fuel is the ECU's
+own reported economy to within 0.5%.
+
+The reference calibrator is checked on two real cars from comma.ai (MIT): the
+comma2k19 example segment and a 2021 drive from openpilot's public CI routes,
+both a Toyota RAV4 with a GPS receiver. From the GPS speed alone it finds the
+vehicle speed in 0x0B4 bytes 5-6 in both, and on the comma2k19 segment all four
+wheel speeds in 0x0AA, returned at 0.01 km/h per bit, the figure in
+openpilot's DBC. Given the comma2k19 reference stamped in UTC against a capture
+on its own clock, it places the reference within 0.17 s of the true offset, and
+that residual is the receiver's latency: 0.16 s by a cross-correlation that
+does not use CanLab. In both drives the signal it writes decodes the car within
+0.9% of openpilot's own decode.
+
 The same recordings check the newer analysis. Multi-frame reassembly rebuilds
 the marine log's 60 GNSS fixes and 60 satellite lists with nothing dropped,
 and the truck's 85 BAM broadcasts, in under 0.05 s; the position agrees with
@@ -794,10 +816,17 @@ A recording of the run is
 
 ```bash
 pip install -e ".[dev]"
-QT_QPA_PLATFORM=offscreen python -m pytest -q     # 673 passed
+QT_QPA_PLATFORM=offscreen python -m pytest -q     # 753 passed
 ruff check canlab tests
 ```
 
+ISO-TP, UDS and J1939 transport are also tested against implementations this
+project did not write (`tests/test_interop.py`): can-isotp as the ECU,
+udsoncan's encoding and DTC parsing, and two can-j1939 nodes holding a real
+RTS/CTS session while CanLab listens. That found an ISO-TP timing defect a
+home-grown responder could not have: the first frame after each flow control
+was sent without the separation time the receiver asked for.
+
 The suite covers the log parsers against fixtures in the genuine formats; DBC
 encode and decode round trips through cantools (little-endian, big-endian,
 signed, extended IDs, multiplexing, value tables); the ARXML and Lua exports
@@ -850,8 +879,8 @@ batch stays flat as the capture grows. Memory is bounded by the ring buffer cap.
   Verify every result before trusting it.
 - The analysis suggests candidates. A confidence figure is a match fraction over
   the frames you loaded, not a statistical proof.
-- **openpilot rlog import** needs pycapnp plus the cereal schema. Without them
-  it raises rather than producing data.
+- **openpilot log import** needs pycapnp (`pip install canlab[openpilot]`); a
+  zstd-compressed log also needs `zstandard`. The schema ships with CanLab.
 - **MDF4** import needs `asammdf` (`pip install canlab[mdf]`).
 - CAN FD is parsed, stored, decoded, injected and replayed end to end, and the
   bit grid follows the message length. It has been tested on a virtual bus,
@@ -865,15 +894,24 @@ batch stays flat as the capture grows. Memory is bounded by the ring buffer cap.
 - The MCP server in the window has no authentication unless you set a token,
   and ChatGPT's connectors cannot send one. Keep it on loopback unless you
   accept that.
-- Adapter detection was verified with the virtual backend and with stand-ins
-  for the USB, serial and sysfs probes; no physical adapter was attached during
-  development.
+- One physical adapter has been used: a CANalyst-II on a Tata Tigor EV, which
+  was detected and captured 111,006 frames. It has no listen-only mode. Every
+  other adapter was verified only with the virtual backend and stand-ins for
+  the USB, serial and sysfs probes.
+- The UDS, ISO-TP, security-access and OBD-II requests are tested against
+  scripted responders and independent implementations, not against a real ECU.
+  No real OBD-II capture was available, so the PID formulas are checked against
+  SAE J1979's worked values.
 - The GVRET backend is written to the protocol in SavvyCAN's source and tested
   against byte streams built to that format, including a scripted board behind
   the bus object. It has not been run against a physical GVRET board.
-- J1939 RTS/CTS reassembly is observe-only and, because no recording in the
-  corpus contains an RTS/CTS session, tested against synthetic frames. BAM and
-  NMEA 2000 fast packets are tested on real recordings.
+- J1939 RTS/CTS reassembly is observe-only. No recording in the corpus contains
+  an RTS/CTS session, so it is tested against synthetic frames and against two
+  can-j1939 nodes on a virtual bus. BAM and NMEA 2000 fast packets are tested
+  on real recordings.
+- J1939 decoding covers 26 parameter groups. SAE sells the full list, and a PGN
+  that is not in the table is named when it is known and otherwise shown by
+  number, never guessed.
 - The capture kit has been run on python-can's virtual backend and a fake bus,
   not in a vehicle. The systemd unit is a starting point.
 - The reference calibrator's lag search is ambiguous for a periodic reference
@@ -912,6 +950,16 @@ from [CANboat](https://github.com/canboat/canboat) (Apache License 2.0, Kees
 Verruijt) by `tools/build_n2k_table.py`; the licence and the changes made are
 in `canlab/core/data/CANBOAT-NOTICE.txt`.
 
+openpilot logs are read with comma.ai's cereal schema
+([openpilot](https://github.com/commaai/openpilot) and
+[opendbc](https://github.com/commaai/opendbc), MIT), vendored with its notice
+under `canlab/core/data/cereal/`. The real-car calibration checks use comma.ai's
+[comma2k19](https://github.com/commaai/comma2k19) example segment (MIT) and a
+drive from openpilot's public CI routes. The interoperability tests run against
+[can-isotp](https://github.com/pylessard/python-can-isotp),
+[udsoncan](https://github.com/pylessard/python-udsoncan) and
+[can-j1939](https://github.com/juergenH87/python-can-j1939) (all MIT).
+
 The SNIFFER tab and its notch, the capture splitter and the GVRET protocol
 follow [SavvyCAN](https://github.com/collin80/SavvyCAN) (MIT), whose sniffer
 window and Bisector are the originals and whose source documents the GVRET

docs/diagnostics.html

@@ -40,6 +40,10 @@ <h2 id="iso-tp">ISO-TP</h2>
    time, and reassembly of responses.</p>
 <p>You rarely touch this directly, but it is the layer everything else rides
    on, so when a scan returns nothing this is often where the problem is.</p>
+<p>It is tested against can-isotp, an independent implementation, in both
+   directions. That test found that the first consecutive frame after each flow
+   control went out without the separation time the receiver had asked for;
+   every consecutive frame is now timed against the one before it.</p>
 
 <h2 id="uds">UDS</h2>
 <p><code>core/uds.py</code> implements ISO 14229 requests: read diagnostic
@@ -69,22 +73,35 @@ <h2 id="security-access">Security access</h2>
 </div>
 
 <h2 id="obd-ii">OBD-II</h2>
-<p><code>core/obd2_pids.py</code> holds the canonical 26-PID table with
-   correct one- and two-byte decoders. Supported-PID discovery walks the
-   continuation windows rather than assuming the first 32, so PIDs above 0x20
-   are found.</p>
+<p><code>core/obd2_pids.py</code> decodes 78 mode 01 PIDs with the SAE J1979
+   formulas. A scan first asks the vehicle which PIDs it supports, walking the
+   continuation windows rather than assuming the first 32, and then reads only
+   those. Trouble codes are read with modes 03 (stored) and 07 (pending), which
+   every OBD-II vehicle answers, and UDS service 0x19 only if they get no reply.
+   A vehicle that answers nothing is reported as silent, not as clean.</p>
 <p>This is the one protocol where you can expect an answer from any compliant
    vehicle without knowing anything about it, which makes it a good first test
    that your interface and wiring work at all.</p>
 
 <h2 id="j1939-and-nmea-2000">J1939 and NMEA 2000</h2>
 <p><code>core/j1939.py</code> decodes parameter group numbers for heavy
-   vehicles, and decodes DM1 active diagnostic trouble codes into SPN, FMI, CM
-   and OC fields.</p>
+   vehicles by the SAE J1939-71 bit layouts in <code>core/j1939_db.py</code>:
+   26 PGNs and 152 parameters, 30 more PGNs named, the preferred source-address
+   table, two-bit switch states, and the error and not-available ranges, which
+   are never shown as readings. DM1 active trouble codes decode into SPN, FMI,
+   CM and OC fields.</p>
+<p>An audit of the earlier table found values read from the wrong bytes and
+   the wrong messages, among them coolant temperature from half of the
+   crankcase pressure. On the real truck log the corrected layouts agree with
+   each other: the brakes' and the engine's road speeds differ by 0.33 km/h,
+   absolute inlet pressure minus boost is the barometer, and lifetime distance
+   over fuel matches the ECU's own economy.</p>
 <p>Marine NMEA 2000 uses the same 29-bit frame, so the data page decides which
-   table applies. Single-frame NMEA 2000 PGNs such as vessel heading, rate of
-   turn, rapid position, course and speed, wind and temperature are decoded.
-   Every layout is checked in the tests against frames from a real recording.</p>
+   table applies. Hand-written decoders, checked against frames from a real
+   recording, cover heading, rate of turn, position, course and speed, wind,
+   temperature and the GNSS fix; every other standard PGN, 216 in all, is
+   decoded from a table distilled from canboat (Apache 2.0). Where both exist
+   they agree on every value on the real recording.</p>
 <p>Messages that span several frames are reassembled by
    <code>core/multiframe.py</code> before decoding: J1939 transport protocol,
    both BAM broadcasts and RTS/CTS sessions between two other nodes (observed
@@ -95,8 +112,10 @@ <h2 id="j1939-and-nmea-2000">J1939 and NMEA 2000</h2>
    of 135 bytes; on the truck log, 85 BAM broadcasts of engine and retarder
    configuration with nothing dropped. The PGN scan in INTELLIGENCE shows the
    reassembled messages; <code>list_pgns</code> and
-   <code>list_transport_messages</code> serve them over MCP. RTS/CTS is tested
-   against synthetic frames, because no recording in the corpus has one.</p>
+   <code>list_transport_messages</code> serve them over MCP. No recording in
+   the corpus has an RTS/CTS session, so that path is tested against synthetic
+   frames and against two can-j1939 nodes holding a real one on a virtual
+   bus.</p>
 
 <h2 id="bus-load-and-health">Bus load and health</h2>
 <p>Two monitor sub-tabs. Load shows utilisation over time. Health tracks error

docs/install.html

@@ -60,7 +60,7 @@ <h2 id="prebuilt-linux-binary">Prebuilt Linux binary</h2>
 <h2 id="optional-pieces">Optional pieces</h2>
 <p>The application works fully offline with none of these. Each unlocks one
    feature and is inert until you use it.</p>
-<div class="table-scroll"><table><thead><tr><th>What</th><th>Install</th><th>Needed for</th></tr></thead><tbody><tr><td>AI providers</td><td><code>pip install anthropic groq</code></td><td>The AI ENGINE tab. Or run a local Ollama server, which needs no package and no key.</td></tr><tr><td>MDF4 logs</td><td><code>pip install asammdf</code></td><td>Opening <code>.mf4</code> and <code>.mdf</code> captures from CANedge and similar loggers.</td></tr><tr><td>openpilot logs</td><td><code>pip install pycapnp</code> plus the cereal <code>log.capnp</code> schema</td><td>Opening <code>.rlog</code> and <code>.qlog</code>. Without both it raises a clear error rather than guessing.</td></tr><tr><td>Vision OCR</td><td><code>pip install opencv-python rapidocr onnxruntime</code></td><td>Reading a reference value off a dashboard video for calibration. These are large; skip unless you need it.</td></tr><tr><td>MCP server</td><td><code>pip install mcp</code></td><td>Exposing the analysis as tools to an MCP client.</td></tr><tr><td>Panda</td><td><code>pip install pandacan</code></td><td>Using a comma.ai Panda as the interface.</td></tr></tbody></table></div>
+<div class="table-scroll"><table><thead><tr><th>What</th><th>Install</th><th>Needed for</th></tr></thead><tbody><tr><td>AI providers</td><td><code>pip install anthropic groq</code></td><td>The AI ENGINE tab. Or run a local Ollama server, which needs no package and no key.</td></tr><tr><td>MDF4 logs</td><td><code>pip install asammdf</code></td><td>Opening <code>.mf4</code> and <code>.mdf</code> captures from CANedge and similar loggers.</td></tr><tr><td>openpilot logs</td><td><code>pip install canlab[openpilot]</code></td><td>Opening openpilot's <code>rlog</code> and <code>qlog</code>, plain or compressed. The cereal schema ships with CanLab. A <code>.zst</code> log also needs <code>zstandard</code>.</td></tr><tr><td>MCP server</td><td><code>pip install mcp</code></td><td>Exposing the analysis as tools to an MCP client.</td></tr><tr><td>Panda</td><td><code>pip install pandacan</code></td><td>Using a comma.ai Panda as the interface.</td></tr></tbody></table></div>
 
 <h2 id="api-keys">API keys</h2>
 <p>Keys for the AI providers go in <strong>Settings &rarr; API KEYS</strong>