From ab6a02d28506ca6f4442f3f5818bf540babb287a Mon Sep 17 00:00:00 2001 From: Sherin Joseph Roy Date: Thu, 24 Sep 2026 01:03:52 +0530 Subject: [PATCH] docs: the audit, the coverage, and the real-data checks behind them README and site: J1939 now 26 PGNs and 152 parameters by the J1939-71 layouts, with the audit that found the old table's wrong bytes and wrong messages and the truck-log cross-checks that confirm the new one; NMEA 2000 now every standard PGN canboat defines (216); OBD-II 78 PIDs, modes 03 and 07 for DTCs, silence reported as silence; ISO-TP tested against can-isotp and the STmin defect that found; openpilot logs open with the bundled schema; the calibrator checked on two real cars from comma.ai. Counts from the runs: 753 unit tests, 99 real-data checks in the two corpora, 33 in the stress run. Limitations updated honestly: one physical adapter has now been used (a CANalyst-II on a Tata Tigor EV, 111,006 frames); UDS, ISO-TP, security access and OBD-II are tested against scripted responders and independent implementations, not a real ECU; J1939 coverage is 26 groups. Removed a false claim: the site's install page and limitations offered a "Vision OCR" option for reading a value off dashboard video. No such code exists. --- README.md | 82 +++++++++++++++++++++++++++++++-------- docs/diagnostics.html | 41 ++++++++++++++------ docs/install.html | 2 +- docs/reference.html | 23 +++++++---- docs/site_src/pages.py | 81 +++++++++++++++++++++++++------------- tests/real_data/README.md | 33 ++++++++++++++-- 6 files changed, 195 insertions(+), 67 deletions(-) diff --git a/README.md b/README.md index 1ca9a20..a6c7691 100644 --- a/README.md +++ b/README.md @@ -321,7 +321,7 @@ that fits it and report the scale and offset. | Vector BLF | via python-can `BLFReader`. | | Vector ASC | via python-can `ASCReader`. | | MDF4 `.mf4` / `.mdf` | CANedge and similar. Needs `pip install canlab[mdf]`. | -| openpilot `.rlog` / `.qlog` | Needs pycapnp and the cereal `log.capnp` schema. Raises a clear error if either is missing. | +| openpilot `rlog` / `qlog` | Plain, `.bz2` or `.zst`, named the way openpilot names them. Needs `pip install canlab[openpilot]` (pycapnp); the cereal schema ships with CanLab. Frames the panda transmitted are kept apart from the car's traffic, and the log's GPS can be used as a calibration reference. | Every parser produces the same columns: `Timestamp, ID, Bus, DLC, Extended, B0..B7`, widened to `B63` when FD frames are present, plus a per-ID `Delta`. The @@ -337,7 +337,7 @@ None of this sends anything anywhere. | Feature | Module | Notes | |---|---|---| | Checksum algorithms | `core/checksums.py` | Parametrised CRC-8 plus OEM variants (Hyundai, Toyota, Honda, Subaru, AUTOSAR), checked against published check values and against commaai/opendbc. | -| J1939 and NMEA 2000 | `core/j1939.py` | Both protocols share the 29-bit frame and split the identifier the same way, so the data page decides which PGN table applies: J1939 PGNs and SPNs, or NMEA 2000's own range with radians, metres per second and kelvin. Multi-frame PGNs are reassembled first (see below) and decoded whole; one frame of one is never decoded alone, because that gives a confident wrong answer. | +| J1939 and NMEA 2000 | `core/j1939.py` | Both protocols share the 29-bit frame and split the identifier the same way, so the data page decides which PGN table applies: J1939 PGNs and SPNs, or NMEA 2000's own range with radians, metres per second and kelvin. Multi-frame PGNs are reassembled first (see below) and decoded whole; one frame of one is never decoded alone, because that gives a confident wrong answer. J1939: 26 PGNs decoded to 152 parameters by their SAE J1939-71 bit positions, 30 more named, the preferred source-address table, and the J1939 error and not-available ranges (`core/j1939_db.py`). NMEA 2000: every standard PGN canboat defines, 216 of them, from a table distilled from [canboat](https://github.com/canboat/canboat) (`core/n2k_db.py`), with the hand-written decoders taking precedence. | | Counter and checksum detection | `core/counter_checksum_detector.py` | Sweeps every message. Counters are whole-byte or per-nibble, with the modulus read from the values seen and reported only when a roll-over was actually observed. | | Checksum algorithm guesser | `core/checksum_guesser.py` | Takes one message and one byte and scores all twelve algorithms, fitting on the first 70% of the capture and validating on the rest. Reports both numbers. | | Byte role classifier | `core/signal_classifier.py` | COUNTER, CHECKSUM, BOOLEAN, PHYSICAL or PADDING per byte. | @@ -478,11 +478,11 @@ period, so keep the window under half of it. | Protocol | Module | Notes | |---|---|---| -| ISO-TP (ISO 15765-2) | `core/isotp.py` | Single and multi-frame transmit with the flow-control handshake and STmin, reassembly, CAN FD escape frames, functional addressing. | +| ISO-TP (ISO 15765-2) | `core/isotp.py` | Single and multi-frame transmit with the flow-control handshake and STmin, reassembly, CAN FD escape frames, functional addressing. Tested against can-isotp, an independent implementation. | | UDS (ISO 14229) | `core/uds.py` | Read DTCs, read ECU identification, service scan (read-only by default), NRC 0x78 response-pending handling, periodic TesterPresent during long scans. | | Security access | `core/security_access.py` | Seed and key algorithms, scripted key functions, rate-limited brute force that stops on the ECU's attempt-limit response. | -| J1939 | `core/j1939.py` | PGN decoding plus DM1 active-DTC decode (SPN, FMI, CM, OC). | -| OBD-II (SAE J1979) | `core/obd2_pids.py` | PID table and supported-PID discovery across continuation windows. | +| J1939 | `core/j1939.py`, `core/j1939_db.py` | PGN and SPN decoding by the J1939-71 layouts, DM1 active-DTC decode (SPN, FMI, CM, OC), and transport-protocol sessions observed without taking part. | +| OBD-II (SAE J1979) | `core/obd2_pids.py` | 78 mode 01 PIDs. A scan asks which PIDs the vehicle supports and reads only those. DTCs are read with modes 03 and 07, the way every OBD-II vehicle answers, with UDS 0x19 as the fallback; a vehicle that does not answer is reported as silent, not clean. | | XCP over CAN | `core/xcp.py` | Read-only client (CONNECT, UPLOAD, SHORT_UPLOAD) and a measurement poller. No memory-write or programming commands are implemented. | | DoIP (ISO 13400) | `core/doip.py` | Vehicle discovery, routing activation, UDS over IP, on stdlib sockets. | @@ -717,12 +717,12 @@ The unit suite uses fixtures and a generated sample. Separately, the whole application is run end to end over real vehicle recordings, because synthetic data agrees with whatever the code assumes. -Two corpora, 93 checks: +Two corpora and two public car logs, 99 checks: | Corpus | What it is | Checks | |---|---|---| | SavvyCAN examples | 12,974 frames, 180 IDs, 11-bit, one bus | 36 | -| CANedge recordings and python-can format files | 2 to 154,896 frames, native MDF4, 11-bit and 29-bit, dual-bus, CAN FD and error frames | 57 | +| CANedge recordings, python-can format files and comma.ai logs | 2 to 154,896 frames, native MDF4, 11-bit and 29-bit, dual-bus, CAN FD and error frames, and two Toyota RAV4 drives from comma.ai | 63 | The second corpus is other people's hardware output, none of it produced here: five CANedge logger recordings in native MDF4 from @@ -748,7 +748,7 @@ the identifier. All are fixed and pinned by tests. A third run pushes the size instead of the variety. It merges the 145,534-frame J1939 truck log and the 154,896-frame two-channel car log into one 300,430-frame capture carrying 11-bit and 29-bit identifiers on three bus tags, then times -every stage against a budget: 30 checks, all passing. +every stage against a budget: 33 checks, all passing. | | | |---|---| @@ -765,6 +765,28 @@ agreeing with itself. The run found a real defect too. The PGN scan crashed on any log containing an active fault code, because those decode to lamps and a list rather than to a value and a unit, and this truck sends 196 of them. +An audit against SAE J1939-71 then found that the J1939 table itself was +wrong in places that produce plausible numbers: coolant temperature read from +half of the crankcase pressure, three parameter groups filed under the wrong +PGN, battery voltage read from the current bytes, every switch decoded as a +whole byte. The corrected layouts are checked on the truck by readings that +must agree: the brakes' front axle speed and the engine's wheel-based speed +differ by 0.33 km/h over 1,957 pairs, absolute inlet pressure minus boost is +the barometer, two distance counters of different resolution agree within the +coarser one's 125 m step, and lifetime distance over lifetime fuel is the ECU's +own reported economy to within 0.5%. + +The reference calibrator is checked on two real cars from comma.ai (MIT): the +comma2k19 example segment and a 2021 drive from openpilot's public CI routes, +both a Toyota RAV4 with a GPS receiver. From the GPS speed alone it finds the +vehicle speed in 0x0B4 bytes 5-6 in both, and on the comma2k19 segment all four +wheel speeds in 0x0AA, returned at 0.01 km/h per bit, the figure in +openpilot's DBC. Given the comma2k19 reference stamped in UTC against a capture +on its own clock, it places the reference within 0.17 s of the true offset, and +that residual is the receiver's latency: 0.16 s by a cross-correlation that +does not use CanLab. In both drives the signal it writes decodes the car within +0.9% of openpilot's own decode. + The same recordings check the newer analysis. Multi-frame reassembly rebuilds the marine log's 60 GNSS fixes and 60 satellite lists with nothing dropped, and the truck's 85 BAM broadcasts, in under 0.05 s; the position agrees with @@ -794,10 +816,17 @@ A recording of the run is ```bash pip install -e ".[dev]" -QT_QPA_PLATFORM=offscreen python -m pytest -q # 673 passed +QT_QPA_PLATFORM=offscreen python -m pytest -q # 753 passed ruff check canlab tests ``` +ISO-TP, UDS and J1939 transport are also tested against implementations this +project did not write (`tests/test_interop.py`): can-isotp as the ECU, +udsoncan's encoding and DTC parsing, and two can-j1939 nodes holding a real +RTS/CTS session while CanLab listens. That found an ISO-TP timing defect a +home-grown responder could not have: the first frame after each flow control +was sent without the separation time the receiver asked for. + The suite covers the log parsers against fixtures in the genuine formats; DBC encode and decode round trips through cantools (little-endian, big-endian, signed, extended IDs, multiplexing, value tables); the ARXML and Lua exports @@ -850,8 +879,8 @@ batch stays flat as the capture grows. Memory is bounded by the ring buffer cap. Verify every result before trusting it. - The analysis suggests candidates. A confidence figure is a match fraction over the frames you loaded, not a statistical proof. -- **openpilot rlog import** needs pycapnp plus the cereal schema. Without them - it raises rather than producing data. +- **openpilot log import** needs pycapnp (`pip install canlab[openpilot]`); a + zstd-compressed log also needs `zstandard`. The schema ships with CanLab. - **MDF4** import needs `asammdf` (`pip install canlab[mdf]`). - CAN FD is parsed, stored, decoded, injected and replayed end to end, and the bit grid follows the message length. It has been tested on a virtual bus, @@ -865,15 +894,24 @@ batch stays flat as the capture grows. Memory is bounded by the ring buffer cap. - The MCP server in the window has no authentication unless you set a token, and ChatGPT's connectors cannot send one. Keep it on loopback unless you accept that. -- Adapter detection was verified with the virtual backend and with stand-ins - for the USB, serial and sysfs probes; no physical adapter was attached during - development. +- One physical adapter has been used: a CANalyst-II on a Tata Tigor EV, which + was detected and captured 111,006 frames. It has no listen-only mode. Every + other adapter was verified only with the virtual backend and stand-ins for + the USB, serial and sysfs probes. +- The UDS, ISO-TP, security-access and OBD-II requests are tested against + scripted responders and independent implementations, not against a real ECU. + No real OBD-II capture was available, so the PID formulas are checked against + SAE J1979's worked values. - The GVRET backend is written to the protocol in SavvyCAN's source and tested against byte streams built to that format, including a scripted board behind the bus object. It has not been run against a physical GVRET board. -- J1939 RTS/CTS reassembly is observe-only and, because no recording in the - corpus contains an RTS/CTS session, tested against synthetic frames. BAM and - NMEA 2000 fast packets are tested on real recordings. +- J1939 RTS/CTS reassembly is observe-only. No recording in the corpus contains + an RTS/CTS session, so it is tested against synthetic frames and against two + can-j1939 nodes on a virtual bus. BAM and NMEA 2000 fast packets are tested + on real recordings. +- J1939 decoding covers 26 parameter groups. SAE sells the full list, and a PGN + that is not in the table is named when it is known and otherwise shown by + number, never guessed. - The capture kit has been run on python-can's virtual backend and a fake bus, not in a vehicle. The systemd unit is a starting point. - The reference calibrator's lag search is ambiguous for a periodic reference @@ -912,6 +950,16 @@ from [CANboat](https://github.com/canboat/canboat) (Apache License 2.0, Kees Verruijt) by `tools/build_n2k_table.py`; the licence and the changes made are in `canlab/core/data/CANBOAT-NOTICE.txt`. +openpilot logs are read with comma.ai's cereal schema +([openpilot](https://github.com/commaai/openpilot) and +[opendbc](https://github.com/commaai/opendbc), MIT), vendored with its notice +under `canlab/core/data/cereal/`. The real-car calibration checks use comma.ai's +[comma2k19](https://github.com/commaai/comma2k19) example segment (MIT) and a +drive from openpilot's public CI routes. The interoperability tests run against +[can-isotp](https://github.com/pylessard/python-can-isotp), +[udsoncan](https://github.com/pylessard/python-udsoncan) and +[can-j1939](https://github.com/juergenH87/python-can-j1939) (all MIT). + The SNIFFER tab and its notch, the capture splitter and the GVRET protocol follow [SavvyCAN](https://github.com/collin80/SavvyCAN) (MIT), whose sniffer window and Bisector are the originals and whose source documents the GVRET diff --git a/docs/diagnostics.html b/docs/diagnostics.html index 43198f9..1d58a1e 100644 --- a/docs/diagnostics.html +++ b/docs/diagnostics.html @@ -40,6 +40,10 @@

ISO-TP

time, and reassembly of responses.

You rarely touch this directly, but it is the layer everything else rides on, so when a scan returns nothing this is often where the problem is.

+

It is tested against can-isotp, an independent implementation, in both + directions. That test found that the first consecutive frame after each flow + control went out without the separation time the receiver had asked for; + every consecutive frame is now timed against the one before it.

UDS

core/uds.py implements ISO 14229 requests: read diagnostic @@ -69,22 +73,35 @@

Security access

OBD-II

-

core/obd2_pids.py holds the canonical 26-PID table with - correct one- and two-byte decoders. Supported-PID discovery walks the - continuation windows rather than assuming the first 32, so PIDs above 0x20 - are found.

+

core/obd2_pids.py decodes 78 mode 01 PIDs with the SAE J1979 + formulas. A scan first asks the vehicle which PIDs it supports, walking the + continuation windows rather than assuming the first 32, and then reads only + those. Trouble codes are read with modes 03 (stored) and 07 (pending), which + every OBD-II vehicle answers, and UDS service 0x19 only if they get no reply. + A vehicle that answers nothing is reported as silent, not as clean.

This is the one protocol where you can expect an answer from any compliant vehicle without knowing anything about it, which makes it a good first test that your interface and wiring work at all.

J1939 and NMEA 2000

core/j1939.py decodes parameter group numbers for heavy - vehicles, and decodes DM1 active diagnostic trouble codes into SPN, FMI, CM - and OC fields.

+ vehicles by the SAE J1939-71 bit layouts in core/j1939_db.py: + 26 PGNs and 152 parameters, 30 more PGNs named, the preferred source-address + table, two-bit switch states, and the error and not-available ranges, which + are never shown as readings. DM1 active trouble codes decode into SPN, FMI, + CM and OC fields.

+

An audit of the earlier table found values read from the wrong bytes and + the wrong messages, among them coolant temperature from half of the + crankcase pressure. On the real truck log the corrected layouts agree with + each other: the brakes' and the engine's road speeds differ by 0.33 km/h, + absolute inlet pressure minus boost is the barometer, and lifetime distance + over fuel matches the ECU's own economy.

Marine NMEA 2000 uses the same 29-bit frame, so the data page decides which - table applies. Single-frame NMEA 2000 PGNs such as vessel heading, rate of - turn, rapid position, course and speed, wind and temperature are decoded. - Every layout is checked in the tests against frames from a real recording.

+ table applies. Hand-written decoders, checked against frames from a real + recording, cover heading, rate of turn, position, course and speed, wind, + temperature and the GNSS fix; every other standard PGN, 216 in all, is + decoded from a table distilled from canboat (Apache 2.0). Where both exist + they agree on every value on the real recording.

Messages that span several frames are reassembled by core/multiframe.py before decoding: J1939 transport protocol, both BAM broadcasts and RTS/CTS sessions between two other nodes (observed @@ -95,8 +112,10 @@

J1939 and NMEA 2000

of 135 bytes; on the truck log, 85 BAM broadcasts of engine and retarder configuration with nothing dropped. The PGN scan in INTELLIGENCE shows the reassembled messages; list_pgns and - list_transport_messages serve them over MCP. RTS/CTS is tested - against synthetic frames, because no recording in the corpus has one.

+ list_transport_messages serve them over MCP. No recording in + the corpus has an RTS/CTS session, so that path is tested against synthetic + frames and against two can-j1939 nodes holding a real one on a virtual + bus.

Bus load and health

Two monitor sub-tabs. Load shows utilisation over time. Health tracks error diff --git a/docs/install.html b/docs/install.html index a88788e..7f6efbd 100644 --- a/docs/install.html +++ b/docs/install.html @@ -60,7 +60,7 @@

Prebuilt Linux binary

Optional pieces

The application works fully offline with none of these. Each unlocks one feature and is inert until you use it.

-
WhatInstallNeeded for
AI providerspip install anthropic groqThe AI ENGINE tab. Or run a local Ollama server, which needs no package and no key.
MDF4 logspip install asammdfOpening .mf4 and .mdf captures from CANedge and similar loggers.
openpilot logspip install pycapnp plus the cereal log.capnp schemaOpening .rlog and .qlog. Without both it raises a clear error rather than guessing.
Vision OCRpip install opencv-python rapidocr onnxruntimeReading a reference value off a dashboard video for calibration. These are large; skip unless you need it.
MCP serverpip install mcpExposing the analysis as tools to an MCP client.
Pandapip install pandacanUsing a comma.ai Panda as the interface.
+
WhatInstallNeeded for
AI providerspip install anthropic groqThe AI ENGINE tab. Or run a local Ollama server, which needs no package and no key.
MDF4 logspip install asammdfOpening .mf4 and .mdf captures from CANedge and similar loggers.
openpilot logspip install canlab[openpilot]Opening openpilot's rlog and qlog, plain or compressed. The cereal schema ships with CanLab. A .zst log also needs zstandard.
MCP serverpip install mcpExposing the analysis as tools to an MCP client.
Pandapip install pandacanUsing a comma.ai Panda as the interface.

API keys

Keys for the AI providers go in Settings → API KEYS diff --git a/docs/reference.html b/docs/reference.html index 76c91f8..d689297 100644 --- a/docs/reference.html +++ b/docs/reference.html @@ -28,7 +28,7 @@

Reference

The details that do not belong anywhere else.

Log formats

-
FormatNotes
SavvyCAN CSVGVRET and SavvyCAN exports. Handles both the hex byte format real exports use and older decimal ones, and the trailing comma SavvyCAN writes after the last data byte.
candump .logcandump -l output, including CAN FD lines.
pcap / pcapngLinux SocketCAN, link type 227, via dpkt.
Vector BLFThrough python-can's reader.
Vector ASCThrough python-can's reader.
MDF4 .mf4 / .mdfCANedge and similar. Needs asammdf.
openpilot .rlog / .qlogNeeds pycapnp and the cereal schema. Raises a clear error if either is missing rather than guessing.
+
FormatNotes
SavvyCAN CSVGVRET and SavvyCAN exports. Handles both the hex byte format real exports use and older decimal ones, and the trailing comma SavvyCAN writes after the last data byte.
candump .logcandump -l output, including CAN FD lines.
pcap / pcapngLinux SocketCAN, link type 227, via dpkt.
Vector BLFThrough python-can's reader.
Vector ASCThrough python-can's reader.
MDF4 .mf4 / .mdfCANedge and similar. Needs asammdf.
openpilot rlog / qlogPlain, .bz2 or .zst. Needs canlab[openpilot]; the schema ships with CanLab. Frames the panda sent itself are kept apart from the car's traffic.

Every parser produces the same columns, so the rest of the application does not care where a capture came from: Timestamp, ID, Bus, DLC, Extended, B0..B7, plus a per-ID Delta.

@@ -80,7 +80,7 @@

Validated against real captures

Testing

The suite runs headless:

-
QT_QPA_PLATFORM=offscreen python -m pytest -q     # 673 passed
+
QT_QPA_PLATFORM=offscreen python -m pytest -q     # 753 passed

Tests that need an optional dependency skip cleanly when it is absent: the MDF4 importer without asammdf, the transport tests without the MCP SDK, the Lua dissector without a Lua runtime.

@@ -99,11 +99,14 @@

Limitations

paths it covers.
  • ARXML export is experimental and is not validated against the AUTOSAR schema.
    • -
  • openpilot rlog import needs pycapnp plus the cereal - schema; without them it raises rather than producing data.
    • -
  • MDF4 needs asammdf. Vision - OCR needs opencv, rapidocr and onnxruntime, which are heavy.
    • -
  • CAN FD parsing and decoding is partial in places.
    • +
  • openpilot logs need pycapnp + (canlab[openpilot]); the schema ships with CanLab.
    • +
  • MDF4 needs asammdf.
    • +
  • CAN FD is parsed, decoded, injected and replayed end to + end, but has been tested on a virtual bus, not on FD hardware.
    • +
  • No real ECU. UDS, ISO-TP, security access and OBD-II are + tested against scripted responders and independent implementations, not + against a vehicle's modules.
  • The gateway needs two hardware channels.
  • The prebuilt binary is Linux x86_64 and unsigned. No macOS or Windows build; run from source there.
    • @@ -121,6 +124,12 @@

    Credits

    bus reverse engineering skills (MIT). The OEM checksum algorithms in core/checksums.py follow commaai/opendbc (MIT).

    +

    NMEA 2000 definitions are distilled from + canboat (Apache 2.0). + openpilot logs are read with comma.ai's cereal schema (MIT), and the + real-car calibration checks use comma.ai's comma2k19 segment (MIT) and a + drive from openpilot's public CI routes. The interoperability tests run + against can-isotp, udsoncan and can-j1939 (all MIT).

    Built on python-can, cantools, PyQt6, pandas, NumPy and pyqtgraph.

    diff --git a/docs/site_src/pages.py b/docs/site_src/pages.py index d7e40b0..4960167 100644 --- a/docs/site_src/pages.py +++ b/docs/site_src/pages.py @@ -163,14 +163,10 @@ def build_pages(*, h2, table, video_card, parts, repo): ["MDF4 logs", "pip install asammdf", "Opening .mf4 and .mdf captures from CANedge " "and similar loggers."], - ["openpilot logs", "pip install pycapnp plus the cereal " - "log.capnp schema", - "Opening .rlog and .qlog. Without both it " - "raises a clear error rather than guessing."], - ["Vision OCR", - "pip install opencv-python rapidocr onnxruntime", - "Reading a reference value off a dashboard video for calibration. These " - "are large; skip unless you need it."], + ["openpilot logs", "pip install canlab[openpilot]", + "Opening openpilot's rlog and qlog, plain or " + "compressed. The cereal schema ships with CanLab. A " + ".zst log also needs zstandard."], ["MCP server", "pip install mcp", "Exposing the analysis as tools to an MCP client."], ["Panda", "pip install pandacan", @@ -841,6 +837,10 @@ def build_pages_4(*, h2, table, repo): time, and reassembly of responses.

    You rarely touch this directly, but it is the layer everything else rides on, so when a scan returns nothing this is often where the problem is.

    +

    It is tested against can-isotp, an independent implementation, in both + directions. That test found that the first consecutive frame after each flow + control went out without the separation time the receiver had asked for; + every consecutive frame is now timed against the one before it.

    {h2("UDS")}

    core/uds.py implements ISO 14229 requests: read diagnostic @@ -870,22 +870,35 @@ def build_pages_4(*, h2, table, repo): {h2("OBD-II")} -

    core/obd2_pids.py holds the canonical 26-PID table with - correct one- and two-byte decoders. Supported-PID discovery walks the - continuation windows rather than assuming the first 32, so PIDs above 0x20 - are found.

    +

    core/obd2_pids.py decodes 78 mode 01 PIDs with the SAE J1979 + formulas. A scan first asks the vehicle which PIDs it supports, walking the + continuation windows rather than assuming the first 32, and then reads only + those. Trouble codes are read with modes 03 (stored) and 07 (pending), which + every OBD-II vehicle answers, and UDS service 0x19 only if they get no reply. + A vehicle that answers nothing is reported as silent, not as clean.

    This is the one protocol where you can expect an answer from any compliant vehicle without knowing anything about it, which makes it a good first test that your interface and wiring work at all.

    {h2("J1939 and NMEA 2000")}

    core/j1939.py decodes parameter group numbers for heavy - vehicles, and decodes DM1 active diagnostic trouble codes into SPN, FMI, CM - and OC fields.

    + vehicles by the SAE J1939-71 bit layouts in core/j1939_db.py: + 26 PGNs and 152 parameters, 30 more PGNs named, the preferred source-address + table, two-bit switch states, and the error and not-available ranges, which + are never shown as readings. DM1 active trouble codes decode into SPN, FMI, + CM and OC fields.

    +

    An audit of the earlier table found values read from the wrong bytes and + the wrong messages, among them coolant temperature from half of the + crankcase pressure. On the real truck log the corrected layouts agree with + each other: the brakes' and the engine's road speeds differ by 0.33 km/h, + absolute inlet pressure minus boost is the barometer, and lifetime distance + over fuel matches the ECU's own economy.

    Marine NMEA 2000 uses the same 29-bit frame, so the data page decides which - table applies. Single-frame NMEA 2000 PGNs such as vessel heading, rate of - turn, rapid position, course and speed, wind and temperature are decoded. - Every layout is checked in the tests against frames from a real recording.

    + table applies. Hand-written decoders, checked against frames from a real + recording, cover heading, rate of turn, position, course and speed, wind, + temperature and the GNSS fix; every other standard PGN, 216 in all, is + decoded from a table distilled from canboat (Apache 2.0). Where both exist + they agree on every value on the real recording.

    Messages that span several frames are reassembled by core/multiframe.py before decoding: J1939 transport protocol, both BAM broadcasts and RTS/CTS sessions between two other nodes (observed @@ -896,8 +909,10 @@ def build_pages_4(*, h2, table, repo): of 135 bytes; on the truck log, 85 BAM broadcasts of engine and retarder configuration with nothing dropped. The PGN scan in INTELLIGENCE shows the reassembled messages; list_pgns and - list_transport_messages serve them over MCP. RTS/CTS is tested - against synthetic frames, because no recording in the corpus has one.

    + list_transport_messages serve them over MCP. No recording in + the corpus has an RTS/CTS session, so that path is tested against synthetic + frames and against two can-j1939 nodes holding a real one on a virtual + bus.

    {h2("Bus load and health")}

    Two monitor sub-tabs. Load shows utilisation over time. Health tracks error @@ -1176,9 +1191,10 @@ def register(app): ["Vector ASC", "Through python-can's reader."], ["MDF4 .mf4 / .mdf", "CANedge and similar. Needs asammdf."], - ["openpilot .rlog / .qlog", - "Needs pycapnp and the cereal schema. Raises a clear error if either is " - "missing rather than guessing."], + ["openpilot rlog / qlog", + "Plain, .bz2 or .zst. Needs " + "canlab[openpilot]; the schema ships with CanLab. Frames the " + "panda sent itself are kept apart from the car's traffic."], ])}

    Every parser produces the same columns, so the rest of the application does not care where a capture came from: Timestamp, ID, Bus, DLC, @@ -1247,7 +1263,7 @@ def register(app): {h2("Testing")}

    The suite runs headless:

    -
    QT_QPA_PLATFORM=offscreen python -m pytest -q     # 673 passed
    +
    QT_QPA_PLATFORM=offscreen python -m pytest -q     # 753 passed

    Tests that need an optional dependency skip cleanly when it is absent: the MDF4 importer without asammdf, the transport tests without the MCP SDK, the Lua dissector without a Lua runtime.

    @@ -1266,11 +1282,14 @@ def register(app): paths it covers.
  • ARXML export is experimental and is not validated against the AUTOSAR schema.
    • -
  • openpilot rlog import needs pycapnp plus the cereal - schema; without them it raises rather than producing data.
    • -
  • MDF4 needs asammdf. Vision - OCR needs opencv, rapidocr and onnxruntime, which are heavy.
    • -
  • CAN FD parsing and decoding is partial in places.
    • +
  • openpilot logs need pycapnp + (canlab[openpilot]); the schema ships with CanLab.
    • +
  • MDF4 needs asammdf.
    • +
  • CAN FD is parsed, decoded, injected and replayed end to + end, but has been tested on a virtual bus, not on FD hardware.
    • +
  • No real ECU. UDS, ISO-TP, security access and OBD-II are + tested against scripted responders and independent implementations, not + against a vehicle's modules.
  • The gateway needs two hardware channels.
  • The prebuilt binary is Linux x86_64 and unsigned. No macOS or Windows build; run from source there.
    • @@ -1288,6 +1307,12 @@ def register(app): bus reverse engineering skills (MIT). The OEM checksum algorithms in core/checksums.py follow commaai/opendbc (MIT).

    +

    NMEA 2000 definitions are distilled from + canboat (Apache 2.0). + openpilot logs are read with comma.ai's cereal schema (MIT), and the + real-car calibration checks use comma.ai's comma2k19 segment (MIT) and a + drive from openpilot's public CI routes. The interoperability tests run + against can-isotp, udsoncan and can-j1939 (all MIT).

    Built on python-can, cantools, PyQt6, pandas, NumPy and pyqtgraph.

    """)) pages.append(( diff --git a/tests/real_data/README.md b/tests/real_data/README.md index fce1a87..e3a88a5 100644 --- a/tests/real_data/README.md +++ b/tests/real_data/README.md @@ -177,14 +177,41 @@ batches, require every batch under 20 ms, and report the events by kind without asserting them (a per-byte baseline fitted on one minute flags the range changes of the next 22). +### J1939 values that must agree + +`phase_j1939` in the stress run checks the J1939 layouts on the truck by +readings that have to agree with each other rather than with an expected +number: the brakes' EBC2 front axle speed against the engine's CCVS +wheel-based speed (0.33 km/h apart over 1,957 pairs, r 0.9997); coolant, oil, +barometer, battery and fuel level each physical and each from the message +J1939-71 puts it in; absolute inlet pressure minus boost pressure equal to the +barometer; VD and HRVD distance within the coarser counter's 125 m step; and +lifetime distance over lifetime fuel against the ECU's own average economy. + +### Reference calibration on real cars + +`phase_reference` fetches comma.ai's comma2k19 example segment (MIT, about +6 MB) into `/comma2k19`: a minute of a Toyota RAV4's CAN bus, a +u-blox receiver, and openpilot's decoded speed. From the GNSS speed alone the +calibrator must find 0x0B4 bytes 5-6 and the four 0x0AA wheel words, return +the wheel scale as 0.01 km/h (openpilot's DBC), place a UTC-stamped reference +within 0.3 s of the true clock offset, and write a signal that decodes the car +within 2% of openpilot. + +`phase_openpilot` fetches a 2021 RAV4 drive from openpilot's public CI routes +(8 MB, `rlog.bz2`) into `/openpilot`, opens it with the bundled +schema (received frames and the panda's own transmissions must add up to +pycapnp's count), and calibrates it against its own GPS. Both phases skip, +rather than fail, when the download is not possible or pycapnp is missing. + ## What this does not cover No real hardware is involved, so none of the following is verified here: a real ECU answering a UDS scan or a security-access seed, a physical CAN adapter, the gateway (which needs two channels), CAN FD (the capture is classic CAN), the AI -providers (no key), openpilot rlog (no cereal schema), vision OCR, and the GUI -as a person drives it, since these scripts call the same slots the buttons call -rather than clicking. +providers (no key), and the GUI as a person drives it, since these scripts call +the same slots the buttons call rather than clicking. One physical adapter has +been used outside these scripts: a CANalyst-II on a Tata Tigor EV. All five parsers must agree on the same capture: 12,974 frames and 180 IDs from the CSV, the candump log, BLF, ASC and pcap alike. A disagreement means one