You rarely touch this directly, but it is the layer everything else rides on, so when a scan returns nothing this is often where the problem is.
+It is tested against can-isotp, an independent implementation, in both + directions. That test found that the first consecutive frame after each flow + control went out without the separation time the receiver had asked for; + every consecutive frame is now timed against the one before it.
core/uds.py implements ISO 14229 requests: read diagnostic
@@ -69,22 +73,35 @@
core/obd2_pids.py holds the canonical 26-PID table with
- correct one- and two-byte decoders. Supported-PID discovery walks the
- continuation windows rather than assuming the first 32, so PIDs above 0x20
- are found.
core/obd2_pids.py decodes 78 mode 01 PIDs with the SAE J1979
+ formulas. A scan first asks the vehicle which PIDs it supports, walking the
+ continuation windows rather than assuming the first 32, and then reads only
+ those. Trouble codes are read with modes 03 (stored) and 07 (pending), which
+ every OBD-II vehicle answers, and UDS service 0x19 only if they get no reply.
+ A vehicle that answers nothing is reported as silent, not as clean.
This is the one protocol where you can expect an answer from any compliant vehicle without knowing anything about it, which makes it a good first test that your interface and wiring work at all.
core/j1939.py decodes parameter group numbers for heavy
- vehicles, and decodes DM1 active diagnostic trouble codes into SPN, FMI, CM
- and OC fields.
core/j1939_db.py:
+ 26 PGNs and 152 parameters, 30 more PGNs named, the preferred source-address
+ table, two-bit switch states, and the error and not-available ranges, which
+ are never shown as readings. DM1 active trouble codes decode into SPN, FMI,
+ CM and OC fields.
+An audit of the earlier table found values read from the wrong bytes and + the wrong messages, among them coolant temperature from half of the + crankcase pressure. On the real truck log the corrected layouts agree with + each other: the brakes' and the engine's road speeds differ by 0.33 km/h, + absolute inlet pressure minus boost is the barometer, and lifetime distance + over fuel matches the ECU's own economy.
Marine NMEA 2000 uses the same 29-bit frame, so the data page decides which - table applies. Single-frame NMEA 2000 PGNs such as vessel heading, rate of - turn, rapid position, course and speed, wind and temperature are decoded. - Every layout is checked in the tests against frames from a real recording.
+ table applies. Hand-written decoders, checked against frames from a real + recording, cover heading, rate of turn, position, course and speed, wind, + temperature and the GNSS fix; every other standard PGN, 216 in all, is + decoded from a table distilled from canboat (Apache 2.0). Where both exist + they agree on every value on the real recording.Messages that span several frames are reassembled by
core/multiframe.py before decoding: J1939 transport protocol,
both BAM broadcasts and RTS/CTS sessions between two other nodes (observed
@@ -95,8 +112,10 @@
list_pgns and
- list_transport_messages serve them over MCP. RTS/CTS is tested
- against synthetic frames, because no recording in the corpus has one.
+ list_transport_messages serve them over MCP. No recording in
+ the corpus has an RTS/CTS session, so that path is tested against synthetic
+ frames and against two can-j1939 nodes holding a real one on a virtual
+ bus.
Two monitor sub-tabs. Load shows utilisation over time. Health tracks error diff --git a/docs/install.html b/docs/install.html index a88788e..7f6efbd 100644 --- a/docs/install.html +++ b/docs/install.html @@ -60,7 +60,7 @@
The application works fully offline with none of these. Each unlocks one feature and is inert until you use it.
-| What | Install | Needed for |
|---|---|---|
| AI providers | pip install anthropic groq | The AI ENGINE tab. Or run a local Ollama server, which needs no package and no key. |
| MDF4 logs | pip install asammdf | Opening .mf4 and .mdf captures from CANedge and similar loggers. |
| openpilot logs | pip install pycapnp plus the cereal log.capnp schema | Opening .rlog and .qlog. Without both it raises a clear error rather than guessing. |
| Vision OCR | pip install opencv-python rapidocr onnxruntime | Reading a reference value off a dashboard video for calibration. These are large; skip unless you need it. |
| MCP server | pip install mcp | Exposing the analysis as tools to an MCP client. |
| Panda | pip install pandacan | Using a comma.ai Panda as the interface. |
| What | Install | Needed for |
|---|---|---|
| AI providers | pip install anthropic groq | The AI ENGINE tab. Or run a local Ollama server, which needs no package and no key. |
| MDF4 logs | pip install asammdf | Opening .mf4 and .mdf captures from CANedge and similar loggers. |
| openpilot logs | pip install canlab[openpilot] | Opening openpilot's rlog and qlog, plain or compressed. The cereal schema ships with CanLab. A .zst log also needs zstandard. |
| MCP server | pip install mcp | Exposing the analysis as tools to an MCP client. |
| Panda | pip install pandacan | Using a comma.ai Panda as the interface. |
Keys for the AI providers go in Settings → API KEYS diff --git a/docs/reference.html b/docs/reference.html index 76c91f8..d689297 100644 --- a/docs/reference.html +++ b/docs/reference.html @@ -28,7 +28,7 @@
The details that do not belong anywhere else.
| Format | Notes |
|---|---|
| SavvyCAN CSV | GVRET and SavvyCAN exports. Handles both the hex byte format real exports use and older decimal ones, and the trailing comma SavvyCAN writes after the last data byte. |
candump .log | candump -l output, including CAN FD lines. |
| pcap / pcapng | Linux SocketCAN, link type 227, via dpkt. |
| Vector BLF | Through python-can's reader. |
| Vector ASC | Through python-can's reader. |
MDF4 .mf4 / .mdf | CANedge and similar. Needs asammdf. |
openpilot .rlog / .qlog | Needs pycapnp and the cereal schema. Raises a clear error if either is missing rather than guessing. |
| Format | Notes |
|---|---|
| SavvyCAN CSV | GVRET and SavvyCAN exports. Handles both the hex byte format real exports use and older decimal ones, and the trailing comma SavvyCAN writes after the last data byte. |
candump .log | candump -l output, including CAN FD lines. |
| pcap / pcapng | Linux SocketCAN, link type 227, via dpkt. |
| Vector BLF | Through python-can's reader. |
| Vector ASC | Through python-can's reader. |
MDF4 .mf4 / .mdf | CANedge and similar. Needs asammdf. |
openpilot rlog / qlog | Plain, .bz2 or .zst. Needs canlab[openpilot]; the schema ships with CanLab. Frames the panda sent itself are kept apart from the car's traffic. |
Every parser produces the same columns, so the rest of the application does
not care where a capture came from: Timestamp, ID, Bus, DLC,
Extended, B0..B7, plus a per-ID Delta.
The suite runs headless:
-QT_QPA_PLATFORM=offscreen python -m pytest -q # 673 passedQT_QPA_PLATFORM=offscreen python -m pytest -q # 753 passedTests that need an optional dependency skip cleanly when it is absent: the
MDF4 importer without asammdf, the transport tests without the
MCP SDK, the Lua dissector without a Lua runtime.
asammdf. Vision
- OCR needs opencv, rapidocr and onnxruntime, which are heavy.canlab[openpilot]); the schema ships with CanLab.asammdf.core/checksums.py follow
commaai/opendbc (MIT).
+NMEA 2000 definitions are distilled from + canboat (Apache 2.0). + openpilot logs are read with comma.ai's cereal schema (MIT), and the + real-car calibration checks use comma.ai's comma2k19 segment (MIT) and a + drive from openpilot's public CI routes. The interoperability tests run + against can-isotp, udsoncan and can-j1939 (all MIT).
Built on python-can, cantools, PyQt6, pandas, NumPy and pyqtgraph.
diff --git a/docs/site_src/pages.py b/docs/site_src/pages.py index d7e40b0..4960167 100644 --- a/docs/site_src/pages.py +++ b/docs/site_src/pages.py @@ -163,14 +163,10 @@ def build_pages(*, h2, table, video_card, parts, repo): ["MDF4 logs", "pip install asammdf",
"Opening .mf4 and .mdf captures from CANedge "
"and similar loggers."],
- ["openpilot logs", "pip install pycapnp plus the cereal "
- "log.capnp schema",
- "Opening .rlog and .qlog. Without both it "
- "raises a clear error rather than guessing."],
- ["Vision OCR",
- "pip install opencv-python rapidocr onnxruntime",
- "Reading a reference value off a dashboard video for calibration. These "
- "are large; skip unless you need it."],
+ ["openpilot logs", "pip install canlab[openpilot]",
+ "Opening openpilot's rlog and qlog, plain or "
+ "compressed. The cereal schema ships with CanLab. A "
+ ".zst log also needs zstandard."],
["MCP server", "pip install mcp",
"Exposing the analysis as tools to an MCP client."],
["Panda", "pip install pandacan",
@@ -841,6 +837,10 @@ def build_pages_4(*, h2, table, repo):
time, and reassembly of responses.
You rarely touch this directly, but it is the layer everything else rides on, so when a scan returns nothing this is often where the problem is.
+It is tested against can-isotp, an independent implementation, in both + directions. That test found that the first consecutive frame after each flow + control went out without the separation time the receiver had asked for; + every consecutive frame is now timed against the one before it.
{h2("UDS")}core/uds.py implements ISO 14229 requests: read diagnostic
@@ -870,22 +870,35 @@ def build_pages_4(*, h2, table, repo):
{h2("OBD-II")}
-
core/obd2_pids.py holds the canonical 26-PID table with
- correct one- and two-byte decoders. Supported-PID discovery walks the
- continuation windows rather than assuming the first 32, so PIDs above 0x20
- are found.
core/obd2_pids.py decodes 78 mode 01 PIDs with the SAE J1979
+ formulas. A scan first asks the vehicle which PIDs it supports, walking the
+ continuation windows rather than assuming the first 32, and then reads only
+ those. Trouble codes are read with modes 03 (stored) and 07 (pending), which
+ every OBD-II vehicle answers, and UDS service 0x19 only if they get no reply.
+ A vehicle that answers nothing is reported as silent, not as clean.
This is the one protocol where you can expect an answer from any compliant vehicle without knowing anything about it, which makes it a good first test that your interface and wiring work at all.
{h2("J1939 and NMEA 2000")}core/j1939.py decodes parameter group numbers for heavy
- vehicles, and decodes DM1 active diagnostic trouble codes into SPN, FMI, CM
- and OC fields.
core/j1939_db.py:
+ 26 PGNs and 152 parameters, 30 more PGNs named, the preferred source-address
+ table, two-bit switch states, and the error and not-available ranges, which
+ are never shown as readings. DM1 active trouble codes decode into SPN, FMI,
+ CM and OC fields.
+An audit of the earlier table found values read from the wrong bytes and + the wrong messages, among them coolant temperature from half of the + crankcase pressure. On the real truck log the corrected layouts agree with + each other: the brakes' and the engine's road speeds differ by 0.33 km/h, + absolute inlet pressure minus boost is the barometer, and lifetime distance + over fuel matches the ECU's own economy.
Marine NMEA 2000 uses the same 29-bit frame, so the data page decides which - table applies. Single-frame NMEA 2000 PGNs such as vessel heading, rate of - turn, rapid position, course and speed, wind and temperature are decoded. - Every layout is checked in the tests against frames from a real recording.
+ table applies. Hand-written decoders, checked against frames from a real + recording, cover heading, rate of turn, position, course and speed, wind, + temperature and the GNSS fix; every other standard PGN, 216 in all, is + decoded from a table distilled from canboat (Apache 2.0). Where both exist + they agree on every value on the real recording.Messages that span several frames are reassembled by
core/multiframe.py before decoding: J1939 transport protocol,
both BAM broadcasts and RTS/CTS sessions between two other nodes (observed
@@ -896,8 +909,10 @@ def build_pages_4(*, h2, table, repo):
of 135 bytes; on the truck log, 85 BAM broadcasts of engine and retarder
configuration with nothing dropped. The PGN scan in INTELLIGENCE shows the
reassembled messages; list_pgns and
- list_transport_messages serve them over MCP. RTS/CTS is tested
- against synthetic frames, because no recording in the corpus has one.
list_transport_messages serve them over MCP. No recording in
+ the corpus has an RTS/CTS session, so that path is tested against synthetic
+ frames and against two can-j1939 nodes holding a real one on a virtual
+ bus.
{h2("Bus load and health")}
Two monitor sub-tabs. Load shows utilisation over time. Health tracks error
@@ -1176,9 +1191,10 @@ def register(app):
["Vector ASC", "Through python-can's reader."],
["MDF4 .mf4 / .mdf",
"CANedge and similar. Needs asammdf."],
- ["openpilot .rlog / .qlog",
- "Needs pycapnp and the cereal schema. Raises a clear error if either is "
- "missing rather than guessing."],
+ ["openpilot rlog / qlog",
+ "Plain, .bz2 or .zst. Needs "
+ "canlab[openpilot]; the schema ships with CanLab. Frames the "
+ "panda sent itself are kept apart from the car's traffic."],
])}
Every parser produces the same columns, so the rest of the application does
not care where a capture came from: The suite runs headless: Tests that need an optional dependency skip cleanly when it is absent: the
MDF4 importer without Timestamp, ID, Bus, DLC,
@@ -1247,7 +1263,7 @@ def register(app):
{h2("Testing")}
+QT_QPA_PLATFORM=offscreen python -m pytest -q # 673 passedQT_QPA_PLATFORM=offscreen python -m pytest -q # 753 passedasammdf, the transport tests without the
MCP SDK, the Lua dissector without a Lua runtime.asammdf. Vision
- OCR needs opencv, rapidocr and onnxruntime, which are heavy.canlab[openpilot]); the schema ships with CanLab.asammdf.core/checksums.py follow
commaai/opendbc (MIT).
NMEA 2000 definitions are distilled from + canboat (Apache 2.0). + openpilot logs are read with comma.ai's cereal schema (MIT), and the + real-car calibration checks use comma.ai's comma2k19 segment (MIT) and a + drive from openpilot's public CI routes. The interoperability tests run + against can-isotp, udsoncan and can-j1939 (all MIT).
Built on python-can, cantools, PyQt6, pandas, NumPy and pyqtgraph.
""")) pages.append(( diff --git a/tests/real_data/README.md b/tests/real_data/README.md index fce1a87..e3a88a5 100644 --- a/tests/real_data/README.md +++ b/tests/real_data/README.md @@ -177,14 +177,41 @@ batches, require every batch under 20 ms, and report the events by kind without asserting them (a per-byte baseline fitted on one minute flags the range changes of the next 22). +### J1939 values that must agree + +`phase_j1939` in the stress run checks the J1939 layouts on the truck by +readings that have to agree with each other rather than with an expected +number: the brakes' EBC2 front axle speed against the engine's CCVS +wheel-based speed (0.33 km/h apart over 1,957 pairs, r 0.9997); coolant, oil, +barometer, battery and fuel level each physical and each from the message +J1939-71 puts it in; absolute inlet pressure minus boost pressure equal to the +barometer; VD and HRVD distance within the coarser counter's 125 m step; and +lifetime distance over lifetime fuel against the ECU's own average economy. + +### Reference calibration on real cars + +`phase_reference` fetches comma.ai's comma2k19 example segment (MIT, about +6 MB) into `