Added Logging and Enhancements in Qos test by litincandela · Pull Request #319 · greearb/lanforge-scripts

litincandela · 2026-07-29T11:06:11Z

This PR improves the stability of robot monitoring and handling of CX/device failures during test execution in QOS Test. It also reduces unnecessary log spam and makes reports more useful for debugging.

Changes

Added a clients_issue.csv report to track issues seen for each device during monitoring.
Logged the actual monitoring time in the report instead of only the configured duration.
Improved monitoring to detect when a CX or device Wi-Fi port disappears, log the warning only once, and log when it recovers.
Added a 40-second retry when all devices stop responding during a test. If they recover, the test continues with the data collected so far.
Added a one-time warning when a CX is not in the Run state (after a 10-second grace period) and log when it starts running again.
Removed repeated "CX Not Found" messages by logging them only once per occurrence.
Stop the entire robot test if devices do not recover within the retry period, instead of continuing with invalid monitoring data.
Prevent report generation failures by safely handling coordinates or rotations that were never reached.
Updated the WebUI navigation status when all devices become unresponsive so the UI doesn't remain stuck waiting for the next step.
Improved missing-CX logs by recording the API endpoint and response keys instead of dumping the full response.
Fixed the not-running grace period so it resets for every coordinate/rotation in robot tests, while band steering continues to use a single grace period for the entire test.

Below is the csv for the clients issues:
clients_issue.csv

Below is one of the logs where we got empty response from the api and how we are handling it:

Logs:

2026-07-29 16:22:41,087 __main__ WARNING: CX '1.179MacMacs-Air.ctinb._TCP_DL_BK-22' is missing from the monitoring data, the device may have disconnected or its connection was not created. Continuing the test with the remaining devices.
URL     : /endp/list?fields=rx rate (last),rx drop %25,name
Response keys: []
2026-07-29 16:22:41,087 __main__ WARNING: CX '1.13androidvivov15_TCP_DL_BK-23' is missing from the monitoring data, the device may have disconnected or its connection was not created. Continuing the test with the remaining devices.
URL     : /endp/list?fields=rx rate (last),rx drop %25,name
Response keys: []
**2026-07-29 16:22:41,087 __main__ WARNING: All devices have stopped responding during monitoring, retrying for up to 40 seconds before ending the monitor loop.
2026-07-29 16:22:46,273 __main__ INFO: Device(s) responded again after 5s, resuming.**
2026-07-29 16:22:46,443 __main__ WARNING: 1.154Macmackbook-air-02_TCP_DL_VO-3-A/B : Endpoint Not Found
Endpoint keys: ['1.86Linlin-16_TCP_DL_VO-0-B', '1.134WinDesktopLatitude_TCP_DL_VO-1-B', '1.146Lintest99-Latitude_TCP_DL_VO-2-B', '1.154Macmackbooir02sAir_TCP_DL_VO-3-B', '1.179MacMacs-Air.ctinb._TCP_DL_VO-4-B', '1.13androidvivov15_TCP_DL_VO-5-B', '1.86Linlin-16_TCP_DL_VI-6-B', '1.146Lintest99-Latitude_TCP_DL_VI-8-B', '1.154Macmackbooir02sAir_TCP_DL_VI-9-B', '1.179MacMacs-Air.ctinb._TCP_DL_VI-10-B', '1.86Linlin-16_TCP_DL_BE-12-B', '1.146Lintest99-Latitude_TCP_DL_BE-14-B', '1.154Macmackbooir02sAir_TCP_DL_BE-15-B', '1.179MacMacs-Air.ctinb._TCP_DL_BE-16-B', '1.86Linlin-16_TCP_DL_BK-18-B', '1.134WinDesktopLatitude_TCP_DL_BK-19-B', '1.146Lintest99-Latitude_TCP_DL_BK-20-B', '1.154Macmackbooir02sAir_TCP_DL_BK-21-B', '1.179MacMacs-Air.ctinb._TCP_DL_BK-22-B', '1.154Macmackbook-air-02_TCP_DL_VO-3-B', '1.134WinDesktopLatitude_TCP_DL_VI-7-B', '1.154Macmackbook-air-02_TCP_DL_VI-9-B', '1.13androidvivov15_TCP_DL_VI-11-B', '1.134WinDesktopLatitude_TCP_DL_BE-13-B', '1.154Macmackbook-air-02_TCP_DL_BE-15-B', '1.13androidvivov15_TCP_DL_BE-17-B', '1.154Macmackbook-air-02_TCP_DL_BK-21-B', '1.13androidvivov15_TCP_DL_BK-23-B', 'dfgbdfb-B', 'cv_udp-1.0-1.sta1--1.0.0-A', 'cv_tcp-1.0-1.sta1--1.0.0-A', 'cv_tcp6-1.0-1.sta1--1.0.0-A', 'cv_udp6-1.0-1.sta1--1.0.0-A', 'cv_udp-1.0-13.wlan0--1.0.0-A', 'cv_tcp-1.0-13.wlan0--1.0.0-A', '1.13androidvivov15_TCP_DL_VO-5-A', '1.13androidvivov15_TCP_DL_VI-11-A', '1.13androidvivov15_TCP_DL_BE-17-A', '1.13androidvivov15_TCP_DL_BK-23-A', 'cv_tcp6-1.0-13.wlan0--1.0.0-A', 'cv_udp6-1.0-13.wlan0--1.0.0-A', '1.86Linlin-16_TCP_DL_VO-0-A', '1.86Linlin-16_TCP_DL_VI-6-A', '1.86Linlin-16_TCP_DL_BE-12-A', '1.86Linlin-16_TCP_DL_BK-18-A', 'cv_udp-1.0-134.wlan0--1.0.0-A', 'cv_tcp-1.0-134.wlan0--1.0.0-A', '1.134WinDesktopLatitude_TCP_DL_VO-1-A', '1.134WinDesktopLatitude_TCP_DL_BK-19-A', '1.134WinDesktopLatitude_TCP_DL_VI-7-A', '1.134WinDesktopLatitude_TCP_DL_BE-13-A', 'cv_tcp6-1.0-134.wlan0--1.0.0-A', 'cv_udp6-1.0-134.wlan0--1.0.0-A', '1.146Lintest99-Latitude_TCP_DL_VO-2-A', '1.146Lintest99-Latitude_TCP_DL_VI-8-A', '1.146Lintest99-Latitude_TCP_DL_BE-14-A', '1.146Lintest99-Latitude_TCP_DL_BK-20-A', 'UNKNOWN.59.1', 'UNKNOWN.63.2', 'UNKNOWN.135.2', 'UNKNOWN.147.2', 'UNKNOWN.159.2', 'UNKNOWN.171.2', 'UNKNOWN.178.2', 'UNKNOWN.182.2', 'UNKNOWN.188.2', 'UNKNOWN.192.2', 'UNKNOWN.65.20', 'UNKNOWN.61.21', '1.179MacMacs-Air.ctinb._TCP_DL_VO-4-A', '1.179MacMacs-Air.ctinb._TCP_DL_VI-10-A', '1.179MacMacs-Air.ctinb._TCP_DL_BE-16-A', '1.179MacMacs-Air.ctinb._TCP_DL_BK-22-A']

goyalsaurabh06 · 2026-07-30T04:07:03Z

            dropb_connections.update({keys[i]: avg_drop_b_per[i]})
-        logger.info("connections download {}".format(connections_download))
-        logger.info("connections {}".format(connections_upload))
+        # logger.info("connections download {}".format(connections_download))


@litincandela please remove this commented out lines if its not required.

goyalsaurabh06 · 2026-07-30T05:58:00Z

            return matched, abort, all_dataframes
        return matched, abort

+    def update_nav_data_for_all_cxs_stopped(self):


@litincandela Description for some functions like this in short

haricharan-candela · 2026-07-30T06:00:41Z

+        # CX names are prefixed with the resource id (e.g. "1.16androidsamsunga7_UDP_UL_BE-8"
+        # -> port "1.16"), mirroring the prefix already parsed in monitor_cx(). Used to report
+        # device issues against a recognizable port.


Just use doc strings for function

haricharan-candela · 2026-07-30T06:03:01Z

+        Polls CX presence in the layer-3 endpoint list while every created CX is missing, giving
+        devices a chance to reappear before the caller gives up. Also honors a user-initiated
+        stop from the webgui (same running.json check used in the monitor loop) during the wait,
+        so a stop request isn't delayed by the full retry window.
+
+        Returns True as soon as at least one CX responds again or the user stops the test, False
+        if `timeout` seconds elapse with every CX still missing and no stop request.


2-3 lines summarized lines are fine

VERIFIED CLI: python3 lf_interop_qos.py --ap_name Cisco \ --mgr 192.168.207.78 \ --mgr_port 8080 \ --upstream eth1 \ --test_duration 1m \ --download 0 \ --upload 1000000 \ --traffic_type lf_udp \ --tos "BK,BE,VI,VO" Signed-off-by: litincandela <litinommi@candelatech.com>

…stop responding VERIFIED CLI: python3 lf_interop_qos.py --ap_name Cisco \ --mgr 192.168.207.78 \ --mgr_port 8080 \ --upstream eth1 \ --test_duration 1m \ --download 0 \ --upload 1000000 \ --traffic_type lf_udp \ --tos "BK,BE,VI,VO" Signed-off-by: litincandela <litinommi@candelatech.com>

…CX Not Found" logs VERIFIED CLI: python3 lf_interop_qos.py --ap_name Cisco \ --mgr 192.168.207.78 \ --mgr_port 8080 \ --upstream eth1 \ --test_duration 1m \ --download 0 \ --upload 1000000 \ --traffic_type lf_udp \ --tos "BK,BE,VI,VO" Signed-off-by: litincandela <litinommi@candelatech.com>

…unresponsive VERIFIED CLI: python3 lf_interop_qos.py --ap_name Cisco \ --mgr 192.168.207.78 \ --mgr_port 8080 \ --upstream eth1 \ --test_duration 1m \ --download 0 \ --upload 1000000 \ --traffic_type lf_udp \ --tos "BK,BE,VI,VO" \ --robot_ip 192.168.204.64 \ --rotation 30,90 \ --coordinate 3,4 \ --robot_test Signed-off-by: litincandela <litinommi@candelatech.com>

haricharan-candela · 2026-07-30T07:31:34Z

        return matched, abort

+    def update_nav_data_for_all_cxs_stopped(self):
+        """Mark the WebUI navigation state as completed."""


Change the doc string

Signed-off-by: litincandela <litinommi@candelatech.com>

…arnings VERIFIED CLI: python3 lf_interop_qos.py --ap_name Cisco \ --mgr 192.168.207.78 \ --mgr_port 8080 \ --upstream eth1 \ --test_duration 1m \ --download 0 \ --upload 1000000 \ --traffic_type lf_udp \ --tos "BK,BE,VI,VO" \ --robot_ip 192.168.204.64 \ --rotation 30,90 \ --coordinate 3,4 \ --robot_test Signed-off-by: litincandela <litinommi@candelatech.com>

…t coordinate VERIFIED CLI: python3 lf_interop_qos.py --ap_name Cisco \ --mgr 192.168.207.78 \ --mgr_port 8080 \ --upstream eth1 \ --test_duration 1m \ --download 0 \ --upload 1000000 \ --traffic_type lf_udp \ --tos "BK,BE,VI,VO" \ --robot_ip 192.168.204.64 \ --rotation 30,90 \ --coordinate 3,4 \ --robot_test Signed-off-by: litincandela <litinommi@candelatech.com>

… reporting VERIFIED CLI: python3 lf_interop_qos.py --ap_name Cisco \ --mgr 192.168.207.78 \ --mgr_port 8080 \ --upstream eth1 \ --test_duration 1m \ --download 0 \ --upload 1000000 \ --traffic_type lf_udp \ --tos "BK,BE,VI,VO" \ --robot_ip 192.168.204.64 \ --rotation 30,90 \ --coordinate 3,4 \ --robot_test Signed-off-by: litincandela <litinommi@candelatech.com>

haricharan-candela self-requested a review July 29, 2026 11:56

haricharan-candela approved these changes Jul 29, 2026

View reviewed changes

goyalsaurabh06 reviewed Jul 30, 2026

View reviewed changes

haricharan-candela reviewed Jul 30, 2026

View reviewed changes

Comment thread py-scripts/lf_interop_qos.py Outdated

haricharan-candela reviewed Jul 30, 2026

View reviewed changes

litincandela added 5 commits July 30, 2026 06:33

litincandela force-pushed the qos_logging branch from 2b4d2ad to 7758e92 Compare July 30, 2026 07:29

haricharan-candela reviewed Jul 30, 2026

View reviewed changes

litincandela added 4 commits July 30, 2026 07:37

lf_base_robo.py: Add update_nav_data_for_all_cxs_stopped helper

e1f31dd

Signed-off-by: litincandela <litinommi@candelatech.com>

litincandela force-pushed the qos_logging branch from 7758e92 to ee2e3c2 Compare July 30, 2026 07:38

haricharan-candela requested a review from goyalsaurabh06 July 30, 2026 08:03

haricharan-candela added the Ready for review Ready for code review and feedback. label Jul 30, 2026

goyalsaurabh06 approved these changes Jul 30, 2026

View reviewed changes

goyalsaurabh06 merged commit 64db77b into greearb:master Jul 30, 2026
2 checks passed

Navigation Menu

Added Logging and Enhancements in Qos test - #319

Added Logging and Enhancements in Qos test#319
goyalsaurabh06 merged 9 commits into
greearb:masterfrom
goyalsaurabh06:qos_logging

litincandela commented Jul 29, 2026 •

edited

Loading

Uh oh!

goyalsaurabh06 Jul 30, 2026

Uh oh!

goyalsaurabh06 Jul 30, 2026

Uh oh!

haricharan-candela Jul 30, 2026

Uh oh!

Uh oh!

haricharan-candela Jul 30, 2026

Uh oh!

haricharan-candela Jul 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

litincandela commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

goyalsaurabh06 Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

goyalsaurabh06 Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

haricharan-candela Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

haricharan-candela Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

haricharan-candela Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

litincandela commented Jul 29, 2026 •

edited

Loading