[Celestica] Leh800bcls: Restart agents per benchmark to prevent cascading test hangs

[gang-tao]

Pre-submission checklist

• [ ✓ ] I've ran the linters locally and fixed lint errors related to the files I modified in this PR. You can install the linters by running pip install -r requirements-dev.txt && pre-commit install
• [ ✓ ] pre-commit run
  clang-format.........................................(no files to check)Skipped
  shellcheck...........................................(no files to check)Skipped
  shfmt................................................(no files to check)Skipped
  trim trailing whitespace.................................................Passed
  fix end of files.........................................................Passed
  check yaml...........................................(no files to check)Skipped
  check json...........................................(no files to check)Skipped
  check for merge conflicts................................................Passed
  ruff check...............................................................Passed
  ruff format..............................................................Passed
  Prevent sai_impl in fboss manifest.......................................Passed

Summary

When running multiple hardware benchmarks in a single batch on multi-switch/multi-NPU platforms, an unhandled exception or crash in a single benchmark case leaves the systemd hardware agents (fboss_hw_agent_oss@0 and @1) in a dead state. Because the benchmark framework's setup/teardown is outer-scoped, subsequent benchmark cases in the loop attempt to run without restarting these dead agents, resulting in all subsequent cases timing out after 1200 seconds.

Root Cause

benchmark_framework.py initialized hardware agents once before the case loop (self.suite.setup) and stopped them after the entire loop (self.suite.teardown). If a scale or config check throws a fatal exception in one case, the hardware agents exit and are never restarted. Subsequent cases attempt to connect to the dead agents, causing serial 1200s timeouts.

Solution

Modified fboss/oss/scripts/run_scripts/fboss_test_runner/frameworks/benchmark_framework.py:

• Initialized self._is_first_run = True in __init__.
• Restored the outer run method to its unified single try-finally wrapper.
• Implemented lazy-restart and try-except error handling inside _run_benchmark_binary.
• Changes similar to the format below are done automatically by pre-commit run.

-        except (
-            subprocess.CalledProcessError,
-            subprocess.TimeoutExpired,
-            OSError,
-        ) as e:
+        except (subprocess.CalledProcessError, subprocess.TimeoutExpired, OSError) as e:

Test Plan

1. python test

[root@a7b22b4d0190 fboss]# pytest fboss/oss/scripts/run_scripts/fboss_test_runner/unittests/test_benchmark_framework.py
================================================================= test session starts ==================================================================
platform linux -- Python 3.12.13, pytest-9.0.3, pluggy-1.6.0
rootdir: /var/FBOSS/fboss/fboss/oss/scripts/run_scripts
collected 24 items

fboss/oss/scripts/run_scripts/fboss_test_runner/unittests/test_benchmark_framework.py ........................                                   [100%]

================================================================== 24 passed in 0.04s ==================================================================

2. benchmark test.
   Ran benchmark batch on the dual-NPU Leh800bcls platform:

• Induced an intentional abort in Case 1.
• Verified Case 1 failed gracefully, torn down, and Case 2 successfully restarted the agents and ran to 100% completion with zero hangs.

Even if earlier cases time out or crash, it won't affect the execution of subsequent ones.

########## Benchmark results written to: benchmark_results_20260727_113239.csv

================================================================================
BENCHMARK RESULTS SUMMARY
================================================================================
HwFswScaleRouteAddBenchmark: OK [NO_THRESHOLD]
HwFswScaleRouteDelBenchmark: OK [NO_THRESHOLD]
HwThAlpmScaleRouteAddBenchmark: OK [NO_THRESHOLD]
HwThAlpmScaleRouteDelBenchmark: OK [NO_THRESHOLD]
HwHgridDUScaleRouteAddBenchmark: OK [NO_THRESHOLD]
HwHgridDUScaleRouteDelBenchmark: OK [NO_THRESHOLD]
HwHgridUUScaleRouteAddBenchmark: OK [NO_THRESHOLD]
HwHgridUUScaleRouteDelBenchmark: OK [NO_THRESHOLD]
HwHgridUUAnticipatedScaleRouteAddBenchmark: OK [NO_THRESHOLD]
HwHgridUUAnticipatedScaleRouteDelBenchmark: OK [NO_THRESHOLD]
HwStatsCollection: OK [NO_THRESHOLD]
runTxSlowPathBenchmark: FAILED
HwEcmpGroupShrink: OK [THRESHOLD PASS]
HwEcmpGroupShrinkWithCompetingRouteUpdates: OK [THRESHOLD PASS]
RxSlowPathBenchmark: OK [NO_THRESHOLD]
HwInitAndExit40Gx10GBenchmark: FAILED
HwInitAndExit100Gx10GBenchmark: FAILED
HwInitAndExit100Gx25GBenchmark: FAILED
HwInitAndExit100Gx50GBenchmark: FAILED
HwInitAndExit100Gx100GBenchmark: FAILED
HwInitAndExit400Gx400GBenchmark: FAILED
RibResolutionBenchmark: OK [NO_THRESHOLD]
RibSyncFibBenchmark: OK [NO_THRESHOLD]
HwUcmp128WidthScaleBenchmark: OK [NO_THRESHOLD]
HwUcmp512WidthScaleBenchmark: OK [NO_THRESHOLD]
cpuLatencyBenchmark: OK [NO_THRESHOLD]
SystemScaleChurnMemoryBenchmark: TIMEOUT
HwTunManagerProbeAndCleanup: OK [NO_THRESHOLD]
HwEcmpBackupGroupSpilloverBenchmark: OK [NO_THRESHOLD]
RouteAddHwEcmpGroupScale128x64Benchmark: OK [NO_THRESHOLD]
RouteAddHwEcmpGroupScale256x64Benchmark: OK [NO_THRESHOLD]
RouteDelHwEcmpGroupScale128x64Benchmark: OK [NO_THRESHOLD]
RouteDelHwEcmpGroupScale256x64Benchmark: OK [NO_THRESHOLD]
BgpRxSlowPathRouteChangeBenchmark: OK [NO_THRESHOLD]
BgpRxSlowPathRouteChangePortFlapBenchmark: OK [NO_THRESHOLD]
RxSlowPathArpBenchmark: TIMEOUT
SystemScaleMemoryBenchmark: TIMEOUT
HwFlowletStatsCollection: FAILED
HwClearInterfacePhyCounters: OK [NO_THRESHOLD]
================================================================================

[meta-codesync]

@AarjunC has imported this pull request. If you are a Meta employee, you can view this in D113927710.