Added observability tests for PCI Spyre adapter by saijanani-c · Pull Request #3192 · avocado-framework-tests/avocado-misc-tests

saijanani-c · 2026-06-10T16:55:06Z

Added tests to validate AIU-SMI observability
functionality on Spyre AIU devices during
inferencing.

Features:

Creates a containerized VLLM environment
Starts continuous inference in the background
Captures aiu-smi metrics continuously for a specified duration
Validates that metrics are successfully captured and contain expected data
Displays all captured metrics in test logs for analysis

saijanani-c · 2026-06-10T17:01:26Z

Logs:

Fetching asset from spyre_observability_tests.py:ObservabilityTests.test_aiu_smi
JOB ID : 6f8d92194eb3dbbd3c64934a9607757adb623341
JOB LOG : /root/avocado-fvt-wrapper/results/job-2026-06-10T22.06-6f8d921/job.log
(1/1) spyre_observability_tests.py:ObservabilityTests.test_aiu_smi;run-2c63: STARTED
(1/1) spyre_observability_tests.py:ObservabilityTests.test_aiu_smi;run-2c63: PASS (233.30 s)
RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML : /root/avocado-fvt-wrapper/results/job-2026-06-10T22.06-6f8d921/results.html
JOB TIME : 250.10 s

saijanani-c · 2026-06-14T11:16:01Z

Logs:

Fetching asset from spyre_observability_tests.py:ObservabilityTests.test_aiu_smi
Fetching asset from spyre_observability_tests.py:ObservabilityTests.test_acelyzer
JOB ID : 08fa912e16b8812912098bf75e4bf3f54f884531
JOB LOG : /root/avocado-fvt-wrapper/results/job-2026-06-14T14.33-08fa912/job.log
(1/2) spyre_observability_tests.py:ObservabilityTests.test_aiu_smi;run-06af: STARTED
(1/2) spyre_observability_tests.py:ObservabilityTests.test_aiu_smi;run-06af: PASS (243.23 s)
(2/2) spyre_observability_tests.py:ObservabilityTests.test_acelyzer;run-06af: STARTED
(2/2) spyre_observability_tests.py:ObservabilityTests.test_acelyzer;run-06af: PASS (690.20 s)
RESULTS : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML : /root/avocado-fvt-wrapper/results/job-2026-06-14T14.33-08fa912/results.html
JOB TIME : 952.04 s

sacsant · 2026-06-17T08:49:07Z

+    def run_cmd(self, cmd):
+        """Execute a command and track failures."""
+        if process.system(cmd, ignore_status=True, sudo=True, shell=True):
+            self.is_fail += 1


Not used any where else. Should this be removed?

sacsant · 2026-06-17T08:57:23Z

+                if result.exit_status == 0:
+                    self.log.info("SELinux set to Permissive mode")
+                else:
+                    self.log.warning(


Given this is a hard requirement, warning wouldn't be sufficient. Should we mark this as failed?

sacsant · 2026-06-17T08:58:26Z

+        tarball = self.fetch_asset('ServiceReport.zip', locations=[
+                                   'https://github.com/linux-ras/ServiceReport'
+                                   '/archive/master.zip'], expire='7d')
+        archive.extract(tarball, self.workdir)


Just extract will not ensure this tool is installed. You will need to explicitly install it.

sacsant · 2026-06-17T08:59:26Z

+
+        self.log.info("Setting up spyre group access")
+        self.run_cmd("servicereport -r -p spyre")
+        self.run_cmd("servicereport -v -p spyre")


Check for failure and mark the test as failed. Spyre setup is a hard requirement before running the tests.

yes look for FAIL in command output

sacsant · 2026-06-17T09:11:51Z

+        self.run_cmd(f"rm -rf {clone_dir}")
+
+        clone_cmd = f"cd /tmp && git clone https://github.com/IBM/aiu-trace-analyzer.git"
+        result = process.run(clone_cmd, shell=True, ignore_status=True)


in teardown() make sure to remove this directory

sacsant · 2026-06-17T09:12:48Z

+
+            script_path = os.path.join(self.workdir, "continuous_inference.py")
+            with open(script_path, 'w') as f:
+                f.write(inference_script)


in teardown() clean up this script.

sacsant · 2026-06-17T09:31:16Z

+            "PASS: AIU-SMI observability test completed successfully")
+        self.log.info("Metrics were successfully captured and validated")
+
+    def _build_acelyzer_podman_command(self):


this function implementation is similar to test_aiu_smi(). some of the code and be moved to a helper function which then both functions can use.

abdhaleegit

Please follow the existing template for spyre test

abdhaleegit · 2026-06-23T08:23:34Z

+            GROUP_ADD: "keep-groups"
+            PORT_MAPPING: "127.0.0.1::8000"
+            METRICS_DURATION: "30"
+```


if yaml is already added.. remove details from readme ..

abdhaleegit · 2026-06-23T08:24:14Z

+- Verifying acelyzer can process trace files
+- Validating performance metrics extraction
+- Confirming trace analysis tools are functioning correctly
+- Measuring AIU device utilization and kernel execution times


too much explanation, optmize overview and test purpose into single

abdhaleegit · 2026-06-23T08:24:32Z

+
+### Observability-Specific Parameters
+- `METRICS_DURATION`: Duration to capture metrics in seconds (default: "30") - for aiu-smi test
+- `TRACE_NUM_REQUESTS`: Number of inference requests for trace generation (default: "10") - for Acelyzer test


remove parameter explanation as already taken care in yaml

abdhaleegit · 2026-06-23T09:41:22Z

+- Test generates 10 inference requests by default (configurable via TRACE_NUM_REQUESTS)
+- Each inference request waits 2 seconds before the next one
+- Trace files are automatically found and analyzed
+- Performance metrics (AIU utilization, kernel times) are extracted and validated


remove trouble shooting and notes , pass fail criteria..

abdhaleegit · 2026-06-23T09:50:36Z

+# Duration to capture metrics in seconds (for aiu-smi test)
+METRICS_DURATION: ""
+# Number of inference requests for trace generation (for Acelyzer test)
+TRACE_NUM_REQUESTS: ""


follow yaml template from here https://github.com/avocado-framework-tests/avocado-misc-tests/blob/master/accelerator/spyre/spyre_RAG_usecase_test.py.data/spyre_RAG_usecase_Granite-3.3-8b-instruct.yaml

abdhaleegit · 2026-06-23T10:53:42Z

+        self.run_cmd(f"rm -rf {clone_dir}")
+
+        clone_cmd = f"cd /tmp && git clone https://github.com/IBM/aiu-trace-analyzer.git"
+        result = process.run(clone_cmd, shell=True, ignore_status=True)


avocado has git utils to clone .. please check

abdhaleegit · 2026-06-23T10:54:08Z

+                "Failed to clone aiu-trace-analyzer: %s", result.stderr_text)
+            self.fail("Failed to clone aiu-trace-analyzer repository")
+
+        self.log.info("Successfully cloned aiu-trace-analyzer")


are we not compiling the source ?

abdhaleegit · 2026-06-23T10:55:13Z

+        self.log.info("Starting container with FLEX environment variables")
+        podman_cmd = self._build_acelyzer_podman_command()
+        self.log.info("Podman command: %s", podman_cmd)
+        container_output = self.run_cmd_out(podman_cmd)


reuse create container run utils

abdhaleegit · 2026-06-23T10:55:33Z

+            self.fail("Failed to create container")
+
+        self.container_id = container_id
+        self.log.info("Container created: %s", container_id)


avoids all above logic

abdhaleegit · 2026-06-23T10:56:05Z

+
+        self.log.info("Copying aiu-trace-analyzer to container")
+        copy_cmd = f"podman cp {clone_dir} {container_id}:/tmp"
+        result = process.run(copy_cmd, shell=True, ignore_status=True)


there exists copy container utils .. please reuse

abdhaleegit · 2026-06-30T05:07:41Z

@saijanani-c any updates here ?

saijanani-c · 2026-07-31T06:33:37Z

JOB ID : 04f28119148799ba21fedb89cb7acd5dedaaf2af
JOB LOG : /root/avocado-fvt-wrapper/results/job-2026-07-28T13.59-04f2811/job.log
(1/4) spyre_observability_tests.py:ObservabilityTests.test_aiu_smi_root;run-dc39: STARTED
(1/4) spyre_observability_tests.py:ObservabilityTests.test_aiu_smi_root;run-dc39: PASS (246.34 s)
(2/4) spyre_observability_tests.py:ObservabilityTests.test_trace_analyzer_root;run-dc39: STARTED
(2/4) spyre_observability_tests.py:ObservabilityTests.test_trace_analyzer_root;run-dc39: PASS (175.78 s)
(3/4) spyre_observability_tests.py:ObservabilityTests.test_aiu_smi_nonroot;run-dc39: STARTED
(3/4) spyre_observability_tests.py:ObservabilityTests.test_aiu_smi_nonroot;run-dc39: PASS (248.89 s)
(4/4) spyre_observability_tests.py:ObservabilityTests.test_trace_analyzer_nonroot;run-dc39: STARTED
(4/4) spyre_observability_tests.py:ObservabilityTests.test_trace_analyzer_nonroot;run-dc39: PASS (176.83 s)
RESULTS : PASS 4 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML : /root/avocado-fvt-wrapper/results/job-2026-07-28T13.59-04f2811/results.html
JOB TIME : 876.76 s

sacsant · 2026-08-03T15:04:39Z

+3. ** test_aiu_smi_nonroot**: Create non-root container, capture aiu-smi metrics
+4. ** test_trace_analyzer_nonroot**: Use non-root container, analyze traces with acelyzer
+
+# Required Parameters


Are all the below mentioned parameters required?

I have changed this section

sacsant · 2026-08-03T15:05:26Z

+avocado run spyre_observability_tests.py: ObservabilityTests.test_trace_analyzer_nonroot - m spyre_observability_tests.py.data/spyre_observability_tests.yaml
+```
+
+# Test Flow


Remove the test flow section. It is additional work to keep this in sync with the implementation.

sacsant · 2026-08-03T15:18:19Z

+                metrics_output = f.read()
+
+            if not metrics_output or len(metrics_output.strip()) == 0:
+                self.log.error("No metrics were captured during the teno st")


typo -- teno st

Has been edited

sacsant · 2026-08-03T15:19:37Z

+
+        repo_url = self.params.get("TRACE_ANALYZER_REPO",
+                                   default="https://github.com/IBM/aiu-trace-analyzer.git")
+        branch = self.params.get("TRACE_ANALYZER_BRANCH", default="main")


This is missing from YAML file.

Has been removed

sacsant · 2026-08-03T15:23:51Z

+
+Run all tests:
+```bash
+avocado run - -max-parallel-tasks = 1 spyre_observability_tests.py - m spyre_observability_tests.py.data/spyre_observability_tests.yaml


test name is spyre_Observability_tests.py. Similarly the .data directory name

Implemented

sacsant · 2026-08-03T15:24:57Z

+- `CONTAINER_TAG`: Container image tag
+- `API_KEY`: Container registry API key(optional)
+- `DEVICE`: Device to mount(default: "/dev/vfio")
+- `PRIVILEGED`: Run privileged(default: "true")


Not used in the test.

Has been removed

sacsant · 2026-08-03T15:25:49Z

+
+# VLLM Options
+- `VLLM_SPYRE_USE_CB`: Enable Spyre callback(e.g., "1")
+- `VLLM_DT_CHUNK_LEN`: Data transfer chunk length(optional)


These are not present in YAML file.

Has been removed

sacsant · 2026-08-03T15:30:17Z

+
+        return container_id
+
+    def test_aiu_smi_root(self):


test_aiu_smi_root and test_aiu_smi_nonroot are near-identical. Create a helper function for common code.

Implemented

sacsant · 2026-08-03T15:31:03Z

+        self.log.info(
+            "PASS: AIU-SMI observability test (root) completed successfully")
+
+    def test_trace_analyzer_root(self):


test_trace_analyzer_root and test_trace_analyzer_nonroot are near-identical. Create a helper function for common code.

implemented

sacsant · 2026-08-03T15:38:02Z

+            else:
+                self.log.warning("Active file not found or empty")
+
+            return "out.json" in output


This is reached when the active CSV is empty or missing. The semantics are: "if the active CSV is absent but the output JSON exists, consider it a pass." This silent degradation is not documented and may mask real failures. If the active CSV is expected to always be present, this should be return False or at minimum accompanied by a log warning.

if the CSV is missing or empty, the test now logs an error and fails.

Added tests to validate AIU-SMI observability and Acelyzer trace analysis functionality on Spyre AIU devices during inferencing. Features: - Creates a containerized VLLM environment - Starts continuous inference in the background - Captures aiu-smi metrics continuously for a specified duration - Validates that metrics are successfully captured and contain expected data - Integrates Acelyzer (aiu-trace-analyzer) for post-processing JSON trace files - Generates FLEX timing traces during inference workloads - Analyzes traces with enhanced statistics for performance analysis - Validates trace generation and successful acelyzer processing Test Methods: 1. test_aiu_smi_root 2. test_trace_analyzer_root 3. test_aiu_smi_nonroot 4. test_trace_analyzer_nonroot Signed-off-by: Sai Janani C <jananic@linux.ibm.com>

saijanani-c · 2026-08-05T08:36:59Z

Logs:

JOB ID     : fd0e27f538835c1b4b4e96c2c5d35f71e6a5037a
JOB LOG    : /root/avocado-fvt-wrapper/results/job-2026-08-05T13.49-fd0e27f/job.log
 (1/4) spyre_observability_tests.py:ObservabilityTests.test_aiu_smi_root;run-dc39: STARTED
 (1/4) spyre_observability_tests.py:ObservabilityTests.test_aiu_smi_root;run-dc39: PASS (246.03 s)
 (2/4) spyre_observability_tests.py:ObservabilityTests.test_trace_analyzer_root;run-dc39: STARTED
 (2/4) spyre_observability_tests.py:ObservabilityTests.test_trace_analyzer_root;run-dc39: PASS (176.23 s)
 (3/4) spyre_observability_tests.py:ObservabilityTests.test_aiu_smi_nonroot;run-dc39: STARTED
 (3/4) spyre_observability_tests.py:ObservabilityTests.test_aiu_smi_nonroot;run-dc39: PASS (259.54 s)
 (4/4) spyre_observability_tests.py:ObservabilityTests.test_trace_analyzer_nonroot;run-dc39: STARTED
 (4/4) spyre_observability_tests.py:ObservabilityTests.test_trace_analyzer_nonroot;run-dc39: PASS (176.49 s)
RESULTS    : PASS 4 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /root/avocado-fvt-wrapper/results/job-2026-08-05T13.49-fd0e27f/results.html
JOB TIME   : 881.67 s

saijanani-c force-pushed the spyre-observability branch from d32e6b6 to 79f6b64 Compare June 10, 2026 16:57

saijanani-c force-pushed the spyre-observability branch from 79f6b64 to 8979066 Compare June 14, 2026 09:26

saijanani-c force-pushed the spyre-observability branch 5 times, most recently from f80b8bd to ec49d79 Compare June 16, 2026 10:08

sacsant suggested changes Jun 17, 2026

View reviewed changes

abdhaleegit self-assigned this Jun 19, 2026

abdhaleegit self-requested a review June 23, 2026 07:56

abdhaleegit requested changes Jun 23, 2026

View reviewed changes

saijanani-c force-pushed the spyre-observability branch from ec49d79 to b1ea413 Compare July 31, 2026 06:32

abdhaleegit self-requested a review August 3, 2026 11:25

sacsant suggested changes Aug 3, 2026

View reviewed changes

saijanani-c force-pushed the spyre-observability branch from b1ea413 to 04ada28 Compare August 5, 2026 08:36

Uh oh!

Conversation

saijanani-c commented Jun 10, 2026

Uh oh!

saijanani-c commented Jun 10, 2026

Uh oh!

saijanani-c commented Jun 14, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abdhaleegit left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abdhaleegit commented Jun 30, 2026

Uh oh!

saijanani-c commented Jul 31, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment