Skip to content

feat(spdm-controller): count unexpected evidence task states#4118

Merged
chet merged 1 commit into
NVIDIA:mainfrom
chet:gh-issue-4109
Jul 24, 2026
Merged

feat(spdm-controller): count unexpected evidence task states#4118
chet merged 1 commit into
NVIDIA:mainfrom
chet:gh-issue-4109

Conversation

@chet

@chet chet commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

SPDM evidence polling already logs and retries any Redfish task state outside Completed, New, Starting, and Running, then transitions to failure after the retry budget is exhausted. Those encounters were visible per device but could not be counted or separated by state.

So, the retry-or-fail branch now emits SpdmEvidenceCollectionTaskStateUnexpected. Bounded task_state and next_action labels make the failure path trendable, while machine, device, task, retry count, and task error stay log-only. The same computed action drives both telemetry and control flow, and the existing error message and retry threshold are unchanged.

The table-driven test enumerates every Redfish TaskState variant plus a missing state, covering normal states, all retrying states, and the retry-to-fail boundary.

Tests added!

Related issues

This supports #4109

Part of #3169

Type of Change

  • Add - New feature or capability
  • Change - Changes in existing functionality
  • Fix - Bug fixes
  • Remove - Removed features or deprecated functionality
  • Internal - Internal changes (refactoring, tests, docs, etc.)

Breaking Changes

  • This PR contains breaking changes

Testing

  • Unit tests added/updated

  • Integration tests added/updated

  • Manual testing performed

  • No testing required (docs, internal refactor, etc.)

  • cargo test -p carbide-spdm-controller evidence_collection_task_states_log_and_count_when_unexpected --lib

  • cargo make format-nightly

  • cargo make clippy

  • cargo make carbide-lints

  • cargo xtask check-event-names

  • cargo xtask check-metric-docs

  • cargo xtask check-workspace-deps

  • cargo make check-licenses

  • cargo make check-bans

  • git diff --check origin/main

Additional Notes

The counter measures unexpected-state encounters rather than unique devices. Pending remains in the existing retry-or-fail branch to keep this PR behavior-preserving; changing its state-machine treatment should be a separate policy decision. The exhaustive classifier intentionally makes future Redfish task-state additions a compile-time decision instead of silently assigning them to the failure path.

@chet
chet requested a review from a team as a code owner July 24, 2026 17:17
@coderabbitai

coderabbitai Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 856a28d3-5447-41f2-804e-f9adca170327

📥 Commits

Reviewing files that changed from the base of the PR and between b9bd173 and 6e62080.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (3)
  • crates/spdm-controller/Cargo.toml
  • crates/spdm-controller/src/handler.rs
  • docs/observability/core_metrics.md
🚧 Files skipped from review as they are similar to previous changes (3)
  • crates/spdm-controller/Cargo.toml
  • docs/observability/core_metrics.md
  • crates/spdm-controller/src/handler.rs

Summary by CodeRabbit

  • Bug Fixes

    • Improved SPDM evidence collection handling by classifying task states and applying a consistent flow for unexpected states.
    • Refined in-progress behavior to wait based on percent completion.
    • Standardized retry vs. failure decisions using the remaining retry budget.
  • Observability

    • Added tracking for unexpected SPDM evidence collection task states, including next-action context.
    • Documented a new counter metric for unexpected task states.
  • Tests

    • Added coverage for normal, missing, and retry-exhausted scenarios to verify emitted events and metric deltas.

Walkthrough

SPDM evidence collection polling now classifies task states, instruments unexpected states with retry or failure actions, and validates the resulting events and counters. The new counter is documented and test-only dependencies are added.

Changes

SPDM evidence collection observability

Layer / File(s) Summary
Task-state classification and retry routing
crates/spdm-controller/src/handler.rs
Evidence collection states are classified as completed, in progress, or unexpected; unexpected states emit instrumentation and transition to retry or failure based on the retry budget.
Instrumentation validation and documentation
crates/spdm-controller/src/handler.rs, crates/spdm-controller/Cargo.toml, docs/observability/core_metrics.md
Tests validate event labels and counter increments for normal, unexpected, missing, retryable, and exhausted states; the metric and test dependencies are added.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant PollEvidenceCollection
  participant TaskStateClassifier
  participant Instrumentation
  participant StateMachine
  PollEvidenceCollection->>TaskStateClassifier: classify task state
  TaskStateClassifier-->>PollEvidenceCollection: Completed, InProgress, or Unexpected
  PollEvidenceCollection->>Instrumentation: emit unexpected state and next_action
  PollEvidenceCollection->>StateMachine: retry collection or transition to Failed
Loading

Possibly related issues

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: counting unexpected SPDM evidence task states.
Description check ✅ Passed The description accurately describes the telemetry, retry behavior, and tests added in this PR.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

chet commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full_review, thanks!

@coderabbitai

coderabbitai Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

@chet: Understood — I’ll perform a complete re-review of PR #4118, including the implementation and its test coverage.

ᕱᕱ

✅ Action performed

Full review finished.

@github-actions

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
crates/spdm-controller/src/handler.rs (1)

242-254: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Avoid cloning each task message.

task.messages.iter().map(|t| t.message.clone()) clones every message string just to join them; itertools::join only needs Display, so borrowing avoids the allocation.

♻️ Proposed fix
-                        let err = task.messages.iter().map(|t| t.message.clone()).join("\n");
+                        let err = task.messages.iter().map(|t| t.message.as_str()).join("\n");

As per path instructions, "avoiding needless clones" for crates/**/*.rs.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/spdm-controller/src/handler.rs` around lines 242 - 254, Update the
error construction in the EvidenceCollectionTaskStateClass::Unexpected branch to
join borrowed task message values rather than cloning each t.message. Preserve
the existing newline-separated output and error field behavior while using
itertools::join with a Display-compatible borrowed iterator.

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@crates/spdm-controller/src/handler.rs`:
- Around line 242-254: Update the error construction in the
EvidenceCollectionTaskStateClass::Unexpected branch to join borrowed task
message values rather than cloning each t.message. Preserve the existing
newline-separated output and error field behavior while using itertools::join
with a Display-compatible borrowed iterator.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 50ed5ba1-f1b2-47aa-9425-58ac04e746a8

📥 Commits

Reviewing files that changed from the base of the PR and between 1bac3d0 and b9bd173.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (3)
  • crates/spdm-controller/Cargo.toml
  • crates/spdm-controller/src/handler.rs
  • docs/observability/core_metrics.md

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
crates/spdm-controller/src/handler.rs (2)

242-256: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Aggregated error string embeds raw newlines into a structured log field.

task.messages.iter().map(|t| t.message.clone()).join("\n") joins multiple messages with "\n" and stores the result in the error #[context] field. Embedded newlines inside a single structured field undermine the logfmt-friendly single-line assumption this codebase's tracing convention relies on. Consider joining with a single-line-safe separator instead.

Based on coding guidelines: "Use logfmt-compatible structured logging."

♻️ Proposed fix
-                        let err = task.messages.iter().map(|t| t.message.clone()).join("\n");
+                        let err = task.messages.iter().map(|t| &t.message).join("; ");
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/spdm-controller/src/handler.rs` around lines 242 - 256, Update the
error aggregation in the EvidenceCollectionTaskStateClass::Unexpected branch to
join task.messages with a single-line-safe separator instead of a raw newline
before assigning the result to the structured error field. Preserve the existing
message order and emit behavior.

Source: Coding guidelines


405-414: 🧹 Nitpick | 🔵 Trivial

Event always logs at error, even on retry.

SpdmEvidenceCollectionTaskStateUnexpected is declared with a fixed log = error regardless of the computed next_action. Since Pending/transient states routinely fall into this branch and retry successfully, every retry attempt is logged as an ERROR, which may generate log noise disproportionate to actual severity (reserving error for the terminal Fail outcome and warn for Retry would better reflect operational impact). Worth confirming whether carbide_instrument::Event supports level selection based on a field value, or whether this is an accepted tradeoff for this event.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/spdm-controller/src/handler.rs` around lines 405 - 414, Update
SpdmEvidenceCollectionTaskStateUnexpected and its emission path so log severity
reflects the computed next_action: use warn for Retry and error for terminal
Fail outcomes, if carbide_instrument::Event supports field-based level
selection. If dynamic selection is unsupported, confirm and document retaining
the fixed error level rather than changing unrelated behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@crates/spdm-controller/src/handler.rs`:
- Around line 242-256: Update the error aggregation in the
EvidenceCollectionTaskStateClass::Unexpected branch to join task.messages with a
single-line-safe separator instead of a raw newline before assigning the result
to the structured error field. Preserve the existing message order and emit
behavior.
- Around line 405-414: Update SpdmEvidenceCollectionTaskStateUnexpected and its
emission path so log severity reflects the computed next_action: use warn for
Retry and error for terminal Fail outcomes, if carbide_instrument::Event
supports field-based level selection. If dynamic selection is unsupported,
confirm and document retaining the fixed error level rather than changing
unrelated behavior.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 0dcf907e-18c4-46ba-8cb4-0dff3cc10d3d

📥 Commits

Reviewing files that changed from the base of the PR and between 1bac3d0 and b9bd173.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (3)
  • crates/spdm-controller/Cargo.toml
  • crates/spdm-controller/src/handler.rs
  • docs/observability/core_metrics.md

SPDM evidence polling now counts Redfish task states that already enter the retry-or-fail branch.

Bounded task-state and next-action labels distinguish each encounter, while machine, device, task, retry count, and task error stay log-only. The existing error message and retry threshold are unchanged.

This supports NVIDIA#4109

Signed-off-by: Chet Nichols III <chetn@nvidia.com>
@chet
chet enabled auto-merge (squash) July 24, 2026 19:03
@chet
chet merged commit 4520d8f into NVIDIA:main Jul 24, 2026
63 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants