Skip to content

feat(api-db): count transaction rollback failures#4117

Merged
chet merged 1 commit into
NVIDIA:mainfrom
chet:gh-issue-4108
Jul 24, 2026
Merged

feat(api-db): count transaction rollback failures#4117
chet merged 1 commit into
NVIDIA:mainfrom
chet:gh-issue-4108

Conversation

@chet

@chet chet commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Explicit transaction cleanup already warned when rollback failed, while WithTransaction discarded a rollback failure after a closure returned an error. Both paths can leave the database result uncertain, but only one was visible and neither was countable.

So, both branches now emit DatabaseTransactionRollbackFailed. A bounded trigger distinguishes explicit cleanup from closure-error cleanup, while the caller location, static context, and rollback error stay log-only. The original closure error still wins, so transaction return behavior is unchanged.

The table-driven Event contract covers both trigger values and verifies the warning fields and counter labels.

Tests added!

Related issues

This supports #4108

Part of #3169

Type of Change

  • Add - New feature or capability
  • Change - Changes in existing functionality
  • Fix - Bug fixes
  • Remove - Removed features or deprecated functionality
  • Internal - Internal changes (refactoring, tests, docs, etc.)

Breaking Changes

  • This PR contains breaking changes

Testing

  • Unit tests added/updated

  • Integration tests added/updated

  • Manual testing performed

  • No testing required (docs, internal refactor, etc.)

  • cargo test -p carbide-api-db database_transaction_rollback_failures_log_and_count_by_trigger --lib

  • cargo make clippy

  • cargo make carbide-lints

  • cargo xtask check-workspace-deps

  • cargo make check-licenses

  • cargo make check-bans

  • git diff --check origin/main

Additional Notes

The test intentionally exercises the Event contract rather than forcing a live SQLx backend to terminate during rollback; manufacturing that failure would couple a unit test to brittle driver behavior. Production wiring covers both existing rollback branches, and the primary transaction error remains unchanged.

Transaction rollback failures now emit one Event from both explicit cleanup and the error path in WithTransaction.

A bounded trigger distinguishes the two paths, while the caller location, static context, and rollback error stay log-only. The original closure error still wins, so this adds visibility without changing transaction return behavior.

This supports NVIDIA#4108

Signed-off-by: Chet Nichols III <chetn@nvidia.com>
@chet
chet requested a review from a team as a code owner July 24, 2026 17:17
@coderabbitai

coderabbitai Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Summary by CodeRabbit

  • New Features
    • Added monitoring for database transaction rollback failures, including trigger, context, caller, and error details.
    • Added a counter metric to track rollback failures by trigger.
  • Documentation
    • Documented the new carbide_database_transaction_rollback_failures_total metric (counter) and its “by trigger” breakdown.

Walkthrough

Transaction rollback failures now emit instrumented events and metrics for explicit cleanup and transaction-closure error paths. Tests validate event metadata and counter increments, and the new counter is documented.

Changes

Transaction rollback instrumentation

Layer / File(s) Summary
Rollback failure event and recorder
crates/api-db/src/lib.rs
Adds rollback-failure triggers, event metadata, error conversion, and instrument emission.
Rollback paths, validation, and metric documentation
crates/api-db/src/lib.rs, docs/observability/core_metrics.md
Routes explicit cleanup and closure-error rollback failures through instrumentation, tests both trigger paths, and documents the counter metric.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant TransactionAPI
  participant PgPool
  participant RollbackRecorder
  participant Instrumentation
  TransactionAPI->>RollbackRecorder: record ExplicitCleanup rollback failure
  PgPool->>RollbackRecorder: record ClosureError rollback failure
  RollbackRecorder->>Instrumentation: emit event and increment counter
  Instrumentation-->>TransactionAPI: rollback failure metadata
  Instrumentation-->>PgPool: rollback failure metadata
Loading

Possibly related issues

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title concisely describes the main change: counting transaction rollback failures in api-db.
Description check ✅ Passed The description accurately matches the added rollback-failure events, triggers, and tests.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

chet commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full_review, thanks!

@coderabbitai

coderabbitai Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

@chet Understood — I’ll perform a full review of all changes in #4117.

🐇🔍

✅ Action performed

Full review finished.

@github-actions

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
crates/api-db/src/lib.rs (1)

988-995: 🎯 Functional Correctness | 🔵 Trivial | 🏗️ Heavy lift

Exercise the rollback paths, not only the event payload.

This test directly calls emit(DatabaseTransactionRollbackFailed::new(...)); it passes even if either rollback branch stops recording failures, and it bypasses DatabaseError::txn_rollback. Retain this event-contract test, but add cases that induce rollback failure through rollback_or_log and with_txn.

As per path instructions, prefer findings about behavior, concurrency, resource lifetimes, and missing tests over style-only comments.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/api-db/src/lib.rs` around lines 988 - 995, Add integration cases that
trigger rollback failures through both rollback branches, using rollback_or_log
and with_txn, rather than only constructing DatabaseTransactionRollbackFailed
directly. Keep the existing event-payload contract test, and verify each induced
failure records the expected rollback-failure behavior/event through the actual
DatabaseError::txn_rollback path.

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@crates/api-db/src/lib.rs`:
- Around line 988-995: Add integration cases that trigger rollback failures
through both rollback branches, using rollback_or_log and with_txn, rather than
only constructing DatabaseTransactionRollbackFailed directly. Keep the existing
event-payload contract test, and verify each induced failure records the
expected rollback-failure behavior/event through the actual
DatabaseError::txn_rollback path.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e3018256-3365-44c1-a451-0804b4ae021e

📥 Commits

Reviewing files that changed from the base of the PR and between 1bac3d0 and 34983ad.

📒 Files selected for processing (2)
  • crates/api-db/src/lib.rs
  • docs/observability/core_metrics.md

@chet
chet enabled auto-merge (squash) July 24, 2026 19:03
@chet
chet merged commit 634c9e6 into NVIDIA:main Jul 24, 2026
63 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants