Skip to content

fix(metering): alarm when a billable meter emit is swallowed - #484

Merged
yilu331 merged 1 commit into
mainfrom
fix/meter-emit-failure-alarm
Sep 9, 2026
Merged

fix(metering): alarm when a billable meter emit is swallowed#484
yilu331 merged 1 commit into
mainfrom
fix/meter-emit-failure-alarm

Conversation

@guangyu-reflexio

@guangyu-reflexio guangyu-reflexio commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Problem

Both metering helpers in routes/_metering.py caught every exception and logged a
warning. That is lost billable usage, and the loss is undetectable:

  • the caller ignores the helper's return value, so a False goes nowhere;
  • _worker_loop's own search.metering.job_failed capture can never fire, because
    these handlers swallow the exception first;
  • downstream, the enterprise Sentry integration is wired at
    event_level=logging.ERROR with Sentry Logs off, so a logger.warning produces
    no event at all. Searching Sentry for one returns a false clean whether or not
    the drop is happening.

The asymmetry between the two helpers is what makes it dangerous.
_meter_applied_learnings performs a config lookup before emitting;
_meter_search_request does not. A config-store blip therefore drops
learning_applied while search_request keeps flowing, and the two meters diverge
with no signal anywhere — a shape that is easily misread as a retrieval bug rather
than a metering one.

Change

capture_anomaly("search.metering.emit_failed", level="error", ...) in both
handlers, tagged with the meter so the two are distinguishable. The logger.warning
stays; this adds the alarm rather than replacing the log.

Tests

test_metering_failure_does_not_break_search_response already drove this exact
failure path and asserted the drop happened, with a comment that it was silent.
That comment is now false, so the test asserts the alarm instead of the silence.

Mutation-verified rather than assumed: removing the applied-learnings capture fails
with assert [] == ['search.metering.emit_failed']. File restored afterwards and
confirmed by SHA-256.

  • tests/server/api_endpoints/test_applied_learnings_metering.py — 24 passed
  • ruff check / ruff format --check clean on both changed files

Scope

Observability only — no metering behaviour changes, and a failed emit still returns
False and still leaves the search response a 200.

Summary by CodeRabbit

  • Bug Fixes

    • Improved handling of metering emission failures so requests continue successfully while failures are logged and reported for monitoring.
    • Added diagnostic reporting for failures involving applied-learning and search-request usage events.
  • Tests

    • Expanded coverage to verify failure diagnostics, error severity, affected meter, and continued request success.

Both metering helpers caught every exception and logged a warning. That is
lost billable usage, and the loss is undetectable: the caller ignores the
helper's return value, and `_worker_loop`'s own `search.metering.job_failed`
capture can never fire because these handlers swallow first. Downstream, the
enterprise Sentry integration is wired at `event_level=logging.ERROR` with
Sentry Logs off, so a `logger.warning` produces no event at all -- searching
Sentry for one returns a false clean whether or not the drop is happening.

The asymmetry is what makes it dangerous. `_meter_applied_learnings` performs
a config lookup before emitting; `_meter_search_request` does not. A
config-store blip therefore drops `learning_applied` while `search_request`
keeps flowing, and the two meters diverge with no signal anywhere -- a shape
that is easily misread as a retrieval bug.

Adds `capture_anomaly("search.metering.emit_failed", level="error", ...)` to
both handlers, tagged with the meter so the two are distinguishable.

The existing failure test asserted the drop happened and commented that it was
silent; that comment is now false, so it asserts the alarm instead.
Mutation-verified: removing the applied-learnings capture fails with
"assert [] == ['search.metering.emit_failed']"; restored and confirmed by
SHA-256. 24 passed in that file; ruff clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 9, 2026

Copy link
Copy Markdown

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Essentials

Run ID: e112b5c9-5d0d-42d3-9e1f-bcf36733b924

📥 Commits

Reviewing files that changed from the base of the PR and between 3ff647c and 3c0ed74.

📒 Files selected for processing (2)
  • reflexio/server/routes/_metering.py
  • tests/server/api_endpoints/test_applied_learnings_metering.py

Included review availability: 1 review is currently available. Your included PR review attempts over the past 7 days set your current allowance at 4 reviews per hour.


📝 Walkthrough

Walkthrough

Metering helpers now report emission failures as search.metering.emit_failed anomalies with error severity, meter names, and organization IDs. Tests verify applied-learnings failure diagnostics and preserved request behavior.

Changes

Metering anomaly reporting

Layer / File(s) Summary
Report metering emission failures
reflexio/server/routes/_metering.py
Applied-learnings and search-request emission failures now call capture_anomaly with the relevant meter and organization ID while retaining error logging.
Validate applied-learnings diagnostics
tests/server/api_endpoints/test_applied_learnings_metering.py
The failure test captures anomaly calls, verifies error-level search.metering.emit_failed reporting for applied_learnings, and confirms that no usage event is emitted.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to 3c0ed

Metering emission failures now produce error-level anomaly events while preserving successful search responses and the existing best-effort metering behavior. No merge-blocking risk remains.

Suggested reviewers: yyiilluu

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 4 functions across 2 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: reporting failed billable metering emissions instead of silently swallowing them.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/meter-emit-failure-alarm

Comment @coderabbitai help to get the list of available commands.

@yilu331
yilu331 merged commit 1232dc9 into main Sep 9, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants