Skip to content

feat(routing): prometheus counters for sticky routing - #423

Open
prajjwalkumar17 wants to merge 6 commits into
mainfrom
feat/sticky-routing-metrics
Open

feat(routing): prometheus counters for sticky routing#423
prajjwalkumar17 wants to merge 6 commits into
mainfrom
feat/sticky-routing-metrics

Conversation

@prajjwalkumar17

@prajjwalkumar17 prajjwalkumar17 commented Sep 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Phase 5 for sticky routing (#393): observability. Merge order: after #422. Partially addresses #417 (Grafana panel + alerting ride the standard metrics endpoint; sandbox rollout and the A/B uplift measurement stay open there).

Two counters on the existing prometheus registry:

  • sticky_routing_decisions_total{outcome} — decide side, counted only when the feature actually engaged (flag on, customer present): overridden (pin diverged from the SR head — exactly the txns labeled STICKY_ROUTING), pinned_agreeing, vetoed (counts existed but every connector failed the eligibility/health filter — the veto doing its job), no_state, read_error.
  • sticky_routing_writes_total{outcome} — feedback side: recorded, decremented (a failure subtracted from a habit), no_habit (failure with nothing to subtract), over_budget (admission guard exhausted — the alarm that sticky key growth hit its ceiling), write_error.

The error outcomes matter because both guards deliberately fail open (Redis trouble must never fail a routing decision or a feedback call) — these counters are the only place that degradation becomes visible.

Important

Builds on phases 1–4; review the single commit on this branch.

🤖 Generated with Claude Code

This was referenced Sep 4, 2026
Open
Open
@prajjwalkumar17
prajjwalkumar17 force-pushed the feat/sticky-routing-metrics branch from 13dd253 to 0f3afac Compare September 6, 2026 08:48
@prajjwalkumar17 prajjwalkumar17 mentioned this pull request Sep 6, 2026
Open
@prajjwalkumar17
prajjwalkumar17 force-pushed the feat/sticky-routing-metrics branch from 0f3afac to 457e008 Compare September 8, 2026 10:15
@prajjwalkumar17 prajjwalkumar17 mentioned this pull request Sep 8, 2026
Open
@prajjwalkumar17 @claude
feat(routing): add sticky routing storage — per-customer net connecto…
64bdc9e 
…r scores

Storage layer only; feedback writes and the decide-side read come in the
stacked follow-ups.

- One Redis hash per (merchant, customer): fields pm:pmt:connector hold a
  NET count — successes increment, gateway failures decrement (floored at
  zero; merchant feedback is the source of truth in both directions).
  Failures never create a key or field, so key-creation stays tied to
  successful payments and the footprint stays bounded. Only positive
  counts qualify as pin candidates. Read is a single HGETALL returning
  counts sorted per exact combo; deliberately no cross-combo fallback
  (transaction-level fallbacks already cover a missing combo).
- Eviction safety: every key volatile with a sliding TTL re-armed only on
  success; per-customer combo cap with lowest-count pruning that drains
  full excess; per-merchant two-window admission budget so new-key growth
  is bounded up front. Defaults overridable via service config.
- Sticky on/off is a merchant-level feature flag (sticky_routing_enabled
  FeatureConf, dashboard plumbing in a follow-up) — deliberately NOT part
  of the euclid rule store, so toggling never touches the algorithm
  lifecycle.
- New wrapper commands: hincrby_with_expire, hincrby, hgetall_map,
  hdel_field.

Measured: 2-combo customer = 136B, 30-combo = 952B, listpack-encoded
(Redis 7.2.7). Refs #393.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@prajjwalkumar17
prajjwalkumar17 force-pushed the feat/sticky-routing-metrics branch from 457e008 to 55c5958 Compare September 8, 2026 10:51
@prajjwalkumar17 prajjwalkumar17 mentioned this pull request Sep 8, 2026
Open
7 tasks
prajjwalkumar17 and others added 5 commits September 9, 2026 14:12
@prajjwalkumar17 @claude
feat(routing): record sticky-routing outcomes from the feedback path
23f953b 
Write side of sticky routing; the decide-side read comes next in the
stack. Merchant feedback is the source of truth in both directions:
successes (CHARGED/AUTHORIZED/PARTIAL_CHARGED) increment the habit,
failure statuses (AUTHENTICATION_FAILED/AUTHORIZATION_FAILED/
JUSPAY_DECLINED/FAILURE) decrement it, floored at zero. Wider lifecycle
statuses (VOIDED, AUTO_REFUNDED, ...) touch neither direction. Failures
never create a key or field.

- update-gateway-score gains optional customerId/paymentMethod/
  paymentMethodType (backward compatible). Payload wins; the decide-time
  GatewayScoringData snapshot (now also carrying customerId) is the
  fallback, so callers need the new fields only when webhooks can
  outlive the snapshot's 30-min TTL.
- Two-pass hook in check_and_update_gateway_score_: payload-only pass
  before the snapshot fetch (survives snapshot expiry, same rationale as
  the ab-test emit), snapshot pass after.
- Deliberately NO dedupe: every feedback event counts, N times if sent
  N times — matching the engine's default SR scoring behavior (its
  feedback locks are opt-in per merchant), and avoiding a lock key per
  payment on the hot path.
- GSM healthy-failure early return is now failure-only: a success with
  errorInfo previously skipped both the SR reward and any snapshot-based
  sticky write.
- Sticky key pm/pmt are case-folded at the key boundary: the snapshot
  stores them uppercased while payloads arrive verbatim, and the two
  write sources must land on one hash field. Connector stays verbatim
  for the eligible-list comparison at decide time.
- openapi + api-ref updated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@prajjwalkumar17 @claude
feat(routing): apply sticky routing in the decide flow
0274dfc 
Read side of sticky routing, completing the loop over the two prior
commits (storage, feedback writes). Sticky is independent of the euclid
rule store: it is gated only by the merchant-level
sticky_routing_enabled feature flag, a customerId on the request, and
an optional per-request stickyRouting override (mirroring
enableMultiObjective).

- New post-scoring override in run_decider_flow, after the cost and
  volume nudges: pin the customer's highest-success-count connector for
  the pm:pmt combo. Precedence is PL/euclid explicit orders first (they
  bound the candidate set and are never overridden), then sticky, then
  SR, then the default ordering — sticky applies over the SR-selection
  family AND the Default approach, so a pin still wins when SR is off.
  Downtime relabels and hedging exploration stay untouched. The pinned
  connector must clear a configurable fraction of the top score
  (STICKY_ROUTING_MIN_SCORE_RATIO, default 0.5), so a pin can never
  resurrect a connector the outage/elimination passes just buried.
  Fails open on any miss.
- Every applied pin reports the new STICKY_ROUTING approach — a
  returning customer's repeat combo reading as SR would confuse
  callers. The label is admitted by the SRv3 producer-isolation and
  explore/exploit gates (mirroring the SR_SELECTION_MULTI_OBJECTIVE
  precedent), so pinned outcomes keep feeding the SR windows and the
  health veto stays live. A divergent pin additionally clears the
  superseded multi-objective/volume-steer claims so cost analytics
  can't credit a pick that didn't reach the customer.
- Sticky keys trim merchant/customer ids; customerId is documented as
  byte-exact, and the UPI doc guidance matches the V2 flow's actual key
  material.

Refs #393.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@prajjwalkumar17 @claude
feat(routing): dashboard-manageable kill switch for sticky routing
d5728a3 
KnownFeature::StickyRouting (slug "sticky-routing") maps to the
sticky_routing_enabled FeatureConf — the same key the feedback write and
decide read already check — so the ops kill switch gets the standard
features API/dashboard treatment with rollout-percentage support.
Partially addresses #417.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@prajjwalkumar17 @claude
test(routing): e2e coverage for the sticky routing loop
35b5def 
Six Playwright API tests against the live stack covering decide ->
update-gateway-score -> decide: a recorded success pins the customer, a
retried payment sticks to the finally-succeeding connector, the
stickyRouting:false opt-out and a disabled merchant flag suppress the
pin, sticky state is scoped to the payment-method combo, and a late
webhook carrying inline customer/pm fields records without the
decide-time snapshot. Score writes are async server-side, so pin
assertions poll. Closes #418's live-stack scope.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@prajjwalkumar17 @claude
feat(routing): prometheus counters for sticky routing
5f7a2cb 
Two IntCounterVecs: sticky_routing_decisions_total{outcome} on the
decide side (overridden / pinned_agreeing / vetoed / no_state /
read_error, counted only when the feature engaged) and
sticky_routing_writes_total{outcome} on the feedback side (recorded /
over_budget / duplicate / write_error). over_budget spiking is the
admission-guard alarm; read_error/write_error surface Redis trouble
(the guards fail open by design, so errors are otherwise invisible).
Partially addresses #417 — Grafana panel and
alerting ride the existing metrics endpoint.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@prajjwalkumar17
prajjwalkumar17 force-pushed the feat/sticky-routing-metrics branch from 55c5958 to 5f7a2cb Compare September 9, 2026 08:45
@prajjwalkumar17 prajjwalkumar17 self-assigned this Sep 9, 2026
@prajjwalkumar17
prajjwalkumar17 marked this pull request as ready for review September 9, 2026 10:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@prajjwalkumar17 prajjwalkumar17

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@prajjwalkumar17