test(routing): sticky routing e2e loop + dashboard-manageable kill switch

[prajjwalkumar17]

Phase 4 for sticky routing (#393) — the kill switch made operable, and the loop proven end-to-end. Merge order: #419 → #420 → #412 → this. Two commits, one concern each:

1. KnownFeature::StickyRouting (slug sticky-routing) — maps to the sticky_routing_enabled FeatureConf the write and read paths check. This flag is THE configuration for sticky routing (independent of the euclid rule store), on the standard /merchant-account/{id}/features/* API with rollout-percentage support. Closes Sticky routing: merchant feature-flag config (sticky_routing_enabled) #414; the dashboard card ships in feat(dashboard): sticky routing toggle in the Multi Objective feature flags #424.
2. Six Playwright API tests (tests/api/routing/sticky-routing.spec.ts) — the full decide → update-gateway-score → decide loop against the live stack: a recorded success pins the customer; a retried payment sticks to the connector that finally succeeded; stickyRouting:false and a disabled merchant flag suppress the pin; sticky state is scoped to the payment-method combo; a late webhook with inline customer/pm fields records without the decide-time snapshot. Score writes are async server-side, so pin assertions poll. All six pass locally in ~4 s.

Closes #418.

Important

This branch builds on phases 1–3, so "Files changed" includes them until predecessors merge and this is rebased. Review commits d5728a3 (kill switch) and 35b5def (e2e suite) only.

🤖 Generated with Claude Code

---

How to test locally

Full manual walkthrough — stack boot (with or without analytics), the pin loop, duplicates/decrement/opt-out, ClickHouse verification, edge scenarios, reference table (click to expand)

[!TIP]
Run from this PR's branch for the full loop (flag via the features API below). The dashboard toggle card ships in #424 and the prometheus counters in #423 — on the top-of-stack branch (feat/sticky-routing-dashboard) every step works, including the counters in step 5 and the dashboard paths.

1. Boot an isolated stack

Run on non-default ports so an existing dev server (8080/5173) is untouched. Routing and sticky do not depend on analytics — but with Kafka + ClickHouse up, Decision Audit shows every decision with its STICKY_ROUTING approach (step 6).

With analytics (Kafka + ClickHouse in Docker/OrbStack — skip compose up if your dev stack already runs them):

cd decision-engine   # worktree on feat/sticky-routing-dashboard
cargo build --no-default-features --features postgres
docker compose -f docker-compose.yaml --profile analytics-clickhouse up -d   # kafka + kafka-init + clickhouse only

DECISION_ENGINE__SERVER__PORT=8092 \
DECISION_ENGINE__METRICS__PORT=9195 \
./target/debug/open_router &

curl -s http://localhost:8092/health   # expect 200

Without analytics (no containers; audit/analytics pages stay empty):

DECISION_ENGINE__SERVER__PORT=8092 \
DECISION_ENGINE__METRICS__PORT=9195 \
DECISION_ENGINE__ANALYTICS__CLICKHOUSE__ENABLED=false \
DECISION_ENGINE__ANALYTICS__KAFKA__ENABLED=false \
./target/debug/open_router &

Optional dashboard (only needed for the toggle UI; the flag can also be flipped by API):

DE_BACKEND_URL=http://127.0.0.1:8092 npm --prefix website run dev -- --port 5274 --strictPort

2. Create a fresh demo merchant + login

export API=http://localhost:8092
export H_ADMIN='x-admin-secret: test_admin'   # from config/development.toml [admin_secret]

curl -s -X POST $API/merchant-account/create -H 'Content-Type: application/json' -H "$H_ADMIN" \
  -d '{"merchant_id":"sticky_demo","gateway_success_rate_based_decider_input":null}'

curl -s -X POST $API/auth/signup -H 'Content-Type: application/json' \
  -d '{"email":"sticky_demo@example.com","password":"Password123!","merchant_id":"sticky_demo"}'

Dashboard login: sticky_demo@example.com / Password123! at http://localhost:5274.

3. Enable the feature flag

Via dashboard: Multi Objective → Feature Flags → Sticky routing (pin returning customers) → Enable. (It sits beside Cost savings / Volume contracts — the flag is the entire config; sticky is independent of the euclid rule store.)

Or via API:

curl -s -X POST $API/merchant-account/sticky_demo/features/sticky-routing \
  -H 'Content-Type: application/json' -H "$H_ADMIN" -d '{"enabled": true}'

Verify:

curl -s $API/merchant-account/sticky_demo/features -H "$H_ADMIN" \
  | python3 -c "import json,sys; print([f for f in json.load(sys.stdin)['features'] if f['feature']=='sticky-routing'])"

4. The core loop: decide → succeed → pinned

Redis is empty before anything happens:

redis-cli KEYS 'sticky_gw_sticky_demo*'

Payment 1 — plain SR decision (no sticky state yet):

curl -s -X POST $API/decide-gateway -H 'Content-Type: application/json' -H "$H_ADMIN" -d '{
  "merchantId":"sticky_demo",
  "eligibleGatewayList":["gigadat","loonio"],
  "rankingAlgorithm":"SR_BASED_ROUTING",
  "eliminationEnabled":false,
  "paymentInfo":{"paymentId":"pay_001","customerId":"cust_42","amount":100.5,"currency":"CAD",
    "paymentType":"ORDER_PAYMENT","paymentMethodType":"RTP","paymentMethod":"INTERAC"}
}' | python3 -m json.tool | head -12

It succeeds on loonio (feedback):

curl -s -X POST $API/update-gateway-score -H 'Content-Type: application/json' -H "$H_ADMIN" -d '{
  "merchantId":"sticky_demo","gateway":"loonio","status":"CHARGED","paymentId":"pay_001",
  "customerId":"cust_42","paymentMethod":"INTERAC","paymentMethodType":"RTP"
}'

customerId/paymentMethod/paymentMethodType are optional when feedback arrives within 30 min of the decide (recovered from the decide-time snapshot); send them for webhooks that can be later. CHARGED / AUTHORIZED / PARTIAL_CHARGED add one; AUTHENTICATION_FAILED / AUTHORIZATION_FAILED / JUSPAY_DECLINED / FAILURE subtract one (floor zero); lifecycle statuses (VOIDED, refunds…) touch neither.

Inspect the state (writes are async — allow ~1 s):

redis-cli KEYS 'sticky_gw_sticky_demo*'
redis-cli HGETALL sticky_gw_sticky_demo_cust_42
redis-cli TTL sticky_gw_sticky_demo_cust_42

Expected: hash sticky_gw_sticky_demo_cust_42 with field INTERAC:RTP:loonio → 1; TTL ≈ 7,776,000 s (90 days, sliding — every success re-arms it). There is deliberately no dedupe: every feedback event counts — the same event sent twice moves the count twice (matching default SR scoring behavior).

Payment 2 — same customer, now pinned:

curl -s -X POST $API/decide-gateway -H 'Content-Type: application/json' -H "$H_ADMIN" -d '{
  "merchantId":"sticky_demo",
  "eligibleGatewayList":["gigadat","loonio"],
  "rankingAlgorithm":"SR_BASED_ROUTING",
  "eliminationEnabled":false,
  "paymentInfo":{"paymentId":"pay_002","customerId":"cust_42","amount":100.5,"currency":"CAD",
    "paymentType":"ORDER_PAYMENT","paymentMethodType":"RTP","paymentMethod":"INTERAC"}
}' | python3 -m json.tool | head -12

Expected: "decided_gateway": "loonio" and "routing_approach": "STICKY_ROUTING" (every applied pin reports this label). A second CHARGED feedback for pay_002 bumps the count to 2.

Duplicates count — truth is truth. There is no dedupe: the same event sent twice moves the count twice (matching default SR scoring behavior):

for i in 1 2; do curl -s -X POST $API/update-gateway-score -H 'Content-Type: application/json' -H "$H_ADMIN" -d '{
  "merchantId":"sticky_demo","gateway":"loonio","status":"CHARGED","paymentId":"pay_002",
  "customerId":"cust_42","paymentMethod":"INTERAC","paymentMethodType":"RTP"}' > /dev/null; done
sleep 1; redis-cli HGET sticky_gw_sticky_demo_cust_42 INTERAC:RTP:loonio

Failures decrement (floor 0). A gateway-failure status subtracts one; at zero the pin releases. A failure on a connector with no habit changes nothing and creates no field:

curl -s -X POST $API/update-gateway-score -H 'Content-Type: application/json' -H "$H_ADMIN" -d '{
  "merchantId":"sticky_demo","gateway":"loonio","status":"FAILURE","paymentId":"pay_003",
  "customerId":"cust_42","paymentMethod":"INTERAC","paymentMethodType":"RTP"}' > /dev/null
sleep 1; redis-cli HGET sticky_gw_sticky_demo_cust_42 INTERAC:RTP:loonio

Per-request opt-out. Add top-level "stickyRouting": false to any decide body — that payment routes by SR (approach ≠ STICKY_ROUTING) while the stored habit stays untouched.

5. Watch the counters

curl -s http://localhost:9195/metrics | grep sticky_routing

• sticky_routing_decisions_total{outcome=…}: overridden (pin beat the SR head), pinned_agreeing (pin matched it), vetoed, no_state, read_error
• sticky_routing_writes_total{outcome=…}: recorded, decremented, no_habit (failure with nothing to subtract), over_budget (admission guard), write_error

6. Verify in analytics (with Kafka + ClickHouse up)

Every decide/feedback emits domain events; the Kafka → ClickHouse hop is async, so allow ~5 s:

curl -s "http://localhost:8123/?user=decision_engine&password=decision_engine" --data \
  "SELECT payment_id, flow_type, gateway, routing_approach, status FROM analytics_domain_events \
   WHERE merchant_id='sticky_demo' ORDER BY created_at_ms DESC LIMIT 10 FORMAT PrettyCompact"

The decide_gateway_decision rows carry routing_approach = STICKY_ROUTING for pinned payments. The same data drives the dashboard's Decision Audit page — search a payment id there to see the full request → decision → feedback timeline.

7. Behaviors worth testing

Scenario	How	Expect
Per-request opt-out	add top-level "stickyRouting": false to the decide body	SR decision, approach ≠ STICKY_ROUTING
Combo isolation	decide same customer with "paymentMethodType":"CARD","paymentMethod":"VISA"	no pin (counts are per PM:PMT combo)
Retry correctness	same paymentId: feedback AUTHORIZATION_FAILED on gigadat, then CHARGED on loonio	only loonio's count grows — the succeeding attempt sticks
Late webhook	feedback with inline customerId/pm/pmt for a paymentId that was never decided (snapshot absent)	count still recorded
Failures erode the habit	feedback FAILURE on the pinned connector (new paymentId)	count −1 (floor 0); at 0 the pin (and its label) disappears; SR score drops too
Health veto	seed several FAILUREs on the pinned connector (other paymentIds) until its SR score < 0.5 × top	pin refused: customer routes by SR to the healthy connector, metrics vetoed; counts remain, habit resumes if it recovers
Explicit rules win	route via priority logic / merchant preference	never overridden by sticky (precedence: PL/euclid → sticky → SR → default)
Flag off	disable the feature card	no sticky reads or writes at all

8. Automated suites

# unit (pure logic: keys, pruning, config, scoring gates)
cargo test --no-default-features --features postgres --lib sticky

# e2e (needs the API from step 1 running)
PW_NO_WEBSERVER=1 API_BASE_URL=http://localhost:8092 \
  npx playwright test tests/api/routing/sticky-routing.spec.ts --project=api

9. Reference

Thing	Value / knob
Sticky hash	sticky_gw_{merchantId}_{customerId} → field {PM}:{PMT}:{connector} = net habit count (success +1, failure −1, floor 0)
Key TTL (sliding)	STICKY_ROUTING_KEY_TTL service-config, default 90 d
Combos per customer	STICKY_ROUTING_MAX_COMBOS_PER_CUSTOMER, default 30 (lowest-count pruned)
New customers per merchant per TTL window	STICKY_ROUTING_MAX_CUSTOMERS_{mid}, default 1,000,000
Health-veto ratio	STICKY_ROUTING_MIN_SCORE_RATIO, default 0.5
Merchant flag	FeatureConf sticky_routing_enabled (dashboard slug sticky-routing)
Memory	~136 B for a 2-combo customer, ~952 B at the 30-combo cap (listpack)

Cleanup when done:

kill $(lsof -ti :8092); redis-cli --scan --pattern 'sticky_gw_sticky_demo*' | xargs -r redis-cli DEL