Skip to content

fix(config): serialize concurrent feature-flag updates on the shared conf row - #431

Open
prajjwalkumar17 wants to merge 1 commit into
mainfrom
claude/gifted-williams-cc51df
Open

fix(config): serialize concurrent feature-flag updates on the shared conf row#431
prajjwalkumar17 wants to merge 1 commit into
mainfrom
claude/gifted-williams-cc51df

Conversation

@prajjwalkumar17

Copy link
Copy Markdown
Member

Problem

POST /merchant-account/{merchant_id}/features/{feature} (KnownFeature::update_conf) updates a FeatureConf stored as one shared service_configuration row per feature (e.g. multi_objective_routing_enabled): it read the row, modified the merchants list in memory, and wrote the whole row back with no locking or compare-and-set.

Two concurrent calls for different merchants lose one update: merchant A enables, merchant B's concurrent enable overwrites the row without A's entry, and A's flag silently reads back as disabled. Reproduced by the sticky-routing Playwright suite when tests ran fully parallel (each test enabling the feature for its own merchant).

Fix

Database-level locking only — no new Redis keys, no schema changes.

  • New service_configuration::update_config_atomic(name, transform): runs the read-modify-write inside one DB transaction holding SELECT ... FOR UPDATE on the row. A concurrent writer blocks until the first commits, then reads the committed value and applies its change on top. After commit it does the same Redis/memory write-through as update_config.
  • KnownFeature::update_conf now passes its add/remove-merchant logic as the transform closure.
  • First-ever write to a key: service_configuration.name has no unique index, so FOR UPDATE has no row to lock and two racers could insert duplicate rows. On postgres the transaction additionally takes pg_advisory_xact_lock(hashtext('service_configuration'), hashtext(key)) — transaction-scoped, nothing persisted. It is savepoint-wrapped so backends without advisory locks degrade to the plain row lock instead of failing the request. On mysql the unindexed name scan already serializes writers via InnoDB next-key locks.
  • Row selection is ORDER BY id LIMIT 1 FOR UPDATE and the update targets the locked id, so behavior stays deterministic even if legacy duplicate rows exist.

read_effective already reads the DB directly, so dashboard reads reflect the committed row immediately; the TTL-bounded cache write-through behavior is unchanged.

Test

New Playwright test (tests/api/merchant/merchant-features.spec.ts): fires 6 parallel enables for distinct merchants against the shared row and asserts every flag survives; parallel disables on teardown exercise the removal path.

  • Against the unfixed binary it fails on the first run with a lost flag (one of its cleanup disables got eaten by the same race).
  • Against the fixed binary: passed 6/6 repeated runs, full spec green, and the DB kept a single clean row for the key.

cargo check passes for both --no-default-features --features postgres and the default mysql feature set; clippy and fmt clean.

Follow-up (out of scope)

Several other callers do the same unlocked find_config_by_name → mutate → update_config pattern on shared rows (sr_auto_calibration, cost_ingestion creds/mapping/overrides/invoice, multi_objective seed_store, routing_rules SR dimensions, merchant hierarchy). They can migrate to update_config_atomic incrementally.

🤖 Generated with Claude Code

…conf row

All merchants share ONE service_configuration row per feature, and
update_conf did an unlocked read-modify-write on it: two concurrent
toggles for different merchants lost one of the updates, so a merchant's
enable silently read back as disabled (reproduced by the sticky-routing
Playwright suite when it ran fully parallel).

Add service_configuration::update_config_atomic, which runs the
read-modify-write inside one DB transaction holding SELECT ... FOR
UPDATE on the row, and rewrite KnownFeature::update_conf on top of it.
The row lock serializes writers whenever the row exists. For the
first-ever write (no row to lock) postgres additionally takes
pg_advisory_xact_lock on the key, savepoint-wrapped so backends without
advisory locks (CockroachDB) degrade to the row lock instead of
failing; on mysql the unindexed name scan already serializes via
next-key locks. No new redis keys or schema changes.

The new Playwright test fires 6 parallel enables for distinct merchants
and asserts every flag survives; against the unfixed binary it fails on
the first run, and the cache write-through after commit keeps redis and
memory caches in sync with the winning row.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant