fix(config): serialize concurrent feature-flag updates on the shared conf row - #431
Open
prajjwalkumar17 wants to merge 1 commit into
Open
fix(config): serialize concurrent feature-flag updates on the shared conf row#431prajjwalkumar17 wants to merge 1 commit into
prajjwalkumar17 wants to merge 1 commit into
Conversation
…conf row All merchants share ONE service_configuration row per feature, and update_conf did an unlocked read-modify-write on it: two concurrent toggles for different merchants lost one of the updates, so a merchant's enable silently read back as disabled (reproduced by the sticky-routing Playwright suite when it ran fully parallel). Add service_configuration::update_config_atomic, which runs the read-modify-write inside one DB transaction holding SELECT ... FOR UPDATE on the row, and rewrite KnownFeature::update_conf on top of it. The row lock serializes writers whenever the row exists. For the first-ever write (no row to lock) postgres additionally takes pg_advisory_xact_lock on the key, savepoint-wrapped so backends without advisory locks (CockroachDB) degrade to the row lock instead of failing; on mysql the unindexed name scan already serializes via next-key locks. No new redis keys or schema changes. The new Playwright test fires 6 parallel enables for distinct merchants and asserts every flag survives; against the unfixed binary it fails on the first run, and the cache write-through after commit keeps redis and memory caches in sync with the winning row. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
POST /merchant-account/{merchant_id}/features/{feature}(KnownFeature::update_conf) updates aFeatureConfstored as one sharedservice_configurationrow per feature (e.g.multi_objective_routing_enabled): it read the row, modified the merchants list in memory, and wrote the whole row back with no locking or compare-and-set.Two concurrent calls for different merchants lose one update: merchant A enables, merchant B's concurrent enable overwrites the row without A's entry, and A's flag silently reads back as disabled. Reproduced by the sticky-routing Playwright suite when tests ran fully parallel (each test enabling the feature for its own merchant).
Fix
Database-level locking only — no new Redis keys, no schema changes.
service_configuration::update_config_atomic(name, transform): runs the read-modify-write inside one DB transaction holdingSELECT ... FOR UPDATEon the row. A concurrent writer blocks until the first commits, then reads the committed value and applies its change on top. After commit it does the same Redis/memory write-through asupdate_config.KnownFeature::update_confnow passes its add/remove-merchant logic as thetransformclosure.service_configuration.namehas no unique index, soFOR UPDATEhas no row to lock and two racers could insert duplicate rows. On postgres the transaction additionally takespg_advisory_xact_lock(hashtext('service_configuration'), hashtext(key))— transaction-scoped, nothing persisted. It is savepoint-wrapped so backends without advisory locks degrade to the plain row lock instead of failing the request. On mysql the unindexednamescan already serializes writers via InnoDB next-key locks.ORDER BY id LIMIT 1 FOR UPDATEand the update targets the lockedid, so behavior stays deterministic even if legacy duplicate rows exist.read_effectivealready reads the DB directly, so dashboard reads reflect the committed row immediately; the TTL-bounded cache write-through behavior is unchanged.Test
New Playwright test (
tests/api/merchant/merchant-features.spec.ts): fires 6 parallel enables for distinct merchants against the shared row and asserts every flag survives; parallel disables on teardown exercise the removal path.cargo checkpasses for both--no-default-features --features postgresand the default mysql feature set; clippy and fmt clean.Follow-up (out of scope)
Several other callers do the same unlocked
find_config_by_name→ mutate →update_configpattern on shared rows (sr_auto_calibration, cost_ingestion creds/mapping/overrides/invoice, multi_objective seed_store, routing_rules SR dimensions, merchant hierarchy). They can migrate toupdate_config_atomicincrementally.🤖 Generated with Claude Code