Describe the bug
On the MongoDB backend, two PutItem calls that upsert the same key at the same moment can fail with InternalServerError. The server log shows MongoDB's E11000 duplicate key error ... during findAndModify: both upserts see no document, both try to insert it, and the loser's error is mapped to StorageError::Internal instead of being retried. MongoDB documents this race for upserts against a unique index and expects the application to retry.
Two code paths leak it:
crates/storage-mongodb/src/data_engine.rs fast path (no condition, no stream, no GSI): find_one_and_replace(..).upsert(true) with .map_err(|e| StorageError::Internal(e.to_string())).- The transactional path used when the table has a GSI or a stream: the same upsert inside the session, with
TxErr::from, which treats only is_transient_write_conflict errors as retryable, so a duplicate-key error becomes TxErr::Fatal(Internal).
The conditional-insert branch already handles E11000 (it maps it to a condition failure with the winner's image); the unconditional upsert branches do not.
To Reproduce
Observed during concurrent read and write churn against a table with a GSI (items inserted and deleted from 8 threads while Scan and Query pages were read): 9 of the PutItem calls returned InternalServerError on a server built from 52c2fc90, and 4 on a server built from main 0a2b3bd3, with this log signature each time:
Kind: Command failed: Error code 11000 (DuplicateKey): Plan executor error during findAndModify :: caused by :: E11000 duplicate key error collection: extenddb_data._ddb_79e1bb8a-... index: _id_ dup key: { _id: "2:q1,5:666.5," }
A narrower probe, 2,000 PutItem calls on one key from 32 threads against a table with one GSI (main at 9e400357, MongoDB 7 standalone with a replica set), produced no InternalServerError but 269 TransactionConflictException responses, which is the retry ceiling in put_item_impl being reached. The same 2,000 calls against a hash-only table (fast path) all succeeded. The E11000 case needs the insert-versus-insert timing, which the churn workload hits and the single-key hammer does not always reach.
Expected behavior
Amazon DynamoDB never fails a PutItem because another PutItem on the same key is in flight; the last writer wins. Neither a 500 nor a TransactionConflictException is a client-visible outcome of two plain PutItem calls on the service.
Actual behavior
InternalServerError on the losing upsert when the race lands on the insert; TransactionConflictException when the retry budget for write conflicts is exhausted.
Suggested fix
Treat error code 11000 from the unconditional upsert as retryable in both paths: on the fast path, loop with backoff the way the transactional path does; in TxErr::from, classify a duplicate-key error from an upsert as Transient. The retry re-reads the document the winner inserted and replaces it, which is the last-writer-wins result.
Environment
- ExtendDB main at 9e40035 (and 0a2b3bd during the original observation)
- MongoDB 7, single-node replica set in a container
- boto3 1.40, Python 3.12
Describe the bug
On the MongoDB backend, two PutItem calls that upsert the same key at the same moment can fail with
InternalServerError. The server log shows MongoDB'sE11000 duplicate key error ... during findAndModify: both upserts see no document, both try to insert it, and the loser's error is mapped toStorageError::Internalinstead of being retried. MongoDB documents this race for upserts against a unique index and expects the application to retry.Two code paths leak it:
crates/storage-mongodb/src/data_engine.rsfast path (no condition, no stream, no GSI):find_one_and_replace(..).upsert(true)with.map_err(|e| StorageError::Internal(e.to_string())).TxErr::from, which treats onlyis_transient_write_conflicterrors as retryable, so a duplicate-key error becomesTxErr::Fatal(Internal).The conditional-insert branch already handles E11000 (it maps it to a condition failure with the winner's image); the unconditional upsert branches do not.
To Reproduce
Observed during concurrent read and write churn against a table with a GSI (items inserted and deleted from 8 threads while Scan and Query pages were read): 9 of the PutItem calls returned
InternalServerErroron a server built from52c2fc90, and 4 on a server built from main0a2b3bd3, with this log signature each time:A narrower probe, 2,000 PutItem calls on one key from 32 threads against a table with one GSI (main at
9e400357, MongoDB 7 standalone with a replica set), produced noInternalServerErrorbut 269TransactionConflictExceptionresponses, which is the retry ceiling input_item_implbeing reached. The same 2,000 calls against a hash-only table (fast path) all succeeded. The E11000 case needs the insert-versus-insert timing, which the churn workload hits and the single-key hammer does not always reach.Expected behavior
Amazon DynamoDB never fails a PutItem because another PutItem on the same key is in flight; the last writer wins. Neither a 500 nor a
TransactionConflictExceptionis a client-visible outcome of two plain PutItem calls on the service.Actual behavior
InternalServerErroron the losing upsert when the race lands on the insert;TransactionConflictExceptionwhen the retry budget for write conflicts is exhausted.Suggested fix
Treat error code 11000 from the unconditional upsert as retryable in both paths: on the fast path, loop with backoff the way the transactional path does; in
TxErr::from, classify a duplicate-key error from an upsert asTransient. The retry re-reads the document the winner inserted and replaces it, which is the last-writer-wins result.Environment