Repository navigation
Release Python GCP Storage v1.1.0 - #964
Conversation
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub. |
2d6458e to
0bf9de7
Compare
0bf9de7 to
3167c40
Compare
3167c40 to
526314a
Compare
526314a to
2b080d3
Compare
There was a problem hiding this comment.
Additional findings (outside current diff — PR may have been updated during review):
-
🔴
sdk/python/storage/keeper_secrets_manager_storage_gcp_kms/keeper_secrets_manager_storage_gcp_kms/storage_gcp_kms.py:300-304— The new KSM-946threading.RLockwrapsread_storage/save_storage/set/delete/delete_all(lines 300-337), butchange_key(lines 277-298) — the heaviest mutator on the class — was left outside the lock. It assignsself.gcp_key_config(line 286) and viaget_key_details()mutatesself.key_purpose_details,self.encryption_algorithm,self.is_asymmetric, then calls__save_config, all touching the same shared state the lock protects. Fix: wrap thechange_keybody inwith self._lock:— RLock is re-entrant so the inner__save_configcall is fine.Extended reasoning...
What the bug is
The KSM-946 fix adds
self._lock = threading.RLock()in__init__(line 52) and wrapsread_storage,save_storage,set,delete, anddelete_allinwith self._lock:(lines 300-337). Butchange_keyat lines 277-298 — sitting in the same class, on the same level, as those wrapped mutators — is the heaviest mutator and is left unprotected.change_keydoes all of the following without acquiring the lock:- Reads
self.config(line 283), conditionally callsself.load_config()(line 285). - Assigns
self.gcp_key_config = new_gcp_key_config(line 286). - Calls
self.get_key_details()(line 287) which mutatesself.key_purpose_details,self.encryption_algorithm, andself.is_asymmetric(storage_gcp_kms.py:70-79). - Calls
self.__save_config({}, force=True)(line 288), which reads all of the above to driveencrypt_bufferrouting.
Every one of these attributes is read by
__save_config/encrypt_bufferon the locked code path thatset/deletetake.The race (concrete, observable)
Starting state: storage initialized with a symmetric
ENCRYPT_DECRYPTkey.is_asymmetric == False.- Thread A calls
change_key(new_key)with a new asymmetric key. It executes line 286 —self.gcp_key_config = new_gcp_key_config— and is preempted beforeget_key_details()runs. State:gcp_key_config = NEW (asymmetric),is_asymmetric = False (OLD),encryption_algorithm = OLD. - Thread B calls
set(KEY_CLIENT_ID, "x"). It acquiresself._lockat line 315 (Thread A is not holding it becausechange_keynever tried to acquire it — RLock is a mutex, but mutexes only exclude threads that try to acquire). B enterssave_storage→__save_config. - Inside
__save_config,encrypt_bufferis invoked withkey_properties=self.gcp_key_config(NEW asymmetric key resource) butis_asymmetric=self.is_asymmetric(stillFalse) andencryption_algorithm=self.encryption_algorithm(still OLD). encrypt_buffertakes the symmetric branch (becauseis_asymmetric == False) and callsencrypt_data_and_validate_crc/client.encryptagainst an asymmetric key resource. GCP rejects the request — and post-PRencrypt_buffernow raises (utils.py:75), so__save_configraises,save_storageraises,setpropagates an inscrutable error.
The mirror-image race — A finishing
get_key_details(sois_asymmetric=True) while still in__save_config, with B reading a mix of NEWis_asymmetric+ OLDgcp_key_config(if B took a snapshot earlier) — produces the same class of torn-state failure. CPythons GIL serializes single attribute writes but not multi-statement sequences, so torn reads across multiple attributes are real.Why the existing code does not prevent it
This is exactly the race class the PR identified and added the lock for. The PR added the lock to five mutators but missed the sixth. There is no acquire on
change_keys path, so the mutex held by B does not exclude A.README CHANGELOG advertises the broader contract
The 1.1.0 CHANGELOG entry this PR adds says: "Fixed thread-safety: added threading.RLock to GCPKeyValueStorage — concurrent set() / delete() calls no longer race on the config dict or the encrypt-and-write sequence (KSM-946)". A reader of that contract would reasonably expect all public mutators on the class to honor the lock.
change_keyis the one mutator that breaks it.Impact
- Concurrent
set/delete/change_keycallers can hit a torn-stateencrypt_bufferinvocation routed through the wrong code path — error or wrong-ciphertext on disk. - The class advertises a thread-safety contract (CHANGELOG) that one of its mutators silently violates.
How to fix
One line of indentation. Wrap
change_keys body inwith self._lock:— same pattern the PR established for the five mutators immediately below it.RLockis re-entrant, so the inner__save_config(which itself does no explicit acquire on this path) and the innerload_configcall are unaffected:def change_key(self, new_gcp_key_config: GCPKeyConfig) -> bool: with self._lock: old_key_configuration = self.gcp_key_config old_crypto_client = self.crypto_client try: ... except Exception as error: ...
Not a duplicate
The existing review comment about
change_key(comment 3228324893 and its repeat at 3229647853) is about the rollback gap — that the except block only restores 2 of the 5 attributes mutated byget_key_details. This bug is about the lock gap — that the function takes no lock at all. Both bugs are in the same function; they are independent defects with independent fixes. The lock fix is one extrawith self._lock:line; the rollback fix is snapshotting and restoring three more attributes. - Reads
There was a problem hiding this comment.
Additional findings (outside current diff — PR may have been updated during review):
-
🟣
sdk/python/storage/keeper_secrets_manager_storage_gcp_kms/keeper_secrets_manager_storage_gcp_kms/storage_gcp_kms.py:260-266— Pre-existing dead code inload_config: the inner except at line 266-269 setsdecryption_error = Trueand immediatelyraise Exception("Failed to parse decrypted config file ..."), so the raise propagates past lines 271-273 to the outer except at line 275. Theif json_error and decryption_error:block at lines 271-273 is therefore unreachable, the "may contain JSON format problems" message never surfaces, and thedecryption_errorlocal (initialized at 234, mutated at 267, never read on any reachable path) is dead state. The structure predates this PR — only md5→sha256 and the narrowing of the OTHER inner except at line 245 changed in this region — but since the file is open, cheap cleanup: drop thedecryption_errorlocal and the unreachable block, or (if the JSON-format message was meant to surface) move the inner raise out of the except.Extended reasoning...
What the bug is
load_config(storage_gcp_kms.py:215-277) has two sibling inner try/except blocks inside its outer try:- try-A (lines 235-246): attempts
contents.decode()andjson.loads()to detect a plain-JSON config. Its except (line 245, narrowed in this PR to(json.JSONDecodeError, UnicodeDecodeError)) stores the failure injson_error. - try-B (lines 260-269): inside the
if json_error:branch, attemptsjson.loads(config_json)on the result ofdecrypt_buffer. Its except (lines 266-269) setsdecryption_error = True, logs, and immediately raisesException("Failed to parse decrypted config file ...").
The raise at line 269 escapes try-B and the surrounding
if json_error:block, propagating directly to the outer except at line 275, which logs and re-raises. So on every path, lines 271-273 (if json_error and decryption_error: raise Exception("may contain JSON format problems")) are unreachable:- try-B succeeds →
decryption_errorstaysFalse→ condition False, branch skipped. - try-B fails →
decryption_erroris set toTruebut the very next statement is a raise → control skips line 271 entirely.
The
decryption_errorlocal at line 234 is therefore dead state — assigned, mutated in one place, never read on any reachable path. The "may contain JSON format problems" message it gates can never surface to the user; the actual exception that reaches the caller is always "Failed to parse decrypted config file" from line 269 (or the original load error from line 225).Step-by-step proof
- Storage initialized; file on disk is an encrypted blob.
load_configreadscontents(encrypted bytes).- Line 236:
contents.decode()raisesUnicodeDecodeError(orjson.loadsraisesJSONDecodeError). - Line 245 catches it:
json_error = err. if json_error:at line 248 is true.decrypt_bufferruns at line 252. Suppose it returns a string that fails to parse as JSON (or returns""from the still-swallowing decrypt path).- Line 261:
json.loads(config_json)raises. - Line 266 catches it. Line 267 sets
decryption_error = True. Line 268 logs. Line 269:raise Exception("Failed to parse decrypted config file ..."). - The raise propagates up. Line 271 is never evaluated. Line 275 catches the exception, logs, re-raises.
The user sees "Failed to parse decrypted config file". They never see "may contain JSON format problems".
Why this is pre-existing, not introduced
Pre-PR, the structure was identical — only
md5 → sha256and the narrowing of try-A's except (line 245) from broadExceptionto(JSONDecodeError, UnicodeDecodeError)changed in this region. The narrowing of try-A is the right KSM-938-adjacent fix (it stops misclassifying KMS errors as JSON errors, addressed elsewhere in this review), but it does not affect the reachability of try-B's except or theif json_error and decryption_error:block — those reside in a sibling try/except, not nested inside try-A. Pre-PR, with broadexcept Exceptionat line 245, the raise at line 269 was also already not caught by that handler, because try-A had already exited beforeif json_error:ran. The dead code predates this PR.Impact
Zero functional impact. The actual user-facing exception ("Failed to parse decrypted config file") is still raised correctly at line 269 and propagates to the caller. The cost is purely dead code: a dead local (
decryption_error), an unreachable branch (lines 271-273), and a never-surfaced error message ("may contain JSON format problems"). Mildly misleading to a future reader — the unreachable branch suggests a code path that does not exist.How to fix
Cheapest cleanup — drop both:
# remove line 234: decryption_error = False # remove line 267: decryption_error = True # remove lines 271-273: the if json_error and decryption_error block
Alternatively, if the "may contain JSON format problems" message was actually intended to surface to the user, move the raise out of try-B's except so
decryption_errorcan flow to line 271:except Exception as err: decryption_error = True self.logger.error(f"Failed to parse decrypted config file: {err}") # raise removed — let flow continue to line 271
Either is one to four lines. Given the PR is already touching this function for the md5→sha256 and except-narrowing changes, it's reasonable cleanup while the file is open. Filing as
pre_existingsince the structural defect was not introduced by this PR. - try-A (lines 235-246): attempts
There was a problem hiding this comment.
An organization admin can view or raise the cap at claude.ai/admin-settings/claude-code. The cap resets at the start of the next billing period.
Once the cap resets or is raised, push a new commit or reopen this pull request to trigger a review.
There was a problem hiding this comment.
An organization admin can view or raise the cap at claude.ai/admin-settings/claude-code. The cap resets at the start of the next billing period.
Once the cap resets or is raised, push a new commit or reopen this pull request to trigger a review.
There was a problem hiding this comment.
An organization admin can view or raise the cap at claude.ai/admin-settings/claude-code. The cap resets at the start of the next billing period.
Once the cap resets or is raised, push a new commit or reopen this pull request to trigger a review.
Security - KSM-764: protobuf upgraded to >=6.33.5, fixing CVE-2026-0994 (JSON recursion DoS, HIGH CVSS 8.2) - KSM-833: cryptography upgraded to >=46.0.5, fixing CVE-2026-26007 (ECDH subgroup attack on SECT curves, HIGH CVSS 8.2) - urllib3 upgraded to 2.6.3, requests to 2.32.4, pyasn1 to 0.6.2 Bug Fixes - KSM-938: initialization no longer silently proceeds with config left unencrypted when service account lacks cloudkms.cryptoKeys.get - KSM-939: KMS errors (network failures, permission denials) now propagate as exceptions - KSM-940: delete_all() no longer leaves the credential file on disk when KMS is unavailable - KSM-941: set() no longer silently swallows PermissionError on read-only config files - KSM-942: change_key() no longer returns True when re-encryption fails; key state rolls back cleanly - KSM-943: AES-GCM nonce corrected to 96-bit per NIST SP 800-38D (was 128-bit); existing encrypted blobs remain readable - KSM-943: MD5 replaced with SHA-256 for config change detection - KSM-944: read_storage() now returns a copy of the config dict instead of a live reference - KSM-944: decrypt_config() autosave default changed from True to False - KSM-945: symmetric decrypt no longer fails when passed a versioned key URI - KSM-946: GCPKeyValueStorage is now thread-safe for concurrent reads and writes - KSM-947: pre-release housekeeping (dead MD5_HASH constant removed, in-workflow test matrix added) - KSM-948: get/set/delete no longer crash with TypeError after bootstrapping from an empty config file Breaking Changes - Python minimum version raised from 3.6 to 3.9.2 - Core SDK minimum version raised from 16.6.6 to 17.2.1 - decrypt_config() autosave default changed from True to False
0f96e56 to
c1ef3da
Compare
Summary
Release branch for v1.1.0 — raises the Python floor to 3.9, aligns the Core dependency with v17.2.1, fixes two HIGH severity CVEs, closes four security bugs, and migrates the publish workflow to PyPI Trusted Publishing (OIDC).
Changes
Bug Fixes
protobuf>=6.33.5direct dependency (CVSS 8.2; protobuf 5.x discontinued, fix requires Python 3.9+)cryptography>=46.0.5direct dependency (CVSS 8.2)cloudkms.cryptoKeys.getwas denied,get_key_details()swallowed the 403 silently,key_purpose_detailswas never set, and__save_config()swallowed the resultingAttributeError— credentials sat on disk as plaintext JSON with no error surfaced.GCPKeyValueStoragenow raises on init when key introspection fails. Merges KSM-938: raise on get_key_details failure to prevent plaintext config #1009.read_storage()live dict reference +decrypt_config()plaintext-on-disk default (KSM-944):read_storage()returnedself.configdirectly, allowing callers to silently corrupt internal state without triggering encryption or a disk write.decrypt_config()defaulted toautosave=True, permanently overwriting the encrypted config file with plaintext credentials when called without arguments.read_storage()now returns a copy;decrypt_config()defaults toautosave=False. Merges fix(storage): fix read_storage() live ref and decrypt_config() plaintext-on-disk default (KSM-944) #1011.Maintenance
keeper-secrets-manager-coredependency floor from>=16.6.6to>=17.2.1importlib_metadatafrom requirements.txt (stdlib since Python 3.8)cloudkms.cryptoKeys.getas a required IAM permissionpublishtoggle for build+SBOM-only runs andrepository-urloverride for TestPyPI dry-runsBreaking Changes
decrypt_config()no longer writes plaintext to disk by default — callers relying on the old default must now passautosave=Trueexplicitly.Related Issues