security: cache derived SCRAM stored key #31054
Open
nguyen-andrew wants to merge 4 commits into
Open
Conversation
Measures the per-call cost of the SCRAM password check that HTTP Basic auth runs on every pandaproxy/schema registry/admin request. The salted password is rederived per request (4096 HMAC rounds), which saturated pandaproxy CPU under unbatched HTTP produce in a production incident: ~473us per SHA-256 validation, so a few thousand requests per second exhaust a core before any request work happens. Covers the wrong password (reject) path, which costs the same as an accept, SHA-512 (~3x SHA-256), and a unique-password case that always pays the full derivation.
Exposes the password -> stored_key derivation as a reusable step so a subsequent commit can memoize it.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a per-shard memoization layer for SCRAM password validation to eliminate repeated PBKDF2-style derivations on hot authentication paths (HTTP Basic auth and SASL/PLAIN), controlled via a new cluster configuration kill switch.
Changes:
- Add
security::scram_credential_cacheand integrate it intovalidate_scram_credentialwith athread_localper-shard cache gated byscram_credential_cache_enabled. - Refactor SCRAM algorithm code to expose
derive_stored_key()and reuse it in password validation. - Add unit tests and an rpbench benchmark covering cached vs uncached behavior and correctness.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/v/security/tests/scram_credential_cache_test.cc | New unit tests validating cache hit/miss behavior, key composition, eviction bounds, and cached credential validation. |
| src/v/security/tests/scram_credential_bench.cc | New perf benchmark exercising repeated/unique password validation cases to quantify caching impact. |
| src/v/security/tests/scram_algorithm_test.cc | Adds a test ensuring derive_stored_key() matches stored keys produced by credential creation. |
| src/v/security/tests/BUILD | Wires new test and benchmark targets into the build. |
| src/v/security/scram_credential_cache.h | Declares the per-shard SCRAM derivation cache and its keying/metrics interface. |
| src/v/security/scram_credential_cache.cc | Implements HMAC-keyed cache entries backed by utils::chunked_kv_cache. |
| src/v/security/scram_authenticator.h | Documents cached validation behavior and introduces a detail::validate_scram_credential overload for caller-provided caches. |
| src/v/security/scram_authenticator.cc | Implements cached derivation path and config-gated use of a per-thread cache. |
| src/v/security/scram_algorithm.h | Adds derive_stored_key() and refactors validate_password() to use it. |
| src/v/security/BUILD | Adds the new cache implementation to the security library and links required deps. |
| src/v/config/configuration.h | Declares the new scram_credential_cache_enabled configuration property. |
| src/v/config/configuration.cc | Defines the new configuration property and its description/default. |
Per-shard memoization of SCRAM password derivations, keyed by an HMAC (random per-instance key) of (mechanism, password, salt, iterations) so plaintext passwords are not retained, holding only the derived stored key: the same datum already persisted in the credential store. The mapping is purely functional, so entries never go stale: updating a credential generates a fresh salt, which changes the cache key. Built on the S3-FIFO chunked_kv_cache, whose scan resistance keeps one-shot wrong-password entries from displacing hot legitimate ones.
Member
Author
|
Force push to address Copilot comments. |
Collaborator
CI test resultstest results on build#86904
test results on build#86936 |
nguyen-andrew
requested review from
bartoszpiekny-redpanda,
dotnwat,
mnajda-redpanda and
pgellert
Contributor
|
/microbench |
Collaborator
|
Performance change detected in https://buildkite.com/redpanda/redpanda/builds/86918#019f4614-84b6-4266-9e37-2288a79527c5: See https://redpandadata.atlassian.net/wiki/x/LQAqLg for docs |
validate_scram_credential now consults a shard-local scram_credential_cache, skipping the salted password derivation for repeat authentications. HTTP Basic auth (pandaproxy, schema registry, admin API) and SASL/PLAIN pay that derivation on every request; in production, unbatched HTTP produce through pandaproxy at a few hundred requests per second saturated a core on it. Cache hits skip the derivation entirely; the uncached path is unchanged. The scram_credential_cache_enabled cluster config (default true) is an operational kill switch.
Member
Author
|
Force push to change default behavior to disabled. |
nguyen-andrew
changed the title
Collaborator
Retry command for Build#86936please wait until all jobs are finished before running the slash command |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
HTTP Basic auth (PandaProxy, Schema Registry, Admin API) and Kafka SASL/PLAIN
rederive the credential's stored SCRAM key on every request: a PBKDF2-style
loop of >=4096 HMAC rounds, run on-reactor before the body is parsed. Under
high-rate authenticated traffic this dominates CPU. A cluster serving a few
hundred HTTP-produce req/s per shard through PandaProxy saturated its reactor
core almost entirely in this path (~45% of on-CPU time in
validate_scram_credential), starving heartbeats.This memoizes the derivation with a bounded per-shard cache, keyed by an HMAC
of (algorithm, password, salt, iterations) and holding only the derived stored
key (already in the credential store). It is purely functional, so a password
change re-salts and re-keys and entries never go stale.
The cache is gated by the
scram_credential_cache_enabledcluster config andships disabled by default. Because it caches the result of password
validation in memory, enabling it by default is deferred pending security team
review; until then it is strictly opt-in and out-of-the-box behavior is
unchanged.
Fixes CORE-16829
Security considerations (for review)
credential store; no plaintext or salted password is retained.
random per-boot key, so a memory dump exposes no offline-crackable material
and keys can't be correlated across shards or reboots.
stored key, so a rotated credential (re-salted, hence re-keyed) or a
cache-key collision fails closed. A false accept would require colliding both
the HMAC key and the SCRAM stored key at once — two independent SHA
collisions, one under a secret key — negligible and no weaker than SCRAM's
existing assumptions.
correct and wrong passwords are cached, so a hit only reveals that an exact
(user, password) was tried recently — not whether it is correct. Correctness
still comes from the 200/401 response, and distinct guesses (a dictionary
attack) always miss. Caching failed validations also keeps bad-credential
floods cheap.
Microbench (
--config=release, single core)Repeat = same credential per call (steady-state auth); unique = fresh password
per call (cache-miss path).
End-to-end (single core, unbatched ~2 KiB HTTP produce)
The cache is disabled by default, so out-of-the-box throughput is the Disabled
column. Enabling it (
scram_credential_cache_enabled=true) raises Basic-auththroughput to match the no-auth baseline; toggling it back off returns to the
Disabled numbers, so the config gates the behavior cleanly in both directions.
Backports Required
Release Notes