Route buckets per backend + make the multi-backend functional test real - #22
Merged
Merged
Conversation
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
…to-end The multi-backend functional test configured s3 + azureblob + filesystem but only ever exercised one backend: the chart had no way to route a bucket to a specific backend, so S3Proxy served every bucket from the first (default) backend. The leg passed without touching azureblob or s3 — a false green. Chart: - Per-backend bucket-locator. Move s3proxy.bucket-locator.N out of the shared config block (which copied one global list into every backend file, so S3Proxy's first-file-wins de-dup meant only the first backend honored it) into each backend's own properties, driven by config.backends.<name>.bucketLocators. - Per-backend credentials. Replace the single shared secret.properties (which emitted only the first-enabled backend's jclouds.credential and appended it to every backend file) with secret-common.properties + per-backend secret-<name>.properties; the merge initContainer appends common + the matching backend secret, so each backend gets only its own credential. Also fixes a latent GCS double-"jclouds.credential=" prefix. - azureblob: emit the provider-agnostic jclouds.endpoint (was the jclouds.azureblob.endpoint typo) and render the computed default endpoint when only account is set. Test: - ci/functional/values/multi-backend.yaml routes smoke-s3-*/smoke-az-*/smoke-fs-* to their backends; azureblob uses the azureblob-sdk provider, which signs correctly against Azurite (the legacy azureblob provider mis-signs -> 403). - ci/functional/assert-routing.sh writes one bucket per backend through s3proxy and verifies each object physically lands on its intended backend (MinIO for s3, Azurite for azureblob, filesystem by elimination) and that no bucket leaks into another backend. - Azurite mock runs with --skipApiVersionCheck: the assertion's az CLI on the CI runner sends a newer x-ms-version than the Azurite image knows, which it would otherwise reject (InvalidHeaderValue). The s3proxy azureblob-sdk path is unaffected. - functional-test.yaml: multi-backend and azureblob legs are now hard gates (all soft_fail removed). No Chart.yaml version bump (handled separately), so verify-version fails by design for now. Verified on kind: multi-backend routing exclusive (s3->MinIO, az->Azurite, fs->filesystem); azureblob-only, s3-only, filesystem-only all pass.
CRThaze
force-pushed
the
CRThaze/s3proxy-multi-backend-routing
branch
from
July 21, 2026 10:18
105af9e to
453d241
Compare
📊 Helm Render Diff SummaryChart
🔍 Detailed Changes📝 Changes in
|
📝 README.md PreviewThe following changes to Click to expand diffdiff --git a/README.md b/README.md
index bf652df..dfcde1c 100644
--- a/README.md
+++ b/README.md
@@ -92,6 +92,12 @@ The following section lists the configurable parameters of the s3proxy chart and
<td><code>string</code></td>
<td><code>""</code></td>
</tr>
+ <tr>
+ <td><code>config.backends.azureblob.bucketLocators</code></td>
+ <td>Buckets routed to this backend (S3Proxy bucket-locator; glob patterns supported). Only relevant when multiple backends are enabled; a bucket matching no backend's list falls through to the first-enabled backend.</td>
+ <td><code>list</code></td>
+ <td><code>[]</code></td>
+ </tr>
<tr>
<td><code>config.backends.azureblob.enabled</code></td>
<td>Enable Azure Blob Storage backend</td>
@@ -188,6 +194,12 @@ The following section lists the configurable parameters of the s3proxy chart and
<td><code>string</code></td>
<td><code>""</code></td>
</tr>
+ <tr>
+ <td><code>config.backends.b2.bucketLocators</code></td>
+ <td>Buckets routed to this backend (S3Proxy bucket-locator; glob patterns supported). Only relevant when multiple backends are enabled; a bucket matching no backend's list falls through to the first-enabled backend.</td>
+ <td><code>list</code></td>
+ <td><code>[]</code></td>
+ </tr>
<tr>
<td><code>config.backends.b2.enabled</code></td>
<td>Enable Backblaze B2 backend</td>
@@ -200,6 +212,12 @@ The following section lists the configurable parameters of the s3proxy chart and
<td><code>string</code></td>
<td><code>"/data/s3proxy"</code></td>
</tr>
+ <tr>
+ <td><code>config.backends.filesystem.bucketLocators</code></td>
+ <td>Buckets routed to this backend (S3Proxy bucket-locator; glob patterns supported). Only relevant when multiple backends are enabled; a bucket matching no backend's list falls through to the first-enabled backend.</td>
+ <td><code>list</code></td>
+ <td><code>[]</code></td>
+ </tr>
<tr>
<td><code>config.backends.filesystem.credential</code></td>
<td>jclouds credential. S3Proxy requires jclouds.credential in every backend properties file; the filesystem backend ignores the value. An empty value falls back to "local".</td>
@@ -224,6 +242,12 @@ The following section lists the configurable parameters of the s3proxy chart and
<td><code>bool</code></td>
<td><code>true</code></td>
</tr>
+ <tr>
+ <td><code>config.backends.googleCloudStorage.bucketLocators</code></td>
+ <td>Buckets routed to this backend (S3Proxy bucket-locator; glob patterns supported). Only relevant when multiple backends are enabled; a bucket matching no backend's list falls through to the first-enabled backend.</td>
+ <td><code>list</code></td>
+ <td><code>[]</code></td>
+ </tr>
<tr>
<td><code>config.backends.googleCloudStorage.clientEmail</code></td>
<td>Service account email or user email</td>
@@ -272,6 +296,12 @@ The following section lists the configurable parameters of the s3proxy chart and
<td><code>string</code></td>
<td><code>""</code></td>
</tr>
+ <tr>
+ <td><code>config.backends.openstackSwift.bucketLocators</code></td>
+ <td>Buckets routed to this backend (S3Proxy bucket-locator; glob patterns supported). Only relevant when multiple backends are enabled; a bucket matching no backend's list falls through to the first-enabled backend.</td>
+ <td><code>list</code></td>
+ <td><code>[]</code></td>
+ </tr>
<tr>
<td><code>config.backends.openstackSwift.enabled</code></td>
<td>Enable OpenStack Swift backend</td>
@@ -344,6 +374,12 @@ The following section lists the configurable parameters of the s3proxy chart and
<td><code>string</code></td>
<td><code>""</code></td>
</tr>
+ <tr>
+ <td><code>config.backends.rackspaceCloudfiles.bucketLocators</code></td>
+ <td>Buckets routed to this backend (S3Proxy bucket-locator; glob patterns supported). Only relevant when multiple backends are enabled; a bucket matching no backend's list falls through to the first-enabled backend.</td>
+ <td><code>list</code></td>
+ <td><code>[]</code></td>
+ </tr>
<tr>
<td><code>config.backends.rackspaceCloudfiles.enabled</code></td>
<td>Enable Rackspace Cloud Files backend</td>
@@ -374,6 +410,12 @@ The following section lists the configurable parameters of the s3proxy chart and
<td><code>bool</code></td>
<td><code>true</code></td>
</tr>
+ <tr>
+ <td><code>config.backends.s3.bucketLocators</code></td>
+ <td>Buckets routed to this backend (S3Proxy bucket-locator; glob patterns supported). Only relevant when multiple backends are enabled; a bucket matching no backend's list falls through to the first-enabled backend.</td>
+ <td><code>list</code></td>
+ <td><code>[]</code></td>
+ </tr>
<tr>
<td><code>config.backends.s3.enabled</code></td>
<td>Enable S3 backend</td>
@@ -416,6 +458,12 @@ The following section lists the configurable parameters of the s3proxy chart and
<td><code>string</code></td>
<td><code>""</code></td>
</tr>
+ <tr>
+ <td><code>config.backends.transient.bucketLocators</code></td>
+ <td>Buckets routed to this backend (S3Proxy bucket-locator; glob patterns supported). Only relevant when multiple backends are enabled; a bucket matching no backend's list falls through to the first-enabled backend.</td>
+ <td><code>list</code></td>
+ <td><code>[]</code></td>
+ </tr>
<tr>
<td><code>config.backends.transient.credential</code></td>
<td>jclouds credential. S3Proxy requires jclouds.credential in every backend properties file; the transient backend ignores the value. An empty value falls back to "local".</td>
@@ -446,12 +494,6 @@ The following section lists the configurable parameters of the s3proxy chart and
<td><code>object</code></td>
<td><code>{}</code></td>
</tr>
- <tr>
- <td><code>config.buckets.locator</code></td>
- <td>Assign specific buckets to different backends (glob patterns supported)</td>
- <td><code>list</code></td>
- <td><code>[]</code></td>
- </tr>
<tr>
<td><code>config.cors.allowCredential</code></td>
<td>Allow credentials</td>
@@ -937,17 +979,27 @@ config:
another-bucket: "actual-bucket-name"
```
-### Bucket Locator
+### Bucket Locator (routing buckets to backends)
-Assign specific buckets to different backends:
+When more than one backend is enabled, assign buckets to a specific backend with
+that backend's own `bucketLocators` list. S3Proxy reads bucket-locators per
+backend, so each list is emitted only into that backend's properties file. Glob
+patterns are supported. A bucket that matches no backend's list falls through to
+the first-enabled backend (the default).
```yaml
config:
- buckets:
- locator:
- - "bucket1"
- - "bucket2"
- - "*.test" # Glob patterns supported
+ backends:
+ s3:
+ enabled: true
+ # ... credentials ...
+ bucketLocators:
+ - "prod-*"
+ - "customer-data"
+ filesystem:
+ enabled: true
+ bucketLocators:
+ - "scratch-*" # everything else also lands here (first-enabled default)
```
## Monitoring
|
This was referenced Jul 21, 2026
CRThaze
added a commit
that referenced
this pull request
Jul 21, 2026
Reconciles the chart changes that merged without a version bump (DND-1442 #21, azureblob endpoint + multi-backend routing #22) together with this PR's HPA autoscaling/v2 fix, the helm-polish template cleanup, and the added LICENSE. Clears the verify-version gate and cuts a 0.1.0 release on merge (minor bump: new azureblob regions value + bucketLocators, no breaking changes).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The
multi-backendfunctional leg enabled s3 + azureblob + filesystem in one release but only ever exercised one backend. The chart had no way to route a bucket to a specific backend, so S3Proxy served every bucket from the first (default) backend; the leg passed green without ever touching azureblob or s3. This change makes it actually route to and verify each backend.Chart changes
s3proxy.bucket-locator.Nwas emitted in the shared config block, so the same global list was copied into every backend's properties file. S3Proxy de-dupes locators across--propertiesfiles (first file wins), so only the first backend ever honored them. Now each backend renders its own locators fromconfig.backends.<name>.bucketLocators, and the globalconfig.buckets.locatoris removed.secret.propertiesemitted only the first-enabled backend'sjclouds.credentialand was appended to every backend file, so azureblob got s3's credential. It is replaced withsecret-common.properties(client auth, appended to all) plus per-backendsecret-<name>.properties; the merge initContainer appends common plus the matching backend secret, so each backend gets only its own credential. This also fixes a latent GCSjclouds.credential=jclouds.credential=double-prefix.jclouds.endpoint(previously thejclouds.azureblob.endpointtypo) and render the computed default endpoint when onlyaccountis set.Test changes
ci/functional/values/multi-backend.yamlroutessmoke-s3-*/smoke-az-*/smoke-fs-*to their backends. azureblob uses theazureblob-sdkprovider, which signs correctly against Azurite; the legacyazureblobprovider mis-signs SharedKey and Azurite returns 403 (verified on kind).ci/functional/assert-routing.sh: it writes one bucket per backend through s3proxy and verifies each object physically lands on its intended backend (MinIO via the S3 API for s3, Azurite via theazCLI for azureblob, filesystem by elimination), and that no bucket leaks into another backend's store.--skipApiVersionCheck. The assertion'sazCLI on the CI runner sends a newerx-ms-versionthan the pinned Azurite image knows, which it would otherwise reject (InvalidHeaderValue). The s3proxy azureblob-sdk path is unaffected..github/workflows/functional-test.yaml: themulti-backendandazurebloblegs are now hard gates (allsoft_failremoved); multi-backend runs the routing assertion instead of the plain round-trip.Verification (local kind, end-to-end)
smoke-s3goes to MinIO,smoke-azto Azurite,smoke-fsto filesystem, with no cross-leaks.helm lint,kubeconform -strict(k8s 1.29, all test-values),actionlint,shellcheck: clean.Notes
Chart.yamlversion bump (handled separately), soverify-versionfails by design for now.README.mdfromREADME.md.gotmplon the PR; only the template is edited here.provideris left asazureblob. Whether to makeazureblob-sdkthe default is left to the dedicated azureblob ticket (DND-1416); this PR only sets it in the functional test values.