Repository navigation
fix: sort map keys when rendering Workbench and Connect config - #172
Conversation
Go randomizes map iteration order. Several config renderers walked maps with reflect.MapRange() and wrote named sections in iteration order, so the same spec could render to different bytes on each reconcile. The configmap-sha and secret-sha pod template annotations then changed and rolled the Deployment. Iterate map keys in sorted order in Connect GenerateGcfg (RPackageRepository sections) and in the Workbench renderers for databricks.conf (secret and deprecated configmap field), launcher.kubernetes.profiles.conf and supervisord programs. Output is otherwise unchanged. Add tests that render each config 50 times and assert identical, sorted output.
|
Claude finished @stevenolen's task in 2m 2s —— View job PR Review
The fix is correct and well-motivated.
Inline comments repeat the same reasoning five times. Each loop site now carries a comment explaining map-iteration non-determinism and hash stability. Since Connect test duplicates helpers available in the same package. The No correctness or security issues found. |
| // Go randomizes map iteration order, so config renderers must iterate maps in | ||
| // a fixed order to produce byte-identical output (and stable hash annotations) | ||
| // across reconciles. | ||
| func sortedMapKeys(m reflect.Value) []reflect.Value { |
There was a problem hiding this comment.
The helper is in workbench_config.go but is also called by connect_config.go. Both are in the same package so it compiles fine, but if someone is grepping for where the sort logic lives they won't find it by looking in connect_config.go. A one-line comment on the connect_config.go call site referencing this helper, or moving it to a shared file (e.g. config_util.go), would make the cross-file dependency more discoverable. Not blocking.
| "Internal": {Url: "https://internal.example.com/cran/latest"}, | ||
| }, | ||
| } | ||
|
|
There was a problem hiding this comment.
requireStableRender and requireInOrder in workbench_config_test.go (same package) do exactly what this test implements inline. Suggest refactoring:
| first, err := c.GenerateGcfg() | |
| require.Nil(t, err) | |
| requireStableRender(t, func() map[string]string { | |
| str, err := c.GenerateGcfg() | |
| require.Nil(t, err) | |
| return map[string]string{"gcfg": str} | |
| }) |
Note: requireInOrder could cover the index checks too. If the return type mismatch makes this awkward, extracting a requireGcfgStable wrapper is the next best option. As-is the duplication is minor but it means the loop count would need updating in two places.
Move sortedMapKeys into config_util.go since both the Workbench and Connect renderers use it, and reuse requireStableRender and requireInOrder in the Connect gcfg ordering test.
|
@khusmann I know you are working on a bunch of config-order sensitivity changes in workbench for configuration precedence rules. This PR to making the order stable is definitely a good change, I wanted to get your thoughts about whether there is any additional ordering support we need to contribute to team operator for your upcoming workbench changes. |
Lytol
left a comment
There was a problem hiding this comment.
Jeeeeesh. That was a much needed fix. 🥇
## [1.32.1](v1.32.0...v1.32.1) (2026-10-02) ### Bug Fixes * sort map keys when rendering Workbench and Connect config ([#172](#172)) ([8e4317f](8e4317f))
|
@statik Yeah, our Helm charts had the same unstable ordering bug this PR addresses (in the PAM volume mounts). rstudio/helm#955 fixes it and also refreshes our snapshot lockfiles, which had drifted over the last few years because of a tooling issue (rstudio/helm#952). The I just went through team-operator's renderers to see if there were any areas with similar problems. Looks like |
Description
Go randomizes map iteration order. Several config renderers walk maps with
reflect.Value.MapRange()and append named sections in iteration order without sorting. When a map has two or more entries, the same spec can render to more than one byte sequence. The rendered content is hashed into pod template annotations, so the annotation flips between values and the Deployment rolls.Affected outputs:
rstudio-connect.gcfg:[RPackageRepository "X"]sections. The site controller always sets two repositories (CRAN and RSPM), so every Connect Deployment created from a Site is affected (connect.posit.team/configmap-sha).databricks.confin the config Secret: affects Deployments using two or more Databricks workspaces (workbench.posit.team/secret-sha).databricks.conffield in the main ConfigMap, andlauncher.kubernetes.profiles.confwith two or more profiles (workbench.posit.team/configmap-sha).programs.confwith two or more programs (rendered ConfigMap content; same pattern, fixed for consistency).We confirmed this on a running cluster: swapping the order of the two RPackageRepository sections reproduces the configmap-sha of the other ReplicaSet exactly.
Why it loops
Each reconcile has roughly a 50% chance (with two entries) of rendering the other order. A changed hash updates the pod template, which creates a new ReplicaSet. The Deployment status updates from that rollout come back through the Deployment watch and trigger another reconcile, which flips the hash again. The result is repeated rollouts with no spec change.
Commit 745819c ("fix: prevent non-deterministic reconcile loops (#121)") sorted map iteration in volumes, secrets and subdir handling but missed these renderers.
Code Flow
sortedMapKeys(reflect.Value)inworkbench_config.goreturnsMapKeys()sorted by string form.ConnectConfig.GenerateGcfg,WorkbenchSecretIniConfig.GenerateConfigMap,WorkbenchIniConfig.GenerateConfigMap(default map branch),WorkbenchProfilesConfig.GenerateConfigMapandWorkbenchConfig.GenerateSupervisorConfigmapiterate the sorted keys and useMapIndexinstead ofMapRange.[*]still sorts first.GenerateGcfgthat claimed reflection order was deterministic for every field.After upgrading, each affected Deployment may roll once if its current hash came from a non-sorted order. After that the hash stays stable.
Category of change
Checklist
just testand all tests passTests
New tests render each affected config 50 times with three map entries, assert the output is identical each time, and assert the sections come out in sorted order. All of them fail against the previous code and pass with this change.
make test(manifests, generate, vet, envtest, race detector) passes. Plainjust testcould not start envtest locally (no kubebuilder binaries installed), so thejust testbox is left unchecked; CI will confirm.