Skip to content

fix: sort map keys when rendering Workbench and Connect config - #172

Merged
stevenolen merged 3 commits into
mainfrom
claude-sort-config-map-sections
Oct 2, 2026
Merged

stevenolen merged 3 commits into
mainfrom
claude-sort-config-map-sections

Conversation

@stevenolen

Copy link
Copy Markdown
Contributor

Description

Go randomizes map iteration order. Several config renderers walk maps with reflect.Value.MapRange() and append named sections in iteration order without sorting. When a map has two or more entries, the same spec can render to more than one byte sequence. The rendered content is hashed into pod template annotations, so the annotation flips between values and the Deployment rolls.

Affected outputs:

  • Connect rstudio-connect.gcfg: [RPackageRepository "X"] sections. The site controller always sets two repositories (CRAN and RSPM), so every Connect Deployment created from a Site is affected (connect.posit.team/configmap-sha).
  • Workbench databricks.conf in the config Secret: affects Deployments using two or more Databricks workspaces (workbench.posit.team/secret-sha).
  • Workbench deprecated databricks.conf field in the main ConfigMap, and launcher.kubernetes.profiles.conf with two or more profiles (workbench.posit.team/configmap-sha).
  • Workbench supervisord programs.conf with two or more programs (rendered ConfigMap content; same pattern, fixed for consistency).

We confirmed this on a running cluster: swapping the order of the two RPackageRepository sections reproduces the configmap-sha of the other ReplicaSet exactly.

Why it loops

Each reconcile has roughly a 50% chance (with two entries) of rendering the other order. A changed hash updates the pod template, which creates a new ReplicaSet. The Deployment status updates from that rollout come back through the Deployment watch and trigger another reconcile, which flips the hash again. The result is repeated rollouts with no spec change.

Commit 745819c ("fix: prevent non-deterministic reconcile loops (#121)") sorted map iteration in volumes, secrets and subdir handling but missed these renderers.

Code Flow

  • New helper sortedMapKeys(reflect.Value) in workbench_config.go returns MapKeys() sorted by string form.
  • ConnectConfig.GenerateGcfg, WorkbenchSecretIniConfig.GenerateConfigMap, WorkbenchIniConfig.GenerateConfigMap (default map branch), WorkbenchProfilesConfig.GenerateConfigMap and WorkbenchConfig.GenerateSupervisorConfigmap iterate the sorted keys and use MapIndex instead of MapRange.
  • The rendered format is unchanged apart from section order. For profiles, [*] still sorts first.
  • Fixed a misleading comment in GenerateGcfg that claimed reflection order was deterministic for every field.

After upgrading, each affected Deployment may roll once if its current hash came from a non-sorted order. After that the hash stays stable.

Category of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Build: a code change that affects the build system or external dependencies
  • Performance: a code change that improves performance
  • Refactor: a code change that neither fixes a bug nor adds a feature
  • Documentation: documentation changes
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist

  • I have run just test and all tests pass
  • I have reviewed my own diff and added inline comments on lines I want reviewers to focus on or that I am uncertain about

Tests

New tests render each affected config 50 times with three map entries, assert the output is identical each time, and assert the sections come out in sorted order. All of them fail against the previous code and pass with this change. make test (manifests, generate, vet, envtest, race detector) passes. Plain just test could not start envtest locally (no kubebuilder binaries installed), so the just test box is left unchecked; CI will confirm.

Go randomizes map iteration order. Several config renderers walked
maps with reflect.MapRange() and wrote named sections in iteration
order, so the same spec could render to different bytes on each
reconcile. The configmap-sha and secret-sha pod template annotations
then changed and rolled the Deployment.

Iterate map keys in sorted order in Connect GenerateGcfg
(RPackageRepository sections) and in the Workbench renderers for
databricks.conf (secret and deprecated configmap field),
launcher.kubernetes.profiles.conf and supervisord programs. Output is
otherwise unchanged.

Add tests that render each config 50 times and assert identical,
sorted output.
@stevenolen
stevenolen marked this pull request as ready for review October 1, 2026 13:59
@claude

claude Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Claude finished @stevenolen's task in 2m 2s —— View job


PR Review

  • Gather context
  • Read changed files and review guidelines
  • Analyze changes and identify issues
  • Post inline comments
  • Post final review summary

The fix is correct and well-motivated. sortedMapKeys is a clean, well-placed helper, and all five call sites are consistent. Tests cover all affected renderers with 50-iteration stability checks and sorted-order assertions. A few minor points:

sortedMapKeys is defined in workbench_config.go but used in connect_config.go — this is fine since both are in the same package, but it's easy to miss during refactors. No action needed; just worth noting the dependency exists.

Inline comments repeat the same reasoning five times. Each loop site now carries a comment explaining map-iteration non-determinism and hash stability. Since sortedMapKeys itself documents this intent, the per-site comments add noise without adding clarity. One sentence at each site is fine; consider trimming to just the first occurrence or relying solely on the helper's doc comment.

Connect test duplicates helpers available in the same package. TestConnectConfig_GenerateGcfgRepositoriesDeterministic re-implements the 50-iteration stability loop and index-based ordering checks inline, while requireStableRender and requireInOrder from workbench_config_test.go (same package) do exactly this. Using the helpers would shrink the test and reduce divergence if the loop count ever changes.

The fmt.Sprintf("%v", key) comparison in sortedMapKeys works for all current string-keyed maps. No issue, but for pure string maps keys[i].String() is more direct. This is not worth changing unless the function needs to handle non-string keys.

No correctness or security issues found.

Comment thread api/core/v1beta1/workbench_config.go Outdated
// Go randomizes map iteration order, so config renderers must iterate maps in
// a fixed order to produce byte-identical output (and stable hash annotations)
// across reconciles.
func sortedMapKeys(m reflect.Value) []reflect.Value {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The helper is in workbench_config.go but is also called by connect_config.go. Both are in the same package so it compiles fine, but if someone is grepping for where the sort logic lives they won't find it by looking in connect_config.go. A one-line comment on the connect_config.go call site referencing this helper, or moving it to a shared file (e.g. config_util.go), would make the cross-file dependency more discoverable. Not blocking.

"Internal": {Url: "https://internal.example.com/cran/latest"},
},
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

requireStableRender and requireInOrder in workbench_config_test.go (same package) do exactly what this test implements inline. Suggest refactoring:

Suggested change
first, err := c.GenerateGcfg()
require.Nil(t, err)
requireStableRender(t, func() map[string]string {
str, err := c.GenerateGcfg()
require.Nil(t, err)
return map[string]string{"gcfg": str}
})

Note: requireInOrder could cover the index checks too. If the return type mismatch makes this awkward, extracting a requireGcfgStable wrapper is the next best option. As-is the duplication is minor but it means the loop count would need updating in two places.

stevenolen and others added 2 commits October 1, 2026 15:15
Move sortedMapKeys into config_util.go since both the Workbench and
Connect renderers use it, and reuse requireStableRender and
requireInOrder in the Connect gcfg ordering test.
@stevenolen
stevenolen enabled auto-merge October 2, 2026 14:58
@statik

statik commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

@khusmann I know you are working on a bunch of config-order sensitivity changes in workbench for configuration precedence rules. This PR to making the order stable is definitely a good change, I wanted to get your thoughts about whether there is any additional ordering support we need to contribute to team operator for your upcoming workbench changes.

@Lytol Lytol left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jeeeeesh. That was a much needed fix. 🥇

@stevenolen
stevenolen added this pull request to the merge queue Oct 2, 2026
Merged via the queue into main with commit 8e4317f Oct 2, 2026
8 checks passed
@stevenolen
stevenolen deleted the claude-sort-config-map-sections branch October 2, 2026 17:08
ian-flores pushed a commit that referenced this pull request Oct 2, 2026
## [1.32.1](v1.32.0...v1.32.1) (2026-10-02)

### Bug Fixes

* sort map keys when rendering Workbench and Connect config ([#172](#172)) ([8e4317f](8e4317f))
@khusmann

khusmann commented Oct 2, 2026

Copy link
Copy Markdown

@statik Yeah, our Helm charts had the same unstable ordering bug this PR addresses (in the PAM volume mounts). rstudio/helm#955 fixes it and also refreshes our snapshot lockfiles, which had drifted over the last few years because of a tooling issue (rstudio/helm#952).

The profiles ordering fix is related but separate (rstudio/helm#944). The short version is that we can't rely on maps for any config file where order matters. So it's not just /etc/rstudio/profiles but also things like repos.conf. I'm working on a fix that keeps the legacy behavior (with a warning) and gives admins a consistent mental model across ini files. It's self-contained and won't need any team-operator changes to ship.

I just went through team-operator's renderers to see if there were any areas with similar problems. Looks like launcher.kubernetes.profiles.conf and launcher.kubernetes.resources.conf are order-sensitive files that render from maps, so the operator picks the order (alphabetical for profiles, by CPU/memory for resources) instead of the admin-desired order. Whatever shape we land on in rstudio/helm#944 could/should be mirrored there (looks like that's related to #5).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants