Repository navigation
Description
Before submitting
- I have searched existing feature requests and discussions to confirm this has not been requested
Problem or motivation
devsy workspace up for a devcontainer with a prebuilt image: reads that image's config from the registry on the machine that runs the CLI. On the kubernetes provider the node pulls the image, and in many clusters the cluster already has its own registry credentials (a node role, a service account's imagePullSecrets, or a secret an admin manages). The client still needs read access to the registry, though only for the image config.
The kubernetes provider has no imageBackend, so inspectImage (pkg/devcontainer/inspect.go) calls image.GetImageConfigForArch. Outside a pod, GetKeychain (pkg/image/auth.go) returns authn.DefaultKeychain, which uses the client's Docker credentials. Devsy then uses only five fields: User, Env, Labels (for devcontainer.metadata), Entrypoint and Cmd.
When the client can't read the registry, up fails before anything happens in the cluster:
Error: start workspace: build image: get image build info: failed to get image
config for architecture amd64: retrieve image <registry>/<repo>:<tag>:
GET https://<registry>/v2/<repo>/manifests/<tag>: unexpected status code
401 Unauthorized
Who benefits: teams on shared clusters. People who start workspaces should need little more than access to their own namespace. Today each of them also needs read access to the registry and a docker login on their laptop (renewed every few hours for short-lived cloud registry tokens). That access allows pulling the whole image, which goes beyond the config Devsy actually reads.
Seen on CLI v1.24.0; code read at 61b3c83 (v1.25.0-beta.6).
Proposed solution
Accept the image details from the user, so the CLI doesn't contact the registry for the inspect step. For example, a provider option or flag such as --image-details <file> that takes the JSON of config.ImageDetails. When it is set, inspectImage skips the registry entirely and fails loudly if the file is missing or invalid. The team that builds the image can produce the file in CI next to the image digest.
This is the smallest change, and it stays predictable because the CLI never tries the registry.
For least privilege, pin the image by digest so the image details always match the image the node pulls.
Scope: a prebuilt image: with no features on the kubernetes provider. On this path extendImage returns without building, so the image config is the only thing the client needs from the registry. Features, Dockerfile builds and prebuilds are out of scope. The existing pull-secret path (EnsurePullSecret copying client credentials when kubernetesPullSecretsEnabled is on) stays as it is.
Alternatives considered
-
Inspect from the cluster (no change for users, but weaker on least privilege). Read the image config inside the cluster and return the five fields to the CLI. As far as I can tell, the kubernetes driver doesn't run a helper pod or job today, so this needs a short-lived pod, RBAC, waiting and cleanup. It could be an
imageBackendfor the kubernetes driver, which would also covergetImageTag.
Downside: the helper pod would hold usable registry credentials, and the in-cluster keychain (kubernetesauth.NewInCluster) reads the service account and itsimagePullSecretsthrough the API, so that identity needsgeton secrets. Anyone who can create pods in the namespace could then use those credentials. With--image-details, devs need no registry access and no extra RBAC. -
Fall back to devcontainer.json alone (opt-in only). When the inspect fails and
remoteUseris set, continue with only devcontainer.json values and warn thatdevcontainer.metadatawasn't merged. Risky as a silent fallback, because missing metadata changes lifecycle behaviour without any visible error.
Workaround today: give users read-only pull access to the repository, plus a registry login on the client.
Area
Providers, Dev containers
Acceptance criteria
- A Kubernetes workspace with a prebuilt
image:(no features) starts when the client has no credentials for the image's registry, as long as the cluster can pull the image. - Remote user, env, entrypoint/cmd and
devcontainer.metadatalabels are applied the same as when the inspect runs on the client. - Invalid or missing image details produce a clear error, not a silent fallback.
- Existing behaviour is unchanged when the option isn't used.
- Documented in the kubernetes provider docs.
Additional context
Requested upstream in loft-sh/devpod#1888 ("Cannot use a private repo with devpod in kubernetes"), which was closed by the stale bot without a response.
Related: inspectImage first calls TargetArchitecture (pkg/driver/kubernetes/target_architecture.go), which lists every node unless the architecture provider option is set. That also needs a cluster-scoped permission. Setting architecture avoids it today, but it may be worth documenting next to the fix.
Activity
wsmoak commentedon Oct 30, 2026
WIP on branch main...wsmoak:devsy:feat/k8s-image-details
It is working locally on k8s in Docker Desktop.
wsmoak commentedon Oct 30, 2026
IMAGE_DETAILS comes from the Go type name. IMAGE_CONFIG_FILE would say more clearly what to pass.
skevetter commentedon Oct 31, 2026
Implementation plan: Kubernetes image configuration supplied by the operator
Status: still outstanding. The current main branch still inspects the image from the client. A contributor implementation exists on
wsmoak:feat/k8s-image-details, reviewed ate20ee457e539eedc43cebd387f5a747cce4f5cb5. No corresponding open or merged PR was found in the issue timeline or searches for this issue and branch. Continue that work on this issue rather than creating a duplicate feature.Main baseline:
9bc12ee041ca854c6f81de95fd8c6e49d7cc028c, reviewed 30 October 2026. The requirements below are proposed changes. This investigation inspected source and the contributor branch; it did not execute a Kubernetes reproduction.1. Verified path and gaps
drivercreate.New/newImageBackendgetImageBuildInfoFromImagecallsinspectImage, which obtains the image configuration from the registry. Outside a pod, the keychain is the client's default Docker keychain.build.go:250-281,inspect.go,auth.go:53-66extendImage,GetExtendedBuildInfoTargetArchitecturelists cluster nodes unlessARCHITECTUREis configured. Agent binary delivery also calls it.target_architecture.go,setup.go:191-220ensurePullSecrets/finalizePodSpec,pullsecrets.gorunPod,optionsEqualThe contributor branch already makes the useful architectural choice: an optional
ImageDetailsDrivercapability and a prebuilt-image override inbuild.go, withIMAGE_CONFIG_FILEwired through provider/options types. Retain that approach.Before merging, close these concrete branch gaps:
loadImageDetailsaccepts an arbitrary supplied ID, cannot establish which requested image the file describes, and does not validate OS/architecture. A provider setting reused for another workspace can silently apply the wrong user, environment or metadata.configmember and must be handled by structure, not by the truthiness of its values.2. Product contract and bounded scope
Use the contributor's clarified name,
IMAGE_CONFIG_FILE, as an advanced Kubernetes provider option. Reuse the existing provider-options path in CLI and Desktop; do not add a competing global--image-detailsflag or a second Desktop launch form.The first version supports an explicitly selected, digest-pinned
devcontainer.jsonimage:with no Features and no image build. The cluster pulls the image using its existing node credentials, service account or explicitly configured pull-secret references. Devsy uses the supplied file for the five image fields it already consumes:User,Env,Labels,Entrypoint, andCmd.Require
ARCHITECTUREto be set for this mode and to match the file. This avoids both inspection-time and agent-delivery node enumeration. On a heterogeneous cluster the operator also uses the existingNODE_SELECTOR/pod template to schedule onto the matching architecture;ARCHITECTUREalone is not a scheduler constraint. Do not introduce an implicit node-selector change as part of this feature.Example after implementation, with client credential injection explicitly disabled for the active context:
devsy context set --option SSH_INJECT_DOCKER_CREDENTIALS=false devsy workspace up ./project --provider kubernetes --ide none \ --provider-option IMAGE_CONFIG_FILE=/absolute/path/project-image.json \ --provider-option ARCHITECTURE=amd64 \ --provider-option KUBERNETES_PULL_SECRETS_ENABLED=falsedevsy context set [context-name] --option KEY=VALUEis the existing context command syntax (cmd/context/set_options.go). The first command changes the selected context's forwarding preference; it is not a new per-workspace flag. Configure bothSSH_INJECT_DOCKER_CREDENTIALS=falseandKUBERNETES_PULL_SECRETS_ENABLED=falsefor the no-client-credential-read acceptance flow. Disabling Kubernetes pull-secret creation alone does not disable the separate SSH credential-injection service.Use an absolute path in the process running the Kubernetes provider. The built-in provider is local, so this is the client machine, including Desktop's local CLI child. Validate absolute Windows paths with the native path API. Do not interpret this as a container path, shell expression, URL, project-relative remote path or Kubernetes Secret name.
When the option is unset, preserve current inspection, credential, build and error behavior. When it is set, invalid input is a hard configuration error and must never trigger a registry fallback. The guarantee covers Devsy's target-image inspection, feature retrieval and build paths in this supported mode. Kubernetes API calls, agent downloads and networking explicitly requested by a user's lifecycle commands remain separate operations.
Reject these combinations before image/feature registry access or pod/PVC changes: Dockerfile/build configuration, Compose, attached existing-container sources, nonempty effective Features including CLI-added Features, forced Dockerless, recovery mode, prebuild-repository lookup, build/push/export requests, and standalone
workspace build. Existing baked-in Feature metadata in the image remains supported: those Features are already installed and are merged as metadata, not fetched again. Error messages identify the unsupported input and instruct the operator to removeIMAGE_CONFIG_FILEor use a plain prebuilt image.Ordinary
upon an existing workspace validates the configured sidecar but retains the existing saved runtime configuration and metadata. Editing the sidecar must not silently restart a running pod or reapply image metadata. Explicit recreation adopts new validated details through the normal recreation flow. Missing or invalid sidecars must not preventstatus,logs,stopordelete.3. Explicit file format and trust boundary
Adopt one documented version-1 wrapper rather than treating every vaguely similar JSON object as the same format. Reuse
config.ImageDetailsConfigfor the five-field payload.{ "schemaVersion": 1, "image": "registry.example.com/team/project@sha256:<64-hex-digest>", "os": "linux", "architecture": "amd64", "config": { "User": "vscode", "Env": ["PATH=/usr/local/bin:/usr/bin:/bin", "EXAMPLE=a=b"], "Labels": { "devcontainer.metadata": "[{\"remoteUser\":\"vscode\"}]" }, "Entrypoint": [], "Cmd": [] } }The digest above is a placeholder, not a runnable image reference. Optional
variantuses the OCI platform convention. The image's build/publish pipeline creates this artifact from the same selected platform image it publishes, then distributes it with that digest. Document a conversion fromcrane config --platform ... <digest-reference>output to the wrapper; raw OCI configuration lacks the manifest-reference binding and is not accepted directly by this version.Validation rules:
configobject, a Linux platform and a fully qualified digest-pinned image reference. Reject trailing JSON, arrays, null, unknown wrapper/payload fields and type mismatches. Emptyconfig: {}is structurally valid and preserves existing defaults.image:; reject tag-only references. Do not interpret Docker inspect's image/config ID as a manifest reference.containerd/platformsdependency; compare the architecture with configuredARCHITECTURE. Reject an unsupported OS or contradictory platform before choosing the agent binary. Document one file per selected platform; a manifest-list digest does not make an amd64 config valid on arm64.Envat every equals sign, expand shell text, or normalize entrypoint/command arguments.devcontainer.metadatais present and nonempty, validate its JSON form before calling the existing metadata merge. Accept an object or array of objects, including an empty array; reject malformed JSON or null entries. CurrentGetImageMetadatacan warn and ignore malformed metadata, which is inappropriate for this explicit authoritative input. Keep existing behavior unchanged outside this mode.ImageDetails.IDfrom the validated requested image reference, not arbitrary file input. Populate OS/architecture/variant and the five Config fields. UseimageBuildInfoFromDetails,GetImageMetadata,GetDevContainerMetadataandMergeConfigurationto retain current user/environment/lifecycle precedence.The file is trusted workspace configuration. Its metadata can contain lifecycle commands, privileges, mounts and environment values just as metadata fetched from an image can. Matching the declared digest prevents accidental cross-image use; it does not cryptographically prove that the supplied config bytes belong to that registry image. The publishing pipeline is responsible for producing the correct artifact. Do not claim registry verification without actually performing it. Do not log or upload the raw file.
4. Exact implementation tasks
A. Provider contract
IMAGE_CONFIG_FILEin the Advanced Options group inproviders/kubernetes/provider.yaml, with default unset and concise help covering local absolute path, supported prebuilt-image scope, versioned format and matchingARCHITECTURE.ImageConfigFile stringwithjson:"imageConfigFile,omitempty"toProviderKubernetesDriverConfiginpkg/provider/provider.go; resolve it inpkg/options/resolve.go:resolveAgentKubernetesConfig. Preserve unrelated current fields.pkg/driver/kubernetes/run.go, stripImageConfigFilefrom both operands inoptionsEqual. Serialize a copied provider-options value with this field blank inrunPod. Do not mutate the live options object. The path alone must not cause pod replacement.B. Parser and optional capability
ImageDetailsDrivercapability inpkg/driver/types.goand the compile-time assertion inpkg/driver/kubernetes/driver.go. Returnnil, nilonly when the provider option is unset. A configured failure returns an error, never an empty successful fallback.pkg/driver/kubernetes/image_details.goor a clearly named replacementimage_config.go. Keep file reading, JSON/platform/reference validation and conversion toImageDetailsas focused helpers.NewKubernetesDriver: that constructor is also used for management operations that must work when the sidecar is gone.C. Runner integration and strict opt-in behavior
pkg/devcontainer/run.go:Up, after effective config/overlay selection is available and beforedispatchByConfigKind, invoke the supplied-details preparation helper. It performs the complete scope and file checks for both fresh and existing workspaces. It must run beforerunSingleContainerstarts the agent-prefetch goroutine. User initialization keeps its current ordering; do not claim rollback of user-suppliedinitializeCommand.pkg/devcontainer/build.go:getImageBuildInfoFromImage, reuse that validated snapshot for the exact selected image and pass it toimageBuildInfoFromDetails. Call ordinaryinspectImageonly when the capability says the option was unset.feature.GetExtendedBuildInfocan fetch a Feature. Inspect the substituted/overlay/CLI-added configuration, not just the original JSON text.pkg/devcontainer/prebuild.go:Build, reject configured-file mode before standalone build, prebuild lookup or publication. Reuse the same capability/request semantics instead of reaching into a concrete Kubernetes type from generic runner code.single.go:buildNewContainerConfig. Invalid input onup --recreatemust leave the running pod and PVC untouched.ARCHITECTUREand file agreement before anyTargetArchitecturecall from this launch; retain existing target-architecture behavior outside this mode.inspectImagefor ordinary Dockerfile bases, intermediate images and other providers. Do not return the supplied details for arbitrary inspection requests.D. Pull credentials and diagnostics
KUBERNETES_PULL_SECRETS_ENABLEDand contextSSH_INJECT_DOCKER_CREDENTIALSbehavior. Document setting both to false for cluster-managed authorization without client Docker credential reads. Use the existingdevsy context set [context-name] --option SSH_INJECT_DOCKER_CREDENTIALS=falsecommand for the context setting and the existing provider option for pull-secret creation. Do not silently change an operator's settings when enablingIMAGE_CONFIG_FILE.imagePullSecrets; never replace an admin-managed reference merely because supplied image config is enabled.ImagePullBackOff; never suggest clientdocker loginas the remedy for a missing cluster pull authorization in this mode.5. Tests and observable acceptance
Deterministic unit/contract coverage
Extend the contributor's
pkg/driver/kubernetes/image_details_test.goandpkg/devcontainer/image_details_test.go:Id, platform mismatch, missingARCHITECTURE, and multi-platform fixtures.=, literal entrypoint/command argv, remote/container user precedence, valid inherited lifecycle metadata and malformed metadata failure.upvalidates the input but preserves the saved pod/runtime metadata until recreation. Moving an otherwise identical local file does not replace a healthy pod.SSH_INJECT_DOCKER_CREDENTIALS=falseandKUBERNETES_PULL_SECRETS_ENABLED=false, client credential-helper and cluster registry-secret read spies must not be called.GitHub Actions integration
Extend the contributor's
e2e/tests/up/provider_kubernetes_image_config_file.goand register its label inhack/ci/suites.yaml. Use the existing Kubernetes harness and owned CI fixtures.ARCHITECTURE, contextSSH_INJECT_DOCKER_CREDENTIALS=falseand providerKUBERNETES_PULL_SECRETS_ENABLED=false. The test identity must lack permission to read the admin's image-pull secret while retaining Devsy's normal namespace/daemon-secret permissions.upsucceeds, the pod runs the expected digest, and the registry records no anonymous/client inspection attempts during the launch interval. The node's authenticated image pull must actually occur; a public image or a cache-only success is insufficient.overrideCommand: falseand timeout-disabled policy are needed to observe an image entrypoint.Run focused Go tests and lint, then the registered CI scenario against the actual candidate CLI and delivered agent. Source-level fake-inspector tests and a file-injected environment marker alone do not complete customer acceptance.
6. Documentation and delivery
sites/docs-devsy-sh/content/docs/managing-providers/kubernetes-registry-access.mdx, register it in that directory'smeta.json, and link it frommanage-providers.mdx.ARCHITECTUREversus scheduling, per-workspace--provider-option, stable absolute paths for Desktop, existing cluster pull configuration, baseline RBAC, and explicit recreation for changed image metadata.developing-providers/driver.mdxto match the final request/result contract and strict fallback rules.Completion: a namespace-scoped client without image-registry credentials starts a digest-pinned prebuilt workspace using cluster pull authorization, obtains the same user/environment/metadata/entrypoint behavior as authenticated inspection, and gets deterministic errors for invalid or unsupported file mode. No helper pod or additional registry-secret access is introduced.
feat(kubernetes): read prebuilt image config from IMAGE_DETAILS file
wsmoak commentedon Oct 31, 2026
I've rebased the branch onto current main (3242016) and fixed these:
WIP remains on branch main...wsmoak:devsy:feat/k8s-image-details
I still need to test the latest changes with a full setup (the original set worked fine) so this isn't ready to review yet.
The commits are now signed. This is 100% Claude Code + Opus 5.5 generated.
wsmoak commentedon Oct 31, 2026
I've tested with these commits cherry picked on top of v1.24.0 with my AWS EKS setup and it all works fine for me.
Shall I open a pull request? I am hesitant because I cannot actually explain every line of this as required in the pull request template. If you'd rather consider it a proof of concept and re-implement from requirements that is fine. I can still sign a CLA if needed.
From the very large comment above, are any of these blocking?
skevetter commentedon Oct 31, 2026
@wsmoak you are welcome to open a PR. If anything is missing, I will add on to your PR.
I am trying something new where I put the plans in the Issue comments section for an agent to pick up the work. The thought being consistent implementation for those looking to use their agents to perform work. Or at least some level of sanity check.
I am a bit overloaded at the moment between personal life obligations, professional work, and this side project. That is to say, I will try my best to get to it when I have available bandwidth but the approaching holidays is also cutting into my availability.
I would love having more time to work on issues/feature request directly but that is not an available opportunity at the moment.
skevetter commentedon Oct 31, 2026
Continued the contributor branch in #1739, preserving contributor authorship. The implementation adds the versioned digest-bound file format, launch-scope validation, per-launch snapshots, and local-path redaction from pod options and container setup payloads. The live private-registry acceptance flow and local validation pass. GitHub CI and hosted reviews are in progress. The contributor CLA signature is still required. No merge has been performed.
skevetter commentedon Nov 1, 2026
The continuation is ready for human merge in #1739 at cf94f45. All required GitHub checks pass, including the 94-job implementation workflow, contributor agreement, and commit signatures. Greptile reviewed the final head at 5/5. Local and hosted CodeRabbit reviews have no actionable findings remaining. The private-registry startup and lifecycle scenario passes locally and in the Kubernetes CI suite. Live AWS/EKS and native Windows Kubernetes launch were not run. No merge has been performed.
skevetter commentedon Nov 11, 2026
completed