Skip to content

Unleash upgrades give no feedback in Console: the progress signal is dropped in nais-api #501

Description

@Starefossen

A user changes an Unleash instance's release channel in Console. The displayed version stays on the old value with a green checkmark, nothing indicates work is happening, and then at some point it flips. A failed upgrade looks the same as a slow one for five minutes.

This traces the signal from the CRD to the browser and identifies where it is lost. Not where it was assumed to be.

The coordination needed is two repos, not three

The natural assumption is Console + nais-api + bifrost. bifrost is not involved.

  • bifrost is not in the read path at all. nais-api runs a Kubernetes informer directly against the unleashes CRD in the management cluster — internal/unleash/dataloader.go:45-49, queries.go:39-41. Console's instance data never passes through bifrost.
  • bifrost also does not drop anything. Its handlers serialize the raw CRD (pkg/api/http/v1/handlers/unleash.go:70, :99, :371), so conditions, resolvedReleaseChannelImage and metadata.generation all go over the wire.

So: nais-api and console-frontend. unleasherator is optional for the base fix but needed for a decent failure story.

Where it dies

nais-api/internal/unleash/models.go:33-66toUnleashInstance, run inside the watcher's converter, so the cache never holds the dropped fields:

instance := &UnleashInstance{
    Name:    u.Name,
    Version: u.Status.Version,
    Ready:   u.Status.Reconciled && u.Status.Connected,
    ...
}

Three of six status fields are read. Dropped by name:

Field Why it matters
Status.ResolvedReleaseChannelImage The target image. This is the field that makes "upgrading" renderable, and it is discarded here.
Status.Conditions Every reason, message and transition time — including the failure message quoted below.
Status.ReleaseChannelName The observed channel. nais-api reads the spec side instead (models.go:62-64), so Console shows desired state that flips instantly.

The CRD's own comment at unleash_types.go:286-288 says "Rather than relying on this value, check the conditions instead." nais-api relies on the booleans and drops the conditions.

The GraphQL schema then has nowhere to put progress: internal/graph/schema/unleash.graphqls:112-142 compresses the entire rollout lifecycle into ready: Boolean!.

A second, smaller drop: bifrost returns the channel's full Image; models.go:208-223 discards it.

Why the version "flips suddenly"

In unleasherator, Status.Version is only written after a successful live connection to the server (unleash_controller.go:1069-1075) and is never cleared or marked stale. So it holds the old value from the moment the spec changes until the rollout completes, then jumps.

During the entire window the CRD reads reconciled: true, connected: true, version: <old> — byte-identical to steady state except for resolvedReleaseChannelImage. waitForDeployment blocks for up to 5 minutes (:58, :929-943) writing no status at all.

So there is no intermediate state to show — but there is a usable target-vs-actual gap, and it is exactly the field nais-api drops.

Smallest coherent fix

nais-api — roughly three struct fields, three schema fields, three assignments:

  • resolvedReleaseChannelImage: String on UnleashInstance (already on the watched CRD)
  • statusMessage: String from conditions[Reconciled].message
  • image: String on UnleashReleaseChannel

console-frontend — compare target tag to version; when they differ show "Upgrading to vX…" and poll while they differ. Both pieces already exist there: an extractVersion() tag parser (+page.svelte:594-607), and a conditional-polling pattern with a document.hidden guard in WorkloadHealth.svelte:152-168.

Today Console fires the mutation and does one refetch (+page.svelte:150-170) — which, given the above, is guaranteed to read the old version. A polling helper exists at :188-193 but is only wired to instance creation, and stops when ready is true — which it already is during an upgrade.

Two traps for whoever picks this up

Do not use releaseChannel.currentVersion as the target. It is derived from instances whose resolved image matches (releasechannel_controller.go:1693-1711) — which precedes the rollout — while the version is harvested from such an instance. It can advertise the old version as current. Use the image tag.

Do not build on the mutation response. The generated bifrost client types status as {connected, version} only (bifrostclient/client.gen.go:108-115), so Reconciled is unreachable and ready is always false on mutation responses (models.go:226-288). Rely on the watcher-backed query.

What this does not fix

  • A failing upgrade still looks like a slow one for five minutes, then becomes a bare X with the old version. The useful text — "Deployment rollout timed out after 5m0s" — is already sitting in conditions[Reconciled].message; exposing statusMessage turns an unexplained X into a sentence, but does not make it appear sooner.
  • Degraded is never set for upgrade failure — only on the deletion path (unleash_controller.go:149,172). And the Unleash controller emits no Kubernetes Events at all.
  • No granularity. A badge and a target version; no pod counts, no ETA.

Related findings worth their own issues

Filed separately, but they bear on any progress UI built here:

  1. Unleash conditions never carry ObservedGeneration, so nothing downstream can tell whether Reconciled=True refers to the new spec or the old one. RemoteUnleash already does this correctly (remoteunleash_controller.go:401) — the fix is one line, copied.
  2. A per-instance progress schema already exists and is dead. ReleaseChannelStatus.InstanceStatus with Phase/StartTime/EndTime/Message/Ready (releasechannel_types.go:133-134,184-202) is never written by any controller. The domain model for this feature was designed and never shipped.
  3. Switching into a channel that is mid-rollout silently gives you the channel's old image (internal/resources/unleash.go:567-588), with nothing anywhere saying so. A user could pick a channel and correctly conclude nothing happened — because nothing did.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions