feat(operator): gate readiness on installation completion

[aweingarten]

📌 Summary

Closes #3419 — gives apl-operator a readiness signal that reflects platform installation, so helm install --wait and kubectl wait --for=condition=Available stop returning before the platform has converged.

On the auth.<domainSuffix>/ready suggestion

@j-zimnowoda thanks — that endpoint (oauth2-proxy behind the ingress) is a genuinely useful external smoke check, and I've documented it in this PR as the complement to the in-cluster gate. It isn't a substitute for one, for four reasons:

1. It doesn't fix the reported bug. helm install --wait still returns in ~30s, because the operator's readinessProbe was pgrep -f 'apl-operator' — "the process exists". Every adopter still has to bolt a bespoke external poll loop onto their bootstrap.
2. It needs things the platform doesn't control. Public DNS resolution and a trusted certificate. Bootstrap runners on private networks, and anyone on a custom CA or ACME staging, either can't reach it or must drop to curl -k, which weakens what the 200 proves.
3. Ambiguous failure modes. A non-200 / NXDOMAIN can't distinguish "platform not ready" from a DNS, cert, or ingress problem — exactly the distinction bootstrap automation needs to decide between "keep waiting" and "fail loudly".
4. It has no notion of which revision converged. After pushing a values change, auth/ready stays 200 throughout. There's no way to know the operator applied your change — the AplStatus-shaped ask in the issue.

So: good liveness check for the ingress/auth chain, not a convergence signal. This PR adds the latter using state the operator already tracks.

What changed

The operator writes a /tmp/ready marker at the single point where the installation phase has resolved to completed — fresh install, recovery install, or a restart on an already-installed cluster — and the readinessProbe tests for that marker:

# now blocks until the helmfile pipeline has actually converged
kubectl wait --for=condition=Available deployment/apl-operator -n apl-operator --timeout=30m
helm install apl … --wait --timeout 30m

• src/operator/k8s.ts — markInstallationComplete() alongside the existing updateHeartbeatFile(); same /tmp emptyDir pattern, so the marker clears on restart and is re-written once the operator re-confirms status.
• src/operator/main.ts — one call, after the install/recovery/already-installed branches converge.
• chart/apl — readinessProbe switched from pgrep to test -f /tmp/ready; progressDeadlineSeconds raised to 3600.
• charts/apl-operator — same gate (it had no readinessProbe at all).
• EXECUTION_FLOW.md — documents the contract, including the already existing apl-installation-status / apl-operator-state ConfigMaps, which answer "did the operator apply my commit?" via commitHash + status. That covered most of the issue's ask already; it just wasn't documented as a contract.

Three deliberate properties

• Readiness latches. It is not cleared while a later apply runs. The reconcile loop applies every ~5 minutes in steady state; flapping Available on each pass would make the condition useless as a gate. Per-apply status stays in apl-operator-state.
• It fails closed. A marker that can't be written, or an installation that keeps retrying, leaves the pod NotReady. The signal never claims a convergence that didn't happen — --wait times out loudly instead of returning early.
• progressDeadlineSeconds: 3600. Otherwise kubectl rollout status reports ProgressDeadlineExceeded at the 600s default while a perfectly healthy first install is still running.

🔍 Reviewer Notes

The default is the one thing worth arguing about. With operator.readiness.gateOnInstallationComplete: true (default here), helm install --wait blocks for the real 10-15 minute install instead of ~30 seconds. Anyone using --wait with helm's 5m default timeout will now see it time out — and with --atomic, roll back. That's the honest behaviour and the point of the issue, but it is a behaviour change, so it's switchable in one value and NOTES.txt prints the --timeout guidance. Happy to flip the default to false (opt-in) if you'd rather not change what existing --wait callers see; say the word and I'll push the one-line change.

Two smaller notes:

• Both operator Deployments (chart/apl for install, charts/apl-operator via helmfile) got the gate so their pod specs don't disagree about readiness across the handover.
• A first-class AplStatus CR with conditions and lastSuccessfulReconcile is still the better long-term answer, and would let kubectl wait --for=condition=Converged work per-revision. This PR deliberately doesn't introduce a CRD — it makes the existing signal truthful. Glad to follow up with the CR if you want it.

🧹 Checklist

• Code is readable, maintainable, and robust.
• Unit tests added/updated — 4 new cases in src/operator/k8s.test.ts (marker written with timestamp, idempotent re-mark on restart, never throws when unwritable so the pod stays NotReady, default path matches the probe). Full operator suite: 89 passed / 1 skipped.

Also verified locally: tsc --noEmit clean, eslint clean (33 pre-existing warnings, 0 errors), helm lint chart/apl passes, and both charts render correctly with the gate on and off. lint:hf was not run — no helmfile binary in my environment; no helmfile templates are touched.

[CasLubbers]

Hi @aweingarten, thanks for your contribution! I will start reviewing and testing your PR soon. Since this touches the installation process, the review will take a bit more time than usual.

[CasLubbers]

So this got me thinking what does it actually mean that the apl-operator is 'ready'. With the current place of markOperatorReady (markInstallationComplete) it only installed the core application through helmfile. If the operator would stop working from this point the platform would not work.
So it would be better to move the execution of this function to: https://github.com/linode/apl-core/blob/main/src/operator/apl-operator.ts#L99. There it would mark the operator ready after it did its first apply run. So all the argocd application exists. From there the platform will always try to heal itself through argocd.
Additionally the function is idempotent. So its fine to call it multiple times.

[CasLubbers]

This flag is not necessary. If we have the marker in code which will always run why not always use it?

[CasLubbers]

There is already a default in the values.yaml

Suggested change

	{{- $readiness := .Values.operator.readiness | default dict }}
	{{- $readiness := .Values.operator.readiness }}

[CasLubbers]

There is already a default in the values.yaml

Suggested change

	progressDeadlineSeconds: {{ $readiness.progressDeadlineSeconds | default 3600 }}
	progressDeadlineSeconds: {{ $readiness.progressDeadlineSeconds }}

[CasLubbers]

This comment is cluttering. One line on why we increased the limit would be better.

[CasLubbers]

Lets remove the if else block and keep this command. Additionally we can remove the comment. It's pretty clear what it does.

[CasLubbers]

This is not entirely true. When apl-operator is done installing the platform is not completly ready. We still have ArgoCD that is applying stuff. It only means that the operator is finished and started is reconcile loop.

[CasLubbers]

The name speaks for itself, we can remove the above comment

[CasLubbers]

I would rename this function so it states what it actually does:

Suggested change

	export function markInstallationComplete(filePath: string = READINESS_FILE): void {
	export function markOperatorReady(filePath: string = READINESS_FILE): void {

[CasLubbers]

This comment can be removed. It does not tell much.

[CasLubbers]

The comment can be removed if we rename the function. Then from the function name it is clear what this function does. That is the clean code we want.

[CasLubbers]

I still need to find a way how I can easily deploy and test your fork.

[Copilot]

Pull request overview

This PR makes apl-operator readiness reflect platform installation completion (helmfile pipeline convergence) instead of mere process liveness, so helm install --wait and kubectl wait --for=condition=Available can be used as reliable bootstrap gates.

Changes:

• Add a /tmp/ready marker written once installation reaches completed, and call it from the post-install convergence point in the operator startup flow.
• Gate operator readinessProbe on the marker file (both the bootstrap chart/apl Deployment and the charts/apl-operator Deployment), and extend rollout budget for the bootstrap Deployment.
• Document the readiness/convergence contract and improve CI workflow output behavior for fork PRs.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/operator/main.ts	Marks installation completion at the single post-install convergence point.
src/operator/k8s.ts	Adds READINESS_FILE and markInstallationComplete() to write /tmp/ready safely.
src/operator/k8s.test.ts	Adds unit coverage for readiness marker behavior and error handling.
src/operator/EXECUTION_FLOW.md	Documents readiness gate semantics and ConfigMap-based introspection contract.
charts/apl-operator/values.yaml	Introduces operator.readiness.gateOnInstallationComplete defaulting to true.
charts/apl-operator/templates/deployment.yaml	Adds readinessProbe gated on /tmp/ready for the helmfile-managed operator Deployment.
chart/apl/values.yaml	Adds readiness gating + rollout budget knobs for the bootstrap operator Deployment.
chart/apl/templates/NOTES.txt	Surfaces the new “wait for Available with a longer timeout” guidance.
chart/apl/templates/deployment.yaml	Switches readiness from pgrep to /tmp/ready (configurable), and raises progressDeadlineSeconds.
.github/workflows/svcaplbot-run-dyff.yml	Always publishes compare output to job summary; adds fork-safe behavior for missing token.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (2)

chart/apl/values.yaml:143

• The comment for operator.readiness.progressDeadlineSeconds says it’s “only relevant when the readiness gate is on”, but progressDeadlineSeconds affects Deployment rollout reporting regardless of readinessProbe configuration. If the intent is “primarily needed when the gate is on”, the wording should be adjusted to avoid misleading operators who disable the gate.

    # Rollout budget for the install window. Only relevant when the readiness
    # gate is on — `kubectl rollout status` fails once it is exceeded.
    progressDeadlineSeconds: 3600

src/operator/k8s.ts:52

• The JSDoc claims the /tmp emptyDir marker is “cleared on every restart”, but emptyDir volumes persist for the lifetime of the Pod (container restarts don’t clear them). This reads as if a container crash/restart would force re-marking, which isn’t necessarily true; please clarify that the marker is cleared when the Pod is recreated (e.g., reschedule/rollout).

 * The marker lives on the pod's /tmp emptyDir, so it is cleared on every
 * restart and re-created as soon as the operator re-confirms the installation
 * status from the apl-installation-status ConfigMap.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/operator/k8s.ts:52

• The comment says the /tmp emptyDir marker is "cleared on every restart", but emptyDir persists for the lifetime of the pod and will survive container restarts within the same pod. If the intent is "cleared when the pod is recreated", the wording should be updated to avoid overstating the guarantee.

 * The marker lives on the pod's /tmp emptyDir, so it is cleared on every
 * restart and re-created as soon as the operator re-confirms the installation
 * status from the apl-installation-status ConfigMap.

charts/apl-operator/templates/deployment.yaml:80

• The readiness gate can keep this Deployment NotReady for 10–15 minutes on a first install, but this chart does not raise spec.progressDeadlineSeconds. With the default 600s deadline, Kubernetes will mark the rollout ProgressDeadlineExceeded while installation is still legitimately running, and Helm/kubectl wait/rollout commands may fail early even though the operator is healthy. Consider adding a higher progressDeadlineSeconds (or making it configurable) when gateOnInstallationComplete is enabled, similar to chart/apl.

          {{- if ne (.Values.operator.readiness | default dict).gateOnInstallationComplete false }}
          # /tmp/ready is written by the operator once the platform installation
          # reaches the 'completed' state — the Deployment reports Available only
          # after the helmfile pipeline has converged, not merely once the process
          # is up. Readiness latches, so steady-state reconciles do not flap it.