Context
When an OSA deployment runs on a Kubernetes cluster with the cluster-autoscaler, scale-down/consolidation may evict any pod to drain an underused node. Hook Jobs routinely run for many minutes up to their timeout (an hour by default) at high CPU; an eviction 50 minutes in throws all that work away and burns a retry attempt — the run limps instead of failing fast or finishing.
Ask
Add the standard opt-out annotation to the pod templates the K8s runners create (infra.k8s.runner hook Jobs, infra.k8s.ingester_runner ingester Jobs):
metadata:
annotations:
cluster-autoscaler.kubernetes.io/safe-to-evict: "false"This tells the autoscaler the pod must not be evicted for consolidation; the node it runs on is only reclaimed once the Job finishes. It is a no-op on clusters without an autoscaler, so it is safe unconditionally — no configuration surface needed.
Notes
- Annotation belongs on the pod template inside the Job spec, not the Job object.
- Worth a quick sweep for other runner-created pods with meaningful in-flight state; the OSA server's own Deployment pods should NOT get it (they are replicated and must stay evictable for drains).
Context
When an OSA deployment runs on a Kubernetes cluster with the cluster-autoscaler, scale-down/consolidation may evict any pod to drain an underused node. Hook Jobs routinely run for many minutes up to their timeout (an hour by default) at high CPU; an eviction 50 minutes in throws all that work away and burns a retry attempt — the run limps instead of failing fast or finishing.
Ask
Add the standard opt-out annotation to the pod templates the K8s runners create (
infra.k8s.runnerhook Jobs,infra.k8s.ingester_runneringester Jobs):This tells the autoscaler the pod must not be evicted for consolidation; the node it runs on is only reclaimed once the Job finishes. It is a no-op on clusters without an autoscaler, so it is safe unconditionally — no configuration surface needed.
Notes