Our runners often deal with very long running jobs that may even take 24h to complete.
For that reason, when deploying a new versions it is not reasonable to manually pause the runner and wait for all the jobs to complete before updating and unpausing. Doing so may require keeping the runner paused while a job takes many hours to complete, preventing other jobs from being processed in the meantime.
Kubernetes actually provides a way to handle this situation automatically via graceful shutdowns. In short, we can set a very long terminationGracePeriodSeconds value and, when getting SIGTERM from Kubernetes, stop polling for new jobs and exit once the last running job completes.
This crate already has an API to drain the jobs, but it needs some careful analysis to provide good guidance on how to integrate that in the worker loop wrt. e.g. cancel safety, with some examples.
See also:
Our runners often deal with very long running jobs that may even take 24h to complete.
For that reason, when deploying a new versions it is not reasonable to manually pause the runner and wait for all the jobs to complete before updating and unpausing. Doing so may require keeping the runner paused while a job takes many hours to complete, preventing other jobs from being processed in the meantime.
Kubernetes actually provides a way to handle this situation automatically via graceful shutdowns. In short, we can set a very long
terminationGracePeriodSecondsvalue and, when gettingSIGTERMfrom Kubernetes, stop polling for new jobs and exit once the last running job completes.This crate already has an API to drain the jobs, but it needs some careful analysis to provide good guidance on how to integrate that in the worker loop wrt. e.g. cancel safety, with some examples.
See also: