ruby: Measure worker slack and let a sample of workers exit idle

[spiliopoulos]

Describe

A ci-queue worker that has drained the queue does not exit. exhausted? is queue_initialized? && size == 0, and size counts pending plus running across the whole build:

def size
  redis.multi do |transaction|
    transaction.llen(key('queue'))     # pending
    transaction.zcard(key('running'))  # in flight, on ANY worker
  end.inject(:+)
end

So every worker stays online until the slowest one finishes. That is deliberate — the comment on the idle branch says why:

we just stay online here in case a test gets retried or times out so we can afford to wait

It is not free. On a Figma permissions build with 20 workers over 141 files:

• two workers ran 105 and 34 tests and stopped 62 milliseconds apart
• Datadog wall-time profiles put roughly 22% of main-thread time in worker.rb:poll → sleep
• that works out to about 28 worker-minutes of paid-for idle on a single build

This PR measures that slack, and makes the number of workers that hold requeue duty tunable.

Measurement

The worker records when its last test finished and exposes the gap between that and leaving the queue:

# Time between the last test finishing and the worker leaving the queue: capacity
# the build paid for and did not use. Nil when the worker never reserved a test.
def slack_duration
  return nil if @last_test_finished_at.nil?

  CI::Queue.time_now - @last_test_finished_at
end

The minitest adapter emits it after queue.poll returns as minitests.queue.worker.slack (a ms timer), tagged waits_for_requeues:true|false and the existing slug:.

Emitted from the adapter rather than the queue because ci/queue stays framework agnostic — there is an rspec adapter too — and Minitest::Queue::Statsd lives under minitest/. It reuses the existing CI_QUEUE_STATSD_ADDR client, so no new dependency.

Control

At init each worker draws a number and compares it to a threshold. Workers above it keep today's behaviour; the rest leave once they have been idle for a grace period, after the existing sleep and backoff.

@waits_for_requeues = Random.new.rand > config.idle_exit_probability

def idle_exit?
  return false if @waits_for_requeues
  return false if @idle_since.nil?

  CI::Queue.time_now - @idle_since >= config.idle_exit_grace
end

added to the existing loop guard:

until shutdown_required? || config.circuit_breakers.any?(&:open?) || exhausted? ||
      max_test_failed? || idle_exit?

env var	default	meaning
CI_QUEUE_IDLE_EXIT_PROBABILITY	0.0	fraction of workers allowed to leave early
CI_QUEUE_IDLE_EXIT_GRACE	30	seconds idle before such a worker leaves

Defaults preserve current behaviour exactly. At 0.0 every draw is above the threshold, so every worker waits, as today.

One non-obvious detail, called out in a code comment: the draw uses Random.new rather than Kernel#rand. The global RNG is seeded from --seed, which is identical across workers, so every worker would otherwise draw the same number and the sample would be all-or-nothing.

poll previously kept idle_since as a local while the class already had an attr_accessor :idle_since and an idle? reading the ivar. The local is now the ivar, so idle_exit? can read it and the existing accessor stops being dead.

Test plan

New tests in test/ci/queue/redis/worker_idle_exit_test.rb (no Redis server needed — ::Redis.new is lazy) and test/ci/queue/configuration_test.rb.

$ bundle exec rake test TEST_FILES="test/ci/queue/configuration_test.rb test/ci/queue/redis/worker_idle_exit_test.rb"
27 tests, 64 assertions, 0 failures, 0 errors, 0 skips

$ bundle exec rake test TEST_FILES="<all Redis-free suites>"
62 tests, 190 assertions, 0 failures, 0 errors, 0 skips

I could not run the Redis-backed suites locally (no Redis available); CI covers those.

Before the metric will land

Two things outside this repo, neither of which this PR can fix:

1. CI_QUEUE_STATSD_ADDR is not set anywhere in figma/figma. Without it queue_config.statsd_endpoint is nil, the StatsdReporter is never registered, and report_worker_slack returns early. It needs setting on the sinatra test jobs.
2. The CI agent does not accept non-local dogstatsd. devex/buildkite/images/linux/config/datadog.yaml sets use_dogstatsd: true and dogstatsd_port: 8125, but not dogstatsd_non_local_traffic, which defaults to false. The agent therefore binds 127.0.0.1:8125 and a UDP packet from a job container to the bridge gateway is dropped silently. Note apm_non_local_traffic: true is set, which is why the Ruby profiler reaches the agent but statsd would not.

Both are one-line changes in figma/figma, and the figma/figma Gemfile ref needs bumping to pick this up.

🤖 Generated with Claude Code

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

^{Reviewed by Cursor Bugbot for commit 86365bb. Configure here.}

[cursor]

Idle workers spam logs and Redis

Medium Severity

Writing idle_since on the instance makes idle? true while polling, which activates a previously dead branch in try_to_reserve_lost_test. That path prints the full running set and issues a Redis zrange on every idle reserve (every 0.5–2s), including when idle_exit_probability is 0.0.

Additional Locations (1)

• ruby/lib/ci/queue/redis/worker.rb#L469-L474

 

^{Reviewed by Cursor Bugbot for commit 86365bb. Configure here.}

[cursor]

Zero idle grace is ignored

Low Severity

from_env uses || 30.0 after to_f, so CI_QUEUE_IDLE_EXIT_GRACE=0 becomes 0.0 and falls back to 30.0. A zero grace is a valid “leave as soon as idle” setting and is already used that way on the constructor path.

 

^{Reviewed by Cursor Bugbot for commit 86365bb. Configure here.}