Skip to content

Schedule fuzz tasks based on target CPU capacity for Chrome - #5462

Draft
hunsche wants to merge 1 commit into
google:masterfrom
hunsche:feature/cpu-aware-schedule-fuzz
Draft

Schedule fuzz tasks based on target CPU capacity for Chrome#5462
hunsche wants to merge 1 commit into
google:masterfrom
hunsche:feature/cpu-aware-schedule-fuzz

Conversation

@hunsche

@hunsche hunsche commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Summary

This pull request introduces CPU-aware scheduling for Chrome fuzzing tasks in schedule_fuzz.py, allowing operators to set a target CPU utilization ceiling rather than relying solely on static Pub/Sub buffer sizes.

Background & Problem

Previously, schedule_chrome_fuzz_tasks scheduled fuzz tasks into the preprocess queue purely based on PREPROCESS_QUEUE_SIZE_LIMIT without visibility into downstream consumption or actual running compute. Because tworker preprocessing takes only seconds while downstream fuzzing runs take minutes to hours, the preprocess queue would drain quickly, causing schedule_fuzz to inject thousands of additional tasks every cron cycle. This led to queue buildups in utask_main and QueueLimitReachedError failures in tworker.

Changes

  1. New Feature Flag: Added CHROME_FUZZ_TARGET_CPUS to FeatureFlags to dynamically configure the target number of active vCPUs for Chrome fuzzing.
  2. CPU Capacity Calculation: When CHROME_FUZZ_TARGET_CPUS is enabled, schedule_fuzz determines remaining capacity by comparing the target against total committed CPUs:
    Committed CPUs = (Active Batch Jobs + Preprocess Tasks + Utask Main Tasks) * 2 vCPUs
  3. Fast Batch Querying: Uses the fast server-side jobs:countByState GCP Batch endpoint across configured regions, memoized with 60-second caching.
  4. Resilient Fallback: Automatically falls back to the original queue-size sizing logic if the feature flag is disabled, empty, invalid, or if the GCP Batch API is temporarily unreachable.
  5. Clean Code & Style: No inline comments (strictly docstrings), self-documenting functions, and minimal diff.
  6. Unit Tests: Added ChromeCpuCapacityTest covering deficit scheduling, capacity saturation, and fallback paths.

Introduces the CHROME_FUZZ_TARGET_CPUS feature flag to allow
dynamically controlling the target number of active CPUs for Chrome
fuzzing workloads.

When enabled, schedule_fuzz aggregates active GCP Batch workloads
and in-flight pipeline queues (preprocess + utask_main) to determine
remaining CPU capacity, preventing downstream queue buildup and
tworker failures.
@hunsche
hunsche requested a review from PauloVLB September 8, 2026 18:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant