Skip to content

runner: Linux job containers run with NO cgroup limits - oom-killer takes the runner listener and the logs #102

Description

@luthermonson

Verified in source: the Linux job container OCI spec (pkg/runtime/runtime.go ~635: WithDefaultSpecForPlatform + WithImageConfig + WithEnv + caps) sets no LinuxResources - no memory.max, no CPU quota. Windows Hyper-V containers DO get limits (4GB/2CPU defaults), Linux gets nothing.

Observed consequence (2026-07-07/08, 4vCPU/16GB VM): >=3 concurrent heavy builds -> kernel oom-killer reaps a whole job container mid-compile -> Runner.Listener dies with it -> GitHub marks the job failed and logs are never uploaded (BlobNotFound), leaving zero diagnostics. Repeated across SDK builds, CI matrix jobs, and E2E lanes.

Proposal:

  1. [runner.linux] memory_bytes / cpu_count config (mirroring the Windows knobs); DEFAULT derived: (vm_memory - 2GB reserve) / max_concurrent, so the sum can never exceed the VM.
  2. Protect the listener: run job step processes in a child cgroup capped slightly below the container limit, or set oom_score_adj so the kernel kills the build process, not Runner.Listener - failed jobs then fail WITH logs.
  3. Surface VM oom events: tail dmesg/kmsg for oom-killer lines in the dispatch agent and mirror them into ephemerd's log with the victim cgroup -> ends the guesswork.
  4. Emit ephemerd_job_memory_limit_bytes so operators can see the derived caps.

Related: fix/claim-retry-resilience (PR #101) handles the claim side of tonight's failures; this handles the execution side.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions