Skip to content

[CI] Preload GPU Docker image in AMD64 runner AMIs - #109

Open
RAMitchell wants to merge 1 commit into
dmlc:mainfrom
RAMitchell:codex/preload-gpu-runner-image
Open

RAMitchell wants to merge 1 commit into
dmlc:mainfrom
RAMitchell:codex/preload-gpu-runner-image

Conversation

@RAMitchell

Copy link
Copy Markdown
Member

Fresh GPU CI runners spend about 140 seconds downloading and extracting xgb-ci.gpu; the same pull took 0.05 seconds with the image cached (measurement). Preload xgb-ci.gpu:main during the normal AMD64 Packer build so new runners inherit those Docker layers from the AMI.

The preload step checks GPU access inside the image, records its digest and size in the build log, removes temporary registry credentials, and stops Docker/containerd before snapshotting. CI must continue pulling its requested tag: newer Docker images remain usable even before the next AMI rebuild. Concise inline comments explain why the step exists and when it can be removed.

The AMD64 image is shared by CPU and GPU runners, so both inherit the extra cached image (measured at 11.64 GB locally). The existing AMI name and 40 GB disk size are retained. ARM64 and Windows builds are unchanged.

Building

The AMD64 build now requires an EC2 instance profile with ECR pull permissions:

cd vm_images/linux-amd64
packer init linux-amd64.pkr.hcl
packer build -var 'iam_instance_profile=YOUR_ECR_READ_PROFILE' linux-amd64.pkr.hcl

The profile needs ecr:GetAuthorizationToken plus ecr:BatchGetImage, ecr:GetDownloadUrlForLayer, and ecr:BatchCheckLayerAvailability for xgb-ci.gpu. The Packer operator needs permission to pass that role. This change does not create IAM resources or automate AMI rebuilds.

Validation

  • Packer 1.16.1 / Amazon plugin 1.8.3 validation passed; an empty instance-profile name is rejected.
  • Packer formatting, repository pre-commit, shell syntax and whitespace checks passed.
  • A temporary mocked AWS/Docker check verified successful preloading, pull-failure propagation and credential cleanup on both paths.
  • No AWS AMI has been built with this change. The first build must verify ECR access, disk capacity, and cached-image availability on a fresh runner. The measured pull-time reduction excludes provisioning and snapshot-read overhead, so it is not a promise of equivalent end-to-end savings.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant