Skip to content

docs(ci): add .buildkite/ scaffold + Buildkite setup runbook#486

Open
mattwilkinsonn wants to merge 6 commits into
mainfrom
sea-587-buildkite-migration--poseidon
Open

docs(ci): add .buildkite/ scaffold + Buildkite setup runbook#486
mattwilkinsonn wants to merge 6 commits into
mainfrom
sea-587-buildkite-migration--poseidon

Conversation

@mattwilkinsonn

@mattwilkinsonn mattwilkinsonn commented Jun 6, 2026

Copy link
Copy Markdown
Contributor

Pull request

Summary

Establishes the Buildkite CI infrastructure by adding the .buildkite/ directory with an orchestrator pipeline, a lints pipeline, a tool-install script, and a comprehensive setup runbook (docs/buildkite-setup.md). The lints job is removed from .github/workflows/ci.yml and migrated to Buildkite, where it runs as the seal-lints downstream pipeline triggered by the orchestrator.

Changes

  • Added .buildkite/README.md describing the pipeline layout, path-filtering strategy, and which workflows intentionally remain on GitHub Actions
  • Added .buildkite/pipelines/orchestrator.yml as the single Buildkite GitHub App entrypoint: runs the Graphite-CI optimizer, then uses monorepo-diff-buildkite-plugin to fan out trigger: steps to per-concern downstream pipelines based on path filters mirroring .github/path-filters/*.yml
  • Added .buildkite/pipelines/lints.yml running check-fmt, actionlint, and the docs site lint/check suite (biome, markdownlint-cli2, astro check) on a linux-amd64 hosted agent with S3-backed bun install cache
  • Added .buildkite/scripts/lints-install-tools.sh to idempotently install rustfmt, actionlint (SHA-pinned), just, nvm/Node 24, and bun on the agent
  • Removed the lints job from .github/workflows/ci.yml
  • Added buildkite-agent and bk CLI installation to just install-deps — via Homebrew on macOS and via pinned release tarballs into $CARGO_HOME/bin on Linux
  • Added docs/buildkite-setup.md covering the full one-time setup: Buildkite org and agent queue configuration, GitHub App installation, required-status-check naming and two-step rollout order, path filtering, Graphite integration, per-pipeline secrets inventory, and S3-backed cache strategy

Notes for reviewers

  • The orchestrator's trigger: steps for seal-test-linux, seal-test-macos, seal-live-tests, seal-clients-even-terminal, and seal-docs are wired but those downstream pipeline YAML files are not included in this PR — they land separately.
  • The runbook documents the two-step required-check rollout order (land pipeline first, add to ruleset after first conclusion) to avoid locking the merge queue.
  • Live API Tests can also be triggered by adding a live-tests label to a PR, independent of path changes, via a dedicated trigger: step in the orchestrator.
  • Downstream pipelines have no direct GitHub App connection — commit statuses (buildkite/seal-lints, buildkite/seal-test-linux, etc.) are posted back to GitHub through the orchestrator's triggered builds.

@linear-code

linear-code Bot commented Jun 6, 2026

Copy link
Copy Markdown

SEA-587

@coderabbitai

coderabbitai Bot commented Jun 6, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Adds Buildkite CI: README and placeholders, an orchestrator pipeline that fans out to downstream pipelines via monorepo-diff, a lints pipeline and installer script, GitHub Actions lints removal, Justfile tooling installs, and a detailed runbook in docs/buildkite-setup.md.

Changes

Buildkite CI Infrastructure Documentation

Layer / File(s) Summary
Buildkite overview and directory index
.buildkite/README.md
README introduces the .buildkite/ layout and orchestrator fan-out model, explains the monorepo-diff static YAML trigger approach, lists which CI workflows remain on GitHub Actions, and points to the detailed runbook and local tooling expectations.
Directory structure placeholders
.buildkite/hooks/.gitkeep, .buildkite/pipelines/.gitkeep
Placeholder .gitkeep files describe the purpose of hooks/ and pipelines/ and note they can be removed when real files are added.
Orchestrator, lints pipeline, and installer script
.buildkite/pipelines/orchestrator.yml, .buildkite/pipelines/lints.yml, .buildkite/scripts/lints-install-tools.sh, .github/workflows/ci.yml
Adds an orchestrator pipeline with a Graphite optimizer and monorepo-diff-based watch: triggers to synchronously trigger downstream pipelines, an always-run seal-lints trigger, a new lints pipeline to run formatting/actionlint/docs checks, an idempotent installer script for lints tooling, and removes the previous lints job from GitHub Actions CI.
Local tooling install changes
Justfile
Extends install-deps to install buildkite-agent and bk on macOS (Homebrew) and Linux (resolve GitHub releases and install binaries into ${CARGO_HOME:-$HOME/.cargo}/bin), with warnings on failures.
Buildkite setup and configuration runbook
docs/buildkite-setup.md
Adds a detailed runbook covering one-time Buildkite/GitHub App setup, commit-status naming and rollout, agent/queue configuration, monorepo path-filter coordination, Graphite integration, secret naming/injection, S3-backed cache configuration and key shapes, developer tooling/workflows, and verification notes.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

🐰 In burrows of YAML the pipelines bloom,

Orchestrator hums and fans out the room,
Hooks and scripts snug in .buildkite/ nests,
Runbook and lints keep our checks at rest,
I hop, I stamp—green checkmarks are my guests.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately summarizes the main change: adding Buildkite CI scaffold and setup runbook.
Description check ✅ Passed The description comprehensively details all changes in the PR, including new files, removed jobs, and updates to existing files.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch sea-587-buildkite-migration--poseidon

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor Author

How to use the Graphite Merge Queue

Add either label to this PR to merge it via the merge queue:

  • Merge Queue - adds this PR to the back of the merge queue
  • Merge Queue Fast Track - for urgent changes, fast-track this PR to the front of the merge queue

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has required the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

@greptile-apps

greptile-apps Bot commented Jun 6, 2026

Copy link
Copy Markdown

Greptile Summary

This PR migrates the lints job from GitHub Actions to Buildkite, establishing the full Buildkite CI scaffold: an orchestrator pipeline with Graphite-CI integration and monorepo-diff path-filtered triggers, a seal-ci Docker base image baking in all Linux tooling, a seal-lints downstream pipeline, a comprehensive setup runbook, and buildkite-agent/bk additions to just install-deps.

  • The lints job is removed from .github/workflows/ci.yml and replaced by a Buildkite seal-lints downstream pipeline run inside the new seal-ci image, covering check-fmt, actionlint, and the docs/site bun-based lint/check suite with --frozen-lockfile.
  • The orchestrator is the single GitHub App entrypoint; downstream pipelines report individual buildkite/<slug> commit statuses for branch-protection consumption, and the runbook documents the two-step rollout order to avoid locking the merge queue.

Confidence Score: 5/5

Safe to merge — all changes are CI infrastructure and documentation with no impact on production code paths.

The PR removes one GHA job and adds Buildkite pipeline YAML, a Dockerfile, a runbook, and local tooling install steps. None of the changes touch runtime code. The two files worth a second look are Dockerfile.ci (unpinned tool versions and a partial integrity check) and orchestrator.yml (mutable plugin reference), but neither introduces a defect that would break a build or expose sensitive data today.

.buildkite/images/Dockerfile.ci — unpinned sccache/just/taplo versions and the arm64 SHA256 gap are worth tidying before the image is used at scale.

Important Files Changed

Filename Overview
.buildkite/images/Dockerfile.ci Base CI image baking all Linux tooling; sccache, just, and taplo resolve latest-at-build-time from the GitHub API (unpinned), and the actionlint SHA256 integrity check is conditional on amd64 only, silently skipping verification for arm64.
.buildkite/pipelines/orchestrator.yml Entrypoint pipeline with Graphite-CI and monorepo-diff fan-out; withgraphite/graphite-ci is referenced via the mutable #main branch rather than a pinned tag or commit SHA, unlike monorepo-diff#v1.10.0 in the same file.
.buildkite/pipelines/lints.yml Lints pipeline correctly pins the docker plugin to v5.13.0, uses --frozen-lockfile, mounts the workspace, and adds auto-retry on agent loss/stop.
.buildkite/pipelines/ci-image.yml Builds and pushes the seal-ci image via OIDC (no PAT), includes a smoke-run after push, and annotates the build with the resolved digest; well-structured.
.github/workflows/ci.yml Clean removal of the lints job; no other jobs are touched.
Justfile Adds buildkite-agent and bk CLI installs via Homebrew (macOS) and pinned-latest release tarballs (Linux); error paths are guarded and warn-only.
docs/buildkite-setup.md Comprehensive runbook covering org setup, agent queues, GitHub App install, OIDC registry policy, required-check rollout order, Graphite integration, secrets inventory, and S3 cache strategy.

Sequence Diagram

sequenceDiagram
    participant GH as GitHub
    participant Orch as seal-ci-orchestrator
    participant GCI as Graphite CI plugin
    participant Lints as seal-lints
    participant MD as monorepo-diff plugin
    participant Down as downstream pipelines

    GH->>Orch: pull_request / push webhook
    Orch->>GCI: "withgraphite/graphite-ci#main"
    GCI-->>Orch: skip or proceed signal
    par after wait step
        Orch->>Lints: trigger (async: false)
        Lints-->>GH: buildkite/seal-lints status
    and
        Orch->>MD: diff against origin/main
        MD->>Down: trigger steps for matched paths
        Down-->>GH: "buildkite/<slug> status"
    end
Loading

Reviews (8): Last reviewed commit: "feat(ci): containerized lints — seal-ci ..." | Re-trigger Greptile

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.buildkite/README.md:
- Around line 37-39: Update the phrasing for the `nightly.yml` bullet so it
doesn't claim "non-x64 runners" while listing `macos-26-intel`; instead describe
that `nightly.yml` runs across multiple architectures/hosts — e.g., "coverage
for multiple architectures/hosts (e.g., `ubuntu-24.04-arm`, `macos-latest`,
`macos-26-intel`)" — so the entry for `nightly.yml` accurately reflects
inclusion of both ARM and Intel mac runners.

In `@docs/buildkite-setup.md`:
- Around line 31-34: The text incorrectly treats "pull_request: synchronize" as
a separate event; update the sentence to say Buildkite listens to the GitHub
'pull_request' event (including actions like 'synchronize') and 'push' events,
so rephrase to clarify that 'synchronize' is an action under 'pull_request'
rather than its own event and ensure the identifiers 'pull_request' and
'synchronize' are used precisely in the sentence.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: ac2e0551-487e-416a-96f3-7661eec839b2

📥 Commits

Reviewing files that changed from the base of the PR and between 207c70f and 15ab235.

📒 Files selected for processing (4)
  • .buildkite/README.md
  • .buildkite/hooks/.gitkeep
  • .buildkite/pipelines/.gitkeep
  • docs/buildkite-setup.md

Comment thread .buildkite/README.md Outdated
Comment thread docs/buildkite-setup.md Outdated
Comment thread docs/buildkite-setup.md
Comment thread .buildkite/README.md
Comment on lines +10 to +21
.buildkite/
├── README.md (this file)
├── pipelines/ (pipeline YAML — one per gate)
│ ├── lints.yml (fmt + actionlint + docs lint)
│ ├── test-linux.yml (clippy + Test Linux + build smoke)
│ ├── test-macos.yml (Test macOS — uses `just nextest-macos-pr`)
│ ├── live-tests.yml (Live API Tests, label/path-gated)
│ ├── clients-even-terminal.yml (Even Terminal — bun lint + test)
│ ├── deploy-docs.yml (Cloudflare Pages deploy + PR preview)
│ └── release.yml (4-target release matrix on tag push)
├── scripts/ (helper scripts invoked from pipelines)
└── hooks/ (agent-side hooks, if any)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 scripts/ directory is listed but has no placeholder

The layout tree shows a scripts/ directory alongside pipelines/ and hooks/, and both of those get .gitkeep files in this PR. scripts/ does not, so it exists only in documentation but not on disk. If another PR adds scripts that depend on the directory existing, the scaffolding will be inconsistent. Consider adding .buildkite/scripts/.gitkeep here while the structure is being established.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
.buildkite/pipelines/orchestrator.yml (1)

169-183: ⚠️ Potential issue | 🟡 Minor

Prevent duplicate seal-live-tests triggers when both LLM paths change and the live-tests label is present

  • Buildkite doesn’t automatically deduplicate multiple trigger steps that target the same downstream pipeline; each matching trigger step creates its own downstream build.
  • .buildkite/pipelines/orchestrator.yml contains two separate seal-live-tests triggers (the LLM path-filtered one plus the label-gated one build.pull_request.labels includes "live-tests"), with no cancellation/dedup mechanism configured here.
  • Mitigate by enabling “Cancel Intermediate Builds” on the seal-live-tests pipeline (or document that the double-run is intentional/acceptable).
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.buildkite/pipelines/orchestrator.yml around lines 169 - 183, The pipeline
has two separate trigger steps that both call the downstream pipeline
"seal-live-tests" (the LLM path-filtered trigger plus the label-gated trigger
labeled ":satellite: Live API Tests (label-triggered)"), which can cause
duplicate downstream builds when a PR both changes LLM paths and has the
"live-tests" label; fix by enabling "Cancel Intermediate Builds" (or equivalent
automatic dedup/cancellation) on the downstream "seal-live-tests" pipeline OR
add documentation/comments near the trigger step explaining the double-run is
intentional/acceptable so maintainers know why duplicates occur; ensure the
change references the trigger name "seal-live-tests" and the label-gated step to
avoid accidental edits elsewhere.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/buildkite-setup.md`:
- Around line 156-160: The PR objectives claim GRAPHITE_TOKEN is no longer
required but the docs and pipeline still reference GRAPHITE_TOKEN and the
orchestrator injects graphite_token: "$$GRAPHITE_TOKEN"; either update the PR
objectives summary to state that GRAPHITE_TOKEN is required, or remove
references and injection of GRAPHITE_TOKEN from docs/buildkite-setup.md and
.buildkite/pipelines/orchestrator.yml; specifically reconcile and change the PR
objectives text to match the actual state or remove the env var usage
(GRAPHITE_TOKEN and graphite_token) from the orchestrator pipeline and docs so
they are consistent.

In `@Justfile`:
- Around line 159-216: The bk download URL is incorrect; update the bk_url
construction in the bk installation block (where bk_tag is resolved) to use the
published asset name pattern "bk_${bk_tag}_linux_${bk_arch}.tar.gz" instead of
"cli_${bk_tag}_linux_${bk_arch}.tar.gz" so the URL becomes
https://github.com/buildkite/cli/releases/download/v${bk_tag}/bk_${bk_tag}_linux_${bk_arch}.tar.gz;
ensure this change is made in the code that sets bk_url (the section using
bk_tag and bk_arch before fetching and extracting bk).

---

Outside diff comments:
In @.buildkite/pipelines/orchestrator.yml:
- Around line 169-183: The pipeline has two separate trigger steps that both
call the downstream pipeline "seal-live-tests" (the LLM path-filtered trigger
plus the label-gated trigger labeled ":satellite: Live API Tests
(label-triggered)"), which can cause duplicate downstream builds when a PR both
changes LLM paths and has the "live-tests" label; fix by enabling "Cancel
Intermediate Builds" (or equivalent automatic dedup/cancellation) on the
downstream "seal-live-tests" pipeline OR add documentation/comments near the
trigger step explaining the double-run is intentional/acceptable so maintainers
know why duplicates occur; ensure the change references the trigger name
"seal-live-tests" and the label-gated step to avoid accidental edits elsewhere.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: b12ca056-d8c2-4579-a0d5-edd3b391488d

📥 Commits

Reviewing files that changed from the base of the PR and between 15ab235 and 4149144.

📒 Files selected for processing (7)
  • .buildkite/README.md
  • .buildkite/pipelines/lints.yml
  • .buildkite/pipelines/orchestrator.yml
  • .buildkite/scripts/lints-install-tools.sh
  • .github/workflows/ci.yml
  • Justfile
  • docs/buildkite-setup.md
💤 Files with no reviewable changes (1)
  • .github/workflows/ci.yml

Comment thread docs/buildkite-setup.md
Comment on lines +156 to +160
plugin and reads `GRAPHITE_TOKEN` from a pipeline-level
Buildkite secret. Set the secret via:

> Buildkite dashboard → Pipeline → Settings → Secrets →
> `GRAPHITE_TOKEN`

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Description: Check if GRAPHITE_TOKEN is referenced in orchestrator pipeline

# Search for GRAPHITE_TOKEN usage in orchestrator and related pipeline files
rg -n 'GRAPHITE_TOKEN' .buildkite/

Repository: sealedsecurity/seal

Length of output: 256


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== Search for GRAPHITE_TOKEN in .buildkite =="
rg -n 'GRAPHITE_TOKEN|graphite_token' .buildkite/

echo
echo "== Search for GRAPHITE_TOKEN in docs/buildkite-setup.md =="
rg -n 'GRAPHITE_TOKEN' docs/buildkite-setup.md

echo
echo "== Show PR objectives / summary context in docs/buildkite-setup.md =="
# Try to find the 'PR objectives' section or the exact summary sentence
rg -n 'PR objectives|Documents that GRAPHITE_TOKEN|Graphite-CI handles deduplication server-side' docs/buildkite-setup.md || true

Repository: sealedsecurity/seal

Length of output: 577


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== Search for GRAPHITE_TOKEN in .buildkite =="
rg -n 'GRAPHITE_TOKEN|graphite_token' .buildkite/

echo
echo "== Search for GRAPHITE_TOKEN in docs/buildkite-setup.md =="
rg -n 'GRAPHITE_TOKEN' docs/buildkite-setup.md

echo
echo "== Show PR objectives / summary context in docs/buildkite-setup.md =="
rg -n 'PR objectives|Documents that GRAPHITE_TOKEN|Graphite-CI handles deduplication server-side' docs/buildkite-setup.md || true

Repository: sealedsecurity/seal

Length of output: 577


Fix the GRAPHITE_TOKEN “no longer required” contradiction in the PR objectives
docs/buildkite-setup.md still instructs setting GRAPHITE_TOKEN (lines 156-160) and lists it for the orchestrator (line 189), and .buildkite/pipelines/orchestrator.yml still injects graphite_token: "$$GRAPHITE_TOKEN". Update the PR objectives summary to match the actual requirement (or change the pipeline/docs if the token truly became unnecessary).

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/buildkite-setup.md` around lines 156 - 160, The PR objectives claim
GRAPHITE_TOKEN is no longer required but the docs and pipeline still reference
GRAPHITE_TOKEN and the orchestrator injects graphite_token: "$$GRAPHITE_TOKEN";
either update the PR objectives summary to state that GRAPHITE_TOKEN is
required, or remove references and injection of GRAPHITE_TOKEN from
docs/buildkite-setup.md and .buildkite/pipelines/orchestrator.yml; specifically
reconcile and change the PR objectives text to match the actual state or remove
the env var usage (GRAPHITE_TOKEN and graphite_token) from the orchestrator
pipeline and docs so they are consistent.

Comment thread Justfile
mattwilkinsonn added a commit that referenced this pull request Jun 7, 2026
@mattwilkinsonn mattwilkinsonn force-pushed the sea-587-buildkite-migration--poseidon branch from 6389736 to 8b87901 Compare June 7, 2026 23:43

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
.buildkite/pipelines/orchestrator.yml (1)

39-39: ⚠️ Potential issue | 🟠 Major

Pin the Graphite Buildkite plugin to an immutable ref (don’t use #main)

In .buildkite/pipelines/orchestrator.yml (line 39), - withgraphite/graphite-ci#main: should be changed to - withgraphite/graphite-ci#<git-tag> or the most robust - withgraphite/graphite-ci#<full-commit-sha>. Buildkite recommends immutable plugin refs (tags/SHAs) and avoiding branch refs like #main; Graphite’s plugin repo does not appear to provide stable version tags, so pinning to a specific commit SHA is the practical option here.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.buildkite/pipelines/orchestrator.yml at line 39, Replace the mutable plugin
ref "- withgraphite/graphite-ci#main:" with an immutable ref by pinning to a
specific tag or full commit SHA (e.g. "- withgraphite/graphite-ci#<git-tag>" or
"- withgraphite/graphite-ci#<full-commit-sha>") so the Buildkite pipeline uses a
fixed plugin version; update the occurrence of "-
withgraphite/graphite-ci#main:" accordingly in the orchestrator pipeline
definition.
Justfile (1)

184-207: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Verify downloaded Buildkite binaries before installing them to PATH.

This block installs executables from GitHub release tarballs without checksum/signature validation, which weakens the trust boundary for local developer tooling.

Suggested hardening
-                    if curl -fsSL -o "$tmp_dir/bk.tar.gz" "$bk_url"; then
+                    checks_url="https://github.com/buildkite/cli/releases/download/v${bk_tag}/checksums.txt"
+                    if curl -fsSL -o "$tmp_dir/checksums.txt" "$checks_url" \
+                       && curl -fsSL -o "$tmp_dir/bk.tar.gz" "$bk_url" \
+                       && (cd "$tmp_dir" && grep "bk_${bk_tag}_linux_${bk_arch}.tar.gz" checksums.txt | sha256sum --check --strict); then
                         tar -xzf "$tmp_dir/bk.tar.gz" -C "$tmp_dir" bk
                         install -m 0755 "$tmp_dir/bk" "$bin_dir/bk"
#!/usr/bin/env bash
set -euo pipefail
echo "Agent release assets:"
gh api repos/buildkite/agent/releases/latest --jq '.assets[].name'
echo
echo "CLI release assets:"
gh api repos/buildkite/cli/releases/latest --jq '.assets[].name'
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@Justfile` around lines 184 - 207, The script currently downloads
buildkite-agent and bk tarballs using ba_url/bk_url and installs files from
tmp_dir without integrity/authenticity checks; update the download-and-install
blocks for buildkite-agent (ba_url, tmp_dir, buildkite-agent) and bk (bk_url,
tmp_dir, bk) to (1) fetch corresponding release asset checksums or signatures
(via the GitHub releases API or provided .sha256/.asc assets), (2) verify the
downloaded tarball using sha256sum --check or gpg --verify before extracting,
(3) abort and emit a clear error if verification fails, and (4) only run
tar/extract and install -m 0755 into $bin_dir once verification succeeds,
ensuring tmp_dir cleanup still happens on both success and failure.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In @.buildkite/pipelines/orchestrator.yml:
- Line 39: Replace the mutable plugin ref "- withgraphite/graphite-ci#main:"
with an immutable ref by pinning to a specific tag or full commit SHA (e.g. "-
withgraphite/graphite-ci#<git-tag>" or "-
withgraphite/graphite-ci#<full-commit-sha>") so the Buildkite pipeline uses a
fixed plugin version; update the occurrence of "-
withgraphite/graphite-ci#main:" accordingly in the orchestrator pipeline
definition.

In `@Justfile`:
- Around line 184-207: The script currently downloads buildkite-agent and bk
tarballs using ba_url/bk_url and installs files from tmp_dir without
integrity/authenticity checks; update the download-and-install blocks for
buildkite-agent (ba_url, tmp_dir, buildkite-agent) and bk (bk_url, tmp_dir, bk)
to (1) fetch corresponding release asset checksums or signatures (via the GitHub
releases API or provided .sha256/.asc assets), (2) verify the downloaded tarball
using sha256sum --check or gpg --verify before extracting, (3) abort and emit a
clear error if verification fails, and (4) only run tar/extract and install -m
0755 into $bin_dir once verification succeeds, ensuring tmp_dir cleanup still
happens on both success and failure.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 57ff2619-cbcf-4754-92a7-104b1ad3af35

📥 Commits

Reviewing files that changed from the base of the PR and between 6389736 and 8b87901.

📒 Files selected for processing (4)
  • .buildkite/pipelines/lints.yml
  • .buildkite/pipelines/orchestrator.yml
  • Justfile
  • docs/buildkite-setup.md

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
.buildkite/pipelines/orchestrator.yml (2)

25-42: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Pin the Graphite-CI plugin to a stable version or commit SHA.

The plugin reference withgraphite/graphite-ci#main tracks the main branch, which means the pipeline can break without warning if the plugin changes. For production CI stability, pin to a specific release tag or commit SHA (e.g., withgraphite/graphite-ci#v1.2.3 or withgraphite/graphite-ci#abc123).

📌 Proposed fix to pin the plugin version
     plugins:
-      - withgraphite/graphite-ci#main:
+      - withgraphite/graphite-ci#v1.0.0:  # Replace with actual stable version/SHA
           # GRAPHITE_TOKEN injected by Buildkite's pipeline-level
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.buildkite/pipelines/orchestrator.yml around lines 25 - 42, The pipeline
step using the Graphite CI plugin (label ":graphite: Graphite CI optimizer",
plugin reference withgraphite/graphite-ci#main) should be pinned to a specific
release tag or commit SHA instead of tracking `#main`; update the plugin
reference `withgraphite/graphite-ci#main` to a fixed identifier like
`withgraphite/graphite-ci#vX.Y.Z` or `withgraphite/graphite-ci#<commit-sha>` to
ensure CI stability and avoid unexpected breakage.

43-53: 🧹 Nitpick | 🔵 Trivial | ⚖️ Poor tradeoff

Verify that synchronous lints trigger aligns with orchestrator performance goals.

The async: false configuration blocks the orchestrator step until the seal-lints pipeline completes. While this ensures correct commit-status ordering for branch protection, it also means the orchestrator agent remains occupied during the entire lints run (~8min cold-cache budget per lints.yml). If downstream pipelines will eventually run concurrently (seal-test-linux, seal-test-macos, etc.), confirm whether the orchestrator should block on lints or trigger all pipelines asynchronously and rely on GitHub's commit-status aggregation.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.buildkite/pipelines/orchestrator.yml around lines 43 - 53, The orchestrator
step for the Lints pipeline currently uses "async: false" which blocks the
orchestrator agent while "seal-lints" runs; change the step with label
":hammer_and_wrench: Lints" (trigger: "seal-lints") to "async: true" so the
orchestrator no longer waits for completion, and add/adjust a brief comment
noting we rely on GitHub commit-status aggregation for final gating; if you
decide to keep synchronous behavior instead, document the performance tradeoff
and leave "async: false" intentionally.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In @.buildkite/pipelines/orchestrator.yml:
- Around line 25-42: The pipeline step using the Graphite CI plugin (label
":graphite: Graphite CI optimizer", plugin reference
withgraphite/graphite-ci#main) should be pinned to a specific release tag or
commit SHA instead of tracking `#main`; update the plugin reference
`withgraphite/graphite-ci#main` to a fixed identifier like
`withgraphite/graphite-ci#vX.Y.Z` or `withgraphite/graphite-ci#<commit-sha>` to
ensure CI stability and avoid unexpected breakage.
- Around line 43-53: The orchestrator step for the Lints pipeline currently uses
"async: false" which blocks the orchestrator agent while "seal-lints" runs;
change the step with label ":hammer_and_wrench: Lints" (trigger: "seal-lints")
to "async: true" so the orchestrator no longer waits for completion, and
add/adjust a brief comment noting we rely on GitHub commit-status aggregation
for final gating; if you decide to keep synchronous behavior instead, document
the performance tradeoff and leave "async: false" intentionally.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 099b1c85-d833-4290-b824-0e6d6f3a6085

📥 Commits

Reviewing files that changed from the base of the PR and between 8b87901 and 97d473b.

📒 Files selected for processing (2)
  • .buildkite/pipelines/lints.yml
  • .buildkite/pipelines/orchestrator.yml

Comment on lines +102 to +104
&& if [ "${actionlint_arch}" = "amd64" ]; then \
echo "${ACTIONLINT_SHA256} /tmp/${tarball}" | sha256sum --check --strict; \
fi \

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHA256 verification is only performed for amd64 architecture but skipped for arm64. This creates a security vulnerability where arm64 builds could download and execute a compromised actionlint binary without integrity verification.

# Fix: Add SHA256 verification for arm64 as well
ARG ACTIONLINT_SHA256_ARM64=<arm64_sha256_here>
...
&& if [ "${actionlint_arch}" = "amd64" ]; then \
       echo "${ACTIONLINT_SHA256}  /tmp/${tarball}" | sha256sum --check --strict; \
   elif [ "${actionlint_arch}" = "arm64" ]; then \
       echo "${ACTIONLINT_SHA256_ARM64}  /tmp/${tarball}" | sha256sum --check --strict; \
   fi \
Suggested change
&& if [ "${actionlint_arch}" = "amd64" ]; then \
echo "${ACTIONLINT_SHA256} /tmp/${tarball}" | sha256sum --check --strict; \
fi \
&& if [ "${actionlint_arch}" = "amd64" ]; then \
echo "${ACTIONLINT_SHA256} /tmp/${tarball}" | sha256sum --check --strict; \
elif [ "${actionlint_arch}" = "arm64" ]; then \
echo "${ACTIONLINT_SHA256_ARM64} /tmp/${tarball}" | sha256sum --check --strict; \
fi \

Spotted by Graphite

Fix in Graphite


Is this helpful? React 👍 or 👎 to let us know.

@mattwilkinsonn mattwilkinsonn force-pushed the sea-587-buildkite-migration--poseidon branch from 29c8c6b to 3013a87 Compare June 9, 2026 22:33
@mattwilkinsonn mattwilkinsonn force-pushed the sea-587-buildkite-migration--poseidon branch from 3013a87 to d096d99 Compare June 9, 2026 22:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant