Skip to content

ci: investigate Debian action preparation 404 and propose runner mitigation #4085

Description

@matthewgrossman

User Story

Matt (MG) needs reliable OpenShell Debian packaging and downstream conformance checks while qualifying changes through the merge queue. After the Python PTY fix #4076 merged, Matt observed the Debian action-download failure and requested this focused investigation and a reviewable mitigation proposal.

Description

Investigate the action preparation failure in Branch E2E run 36931039020, and prepare a concrete runner configuration proposal if the failure cannot be corrected inside an OpenShell workflow.

On October 1, 2026 at 21:58:38 UTC, runner 2.337.0 failed to download the existing actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c archive from codeload. Only Set up job ran. Its ActionNotFoundException corresponds to HTTP 404 in the exact runner source and bypasses the existing three-attempt retry loop. Neither workflow steps nor action-body retries can intercept it. Same-pin Debian packaging succeeded in run 36914303849.

Impact / Why This Matters

The failure canceled the arm64 sibling and skipped package-dependent integration lanes, including K3s conformance. Manual reruns spend qualification time without addressing the failing boundary. The underlying reason for the 404 remains unproven.

Context

OpenShell SHA: 76cfd0e31d5e1633db7ccd86ad9023ef7a2461b2; NVIDIA managed Linux amd64 CPU runner, runtime v1.9.0, image 24b0d8a. There is no demonstrated OpenShell deployment reproduction because packaging never started. The original CI job is the observed reproduction; repeatability is unproven. This internal investigation task does not claim a runtime bug.

Related tracker: #3954. This task covers diagnosis and a reviewable proposal only; fleet rollout and proof of prevention remain explicit infrastructure dependencies. Do not close the broader testing tracker.

Definition of Done

  • Preserve the first-failure evidence and compare successful same-pin runs.
  • Verify actual runner retry/cache controls in primary sources; do not invent a workflow retry setting.
  • Prepare a concrete configuration proposal with bounded population retries, failure behavior, integrity/ownership requirements, and fleet validation criteria.
  • Validate the proposal's commands with controlled transient/permanent failures where practical.
  • Run and report Debian packaging and downstream E2E on the draft PR SHA with matching artifacts, distinguishing baseline results from mitigation activation.
  • Keep the PR draft and explicitly report runner-admin rollout limitations.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions