Skip to content

release-health.yml: second concurrent report from one release lost a git push, left a stale dashboard row #34

Description

@bdbarnett

What happened

audiocomponents' first release (v0.2.0, tonight) publishes two distributions
from one tag — pydevices-audioinstruments and pydevices-audioeffects
each with its own report-release-health job, same as audioif has always
done for its three distributions. Both dispatched release-health events
within about 2 seconds of each other:

  • pydevices-audioinstruments report: run 33814572117success, committed 3c03ced at 22:46:03Z.
  • pydevices-audioeffects report: run 33814574688failed.

The release-health concurrency group did serialize the jobs correctly —
run 2's update job started at 22:46:10Z, three seconds after run 1's
completed at 22:46:07Z, so the runs did not overlap. But run 2's checkout
still did not see 3c03ced, so its own commit was built on the pre-run-1
tip; the push was then rejected outright:

[main e9d1b43] release-health: pydevices-audioeffects 0.2.0
To https://github.com/PyDevices/.github
 ! [rejected]        main -> main (fetch first)
error: failed to push some refs to 'https://github.com/PyDevices/.github'

report-release-health runs continue-on-error: true, so this correctly
did not fail the audiocomponents release. But it left
RELEASE_HEALTH.md's pydevices-audioeffects row stale at 0.1.1
(audioif's last release of it) even though pydevices-audioinstruments
right above it shows the correct 0.2.0, and pypi_publish for
audioeffects never got recorded for this release.

Likely cause

reusable-synchronize-mip-package.yml's commit step already has this exact
problem solved — "rebase-and-retry ... up to 3 attempts" — specifically
because "a repository_dispatch payload freezes github.sha at creation
time" and multiple profiles from one release can race
(publishing-automation.md, "Request → queue → sync", point 5). The MIP
queue is also why that job re-checks-out from the branch tip rather than
the dispatch-frozen SHA.

release-health.yml's Commit step does neither: it's a single
git add && git commit && git push with no retry, and relies solely on the
release-health concurrency group to prevent exactly this collision. That
group serializes job execution, but apparently not tightly enough against
back-to-back runs from the same release (checkout right after a push that
landed 3 seconds earlier) to guarantee a push always succeeds.

Suggested fix

Give the Commit step in release-health.yml the same rebase-and-retry
loop reusable-synchronize-mip-package.yml already uses, rather than a
bare git push.

Current state

RELEASE_HEALTH.md's pydevices-audioeffects row is stale (0.1.1, audioif's
run) as of this filing. Not urgent — cosmetic only, the release it describes
(audiocomponents v0.2.0) shipped correctly on TestPyPI — but it will keep
happening on every future audiocomponents or audioif release, since both
report multiple distributions from one tag.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions