develop CI watch: red or missing push runs

[umeradl]

This is a long-running thread for reporting problems with CI on develop. Keep it open indefinitely: each incident gets its own new comment thread here.

Maintainers on CI watch: @Santiagocetran and @robertocarlous

Why this exists

Most merges into develop are done locally and pushed directly, and until now CI never ran on those pushes. PR No. 168 adds a push trigger, so every commit pushed to develop now gets its own CI run (Solidity, Python, Docs, CI OK). That run doesn't block anything, because the commit has already landed. It only shows whether the landed code is healthy, so someone has to look at it.

How it works

Normally, I handle it. When I push to develop, I make sure the push's CI run passes, or I add a fix commit on top to make it pass.

When it becomes critical, you step in. If a red (failed) or missing (never started) run on develop has gone unaddressed for about 6 hours or more, and more commits have landed on top of it, then:

• the breakage is being built on, and
• every later commit inherits the failure, so it's harder to tell which change broke what.

At that point, please don't wait for me: open a thread here (format below).

How to check

# latest push runs on develop: event, status, commit
gh run list --branch develop --event push --limit 10

# the run for one exact commit
gh run list --commit <sha>

Or open the Actions tab (filter: branch develop, event push), or the commit on GitHub, where a ✓/✗ next to it links to the run.

To find the commit that introduced the failure, walk back through the list to the oldest red run after the last green one. That commit is usually the culprit, not the newest one.

Reporting an incident: open a new top-level comment here

One thread per incident. Please don't reply inside someone else's incident thread unless it's the same failure. Include:

• Commit: the full SHA of the first failing (or run-less) commit on develop, plus the latest develop SHA if more have landed since
• Run: URL of the failing run, or "no run appeared"
• Failing job / test: e.g. Python → tests/test_x.py::TestY::test_z, with the key error line
• What's actually wrong: your read of the cause, if you have one (e.g. "test depends on the local config file", "contract change broke the upgrade-validation test")
• Since when: how long it has been red, and how many commits landed on top
• Proposed fix: optional. If you open a fix PR, link it.

I'll reply in the thread. Once it's fixed, the thread gets a final comment with the fixing commit SHA and the green run URL, so this discussion doubles as the incident log.

Notes

• This starts once PR No. 168 is merged. Before that, develop pushes have no CI run at all.
• The ruff (advisory) and forge lint (advisory) steps can show ##[error] without failing the job. Only a red job or a red CI OK counts as an incident.
• Related: CI/CD proposal in discussion No. 74.