Skip to content

feat(ci): add /explain-ci bot command for PR failure help - #60821

Open
miaulalala wants to merge 2 commits into
masterfrom
feat/noid/explain-ci-bot
Open

feat(ci): add /explain-ci bot command for PR failure help#60821
miaulalala wants to merge 2 commits into
masterfrom
feat/noid/explain-ci-bot

Conversation

@miaulalala

@miaulalala miaulalala commented May 28, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds .github/workflows/ci-bot.yml: a new slash-command bot triggered by commenting /explain-ci on any pull request
  • When invoked, nextcloud-bot reacts to the comment with 👍 and posts a single summary of all currently-failing CI checks, explaining what each one validates and how to fix it locally
  • Covers all ~40 developer-facing PR workflows with specific fix instructions
  • Extends autotest.sh with Docker-backed service helpers for local reproduction of external-service CI failures

How the bot works

The bot uses the issue_comment trigger (same pattern as /compile and /update-3rdparty), so it has access to COMMAND_BOT_PAT and works on fork PRs too. It:

  1. Fetches all completed workflow runs for the PR's current commit SHA via the Actions API
  2. Deduplicates by workflow name (takes the most recent run per workflow)
  3. Looks up each failing workflow in a static explanation map
  4. Posts a formatted summary, or a "no failures found" message if everything is green

Workflows covered

All PHPUnit variants (SQLite, MariaDB, MySQL, PostgreSQL, OCI, nodb, 32bits, memcached, key-value store, sharding, primary object store, all files_external backends), ESLint, Stylelint, PHP lint, PHP-CS-Fixer, Psalm, Cypress, Playwright, Node build/tests, REUSE, OpenAPI, Rector, Behat integration tests (DAV, Litmus, SQLite), object storage tests (S3, Azure, Swift), Samba Kerberos SSO, Code checkers, block checks (fixup commits, unconventional commits, outdated 3rdparty/), AI Policy, and CodeQL.

The AI Policy entry also flags a known issue on fork PRs (nextcloud/.github#767): the label-write step can 403 and fail the whole job even when there is no real trailer violation. The bot's explanation calls this out so contributors do not mistake it for a real policy failure.

New autotest.sh Docker service helpers

autotest.sh now supports spinning up Docker-backed external services automatically, making it possible to reproduce CI failures locally without any manual container setup:

Variable What it does
EXTERNAL_STORAGE=ftp Starts vsftpd, runs files_external FTP tests
EXTERNAL_STORAGE=sftp Starts atmoz/sftp, runs files_external SFTP tests
EXTERNAL_STORAGE=smb Starts Samba, runs files_external SMB tests
EXTERNAL_STORAGE=webdav Starts Apache WebDAV, runs files_external WebDAV tests
EXTERNAL_STORAGE=amazons3 Starts LocalStack, runs files_external S3 tests
PRIMARY_STORAGE_CONFIG=s3 USEDOCKER=1 Starts MinIO as primary object store
PRIMARY_STORAGE_CONFIG=azure USEDOCKER=1 Starts Azurite as primary object store
ENABLE_MEMCACHE=memcached USEDOCKER=1 Starts the same Memcached image CI uses (ghcr.io/nextcloud/continuous-integration-memcached), copies tests/memcached.config.php

Each helper installs Nextcloud, enables the relevant app/config, waits for the service to be healthy, runs the tests, and tears down the container, so there is no manual cleanup needed. Pre-flight checks catch missing system dependencies (e.g. php-smbclient, smbclient binary) before wasting time on an install.

I checked for existing overlap before adding these: autotest-external.sh and the apps/files_external/tests/env/start-*.sh scripts cover similar ground for files_external backends, but current CI (e.g. files-external-ftp.yml) no longer calls them, it sets up services directly in the workflow instead. Those scripts are legacy and unused by CI today; removing them is a separate cleanup and out of scope here.

Example usage

Comment /explain-ci on a PR with failing checks. nextcloud-bot replies:

CI failure summary

Found 2 failing check(s) on commit abc1234:


Lint eslint

What this checks: Checks JavaScript/TypeScript/Vue source files for ESLint rule violations.

How to fix locally:

  • Run npm run lint to see all violations.
  • Run npm run lint:fix to auto-fix many of them, then review remaining errors manually.

View failed run

Test plan

  • Comment /explain-ci on a PR with at least one failing check, bot reacts with 👍 and posts summary
  • Comment /explain-ci on a PR where all checks pass, bot replies "no failing checks found"
  • Comment /explain-ci on a fork PR, bot works (issue_comment runs in base repo context)
  • Re-run a failed check, then comment /explain-ci, only still-failing checks appear in summary
  • EXTERNAL_STORAGE=ftp NOCOVERAGE=1 ./autotest.sh runs all FTP tests and passes
  • EXTERNAL_STORAGE=amazons3 NOCOVERAGE=1 ./autotest.sh runs all S3 tests and passes
  • PRIMARY_STORAGE_CONFIG=s3 USEDOCKER=1 NOCOVERAGE=1 ./autotest.sh sqlite lib/Files/ObjectStore/S3Test.php runs MinIO tests and passes
  • ENABLE_MEMCACHE=memcached USEDOCKER=1 NOCOVERAGE=1 ./autotest.sh sqlite lib/Path/To/SpecificTest.php starts Memcached on port 11211 and passes

Note on upcoming compiled-assets change

Per the engineering list announcement (2026-07-21), asset compilation is moving out of the PR workflow into a post-merge step starting early August 2026, with a new check to reject PRs that include compiled assets. That workflow (update-node-dist.yml) is not on master yet, so the bot does not cover it in this PR. Once it lands, the Node explanation entry should be revisited and a new entry added for the compiled-assets check.

🤖 Generated with Claude Code

@miaulalala
miaulalala requested a review from a team as a code owner May 28, 2026 20:42
@miaulalala
miaulalala requested review from ArtificialOwl, icewind1991, leftybournes and salmart-dev and removed request for a team May 28, 2026 20:42
@miaulalala
miaulalala marked this pull request as draft May 28, 2026 21:18
@miaulalala
miaulalala force-pushed the feat/noid/explain-ci-bot branch 2 times, most recently from 063c0a7 to a767e76 Compare June 1, 2026 05:39
@miaulalala
miaulalala requested a review from provokateurin June 1, 2026 06:59
@miaulalala miaulalala self-assigned this Jun 1, 2026
Comment thread autotest.sh
with:
github-token: ${{ secrets.COMMAND_BOT_PAT }}
script: |
const WORKFLOW_EXPLANATIONS = {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I fear that the keys will get out of sync and we won't notice. I don't have a good solution for it though.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wdym? The token?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, you mean the workflow names? Well, no use worrying about it for now. This is a test run more or less. If the bot works well for our community contributors, we can move it to the workflow repo and find a different solution for it.

@susnux susnux added this to the Nextcloud 35 milestone Jun 9, 2026
- Adds `.github/workflows/ci-bot.yml`: slash command `/explain-ci` on
  any PR triggers nextcloud-bot to post a summary of all currently-
  failing checks with purpose, fix steps, Docker commands, and
  prerequisites. Covers all ~40 PR-facing workflows. Uses the
  `issue_comment` trigger (works on fork PRs) and posts via
  `COMMAND_BOT_PAT`, matching the `/compile` and `/update-3rdparty`
  pattern.

- Extends `autotest.sh` with Docker-backed service helpers so external-
  service CI failures can be reproduced locally without manual setup:
  - `EXTERNAL_STORAGE=ftp/sftp/smb/webdav/amazons3` — spins up the
    relevant service container, installs Nextcloud, enables
    files_external, runs the storage tests, and tears down the container
  - `PRIMARY_STORAGE_CONFIG=s3/azure USEDOCKER=1` — spins up MinIO or
    Azurite as primary object store
  - `ENABLE_MEMCACHE=memcached USEDOCKER=1` — spins up Memcached
  - Pre-flight checks warn about missing system dependencies before
    wasting time on an install

- Adds `tests/memcached.config.php` for Memcached cache configuration

AI-Assisted-By: claude-sonnet-4-6 <noreply@anthropic.com>
Signed-off-by: Anna Larch <anna@nextcloud.com>
…n-ci

Add explanation-map entries for the three workflows introduced upstream
since this branch was created, and fix the Memcached Docker helper in
autotest.sh to use the same image/port as the CI service
(ghcr.io/nextcloud/continuous-integration-memcached on 11211) instead of
a mismatched generic memcached image on 11212.

Signed-off-by: Anna Larch <anna@nextcloud.com>
@miaulalala
miaulalala force-pushed the feat/noid/explain-ci-bot branch from a767e76 to bc14d46 Compare July 21, 2026 19:28
@miaulalala
miaulalala marked this pull request as ready for review July 21, 2026 19:33
@miaulalala
miaulalala requested a review from provokateurin July 21, 2026 19:33
@miaulalala miaulalala added 3. to review Waiting for reviews feature: workflows and removed 2. developing Work in progress labels Jul 21, 2026
@susnux

susnux commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Nice idea!

I fear this will get out of sync very quickly with what workflows do etc.
Also I think such slash commands are pretty undiscoverable as nothing in github advertises they exist.

Why not instead improve the failing workflows?
Instead of just failing they should instead print a nice reason why they failed, e.g. we did a first step in cypress.

Error: Skipping Cypress: draft state, missing labels or no cypress path changes.

Which could of course improved, a bit. But then you directly can see why it failed and also you can explain what it does.
And the best part here is that the documentation lives next to the code so its less likely to get out-of-sync.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants