Publish all project documentation on smallrye.io, from the projects' repositories

.github/workflows/build.yml

@@ -20,7 +20,7 @@ on:
       - 'NOTICE'
       - 'README*'
   # Daily, so a post dated later than the merge publishes on its date
-  # (future-dated posts are left out until then), and the docs pick up new releases.
+  # (future-dated posts are left out until then).
   schedule:
     - cron: "0 6 * * *"
   workflow_dispatch:
@@ -34,33 +34,13 @@ jobs:
   build:
     runs-on: ubuntu-latest
 
-    services:
-      kroki:
-        image: yuzutech/kroki
-        env:
-          KROKI_MERMAID_HOST: kroki-mermaid
-        ports:
-          - 8000:8000
-      kroki-mermaid:
-        image: yuzutech/kroki-mermaid
-
     steps:
       - name: checkout
         uses: actions/checkout@v4
         with:
           # The sitemap plugin reads each page's last change from git history.
           fetch-depth: 0
 
-      # Before the site: the Roq build leaves a node_modules/ in the repository root, and
-      # npm would install Antora there instead of in docs/.
-      - name: generate docs
-        run: |
-          cd docs
-          # asciidoctor-kroki must stay on 0.18.1: since 1.0.0, its extensions are
-          # async and require Asciidoctor.js 4, while Antora 2.3 uses Asciidoctor.js 1.5.9
-          npm i @antora/cli@2.3 @antora/site-generator-default@2.3 asciidoctor-kroki@0.18.1
-          ./node_modules/@antora/cli/bin/antora --stacktrace generate antora-playbook.yml
-
       - name: generate website
         uses: quarkiverse/quarkus-roq@v1.1
         with:
@@ -72,8 +52,23 @@ jobs:
       - name: put the stylesheet in the pages
         run: java .github/InlineStyles.java target/roq
 
-      - name: add docs to the website
-        run: cp -r docs/build/site target/roq/docs
+      # GitHub Pages cannot redirect: the documentation addresses this site published before
+      # (Antora's /docs/<project>/<release>/<page>.html) get small redirect pages.
+      - uses: jbangdev/setup-jbang@2b1b465a7b75f4222b81426f23a01e013aa7b95c # v0.1.1
+        with:
+          version: 0.138.0
+
+      # An older documentation version stores only the pages that differ from the next newer version:
+      # write the others, each in its own version's frame.
+      - name: add the documentation pages versions share
+        run: jbang tools/docs/Docs.java inherit target/roq
+
+      # Search for the documentation pages: a static index per project, filtered by version.
+      - name: index the documentation for search
+        run: tools/docs/search-index.sh target/roq
+
+      - name: add redirects for moved documentation
+        run: jbang tools/docs/Docs.java redirects target/roq
 
       - name: attach website
         uses: actions/upload-artifact@v4

.github/workflows/docs-check.yml

@@ -0,0 +1,91 @@
+name: Check project documentation
+
+# Reusable: a SmallRye project calls this on pull requests that touch its docs. It renders the
+# pull request's AsciiDoc with the smallrye.io site exactly as it will be published, fails on a
+# missing include or a page that does not render, and attaches the rendered docs as an artifact.
+#
+#   jobs:
+#     docs:
+#       uses: smallrye/smallrye.github.io/.github/workflows/docs-check.yml@develop
+#       with:
+#         project: smallrye-graphql      # the key in smallrye.github.io data/docs.yml
+
+on:
+  workflow_call:
+    inputs:
+      project:
+        description: 'Project key in data/docs.yml'
+        type: string
+        required: true
+
+# It renders pull-request content: read-only, no secrets.
+permissions:
+  contents: read
+
+jobs:
+  check:
+    runs-on: ubuntu-latest
+    # Diagram blocks ([mermaid], [plantuml], ...) are rendered to SVG by the sync (tools/docs/Docs.java).
+    services:
+      kroki:
+        image: yuzutech/kroki
+        env:
+          KROKI_MERMAID_HOST: kroki-mermaid
+        ports:
+          - 8000:8000
+      kroki-mermaid:
+        image: yuzutech/kroki-mermaid
+    env:
+      KROKI_URL: http://localhost:8000
+    steps:
+      - name: checkout the project
+        uses: actions/checkout@v4
+        with:
+          path: project
+
+      - name: checkout smallrye.io
+        uses: actions/checkout@v4
+        with:
+          repository: smallrye/smallrye.github.io
+          ref: develop
+          path: site
+
+      - uses: actions/setup-java@v4
+        with:
+          distribution: temurin
+          java-version: 21
+          cache: maven
+          cache-dependency-path: site/pom.xml
+
+      - uses: jbangdev/setup-jbang@2b1b465a7b75f4222b81426f23a01e013aa7b95c # v0.1.1
+        with:
+          version: 0.138.0
+
+      - name: copy the docs into the site
+        env:
+          PROJECT: ${{ inputs.project }}
+        run: jbang site/tools/docs/Docs.java sync --strict --project "$PROJECT" --version main --source project
+
+      - name: render the site
+        working-directory: site
+        env:
+          QUARKUS_ROQ_GENERATOR_BATCH: 'true'
+        run: |
+          ./mvnw -B -q package quarkus:run -DskipTests 2>&1 | tee ../render.log
+          # Asciidoctor warnings (such as a section level out of sequence) are reported; a page that
+          # fails to render, or an include that cannot be resolved, fails the check.
+          grep -E 'WARNING \[asciidoctor\]' ../render.log | sed 's/^.*WARNING \[asciidoctor\] ([^)]*) /::warning::/' || true
+          if grep -E 'Roq request failed|Error occurred while rendering|SEVERE \[asciidoctor\]' ../render.log; then
+            echo "::error::The documentation does not render cleanly, see the lines above"
+            exit 1
+          fi
+
+      - name: add the pages versions share
+        run: jbang site/tools/docs/Docs.java inherit site/target/roq
+
+      - name: attach the rendered docs
+        uses: actions/upload-artifact@v4
+        with:
+          name: docs-${{ inputs.project }}
+          path: site/target/roq
+          retention-days: 7

.github/workflows/docs-sync.yml

@@ -0,0 +1,101 @@
+name: Sync project documentation
+
+# Pulls each project's AsciiDoc documentation (data/docs.yml) into content/docs/ and commits it to
+# develop, which then builds and deploys the site. Nightly it picks up new commits on the projects'
+# main branches and new release tags; it can also be run by hand for one project or one version.
+
+on:
+  schedule:
+    - cron: "30 4 * * *"
+  workflow_dispatch:
+    inputs:
+      project:
+        description: 'Project key from data/docs.yml (empty: all projects)'
+        required: false
+      version:
+        description: 'Version id (main, latest, a minor such as 2.9, a stream id); with ref, syncs exactly that'
+        required: false
+      ref:
+        description: 'Branch or tag to sync the version from'
+        required: false
+      force:
+        description: 'Sync even when nothing changed upstream'
+        type: boolean
+        default: false
+
+concurrency:
+  group: docs-sync
+  cancel-in-progress: false
+
+jobs:
+  sync:
+    if: github.repository_owner == 'smallrye'
+    runs-on: ubuntu-latest
+    # Diagram blocks ([mermaid], [plantuml], ...) are rendered to SVG by the sync (tools/docs/Docs.java).
+    services:
+      kroki:
+        image: yuzutech/kroki
+        env:
+          KROKI_MERMAID_HOST: kroki-mermaid
+        ports:
+          - 8000:8000
+      kroki-mermaid:
+        image: yuzutech/kroki-mermaid
+    env:
+      KROKI_URL: http://localhost:8000
+    steps:
+      - uses: actions/create-github-app-token@v1
+        id: app-token
+        with:
+          app-id: ${{ vars.CI_APP_ID }}
+          private-key: ${{ secrets.CI_APP_PRIVATE_KEY }}
+
+      - uses: actions/checkout@v4
+        with:
+          ref: develop
+          # A push made with the app token (not GITHUB_TOKEN) starts the website build.
+          token: ${{ steps.app-token.outputs.token }}
+
+      - uses: actions/setup-java@v4
+        with:
+          distribution: temurin
+          java-version: 21
+
+      - uses: jbangdev/setup-jbang@2b1b465a7b75f4222b81426f23a01e013aa7b95c # v0.1.1
+        with:
+          version: 0.138.0
+
+      # Releases still documented with MkDocs are converted with pandoc.
+      - name: install pandoc
+        run: sudo apt-get update -qq && sudo apt-get install -y -qq pandoc
+
+      - name: sync
+        env:
+          PROJECT: ${{ inputs.project }}
+          VERSION: ${{ inputs.version }}
+          REF: ${{ inputs.ref }}
+          FORCE: ${{ inputs.force }}
+        run: |
+          if [ -n "$VERSION" ] && [ -n "$REF" ]; then
+            args=(--project "$PROJECT" --version "$VERSION" --ref "$REF")
+          else
+            args=(--auto)
+            [ -n "$PROJECT" ] && args+=(--project "$PROJECT")
+            [ "$FORCE" = true ] && args+=(--force)
+          fi
+          jbang tools/docs/Docs.java sync "${args[@]}" --summary "$GITHUB_STEP_SUMMARY"
+
+      - name: commit
+        run: |
+          git add -A content/docs data/docversions.yml
+          if git diff --cached --quiet; then
+            echo "Documentation unchanged"
+            exit 0
+          fi
+          slug='${{ steps.app-token.outputs.app-slug }}'
+          git config user.name "$slug[bot]"
+          git config user.email "$(gh api "/users/$slug%5Bbot%5D" --jq .id)+$slug[bot]@users.noreply.github.com"
+          git commit -q -m "Sync project documentation"
+          git push origin develop
+        env:
+          GH_TOKEN: ${{ steps.app-token.outputs.token }}

.gitignore

@@ -6,3 +6,4 @@ package-lock.json
 .idea
 .vscode
 .quarkus
+__pycache__/

README.md

@@ -17,6 +17,9 @@ To generate the static site the way CI does:
 
 ```bash
 QUARKUS_ROQ_GENERATOR_BATCH=true ./mvnw package quarkus:run
+jbang tools/docs/Docs.java inherit target/roq     # the documentation pages older versions share
+tools/docs/search-index.sh target/roq             # the documentation search (Pagefind, needs Node)
+jbang tools/docs/Docs.java redirects target/roq   # the documentation's old addresses
 ```
 
 The output lands in `target/roq/`.
@@ -51,22 +54,107 @@ Add `content/posts/YYYY-MM-DD-name.adoc`. The header carries the post's details:
 `:page-author:` is a key in `data/authors.yml`; add yourself there first. A post dated in the
 future is published by the first build on or after that date (the workflow runs daily).
 
-## Building the documentation
+## Project documentation
 
-The documentation at `/docs/` is built separately with Antora, from the repositories listed in
-`docs/antora-playbook.yml`, and copied into the site by CI.
+Each project keeps its documentation in its own repository, as AsciiDoc laid out like the
+Quarkus guides (`docs/src/main/asciidoc/`: one page per `.adoc` file, `_nav.yml` for the sidebar,
+`_attributes.adoc` for shared attributes, examples included from compiled sources). This site
+pulls it in and publishes it at `/docs/<project>/<version>/`.
 
-1. Install Antora. See [Antora installation](https://docs.antora.org/antora/2.3/install/install-antora/).
-2. Go to the `docs` directory and run
+| What | Where |
+| --- | --- |
+| The projects, their repositories and the versions followed | `data/docs.yml` |
+| The synced pages, one directory per version | `content/docs/<project>/<version>/` |
+| Synced versions, their refs, release dates and navigation (generated) | `data/docversions.yml` |
+| The documentation layout and its sidebar | `templates/layouts/doc.html`, `templates/partials/docs-nav.html` |
+| The `/docs/` hub | `content/docs/index.html` |
+| Sync and conversion tools | `tools/docs/` |
+
+The tools are one [JBang](https://www.jbang.dev) program, `tools/docs/Docs.java`: Java and JBang are
+all they need (pandoc, or podman to run it in a container, only to convert releases still
+documented with MkDocs).
+
+`sync` copies a project's docs from a branch or tag (or a local checkout) into `content/docs/`.
+The `Sync project documentation` workflow runs it nightly: each project's `main` branch becomes
+`main`, its newest release tag becomes `latest`, and the minors of the latest two major lines become
+`<major>.<minor>` (`/docs/smallrye-mutiny/2.9/`, from 2.9's newest release; older major lines keep one
+version each). Versions that disappear (an older minor taken away with `from`) are removed. `hidden`
+keeps a version published but out of the version drop-down; `data/docs.yml` explains both. The commit
+to `develop` then deploys. A release whose docs are still MkDocs or Antora is converted on the fly by
+`mkdocs2adoc` or `antora2adoc`, which are also the tools to convert a project's sources once.
+
+Most minors change few pages, and Roq can only build a few thousand. So an older version stores only
+the pages that read differently from the same page in the next newer version: its text, the examples
+and files it includes, and the attributes it uses (a page showing `{project-version}` is always
+stored). The others are listed in the version's `_inherited.yml`. After Roq, `Docs.java inherit
+target/roq` writes them into the generated site: the stored page, its links moved to the version,
+with the version's own picker, sidebar, notice and Javadoc version. `quarkus:dev` shows only the
+stored pages of older versions.
+
+Each documentation page has a search of its version: `tools/docs/search-index.sh` builds a
+[Pagefind](https://pagefind.app) index per project after Roq (`/pagefind/<project>/`, filtered by
+version), loaded by the browser only when someone searches. In `quarkus:dev` and the project
+previews there is no index, and the search says so. The header also links the project's
+repository, with its latest release; the browser fetches its stars and forks from GitHub.
+
+```bash
+jbang tools/docs/Docs.java sync --auto                             # what the nightly job does
+jbang tools/docs/Docs.java sync --project smallrye-graphql --version main --source ../smallrye-graphql
+jbang tools/docs/Docs.java --help                                  # every tool
+```
+
+### Previewing a project's documentation
+
+In a project's checkout, `jbang docs-preview@smallrye/smallrye.github.io` (or `./mvnw -Pdocs-preview`
+where the project has that profile) shows its documentation on a local smallrye.io at
+`http://localhost:8080/docs/<project>/main/`, and publishes it again whenever a page or an included
+example changes. It downloads this site's sources once a day; `--site ../smallrye.github.io` uses a
+checkout instead, to try changes to the site itself.
+
+Projects call `.github/workflows/docs-check.yml` from their pull requests: it renders their docs
+with this site and fails on a broken include or a page that does not render.
+
+Converting a project once (then commit the result in the project's repository):
+
+```bash
+jbang tools/docs/Docs.java mkdocs2adoc --mkdocs ../project/mkdocs.yml --out ../project/docs/src/main/asciidoc
+jbang tools/docs/Docs.java antora2adoc --antora ../project/doc --out ../project/docs/src/main/asciidoc
+```
+
+Diagram blocks (`[mermaid]`, `[plantuml]`, ...) are rendered to SVG during the sync, by a Kroki
+server (`KROKI_URL`, https://kroki.io by default; the workflows run their own).
+
+On the dark theme, the docs pages show a dark version of each drawing (`DarkImages.java`, part of the
+sync), so diagrams do not sit on a light block:
+
+- a dark version the project drew itself, next to the image: `x.dark.png` for `x.png`, or `x-dark.png`
+  for `x-light.png`;
+- otherwise the sync makes one: diagrams get their colours turned around (same hue, lightness inverted),
+  and so do PNGs that look like drawings (mostly white or transparent, with darker lines), their white
+  becoming transparent. Screenshots and photos keep a light paper behind them;
+- a project opts an image out with the `no-dark` role (`image::x.png[...,role=no-dark]`), or in with
+  `dark-invert` (an SVG, or a PNG the check passes over).
+
+`jbang tools/docs/Docs.java dark-images content/docs/<project>/<version>` makes them again for a
+synced version.
+
+### Old addresses
 
-        antora generate antora-playbook.yml --fetch
+The documentation addresses this site published before (Antora's
+`/docs/<project>/<release>/<page>.html`) are listed under `legacy` in `data/docs.yml`. The build
+writes a redirect page for each of them (`Docs.java redirects`), and
+`Docs.java rewrite-links --check target/roq content/posts/*.adoc` points blog links at the new
+addresses, checking every page and anchor against the generated site.
 
-3. The HTML for the SmallRye documentation is now in `docs/build/site`.
+The projects' old MkDocs sites (their `gh-pages` branches, at `smallrye.io/<project>/<version>/`) are
+replaced by redirect pages: `Docs.java gh-pages-redirects --project <key> --checkout <gh-pages checkout>
+--site target/roq` sends every page of every version to the same page of the version this site
+publishes for that minor (or major) line, or the next newer one. Javadoc stays as it is.
 
 ## Deployment
 
 Pushing to `develop` runs `.github/workflows/build.yml`: it generates the site with the Roq
-GitHub Action, builds the documentation with Antora, and pushes the result to the `main`
+GitHub Action (project docs included), adds the redirects for old documentation addresses, and pushes the result to the `main`
 branch, which GitHub Pages serves. Pull requests build the site without deploying it. The
 workflow also runs daily and can be started by hand from the Actions tab.