From 3d55d90ab12c68e908104ef8fcd26f805a807df0 Mon Sep 17 00:00:00 2001 From: Dimitrie Hoekstra Date: Mon, 3 Aug 2026 11:35:38 +0200 Subject: [PATCH 1/4] Resolve product docs from the caller's checkout The docs source is now resolved in a documented order: FLOWFUSE_DOCS_LOCAL, a sibling flowfuse checkout, the snapshot committed to live, then a clone. Sibling detection is the convention CI depends on, so a docs PR is validated against its own changes rather than whatever main happens to be. Clone with full history so each page is dated from its own last commit, and have the build workflow commit the resolved docs onto live so production deploys use a pinned snapshot instead of cloning at deploy time. --- .claude/CLAUDE.md | 9 +- .github/workflows/build.yml | 17 ++++ README.md | 26 ++++- netlify.toml | 5 + nuxt/lib/docs-sync.mjs | 197 ++++++++++++++++++++++++++++++++++++ nuxt/lib/docs-sync.test.mjs | 77 ++++++++++++++ nuxt/modules/docs-source.ts | 136 +------------------------ package.json | 4 +- scripts/sync_docs.mjs | 12 +++ 9 files changed, 343 insertions(+), 140 deletions(-) create mode 100644 nuxt/lib/docs-sync.mjs create mode 100644 nuxt/lib/docs-sync.test.mjs create mode 100644 scripts/sync_docs.mjs diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 96c11e5373..1a0812f46d 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -17,7 +17,7 @@ The site is migrating from Eleventy (11ty) to Nuxt 3. Nuxt is the primary framew | Section | Status | |---------|--------| | `/handbook/**` | **Migrated** — served by Nuxt (`nuxt/content/handbook/`) | -| `/docs/**` | **Migrated** — served by Nuxt; source cloned from `flowfuse/flowfuse` at build time | +| `/docs/**` | **Migrated** — served by Nuxt; source resolved from `flowfuse/flowfuse` at build time | | All other routes | Still on 11ty, proxied through Nuxt in dev | ### Production build order @@ -26,7 +26,7 @@ The site is migrating from Eleventy (11ty) to Nuxt 3. Nuxt is the primary framew clean:nuxt → build:js:nuxt → prod:postcss-nuxt → prod:eleventy-nuxt → prod:nuxt ``` -The `docs-source` Nuxt module runs automatically during `prod:nuxt` and sparse-clones `docs/` from `flowfuse/flowfuse` (public repo, no token needed). 11ty outputs to `nuxt/public/` so Nuxt can serve 11ty-generated assets. `nuxt/public/` is gitignored (fully build-generated). +The `docs-source` Nuxt module runs automatically during `prod:nuxt` and calls `nuxt/lib/docs-sync.mjs` to resolve `docs/` from `flowfuse/flowfuse` (see **Local docs development** below). 11ty outputs to `nuxt/public/` so Nuxt can serve 11ty-generated assets. `nuxt/public/` is gitignored (fully build-generated). ## Dev commands @@ -35,12 +35,13 @@ npm start # all watchers in parallel (11ty + nuxt + postcss + bluep npm run dev # eleventy + postcss + nuxt only npm run dev:eleventy # 11ty only, port 8080 (legacy; most work doesn't need this) npm run dev:nuxt # Nuxt only, port 3000 — use this for handbook, docs, and migrated pages +npm run docs # resolve product docs into nuxt/content/docs, no build npm run build # production build ``` > When working on the handbook, docs, or other migrated sections, `npm run dev:nuxt` is sufficient. `npm start` is only needed when also touching 11ty-served pages. > -> **Local docs development:** set `FLOWFUSE_DOCS_LOCAL=/path/to/flowfuse` to point the docs module at a local checkout instead of cloning from GitHub. If the env var is not set and `nuxt/content/docs/` already exists, that cached copy is used. If neither is true, the module clones fresh from GitHub (public, no token needed). +> **Local docs development:** a checkout of `flowfuse/flowfuse` sitting next to this repo (`../flowfuse`) is picked up automatically, with no configuration. Full resolution order, which every build logs: `FLOWFUSE_DOCS_LOCAL` (explicit path, and a path that does not exist is an error), then a sibling checkout, then the snapshot committed to `live` when `FLOWFUSE_DOCS_SNAPSHOT` is set (Netlify only), then a clone of `FLOWFUSE_DOCS_REF` (default `main`). CI relies on the sibling rule: `FlowFuse/flowfuse`'s `Publish Documentation` workflow checks itself out next to the website so a docs PR is validated against its own changes. ## Directory layout @@ -61,7 +62,7 @@ nuxt/ │ ├── handbook/ # Handbook pages (edit here) │ └── docs/ # Product docs (build-generated, gitignored — do not edit) ├── modules/ -│ └── docs-source.ts # Clones docs from flowfuse/flowfuse at build time +│ └── docs-source.ts # Wires docs into Nuxt; resolution lives in nuxt/lib/docs-sync.mjs ├── composables/ │ ├── useHandbookNav.ts │ └── useDocsNav.ts diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 4385752e15..1db4b1c56f 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -16,6 +16,14 @@ jobs: uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: path: 'website' + - name: Check out FlowFuse/flowfuse repository (to access the docs) + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + repository: 'FlowFuse/flowfuse' + ref: main + path: 'flowfuse' + # Full history: each docs page is dated from its own last commit. + fetch-depth: 0 - name: Generate a token id: generate_token uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0 @@ -37,8 +45,17 @@ jobs: node-version: 24 cache: 'npm' cache-dependency-path: './website/package-lock.json' + - run: npm run docs + working-directory: 'website' - run: npm run blueprints working-directory: 'website' + - name: Commit Latest Docs + run: | + cd ./website + git config --local user.email "41898282+github-actions[bot]@users.noreply.github.com" + git config --local user.name "github-actions[bot]" + git add nuxt/content/docs nuxt/public/docs -A -f + git commit -a -m "Bot: update docs" - name: Commit Latest Blueprints run: | cd ./website diff --git a/README.md b/README.md index e913bbfba9..4ba7ca00f0 100644 --- a/README.md +++ b/README.md @@ -5,8 +5,8 @@ This repository contains the source of the FlowFuse website. It is hosted on Netlify with each commit to the `main` branch being automatically deployed to the live site. -This works by a GitHub action automatically updating the `live` branch to includes documentation pulled from the `main` branch of the [FlowFuse/flowfuse](https://github.com/FlowFuse/flowfuse) -repository, when changes are pushed to `main`. +This works by the [Build Site](.github/workflows/build.yml) action updating the `live` branch, committing onto it the +product documentation pulled from the `main` branch of [FlowFuse/flowfuse](https://github.com/FlowFuse/flowfuse). Netlify is then configured to watch the `live` branch for any changes, once detected, it will automatically pull the contents of this branch (docs included) and deploy to our production site. @@ -95,6 +95,23 @@ The documentation for FlowFuse is maintained in the core [FlowFuse repo](https:/ The `npm run dev` (and `npm start`) commands will retrieve the documentation from that folder and inject them into the site automatically. The docs will be available at http://localhost:3000/docs. +Nothing needs configuring for that to happen. Every build resolves the docs in this order, and logs which one it used: + +| Order | Source | Used when | +|-------|--------|-----------| +| 1 | `FLOWFUSE_DOCS_LOCAL=/path/to/flowfuse` | The env var is set. A path that does not exist is an error, not a fallback. | +| 2 | A sibling checkout: `../flowfuse`, `../flowforge` or `../dev-env/packages/flowfuse` | One of those has a `docs/` directory. This is what CI relies on. | +| 3 | The snapshot committed to `live` | `FLOWFUSE_DOCS_SNAPSHOT` is set, which Netlify does. Production deploys never clone. | +| 4 | A clone of `FLOWFUSE_DOCS_REF` (default `main`) | Nothing above applied. | + +`npm run docs` runs that resolution on its own, without a full build, writing `nuxt/content/docs` and `nuxt/public/docs`. Both are generated, and neither is committed on `main`. + +If the docs and handbook pages fail to render locally while the rest of the site is fine, you are hitting [nuxt#35253](https://github.com/nuxt/nuxt/issues/35253). Give the build its own temp directory: + +```bash +export TMPDIR=/tmp/nuxt +``` + ## How to add blog posts See the [Blog section of the Marketing Handbook](https://flowfuse.com/handbook/marketing/content-strategy/blog/) for instructions on writing and publishing blog posts. @@ -109,7 +126,10 @@ To make a documentation update *and* make it live on the website: 1. PR the documentation update to the `main` branch of [FlowFuse/flowfuse](https://github.com/FlowFuse/flowfuse) 2. Get the PR reviewed and merged in the normal manner. -3. Manually kick-off a website rebuild by clicking 'Run workflow' on [this page](https://github.com/FlowFuse/website/actions/workflows/build.yml). + +That repository's `Publish Documentation` workflow builds this site against the PR's docs before it can merge, then +triggers a website rebuild once it lands. A rebuild can also be started by hand with 'Run workflow' on +[this page](https://github.com/FlowFuse/website/actions/workflows/build.yml). ## Acknowledgements diff --git a/netlify.toml b/netlify.toml index 39cf145a24..580aacd7f4 100644 --- a/netlify.toml +++ b/netlify.toml @@ -1,3 +1,8 @@ +[build.environment] + # Deploy the docs snapshot the build workflow committed to 'live' rather than cloning + # flowfuse at deploy time. Branches without a snapshot fall back to cloning. + FLOWFUSE_DOCS_SNAPSHOT = "1" + [[headers]] for = "/*" [headers.values] diff --git a/nuxt/lib/docs-sync.mjs b/nuxt/lib/docs-sync.mjs new file mode 100644 index 0000000000..94ace88d83 --- /dev/null +++ b/nuxt/lib/docs-sync.mjs @@ -0,0 +1,197 @@ +// Resolves the FlowFuse product docs for a build and copies them into nuxt/content/docs. +// Kept free of Nuxt imports so `scripts/sync_docs.mjs` can run it before `npm install`. + +import { execFileSync } from 'node:child_process' +import { cpSync, existsSync, mkdirSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs' +import { join, relative } from 'node:path' +import { tmpdir } from 'node:os' + +import { processMarkdown } from './docs-markdown.mjs' + +const REPO_URL = 'https://github.com/FlowFuse/flowfuse.git' +const DEFAULT_REF = 'main' +const CLONE_ATTEMPTS = 3 +const CLONE_BACKOFF_MS = 2000 + +// Whatever checkout sits next to the website repo wins. CI puts the flowfuse repo there, +// so a build validates the docs of the caller's checkout rather than whatever main +// happens to be. The release pipeline depends on this. +const SIBLING_PATHS = ['../dev-env/packages/flowfuse', '../flowfuse', '../flowforge'] + +export const MANIFEST_FILE = '.source.json' + +const sleep = (ms) => new Promise(resolve => setTimeout(resolve, ms)) + +/** + * Decide where the docs come from. Pure: touches nothing, so the precedence is testable. + * + * 1. `FLOWFUSE_DOCS_LOCAL` - an explicit checkout path + * 2. a sibling checkout of flowfuse + * 3. the snapshot committed to the `live` branch (`FLOWFUSE_DOCS_SNAPSHOT` builds only) + * 4. a clone of `FLOWFUSE_DOCS_REF` + */ +export function resolveSource ({ repoRoot, contentDocsDir, env = process.env, exists = existsSync }) { + const local = env.FLOWFUSE_DOCS_LOCAL + if (local) { + const docsDir = local.endsWith('/docs') ? local : join(local, 'docs') + // A typo here would otherwise fall through and quietly publish main's docs. + if (!exists(docsDir)) { + throw new Error(`FLOWFUSE_DOCS_LOCAL is set but ${docsDir} does not exist`) + } + return { kind: 'local', docsDir } + } + + for (const sibling of SIBLING_PATHS) { + const docsDir = join(repoRoot, sibling, 'docs') + if (exists(docsDir)) { + return { kind: 'sibling', docsDir } + } + } + + if (env.FLOWFUSE_DOCS_SNAPSHOT && exists(join(contentDocsDir, MANIFEST_FILE))) { + return { kind: 'snapshot' } + } + + return { kind: 'clone', ref: env.FLOWFUSE_DOCS_REF || DEFAULT_REF } +} + +/** + * Sparse-clone the docs into a temp dir and return its path. + * + * A transient network failure here would otherwise fail the entire production deploy, so + * each attempt gets a clean temp dir and the network steps are retried with backoff. The + * caller owns cleanup of the returned dir. + */ +async function cloneDocs (ref, logger) { + let lastError + + for (let attempt = 1; attempt <= CLONE_ATTEMPTS; attempt++) { + const tmpDir = join(tmpdir(), `flowfuse-docs-${process.pid}-${attempt}`) + if (existsSync(tmpDir)) rmSync(tmpDir, { recursive: true, force: true }) + + try { + // Blobless but not shallow: dating a page needs that page's history, and a + // --depth=1 clone stamps every page with the same commit date. + execFileSync('git', ['clone', '--filter=blob:none', '--no-checkout', REPO_URL, tmpDir], { stdio: 'pipe' }) + execFileSync('git', ['sparse-checkout', 'set', 'docs'], { cwd: tmpDir, stdio: 'pipe' }) + execFileSync('git', ['checkout', ref], { cwd: tmpDir, stdio: 'pipe' }) + return tmpDir + } catch (err) { + lastError = err + if (existsSync(tmpDir)) rmSync(tmpDir, { recursive: true, force: true }) + + if (attempt === CLONE_ATTEMPTS) break + + const wait = CLONE_BACKOFF_MS * attempt + logger.warn(`Docs clone attempt ${attempt}/${CLONE_ATTEMPTS} failed, retrying in ${wait}ms`) + await sleep(wait) + } + } + + const reason = lastError instanceof Error ? lastError.message : String(lastError) + throw new Error(`Failed to clone FlowFuse docs from ${REPO_URL} after ${CLONE_ATTEMPTS} attempts: ${reason}`) +} + +function gitOutput (cwd, args) { + try { + return execFileSync('git', args, { cwd, encoding: 'utf8' }).trim() + } catch { + return '' + } +} + +function copyDocsDir (srcDir, repoRoot, contentDir, publicDir, version) { + mkdirSync(contentDir, { recursive: true }) + mkdirSync(publicDir, { recursive: true }) + + for (const entry of readdirSync(srcDir, { withFileTypes: true })) { + if (entry.name.startsWith('.')) continue + + const srcPath = join(srcDir, entry.name) + const destName = entry.name === 'README.md' ? 'index.md' : entry.name + + if (entry.isDirectory()) { + copyDocsDir(srcPath, repoRoot, join(contentDir, entry.name), join(publicDir, entry.name), version) + } else if (entry.name.endsWith('.md')) { + const relFromRepo = relative(repoRoot, srcPath) + const originalPath = relative(join(repoRoot, 'docs'), srcPath) + + // Argument array, not a shell string: relFromRepo comes from filenames in the + // source repo, so quoting it into a shell command would be an injection path. + const updated = gitOutput(repoRoot, ['log', '-1', '--pretty=format:%ci', '--', relFromRepo]) + + const raw = readFileSync(srcPath, 'utf8') + writeFileSync(join(contentDir, destName), processMarkdown(raw, originalPath, updated, version), 'utf8') + } else { + cpSync(srcPath, join(publicDir, entry.name)) + } + } +} + +function writeDocs ({ docsDir, sourceRoot, contentDocsDir, publicDocsDir, kind, ref }) { + let version = '' + try { + version = JSON.parse(readFileSync(join(sourceRoot, 'package.json'), 'utf8')).version || '' + } catch { /* not fatal */ } + + rmSync(contentDocsDir, { recursive: true, force: true }) + rmSync(publicDocsDir, { recursive: true, force: true }) + copyDocsDir(docsDir, sourceRoot, contentDocsDir, publicDocsDir, version) + + const manifest = { + source: kind, + ref: ref || gitOutput(sourceRoot, ['rev-parse', '--abbrev-ref', 'HEAD']), + sha: gitOutput(sourceRoot, ['rev-parse', 'HEAD']), + version, + syncedAt: new Date().toISOString(), + } + writeFileSync(join(contentDocsDir, MANIFEST_FILE), `${JSON.stringify(manifest, null, 2)}\n`, 'utf8') + + return manifest +} + +/** + * Populate nuxt/content/docs and nuxt/public/docs, and return the manifest describing + * what was published. + */ +export async function syncDocs ({ repoRoot, nuxtRoot, env = process.env, logger = console } = {}) { + const contentDocsDir = join(nuxtRoot, 'content', 'docs') + const publicDocsDir = join(nuxtRoot, 'public', 'docs') + const source = resolveSource({ repoRoot, contentDocsDir, env }) + + if (source.kind === 'snapshot') { + const manifest = JSON.parse(readFileSync(join(contentDocsDir, MANIFEST_FILE), 'utf8')) + logger.info(`Using committed docs snapshot: ${manifest.ref} ${manifest.sha.slice(0, 8)} synced ${manifest.syncedAt}`) + return manifest + } + + let manifest + if (source.kind === 'clone') { + logger.info(`Cloning FlowFuse docs from ${source.ref}...`) + const tmpDir = await cloneDocs(source.ref, logger) + try { + manifest = writeDocs({ + docsDir: join(tmpDir, 'docs'), + sourceRoot: tmpDir, + contentDocsDir, + publicDocsDir, + kind: source.kind, + ref: source.ref, + }) + } finally { + if (existsSync(tmpDir)) rmSync(tmpDir, { recursive: true, force: true }) + } + } else { + logger.info(`Using ${source.kind} docs from ${source.docsDir}`) + manifest = writeDocs({ + docsDir: source.docsDir, + sourceRoot: join(source.docsDir, '..'), + contentDocsDir, + publicDocsDir, + kind: source.kind, + }) + } + + logger.info(`Docs synced from ${manifest.source} (${manifest.ref} ${manifest.sha.slice(0, 8) || 'unknown'}, version ${manifest.version || 'unknown'})`) + return manifest +} diff --git a/nuxt/lib/docs-sync.test.mjs b/nuxt/lib/docs-sync.test.mjs new file mode 100644 index 0000000000..19b1ac1873 --- /dev/null +++ b/nuxt/lib/docs-sync.test.mjs @@ -0,0 +1,77 @@ +import { test } from 'node:test' +import assert from 'node:assert/strict' + +import { MANIFEST_FILE, resolveSource } from './docs-sync.mjs' + +const repoRoot = '/repo/website' +const contentDocsDir = '/repo/website/nuxt/content/docs' + +const resolve = (env, present = []) => resolveSource({ + repoRoot, + contentDocsDir, + env, + exists: (path) => present.includes(path), +}) + +test('an explicit path wins over a sibling checkout', () => { + const source = resolve( + { FLOWFUSE_DOCS_LOCAL: '/elsewhere/flowfuse' }, + ['/elsewhere/flowfuse/docs', '/repo/flowfuse/docs'], + ) + + assert.deepEqual(source, { kind: 'local', docsDir: '/elsewhere/flowfuse/docs' }) +}) + +test('an explicit path already ending in /docs is used as given', () => { + const source = resolve( + { FLOWFUSE_DOCS_LOCAL: '/elsewhere/flowfuse/docs' }, + ['/elsewhere/flowfuse/docs'], + ) + + assert.equal(source.docsDir, '/elsewhere/flowfuse/docs') +}) + +test('a mistyped explicit path throws rather than falling back', () => { + assert.throws( + () => resolve({ FLOWFUSE_DOCS_LOCAL: '/typo' }, ['/repo/flowfuse/docs']), + /FLOWFUSE_DOCS_LOCAL is set but/, + ) +}) + +test('a sibling checkout is found without configuration', () => { + const source = resolve({}, ['/repo/flowfuse/docs']) + + assert.deepEqual(source, { kind: 'sibling', docsDir: '/repo/flowfuse/docs' }) +}) + +test('the dev-env checkout is preferred over a bare sibling', () => { + const source = resolve({}, ['/repo/dev-env/packages/flowfuse/docs', '/repo/flowfuse/docs']) + + assert.equal(source.docsDir, '/repo/dev-env/packages/flowfuse/docs') +}) + +test('a sibling checkout wins over a committed snapshot', () => { + const source = resolve( + { FLOWFUSE_DOCS_SNAPSHOT: '1' }, + ['/repo/flowfuse/docs', `${contentDocsDir}/${MANIFEST_FILE}`], + ) + + assert.equal(source.kind, 'sibling') +}) + +test('the committed snapshot is used when a build asks for it', () => { + const source = resolve({ FLOWFUSE_DOCS_SNAPSHOT: '1' }, [`${contentDocsDir}/${MANIFEST_FILE}`]) + + assert.deepEqual(source, { kind: 'snapshot' }) +}) + +test('a snapshot left on disk is ignored unless the build asks for it', () => { + const source = resolve({}, [`${contentDocsDir}/${MANIFEST_FILE}`]) + + assert.equal(source.kind, 'clone') +}) + +test('cloning falls back to main and honours an explicit ref', () => { + assert.deepEqual(resolve({}), { kind: 'clone', ref: 'main' }) + assert.equal(resolve({ FLOWFUSE_DOCS_REF: 'maintenance' }).ref, 'maintenance') +}) diff --git a/nuxt/modules/docs-source.ts b/nuxt/modules/docs-source.ts index 287731420b..c25a3f8fa6 100644 --- a/nuxt/modules/docs-source.ts +++ b/nuxt/modules/docs-source.ts @@ -1,107 +1,14 @@ import { defineNuxtModule, useLogger } from '@nuxt/kit' -import { execFileSync } from 'node:child_process' -import { mkdirSync, cpSync, writeFileSync, readFileSync, rmSync, existsSync, readdirSync } from 'node:fs' -import { join, basename, relative, dirname } from 'node:path' -import { tmpdir } from 'node:os' +import { existsSync, readdirSync } from 'node:fs' +import { join, basename, dirname } from 'node:path' // Lives in nuxt/lib/, not alongside this file: Nuxt auto-registers everything in // nuxt/modules/ as a Nuxt module, so a plain helper there fails the build. // @ts-ignore untyped module, kept as plain JS so `node --test` can run it directly -import { processMarkdown } from '../lib/docs-markdown.mjs' +import { syncDocs } from '../lib/docs-sync.mjs' const logger = useLogger('docs-source') -const CLONE_ATTEMPTS = 3 -const CLONE_BACKOFF_MS = 2000 - -const GROUP_ORDER = [ - 'FlowFuse User Manuals', - 'Device Agent', - 'FlowFuse Cloud', - 'FlowFuse Self-Hosted', - 'Support', - 'Contributing', -] - -const sleep = (ms: number) => new Promise(resolve => setTimeout(resolve, ms)) - -/** - * Sparse-clone the docs from FlowFuse/flowfuse into a temp dir and return its path. - * - * A transient network failure here would otherwise fail the entire production deploy, so - * each attempt gets a clean temp dir and the network steps are retried with backoff. The - * caller owns cleanup of the returned dir. - */ -async function cloneDocs(): Promise { - const repoUrl = 'https://github.com/FlowFuse/flowfuse.git' - let lastError: unknown - - for (let attempt = 1; attempt <= CLONE_ATTEMPTS; attempt++) { - const tmpDir = join(tmpdir(), `flowfuse-docs-${process.pid}-${attempt}`) - if (existsSync(tmpDir)) rmSync(tmpDir, { recursive: true, force: true }) - - try { - execFileSync('git', ['clone', '--filter=blob:none', '--no-checkout', '--depth=1', repoUrl, tmpDir], { stdio: 'pipe' }) - execFileSync('git', ['sparse-checkout', 'set', 'docs'], { cwd: tmpDir, stdio: 'pipe' }) - execFileSync('git', ['checkout'], { cwd: tmpDir, stdio: 'pipe' }) - return tmpDir - } catch (err) { - lastError = err - if (existsSync(tmpDir)) rmSync(tmpDir, { recursive: true, force: true }) - - if (attempt === CLONE_ATTEMPTS) break - - const wait = CLONE_BACKOFF_MS * attempt - logger.warn(`Docs clone attempt ${attempt}/${CLONE_ATTEMPTS} failed, retrying in ${wait}ms`) - await sleep(wait) - } - } - - const reason = lastError instanceof Error ? lastError.message : String(lastError) - throw new Error(`Failed to clone FlowFuse docs from ${repoUrl} after ${CLONE_ATTEMPTS} attempts: ${reason}`) -} - -function copyDocsDir( - srcDir: string, - repoRoot: string, - contentDir: string, - publicDir: string, - version: string, -) { - mkdirSync(contentDir, { recursive: true }) - mkdirSync(publicDir, { recursive: true }) - - for (const entry of readdirSync(srcDir, { withFileTypes: true })) { - if (entry.name.startsWith('.')) continue - - const srcPath = join(srcDir, entry.name) - const destName = entry.name === 'README.md' ? 'index.md' : entry.name - - if (entry.isDirectory()) { - copyDocsDir(srcPath, repoRoot, join(contentDir, entry.name), join(publicDir, entry.name), version) - } else if (entry.name.endsWith('.md')) { - const relFromRepo = relative(repoRoot, srcPath) - const docsRoot = join(repoRoot, 'docs') - const originalPath = relative(docsRoot, srcPath) - - let updated = '' - try { - // Argument array, not a shell string: relFromRepo comes from filenames in the - // cloned repo, so quoting it into a shell command would be an injection path. - updated = execFileSync('git', ['log', '-1', '--pretty=format:%ci', '--', relFromRepo], { - cwd: repoRoot, encoding: 'utf8', - }).trim() - } catch { /* not fatal */ } - - const raw = readFileSync(srcPath, 'utf8') - const processed = processMarkdown(raw, originalPath, updated, version) - writeFileSync(join(contentDir, destName), processed, 'utf8') - } else { - cpSync(srcPath, join(publicDir, entry.name)) - } - } -} - function collectRoutes(dir: string, basePath: string): string[] { const routes: string[] = [] for (const entry of readdirSync(dir, { withFileTypes: true })) { @@ -121,43 +28,8 @@ export default defineNuxtModule({ async setup(_options, nuxt) { const nuxtRoot = nuxt.options.rootDir const contentDocsDir = join(nuxtRoot, 'content', 'docs') - const publicDocsDir = join(nuxtRoot, 'public', 'docs') - const localPath = process.env.FLOWFUSE_DOCS_LOCAL - - if (localPath) { - logger.info(`Using local docs from ${localPath}`) - const docsDir = localPath.endsWith('/docs') ? localPath : join(localPath, 'docs') - if (!existsSync(docsDir)) { - logger.warn(`FLOWFUSE_DOCS_LOCAL path not found: ${docsDir}`) - } else { - let version = '' - try { - const pkg = JSON.parse(readFileSync(join(dirname(docsDir), 'package.json'), 'utf8')) - version = pkg.version || '' - } catch { /* ignore */ } - if (existsSync(contentDocsDir)) rmSync(contentDocsDir, { recursive: true, force: true }) - if (existsSync(publicDocsDir)) rmSync(publicDocsDir, { recursive: true, force: true }) - copyDocsDir(docsDir, dirname(docsDir), contentDocsDir, publicDocsDir, version) - logger.success('Local docs copied') - } - } else if (existsSync(contentDocsDir)) { - logger.info('Using existing content/docs (set FLOWFUSE_DOCS_LOCAL to refresh)') - } else { - logger.info('Cloning FlowFuse docs...') - const tmpDir = await cloneDocs() - try { - const pkg = JSON.parse(readFileSync(join(tmpDir, 'package.json'), 'utf8')) - const version: string = pkg.version || '' - - if (existsSync(contentDocsDir)) rmSync(contentDocsDir, { recursive: true, force: true }) - if (existsSync(publicDocsDir)) rmSync(publicDocsDir, { recursive: true, force: true }) - copyDocsDir(join(tmpDir, 'docs'), tmpDir, contentDocsDir, publicDocsDir, version) - logger.success(`Docs cloned (version ${version})`) - } finally { - if (existsSync(tmpDir)) rmSync(tmpDir, { recursive: true, force: true }) - } - } + await syncDocs({ repoRoot: dirname(nuxtRoot), nuxtRoot, logger }) if (!existsSync(contentDocsDir)) return diff --git a/package.json b/package.json index 8d083fbe5f..a3c656eb40 100644 --- a/package.json +++ b/package.json @@ -23,12 +23,14 @@ "dev:postcss": "dotenv -v TAILWIND_MODE=watch -- npx postcss ./src/css/style.css -o ./_site/css/style.css --config ./postcss.config.js -w", "dev:postcss-nuxt": "dotenv -v TAILWIND_MODE=watch -- npx postcss ./src/css/style.css -o ./nuxt/public/css/style.css --config ./postcss.config.js -w", "blueprints": "node scripts/copy_blueprints.js", + "docs": "node scripts/sync_docs.mjs", "index:algolia": "node scripts/index-algolia.js", "dev:eleventy": "dotenv -- npx @11ty/eleventy --serve --port 8080 --quiet", + "dev:nuxt": "dotenv -- npm run dev --workspace=nuxt", "old_dev:eleventy": "dotenv -v NODE_ENV=development -- ELEVENTY_ENV=development npx @11ty/eleventy --serve --quiet", "prod:eleventy": "npx @11ty/eleventy", "prod:postcss": "postcss ./src/css/style.css -o ./_site/css/style.css --config ./postcss.config.js", - "clean:nuxt": "npx del-cli 'nuxt/public/!(img|handbook|images)' 'nuxt/.output' 'nuxt/.netlify' && npx mkdirp 'nuxt/public/js' 'nuxt/public/css'", + "clean:nuxt": "npx del-cli 'nuxt/public/!(img|handbook|images|docs)' 'nuxt/.output' 'nuxt/.netlify' && npx mkdirp 'nuxt/public/js' 'nuxt/public/css'", "build:js:nuxt": "terser -c -m -o nuxt/public/js/cc.min.js node_modules/vanilla-cookieconsent/dist/cookieconsent.umd.js src/js/cookieconsent-config.js && cp node_modules/@flowfuse/flow-renderer/index.min.js nuxt/public/js/flowrenderer.min.js", "prod:postcss-nuxt": "postcss ./src/css/style.css -o ./nuxt/public/css/style.css --config ./postcss.config.js", "prod:eleventy-nuxt": "npx @11ty/eleventy --output=./nuxt/public/", diff --git a/scripts/sync_docs.mjs b/scripts/sync_docs.mjs new file mode 100644 index 0000000000..5c91457ed8 --- /dev/null +++ b/scripts/sync_docs.mjs @@ -0,0 +1,12 @@ +#!/usr/bin/env node +// Populates nuxt/content/docs outside of a Nuxt build, so CI can commit the result as a +// snapshot. Uses only node builtins: this runs before `npm install`. + +import { dirname, join } from 'node:path' +import { fileURLToPath } from 'node:url' + +import { syncDocs } from '../nuxt/lib/docs-sync.mjs' + +const repoRoot = join(dirname(fileURLToPath(import.meta.url)), '..') + +await syncDocs({ repoRoot, nuxtRoot: join(repoRoot, 'nuxt') }) From fe17fc9c96c65648af514ae8cbca3eacd080399e Mon Sep 17 00:00:00 2001 From: Dimitrie Hoekstra Date: Mon, 3 Aug 2026 11:43:43 +0200 Subject: [PATCH 2/4] Set FLOWFUSE_DOCS_SNAPSHOT in the existing build.environment table A second [build.environment] table is a TOML redefinition, which Netlify could not parse. --- netlify.toml | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/netlify.toml b/netlify.toml index 580aacd7f4..618adc4890 100644 --- a/netlify.toml +++ b/netlify.toml @@ -1,8 +1,3 @@ -[build.environment] - # Deploy the docs snapshot the build workflow committed to 'live' rather than cloning - # flowfuse at deploy time. Branches without a snapshot fall back to cloning. - FLOWFUSE_DOCS_SNAPSHOT = "1" - [[headers]] for = "/*" [headers.values] @@ -75,6 +70,9 @@ publish = "nuxt/dist" [build.environment] NODE_OPTIONS = "--max-old-space-size=4096" +# Deploy the docs snapshot the build workflow commits to 'live' instead of cloning +# flowfuse at deploy time. Branches carrying no snapshot fall back to cloning. +FLOWFUSE_DOCS_SNAPSHOT = "1" [functions] directory = "nuxt/.netlify/functions-internal" From 801f6f286e3901f05d04050d1bbaced25ac9b665 Mon Sep 17 00:00:00 2001 From: ZJ van de Weg Date: Mon, 3 Aug 2026 16:57:13 -0700 Subject: [PATCH 3/4] Drop the live-branch docs snapshot, clone docs at build time instead Address review feedback: stop committing a docs snapshot to `live` and setting FLOWFUSE_DOCS_SNAPSHOT in Netlify. Every build (including Netlify previews) now resolves docs via the existing local/sibling/clone chain in docs-sync.mjs, with the clone kept blobless rather than shallow. --- .claude/CLAUDE.md | 2 +- .github/workflows/build.yml | 17 ----------------- README.md | 7 +++---- netlify.toml | 3 --- nuxt/lib/docs-sync.mjs | 17 +++-------------- nuxt/lib/docs-sync.test.mjs | 25 +------------------------ 6 files changed, 8 insertions(+), 63 deletions(-) diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 1a0812f46d..517e1535a1 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -41,7 +41,7 @@ npm run build # production build > When working on the handbook, docs, or other migrated sections, `npm run dev:nuxt` is sufficient. `npm start` is only needed when also touching 11ty-served pages. > -> **Local docs development:** a checkout of `flowfuse/flowfuse` sitting next to this repo (`../flowfuse`) is picked up automatically, with no configuration. Full resolution order, which every build logs: `FLOWFUSE_DOCS_LOCAL` (explicit path, and a path that does not exist is an error), then a sibling checkout, then the snapshot committed to `live` when `FLOWFUSE_DOCS_SNAPSHOT` is set (Netlify only), then a clone of `FLOWFUSE_DOCS_REF` (default `main`). CI relies on the sibling rule: `FlowFuse/flowfuse`'s `Publish Documentation` workflow checks itself out next to the website so a docs PR is validated against its own changes. +> **Local docs development:** a checkout of `flowfuse/flowfuse` sitting next to this repo (`../flowfuse`) is picked up automatically, with no configuration. Full resolution order, which every build logs: `FLOWFUSE_DOCS_LOCAL` (explicit path, and a path that does not exist is an error), then a sibling checkout, then a clone of `FLOWFUSE_DOCS_REF` (default `main`) — this is what Netlify production deploys use. CI relies on the sibling rule: `FlowFuse/flowfuse`'s `Publish Documentation` workflow checks itself out next to the website so a docs PR is validated against its own changes. ## Directory layout diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 1db4b1c56f..4385752e15 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -16,14 +16,6 @@ jobs: uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: path: 'website' - - name: Check out FlowFuse/flowfuse repository (to access the docs) - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - repository: 'FlowFuse/flowfuse' - ref: main - path: 'flowfuse' - # Full history: each docs page is dated from its own last commit. - fetch-depth: 0 - name: Generate a token id: generate_token uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0 @@ -45,17 +37,8 @@ jobs: node-version: 24 cache: 'npm' cache-dependency-path: './website/package-lock.json' - - run: npm run docs - working-directory: 'website' - run: npm run blueprints working-directory: 'website' - - name: Commit Latest Docs - run: | - cd ./website - git config --local user.email "41898282+github-actions[bot]@users.noreply.github.com" - git config --local user.name "github-actions[bot]" - git add nuxt/content/docs nuxt/public/docs -A -f - git commit -a -m "Bot: update docs" - name: Commit Latest Blueprints run: | cd ./website diff --git a/README.md b/README.md index 4ba7ca00f0..f5833cf998 100644 --- a/README.md +++ b/README.md @@ -6,9 +6,9 @@ This repository contains the source of the FlowFuse website. It is hosted on Netlify with each commit to the `main` branch being automatically deployed to the live site. This works by the [Build Site](.github/workflows/build.yml) action updating the `live` branch, committing onto it the -product documentation pulled from the `main` branch of [FlowFuse/flowfuse](https://github.com/FlowFuse/flowfuse). +blueprints pulled from [FlowFuse/blueprint-library](https://github.com/FlowFuse/blueprint-library). -Netlify is then configured to watch the `live` branch for any changes, once detected, it will automatically pull the contents of this branch (docs included) and deploy to our production site. +Netlify is then configured to watch the `live` branch for any changes, once detected, it will automatically pull the contents of this branch and deploy to our production site. Product documentation is not part of that snapshot — Netlify clones it directly from `main` of [FlowFuse/flowfuse](https://github.com/FlowFuse/flowfuse) during its own build. ## Repository structure @@ -101,8 +101,7 @@ Nothing needs configuring for that to happen. Every build resolves the docs in t |-------|--------|-----------| | 1 | `FLOWFUSE_DOCS_LOCAL=/path/to/flowfuse` | The env var is set. A path that does not exist is an error, not a fallback. | | 2 | A sibling checkout: `../flowfuse`, `../flowforge` or `../dev-env/packages/flowfuse` | One of those has a `docs/` directory. This is what CI relies on. | -| 3 | The snapshot committed to `live` | `FLOWFUSE_DOCS_SNAPSHOT` is set, which Netlify does. Production deploys never clone. | -| 4 | A clone of `FLOWFUSE_DOCS_REF` (default `main`) | Nothing above applied. | +| 3 | A clone of `FLOWFUSE_DOCS_REF` (default `main`) | Nothing above applied. This is what Netlify production deploys use. | `npm run docs` runs that resolution on its own, without a full build, writing `nuxt/content/docs` and `nuxt/public/docs`. Both are generated, and neither is committed on `main`. diff --git a/netlify.toml b/netlify.toml index 618adc4890..39cf145a24 100644 --- a/netlify.toml +++ b/netlify.toml @@ -70,9 +70,6 @@ publish = "nuxt/dist" [build.environment] NODE_OPTIONS = "--max-old-space-size=4096" -# Deploy the docs snapshot the build workflow commits to 'live' instead of cloning -# flowfuse at deploy time. Branches carrying no snapshot fall back to cloning. -FLOWFUSE_DOCS_SNAPSHOT = "1" [functions] directory = "nuxt/.netlify/functions-internal" diff --git a/nuxt/lib/docs-sync.mjs b/nuxt/lib/docs-sync.mjs index 94ace88d83..4274151d17 100644 --- a/nuxt/lib/docs-sync.mjs +++ b/nuxt/lib/docs-sync.mjs @@ -27,10 +27,9 @@ const sleep = (ms) => new Promise(resolve => setTimeout(resolve, ms)) * * 1. `FLOWFUSE_DOCS_LOCAL` - an explicit checkout path * 2. a sibling checkout of flowfuse - * 3. the snapshot committed to the `live` branch (`FLOWFUSE_DOCS_SNAPSHOT` builds only) - * 4. a clone of `FLOWFUSE_DOCS_REF` + * 3. a clone of `FLOWFUSE_DOCS_REF` */ -export function resolveSource ({ repoRoot, contentDocsDir, env = process.env, exists = existsSync }) { +export function resolveSource ({ repoRoot, env = process.env, exists = existsSync }) { const local = env.FLOWFUSE_DOCS_LOCAL if (local) { const docsDir = local.endsWith('/docs') ? local : join(local, 'docs') @@ -48,10 +47,6 @@ export function resolveSource ({ repoRoot, contentDocsDir, env = process.env, ex } } - if (env.FLOWFUSE_DOCS_SNAPSHOT && exists(join(contentDocsDir, MANIFEST_FILE))) { - return { kind: 'snapshot' } - } - return { kind: 'clone', ref: env.FLOWFUSE_DOCS_REF || DEFAULT_REF } } @@ -157,13 +152,7 @@ function writeDocs ({ docsDir, sourceRoot, contentDocsDir, publicDocsDir, kind, export async function syncDocs ({ repoRoot, nuxtRoot, env = process.env, logger = console } = {}) { const contentDocsDir = join(nuxtRoot, 'content', 'docs') const publicDocsDir = join(nuxtRoot, 'public', 'docs') - const source = resolveSource({ repoRoot, contentDocsDir, env }) - - if (source.kind === 'snapshot') { - const manifest = JSON.parse(readFileSync(join(contentDocsDir, MANIFEST_FILE), 'utf8')) - logger.info(`Using committed docs snapshot: ${manifest.ref} ${manifest.sha.slice(0, 8)} synced ${manifest.syncedAt}`) - return manifest - } + const source = resolveSource({ repoRoot, env }) let manifest if (source.kind === 'clone') { diff --git a/nuxt/lib/docs-sync.test.mjs b/nuxt/lib/docs-sync.test.mjs index 19b1ac1873..ad9c267e7d 100644 --- a/nuxt/lib/docs-sync.test.mjs +++ b/nuxt/lib/docs-sync.test.mjs @@ -1,14 +1,12 @@ import { test } from 'node:test' import assert from 'node:assert/strict' -import { MANIFEST_FILE, resolveSource } from './docs-sync.mjs' +import { resolveSource } from './docs-sync.mjs' const repoRoot = '/repo/website' -const contentDocsDir = '/repo/website/nuxt/content/docs' const resolve = (env, present = []) => resolveSource({ repoRoot, - contentDocsDir, env, exists: (path) => present.includes(path), }) @@ -50,27 +48,6 @@ test('the dev-env checkout is preferred over a bare sibling', () => { assert.equal(source.docsDir, '/repo/dev-env/packages/flowfuse/docs') }) -test('a sibling checkout wins over a committed snapshot', () => { - const source = resolve( - { FLOWFUSE_DOCS_SNAPSHOT: '1' }, - ['/repo/flowfuse/docs', `${contentDocsDir}/${MANIFEST_FILE}`], - ) - - assert.equal(source.kind, 'sibling') -}) - -test('the committed snapshot is used when a build asks for it', () => { - const source = resolve({ FLOWFUSE_DOCS_SNAPSHOT: '1' }, [`${contentDocsDir}/${MANIFEST_FILE}`]) - - assert.deepEqual(source, { kind: 'snapshot' }) -}) - -test('a snapshot left on disk is ignored unless the build asks for it', () => { - const source = resolve({}, [`${contentDocsDir}/${MANIFEST_FILE}`]) - - assert.equal(source.kind, 'clone') -}) - test('cloning falls back to main and honours an explicit ref', () => { assert.deepEqual(resolve({}), { kind: 'clone', ref: 'main' }) assert.equal(resolve({ FLOWFUSE_DOCS_REF: 'maintenance' }).ref, 'maintenance') From 4eb0cd82d89c4b278b336faf6499903c0a33d8bf Mon Sep 17 00:00:00 2001 From: ZJ van de Weg Date: Mon, 3 Aug 2026 17:03:23 -0700 Subject: [PATCH 4/4] Remove redundant TMPDIR workaround note from README nuxt/package.json's dev script already sets TMPDIR=/tmp, restored by the merge from main, so the manual workaround for nuxt#35253 no longer applies. --- README.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/README.md b/README.md index f5833cf998..48d9b63fa3 100644 --- a/README.md +++ b/README.md @@ -105,12 +105,6 @@ Nothing needs configuring for that to happen. Every build resolves the docs in t `npm run docs` runs that resolution on its own, without a full build, writing `nuxt/content/docs` and `nuxt/public/docs`. Both are generated, and neither is committed on `main`. -If the docs and handbook pages fail to render locally while the rest of the site is fine, you are hitting [nuxt#35253](https://github.com/nuxt/nuxt/issues/35253). Give the build its own temp directory: - -```bash -export TMPDIR=/tmp/nuxt -``` - ## How to add blog posts See the [Blog section of the Marketing Handbook](https://flowfuse.com/handbook/marketing/content-strategy/blog/) for instructions on writing and publishing blog posts.