-
Notifications
You must be signed in to change notification settings - Fork 19
nuxt: Sitemap enrichment #5546
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+222
−15
Merged
nuxt: Sitemap enrichment #5546
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| // Resolves a file's sitemap `lastmod` from git history rather than build time, so a | ||
| // static-generated site doesn't stamp every URL with the deploy timestamp. Builds the | ||
| // whole repo-root -> commit-date map in one `git log` walk (memoized per repoRoot) | ||
| // instead of spawning `git log` per file, which would be one process per content page - | ||
| // the same N+1 GitLab hit in Gitaly's ListLastCommitsForTree RPC (used to render a | ||
| // "last commit" column per file in the web file browser), which originally ran | ||
| // `git log -1` once per tree entry. GitLab's fix - upstreamed into git.git as the | ||
| // `git last-modified` builtin (Git 2.52, née GitHub's internal "blame-tree", used for | ||
| // GitHub's own file-browser column since 2012) - walks commit history once and, at each | ||
| // commit, diffs trees to see which of the *still-unresolved* requested paths changed, | ||
| // pruning whole unchanged subtrees via tree-object-id ("treesame") comparisons instead of | ||
| // diffing file-by-file, and stops early once every path has been resolved. That is a | ||
| // more efficient shared walk (matches Gitaly/GitHub's write-up: revisiting the same | ||
| // commits per file is "twice the necessary" work); this module's `--name-only` walk | ||
| // below shares the same core insight - one walk, not one process per file, and the first | ||
| // occurrence of a path scanning newest-first is its most recent commit - but reads the | ||
| // whole history as flat text and dedupes in JS rather than pruning the walk itself, which | ||
| // is fine at this repo's history size (a few seconds, once, memoized) but wouldn't scale | ||
| // to a git-last-modified-sized monorepo. | ||
| import { execFileSync } from 'node:child_process' | ||
|
|
||
| // A NUL byte can't appear in a file path, so it safely marks a commit-date line apart | ||
| // from the `--name-only` file lines that follow it. `%x00` asks git to emit the byte | ||
| // into its output; the argv string itself only ever contains the ASCII text "%x00". | ||
| const NUL = '\u0000' | ||
| const mapCache = new Map() | ||
|
|
||
| /** | ||
| * Pure parser for `git log --pretty=format:%x00%ci --name-only` output - split out from | ||
| * buildLastmodMap so the newest-first/first-occurrence-wins logic can be unit tested | ||
| * against fixture strings without shelling out to git or touching a real repo. | ||
| * | ||
| * @param {string} output raw stdout from the git log invocation above | ||
| * @returns {Map<string, string>} file path -> most recent commit date | ||
| */ | ||
| export function parseGitLogOutput (output) { | ||
| const map = new Map() | ||
| let currentDate = null | ||
| for (const line of output.split('\n')) { | ||
| if (line.startsWith(NUL)) { | ||
| currentDate = line.slice(1) | ||
| continue | ||
| } | ||
| if (!line || map.has(line)) continue | ||
| map.set(line, currentDate) | ||
| } | ||
| return map | ||
| } | ||
|
|
||
| function buildLastmodMap (repoRoot) { | ||
|
ZJvandeWeg marked this conversation as resolved.
|
||
| let output | ||
| try { | ||
| // Newest-first (git log's default order): the first time a path is seen while | ||
| // walking top-to-bottom is its most recent commit. | ||
| output = execFileSync( | ||
| 'git', | ||
| ['log', '--pretty=format:%x00%ci', '--name-only'], | ||
| { cwd: repoRoot, encoding: 'utf8', maxBuffer: 1024 * 1024 * 256 } | ||
| ) | ||
| } catch (err) { | ||
| console.warn(`[sitemap] git log failed in ${repoRoot}, lastmod will be omitted: ${err.message}`) | ||
| return new Map() | ||
| } | ||
|
|
||
| return parseGitLogOutput(output) | ||
| } | ||
|
|
||
| /** | ||
| * @param {string} repoRoot absolute path to the git repository root | ||
| * @param {string} relativePath path to the file, relative to repoRoot (e.g. "src/blog/2024/01/post.md") | ||
| * @returns {string|undefined} the ISO-ish commit date git log reports, or undefined if unknown | ||
| */ | ||
| export function getGitLastmod (repoRoot, relativePath) { | ||
| if (!mapCache.has(repoRoot)) { | ||
| mapCache.set(repoRoot, buildLastmodMap(repoRoot)) | ||
| } | ||
| return mapCache.get(repoRoot).get(relativePath) | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,120 @@ | ||
| import { existsSync } from 'node:fs' | ||
| import { dirname, join } from 'node:path' | ||
| import { queryCollection } from '@nuxt/content/server' | ||
| import { getGitLastmod } from '../../../lib/git-lastmod.mjs' | ||
|
|
||
| // docs/handbook/changelog/blog/ebooks/whitepapers deliberately carry no `sitemap` schema | ||
| // field in content.config.ts - @nuxtjs/sitemap's @nuxt/content integration re-splices | ||
| // each collection's onUrl/filter as raw source text into a generated file with no closure | ||
| // over this module's imports or helpers, which git-based lastmod needs. This route does | ||
| // the same job (loc/lastmod/images) as a normal Nitro handler instead. | ||
|
|
||
| const toAbsoluteUrl = (event: Parameters<typeof getSiteConfig>[0], path: string) => | ||
| (path.startsWith('http') ? path : `${getSiteConfig(event).url}${path}`) | ||
|
|
||
| // Runs at prerender time (during `nuxt generate`), inside the git checkout, so walking up | ||
| // from cwd to find `.git` is robust whether the build is invoked from the repo root or the | ||
| // `nuxt/` workspace - unlike at request time in the deployed function, where no .git exists. | ||
| function findRepoRoot (start: string): string { | ||
| let dir = start | ||
| for (let i = 0; i < 6; i++) { | ||
| if (existsSync(join(dir, '.git'))) return dir | ||
| const parent = dirname(dir) | ||
| if (parent === dir) break | ||
| dir = parent | ||
| } | ||
| return start | ||
| } | ||
|
|
||
| const REPO_ROOT = findRepoRoot(process.cwd()) | ||
|
|
||
| type ContentEntry = Record<string, unknown> & { path?: string, stem: string } | ||
| type SitemapUrl = { loc: string, lastmod?: Date, images?: { loc: string }[] } | ||
|
|
||
| interface ContentSource { | ||
| collection: string | ||
| // Repo-relative directory the collection's files live in, for git-log lastmod. | ||
| // Omit when a collection derives lastmod another way (e.g. docs' `updated` field). | ||
| fileRoot?: string | ||
| filter?: (entry: ContentEntry) => boolean | ||
| lastmod?: (entry: ContentEntry) => string | undefined | ||
| images?: (entry: ContentEntry) => string[] | ||
| rewriteLoc?: (loc: string) => string | ||
| } | ||
|
|
||
| const stringField = (entry: ContentEntry, key: string) => { | ||
| const value = entry[key] | ||
| return typeof value === 'string' && value ? value : undefined | ||
| } | ||
|
|
||
| const CONTENT_SOURCES: ContentSource[] = [ | ||
| { | ||
| collection: 'docs', | ||
| // Already git-derived once, at docs-sync time, against the flowfuse/flowfuse repo | ||
| // this content came from - not this repo's history. | ||
| lastmod: entry => stringField(entry, 'updated'), | ||
| // Redirect stub pages (section index pages that just 301 elsewhere) aren't a | ||
| // real destination. | ||
| filter: entry => entry.layout !== 'redirect', | ||
| }, | ||
| { collection: 'handbook', fileRoot: 'nuxt/content' }, | ||
| { collection: 'changelog', fileRoot: 'src' }, | ||
| { | ||
| collection: 'blog', | ||
| fileRoot: 'src', | ||
| // Prefer the author-set `lastUpdated` frontmatter field - the same one the blog | ||
| // page itself uses for its "Updated" date and JSON-LD dateModified - over the git | ||
| // history date, so an editorial update date isn't silently overridden by whatever | ||
| // last touched the file (a repo-wide lint fix, a typo pass elsewhere, etc). | ||
| lastmod: entry => stringField(entry, 'lastUpdated') ?? getGitLastmod(REPO_ROOT, `src/${entry.stem}.md`), | ||
| images: entry => [stringField(entry, 'image')].filter((path): path is string => Boolean(path)), | ||
| }, | ||
| { | ||
| collection: 'ebooks', | ||
| fileRoot: 'nuxt/content', | ||
| images: entry => ['image', 'coverImage', 'thumbnail', 'secondaryImage', 'tertiaryImage'] | ||
| .map(key => stringField(entry, key)) | ||
| .filter((path): path is string => Boolean(path)), | ||
| }, | ||
| { | ||
| collection: 'whitepapers', | ||
| fileRoot: 'nuxt/content', | ||
| images: entry => ['image', 'thumbnail'] | ||
| .map(key => stringField(entry, key)) | ||
| .filter((path): path is string => Boolean(path)), | ||
| rewriteLoc: loc => loc.replace(/^\/whitepapers\//, '/whitepaper/'), | ||
| }, | ||
| ] | ||
|
|
||
| export default defineSitemapEventHandler(async (event) => { | ||
| const urls: SitemapUrl[] = [] | ||
|
|
||
| for (const source of CONTENT_SOURCES) { | ||
| try { | ||
| const entries = await queryCollection(event, source.collection as never).all() as ContentEntry[] | ||
| for (const entry of entries) { | ||
| // `.navigation` entries are @nuxt/content's per-directory nav metadata, | ||
| // not real pages - @nuxtjs/sitemap's own content integration excludes | ||
| // them the same way. | ||
| if (!entry.path || entry.path.endsWith('.navigation')) continue | ||
| if (source.filter && !source.filter(entry)) continue | ||
|
|
||
| const url: SitemapUrl = { loc: source.rewriteLoc ? source.rewriteLoc(entry.path) : entry.path } | ||
|
|
||
| const lastmod = source.lastmod | ||
| ? source.lastmod(entry) | ||
| : (source.fileRoot ? getGitLastmod(REPO_ROOT, `${source.fileRoot}/${entry.stem}.md`) : undefined) | ||
| if (lastmod) url.lastmod = new Date(lastmod) | ||
|
|
||
| const images = source.images?.(entry) ?? [] | ||
| if (images.length) url.images = images.map(loc => ({ loc: toAbsoluteUrl(event, loc) })) | ||
|
|
||
| urls.push(url) | ||
| } | ||
| } catch (err) { | ||
| console.error(`[sitemap] failed to query "${source.collection}" collection for content-urls:`, err) | ||
| } | ||
| } | ||
|
|
||
| return urls | ||
| }) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.