Skip to content
This repository was archived by the owner on Sep 9, 2026. It is now read-only.
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 21 additions & 1 deletion packages/compiler/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,24 @@ for (const file of files) fs.copyFileSync(file, path.join(publicDir, path.basena

每次浏览器构建都会拿 esbuild 的 metafile 对着这份清单自检:产物改了名、被挪进子目录、被拆出新 chunk、或某个资源开始 import 别的文件(`import` / `require` / 动态 import 都算,一带上就不再是自包含的单文件),以及清单里的文件名和 `package.json` 的 exports 对不上(改名只改了一边),构建当场失败,而不是几个月后在某个宿主那里 404。`toolchain.browser.js` 不在清单里——它由宿主用自己的打包器 import(`@dimina-kit/compiler/toolchain`),拷过去也没人 fetch。

### 自定义文件类型(方言):声明一次,编译器和宿主用同一份

`options.fileTypes` 让工程用自己的扩展名,比如千岛(qd)方言的 `.qdml`/`.qdss`/`.qds` 对应 `.wxml`/`.wxss`/`.wxs`。麻烦的是这份配置不止编译器要用:编辑器的语言映射、模板校验、预览时找页面模板,宿主自己也要按同一套规则判断文件角色。各处手写 `/\.(wxml|qdml)$/` 的结果是漏掉内置的 `.ddml`,而且同一个方言在几个仓库里各抄一份,谁改了另一边不会知道。

`@dimina-kit/compiler/file-types` 就是那一份(同时发 ESM 和 CJS,无依赖):

```js
import { QD_FILE_TYPES, resolveFileTypes, hasExt } from '@dimina-kit/compiler/file-types'

await pool.compile({ files, workPath, options: { fileTypes: QD_FILE_TYPES } })

const { templateExts, styleExts, viewScriptExts, viewScriptTags } = resolveFileTypes(QD_FILE_TYPES)
// templateExts: ['.wxml', '.ddml', '.qdml'] ← 内置在前,自定义在后,顺序即查找优先级
hasExt('pages/index/index.QDML', templateExts) // true,大小写不敏感
```

`resolveFileTypes()` 算的是**编译器这次实际会认的**扩展名和内联标签:合并内置项、规范化(去空白、转小写、补一个前导点)、去重,并丢掉占用其他角色或 `.js`/`.ts`/`.json` 的项——`template: ['js']` 会把页面逻辑当模板解析,所以直接不接受。规则是照编译器 `env.js` 的 `normalizeFileTypes` 写的(直接 import 它会把 `node:fs` 和整个配置解析一起拉进来),`test:file-types` 读 env.js 源码比对内置列表和两条校验正则,上游一改这里就红。

## 架构

本包是**编译器与文件系统之间的一层适配**,再往上叠一层**编排**。真正的编译逻辑在 `dimina` 子模块的 `@dimina/compiler`,本包用一个**无后端的 fs 转发 shim** 把它每一次 `fs.xxx` 指向下游注入的 fs;`pool` 则在上面替下游管好 worker 池与并行——下游不再手写任何 worker/合并逻辑。
Expand Down Expand Up @@ -437,7 +455,7 @@ pnpm --filter @dimina-kit/compiler build:types # 仅 dist/types/*.d.ts

## 测试

`pnpm --filter @dimina-kit/compiler test`(也就是 `turbo run test` 会跑到的那份)包含四份契约测试——静态资源清单(`test:browser-assets`)、错误码(`test:error-codes`)、二进制入参播种(`test:binary-seed`)和二进制产物保真(`test:binary-outputs`)。最后一份要拿 `dist` 里的真实 bundle 跑,但它自己不构建:turbo 里 `@dimina-kit/compiler#test` 依赖本包的 `build`,构建只发生一次,测试期间没有人再往 `dist` 写。脱离 turbo 单跑时用带构建的 `test:binary-outputs``test:binary-outputs:prebuilt` 是不构建的那个入口)。下面这些各自要先构建,按需单跑。
`pnpm --filter @dimina-kit/compiler test`(也就是 `turbo run test` 会跑到的那份)包含六份契约测试——静态资源清单(`test:browser-assets`)、错误码(`test:error-codes`)、二进制入参播种(`test:binary-seed`)、二进制产物保真(`test:binary-outputs`)、方言扩展名(`test:file-types`)和方言穿过编译池(`test:pool-filetypes`)。后两类里要拿 `dist` 真实 bundle 跑的那几份自己不构建:turbo 里 `@dimina-kit/compiler#test` 依赖本包的 `build`,构建只发生一次,测试期间没有人再往 `dist` 写。脱离 turbo 单跑时用带构建的 `test:binary-outputs` / `test:pool-filetypes`(带 `:prebuilt` 后缀的是不构建的那个入口)。下面这些各自要先构建,按需单跑。

测试里用 memfs 扮演「下游 fs」:

Expand All @@ -455,6 +473,7 @@ pnpm --filter @dimina-kit/compiler test:stage-load-retry # stage 工
pnpm --filter @dimina-kit/compiler test:browser-assets # 静态资源清单:改名/新 chunk/出现静态 import 都会被构建期检查拦下
pnpm --filter @dimina-kit/compiler test:error-codes # 错误码:worker 自己判定的失败(工具链导入)带着码原样传到调用方,其余记为 compiler-stage-error
pnpm --filter @dimina-kit/compiler test:stage-toolchain # 真实 stage worker bundle:每个 stage 都加载工具链、按 URL 记忆、导入失败带错误码
pnpm --filter @dimina-kit/compiler test:file-types # 方言扩展名:合并/规范化行为,以及内置列表与校验正则和编译器 env.js 逐字一致
pnpm --filter @dimina-kit/compiler test:binary-seed # 入参里的 Uint8Array 播种成文件(memfs 自己的 fromJSON 会把它变成目录)
pnpm --filter @dimina-kit/compiler test:binary-outputs # 页面引用的图片走完整编译后逐字节相同,文本产物仍是字符串
```
Expand All @@ -471,6 +490,7 @@ pnpm --filter @dimina-kit/compiler test:binary-outputs # 页面引
- `src/pool-node.js` — **Node 编排池** `createNodeCompilerPool` + dmcc drop-in 默认导出 `build()`:常驻 worker_threads、真实磁盘、全局 build 串行、死 worker 懒复活、idle 自动收缩(`idleShrinkMs`)。
- `src/stage-worker-node.js` — Node 常驻 stage worker:spawn 时按 workerData 里的 stage 身份预热本 stage 工具链,恢复 storeInfo → `runStage(stage, { sourcemap })` 写共享 staging 目录;应答 `{ type: 'introspect' }` 报告本 realm 已加载的重依赖。
- `src/toolchain.js` — 写 `toolchainSetupURL` 模块的可选助手(`installOxc` / `installEsbuildFromURL`,后者内置 esbuild-wasm 静态资源的 Blob-URL 兜底)。导出为 `@dimina-kit/compiler/toolchain`。
- `src/file-types.js` — 自定义文件类型(方言)的权威声明:内置扩展名、合并规则 `resolveFileTypes`、qd 方言常量 `QD_FILE_TYPES`。导出为 `@dimina-kit/compiler/file-types`(ESM + CJS)。
- `src/browser-assets.js` — 浏览器静态资源清单与契约(`COMPILER_BROWSER_ASSETS` / `resolveBrowserAssets`,见上文),构建期检查也用它。导出为 `@dimina-kit/compiler/browser-assets`。
- `src/error-codes.js` — 两个 pool 共用的错误码表 `COMPILER_ERROR_CODES` 与判定 `isInfrastructureError`(见上文),经 `./pool` 与 `./pool-node` 再导出。
- `src/failure-hints.js` — Node 侧「一条原始报错文字该记哪个码」的判定(`errorCodeForMessage` / `tagFailure`),以及 oxc 绑定缺失、esbuild 二进制被封在 app.asar 这两种打包问题的中文提示(`oxcNativeBindingHint` / `esbuildAsarSpawnHint`,经 `./pool-node` 再导出)。单独成文件是为了让它不牵连 `worker_threads` 和编译器实体,`test:error-codes` 能直接驱动。
Expand Down
11 changes: 9 additions & 2 deletions packages/compiler/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,11 @@
"require": "./dist/browser-assets.cjs",
"default": "./dist/browser-assets.js"
},
"./file-types": {
"types": "./dist/types/file-types.d.ts",
"require": "./dist/file-types.cjs",
"default": "./dist/file-types.js"
},
"./package.json": "./package.json"
},
"files": [
Expand All @@ -69,7 +74,8 @@
"test:realm-reuse": "node scripts/build-compiler.js node && node --import ./scripts/register-kit.js scripts/test-realm-reuse.js",
"test:pool-node": "node scripts/build-compiler.js node && node --import ./scripts/register-kit.js scripts/test-pool-node.js",
"test:pool-scopehash": "node scripts/build-compiler.js node && node --import ./scripts/register-kit.js scripts/test-pool-scopehash.js",
"test": "pnpm run test:browser-assets && pnpm run test:error-codes && pnpm run test:binary-seed && pnpm run test:binary-outputs:prebuilt",
"test": "pnpm run test:browser-assets && pnpm run test:error-codes && pnpm run test:binary-seed && pnpm run test:binary-outputs:prebuilt && pnpm run test:file-types && pnpm run test:pool-filetypes:prebuilt",
"test:file-types": "node scripts/test-file-types.js",
"test:binary-seed": "node scripts/test-binary-seed.js",
"test:binary-outputs": "node scripts/build-compiler.js node && node scripts/build-compiler.js browser && pnpm run test:binary-outputs:prebuilt",
"test:binary-outputs:prebuilt": "node --import ./scripts/register-kit.js scripts/test-binary-outputs.js",
Expand All @@ -88,7 +94,8 @@
"test:stage-toolchain": "node scripts/build-compiler.js browser && node scripts/test-stage-toolchain.js",
"test:stage-worker-message-order": "node scripts/build-compiler.js node && node scripts/test-stage-worker-message-order.js",
"test:stage-load-retry": "node scripts/build-compiler.js node && node scripts/test-stage-load-retry.js",
"test:pool-filetypes": "node scripts/build-compiler.js node && node scripts/build-compiler.js browser && node scripts/test-pool-filetypes.js"
"test:pool-filetypes": "node scripts/build-compiler.js node && node scripts/build-compiler.js browser && pnpm run test:pool-filetypes:prebuilt",
"test:pool-filetypes:prebuilt": "node scripts/test-pool-filetypes.js"
},
"dependencies": {
"@babel/parser": "^7.29.7",
Expand Down
23 changes: 13 additions & 10 deletions packages/compiler/scripts/build-compiler.js
Original file line number Diff line number Diff line change
Expand Up @@ -246,17 +246,20 @@ for (const b of builds) {
// The static-asset manifest itself: dependency-free string code, emitted in both
// modes (either build alone leaves a usable dist) and in both formats, because the
// hosts that copy these files are as often CommonJS build scripts as ESM ones.
for (const [format, outfile] of [['esm', 'browser-assets.js'], ['cjs', 'browser-assets.cjs']]) {
await esbuild.build({
entryPoints: [path.join(root, 'src/browser-assets.js')],
outfile: path.join(root, 'dist', outfile),
bundle: true,
format,
target: ['es2022'],
logLevel: 'warning',
})
// src/file-types.js(方言扩展名)走同一条路:宿主的编辑器配置和文件分类也常常在 CJS 里。
for (const name of ['browser-assets', 'file-types']) {
for (const [format, ext] of [['esm', 'js'], ['cjs', 'cjs']]) {
await esbuild.build({
entryPoints: [path.join(root, `src/${name}.js`)],
outfile: path.join(root, 'dist', `${name}.${ext}`),
bundle: true,
format,
target: ['es2022'],
logLevel: 'warning',
})
}
console.log(`✅ built dist/${name}.js + dist/${name}.cjs`)
}
console.log('✅ built dist/browser-assets.js + dist/browser-assets.cjs')

// The browser bundles double as static files a host copies and serves. Their names
// and the "self-contained, imports nothing" rule are stated once in
Expand Down
93 changes: 93 additions & 0 deletions packages/compiler/scripts/test-file-types.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
// src/file-types.js 把编译器的合并规则抄了一份出来(env.js 直接 import 会拉进 node:fs 和
// 整个配置解析,宿主只想知道扩展名)。抄来的东西会漂:所以这里直接读 env.js 源码,比对内置
// 列表、保留扩展名和两条规范化正则——上游加一种内置方言或改一条正则,这个测试就红,而不是
// 等某个宿主把文件分错类。后半段是合并行为本身。
import { readFileSync } from 'node:fs'
import { fileURLToPath } from 'node:url'
import {
BUILTIN_STYLE_EXTS,
BUILTIN_TEMPLATE_DIRECTIVE_PREFIXES,
BUILTIN_TEMPLATE_EXTS,
BUILTIN_VIEW_SCRIPT_EXTS,
BUILTIN_VIEW_SCRIPT_TAGS,
QD_FILE_TYPES,
RESERVED_EXTS,
hasExt,
resolveFileTypes,
} from '../src/file-types.js'

let failed = 0
const chk = (cond, msg) => { if (cond) { console.log(`✅ ${msg}`) } else { console.log(`❌ ${msg}`); failed++ } }
const same = (a, b) => JSON.stringify(a) === JSON.stringify(b)

const ENV_PATH = fileURLToPath(new URL('../../../dimina/fe/packages/compiler/src/env.js', import.meta.url))
const env = readFileSync(ENV_PATH, 'utf8')

const listOf = (name) => {
const m = env.match(new RegExp(`const ${name} = \\[([^\\]]*)\\]`))
if (!m) throw new Error(`${name} 不在 ${ENV_PATH} 里了——编译器改了内置文件类型的写法,先看它现在怎么写`)
return [...m[1].matchAll(/'([^']*)'/g)].map((x) => x[1])
Comment on lines +27 to +29

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Verify normalization parity through compiler behavior

When upstream changes merge ordering, cross-role filtering, directive-prefix derivation, or any other part of normalizeFileTypes without changing these constant literals and two regexes, this test remains green while the exported resolver can silently diverge from the compiler. Conversely, syntax-only refactors break these source parsers despite unchanged behavior. The real-pool test covers only the QD happy path, so replace or supplement this source-text comparison with an edge-case matrix exercised through the actual compiler boundary.

AGENTS.md reference: AGENTS.md:L21-L21

Useful? React with 👍 / 👎.

}
const regexOf = (fnName) => {
const start = env.indexOf(`function ${fnName}(`)
if (start < 0) throw new Error(`${fnName} 不在 ${ENV_PATH} 里了——先看编译器现在怎么校验扩展名`)
const m = env.slice(start).match(/!(\/\S+?\/)\.test\(v\)/)
if (!m) throw new Error(`${fnName} 里的规范化正则找不到了——先看编译器现在怎么校验扩展名`)
return m[1]
}

chk(same([...BUILTIN_TEMPLATE_EXTS], listOf('DEFAULT_TEMPLATE_EXTS')),
`内置模板扩展名与编译器一致(${BUILTIN_TEMPLATE_EXTS.join(' ')})`)
chk(same([...BUILTIN_TEMPLATE_DIRECTIVE_PREFIXES], listOf('DEFAULT_TEMPLATE_DIRECTIVE_PREFIXES')),
`内置模板指令前缀与编译器一致(${BUILTIN_TEMPLATE_DIRECTIVE_PREFIXES.join(' ')})`)
chk(same([...BUILTIN_STYLE_EXTS], listOf('DEFAULT_STYLE_EXTS')),
`内置样式扩展名与编译器一致(${BUILTIN_STYLE_EXTS.join(' ')})`)
chk(same([...BUILTIN_VIEW_SCRIPT_EXTS], listOf('DEFAULT_VIEW_SCRIPT_EXTS')),
`内置视图脚本扩展名与编译器一致(${BUILTIN_VIEW_SCRIPT_EXTS.join(' ')})`)
chk(same([...BUILTIN_VIEW_SCRIPT_TAGS], listOf('DEFAULT_VIEW_SCRIPT_TAGS')),
`内置视图脚本标签与编译器一致(${BUILTIN_VIEW_SCRIPT_TAGS.join(' ')})`)

{
const block = env.slice(env.indexOf('const RESERVED_EXTS = new Set(['))
const extra = [...block.slice(0, block.indexOf('])')).matchAll(/'([^']*)'/g)].map((x) => x[1])
chk(same(RESERVED_EXTS.filter((e) => !BUILTIN_TEMPLATE_EXTS.includes(e) && !BUILTIN_STYLE_EXTS.includes(e) && !BUILTIN_VIEW_SCRIPT_EXTS.includes(e)), extra),
`保留扩展名里非内置的那几个与编译器一致(${extra.join(' ')})`)
}

chk(regexOf('normalizeExt') === '/^[a-z0-9_-]+$/', `扩展名校验正则与编译器一致(${regexOf('normalizeExt')})`)
chk(regexOf('normalizeTag') === '/^[a-z][a-z0-9_-]*$/', `标签名校验正则与编译器一致(${regexOf('normalizeTag')})`)

// 合并行为
{
const r = resolveFileTypes(QD_FILE_TYPES)
chk(same(r.templateExts, ['.wxml', '.ddml', '.qdml']), `qd 方言的模板扩展名(${r.templateExts.join(' ')})`)
chk(same(r.styleExts, ['.wxss', '.ddss', '.less', '.scss', '.sass', '.qdss']), `qd 方言的样式扩展名(${r.styleExts.join(' ')})`)
chk(same(r.viewScriptExts, ['.wxs', '.qds']), `qd 方言的视图脚本扩展名(${r.viewScriptExts.join(' ')})`)
chk(same(r.viewScriptTags, ['wxs', 'dds', 'qds']), `视图脚本扩展名同时派生内联标签(${r.viewScriptTags.join(' ')})`)
chk(r.templateDirectivePrefixes.includes('qd'), `自定义模板扩展名派生出指令前缀(${r.templateDirectivePrefixes.join(' ')})`)
}

{
const r = resolveFileTypes()
chk(same(r.templateExts, [...BUILTIN_TEMPLATE_EXTS]), '不传 fileTypes 就只有内置项')
r.templateExts.push('.mine')
chk(!BUILTIN_TEMPLATE_EXTS.includes('.mine'), '返回的是副本,改它不会污染内置列表')
}

chk(same(resolveFileTypes({ template: ['js', 'ts', 'json', 'wxss'] }).templateExts, ['.wxml', '.ddml']),
'占用逻辑/配置/其他角色扩展名的自定义项被丢弃')
chk(same(resolveFileTypes({ template: ['.QDML', 'qdml', ' qdml '] }).templateExts, ['.wxml', '.ddml', '.qdml']),
'带点、大写、带空白的写法规范化成同一项,且只留一份')
chk(same(resolveFileTypes({ template: ['a/b', 'q*d', '', ' '] }).templateExts, ['.wxml', '.ddml']),
'带路径分隔符或元字符的项被丢弃')
chk(same(resolveFileTypes({ viewScript: ['9qd'] }).viewScriptTags, ['wxs', 'dds']),
'数字开头不能当标签名(扩展名可以,标签名不行)')
chk(same(resolveFileTypes({ viewScript: ['9qd'] }).viewScriptExts, ['.wxs', '.9qd']),
'同一项作为扩展名仍然有效')

chk(hasExt('pages/index/index.QDML', resolveFileTypes(QD_FILE_TYPES).templateExts), 'hasExt 大小写不敏感')
chk(!hasExt('pages/index/index.json', resolveFileTypes(QD_FILE_TYPES).templateExts), 'hasExt 不误判 .json')
chk(!hasExt(undefined, BUILTIN_TEMPLATE_EXTS), 'hasExt 对非字符串返回 false')

console.log(failed ? `\n❌ ${failed} 条文件类型断言失败。` : '\n✅ 方言常量与编译器的合并规则一致。')
process.exit(failed ? 1 : 0)
8 changes: 5 additions & 3 deletions packages/compiler/scripts/test-pool-filetypes.js
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,7 @@
// options-threading fix and passes (green) with it — no reimplementation of the
// protocol handlers under test.
//
// Fixture mirrors qdmp's e2e "qd-app" (see
// ~/code/qdmp/main/packages/qdmp-devtools/e2e/qd-app): a page using .qdml/.qdss/.qds
// Fixture is a minimal qd-dialect app: a page using .qdml/.qdss/.qds
// instead of .wxml/.wxss/.wxs, with a <wxs src="./index.qds" module="m" /> view
// script and a `{{ m.shout(title) }}` mustache expression — so a correct compile
// must recognize the custom template AND the custom view-script extension.
Expand All @@ -26,6 +25,7 @@
// must run first.
import { transform } from 'esbuild'
import 'oxc-parser'
import { QD_FILE_TYPES } from '../src/file-types.js'

await transform('const __warm = 1', {})

Expand Down Expand Up @@ -53,7 +53,9 @@ const FIXTURE_FILES = {
'pages/index/index.json': JSON.stringify({ navigationBarTitleText: 'QD Ext Index' }),
'pages/index/index.qds': "function shout(text) {\n return text + '!'\n}\n\nmodule.exports = {\n shout: shout,\n}\n",
}
const FILE_TYPES_OPTIONS = { fileTypes: { template: ['qdml'], style: ['qdss'], viewScript: ['qds'] } }
// 用导出的常量本身,而不是再抄一份字面量:这样这个测试同时证明 QD_FILE_TYPES 就是能让
// .qdml/.qdss/.qds 工程编出来的那份配置。
const FILE_TYPES_OPTIONS = { fileTypes: QD_FILE_TYPES }

let failed = false
const chk = (cond, msg) => { if (!cond) { failed = true; console.error(`❌ ${msg}`) } else console.log(`✅ ${msg}`) }
Expand Down
2 changes: 1 addition & 1 deletion packages/compiler/src/compile-core.js
Original file line number Diff line number Diff line change
Expand Up @@ -516,7 +516,7 @@ let compileChain = Promise.resolve()
* Compile a mini-program against a caller-injected fs. Calls are serialized per
* realm (see the singleton note above). Convenience wrapper that runs
* `setupCompile` + all stages + `collectOutputs` in one realm.
* @param {{ fs: object, workPath?: string, options?: { fileTypes?: { template?: string[], style?: string[], viewScript?: string[] } } }} opts
* @param {{ fs: object, workPath?: string, options?: { fileTypes?: { template?: readonly string[], style?: readonly string[], viewScript?: readonly string[] } } }} opts
* fs: a node:fs replacement (sync subset: existsSync/readFileSync/
* readdirSync{withFileTypes}/statSync/writeFileSync/mkdirSync{recursive}/
* copyFileSync/rmSync), already seeded with the project source under
Expand Down
Loading
Loading