v0.4.0 Wave 1: breaking changes, live-bug fixes, and release-safety gates

.devflow/features/bundler-plugins/KNOWLEDGE.md

@@ -0,0 +1,187 @@
+---
+feature: bundler-plugins
+name: Bundler Plugins (bundler-utils + Vite/Rollup/Webpack/Rspack)
+description: "Use when adding a new bundler integration, modifying the emitted-module contract, debugging HMR behavior, working on the CJS compatibility shim, updating the transformer/loader factory, registering a new package in the release pipeline, or investigating why a .mds file emits unexpected output. Keywords: createMdsTransformer, createMdsLoader, bundler-utils, vite-plugin, rollup-plugin, webpack-loader, rspack-loader, addWatchFile, addDependency, handleHotUpdate, emitted module contract, export default string, export default Message[], safeJsonForJs, escapeForJs, metadata, kind, markdown, messages, discriminated union, mds.d.ts, MdsMessage, string | MdsMessage[]."
+category: component-patterns
+directories: ["packages/bundler-utils/", "packages/vite-plugin/", "packages/rollup-plugin/", "packages/webpack-loader/", "packages/rspack-loader/"]
+referencedFiles:
+  - packages/bundler-utils/src/transform.ts
+  - packages/bundler-utils/src/types.ts
+  - packages/bundler-utils/src/loader.ts
+  - packages/bundler-utils/src/frontmatter.ts
+  - packages/bundler-utils/src/lazy-init.ts
+  - packages/bundler-utils/mds.d.ts
+  - packages/bundler-utils/src/index.ts
+  - packages/vite-plugin/src/index.ts
+  - packages/rollup-plugin/src/index.ts
+  - packages/webpack-loader/src/index.ts
+  - packages/rspack-loader/src/index.ts
+created: 2026-06-26
+updated: 2026-06-26
+---
+
+# Bundler Plugins (bundler-utils + Vite/Rollup/Webpack/Rspack)
+
+## Overview
+
+`packages/bundler-utils/` is the shared transformation layer consumed by four bundler plugins: `vite-plugin`, `rollup-plugin`, `webpack-loader`, `rspack-loader`. It implements `createMdsTransformer` (used by Vite/Rollup) and `createMdsLoader` (used by Webpack/Rspack). After the intrinsic-output refactor, the emitted JS module branches on the compiled `kind` — a markdown `.mds` emits a string default export, a messages `.mds` emits a `Message[]` default export. The published `mds.d.ts` ambient declaration reflects this widened type.
+
+## Core Responsibilities
+
+- `transform.ts`: compile `.mds` files via `MdsApi.compileFile`, emit the JS module source (`export default`), serialize metadata
+- `loader.ts`: webpack/rspack integration via `createMdsLoader`
+- `frontmatter.ts`: `shouldTransform(id)` — decides if a module ID refers to an `.mds` file
+- `lazy-init.ts`: `LazyInit<T>` — ensures `mds.init()` is awaited exactly once per transformer instance
+- Does NOT: implement compilation logic, manage caching, handle HMR (delegated to plugin wrappers)
+
+## Standard Structure
+
+### Emitted module contract (post-refactor)
+
+The emitted JS module branches on `result.kind`:
+
+```typescript
+// transform.ts — inside transform()
+let defaultExport: string;
+if (result.kind === 'markdown') {
+  // Escape the string for embedding in a double-quoted JS literal
+  defaultExport = `export default "${escapeForJs(result.output)}";\n`;
+} else {
+  // kind === 'messages' — serialize the messages array as safe inline JSON
+  defaultExport = `export default ${safeJsonForJs(result.messages)};\n`;
+}
+
+const code =
+  defaultExport +
+  `export const metadata = ${safeJsonForJs({ warnings: result.warnings, dependencies: result.dependencies })};\n`;
+```
+
+So for a markdown `.mds`: `export default "..."` (string)
+For a messages `.mds`: `export default [{role:"...", content:"..."}]` (array literal)
+
+Both emit `export const metadata = { warnings: [...], dependencies: [...] };`
+
+### safeJsonForJs vs escapeForJs
+
+These two serializers have different contracts and must not be swapped:
+
+- `escapeForJs(str: string): string` — escapes special chars for embedding inside a double-quoted JS string literal (`"..."`)
+- `safeJsonForJs(value: unknown): string` — `JSON.stringify` + escapes `<`, U+2028, U+2029 for safe inline `<script>` embedding; used for array/object literals in `export default`
+
+```typescript
+// escapeForJs handles: \, ", \n, \r, \0, U+2028, U+2029
+export default "${escapeForJs(result.output)}"  // for strings
+
+// safeJsonForJs handles: <, U+2028, U+2029 (JSON.stringify handles the rest)
+export default ${safeJsonForJs(result.messages)} // for objects/arrays
+```
+
+`safeJsonForJs` is exported from `transform.ts` so tests can verify escape behavior directly.
+
+### U+2028/U+2029 construction pattern
+
+Literal U+2028 (line separator) and U+2029 (paragraph separator) cannot appear in regex literals or object key literals — the JS parser treats them as line terminators. Always use:
+
+```typescript
+// Regex: use new RegExp() string
+const JS_ESCAPE_RE = new RegExp('[\\\\\"\\n\\r\\0\\u2028\\u2029]', 'g');
+
+// Map keys: computed property assignment after the literal
+JS_ESCAPE_MAP[String.fromCodePoint(0x2028)] = '\\u2028';
+JS_ESCAPE_MAP[String.fromCodePoint(0x2029)] = '\\u2029';
+```
+
+### Published type declaration (mds.d.ts)
+
+```typescript
+// packages/bundler-utils/mds.d.ts
+interface MdsMessage { role: string; content: string; }
+
+declare module '*.mds' {
+  // Widened to union: string for markdown, MdsMessage[] for messages
+  const content: string | MdsMessage[];
+  export default content;
+  export const metadata: { warnings: string[]; dependencies: string[] };
+}
+```
+
+TypeScript consumers must narrow on `Array.isArray(content)` or similar to distinguish the two variants. The declaration uses a local `MdsMessage` interface (not importing from `@mdscript/mds`) to avoid a dependency cycle.
+
+### MdsApi interface (bundler-utils/src/types.ts)
+
+```typescript
+export interface MdsApi {
+  compileFile(path: string, options?: { vars?: Record<string, unknown> }): Promise<CompileResult>;
+  init(): Promise<void>;
+}
+
+export type CompileResult = MarkdownResult | MessagesResult;
+
+export interface MarkdownResult {
+  kind: 'markdown'; output: string; warnings: string[]; dependencies: string[];
+}
+export interface MessagesResult {
+  kind: 'messages'; messages: Message[]; warnings: string[]; dependencies: string[];
+}
+```
+
+The `MdsApi` interface intentionally omits `InitOptions` (bundler plugins always call `init()` with no arguments).
+
+### createMdsTransformer
+
+```typescript
+// Returns: { shouldTransform(id): boolean, transform(id): Promise<TransformResult> }
+export function createMdsTransformer(mds: MdsApi, options?: MdsPluginOptions): { ... }
+```
+
+Stateful: `LazyInit<void>` ensures `mds.init()` is awaited once. The `id` passed to `transform(id)` is trusted (sourced from bundler module pipeline); query/hash stripping is the plugin's responsibility.
+
+## Dependency Patterns
+
+Bundler plugin factories (`vite-plugin`, etc.) import `createMdsTransformer` or `createMdsLoader` from `bundler-utils` and pass in the `@mdscript/mds` module import. The `MdsApi` structural type means the real package satisfies it without an `implements` declaration.
+
+Test mocks for `MdsApi.compileFile` must return the new discriminated-union shape:
+
+```javascript
+// Correct mock for transform tests
+compileFile: async (id) => ({ kind: 'markdown', output: '# Test', warnings: [], dependencies: [] })
+// or
+compileFile: async (id) => ({ kind: 'messages', messages: [{role:'user',content:'hi'}], warnings: [], dependencies: [] })
+```
+
+Old mocks returning `{ output: ..., warnings: [], deps: [] }` (flat shape) are wrong — update them.
+
+## Error Handling
+
+Compilation errors from `mds.compileFile()` propagate through `transform()` as thrown exceptions. The bundler plugin wrappers (Vite, Rollup, Webpack) catch them and format them into bundler-specific error reporting via `FormattedError`.
+
+## Anti-Patterns
+
+- **Using `escapeForJs` for objects/arrays** — only valid for string values inside `"..."`. Use `safeJsonForJs` for `export default [{...}]`.
+- **Using `safeJsonForJs` for string default exports** — it won't add JS string literal quotes; use `escapeForJs` and wrap in `"..."`.
+- **Using literal U+2028/U+2029 in regex or map keys** — JS parser treats them as line terminators; use `new RegExp()` and `String.fromCodePoint()`.
+- **Importing `Message` type from `@mdscript/mds` in `mds.d.ts`** — creates a dependency cycle; use the local `MdsMessage` interface.
+- **Emitting `export default null` or `export default undefined` for messages** — the inactive case must be omitted at the napi level; the transformer always has a real messages array.
+- **Calling `mds.init()` unconditionally on every transform** — `LazyInit` ensures it's called once; re-calling it is wasteful and may break WASM backends.
+
+## Gotchas
+
+- `metadata` is emitted as a named export on BOTH kinds. Consumers that only use `metadata.dependencies` for HMR registration don't need to branch on kind.
+- The `shouldTransform(id)` function (`frontmatter.ts`) checks the file extension; it does NOT read file content. A `.mds` file with type-mds frontmatter is handled by `shouldTransform` returning true for the extension alone.
+- Watch mode: `addWatchFile` / `addDependency` are called with `result.dependencies` from the transform result, regardless of kind. HMR works identically for markdown and messages mode.
+- `LazyInit` stores state per transformer instance. Each call to `createMdsTransformer` creates an independent lazy initializer — `D3 createMdsLoader factory independent state`.
+- Linux HMR e2e tests are gated behind `MDS_HMR=1` env var; they are timing-sensitive and may flake on re-run.
+
+## Key Files
+
+- `packages/bundler-utils/src/transform.ts` — `createMdsTransformer`, `safeJsonForJs`, `escapeForJs`, kind-branching emit logic
+- `packages/bundler-utils/src/types.ts` — `MdsApi`, `CompileResult`, `MarkdownResult`, `MessagesResult`, `Message`, `TransformResult`
+- `packages/bundler-utils/mds.d.ts` — ambient `*.mds` module declaration; `string | MdsMessage[]` default export type
+- `packages/bundler-utils/src/loader.ts` — `createMdsLoader` (Webpack/Rspack variant)
+- `packages/bundler-utils/src/index.ts` — re-exports `MarkdownResult`, `MessagesResult`, `Message` from types
+
+## Related
+
+- Feature: mds-js — the `@mdscript/mds` package that implements `MdsApi`; `CompileResult` union defined in `packages/mds/src/types.ts` mirrors the `MdsApi.compileFile` return type
+- Feature: mds-napi — the native backend that produces the discriminated-union result consumed via `MdsApi`
+- Feature: mds-compiler — `CompiledOutput::Messages(Vec<Message>)` is the Rust type that drives messages mode; its wire format shapes the JS `MessagesResult`

.devflow/features/index.md

@@ -1,5 +1,6 @@
 # Feature Knowledge Index
 
 - **mds-fmt** — crates/mds-core/src, crates/mds-cli/src — Use when modifying the mds fmt formatter engine (crates/mds-core/src/formatter.rs), the mds fmt CLI subcommand (crates/mds-cli/src/fmt.rs), any change to mds-core's output model (clean_output, evaluate_nodes, @message/@define body evaluation, the lexer's fence recognition) that could silently break the formatter's compile-equivalence guarantee, or changes to the shared directory walker (output.rs). Keywords: mds fmt, format_str, format_str_with, format_str_named, FormatterInvariant, clean_output, compile-equivalence, idempotent, assert_equivalent, structural_equivalent, strip_trailing_insignificant_text, in_raw_content, raw_content_spans, protected_spans, R1 R2 R3 R4, safety gate, token lossiness, @message body, @define body, @block body, FmtConfig, FmtFlags, interior-verbatim contract, try_scan_fence_at, FenceMatch, deep_merge_yaml, RESERVED_MERGE_KEYS, is_default_excluded_dir, is_within_default_excluded_dir, walker exclusions, node_modules, hidden dirs, effective_parent, bare filename, atomic_write_file.
-- **mds-lint** — crates/mds-core/src/lint, crates/mds-cli/src, crates/mds-wasm/src, crates/mds-napi/src, crates/mds-python/src, packages/mds/src — Use when adding or modifying lint rules, extending the --fix pipeline, changing the JSON wire format, wiring lint into a binding layer, debugging unexpected exit codes and reverify gate refusals, or working on the ESC/bidi/newline injection defences. Keywords: mds lint, LintDiagnostic, fix_removals, fix_edits, TextEdit, FixLineSpan, diag_to_edits, LintResult, LintConfig, to_canonical_json, fix tier, reverify gate, FixOutcome, PartiallyFixed, apply_fixes_incremental, preview_fixes, PreviewOutcome, set_diag_display_path, AnalysisContext, ElseifBranch, end_offset, DefineFact, assertKnownKeys, CheckOptions, unreachable-branch, unused-variable, duplicate-import, empty-block, legacy-interpolation, is_output_neutral, all_output_neutral, Tier A Tier B Tier C, structural-standalone, compile-clean, is_standalone, sanitize_control_chars, sanitize_control_chars_wire, named_source_for_render, neutralize_source_for_render, SanitizedReport, SanitizedNode, MAX_AUX_DEPTH, EscapeMode, HUMAN WIRE, eprint_warning, safe_path, safe_inline, safe_file_display, preview_text_for, print_discipline, reverify_failure_reason, LintDirCtx, config_cache, dedup_contained_or_identical, EXIT 0 1 2 3, render_error_sanitized, eprint_error, display_sanitized, MdsError::display_sanitized, ESC-injection, CWE-150, CWE-117, bidi, Trojan-Source, CVE-2021-42574, U+061C, U+202E, U+FEFF, U+2028, U+2029, PF-014, PF-005, construction-time sanitization, per-field rule, Cow, #176.
+- **mds-lint** — crates/mds-core/src/lint, crates/mds-cli/src, crates/mds-wasm/src, crates/mds-napi/src, crates/mds-python/src, packages/mds/src — Use when adding or modifying lint rules, extending the --fix pipeline, changing the JSON wire format, wiring lint into a binding layer, debugging unexpected exit codes and reverify gate refusals, or working on the ESC/bidi/newline injection defences. Keywords: mds lint, LintDiagnostic, fix_removals, fix_edits, TextEdit, FixLineSpan, diag_to_edits, LintResult, LintConfig, to_canonical_json, fix tier, reverify gate, FixOutcome, PartiallyFixed, apply_fixes_incremental, preview_fixes, PreviewOutcome, set_diag_display_path, AnalysisContext, ElseifBranch, end_offset, DefineFact, assertKnownKeys, CheckOptions, unreachable-branch, unused-variable, duplicate-import, empty-block, legacy-interpolation, is_output_neutral, all_output_neutral, Tier A Tier B Tier C, structural-standalone, compile-clean, is_standalone, sanitize_control_chars, sanitize_control_chars_wire, named_source_for_render, neutralize_source_for_render, SanitizedReport, SanitizedNode, MAX_AUX_DEPTH, EscapeMode, HUMAN WIRE, eprint_warning, safe_path, safe_inline, safe_file_display, preview_text_for, print_discipline, reverify_failure_reason, LintDirCtx, config_cache, dedup_contained_or_identical, EXIT 0 1 2 3, render_error_sanitized, eprint_error, display_sanitized, MdsError::display_sanitized, ESC-injection, CWE-150, CWE-117, bidi, Trojan-Source, CVE-2021-42574, U+061C, U+202E, U+FEFF, U+2028, U+2029, PF-014, PF-005, construction-time sanitization, per-field rule, Cow, #176, ADR-008, ResultSink, from_rules_checked, relative_display, write_bytes, PF-020, #309, emit-ordering.
 - **source-map-security** — crates/mds-core/src, crates/mds-cli/src, packages/mds/src — Use when working with Source Map v3 generation, sources[] path relativization, the relativize_source choke-point, FileSystem::source_root(), CompileOptions.source_map_base, cross-surface source-map parity tests, or the Windows verbatim UNC path fix. Keywords: source map, sources[], relativize_source, source_map_base, source_root, path containment, basename fallback, PF-005, ADR-005, SEC-3, Windows verbatim UNC, path_to_unified, compute_source_map_base, apply_source_map_file_label, CF-SM2, V-SM1, differential test, two-level anchoring, map-relative, root-relative.
+- **mds-js** — packages/mds/src, packages/mds/__test__ — Use when modifying the JS/TS public API surface, adding backend methods, changing option types, debugging basePath rejection behaviour, changing result types, updating the backend contract, working on WASM/native backend validation, or debugging why a backend result is rejected. Keywords: compileFile, compile, check, checkFile, lint, lintFile, lintVirtual, CompileResult, MarkdownResult, MessagesResult, CheckResult, LintResult, LintDiagnostic, LintFileOptions, CompileFileOptions, FileOptions, assertResultShape, validateBackendMethods, METHOD_KEYS, forwardOpts, assertKnownKeys, getBasePathError, BASEPATH_REJECTORS, BASE_METHODS, NODE_METHODS, WASM_EXPORTS, discriminated union, kind, mds::invalid_backend_result, mds::invalid_options, basePath, synchronous throw, native.ts, wasm.ts, contract.ts, types.ts, node.ts, browser.ts, options.ts.