perf: add deterministic synthetic benchmark corpus

[seonghobae]

Scope and ownership

Refs #375. This branch is the established single-writer implementation lane for Inkspan's deterministic large-document performance evidence. Protected main remains the only shipped implementation authority. No SHA, merge-base, check result, review count, or workflow state recorded in this PR body is lifecycle authority; refetch those values at decision time.

The lane remains standalone, deterministic, provider-neutral, and network/model independent. It contains deterministic synthetic Markdown/Office fixtures, strict summary/comparison primitives, bounded Markdown and revision-evidence measurement harnesses, retained-memory settling analysis, demo-vendor chunking, an Office render duration/peak-RSS measurement primitive bound to the canonical synthetic fixture lock, and a one-command suite that composes Markdown + revision-evidence measurements. Packed mode measures exact modules extracted from one verified npm artifact and records package/runtime/source/reference-hardware provenance.

This work does not by itself establish production performance budgets, a supported large-document envelope, complete packed-editor/browser/IME/Yjs/Office matrix acquisition, leak freedom, or release acceptance.

Executable contract

Benchmark producers and fixture generators fail closed on unsafe symlink/non-directory ancestors, unsafe leaf targets, pre-existing hard-linked outputs, bounded-input violations, and unverifiable package/module provenance. Packed mode requires the exact package tarball digest and package identity, the active Node runtime identity, the source checkout identity, and a reference-hardware identifier.

benchmarks/run-current-suite.mjs additionally fails closed unless the benchmark checkout cleanliness guard succeeds before delegating to benchmarks/run-current-suite-core.mjs. This closes the false-provenance class where modified or untracked source could otherwise produce evidence labeled with an unchanged HEAD SHA. The failure is bounded and does not disclose dirty file paths.

office/benchmarks/measure_render.py accepts only a committed synthetic Office fixture whose exact byte count and SHA-256 match benchmarks/office-fixtures.lock.json. It requires a clean checkout and verified source revision, rejects unbounded iteration counts before inspecting caller-selected input, uses a fresh Python child process for each render sample, and records render duration plus process peak RSS with p50/p75/p95/max summaries and runtime/reference-hardware provenance. Ordinary evidence contains fixture identity/hash/size and measurements, never the document body or caller path. It performs no network, credential, service, database, or model operation.

Failure-contract / TDD lineage

A direct reproduction against the predecessor implementation established that git rev-parse HEAD alone cannot distinguish clean source from tracked or untracked worktree mutations. The narrow repair added a clean-checkout guard plus isolated temporary-repository contract tests for clean acceptance and dirty-state rejection; packed-suite tests exercise the delegated clean path. The test fixture is isolated from the repository checkout so parallel benchmark tests are not contaminated by a temporary dirty worktree.

The Office measurement contract was added test-first on the canonical performance branch: it requires lock-bound synthetic input, stable privacy-safe rejection of arbitrary/private content, bounded iteration work, isolated repeated samples, duration and peak-RSS evidence, and source/runtime/reference-hardware provenance. The immediately superseded test-only generation was cancelled before terminal hosted RED evidence, so it is lineage rather than passing evidence; current-head verification must be read live and predecessor/cancelled runs never transfer.

Earlier RED/GREEN iterations established output-symlink, hard-link, measurement privacy/resource, suite ordering, failure-atomicity, packed-artifact identity, runtime identity, source-SHA, summary/comparison, and retained-memory-analysis contracts. Historical workflow results document lineage only; they never transfer to a later head or base.

Remaining #375 acceptance work

This PR does not close #375 until protected evidence covers the applicable acceptance boundary. Remaining product work includes:

• repeated acquisition from the actual packed editor/runtime on accepted reference hardware and the committed deterministic corpus;
• trusted interaction, IME, mount/hydration, serialization, revision, autosave, Yjs, and print measurements at the source owner that can produce them without crossing Inkspan's host boundary;
• full supported Office fixture/profile acquisition across DOCX/XLSX/PPTX runtimes, publication-path measurement where applicable, and evidence-derived conversion budgets; the current Office primitive is measurement infrastructure, not a published support envelope;
• an accepted repeated-run retained-memory/leak/settling acquisition protocol rather than analysis of synthetic samples alone;
• support budgets derived from measured evidence rather than raised after a regression;
• accepted support-envelope architecture/operability/README claims;
• appropriately owned bounded PR smoke evidence plus scheduled/manual full evidence; and
• stable over-limit behavior wherever Inkspan owns the bounded API.

Browser/IME work, reference-host integration, release workflow, and organization-required workflow behavior remain with their established owners. Do not create competing source writers merely to make this performance lane appear complete.

Decision-time acceptance rule

Before any readiness, merge, release, closure, ownership, or support-envelope decision, independently refetch and reconcile at least:

• protected main, this PR's exact head, its independently resolved live base, ancestry/divergence, mergeability, and changed paths;
• all open Inkspan PRs/issues that can own or conflict with the affected source boundary;
• formal reviews, last-push approval requirements, and unresolved review threads;
• repository and organization-required workflow runs plus individual jobs and their actual checkout/source SHAs;
• live rulesets/branch governance and required-review policy;
• applicable package, browser, Office, coverage, security, SBOM, provenance, reproducibility, release, rollback, and operability evidence; and
• releases/tags/registry state when making a publication or buyer-facing support claim.

Pending, queued, in-progress, skipped-required, cancelled, absent, neutral, failed, stale, predecessor, wrong-checkout, synthetic-source-only, status-only, model-only, or vacuous evidence is non-passing. Automated comments/reviews are technical input, not qualifying independent approval. Any material head/base/ruleset movement invalidates the corresponding decision evidence.

Do not self-approve, weaken gates, transfer predecessor evidence, fabricate release identity, create a competing CI/security writer, or represent branch behavior as protected-main shipped truth.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}