Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,12 @@

本文件记录面向用户和集成方的重要变化。完整的发布叙事、升级说明和下载入口见对应版本的中文发布说明;实现证据和测试门禁见技术发布说明。

## 未发布(候选)

- 恢复 #51 等长 provider 原地更新,将字节恢复纳入 #71 逐目标事务;覆盖 Node POSIX 和 Windows worker。
- 合并正文扫描;新增显式 `sync --fast` / `switch --fast`,只读首行、保留模型、提示未执行检查,不支持原地更新时不隐式重写全文。
- 备份保持官方 v2,并继续支持 v1/v2 恢复;字节记录是可选扩展,不改变旧工具所需的标准恢复数据。稳定文件保留原 mtime,并处理并发追加与回滚。详见[兼容边界](docs/adr/proposed-transactional-provider-bytes.md)。

## [0.5.0] - 2026-08-15

### 新增
Expand Down
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,10 @@ codex-provider sync

`switch` 默认会在目标 Provider section 定义了 `model` 时同步根级 `model`。使用 `--keep-root-model` 保留当前值,或使用 `--model <name>` 显式指定。

本分支新增 `sync --fast` / `switch <provider-id> --fast`:只读 rollout 首行,保留模型,不检查历史用户消息和加密内容;不支持原地更新时在写入前报错,不自动全量重写。备份继续使用官方 v2 格式并支持恢复 v1/v2;旧工具按原有方式恢复,新代码可利用可选字节记录原地恢复。详见[设计与兼容边界](docs/adr/proposed-transactional-provider-bytes.md)。

建议使用统一长度的 ASCII Provider ID,优先 6 个字符(如将 `provider_a` 写作 `prov_a`),因为内置 `openai` 是 6 个字符,相对最通用;历史文件很多或很大时,不等长替换会重写整个 rollout、产生大量硬盘写入,等长且符合条件时可 in-place 替换。

SQLite Home 解析顺序:`--sqlite-home` → `config.toml` 根级 `sqlite_home` → `CODEX_SQLITE_HOME` → `<Codex Home>/sqlite`。只有默认布局会回退到 `<Codex Home>/state_5.sqlite`。

## 当前架构
Expand Down
4 changes: 4 additions & 0 deletions docs/README_EN.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,10 @@ codex-provider sync

By default, `switch` also updates the root-level `model` when the target provider section defines one. Use `--keep-root-model` to preserve the current value, or `--model <name>` to set it explicitly.

This branch adds `sync --fast` / `switch <provider-id> --fast`: read only rollout headers, preserve models, and leave historical user-message/encryption checks unperformed. Unsupported in-place updates fail before mutation, without automatic full rewriting. Backups retain the official v2 format and v1/v2 restore support; older tools use their existing restore path, while updated code can use optional byte records for in-place recovery. See [design and compatibility](adr/proposed-transactional-provider-bytes.md).

We recommend uniform-length ASCII provider IDs, preferably six characters (for example, `provider_a` as `prov_a`), because built-in `openai` has six characters and is the most common compatibility target. With many or large histories, different lengths require whole-rollout rewrites and substantial disk writes; eligible equal-length values can be replaced in place.

SQLite Home resolution order: `--sqlite-home` → root-level `sqlite_home` in `config.toml` → `CODEX_SQLITE_HOME` → `<Codex Home>/sqlite`. Only the default layout falls back to `<Codex Home>/state_5.sqlite`.

## Current Architecture
Expand Down
76 changes: 76 additions & 0 deletions docs/adr/proposed-transactional-provider-bytes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# Proposed: transactional provider byte updates and explicit fast scope

- Status: Proposed, not maintainer-approved or released
- Date: 2026-08-28
- Base: upstream main c7ff852 (v0.5.0)

## Context

[PR #51](https://github.com/Dailin521/codex-provider-sync/pull/51), by cccat6
(7231881, cdcde35, 84a60d3), avoided whole-rollout writes for equal-length IDs.
[PR #71](https://github.com/Dailin521/codex-provider-sync/pull/71) removed that
path while adding durable per-target recovery for #69. Restore the optimization
inside that journal, not around it. A separate fast scope addresses body reads.

## Proposal

- Keep current default synchronization semantics and full-rewrite fallback.
- Add a manifest-bound provider-byte strategy, durable before `applying`.
- Revalidate identity/header/bytes, write through the same handle, flush and
verify; undo through that handle/strategy without rename or tail truncation.
- Preserve original mtime for unchanged-size files. If an append races POSIX
stat/utimes, reassert only the guarded bytes for a fresh kernel write time;
never backdate again. Recovery also repairs this interrupted timestamp step.
Windows preserves timestamps under its exclusive handle. `updated_at` stays
unchanged; History's existing ordering and duplicate selection remain intact.
- Keep official v2 metadata/manifests and v1/v2 restore support. `mutation` is
optional: all standard original header/model/time/DB fields remain intact.
Older readers may ignore it and use their existing restore path; updated
readers use it for guarded byte-level undo. No private format version is added.
- Opt-in `--fast` narrows diagnostics/model scope, never durability or recovery.
Unsupported headers fail preflight rather than silently copying the body.
- Full mode fuses encryption, user-event and model checks in one body pass.
- Fast mode caps headers at 1 MiB, preserves root/history models and user-event
flags, retains provider/cwd/workspace repair, and reports unchecked diagnostics.
`--model` conflicts with `--fast`. Busy/changed targets retain partial outcomes.
- No credentials, account tooling, process management, new database or cache.

## Recovery boundaries

Only a unique unescaped ASCII provider value, equal encoded length and no
model rewrite qualifies. Validate file identity, size, header and target bytes
before writing; a stale precondition is skipped, never a full-rewrite fallback.
Normal undo and crash recovery use the same byte strategy. Old/new bytes or
one contiguous run of new bytes among old bytes are recoverable under the
append-only writer model. Unknown/disjoint edits, truncation and replaced
identities fail closed. Hardlinked targets are ineligible. Inode identity is
not a permanent proof against reuse; writes are not atomic to concurrent readers.

Cross-store restore preflights byte conflicts before changing config/SQLite;
rollout-only compensation attempts all targets even when one conflicts. Applying
and applied targets recover from the immutable manifest; damaged journals use
all candidates. Failure retains recoveryRequired and evidence. Full rewrite and
old-backup paths retain their existing active-writer limitations.

## Acceptance questions

Format interoperability does not retrofit in-place recovery into older tools:
their original full-rewrite and active-writer restrictions still apply. Verify
both directions with unmodified upstream Node/.NET readers and producers, as
well as recovery from applying-only partial writes. This does not claim that
old binaries implement the new inode-preserving strategy.

Fast scope is new behavior: discuss it separately from restoring #51. If the
unmerged #90 becomes main, integrate scope into its Plan/Revision/Restore v2
contracts before claiming support. No new public method, lock protocol or
second transaction system is necessary on the current base.

## Evidence

Focused tests: `in-place-transaction.test.js`, `fast-sync.test.js`, and the
Windows worker/native-helper tests. They cover short writes, flush failure,
crashes, torn journals, rollback conflicts, append/mtime races, inode and tail
preservation, fast preflight and default behavior. `scripts/benchmark-provider-io.mjs`
measures disposable fixtures; report logical I/O separately from kernel writes.
Native, simulated and unrun coverage belong in the PR validation report.
Process-exit tests do not prove arbitrary power-loss or hostile-writer safety.
2 changes: 2 additions & 0 deletions docs/architecture/contracts/CLI_CONTRACT_ZH.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# CLI 命令兼容合同

> 本分支候选增量(未发布):`sync --fast` 和 `switch <provider-id> --fast` 只扫描首行,要求原地 provider 更新,保留模型并提示未执行的历史检查。`--fast` 不接收值、不支持其他命令,与 `--model` 互斥,允许冗余的 `--keep-root-model`。默认同步范围不变;摘要增加 `In-place rollout updates`。不支持的快速操作在业务写入前失败,不自动全量重写。详见[候选 ADR](../../adr/proposed-transactional-provider-bytes.md)。

> 状态:Phase 0 兼容基线
>
> 基线版本:`@dailin521/codex-provider-sync` v0.5.0
Expand Down
11 changes: 11 additions & 0 deletions docs/architecture/contracts/CORE_EXTERNAL_BEHAVIOR_ZH.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,17 @@

## 1. 文档目的

本分支候选增量(未发布,待上游评审):

- Node `runSync` / `runSwitch` 新增 `fast=false`。默认保留原检查范围,合并为一次正文扫描。
- 合格的等长 provider 使用官方 v2 manifest 中的可选 mutation 记录;新代码执行、回滚和崩溃恢复保留同一 inode/file ID,开始写入后不得回退全文重写。
- 稳定文件保留原 mtime;POSIX 检测并修复 stat/utimes 期间的追加竞争,Windows 使用独占句柄;`updated_at` 和 History 的去重/排序规则不变。
- `fast=true` 只读首行(上限 1 MiB)及属性,保留根级/历史模型,不检查用户事件与加密内容;保留 provider、首行 cwd、workspace 修复及数据库事务。
- 快速模式静态不支持项在备份和业务写入前以 `FAST_MODE_UNSUPPORTED` 失败,不隐式全量重写;运行时 busy/changed 沿用 partial 分类。
- 结果增加 `inPlaceSessionFiles`;快速结果另含 `scanScope="metadata"`、`unchecked=["historyModels","userEventFlags","encryptedContent"]`、`encryptedContentCounts=null` 和未检查警告。
- metadata/manifest 均保持官方 v2,继续读取 v1/v2;标准恢复字段不变。旧 Node/.NET 可忽略新增记录并按原流程恢复,不代表旧版本自动具备原地恢复能力。
- 详见[候选 ADR](../../adr/proposed-transactional-provider-bytes.md)。V1/#90 的 Plan/Revision/Restore v2 接入不属于本分支已完成能力。

本文冻结 vNext 迁移开始时 Node 实现已经提供的外部行为。这里的“外部”不仅指 npm 最终用户,也包括当前 CLI 与 Local Web UI 对 Node service 的真实依赖。

本文解决三个问题:
Expand Down
2 changes: 2 additions & 0 deletions docs/architecture/contracts/ERROR_CODES_ZH.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@

## 1. 目的

本分支候选增量(未发布):`FAST_MODE_UNSUPPORTED` 表示首行无效/超过快速读取上限,或待修改文件无法采用等长原地策略。发生在备份和业务 mutation 之前,修正输入或显式选择完整同步后可重试,不要求 recovery。`IN_PLACE_RESTORE_FAILED` 仅作为内部原因;服务层仍以既有 `SYNC_FAILED_ROLLED_BACK` / `RECOVERY_REQUIRED` 表达最终恢复结果。不得将未知字节、截断或文件替换伪装成安全回退。

本文冻结 vNext 的错误分类、兼容映射和演进规则,使调用方依据稳定的 `code` 决策,而不是解析自然语言 `message`、异常类型名或堆栈。

本文不表示当前代码已经完成统一。当前 Node、Web 与 .NET 仍存在不同大小写、命名和结构;这些现状被列为 Legacy Surface,由后续结构化错误 PR 通过 Adapter 渐进收口。
Expand Down
2 changes: 2 additions & 0 deletions docs/migration/BEHAVIOR_FIXTURES_ZH.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# vNext 行为兼容 Fixture 清单

本分支候选夹具:`test/in-place-transaction.test.js` 验证短写、崩溃、损坏 journal、逐目标回滚、追加/mtime 竞争与 History 副本选择;`test/fast-sync.test.js` 验证快速范围、无正文流、模型/SQLite/cwd、前置失败和格式兼容;`test/windows-rewrite-worker.test.js` 与 `test/windows-provider-bytes.ps1` 验证协议和原生独占句柄。

> **状态:Accepted(阶段 0 语义清单;共享 Corpus 尚未创建)**
>
> **日期:2026-08-24**
Expand Down
69 changes: 69 additions & 0 deletions scripts/benchmark-provider-io.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
import assert from "node:assert/strict";
import { createHash } from "node:crypto";
import fs from "node:fs";
import fsp from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { performance } from "node:perf_hooks";
import { runSync } from "../src/service.js";

// Disposable, bounded-memory benchmark. Never use an existing Codex Home.
delete process.env.CODEX_SQLITE_HOME;
const mib = Number(process.argv[2] ?? 32);
if (!Number.isInteger(mib) || mib < 1 || mib > 256) throw new Error("Size must be 1..256 MiB.");
const root = await fsp.mkdtemp(path.join(os.tmpdir(), "provider-io-benchmark-"));
const line = JSON.stringify({ type: "event_msg", payload: { text: "x".repeat(4000) } }) + "\n";
const block = Buffer.from(line.repeat(128));

async function counters() {
if (process.platform !== "linux") return null;
const text = await fsp.readFile("/proc/self/io", "utf8");
return Object.fromEntries(text.trim().split("\n").map(line => {
const [key, value] = line.split(": ");
return [key, Number(value)];
}));
}

async function hashTail(file, offset) {
const hash = createHash("sha256");
for await (const chunk of fs.createReadStream(file, { start: offset })) hash.update(chunk);
return hash.digest("hex");
}

try {
const results = [];
for (const mode of ["full-equal", "fast-equal", "full-unequal"]) {
const home = path.join(root, mode);
await fsp.mkdir(path.join(home, "sessions"), { recursive: true });
await fsp.writeFile(path.join(home, "config.toml"), 'model_provider="prov_a"\n');
const file = path.join(home, "sessions", "rollout-fixture.jsonl");
const header = JSON.stringify({ type: "session_meta", payload: {
id: "fixture", model_provider: mode === "full-unequal" ? "provider_old" : "openai"
} }) + "\n";
const h = await fsp.open(file, "w");
try {
await h.writeFile(header);
await h.writeFile('{"type":"event_msg","payload":{"type":"user_message","message":"fixture"}}\n');
for (let n = 0; n < mib * 1024 * 1024; n += block.length) await h.writeFile(block);
await h.sync();
} finally { await h.close(); }
const beforeStat = await fsp.stat(file, { bigint: true });
const beforeHash = await hashTail(file, Buffer.byteLength(header));
const before = await counters();
const start = performance.now();
const result = await runSync({ codexHome: home, fast: mode === "fast-equal" });
const ms = performance.now() - start;
const after = await counters();
const afterStat = await fsp.stat(file, { bigint: true });
const headerAfter = header.replace(mode === "full-unequal" ? "provider_old" : "openai", "prov_a");
assert.equal(await hashTail(file, Buffer.byteLength(headerAfter)), beforeHash);
const delta = before && Object.fromEntries(["rchar", "wchar", "read_bytes", "write_bytes"].map(k => [k, after[k] - before[k]]));
results.push({ mode, rolloutBytes: Number(beforeStat.size), ms: Math.round(ms),
inPlace: result.inPlaceSessionFiles, sameInode: beforeStat.ino === afterStat.ino,
...(delta ? { processIo: delta } : {}) });
}
console.log(JSON.stringify({ platform: process.platform, results,
note: "Warm-cache synthetic benchmark. rchar/wchar are logical process I/O, not SSD wear; kernel write_bytes excludes device-internal amplification." }, null, 2));
} finally {
await fsp.rm(root, { recursive: true, force: true });
}
28 changes: 22 additions & 6 deletions src/backup.js
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import {
GLOBAL_STATE_BACKUP_FILE_BASENAME,
GLOBAL_STATE_FILE_BASENAME
} from "./constants.js";
import { restoreSessionChanges } from "./session-files.js";
import { restoreSessionChanges, validateProviderMutationDescriptor, validateProviderByteRestore } from "./session-files.js";
import {
assertSqliteWritable,
createSqliteOnlineBackup,
Expand Down Expand Up @@ -156,6 +156,12 @@ async function validateSessionManifestEntries(entries, codexHome) {
throw new Error(`Backup session manifest contains a duplicate rollout target: ${entry.path}`);
}
seen.add(key);
if (entry.mutation) {
if (entry.modelOnlyChange || entry.originalTurnContextModels?.length) {
throw new Error(`Provider byte mutation cannot also restore model fields: ${entry.path}`);
}
validateProviderMutationDescriptor(entry.mutation, entry.path, entry.originalFirstLine, entry.originalSeparator);
}
await assertNoLinkedPathSegments(lexicalRoot, target);
const [canonicalRoot, canonicalTarget] = await Promise.all([
fs.realpath(lexicalRoot),
Expand Down Expand Up @@ -242,6 +248,7 @@ export async function createBackup({
codexHome,
targetProvider,
sessionChanges,
fast = false,
configPath,
configBackupText
}) {
Expand Down Expand Up @@ -294,6 +301,7 @@ export async function createBackup({
namespace: BACKUP_NAMESPACE,
codexHome,
targetProvider,
...(fast ? { scanScope: "metadata" } : {}),
createdAt: new Date().toISOString(),
// Keep the full pre-mutation source of truth for the lifetime of the
// backup. appliedPaths is only a compatibility hint for backups without a
Expand All @@ -305,6 +313,9 @@ export async function createBackup({
originalFirstLine: change.originalFirstLine,
originalSeparator: change.originalSeparator,
originalMtimeMs: change.originalMtimeMs,
// Optional byte-level undo; the standard v2 fields remain sufficient
// for older readers to use their existing full-header restore path.
...(change.inPlaceMutation ? { mutation: change.inPlaceMutation } : {}),
// Per-line record of the original turn_context.model values
// so a failed rollback can put the per-turn model back to
// what it was before the sync. Without this, a restore
Expand All @@ -328,6 +339,7 @@ export async function createBackup({
codexHome,
sqliteHome: actualSqliteHome,
targetProvider,
...(fast ? { scanScope: "metadata" } : {}),
createdAt: sessionManifest.createdAt,
dbFiles: copiedDbFiles,
sqliteDbFiles: copiedSqliteDbFiles,
Expand All @@ -344,11 +356,8 @@ export async function updateSessionBackupManifest(backupDir, sessionChanges, opt
const sessionManifest = JSON.parse(await fs.readFile(manifestPath, "utf8"));
const metadata = JSON.parse(await fs.readFile(metadataPath, "utf8"));

// Promote older manifests to the v2 schema so restoreSessionChanges
// can rely on the per-line `originalTurnContextModels` field.
if (sessionManifest.version !== 2) {
sessionManifest.version = 2;
}
// Preserve the official v1-to-v2 model-snapshot upgrade.
if (sessionManifest.version !== 2) sessionManifest.version = 2;

const filesByPath = new Map(
(sessionManifest.files ?? []).map((entry) => [pathComparisonKey(entry.path), entry])
Expand Down Expand Up @@ -606,6 +615,13 @@ export async function restoreBackup(backupDir, storageOrCodexHome, options = {})
} else {
sessionRestoreEntries = await selectSessionRestoreEntries(backupDir, sessionManifest);
}
// Preflight before rewinding other stores. Rollout-only compensation must
// attempt every target, even if another target has conflicting bytes.
if (restoreConfig || restoreGlobalState || restoreDatabase) {
for (const entry of sessionRestoreEntries) {
if (entry.mutation) await validateProviderByteRestore(entry);
}
}
}

let stateDb = null;
Expand Down
Loading