diff --git a/content/blog/design/checksum-verify.md b/content/blog/design/checksum-verify.md new file mode 100644 index 00000000..cab2cbb8 --- /dev/null +++ b/content/blog/design/checksum-verify.md @@ -0,0 +1,182 @@ +--- +title: "Read-Only Checksum Audit and Reliable CLI Output" +linkTitle: "Checksum Verify" +date: 2026-08-29 +lastmod: 2026-08-29 +author: "Ruohang Feng" +summary: > + MCLI can audit stored S3 checksums against logical object bytes without mutating data. This record defines selection, classification, report and exit semantics, and the non-TTY output contract required by pipelines and CI. +tags: [Design, S3, Compatibility, Checksum, mcli] +weight: 28 +draft: true +url: "/blog/design/checksum-verify/" +--- + +This is the design and implementation record for MCLI's read-only checksum +verification workflow and [pgsty/mc#5](https://github.com/pgsty/mc/issues/5), +the non-TTY output defect found during release review. + +> **Status:** implementation, local commits, real-S3 probes, subprocess tests, +> full unit/race, lint, vet, branding, credits, cross-builds, and clean local +> provenance are complete. Push, pull request, hosted CI, release, Server image +> integration, and public documentation deployment remain separate gates.
+> **Owner:** [`pgsty/mc`](https://github.com/pgsty/mc).
+> **Tracking:** [pgsty/mc#5](https://github.com/pgsty/mc/issues/5).
+> **Safety boundary:** verification is read-only; repair is not part of this +> command. + +## Too Long; Didn't Read (TL;DR) {#tldr} + +Historical CopyObject implementations could calculate a stored additional +checksum over transformed storage bytes instead of the logical bytes returned +by S3. `mcli checksum verify` inventories objects and independently streams the +logical body through the recorded algorithm. Each candidate becomes `MATCH`, +`MISMATCH`, `NO_CHECKSUM`, `UNKNOWN_*`, or `SKIPPED_*`. + +The first implementation worked in a terminal but printed nothing when stdout +was redirected. MCLI automatically marked non-TTY execution as quiet to disable +progress UI, and the new command accidentally treated that internal state as a +user request to suppress audit records. The repair separates semantic output +from progress suppression without changing global quiet behavior or enabling +progress bars in CI. + +## Command and scope {#scope} + +```console +mcli checksum verify ALIAS/BUCKET/OBJECT +mcli checksum verify --recursive ALIAS/BUCKET[/PREFIX] +mcli checksum verify --manifest candidates.jsonl ALIAS +``` + +Version one supports CRC32, CRC32C, CRC64NVME, SHA1, and SHA256 checksums marked +as `FULL_OBJECT`. It can select one object, an exact VersionID, current objects +under a prefix, all versions, or exact entries from a JSON Lines manifest. It +also supports SSE-C key mappings, time and size filters, dry-run estimation, +bounded workers, download limits, JSON output, and an optional JSON Lines report. + +It does not verify `COMPOSITE` checksums, infer type from an ETag, inspect +`xl.meta`, identify the historical writer with certainty, or repair metadata. + +## Server checksum compatibility boundary {#server-boundary} + +Amazon S3 currently defines ten additional-checksum algorithms. This SILO +release implements CRC32, CRC32C, CRC64NVME, SHA1, and SHA256. MD5, SHA512, +XXHASH64, XXHASH3, and XXHASH128 are not persisted or verified yet. + +An unsupported `x-amz-checksum-*` value or trailer is rejected with HTTP 400 +`InvalidArgument`; SILO no longer accepts the object while silently discarding +the caller's integrity assertion. This also means cross-vendor replication of +an object carrying one of the five unsupported algorithms fails visibly and +must be retried with a supported algorithm. Supported SDK control headers such +as `x-amz-sdk-checksum-algorithm` remain accepted when paired with a supported +value or trailer. + +CRC64NVME is a full-object checksum only. Requests combining CRC64NVME with +`COMPOSITE` are rejected instead of being silently converted to `FULL_OBJECT`. +Comma-separated checksum trailer names are parsed individually; declaring more +than one checksum trailer remains invalid. + +The release deliberately rejects rather than adding five new persisted +checksum type identifiers. Older nodes cannot interpret such identifiers +safely during a rolling upgrade. Adding those algorithms requires a separate +storage and mixed-version compatibility design. + +## Read-only data path {#data-path} + +For every selected object, MCLI: + +1. sends `HEAD` with checksum mode enabled and retains every supported checksum + plus `ChecksumType`; +2. rejects unsupported or ambiguous states as `UNKNOWN_*` instead of guessing; +3. streams `GET` logical bytes through bounded hashers without writing the body + to disk; +4. uses VersionID pinning, or `If-Match` plus a second `HEAD` for mutable + unversioned/null objects; +5. compares independently calculated values with the stored values. + +The S3 boundary allows LIST, HEAD, and GET only. Tests fail if a write method +reaches the mock endpoint. + +## Result and exit contract {#result-contract} + +Every candidate produces one stable result: + +| Result | Meaning | +|:--|:--| +| `MATCH` | Every supported stored checksum matches the returned logical bytes | +| `MISMATCH` | At least one stored checksum differs | +| `NO_CHECKSUM` | No additional checksum exists; the body is not read | +| `WOULD_VERIFY` | Dry-run found a supported full-object checksum | +| `UNKNOWN_*` | MCLI cannot make a reliable statement | +| `SKIPPED_*` | A filter intentionally excluded the object | + +`--fail-on` accepts `mismatch`, `unknown`, `any`, or `none`. The default `any` +returns exit 1 for mismatches and incomplete verification. Dry-run does not +apply `--fail-on`. Argument, authentication, enumeration, and report-write +failures remain command failures rather than object classifications. + +In particular, `SKIPPED_TOO_LARGE` makes the default `any` return exit 1 because +the size cap leaves the audit incomplete. Time-filter and delete-marker skips do +not fail by themselves. + +## Output and automation contract {#output-contract} + +Object records and the final summary are semantic output: + +- Unless the caller explicitly sets `--quiet`, `-q`, or `MC_QUIET=true`, stdout + receives every object record and the final summary in both TTY and non-TTY + execution. +- Non-TTY `--json` emits exactly one compact JSON value per line. TTY JSON keeps + MCLI's existing pretty presentation. +- Global flags work at the app, `checksum`, and `verify` levels. +- `--report` is independent of stdout. It still writes object records and the + final summary as JSON Lines when explicit quiet suppresses stdout. +- Output transport does not change `--fail-on` decisions. + +The distinction matters because MCLI's historical `globalQuiet` has two inputs: +an explicit quiet flag and an automatic non-TTY state used to disable progress +UI. Changing that global would risk re-enabling progress output across copy, +get, put, mirror, and other commands. + +The selected repair is command-local. It walks the full CLI context chain for +explicit quiet/JSON flags because the CLI library's `GlobalBool` stops at the +nearest ancestor flag set. It also restores JSON Lines mode inside the checksum +action because nested `Before` hooks can reset it after an app-level `--json`. +No other command's progress or output behavior changes. + +## Report, secrets, and operational cost {#operations} + +On POSIX systems report files are created with mode `0600`; Windows relies on +the account's filesystem ACLs. Reports must not already exist and contain +metadata/results rather than object bodies or SSE-C keys. The manifest likewise +contains only bucket, key, and optional VersionID. + +Verification downloads every supported object body. Operators should use +`--dry-run`, `--max-size`, time filters, `--max-workers`, and the global download +limit to bound cost and load. `NO_CHECKSUM` and `UNKNOWN_*` counts must remain +visible; neither may be presented as successful verification. + +## What a mismatch proves {#meaning} + +A mismatch proves only that the additional checksum returned at verification +time does not describe the logical bytes returned at verification time. It does +not prove that a particular historical compression defect created the object, +and it is not an external source-of-truth comparison. + +Do not overwrite checksum metadata in place. Audit and classify first. For a +confirmed, operationally relevant mismatch, prefer a new key or new version, +verify the replacement, then switch consumers deliberately. Leave `UNKNOWN_*` +objects out of automatic repair. + +## Verification record and release boundary {#verification} + +The local acceptance matrix covers TTY human/JSON, non-TTY pipes, regular-file +redirects, app/parent/leaf JSON and quiet flags, environment quiet, report under +quiet, report-write failure, and MISMATCH/UNKNOWN exit status. It also includes +real historical `MATCH`, `MISMATCH`, and unsupported-composite objects on a local +S3 server. + +Local commits are not a release. Close [pgsty/mc#5](https://github.com/pgsty/mc/issues/5) +only after the code and this decision record are merged and hosted `main` CI is +green. A signed tag, packages, container image, bundled Server client, public +deployment, and production audit remain later, separately evidenced gates. diff --git a/content/blog/design/checksum-verify.zh.md b/content/blog/design/checksum-verify.zh.md new file mode 100644 index 00000000..476d9c58 --- /dev/null +++ b/content/blog/design/checksum-verify.zh.md @@ -0,0 +1,156 @@ +--- +title: "只读 Checksum 审计与可靠的 CLI 输出契约" +linkTitle: "Checksum Verify" +date: 2026-08-29 +lastmod: 2026-08-29 +author: "冯若航" +summary: > + MCLI 可以在不修改数据的前提下,将 S3 已存 checksum 与对象逻辑字节重新比对。本文定义候选选择、结果分类、report、退出码,以及管道和 CI 所依赖的 non-TTY 输出契约。 +tags: [设计, S3, 兼容性, Checksum, mcli] +weight: 28 +draft: true +url: "/zh/blog/design/checksum-verify/" +--- + +本文是 MCLI 只读 checksum 校验流程,以及发布审查中发现的 non-TTY 输出缺陷 +[pgsty/mc#5](https://github.com/pgsty/mc/issues/5) 的设计与实现记录。 + +> **状态:** 实现、本地提交、真实 S3 探针、subprocess 测试、完整 unit/race、 +> lint、vet、品牌、CREDITS、跨平台构建及本地干净溯源已经完成。push、PR、 +> 托管 CI、发布、Server 镜像集成与公共文档部署仍是相互独立的后续门禁。
+> **归属:** [`pgsty/mc`](https://github.com/pgsty/mc)。
+> **跟踪:** [pgsty/mc#5](https://github.com/pgsty/mc/issues/5)。
+> **安全边界:** 本命令只读校验,不负责修复。 + +## 太长不看(TL;DR) {#tldr} + +历史 CopyObject 实现可能在转换后的存储字节上计算 additional checksum,而不是在 +S3 返回给客户端的逻辑对象字节上计算。`mcli checksum verify` 会筛选对象,独立地 +把逻辑对象流送入已记录的算法,并把每个候选分类为 `MATCH`、`MISMATCH`、 +`NO_CHECKSUM`、`UNKNOWN_*` 或 `SKIPPED_*`。 + +首版实现在终端中工作正常,但 stdout 被重定向时完全不输出。MCLI 为了在非终端 +环境中禁用进度 UI,会自动把执行状态标记为 quiet;新命令错误地把这项内部状态 +理解成了“用户要求隐藏审计结果”。修复将语义输出与进度抑制分离,同时不改变 +全局 quiet 行为,也不会在 CI 中重新打开进度条。 + +## 命令与范围 {#scope} + +```console +mcli checksum verify ALIAS/BUCKET/OBJECT +mcli checksum verify --recursive ALIAS/BUCKET[/PREFIX] +mcli checksum verify --manifest candidates.jsonl ALIAS +``` + +V1 支持标记为 `FULL_OBJECT` 的 CRC32、CRC32C、CRC64NVME、SHA1 与 SHA256。 +候选可以是单个对象、精确 VersionID、前缀下的当前对象、所有版本,或 JSON Lines +manifest 给出的精确条目。它还支持 SSE-C key 映射、时间与大小过滤、dry-run 成本 +估计、有界 worker、下载限速、JSON 输出和可选的 JSON Lines report。 + +V1 不验证 `COMPOSITE` checksum,不从 ETag 推断类型,不读取 `xl.meta`,不能确定 +历史 writer,也不会修复 metadata。 + +## Server checksum 兼容边界 {#server-boundary} + +Amazon S3 当前定义了十种附加 checksum 算法。本次 SILO 实现 CRC32、 +CRC32C、CRC64NVME、SHA1 与 SHA256;MD5、SHA512、XXHASH64、XXHASH3、 +XXHASH128 尚未持久化或校验。 + +不支持的 `x-amz-checksum-*` 值或 trailer 现在会明确返回 HTTP 400 +`InvalidArgument`,SILO 不再一边返回成功、一边静默丢弃调用方的完整性断言。 +这也意味着:从其他厂商复制携带上述五种未支持算法的对象时,复制会显式失败, +必须改用已支持算法后重试。`x-amz-sdk-checksum-algorithm` 等 SDK 控制头本身 +仍可使用,但必须配合已支持的 checksum 值或 trailer。 + +CRC64NVME 只支持整对象 checksum。CRC64NVME 与 `COMPOSITE` 的组合会被拒绝, +不再静默改写为 `FULL_OBJECT`。逗号分隔的 checksum trailer 会逐项解析; +同时声明多个 checksum trailer 仍属于非法请求。 + +本次选择“明确拒绝”,而不是直接增加五种持久化 checksum 类型标识。 +旧节点无法在滚动升级期间安全解释新标识;完整支持这些算法需要单独设计 +存储格式与混合版本兼容方案。 + +## 只读数据路径 {#data-path} + +对每个候选对象,MCLI: + +1. 使用 checksum mode 执行 `HEAD`,保留所有支持的 checksum 与 `ChecksumType`; +2. 对不支持或含糊的状态返回 `UNKNOWN_*`,绝不猜测; +3. 将 `GET` 返回的逻辑字节流送入有界 hasher,不把对象体写入磁盘; +4. 对固定版本使用 VersionID;对可变的未版本化/null 对象使用 `If-Match`,并在 + 读取后再次 `HEAD`; +5. 将独立计算结果与已存值比较。 + +S3 边界只允许 LIST、HEAD 与 GET;如果 mock endpoint 收到写方法,测试必须失败。 + +## 结果与退出码契约 {#result-contract} + +每个候选只产生一个稳定结果: + +| 结果 | 含义 | +|:--|:--| +| `MATCH` | 所有支持的已存 checksum 都匹配逻辑对象字节 | +| `MISMATCH` | 至少一个已存 checksum 不同 | +| `NO_CHECKSUM` | 没有 additional checksum,因此不读取对象体 | +| `WOULD_VERIFY` | dry-run 找到可验证的 full-object checksum | +| `UNKNOWN_*` | MCLI 无法给出可靠判断 | +| `SKIPPED_*` | 过滤器主动排除了对象 | + +`--fail-on` 支持 `mismatch`、`unknown`、`any` 与 `none`。默认 `any` 会在 mismatch +或校验不完整时返回 exit 1。dry-run 不应用 `--fail-on`。参数、认证、枚举与 report +写入失败属于命令失败,而不是对象分类。 + +其中,`SKIPPED_TOO_LARGE` 会让默认 `any` 返回 exit 1,因为大小上限使审计不完整; +时间过滤与 delete-marker skip 本身不会触发失败。 + +## 输出与自动化契约 {#output-contract} + +对象记录和最终 summary 都是命令的语义输出: + +- 除非调用方显式设置 `--quiet`、`-q` 或 `MC_QUIET=true`,TTY 与 non-TTY stdout + 都必须收到全部对象记录和最终 summary。 +- non-TTY `--json` 每行输出一个紧凑 JSON 值;TTY JSON 保留 MCLI 既有的美化格式。 +- 全局参数在 app、`checksum` 与 `verify` 三层位置都必须生效。 +- `--report` 独立于 stdout;即使显式 quiet 让 stdout 静默,它仍会写入对象记录和 + 最终 summary 的 JSON Lines。 +- 输出通道不会改变 `--fail-on` 的判定。 + +这一区分之所以必要,是因为 MCLI 历史上的 `globalQuiet` 有两个来源:用户显式的 +quiet 参数,以及拿不到终端尺寸时自动启用、用于关闭进度 UI 的 non-TTY 状态。 +直接修改这个全局量,可能让 copy、get、put、mirror 等命令在 CI 中重新输出进度条。 + +最终修复只作用于 checksum 命令。它沿完整 CLI context 链查找显式 quiet/JSON +参数,因为 CLI 库的 `GlobalBool` 会停在最近的祖先 flag set;同时在 checksum action +内部恢复 JSON Lines,因为嵌套 `Before` hook 可能在 app-level `--json` 之后重置它。 +其他命令的进度与输出行为均不改变。 + +## Report、秘密与运行成本 {#operations} + +POSIX 系统上的 Report 文件以 `0600` 新建;Windows 使用账户的文件系统 ACL。 +目标必须不存在;它只包含 metadata 与结果,不包含对象体 +或 SSE-C key。Manifest 同样只保存 bucket、key 与可选 VersionID。 + +校验会下载每个受支持对象的完整逻辑内容。运维人员应使用 `--dry-run`、`--max-size`、 +时间过滤、`--max-workers` 与全局下载限速控制成本和负载。`NO_CHECKSUM` 与 +`UNKNOWN_*` 数量必须显式展示,二者都不能被包装成“校验成功”。 + +## Mismatch 能证明什么 {#meaning} + +Mismatch 只能证明:校验时 endpoint 返回的 additional checksum,不能描述同一时刻 +返回的逻辑对象字节。它不能单独证明对象一定由某个历史压缩缺陷生成,也不是与外部 +真值的比较。 + +不要原地覆盖 checksum metadata。应先只读审计和分类。对已经确认且确有业务影响的 +mismatch,优先写入新 key 或新 version,验证替代对象后再显式切换消费者; +`UNKNOWN_*` 对象不得进入自动修复。 + +## 验证记录与发布边界 {#verification} + +本地验收矩阵覆盖 TTY human/JSON、non-TTY pipe、普通文件重定向、app/parent/leaf +三层 JSON 与 quiet、环境变量 quiet、quiet 下的 report、report 写失败,以及 +MISMATCH/UNKNOWN 退出码。真实本地 S3 还覆盖了历史 `MATCH`、`MISMATCH` 与不支持 +的 composite 对象。 + +本地 commit 不是正式发布。只有代码与本文决策记录合并、托管 `main` CI green 后, +才能关闭 [pgsty/mc#5](https://github.com/pgsty/mc/issues/5)。签名 tag、软件包、容器 +镜像、Server 内置客户端、公共部署与生产审计仍是之后需要独立证明的门禁。 diff --git a/content/blog/release/silo-next-hardening.md b/content/blog/release/silo-next-hardening.md new file mode 100644 index 00000000..0d2606eb --- /dev/null +++ b/content/blog/release/silo-next-hardening.md @@ -0,0 +1,130 @@ +--- +title: "SILO Next Release: Pre-Release Hardening Notes" +linkTitle: "SILO Next Hardening" +date: 2026-08-29 +lastmod: 2026-08-29 +author: "Ruohang Feng" +description: "Draft compatibility, integrity, site-replication, client, and release-pipeline notes for the SILO release after 20260806." +tags: [Release, Draft, Compatibility, Checksum, S3] +weight: 9 +draft: true +url: "/blog/release/silo-next-hardening/" +--- + +> **Draft — no release tag, package, or image exists yet.** This page records +> the code and compatibility boundary being validated before the release after +> `RELEASE.2026-08-06T00-00-00Z`. Commit IDs, artifact hashes, and the final +> version will be filled only after source PRs and hosted CI are complete. + +## Integrity and encryption fixes {#integrity} + +- Metadata-only `CopyObject` of a null version now records compression metadata + for the bytes the object layer actually stores. Both compression directions + and SSE-C key rotation are covered; the previous failure modes ranged from a + loud `s2: corrupt input` to silent short-object reads. +- Zero-byte SSE-C reads authenticate a supplied customer key even though no + decryptor is constructed. GET, CopyObject, and UploadPartCopy now match the + non-empty-object 403 behavior; internal no-decryption, replication, restore, + range, and request-precondition ordering are preserved. +- `GetObjectAttributes` now unseals the SSE-C key before returning plaintext + object size or completed multipart layout. Correct, wrong, and missing keys + are covered for zero-byte and non-empty objects. +- CopyObject checksum metadata is decrypted with the destination key when the + destination is re-encrypted, and explicit multipart checksum-type assertions + are no longer accepted without their value. + +## Checksum compatibility {#checksum} + +SILO implements CRC32, CRC32C, CRC64NVME, SHA1, and SHA256. Requests asserting +MD5, SHA512, XXHASH64, XXHASH3, XXHASH128, or an unknown future +`x-amz-checksum-*` value/trailer now return 400 `InvalidArgument` rather than +200 with the assertion silently discarded. This includes PutObject, +CreateMultipartUpload, UploadPart, CopyObject, and UploadPartCopy. + +CRC64NVME is full-object only. CRC64NVME plus `COMPOSITE` is rejected instead +of canonicalized silently. A cross-vendor replication job carrying one of the +unsupported algorithms will now fail visibly and must be retried with a +supported algorithm. See the [checksum verification design](/blog/design/checksum-verify/). + +## CORS and bucket metadata {#cors-metadata} + +- Per-bucket CORS replication, tombstones, strict XML/wire validation, status, + heal, and recovery are already merged and documented in the CORS design + records. +- Requests without an `Origin` header bypass per-bucket CORS metadata entirely. + Admin, Console, health, and ordinary non-browser traffic no longer pay failed + bucket-metadata reads before authentication. Operational or invalid-document + errors remain fail-closed; a missing/no-config bucket still uses global CORS. +- Invalid CORS accepted by an earlier development build can be repaired through + a valid `PUT ?cors` or `DELETE ?cors`. Do that before changing unrelated + bucket metadata. + +## Site replication and Object Lock {#site-replication} + +- Live, initial-sync, and heal events now put Object Lock XML in + `ObjectLockConfig`. New receivers retain a legacy `Tags` fallback for rolling + upgrades. +- Adopting a pre-existing same-name bucket preserves its full Object Lock + retention and enabled custom versioning rules, including excluded prefixes. + Missing configs are bootstrapped; suspended or invalid versioning is enabled + deliberately. +- Per-site status totals are derived from each site's valid payload rather than + cumulative counters. Bucket policy and quota totals are now accurate; + malformed fields emit bounded diagnostics without suppressing unrelated + bucket statistics. Reported totals may therefore decrease or move to the + correct site after upgrade. + +The wider source-timestamp/tombstone repair for policy, tags, SSE, quota, +versioning, and Object Lock remains staged behind a peer-capability design. +This release must not claim that every inherited metadata type converges under +all delayed/mixed-version event schedules. + +## MCLI and image boundary {#mcli} + +The client source is split into independent review units: + +- read-only `mcli checksum verify`, including reliable non-TTY JSON Lines and + explicit quiet/report/exit semantics; +- strict validation on policy-write commands while historical reads remain + permissive; +- regular PUT semantics for empty `mcli pipe` input, changing its ETag from a + one-part multipart form to the standard empty-object MD5; +- tag-idempotent Release workflow retries. + +The Server image must not promise the audit command until a new immutable MCLI +release exists and `Dockerfile.goreleaser` pins its real amd64/arm64 asset +checksums. The `mc` compatibility alias remains. + +## Security and authorization compatibility {#authorization} + +- Group enable/disable authorization is selected from the requested target + status. A principal with only the enable action can no longer disable a + group, and vice versa. +- New MCLI policy-write commands reject unknown fields, bare ARN statements, + conflicting Resource/NotResource, empty statements, and missing Version for + named policies. Existing stored/session policies remain readable under their + historical compatibility rules. +- Config environment-file discovery preserves named targets and broader valid + environment names. In `silo-pkg`, only exact `env://` and `env+tls://` values + are remote references; ordinary values such as `envreview` remain literal. + +## Release pipeline {#release-pipeline} + +Both MCLI and Server release jobs serialize by resolved tag, verify tag-to-HEAD +identity, reject published/duplicate state, and replace only one unfinalized +Draft from scratch. The Server build lane refuses to replace a Draft carrying +the finalize lane's GPG-derived provenance marker, so retries cannot silently +restore SBOMs or attestations for unsigned RPM bytes. + +Before publication, a controlled Draft/retry exercise must prove one Draft and +one copy of every asset, finalized-Draft refusal, and published-release +immutability. Tags remain immutable and are never moved. + +## Deliberately deferred {#deferred} + +- full source-time/tombstone convergence across all inherited site-replication + metadata types, pending a mixed-version capability channel; +- prefix, delimiter, and pagination support for `ListMultipartUploads`, which + requires redesigning the hashed multipart index and in-memory cache; +- persistence of the five newer checksum algorithms, which requires an on-disk + and rolling-upgrade compatibility design. diff --git a/content/blog/release/silo-next-hardening.zh.md b/content/blog/release/silo-next-hardening.zh.md new file mode 100644 index 00000000..7d250106 --- /dev/null +++ b/content/blog/release/silo-next-hardening.zh.md @@ -0,0 +1,109 @@ +--- +title: "SILO 下一版本:发布前加固说明" +linkTitle: "SILO 下一版加固" +date: 2026-08-29 +lastmod: 2026-08-29 +author: "冯若航" +description: "SILO 20260806 后续版本的兼容性、完整性、站点复制、客户端与发布管线草案。" +tags: [发布, 草稿, 兼容性, Checksum, S3] +weight: 9 +draft: true +url: "/zh/blog/release/silo-next-hardening/" +--- + +> **草稿——目前没有对应的 release tag、软件包或镜像。** 本页记录 +> `RELEASE.2026-08-06T00-00-00Z` 后续版本在发布前验证的代码与兼容边界。 +> 只有源码 PR、远端 CI 与最终候选验收完成后,才会填写最终版本、提交与产物摘要。 + +## 完整性与加密修复 {#integrity} + +- null version 的 metadata-only `CopyObject` 现在按对象层实际写入的字节记录压缩 + metadata。压缩两个方向与 SSE-C 换钥均有覆盖;原故障既可能报 + `s2: corrupt input`,也可能静默返回被截短的对象。 +- 零字节 SSE-C 读取虽然不会创建 decryptor,仍会验证调用方提供的客户密钥。 + GET、CopyObject 与 UploadPartCopy 现在与非空对象一样对错误密钥返回 403, + 同时保持内部无解密、replication、restore、range 与 precondition 顺序。 +- `GetObjectAttributes` 在返回明文对象大小或已完成分段布局前会解封 SSE-C 密钥; + 零字节与非空对象的正确、错误、缺失密钥均有测试。 +- CopyObject 重新加密时使用目标密钥解密 checksum metadata;显式 multipart + checksum type 也不能再缺少对应值而被接受。 + +## Checksum 兼容边界 {#checksum} + +SILO 实现 CRC32、CRC32C、CRC64NVME、SHA1 与 SHA256。请求声明 MD5、SHA512、 +XXHASH64、XXHASH3、XXHASH128 或未知的 `x-amz-checksum-*` 值/trailer 时, +现在明确返回 400 `InvalidArgument`,不再返回 200 却静默丢弃完整性断言。 +该规则覆盖 PutObject、CreateMultipartUpload、UploadPart、CopyObject 与 +UploadPartCopy。 + +CRC64NVME 只允许整对象 checksum;CRC64NVME + `COMPOSITE` 会被拒绝, +不再静默规范化。跨厂商复制若携带未支持算法,将显式失败,必须改用支持算法重试。 +详见 [checksum 校验设计](/zh/blog/design/checksum-verify/)。 + +## CORS 与桶 metadata {#cors-metadata} + +- per-bucket CORS 的复制、tombstone、严格 XML/wire 校验、status、heal 与恢复 + 已合并,并由两篇 CORS 设计文档记录。 +- 没有 `Origin` 的请求会完全跳过 per-bucket CORS metadata。Admin、Console、 + health 与普通非浏览器流量不再在鉴权前重复执行失败的 metadata 读取。 + 运行故障与非法文档仍 fail-closed;不存在或未配置 CORS 的桶继续走全局 CORS。 +- 旧开发版本接受的非法 CORS 可通过合法 `PUT ?cors` 或 `DELETE ?cors` 修复; + 修复前不要修改该桶的其他 metadata。 + +## 站点复制与 Object Lock {#site-replication} + +- live、initial-sync 与 heal 事件统一在 `ObjectLockConfig` 字段携带 Object Lock XML; + 新 receiver 保留对旧 `Tags` 字段的滚动升级兼容。 +- 接管同名既有桶时保留完整 Object Lock retention 与已启用的自定义 versioning + 规则(含 excluded prefixes)。缺失配置会初始化;Suspended 或非法 versioning + 会被明确启用。 +- 每站点 totals 按该站点自己的有效 payload 统计,不再使用累计计数。Bucket policy + 与 quota 统计得到补齐;非法字段输出有界诊断,且不会吞掉同站点的其他桶统计。 + 升级后,统计数可能下降或移动到真正持有配置的站点,这是修正后的结果。 + +Policy、Tags、SSE、Quota、Versioning、Object Lock 的完整 source-time/tombstone +收敛仍需 peer capability 设计后分阶段交付。本版本不得宣称所有继承 metadata +类型在任意延迟事件与混合版本下都能完全收敛。 + +## MCLI 与镜像边界 {#mcli} + +客户端源码拆为四个独立审查单元: + +- 只读 `mcli checksum verify`,含可靠的非 TTY JSON Lines,以及明确的 + quiet/report/退出码语义; +- policy 写入命令严格校验,同时继续宽容读取历史策略; +- 空 `mcli pipe` 改用普通 PUT,其 ETag 从单段 multipart 形式变为标准空对象 MD5; +- 同 tag 可幂等重试的 Release workflow。 + +只有新的不可移动 MCLI release 真实存在,且 `Dockerfile.goreleaser` 钉住其 +amd64/arm64 资产真实摘要后,Server 镜像才可以承诺内置 checksum audit 命令。 +`mc` 兼容别名保持不变。 + +## 安全与鉴权兼容性 {#authorization} + +- Group enable/disable 按目标状态选择鉴权动作;只有 enable 权限的主体不能再 + disable group,反之亦然。 +- 新版 MCLI 的 policy 写入拒绝未知字段、裸 ARN、Resource/NotResource 冲突、 + 空 Statement,以及 named policy 缺失 Version;既有 stored/session policy + 仍按历史兼容规则读取。 +- config environment file 能保留 named target 与更广的合法环境变量名。 + `silo-pkg` 只把精确 `env://` / `env+tls://` 视为远程引用,`envreview` + 等普通值保持字面含义。 + +## 发布管线 {#release-pipeline} + +MCLI 与 Server release job 都按解析后的 tag 串行化、验证 tag→HEAD、拒绝 published +或重复状态,并只整体替换一个尚未 finalize 的 Draft。Server build lane 检测到 +finalize lane 的 GPG provenance marker 后会拒绝替换,重试不会把已签名 RPM 的 +SBOM/attestation 悄悄退回未签名字节版本。 + +发布前仍必须完成受控 Draft/retry 验收:只剩一个 Draft、每个资产只有一份、 +finalized Draft 拒绝重建、published release 不可覆盖。Tag 不可移动,也绝不改写。 + +## 明确延期 {#deferred} + +- 所有继承 site-replication metadata 类型的 source-time/tombstone 完整收敛, + 等待混合版本 capability channel; +- `ListMultipartUploads` 的 prefix/delimiter/pagination 支持,该问题需要重构 + 哈希 multipart 索引与内存缓存; +- 五种新 checksum 算法的持久化支持,该工作需要单独的磁盘格式与滚动升级设计。