feat(mp4,storage): trailer 零重写阶段 A (fallocate INSERT_RANGE 原地插 moov)#2
Merged
Conversation
* 'develop' of https://github.com/eanfs/monibuca: with plan e2e test fix(storage): trailerWriteBytesPerSec 改 atomic + 补注释 (第二轮评审) fix(storage,mp4): 修评审发现的 trailer 临时文件失败丢数据缺陷 docs(storage): TrailerWriteRateMBps 配置项示例注释 feat(mp4): trailer 临时文件写入接入 storage 限速器 feat(storage): 加 trailer 写盘限速器 + TrailerWriteRateMBps 配置 (默认 0=不限速) feat(mp4): writeTrailerTask 走 TempFileFinalizer 快路径, 消除 trailer 全量回拷 feat(storage/s3,oss,cos): 三后端实现 FinalizeFromTemp, 直接以现成文件为上传源 feat(storage/local): LocalFile 实现 FinalizeFromTemp (rename, 跨设备回退复制) feat(storage): 加 TempFileFinalizer 接口, 为 trailer 重写消除回拷做准备 docs(plan): MP4 trailer 重写 磁盘 IO 削减 实施计划
为 MP4 trailer 零重写优化 (2026-05-21 plan 阶段 A) 做准备: - rangeinsert.go: RangeInserter 接口 + ErrRangeInsertUnsupported 哨兵错误 - fallocate_linux.go: InsertRange/CollapseRange 封装 unix.Fallocate - fallocate_other.go: 非 Linux stub, 恒返回 unsupported - fallocate_test.go: INSERT/COLLAPSE 行为单测 (//go:build linux) INSERT_RANGE 真实行为依赖 ext4/xfs, darwin 开发环境会因 build tag 跳过 fallocate_test, 留待阶段 A 验收在 Linux 环境跑。
LocalFile/S3File/OSSFile/COSFile 各加 LocalFd() *os.File, 暴露底层 本地文件句柄 (对象存储后端为上传前的本地暂存文件), 供 MP4 trailer 用 fallocate 原地插 moov。各加编译期断言 var _ RangeInserter。
writeTrailerTask.Run() 加 fast path 分派: progressive MP4 优先用 fallocate(INSERT_RANGE) 在文件头就地撑开 moov 空间, mdat 数据零移动, trailer 磁盘写从 O(整个 mdat) 降到 O(moov)。 - runInsertRangeFastPath: 判定标准布局 -> 算 insertLen(4K 对齐) -> shiftSampleOffsets 校正 chunk offset -> INSERT_RANGE -> writeMoovHead 写 [ftyp][moov][free] -> 截掉 Start() 的尾部 moov 兜底副本 -> Close - writeMoovHead: 写头部 + 把暴露的旧 [ftyp][free] 覆盖为 free box - 失败处理: INSERT 后写头失败 -> CollapseRange 撤销, 退回 moov-在-尾 的可播文件; Close/上传失败 -> 登记 pending 补传 - 文件系统不支持 INSERT_RANGE / fMP4 / 非标准布局 -> 回退原全量重写路径 fast path 真实生效依赖 Linux + ext4/xfs, 端到端验证(录制->ffprobe moov-first)归阶段 A 验收 (Task A4, 130 环境)。
31 路 x 15min 真实录制验收: 31/31 走 INSERT_RANGE fast path, trailer 写入从 ≈文件大小 降到 ≈moov 大小 (cam1 48MB->100KB, -99.8%), 产物标准 moov-first MP4 (ffprobe 验证 ftyp->moov->free->mdat), 0 上传失败 / 0 RequestTimeout / record 全入库。
进程级 /proc/pid/io 精确采集分离 trailer 写盘与上传落盘: - trailer 阶段 monibuca 写盘 46MB (31 路 moov), 对比 fallback 全量 重写 ≈13GB, 降约 99.6% - 整盘 iostat 峰值 567MB/s 为 MinIO 接收上传落盘, 与 trailer 无关 - 31/31 fast path, 0 异常/超时/失败, 31 路入库, ffprobe moov-first
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
2026-05-21-trailer-zero-rewriteplan 阶段 A:用fallocate(FALLOC_FL_INSERT_RANGE)在 MP4 文件头就地为 moov 撑开空间,mdat数据零移动,把 record stop 时的 trailer 重写从O(整个文件)降到O(moov)。pkg/storage的RangeInserter接口 +InsertRange/CollapseRange(fallocate封装,Linux 实现 + 非 Linux stub)LocalFile/S3File/OSSFile/COSFile四后端实现RangeInserter,暴露底层本地文件句柄writeTrailerTask.Run()加 fast path 分派:标准 progressive MP4 走INSERT_RANGE原地插 moov;文件系统不支持 / fMP4 / 非标准布局时回退原有的全量重写路径(零删除保留)验证(130 UAT,镜像
v5.2.5.2605211340)pkg/storage单测 PASS;130 真 ext4 上INSERT_RANGE/COLLAPSE_RANGEsyscall 实证 PASSffprobe产物 box 序ftyp→moov→free→free→mdat/proc/pid/io采集:trailer 阶段 monibuca 写盘 46 MB(31 路 moov),对比 fallback 全量重写 ≈13 GB,降约 99.6%;0 上传失败 / 0RequestTimeout/ 31 路全入库Test plan
pkg/storage单测(darwin)INSERT_RANGE/COLLAPSEsyscall 实证(130 ext4)详见
docs/superpowers/plans/2026-05-21-trailer-zero-rewrite.md(plan)及*-stageA-report.md/*-30min-io-report.md(验收报告)。