fix(upload): 录像上传可靠性 P1/P2 修复(崩溃卡死/孤儿文件/盘满/补传耗尽)#4
Merged
Conversation
承接 2026-05-22 上传可靠性 review。3 个 Phase: - Phase 1: P1-a Uploading 状态崩溃卡死 + P1-b 上传成功 DB 入库失败孤儿文件 - Phase 2: P2-b pending 目录磁盘水位保护 - Phase 3: P2-a 补传次数耗尽告警 + 人工介入 API 含 4 项实现前调研结论。
P1-a Uploading 状态崩溃卡死: - UploadTask 加 UploadStartedAt 字段 - MarkUploading 改原子抢占(WHERE status=Failed 条件更新), 返回 claimed - 新增 ReclaimStaleUploading: 把卡 Uploading 超 35min 的任务扫回 Failed(不计 retry_count), 进程崩溃残留可重新补传 - UploadRetryScheduler.Tick 每轮先回收; retryUpload 用 CAS 防并发重传 P1-b 上传成功但 DB 入库失败 -> 孤儿文件: - WriteTailDeferred 闭包改返回 error(record_streams 写失败=孤儿) - writeTrailerTask 捕获该 error, reportOrphan 打日志 + 告警入 alarm_info - 新增 RaiseUploadAlarm 告警 helper(带去重) upload_task_test.go 覆盖 ReclaimStaleUploading / MarkUploading 抢占 / 回收后可补传, 3/3 PASS。
MinIO 长期故障时 pending 文件堆积会撑爆本地盘、拖垮新录制。加水位保护: - UploadConfig 加 PendingMaxSizeMB / PendingMaxFiles / PendingDiskMinFreeMB (默认 0=不限, 向后兼容); 接入 ServerConfig.Upload, InitUploadManager 从配置读取(原为硬编码) - MoveToPendingDir 入口做水位检查, 超阈值返回 ErrPendingDirFull —— 宁可 丢一个录像也不让磁盘写满拖垮全盘 - GetDiskFreeBytes 分平台实现(statfs_unix / statfs_windows, Windows 降级) - record.go recoverToPending / recoverFastPathFailure 识别 ErrPendingDirFull 触发 AlarmDiskSpaceFull 告警 - UploadRetryScheduler.Tick 加 pending 水位巡检, 达阈值 80% 告警 upload_manager_test.go 覆盖用量统计 / 水位拒绝 / 无限制兼容, 4/4 PASS。
补传 retry_count 耗尽(达 MaxRetries)后系统永久放弃, 原本无告警、无介入: - retryUpload 检测耗尽 -> RaiseUploadAlarm 写 AlarmStorageException 告警 - 新增 QueryExhaustedUploads: 查耗尽任务(retry_count >= max_retries) - 新增 ResetUploadForRetry: 重置任务重试状态, 重新拉入补传循环 - mp4 plugin 加运维 HTTP 端点(经 RegisterHandler): GET /mp4/api/upload/exhausted 列出耗尽任务 POST /mp4/api/upload/retry?id=N 一键重新拉起 upload_task_test.go 加 QueryExhaustedUploads / ResetUploadForRetry 单测, 全 5 例 PASS。 Task 3.4(孤儿 RecordStream 自动补偿队列)未做, plan 已标注修订: 孤儿场景罕见 + Phase 1 已有感知告警, 完整自动补偿成本超 plan 设想, 留作后续专项。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
承接 2026-05-22 上传可靠性 code review,修复 4 个 P1/P2 级缺陷。分 3 个 Phase,每个 Phase 独立验证。P0(
pending_uploads/m7s.db不持久化)已通过 130 docker-compose bind mount 解决,不在本 PR。Phase 1 — P1-a Uploading 崩溃卡死 + P1-b 孤儿文件
UploadTask加UploadStartedAt;MarkUploading改原子抢占(WHERE status=Failed条件更新)防并发重传ReclaimStaleUploading:卡Uploading超 35min 的任务扫回Failed(不计 retry_count),进程崩溃残留可重新补传;UploadRetryScheduler.Tick每轮回收WriteTailDeferred闭包改返回error(record_streams写失败=孤儿);writeTrailerTask.reportOrphan错误日志 + 告警Phase 2 — P2-b pending 目录磁盘水位保护
UploadConfig加PendingMaxSizeMB/PendingMaxFiles/PendingDiskMinFreeMB(默认 0=不限,向后兼容),接入ServerConfig.UploadMoveToPendingDir入口水位检查,超阈值返回ErrPendingDirFull—— 宁可丢一个录像也不让磁盘写满拖垮全盘GetDiskFreeBytes分平台(statfs_unix/statfs_windows);Tick加 pending 水位巡检告警Phase 3 — P2-a 补传耗尽告警 + 人工介入
retry_count耗尽 →AlarmStorageException告警QueryExhaustedUploads/ResetUploadForRetry;mp4 plugin 加运维端点GET /mp4/api/upload/exhausted、POST /mp4/api/upload/retry?id=N新增告警 helper
RaiseUploadAlarm(写alarm_info表,带去重)。未含 / 待后续
RecordStream自动补偿队列)未做:孤儿场景罕见 + Phase 1 已有感知告警,完整自动补偿需改WriteTailDeferred设计,plan 已标注。kill -9验证回收)待部署新镜像后跑。Test plan
upload_task_test.go5 例:ReclaimStaleUploading/MarkUploading并发抢占 / 回收后可补传 /QueryExhaustedUploads/ResetUploadForRetryupload_manager_test.go4 例:pending 用量统计 / 水位拒绝ErrPendingDirFull/ 无限制兼容 /GetDiskFreeByteskill -9→ 重启 →reclaimed stale uploading→ 文件补传成功