S3 API cannot persist large files (5GB safetensors) — uploads transfer data but 404 on HEAD

[karabil]

Summary

The S3-compatible API for network volumes cannot persist large files (~5GB). Small files (≤500MB) work fine. The data transfers successfully but files return 404 on HEAD after upload completes.

Reproduction

1. Create network volume in US-KS-2 (40GB)
2. Download ByteDance-Seed/UI-TARS-1.5-7B model (7 safetensors × ~5GB = 33GB total)
3. Upload via aws s3 cp --recursive or single-file uploads

aws s3 cp model.safetensors \
  s3://tmz3bxcv6a/models/model.safetensors \
  --region us-ks-2 \
  --endpoint-url https://s3api-us-ks-2.runpod.io

4. AWS CLI reports success: Completed 30.9 GiB/30.9 GiB
5. Verify: aws s3 ls s3://tmz3bxcv6a/models/ — only small files listed
6. HEAD: aws s3api head-object --bucket tmz3bxcv6a --key models/model.safetensors → 404 Not Found

Evidence

What works

File size	Result
100MB	✅ Upload + persist
500MB	✅ Upload + persist

What fails

Test	Details
30.9 GB total uploaded	Data transferred but 0/7 safetensors persisted
Multipart chunk sizes tried	Default (8MB), 256MB, 512MB
Methods tried	Recursive upload, serial single-file, s3 cp, s3api
Source locations	macOS (mac-pro-1), Linux (ubuntu-2), both on Bezeq ISP, Israel

Environment

• Network volume: US-KS-2, 40GB
• Endpoint: https://s3api-us-ks-2.runpod.io (behind Cloudflare CDN, 8ms ping)
• AWS CLI: 1.45.18
• S3 credentials: Valid (small files work)
• Local bandwidth: 37 MB/s to Cloudflare, ~100 Mbps upload

Expected behavior

5GB files should persist on the network volume via the S3 API, same as smaller files.

Impact

Cannot seed network volumes with large model weights without spinning up a pod. The S3 API is the recommended way to pre-populate volumes, but it is broken for model-sized files, forcing users to provision GPU pods just to download models to volumes.