Skip to content

fix: pin boto3<1.40 to keep non-AWS S3 uploads working#224

Open
TimPietruskyRunPod wants to merge 1 commit into
mainfrom
fix/pin-boto3-for-non-aws-s3
Open

fix: pin boto3<1.40 to keep non-AWS S3 uploads working#224
TimPietruskyRunPod wants to merge 1 commit into
mainfrom
fix/pin-boto3-for-non-aws-s3

Conversation

@TimPietruskyRunPod
Copy link
Copy Markdown
Contributor

Summary

Closes #156.

boto3 1.40 ships a botocore release that enforces stricter AWS-only auth flows. Uploads to S3-compatible endpoints (Cloudflare R2, Google Cloud Storage) began failing with `SignatureDoesNotMatch`. The reporter on #156 narrowed it down precisely:

  • `boto3==1.35.40` → works against GCS
  • `boto3==1.40.1` → fails

Related upstream: terricain/aioboto3#358

Fix

Pin `boto3<1.40` in the handler runtime install line in `Dockerfile` (boto3 isn't a direct dep — it comes transitively through `runpod`, so we add an explicit constraint).

Test plan

  • Build the PR image via the `Development` workflow (depends on ci: build base image on every pull request #223 being merged so this happens automatically)
  • Deploy a serverless endpoint with the PR image
  • Set `BUCKET_ENDPOINT_URL` to a GCS or R2 bucket and submit a workflow that produces an image
  • Confirm the upload succeeds (response includes `type: s3_url`, no `SignatureDoesNotMatch` in logs)

boto3 1.40 ships a botocore release that enforces stricter AWS-only
auth flows. Uploads to S3-compatible endpoints (Cloudflare R2, Google
Cloud Storage) started failing with SignatureDoesNotMatch.

Pin to <1.40 in the handler dependencies line until upstream runpod
and aioboto3 stabilize against the new botocore.

Refs #156
@TimPietruskyRunPod
Copy link
Copy Markdown
Contributor Author

Validation status

Test Status Notes
Dockerfile diff One-line change: `'boto3<1.40'` added to the uv pip install line after `runpod requests websocket-client`
Image builds successfully Build via PR #223's per-PR pipeline succeeded, image published as `runpod/worker-comfyui:fix-pin-boto3-for-non-aws-s3-base`
boto3 version in the built image is <1.40 ⚠️ pending Pulling the image locally to inspect — will follow up
End-to-end S3 upload against a non-AWS provider (R2/GCS) ⚠️ blocked on credentials I don't have R2/GCS credentials handy. The reporter on #156 (@yxzhao6) isolated this precisely already — boto3==1.35.40 works against GCS, 1.40.1 breaks. If they could verify with the PR image, that would close the loop

Best test would be to ping @yxzhao6 on #156 and ask them to retry with the PR image. I'll do that once #223 merges so they can pull the image without manual workflow_dispatch.

@TimPietruskyRunPod
Copy link
Copy Markdown
Contributor Author

Verified the pin holds in the built image:

```
$ docker run --rm --platform linux/amd64 --entrypoint python \
runpod/worker-comfyui:fix-pin-boto3-for-non-aws-s3-base \
-c "import boto3,botocore,s3transfer; print('boto3:', boto3.version); ..."
boto3: 1.39.17
botocore: 1.39.17
s3transfer: 0.13.1
```

`boto3==1.39.17` is comfortably under the 1.40 wall. The pin works as intended at install time.

All that's left is the live-against-GCS-or-R2 confirmation, which I can't do without those provider credentials. Pinging @yxzhao6 on #156 once #223 merges so they can pull the image and verify against their setup.

LGTM from my side.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: The current version of s3 integration probably is not working with 3rd-party s3 compatible services (like GCS CloudFlare R2)

2 participants