feat: support S3 data connections on non-AWS providers using temp credentials

[pwgardipee]

Before submitting

• Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure to update the docs?
• Did you write any new necessary tests?

What does this PR do?

Enables S3 data connections marked available on non-AWS providers to be resolved and accessed directly (bypassing the FUSE mount) from non-AWS Studios (GCP, Nebius, etc.), by minting temporary project-role credentials — mirroring the existing R2 / lightning_storage flow.

Previously _resolve_s3_connections returned a Dir without a data_connection_id, so the s3:// path fell back to ambient AWS credentials (instance profile / shared credentials file). Off AWS there are none, so S3Client._create_client crashed:

AttributeError: 'NoneType' object has no attribute 'access_key'

Changes

• streaming/resolver.py: _resolve_s3_connections now threads data_connection_id through when the connection has available_in_non_aws_providers set (parallel to _resolve_lightning_storage). AWS-only connections leave it unset and keep using ambient credentials.
• streaming/client.py: extracted the shared token-login + temp-bucket-credentials fetch into _login_and_get_temp_bucket_credentials, reused by both R2 and S3. S3Client now mints temporary project-role credentials when a data_connection_id is present (real AWS S3 — no custom endpoint). This covers both the read (S3Downloader) and write (S3FsProvider) paths, since both share the s3:// scheme and S3Client.
• Credential refresh is handled by the existing refetch_interval mechanism.
• Added unit tests for the resolver and client paths.

Note

Requires the lightning_sdk V1AwsDataConnection model to expose available_in_non_aws_providers (added separately in the SDK). Without it the field is dropped on deserialize and the connection continues to use ambient credentials.

PR review

Anyone in the community is free to review the PR once the tests have passed.

Did you have fun?

🙃

Made with Cursor

[gitguardian]

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
5685611	Triggered	Generic High Entropy Secret	5cbe99f	tests/streaming/test_resolver.py	View secret

🛠 Guidelines to remediate hardcoded secrets

1. Understand the implications of revoking this secret by investigating where it is used in your code.
2. Replace and store your secret safely. Learn here the best practices.
3. Revoke and rotate this secret.
4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

• following these best practices for managing and storing secrets including API keys and other credentials
• install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

---

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81%. Comparing base (72b25d8) to head (5cbe99f).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@         Coverage Diff         @@
##           main   #844   +/-   ##
===================================
  Coverage    81%    81%           
===================================
  Files        54     54           
  Lines      7671   7683   +12     
===================================
+ Hits       6198   6210   +12     
  Misses     1473   1473           

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.