feat(storage): implement appendable upload connector

[vsharonlynn]

This PR comes after PR #5932.

[gemini-code-assist]

Code Review

This pull request introduces the appendable_object_writer module and supporting bidi_write infrastructure to enable bidirectional streaming writes for Google Cloud Storage. Key feedback includes avoiding .expect() on parameter conversions to prevent panics, simplifying complex string-folding logic, removing an unnecessary clone of the request object, and ensuring the background worker fails loudly with an explicit error if the stream closes unexpectedly while flushes are pending.

[codecov]

Codecov Report

❌ Patch coverage is 97.05506% with 23 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.75%. Comparing base (affbeb2) to head (8fcbb33).
⚠️ Report is 25 commits behind head on main.

Files with missing lines	Patch %	Lines
src/storage/src/storage/bidi_write/connector.rs	97.60%	12 Missing ⚠️
src/storage/src/storage/bidi_write/state.rs	97.23%	7 Missing ⚠️
src/storage/src/storage/bidi_write/mocks.rs	85.71%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5956      +/-   ##
==========================================
+ Coverage   97.73%   97.75%   +0.02%     
==========================================
  Files         244      247       +3     
  Lines       61096    61799     +703     
==========================================
+ Hits        59710    60412     +702     
- Misses       1386     1387       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[joshuatants]

We're gone through this extensively offline.

[coryan]

This is still too large. Aim for at most 200 lines of new code, which probably means 400 to 600 lines per PR. 1600 lines is simply too large.

[vsharonlynn]

This is still too large. Aim for at most 200 lines of new code, which probably means 400 to 600 lines per PR. 1600 lines is simply too large.

Ok. I am going to split this PR into smaller PRs. I'll update once done. Thank you and sorry about this!

[vsharonlynn]

This is still too large. Aim for at most 200 lines of new code, which probably means 400 to 600 lines per PR. 1600 lines is simply too large.

Ok. I am going to split this PR into smaller PRs. I'll update once done. Thank you and sorry about this!

I split this PR into three PRs:

• This current PR is now for connector-related changes only, which includes some changes in redirect and mocks for testing.
• feat(storage): implement appendable upload background worker #6001 for background worker. Still in draft, will wait for this PR to be merged before rebasing and sending it out.
• feat(storage): implement appendable upload transport #6002 for transport. Still in draft, will wait for 6001 to be merged before rebasing and sending it out.

Thank you.

[coryan]

Why a Box<WriteObjectSpec> ?

[vsharonlynn]

Without Box it fails the clippy::large_enum_variant check.

AppendObjectSpecState can either be Append or Write.
Write contains WriteObjectSpec. WriteObjectSpec is large, contains Option<Object>. And Object is a large struct containing many fields.
In Rust, an enum takes a memory of the largest variant, so if WriteObjectSpec is not boxed, AppendObjectSpecState with Append variant will take up memory that fits WriteObjectSpec which is unnecessarily large.

Should I do this way with Box; or without Box but will need #[allow(clippy::large_enum_variant)] macro?

Thanks!

[coryan]

LGTM.

For future reference, avoid PRs with 1,000 lines of code (such as this one). It makes it hard to review them thoroughly. This may contradict what other reviewers want (a full picture of all the changes). Consider a draft PR with the full thing and smaller PRs that are part of the changes.

[coryan]

optional: the code coverage says this case is not tested. Seems like an important behavior to test