Add Google Cloud Storage reader support - #6467
Conversation
DALI can read from S3 but not from GCS, so datasets on Google Cloud have to be staged elsewhere first. Add a gs:// backend mirroring the existing S3 layer: FileStream::Open and discover_files gain a gs:// branch, guarded by a new BUILD_GCS/GCS_ENABLED option that auto-disables when google-cloud-cpp is absent. Three details differ from the S3 path. google::cloud::storage::Client is not documented as safe for concurrent use of a single instance, while copies of it are and share the connection pool, so GCSClientManager hands out copies and GCSFileStream owns one instead of borrowing a shared pointer. There is no InitAPI/ShutdownAPI equivalent to serialize, as the library initializes libcurl lazily. Reads land directly in the caller's buffer via std::istream::read, so no preallocated stream buffer is needed, and ReadRange is right-open unlike the inclusive HTTP byte range. Download checksum validation is off by default: DALI only issues ranged reads and GCS reports checksums for whole objects, so a per-read CRC32C cannot be validated end-to-end. DALI_GCS_VERIFY_CHECKSUMS re-enables it, DALI_GCS_ENDPOINT_URL overrides the endpoint, and DALI_GCS_ANONYMOUS selects insecure credentials for public buckets and emulators. Signed-off-by: Michal Zientkiewicz <michalz@nvidia.com>
|
CI MESSAGE: [65633875]: BUILD STARTED |
|
CI MESSAGE: [65634727]: BUILD STARTED |
|
CI MESSAGE: [65640124]: BUILD STARTED |
|
CI MESSAGE: [65634727]: BUILD FAILED |
|
CI MESSAGE: [65640124]: BUILD PASSED |
sylvesterkaczmarek
left a comment
sylvesterkaczmarek
left a comment
There was a problem hiding this comment.
I think this needs to handle GCS directory-marker objects explicitly.
ListObjects can return zero-byte objects whose names end in /, which are commonly created as folder placeholders. For a marker such as dataset/class/, the relative path may contain only the class component, but the code currently asserts path_elems >= 2 and then immediately accesses both the first and second components.
In debug builds that can abort; with assertions disabled the iterator increment can run past the end. Could entries with fewer than two components, particularly trailing-slash directory markers, simply be skipped before extracting subdir and fname?
Signed-off-by: Michal Zientkiewicz <michalz@nvidia.com>
|
CI MESSAGE: [65813390]: BUILD STARTED |
|
CI MESSAGE: [65813390]: BUILD FAILED |
|
CI MESSAGE: [65813390]: BUILD PASSED |
3 participants
Category:
New feature (non-breaking change which adds functionality)
Description:
DALI can read from S3 but not from GCS, so datasets on Google Cloud have to be staged elsewhere first. Add a gs:// backend mirroring the existing S3 layer: FileStream::Open and discover_files gain a gs:// branch, guarded by a new BUILD_GCS/GCS_ENABLED option that auto-disables when google-cloud-cpp is absent.
Three details differ from the S3 path. google::cloud::storage::Client is not documented as safe for concurrent use of a single instance, while copies of it are and share the connection pool, so GCSClientManager hands out copies and GCSFileStream owns one instead of borrowing a shared pointer. There is no InitAPI/ShutdownAPI equivalent to serialize, as the library initializes libcurl lazily. Reads land directly in the caller's buffer via std::istream::read, so no preallocated stream buffer is needed, and ReadRange is right-open unlike the inclusive HTTP byte range.
Download checksum validation is off by default: DALI only issues ranged reads and GCS reports checksums for whole objects, so a per-read CRC32C cannot be validated end-to-end. DALI_GCS_VERIFY_CHECKSUMS re-enables it, DALI_GCS_ENDPOINT_URL overrides the endpoint, and DALI_GCS_ANONYMOUS selects insecure credentials for public buckets and emulators.
Additional information:
Affected modules and functionalities:
Key points relevant for the review:
Tests:
Checklist
Documentation
DALI team only
Requirements
REQ IDs: N/A
JIRA TASK: N/A