Transfer requests are neither queued nor deduplicated — N identical downloads write N copies

[jordlee]

Environment: ILCE-7M5 (CFCA6014E092) · USB · Camera Remote SDK V2.02.00 · darwin-arm64 (macOS 15) · server 3.0.0 · remote-transfer

Summary

The server neither queues nor deduplicates RemoteTransfer requests. Two independent consequences:

1. A request issued while another transfer is in flight is rejected outright with 0x8D03 rather than being queued. There is no way for a client to submit work and let the server sequence it, and no exposed "transfer in progress" state it could poll to avoid the collision.
2. Repeated identical requests each create a new file rather than being coalesced or rejected.

Observed

Four successive POST .../131201/1/download calls for the same content_id/file_id, spaced 2/5/10/20 s apart, all returned 202 and all wrote separate copies:

DSC09991.JPG    = 10438049
DSC09991(1).JPG = 10438049
DSC09991(2).JPG = 10438049
DSC09991(3).JPG = 10438049

Same bytes, four times, ~40 MB of duplicate I/O over USB for one logical request.

Why it bites

A client that cannot tell whether its 202 will complete — see #44, where a transfer can be dropped with no terminal event — has an obvious and reasonable strategy: retry. Doing so silently multiplies files on disk and transfer time on the wire, and each retry that lands while the previous attempt is still running is rejected with a message that reads as a hard failure (filed separately).

Suggested

• Serialize transfers in a queue, so a request submitted during an in-flight transfer is accepted and run in turn rather than rejected.
• Coalesce or reject an identical in-flight (cameraId, contentId, fileId, savePath) rather than starting a second copy.
• Expose the in-flight transfer (and any queue depth) so clients can make informed decisions instead of guessing.

Found while running the cross-platform transfer sweep in #40.

[jordlee]

Still open after #54 — plus a second collision case found during regression testing

#54 fixes how the rejection is reported (#49: 0x8D03 is now a retryable 409 rather than a hard 400), but it does not queue or deduplicate anything. Everything in this issue still stands.

New: the collision is not limited to identical requests

The original report covers N identical download calls producing N copies. Regression testing surfaced a second, arguably worse case — the full file, thumbnail and screennail of the same content collide with each other:

POST .../files/131305/1/download     -> DSC00081.JPG      4,173,653 bytes
POST .../files/131305/1/thumbnail    -> DSC00081(1).JPG     108,415 bytes
POST .../files/131305/1/screennail   -> DSC00081(2).JPG     155,313 bytes

Three genuinely different artifacts, named only by arrival order. Nothing in the filename says which is the full image and which is the preview; the only distinguishing signal is file size, and that is a heuristic, not a contract. A client that pulls a thumbnail for triage and the full file on demand — the exact flow the thumbnail endpoint exists for — cannot tell its own outputs apart on disk.

This also limits what #54 could fix in the events

#54 adds contentId / fileId to transferProgress (#47), which resolves "which file finished". It does not resolve "which artifact finished", because all three requests above emit an identical identity:

contentId=131305 fileId=1  savedPath=.../DSC00081.JPG
contentId=131305 fileId=1  savedPath=.../DSC00081(1).JPG
contentId=131305 fileId=1  savedPath=.../DSC00081(2).JPG

The spec's documented payload has no field for the transfer type, so #54 matched the spec rather than exceeding it — but the spec is incomplete here. Adding a type (file / thumbnail / screennail) to transferProgress would close it, and is worth doing alongside whatever naming fix lands here.

Suggested, updated

• Serialize transfers in a queue so a request arriving during an in-flight transfer is accepted and run in turn, rather than rejected with a 409 the caller has to poll around.
• Coalesce or reject an identical in-flight (cameraId, contentId, fileId, type, savePath) instead of starting a second copy.
• Disambiguate destination filenames by artifact type (e.g. a _thumb / _screen suffix) rather than by an arrival-order counter.
• Add type to transferProgress so events are mappable to artifacts.
• Expose the in-flight transfer and any queue depth so clients can avoid the collision instead of discovering it.

Environment for the new observation: ILCE-7M5 (CFCA6014E092) · USB · SDK V2.02.00 · darwin-arm64 · server 3.0.0 · remote-transfer.