Skip to content

feat(uploads): bind presigned PUT URLs to the file row (x-amz-meta-upload-id) and return uploadHeaders - #1983

Merged
pyramation merged 3 commits into
mainfrom
feat/upload-id-bound-presigned-put
Nov 11, 2026
Merged

pyramation merged 3 commits into
mainfrom
feat/upload-id-bound-presigned-put

Conversation

@pyramation

Copy link
Copy Markdown
Contributor

Summary

#1981 binds a presigned PUT to the bytes (x-amz-checksum-sha256). It does not bind it to the upload. A content-addressed key can already hold matching bytes from another upload: either a deleted row whose object is waiting for GC, or a replaced row whose old URL is still valid. If confirm only checks that an object with the right size and type is there, a caller who never PUT can have those bytes adopted as their own file (constructive-planning#2477).

This PR signs the file row's id into the URL, so the stored object records which row's upload wrote it:

generatePresignedPutUrl(s3, key, contentType, size, expiry, contentHash, fileId)
// PutObjectCommand({ ..., ChecksumSHA256, Metadata: { 'upload-id': fileId } })
// unhoistableHeaders: { x-amz-checksum-sha256, x-amz-meta-upload-id }
  • A PUT without x-amz-meta-upload-id, or with another row's id, fails the signature (403). The header is signed and not hoisted into the query string.
  • headObject now also returns uploadId (from Metadata['upload-id']). The constructive-db confirm worker will compare it with row.id and treat a mismatch as a missing object (constructive-db#4333).
  • uploadHeaders: the requestUploadUrl/upload<X>File payload and the bulk file payload now return the exact headers the PUT must send. They come from the new presignedPutHeaders(contentType, contentHash, uploadId), the same inputs the URL was signed with: { 'Content-Type', 'x-amz-checksum-sha256', 'x-amz-meta-upload-id' }. Clients forward them as-is, so adding the next signed header won't require a client change. The field is null on dedup.
  • Clients:
    • upload-client: uploadFile (both the fetch and XHR paths) forwards uploadHeaders. putToPresignedUrl(url, body, type, { headers }) sends them as given, and storageTransport.putObject(..., headers) passes them through.
    • graphql-query's storage client selects uploadHeaders and hands it to the transport.
  • Bucket CORS (buildUploadCorsRules / buildPrivateCorsRules) allows x-amz-meta-upload-id, so browser PUTs pass preflight.

Checked against RustFS before writing this:

  • A URL not signed over the header accepts an extra x-amz-meta-upload-id (200) and stores it as metadata. Clients can therefore start sending it before this ships, the same way dashboard#321 and desktop#269 shipped ahead of feat(uploads): bind presigned PUT URLs to the file's SHA-256 checksum #1981.
  • A URL signed over the header refuses a PUT that omits it (missing signed header) or carries a different id (SignatureDoesNotMatch).

Tests (RustFS, s3-signer.integration.test.ts): the header is signed and not in the query string; the issued headers are accepted and HEAD returns uploadId; a PUT missing the header, or with another id, gets 403 and nothing is stored.

Link to Devin session: https://app.devin.ai/sessions/2429c26551c748bf95c229c5b35a7ac4
Open in Devin Desktop: https://app.devin.ai/desktop/session/2429c26551c748bf95c229c5b35a7ac4?variant=devin
Requested by: @pyramation

@devin-ai-integration

Copy link
Copy Markdown
Contributor

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@luxor-code-review

luxor-code-review Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Review complete. 🟠 1 high · 🟡 2 medium

💬 Inline comments (3)

  • 🟠 Client queries hard-code uploadHeaders, failing against schemas lacking it — queries.ts:31
  • 🟡 Presigned PUT now signed over x-amz-meta-upload-id, breaking old clients — s3-signer.ts:104
  • 🟡 Existing buckets keep CORS rules without x-amz-meta-upload-id — cors.ts:42

The change makes row-bound uploads: the S3 signer signs x-amz-meta-upload-id as an unhoistable header naming the file row, the GraphQL upload payload returns the exact headers the client must send, and the upload clients (fetch and XHR paths) forward them verbatim instead of rebuilding locally, with a local-rebuild fallback for older servers. CORS allowlists in the bucket provisioner are extended for the new header. Overall the implementation is internally consistent — header construction, signature coverage, and both client paths agree — but deployment-compat gaps remain around older clients, old servers, and existing buckets.

Files Change
graphile/graphile-presigned-url-plugin Signs x-amz-meta-upload-id as an unhoistable header, returns uploadHeaders in the upload payload, and surfaces row metadata from HEAD
graphql/query, packages/upload-client Clients select and forward server-issued uploadHeaders across fetch/XHR/transport paths, with a local fallback when absent
packages/bucket-provisioner Adds x-amz-meta-upload-id to the upload and private CORS rule allowlists

Reviewed commit: 7a879d3

@blacksmith-sh

This comment has been minimized.

@luxor-code-review luxor-code-review Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR binds presigned S3 PUTs to their file row via an x-amz-meta-upload-id metadata header and switches clients to forward server-issued uploadHeaders as-is.

Key findings

  • 🟠 Client queries hard-code uploadHeaders, failing against schemas lacking it — queries.ts:31
  • 🟡 Presigned PUT now signed over x-amz-meta-upload-id, breaking old clients — s3-signer.ts:104
  • 🟡 Existing buckets keep CORS rules without x-amz-meta-upload-id — cors.ts:42

filename: $filename
) {
uploadUrl
uploadHeaders

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 bug · high

Client queries hard-code uploadHeaders, failing against schemas lacking it

buildRequestUploadUrlQuery now unconditionally selects uploadHeaders (packages/upload-client/src/queries.ts:31), but the server only adds the field to the payload type when a JSON scalar exists — when it is absent the payload "simply omits the field" (graphile/graphile-presigned-url-plugin/src/plugin.ts:215). Against any server whose schema lacks the field — older deployed plugin versions, or a server without a JSON scalar — GraphQL validation rejects the whole mutation and every upload in the client errors before any PUT is attempted, so the locally computed header fallback is unreachable. The same hard-coded selection exists in the _meta-driven document builder: UPLOAD_PAYLOAD_FIELDS in graphql/query/src/storage/document.ts at line 15.

📋 Prompt for AI Agents

In packages/upload-client/src/queries.ts around line 31 (and graphql/query/src/storage/document.ts line 15 for UPLOAD_PAYLOAD_FIELDS), make the uploadHeaders selection resilient: keep the current query as the default and add a legacy variant (e.g. an options flag or a second exported constant) that omits the uploadHeaders field. In packages/upload-client/src/upload.ts, fall back to the legacy query when the GraphQL executor returns a validation error mentioning uploadHeaders (or via a caller-provided flag), since the server plugin omits this field from the schema whenever no JSON scalar exists or when an older plugin version is deployed. Then use the local presignedPutHeaders fallback path for the headers.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Accepted as intended, with no change. Before launch, the server and client packages ship together in one publish, so the client gets no feature detection and no fallback for a schema that lacks uploadHeaders.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The user's response is a "no fix" claim — they're explicitly stating they accepted the finding as intended and made no change, citing that server and client ship together. However, the diff still shows uploadHeaders unconditionally selected with no fallback or legacy variant added. The compatibility risk remains: any deployment where the client and server are not perfectly in sync (e.g., staged rollouts, independent deployments, or environments running an older plugin version) will see GraphQL validation errors for every upload. The "ships together" assumption does not hold across all deployment scenarios and does not address the schema-conditional nature of the field on the server side. The finding remains unresolved.

});
const unhoistableHeaders = new Set([
...(contentHash ? [CHECKSUM_SHA256_HEADER] : []),
...(uploadId ? [UPLOAD_ID_HEADER] : []),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 compatibility · medium

Presigned PUT now signed over x-amz-meta-upload-id, breaking old clients

The server now binds the presigned PUT signature to x-amz-meta-upload-id: presignedPutHeaders adds the header when uploadId is given and generatePresignedPutUrl marks it as unhoistable so S3 requires it to match the signature (graphile/graphile-presigned-url-plugin/src/plugin.ts:858). Clients built against the previous release — including published versions of @constructive-io/upload-client and any hand-rolled uploader — send only Content-Type and x-amz-checksum-sha256, so their PUTs now fail with SignatureDoesNotMatch. There is no opt-out or version negotiation, so every upload from an un-upgraded client breaks until it is rebuilt against the new contract.

📋 Prompt for AI Agents

In graphile/graphile-presigned-url-plugin/src/s3-signer.ts (unhoistableHeaders construction, around lines 92-111) and graphile/graphile-presigned-url-plugin/src/plugin.ts (the generatePresignedPutUrl call passing fileId, around line 858), make the x-amz-meta-upload-id header binding opt-in: add a flag (e.g. bindUploadIdHeader) to the plugin options/storage config that defaults to false, thread it through presignedPutHeaders and generatePresignedPutUrl, and only include the header/metadata when it is enabled — otherwise clients built against the previous signing contract fail with SignatureDoesNotMatch. Keep the confirm-worker path tolerant of uploads without the metadata when binding is disabled.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is expected, and it is what the change is for: a PUT that is not bound to its row is refused. Our clients already send the header: dashboard#322 and constructive-desktop#272 are merged, and their PUTs carry x-amz-meta-upload-id: <fileId>. A URL that is not signed over the header accepts it, so they shipped ahead of this PR.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The diff does not show the fix described in the original finding. The full s3-signer.ts and plugin.ts diffs confirm that uploadId (the fileId) is unconditionally passed to generatePresignedPutUrl and signed into the URL as x-amz-meta-upload-id with no opt-in flag — it is always on.

The user's claim that dashboard#322 and constructive-desktop#272 are already merged and shipping the header may well be true, but it does not address the compatibility risk for any other client (hand-rolled uploaders, third-party integrations, older mobile builds still in the field) that was not updated before this PR merges. The original finding is about the absence of an opt-out mechanism, not about whether specific known clients were updated.

The recommended fix — a bindUploadIdHeader flag defaulting to false so the new signing contract is opt-in — is not present anywhere in the diff. Until that flag (or equivalent version negotiation) is added, the finding stands.

'Content-MD5',
'x-amz-content-sha256',
'x-amz-checksum-sha256',
'x-amz-meta-upload-id',

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 bug · medium

Existing buckets keep CORS rules without x-amz-meta-upload-id

buildUploadCorsRules and buildPrivateCorsRules now allow x-amz-meta-upload-id (packages/bucket-provisioner/src/cors.ts:42), but this only reaches buckets provisioned after deploy: updateCors() in provisioner.ts has no production callers and the reconciler is a no-op for buckets already reconciled. Existing buckets keep their old rule sets, so a browser PUT that sends the new upload-id header fails the CORS preflight against them. Re-provisioning or a migration step is needed for existing buckets to pick up the header.

📋 Prompt for AI Agents

In packages/bucket-provisioner, make the new x-amz-meta-upload-id allowedHeader (src/cors.ts lines 42 and 87) reach already-provisioned buckets. Currently CORS is only applied on fresh provisioning: updateCors() in src/provisioner.ts has no production callers and the storage reconciler skips already-reconciled buckets. Implement a convergence mechanism (e.g. store a fingerprint/hash of the built CORS rules per bucket and have the reconciler call setCors/updateCors when the stored fingerprint differs from the current buildUploadCorsRules/buildPrivateCorsRules output), or document a rollout step that calls updateCors for every existing bucket, since a browser PUT carrying x-amz-meta-upload-id otherwise fails the preflight against buckets provisioned before this change.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no backfill, by design. Databases and buckets get reprovisioned before launch, and #1982 (merged) gives every logical bucket a fresh physical name anyway. Each bucket is therefore created with the current buildUploadCorsRules/buildPrivateCorsRules, which include x-amz-meta-upload-id.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The claim is that buckets are always reprovisioned fresh (via #1982 giving each logical bucket a new physical name), so every bucket is created after this deploy with the updated CORS rules that include 'x-amz-meta-upload-id'. This would indeed resolve the concern if true, but I cannot verify the reprovisioning behavior from the current diff alone. The files changed in this PR don't include provisioner.ts or the reconciler, so I can't confirm whether the fresh-bucket guarantee holds.

@pyramation
pyramation merged commit 2404e15 into main Nov 11, 2026
19 checks passed
@pyramation
pyramation deleted the feat/upload-id-bound-presigned-put branch November 11, 2026 03:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant