Skip to content

feat(webhooks): verifyAndParse* API for compressed payloads (CHA-3071)#255

Open
nijeesh-stream wants to merge 10 commits intomainfrom
nijeeshjoshy/cha-3071-compress-webhook-payloads
Open

feat(webhooks): verifyAndParse* API for compressed payloads (CHA-3071)#255
nijeesh-stream wants to merge 10 commits intomainfrom
nijeeshjoshy/cha-3071-compress-webhook-payloads

Conversation

@nijeesh-stream
Copy link
Copy Markdown
Contributor

@nijeesh-stream nijeesh-stream commented May 6, 2026

Summary

Adds first-class support for gzip-compressed webhook payloads (HTTP webhooks, SQS, SNS) and exposes a stable verifyAndParse* API that mirrors the cross-SDK contract published in Webhooks Overview.

New public API (io.getstream.chat.java.models.App)

Primitives:

  • gunzipPayload(byte[]) — gzip-magic-byte detection, no-op when not compressed
  • decodeSqsPayload(String) — base64 decode then gunzip-if-magic
  • decodeSnsPayload(notificationBody) — JSON-parse the SNS HTTP notification envelope, extract the inner Message, then run the SQS pipeline. Falls through to a pre-extracted Message string when the input is not a JSON envelope
  • verifySignature(byte[], String, String) — HMAC-SHA256 over the uncompressed body, with a constant-time comparison (matters for the HTTP webhook path where the X-Signature header is exposed publicly; SQS / SNS deliveries arrive over AWS-internal transports where timing-attack resistance is not strictly required)
  • parseEvent(byte[]) — JSON → typed Event via Jackson

Composites (return a typed Event):

  • verifyAndParseWebhook(byte[], String, String)
  • verifyAndParseSqs(String, String, String)
  • verifyAndParseSns(String, String, String)

Each composite has an overload that resolves the secret from the configured singleton client.

Backwards compatibility

verifyWebhook and verifyWebhookSignature are preserved and delegate to the new verifySignature helper. Existing callers continue to work for plain (uncompressed) bodies.

Tests

WebhookCompressionTest covers plain / gzip / base64 / base64+gzip payloads, signature mismatches, malformed bytes, and JSON parsing into Event. Linked Linear ticket: CHA-3071.

Golden test fixtures (Tommaso)

Added shared reference fixtures to the test suite so future SDKs can sanity-check decoders against the same payloads:

aGVsbG93b3JsZA==                          -> helloworld   (base64)
H4sIAGrYAWoAA8tIzcnJL88vykkBAK0g6/kKAAAA -> helloworld   (base64 + gzip)

Test plan

  • ./gradlew test --tests io.getstream.chat.java.WebhookCompressionTest — all green (run via amazoncorretto:11 Docker image)
  • ./gradlew build — passes

Adds App.decompressWebhookBody and App.verifyAndDecodeWebhook so
handlers can accept the new outbound webhook compression
(GetStream/chat#13222) without changing how X-Signature is verified.

decompressWebhookBody returns the body unchanged when Content-Encoding
is null or empty, gunzips with java.util.zip.GZIPInputStream when the
header is gzip (case-insensitive, trimmed), and throws
IllegalStateException for any other value with a message that points
the operator at the app's webhook_compression_algorithm setting.

verifyWebhookSignature gains a byte[] overload so the existing String
overload no longer round-trips through UTF-8 unnecessarily, and the
equality check moves to MessageDigest.isEqual so comparison is
constant-time.

verifyAndDecodeWebhook chains decompression with the HMAC check and
returns the raw JSON when the signature matches; SecurityException is
thrown otherwise. The signature is always computed over the
uncompressed bytes, matching the server.

The webhook docs are updated with the new Content-Encoding header row
and a worked example using verifyAndDecodeWebhook.

Tests cover gzip round-trip, null/empty/whitespace passthrough, case-
insensitive Content-Encoding, invalid gzip bytes, every non-gzip
encoding rejected with a clear hint, byte[] / String HMAC overload
parity, signature mismatch, and the regression case where the
signature was computed over the compressed bytes.

Co-authored-by: Cursor <cursoragent@cursor.com>
nijeesh-stream and others added 2 commits May 7, 2026 12:33
Extends `decompressWebhookBody` and `verifyAndDecodeWebhook` with an
optional `payloadEncoding` argument. When set to "base64" (the wrapper
Stream applies for SQS / SNS firehose so the message stays valid UTF-8
over the queue), the body is base64-decoded before gzip decompression.

The HMAC signature continues to be computed over the innermost
(uncompressed, base64-decoded) JSON, so the verification rule is
invariant across HTTP webhooks and SQS / SNS.

`null` / `""` for payloadEncoding is a no-op, so the HTTP webhook path
is byte-identical to before this change. The existing 3-argument
overloads of `decompressWebhookBody` and `verifyAndDecodeWebhook` are
preserved for backward compatibility.

Co-authored-by: Cursor <cursoragent@cursor.com>
Replaces verifyAndDecodeWebhook / decompressWebhookBody on App with
the cross-SDK contract documented at
https://getstream.io/chat/docs/node/webhooks_overview/.

Static helpers on App:

  Primitives:
    ungzipPayload     - gzip magic-byte detection + inflate
    decodeSqsPayload  - base64 then ungzip-if-magic (String -> byte[])
    decodeSnsPayload  - alias for decodeSqsPayload
    verifySignature   - constant-time HMAC-SHA256 comparison
                        (parameter order matches the cross-SDK spec)
    parseEvent        - JSON -> typed Event via Jackson

  Composite (return Event):
    verifyAndParseWebhook
    verifyAndParseSqs
    verifyAndParseSns

Each composite has a singleton-secret overload that pulls the API
secret from Client.getInstance(), so handler code stays terse.

The composite functions auto-detect compression from body bytes,
keeping the same handler correct whether or not Stream is currently
compressing payloads, and behind middleware that auto-decompresses.

Backward compatibility:
  * App.verifyWebhook(body, signature) -> bool kept unchanged.
  * App.verifyWebhookSignature(...) overloads kept; they now delegate
    to verifySignature internally.

Co-authored-by: Cursor <cursoragent@cursor.com>
@nijeesh-stream nijeesh-stream changed the title [CHA-3071] feat: decode gzip-compressed webhook bodies feat(webhooks): verifyAndParse* API for compressed payloads (CHA-3071) May 8, 2026
nijeesh-stream and others added 5 commits May 8, 2026 16:53
RFC 1952 defines the gzip magic number as the two-byte sequence
1F 8B; the third byte (CM) is informational and not part of the
identifier. Trim the magic check from three bytes to two to match
the spec and stay consistent with the reference implementations
in the public docs.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Surface the dual-API pattern at the level callers actually hold a
reference to: the Client interface now ships default instance
methods (verifyAndParseWebhook, verifyAndParseSqs, verifyAndParseSns)
that delegate to the existing App static helpers using the client's
configured API secret. Mirrors the dual-API surface in stream-chat-go
and the other server SDKs and matches the call shape documented at
getstream.io/chat/docs/$framework/webhooks_overview/.

The static App helpers and their singleton overloads remain
available for stateless callers (Lambdas, queue consumers, tests).

Co-authored-by: Cursor <cursoragent@cursor.com>
The default ObjectMapper fails on unknown fields, can't parse Stream's
ISO date formats, and rejects unknown enum values. That made webhook
parsing diverge from the rest of the SDK and would break the first time
Stream added a new field, enum value, or date variant server-side.

Reuse a single mapper configured like DefaultClient's Retrofit mapper:
- FAIL_ON_UNKNOWN_PROPERTIES = false
- READ_UNKNOWN_ENUM_VALUES_USING_DEFAULT_VALUE = true
- StdDateFormat with UTC + colon-in-timezone

Also avoids allocating a fresh mapper on every webhook delivery.

Co-authored-by: Cursor <cursoragent@cursor.com>
The merged docs referenced App.verifyAndDecodeWebhook and
App.decompressWebhookBody, neither of which exists. Rewrite the Java
snippets against the actual API (verifyAndParseWebhook, ungzipPayload,
verifySignature, parseEvent, plus the SQS / SNS variants and the
Client.* instance overloads) so the examples compile.

Co-authored-by: Cursor <cursoragent@cursor.com>
Copy link
Copy Markdown
Contributor

@mogita mogita left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cross-SDK review pass for CHA-3071. Four inline comments — see below.

Comment thread src/main/java/io/getstream/chat/java/models/App.java Outdated
Comment thread src/main/java/io/getstream/chat/java/models/App.java
Comment thread src/main/java/io/getstream/chat/java/models/App.java
Comment thread src/test/java/io/getstream/chat/java/WebhookCompressionTest.java
decodeSnsPayload now JSON-parses the SNS HTTP notification envelope
({"Type":"Notification","Message":"..."}) and extracts the inner
Message field before running the SQS pipeline. Falls through to the
pre-extracted Message string when the input is not a JSON envelope so
existing call sites keep working.

Test adds a realistic SNS HTTP notification body fixture and exercises
both the new envelope path and the existing pre-extracted Message path.

Co-authored-by: Cursor <cursoragent@cursor.com>
@mogita
Copy link
Copy Markdown
Contributor

mogita commented May 11, 2026

Cross-SDK coordination: unifying webhook exception types

After the review pass across all 6 SDKs in this rollout and team discussion, we're consolidating the new webhook exception strategy to a single unified exception class rather than the split (signature vs parse exceptions) being introduced in this PR.

The Webhook Handling Spec on Notion (CHA-2961) has been revised to reflect this — §5.2 / §5.3 / §7 now specify a single class.

Why unified: From a customer's perspective, all failure modes — signature mismatch, gzip decompression failure, base64 decode failure, SNS envelope failure, JSON parse failure, missing schema field — terminate at the same catch block in customer code. A signature/parse split adds structural complexity without changing customer behavior. Customers who want to filter security logs for signature mismatches specifically can do so via exception message text or Throwable#getCause() chain.

Class name family: InvalidWebhookException — "Invalid" covers all failure modes accurately, consistent with JDK naming patterns (IllegalArgumentException, InvalidObjectException, etc.).

Per-SDK naming across the rollout:

SDK Class name
JS InvalidWebhookError (extends Error)
Python InvalidWebhookError
Go sentinel ErrInvalidWebhook + struct InvalidWebhookError
Java InvalidWebhookException (extends existing StreamException)
PHP InvalidWebhookException (extends existing StreamException)
Ruby StreamChat::InvalidWebhookError (extends StandardError)
.NET StreamInvalidWebhookException (extends StreamBaseException)

Asks for this PR:

  1. Introduce io.getstream.chat.java.exceptions.InvalidWebhookException extending existing StreamException, replacing the IllegalStateException (or any per-mode exceptions) introduced in this PR
  2. Wrap all failure paths into this single type — signature mismatch, gzip failure, base64 failure, SNS envelope failure, JSON parse failure, missing type/schema failure
  3. Attach a human-readable message identifying which failure mode fired (e.g. "signature mismatch", "invalid base64", "missing type field") so customers can filter on message content
  4. parseEvent return-type fix (already raised in inline comment on App.java:1599) — returning Optional<Event> or modeling UnknownEvent as a real subclass interacts with this exception change; both should be resolved together
  5. Legacy verifyWebhookSignature (returning boolean) stays unchanged — back-compat preserved
  6. Update JUnit tests to assert against the new exception name; for mode-specific tests, also assert on message-content substrings

This same comment is being posted on all 6 SDK PRs (JS / Go / Ruby / PHP / Java / .NET) for coordination. Happy to discuss naming or scope tradeoffs.

…n fixtures (CHA-3071)

Per Tommaso's suggestion, align the gzip helper with the GNU `gunzip`
command name. The function was added in this PR and not yet released,
so this is a straight rename with no back-compat alias.

Adds Tommaso's reference fixtures to the test suite as named cases so
future SDKs can sanity-check against the same payloads:

  aGVsbG93b3JsZA==                          -> helloworld   (base64)
  H4sIAGrYAWoAA8tIzcnJL88vykkBAK0g6/kKAAAA -> helloworld   (base64+gzip)

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants