Skip to content

feat: allow the Kafka record key deserializer to be configured - #59

Merged
javsanbel2 merged 1 commit into
mainfrom
feat/configurable-key-deserializer
Sep 1, 2026
Merged

javsanbel2 merged 1 commit into
mainfrom
feat/configurable-key-deserializer

Conversation

@javsanbel2

Copy link
Copy Markdown
Contributor

📝 Description

Drone Fly reads Kafka record keys with a LongDeserializer — the default in the Apiary KafkaMessageReader, matching the key written by the Apiary Hive Metastore listener. That default is fine, but it currently cannot be changed.

A topic populated by a different producer may key its records with another type, for example a String <database>.<table>. Reading such a topic fails on every record:

SerializationException: Size of data received by LongDeserializer is not 8

This is worse than a rejected record. The consumer offset never advances past a record it cannot deserialize, so Drone Fly retries the same offset indefinitely and stops processing entirely — no listeners firing, and no obvious signal beyond the repeated exception.

Setting apiary.messaging.consumer.key.deserializer did not help before this change: those properties reach withConsumerProperties(), which merges them with collisions resolved in favour of the builder's own defaults, so the value was silently discarded.

Changes

  • Resolve the key deserializer from the consumer properties and pass it to the builder explicitly via withKeyDeserializer, added in apiary-extensions 8.2.5 (ExpediaGroup/apiary-extensions#148).
  • The default remains LongDeserializer, so existing deployments are unaffected — no configuration change required for anyone consuming an Apiary HMS listener topic.
  • Configure it through the existing consumer-properties mechanism, so no new plumbing is introduced:
    apiary.messaging.consumer.key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
    
    or the environment variable APIARY_MESSAGING_CONSUMER_KEY_DESERIALIZER.
  • Upgrade apiary-extensions from 8.2.0 to 8.2.5.
  • Document the setting and the failure mode in the README; add a 1.0.10 entry to CHANGELOG.md (the pom is already at 1.0.10-SNAPSHOT).

The record key is not used to process the event — events are read from the record value — so any deserializer that can read the key is safe.

✅ Test plan

  • New CommonBeansTest: defaults to LongDeserializer when unset, uses the configured value when present.
  • mvn test on drone-fly-app — Tests run: 35, Failures: 0, Errors: 0, Skipped: 0.
  • mvn test-compile across the reactor, including drone-fly-integration-tests, to confirm the 8.2.0 → 8.2.5 upgrade does not break compilation.

🔗 Related Issues

Depends on ExpediaGroup/apiary-extensions#148, released in apiary-extensions 8.2.5.

🤖 Generated with opencode

Drone Fly reads record keys with a LongDeserializer, the default in the Apiary
KafkaMessageReader, which matches the key written by the Apiary Hive Metastore
listener. A topic populated by a different producer may key its records with
another type, and reading those fails on every record with:

  SerializationException: Size of data received by LongDeserializer is not 8

The consumer offset does not advance past a record it cannot deserialize, so
Drone Fly retries the same offset indefinitely and stops processing entirely.

Resolve the key deserializer from the consumer properties and pass it to the
builder explicitly, defaulting to LongDeserializer so existing deployments are
unaffected. It cannot be picked up from withConsumerProperties, because those
values do not override the builder's own defaults; apiary-extensions 8.2.5 adds
withKeyDeserializer for this purpose.

The record key is not used to process the event, so any deserializer that can
read the key is safe.

Co-authored-by: Claude Opus <noreply@anthropic.com>
@javsanbel2
javsanbel2 requested a review from a team as a code owner September 1, 2026 09:47
@javsanbel2
javsanbel2 merged commit a73fb67 into main Sep 1, 2026
1 check passed
@javsanbel2
javsanbel2 deleted the feat/configurable-key-deserializer branch September 1, 2026 09:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants