feat: allow the Kafka record key deserializer to be configured - #59
Merged
Merged
Conversation
Drone Fly reads record keys with a LongDeserializer, the default in the Apiary KafkaMessageReader, which matches the key written by the Apiary Hive Metastore listener. A topic populated by a different producer may key its records with another type, and reading those fails on every record with: SerializationException: Size of data received by LongDeserializer is not 8 The consumer offset does not advance past a record it cannot deserialize, so Drone Fly retries the same offset indefinitely and stops processing entirely. Resolve the key deserializer from the consumer properties and pass it to the builder explicitly, defaulting to LongDeserializer so existing deployments are unaffected. It cannot be picked up from withConsumerProperties, because those values do not override the builder's own defaults; apiary-extensions 8.2.5 adds withKeyDeserializer for this purpose. The record key is not used to process the event, so any deserializer that can read the key is safe. Co-authored-by: Claude Opus <noreply@anthropic.com>
HamzaJugon
approved these changes
Sep 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📝 Description
Drone Fly reads Kafka record keys with a
LongDeserializer— the default in the ApiaryKafkaMessageReader, matching the key written by the Apiary Hive Metastore listener. That default is fine, but it currently cannot be changed.A topic populated by a different producer may key its records with another type, for example a
String<database>.<table>. Reading such a topic fails on every record:This is worse than a rejected record. The consumer offset never advances past a record it cannot deserialize, so Drone Fly retries the same offset indefinitely and stops processing entirely — no listeners firing, and no obvious signal beyond the repeated exception.
Setting
apiary.messaging.consumer.key.deserializerdid not help before this change: those properties reachwithConsumerProperties(), which merges them with collisions resolved in favour of the builder's own defaults, so the value was silently discarded.Changes
withKeyDeserializer, added inapiary-extensions8.2.5 (ExpediaGroup/apiary-extensions#148).LongDeserializer, so existing deployments are unaffected — no configuration change required for anyone consuming an Apiary HMS listener topic.APIARY_MESSAGING_CONSUMER_KEY_DESERIALIZER.apiary-extensionsfrom 8.2.0 to 8.2.5.1.0.10entry toCHANGELOG.md(the pom is already at1.0.10-SNAPSHOT).The record key is not used to process the event — events are read from the record value — so any deserializer that can read the key is safe.
✅ Test plan
CommonBeansTest: defaults toLongDeserializerwhen unset, uses the configured value when present.mvn testondrone-fly-app—Tests run: 35, Failures: 0, Errors: 0, Skipped: 0.mvn test-compileacross the reactor, includingdrone-fly-integration-tests, to confirm the 8.2.0 → 8.2.5 upgrade does not break compilation.🔗 Related Issues
Depends on ExpediaGroup/apiary-extensions#148, released in
apiary-extensions8.2.5.🤖 Generated with opencode