AWS Kinesis Input: Add optional store_full_message field - #26820
AWS Kinesis Input: Add optional store_full_message field#26820Nithin-Kasam wants to merge 10 commits into
Conversation
# Conflicts: # graylog2-web-interface/src/integrations/aws/cloudwatch/FormAdvancedOptions.tsx
|
Attaching Set up and testing guide : |
| public abstract boolean addFlowLogPrefix(); | ||
|
|
||
| @JsonProperty(STORE_FULL_MESSAGE) | ||
| public abstract boolean storeFullMessage(); |
There was a problem hiding this comment.
This makes store_full_message a required property on the input create request, so any existing client that doesn't send it now fails in build() with Missing required properties: storeFullMessage. I think we can keep backwards API compatibility by giving it a default in the builder instead of requiring it:
For example:
@JsonCreator
public static Builder create() {
return new AutoValue_AWSInputCreateRequest.Builder()
.storeFullMessage(false);
}
| .streamName("a-stream") | ||
| .batchSize(10000) | ||
| .addFlowLogPrefix(true) | ||
| .storeFullMessage(true) |
There was a problem hiding this comment.
Could you expand this test while you're here? It asserts all the neighbouring config keys on the captured InputCreateRequest but not the new one, so the configuration.put(AWSCodec.CK_STORE_FULL_MESSAGE, ...) line in AWSService isn't actually covered. Something like this down with the other configuration assertions:
assertEquals(true, input.configuration().get(AWSCodec.CK_STORE_FULL_MESSAGE));
|
|
||
| // Load the codec by message type. | ||
| final AWSMessageType awsMessageType = AWSMessageType.valueOf(configuration.getString(CK_AWS_MESSAGE_TYPE)); | ||
| LOG.debug("Using AWS codec for message type: {}", awsMessageType); |
There was a problem hiding this comment.
Was there a specific reason to add the logging in this PR? It looks unrelated to the feature. This debug line also fires per message for a value that's fixed for the life of the input, and the LOG.error below duplicates what the InputProcessingException on the next line already carries, so an unknown message type ends up logged twice for the same problem.
There was a problem hiding this comment.
LOG is never used in the code hence I have added those.Shall I remove LOG ?
There was a problem hiding this comment.
I think you can go ahead and remove it.
| if (CollectionUtils.isNotEmpty(logEvent.subscriptionFilters())) { | ||
| result.addField(FIELD_SUBSCRIPTION_FILTERS, logEvent.subscriptionFilters()); | ||
| } | ||
| if (configuration.getBoolean(AWSCodec.CK_STORE_FULL_MESSAGE, false)) { |
There was a problem hiding this comment.
This reads the config on every decoded message. I think that should be resolved once per message instead in the constructor, similar to:
There was a problem hiding this comment.
LGTM and tested successfully.
- New option shows in the input's advanced options and persists on new inputs, on or off.
- Enabled: payload lands in
full_messageas expected. Disabled: omitted. - Backwards compatibility tested successfully. Input without
store_full_messagefield in Mongo still starts and ingests file without option enabled.
2 participants
Description
Add an optional "Store full message" boolean configuration to the AWS Kinesis/CloudWatch input. When enabled, the codecs store the Kinesis log event payload in the full_message field.
Motivation and Context
The AWS Kinesis/CloudWatch input does not currently preserve the original log payload. In raw CloudWatch mode, the message field is overwritten during parsing. In Flow Log mode, message is replaced with a generated summary. The new AWS WAF Illuminate pack parses WAF JSON from the message field, and any content not parsed is lost to the customer.
An optional full_message field preserves the original payload.
closes https://github.com/Graylog2/graylog-plugin-enterprise/issues/14362
How Has This Been Tested?
-> Configured Kinesis input by enabling full message and verfied the logs contains full_message field.
-> Added unit testcases.
Screenshots (if appropriate):
Types of changes
Checklist: