Skip to content

Batch processing failures #770

@slifty

Description

@slifty

Something claude flagged related to batch failures which I think will probably end up being out of scope for this PR. In the context of batches a single message failure rejects the entire batch and SQS would retry all of them.

Claude suggested that we could do better by adding support for batch failure reporting:

  export const handler: SQSHandler = Sentry.wrapHandler(
    async (event: SQSEvent) => {
      const s3Client = new S3Client({ region: process.env["AWS_REGION"] ?? "" });
      const results = await Promise.allSettled(
        event.Records.map(async (message) => {
          // ... existing per-message logic, but throw on failure ...
          // (return message.messageId from .catch handlers, or just let throw propagate)
        }),
      );
      const batchItemFailures = results
        .map((r, i) =>
          r.status === "rejected"
            ? { itemIdentifier: event.Records[i]!.messageId }
            : null,
        )
        .filter((x): x is { itemIdentifier: string } => x !== null);
      return { batchItemFailures };
    },
  );

Which apparently would also require some terraform changes to handle batch item failure reports:

  resource "aws_lambda_event_source_mapping" "file_copier_dev_event_source_mapping" {
    event_source_arn                   = aws_sqs_queue.file_copier_dev_queue.arn
    function_name                      = aws_lambda_function.file_copier_dev_lambda.arn
    batch_size                         = 10
    maximum_batching_window_in_seconds = 0
    function_response_types            = ["ReportBatchItemFailures"]
  }

If this was a pattern we wanted to actually pursue I think it would apply to other lambdas as well.

Anyway, thought I'd share in case it was worth a followup issue! Not suggesting a change in this PR.

Originally posted by @slifty in #757 (comment)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions