Skip to content

Add retry with exponential backoff for webhook notifications #15

@roeimichael

Description

@roeimichael

Problem

In webhook.py, if the HTTP request times out or gets a 5xx/429 response, it returns False and the notification is permanently lost. There is no retry mechanism, no queue for failed notifications, and no way to know a notification was dropped. Slack rate limiting (429) silently discards anomaly alerts.

What to do

  1. Add retry logic to send_webhook() in src/context_analyzer_tool/notify/webhook.py:
    • 3 attempts with exponential backoff (1s, 2s, 4s delays)
    • For 429 responses, respect the Retry-After header
    • Log each retry attempt
  2. (Optional stretch) Persist failed payloads to a queue table and retry them periodically

Files to look at

  • src/context_analyzer_tool/notify/webhook.py, lines 181-230
  • Consider using tenacity library or a simple manual retry loop with asyncio.sleep

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: notificationsAlerts, webhooks, and notification dispatchdifficulty: intermediateRequires some familiarity with the codebaseenhancementNew feature or requesthelp wantedExtra attention is needed

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions