Skip to content

events: SQS, Azure Queue and R2 Queue sources; events setup - #55

Merged
calvin-archastro merged 1 commit into
mainfrom
feat/event-sources
Nov 2, 2026
Merged

calvin-archastro merged 1 commit into
mainfrom
feat/event-sources

Conversation

@calvin-archastro

@calvin-archastro calvin-archastro commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Adds three bucket-notification sources to pail events and a pail events setup command. A notification is still only a nudge; the watcher re-reads, the safety poll covers losses.

Sources

Service Flag Reads
S3 --sqs-queue URL SQS: S3 event notifications, EventBridge, either wrapped by SNS (SigV4; creds: pail provider, AWS_* env, then aws configure export-credentials)
Azure Blob --azure-queue URL Storage Queue: Event Grid BlobCreated/BlobDeleted (Event Grid or CloudEvents schema, JSON or base64); SAS in the URL or a provider token
R2 --r2-queue ACCOUNT/QUEUE Cloudflare Queues HTTP pull; CLOUDFLARE_API_TOKEN or provider token
  • Also PAIL_EVENTS_SOURCE=sqs:URL (gcs:, azure:, r2:, poll) and config [events "s3://acme/app"] source = …. Order: flag, env, config, poll.
  • Unparseable message: logged, becomes Hint::Scan (full re-list). Never dropped silently. Wrong bucket/prefix: deleted, ignored.
  • Delete only after the events are out and the cursor is saved.
  • MinIO: not done; follow-up (webhook listener or NATS).

pail events setup --consumer NAME

Prints gcloud / aws / az / wrangler commands for <repo>/refs/ and <repo>.actions/ into pail-NAME. Print-only by default. --apply asks y/N (--yes skips) and refuses while a placeholder is unresolved. Not run by the author.

Model

No change. Every source is the model's notification; unparseable = the safety-poll scan; delete = finish (needs emitted and saved). Written up in verify/README.md.

Tests

  • Parsers on sample payloads from each provider's docs; SigV4 against AWS's documented example; HMAC against RFC 4231.
  • Source loops against in-process HTTP fakes (SQS, Azure Queue, R2 pull): delete only after the cursor holds the commit, unparseable still scans, elsewhere-messages ignored.
  • Setup plans per provider; apply with a fake runner (declined, --yes, stop on failure, placeholders refused).
  • No cloud calls. Azure and R2 are unverified against the real services; latency unmeasured (marked so in docs).
  • cargo fmt --check, cargo test --release, npm run check, npm test pass. ci.yml unchanged.

🤖 Generated with Claude Code

New pull-based hint sources for `pail events`, one per service's queue:
--sqs-queue (S3 notifications or EventBridge, SigV4), --azure-queue (Event
Grid -> Storage Queue), --r2-queue ACCOUNT/QUEUE (Cloudflare Queues HTTP pull).
Also PAIL_EVENTS_SOURCE and a store-config `events.<url>.source`.

A notification stays a nudge: a message that does not parse is logged and
becomes Hint::Scan (a full re-list); a message is deleted only after its
events are out and the cursor is saved.

`pail events setup --consumer NAME` prints the gcloud/aws/az/wrangler commands
that wire refs/ and .actions/ into a per-consumer queue; --apply runs them
after a y/N question (--yes skips it).

No model change: every source is the model's notification (verify/README.md).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@calvin-archastro
calvin-archastro merged commit 584fbb0 into main Nov 2, 2026
7 checks passed
@calvin-archastro
calvin-archastro deleted the feat/event-sources branch November 2, 2026 21:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant