Messaging and jobs redesign: one IMessageBus, durable jobs runtime, Redis Streams + AWS SQS/SNS transports

[ejsmith]

Adds a unified IMessageBus, typed background jobs through IJobClient, and shared monitoring for jobs and broker-delivered work.

Developer experience

Inject the ordinary application APIs:

await bus.SendAsync(new SendReceipt(1001, "dev@example.com"));
await bus.PublishAsync(new OrderPlaced(1001, "Espresso Machine"));

var job = await jobs.EnqueueAsync<ResizeImageJob, ResizeArgs>(
    new ResizeArgs("product.png", 640, 480));
var completed = await job.WaitForCompletionAsync(TimeSpan.FromSeconds(30));

• Messaging: competing consumers, durable service subscriptions, and separate best-effort SubscribeNodeAsync broadcasts. Implement IMessageHandler<T>; automatic batching keeps single-message calls efficient.
• Jobs: implement IJob or IJob<TArgs> for delays, progress, cancellation, persisted retries, and CRON. IScheduledJobManager supports schedule edits and manual runs.
• One job store: IJobRuntimeStore, JobState, and IJobMonitor cover ordinary jobs and tracked broker work, including paginated history, counts, metadata, heartbeats, and cancellation. Choose Jobs.UseInMemory() or Jobs.UseRedis().
• Delivery controls: scoped handlers, bounded concurrency, prefetch, lease renewal, graceful drain, health checks, and returned MessageOutcome values for success, retry, dead letter, or unsettled work. MessageProcessingContext supplies progress and cooperative cancellation.
• Operations: native Redis resource locks, queue statistics, dead-letter inspection/replay/deletion, and managed node-subscription cleanup. Fresh attempt tokens prevent stale updates from overwriting a newer attempt.

AddFoundatio() configures producers; AddFoundatioWorker(...) starts configured processing. The Quickstart runs without Docker; the messaging sample shows LocalStack, Redis, multiple replicas, and monitoring. Mediator #335 builds queued handlers and live events on these APIs.

Performance

Ordinary sends avoid batch bookkeeping; headers share immutable snapshots; in-memory receipts and unused settlement/renewal gates allocate less. Redis cancellation now expires and reads one job atomically.

Median jobs/sec, previous native version → this update:

Workload	Before	After
In memory, concurrency 1	81,347	87,539
In memory, concurrency 8	80,149	84,685
In memory, concurrency 64	100,073	97,236
Redis tracking, longer validation	6,380	6,503

In-memory allocations fall 15% (9% tracked). Redis throughput varied across short runs; longer runs used 13% less CPU and 3% fewer allocated bytes, with roughly level throughput. System .NET 10.0.11, Release, repeated local runs. Workload sizes, latency and raw results. Earlier messaging/MassTransit comparison.

Additional details

Broker-owned records never enter job-worker claims or recovery. The broker owns leases, retries and settlement; terminal tracking requires confirmed settlement. Queue delivery is at least once, node broadcasts are best effort, and enqueue/replay are not transactions with application data.

Validation: full build, 2,229 tests passed, 24 existing skips, Redis/LocalStack conformance and Quickstart verification. Coverage includes send failure outcomes, cancellation expiry, immutable headers, concurrent settlement and stale receipts. The broader comparison verified 4.32M jobs plus 46,504 jobs under cancellation and worker restarts. Its report retains two known CLR aborts that passed same-runtime retries. LocalStack results are not production AWS capacity.

[ejsmith]

Pushed two commits addressing the messaging/jobs design-review feedback. All changes are green against the messaging, queue, and jobs suites (plus the conformance harness); full core suite passes.

fix: address messaging/jobs design-review feedback

Durable jobs + CRON

• Hosted runtime driver (JobRuntimeService + AddJobRuntimeService()): pumps occurrence materialization, due-dispatch, recovery, and queued-job execution. The runtime previously never ran end-to-end.
• Lease ownership enforced in TryTransitionAsync (expectedNodeId) so a stale worker can't overwrite the reclaiming node's terminal state.
• Lease renewal heartbeat during execution; the run is cancelled if the lease is lost (RenewClaimAsync was dead code).
• Strong node identity (machine:pid:token), capped exponential retry backoff.
• Replaced the hand-rolled cron parser with the vendored Cronos (moved into core Foundatio.Cronos); materialize every missed occurrence in the misfire window, not just the latest.

Messaging

• Resilient consumer/subscription loops — a poison message or transient receive error no longer silently kills the consumer.
• Fixed an in-memory push-path double-settle (tolerate already-settled receipts in the safety-net abandon).
• In-memory transport now honors visibility timeouts (reaper redelivers unsettled messages) so its advertised at-least-once guarantee is real.
• Log handler exceptions instead of swallowing them; drop the misleading write-only content-type header; add ReceiveDeadLetteredAsync so poison payloads are inspectable.

Conformance harness

• Silent capability skips → Assert.Skip; ordering assertions gated on the declared OrderingGuarantee; added visibility-timeout, competing-consumer, and DLQ-read scenarios.

refactor: hoist shared MessageQueue/PubSub behavior into MessageClientCore (M1, M2)

• Removed the large duplication between MessageQueue and PubSub via an internal MessageClientCore; unified the two handle classes and the two HandleMessageAsync overloads. Pub/sub subscriptions now also honor RedeliveryBackoff (prior drift).
• Enforce ITransportInfo.MaxBatchSize by chunking oversized sends.

Still intentionally not addressed (no external provider on this branch): proving the harness against a real transport before cutting the public API — that's the phased-rollout gate from the plan.