Add large-payload blob auto-purge (opt-in singleton job, worker/SDK side) by YunchuWang · Pull Request #758 · microsoft/durabletask-dotnet

YunchuWang · 2026-07-08T19:51:11Z

Summary

Large orchestration payloads are externalized to Azure Blob Storage by the AzureBlobPayloads extension as blob:v1:<container>:<blobName> tokens. The DTS backend stores those tokens but cannot delete the backing blobs — it has no storage credentials; only this SDK can. This PR implements the worker/SDK side of large-payload blob auto-purge.

Design (opt-in singleton durable entity + orchestration job)

Instead of an always-on background stream, this mirrors the existing src/ExportHistory feature: an opt-in, whole-scheduler singleton durable entity + orchestration job that drains soft-deleted payload rows the backend exposes and deletes their blobs, then acks so the backend can hard-delete the rows.

PayloadStore.DeleteAsync — added as a virtual method (default throws NotSupportedException, so it is non-breaking for existing external subclasses). BlobPayloadStore overrides it to decode the token and call DeleteIfExistsAsync (idempotent — deleting a missing blob is a no-op).
BlobPurgeJob (TaskEntity singleton) — Create is a no-op when already Active so racing client processes don't disturb the running job (intentionally softer than ExportJob.Create, which throws). Run starts a fixed-instance-id orchestrator.
BlobPurgeJobOrchestrator (perpetual) — each cycle: fetch a batch of tombstones, delete the blobs with capped parallelism (32), ack only the successful deletions (failed tokens stay tombstoned to retry), idle on a 1-minute timer when there's nothing to purge, and ContinueAsNew every 5 cycles to keep history small. Activities use a small retry policy.
ExecuteBlobPurgeJobOperationOrchestrator — client → entity bridge (mirrors export).
Activities (constructor DI): GetTombstonedPayloadsActivity, DeleteExternalBlobActivity (returns false + logs on failure so one bad token can't fail the batch), AckPurgedPayloadsActivity.
Purge RPCs on the core client — the worker fetch/ack RPCs are now first-class methods on DurableTaskClient (GetTombstonedPayloadsAsync/AckPurgedPayloadsAsync), overridden in GrpcDurableTaskClient — mirroring how ExportHistory added ListInstanceIdsAsync/GetOrchestrationHistoryAsync. The purge activities inject DurableTaskClient directly; no dedicated gRPC client is needed. AzureManaged reuses GrpcDurableTaskClient, so it inherits these methods with no extra client.
Options — LargePayloadStorageOptions.AutoPurge (opt-in, default false) and PayloadPurgeBatchSize (default 500).
Enablement — client-side BlobPurgeJobStarter (IHostedService) ensures the singleton job when AutoPurge is enabled, on a background task that does not block host startup and retries until the backend is reachable. The worker always registers the entity/orchestrators/activities (not gated on AutoPurge) so a client-enabled job always has something to execute.

gRPC contract

Two new unary RPCs added to TaskHubSidecarService in src/Grpc/orchestrator_service.proto (worker is the client; wire paths /TaskHubSidecarService/GetTombstonedPayloads and /AckPurgedPayloads):

rpc GetTombstonedPayloads(GetTombstonedPayloadsRequest) returns (GetTombstonedPayloadsResponse);
rpc AckPurgedPayloads(AckPurgedPayloadsRequest) returns (AckPurgedPayloadsResponse);

message TombstonedPayload { int32 partitionId = 1; int64 instanceKey = 2; int64 payloadId = 3; string token = 4; }
message PayloadPurgeAck   { int32 partitionId = 1; int64 instanceKey = 2; int64 payloadId = 3; }
message GetTombstonedPayloadsRequest  { int32 limit = 1; }
message GetTombstonedPayloadsResponse { repeated TombstonedPayload payloads = 1; }
message AckPurgedPayloadsRequest      { repeated PayloadPurgeAck acks = 1; }
message AckPurgedPayloadsResponse     { }

C# stubs are generated at build time by Grpc.Tools (not committed). The authoritative proto change is a follow-up in microsoft/durabletask-protobuf#76; once it merges, src/Grpc/orchestrator_service.proto should be re-synced from upstream (content identical to what's here). The vendored change lets this PR build/test standalone.

Testing

dotnet build Microsoft.DurableTask.sln — succeeds, 0 errors.
dotnet test test/AzureBlobPayloads.Tests — 11 passed (BlobPayloadStore delete/idempotency + BlobPurgeJob.Create no-op-when-Active + options defaults).
dotnet test test/ExportHistory.Tests — 147 passed (shared patterns unaffected).
dotnet test test/Client/Core.Tests + test/Client/Grpc.Tests — 43 + 47 passed (core DurableTaskClient / GrpcDurableTaskClient edits).

Notes / deviations

BlobPurgeJobStatus.Pending is the 0/default value (mirroring ExportJobStatus.Pending=0) so a freshly initialized entity never appears Active.
Added a RecordPurged entity op to track a cumulative PurgedCount.

Draft — not for merge until the authoritative proto PR lands and the backend serves the RPCs on TaskHubSidecarService.

Co-authored-by: Copilot App 223556219+Copilot@users.noreply.github.com

Large orchestration payloads are externalized to Azure Blob Storage as `blob:v1:<container>:<blobName>` tokens. The DTS backend stores those tokens but cannot delete the backing blobs (it has no storage credentials) — only this SDK can. This adds an opt-in, whole-scheduler singleton durable entity + orchestration job (mirroring src/ExportHistory) that drains payload rows the backend has soft-deleted and deletes their blobs, then acks so the backend can hard-delete the rows. Design: - PayloadStore.DeleteAsync is virtual (default throws NotSupportedException so it is non-breaking for existing external subclasses); BlobPayloadStore overrides it to decode the token and call DeleteIfExistsAsync (idempotent). - BlobPurgeJob (TaskEntity singleton): Create is a no-op when already Active so racing client processes don't disturb the running job; Run starts a fixed-id orchestrator. - BlobPurgeJobOrchestrator (perpetual): fetch a batch of tombstones, delete the blobs with capped parallelism, ack the successful deletions (failed tokens stay tombstoned to retry), idle on a timer when empty, ContinueAsNew periodically. - ExecuteBlobPurgeJobOperationOrchestrator bridges client -> entity. - Two new unary RPCs on TaskHubSidecarService: GetTombstonedPayloads / AckPurgedPayloads (authoritative proto follow-up: microsoft/durabletask-protobuf#76). - LargePayloadStorageOptions gains AutoPurge (opt-in, default false) and PayloadPurgeBatchSize (default 500). - Client-side BlobPurgeJobStarter (IHostedService) ensures the singleton job when AutoPurge is enabled, without blocking host startup. Worker always registers the entity/orchestrators/activities so a client-enabled job has something to run. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…er simplification - Drop the `Dto` suffix now that the payload records are first-class public types in `Microsoft.DurableTask.Client` (`TombstonedPayload`, `PayloadPurgeAck`). - Collapse the magic `500` batch-size literal into a single `BlobPurgeConstants.DefaultBatchSize` used everywhere. - Rename `BlobPurgeJobStatus.Stopped` -> `Pending` (still the zero value) and remove the dead `Failed` member (nothing ever set it; the job self-heals). - Make the perpetual orchestrator self-heal: wrap each cycle in try/catch so a transient backend/entity/activity failure logs, backs off, and continues instead of failing the orchestration and killing the eternal loop. - Ack poison tokens: `DeleteExternalBlobActivity` now returns a three-way `BlobDeleteResult` (Deleted/Discarded/Retry). Malformed tokens are discarded and acked so the backend can clear the stuck row instead of re-streaming it forever; transient failures stay tombstoned to retry. - Replace the single-value `BlobPurgeJobCreationOptions` record with a plain `int` on `BlobPurgeJob.Create`. - Guard the client fetch RPC: `GetTombstonedPayloadsAsync` throws `ArgumentOutOfRangeException` unless `0 < limit < 1000`. - Simplify `BlobPurgeJobStarter` to a fixed-instance-id fire-once: drop the entity-active pre-check and schedule the Create bridge once with a fixed instance id, retrying only until the backend is reachable. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Avoid needlessly re-running a Completed bridge orchestration on every host restart (fixed id + no dedupe means the backend would purge+replace a terminal instance). Check the existing bridge via GetInstanceAsync and only (re)schedule when it is absent or ended Failed/Terminated, so a failed setup still self-heals. Handle the schedule race with OrchestrationAlreadyExistsException. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…bActivity The Azure Storage SDK already retries transient failures internally (connection errors + HTTP 408/429/5xx with backoff), so an escaped exception means those retries were exhausted. Treating every escaped exception as Retry mis-classified permanent service rejections (e.g. Status 400 InvalidResourceName from a malformed decoded blob name) as transient, causing an infinite re-drain of a poison token. Add a RequestFailedException Status 400 -> Discarded branch (ack so the backend clears the row); keep 403/408/429/5xx/timeouts/cancellation as Retry. Document the doc-verified exception model on the activity and add focused tests for the 400 -> Discarded and non-400 -> Retry outcomes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

… AutoPurge Address review #B (store duplication) and #C (conditional-DI of the starter): - TryAddSingleton the PayloadStore so we reuse an already-registered shared store instead of creating a second, redundant one. - Register the auto-purge starter only when the configure overload opted into AutoPurge (peek the flag via a probe options instance); UseExternalizedPayloadsCore no longer registers it unconditionally, so the no-arg overload never starts it. - The starter no longer self-checks the flag in StartAsync (registration already gates it); remove the now-unused BlobPurgeDisabled log and refresh its xmldoc. - Add DI-registration tests proving the starter is registered iff AutoPurge is on. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

The store's consumer is the interceptor wired up in UseExternalizedPayloadsCore, so Core is the single fallback registration site. Move the TryAddSingleton<PayloadStore> out of each configure overload and into Core, symmetrically for the client and worker extensions. TryAdd keeps reusing a shared store (AddExternalizedPayloadStore or the sibling builder) and never creates a redundant one. The worker overload switches from AddSingleton to the shared TryAddSingleton in Core as part of the move. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…store Restore the original design: the client never registers a PayloadStore, it only consumes a shared/external one (via AddExternalizedPayloadStore or an in-process worker). Only the worker self-registers a fallback store. Remove the client Core's TryAddSingleton<PayloadStore> block (and its now-unused Microsoft.Extensions.DependencyInjection.Extensions using); the client Core PostConfigure still resolves PayloadStore from the shared/worker registration. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

This PR adds an opt-in, worker/SDK-driven auto-purge mechanism for Azure Blob externalized payloads, enabling the SDK (which has storage credentials) to delete backing blobs after the backend soft-deletes payload rows and exposes tombstones via new gRPC RPCs.

Changes:

Adds a singleton durable-entity + perpetual orchestrator “blob purge job” with activities to fetch tombstones, delete blobs, and ACK successful deletions.
Extends the public client surface (DurableTaskClient + GrpcDurableTaskClient) with purge RPC methods and adds the corresponding RPCs/messages to the vendored sidecar proto.
Introduces LargePayloadStorageOptions.AutoPurge + PayloadPurgeBatchSize, a client-side hosted starter, and a new test project covering deletion/token behavior, DI, job create semantics, and options defaults.

Reviewed changes

Copilot reviewed 29 out of 29 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
test/AzureBlobPayloads.Tests/PayloadStore/BlobPayloadStoreTests.cs	Adds unit coverage for token decoding and idempotent blob deletion behavior.
test/AzureBlobPayloads.Tests/Options/LargePayloadStorageOptionsTests.cs	Verifies AutoPurge and batch-size default values.
test/AzureBlobPayloads.Tests/DependencyInjection/UseExternalizedPayloadsTests.cs	Validates conditional DI registration of the hosted purge starter.
test/AzureBlobPayloads.Tests/AzureBlobPayloads.Tests.csproj	Introduces a dedicated test project for the AzureBlobPayloads extension.
test/AzureBlobPayloads.Tests/AutoPurge/DeleteExternalBlobActivityTests.cs	Covers poison vs transient failure classification for delete activity.
test/AzureBlobPayloads.Tests/AutoPurge/BlobPurgeJobTests.cs	Tests singleton entity Create semantics and batch-size persistence/defaulting.
src/Grpc/orchestrator_service.proto	Adds new unary RPCs and messages for tombstone fetch + purge ACK.
src/Extensions/AzureBlobPayloads/PayloadStore/PayloadStore.cs	Adds virtual `DeleteAsync` to support deletable external stores without breaking subclasses.
src/Extensions/AzureBlobPayloads/PayloadStore/BlobPayloadStore.cs	Implements token-based blob deletion via `DeleteIfExistsAsync(IncludeSnapshots)`.
src/Extensions/AzureBlobPayloads/Options/LargePayloadStorageOptions.cs	Adds AutoPurge enablement + batch-size option.
src/Extensions/AzureBlobPayloads/DependencyInjection/DurableTaskWorkerBuilderExtensions.AzureBlobPayloads.cs	Ensures purge entity/orchestrations/activities are registered and avoids redundant PayloadStore registration.
src/Extensions/AzureBlobPayloads/DependencyInjection/DurableTaskClientBuilderExtensions.AzureBlobPayloads.cs	Conditionally registers the hosted starter when AutoPurge is enabled via configuration callback.
src/Extensions/AzureBlobPayloads/AutoPurge/Orchestrations/ExecuteBlobPurgeJobOperationOrchestrator.cs	Adds client→entity bridge orchestration for driving entity ops through orchestration surface.
src/Extensions/AzureBlobPayloads/AutoPurge/Orchestrations/BlobPurgeJobOrchestrator.cs	Implements the perpetual purge loop with capped parallelism, idle delay, and periodic ContinueAsNew.
src/Extensions/AzureBlobPayloads/AutoPurge/Models/BlobPurgeJobStatus.cs	Adds job status model for the singleton entity.
src/Extensions/AzureBlobPayloads/AutoPurge/Models/BlobPurgeJobState.cs	Adds persisted entity state (timestamps, counts, batch size, last error).
src/Extensions/AzureBlobPayloads/AutoPurge/Models/BlobDeleteResult.cs	Adds delete outcome enum to drive ACK vs retry decisions.
src/Extensions/AzureBlobPayloads/AutoPurge/Logs.cs	Adds structured log messages for purge job lifecycle and failures.
src/Extensions/AzureBlobPayloads/AutoPurge/Entity/BlobPurgeJob.cs	Adds singleton entity to manage activation and orchestrator startup.
src/Extensions/AzureBlobPayloads/AutoPurge/Constants/BlobPurgeConstants.cs	Centralizes job IDs and defaults for purge job components.
src/Extensions/AzureBlobPayloads/AutoPurge/Client/BlobPurgeJobStarter.cs	Adds hosted service to ensure the singleton purge job without blocking startup.
src/Extensions/AzureBlobPayloads/AutoPurge/Activities/GetTombstonedPayloadsActivity.cs	Fetches tombstoned payloads via injected DurableTaskClient.
src/Extensions/AzureBlobPayloads/AutoPurge/Activities/DeleteExternalBlobActivity.cs	Deletes a blob token and classifies outcomes to avoid batch failure.
src/Extensions/AzureBlobPayloads/AutoPurge/Activities/AckPurgedPayloadsActivity.cs	ACKs successful deletions to backend via DurableTaskClient.
src/Client/Grpc/GrpcDurableTaskClient.cs	Implements the new purge RPC client methods.
src/Client/Core/TombstonedPayload.cs	Adds serializable model for purge tombstones across activity boundary.
src/Client/Core/PayloadPurgeAck.cs	Adds serializable model for purge ACKs across activity boundary.
src/Client/Core/DurableTaskClient.cs	Adds new virtual purge methods to the public core client API.
Microsoft.DurableTask.sln	Adds the new AzureBlobPayloads test project to the solution.

+        LargePayloadStorageOptions opts = this.options.Get(this.builderName);
+        int batchSize = opts.PayloadPurgeBatchSize > 0 ? opts.PayloadPurgeBatchSize : BlobPurgeConstants.DefaultBatchSize;
+


+        this.State.Status = BlobPurgeJobStatus.Active;
+        this.State.PurgeBatchSize = purgeBatchSize > 0 ? purgeBatchSize : BlobPurgeConstants.DefaultBatchSize;
+        this.State.CreatedAt ??= DateTimeOffset.UtcNow;
+        this.State.LastModifiedAt = DateTimeOffset.UtcNow;
+        this.State.LastError = null;


+        int batchSize = input.PurgeBatchSize > 0 ? input.PurgeBatchSize : BlobPurgeConstants.DefaultBatchSize;
+        int processedCycles = input.ProcessedCycles;


+        P.GetTombstonedPayloadsResponse response = await this.sidecarClient.GetTombstonedPayloadsAsync(
+            new P.GetTombstonedPayloadsRequest { Limit = limit },
+            cancellationToken: cancellation);
+
+        List<TombstonedPayload> result = new(response.Payloads.Count);
+        foreach (P.TombstonedPayload payload in response.Payloads)
+        {
+            result.Add(new TombstonedPayload(
+                payload.PartitionId, payload.InstanceKey, payload.PayloadId, payload.Token));
+        }
+
+        return result;
+    }


+            return;
+        }
+
+        await this.sidecarClient.AckPurgedPayloadsAsync(request, cancellationToken: cancellation);


YunchuWang force-pushed the yunchuwang-wangbill-blob-payload-autopurge-sdk branch 2 times, most recently from 82ae04d to 0ac2dc3 Compare July 8, 2026 20:50

github-code-quality Bot found potential problems Jul 8, 2026

View reviewed changes

Comment thread src/Extensions/AzureBlobPayloads/AutoPurge/BlobPayloadAutoPurgeService.cs Fixed

YunchuWang force-pushed the yunchuwang-wangbill-blob-payload-autopurge-sdk branch 4 times, most recently from 0752610 to cab0e9a Compare July 13, 2026 19:07

YunchuWang changed the title ~~Add large-payload blob auto-purge (worker/SDK side)~~ Add large-payload blob auto-purge (opt-in singleton job, worker/SDK side) Jul 13, 2026

github-code-quality Bot found potential problems Jul 13, 2026

View reviewed changes

Comment thread src/Extensions/AzureBlobPayloads/AutoPurge/Orchestrations/BlobPurgeJobOrchestrator.cs Fixed

YunchuWang force-pushed the yunchuwang-wangbill-blob-payload-autopurge-sdk branch from cab0e9a to c05b15a Compare July 13, 2026 20:38

YunchuWang force-pushed the yunchuwang-wangbill-blob-payload-autopurge-sdk branch from c05b15a to 306d19f Compare July 13, 2026 22:43