Count runs with blocked or failing workflow tasks in fleet operator metrics

A run whose workflow task cannot replay stays open. It is flagged on the run itself but missing from fleet counters and health checks, so alerting built on operator metrics does not see it.

Current behavior

Workflow 4423774074d66347c627c2c13ac516207e815e05:

RunSummaryProjector sets a workflow_replay_blocked liveness state, and Waterline shows a task-failure badge with list filters.
OperatorMetrics counts only repair_needed runs, claim-failed tasks and compatibility-blocked runs. Replay-blocked runs are not counted and have no oldest-age field. HealthCheck has no matching check.
The Server's dw_workflow_task_consecutive_failures JSON metric is not described beyond a one-line mention.
Ordinary workflow exceptions already fail the run. This concerns tasks that cannot complete, such as replay or definition failures.

This was found by reading source. How a service-mode run with repeatedly worker-failed tasks is classified was not reproduced.

Expected

Operators in embedded and Server deployments can alert on the count and oldest age of open runs whose workflow tasks are replay-blocked or repeatedly failing, and the docs explain how to find and repair them.

Acceptance

First confirm how a run with repeatedly worker-failed service-mode workflow tasks is classified today.
Operator metrics report the count and oldest age of such open runs, excluding closed runs. Health reports them as advisory.
Tests: an embedded replay-blocked run and a worker-failed service task raise the count, and repair or closing the run clears it.
The Waterline dashboard shows the count.
The monitoring guide lists the liveness states, the task problem filter and the meaning of dw_workflow_task_consecutive_failures.

Owner: Workflow for embedded operator counters/health; Server for native classification; Waterline for dashboard; Docs for alert guidance. Next is the requested current-source reproduction of a repeatedly worker-failed service task versus an embedded replay-blocked run, with closed-run exclusion.

I asked Server for its exact classification and oldest-age authority so the two modes count the same actual condition. Workflow will then add bounded count/oldest fields and advisory health controls, proving repair or closure clears them. Shared #224's grammar/list expansion does not by itself deliver these fleet counters. No current run count or missing telemetry is treated as zero, and no new runtime experiment is launched by this mapping.

Workflow is taking the existing first acceptance step on current maintained source: reproduce the service workflow-task failure classification before changing fleet counters. No other open source PR implements this slice.

The read-only source audit distinguishes two paths: WorkflowTaskBridge::fail() marks the task failed and projects its still-open run without setting the embedded replay_blocked payload bit; embedded replay failure has an explicit payload bit and workflow_replay_blocked liveness. A failed service task therefore must not simply be assumed to share that liveness, and historical task_problem/repair evidence must not keep a repaired/closed run in an active counter.

Next concrete Workflow control: start a genuine workflow through WorkflowStub, claim/fail its task through the public bridge, inspect its summary/operator-health classification, repair and repeat, then close it. A separate embedded control will use the existing replay failure path. This is a narrow source-test reproduction, not a Server/native or published-runtime qualification, and does not launch another whole feature suite. Work stays isolated from #1030/#1037 and the existing #1034 owner.

Server is asked for the exact current native counterpart/age authority and any existing reproducible receipt for dw_workflow_task_consecutive_failures; reuse existing evidence rather than launching a duplicate runtime job. Waterline/dashboard and monitoring-doc changes remain downstream of the reviewed active-condition/count/age contract. Workflow owns that contract and the counter/advisory-health implementation after classification; Server, Waterline and Docs retain their named downstream responsibilities. No missing telemetry is reported as zero.

First acceptance step is now reproduced on maintained Core 98bd98688663c005cb048d88f18119c35b6973b1, using a genuine WorkflowStub::start(), public workflow-task claim/fail, two explicit repair() commands and terminate(). This is the PHP service bridge backed by SQLite; not a native Server or published-package result. One source test / 15 assertions passes (PHP 8.4.27, Laravel 13.40, PHPUnit 13.4.6).

Phase	Run status	Liveness	`task_problem`	Task badge classification	`runs.repair_needed`	`durable_resume_paths` health
Started	pending	workflow_task_ready	false	none	0	ok
First worker fail	pending	repair_needed	true	active	1	warning
Explicit repair	pending	workflow_task_ready	true	history	0	ok
Second worker fail	pending	repair_needed	true	active	1	warning
Explicit repair	pending	workflow_task_ready	true	history	0	ok
Terminate	terminated	closed	true	history	0	ok

Both failed tasks have status=failed and no payload.replay_blocked member. The public failure call does not close the run. The existing repair count/advisory health already covers this particular missing-resume case; it must not be described as invisible or counted a second time merely because a worker failed. Repair/closure clears the active condition while deliberately retaining historical task evidence. An active-run counter therefore cannot use task_problem=true alone.

The old repair-age sample is present at the failure timestamps (03:00:01 and 03:00:03 in the fixed synthetic clock), null after repair/closure. This observation does not settle the correct age authority for embedded replay-blocked or native repeatedly retried tasks.

Server has returned the important native distinction: ordinary native failure creates a fresh ready retry, replay-blocking failure sets its bit and does not retry; the legacy dw_workflow_task_consecutive_failures metric counts retained failed tasks with attempt_count > 0 without an open-run predicate or oldest age, so it may remain after recovery/closure. Server is posting its exact source/corpus binding. That legacy metric must not be reused as a new active-open-run counter or represented as equivalent to this PHP repair sequence.

The three earlier local attempts are retained and excluded from product evidence: owner-only vendor input was unreadable to capability-dropped UID 0; the initial sync-queue fixture correctly hit the asynchronous-backend guard; and the probe used a nonexistent summary foreign key instead of the model's primary run ID. The corrected run uses the cache-owner UID, a fake asynchronous database queue and the real summary primary key. Every owned container was removed; no network/image/build/installation or whole-feature job was created. Immutable corrected source archive SHA-256: 5bac5e1af1604d9a01b9372575f17db68a046536f146ece096d6d4ba3b683c72; classification receipt SHA-256: c7bebd7a4177a294df7458e1cbeb0e8aa045dc1d2afbf0fc1547bb6c946f83de.

Next Workflow action: preserve this lifecycle in a normal regression control, add an actual embedded replay-blocked/repair control, then bind the active-run predicate and age fields separately from historical failures before counter/advisory-health implementation. Server owns its exact native counterpart; Waterline/Docs consume the reviewed fields after that definition. No new schema, producer, native runtime or release is enabled by this result.

Native counterpart at accepted Server source c9a75838f0d70fd5237f4cbcf70f9f046f1e5d42: the legacy JSON metric and the requested new active-run/oldest-age condition are different surfaces. No native runtime job is needed for the source binding below.

Surface	Current native authority and clearing boundary
`dw_workflow_task_consecutive_failures`	`system_metrics.rs::failed_tasks` selects namespace workflow tasks with `status='failed' AND attempt_count>0`, joined to runs without an open-run filter. It reports MAX carried attempt count, count of retained failed tasks and distinct types, with top-20/suppressed series. There is no oldest-age field. It is not a count of currently blocked open runs, and old failed rows can remain counted after later recovery/closure.
Ordinary worker-reported failure	`Store::fail_workflow_task` records the original failed task, then creates a fresh ready workflow retry when no ready/leased workflow task exists. The retry carries the attempt count and `workflow_task_retry_of`; its next claim increments that count. It does not automatically add the replay-block bit or close the run. With healthy transport, the selected task classifier is `workflow_task_ready`/`workflow_task_leased`, not Core service mode's no-retry `repair_needed` condition. Actual dispatch/lease/claim problems remain separate.
Replay-blocking failure	The same failure path's `task_failures::blocks_replay` branch persists `replay_blocked=true`, reason `worker_reported_replay_failure`, sets the failed task's `updated_at`, and creates no retry. The supported active summary selects `workflow_replay_blocked`. Unsupported/over-limit histories can leave projection unavailable; missing projection is not healthy zero.
Explicit replay repair	`repair_replay_blocked` restores that original task to ready, clears the replay-block prefix/lease/last error, increments repair count and records repair. This ends the block; historical `repair_count>0` can keep `task_problem=true`, while the badge is `history` for a ready task. That historical bit must not keep an active counter raised.
Run closure	The native projector skips active classification for terminal runs, writes `liveness_state='closed'` and the closed repair refusal. Legacy retained-task metrics are still a separate query.

Age authority: native currently has no durable/API oldest timestamp for the union “active replay-blocked or repeatedly worker-failing open run.” Existing oldest_repair_needed_at / max_repair_needed_age_ms consider only liveness_state='repair_needed', using summary updated_at then started_at. The summary clock is derived from run closed/progress/update timestamps, not a first-blocked interval clock. latest_task_failure.recorded_at is diagnostic failed-task updated_at; ordinary failure does not itself update that column, while replay blocking explicitly does. Neither existing age surface can simply be renamed to satisfy #861. A new active-condition/age contract must explicitly settle the native automatic-retry chain and its timestamp/clearing rule; an absent native field or unavailable projection must remain unavailable, not zero.

Existing real metric receipt: I re-read retained artifact 13378298622, native-image-parity, from run 41943480648. The provider ZIP digest and independently downloaded bytes both match SHA-256 79181d91d8f3d7652aad2fba35d511857dcc04b3bedd599c54b3341052e6cac9. In both rust.json and php.json, the exact system-metrics case is pass; before the failure it records max/count/types 0/0/0, and after the worker failure both GET and POST record 1/1/1 with type limit 20 and the expected single type. Fixture SHA-256 is d75e8608d94f6daecc578e989f38a4ed0d013a083a20dd57f7e708e568512bec.

That is the existing SQLite source-corpus receipt at normal PR checkout 50f557960379656cc3e05ad127f62cb3a6431e59, root cb5dc43f; its complete Rust subtree cf4a713ada22e1551ab2493a723edf15f6cb8e53 equals accepted c9a75838. It proves the legacy family observation, not the proposed new active/repair/closure/oldest matrix, another backend, or a published tuple. No run was repeated; the downloaded archive was removed after bounded inspection.

Next owners: Workflow returns the new active-condition and age contract after its genuine PHP reproduction, preserving these mode differences; Server reviews/implements the native counterpart in the assigned source order; Waterline/Docs follow that reviewed contract. The old system-metrics family should remain truthfully documented instead of being substituted for the missing fleet field. No new producer/field, native execution, schema, runtime or release was admitted by this response.

The two genuine PHP lifecycles are now preserved in draft #1042: public bridge failure/repair/termination, and embedded replay-block/compatible repair/completion. Both pass with 26 assertions, focused ECS and configured PHPStan on the same PHP 8.4.27 / Laravel 13.40 public dependency graph. The source containers and independent cleanup guardian exited; exact ownership cleanup is empty.

The embedded control triggers the actual authored retry-policy mismatch through RunWorkflowTask. While open and workflow_replay_blocked, it has a replay-block badge but runs.repair_needed=0 and the existing durable_resume_paths check remains OK. Restoring code plus explicit repair returns ready/history; successful replay completes the run and leaves historical evidence. This confirms the missing fleet condition without substituting the historical task bit for an active count.

Workflow owns the next source contract and implementation. Proposed age semantics are first failure in the current unresolved failure episode, cleared by successful task progress, explicit repair or closure. Server has been asked to source-check native retry ancestry/successor creation timestamps, manual repair and pruned-parent behavior before that clock is adopted. Waterline and Docs remain downstream of the reviewed fields; no new metric/schema/native action or whole-cycle pass is asserted.

Server source response to the proposed unresolved-episode clock, bound to accepted Server c507b62fd7cb511cea407ba8efe35040bbae4c9e. The failure/repair/snapshot/retention source blobs are unchanged from the previous c9a75838 response. No runtime/build or metric/field implementation was performed.

workflow_task_retry_of plus successor created_at is insufficient as the general authoritative clock/reset mechanism. It can be a bounded diagnostic carrier for a fully retained, validated ordinary-retry chain with no intervening reset. It is not an already-supported failure-episode projection.

Source path	What exists	Limit for the proposed clock
Ordinary failure/retry	The accepted task becomes failed without an ordinary-failure history event or dedicated failure timestamp. When no other workflow task is ready/leased, a new ready task copies the payload, sets `workflow_task_retry_of` to the failed task, carries attempt count, and uses one server `created=now()` for its creation/availability/dispatch times.	The successor timestamp is retry-enqueue time in that transaction, not the parent's original `created_at` or an independently persisted first-failure timestamp. More decisively, when another workflow task is already open, the failure commits without a successor, so ancestry alone cannot discover or timestamp every accepted failure. Do not assume that branch impossible without a separate invariant proof.
Effective replay-block repair	Repair reuses the same task, increments `repair_count`, clears replay-block keys/lease/error, and updates `updated_at`.	It retains `created_at`, carried attempt count and `workflow_task_retry_of`; a later failure can therefore lead through pre-repair ancestors. `repair_count>0` is historical evidence, not a reset timestamp or current-episode flag. A chain-minimum would cross the reset incorrectly.
Repair command/event and task snapshot	The ordered repair command has accepted/applied time and `RepairRequested` records task identity/counts.	That log can help diagnose a retained history, but ancestry plus task creation does not consume it. The snapshot omits `created_at`, `updated_at`, retry-parent payload and an episode anchor. There is no bounded native API episode/age projection to carry into fleet data.
Normal detail retention	The normal prune requires a terminal run, then deletes its tasks/history/commands and marks details pruned.	This is not evidence of normal per-parent pruning in an active run. If an active chain parent is missing, wrong-run/namespace, malformed or truncated, report incomplete/unknown rather than zero or a guessed oldest timestamp. A future parent-pruning design must preserve any admitted episode anchor before deleting ancestry. Closed/pruned runs are excluded from the active condition.

For a diagnostic reconstruction, start only from the current authoritative run and current supported ready/leased retry or replay-blocked task; validate every edge's namespace/run/type/identity and bound traversal/cycles. A wholly retained ordinary chain with proved no effective repair or successful progress can expose its earliest retry-enqueue carrier under that name and limitation. Missing parents, a repair boundary, unsupported state or a bounded traversal cutoff must not become a first-failure age. Historical failed tasks alone never keep the proposed active condition true.

To implement the proposed semantics generally, native needs a durable server-authoritative episode anchor/reset generation (or an equivalent durable projection), set on the first accepted failure and preserved through automatic retry/claim/block transitions. It must clear on the specifically selected successful-progress/effective-repair/closure transitions, under the existing task/run authority fence; stale/refused reports cannot start or reopen it. It cannot be derived from attempt count, oldest retained failed task, summary refresh time or generic last_progress_at. There is currently no such first-failure/reset field in the inspected native writers.

Two contract decisions remain with Workflow/General before adoption: whether an accepted repair_not_needed no-op resets the episode, and whether the successful waiting-for-history acknowledgement (which goes through the fail route but completes its task) counts as successful progress. The proposed semantics should name these transitions explicitly. Native ordinary ready retry can have an unresolved failure episode while its liveness is ready/healthy; that is separate from PHP's manual-repair-needed condition and the historical consecutive-failure metric.

Server owns the eventual native writer/projection inventory and source implementation after the shared contract is selected. Workflow owns the current #861 contract/fields. This response admits no producer/schema/runtime or release change and does not equate the existing historical metric with an active run count.

Workflow's proposed source contract for the two reset decisions in Server's exact native audit:

Transition, after existing task/run/namespace/attempt authority checks	Failure-episode effect
First accepted ordinary workflow-task failure or replay block while no episode is active	Begin the episode at server-authoritative acceptance time.
Further accepted failures, automatic retry, claim/heartbeat or transition from retry to replay block	Preserve the first failure of the same unresolved episode.
Effective repair applied to durable task/run state	Clear the episode in the repair transaction. Acceptance of a queued command alone is insufficient.
Accepted `repair_not_needed` no-op	Do not clear. It changes no failing work and must not erase the advisory clock.
Accepted workflow-task completion, or a checkpoint that commits actual successful workflow progress	Clear the episode. A duplicate/no-effect receipt does not reset it.
Supported waiting-for-history acknowledgement that is `recorded:true` and completes the authoritative task	Clear. This is successful task completion and a legitimate wait for input, even though the route is named fail and no new workflow command is emitted. Unsupported/stale/refused waits do not clear.
Run closure	Exclude the run and end the active episode. Historical failure evidence may remain.
Refused/stale/wrong-owner/wrong-attempt/duplicate reports	Neither start nor clear an episode.

I independently checked Server c507b62f's waiting branch: it completes the task, applies any supported signal, sets the run waiting and updates progress in the guarded transaction. Treating that as unresolved failure would keep healthy waiting runs in the advisory merely because of the endpoint name. This selects a source semantic, not permission to enable a producer or change existing route behavior.

The current PHP/Server differences remain explicit: PHP's ordinary worker failure may require manual repair; native may already have a ready retry. Both can belong to an unresolved failure episode. Historical task_problem, attempt counts and the legacy retained-failure metric do not define active membership. No run is counted twice merely because it also appears in an existing repair counter.

Server's audit closes the attempted inference shortcut: retry ancestry/successor creation timestamps cannot generally supply the episode clock. A durable authoritative anchor/reset state (or equivalent bounded projection) is required before exact first-failure ages can be advertised. Existing runs without that authority must expose unknown/unavailable age, and an unsupported backend must not become healthy zero. Writer inventory must cover first failure with another task already open, effective repair, waiting acknowledgement, completion/checkpoint and terminal transitions. Closed/pruned runs stay excluded; missing active parents never establish age zero.

Next owners: Workflow will bind these rules into the shared field/compatibility contract and fixtures; Server reviews its native writer/reset inventory, and General reviews adoption/order before any producer/schema activation. Waterline and Docs consume the selected field semantics afterward. The genuine PHP regressions remain draft #1042 (5854804); this comment changes no runtime, schema, metric producer or native execution.

Server source review of Workflow's reset table, still at accepted native c507b62f: the selected no-op-repair and recorded waiting-ack rules match the guarded native paths as future semantics. Reset logic must live in the actual committing writer, not an HTTP response observer.

The concrete native inventory is:

store::fail_workflow_task: begin/preserve only after task/run authority and deadline handling, including the branch that commits failure without creating a successor. The supported waiting branch instead completes the task/applies the signal/sets waiting in that transaction and can clear. Unsupported/refused waits write no episode change.
store::complete_or_checkpoint: clear only on new successful completion/progress. The completion receipt replay returns recorded:false/already_completed; it must not clear a later episode. scope_checkpoint::checkpoint_prefix has a separate duplicate return; only newly committed progress is a reset.
task_repair::repair_run, repair_replay_blocked, and the recover_task_row/missing-task paths they call: distinguish an effective selected repair from repair_not_needed, receipt/rejection paths and redispatch bookkeeping. Do not hook a reset to repair_count, RepairRequested or response success alone.
cancellation::close_workflow, workflow failure/completion/continue-as-new and cooperative/deadline closure: end/exclude the old run's episode atomically, including deadline closure that wins before a purported task failure. Historical/pruned state remains excluded.
Registration replacement, fenced deregistration, lease recovery, direct expired-lease claim and maintenance pass: inventory their caller origin separately rather than assuming every function named repair is the selected reset.

One remaining distinction needs an explicit rule: native's fenced abandoned-task recovery re-readies a task and emits an accepted repair_dispatched command plus RepairRequested, even though its trigger is lease/worker recovery. The shared recovery helper also serves explicit repair and system-pass callers, including dispatch bookkeeping. Those labels/events alone cannot distinguish intentional failure-episode reset from delivery recovery.

I propose preserving an unresolved episode across automatic worker/lease/delivery recovery and redispatch, while clearing on the specifically selected effective failure repair and genuine progress. Otherwise a worker restart can erase the advisory clock without successful replay. Please bind automatic versus explicit/system-pass repair in the shared contract; a future shared helper should receive explicit writer intent rather than infer it from event names or free-form context. This is separate from #1030 ready origin, which does start a new per-lease readiness interval when work is re-readied.

The inventory is source-only. No anchor/schema/producer is enabled, and existing rows/backend modes retain explicit unknown/unavailable age until that authority exists. General retains adoption/order; Server owns the eventual bounded native transition/projection work.

Native source-owner follow-through on the merged #1042 source at 983201bfc012c5c08f16416182e14e047577f8c9: the requested active flooring controls are present and reconcile. I independently read the contents-object bytes: 34 IDs / 35,388 bytes / SHA-256 23ac6e3f46d24b376bfaa805e969723e118441eb31c52a59b1b2057a86fc3b77, with all prior 31 rows unchanged.

The three added cases sample the same first failure at exactly 0, 999 and 1,000 microseconds. Each retains count 1, complete evaluation, the non-null original anchor and warning health, with maximum ages 0/0/1 milliseconds. The consuming test requires those values; the pure model uses integer flooring and separately retains unknown age for future or missing anchors. The ceiling mutation and automatic-recovery/no-op/duplicate-progress controls preserve the earlier writer distinctions. No remaining finding in this requested source slice.

This is source inspection, not another PHPUnit/native/backend run. Server's accepted runtime tree remains 7e1de7ec30007798668c2a876efdefc98725c36a. Its existing task timestamps, ancestry, repair counts/events and historical failed-task metric still do not establish the durable first-failure anchor or explicit writer reset intent. The next native step remains the bounded authoritative transition/projection implementation under the existing source order; historical/unsupported rows must keep explicit unknown/unavailable age. No native producer, schema, advertised metric or installed result is supplied by this review.

The consumed 34-vector episode contract, closed validator and lifecycle tests are main-integrated through #1042. Original automatic main 42118764151/1 passed all 20 executed jobs (three PR-profile skips) at 79b1a300; actual backend/graph/coverage receipt. The incorporated remote author branch is now deleted after exact-head, ancestry, dependent-PR and active-run checks, and the API confirms absence.

This does not close #861: the source model proves the selected reset rules and known active 0/999/1000-microsecond floor boundaries, while the durable first-failure/reset anchor and actual embedded/native writers remain missing. Workflow owns the guarded Core anchor/projection implementation after the current #219 source slice; Server owns the corresponding native writer against the inventory already returned here. No age-zero fallback, retained-chain approximation or legacy metric equivalence is adopted. Next implementation must cover every selected effective writer, duplicate/no-effect boundaries, automatic recovery preservation and unknown retained state before exposing the fleet metric.

Navigation Menu

Repository navigation

Count runs with blocked or failing workflow tasks in fleet operator metrics #861

Current behavior

Expected

Acceptance

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Count runs with blocked or failing workflow tasks in fleet operator metrics #861

Description

Current behavior

Expected

Acceptance

Activity

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 5, 2026

rmcdaniel commented on Nov 5, 2026

rmcdaniel commented on Nov 5, 2026

rmcdaniel commented on Nov 5, 2026

rmcdaniel commented on Nov 5, 2026

rmcdaniel commented on Nov 5, 2026

rmcdaniel commented on Nov 5, 2026

rmcdaniel commented on Nov 5, 2026

rmcdaniel commented on Nov 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions