Repository navigation
Prometheus latency buckets on SQLite count every duration in every bucket #683
Description
Delivered — October 27, 2026
The source correction is published in Server 2.11.5, and actual installed SQLite HTTP acceptance passes the original two-second workflow / 1.5-second activity case. Both samples are excluded from the 0.1, 0.5 and 1-second buckets and included from 2.5 seconds upward; counts and sums are verified through the packaged Apache endpoint. Previous MySQL/PostgreSQL source qualification remains linked in the receipt. Cloud completed the acknowledged acceptance handoff and verified fixture cleanup.
Goal
On SQLite, GET /api/system/prometheus-metrics latency histograms (dw_workflow_run_latency_seconds and dw_activity_execution_latency_seconds) count each duration only in buckets whose bound is at or above it, as they already do on MySQL and PostgreSQL.
Current behavior
Published Server 2.11.1 on SQLite counts every duration in every bucket. A workflow that ran 2.673 s and an activity that ran 0.903 s both appear in the 0.1 bucket:
"latency_seconds": {"count": 1, "sum": 2.673,
"buckets": {"0.1": 1, "0.5": 1, "1": 1, "2.5": 1, "5": 1, "10": 1, "30": 1, "60": 1, "300": 1, "900": 1}}Every SQLite deployment that scrapes this endpoint therefore reports all runs and activities as finishing within 100 ms.
Cause
PrometheusMetricsSummary::selectLatencyBuckets binds each bound as a query parameter. Laravel binds PHP floats with PDO::PARAM_STR, so SQLite receives the text '0.1'. The duration expressions (duration_ms / 1000.0, and the julianday difference for activities) have no column affinity, so SQLite does not convert the text, and it orders every number before every text value. duration <= '0.1' is therefore always true.
Minimal reproduction in the published image:
docker run --rm --entrypoint php ghcr.io/durable-workflow/server@sha256:bf8bed13f5f809ee3425233180a581353eac4683adbee5c4ecbf63bb18bbd102 -r '
$p = new PDO("sqlite::memory:");
$p->exec("CREATE TABLE t(duration_ms INTEGER)"); $p->exec("INSERT INTO t VALUES (2000)");
$s = $p->prepare("SELECT SUM(CASE WHEN duration_ms / 1000.0 <= ? THEN 1 ELSE 0 END) FROM t");
$s->bindValue(1, 0.1, PDO::PARAM_STR); $s->execute(); var_dump($s->fetchColumn());'
# int(1): a 2 s duration counted in the 0.1 s bucketAcceptance
- On SQLite, a 2 s workflow run and a 1.5 s activity count in the
2.5bucket and above, and not in0.1,0.5or1. - MySQL and PostgreSQL results are unchanged.
- A regression test covers SQLite.
Found while porting the summary to the native server (#325). Native compares durations with numeric bounds and reports the cumulative histogram on every backend. Its parity fixture records this SQLite behavior as a reference defect, not a contract.
Activity
rmcdaniel commentedon Oct 23, 2026
Source correction delivered, publication pending
PHP correction #688 is
normally merged at
b4192a3472aae80e646e863c45840e52cd73e70f, identical to thequalified whole tree
23742b117a8e8acf8cfa1e92ebb30111996c2192from candidated7cc197e5. Sixteen candidate checks pass, with one explicit skip and zeroOPEN/CONFIRMED Sonar findings. The completed remote branch is verified absent.
The new metrics-route regression verifies a two-second workflow and 1.5-second
activity. All seven metrics cases pass with 111 assertions each on SQLite,
PostgreSQL 17.11 and MySQL 8.0.46, using PHP 8.3.36 and normal locked Core
2.11.1. The preceding selector fails the focused SQLite case and passes both
network counterparts. MySQL/PostgreSQL comparison SQL remains unchanged.
Final feature run 39891035128 passes 3,039 cases / 68,266 assertions, with 37
existing skips and six deprecations. Original 050688c / 39888653299 remains
failed: my new fixture's Carbon binding lost the fractional closing timestamp.
The fixture now preserves its intended 1.5 seconds. Assertions and production
correction are unchanged. Fresh main 39892778540 and all other applicable main
checks independently pass.
The issue remains open. The dependency is the qualified Server image/release
under shared #237 and its image security follow-through. It unblocks when a
release containing this correction is published and the SQLite histogram
regression passes against that immutable image. Source success alone does not
complete customer-visible delivery. Native/shared/frozen, schemas, dependencies
and package versions are outside this patch.
All 243 MiB of local dependencies/scratch, both DBs and their network were
retired at 02:42:27 UTC, before 03:15. Both DB exits are zero without OOM.
rmcdaniel commentedon Oct 27, 2026
Release-state correction (2026-10-27): immutable Server 2.11.5 source 721b283 contains the #688 merge b4192a3; exact ancestry was reverified. Protected release run https://github.com/durable-workflow/server/actions/runs/40478372197 completed successfully and published the immutable release. The former shared #237 / Security / publisher dependency is superseded: no Security review, credential action, or operator action is pending. This issue remains open only for focused installed acceptance against exact 2.11.5: on SQLite, exercise the accepted 2-second workflow and 1.5-second activity case and verify the endpoint excludes them from lower buckets and includes them from 2.5 seconds upward. Source tests and publication alone are not being substituted for that scenario.
rmcdaniel commentedon Oct 27, 2026
General acknowledged Cloud ownership of the remaining installed acceptance for this issue: run the unchanged SQLite 2-second workflow / 1.5-second activity histogram case against immutable published Server 2.11.5, authenticate the real Prometheus endpoint, and verify all cumulative buckets, count and sum. No conflicting General fixture is active for #683. Cloud will retain exact artifact/evidence and close only after the published case passes. This handoff does not change Rust ownership, product source, publisher authorization or live Cloud resources.
rmcdaniel commentedon Oct 27, 2026
Installed acceptance complete on published Server 2.11.5, exact GHCR index
sha256:f4f2ec64a6522ee18a31542a30554873dbaa2f5d3af3ccb52deb914591699c20, image configsha256:8522ae607faa0c406c9335bd25ed559847e1af141da7f7791731acd1dbeeb7f7, source721b2831975cc898cc39d7528c5ba7e8441fe744. PHP is 8.3.36, SQLite 3.46.1, and the image's three package-provenance receipts identify published Core 2.15.1 at07e0d0e1a1185bd0a3bbd4e3fb56511cc57ca9bc. The actual metrics implementation SHA256030d678b3fe0702d139ce2843a87ab97f907ed94792100010fc3037dbe6a895ematches that published Server tag.The unchanged published regression fixture was loaded into a fresh disposable SQLite database: one completed 2-second workflow and one completed 1.5-second activity, with the activity's fractional closing timestamp retained. The image's normal bootstrap and entrypoint started its packaged Apache HTTP runtime. The real
/api/system/prometheus-metricsendpoint returned 401 without a token and 200 with the fixture admin token and normal control-plane/worker headers.Both returned histogram series have sample count 1 and these exact cumulative buckets:
Workflow sum is 2 seconds. Activity sum is 1.499972 seconds from SQLite's Julian-day calculation, within 0.0001 seconds of the 1.5-second fixture. Queue, workflow type, activity type, schema and namespace also match. All 35 consumer assertions passed.
This completes the missing installed SQLite acceptance. The earlier source/regression and MySQL/PostgreSQL results remain their stated evidence; no new installed network-database matrix is claimed. No product source, package, release, publisher or live deployment changed.
General's acknowledged handoff was accepted by Cloud. The isolated fixture used UID1000, a read-only image filesystem, a private Docker network and no host ports. Its containers and network were removed and absence verified, with no OOM. Task database/scripts/logs are removed after this durable receipt; shared published images and other owners' resources are preserved. Closing #683 on the delivered source, published image and actual endpoint acceptance.