Skip to content

ops(publish): temporarily force 50 workers#729

Merged
brokemac79 merged 1 commit into
mainfrom
codex/csw-049-temp-publication-50
Jul 21, 2026
Merged

ops(publish): temporarily force 50 workers#729
brokemac79 merged 1 commit into
mainfrom
codex/csw-049-temp-publication-50

Conversation

@brokemac79

Copy link
Copy Markdown
Contributor

Summary

Problem

After the emergency rollback in #728, the durable capacity controller remained at the four-worker ceiling written before the rollback. The controller had begun gradual recovery, but the incident backlog remained above 500 items and useful delivery was still below arrivals.

This is an operator-directed temporary throughput override intended to clear the incident backlog. It is not presented as a root-cause repair.

Implementation

Set these production Worker bindings to 50:

  • EXACT_REVIEW_PUBLICATION_MIN_CONCURRENT
  • EXACT_REVIEW_PUBLICATION_BASE_CONCURRENT
  • EXACT_REVIEW_PUBLICATION_MAX_CONCURRENT

Setting all three values is intentional: on deployment, the controller clamps the persisted four-worker ceiling and demand target to 50 without resetting or directly editing Durable Object state.

Validation

pnpm run build:dashboard
  passed

node --input-type=module -e <capacity assertion>
  capacity override proof: 50 at backlog 0/100/600

npx --yes wrangler@4.107.0 deploy --dry-run --config dashboard/wrangler.toml
  passed; reported publication min/base/max bindings as 50/50/50

git diff --check
git diff --cached --check
  passed

At the maintainer's explicit emergency direction, the normal Codex review loop was stopped and waived for this configuration-only incident override.

Risks and rollout

  • This admits 50 publication workflows but does not create 50 parallel state writers. fix(publish): coordinate every state branch writer #724 still serializes mutation of the single shared state ref.
  • The larger cohort can increase state-lease waiting, retries, and 180-second contention failures. Monitor useful published + superseded throughput, pending depth, retry amplification, and state-contention dead letters rather than treating workflow completion alone as recovery.
  • If useful delivery does not improve promptly, revert this override instead of increasing it further.

Rollback

Restore the prior adaptive values after the backlog clears or if contention worsens:

EXACT_REVIEW_PUBLICATION_MIN_CONCURRENT = "4"
EXACT_REVIEW_PUBLICATION_BASE_CONCURRENT = "24"
EXACT_REVIEW_PUBLICATION_MAX_CONCURRENT = "48"

No queue reset, replay, dead-letter mutation, gate change, lease-timeout change, or state-layout change is included.

Related: #725, #727, #728

@brokemac79
brokemac79 merged commit 360074f into main Jul 21, 2026
8 checks passed
@brokemac79
brokemac79 deleted the codex/csw-049-temp-publication-50 branch July 21, 2026 00:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant