Skip to content

(MOT-3748) fix(harness): preserve registrations across engine reloads - #666

Merged
ytallo merged 6 commits into
mainfrom
fix/harness-1.6.5-reattach
Aug 1, 2026
Merged

(MOT-3748) fix(harness): preserve registrations across engine reloads#666
ytallo merged 6 commits into
mainfrom
fix/harness-1.6.5-reattach

Conversation

@ytallo

@ytallo ytallo commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Published Harness workers can lose their registered functions when the engine reloads another worker, causing calls to fail with function_not_found. The same release also had a startup incompatibility: its queue recovery option was sent to older published queue workers, so Harness exited during installation.

Technical details

  • Pin Harness and its workspace test crates to the reconnect-capable iii-sdk/iii-helpers 0.21.8 release.
  • Align the manifest with queue ^0.21.2.
  • Retry queue provisioning without redeliver_on_engine_restart when an older queue worker rejects that field, preserving startup compatibility while the queue upgrade is rolled out.
  • Prepare the patch release as Harness 1.6.7.
  • Validation: locked check, release build, 298 unit tests, and a local install smoke using the local Harness binary with published latest dependencies, including provider reload.

Fixes MOT-3748

Summary by CodeRabbit

  • New Features

    • Added automated Harness quickstart validation for eligible latest and next releases.
    • Added reusable workflow support with configurable release channels and required API authentication.
  • Bug Fixes

    • Improved queue setup compatibility by retrying with a legacy configuration when advanced settings are unsupported.
    • Updated Harness and queue components for reconnect and live span-start support.
  • Chores

    • Updated package and dependency versions across Harness components and tests.

@vercel

vercel Bot commented Aug 1, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
workers Ready Ready Preview Aug 1, 2026 3:43pm
workers-tech-spec Ready Ready Preview Aug 1, 2026 3:43pm

Request Review

@coderabbitai

coderabbitai Bot commented Aug 1, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The PR adds reusable Harness quickstart workflow inputs and secrets, connects eligible releases to Harness smoke testing, updates Harness dependencies, and adds queue provisioning fallback for engines that reject restart-redelivery configuration.

Changes

Harness release and runtime compatibility

Layer / File(s) Summary
Reusable quickstart contract
.github/workflows/harness-quickstart.yml
The workflow accepts a channel input and SLACK_BOT_TOKEN and ZAI_API_KEY secrets through workflow_call.
Release smoke-test orchestration
.github/workflows/release.yml
The release setup detects Harness workers and dependencies. Eligible successful latest and next releases invoke the reusable Harness quickstart workflow.
Queue fallback and dependency updates
harness/Cargo.toml, harness/iii.worker.yaml, harness/src/queue.rs, harness/tests/*
Harness dependencies and the queue requirement are updated. Queue provisioning retries with a legacy definition when the engine rejects redeliver_on_engine_restart. Tests cover fallback detection and the legacy payload.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ReleaseSetup
  participant RegistryPublication
  participant HarnessQuickstart
  ReleaseSetup->>ReleaseSetup: Detect Harness worker or dependency
  ReleaseSetup->>RegistryPublication: Publish release
  RegistryPublication->>HarnessQuickstart: Invoke quickstart with channel
Loading

Possibly related PRs

Poem

A rabbit saw queues hop anew,
With fallback paths when fields would fail.
Harness releases joined the trail,
Quickstarts carried channels too,
And fresh dependencies cleared the rail.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main fix: preserving Harness registrations when the engine reloads another worker.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/harness-1.6.5-reattach

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

skill-check — worker

0 verified, 51 skipped (no docs/).

Layer Result
structure
vale
ai
render

Four for four. Nicely done.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/release.yml:
- Around line 283-286: Update the reusable workflow invocation of
harness-quickstart.yml to replace secrets: inherit with an explicit secrets
mapping containing only SLACK_BOT_TOKEN and ZAI_API_KEY, preserving the existing
channel input.

In `@harness/src/queue.rs`:
- Around line 46-55: Update ensure_turn_queue around the bounded primary retry
loop so a legacy schema fallback triggered by is_legacy_queue_schema_error can
perform its own retry even when the primary DEFINE_ATTEMPTS budget is exhausted.
Ensure legacy_turn_queue_definition() is actually submitted before returning the
error, and add a regression test covering the schema error on the final primary
attempt and successful fallback provisioning.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 4fa84a8e-eb0b-41ef-96d6-64bd116f0b52

📥 Commits

Reviewing files that changed from the base of the PR and between 12700ae and 94218fc.

⛔ Files ignored due to path filters (1)
  • harness/Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (8)
  • .github/workflows/harness-quickstart.yml
  • .github/workflows/release.yml
  • harness/Cargo.toml
  • harness/iii.worker.yaml
  • harness/src/queue.rs
  • harness/tests/e2e/Cargo.toml
  • harness/tests/integration/Cargo.toml
  • harness/tests/manifest.rs

Comment on lines +283 to +286
uses: ./.github/workflows/harness-quickstart.yml
with:
channel: ${{ needs.setup.outputs.registry_tag }}
secrets: inherit

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Pass only the declared secrets to the reusable workflow.

secrets: inherit exposes every secret available to the release job. The quickstart contract requires only SLACK_BOT_TOKEN and ZAI_API_KEY. Pass those names explicitly.

Proposed fix
     with:
       channel: ${{ needs.setup.outputs.registry_tag }}
-    secrets: inherit
+    secrets:
+      SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
+      ZAI_API_KEY: ${{ secrets.ZAI_API_KEY }}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
uses: ./.github/workflows/harness-quickstart.yml
with:
channel: ${{ needs.setup.outputs.registry_tag }}
secrets: inherit
uses: ./.github/workflows/harness-quickstart.yml
with:
channel: ${{ needs.setup.outputs.registry_tag }}
secrets:
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
ZAI_API_KEY: ${{ secrets.ZAI_API_KEY }}
🧰 Tools
🪛 zizmor (1.28.0)

[warning] 283-283: secrets unconditionally inherited by called workflow (secrets-inherit): this reusable workflow

(secrets-inherit)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/release.yml around lines 283 - 286, Update the reusable
workflow invocation of harness-quickstart.yml to replace secrets: inherit with
an explicit secrets mapping containing only SLACK_BOT_TOKEN and ZAI_API_KEY,
preserving the existing channel input.

Source: Linters/SAST tools

Comment thread harness/src/queue.rs
Comment on lines +46 to +55
if restart_redelivery_requested && is_legacy_queue_schema_error(&last_error) {
tracing::warn!(
queue = TURN_QUEUE,
error = %last_error,
"queue worker does not support restart redelivery; retrying with the legacy queue schema"
);
payload = legacy_turn_queue_definition();
restart_redelivery_requested = false;
continue;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Allow the fallback to consume a retry after the primary budget is exhausted.

If the unsupported-field error occurs on the final DEFINE_ATTEMPTS iteration, this branch changes payload and executes continue, but the bounded loop has no next iteration. ensure_turn_queue then returns the schema error without sending legacy_turn_queue_definition().

The supplied harness/src/main.rs flow waits for queue provisioning before it announces readiness. Give the legacy fallback its own retry budget, and add a regression test that returns the schema error on the final primary attempt.

Also applies to: 132-162

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@harness/src/queue.rs` around lines 46 - 55, Update ensure_turn_queue around
the bounded primary retry loop so a legacy schema fallback triggered by
is_legacy_queue_schema_error can perform its own retry even when the primary
DEFINE_ATTEMPTS budget is exhausted. Ensure legacy_turn_queue_definition() is
actually submitted before returning the error, and add a regression test
covering the schema error on the final primary attempt and successful fallback
provisioning.

@ytallo
ytallo merged commit 4b056e5 into main Aug 1, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant