-
Notifications
You must be signed in to change notification settings - Fork 48
fix(wizard): retry skill download body reads and stop losing tasks to timeouts #1170
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -111,8 +111,11 @@ export async function fetchSkillMenu( | |
| const menuUrl = `${skillsBaseUrl}/skill-menu.json`; | ||
| try { | ||
| logToFile(`fetchSkillMenu: fetching from ${menuUrl}`); | ||
| const resp = await fetchWithRetry(menuUrl, opts); | ||
| const data = (await resp.json()) as SkillMenu; | ||
| const data = (await fetchWithRetry( | ||
| menuUrl, | ||
| (resp) => resp.json(), | ||
| opts, | ||
| )) as SkillMenu; | ||
| for (const [category, entries] of Object.entries(data.categories)) { | ||
| data.categories[category] = entries.flatMap(expandBundleEntry); | ||
| } | ||
|
|
@@ -179,13 +182,23 @@ function extractBundle( | |
| return written; | ||
| } | ||
|
|
||
| /** | ||
| * Per-attempt budget for a skill download. Skill bundles are multi-megabyte, so | ||
| * a slow connection needs longer to stream the body than the default menu/prompt | ||
| * budget allows. | ||
| */ | ||
| const DOWNLOAD_TIMEOUT_MS = 120000; | ||
|
|
||
| /** Download a URL to a buffer, retrying transient failures with backoff. */ | ||
| async function downloadWithRetry( | ||
| url: string, | ||
| opts: RetryOpts = {}, | ||
| ): Promise<Uint8Array> { | ||
| const resp = await fetchWithRetry(url, opts); | ||
| return new Uint8Array(await resp.arrayBuffer()); | ||
| return fetchWithRetry( | ||
| url, | ||
| async (resp) => new Uint8Array(await resp.arrayBuffer()), | ||
| { timeoutMs: DOWNLOAD_TIMEOUT_MS, ...opts }, | ||
| ); | ||
|
Comment on lines
+197
to
+201
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nested retries can hold one task for 12 minutesWhy we think it's a valid issue
Issue descriptionEach download now allows three 120-second attempts. The orchestrator then retries a task that throws after these attempts. A stalled body can therefore hold one task for about 12 minutes before it fails. A task with multiple skills can wait longer. The final task waits for this work, so the whole wizard can appear stuck. Suggested fixDo not apply the queue retry after Prompt to fix with AI (copy-paste)
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirmed the arithmetic and the topology, and I'm escalating this for a human rather than applying the suggested fix. The suggested remedy — make an exhausted skill download terminal and skip the queue-level requeue — directly undermines what this PR exists to do: survive a transient mid-body stall by retrying. Removing the task-level second chance kills resilience against the exact failure this change targets, which the review note itself flags. So I won't implement it. The valid signal is real: this PR doubled the worst-case download hang. The per-attempt download budget went from 60s to 120s (deliberate — skill bundles are multi-megabyte), so a fully-stalled connection can now hold one task ~12 minutes across its two attempts instead of ~6. The nesting itself (3 fetch attempts under a task-level retry) predates this PR. The candidate mitigations are design decisions, not contained fixes safe to make unattended:
A human should decide whether to accept the doubled worst case as the price of the retry resilience this PR adds, or invest in one of those levers. No code was changed. |
||
| } | ||
|
|
||
| /** How to place a skill and what triages it — `triage` is stated by every caller so none inherits a silent default. */ | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The install recovery text is never shown
Why we think it's a valid issue
src/lib/agent/runner/sequence/orchestrator/orchestrator-runner.ts:986-989takes — the executor's catch, the queue's failure record, the end-of-drain verdict, and the abort renderer.executor.ts:53-61catches the throw, writes it to the debug log, and captures it as an exception; it never writes the message onto the task. The task is therefore stillRunning, soexecutor.ts:66-75requeues it while attempts remain and then callsstore.fail(task.id, { type: 'no-report', message: 'Task ended without calling complete_task.' }). That literal replaces the install cause inQueuedTask.error(queue.ts:87,queue.ts:343).verdict.requiredFailedTypesand the blocked count (orchestrator-runner.ts:1118-1135). It producesThe wizard was unable to set up PostHog: the <types> step failed. Please report this to: <email>. It passes nodetail, andwizard-abort.ts:110-117setserrorDetailonly whendetailis given. So the writable-directory hint never reaches the screen.Skill "<id>" for task "<type>" could not be installed (...)plus the same hint, and it died in the same catch. The PR changed only the wording, and it addedresult.messageto the log line (orchestrator-runner.ts:972-976), so local diagnosis got better, not worse.executor.ts:57-60captures the real exception withstep: 'orchestrator_run_task'andtask_type, and the log line carries the variant id and the underlying message.consider. The run still fails loudly rather than proceeding blind, the true cause is captured in analytics and the log, and only a self-service hint for one narrow cause is lost. The suggested remedy spans three modules — a typed error in the queue, executor storage, and cause-specific abort copy — which is out of proportion to a pre-existing copy gap on a retry-fix PR.Issue description
runOnecatches this error and leaves the task in theRunningstate. After retries end, the executor replaces the cause withno-report. The final abort only states that a step failed. The user never receives useful network or permissions guidance.Suggested fix
Use a typed install error. Have the executor store
skill-install-failedwith a safe cause while preserving task retries. After the last attempt, makewizardAbortshow stable, cause-specific guidance. Keep skill and task IDs in event properties so exception grouping stays stable.Prompt to fix with AI (copy-paste)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Confirmed this is a real gap at the current head, and it's the pre-existing behavior — this PR only made the throw message constant (for error-tracking grouping) and didn't touch the display path, so local diagnosis got slightly better, not worse.
What happens today: the
A required skill could not be installed …throw is caught by the executor, which captures it for analytics but doesn't record the cause on the task. Because the task is still in the Running state, it's retried and then failed with the genericTask ended without calling complete_task.reason, which overwrites the install cause. The end-of-drain abort screen is then built only from the failed step types and passes nodetail, and the abort renderer only shows extra detail whendetailis provided — so the 'check that the project directory is writable' hint never reaches the user. The sentence kept in the throw is effectively dead copy for the user (the true cause is still captured in analytics and the debug log).I'm escalating rather than fixing it here. A correct fix means (1) preserving a typed install-failure cause through the queue/executor instead of letting the
no-reportfailure overwrite it, and (2) rendering cause-specific recovery copy in the abort, threaded throughdetail. That's a cross-module design change (a new typed error carried through three modules), it's outside this PR's retry/timeout scope, and its only real proof is running the wizard against an unwritable install directory to see the outro — not something to land unattended on this PR.Decision for a human: either track this as a separate, focused change (surface the install cause on the abort screen), or accept the current behavior — the run still fails loudly and the true cause is in analytics and the log; only the one self-service hint for the unwritable-directory case is lost.