Skip to content

fix(functions): report discovery crashes instead of blaming the timeout - #10911

Open
IzaakGough wants to merge 1 commit into
mainfrom
@invertase/fix-issue-7775
Open

fix(functions): report discovery crashes instead of blaming the timeout#10911
IzaakGough wants to merge 1 commit into
mainfrom
@invertase/fix-issue-7775

Conversation

@IzaakGough

@IzaakGough IzaakGough commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Fixes #7775 (and #5888)

Problem

User code failed to load. Cannot determine backend specification is common and hard to act on. Four reasons:

  1. It says "timeout" even when your code crashed. Discovery polls the admin server until it answers. If the child dies while loading we just keep getting ECONNREFUSED until the budget runs out, so the real error never surfaces. detectFromOutputPath already handles this; HTTP discovery didn't.

  2. FUNCTIONS_DISCOVERY_TIMEOUT is in seconds but the error printed milliseconds, and named neither the variable nor the unit:

    FUNCTIONS_DISCOVERY_TIMEOUT=1   ->   "Timeout after 1000"
    

    So the =30000 people pass around is 8.3 hours, not 30 seconds. It "works" by disabling the timeout.

  3. The poll loop had no backoff. Measured against a refused port: ~144k attempts/sec, ~110% of one core, taken from the process we're waiting on.

  4. The timeout timer was never cleared on the success path.

Changes

  • detectFromPort races the child's exit, so a crash reports its exit code and stderr. serveAdmin returns { kill, serverExited }.
  • Exit listeners move to spawn time. Neither event replays, so a server that died before kill() ran left a shutdown that never resolved. (Same bug as the Python one in Killed deploy orphans the Python discovery server (serving.py); orphans wedge and later deploys hang forever on connect ETIMEDOUT #10847.)
  • FUNCTIONS_DISCOVERY_TIMEOUT accepts s and ms suffixes. Bare numbers still mean seconds, but a suspiciously large one now warns.
  • Timeout message names the variable and uses seconds.
  • 100ms backoff between polls, raced against the timeout. Matches functionsRuntimeWorker.
  • clearTimeout on success.

Before / after

Slow load:

- Timeout after 10000. See https://...'
+ Timed out after 10s. If your code is slow to load, set FUNCTIONS_DISCOVERY_TIMEOUT
+ to allow more time (in seconds, e.g. FUNCTIONS_DISCOVERY_TIMEOUT=60). See https://...

Crash while loading, which previously showed the message above after a full 10s:

+ The functions process exited with code 134 before it could be analyzed.
+
+ FATAL ERROR: JavaScript heap out of memory

Testing

Unit tests for the env var parsing, the timeout message, and discovery ending on server exit. Existing detectFromPort tests unchanged. Also verified end to end against a real emulator, both slow-loading and crashing codebases.

Function discovery polls the admin server until it answers, so a codebase that
died during module load looked identical to one that was merely slow: both
surfaced "User code failed to load. Cannot determine backend specification"
after the full 10s budget, with the actual error nowhere in sight. Race the
child's exit against the poll so a crash reports its own exit code and stderr
immediately.

The attendant listeners move to spawn time. Neither exit nor error replays, so
a server that died before kill() ran left a listener that could never fire and
a shutdown that never resolved.

Also fixes the escape hatch people reach for when they hit this:

- FUNCTIONS_DISCOVERY_TIMEOUT is read in seconds but the error printed
  milliseconds and named neither the variable nor the unit. Setting it to 30000
  in the belief it meant 30s asked for 8.3 hours, disabling the timeout rather
  than extending it. Bare numbers still mean seconds, an explicit s or ms suffix
  is now accepted, and a suspiciously large bare value warns.
- The timeout message now states the elapsed time in seconds and tells the user
  which variable to set.
- The poll loop had no backoff, spinning at ~144k connection attempts per second
  and saturating a core taken from the process it was waiting on. It now sleeps
  100ms between attempts, matching functionsRuntimeWorker.
- The timeout timer is now cleared on the success path.

Fixes #7775

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request improves function discovery error handling by reporting server crashes immediately instead of masking them as timeouts, and refactors FUNCTIONS_DISCOVERY_TIMEOUT parsing to support explicit unit suffixes. The review feedback highlights three key improvements: handling process spawn errors in serverExited to prevent potential hangs, utilizing the existing sleep utility from src/utils.ts per the style guide, and limiting the stderr buffer size to avoid potential memory issues.

Comment on lines +296 to +311
const serverExited = new Promise<never>((_resolve, reject) => {
childProcess.once("exit", (code, signal) => {
if (!armed) {
return;
}
const how = code === null ? `signal ${String(signal)}` : `code ${code}`;
const details = stderr.trim();
reject(
new FirebaseError(
`User code failed to load. Cannot determine backend specification. ` +
`The functions process exited with ${how} before it could be analyzed.` +
(details ? `\n\n${details}` : ""),
),
);
});
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If the child process fails to start/spawn (e.g., due to a system error like ENOENT or EACCES), it will emit an "error" event instead of an "exit" event. Since serverExited only listens to "exit", it will never reject on spawn errors, causing discovery to hang until the timeout. Add an "error" event listener to serverExited to reject immediately with the spawn error.

    const serverExited = new Promise<never>((_resolve, reject) => {
      childProcess.once("exit", (code, signal) => {
        if (!armed) {
          return;
        }
        const how = code === null ? `signal ${String(signal)}` : `code ${code}`;
        const details = stderr.trim();
        reject(
          new FirebaseError(
            `User code failed to load. Cannot determine backend specification. ` +
              `The functions process exited with ${how} before it could be analyzed.` +
              (details ? `\\n\\n${details}` : ""),
          ),
        );
      });
      childProcess.once("error", (err) => {
        if (!armed) {
          return;
        }
        reject(
          new FirebaseError(
            `User code failed to load. Cannot determine backend specification. ` +
              `The functions process failed to start: ${err.message}`,
            { original: err },
          ),
        );
      });
    });

// timeout rather than extending it.
const SUSPICIOUS_SECONDS = 600;

const sleep = (ms: number): Promise<void> => new Promise((resolve) => setTimeout(resolve, ms));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Use the existing sleep utility from src/utils.ts instead of redefining it, as per the repository style guide.

Suggested change
const sleep = (ms: number): Promise<void> => new Promise((resolve) => setTimeout(resolve, ms));
import { sleep } from "../../../../utils";
References
  1. Look for existing utilities first: Before writing common helper functions (e.g., for logging, file system operations, promises, string manipulation), check src/utils.ts to see if a suitable function already exists. (link)

Comment on lines +288 to +291
let stderr = "";
childProcess.stderr?.on("data", (chunk: Buffer) => {
stderr += chunk.toString();
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Buffer stderr with a size limit to prevent potential memory issues or Out-Of-Memory (OOM) errors if the child process produces excessive logs.

Suggested change
let stderr = "";
childProcess.stderr?.on("data", (chunk: Buffer) => {
stderr += chunk.toString();
});
let stderr = "";
childProcess.stderr?.on("data", (chunk: Buffer) => {
if (stderr.length < 64 * 1024) {
stderr += chunk.toString();
}
});

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Issue with Global googleapis Import Causing Firebase Functions to Fail Locally with "Cannot Determine Backend Specification" Error

2 participants