fix(functions): report discovery crashes instead of blaming the timeout - #10911
fix(functions): report discovery crashes instead of blaming the timeout#10911IzaakGough wants to merge 1 commit into
Conversation
Function discovery polls the admin server until it answers, so a codebase that died during module load looked identical to one that was merely slow: both surfaced "User code failed to load. Cannot determine backend specification" after the full 10s budget, with the actual error nowhere in sight. Race the child's exit against the poll so a crash reports its own exit code and stderr immediately. The attendant listeners move to spawn time. Neither exit nor error replays, so a server that died before kill() ran left a listener that could never fire and a shutdown that never resolved. Also fixes the escape hatch people reach for when they hit this: - FUNCTIONS_DISCOVERY_TIMEOUT is read in seconds but the error printed milliseconds and named neither the variable nor the unit. Setting it to 30000 in the belief it meant 30s asked for 8.3 hours, disabling the timeout rather than extending it. Bare numbers still mean seconds, an explicit s or ms suffix is now accepted, and a suspiciously large bare value warns. - The timeout message now states the elapsed time in seconds and tells the user which variable to set. - The poll loop had no backoff, spinning at ~144k connection attempts per second and saturating a core taken from the process it was waiting on. It now sleeps 100ms between attempts, matching functionsRuntimeWorker. - The timeout timer is now cleared on the success path. Fixes #7775
There was a problem hiding this comment.
Code Review
This pull request improves function discovery error handling by reporting server crashes immediately instead of masking them as timeouts, and refactors FUNCTIONS_DISCOVERY_TIMEOUT parsing to support explicit unit suffixes. The review feedback highlights three key improvements: handling process spawn errors in serverExited to prevent potential hangs, utilizing the existing sleep utility from src/utils.ts per the style guide, and limiting the stderr buffer size to avoid potential memory issues.
| const serverExited = new Promise<never>((_resolve, reject) => { | ||
| childProcess.once("exit", (code, signal) => { | ||
| if (!armed) { | ||
| return; | ||
| } | ||
| const how = code === null ? `signal ${String(signal)}` : `code ${code}`; | ||
| const details = stderr.trim(); | ||
| reject( | ||
| new FirebaseError( | ||
| `User code failed to load. Cannot determine backend specification. ` + | ||
| `The functions process exited with ${how} before it could be analyzed.` + | ||
| (details ? `\n\n${details}` : ""), | ||
| ), | ||
| ); | ||
| }); | ||
| }); |
There was a problem hiding this comment.
If the child process fails to start/spawn (e.g., due to a system error like ENOENT or EACCES), it will emit an "error" event instead of an "exit" event. Since serverExited only listens to "exit", it will never reject on spawn errors, causing discovery to hang until the timeout. Add an "error" event listener to serverExited to reject immediately with the spawn error.
const serverExited = new Promise<never>((_resolve, reject) => {
childProcess.once("exit", (code, signal) => {
if (!armed) {
return;
}
const how = code === null ? `signal ${String(signal)}` : `code ${code}`;
const details = stderr.trim();
reject(
new FirebaseError(
`User code failed to load. Cannot determine backend specification. ` +
`The functions process exited with ${how} before it could be analyzed.` +
(details ? `\\n\\n${details}` : ""),
),
);
});
childProcess.once("error", (err) => {
if (!armed) {
return;
}
reject(
new FirebaseError(
`User code failed to load. Cannot determine backend specification. ` +
`The functions process failed to start: ${err.message}`,
{ original: err },
),
);
});
});| // timeout rather than extending it. | ||
| const SUSPICIOUS_SECONDS = 600; | ||
|
|
||
| const sleep = (ms: number): Promise<void> => new Promise((resolve) => setTimeout(resolve, ms)); |
There was a problem hiding this comment.
Use the existing sleep utility from src/utils.ts instead of redefining it, as per the repository style guide.
| const sleep = (ms: number): Promise<void> => new Promise((resolve) => setTimeout(resolve, ms)); | |
| import { sleep } from "../../../../utils"; |
References
- Look for existing utilities first: Before writing common helper functions (e.g., for logging, file system operations, promises, string manipulation), check src/utils.ts to see if a suitable function already exists. (link)
| let stderr = ""; | ||
| childProcess.stderr?.on("data", (chunk: Buffer) => { | ||
| stderr += chunk.toString(); | ||
| }); |
There was a problem hiding this comment.
Buffer stderr with a size limit to prevent potential memory issues or Out-Of-Memory (OOM) errors if the child process produces excessive logs.
| let stderr = ""; | |
| childProcess.stderr?.on("data", (chunk: Buffer) => { | |
| stderr += chunk.toString(); | |
| }); | |
| let stderr = ""; | |
| childProcess.stderr?.on("data", (chunk: Buffer) => { | |
| if (stderr.length < 64 * 1024) { | |
| stderr += chunk.toString(); | |
| } | |
| }); |
Fixes #7775 (and #5888)
Problem
User code failed to load. Cannot determine backend specificationis common and hard to act on. Four reasons:It says "timeout" even when your code crashed. Discovery polls the admin server until it answers. If the child dies while loading we just keep getting
ECONNREFUSEDuntil the budget runs out, so the real error never surfaces.detectFromOutputPathalready handles this; HTTP discovery didn't.FUNCTIONS_DISCOVERY_TIMEOUTis in seconds but the error printed milliseconds, and named neither the variable nor the unit:So the
=30000people pass around is 8.3 hours, not 30 seconds. It "works" by disabling the timeout.The poll loop had no backoff. Measured against a refused port: ~144k attempts/sec, ~110% of one core, taken from the process we're waiting on.
The timeout timer was never cleared on the success path.
Changes
detectFromPortraces the child's exit, so a crash reports its exit code and stderr.serveAdminreturns{ kill, serverExited }.kill()ran left a shutdown that never resolved. (Same bug as the Python one in Killed deploy orphans the Python discovery server (serving.py); orphans wedge and later deploys hang forever on connect ETIMEDOUT #10847.)FUNCTIONS_DISCOVERY_TIMEOUTacceptssandmssuffixes. Bare numbers still mean seconds, but a suspiciously large one now warns.functionsRuntimeWorker.clearTimeouton success.Before / after
Slow load:
Crash while loading, which previously showed the message above after a full 10s:
Testing
Unit tests for the env var parsing, the timeout message, and discovery ending on server exit. Existing
detectFromPorttests unchanged. Also verified end to end against a real emulator, both slow-loading and crashing codebases.