[superlog] Use warn instead of error for AI tool call failures in request logger#487
[superlog] Use warn instead of error for AI tool call failures in request logger#487superlog-app[bot] wants to merge 1 commit into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
2 Skipped Deployments
|
|
The latest updates on your projects. Learn more about Unkey Deploy
|
|
Found 7 test failures on Blacksmith runners: Failures
|
Greptile SummaryThis PR fixes a false-positive alerting problem where AI insights jobs completing successfully were being emitted at ERROR level whenever any tool call (e.g. invalid SQL generation, wrong UUID passed to
Confidence Score: 5/5Safe to merge — the change is minimal and well-scoped, touching only one code path in the tool logger's request-scoped error handler. The change is a single-line swap that correctly addresses false-positive ERROR emissions without touching any job-failure paths. The fallback ( No files require special attention — the single changed file has a well-contained, easy-to-verify modification. Important Files Changed
Sequence Diagram%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant Tool as AI Tool
participant TL as createToolLogger
participant RL as RequestLogger (evlog)
participant Emit as logger.emit()
Note over Tool,Emit: Before this PR (tool call fails mid-job)
Tool->>TL: toolLogger.error("invalid SQL", ctx)
TL->>RL: requestLogger.error(new Error(msg), ctx)
RL-->>RL: "hasError = true"
Note over RL: job succeeds, catch block not hit
RL->>Emit: "level = hasError ? error : ..."
Emit-->>Emit: emits at ERROR level
Note over Tool,Emit: After this PR (tool call fails mid-job)
Tool->>TL: toolLogger.error("invalid SQL", ctx)
TL->>RL: requestLogger.warn(msg, ctx)
RL-->>RL: "hasWarn = true"
Note over RL: job succeeds, catch block not hit
RL->>Emit: "level = hasError ? error : hasWarn ? warn : info"
Emit-->>Emit: emits at WARN level
Note over Tool,Emit: Actual job failure (unchanged)
Tool->>TL: toolLogger.error(...)
TL->>RL: requestLogger.warn(...)
Note over RL: catch block in jobs.ts fires
RL->>Emit: logger.error(err)
RL-->>RL: "hasError = true"
Emit-->>Emit: emits at ERROR level
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant Tool as AI Tool
participant TL as createToolLogger
participant RL as RequestLogger (evlog)
participant Emit as logger.emit()
Note over Tool,Emit: Before this PR (tool call fails mid-job)
Tool->>TL: toolLogger.error("invalid SQL", ctx)
TL->>RL: requestLogger.error(new Error(msg), ctx)
RL-->>RL: "hasError = true"
Note over RL: job succeeds, catch block not hit
RL->>Emit: "level = hasError ? error : ..."
Emit-->>Emit: emits at ERROR level
Note over Tool,Emit: After this PR (tool call fails mid-job)
Tool->>TL: toolLogger.error("invalid SQL", ctx)
TL->>RL: requestLogger.warn(msg, ctx)
RL-->>RL: "hasWarn = true"
Note over RL: job succeeds, catch block not hit
RL->>Emit: "level = hasError ? error : hasWarn ? warn : info"
Emit-->>Emit: emits at WARN level
Note over Tool,Emit: Actual job failure (unchanged)
Tool->>TL: toolLogger.error(...)
TL->>RL: requestLogger.warn(...)
Note over RL: catch block in jobs.ts fires
RL->>Emit: logger.error(err)
RL-->>RL: "hasError = true"
Emit-->>Emit: emits at ERROR level
|
Summary
Insights jobs that completed successfully were being logged at
ERRORlevel whenever any AI tool call failed during execution (e.g. the agent generating invalid SQL, orcreate_annotationreceiving a domain name instead of a UUID). This produced false-positive ERROR incidents even though the job recovered and generated insights successfully.The root cause is in
createToolLogger.error()inpackages/ai/src/ai/tools/utils/logger.ts. When a tool fails, it calledrequestLogger.error(err, context). In evlog,RequestLogger.error()permanently setshasError = trueon the request-scoped logger. The finallogger.emit()call then useslevel = hasError ? "error" : hasWarn ? "warn" : "info", so even ajob_status: "succeeded"event was emitted at ERROR level.The fix changes the request-logger call from
requestLogger.error(err, context)torequestLogger.warn(message, context). Tool-level failures sethasWarn = trueinstead ofhasError = true, so succeeded jobs are emitted atwarnorinfolevel. Actual job failures (caught injobs.tscatch block, which explicitly callslogger.error(err)) remain at ERROR level. The error details (message, SQL, context) are still captured in the wide event via the merged context parameter.Alternative: use
requestLogger.info()if even WARN-level events for jobs-with-tool-failures are unwanted. That would produce INFO regardless of tool failures, at the cost of slightly less signal on partial failures.Incident on Superlog
Was this PR helpful? Leave feedback — goes straight to the Superlog team.
Summary by cubic
Downgraded tool-call failures from ERROR to WARN in the request logger to stop successful insight jobs from emitting ERROR-level events and triggering false incidents.
packages/ai/src/ai/tools/utils/logger.ts, changedrequestLogger.error(err, ...)torequestLogger.warn(message, ...)increateToolLogger.error.Written for commit 8c73ced. Summary will update on new commits.