Skip to content

Fix Redis crash loop from unhandled socket errors - #18

Merged
UsamaSadiq merged 3 commits into
foss-sandboxfrom
usama/fix-redis-crash-loop
Jul 13, 2026
Merged

Fix Redis crash loop from unhandled socket errors#18
UsamaSadiq merged 3 commits into
foss-sandboxfrom
usama/fix-redis-crash-loop

Conversation

@UsamaSadiq

@UsamaSadiq UsamaSadiq commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Description

Twenty pods crash-loop on the FOSS sandbox because the node-redis clients (session storage and cache storage) lack error handlers. When Memorystore (Valkey) drops idle TLS connections, node-redis emits an error event with no listener, which Node treats as fatal (exit 1). The .connect().catch() in session storage also re-throws, producing an unhandled rejection that breaks built-in auto-reconnect.

This patch fixes both the session-storage (connect-redis) and cache-storage (cache-manager-redis-yet) Redis clients:

  • Adds .on('error', ...) handler to both clients to prevent unhandled error crashes
  • Replaces re-throwing .catch() with log-only handler so auto-reconnect works (session client)
  • Switches cache storage from redisStore (which creates its own client without error handling) to redisInsStore with a pre-built client that has a proper error handler attached
  • Adds pingInterval: 30s to keep idle connections alive and avoid Memorystore reaping
  • Adds socket.keepAlive + keepAliveInitialDelay for TCP-level keepalive
  • Adds socket.reconnectStrategy with exponential backoff capped at 5s

The ioredis clients (BullMQ, pub/sub) are not affected -- they have built-in reconnect and error handling.

Testing

  • Rebuild Twenty Docker image and redeploy to sandbox
  • Confirm pod stays up past the previous crash window (~1-2 min after boot)
  • Wait 10+ minutes or kill the Redis connection from Memorystore side -- pod should log the error and reconnect, not crash
  • Check logs for Session Redis client error: / Cache Redis client error: lines (expected, non-fatal) instead of SocketClosedUnexpectedlyError stack traces

node-redis session and cache clients crash the process when
Memorystore drops idle TLS connections. Add error handlers,
pingInterval, TCP keepalive, and reconnect strategy to both
the session-storage and cache-storage Redis clients.
@UsamaSadiq
UsamaSadiq force-pushed the usama/fix-redis-crash-loop branch from 089c3b9 to 8645ab6 Compare July 9, 2026 12:10
Comment on lines +36 to +37
keepAlive: true,
keepAliveInitialDelay: 30_000,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[💡 Suggestion]: keepAlive: true isn't the right type for node-redis v4, and keepAliveInitialDelay is ignored

In @redis/client 1.6 (redis 4.7), socket.keepAlive is typed number | false, and the client applies it as socket.setKeepAlive(keepAlive !== false, keepAlive || 0) — it never reads keepAliveInitialDelay. So keepAlive: true fails tsc (TS2322), and at runtime the intended 30s delay is dropped (falls back to the OS keepalive default, ~2h). The SWC image build strips types so it won't break the Docker build, but the typecheck target will. Collapse to a single numeric value:

socket: {
  keepAlive: 30_000,
  reconnectStrategy: (retries: number) => Math.min(retries * 200, 5_000),
},

Not blocking the crash fix, pingInterval: 30_000 is what actually keeps the Memorystore connection warm, so the core fix holds regardless.

Comment on lines +62 to +63
keepAlive: true,
keepAliveInitialDelay: 30_000,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[💡 Suggestion]: Same keepAlive type/runtime issue here as in the cache factory

socket.keepAlive is number | false in @redis/client 1.6, and node-redis ignores keepAliveInitialDelay (it uses the numeric keepAlive as the delay). keepAlive: true fails tsc and doesn't apply the intended 30s delay at runtime. Use:

socket: {
  keepAlive: 30_000,
  reconnectStrategy: (retries: number) => Math.min(retries * 200, 5_000),
},

...cacheModuleOptions,
store: redisStore,
url: redisUrl,
store: redisInsStore(redisClient, {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[💡 Suggestion]: redisInsStore(redisClient, …) needs a RedisClientType cast to pass tsc

createClient() without functions/scripts infers generics that don't match redisInsStore's RedisClientType parameter (RedisFunctions vs Record<string, never>), so this line fails tsc with TS2345 (reproduced against the pinned redis 4.7 / cache-manager-redis-yet 4.1.2, and it persists even with a single @redis/client copy). SWC strips types so the Docker image build is unaffected, but the typecheck target will fail. Add a cast:

import { createClient, type RedisClientType } from 'redis';
// ...
store: redisInsStore(redisClient as RedisClientType, { ttl: cacheStorageTtl * 1000 }),

@NIXKnight NIXKnight left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix correctly resolves the crash loop: both redis clients now register an error listener before connect, and dropping the session re-throw restores auto-reconnect. Verified the node-redis v4.7 options and redisInsStore usage against the pinned versions. Approving.

- Use keepAlive: 30_000 (number) instead of keepAlive: true (boolean)
  since node-redis v4 types socket.keepAlive as number | false
- Remove keepAliveInitialDelay which node-redis ignores
- Cast redisClient as RedisClientType for redisInsStore compatibility
@UsamaSadiq
UsamaSadiq merged commit 333bd42 into foss-sandbox Jul 13, 2026
59 of 60 checks passed
@github-actions

Copy link
Copy Markdown
Fails
🚫

node failed.

Log

Details
�[31mError: �[39m SyntaxError: Unexpected token '<', "<!DOCTYPE "... is not valid JSON
    at JSON.parse (<anonymous>)
�[90m    at parseJSONFromBytes (node:internal/deps/undici/undici:4387:19)�[39m
�[90m    at successSteps (node:internal/deps/undici/undici:7041:27)�[39m
�[90m    at readAllBytes (node:internal/deps/undici/undici:5958:13)�[39m
�[90m    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)�[39m
danger-results://tmp/danger-results-ca709736.json

Generated by 🚫 dangerJS against 43106d4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants