Data Badly Delayed

[ryanhamilton]

Data Badly Delayed
Happens cross-dashboard when large number of simultaneous users.
Opening this issue to record some investigation and implemented solutions

[ryanhamilton]

1. Specific popup for panel that is causing the delay.
   https://github.com/user-attachments/assets/d7f04f49-64fc-42a2-9d28-af3b36f64249

[ryanhamilton]

So there's logic for querying every server to get a server tree.

The scenario.

1. Every HTTP handler ran on Netty's event-loop threads, because no thread selection was configured and Micronaut's default is to leave handlers there.
2. Two handlers make blocking kdb calls with no timeout: One is the server-tree endpoint
3. WebSocket sessions are pinned to loop threads, so every dashboard whose session happens to share that thread stops getting pings and results.
4. Enough stuck loop threads and the whole instance shows the badly-delayed banner, while a plain page request still returns fine because it lands on an unblocked thread.
5. The change moves all handlers to the IO worker pool with one config key, so a hung call holds an IO thread instead and the loop keeps serving sessions.

DAnger now is that threads build up while blocked but next I'll put a deadline on them.

[ryanhamilton]

OK now kdb queries timeout after 2 minutes. So it doesn't freeze on one bad server forever.
2 Minutes is a long wait but it now shows the error should also help.
There is a shorter 10 seconds wait on getting a handle.

[ryanhamilton]

Panels made more independent AND do not hold lock during querying.
Previously they held a lock while querying and other panels would wait for them to return result.
Now they report stale while querying happening and move onto their next query.

panels-independent.mp4

[ryanhamilton]

1. The outage is reproduced and fixed in code. We built a harness that drives 45 dashboards for 29 users against frozen sandboxes and silent servers. On the current build it reproduced the wipeout: 20 of 32 dashboards that never touch a bad source went badly delayed. On the fixed build it is 0 of 32, and dashboards that do touch a bad source freeze once for one deadline and then recover.
2. Three causes, three fixes, all in the next release. Blocking HTTP calls ran on the network threads that serve WebSockets, so one hung sandbox login froze every user sharing that thread. The kdb driver had no timeout anywhere. And the shared query cache ran each database read under a lock that unrelated dashboards could queue behind.
3. New settings, all environment variables, no redeploy to change: KDB_CONNECT_TIMEOUT_MS default 10 seconds for connect and login, KDB_QUERY_TIMEOUT_MS default 120 seconds for a response, KDB_TIMEOUT_BACKOFF_MS default 30 seconds during which a server that timed out is skipped. Zero disables each. Setting KDB_TIMEOUT_BACKOFF_MS=0 is the rollback for the skip behaviour.
4. What users will see instead of a frozen dashboard: a panel waiting more than 10 seconds shows "Waiting for Sandbox A, 24s" on the panel; a timed-out panel shows the timeout error naming the server; during the skip window it shows "Skipping Sandbox A for 30s after a timeout". Other panels keep refreshing.
5. A query longer than 120 seconds is now cut and re-issued, and q keeps computing the abandoned one. If any legitimate PROD query runs that long, the default must go up, which is why question 1 below matters before release.
6. Appearance? #149 explains the pool errors in the September log; closing a dashboard no longer closes everyone's pools.

Questions

1. Query durations from the query log, gateway side and sandbox side if available: p99 and maximum for legitimate dashboard queries over a normal week. This decides whether 120 seconds is safe and whether 30 seconds of skip is tolerable on PROD.
2. A thread dump during the next hang on the current build, jcmd Thread.print or kill -3, plus the 10 minutes of log around it. It confirms the event-loop diagnosis on their hardware rather than ours.
3. Core count of the Pulse host. Their thread names imply 90-plus network threads, which means about 48 cores or an explicit setting.
4. Are sandboxes shared between users, and are they placed on the same dashboards as PROD panels? Our model assumes yes.
5. When a sandbox misbehaves, how? Busy computing, sitting in debug mode, or the process dead. Debug mode blocks even the login and is the case where the 10-second path applies.
6. Gateway handle policy: closes after one query, after idle time, or both, and the values. Decides whether close-after-use stays the default.
7. F5 behaviour on a dropped connection: reset, or black-holed. Black-holing is what the deadlines exist for.
8. Any dashboards using kdb subscriptions? They are exempt from the read deadline by design.
9. How many servers are registered in Pulse, gateways plus sandboxes, roughly.