https://boristane.com/blog/on-call-is-now-theatre/

[utterances-bot]

On-Call Is Now Theatre | Boris Tane

On-call was calibrated to the cost of waking up a human. That cost just went to zero. Self-operating software is coming, and your rotation is the last thing standing in its way.

https://boristane.com/blog/on-call-is-now-theatre/

[dhftah]

(Discovered via the always compelling Lorin Hochstein

I'm not sure I'm fully on board with this just yet. I agree these large language models have gotten fundamentally better in their ability to reason through what's wrong with a production system, and that reasoning capability goes up significantly with an amount of data, the data being well constructed in a way the LLM can reason about (e.g. a temporal knowledge graph) and tools to gather context from other systems (read: GitHub).

However. LLMs aren't free, and putting them in front of ${N} alerts is expensive. There needs to be a trigger condition that's still anchored in the customer experience (the highest signal I haven't yet had a chance to implement being meaningful availability on top of critical business operations. There, I agree, we can do an awful lot before a human is notified.

Release is also an interesting checkpoint — IME the absolute majority of incidents are triggered by change, and change is a far more infrequent condition (at least driven by engineers / pull requests) than more transient behaviors such as infrastructure reshuffling or customer behavior change that tend otherwise to cause these problems.

That said, I'm confident there will still need to be a human on-call, or at least, reviewing these transcripts. If for no other reason than when there are problems the AI can't solve (or solves confidently but incorrectly), the human needs to understand the system reasonably well.