Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/sharder-exit-hooks.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@wolfstar/plugin-sharder": minor
---

`ShardClient#addExitHook` runs a hook before the shard stops with `process.exit` (on `exit()`, `restart()`, a close from the manager, or the manager going away), which skips `beforeExit`, so traces or metrics can be flushed first. `ShardManager#addDestroyHook` runs a hook at the end of `destroy()`, once the shards are closed.
13 changes: 13 additions & 0 deletions packages/plugin-sharder/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,19 @@ first and closes the old one once the new one is ready, for close to no downtime
`manager.reshard(layout)` does the same for a whole new layout. When the manager dies, a process
shard emits `disconnect` and exits when nothing listens to it; a worker thread stops with it.

A shard stops with `process.exit`, which skips `beforeExit`, so whatever a library flushes there
(traces, metrics) is lost. `shard.addExitHook(hook)` runs a hook first: on `shard.exit()` and
`shard.restart()`, after the close handler when the manager closes the shard, and when the manager
is gone and nothing listens to `disconnect`. The hooks count towards the close timeout.
`manager.addDestroyHook(hook)` runs a hook at the end of `manager.destroy()`, once the shards are
closed, even when closing one failed. In both, a hook that throws is reported through `error` and
the next ones still run, and the call returns a function that removes the hook.

```ts
shard.addExitHook(() => metrics.flush());
manager.addDestroyHook(() => metrics.flush());
```

### Gateway shards

The manager paces identifies across every process: `shard.identifyThrottler` is `@discordjs/ws`'s
Expand Down
47 changes: 46 additions & 1 deletion packages/plugin-sharder/src/ShardClient.ts
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,11 @@ export interface ClientTransport {
exit(code: number): void;
}

/**
* Runs before a shard's process or thread stops, see {@link ShardClient.addExitHook}.
*/
export type ShardExitHook = () => unknown;

/**
* The options of a {@link ShardClient}.
*/
Expand Down Expand Up @@ -207,6 +212,9 @@ export class ShardClient extends EventEmitter<ShardClientEvents> {
#watchdog: NodeJS.Timeout | null = null;
#requestHandler: RequestHandler<{ from: number | null }> | null = null;
#closeHandler: (() => unknown) | null = null;
readonly #exitHooks = new Set<ShardExitHook>();
// The hooks in progress, shared by the paths that stop the shard, so none exits before the first run is done.
#exiting: Promise<void> | null = null;
#shardHandler: ShardHandler | null = null;

public constructor(options: ShardClientOptions = {}) {
Expand All @@ -227,7 +235,7 @@ export class ShardClient extends EventEmitter<ShardClientEvents> {
this.#transport.onDisconnect(() => {
this.#stopWatchdog();
if (this.listenerCount("disconnect") > 0) this.emit("disconnect");
else this.#transport.exit(0);
else void this.#runExitHooks().finally(() => this.#transport.exit(0));
});

this.#pingTimeout = context.pingTimeout;
Expand Down Expand Up @@ -455,6 +463,28 @@ export class ShardClient extends EventEmitter<ShardClientEvents> {
return this;
}

/**
* Runs a hook before the shard stops: on {@link ShardClient.exit}, on {@link ShardClient.restart}, once the close
* handler of a close from the manager resolved, and when the manager is gone and nothing listens to `disconnect`.
* The place to flush what is left to export, such as traces: the shard stops with `process.exit`, which skips
* `beforeExit`.
*
* @remarks
* The hooks run one after the other, in the order they were added, before the shard tells the manager it stops. A
* hook that throws is reported through the `error` event (or logged when nothing listens) and does not stop the
* others, nor the shard. They run once: a path that stops the shard while they run (the manager going away during
* `exit()`) waits for them. Adding the same function twice adds it once.
*
* @param hook The hook.
* @returns A function that removes the hook.
*/
public addExitHook(hook: ShardExitHook): () => void {
this.#exitHooks.add(hook);
return () => {
this.#exitHooks.delete(hook);
};
}

/**
* Sets how the manager starts and closes the gateway shards of this shard, for `manager.startShard` & co.
*
Expand Down Expand Up @@ -486,10 +516,25 @@ export class ShardClient extends EventEmitter<ShardClientEvents> {

async #stop(status: ShardStatus, code: number): Promise<void> {
this.#stopWatchdog();
await this.#runExitHooks();
await this.#signal(status).catch(() => undefined);
this.#transport.exit(code);
}

#runExitHooks(): Promise<void> {
this.#exiting ??= (async () => {
// A copy: a hook added while the hooks run is not run this time.
for (const hook of Array.from(this.#exitHooks)) {
try {
await hook();
} catch (error) {
this.#report(error);
}
}
})();
return this.#exiting;
}

async #write(packet: Packet): Promise<void> {
await this.#transport.send(await this.#codec.encode(packet, { channelId: this.id }));
}
Expand Down
45 changes: 42 additions & 3 deletions packages/plugin-sharder/src/ShardManager.ts
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,11 @@ export interface ShardLayoutOptions {
recommended?: RecommendedShardCountOptions;
}

/**
* Runs at the end of {@link ShardManager.destroy}, see {@link ShardManager.addDestroyHook}.
*/
export type ShardManagerDestroyHook = () => unknown;

/**
* The options of a {@link ShardManager}.
*/
Expand Down Expand Up @@ -314,6 +319,7 @@ export class ShardManager extends EventEmitter<ShardManagerEvents> {
readonly #readyTimes: number[] = [];
#queue: Promise<void> = Promise.resolve();
#initialized = false;
readonly #destroyHooks = new Set<ShardManagerDestroyHook>();

public constructor(options: ShardManagerOptions = {}) {
super();
Expand Down Expand Up @@ -622,9 +628,42 @@ export class ShardManager extends EventEmitter<ShardManagerEvents> {
* Closes every shard for good, and releases the strategy.
*/
public async destroy(): Promise<void> {
await Promise.all(this.channels.map((channel) => channel.close()));
if (this.#initialized) await this.strategy.destroy?.();
this.#initialized = false;
try {
// Every close settles before the hooks run, so a shard still stopping does not emit after them.
const closed = await Promise.allSettled(this.channels.map((channel) => channel.close()));
const failed = closed.find((result) => result.status === "rejected");
if (failed) throw failed.reason;
if (this.#initialized) await this.strategy.destroy?.();
this.#initialized = false;
} finally {
// Even when closing failed: the process is likely to exit next, and what is left should still go out.
for (const hook of Array.from(this.#destroyHooks)) {
try {
await hook();
} catch (error) {
this.reportError(error);
}
}
}
}

/**
* Runs a hook at the end of every {@link ShardManager.destroy}, once the shards are closed: the place to flush what
* is left to export, such as traces, before the manager's process exits.
*
* @remarks
* The hooks run one after the other, in the order they were added, even when closing the shards failed. A hook that
* throws is reported through the `error` event (or logged when nothing listens) and does not stop the others. Adding
* the same function twice adds it once.
*
* @param hook The hook.
* @returns A function that removes the hook.
*/
public addDestroyHook(hook: ShardManagerDestroyHook): () => void {
this.#destroyHooks.add(hook);
return () => {
this.#destroyHooks.delete(hook);
};
}

/**
Expand Down
Loading
Loading