Skip to content

Horizon: let the template fork workers for supervisors inside the master - #7

Open
volodymyr-aloware wants to merge 2 commits into
tevkert:mainfrom
aloware:pr/inline-supervisors
Open

volodymyr-aloware wants to merge 2 commits into
tevkert:mainfrom
aloware:pr/inline-supervisors

Conversation

@volodymyr-aloware

Copy link
Copy Markdown

Some apps run Horizon's supervisors inside the master process instead of one process each, to save the memory of a booted app per supervisor. Today such an app has to run queue-prefork stock: its own AddSupervisor binding fails the preflight, and nothing could fork its workers anyway, since workers are forked only by a unit's fork server at its supervisor loop's request.

This PR lets the template fork those workers directly, on the master's request. In our measurements the two savings add up: the in-process supervisors cost nothing extra, and every worker shares the template's compiled code.

What changes

WorkerSet (first commit, no behaviour change). The fork server's handling of horizon:work workers (fork, admit, pause state, cancel, signal, ready, reap) moves into src/Server/WorkerSet.php so that a second server can use it. UnitServer keeps its loop and passes it every worker message as before.

The template forks workers (second commit).

  • ForkedProcess sends its role in spawn. For role: worker, the template forks a horizon:work worker through its own WorkerSet, routes admit, pause_state, cancel and signal for worker ids, and reaps them. Anything else still forks a unit, so existing masters are unaffected.
  • The template TERMs the workers it forked when its master is gone (hosted: in masterExited(), before it exits or execs the stock master; spawned: when the master is gone or its channel closes), so none outlives it. The fork blocks the relayed signals until the child has reset its handlers.
  • Master::start() takes an inline flag, defaulting to the new config horizon_inline (QUEUE_PREFORK_HORIZON_INLINE). Inline mode skips Preflight's bindings check only, applies the off switches itself (as Wiring does before binding the package's command), and registers its own tick on MasterSupervisorLooped, since the app's command replaces PreforkHorizonCommand.
  • Master::workerProcess(SupervisorOptions) hands out a forked handle, or null for stock, with the unit breaker's rule kept per supervisor.
  • ForksWorkersFromTemplate is a trait for the app's ProcessPool subclass. createProcess() asks the master for a forked handle. Horizon restarts a stopped worker on the handle it has, so monitor() first replaces a stopped forked handle whose template is gone, or whose supervisor the breaker turned stock, with a new one.
  • queue-prefork:doctor checks this mode when horizon_inline is set.
  • README gets a "Supervisors inside the master" section (setup, fallbacks, and why horizon:purge must not run in this mode); CHANGELOG gets an Unreleased entry.

Limitations

  • In a master that runs most supervisors inline, a supervisor kept in its own process is exec'd as stock; AddSupervisor::createProcess() could use unitProcess() for it, but that is left out here.
  • When the breaker trips, workers still running keep their forked handle until they stop; they are replaced as they exit, so for a while such a pool mixes forked and stock workers.

Happy to rename the config key, split the PR, or move the trait wherever it fits better.

A unit's fork server forks, admits, signals and reaps the horizon:work
workers of its supervisor loop. That handling moves into WorkerSet,
unchanged, so another server can fork workers the same way. UnitServer
keeps the loop and passes it every worker message, as before.
Some apps run Horizon's supervisors inside the master process instead of
one process each. Their workers can now be forked as well: the master asks
its template for each worker (a spawn with role: worker), and the template
forks it through WorkerSet, with no fork server in between. The template
TERMs the workers it forked when its master is gone, so none outlives it.

On the master's side, Master::start() in inline mode
(queue-prefork.horizon_inline) skips the check for the app's own
AddSupervisor binding, applies the off switches itself and pumps on
MasterSupervisorLooped. Master::workerProcess() hands out forked handles
with a crash breaker per supervisor, and ForksWorkersFromTemplate lets the
app's process pool use them. Horizon restarts a worker on the handle it
has, so a stopped handle whose template is gone, or whose supervisor the
breaker turned stock, is replaced with a new one, forked again or stock.
The doctor reads horizon_inline as well.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant