Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@

All notable changes to this project are documented here. The project uses [Semantic Versioning](https://semver.org/).

## [Unreleased]

- **Supervisors inside the master:** an app that runs Horizon's supervisors in the master process can have the template fork their workers directly (`horizon_inline`, `Master::start()`, `ForksWorkersFromTemplate`). The template's worker handling is shared with the fork servers (`WorkerSet`), and the template TERMs the workers it forked when its master is gone.

## [1.1.2] - 2026-11-20

- **Stack traces in Flare, Sentry and Nightwatch:** a forked worker's stack trace ends in the launcher the template was started from, not in `artisan`. The launcher was a copy outside `vendor/`, so error trackers counted it as the app's own code. Flare drops every frame above the first frame of the app's code, so for a job failing in vendor code (a package's job, for example) only the launcher was left. Sentry marked it as the app's code, and Nightwatch placed such exceptions there. Templates now start through a link to the package's own `bin/template.php`, so traces name it in `vendor/`, and error trackers see the trace as in stock. Reported by Funsho Arashi (#5).
Expand Down
9 changes: 9 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,15 @@ Each `queue:work` process hands itself to a shared template before Laravel boots
- **Horizon works as usual,** because supervisor and worker code is Horizon's own: the dashboard, `horizon:terminate`, `pause`, `continue`, `purge`, autoscaling, and supervisord's `stopasgroup` and `killasgroup`.
- **Fallbacks.** A failed check runs the stock master in the same pid. If the template dies on Linux, Horizon exits and your process manager restarts it. A supervisor whose workers keep dying before their first loop is replaced by a stock one; a supervisor and its workers are always all forked or all stock. Each fallback logs a `queue-prefork:` error line and fires an event.

### Supervisors inside the master

Some apps run Horizon's supervisors inside the master instead of one process each. Their workers can still be forked: the template forks each worker itself, on the master's request, with no fork server in between.

- **Turn it on** with `QUEUE_PREFORK_HORIZON_INLINE=true` (config `horizon_inline`), which the doctor reads too.
- **In the app's own `horizon` command,** call `app(Tevkert\QueuePrefork\Master::class)->start()` before the provisioning plan deploys, when the container has that class bound (the package binds it only for `artisan horizon`). It returns false and changes nothing when the package is switched off or a check fails.
- **Give each in-process supervisor a process pool that uses `Tevkert\QueuePrefork\Horizon\ForksWorkersFromTemplate`.** Its workers are forked while the master runs this way and exec'd as stock otherwise. A stopped worker whose template is gone, or whose supervisor the crash breaker turned stock, gets a new handle, forked again or stock, instead of being restarted on the old one.
- **Don't run `horizon:purge` in this mode.** It looks for workers among the children of supervisor processes, and these workers are the template's children.

### Fork safety

Before forking, a template audits itself: sockets, epoll, eventfd or extra threads that extensions open at startup mean stock workers. ddtrace, Elastic APM and gRPC with fork support (`grpc.enable_fork_support=1`, which rebuilds its threads and poller in each fork) are allowed; workers run stock with New Relic, gRPC without fork support, datadog-profiling and Blackfire.
Expand Down
5 changes: 5 additions & 0 deletions config/queue-prefork.php
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,11 @@
'queue_work' => env('QUEUE_PREFORK_QUEUE_WORK', true),
'horizon' => env('QUEUE_PREFORK_HORIZON', true),

// The app runs Horizon's supervisors inside the master and asks the template only for their workers
// (Master::start(inline: true), ForksWorkersFromTemplate). Its own AddSupervisor binding is then expected;
// the doctor checks this mode instead of the forked supervisors.
'horizon_inline' => env('QUEUE_PREFORK_HORIZON_INLINE', false),

// The templates' opcache. Its shared memory holds the compiled code of the whole app.
'opcache' => [
'memory_consumption' => env('QUEUE_PREFORK_OPCACHE_MB', 192),
Expand Down
7 changes: 7 additions & 0 deletions src/Client/ForkedProcess.php
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,7 @@ public function start(?callable $callback = null, array $env = []): void
$this->broker->request("spawn:{$this->id}", [
't' => 'spawn',
'id' => $this->id,
'role' => $this->role,
'cmd' => $this->line,
'cwd' => $this->directory,
'env' => getenv(),
Expand Down Expand Up @@ -216,6 +217,12 @@ public function ready(): void
}
}

/** Whether the server this handle forks through is gone: starting it again can only fail. */
public function serverGone(): bool
{
return $this->broker->lost;
}

/** @internal */
public function serverLost(): void
{
Expand Down
3 changes: 2 additions & 1 deletion src/Console/DoctorCommand.php
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,8 @@ public function handle(): int
$horizon = class_exists(MasterSupervisor::class);
$release = realpath(base_path()) ?: base_path();
$pidOne = PHP_OS_FAMILY === 'Linux' ? str_replace("\0", ' ', (string) @file_get_contents('/proc/1/cmdline')) : '';
$reasons = $horizon ? [...Switches::off($release, 'horizon'), ...Preflight::check($this->laravel), ...self::pidOne($pidOne, 'horizon')] : [];
$inline = (bool) $this->laravel['config']->get('queue-prefork.horizon_inline', false);
$reasons = $horizon ? [...Switches::off($release, 'horizon'), ...Preflight::check($this->laravel, inline: $inline), ...self::pidOne($pidOne, 'horizon')] : [];
$queueWork = self::queueWork($release, self::pidOne($pidOne, 'queue:work'));
$warnings = [...self::hostWarnings($horizon), ...$this->earlyMismatches($release)];

Expand Down
59 changes: 59 additions & 0 deletions src/Horizon/ForksWorkersFromTemplate.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
<?php

namespace Tevkert\QueuePrefork\Horizon;

use Laravel\Horizon\WorkerProcess;
use Tevkert\QueuePrefork\Client\ForkedProcess;
use Tevkert\QueuePrefork\Master;

/**
* For the ProcessPool of a supervisor that runs inside the Horizon master: its workers are forked by the
* template once the master started in inline mode (Master::start(), queue-prefork.horizon_inline), and
* exec'd as stock otherwise.
*/
trait ForksWorkersFromTemplate
{
protected function createProcess()
{
$process = $this->preforkMaster()?->workerProcess($this->options);

return $process === null ? parent::createProcess() : new WorkerProcess($process);
}

public function monitor()
{
$this->replaceRetiredWorkers();

parent::monitor();
}

/**
* Horizon restarts a stopped worker on the handle it has. A forked handle whose template is gone, or whose
* supervisor the crash breaker turned stock, gets a new one instead: forked again, or stock.
*/
protected function replaceRetiredWorkers(): void
{
$master = $this->preforkMaster();

foreach ($this->processes as $index => $worker) {
$process = $worker->process;

if (! $process instanceof ForkedProcess || $process->isRunning()) {
continue;
}

if ($process->serverGone() || $master?->retires($process, $this->options->name)) {
$this->processes[$index] = $this->createProcess()->handleOutputUsing(function ($type, $line) {
call_user_func($this->output, $type, $line);
});
}
}
}

private function preforkMaster(): ?Master
{
$app = app();

return $app->bound(Master::class) ? $app->make(Master::class) : null;
}
}
102 changes: 100 additions & 2 deletions src/Master.php
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

use Illuminate\Contracts\Foundation\Application;
use Illuminate\Support\Facades\Redis;
use Laravel\Horizon\Events\MasterSupervisorLooped;
use Laravel\Horizon\MasterSupervisor;
use Laravel\Horizon\SupervisorOptions;
use Tevkert\QueuePrefork\Client\Broker;
Expand Down Expand Up @@ -75,20 +76,40 @@ final class Master
/** @var array<string, ForkedProcess> supervisor name => current unit handle */
private array $units = [];

/** @var array<string, list<float>> supervisor name => recent exits of its workers before their first loop */
private array $earlyWorkerExits = [];

public function __construct(private Application $app) {}

public function start(?callable $warn = null): bool
/**
* @param bool|null $inline the app runs its supervisors in this process and asks for their workers through
* workerProcess(), so its own AddSupervisor binding is expected, not a conflict;
* null reads queue-prefork.horizon_inline, which the doctor reads too
*/
public function start(?callable $warn = null, ?bool $inline = null): bool
{
Runtime::preload();
$inline ??= (bool) $this->app['config']->get('queue-prefork.horizon_inline', false);

if ($inline && ($off = $this->inlineOff()) !== null) {
$this->reason = $off;

return false;
}

try {
$reasons = Preflight::check($this->app);
$reasons = Preflight::check($this->app, inline: $inline);

if ($reasons === [] && $this->startTemplate()) {
$this->active = true;
$this->mode = 'prefork';
Report::event('PreforkStarted', ['template' => $this->template['pid'] ?? null], 'info');

if ($inline) {
// The app's own horizon command runs the loop: nothing else registers this.
$this->app['events']->listen(MasterSupervisorLooped::class, fn () => $this->tick());
}

return true;
}

Expand All @@ -113,6 +134,21 @@ public function start(?callable $warn = null): bool
return false;
}

/**
* Why an inline start stays stock: Wiring checks the same switches before it binds the package's own
* command, which an app running its supervisors in the master replaces with its own.
*/
private function inlineOff(): ?string
{
$off = Support\Switches::off(base_path(), 'horizon');

if ($off !== []) {
return implode('; ', $off);
}

return Settings::enabled($this->app, 'horizon') ? null : 'switched off ('.Settings::switchSource($this->app, 'horizon').')';
}

/** Called on every MasterSupervisorLooped (and so also while paused). */
public function tick(): void
{
Expand Down Expand Up @@ -158,6 +194,68 @@ public function unitProcess(SupervisorOptions $options): ?ForkedProcess
return $this->units[$options->name] = $unit;
}

/**
* A worker forked by the template for a supervisor that runs in this process, or null for a stock one:
* when nothing forks, for a command line the template does not run, or once the supervisor's workers kept
* exiting before their first loop. A handle once forked stays forked when its worker restarts.
*/
public function workerProcess(SupervisorOptions $options): ?ForkedProcess
{
if (! $this->canForkUnits() || isset($this->stockNames[$options->name])) {
return null;
}

$line = $options->toWorkerCommand();

try {
CommandLine::parse($line, ['horizon:work']);
} catch (Throwable $e) {
$this->stockNames[$options->name] = $e->getMessage();

return null;
}

$worker = new ForkedProcess($this->broker, 'worker', $line, $options->directory ?? base_path(), ! config('horizon.fast_termination'));
$worker->onExit = function (ForkedProcess $process, int $status, bool $wasReady) use ($options): void {
$this->workerExited($options->name, $process, $status, $wasReady);
};

return $worker;
}

/**
* Whether a stopped forked worker of $supervisor needs a new handle: Horizon restarts a worker on the
* handle it has, which still forks through a template that is gone, or for a supervisor now stock.
*/
public function retires(ForkedProcess $worker, string $supervisor): bool
{
return $worker->serverGone() || isset($this->stockNames[$supervisor]);
}

/** The unit breaker of Link::workerExited(), kept here per supervisor, as no loop of its own runs it. */
private function workerExited(string $supervisor, ForkedProcess $process, int $status, bool $wasReady): void
{
if ($wasReady || $status === 0 || $process->parentSignalled || in_array($status, [137, 75, 72], true)) {
return;
}

$now = microtime(true);

if ($now - $process->spawnedAt > 120) {
return;
}

$window = (float) Runtime::config('breaker_window', 300);
$exits = array_values(array_filter($this->earlyWorkerExits[$supervisor] ?? [], fn (float $at) => $now - $at < $window));
$exits[] = $now;
$this->earlyWorkerExits[$supervisor] = $exits;

if (count($exits) >= (int) Runtime::config('breaker_exits', 5)) {
$this->stockNames[$supervisor] = 'crash breaker';
Report::event('BreakerTripped', ['scope' => 'supervisor', 'supervisor' => $supervisor, 'status' => $status]);
}
}

public function statusDir(): string
{
return $this->statusDir;
Expand Down
14 changes: 11 additions & 3 deletions src/Preflight.php
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,11 @@
*/
final class Preflight
{
/** @return list<string> */
public static function check(Application $app, bool $probes = true): array
/**
* @param bool $inline the app's supervisors run in the master, which forks only their workers
* @return list<string>
*/
public static function check(Application $app, bool $probes = true, bool $inline = false): array
{
$reasons = [];
$config = (array) $app['config']->get('queue-prefork', []);
Expand Down Expand Up @@ -53,7 +56,12 @@ public static function check(Application $app, bool $probes = true): array
$horizon = self::horizonVersion();

// The bindings and command strings are Horizon's too: read them only from a Horizon that has them.
$reasons = array_merge($reasons, $horizon, $horizon === [] ? [...self::bindings($app), ...self::commandStrings($app)] : [], self::extensions($config));
$reasons = array_merge(
$reasons,
$horizon,
$horizon === [] ? [...($inline ? [] : self::bindings($app)), ...self::commandStrings($app)] : [],
self::extensions($config),
);

if ($probes && $reasons === []) {
$reasons = array_merge($reasons, self::probes($app));
Expand Down
3 changes: 2 additions & 1 deletion src/Runtime.php
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@ final class Runtime
Master::class, MasterHandoff::class, Preflight::class, QueueWorkerLink::class, Runtime::class, Settings::class,
Wiring::class, WorkerLink::class, Server\ChildJob::class, Server\ChildRoutine::class, Server\Opcache::class,
Server\RoleGuard::class, Server\SharedLog::class, Server\Signals::class, Server\Template::class,
Server\UnitServer::class, Server\WorkServer::class, Server\WorkerContext::class, Support\Audit::class, Support\Channel::class,
Server\UnitServer::class, Server\WorkServer::class, Server\WorkerContext::class, Server\WorkerSet::class,
Support\Audit::class, Support\Channel::class,
Support\CommandLine::class, Support\DotEnv::class, Support\EarlyConfig::class, Support\Fd::class,
Support\Identity::class, Support\LineBufferFilter::class, Support\LineBufferedOutput::class,
Support\LogTargets::class, Support\Native::class, Support\Process::class, Support\RuntimeDir::class,
Expand Down
Loading