Starting a Trinity Ray job, spawns many more secondary Ray jobs

[vadimkantorov]

These seem to correspond to vllm engines. Is it expected that these vllm instances get spawned as Jobs, and not as actors within the main job?

Thanks!

Process 1066282: /mnt/local/venv_trinity/bin/python3 -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=88, pipe_handle=90) --multiprocessing-fork
Python v3.12.11 (/usr/bin/python3.12)

Thread 1066282 (active+gil): "MainThread"
    fn (torch/_dynamo/utils.py:4353)
    _fn (torch/_dynamo/eval_frame.py:838)
    call_size (torch/_dynamo/utils.py:4355)
    forward (vllm/model_executor/models/qwen2.py:337)
    __call__ (vllm/compilation/decorators.py:246)
    forward (vllm/model_executor/models/qwen3.py:303)
    _call_impl (torch/nn/modules/module.py:1762)
    _wrapped_call_impl (torch/nn/modules/module.py:1751)
    execute_model (vllm/v1/worker/gpu_model_runner.py:1374)
    decorate_context (torch/utils/_contextlib.py:116)
    execute_model (vllm/v1/worker/gpu_worker.py:308)
    decorate_context (torch/utils/_contextlib.py:116)
    run_method (vllm/utils/__init__.py:2736)
    collective_rpc (vllm/executor/uniproc_executor.py:57)
    execute_model (vllm/v1/executor/abstract.py:87)
    execute_model (vllm/v1/engine/core.py:212)
    step (vllm/v1/engine/core.py:235)
    _process_engine_step (vllm/v1/engine/core.py:631)
    run_busy_loop (vllm/v1/engine/core.py:606)
    run_engine_core (vllm/v1/engine/core.py:579)
    run (multiprocessing/process.py:108)
    _bootstrap (multiprocessing/process.py:314)
    _main (multiprocessing/spawn.py:135)
    spawn_main (multiprocessing/spawn.py:122)
    <module> (<string>:1)
Thread 1066609 (idle): "Thread-1 (_report_usage_worker)"
    _report_continuous_usage (vllm/usage/usage_lib.py:233)
    _report_usage_worker (vllm/usage/usage_lib.py:168)
    run (threading.py:1012)
    _bootstrap_inner (threading.py:1075)
    _bootstrap (threading.py:1032)
Thread 1066700 (idle): "Thread-2"
    wait (threading.py:359)
    wait (threading.py:655)
    run (tqdm/_monitor.py:60)
    _bootstrap_inner (threading.py:1075)
    _bootstrap (threading.py:1032)
Thread 1066760 (idle): "Thread-3"
    wait (threading.py:359)
    wait (threading.py:655)
    run (tqdm/_monitor.py:60)
    _bootstrap_inner (threading.py:1075)
    _bootstrap (threading.py:1032)
Thread 1067483 (idle): "Thread-4 (process_input_sockets)"
    poll (zmq/sugar/poll.py:106)
    process_input_sockets (vllm/v1/engine/core.py:734)
    run (threading.py:1012)
    _bootstrap_inner (threading.py:1075)
    _bootstrap (threading.py:1032)
Thread 1067487 (idle): "Thread-5 (process_output_sockets)"
    wait (threading.py:355)
    get (queue.py:171)
    process_output_sockets (vllm/v1/engine/core.py:779)
    run (threading.py:1012)
    _bootstrap_inner (threading.py:1075)
    _bootstrap (threading.py:1032)
Thread 1074225 (idle): "ray_listen_error_messages"
    listen_error_messages (ray/_private/worker.py:2338)
    run (threading.py:1012)
    _bootstrap_inner (threading.py:1075)
    _bootstrap (threading.py:1032)
Thread 1074255 (idle): "ray_print_logs"
    print_logs (ray/_private/worker.py:1006)
    run (threading.py:1012)
    _bootstrap_inner (threading.py:1075)
    _bootstrap (threading.py:1032)

[pan-x-c]

It is because vllm's v1 engine will be started as a separate process, and we currently do not have a good solution for this. The V0 engine doesn't have this problem, but it's no longer maintained, and there may be potential bugs.

[saisharan0103]

This appears to hit the vLLM engine-process path, with the captured stack ending in vllm/v1/engine/core.py:run_engine_core under a multiprocessing spawn. trinity/common/models/vllm_patch/__init__.py is the part I would inspect first to see whether Trinity is letting vLLM start Ray-visible worker jobs instead of keeping them scoped under the main Ray job. For a PR, I would start by tightening that engine launch path or documenting the expected Ray job topology.