Why is the vLLM backend commented out?

[myym0]

When using vLLM as the inference backend, the generation of QA pairs gets stuck. The request has been sent to the vLLM backend, but the generation takes a very long time and does not complete.
The WeChat group seems to have expired. Could you please provide the contact information for another commu

[ChenZiHong-Gavin]

Could you provide more details(version of vllm, error log, etc.)

[ChenZiHong-Gavin]

vllm is commented out because the local inference method is not fully implemented yet. We will implement it soon.
You can use vllm serve to deploy the model yourself, and use openai_client.

[angelyix]

Hi, same question here. I attempted to uncomment the section regarding the VLLM inference engine in graphgen/operators/init/init_llm.py and changed the hard-coded request_id="graphgen_req" in vllm_wrapper.py:lines 70-75 to a unique request ID. However, during the final QA extraction step, the request is sent but the progress bar gets stuck, as shown below:
[4/4] Generating QAs: 0%| | 0/4 [00:00<?, ?batch/s] INFO 12-08 08:35:03 [async_llm.py:252] Added request graphgen_req_682689db. INFO 12-08 08:35:03 [async_llm.py:252] Added request graphgen_req_e33cbf47. ...
I'm trying to understand what might be causing this blockage. Interestingly, this issue doesn't occur during the “Extracting entities and relationships from chunks” step.
My torch and vllm version is：torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 vllm==0.8.5.post1

[myym0]

vllm is commented out because the local inference method is not fully implemented yet. We will implement it soon. You can use vllm serve to deploy the model yourself, and use openai_client.

First, thank you for your response.

1.I have already tried deploying the model using vLLM and the openai_client, but the generation speed is relatively slow.
After removing the comment symbols for the vLLM backend in the source code, I encountered the same issue when running QA generation based on the vLLM backend.
[4/4] Generating QAs: 0%| | 0/4 [00:00<?, ?batch/s]
INFO 12-08 08:35:03 [async_llm.py:252] Added request graphgen_req_682689db.
INFO 12-08 08:35:03 [async_llm.py:252] Added request graphgen_req_e33cbf47. ...

Here, I borrowed someone else’s example and observed from the logs that requests have been added, but the process gets stuck after adding the requests.
So, I set export VLLM_LOGGING_LEVEL=DEBUG and re-ran the script, and the logs showed:
[4/4] Generating QAs: 0%| | 0/4 [00:00<?, ?batch/s]
INFO 12-08 08:35:03 [async_llm.py:252] Added request graphgen_req_682689db.
INFO 12-08 08:35:03 [async_llm.py:252] Added request graphgen_req_e33cbf47. ...
DEBUG 12-09 03:50:40 [core.py:421] EngineCore waiting for work.

The log “EngineCore waiting for work.” appeared, which seems to indicate a deadlock issue during asynchronous request handling.

[ChenZiHong-Gavin]

@angelyix @myym0
Hi, thank you for reporting the problem. I've successfully reproduced it.
The "hang" or freezing behavior you are experiencing may be caused by a conflict between vLLM's asynchronous engine and the event loop management in your GraphGen framework.

We will change the way to init local model using ray after PR #110 in the following week.
This problem will be also fixed after this PR.
Before that, you may use vllm serve to deploy the model or sglang backend.

[angelyix]

plus, we add

@angelyix @myym0 Hi, thank you for reporting the problem. I've successfully reproduced it. The "hang" or freezing behavior you are experiencing may be caused by a conflict between vLLM's asynchronous engine and the event loop management in your GraphGen framework.

We will change the way to init local model using ray after PR #110 in the following week. This problem will be also fixed after this PR. Before that, you may use vllm serve to deploy the model or sglang backend.

Hello, looking forward to your update. Also, I'd like to know if there's a noticeable difference in speed when using the VLLM server for API calls vs using the SYNTHESIZER_BACKEND=vllm setting supported by Graphgen?

[ChenZiHong-Gavin]

plus, we add

@angelyix @myym0 Hi, thank you for reporting the problem. I've successfully reproduced it. The "hang" or freezing behavior you are experiencing may be caused by a conflict between vLLM's asynchronous engine and the event loop management in your GraphGen framework.
We will change the way to init local model using ray after PR #110 in the following week. This problem will be also fixed after this PR. Before that, you may use vllm serve to deploy the model or sglang backend.

Hello, looking forward to your update. Also, I'd like to know if there's a noticeable difference in speed when using the VLLM server for API calls vs using the SYNTHESIZER_BACKEND=vllm setting supported by Graphgen?

Using the VLLM server for API calls may be slightly slower due to HTTP/serialization overhead, but not by much (you may find evidence in vllm-project/vllm#2019 (comment)).
It is recommended to enable --enable-prefix-caching when using vllm serve to reuse the KV cache.
If you observe that GraphGen becomes unbearably slow when using API calls, please open a new issue.

[ChenZiHong-Gavin]

@angelyix @myym0
Hi,
The issue is fixed after PR: #110.
Make sure you update the latest code and have sufficient GPU and VRAM for tasks that require the trainee model.
If you have any further problems, don't hesitate to open new issues.