ERROR - DagFileProcessorManager (PID=1234 last sent a heartbeat 50.42 seconds ago! Restarting it

[sumith0810]

Hi Team,
I have recently installed airflow 2.1.3 using apache-airflow helm repo. But post the installation, The Dag files are not getting displayed on the UI. The reason could be the scheduler getting terminated consistently. Below is the error. Can anyone please help me with the below issue?

> [2021-10-27 10:07:30,954] {manager.py:254} INFO - Launched DagFileProcessorManager with pid: 24359
> [2021-10-27 10:07:30,976] {settings.py:51} INFO - Configured default timezone Timezone('UTC')
> [2021-10-27 10:08:21,675] {manager.py:414} ERROR - DagFileProcessorManager (PID=24359) last sent a heartbeat 50.73 seconds ago! Restarting it
> [2021-10-27 10:08:21,676] {process_utils.py:100} INFO - Sending Signals.SIGTERM to GPID 24359
> [2021-10-27 10:08:21,931] {process_utils.py:66} INFO - Process psutil.Process(pid=24359, status='terminated', exitcode=0, started='10:07:29') (24359) terminated with exit code 0
> [2021-10-27 10:08:21,938] {manager.py:254} INFO - Launched DagFileProcessorManager with pid: 24404
> [2021-10-27 10:08:21,963] {settings.py:51} INFO - Configured default timezone Timezone('UTC')

[lucasfonsecads]

Do you still have this problem?

[aditiagarwal30nov]

Yes, I am facing the same problem, any leads?

[lucasfonsecads]

Could you check if your dag folder is in airflow.cfg?
Chek if it's the same as this example:
/$AIRFLOW_HOME/dags
And you $AIRFLOW_HOM is equal to PWD

[sipri83]

I am facing the same problem with 2.2.0/pythong 3.8

It keeps on restarting and that hampers New DAG parsing . After restart also it keeps on happening for sometime .

[2022-01-10 07:32:09,331] {manager.py:321} ERROR - DagFileProcessorManager (PID=63) last sent a heartbeat 50.47 seconds ago! Restarting it
[2022-01-10 07:32:59,982] {manager.py:321} ERROR - DagFileProcessorManager (PID=69) last sent a heartbeat 50.48 seconds ago! Restarting it
[2022-01-10 07:33:50,334] {manager.py:321} ERROR - DagFileProcessorManager (PID=73) last sent a heartbeat 50.18 seconds ago! Restarting it
[2022-01-10 07:34:41,129] {manager.py:321} ERROR - DagFileProcessorManager (PID=74) last sent a heartbeat 50.62 seconds ago! Restarting it
[2022-01-10 07:35:31,780] {manager.py:321} ERROR - DagFileProcessorManager (PID=76) last sent a heartbeat 50.48 seconds ago! Restarting it

[lucasfonsecads]

For this error, you could try to change the scheduler_health_check_threshold to 240.
Please let me know if this work.

[sipri83]

tried that it didn't help much but processor_timeout i changed and it's helping a bit probably, still monitoring.

[lucasfonsecads]

Waiting for your update.

[sipri83]

One Cluster with 6 Cores and 10 parsing process - 5 scheduler pods with the setting it did hold good,
processor_timeout=120 and scheduler_health_check_threshold=120
however other cluster with 4 cores and 5 parsing process - 5 schedulers pods - still monitoring

[ckvtvm-arch]

can you please let me if this is resolved

[sipri83]

not completely, it does help, but still it happens every now and then.. but it tries after sometime and then it works. One more thing is.. sometimes it also leaves behind defunc processes so have scheduled recycling pod automatically every few hrs in a day.

[bibekyess]

@sipri83 Have you solved it? I am also facing exactly the same issue. Let me know if you managed to solve it.

[appunni-m]

facing same issue here, some times some dags gets processed otherwise nothing is getting processed, has more than 1000 dags in the Dag folder, 43 pending to be processed

[waqas-saeed]

I was having this issue when running dags locally. it turned out that my docker on my macbook was somehow causing this. I had to rest my docker to factory default and this issue went away.

[smonani]

I was not able to get around this issue when trying to use Airflow 2.1.0 on Docker Desktop using WSL2 on a Windows Server VM. After trying countless things, increments and configuration, I followed the following steps which resolved the issue:
From docker desktop, go to troubleshooting option and reset to factory settings. Then redo any customizations you had. Then, try running airflow again.

In my case, there was also another issue where I had to update the wsl using the command "wsl.exe --update".

I hope this helps anyone who is trying to do this on Windows Server or a VM.

[parassetia889]

I tried all of the above resolution, still facing the issue

[Starkie]

In my case, this error was caused by having the .venv folder inside the DAGs folder.

I was able to fix it by adding an .airflowignore file that skipped parsing all these files:

**/.venv
**/__pycache__
**/.git
**/.pytest_cache

[DixT-180]

The place where I worked had this exact issue: the Airflow UI wasn't reflecting DAG changes. DAGs that had been updated or removed weren't being reflected, and the UI appeared to be stuck showing the state from several days earlier.

The issue was resolved by clearing the Airflow metadata database — either by truncating the relevant DAG-related metadata tables or, as a more drastic option, dropping and recreating the entire Airflow metadata database.

After recreating the database, the Airflow schema had to be initialized/migrated again before starting the scheduler and webserver.