1.13 is on apache-airflow==3.3.0 and still carries both pre-fix behaviours:
get_dagbag drops read_dags_from_db=True for Airflow >= 3.0, so collect_dags walks the entire DAG folder on every deploy — one deploy costs O(total DAGs), a bulk deploy O(total DAGs²). The walk also masks process_file, which hits its only_if_updated early return and returns no dags.deploy-pipelines budgets 120s / 20 = 6.0s per pipeline against real latencies of 10–24s, so it abandons chunks that are still succeeding, marks all 20 FAILED, and exits 1 — which restartPolicy: OnFailure then replays on top of in-flight work.
Fixed on main by #32005 (issue #32006) and #31976 (issue #32004). This issue tracks landing both on 1.13.
Backport PR: #32100
1.13is onapache-airflow==3.3.0and still carries both pre-fix behaviours:get_dagbagdropsread_dags_from_db=Truefor Airflow >= 3.0, socollect_dagswalks the entire DAG folder on every deploy — one deploy costs O(total DAGs), a bulk deploy O(total DAGs²). The walk also masksprocess_file, which hits itsonly_if_updatedearly return and returns no dags.deploy-pipelinesbudgets120s / 20 = 6.0sper pipeline against real latencies of 10–24s, so it abandons chunks that are still succeeding, marks all 20FAILED, and exits 1 — whichrestartPolicy: OnFailurethen replays on top of in-flight work.Fixed on
mainby #32005 (issue #32006) and #31976 (issue #32004). This issue tracks landing both on1.13.Backport PR: #32100