Airflow + Spark Example

A minimal pipeline that reads a CSV, aggregates data with PySpark, and writes results back.

Stack

Service	URL
Airflow UI	http://localhost:8080 (admin / admin)
Spark Master UI	http://localhost:8081

Quick start

# 1. Start all services (first run installs providers and migrates DB)
docker-compose up

# 2. Register the Spark connection (run once, in a second terminal)
bash setup_connection.sh

# 3. Open http://localhost:8080, enable and trigger the "spark_example" DAG

What the pipeline does

log_start → spark_aggregate_job → log_end

1. log_start — Python task that logs a start message.
2. spark_aggregate_job — submits dags/spark_job.py to the Spark cluster via SparkSubmitOperator. The job:
   • reads data/sample_data.csv
   • groups by city, computing count / avg salary / max salary / min age
   • writes CSV results to data/output/
3. log_end — Python task that logs completion info.

Project layout

.
├── dags/
│   ├── spark_example_dag.py   # DAG definition
│   └── spark_job.py           # PySpark job submitted to cluster
├── data/
│   ├── sample_data.csv        # Input data
│   └── output/                # Results written here after run
├── docker-compose.yml
└── setup_connection.sh

"# airflow"

The depends_on: airflow-init has been removed from both airflow-webserver and airflow-scheduler. Both services still inherit depends_on: postgres: condition: service_healthy from the common anchor, so Postgres readiness is still enforced.

Now in Docker Desktop, stop and delete the existing containers for this project, then start them again. The stale container ID error will not come back.

Note: airflow-init (DB migration + admin user creation) needs to have run at least once before the webserver/scheduler will work. If this is a fresh setup, start airflow-init first, wait for it to complete, then start the rest.