Create Pack Apache Airflow 1.22.0

packs/apache-airflow-1.22.0/README.md

@@ -0,0 +1,122 @@
+# Apache Airflow
+
+## Overview
+
+Apache Airflow is an open source platform for developing, scheduling, and monitoring workflows. It allows you to define workflows as code, organize tasks into dependencies, execute them on a schedule, and monitor their execution through a web interface.
+
+Airflow workflows are defined using Python and are represented as Directed Acyclic Graphs (DAGs). A DAG defines the tasks that need to be executed and the order in which they should run.
+
+This pack installs Apache Airflow using the official Apache Airflow Helm chart.
+
+## Prerequisites
+
+Before installing this pack, ensure the following requirements are met:
+
+- Kubernetes 1.29 or later.
+- A running Kubernetes cluster.
+- Sufficient cluster resources to deploy Apache Airflow.
+- Network connectivity between the Airflow components.
+- Access to the required container images.
+
+## Pack Contents
+
+This pack deploys the following components:
+
+- Apache Airflow API Server
+- Apache Airflow Scheduler
+- Apache Airflow DAG Processor
+- Apache Airflow Triggerer
+- Apache Airflow Workers
+- Apache Airflow Webserver
+- PostgreSQL
+- Redis
+- StatsD
+- Kubernetes Services
+- Service Accounts
+- RBAC resources
+- ConfigMaps
+- Secrets
+
+## Configuration
+
+The pack installs Apache Airflow into the `airflow` namespace by default.
+
+The pack uses Apache Airflow version `3.2.2` with Helm chart version `1.22.0`.
+
+The following table describes the primary configuration parameters.
+
+| Parameter | Description | Default |
+|----------|-------------|---------|
+| `pack.namespace` | Namespace where Apache Airflow is installed. | `airflow` |
+| `executor` | Executor used by Apache Airflow to execute tasks. | `CeleryExecutor` |
+| `defaultAirflowRepository` | Default Apache Airflow container image repository. | `apache/airflow` |
+| `defaultAirflowTag` | Default Apache Airflow container image tag. | `3.2.2` |
+| `airflowVersion` | Apache Airflow version used by the chart. | `3.2.2` |
+| `postgresql.enabled` | Enables the PostgreSQL dependency. | `true` |
+| `redis` | Configuration for the Redis dependency used by Celery. | Enabled |
+| `flower.enabled` | Enables the Flower web interface for Celery. | `false` |
+| `pgbouncer.enabled` | Enables PgBouncer for PostgreSQL connection pooling. | `false` |
+
+Additional configuration options can be customized through the Helm chart values.
+
+## Installation
+
+Deploy the Apache Airflow pack from Palette.
+
+Wait until the pack reaches the **Healthy** state before proceeding with validation.
+
+## Validation
+
+Verify that the Apache Airflow components are running successfully.
+
+```bash
+kubectl get all -n airflow
+```
+
+Verify the status of the Airflow pods.
+
+```bash
+kubectl get pods -n airflow
+```
+
+Review the logs of an Airflow component if necessary.
+
+```bash
+kubectl logs deployment/airflow-api-server -n airflow
+```
+
+## Accessing the Airflow UI
+
+If an Ingress is not configured, forward the Airflow API Server service locally.
+
+```bash
+kubectl port-forward svc/airflow-api-server 8080:8080 -n airflow
+```
+
+Open the Airflow web interface using a web browser.
+
+```text
+http://localhost:8080
+```
+
+## Workflow Validation
+
+After accessing the Airflow interface, create or deploy a DAG to validate workflow execution.
+
+A DAG can be defined using Python and should contain one or more tasks with defined dependencies.
+
+Verify that:
+
+- The DAG is detected by Airflow.
+- The DAG can be triggered successfully.
+- The tasks are scheduled and executed.
+- The task status changes to **Success**.
+- Task logs can be viewed from the Airflow UI.
+
+
+## References
+
+- https://airflow.apache.org/
+- https://github.com/apache/airflow
+- https://airflow.apache.org/docs/apache-airflow/3.2.2/
+

packs/apache-airflow-1.22.0/charts/airflow/.helmignore

@@ -0,0 +1,42 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Patterns to ignore when building packages.
+# This supports shell glob matching, relative path matching, and
+# negation (prefixed with !). Only one pattern per line.
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.bak
+*.tmp
+*~
+# Various IDEs
+.project
+.idea/
+*.tmproj
+bin
+
+# We do not want to include our Python Helm Chart Unit test files
+tests

packs/apache-airflow-1.22.0/charts/airflow/.pre-commit-config.yaml

@@ -0,0 +1,118 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+---
+default_stages: [pre-commit, pre-push]
+minimum_prek_version: '0.2.0'
+default_language_version:
+  python: python3
+  node: 22.19.0
+  golang: 1.24.0
+repos:
+  - repo: https://github.com/Lucas-C/pre-commit-hooks
+    rev: ad1b27d73581aa16cca06fc4a0761fc563ffe8e8  # frozen: v1.5.6
+    hooks:
+      - id: insert-license
+        name: Add license for all Helm template files
+        files: ^templates/.*
+        args:
+          - --comment-style
+          - "{{/*||*/}}"
+          - --license-filepath
+          - ../scripts/ci/license-templates/LICENSE.txt
+          - --fuzzy-match-generates-todo
+  - repo: local
+    hooks:
+      - id: update-chart-dependencies
+        name: Update chart dependencies to latest (manual)
+        entry: ../scripts/ci/prek/update_chart_dependencies.py
+        stages: ['manual']
+        language: python
+        files: ^\.pre-commit-config\.yaml$|^../scripts/ci/prek/update_build_dependencies\.py$
+        pass_filenames: false
+        require_serial: true
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: 3e8a8703264a2f4a69428a0aa4dcb512790b2c8c  # frozen: v6.0.0
+    hooks:
+      - id: pretty-format-json
+        name: Format JSON files
+        args:
+          - --autofix
+          - --no-sort-keys
+          - --indent
+          - "4"
+        files:
+          (?x)
+          ^values\.schema\.json$|
+          ^values_schema\.schema\.json$
+        pass_filenames: true
+  - repo: local
+    hooks:
+      - id: lint-helm-chart
+        name: Lint Helm Chart
+        entry: ../scripts/ci/prek/lint_helm.py
+        language: python
+        pass_filenames: false
+        files: ^.*
+        require_serial: true
+      - id: validate-chart-annotations
+        name: Validate chart annotations
+        entry: ../scripts/ci/prek/validate_chart_annotations.py
+        language: python
+        pass_filenames: false
+        files: ^Chart\.yaml$
+      - id: kubeconform
+        name: Kubeconform check on our helm chart
+        entry: ../scripts/ci/prek/check_kubeconform.py
+        language: python
+        pass_filenames: false
+        files: ^.*
+        require_serial: true
+      - id: lint-json-schema
+        name: Lint chart/values.schema.json
+        entry: ../scripts/ci/prek/lint_json_schema.py
+        args:
+          - --spec-file
+          - values_schema.schema.json
+          - values.schema.json
+        language: python
+        pass_filenames: false
+        files: ^values\.schema\.json$|^values_schema\.schema\.json$
+        require_serial: true
+      - id: update-vendored-in-k8s-json-schema
+        name: Vendor k8s definitions into values.schema.json
+        entry: ../scripts/ci/prek/vendor_k8s_json_schema.py
+        language: python
+        files: ^values\.schema\.json$
+      - id: lint-json-schema
+        name: Lint chart/values.yaml
+        entry: ../scripts/ci/prek/lint_json_schema.py
+        args:
+          - --enforce-defaults
+          - --spec-file
+          - values.schema.json
+          - values.yaml
+        language: python
+        pass_filenames: false
+        files: ^values\.yaml$|^values\.schema\.json$
+        require_serial: true
+      - id: lint-chart-schema
+        name: Lint chart/values.schema.json file
+        entry: ../scripts/ci/prek/chart_schema.py
+        language: python
+        pass_filenames: false
+        files: ^values\.schema\.json$
+        require_serial: true

packs/apache-airflow-1.22.0/charts/airflow/Chart.lock

@@ -0,0 +1,6 @@
+dependencies:
+- name: postgresql
+  repository: https://charts.bitnami.com/bitnami
+  version: 13.2.24
+digest: sha256:07f12ed410f106bf13eca69df16a1ef6690c4d4bfcb037943bbff6e71a22201d
+generated: "2023-12-09T17:23:53.209725+01:00"