Kubernetes LoadBalancer Service Monitor

This tool monitors Kubernetes LoadBalancer services for changes and starts a Woodpecker CI pipeline in response to those changes.

Overview

The service monitor watches for any changes (creation, modification, or deletion) to LoadBalancer services across all namespaces in your Kubernetes cluster. When a change is detected, it starts a manual Woodpecker pipeline that deploys the loadbalancer configs.

Prerequisites

• Kubernetes cluster access
• Woodpecker API token (mint one at <woodpecker-url>/user/cli-and-api)
• Docker (for building the container image)
• kubectl configured with cluster access

Configuration

1. Build the Docker Image

docker build -t your-registry/service-monitor:latest .
docker push your-registry/service-monitor:latest

2. Create Credentials Secret

kubectl create namespace monitoring
kubectl create secret generic service-monitor-credentials \
    --namespace monitoring \
    --from-literal=WOODPECKER_URL='https://your-woodpecker' \
    --from-literal=WOODPECKER_TOKEN='YOUR_WOODPECKER_TOKEN' \
    --from-literal=WOODPECKER_REPO='owner/repo' \
    --from-literal=TENANT='your-tenant' \
    --from-literal=PROJECT='your-project' \
    --from-literal=MATTERMOST_WEBHOOK_URL='OPTIONAL_WEBHOOK' \
    --from-literal=K8S_CLUSTER_NAME='OPTIONAL_CLUSTER_NAME'

Optional keys: PLAYBOOK (defaults to playbook.yaml, relative to <tenant>/<project>/ansible/) and ANSIBLE_ARGS (extra ansible-playbook args, e.g. --tags k8s-update).

3. Update Deployment Image

Edit k8s/deployment.yaml and update the image field with your registry path:

image: your-registry/service-monitor:latest

4. Deploy to Kubernetes

kubectl apply -f k8s/rbac.yaml
kubectl apply -f k8s/deployment.yaml

Verification

Check if the pod is running:

kubectl get pods -n monitoring

View the logs:

kubectl logs -n monitoring -l app=service-monitor -f

How It Works

1. The service monitor uses the Kubernetes API to watch for changes in LoadBalancer services
2. When a change is detected (debounced to once per 3 minutes), it starts a manual Woodpecker pipeline on branch main with:
   • QUOKKA_TOOL=ansible
   • QUOKKA_PROJECT_DIR=$TENANT/$PROJECT
   • QUOKKA_PLAYBOOK=$PLAYBOOK
   • QUOKKA_ANSIBLE_ARGS=$ANSIBLE_ARGS (only when set)

Troubleshooting

Check Pod Status

kubectl describe pod -n monitoring -l app=service-monitor

Check Logs

kubectl logs -n monitoring -l app=service-monitor -f

Common Issues

1. Pod can't pull image: Check your image registry credentials and image path
2. Permission denied: Verify RBAC permissions are correctly configured
3. Pipeline not starting: Check the Woodpecker token validity and that the repo is activated in Woodpecker

Security Considerations

• The service runs with minimal permissions using RBAC
• The container runs as a non-root user
• The filesystem is read-only
• The container has resource limits defined

Maintenance

• Regularly update the dependencies in requirements.txt
• Monitor the pod's resource usage and adjust limits as needed
• Rotate the Woodpecker token periodically
• Keep the Docker base image updated for security patches