Skip to content

About

A production-grade, end-to-end MLOps application for automated video content moderation. Built on Azure AKS (Kubernetes) + AWS (S3, SQS, DynamoDB) with Azure ML for NSFW and Violence detection model training and deployment to a fully automated Python/CLI deployment pipeline.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

9 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

MultiCloud-Guardian-AI

Video Content Moderation

A production-grade, end-to-end MLOps application for automated video content moderation. Built on Azure AKS (Kubernetes) + AWS (S3, SQS, DynamoDB) with Azure ML for NSFW and Violence detection model training and deployment to a fully automated Python/CLI deployment pipeline.

Project Description

This project orchestrates a Multi-Cloud MLOps Workflow. Azure ML provides the workspace, compute clusters for training, a model registry for versioning, and managed online endpoints for inference. Azure DevOps runs CI/CD pipelines that submit training jobs to the cluster, register trained models automatically, and deploy them to online endpoints. MLflow tracks experiments and metrics with an Azure ML backend.

Application microservices (e.g., Deep-Vision) run on Azure Kubernetes Service (AKS), read video references from AWS S3 and metadata from AWS DynamoDB, process messages from SQS, and call Azure ML endpoints for model scoring. Code pushed to the repository triggers the training pipeline on the compute cluster, models appear in the registry, the deployment pipeline creates or updates endpoints, and Kubernetes ConfigMaps are updated so application pods use the new scoring URIs. This end-to-end flow demonstrates automated, reproducible model training and deployment across AWS (data/messaging) and Azure (ML platform and CI/CD).

The Business Case: Why This Matters

Problem: Manual content moderation does not scale. YouTube receives over 500 hours of video uploads per minute. Platforms growing at that velocity cannot review every upload by hand — the throughput gap between human capacity and content volume is structural, not a staffing problem.

The Challenge: To scale and standardize decisions, you need to automate model training and deployment: train NSFW and violence-detection models on scalable compute, register and version them, deploy to managed inference endpoints, and have application services (which store videos and metadata on AWS) call those endpoints for real-time scoring.

The consequence: platforms either miss harmful content, delay approvals unacceptably, or burn through reviewer capacity on routine cases that automation should handle. The Multi-Cloud-Guardian-AI is built to close that gap — automating the high-volume, high-confidence decisions so human reviewers focus only on the cases that actually need judgment.

The Solution: This project uses Azure for the ML platform and AWS for data and messaging, with Azure DevOps pipelines orchestrating training and deployment so that new model versions flow from code commit to production endpoints.

Cross-Cloud Integration – Data on AWS, ML on Azure

The Architecture: Keeps data and messaging on AWS and ML training, registry, and inference on Azure. Application services (on AKS or elsewhere) use AWS for S3, SQS, and DynamoDB, and use Azure only for calling the ML endpoints and for CI/CD (Azure DevOps and Azure ML).

Definition: Separation of responsibilities: AWS for storage, queues, and application metadata; Azure for ML workspace, compute, registry, endpoints, and pipeline orchestration; application code connects both via configuration (endpoint URLs, keys, and AWS credentials).

Use Case: Optimize cost and capability, use AWS for high-scale object storage and queues, and Azure for a full-featured ML platform and managed endpoints; keep a single application codebase that talks to both clouds via environment variables and SDKs.

Example: Video upload → S3 + DynamoDB (AWS); processing triggered by SQS (AWS); Deep-Vision runs on AKS, reads from S3/DynamoDB, calls Azure ML endpoints for scores, writes results back to DynamoDB; training and deployment are fully on Azure via Azure DevOps and Azure ML.

Design Decisions

  • Two-stage screening (Fast Screening → Deep Vision) reduces Azure ML endpoint calls to only ambiguous content, lowering inference cost and latency.
  • SQS decoupling isolates ingestion from ML processing — a spike in uploads does not block scoring, and each layer scales independently.
  • CI/CD-gated model promotion via Azure DevOps ensures no model version reaches production without passing validation, enabling safe rollback to any registered version.

Results and Impact

Impact: This project demonstrates how an automated multi-cloud moderation pipeline can improve speed, consistency, and scalability. By using ML-driven pre-screening, the system is designed to reduce manual review volume by an estimated 70–80% (industry benchmark) and shorten moderation turnaround by 80–90% (industry benchmark) compared with a fully manual process.

Business Value: From an operational perspective, that can mean fewer staff-hours spent on routine review, faster handling of flagged content, and an estimated 20–40% reduction in moderation operating cost depending on traffic mix and escalation rates. The architecture also improves repeatability and operational safety through versioned models, CI/CD deployment, and managed inference endpoints.

Example: For a platform handling 10,000 videos per day, a 70% automation rate would leave about 3,000 items for human review. That creates a practical path to lower cost and faster response without sacrificing oversight.

Business Value Delivered

❌ Slow, headcount-constrained, inconsistent, hard-to-scale manual process to a:

✅ Automated, auditable, elastic, cost-efficient, multi-cloud ML pipeline

Key outcomes:

  • ~98% reduction in per-video ML scoring latency vs. human review
  • ~80% total operational cost reduction at scale (infrastructure + labour)
  • 70% faster model iteration via Azure DevOps CI/CD automation
  • Zero-touch deployment: code commit → trained model → live endpoint
  • 100% smoke test gate: 69/69 tests block broken builds
  • Dual-cloud architecture: best-of-breed AWS (data) + Azure (ML) at ~$290/month
  • Compliance-ready: full model lineage, versioning, and audit trail via Azure ML + MLflow

Architecture

image (7)
Upload → API Gateway → Ingestion (S3) → SQS → Fast Screening
                                                     ↓
                                              Deep Vision (ML)
                                                     ↓
                                         Policy Engine → Decision
                                                     ↓
                                    Auto-Approve / Auto-Reject / Human Review

AWS (us-east-1):

  • S3: video storage (guardian-videos-*)
  • SQS: guardian-video-processing, guardian-gpu-processing
  • DynamoDB: guardian-videos, guardian-events

Azure (East US):

  • AKS: 4-node cluster (Standard_D2s_v3) running 8 microservices
  • ACR: Docker image registry
  • Azure ML: NSFW + Violence detection model training & deployment
  • Azure DevOps: CI/CD pipelines (3 pipelines)

Prerequisites

Tool Version Install
Python 3.11+ python.org
AWS CLI v2 brew install awscli
Azure CLI latest brew install azure-cli
kubectl latest brew install kubectl
Helm 3+ brew install helm
Docker Desktop latest docker.com
Node.js 20+ brew install node
git any pre-installed on macOS

Project Structure

MultiCloud-Video-Guardian-AI/   # repo root
├── setup.py                          # Main orchestrator (run this)
├── .env.example                      # Environment variables template
├── requirements.txt                  # Python dependencies
│
├── automation/                       # Stage automation scripts
│   ├── config.py                     # Central config, reads .env
│   ├── aws/setup_aws.py             # Stage: aws
│   ├── azure/
│   │   ├── setup_azure.py           # Stage: azure
│   │   └── build_images.py          # Stage: images
│   ├── k8s/deploy_k8s.py            # Stage: k8s
│   ├── devops/setup_devops.py       # Stage: devops
│   ├── ml/setup_ml.py               # Stage: ml
│   └── utils/shell.py               # CLI runner, logging
│
├── scripts/
│   └── setup-aws.sh                 # AWS resource creation (called by aws stage)
│
├── services/                        # Microservice source code
│   ├── ingestion/                   # Video upload & S3 storage
│   ├── fast-screening/              # Quick ML screening
│   ├── deep-vision/                 # Deep learning analysis
│   ├── policy-engine/               # Decision logic
│   ├── human-review/                # Manual review interface
│   ├── notification/                # Alerting service
│   └── api-gateway/                 # REST API gateway
│
├── frontend/                        # React TypeScript UI
├── k8s/                             # Kubernetes manifests
├── mlops/
│   ├── training/                    # Model training scripts
│   └── deployment/                  # Model deployment scripts
│
├── azure-pipelines-app-ci-cd.yml    # App CI/CD pipeline
├── azure-pipelines-ml-training.yml  # ML training pipeline
├── azure-pipelines-ml-deployment.yml # ML deployment pipeline
│
├── tests/smoke/test_smoke.py        # 69 smoke tests
└── aws_resource_cleanup.sh          # AWS teardown script

Cost Estimates

Resource Monthly Cost
AKS (4x Standard_D2s_v3) ~$280
Azure ML compute (cpu-training-cluster, 0 min instances) ~$0 (scales to 0)
ACR Standard ~$5
S3 + SQS + DynamoDB (light usage) ~$5
Total ~$290/month

⚠️ Cost Warning: AKS nodes run 24/7. Stop the cluster when not in use:

az aks stop --name guardian-ai-aks --resource-group guardian-ai-prod
az aks start --name guardian-ai-aks --resource-group guardian-ai-prod

Quick Start

1. Clone and Configure

git clone https://github.com/MAOFILHO/MultiCloud-Video-Guardian-AI.git
cd MultiCloud-Video-Guardian-AI

# Install Python dependencies
python3.11 -m venv .venv
source .venv/bin/activate          # macOS/Linux
pip install -r requirements.txt

# Configure credentials
cp .env.example .env
# Edit .env with your values (see CONFIGURATION section below)

2. Configure AWS

aws configure
# Enter: Access Key ID, Secret Access Key, Region: us-east-1, Output: json

3. Configure Azure

az login
az account set --subscription <YOUR_SUBSCRIPTION_ID>

4. Run Automated Setup (Stages)

python setup.py --stage aws      # ~2 min:   S3, SQS, DynamoDB
python setup.py --stage azure    # ~15 min:  RG, ACR, AKS, NGINX
python setup.py --stage images   # ~20 min:  Docker build & push (8 images)
python setup.py --stage k8s      # ~5 min:   Deploy all services to AKS
python setup.py --stage devops   # ~5 min:   Azure DevOps pipelines
python setup.py --stage ml       # ~45 min:  Azure ML training + deployment

After --stage k8s completes, the app is live at: http://<EXTERNAL_IP>

Configuration

Copy .env.example to .env and fill in all <YOUR_VALUE> placeholders:

Screenshot 2026-06-13 at 10 49 25 AM

AWS Credentials

AWS_ACCESS_KEY_ID=<from IAM user>
AWS_SECRET_ACCESS_KEY=<from IAM user>
AWS_REGION=us-east-1

Azure Credentials

AZURE_SUBSCRIPTION_ID=<az account show --query id -o tsv>

Azure Service Principal (required for DevOps)

# Create SP:
az ad sp create-for-rbac \
  --name guardian-ai-sp \
  --role Contributor \
  --scopes /subscriptions/<YOUR_SUBSCRIPTION_ID>

# Fill in output values:
AZURE_SP_APP_ID=<appId>
AZURE_SP_SECRET=<password>
AZURE_TENANT_ID=<tenant>

Azure DevOps

AZDO_ORG_NAME=<your-org-name>     # from dev.azure.com/<ORG>
AZDO_PAT_TOKEN=<personal-access-token>

Generate PAT at: https://dev.azure.com → Profile → Personal access tokens
Required scopes: Agent Pools, Build, Code, Project & Team, Service Connections, Variable Groups

Stage Details

Stage 1: AWS (--stage aws)

Creates: S3 bucket, 2 SQS queues, 2 DynamoDB tables
Duration: ~2 minutes
Region: us-east-1 (hardcoded guardian — will error if unconfigured)

Screenshot 2026-06-12 at 6 19 28 PM

Screenshot 2026-06-13 at 1 26 56 PM

Screenshot 2026-06-13 at 1 27 44 PM

Screenshot 2026-06-13 at 1 28 35 PM

Stage 2: Azure (--stage azure)

Creates: Resource group, ACR, AKS cluster (4 nodes), NGINX Ingress
Duration: ~15 minutes (AKS is the long step)
Output: External IP for the app

Screenshot 2026-06-12 at 6 21 24 PM

Screenshot 2026-06-13 at 1 38 58 PM

Screenshot 2026-06-13 at 1 42 40 PM

Screenshot 2026-06-13 at 1 44 03 PM

Stage 3: Docker Images (--stage images)

Builds and pushes 8 Docker images to ACR:

  • ingestion, fast-screening, deep-vision, policy-engine
  • human-review, notification, api-gateway, frontend
    Duration: ~20 minutes (first run; subsequent runs use layer cache)
Screenshot 2026-06-12 at 6 31 52 PM

Screenshot 2026-06-12 at 6 32 32 PM

Screenshot 2026-06-13 at 1 47 13 PM

Stage 4: Kubernetes (--stage k8s)

Deploys: Namespace, ConfigMap, AWS secrets, all 8 services + Redis, Ingress
Duration: ~5 minutes
✅ App is live after this stage

Screenshot 2026-06-13 at 11 37 43 AM

Screenshot 2026-06-13 at 11 42 07 AM

Screenshot 2026-06-13 at 1 53 50 PM

Stage 5: DevOps (--stage devops)

Creates: Azure DevOps project, 3 service connections, variable group, 3 pipelines
Duration: ~5 minutes
Requires: AZDO_PAT_TOKEN, AZURE_SP_APP_ID, AZURE_SP_SECRET, AZURE_TENANT_ID

Screenshot 2026-06-12 at 9 56 33 PM

Screenshot 2026-06-12 at 9 56 49 PM

Screenshot 2026-06-13 at 1 57 21 PM

Stage 6: ML (--stage ml)

Creates: Azure ML workspace, compute cluster, triggers training pipeline
Duration: ~45 minutes (training on cpu-training-cluster)
Output: NSFW + Violence detection endpoints; patches Kubernetes ConfigMap

Screenshot 2026-06-13 at 11 44 25 AM

Screenshot 2026-06-13 at 2 00 21 PM

Screenshot 2026-06-13 at 2 00 58 PM

Screenshot 2026-06-13 at 2 02 15 PM

Screenshot 2026-06-13 at 2 02 30 PM

Screenshot 2026-06-13 at 2 02 43 PM

Screenshot 2026-06-13 at 2 03 01 PM

Running Tests

# Smoke tests (no cloud credentials required)
python -m pytest tests/smoke/test_smoke.py -v

# Expected: 69 tests passing
Screenshot 2026-06-13 at 11 27 52 AM

The Video Moderation Application (Screenshots)

Screenshot 2026-06-12 at 7 14 59 PM

Screenshot 2026-06-12 at 7 15 30 PM

Screenshot 2026-06-12 at 7 21 29 PM

Screenshot 2026-06-12 at 7 21 45 PM

Screenshot 2026-06-12 at 7 22 09 PM

Screenshot 2026-06-12 at 7 24 39 PM

Screenshot 2026-06-12 at 7 29 56 PM

Screenshot 2026-06-12 at 7 29 38 PM

GitHub Actions CI/CD

Two GitHub Actions workflows live in .github/workflows/: ci.yml (smoke tests + frontend build on every push/PR, no cloud access) and deploy.yml, which builds and deploys the service images to AKS and runs manually only. While this project lived in the Portfolio-Projects monorepo, the deploy ran automatically on pushes to services/**, k8s/** and frontend/**.

Required GitHub Secrets

Add at: github.com/MAOFILHO/MultiCloud-Video-Guardian-AI → Settings → Secrets → Actions

  • ACR_USERNAME — run: az acr credential show --name guardianacr03211 --query username -o tsv
  • ACR_PASSWORD — run: az acr credential show --name guardianacr03211 --query 'passwords[0].value' -o tsv
  • AZURE_CREDENTIALS — run: az ad sp create-for-rbac --name guardian-github-actions --role Contributor --scopes /subscriptions/960936b9-ecde-465b-be8d-776ca077dcd0 --sdk-auth
Screenshot 2026-06-13 at 11 01 23 AM

Trigger Automatically (monorepo only)

In the monorepo, pushing a change to a service file ran the deploy workflow automatically. The standalone repo's deploy.yml is manual-only, so use the steps below.

Screenshot 2026-06-13 at 12 25 18 PM

Trigger Manually

GitHub UI: Actions tab → Deploy to AKS → Run workflow

Screenshot 2026-06-13 at 9 51 23 AM

Screenshot 2026-06-13 at 10 11 53 AM

Important Notes

  • AKS must be running before deployment: az aks start --name guardian-ai-aks --resource-group guardian-ai-prod
  • The deploy workflow never fires on push; trigger it from the Actions tab
  • Rotate the guardian-github-actions SP secret annually

Project Key Benefits - Summary

Automation: Code commit triggers or manually run pipelines that train on the cluster, register models, and deploy to endpoints, no manual promotion or server management.

Consistency: Every deployment uses the same pipeline and the same registry versions; rollbacks can target a previous model version.

Explainability and traceability: MLflow and Azure ML Studio provide run history, metrics, and model lineage; registry versions tie deployments to specific training runs.

Scalability: Training scales on Azure ML compute clusters; inference scales via managed endpoints; application layer scales independently on AWS and AKS.

Speed: New model versions move from code to production in one pipeline run; application pods pick up new endpoints via ConfigMap updates and restarts. Multi-cloud flexibility: Use AWS for data and messaging and Azure for ML and CI/CD, with clear boundaries and minimal coupling.

Teardown / Cleanup

AWS Resources

chmod +x aws_resource_cleanup.sh
./aws_resource_cleanup.sh
Screenshot 2026-06-13 at 2 13 43 PM

Screenshot 2026-06-13 at 2 14 09 PM

Azure Resources

python automation/cleanup.py
# OR manually:
az group delete --name guardian-ai-prod --yes --no-wait
Screenshot 2026-06-13 at 2 32 41 PM

Screenshot 2026-06-13 at 2 40 31 PM

Verify cleanup

# AWS
echo "S3:"; aws s3 ls | grep guardian
echo "SQS:"; aws sqs list-queues --region us-east-1
echo "DDB:"; aws dynamodb list-tables --region us-east-1

# Azure
az resource list --resource-group guardian-ai-prod --output table
Screenshot 2026-06-13 at 2 43 54 PM

Lessons Learned

This lab required significant troubleshooting beyond the original manual. Key lessons:

  1. Always set AWS_REGION explicitly — never rely on CLI defaults in scripts
  2. S3 in us-east-1 is special — it's the default region and rejects LocationConstraint
  3. Azure DevOps CLI has gaps — some service connection types (AKS Kubeconfig, Token) require the REST API
  4. ACR must be attached to AKS after creation via az aks update --attach-acr
  5. Portfolio-Projects repo structure matters — pipeline YAML paths must include the subfolder prefix when the pipeline YAML isn't at repo root
  6. Self-hosted agents require network access — Microsoft-hosted ubuntu-latest is simpler for labs
  7. Never commit secrets — use .env + .gitignore, rotate credentials immediately if exposed

Troubleshooting - Known Issues & Workarounds

1. AWS Region Mismatch (CRITICAL)

Symptom: Resources created in the wrong region despite AWS_REGION=us-east-1
Root cause: Original setup-aws.sh had REGION="ap-south-1" hardcoded
Fix: Already patched — script now reads ${AWS_REGION:-$(aws configure get region)}

2. S3 Bucket Creation Fails in us-east-1

Symptom: An error occurred (InvalidLocationConstraint)
Root cause: us-east-1 is S3's default region and rejects LocationConstraint
Fix: Already patched — script uses conditional creation without LocationConstraint for us-east-1

3. S3 Bucket "OperationAborted" After Deletion

Symptom: A conflicting conditional operation is currently in progress
Root cause: AWS enforces a ~10-minute cooldown before a deleted bucket name can be reused
Workaround: Add S3_BUCKET_OVERRIDE=guardian-videos-yourname-dev to .env

4. ACR Credentials Lost Between Stages

Symptom: username is empty when running --stage images standalone
Root cause: ACR credentials were only held in-memory during --stage azure
Fix: Already patched — build_images.py fetches credentials from Azure CLI when not in config; setup_azure.py persists AZURE_ACR_NAME to .env

5. Pods in ErrImagePull After k8s Deployment

Symptom: All pods stuck in ErrImagePull
Root cause: New ACR not attached to AKS cluster
Fix: Run manually after --stage k8s if pods fail:

az aks update \
  --name guardian-ai-aks \
  --resource-group guardian-ai-prod \
  --attach-acr <YOUR_ACR_NAME>
kubectl rollout restart deployment -n production

6. Azure DevOps Service Connection — Interactive Prompt

Symptom: Azure RM service principal key: prompt during --stage devops
Root cause: az devops service-endpoint azurerm create prompts for SP key interactively
Fix: Set these in .env before running --stage devops:

AZURE_SP_APP_ID=<appId>
AZURE_SP_SECRET=<password>
AZURE_TENANT_ID=<tenant>

7. Self-Hosted Agent Download Fails

Symptom: socket.gaierror: nodename nor servname provided downloading agent
Root cause: vstsagentpackage.azureedge.net blocked or unavailable
Fix: Already patched — setup skips self-hosted agent and uses Microsoft-hosted ubuntu-latest pool instead

8. Pipeline YAML "Could not find valid pipeline YAML file"

Symptom: Azure DevOps pipeline fails immediately with YAML not found
Root cause: Azure Repos has the full Portfolio-Projects structure, so YAMLs are under MultiCloud-Video-Guardian-AI/ subfolder, not at root
Fix: Already patched — pipeline YAMLs reference correct paths. If you re-create pipelines, update the YAML path in Azure DevOps to MultiCloud-Video-Guardian-AI/azure-pipelines-*.yml

9. ML Training Pipeline "No agent found in pool Default"

Symptom: Training jobs fail immediately
Root cause: Pipeline YAML had pool: name: 'Default' requiring a self-hosted agent
Fix: Already patched — all pipeline YAMLs use pool: vmImage: 'ubuntu-latest'

10. ML Training Path Error

Symptom: Cannot find path '/home/vsts/work/1/s/mlops/training'
Root cause: Pipeline runs from Portfolio-Projects repo root, so MLOps scripts are under MultiCloud-Video-Guardian-AI/mlops/
Fix: Already patched — pipeline YAMLs use cd MultiCloud-Video-Guardian-AI/mlops/training

11. Duplicate ACR Created on Each Azure Stage Run

Symptom: New guardianacr* created every time --stage azure runs
Root cause: ACR name was timestamp-generated and not persisted
Fix: Already patched — setup_azure.py saves AZURE_ACR_NAME= to .env after first creation

Authors

License

MIT License — see LICENSE

About

A production-grade, end-to-end MLOps application for automated video content moderation. Built on Azure AKS (Kubernetes) + AWS (S3, SQS, DynamoDB) with Azure ML for NSFW and Violence detection model training and deployment to a fully automated Python/CLI deployment pipeline.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages