A production-grade, end-to-end MLOps application for automated video content moderation. Built on Azure AKS (Kubernetes) + AWS (S3, SQS, DynamoDB) with Azure ML for NSFW and Violence detection model training and deployment to a fully automated Python/CLI deployment pipeline.
This project orchestrates a Multi-Cloud MLOps Workflow. Azure ML provides the workspace, compute clusters for training, a model registry for versioning, and managed online endpoints for inference. Azure DevOps runs CI/CD pipelines that submit training jobs to the cluster, register trained models automatically, and deploy them to online endpoints. MLflow tracks experiments and metrics with an Azure ML backend.
Application microservices (e.g., Deep-Vision) run on Azure Kubernetes Service (AKS), read video references from AWS S3 and metadata from AWS DynamoDB, process messages from SQS, and call Azure ML endpoints for model scoring. Code pushed to the repository triggers the training pipeline on the compute cluster, models appear in the registry, the deployment pipeline creates or updates endpoints, and Kubernetes ConfigMaps are updated so application pods use the new scoring URIs. This end-to-end flow demonstrates automated, reproducible model training and deployment across AWS (data/messaging) and Azure (ML platform and CI/CD).
Problem: Manual content moderation does not scale. YouTube receives over 500 hours of video uploads per minute. Platforms growing at that velocity cannot review every upload by hand — the throughput gap between human capacity and content volume is structural, not a staffing problem.
The Challenge: To scale and standardize decisions, you need to automate model training and deployment: train NSFW and violence-detection models on scalable compute, register and version them, deploy to managed inference endpoints, and have application services (which store videos and metadata on AWS) call those endpoints for real-time scoring.
The consequence: platforms either miss harmful content, delay approvals unacceptably, or burn through reviewer capacity on routine cases that automation should handle. The Multi-Cloud-Guardian-AI is built to close that gap — automating the high-volume, high-confidence decisions so human reviewers focus only on the cases that actually need judgment.
The Solution: This project uses Azure for the ML platform and AWS for data and messaging, with Azure DevOps pipelines orchestrating training and deployment so that new model versions flow from code commit to production endpoints.
The Architecture: Keeps data and messaging on AWS and ML training, registry, and inference on Azure. Application services (on AKS or elsewhere) use AWS for S3, SQS, and DynamoDB, and use Azure only for calling the ML endpoints and for CI/CD (Azure DevOps and Azure ML).
Definition: Separation of responsibilities: AWS for storage, queues, and application metadata; Azure for ML workspace, compute, registry, endpoints, and pipeline orchestration; application code connects both via configuration (endpoint URLs, keys, and AWS credentials).
Use Case: Optimize cost and capability, use AWS for high-scale object storage and queues, and Azure for a full-featured ML platform and managed endpoints; keep a single application codebase that talks to both clouds via environment variables and SDKs.
Example: Video upload → S3 + DynamoDB (AWS); processing triggered by SQS (AWS); Deep-Vision runs on AKS, reads from S3/DynamoDB, calls Azure ML endpoints for scores, writes results back to DynamoDB; training and deployment are fully on Azure via Azure DevOps and Azure ML.
- Two-stage screening (Fast Screening → Deep Vision) reduces Azure ML endpoint calls to only ambiguous content, lowering inference cost and latency.
- SQS decoupling isolates ingestion from ML processing — a spike in uploads does not block scoring, and each layer scales independently.
- CI/CD-gated model promotion via Azure DevOps ensures no model version reaches production without passing validation, enabling safe rollback to any registered version.
Impact: This project demonstrates how an automated multi-cloud moderation pipeline can improve speed, consistency, and scalability. By using ML-driven pre-screening, the system is designed to reduce manual review volume by an estimated 70–80% (industry benchmark) and shorten moderation turnaround by 80–90% (industry benchmark) compared with a fully manual process.
Business Value: From an operational perspective, that can mean fewer staff-hours spent on routine review, faster handling of flagged content, and an estimated 20–40% reduction in moderation operating cost depending on traffic mix and escalation rates. The architecture also improves repeatability and operational safety through versioned models, CI/CD deployment, and managed inference endpoints.
Example: For a platform handling 10,000 videos per day, a 70% automation rate would leave about 3,000 items for human review. That creates a practical path to lower cost and faster response without sacrificing oversight.
❌ Slow, headcount-constrained, inconsistent, hard-to-scale manual process to a:
✅ Automated, auditable, elastic, cost-efficient, multi-cloud ML pipeline
Key outcomes:
- ~98% reduction in per-video ML scoring latency vs. human review
- ~80% total operational cost reduction at scale (infrastructure + labour)
- 70% faster model iteration via Azure DevOps CI/CD automation
- Zero-touch deployment: code commit → trained model → live endpoint
- 100% smoke test gate: 69/69 tests block broken builds
- Dual-cloud architecture: best-of-breed AWS (data) + Azure (ML) at ~$290/month
- Compliance-ready: full model lineage, versioning, and audit trail via Azure ML + MLflow
Upload → API Gateway → Ingestion (S3) → SQS → Fast Screening
↓
Deep Vision (ML)
↓
Policy Engine → Decision
↓
Auto-Approve / Auto-Reject / Human Review
AWS (us-east-1):
- S3: video storage (
guardian-videos-*) - SQS:
guardian-video-processing,guardian-gpu-processing - DynamoDB:
guardian-videos,guardian-events
Azure (East US):
- AKS: 4-node cluster (
Standard_D2s_v3) running 8 microservices - ACR: Docker image registry
- Azure ML: NSFW + Violence detection model training & deployment
- Azure DevOps: CI/CD pipelines (3 pipelines)
| Tool | Version | Install |
|---|---|---|
| Python | 3.11+ | python.org |
| AWS CLI | v2 | brew install awscli |
| Azure CLI | latest | brew install azure-cli |
| kubectl | latest | brew install kubectl |
| Helm | 3+ | brew install helm |
| Docker Desktop | latest | docker.com |
| Node.js | 20+ | brew install node |
| git | any | pre-installed on macOS |
MultiCloud-Video-Guardian-AI/ # repo root
├── setup.py # Main orchestrator (run this)
├── .env.example # Environment variables template
├── requirements.txt # Python dependencies
│
├── automation/ # Stage automation scripts
│ ├── config.py # Central config, reads .env
│ ├── aws/setup_aws.py # Stage: aws
│ ├── azure/
│ │ ├── setup_azure.py # Stage: azure
│ │ └── build_images.py # Stage: images
│ ├── k8s/deploy_k8s.py # Stage: k8s
│ ├── devops/setup_devops.py # Stage: devops
│ ├── ml/setup_ml.py # Stage: ml
│ └── utils/shell.py # CLI runner, logging
│
├── scripts/
│ └── setup-aws.sh # AWS resource creation (called by aws stage)
│
├── services/ # Microservice source code
│ ├── ingestion/ # Video upload & S3 storage
│ ├── fast-screening/ # Quick ML screening
│ ├── deep-vision/ # Deep learning analysis
│ ├── policy-engine/ # Decision logic
│ ├── human-review/ # Manual review interface
│ ├── notification/ # Alerting service
│ └── api-gateway/ # REST API gateway
│
├── frontend/ # React TypeScript UI
├── k8s/ # Kubernetes manifests
├── mlops/
│ ├── training/ # Model training scripts
│ └── deployment/ # Model deployment scripts
│
├── azure-pipelines-app-ci-cd.yml # App CI/CD pipeline
├── azure-pipelines-ml-training.yml # ML training pipeline
├── azure-pipelines-ml-deployment.yml # ML deployment pipeline
│
├── tests/smoke/test_smoke.py # 69 smoke tests
└── aws_resource_cleanup.sh # AWS teardown script
| Resource | Monthly Cost |
|---|---|
| AKS (4x Standard_D2s_v3) | ~$280 |
| Azure ML compute (cpu-training-cluster, 0 min instances) | ~$0 (scales to 0) |
| ACR Standard | ~$5 |
| S3 + SQS + DynamoDB (light usage) | ~$5 |
| Total | ~$290/month |
⚠️ Cost Warning: AKS nodes run 24/7. Stop the cluster when not in use:az aks stop --name guardian-ai-aks --resource-group guardian-ai-prod az aks start --name guardian-ai-aks --resource-group guardian-ai-prod
git clone https://github.com/MAOFILHO/MultiCloud-Video-Guardian-AI.git
cd MultiCloud-Video-Guardian-AI
# Install Python dependencies
python3.11 -m venv .venv
source .venv/bin/activate # macOS/Linux
pip install -r requirements.txt
# Configure credentials
cp .env.example .env
# Edit .env with your values (see CONFIGURATION section below)aws configure
# Enter: Access Key ID, Secret Access Key, Region: us-east-1, Output: jsonaz login
az account set --subscription <YOUR_SUBSCRIPTION_ID>python setup.py --stage aws # ~2 min: S3, SQS, DynamoDB
python setup.py --stage azure # ~15 min: RG, ACR, AKS, NGINX
python setup.py --stage images # ~20 min: Docker build & push (8 images)
python setup.py --stage k8s # ~5 min: Deploy all services to AKS
python setup.py --stage devops # ~5 min: Azure DevOps pipelines
python setup.py --stage ml # ~45 min: Azure ML training + deploymentAfter --stage k8s completes, the app is live at: http://<EXTERNAL_IP>
Copy .env.example to .env and fill in all <YOUR_VALUE> placeholders:
AWS_ACCESS_KEY_ID=<from IAM user>
AWS_SECRET_ACCESS_KEY=<from IAM user>
AWS_REGION=us-east-1AZURE_SUBSCRIPTION_ID=<az account show --query id -o tsv># Create SP:
az ad sp create-for-rbac \
--name guardian-ai-sp \
--role Contributor \
--scopes /subscriptions/<YOUR_SUBSCRIPTION_ID>
# Fill in output values:
AZURE_SP_APP_ID=<appId>
AZURE_SP_SECRET=<password>
AZURE_TENANT_ID=<tenant>AZDO_ORG_NAME=<your-org-name> # from dev.azure.com/<ORG>
AZDO_PAT_TOKEN=<personal-access-token>Generate PAT at: https://dev.azure.com → Profile → Personal access tokens
Required scopes: Agent Pools, Build, Code, Project & Team, Service Connections, Variable Groups
Creates: S3 bucket, 2 SQS queues, 2 DynamoDB tables
Duration: ~2 minutes
Region: us-east-1 (hardcoded guardian — will error if unconfigured)
Creates: Resource group, ACR, AKS cluster (4 nodes), NGINX Ingress
Duration: ~15 minutes (AKS is the long step)
Output: External IP for the app
Builds and pushes 8 Docker images to ACR:
ingestion,fast-screening,deep-vision,policy-enginehuman-review,notification,api-gateway,frontend
Duration: ~20 minutes (first run; subsequent runs use layer cache)
Deploys: Namespace, ConfigMap, AWS secrets, all 8 services + Redis, Ingress
Duration: ~5 minutes
✅ App is live after this stage
Creates: Azure DevOps project, 3 service connections, variable group, 3 pipelines
Duration: ~5 minutes
Requires: AZDO_PAT_TOKEN, AZURE_SP_APP_ID, AZURE_SP_SECRET, AZURE_TENANT_ID
Creates: Azure ML workspace, compute cluster, triggers training pipeline
Duration: ~45 minutes (training on cpu-training-cluster)
Output: NSFW + Violence detection endpoints; patches Kubernetes ConfigMap
# Smoke tests (no cloud credentials required)
python -m pytest tests/smoke/test_smoke.py -v
# Expected: 69 tests passingTwo GitHub Actions workflows live in .github/workflows/: ci.yml (smoke tests + frontend build on every push/PR, no cloud access) and deploy.yml, which builds and deploys the service images to AKS and runs manually only. While this project lived in the Portfolio-Projects monorepo, the deploy ran automatically on pushes to services/**, k8s/** and frontend/**.
Add at: github.com/MAOFILHO/MultiCloud-Video-Guardian-AI → Settings → Secrets → Actions
ACR_USERNAME— run:az acr credential show --name guardianacr03211 --query username -o tsvACR_PASSWORD— run:az acr credential show --name guardianacr03211 --query 'passwords[0].value' -o tsvAZURE_CREDENTIALS— run:az ad sp create-for-rbac --name guardian-github-actions --role Contributor --scopes /subscriptions/960936b9-ecde-465b-be8d-776ca077dcd0 --sdk-auth
In the monorepo, pushing a change to a service file ran the deploy workflow automatically. The standalone repo's deploy.yml is manual-only, so use the steps below.
GitHub UI: Actions tab → Deploy to AKS → Run workflow
- AKS must be running before deployment:
az aks start --name guardian-ai-aks --resource-group guardian-ai-prod - The deploy workflow never fires on push; trigger it from the Actions tab
- Rotate the
guardian-github-actionsSP secret annually
Automation: Code commit triggers or manually run pipelines that train on the cluster, register models, and deploy to endpoints, no manual promotion or server management.
Consistency: Every deployment uses the same pipeline and the same registry versions; rollbacks can target a previous model version.
Explainability and traceability: MLflow and Azure ML Studio provide run history, metrics, and model lineage; registry versions tie deployments to specific training runs.
Scalability: Training scales on Azure ML compute clusters; inference scales via managed endpoints; application layer scales independently on AWS and AKS.
Speed: New model versions move from code to production in one pipeline run; application pods pick up new endpoints via ConfigMap updates and restarts. Multi-cloud flexibility: Use AWS for data and messaging and Azure for ML and CI/CD, with clear boundaries and minimal coupling.
chmod +x aws_resource_cleanup.sh
./aws_resource_cleanup.shpython automation/cleanup.py
# OR manually:
az group delete --name guardian-ai-prod --yes --no-wait# AWS
echo "S3:"; aws s3 ls | grep guardian
echo "SQS:"; aws sqs list-queues --region us-east-1
echo "DDB:"; aws dynamodb list-tables --region us-east-1
# Azure
az resource list --resource-group guardian-ai-prod --output tableThis lab required significant troubleshooting beyond the original manual. Key lessons:
- Always set
AWS_REGIONexplicitly — never rely on CLI defaults in scripts - S3 in us-east-1 is special — it's the default region and rejects
LocationConstraint - Azure DevOps CLI has gaps — some service connection types (AKS Kubeconfig, Token) require the REST API
- ACR must be attached to AKS after creation via
az aks update --attach-acr - Portfolio-Projects repo structure matters — pipeline YAML paths must include the subfolder prefix when the pipeline YAML isn't at repo root
- Self-hosted agents require network access — Microsoft-hosted
ubuntu-latestis simpler for labs - Never commit secrets — use
.env+.gitignore, rotate credentials immediately if exposed
Symptom: Resources created in the wrong region despite AWS_REGION=us-east-1
Root cause: Original setup-aws.sh had REGION="ap-south-1" hardcoded
Fix: Already patched — script now reads ${AWS_REGION:-$(aws configure get region)}
Symptom: An error occurred (InvalidLocationConstraint)
Root cause: us-east-1 is S3's default region and rejects LocationConstraint
Fix: Already patched — script uses conditional creation without LocationConstraint for us-east-1
Symptom: A conflicting conditional operation is currently in progress
Root cause: AWS enforces a ~10-minute cooldown before a deleted bucket name can be reused
Workaround: Add S3_BUCKET_OVERRIDE=guardian-videos-yourname-dev to .env
Symptom: username is empty when running --stage images standalone
Root cause: ACR credentials were only held in-memory during --stage azure
Fix: Already patched — build_images.py fetches credentials from Azure CLI when not in config; setup_azure.py persists AZURE_ACR_NAME to .env
Symptom: All pods stuck in ErrImagePull
Root cause: New ACR not attached to AKS cluster
Fix: Run manually after --stage k8s if pods fail:
az aks update \
--name guardian-ai-aks \
--resource-group guardian-ai-prod \
--attach-acr <YOUR_ACR_NAME>
kubectl rollout restart deployment -n productionSymptom: Azure RM service principal key: prompt during --stage devops
Root cause: az devops service-endpoint azurerm create prompts for SP key interactively
Fix: Set these in .env before running --stage devops:
AZURE_SP_APP_ID=<appId>
AZURE_SP_SECRET=<password>
AZURE_TENANT_ID=<tenant>Symptom: socket.gaierror: nodename nor servname provided downloading agent
Root cause: vstsagentpackage.azureedge.net blocked or unavailable
Fix: Already patched — setup skips self-hosted agent and uses Microsoft-hosted ubuntu-latest pool instead
Symptom: Azure DevOps pipeline fails immediately with YAML not found
Root cause: Azure Repos has the full Portfolio-Projects structure, so YAMLs are under MultiCloud-Video-Guardian-AI/ subfolder, not at root
Fix: Already patched — pipeline YAMLs reference correct paths. If you re-create pipelines, update the YAML path in Azure DevOps to MultiCloud-Video-Guardian-AI/azure-pipelines-*.yml
Symptom: Training jobs fail immediately
Root cause: Pipeline YAML had pool: name: 'Default' requiring a self-hosted agent
Fix: Already patched — all pipeline YAMLs use pool: vmImage: 'ubuntu-latest'
Symptom: Cannot find path '/home/vsts/work/1/s/mlops/training'
Root cause: Pipeline runs from Portfolio-Projects repo root, so MLOps scripts are under MultiCloud-Video-Guardian-AI/mlops/
Fix: Already patched — pipeline YAMLs use cd MultiCloud-Video-Guardian-AI/mlops/training
Symptom: New guardianacr* created every time --stage azure runs
Root cause: ACR name was timestamp-generated and not persisted
Fix: Already patched — setup_azure.py saves AZURE_ACR_NAME= to .env after first creation
- Marcos Oliveira — VP AI/ML Engineering
github.com/MAOFILHO | Portfolio-Projects
MIT License — see LICENSE