Skip to content
BranchesTags

Repository files navigation

Deepfake Detector

https://huggingface.co/spaces/knmrfr/deepfake-detector-demo?logs=build

Note: This will only work for GAN-based and Stable-Diffusion deepfakes so it won't work on things that are fully AI generated

An image-based deepfake detection web app that classifies face images as Real or Deepfake using a two-branch fusion model: a DoRA-fine-tuned CLIP ViT-L/14 backbone (spatial features) combined with a lightweight CNN over the 2D FFT magnitude spectrum (frequency artifacts). A Flask backend serves predictions; a React/Vite frontend provides the UI.

Images

image
image
image

Setup

pip install -r requirements.txt

# Configure HuggingFace auth (required to pull CLIP weights)
cp .env.example .env
# then edit .env and set HF_TOKEN=<your-token>

# One-time: prep the OpenRL DeepFakeFace diffusion dataset
python scripts/download_deepfakeface.py

python train.py       # fine-tune the model (also pulls Kaggle datasets via kagglehub)
python evaluate.py    # evaluate on held-out test sets
python app.py         # start the Flask API at http://127.0.0.1:5001

# Frontend (optional)
cd frontend
npm install
npm run dev

How It Works

  1. User uploads a face image.
  2. The image is passed through the two-branch detector:
    • Branch 1: CLIP ViT-L/14 vision encoder (spatial semantics).
    • Branch 2: 2D FFT magnitude spectrum → small CNN (frequency artifacts).
    • A fusion MLP head produces the Real / Deepfake logits.
  3. The label and softmax confidence are returned to the frontend.

Training Recipe

  • Backbone: openai/clip-vit-large-patch14
  • Adapter: DoRA (r=16, α=32) on all CLIP attention projections (q_proj, k_proj, v_proj, out_proj), with LoRA dropout 0.1.
  • Loss: 0.7 × Supervised Contrastive + 0.3 × Cross-Entropy (label smoothing 0.1). Projection head (128-d, L2-normalized) used only during training.
  • Optimizer: AdamW (lr=2e-4, weight decay 0.01), gradient clipping at 1.0.
  • Schedule: CosineAnnealingWarmRestarts (T_0=5, η_min=1e-6).
  • Precision: AMP (float16) on MPS / CUDA.
  • Augmentation: Albumentations pipeline — horizontal flip, rotate, random resized crop, color jitter, simulated social-media degradation (downscale → JPEG 20–70 → upscale → blur), JPEG compression, Gaussian blur, downscale, and a stochastic high-pass filter (p=0.15).
  • Batch sampling: Balanced sampler with 6 groups (3 datasets × 2 classes), 12 samples per group at batch size 72.
  • Validation metric: ROC-AUC; early stopping with patience 3; per-epoch versioned DoRA snapshots.

Optimizations

Training Efficiency

  • DoRA adapter instead of full fine-tuning — only the decomposed low-rank updates on CLIP's q/k/v/out_proj layers are trained; the 300M+ backbone weights stay frozen. Massive VRAM savings, and DoRA typically closes the gap to full fine-tuning that plain LoRA leaves on the table.
  • Automatic Mixed Precision (float16 autocast) — roughly 2× memory reduction on MPS/CUDA and faster matmuls on tensor-core hardware.
  • Gradient accumulation (ACCUM_STEPS) — lets a small per-step batch simulate a much larger effective batch without the memory cost.
  • Gradient clipping at max_norm=1.0 — stabilizes DoRA + SupCon updates, which can spike early in training.
  • CosineAnnealingWarmRestarts — resume-friendly LR schedule; periodic restarts help escape flat regions without manual LR tuning.
  • Early stopping (patience=3 on val AUC) — avoids wasted epochs once the model plateaus.
  • Device auto-selection — CUDA → MPS → CPU, with num_workers=4 and pin_memory=True enabled automatically on CUDA only (MPS / CPU get num_workers=0 to avoid Python multiprocessing stalls on macOS).
  • CPU-side Albumentations pipeline — all augmentation runs in NumPy/OpenCV so it doesn't contend with the MPS GPU mid-step.

Data Pipeline

  • Balanced batch sampler — each batch draws equally from 6 groups (3 datasets × 2 classes). Prevents the larger dataset / majority class from dominating gradients and ensures every step sees every generator type (StyleGAN, face-swap, diffusion).
  • Stochastic high-pass filter augmentation (p=0.15) — forces the network to learn frequency-domain cues even when the FFT branch alone would not be enough.
  • Simulated social-media degradation — downscale → JPEG 20–70 → upscale → blur, approximating the Instagram/TikTok transcode pipeline so the model generalizes to "in-the-wild" deepfakes rather than pristine dataset images.
  • Three datasets combined — manjilkarki deepfakes (mixed Kaggle), xhlulu 140k (StyleGAN), and OpenRL DeepFakeFace (Stable Diffusion + InsightFace face-swap). Together they span GAN, face-swap, and diffusion generators.
  • Label smoothing (0.1) — prevents the CE head from producing overconfident logits, which also improves calibration of the softmax score shown in the UI.

Loss & Representation

  • SupCon (0.7) + CE (0.3) — SupCon shapes the fused embedding space so same-class samples cluster together regardless of generator, while CE maintains a clean decision boundary. The projection head is training-only and discarded for inference.
  • Two-branch fusion (CLIP + FFT) — CLIP handles spatial semantics; the FFT CNN captures spectral peaks and blending artifacts CLIP cannot see in pixel space. Concatenated before the classifier.
  • log1p + fftshift on FFT magnitude — compresses the dynamic range of the spectrum and centers the DC component, making the distribution easier for a small CNN to learn.

Checkpointing & Reproducibility

  • Best-only saving, tracked by val AUC — head_weights.pt and the DoRA adapter are overwritten only when validation AUC improves.
  • Versioned per-epoch snapshots (dora_epoch{N}_auc{X}) — any prior epoch can be rolled back to without re-training.
  • train_state.pt — stores optimizer, scheduler, epoch, best-AUC, and patience counter in sync with the best weights, so python train.py can safely resume from interruption.
  • DoRA merge at end of training (merge_and_unload) — the final inference checkpoint is a plain CLIPVisionModel with the adapter folded in, so app.py does not need PEFT at serve time.

Inference & Serving

  • Whole-image inference — the uploaded image is passed directly through the model, matching the training/evaluation pipeline so production behavior aligns with reported AUC.
  • torch.no_grad() everywhere in /predict — no autograd graph is built for inference.

Evaluation

  • AUC-ROC + EER as primary metrics, plus full classification report and confusion matrix.
  • Three modes: combined (all datasets pooled), per-dataset, and Leave-One-Out cross-generator (train on two, evaluate on the held-out third) to measure generalization to unseen generator types.
  • Test-Time Augmentation (TTA) — predictions are averaged over the original and horizontally-flipped image.

Citations & References

Pretrained Backbone

Methods & Techniques

Datasets

Core Libraries

Backend

Frontend

License

This repository is for research and educational use. Please respect the upstream licenses of the datasets and pretrained weights:

  • CLIP ViT-L/14 weights — MIT License (OpenAI).
  • Kaggle datasets — see each dataset page for terms of use.
  • OpenRL DeepFakeFace — see the HuggingFace dataset page for terms of use.

About

This is an image-based deepfake detector that classifies if images are Real or Deepfake using a two-branch fusion model: a DoRA-fine-tuned CLIP + ViT-L/14 backbone for spatial features combined with a small CNN over the 2D FFT magnitude spectrum for frequency artifacts. Trained with a SupCon + cross-entropy loss across three datasets

huggingface.co/spaces/knmrfr/deepfake-detector-demo?logs=build

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages