$VCP\text{-}Attack$: Visual-Contrastive Projection for Transferable Black-Box Targeted Attacks on Large Vision-Language Models

📌 Update: This project has been Accepted by CVPR 2026 .

The full paper and appendix are available in the full_paper directory of this project:

• Full Paper PDF

• Appendix PDF

---

🔥 Overview

VCP-Attack proposes a novel visual-contrastive projection framework for transferable targeted adversarial attacks against Large Vision-Language Models (LVLMs).
Our method enhances semantic consistency across multiple models via adaptive subspace projection and feature-level contrastive guidance.

---

🎯 Attack Effectiveness

Figure 1: Transferable targeted attack performance of VCP-Attack on multiple state-of-the-art LVLMs.

---

📊 Quantitative Results (Table)

Figure 2: Attack success rates across open- and closed-source LVLMs.

---

🖼️ Adversarial Example Visualization

Figure 3: Comparison of adversarial examples generated by different methods. Our VCP-Attack yields semantically aligned and transferable perturbations.

---

⚙️ Method Overview

Figure 4: Overall pipeline of the proposed VCP-Attack. We optimize shared perturbations with visual contrastive supervision and cross-model PCA projection.

---

🗂️ Project Structure

VCP-Attack/
├── vcpattack/                # Core attack implementation package
│   ├── attack.py             # Main attack pipeline
│   └── pca.py                # Adaptive PCA projection module
├── vcp_attack.py             # Backward-compatible CLI entrypoint
├── pca_m.py                  # Backward-compatible import shim
├── evaluation/               # Caption generation + ASR evaluation scripts
├── tools/                    # Dataset preprocessing helpers
├── image/readme/             # README figures
└── VCP-ATTACK-Examples-100/  # Released adversarial examples

📦 Requirements

To install dependencies, use:

conda create -n vcp_attack python=3.9.21
conda activate vcp_attack
pip install -r requirements.txt

🚀 Running the Attack

To run the attack pipeline, make sure you have prepared both the datasets and the surrogate models as follows:

---

📂 Required Dataset

We use two datasets in our attack pipeline:

1. NIPS 2017 Adversarial Attack Dataset

We use 1,000 clean images from the [NIPS 2017 Adversarial Attack and Defense Competition], which is based on ImageNet.

• 🔗 Download link (Kaggle):
  https://tianchi.aliyun.com/dataset/88668 These serve as clean input images to be attacked.

---

2. ImageNet-1K Validation Set (Reference Pool)

We use the full [ImageNet-1K validation set](50,000 images) as a reference pool for contrastive supervision.

• 🔗 Download link (Kaggle):
  https://www.kaggle.com/datasets/sautkin/imagenet1kvalid
• 🔧 Preprocess the downloaded images using:

python tools/process_imagenet.py

---

🤖 Required Surrogate Models

Our method relies on an ensemble of frozen CLIP-based vision encoders. (Models can be auto-downloaded or loaded from local path by setting config.)

python vcp_attack.py

---

🧪 Evaluation

cd evaluation
python _internVL3-14b_img2txt.py --img_path your generated adversarial examples path --output_path output txt path --txt_name txt name #(model you want to evaluate)# 
python evaluate_captions_1000.py --attack_text_path your txt path --output_path your output path

🙏 Acknowledgements

We gratefully acknowledge the open-source community and previous research efforts for their invaluable contributions. This project builds upon the foundations laid by the following works:

• OpenCLIP
• HuggingFace
• AnyAttack
• M-Attack
• FOA-Attack
• AdvDiffVLM
• AttackVLM