Score-based Membership Inference on Diffusion Models (TMLR)

VINE Lab, Vanderbilt University

Mingxing (Ethan) Rao, Bowen Qu, Daniel Moyer

Membership inference attacks (MIAs) against Diffusion Models (DMs) raise pressing privacy concerns by revealing whether a sample was part of the training set. While existing methods typically rely on measuring reconstruction error across multiple denoising steps as a test statistic, they often incur significant computational overhead. In this work, we present a simple yet successful attack statistic using only the predicted noise vectors from the DM’s denoiser, or equivalently, the score. Specifically, we show that the expected denoiser output points toward a kernel-weighted local mean of nearby training samples, such that its norm encodes proximity to the training set and thereby reveals membership. Building on this observation, we propose SimA, a single-query attack that provides a principled, efficient alternative to existing multi-query methods. SimA consistently achieves superior performance across variants of DMs and the Latent Diffusion Models (LDMs) on eight different datasets. Its Monte Carlo variant (SimA-MC) exhibits state-of-the-art performance across all experiments, significantly outperforming baseline methods in terms of TPR@1%FPR. These results demonstrate that complex reconstruction trajectories are unnecessary for effective membership inference, establishing SimA as a highly efficient benchmark for auditing privacy in DMs and LDMs.

Requirements

Please refer to README.md of different model sub-folders.

Experiment Details

Model	Member	Held-out	Pre-trained	Fine-tuned	Splits	Resolution	Cond.
DDPM – CIFAR-10	CIFAR-10	CIFAR-10	No	–	25k/25k	32	–
DDPM – CIFAR-100	CIFAR-100	CIFAR-100	No	–	25k/25k	32	–
DDPM – STL10-U	STL10-U	STL10-U	No	–	50k/50k	32	–
DDPM – CelebA	CelebA	CelebA	No	–	30k/30k	32	–
Guided Diffusion – ImageNet-1k	ImageNet-1k	ImageNetV2	Yes	No	3k/3k	256	class
Latent Diffusion Model – ImageNet-1k	ImageNet-1k	ImageNetV2	Yes	No	3k/3k	256	class
Stable Diffusion V1-4 – Pokémon	Pokémon	Pokémon	Yes	Yes	416/417	512	text
Stable Diffusion V1-4 – COCO2017-Val	COCO2017-Val	COCO2017-Val	Yes	Yes	2.5k/2.5k	512	text
Stable Diffusion V1-4 – Flickr30k	Flickr30k	Flickr30k	Yes	Yes	10k/10k	512	text
Stable Diffusion V1-5 – LAION-Aesthetics v2 5+ / LAION-2B-MultiTranslated	LAION-Aesthetics v2 5+	LAION-2B-MultiTranslated	Yes	No	2.5k/2.5k	512	text
Stable Diffusion V1-5 – LAION-Aesthetics v2 5+ / COCO2017-Val	LAION-Aesthetics v2 5+	COCO2017-Val	Yes	No	2.5k/2.5k	512	text

---

All the experiment scripts can be found in script.sh. For all experiments, please choose --attacker in [SimA, SecMi, PIA, PFAMI, Loss]

DDPM 🔗

Please download partial dataset splits and checkpoints here for DDPM. For LDM and Stable Diffusion, we used the resutls from this paper and the experiment is here.

CIFAR-10

Train

python /DDPM/main.py --dataset_root /path/to/dataset_root --dataset CIFAR10 --logdir /path/to/CIFAR10/logs/ --parallel

Attack

python DDPM/attack.py --checkpoint  /path/to/CIFAR10/cifar10_ckpt.pt --dataset_root /path/to/dataset_root --dataset CIFAR10 --img_size 32 --attacker SimA --attack_num 20 --interval 10

CIFAR-100

Train

python DDPM/main.py --dataset_root /path/to/dataset_root --dataset CIFAR100 --logdir /path/to/CIFAR100/logs/ --parallel

Attack

python DDPM/attack.py --checkpoint  /path/to/CIFAR100/cifar100_ckpt.pt --dataset_root /path/to/dataset_root --dataset CIFAR100 --img_size 32 --attacker SimA --attack_num 20 --interval 10

STL10-U

Train

python DDPM/main.py --dataset_root /path/to/dataset_root --dataset STL10-U --logdir /path/to/STL10-U/logs/ --batch_size 1024 --parallel

Attack

python DDPM/attack.py --checkpoint  /path/to/STL10-U/logs/stl10u_ckpt.pt --dataset_root /path/to/dataset_root --dataset STL10-U --img_size 32 --attacker SimA --attack_num 20 --interval 10

CelebA

Train

python DDPM/main.py --dataset_root /path/to/dataset_root --dataset CELEBA --logdir /path/to/CELEBA/logs/ --batch_size 1024 --parallel

Attack

python DDPM/attack.py \
    --checkpoint  /path/to/CELEBA/logs/celeba_ckpt.pt \
    --dataset_root /path/to/dataset_root \
    --dataset CELEBA \
    --img_size 32 \
    --attacker SimA \
    --attack_num 20 \
    --interval 10

Guided Diffusion 🔗

Member/ Held-out: ImageNet-1K/ ImageNetV2

Attack

python guided-diffusion/INv2_attack.py --IMN1k /path/to/imagenet-1k/train --IMNv2 /path/to/IMAGENETv2/ImageNetV2-matched-frequency --ckpt_path /path/to/IMAGENET1K/256x256_diffusion_uncond.pt --cond=False --attacker SimA

Latent Diffusion Model 🔗

Member/ Held-out: ImageNet-1K/ ImageNetV2

Attack

python latent-diffusion/INv2_attack.py --IMN1k /path/to/imagenet-1k/train --IMNv2 /path/to/IMAGENETv2/ImageNetV2-matched-frequency  --ckpt_path /path/to/imagenet1k_ldm.ckpt --config_path latent-diffusion/configs/latent-diffusion/cin256-v2.yaml --cond=False --attacker SimA

Stable Diffusion V1-4 🔗

Please download all dataset splits here and email me for the checkpoints

Pokémon

Fine-tune

accelerate launch diffusers/examples/text_to_image/train_text_to_image.py \
  --pretrained_model_name_or_path="CompVis/stable-diffusion-v1-4" \
  --train_data_dir="/path/to/POKEMON/pokemon_blip_splits" \
  --use_ema \
  --resolution=512 --center_crop --random_flip \
  --train_batch_size=1 \
  --gradient_accumulation_steps=4 \
  --gradient_checkpointing \
  --mixed_precision="fp16" \
  --max_train_steps=15000 \
  --learning_rate=1e-05 \
  --max_grad_norm=1 \
  --lr_scheduler="constant" --lr_warmup_steps=0 \
  --output_dir="/path/to/POKEMON/logs/sd-pokemon-model" 

Attack

cd diffusers
python -m src.mia.attack --attacker SimA --dataset pokemon --ckpt-path /path/to/POKEMON/logs/sd-pokemon-model/

--unconditional: for unconditional attack

COCO2017-Val

Fine-tune

accelerate launch diffusers/examples/text_to_image/train_text_to_image.py \
  --pretrained_model_name_or_path="CompVis/stable-diffusion-v1-4" \
  --train_data_dir="/path/to/MSCOCO/coco2017_val_splits" \
  --use_ema \
  --resolution=512 --center_crop --random_flip \
  --train_batch_size=1 \
  --gradient_accumulation_steps=4 \
  --gradient_checkpointing \
  --mixed_precision="fp16" \
  --max_train_steps=50000 \
  --learning_rate=1e-05 \
  --max_grad_norm=1 \
  --lr_scheduler="constant" --lr_warmup_steps=0 \
  --output_dir="/path/to/MSCOCO/logs/sd-mscoco-model" \
  --caption_column="captions"

Attack

cd diffusers
python -m src.mia.attack --attacker SimA --dataset coco --ckpt-path /path/to/MSCOCO/logs/sd-mscoco-model/

--unconditional: for unconditional attack

Flickr30k

Fine-tune

accelerate launch diffusers/examples/text_to_image/train_text_to_image.py \
  --pretrained_model_name_or_path="CompVis/stable-diffusion-v1-4" \
  --train_data_dir="/path/to/FLICKR/flickr30k_splits" \
  --use_ema \
  --resolution=512 --center_crop --random_flip \
  --train_batch_size=1 \
  --gradient_accumulation_steps=4 \
  --gradient_checkpointing \
  --mixed_precision="fp16" \
  --max_train_steps=200000 \
  --learning_rate=1e-05 \
  --max_grad_norm=1 \
  --lr_scheduler="constant" --lr_warmup_steps=0 \
  --output_dir="/path/to/FLICKR/logs/sd-flickr-model" \
  --caption_column="caption"

Attack

cd diffusers
python -m src.mia.attack --attacker SimA --dataset flickr --ckpt-path /path/to/FLICKR/logs/sd-flickr-model/

--unconditional: for unconditional attack

Stable Diffusion V1-5 🔗

Member/ Held-out: LAION-Aesthetics v2 5+ / LAION-2B-MultiTranslated

Attack

cd diffusers
python -m src.mia.attack --attacker SimA --dataset laion-aesthetic_coco   --ckpt-path runwayml/stable-diffusion-v1-5 

--unconditional: for unconditional attack

Member/ Held-out: LAION-Aesthetics v2 5+ / COCO2017-Val

Attack

cd diffusers
python -m src.mia.attack --attacker SimA --dataset laion-aesthetic_laion-multitrans   --ckpt-path runwayml/stable-diffusion-v1-5

--unconditional: for unconditional attack

Remarks

With all checkpoints and data splits(here), you should be able to reproduce all the results reported in the paper.

Please email Mingxing (mingxing.rao@vanderbilt.edu) if you have additional questions for reproduction.

BibTeX

@article{
rao2026scorebased,
title={Score-based Membership Inference on Diffusion Models},
author={Mingxing Rao and Bowen Qu and Daniel Moyer},
journal={Transactions on Machine Learning Research},
issn={2835-8856},
year={2026},
url={https://openreview.net/forum?id=Ckvsu5xRmf},
note={}
}