ChimeraLoRA

This repository is the official implementation of "ChimeraLoRA: Multi-Head LoRA-Guided Synthetic Datasets" accepted by CVPR 2026.

Project Overview

ChimeraLoRA is a research project for synthetic dataset generation with few-shot guidance. It implements three methods:

• LoFT: Per-instance LoRA adapters fused together for image generation
• DataDream: Class-level or dataset-level LoRA finetuning for generation
• ChimeraLoRA: Shared LoRA-A matrix with per-instance LoRA-B matrices, enabling efficient multi-instance training

The pipeline has three core stages: SD finetuning → Image generation → Classification training, with an optional mask generation step beforehand that enables semantic boosting.

Architecture

Pipeline Stages

0. mask-generation/ (optional): Generates object masks (enclosing boxes) for the few-shot real images via GroundingDINO + SAM. Enables semantic boosting in SD finetuning. See mask-generation/README.md.
1. sd-finetune/: Finetunes Stable Diffusion 2.1 with LoRA adapters on few-shot real images
2. generation/: Uses finetuned LoRA weights to generate synthetic training images
3. classification/: Trains CLIP or ResNet50 classifiers on synthetic data, evaluates on real test data

Semantic Boosting

When mask_dir (in sd-finetune/local.yaml) points at a directory of masks, SD finetuning crops/zooms each few-shot image to its object's enclosing box instead of using a plain center/random crop, so the LoRA focuses on the object rather than the background. Runs that use it are tagged with the _sb suffix. Masks are JSON files of the form {"enclosing_box": [x_min, y_min, x_max, y_max]}, produced by mask-generation/. Leave mask_dir pointing at a non-existent path to disable semantic boosting.

Method Architectures

• LoFT (main.py): Trains independent LoRA adapters per instance, fuses them at generation time via interpolation
• DataDream (main.py): Single LoRA adapter trained on all images (class-level or dataset-level)
• ChimeraLoRA (main_chimera.py): Factorized approach with shared LoRA-A matrix (captures common features) and per-instance LoRA-B matrices (captures instance-specific details). Outputs saved as B{i}_each/ subdirectories containing individual adapters

Configuration System

Each folder has a local.yaml for machine-specific paths:

• sd-finetune/local.yaml: output_dir, fewshot_dir, model_dir, mask_dir
• generation/local.yaml: save_dir, sd_lora_dir, model_dir
• classification/local.yaml: output_dir, synth_train_data_dir, real_test_data_dir, metadata_dir, clip_download_root

Key Data Structures

Class names and templates are defined in sd-finetune/util.py (referenced by all stages):

• CLASSNAMES: Dict mapping dataset names to class lists
• TEMPLATES_SMALL: Text prompt templates for generation

Supported datasets: imagenet, imagenet100, pets, eurosat, dtd, flowers102, cars, fgvc_aircraft, food101, caltech101, sun397

Common Commands

Installation

pip install -r requirements.txt

SD Finetuning (LoFT method - per instance)

cd sd-finetune
accelerate launch main.py \
  --dataset=pets \
  --target_class_idx=0 \
  --instance_idx=0 \
  --n_shot=4 \
  --fewshot_seed=seed0 \
  --num_train_epochs=600 \
  --rank=4 \
  --learning_rate=1e-3

SD Finetuning (DataDream method - per class or dataset)

cd sd-finetune
# Class-level: set target_class_idx, leave instance_idx=None
# Dataset-level: set both to None
accelerate launch main.py \
  --dataset=pets \
  --target_class_idx=0 \
  --n_shot=16 \
  --num_train_epochs=200 \
  --rank=16 \
  --train_text_encoder=True

SD Finetuning (ChimeraLoRA method - shared A, per-instance B)

cd sd-finetune
accelerate launch main_chimera.py \
  --dataset=pets \
  --n_shot=4 \
  --fewshot_seed=seed0 \
  --num_train_epochs=400 \
  --rank=16 \
  --learning_rate_A=1e-4 \
  --learning_rate_B=1e-3 \
  --method=JOINT \
  --train_text_encoder=True

Image Generation

cd generation
# LoFT generation
python main.py \
  --dataset=pets \
  --method=loft \
  --n_shot=4 \
  --fewshot_seed=seed0 \
  --loft_interpolation_weight=0.5 \
  --n_image_per_class=500 \
  --bs=5

# ChimeraLoRA generation (uses Dirichlet sampling over B adapters)
python main.py \
  --dataset=pets \
  --method=chimera_dir \
  --n_shot=4 \
  --fewshot_seed=seed0 \
  --n_image_per_class=500 \
  --bs=5

Classification Training

cd classification
python main.py \
  --method=loft \
  --loft_interpolation_weight=0.5 \
  --dataset=pets \
  --n_img_per_cls=100 \
  --n_shot=4 \
  --model_type=clip \
  --epochs=60 \
  --lr=1e-4 \
  --batch_size_per_gpu=256

Key Parameters

Method Selection

• method=loft: Uses per-instance LoRA fusion with loft_interpolation_weight (0.5 = equal blend of 2 LoRAs)
• method=datadream-cls: Class-level LoRA
• method=datadream-dset: Dataset-level LoRA
• method=chimera / method=chimera_dir: ChimeraLoRA with Dirichlet-sampled weights for generation

Few-shot Settings

• n_shot: Number of real images per class (commonly 1, 4, 16)
• fewshot_seed: Data split seed (e.g., "seed0")

LoRA Configuration

• rank: LoRA rank (LoFT uses 4, DataDream uses 16, ChimeraLoRA uses 4-16)
• target_class_idx: Class index for class-level training (None = dataset-level)
• instance_idx: Instance index for LoFT (None = not LoFT method)

ChimeraLoRA-Specific Parameters

• learning_rate_A: Learning rate for shared A matrix (typically 1e-4)
• learning_rate_B: Learning rate for per-instance B matrices (typically 1e-3)
• method: Training mode - JOINT (train A+B together) or ALT (alternating optimization)
• step: Number of steps between alternating A/B updates (for ALT mode)

Data Organization

Few-shot real images should be organized as:

$FEWSHOT_DIR/$DATASET/shot{N}_seed{S}/$CLASS_NAME/*.jpg

Generated synthetic images are saved to:

$SAVE_DIR/$DATASET/{method}_avg_{params}/shot{N}_{seed}/$CLASS_NAME/*.png

Model Types (Classification)

• clip: CLIP ViT-B/16 with LoRA finetuning (default)
• resnet50: ResNet50 trained from scratch