DIP-KD: Diverse Image Priors for Black-box Data-free Knowledge Distillation

Official implementation of the paper "Diverse Image Priors for Black-box Data-free Knowledge Distillation", accepted at ICCCI 2026.

Repository Structure

• models/ — neural network components
  • models/classifiers/ — AlexNet, LeNet-5, ResNet (standard and CIFAR variants)
  • models/GANs/ — GAN, ProGAN, StyleGAN architectures and utilities
  • models/distillers/ — distillation pipeline and self-boosting utilities
    • models/distillers/selfboosting_utils/ — contrastive synthesizer, coupled distiller, samplers, losses, filters
• configs/selfboosting/ — experiment configuration files (YAML/JSON)
  • teacher/ — teacher pre-training configs
  • synth/ — image synthesis configs
  • distillcontr/ — contrastive distillation configs
  • distilldfad/ — DFAD-style distillation configs
  • distilliter/ — iterative distillation configs
  • distillraw/ — raw distillation configs
  • adapt/ — domain adaptation configs
  • convergence/ — convergence study configs
  • crossarch/ — cross-architecture transfer configs
  • vizembed/ — embedding visualisation configs
• utils/ — training infrastructure
  • utils/callbacks/ — checkpointing, logging, schedulers, WandB integration
  • utils/data/ — dataset loaders (CIFAR, Tiny ImageNet, Imagenette, SVHN, USPS)
  • utils/metrics/ — accuracy, DDP-aware metrics
  • utils/trainers/ — trainer base class and utilities
  • utils/ddp/ — distributed data parallel helpers
• results/ — experiment notebooks and reporting utilities

Quick Start

1. Setup Environment

conda env create -f environment.yml
conda activate aim2

2. Train a Teacher

python -m models.distillers.selfboosting --config configs/selfboosting/teacher/teacher-cifar10-resnet18.yaml

3. Run Distillation

# Contrastive distillation (DIP-KD)
python -m models.distillers.selfboosting --config configs/selfboosting/distillcontr/distillcontr-cifar10-resnet18.yaml

Configs follow the naming convention <experiment>-<dataset>-<architecture>.yaml. Select a config from configs/selfboosting/ to reproduce any experiment in the paper.

Method Overview

DIP-KD operates in three phases under the restrictive black-box setting — only top-1 predictions from the teacher are available, with no access to training data:

1. Synthesis — generates diverse image priors via multi-scale noise sampling with random augmentations, covering varied visual semantics across classes.
2. Contrast — applies contrastive learning through a ContrastiveSynthesizer to sharpen inter-class distinctions among synthetic samples.
3. Distillation — a CoupledDistiller uses a primer student to enable soft-probability knowledge transfer across multiple generations.

Citation

If you find this work useful, please cite:

@inproceedings{vo2026diverse,
  title     = {Diverse Image Priors for Black-box Data-free Knowledge Distillation},
  author    = {Vo, Tri-Nhan and Nguyen, Dang and Le, Trung and Do, Kien and Gupta, Sunil},
  booktitle = {Proceedings of the International Conference on Computational Collective Intelligence (ICCCI)},
  year      = {2026},
}