DiffDis

DiffDis: Empowering Generative Diffusion Model with Cross-Modal Discrimination Capability

Runhui Huang, Jianhua Han, Guansong Lu, Xiaodan Liang, Yihan Zeng, Wei Zhang, and Hang Xu

ICCV 2023

Sampled image using the prompt: A large bed sitting next to a small Christmas Tree surrounded by pictures

This folder provides a re-implementation of this paper in PyTorch, developed as part of the course METU CENG 796 - Deep Generative Models. The re-implementation is provided by:

• Furkan Genç, [email protected]
• Barış Sarper Tezcan, [email protected]

Please see the jupyter notebook file main.ipynb for a summary of paper, the implementation notes and our experimental results

Project Setup

Setup Environment

To set up the environment for this project, please follow the steps below:

1. Create a new conda environment named DiffDis:

   conda create -n DiffDis

2. Activate the newly created environment:

   conda activate DiffDis

3. Install PyTorch and related libraries with CUDA support:

   conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia

4. Install additional required Python packages from the requirements.txt file:

   pip install -r requirements.txt

Change Directory

Navigate to the scripts directory where the project scripts are located:

cd scripts

Download Data

To download the necessary data for the project, execute the following script:

bash download_data.sh

Download and Extract CC3M Dataset

To download and extract the CC3M dataset, run the following scripts in order:

1. Download the CC3M dataset:

   bash download_cc3m_dataset.sh

2. Extract the downloaded CC3M dataset:

   bash extract_cc3m_dataset.sh

Usage

Change Directory

Navigate to the code directory where the project code is located:

cd code

1. Training: Run the following command to train the model:

   python train.py

2. Testing: Run the following command to test the model:

   python test.py

3. Inference: Run the following command to generate images:

   python inference.py

Note: Adjust the config.py parameters according to your needs before running the scripts.

Pre-trained Models

The model trained on the CC3M dataset for two-thirds of an epoch can be found in this link.