Make images as condition not text

[sapkun]

great job! I have a question that is ok that make images as condition instead of text? for example, as for original dataset the dataset structure is like this:

fill50k/train.jsonl
fill50k/images/X.png
fill50k/conditioning_images/X.png

can i use the conditioning_images as condition, and change the relevent part of the code in here:

class MyDataset(Dataset):
    def __init__(self):
        self.data = []
        with open('./training/cfill50k/prompt.json', 'rt') as f:
            for line in f:
                self.data.append(json.loads(line))

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        item = self.data[idx]

        image_filename = item['image']
        source = item['source']
        target = item['target']

        image = cv2.imread('./training/cfill50k/' + image_filename)

        # Do not forget that OpenCV read images in BGR order.
        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

        image = (image.astype(np.float32) / 127.5) - 1.0

        return dict(jpg=image, source=source, txt=target)

Thank you!

[XiaMu799]

Hello, have you solved this problem?

[beautifulchoi]

I think this framework only use x0 as image condition. to use additional condition. (see algorithm on Appendix A https://arxiv.org/pdf/2310.13165#page=18.09)
So, I think you can give other image condition instead of x0

change "control" variable to your image condition instead of original image in get_input function.