Joint synthesis of images and labels

[naga-karthik]

Hello! Thank you for maintaining this wonderful repository! Makes it quite easy to get started with diffusion models.

I have recently started working with diffusion models and am noticing that several diffusion papers talk only about image synthesis. I am interesting in generating a synthetic dataset containing both images and labels and I am wondering if there are any papers out there doing such type of joint synthesis.

Alternatives considered:

1. One workaround is that you generate high-quality images with diffusion models and use an existing model to get the labels for the synthetic images. My argument here is that, if there's already a model that works well on some dataset/task, then what's the need for a synthetic dataset in the first place? So, in my specific setting, I don't have a good model to get the predictions so I am looking to synthetic generate both images and labels using diffusion models.

2. I have looked into a few papers that generate labels conditioned on multi-modal images (for e.g. this paper). However, in such cases, I see that the labels are generated for existing (real) images for which the ground-truth already exist (i.e. the distribution of the labels isn't new, IIUC).

Any suggestions on how to approach this problem with monai-generative? OR, is it the case that joint synthesis is hard without proper ground-truth image-label pairs? For context, I am working with spinal cord MRI lesions.

[marksgraham]

Hi there,

It is indeed possible, one simple approach is to treat the image + label as a two-channel image and generate them jointly. You would need ground truth paired image and label data to train this on though, do you not have that?

[naga-karthik]

Hi @marksgraham ! Thanks for your reply! In fact, I was just starting to think along this direction (i.e. generating the labels by concatenating in the channel dimension as I do have image/label pairs in my dataset). As I didn't come across any tutorial that synthesizes both images and labels (apart from the counterfactual one), I wasn't so sure whether this would work. Anyway, thanks for your suggestion, I will try it out! :)