LoRA integration?

[dellis23]

How hard would it be to add an option for integrating LoRA files into this? I'm wondering if it's something I can tackle as a relative newbie or whether it'd be a significant amount of undertaking / require a pretty deep understanding of the systems in order to do it correctly.

[dellis23]

I've attempted two approaches. The first: try the instructions here. That didn't work. I wish I had the trail of breadcrumbs for why I concluded that, so I could link them, but I lost them.

The more promising approach: try merging the lora with a SD checkpoint (this worked, at least with the SD web ui), then convert to diffuser format with this script. This seems to get really close, but after the image appears to be done generating with the stable diffusion videos run_app script, I get this error:

  File "z:\downloads\stable-diffusion-videos\stable_diffusion_videos\app.py", line 128, in fn_images
    image_filepaths = generate_images_fn(**kwargs)
  File "z:\downloads\stable-diffusion-videos\stable_diffusion_videos\image_generation.py", line 192, in generate_images
    outputs = pipeline(
  File "Z:\Downloads\stable-diffusion-videos\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "z:\downloads\stable-diffusion-videos\stable_diffusion_videos\stable_diffusion_pipeline.py", line 433, in __call__
    image = self.vae.decode(latents).sample
  File "Z:\Downloads\stable-diffusion-videos\venv\lib\site-packages\diffusers\models\autoencoder_kl.py", line 185, in decode
    decoded = self._decode(z).sample
  File "Z:\Downloads\stable-diffusion-videos\venv\lib\site-packages\diffusers\models\autoencoder_kl.py", line 171, in _decode
    z = self.post_quant_conv(z)
  File "Z:\Downloads\stable-diffusion-videos\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "Z:\Downloads\stable-diffusion-videos\venv\lib\site-packages\torch\nn\modules\conv.py", line 463, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "Z:\Downloads\stable-diffusion-videos\venv\lib\site-packages\torch\nn\modules\conv.py", line 459, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (struct c10::Half) and bias type (float) should be the same

I've tried everything I can find online for fixing it, like moving the VAE to cuda, but I haven't found anything that works yet.

[dellis23]

I tried changing L101 in image_generation.py to yield batch_idx, embeds_batch.type(torch.cuda.FloatTensor), noise_batch.type(torch.cuda.FloatTensor), and the error reversed!

RuntimeError: Input type (float) and bias type (struct c10::Half) should be the same

[dellis23]

The weird thing is, if I print the dtypes of input and bias in torch\nn\modules\conv.py, I get that they are both float16. Not sure where the full-sized float is coming from.

[dellis23]

Okay, I finally got a combo that worked.

I updated the torch_dtype arg in the from_pretrained call in run_app.py to:

torch_dtype=torch.float32

Then I updated the image_generation.py line to:

yield batch_idx, embeds_batch.type(torch.cuda.HalfTensor), noise_batch.type(torch.cuda.FloatTensor)

I've noticed it takes a significantly higher number of iteration steps to get to something as high quality as I'm getting in the SD webui though (~80 vs 20). Not sure if this is the nature of this particular repo, or something still lacking in the updates I've done here.