[Feature Request]: new sampler "lcm" for lcm-loras

[light-and-ray]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

This feature requires not so much: no a new model class support, only a new sampler

LCM had major update and now we can use it like a regular lora:
https://huggingface.co/latent-consistency/lcm-lora-sdv1-5
https://huggingface.co/latent-consistency/lcm-lora-ssd-1b
https://huggingface.co/latent-consistency/lcm-lora-sdxl

We only need to rename them and put into lora models directory. Set sampling steps to 4, CFG Scale to 1.0. Sampling method to "DPM2" or "Euler a". It gives decent results, but for better work it requires a special sampler, with is similar to others but with little change. You can look how it works in ComfyUI: Comfy-Org/ComfyUI@002aefa

Proposed workflow

1. Go to Sampling method
2. Select LCM sampler

Additional information

How it works now

CompyUI's recent update:

[BahzBeih]

I'm awaiting the integration of the LCM sampler into AUTOMATIC1111, While AUTOMATIC1111 is an excellent program, the implementation of new features, such as the LCM sampler and consistency VAE, appears to be sluggish. The consistency VAE is currently only accessible in beta, while comfyui already offers both consistency VAE and LCM sampler support at no time :(

[camoody1]

I'm awaiting the integration of the LCM sampler into AUTOMATIC1111, While AUTOMATIC1111 is an excellent program, the implementation of new features, such as the LCM sampler and consistency VAE, appears to be sluggish. The consistency VAE is currently only accessible in beta, while comfyui already offers both consistency VAE and LCM sampler support at no time :(

Let's not forget that TensorRT for SDXL is sitting on the Dev branch, too. I desperately need all three of these moved up into the Main branch.

[fgtm2023]

I'm awaiting the integration of the LCM sampler into AUTOMATIC1111, While AUTOMATIC1111 is an excellent program, the implementation of new features, such as the LCM sampler and consistency VAE, appears to be sluggish. The consistency VAE is currently only accessible in beta, while comfyui already offers both consistency VAE and LCM sampler support at no time :(

Let's not forget that TensorRT for SDXL is sitting on the Dev branch, too. I desperately need all three of these moved up into the Main branch.

let's hope this will be soon 🤞🤞🤞🤞

[unphased]

Someone reported nearly 5 gens per second on a 3090 with both TensorRT and LCM Lora. We're living in a pretty good timeline over here. Can't wait to see all this stuff get in so we can use it all together. Never ever thought I'd see a factor of 15+ speed up this quick.

[iChristGit]

Someone reported nearly 5 gens per second on a 3090 with both TensorRT and LCM Lora. We're living in a pretty good timeline over here. Can't wait to see all this stuff get in so we can use it all together. Never ever thought I'd see a factor of 15+ speed up this quick.

Makes me feel happy with my 3090 Ti!
Can you elaborate more on TensorRT? is that something that can be enabled on automatic1111? arent sdp and xformers the only options?

[light-and-ray]

I could make tensorrt works with lcm only after merging lcm lora into model (and training difference in supermerger)

[masonintokyo]

I’m looking forward to this feature too.
I wanna use contrlnets with it.

[iChristGit]

Someone reported nearly 5 gens per second on a 3090 with both TensorRT and LCM Lora. We're living in a pretty good timeline over here. Can't wait to see all this stuff get in so we can use it all together. Never ever thought I'd see a factor of 15+ speed up this quick.

I just installed TensorRT and yeah its amazing (3090Ti)
SDXL 1024x1024 from 11 seconds to 3.5 seconds an image! its a must!

[camoody1]

@iChristGit I assume you ran this on the Dev branch?

[iChristGit]

@iChristGit I assume you ran this on the Dev branch?

Yeah git checkout dev

[light-and-ray]

Someone from Reddit implemented it 4 day ago
https://www.reddit.com/r/StableDiffusion/comments/17ti2zo/you_can_add_the_lcm_sampler_to_a1111_with_a/

You can add the LCM sampler to A1111 with a little trick Tutorial | Guide

So I was trying out the new LCM LoRA and found out the sampler is missing in A1111. As as long shot I just copied the code from Comfy, and to my surprise it seems to work. I think.

You have to make two small edits with a text editor. Here's how you do it:

Edit the file sampling.py found at this path:

...\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py

Add the following at the end of the file:

@torch.no_grad()
def sample_lcm(model, x, sigmas, extra_args=None, callback=None, disable=None, noise_sampler=None):
    extra_args = {} if extra_args is None else extra_args
    noise_sampler = default_noise_sampler(x) if noise_sampler is None else noise_sampler
    s_in = x.new_ones([x.shape[0]])
    for i in trange(len(sigmas) - 1, disable=disable):
        denoised = model(x, sigmas[i] * s_in, **extra_args)
        if callback is not None:
            callback({'x': x, 'i': i, 'sigma': sigmas[i], 'sigma_hat': sigmas[i], 'denoised': denoised})

        x = denoised
        if sigmas[i + 1] > 0:
            x += sigmas[i + 1] * noise_sampler(sigmas[i], sigmas[i + 1])
    return x

The second change is done in the file sd_samplers_kdiffusion.py found here:

...\stable-diffusion-webui-new\modules\sd_samplers_kdiffusion.py

On line 39 add this:

('LCM Test', 'sample_lcm', ['lcm'], {}),

That should give you a new sampler option called 'LCM Test'.

[light-and-ray]

But also there was found a bug in LCM Scheduler, and the algorithm will be updated
huggingface/diffusers#5815

[light-and-ray]

There is almost no difference between "Euler a" and "LCM Test" for SD1. But for SDXL it solves all the problems

[light-and-ray]

I've modified patch from reddit to not edit external repository:

diff --git a/modules/sd_samplers_extra.py b/modules/sd_samplers_extra.py
index 1b981ca8..d154a2b6 100644
--- a/modules/sd_samplers_extra.py
+++ b/modules/sd_samplers_extra.py
@@ -72,3 +72,20 @@ def restart_sampler(model, x, sigmas, extra_args=None, callback=None, disable=No
         last_sigma = new_sigma

     return x
+
+
+@torch.no_grad()
+def sample_lcm(model, x, sigmas, extra_args=None, callback=None, disable=None, noise_sampler=None):
+    extra_args = {} if extra_args is None else extra_args
+    noise_sampler = k_diffusion.sampling.default_noise_sampler(x) if noise_sampler is None else noise_sampler
+    s_in = x.new_ones([x.shape[0]])
+    for i in tqdm.auto.trange(len(sigmas) - 1, disable=disable):
+        denoised = model(x, sigmas[i] * s_in, **extra_args)
+        if callback is not None:
+            callback({'x': x, 'i': i, 'sigma': sigmas[i], 'sigma_hat': sigmas[i], 'denoised': denoised})
+
+        x = denoised
+        if sigmas[i + 1] > 0:
+            x += sigmas[i + 1] * noise_sampler(sigmas[i], sigmas[i + 1])
+    return x
+
diff --git a/modules/sd_samplers_kdiffusion.py b/modules/sd_samplers_kdiffusion.py
index 8a8c87e0..b6c3dc44 100644
--- a/modules/sd_samplers_kdiffusion.py
+++ b/modules/sd_samplers_kdiffusion.py
@@ -36,6 +36,7 @@ samplers_k_diffusion = [
     ('DPM2 a Karras', 'sample_dpm_2_ancestral', ['k_dpm_2_a_ka'], {'scheduler': 'karras', 'discard_next_to_last_sigma': True, "uses_ensd": True, "second_order": True}),
     ('DPM++ 2S a Karras', 'sample_dpmpp_2s_ancestral', ['k_dpmpp_2s_a_ka'], {'scheduler': 'karras', "uses_ensd": True, "second_order": True}),
     ('Restart', sd_samplers_extra.restart_sampler, ['restart'], {'scheduler': 'karras', "second_order": True}),
+    ('LCM Test', sd_samplers_extra.sample_lcm, ['lcm'], {}),
 ]