Question About Reproducing W4A4 Results in the Paper, FP32/W8A8/W4A8 Reproduced Successfully, but W4A4 failed

[nku-ligl]

1.Results
Hi, thanks for releasing the code — it's a really nice work.

I successfully reproduced the FP32, W8A8, and W4A8 results on LSUN-Bedrooms, but I am having significant difficulty reproducing the W4A4 results.

As shown in the table below, the FP32/W8A8/W4A8 results are relatively close to the reported values, while W4A4 shows a much larger gap.

Dataset	Setting	FID
LSUNBedrooms (LDM-4)	FP32 (my reproduction)	3.03
LSUNBedrooms (LDM-4)	FP32 (paper)	2.95
LSUNBedrooms (LDM-4)	W8A8 (my reproduction)	2.93
LSUNBedrooms (LDM-4)	W8A8 (paper)	3.03
LSUNBedrooms (LDM-4)	W4A8 (my reproduction)	3.40
LSUNBedrooms (LDM-4)	W4A8 (paper)	3.26
LSUNBedrooms (LDM-4)	W4A4 (my reproduction)	11.36
LSUNBedrooms (LDM-4)	W4A4 (paper)	5.64

2.Command:
python sample_diffusion_ldm_bedroom.py
-r models/ldm/lsun_beds256/model.ckpt
-n 50000
--batch_size 20
-c 200
-e 1.0
--seed 40
--ptq
--weight_bit 4
--quant_mode qdiff
--cali_st 20
--cali_batch_size 32
--cali_n 256
--quant_act
--act_bit 4
--a_sym
--a_min_max
--cali_data_path datasets/bedroom_sample2040_allst.pt
-l results/QuEST/bedroom/w4a4

3.Additional Notes

I noticed the previous issue:"Strange Results Produced on W4A4 LSUN-Bedrooms Dataset Calibration",
Following the discussion there: I did NOT use running_stat when executing w4a4 quantization, and I set aq_params['channel_wise'] = True.
However, the W4A4 result is still significantly worse than the paper.

I also tested LSUN-Churches with W4A4: the generated samples are mostly corrupted. For LSUN-Bedrooms, although the FID is worse than reported result in paper, it still generates semantically reasonable images.

I also tried to investigate the issue by reading the source code myself, but I could not identify the cause.

Could you please give some advice on whether there are any additional hyperparameters, calibration settings, or implementation details required to reproduce the reported W4A4 results?

Thanks again for the great work.

4.Generated W4A4 samples

W4A4_LSUN_Bedroom

W4A4_LSUN_Churches

[hatchetProject]

Thanks for your interest in our work! The current fine-tuning setup is configured for ImageNet. Achieving good performance on other datasets typically requires fine-tuning different modules, which can be adjusted in the pd_optimize_timewise() function. If I recall correctly, for LSUN Bedroom ww first fine-tune the time-embedding layers, followed by only the Q, K, V weights (no feed-forward layers). LSUN Churches should follow a similar recipe, or possibly just fine-tuning the time-embedding layers alone.

[nku-ligl]

Sorry for the late reply and thank you so much for your response.

I have checked the code and found that the functions pd_optimize_timeembed() and pd_optimize_timewise() are sourced from the file post_layer_recon.py. I noticed that different quantization codes use different versions of post_layer_recon, and the corresponding pd_optimizer_timewise function also differs across model variants.

For example, sample_diffusion_ldm_bedroom.py and sample_diffusion_ldm_church.py import pd_optimizer_timewise with the statement import qdiff.post_layer_recon_uncond, whereas sample_diffusion_ldm_imagenet.py imports it via import qdiff.post_layer_recon_imagenet.

Hence, I guess that the pd_optimize_timeembed() function is not the main reason behind the unsuccessful reproduction of the w4a4 quantization result. As you noted that the current code configuration is adapted for ImageNet, I will test the w4a4 performance on the ImageNet dataset later.

Thanks again for your reply.

[hatchetProject]

Thank you for the reminder, I do have the unconditional generation separated from the ImageNet one in this repo, so the choice of layers is probably not the problem.

I checked my experiment logs, for LSUN-Bedroom, actually only doing channel_wise and time embedding layer finetuning could give an FID of 6.41. Maybe you could check on that, since it is pretty fast. You can also try to enable / disable running_stat to see if it works.

[nku-ligl]

Sorry for the late reply — it took a while to generate the images. Following your feedback, I re-ran the experiment for w4a4_lsun_bedroom, but the results turned out unsatisfactory.

In my setup, I set channel_wise=True and executed the pd_optimize_timeembed function, while commented the pd_optimize_timewise function. Could I have misunderstood the your reply?

w4a4_running_stat:

w4a4_without_running_stat:

FID results:

dataset	bit-width	running_stat	pd_optimize_timeembed	pd_optimize_timewise	FID
lsun_bedroom	w4a4	True	True	False	384.46
lsun_bedroom	w4a4	False	True	False	233.50

[hatchetProject]

The results look strange. Actually, if you set channel_wise=True (actually in the current code it is defaulted to True if act_bit=4), the raw quantized model should already produce some readable results, instead of corrupting entirely. Have you loaded the 4-bit weight-quantized model first? You can save a 4-bit weight-quantized model, see its W4A32 results, then load this model and then use the code to quantize the activations (with --resume_w argument). It will be faster and easier to debug.

If these doesn't work, I honestly don't know what is happening. You can try alternatives such as EfficientDM, or a more powerful method like SVDQuant, but I don't know if it supports UNet and LSUN datasets.

[nku-ligl]

Many thanks for your response. I shall try it out. Hope your research goes well and you enjoy your life.