Making a "quantize-ggml_16bit-to-gptq.py" script?

[BadisG]

Hello,

I know the quantize.py converts a ggml 16 bits into a ggml 4 bits RTN.
Do you think it's possible to create a script that converts a ggml 16 bits into a ggml 4bits GPTQ?

Referring to this repository, it appears that the current implementation of the quantization relies only on GPU, which demands a significant amount of VRAM and might not be suitable for the average user.

A new script, which we could call "quantize-ggml_16bit-to-gptq.py", could be designed to use only CPU and RAM resources, making it more accessible to the general public.

[FNsi]

Maybe you can check ggml.c to edit the q4 type?

[prusnak]

quantize.py does not do any conversion by itself - it just calls ./quantize, so you might want to dig into examples/quantize/quantize.cpp for answers

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.