Need support for GemmaForCausalLM

[maziyarpanahi]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Feature Description

Please provide a detailed written description of what you were trying to do, and what you expected llama.cpp to do as an enhancement.

Google just released Gemma models for 7B and 2B under GemmaForCausalLM arch. https://huggingface.co/models?other=gemma

Motivation

Please provide a detailed written description of reasons why this feature is necessary and how it is useful to llama.cpp users.

Possible Implementation

If you have an idea as to how it can be implemented, please write a detailed description. Feel free to give links to external sources or share visuals that might be helpful to understand the details better.

Maybe GemmaForCausalLM can be replicated via Llama-2 or Mistral to convert it to GGUF? Clearly there is a way since they also offer GGUF model in the same model's card, how it is twice the size of the model is beyond me! (maybe it's 32bit)

[maziyarpanahi]

My bad! 580111d

[qnixsynapse]

Gemma already has a gguf but I am not able to quant it for some reason! (I only tried on colab because of its size which is 34GB)

[maziyarpanahi]

I pulled the latest from the main branch and re-build it. I cannot convert to 16bit GGUF and quantized from there, the quant models fail with:

llama_model_load: error loading model: create_tensor: tensor 'output.weight' not found
llama_load_model_from_file: failed to load model
llama_init_from_gpt_params: error: failed to load model 'quantized/google/gemma-7b/gemma-7b.Q4_K_S.gguf'
main: error: unable to load model

[hannibalhuang]

not sure if @ggerganov could have time to take a look at the problem

[rahuldshetty]

@akarshanbiswas I was able to quant it by starting from the gguf weights released on official repo and with the latest llama.cpp branch. I've uploaded some of the quantized version in my HF repo.

[qnixsynapse]

@rahuldshetty Do you have quants of the 7b instruction model?

[akx]

The model has a output_norm.weight tensor though.

[rahuldshetty]

@akarshanbiswas Just uploaded 4bit quantized version of Gemma-7b instruction model.

[qnixsynapse]

@rahuldshetty Awesome. Thank you.

[remghoost]

@rahuldshetty Hey, thanks for the quick responses!

I seem to be unable to run that quant as well...?
It doesn't even try and load the model.

Are you using a specific branch for inference...?

Tested with both llamacpp (b2222) and koboldcpp, both failing to load the model.

D:\llm\llamacpp\b2222 python 11>.\server.exe -m "D:\llm\llamacpp\gemma-7b-it-Q4_K_M.gguf"
ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no
ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
ggml_init_cublas: found 1 CUDA devices:
  Device 0: NVIDIA GeForce GTX 1060 6GB, compute capability 6.1, VMM: yes
{"timestamp":1708531713,"level":"INFO","function":"main","line":2533,"message":"build info","build":2222,"commit":"88c46cbd"}
{"timestamp":1708531713,"level":"INFO","function":"main","line":2540,"message":"system info","n_threads":10,"n_threads_batch":-1,"total_threads":12,"system_info":"AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 0 | "}

llama server listening at http://127.0.0.1:8080

{"timestamp":1708531713,"level":"INFO","function":"main","line":2692,"message":"HTTP server listening","hostname":"127.0.0.1","port":"8080"}

D:\llm\llamacpp\b2222 python 11>

Spot checked b2222 with the OpenHermes-2.5-Mistral-7B-16k-GGUF (Q4_K_S) and it loads fine, so I'm guessing it's not an error on my part...

[postmasters]

According to the technical paper, there are more than 8 billion parameters. I suspect that, including compute buffer, your GPU of 6GB may not be able to load this Q4 quantized model.

[rahuldshetty]

I think @postmasters is right. I was able to run the inference on Google Colab using the same weights. Refer: https://colab.research.google.com/drive/1uVIz_Y6mdjjRgnfC7X2jplHz7aUiQw0I?usp=sharing

[remghoost]

Hmm. I'll test it again locally in a while.

That colab link does indeed work.

It should still be able to run with CPU only inference though, correct...?
Even when attempting to not offload any of the layers to my GPU, it still didn't want to run...

It seems as if it is a problem on my end though. My apologies.
I'll do more research. Thank you for the information!

[maziyarpanahi]

@akarshanbiswas I was able to quant it by starting from the gguf weights released on official repo and with the latest llama.cpp branch. I've uploaded some of the quantized version in my HF repo.

Thank you @rahuldshetty for your work. I guess we can use the official GGUF Google has provided, but there are 2 issues:

• why is that thing 2.5x larger than the original model? Is it 32bit? (out of curiosity, quant from 32bit would be even better I guess)
• how did they make that GGUF file?