py : add Gemma conversion from HF models

[ggerganov]

# gemma-2b
python3 convert-hf-to-gguf.py ~/Data/huggingface/gemma-2b/ --outfile models/gemma-2b/ggml-model-f16.gguf --outtype f16

# gemma-7b
python3 convert-hf-to-gguf.py ~/Data/huggingface/gemma-7b/ --outfile models/gemma-7b/ggml-model-f16.gguf --outtype f16

[twoxfh]

I successfully created a 2b GGUF and loaded model with the server in master. Thanks!

[Yefori-Go]

Works well for me

[postmasters]

I notice that the HF config.json says there are 256000 tokens. But the embedding layer is 256128 x d_model. Not sure if there would be latent issues later.

[ggerganov]

I noticed that as well, but I think the actual tensor shape in the safetensors files is [2048, 256000], instead of [2048, 256128] (Gemma-2B), leading to discrepancy with the published FP32 GGUF files. So we probably have to pad with 0s? But what would be the point of that? Not sure - would be helpful if we get some more eyes on this

[postmasters]

I just checked. The original internal checkpoint uses [256128, 3072] for 7B. Perhaps the The conversion from that checkpoint to SafeTensors has dropped the last hundred tokens.

[Ronnie-Leon76]

I have tried quantizing a fine-tuned gemma-7b model that was loaded as 4-bit but I get the error: Can not map tensor 'model.layers.0.mlp.down_proj.weight.absmax' above. @ggerganov I'll appreciate it if you help me resolve this issue.

[diogo-garcia]

I have tried quantizing a fine-tuned gemma-7b model that was loaded as 4-bit but I get the error: Can not map tensor 'model.layers.0.mlp.down_proj.weight.absmax' above. @ggerganov I'll appreciate it if you help me resolve this issue.

I recently downloaded the Meta-Llama-3-8b model from huggingface and attempted to convert it to GGUF format using the following command:

python3 /root/llama.cpp/convert-hf-to-gguf.py /root/models/meta-llama-3-8b --outfile /root/models/meta-llama-3-8b.gguf --outtype f32

However, I encountered the same error. The specific error message I received was:

File "/root/llama.cpp/convert-hf-to-gguf.py", line 182, in map_tensor_name
raise ValueError(f"Can not map tensor {name!r} {try_suffixes} {self.tensor_map}")
ValueError: Can not map tensor 'model.layers.0.mlp.down_proj.weight.absmax'

I also tried modifying the function map_tensor_name in the script to change the suffixes from .weight to .weight_map, which resolved the previous error, but now I am getting a new error:

File "/root/llama.cpp/convert-hf-to-gguf.py", line 182, in map_tensor_name
raise ValueError(f"Can not map tensor {name!r} {try_suffixes} {self.tensor_map}")
ValueError: Can not map tensor 'model.embed_tokens.weight'

The reason I made this change is that the structure of model.safetensors.index.json indicates that the weights are mapped with suffixes like .weight_map. Here is an excerpt from the JSON file for reference:

{
"metadata": {
"total_size": 6027779904
},
"weight_map": {
"lm_head.weight": "model-00002-of-00002.safetensors",
"model.embed_tokens.weight": "model-00001-of-00002.safetensors",
"model.layers.0.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.0.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.0.mlp.down_proj.weight.absmax": "model-00001-of-00002.safetensors",
"model.layers.0.mlp.down_proj.weight.quant_map": "model-00001-of-00002.safetensors",
...

Could anyone please provide guidance on how to properly map these tensors, or if there's a different approach needed for this conversion? Thank you!