gemma : use more bits for the token_embd.weight tensor

[ggerganov]

Based on some anecdotal runs with Q4 quantizations, it seems that the quality of the generated responses is very sensitive to the type of the token_embd.weight tensor:

make -j && ./main -m models/gemma-2b/ggml-model-q4_k_m.gguf -p "I believe the meaning of life is" -n 64 -s 3 -ngl 99

...

 I believe the meaning of life is to do what you want, not what others tell you to do. Interehavior that looks like obstinacy is obstinacy in fact Interehavior that looks obstinacy is obstinacy Interehavior Interehavior Interehavior Interehavior Interehavior Interehavior Interehavior Interehavior Interehavior Interehavior Interehavior Interehavior Interehavior Interehavior Intere

Quantizing this tensor to Q8_0 seems like a safe bet. Thought, there might be better strategies

[slaren]

I imagine that for models that share the same tensor for token_embd and output, using the quantization type of the output tensor would be a safe bet.

[hannibalhuang]

#5651 as well

[ggerganov]

I imagine that for models that share the same tensor for token_embd and output, using the quantization type of the output tensor would be a safe bet.

I changed it as suggested. Did a couple of ppl runs with Gemma 2B:

F16:    PPL = 9.1298 +/- 0.06169
Q4_K_M: PPL = 9.4794 +/- 0.06483 (token_embd.weight is Q8_0, 531.25 MiB)
Q4_K_M: PPL = 9.6781 +/- 0.06653 (token_embd.weight is Q6_K, 410.16 MiB)

For comparison, this is the PPL on master:

Q4_K_M: PPL = 13.3236 +/- 0.10038 (token_embd.weight is Q4_K, 281.25 MiB)

Also, here is the speed on M2 Ultra using different types for the tensor:

model	size	params	backend	ngl	test	t/s
gemma 2B F16 (guessed)	4.67 GiB	2.51 B	Metal	99	pp 512	1702.22 ± 193.56
gemma 2B F16 (guessed)	4.67 GiB	2.51 B	Metal	99	tg 128	68.23 ± 1.36
gemma 2B Q4_K - Medium (Q8_0)	1.63 GiB	2.51 B	Metal	99	pp 512	1490.68 ± 160.39
gemma 2B Q4_K - Medium (Q8_0)	1.63 GiB	2.51 B	Metal	99	tg 128	120.98 ± 1.93
gemma 2B Q4_K - Medium (Q6_K)	1.51 GiB	2.51 B	Metal	99	pp 512	1389.83 ± 122.54
gemma 2B Q4_K - Medium (Q6_K)	1.51 GiB	2.51 B	Metal	99	tg 128	127.55 ± 2.19

build: 488bd97 (2232)

[jingnanzhou]

@ggerganov , FYI: llama-cpp-python does not work for gemma gguf either