ggml : reuse quantum structs across backends

[ggerganov]

Trying to see if we can have the quantum structs declared in a single place (e.g. ggml-common.h)

• The #define GGML_COMMON_AGGR data is needed because nvcc does not allow members with constructors (half) in anonymous struct
• ggml-common.h is structured in such a way to allow the following usage:

// some-file.h
#define GGML_COMMON_DECL_C
#include "ggml-common.h"

// some-file.c
#include "some-file.h"

#define GGML_COMMON_IMPL_C
#include "ggml-common.h"

• On the CPU, block_q8_1 now uses fp16 instead of fp32 for d and s. This was already the case for CUDA and to make things similar, I decided to apply the same format on the CPU. This type is used only to quantize activations, so it is not breaking any existing models

TODO:

• CPU
• CUDA
• Metal
• SYCL
• Vulkan (might not be applicable here)
• Kompute (might not be applicable here)

[ggerganov]

I wonder if this change is worth it. @slaren what do you think?

We reduce code duplication, but maybe the macros are bit too much. Any better way to do this?

[slaren]

I am not sure if this would be better, but it is also possible to access the data as a half2 with a reinterpret cast, ie. *(half2*)&x.d. That wouldn't require changes to the structs for CUDA.

[ggerganov]

I am not sure if this would be better, but it is also possible to access the data as a half2 with a reinterpret cast, ie. *(half2*)&x.d. That wouldn't require changes to the structs for CUDA.

Yup, that's an option, but I'm also not sure it would be better

I'm 50/50 about merging this. We can potentially adopt this pattern and use it for other stuff that can be shared by introducing more GGML_COMMON_XXX guards. But on the other hand it is a little ugly. So I don't know

[slaren]

Avoiding code duplication is definitely good. We can always iterate over it in the future if we figure a cleaner way to do this.

[ikawrakow]

I think these CUDA-specific macros (QIX, QRX) shouldn't be propagated to the other backends. If it so happens that another back-end uses these, it would be better to just duplicate there.

[ggerganov]

Ok, will merge this after the CI is green

[ggerganov]

These thread sanitizer failures seem unrelated to our changes:

https://github.com/ggerganov/llama.cpp/actions/runs/8246447769/job/22552498578?pr=5943#step:5:27

Maybe related to this: google/sanitizers#1716?
Though the runner seems to be running kernel 6.5.0 so not sure