Support MXINT4 scheme

[mengniwang95]

Description

Support MXINT4 scheme

How to use:

model quantization:

CUDA_VISIBLE_DEVICES=0 auto-round --model /models/Llama-3.2-3B/ --scheme MXINT4 --iters 0 --format auto_round

inference with transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "tmp_autoround/Llama-3.2-3B-mxint-w4g32/"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")
input_ids = tokenizer("Hello my name is", return_tensors="pt").input_ids.to(
    model.device
)
output = model.generate(input_ids, max_new_tokens=100)
print(tokenizer.decode(output[0]))

results:

BF16
hf ({'pretrained': '/models/Llama-3.1-8B-Instruct', 'add_bos_token': True}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: 64

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
hellaswag	1	none	0	acc	↑	0.5977	±	0.0049
		none	0	acc_norm	↑	0.7954	±	0.0040
piqa	1	none	0	acc	↑	0.8014	±	0.0093
		none	0	acc_norm	↑	0.8101	±	0.0092

hf ({'pretrained': '/models/Llama-3.1-8B-Instruct', 'add_bos_token': True}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: 16

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k_llama	3	flexible_extract	8	exact_match	↑	0.8620	±	0.0095
		strict_match	8	exact_match	↑	0.8567	±	0.0097
mmlu_llama	1	strict_match		exact_match	↑	0.6946	±	0.0037

MXINT4
hf ({'pretrained': 'tmp_autoround/Llama-3.1-8B-Instruct-mxint-w4g32/', 'add_bos_token': True}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: 64

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
hellaswag	1	none	0	acc	↑	0.5460	±	0.0050
		none	0	acc_norm	↑	0.7396	±	0.0044
piqa	1	none	0	acc	↑	0.7535	±	0.0101
		none	0	acc_norm	↑	0.7693	±	0.0098

hf ({'pretrained': 'tmp_autoround/Llama-3.1-8B-Instruct-mxint-w4g32', 'add_bos_token': True}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: 16

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k_llama	3	flexible_extract	8	exact_match	↑	0.6285	±	0.0133
		strict_match	8	exact_match	↑	0.5663	±	0.0137
mmlu_llama	1	strict_match		exact_match	↑	0.5786	±	0.0040

Type of Change

• Bug fix
• New feature
• Documentation update
• Performance improvement
• Code refactoring
• Other (please specify):

[hshen14]

@lvliang-intel pls review also.

[mengniwang95]

/azp run Unit-Test-CUDA-AutoRound

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[lkk12014402]

LGTM

[mengniwang95]

/azp run Unit-Test-CUDA-AutoRound

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).