[BUG] [0.8.1] INT8 model loading/inference issue

[sindhuvahinis]

Describe the bug

We conducted tests on OPT/GPTJ/GPT-Neox/BLOOM 7B INT8, these models are all producing garbage outputs on DeepSpeed 0.8.1

OPT model is NCCL communication issue
GPT-NeoX 20B is producing garbage
BLOOM-7B: shape '[1, 4, 32, 384]' is invalid for input of size 16384

How we tested?
We generated int8 checkpoints of the model and then loaded them back. Example of doing the same with DS inference test suite.

deepspeed --num_nodes 1 \
    --num_gpus 8 \
    inference-test.py \
    --use_kernel \
    --ds_inference \
    --use_meta_tensor \
    --name EleutherAI/gpt-neox-20b \
    --checkpoint_path /tmp/ws/gpt-neox-20b/ \
    --save_mp_checkpoint_path /tmp/ws/sharded-gpt-neox-20b/ \
    --dtype int8

deepspeed --num_nodes 1 \
    --num_gpus 8     \
    inference-test.py     \
    --use_kernel     \
    --ds_inference     \
    --use_meta_tensor \
    --name EleutherAI/gpt-neox-20b     \
    --checkpoint_path /tmp/ws/sharded-gpt-neox-20b/ \
    --dtype int8

More info this.
#2770

Creating a new issue to track the int8 checkpoint loading issue.

[lanking520]

@HeyangQin

[HeyangQin]

Hi @lanking520 @sindhuvahinis, PR #2875 has been merged to address part of the issue. For now, the INT saving / loading is still not fully functional due to kernel issues. I would suggest you to use the workaround of saving checkpoints with fp32/fp16 and then load it with int8 to get around this issue for the time being.

[lanking520]

Thanks for the info. Given the above context, at least the INT8 inference (load from FP16 ckpt) should work as expected?

[lanking520]

@HeyangQin So I would assume developer should follow this path:

• Load a model (e.g GPT-NeoX20B) and save to DS_sharded checkpoint in FP16.
• Using ckpt loading on FP16 and set the dtype in init_inference to torch.int8

And this should work as expected. The only drawback is, developer may still facing runtime GPU OOM issue when converting FP16 to INT8 during runtime

[crazycth]

@HeyangQin Load BLOOM model with FP16 checkpoint and then set dtype=int8 in init_inference not work : (

Could u please answer this issue: #2923 , and I found some people face the same problem .