[BUG]: hybrid_parallel_plugin 模型保存问题

[zryowen123]

🐛 Describe the bug

使用shardformer分支中hybrid_parallel_plugin的模型保存代码测试，模型保存不完整
原始模型llama2-13b 大约24G大小
使用hybrid_parallel_plugin训练完后保存只有13G大小
训练设置 pp=2, tp=2 , 4 *a800

Environment

No response

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

Title: [BUG]: hybrid_parallel_plugin model saving problem

[Fridge003]

请问您训练用的精度是fp16的混合精度吗。如果开了fp16的混合精度运算，在存储的时候就会直接存dtype=fp16的模型参数，所以参数占的空间减小了一半

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

May I ask if the precision you use for training is the mixed precision of fp16?

[zryowen123]

是是fp16的混合精度训练的，模型大小的问题了解了，但加载保存的模型推理时遇到了下面的错误
in :17 │
│ │
│ 14 inputs.pop('token_type_ids', None) │
│ 15 inputs = inputs.to('cpu') │
│ 16 model.eval() │
│ ❱ 17 model.to('cpu') │
│ 18 │
│ 19 outputs = model.generate( │
│ 20 │ │ │ │ **inputs, │
│ │
│ /opt/conda/lib/python3.8/site-packages/transformers/modeling_utils.py:1902 in to │
│ │
│ 1899 │ │ │ │ " model has already been set to the correct devices and casted to the co │
│ 1900 │ │ │ ) │
│ 1901 │ │ else: │
│ ❱ 1902 │ │ │ return super().to(*args, **kwargs) │
│ 1903 │ │
│ 1904 │ def half(self, *args): │
│ 1905 │ │ # Checks if the model has been loaded in 8-bit │
│ │
│ /opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:989 in to │
│ │
│ 986 │ │ │ │ │ │ │ non_blocking, memory_format=convert_to_format) │
│ 987 │ │ │ return t.to(device, dtype if t.is_floating_point() or t.is_complex() else No │
│ 988 │ │ │
│ ❱ 989 │ │ return self._apply(convert) │
│ 990 │ │
│ 991 │ def register_backward_hook( │
│ 992 │ │ self, hook: Callable[['Module', _grad_t, _grad_t], Union[None, Tensor]] │
│ │
│ /opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:641 in _apply │
│ │
│ 638 │ │
│ 639 │ def _apply(self, fn): │
│ 640 │ │ for module in self.children(): │
│ ❱ 641 │ │ │ module._apply(fn) │
│ 642 │ │ │
│ 643 │ │ def compute_should_use_set_data(tensor, tensor_applied): │
│ 644 │ │ │ if torch._has_compatible_shallow_copy_type(tensor, tensor_applied): │
│ │
│ /opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:641 in _apply │
│ │
│ 638 │ │
│ 639 │ def _apply(self, fn): │
│ 640 │ │ for module in self.children(): │
│ ❱ 641 │ │ │ module._apply(fn) │
│ 642 │ │ │
│ 643 │ │ def compute_should_use_set_data(tensor, tensor_applied): │
│ 644 │ │ │ if torch._has_compatible_shallow_copy_type(tensor, tensor_applied): │
│ │
│ /opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:641 in _apply │
│ │
│ 638 │ │
│ 639 │ def _apply(self, fn): │
│ 640 │ │ for module in self.children(): │
│ ❱ 641 │ │ │ module._apply(fn) │
│ 642 │ │ │
│ 643 │ │ def compute_should_use_set_data(tensor, tensor_applied): │
│ 644 │ │ │ if torch._has_compatible_shallow_copy_type(tensor, tensor_applied): │
│ │
│ /opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:641 in _apply │
│ │
│ 638 │ │
│ 639 │ def _apply(self, fn): │
│ 640 │ │ for module in self.children(): │
│ ❱ 641 │ │ │ module._apply(fn) │
│ 642 │ │ │
│ 643 │ │ def compute_should_use_set_data(tensor, tensor_applied): │
│ 644 │ │ │ if torch._has_compatible_shallow_copy_type(tensor, tensor_applied): │
│ │
│ /opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:641 in _apply │
│ │
│ 638 │ │
│ 639 │ def _apply(self, fn): │
│ 640 │ │ for module in self.children(): │
│ ❱ 641 │ │ │ module._apply(fn) │
│ 642 │ │ │
│ 643 │ │ def compute_should_use_set_data(tensor, tensor_applied): │
│ 644 │ │ │ if torch._has_compatible_shallow_copy_type(tensor, tensor_applied): │
│ │
│ /opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:664 in _apply │
│ │
│ 661 │ │ │ # track autograd history of param_applied, so we have to use │
│ 662 │ │ │ # with torch.no_grad(): │
│ 663 │ │ │ with torch.no_grad(): │
│ ❱ 664 │ │ │ │ param_applied = fn(param) │
│ 665 │ │ │ should_use_set_data = compute_should_use_set_data(param, param_applied) │
│ 666 │ │ │ if should_use_set_data: │
│ 667 │ │ │ │ param.data = param_applied │
│ │
│ /opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:987 in convert │
│ │
│ 984 │ │ │ if convert_to_format is not None and t.dim() in (4, 5): │
│ 985 │ │ │ │ return t.to(device, dtype if t.is_floating_point() or t.is_complex() els │
│ 986 │ │ │ │ │ │ │ non_blocking, memory_format=convert_to_format) │
│ ❱ 987 │ │ │ return t.to(device, dtype if t.is_floating_point() or t.is_complex() else No │
│ 988 │ │ │
│ 989 │ │ return self._apply(convert)
NotImplementedError: Cannot copy out of meta tensor; no data!

模型加载的代码：
model = LlamaForCausalLM.from_pretrained(
model_path,
load_in_8bit=False,
device_map="auto",
torch_dtype=torch.float16
)

[Fridge003]

不好意思，HybridParallelPlugin的加载功能目前还没有完成好，但预计在这周可以完成。请您持续关注我们仓库的更新~

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

Sorry, the loading function of HybridParallelPlugin has not been completed yet, but it is expected to be completed this week. Please continue to pay attention to the update of our warehouse~

[zryowen123]

加载还需要定制化么，能否转成huggingface model通用的加载格式么

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

Does the loading still need to be customized? Can it be converted into a common loading format for huggingface model?

[Fridge003]

加载还需要定制化么，能否转成huggingface model通用的加载格式么

目前需要用booster.load做定制化的加载，我们会在不久后的更新中兼容huggingface的from_pretrained加载方式，谢谢您的意见！

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

Does the loading still need to be customized? Can it be converted into a common loading format for huggingface model?

Currently need to use the customized loading of booster.load, we will be compatible with the from_pretrained loading method of huggingface in an update in the near future, thank you for your comments!