Hi,
I try to convert MT models (from WMT 2019) to ONNX, but I catch a lot of problems during this process. I experimented with a single_model pipeline.
I try to export Encoder and Decoder models separately.
TransformerEncoder.
The main problem with Encoder is that torch.onnx.export(...) can't work with NoneType arguments. But some arguments of TransformerEncoder get as a NoneType by default for a single model (for example, cls_input has a None type for single_model mode). To work around it, I removed all arguments with None type from interfaces. After, I apply prepare_to_onnx_export_(...) for all modules of TransformerEncoder recursively. After that model can be serialized to ONNX, but some nodes of the graph have a custom type (Aten), so, this model can't be executed using onnxruntime, TensorRT, OpenVINO etc.
TransformerDecoder.
Decoder has the same problems with NoneType arguments.
But after when we apply the same workarounds as in Encoder model and try to convert to ONNX, we get some errors:
-
"UserWarning: ONNX export failed on ATen operator triu because torch.onnx.symbolic_opset9.triu does not exist".
This error related to TransformerDecoder.buffered_future_mask(...) function.
-
"RuntimeError: Unsupported: ONNX export of Slice with dynamic inputs. DynamicSlice is a deprecated experimental op. Please use statically allocated variables or export to a higher opset version.".
And I can't to find which module throws this error (trace broke after decoder call).
So, and I can't move forward with thуs errors. Does anybody have a positive experience in export fairseq models to ONNX?
Hi,
I try to convert MT models (from WMT 2019) to ONNX, but I catch a lot of problems during this process. I experimented with a single_model pipeline.
I try to export Encoder and Decoder models separately.
TransformerEncoder.
The main problem with Encoder is that torch.onnx.export(...) can't work with NoneType arguments. But some arguments of TransformerEncoder get as a NoneType by default for a single model (for example, cls_input has a None type for single_model mode). To work around it, I removed all arguments with None type from interfaces. After, I apply prepare_to_onnx_export_(...) for all modules of TransformerEncoder recursively. After that model can be serialized to ONNX, but some nodes of the graph have a custom type (Aten), so, this model can't be executed using onnxruntime, TensorRT, OpenVINO etc.
TransformerDecoder.
Decoder has the same problems with NoneType arguments.
But after when we apply the same workarounds as in Encoder model and try to convert to ONNX, we get some errors:
"UserWarning: ONNX export failed on ATen operator triu because torch.onnx.symbolic_opset9.triu does not exist".
This error related to TransformerDecoder.buffered_future_mask(...) function.
"RuntimeError: Unsupported: ONNX export of Slice with dynamic inputs. DynamicSlice is a deprecated experimental op. Please use statically allocated variables or export to a higher opset version.".
And I can't to find which module throws this error (trace broke after decoder call).
So, and I can't move forward with thуs errors. Does anybody have a positive experience in export fairseq models to ONNX?