Add self-detecting on-the-fly bfloat16->float16 conversion pass - #741
Conversation
sfatimar
left a comment
There was a problem hiding this comment.
I am more aligned with this change.
a02a919 to
c594c4d
Compare
| DumpOpenVINOEPModel(onnx_model_path_name, model_proto.get(), fused_node); | ||
| ORT_ENFORCE(status.IsOK(), status.ErrorMessage()); | ||
| return model_proto; | ||
| } else if (HasBf16(subgraph)) { |
There was a problem hiding this comment.
Is a check needed for enable_qdq_optimizer . Should you check for GPU here ? Please let me know if you support ep context graphs
There was a problem hiding this comment.
Not necessarily, this is a universal pass, which works for all the IPs.
UPD to an edited comment: this is an else condition for all the qdq_scales-related graph modifications. Overall, qdq_scales and bfloat16 are mutually exclusive, so the current logic is the following: if qdq_scaling pass is requested, we go there with two different paths for NPU and GPU. Else, if the model has bfloat16 initializers, we convert them to fp16 in this pass. Otherwise, we just transfer the model directly to openvino.
Regarding ep context graphs: no, they're not supported, since they're basically an encapsulated OVIR and we can only redirect it to OV, nothing more. So if there is a request from a customer to work with bfloat16 ep context models, we'd need to solve it on the OV side.
sfatimar
left a comment
There was a problem hiding this comment.
Changes look good. Please look at review comments and see if you have subscribed to Coding Style.
|
Please update branch. |
|
@mklimenk |
|
@vthaniel please approve |
|
Reviewer has approved and internal_ci has passed. |
* Add on-the-fly bfloat16->float16 conversion pass * Fix undetected bfloat16 initializers * Remove the option and make the logic implicit * Add tests * Rename detection function * Fix CI for strict aliasing rules --------- Co-authored-by: Vishnudas Thaniel S <vishnudas.thaniel.s@intel.com>
A follow-up to #740 with changed logic. Instead of relying on an external configuration key, perform bfloat16->float16 conversion in case there is at least one tensor in bfloat16 in the model.
https://jira.devtools.intel.com/browse/CVS-170592