Add model compilation in ORT perf test - #25797
Conversation
|
@jywu-msft @chilo-ms Can you please review this? |
|
/azp run Linux QNN CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline,Windows x64 QNN CI Pipeline |
|
Azure Pipelines successfully started running 5 pipeline(s). |
|
/azp run Linux QNN CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline,Windows x64 QNN CI Pipeline |
|
Commenter does not have sufficient privileges for PR 25797 in repo microsoft/onnxruntime |
97d7cda to
39e4ef5
Compare
|
general question. why is this PR adding two ways to compile the model? i.e., with onnxruntime_perf_test and compile_ep_context_model.py. IMO, onnxruntime_perf_test already does a lot - do we need to add more functionality to it or can we just use the python script to first generate the compiled model? |
|
/azp run Linux QNN CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline,Windows x64 QNN CI Pipeline |
|
Azure Pipelines successfully started running 5 pipeline(s). |
The core utility is the same. Just added the same implementation for Python and C++ to reduce dependency overhead and devs can use whichever language they are already working on. Python is quicker to test and prototype. |
|
/azp run Linux QNN CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline,Windows x64 QNN CI Pipeline |
|
Azure Pipelines successfully started running 5 pipeline(s). |
I haven't used this, but a potential concern is having two separate implementations to maintain. is having both paths worth that maintenance overhead? |
IMO, it is not a big maintenance overhead. The python script is independent and does just one and only thing - generate EP context model. For python usecases like LLMs with ORT GenAI and Diffusion pipelines, a python based script is desirable for developers to compile model ahead-of-time, without needing to have the C/C++ build. |
|
Discussed offline. Let's remove the python script for now and only merge the part in perf test. |
|
/azp run Linux QNN CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline,Windows x64 QNN CI Pipeline |
|
Azure Pipelines successfully started running 4 pipeline(s). |
This PR adds model compilation support in ort perf test * Add `--compile_ep_context` flag in onnxruntime_perf_test. This generates EP context model and prints out compilation time and perf statistics for the compiled model * Prints `Compile time cost` for EP context compilation Sample usage: ```sh $ ./onnxruntime_perf_test -e trt-rtx -I -r 1 "/path/to/model.onnx" --compile_ep_context --compile_model_path "/path/to/model_ctx.onnx" ``` Output: ```sh Compile time cost: 17.8012 s Session creation time cost: 0.966619 s First inference time cost: 8151 ms Total inference time cost: 8.08084 s Total inference requests: 1 Average inference time cost: 8080.84 ms Total inference run time: 8.08085 s Number of inferences per second: 0.123749 Avg CPU usage: 6 % Peak working set size: 7861874688 bytes Avg CPU usage:6 Peak working set size:7861874688 Runs:1 Min Latency: 8.08084 s Max Latency: 8.08084 s P50 Latency: 8.08084 s P90 Latency: 8.08084 s P95 Latency: 8.08084 s P99 Latency: 8.08084 s P999 Latency: 8.08084 s ```
This PR adds model compilation support in ort perf test * Add `--compile_ep_context` flag in onnxruntime_perf_test. This generates EP context model and prints out compilation time and perf statistics for the compiled model * Prints `Compile time cost` for EP context compilation Sample usage: ```sh $ ./onnxruntime_perf_test -e trt-rtx -I -r 1 "/path/to/model.onnx" --compile_ep_context --compile_model_path "/path/to/model_ctx.onnx" ``` Output: ```sh Compile time cost: 17.8012 s Session creation time cost: 0.966619 s First inference time cost: 8151 ms Total inference time cost: 8.08084 s Total inference requests: 1 Average inference time cost: 8080.84 ms Total inference run time: 8.08085 s Number of inferences per second: 0.123749 Avg CPU usage: 6 % Peak working set size: 7861874688 bytes Avg CPU usage:6 Peak working set size:7861874688 Runs:1 Min Latency: 8.08084 s Max Latency: 8.08084 s P50 Latency: 8.08084 s P90 Latency: 8.08084 s P95 Latency: 8.08084 s P99 Latency: 8.08084 s P999 Latency: 8.08084 s ```
This PR adds model compilation support in ort perf test
--compile_ep_contextflag in onnxruntime_perf_test. This generates EP context model and prints out compilation time and perf statistics for the compiled modelCompile time costfor EP context compilationSample usage:
Output: