[RFC] Runtime-derived ExecuTorch backend arguments

[dbort]

This document proposes a way for applications to pass runtime-derived arguments to backends, letting backends behave differently based on information only known at runtime.

Note that this is not an official RFC process, just something I put together to share this document.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/1262

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[robell]

A common interface is good as specifying backend properties in a consistent way is useful, even if partitioning is fixed. I assume the arguments can be of all types (see refs below but we pass bool, string, scalars)

Only delta I see is that we have some interfaces with runtime_init time arguments, see ArmNN Instance level options in the below link.

One route to implementation on android is TOSA on ACL (Cortex-A) and TOSA on GPU so having a common partitioner/ArmBackend in python and a delegate device which JiT’s to a specific target IP is quite likely. ArmNN has the same capabilities.

The ArmNN api would also accept a target device and other properties at runtime if extended to consume a TOSA representation or other IR.

https://arm-software.github.io/armnn/latest/runtimeoptions.html
https://arm-software.github.io/ComputeLibrary/latest/architecture.xhtml#architecture_opencl_tuner

We’d also like to take Symint like data in future, which can be specialised with input sizes on devices which need to know this information statically. (finalise shapes at init time)

We don’t need the ability to adjust options between execute() calls, most are build time and heavyweight and it’s expected a new instance is created for such heavy activities.

[dbort]

Thanks for taking a look @robell, and for the detailed background.

We don’t need the ability to adjust options between execute() calls, most are build time and heavyweight and it’s expected a new instance is created for such heavy activities.

Other backend authors gave the same feedback, so I'm going to de-prioritize that feature for now. There are some backends that could benefit from it, but I don't want to complicate the user experience unnecessarily until I hear specific requests for per-execute()-call-args.

Only delta I see is that we have some interfaces with runtime_init time arguments, see ArmNN Instance level options in the below link.

Interesting, thanks for bringing this up. This is a different problem, and one that I think we could first try addressing with a convention rather than an API change. A couple possibilities, though I'm not sure of the subtleties of exactly when the setup needs to happen:

1. Applications could manually create and register their ArmBackend instance (instead of relying on load-time initialization), giving them a chance to pass parameters to its constructor. The ctor could then coordinate this global ArmNN setup. Though it might need some internal global state to protect against the situation where multiple ArmBackend instances are created.
2. The ArmNN backend could provide a header for applications to include, and that header could declare an init function that takes the appropriate parameters. Internally, the ArmNN init function and runtime_init() could share a global state variable with states like kUnintialized, kArmNNInitialized, kExecutorchRuntimeInitialized. Calling the ArmNN init function could assert that the state is kUninitialized, set the global parameters, then advance the state to kArmNNInitialized. Similarly, runtime_init() could assert that the state is kArmNNInitialized and then advance the state to kExecutorchRuntimeInitialized. This would ensure that the application called the ArmNN init function before calling runtime_init(). This might be too strict, though.

[cccclai]

Follow up discussion in #10216