Allow users to bind arbitrary memory using raw pointers - #10428
Merged
Conversation
skottmckay
reviewed
Jan 31, 2022
| /// <param name="elementType">element type</param> | ||
| /// <param name="pointer">the actual pointer to memory</param> | ||
| /// <param name="sizeInBytes">size of the allocation in bytes</param> | ||
| public OrtExternalAllocation(OrtMemoryInfo memInfo, long[] shape, Tensors.TensorElementType elementType, IntPtr pointer, long sizeInBytes) |
Contributor
There was a problem hiding this comment.
Do we need this to be provided separately to the size that can be inferred from the shape? #Closed
Contributor
Author
There was a problem hiding this comment.
I was going back and forth on this. Finally, I decided to supply this argument/property to validate the user knows what they are doing. It is a common mistake to make a wrong choice betwen then number of elements and the buffer size in bytes.
skottmckay
reviewed
Jan 31, 2022
| if (width < 1) | ||
| { | ||
| throw new OnnxRuntimeException(ErrorCode.InvalidArgument, "Unsupported data type (such as string)"); | ||
| } |
Contributor
There was a problem hiding this comment.
nit: include elementType in the error message #Closed
skottmckay
reviewed
Jan 31, 2022
| int width; | ||
| TensorElementTypeConverter.GetTypeAndWidth(elementType, out type, out width); | ||
| if (width < 1) | ||
| { |
Contributor
There was a problem hiding this comment.
nit: can we update GetTypeAndWidth to return true/false if successful? if not, would checking type==null be a little more obvious than width < 1? #Closed
| } | ||
|
|
||
| /// <summary> | ||
| /// Bind externally allocated memory as input |
Member
There was a problem hiding this comment.
nit: Should this description also include a line similar to the other BindInput() overload (OrtMemoryAllocation continues to own the chunk of native memory and should be alive until the end of execution.) for completeness ? #Resolved
edgchen1
reviewed
Jan 31, 2022
| Assert.Equal(outputData, tensor.ToArray<float>(), new FloatComparer()); | ||
| } | ||
| } | ||
| // 3. Pretend we are using external allocation which is currently on CPU |
Contributor
edgchen1
reviewed
Feb 1, 2022
| throw new NotSupportedException(nameof(NativeOnnxTensorMemory<T>) + " does not support T = " + nameof(T)); | ||
| { | ||
| var message = String.Format("The NativeOnnxTensorMemory<T> type being instantiated for T = : {0} while supplied OrtValue contains T = {1}", | ||
| nameof(T), nameof(type)); |
Contributor
edgchen1
reviewed
Feb 1, 2022
| /// The model will read the specified input from that memory | ||
| /// possibly avoiding the need to copy between devices. The user code continues to own | ||
| /// the chunk of externally allocated memory, and the allocation should be alive until the end of execution. | ||
| /// by the Tensor of the given size. |
Contributor
edgchen1
reviewed
Feb 1, 2022
| if (elementType == TensorElementType.String) | ||
| { | ||
| throw new OnnxRuntimeException(ErrorCode.InvalidArgument, | ||
| "Can not use map managed strings buffer to native OrtValue"); |
Contributor
yuslepukhin
force-pushed
the
yuslepukhin/bind_external
branch
from
February 1, 2022 19:21
af53a53 to
dd43824
Compare
edgchen1
approved these changes
Feb 1, 2022
hariharans29
approved these changes
Feb 2, 2022
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
Introduce OrtExternalAllocation class that allows to supply type, shape and raw memory pointer for binding and other purposes. Introduce reasonable shape checks against the supplied size.
Could not find a suitable framework class that would give its memory back as a raw Ptr.
Motivation and Context
OrtIoBinding does not currently allow binding of raw native and/or device-based buffers that were user-allocated or potentially by another framework. Those are usually expressed by
IntPtr.Re: #10180