Handle views in advanced indexing

[manopapad]

Copies that involve advanced indexing are implemented with a scatter/gather copy. The current code ignores the transforms on the RegionFields, i.e. it doesn't take into account the translation from NumPy (local) index space (every view's indices start from 0) to Legion (global) index space (every subregion's indices start wherever that subregion is placed within the parent region).

In the general case the base, index and value arrays can all be views:

a = np.arange(10)
b = np.arange(10)
c = np.arange(9, -1, -1)
x = a[2:7][ c[5:10] ] # __get_item__
a[2:7][ c[5:10] ] = b[3:8] # __set_item__

The logic to handle the general case of __get_item__ might be:

• use the RegionField backing the newly created result array (which has no transform) as dst in the gather/scatter copy
• use root of base array as src
• if neither the base nor the index array are views, use index array's RegionField as src_indirect
• if only the index array is a view, materialize it as a new array, getting rid of any transforms (like doing c[5:10] * 1 above), use the final array as src_indirect
• otherwise, apply the base array's transform (\x. x+2 in the example above) on each element of the index array (the transform operation will naturally get rid of the index's transform, if any), use the output array as src_indirect

And for __set_item__:

• use value array's RegionField as src in the gather/scatter copy
• if the value array is a view, create a new field to materialize its transform (stores the mapping 0:5 -> 3:8 for the example above), use that as src_indirect
• use root of base array as dst
• if neither the base nor the index array are views, use index array's RegionField as dst_indirect
• if only the index array is a view, materialize it as a new array, getting rid of any transforms (like doing c[5:10] * 1 above), use the final array as dst_indirect
• otherwise, apply the base array's transform (\x. x+2 in the example above) on each element of the index array (the transform operation will naturally get rid of the index's transform, if any), use the output array as dst_indirect

The work on StanfordLegion/legion#705 would allow us to avoid materializing at least some of these fields.

Edit: add some more detail, enumerate more cases, fix typos

[manopapad]

This is mostly fixed in https://github.com/manopapad/legate.numpy/tree/slicing, except for views on the RHS of __set_item__, which will be trivial to support once StanfordLegion/legion#705 is fixed.

[manopapad]

#235 implements advanced indexing support. The final implementation was a little different than the description above:

For __get_item__:

• use newly created result array as dst
• if the index array is a view, make a copy to remove its transform
• use (copy of) index array as src_indirect
• if the base array is a view, make a copy to remove its transform
• use (copy of) base array as src

For __set_item__:

• if the value array is a view, make a copy to remove its transform
• use (copy of) value array as src
• if the index array is a view, make a copy to remove its transform
• use (copy of) index array as dst_indirect
• if the base array is a view, make a copy to remove its transform -- this sidesteps the issue of needing to convert from "local" NumPy-level indices to "global" Legion-level indices
• use (copy of) base array as dst
• after the operation, if we made a copy of the base array, copy the new contents back into the original base array (at the appropriate offset)

Keeping this issue open to track the work on incorporating the Legion-level support for attaching transforms on copies: StanfordLegion/legion#705.