Experimental catalyst conversions for low-dimension tensors #35
Description
Directly integrate with the logical plan of Catalyst when working on small dimensions.
For columns containing scalars or vectors, perform an unsafe memory copy to a buffer instead of using scala collections, and provide a fallback for higher-order tensors. It is recommended to flatten the operations for higher order tensors because the Tungsten representation is not very efficient from a memory perspective.
Implementation for the 4 basic types will duplicate some code, so investigate a templating library or scala macros to generate the code for each type?
The benchmark for this task will be the computation of the covariance matrix of a dataset (hopefully, it can get faster than the current computation in Spark).