Vortex GPU support

[robert3005]

Part of making Vortex a truly universal format is making sure we can efficiently decode it on a GPU with minimal CPU involvement

Planning - File Format

We need to be able to plan the scan of a file and be able to figure out decoding kernels to use without reading data buffers. Currently the layout metadata is outlined in the footer of the file but the array metadata lives as the header of data segments. We need to have access to those array metadata buffers without having to read data buffers.

Planning - API

Having GPU resident arrays and computation means that the CPU, which does the planning work, doesn't operate on arrays as they are defined at the moment. We need to be able to perform transformations on arrays that 1) don't have main memory resident data, 2) don't assume that arrays have buffers associated with them

Pipelining

In our limited benchmarking we have realised that pipelining achieves A LOT better performance than dispatching multiple kernels. We need to be able to generate pipelined kernels for arrays that are going to be used on gpus

Patches

Pipelining patches requires storing patches in a layout that's friendly to pipelining with fastlanes. This means that we have to be able to store patches offset and those offsets have to be in transposed order.

Strings

GSST is an implementation of FSST algorithm that's GPU friendly. We should explore integrating GSST into our FSST encoding (which should be backwards compatible) and make sure we can replicate the results.

[a10y]

GSST doesn't appear to be open source anymore. It looks like there's a related project from a different TU masters thesis with code here https://github.com/timanema/fsst-gpu