PCodec-inspired ideas

[AdamGS]

After spending about a couple of days playing around with pcodec, reading the paper (which I’m still going through in more depth), and talking to multiple people, here are the main ideas/things I have:

1. IntMult and FloatMult encodings that generalize the form a * m + n + patch (forgive my lack of latex skills), seems extremely useful for a lot of data that is stored in in large ptypes but has finite scale (currency, timestamps etc). Another paths pcodec seems to use a lot is recursive FOR and consecutive delta encoding (think timestamps taken one after the other every 1ms, high variance overall very low locally).
2. Change/mix sampling logic from re-compressing to pcodec statistical based approach which tries to estimate bits-saved-per-value.
3. Something we’ve been talking about for a long time and @nick discussed here (Heavier compression encodings (zstd, pco) #2984) with one of the original authors is different encodings and heuristics for different use cases. If you don’t care as much about filtering/random access, pcodec can be extremely useful. Our current compressor isn’t very externally configurable or extendable, and the way we use it (VortexLayoutStrategy) isn’t either.
   Having said that, pcodec might still prove useful for ints/floats without much changes around it? Not sure yet I understand it well enough to have a good sense as to what’s the minimal amount of work it has to do to access a single value, but if its low enough and it saves on enough IO it might prove as a net win in certain setups.
4. Very basic profiling of compressing data showed we basically spend half the time trying out dictionaries for numerical columns, which seems crazy expensive and will only rarely prove useful. I think we should either not even try it or just bail much earlier if the cardinality is too high (if we already have 100L unique value out of 100K values, its very unlikely that dict will be useful).
5. Another path we can go down is replacing the cardinality estimation with some sketch that can be faster than the current HashMap based one (there’s that one TUM paper we keep pointing at, but its a pretty big topic in CS basically infinite ways of doing that).