[Python] Implement pickling for Column, ChunkedArray, RecordBatch, Table

[asfimport]

At the moment the types pyarrow.Column/ChunkedArray/RecordBatch/Table cannot be pickled. Although it may not be the fastest way to transport them from one process to another, it is a very comfortable one. We should implement a __reduce__() for all of them.

Reporter: Wes McKinney / @wesm
Assignee: Antoine Pitrou / @pitrou

Related issues:

• [Python] Add support for pickling to buffers and arrays (incorporates)

PRs and other links:

• GitHub Pull Request #2292

_{Note: This issue was originally created as ARROW-1715. Please see the migration documentation for further details.}

[asfimport]

Uwe Korn / @xhochy:
@wesm Do you have a better idea here on how to do simple serialization instead of wrapping the Array and Column into small RecordBatches and then using the IPC code?

[asfimport]

Wes McKinney / @wesm:
Not really, this will introduce a little bit of overhead but I'm not sure what are the best alternatives – we could support "Pythonizing" the internal ArrayData object and serializing that

[asfimport]

Antoine Pitrou / @pitrou:
It seems pickling arrays can simply reuse Array.buffers() and Array.from_buffers() (once ARROW-2281 is resolved). That will involve copying data but that's unavoidable with pickle.

Is there any advantage in using the IPC code instead?

[asfimport]

Uwe Korn / @xhochy:
No, the advantage of the IPC code was that it is already there. Using buffers() and from_buffers() is definitely the better approach as this will allow pickling with less copies of the data.

[asfimport]

Wes McKinney / @wesm:
It'd be worth comparing a naive pickle version with a pickle version using pyarrow.serialize(...).to_components(). The component-based serialization minimizes copying. I'm due to write a blog post about it, as the difference with large objects can be quite significant

[asfimport]

Antoine Pitrou / @pitrou:
pa.serialize() currently doesn't seem to support Arrow arrays or other arrow types.

[asfimport]

Wes McKinney / @wesm:
Right, what I'm saying is that we could unpack the Array into its buffer + metadata representation (to use from_buffers on the way back from pickle). I could take a shot at this, maybe next week. I'm not sure which strategy will be more efficient without some closer analysis.

[asfimport]

Antoine Pitrou / @pitrou:
I think it will be down to differences in metadata serialization overhead, because in both cases you end up copying the buffer contents into the pickle stream (and actually there's a second copy before that, as pickle will only allow serializing a bytes object, not a zero-copy buffer).

[asfimport]

Antoine Pitrou / @pitrou:
Related: ARROW-2660 (zero-copy pickling).

[asfimport]

Dave Hirschfeld / @dhirschfeld:
This has come up in the context of dask.distributed also:
dask/distributed#2103

[asfimport]

Uwe Korn / @xhochy:
Issue resolved by pull request 2292
#2292