Unified arrow API

[evertlammerts]

Problem

The Arrow API of the Python client regularly causes confusion. The most important issues seem to be that:

• The Relational API has a total of six functions, four of which are aliases, three of which are unique to the Relational API
• The naming of the two core functions is not as intuitive as it could be.

Also see #97

Core Functions

The Connection API (and with it, the duckdb module) and the Relational API have two core functions to create Arrow objects:

• fetch_record_batch() -> pyarrow.lib.RecordBatchReader
• fetch_arrow_table() -> pyarrow.lib.Table

The Connection API has another function to create a Relation from an Arrow object:

• arrow(arrow_object, connection = None) -> DuckDBPyRelation

Aliases

The Connection and Relational APIs both have an alias for fetch_record_batch():

• arrow() -> pyarrow.lib.RecordBatchReader This function was the first we exposed in the API and is probably most often used. It changed return type over the course of 1.4.X, from Table to RecordBatchReader, which caused a number of issues.

The Relational API has three more aliases:

• to_arrow_table() -> pyarrow.lib.Table
• fetch_arrow_reader() -> pyarrow.lib.RecordBatchReader
• record_batch() -> pyarrow.lib.RecordBatchReader (this has been deprecated for a while)

Changes

v1.5.0 API

The Connection and Relational APIs will have the following functions:

• to_arrow_reader() -> pyarrow.lib.RecordBatchReader
• to_arrow_table() -> pyarrow.lib.Table
• arrow() -> pyarrow.lib.RecordBatchReader Note: we will not deprecate this function in v1.5.0, but we will discourage its use in both the documentation and the docstring. We encourage users to use to_arrow_reader() instead.

The Connection API will keep this function:

• arrow(arrow_object, connection = None) -> DuckDBPyRelation

v1.5.0 Deprecated API

The fetch_* functions will be deprecated in v1.5.0 (which will be emitted as a DeprecationWarning) and removed in v1.6.0:

• fetch_record_batch() -> pyarrow.lib.RecordBatchReader
• fetch_arrow_table() -> pyarrow.lib.Table
• fetch_arrow_reader() -> pyarrow.lib.RecordBatchReader

v1.5.0 Removed API

• Relation::record_batch() -> pyarrow.lib.RecordBatchReader will be removed.

What's in a Name

Arrow's ADBC Driver Manager API uses the fetch_* naming convention (docs):

• fetch_arrow_table()
• fetch_record_reader()
• fetch_df()

This is what we've adopted, in spite of our own (not consistently applied) convention of using to_*:

• to_csv
• to_df
• to_parquet
• to_table
• to_view

We however also provide:

• fetch_df
• fetch_df_chunk

Looking at other libraries (Vortex, Pandas, etc) there is precedent to move from the fetch_* prefix to the to_* prefix, which seems the preferred way of expressing a conversion.

[NickCrews]

This looks great. My only question is, why is Relation::record_batch() the only one to get removed, why is it not merely deprecated like the rest of the deprecations? Without knowing your reasoning I would weakly prefer to also deprecate it, for consistency and just to ease the migration path of any users.

[evertlammerts]

This looks great. My only question is, why is Relation::record_batch() the only one to get removed, why is it not merely deprecated like the rest of the deprecations? Without knowing your reasoning I would weakly prefer to also deprecate it, for consistency and just to ease the migration path of any users.

@NickCrews Relation::record_batch() was already deprecated since 1.4.0

[NickCrews]

Oh, excellent, then that is a great plan. Thanks for your work here!