Add Apache Arrow provider

[dabla]

Apache Arrow provider (apache-airflow-providers-apache-arrow)

Adds a new Airflow provider backed by Apache Arrow and the
Arrow Database Connectivity (ADBC) standard.

What's included

AdbcHook

A general-purpose DbApiHook subclass that connects to any ADBC-compatible database via the
adbc-driver-manager. It exposes the full Airflow SQL hook surface (get_records, run,
insert_rows, …) while internally operating on Arrow RecordBatch objects for zero-copy,
columnar data transfer.

Key capabilities:

• Arrow-native inserts — insert_rows converts rows to Arrow RecordBatch objects and
  uses the cursor's native bind API when available, falling back to executemany otherwise.
• Chunked bulk loading — rows are committed in configurable chunks (commit_every) to
  bound transaction size and lock duration.
• Dynamic driver discovery — the driver shared library is located automatically from the
  installed wheel; the entrypoint can also be overridden via connection extras.
• Dialect-aware — SQL placeholder style is normalised per-dialect before execution.
• Connection extras — driver, entrypoint, db_kwargs, conn_kwargs, and dialect
  are all configurable from the Airflow connection UI without code changes.

Supported databases (via optional extras)

Extra	Driver package
sqlite	adbc-driver-sqlite
postgresql	adbc-driver-postgresql
snowflake	adbc-driver-snowflake
bigquery	adbc-driver-bigquery
flightsql	adbc-driver-flightsql

Installation

# Core provider only
pip install apache-airflow-providers-apache-arrow

# With a specific driver
pip install "apache-airflow-providers-apache-arrow[postgresql]"

# With all bundled drivers
pip install "apache-airflow-providers-apache-arrow[all]"

Requirements

• apache-airflow >= 2.11.0
• apache-airflow-providers-common-sql >= 1.28.2
• adbc-driver-manager >= 1.7.0
• pyarrow >= 16.1.0 (Python < 3.13) / >= 18.0.0 (Python ≥ 3.13)
• more-itertools >= 9.0.0

Link to the discussion thread on the devlist: https://lists.apache.org/thread/v1nxnr5gocvzs12pknjxkw64mrtpg2o0

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[potiuk]

Just for anyone looking here - this is a draft for discussion between me, @dabla and @zeroshade - we will still need to start a DISCUSSION thread for the new provider - and we think Arrow and ADBC is a good addition. But we have to first discuss the approach :)

[vikramkoka]

As @potiuk mentioned, I believe this needs a devlist conversation first.
Looking forward to it

[potiuk]

As @potiuk mentioned, I believe this needs a devlist conversation first. Looking forward to it

Yep. This one is mostly to gather learnigs, get feedback from @zeroshade and see how we can turn it into a "convincing" devlist proposal - by showing some use cases and small POC of implementation and what it allows :).

We'll experiment a bit with it and gather our thoughts and see what can come out of it.

[dabla]

Just for anyone looking here - this is a draft for discussion between me, @dabla and @zeroshade - we will still need to start a DISCUSSION thread for the new provider - and we think Arrow and ADBC is a good addition. But we have to first discuss the approach :)

A discussion was already started on the devlist in beginning of march this year.