Providers: Make psycopg (v3) the default synchronous Postgres driver

[Dev-iL]

closes:

• pgvector operator's register_vector always assumes a psycopg2 connection #69443

related:

• Remove psycopg2 and migrate the sync Postgres driver to psycopg3 #68453
• Core: Make psycopg (v3) the default synchronous Postgres driver #69469

---

Makes psycopg (v3) the default synchronous Postgres driver across the providers, mirroring the async default switch (#69089). psycopg3 is used wherever it actually works (SQLAlchemy 2.0+); psycopg2-binary stays installed by default as a fallback so nothing regresses on the Airflow versions this provider still supports.

Why psycopg2 stays a default dependency (not demoted to an extra)

An earlier revision of this PR moved psycopg2-binary to an opt-in [psycopg2] extra. That regresses every core older than 3.4.0, so it now stays a default dependency — gated to become opt-in only once the minimum supported Airflow is 3.4.0. The reasons mirror #69690 (which kept asyncpg installed by default for the same class of reason on the async side):

• Core normalizes the sync metadata-DB conn to postgresql+psycopg2://. On every currently released core, configuration.py::_upgrade_postgres_metastore_conn rewrites a bare postgresql:// / legacy postgres:// / postgres+psycopg2:// sql_alchemy_conn to explicit postgresql+psycopg2://, and settings.py builds the metadata engine from that string with no psycopg fallback. Dropping psycopg2 breaks the metadata DB.
• Airflow 2.11 ships SQLAlchemy 1.4, which has no native postgresql+psycopg (v3) dialect at all. Since this provider still supports apache-airflow>=2.11.0, psycopg2 is the only viable sync driver there.

So psycopg[binary], psycopg2-binary, and asyncpg are all default dependencies. The removal of the psycopg2/asyncpg defaults is deferred, gated on the floor bump to 3.4.0, and noted at the workaround site (tracked at #68453).

Relationship to #69469 (the core half)

The sync switch for Airflow's own metadata DB lives in core #69469: it flips _upgrade_postgres_metastore_conn to derive postgresql+psycopg:// when psycopg (v3) is importable (gated to land in 3.4.0), mirroring the async _USE_PSYCOPG3 guard.

Because this PR keeps both sync drivers installed by default, it is robust to #69469's presence and to merge order: whether core normalizes the conn to +psycopg2:// (pre-#69469) or +psycopg:// (post-#69469), the driver behind that scheme is always present. The two PRs can merge in either order and release independently without a broken window. The eventual removal of the psycopg2 default is what's coupled to #69469 shipping in every supported core — deferred to the 3.4.0 floor bump.

What changed (6 commits, one per package)

1. providers/postgres — PostgresHook no longer requires psycopg2 at import time: the psycopg2-specific connection / cursor / execute_values imports are lazy/guarded and raise a clear AirflowOptionalProviderFeatureException if psycopg2 is genuinely needed but missing. A new [psycopg2] extra is added for the eventual opt-in. psycopg2-binary remains a default dependency for the backward-compat reasons above; a regression test asserts both psycopg2-binary and asyncpg stay default (non-extra) dependencies.
2. providers/celery — the [celery] result_backend derivation from a driverless sql_alchemy_conn now targets postgresql+psycopg:// (matching the new sync default) when psycopg (v3) is importable, falling back to postgresql+psycopg2:// otherwise — mirroring the same guard PostgresHook uses.
3. providers/google — BigQueryToPostgresOperator's psycopg2-specific register_adapter(list/dict, Json) now only runs when the resolved PostgresHook connection is actually on psycopg2 (a no-op on psycopg3), so the module imports cleanly without psycopg2 installed.
4. providers/pgvector — PgVectorIngestOperator._register_vector now calls the registration helper matching the connection's actual driver (pgvector.psycopg2.register_vector vs. pgvector.psycopg.register_vector) instead of always assuming psycopg2, fixing a pre-existing bug. Its "psycopg (v3) integration missing" error now points at pip install --upgrade pgvector.
5. providers/amazon — RedshiftSQLHook.get_sqlalchemy_engine() resolves an explicit driver (prefers psycopg on SQLAlchemy 2.0+, falls back to psycopg2, clear error if neither) instead of relying on SQLAlchemy's implicit psycopg2 default for a bare postgresql:// URI. get_uri()'s own bare-scheme contract is untouched. Also updates the Batch/ECS/Lambda executor docs' example connection strings to postgresql+psycopg://.
6. providers/common/sql — DbApiHook.get_sqlalchemy_engine() retries with an explicit driver only for the specific "bare postgresql scheme, driver import failed" case; any other scheme's failure still propagates unchanged, keeping the class DB-agnostic. The retry prefers psycopg only on SQLAlchemy 2.0+ (which has the native postgresql+psycopg dialect) and falls back to psycopg2 otherwise, so it stays correct on Airflow 2.11's SQLAlchemy 1.4. This is the general
   safety net: any current or future DbApiHook subclass with a postgres-compatible conn_type that doesn't override sqlalchemy_url is covered by this fix, not just the two concrete cases (amazon, google) found so far.

Test plan

• New/updated tests per commit cover: the lazy-psycopg2 fallback itself; both USE_PSYCOPG3 branches in celery/google/pgvector; PgVectorIngestOperator registering against both a psycopg2-backed and a psycopg3-backed connection; and get_sqlalchemy_engine's driver-resolution / retry / re-raise / no-retry-for-other-schemes behavior in both amazon and common.sql — including the new SQLAlchemy-1.4 case that must fall back to psycopg2 rather than psycopg (v3).
• A packaging regression test asserts psycopg2-binary and asyncpg remain default (non-extra) dependencies of apache-airflow-providers-postgres, so a future change can't silently demote either driver back to an extra and reintroduce the <3.4.0 regression.

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[shahar1]

Blocking to avoid accidental merges until we figure out this one:
#69526 (comment)

[github-actions]

uv.lock on main just moved via #69694 ("[main] Upgrade important CI environment"), commit c8d313a and this PR currently conflicts.

Quickest fix:

git fetch upstream main && git rebase upstream/main
rm uv.lock && uv lock
git add uv.lock && git rebase --continue
git push --force-with-lease

Automated nudge — ignore if you're not ready to rebase. This comment is updated in place on future uv.lock bumps.

.