Add Apache Airflow Provider Registry by kaxil · Pull Request #62261 · apache/airflow

kaxil · 2026-02-20T22:14:23Z

Devlist Discussion: https://lists.apache.org/thread/7n4pklzcc4lxtxsy9g69ssffg9qbdyvb

A static-site provider registry for discovering and browsing Airflow providers and their modules. Deployed at airflow.apache.org/registry/ alongside the existing docs infrastructure (S3 + CloudFront).

Staging preview: https://airflow.staged.apache.org/registry/

Acknowledgments

Many of you know the Astronomer Registry, which has been the go-to for discovering providers for years. Big thanks to Astronomer and @josh-fell for building and maintaining it. This new registry is designed to be a community-owned successor on airflow.apache.org, with the eventual goal of redirecting registry.astronomer.io traffic here once it's stable. Thanks also to @ashb for suggesting and prototyping the Eleventy-based approach.

What it does

The registry indexes all 99 official providers and 840 modules (operators, hooks, sensors, triggers, transfers, bundles, notifiers, secrets backends, log handlers, executors) from the existing
providers/*/provider.yaml files and source code in this repo. No external data sources beyond PyPI download stats.

Pages:

Homepage — search bar (Cmd+K), stats counters, featured and new providers
Providers listing — filterable by lifecycle stage (stable/incubation/deprecated), category, and sort order (downloads, name, recently updated)
Provider detail — module counts by type, install command with extras/version selection, dependency info, connection builder, and a tabbed module browser with category sidebar and per-module search
Explore by Category — providers grouped into Cloud, Databases, Data Warehouses, Messaging, AI/ML, Data Processing, etc.
Statistics — module type distribution, lifecycle breakdown, top providers by downloads and module count
JSON API — /api/providers.json, /api/modules.json, per-provider endpoints for modules, parameters, and connections

Connection Builder — pick a connection type (e.g. aws, redshift), fill in the form fields with placeholders and sensitivity markers, and export as URI, JSON, or environment variable format. Fields are
extracted from provider.yaml connection metadata.

Screenshots

Homepage

Light	Dark

Providers List

Light	Dark

Provider Detail (Amazon)

Light	Dark

Module Browser

Light	Dark

Connection Builder

Light	Dark

Explore by Category

Light	Dark

Statistics

Light	Dark

Motivation

With AIP-95 approved, Airflow now has a formal provider lifecycle (incubation, production, mature, deprecated). That opens the door for accepting more community-built providers and giving them an official home, while setting clear expectations about maturity and support. But lifecycle stages only work if users can actually see them. Right now there's no place on airflow.apache.org where someone can browse providers, check their lifecycle stage, or discover what modules they ship.

This registry fills that gap:

Surface governance visibly — AIP-95 lifecycle stages are first-class citizens in the discovery experience (badges, filters, explore by stage)
Stay in sync automatically — generates directly from provider.yaml files in the repo, no separate data pipeline or manual curation
Community-owned — an official Apache project resource on airflow.apache.org

Architecture

provider.yaml + source code (providers/*/)
        │
        ▼
extract_metadata.py     ← AST-parses Python files, fetches PyPI stats
        │
        ▼
registry/src/_data/
  ├── providers.json    ← 99 providers with metadata, quality scores
  ├── modules.json      ← 840 modules with import paths, docstrings
  └── search-index.json ← Pagefind custom records
        │
        ▼
Eleventy build          ← Generates 2,740 static HTML pages
        │
        ▼
Pagefind postbuild      ← Builds search index from custom records
        │
        ▼
S3 sync + CloudFront    ← registry-build.yml workflow

Four Python extraction scripts run at build time:

Script	What it does	Runs in
`extract_metadata.py`	Parses provider.yaml, AST-parses source for class names/docstrings, fetches PyPI stats and release dates	CI (host Python)
`extract_versions.py`	Reads older provider versions from git tags	CI (host Python)
`extract_parameters.py`	Inspects constructor signatures via runtime import	Breeze (needs provider packages installed)
`extract_connections.py`	Extracts connection form fields from provider.yaml + hook classes	Breeze (needs provider packages installed)

The site itself is vanilla HTML/CSS/JS built with Eleventy — no React, no bundler. Search uses Pagefind (client-side, loads lazily on first search interaction). Fonts are self-hosted (Plus Jakarta Sans, JetBrains
Mono).

Design decisions worth calling out

Why AST parsing instead of runtime import? extract_metadata.py runs on the CI host without installing 100+ provider packages. It reads .py files and extracts class names, base classes, and docstrings from
the AST. This means it works with just pyyaml as a dependency. The trade-off: it can't resolve dynamic class definitions or runtime-computed attributes. For the 99 providers currently in the repo, AST parsing
captures everything.

Why four separate scripts? extract_parameters.py and extract_connections.py need runtime access to provider classes (to inspect __init__ signatures and call get_connection_form_widgets()). They run
inside Breeze where all providers are installed. extract_metadata.py and extract_versions.py only need filesystem access and run on the host. Keeping them separate means the CI workflow can run the fast
scripts (metadata) without spinning up Breeze, while parameter/connection extraction is a separate optional step.

Why Eleventy? Static site generators produce zero-JS pages by default. The registry works without JavaScript — filtering and search are layered on top progressively. Eleventy also has no opinion on frontend
frameworks, which keeps the dependency surface small (the lockfile has ~30 packages total).

Path prefix handling: The site deploys at /registry/ on airflow.apache.org but runs at / during local dev. Eleventy's pathPrefix config handles this via the REGISTRY_PATH_PREFIX env var. Templates use
the | url filter, and client-side JS reads window.__REGISTRY_BASE__ (injected in base.njk).

Module filtering: The extraction script filters classes based on type-specific suffix patterns (e.g. Operator, Hook, Sensor suffixes for their respective types) and base class inheritance. This avoids
indexing helper classes, dataclasses, and exceptions that happen to live in operator/hook modules.

What's NOT changed

No modifications to any provider package code
No changes to provider.yaml schema
No changes to core Airflow code
No new Python dependencies for Airflow itself
No changes to existing documentation build process

What's NOT included (future work)

apache/airflow-site PR for .htaccess rewrite and nav link
apache/airflow-site-archive — update s3-to-github.yml and github-to-s3.yml workflows to sync the registry/ S3 prefix (same pattern as docs/)
Third-party provider support (Cosmos, Great Expectations, etc.) — will use .airflow-registry.yaml files in provider repos. (Example Integration)
LLM-friendly exports (llms.txt) and "Copy for AI" buttons
Redirect registry.astronomer.io traffic once the official registry is stable
Explicit categories in provider.yaml (replacing keyword matching)
Version changelog/diff on provider detail pages
Example DAGs for Providers
Integration with pre-commit/prek checks

How to test locally

# 1. Extract metadata
uv run python dev/registry/extract_metadata.py

# 2. Install Node dependencies
cd registry && pnpm install

# 3. Start dev server at http://localhost:8080
pnpm dev

Copilot

Pull request overview

This PR adds a comprehensive Apache Airflow Provider Registry - a static-site catalog for discovering and browsing all 99 official Airflow providers and 840+ modules (operators, hooks, sensors, triggers, transfers, and more). The registry is built with Eleventy (static site generator), uses Pagefind for client-side search, and will be deployed to airflow.apache.org/registry/.

Changes:

Data extraction pipeline using AST parsing and PyPI API to generate provider/module metadata JSON
Static site built with Eleventy (vanilla HTML/CSS/JS, no React) with search, filtering, and interactive connection builder
CI/CD workflows for automated registry builds and S3 deployment with breeze integration

Reviewed changes

Copilot reviewed 64 out of 168 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
`registry/src/js/*.js`	Client-side JavaScript for search modal, provider filtering, connection builder, theme toggle
`registry/src/*.njk`	Nunjucks templates for homepage, provider listing, statistics, explore pages
`registry/src/_data/*.js`	Eleventy data files for stats computation, category mappings, version management
`registry/.eleventy.js`	Eleventy configuration with custom filters for number formatting
`.github/workflows/registry-*.yml`	CI workflows for testing and building/publishing the registry
`dev/breeze/src/airflow_breeze/commands/registry_commands.py`	Breeze CLI command for registry data extraction
`dev/registry/tests/`	Test infrastructure for extraction scripts
`registry/README.md`	Comprehensive documentation

Files not reviewed (1)

registry/pnpm-lock.yaml: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

jscheffl

Looks already very good.

I assume the build must be made after a providers release? Can you add it to the descriptions of the providers release docs in dev/README_RELEASE_PROVIDERS.md so that the update is included when release manager upgrades the site? (Or are all update steps in the docs pipeline? Then the release manager needs to check published providers are updated at least)

kaxil · 2026-02-21T23:16:13Z

Thanks for the review @jscheffl , appreciate it since this is too big

I assume the build must be made after a providers release? Can you add it to the descriptions of the providers release docs in dev/README_RELEASE_PROVIDERS.md so that the update is included when release manager upgrades the site? (Or are all update steps in the docs pipeline? Then the release manager needs to check published providers are updated at least)

I had added the following in the PR:

airflow/dev/README_RELEASE_PROVIDERS.md

Lines 1331 to 1337 in 20ef65c

    
           > [!NOTE] 
        
           > The **Provider Registry** at `airflow.apache.org/registry/` is rebuilt automatically as part of the 
        
           > `publish-docs-to-s3.yml` workflow (it calls `registry-build.yml` as a post-publish job). The registry 
        
           > extracts metadata from `provider.yaml` files and PyPI, so it picks up new/updated providers without 
        
           > manual intervention. If you need to rebuild the registry independently, trigger the `registry-build.yml` 
        
           > workflow manually. See [`registry/README.md`](../registry/README.md) for details.

Does that look ok or did you have something else in mind?

gopidesupavan

LGTM, overall its really great. not an expert on frontend UI stuff :)

Do we have to do anything on airflow-archive repo to sync? today we keep a record of complete site in archive repo . may be this also falls in to same?

…lback The "View latest" banner on older version pages was statically rendered at build time, so it didn't reflect the actual latest version deployed in S3. Now the banner (link + version label) updates dynamically from versions.json, matching the dropdown behavior. Also fix publish_registry_versions.py to fall back to the highest deployed version when the declared latest from providers.json isn't actually deployed in S3 yet. Fix code review findings: CallerReference collision, sort order, XSS, URI encoding - Add timestamp to CloudFront CallerReference to prevent collision on re-runs - Fix sort_versions_desc: invalid versions now sort last (not first) in descending order, preventing malformed S3 dirs from becoming "latest" - Build JSON-LD breadcrumb via dump|safe to prevent XSS via provider names - Escape single quotes in generateEnvVar shell output - URI-encode host (with IPv6 bracket wrapping) and port in connection URI builder - Fix "doscs" typo in publish-docs-to-s3.yml

When publish-docs-to-s3 publishes provider docs, it now triggers registry-build with just that provider ID. The registry workflow downloads existing data from S3, extracts only the updated provider, merges incremental results into the full dataset, and rebuilds the site. This reduces extraction time from ~12 minutes (99 providers) to ~30 seconds (1 provider). Full builds (no --provider flag) remain unchanged.

The extraction script was generating non-existent class names when AST parsing couldn't find matching classes in a module. For example, common-sql produced "SqlOperator" (doesn't exist) instead of "SQLExecuteQueryOperator" (the real class). Root causes fixed: - No transitive inheritance resolution: classes inheriting from intermediate bases (e.g., SQLExecuteQueryOperator → BaseSQLOperator → BaseOperator) were invisible - No handling of generic subscript syntax: AwsBaseOperator[S3Hook] was not parsed (ast.Subscript node) - Type-suffix pattern filter rejected valid classes like GenericTransfer - Fallback fabricated phantom class names from module names Changes: - Add build_global_inheritance_map() to scan all provider src/ dirs and build a cross-file class-to-bases map (~2300 classes indexed) - Add _extract_base_class_name() to handle ast.Name, ast.Attribute, and ast.Subscript nodes - Rewrite extract_classes_from_python_file() with transitive inheritance resolution using both local and global maps - Remove phantom class fallbacks and type-suffix pattern filter - Use get_module_type_base_classes() (broader set) instead of the restricted inline base_class_map Impact: Amazon goes from ~49 phantom entries to 391 real modules; Google from ~250 to 753; common-sql phantoms eliminated entirely.

- Add 6 unit tests covering directory scanning, subscript bases, missing/broken files, first-definition-wins dedup, and multiple bases - Cache get_module_type_base_classes() per extract_modules_from_yaml call instead of re-creating the dict per module - Clarify first-definition-wins comment scope (cross-provider)

Address review feedback to stop maintaining a parallel logo inventory in registry/public/logos/. Logos now live in their canonical location under providers/*/docs/integration-logos/ and are copied to the registry build output at CI time. Changes: - extract_metadata.py: write logos to dev/registry/logos/ (mounted in breeze) instead of registry/public/logos/; fix stale docstring that claimed objects.inv as a data source - registry-build.yml: copy logos from dev/registry/logos/ to registry/public/logos/ before the 11ty build - Move 15 logos that only existed in registry/ into their provider directories; add logo entries to their provider.yaml integrations - Add registry/public/logos/ and dev/registry/logos/ to .gitignore - Remove 87 checked-in logo files from registry/public/logos/

Replace manual provider.yaml discovery and hook class importing with ProvidersManager, which already handles both YAML-declared connection metadata and Python hook fallback. Follows the same WTForms mocking pattern used by the UI connections service (HookMetaService). This reduces the script from 414 to 312 lines by removing duplicated discovery logic (discover_provider_yamls, import_hook_class, extract_conn_fields_from_widgets, extract_ui_behaviour_from_hook, extract_connection_type) and delegating to ProvidersManager's existing implementation. Add unit tests for the serialization helpers (build_standard_fields, build_custom_fields, package_name_to_provider_id).

Manually constructing documentation URLs from module paths is fragile — if Sphinx reorganizes its output layout, links break. Inventory files are published for all released providers on S3 and map every documented symbol to its actual URL. Add read_inventory() and fetch_provider_inventory() to parse and download objects.inv files with 12-hour TTL caching. All module types (operators, hooks, sensors, triggers, transfers, bundles, notifiers, secrets, logging, executors, decorators) now resolve docs URLs through inventory lookup with fallback to the previous manual construction for unpublished providers. Inventory fetching runs in parallel (ThreadPoolExecutor, 8 workers) before the main extraction loop. Corrupted cache files are handled gracefully (try/except around read_inventory), and stale cache is served on refetch failure rather than discarding it. Also documents design decisions in AGENTS.md and README.md: why AST parsing over runtime import, why four separate scripts, module filtering via base class inheritance (not suffix matching), inventory approach, Eleventy choice, and path prefix handling.

…alidation provider.yaml correctness is already guaranteed by run_provider_yaml_files_check.py (check-provider-yaml-valid pre-commit hook). Document how extract_metadata.py builds on that: provider.yaml lists operators/hooks/sensors at the module level, but the registry needs individual class names within each module — that's the gap AST parsing fills. For class-level entries (notifications, secrets, etc.), provider.yaml already has full class paths and no discovery is needed.

extract_metadata.py discovers classes via AST parsing on the host, while extract_parameters.py imports them at runtime in breeze. These two systems can disagree silently: AST may find classes that can't be imported, miss classes, or get the wrong type. Add a --discover flag that imports provider modules directly from provider.yaml, compares results against the AST-produced modules.json, and reports discrepancies (phantoms, misses, type mismatches). When --discover is not passed, existing behavior is unchanged. This is Phase 1 (validate). Once discrepancies are understood, Phase 2 will have extract_parameters.py produce the full modules.json and remove AST parsing from extract_metadata.py.

extract_parameters.py now produces modules.json (the full module catalog) via runtime inspection inside breeze, using issubclass() checks and inspect.getmembers() instead of AST parsing. Each entry includes all 11 fields: id, name, type, import_path, module_path, short_description, docs_url (from Sphinx inventory), source_url (with line numbers), category, provider_id, and provider_name. extract_metadata.py is simplified to only produce providers.json — all AST parsing functions (extract_classes_from_python_file, build_global_inheritance_map, extract_modules_from_yaml, etc.) and the Module dataclass are removed. Also: - Replace copy-paste module-type blocks in count_modules_by_type() and extract_modules_from_yaml() with table-driven loops - Replace regex TOML parsing in extract_versions.py with tomllib - Fix naive datetime in inventory cache TTL check

Only include provider versions in static HTML when their version data has actually been built, so local backfills work without exposing broken links in CI. Also let extract_versions.py process multiple requested providers concurrently so manual staging rebuilds finish faster.

Handle callable task decorator symbols via inspect.signature so parameters.json includes decorator metadata instead of failing on __mro__ during extraction.

Show the homepage provider spotlight based on monthly PyPI downloads so the section matches its label instead of rendering the first four providers alphabetically.

kaxil · 2026-03-07T04:03:24Z

In the interest of ensuring reviews become manageable I merged the PR. If there are any issues in CI or around doc publishing, please let me know.

Nothing would show up in the "live" (non-staging site) until I add .htaccess. I plan to work on incremental updates to handle some of the todos and review comments as separate PRs next week with a goal to get the Airflow Registry live around the same timing as Airflow 3.2 or during the week of 16th March.

gopidesupavan · 2026-03-11T03:20:10Z

#protm

@josh-fell

Devlist Discussion: https://lists.apache.org/thread/7n4pklzcc4lxtxsy9g69ssffg9qbdyvb A static-site provider registry for discovering and browsing Airflow providers and their modules. Deployed at `airflow.apache.org/registry/` alongside the existing docs infrastructure (S3 + CloudFront). Staging preview: https://airflow.staged.apache.org/registry/ ## Acknowledgments Many of you know the [Astronomer Registry](https://registry.astronomer.io), which has been the go-to for discovering providers for years. Big thanks to **Astronomer** and @josh-fell for building and maintaining it. This new registry is designed to be a community-owned successor on `airflow.apache.org`, with the eventual goal of redirecting `registry.astronomer.io` traffic here once it's stable. Thanks also to @ashb for suggesting and prototyping the Eleventy-based approach. ## What it does The registry indexes all 99 official providers and 840 modules (operators, hooks, sensors, triggers, transfers, bundles, notifiers, secrets backends, log handlers, executors) from the existing `providers/*/provider.yaml` files and source code in this repo. No external data sources beyond PyPI download stats. **Pages:** - **Homepage** — search bar (Cmd+K), stats counters, featured and new providers - **Providers listing** — filterable by lifecycle stage (stable/incubation/deprecated), category, and sort order (downloads, name, recently updated) - **Provider detail** — module counts by type, install command with extras/version selection, dependency info, connection builder, and a tabbed module browser with category sidebar and per-module search - **Explore by Category** — providers grouped into Cloud, Databases, Data Warehouses, Messaging, AI/ML, Data Processing, etc. - **Statistics** — module type distribution, lifecycle breakdown, top providers by downloads and module count - **JSON API** — `/api/providers.json`, `/api/modules.json`, per-provider endpoints for modules, parameters, and connections **Connection Builder** — pick a connection type (e.g. `aws`, `redshift`), fill in the form fields with placeholders and sensitivity markers, and export as URI, JSON, or environment variable format. Fields are extracted from provider.yaml connection metadata.

kaxil requested review from amoghrajesh, ashb, bugraoz93, ephraimbuddy, gopidesupavan, jason810496, jedcunningham, jscheffl and potiuk as code owners February 20, 2026 22:14

boring-cyborg Bot added area:dev-tools backport-to-v3-1-test labels Feb 20, 2026

github-advanced-security AI found potential problems Feb 20, 2026

View reviewed changes

Comment thread registry/src/js/provider-detail.js Fixed

kaxil requested a review from Copilot February 20, 2026 22:17

Copilot started reviewing on behalf of kaxil February 20, 2026 22:18 View session

Copilot AI reviewed Feb 20, 2026

View reviewed changes

kaxil force-pushed the registry/provider-registry branch from 7b197fc to 3c03326 Compare February 20, 2026 22:26

kaxil removed the backport-to-v3-1-test label Feb 20, 2026

kaxil force-pushed the registry/provider-registry branch 4 times, most recently from 535d259 to b435be6 Compare February 21, 2026 03:50