Add Kubernetes Secrets Backend to cncf.kubernetes provider

[piotrlinski]

Add a new secrets backend that reads Airflow connections, variables, and configuration from Kubernetes Secrets via the Kubernetes API.

This enables a native integration for Airflow deployments running on Kubernetes, allowing secrets to be managed through Kubernetes-native tooling or synced from external stores using External Secrets Operator (ESO).

Key design decisions:

• Delegates to KubernetesHook(conn_id=None, in_cluster=True) for client management, reusing its timeout handling, TCP keepalive, and properly configured API client rather than creating a raw CoreV1Api directly
• Uses conn_id=None to avoid circular dependency (the secrets backend resolves connections, so it cannot depend on one)
• Auto-detects namespace from pod service account metadata (/var/run/secrets/kubernetes.io/serviceaccount/namespace) since the hook's get_namespace() reads from connection extras which are not available here
• Secret names are built as {prefix}-{key} with automatic sanitization (underscores to hyphens, lowercase) to conform to Kubernetes DNS naming rules
• Each secret type (connections, variables, config) can be independently disabled by setting its prefix to None

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[jscheffl]

Thanks for adding. That makes sense. Not some static check fail because secret backends need to be registered into provider.yaml that described providers capabilties.

[SameerMesiah97]

Looks good overall. This is a solid addition that is well-implemented.

It appears that this backend is explicitly scoped to in-cluster usage and bypasses KubernetesHook, which is reasonable here, as that could introduce circular dependencies.

There are 2 non-blocking suggestions I have left if you want to improve it further. And I would resolve the CI failures. One looks like a missing key from get_provider_info (@jscheffl has pointed this out). The other seems unrelated to your change but I would check anyway.

[Nataneljpwd]

Looks good overall, I have left a few comments, I would appreciate it if you could answer them.

[piotrlinski]

thanks for the review and comments, however before I will address them, I would like to also get your opinion on the secrets discovery method I chose - the secret name mapping... which after more thinking is not a good ide. What do you think about the options (I would go for option 2)

---
# labels (option 1)
# backend_kwargs = {"connections_lable_type_name": "secret-type", "connections_lable_name_name": "secret-name", ...}
apiVersion: v1
kind: Secret
metadata:
  name: anything
  labels:
    airflow.apache.org/secret-type: connection
    airflow.apache.org/secret-name: <conn_name>
data:
  value: <conn_value>

---
# labels (options 2)
# backend_kwargs = {"connections_secret_lable_name": "connection-name", ... }
apiVersion: v1
kind: Secret
metadata:
  name: anything
  labels:
    airflow.apache.org/connection-name: <conn_name>
data:
  value: <conn_value>



---
# single secret to manage all connections (option 3)
# backend_kwargs = {"connections_secret_name": "airflow-connections", ... }
apiVersion: v1
kind: Secret
metadata:
  name: airflow-connections
data:
  <conn_name1>: <conn_value>
  <conn_name2>: <conn_value>
 

labels search could be equipped with same cacheing feature, if a secrets has been discovered on previous runs, it does not perform search (i knows that the secret/connection exists)

[Nataneljpwd]

thanks for the review and comments, however before I will address them, I would like to also get your opinion on the secrets discovery method I chose - the secret name mapping... which after more thinking is not a good ide. What do you think about the options (I would go for option 2)

---
# labels (option 1)
# backend_kwargs = {"connections_lable_type_name": "secret-type", "connections_lable_name_name": "secret-name", ...}
apiVersion: v1
kind: Secret
metadata:
  name: anything
  labels:
    airflow.apache.org/secret-type: connection
    airflow.apache.org/secret-name: <conn_name>
data:
  value: <conn_value>

---
# labels (options 2)
# backend_kwargs = {"connections_secret_lable_name": "connection-name", ... }
apiVersion: v1
kind: Secret
metadata:
  name: anything
  labels:
    airflow.apache.org/connection-name: <conn_name>
data:
  value: <conn_value>



---
# single secret to manage all connections (option 3)
# backend_kwargs = {"connections_secret_name": "airflow-connections", ... }
apiVersion: v1
kind: Secret
metadata:
  name: airflow-connections
data:
  <conn_name1>: <conn_value>
  <conn_name2>: <conn_value>
 

labels search could be equipped with same cacheing feature, if a secrets has been discovered on previous runs, it does not perform search (i knows that the secret/connection exists)

Option 1 seems reasonable, it allows to distinguish between connections and variables easily, while option 2 does not, so here I would prefer option 1, yet if you decide to make it with labels, you need to make sure that there are no duplicated labels before the name was the unique constraint enforced by k8s, now you will need to enforce uniqueness by yourself, in case you create, update or delete a secret.

About the caching, we can just do a k8s query with a resource version of 0 (which is very quick) yet we will be updated on when the connection or variable is deleted, and so caching on airflow side is probably not worth implementing, as we can rely on k8s api server cache.

[piotrlinski]

in opotion 2, i did not add it, but it would go like

• airflow.apache.org/connection-name
• airflow.apache.org/variable-name
• airflow.apache.org/config-name

as per maintainig uniquness on secrets/labels, this is not a backend role to maintain proper secrets deployment, secrets backend reads connections, if the connections are wrongly deployed to a namespace this is a totally different problem (this is how I see it )

[Nataneljpwd]

in opotion 2, i did not add it, but it would go like

• airflow.apache.org/connection-name
• airflow.apache.org/variable-name
• airflow.apache.org/config-name

as per maintainig uniquness on secrets/labels, this is not a backend role to maintain proper secrets deployment, secrets backend reads connections, if the connections are wrongly deployed to a namespace this is a totally different problem (this is how I see it )

So it would have a hardcoded label prefix plus a name for the connection/variable/config, alright, seems reasonable and probably a good Idea, yet I still don't understand what is a 'config'? is it an airflow configuration? as I do not know about any airflow object of type config other than the core airflow configuration, which from what I remember, cannot be pulled at runtime.

Option 1 still seems like the better option, as in my opinion it will look cleaner in code rather than using formatted strings in a bunch of places, but it is not a must, nor is it a blocker from my side, just personal preference.

Any of the options, 1 or 2 will suffice

[SameerMesiah97]

thanks for the review and comments, however before I will address them, I would like to also get your opinion on the secrets discovery method I chose - the secret name mapping... which after more thinking is not a good ide. What do you think about the options (I would go for option 2)

---
# labels (option 1)
# backend_kwargs = {"connections_lable_type_name": "secret-type", "connections_lable_name_name": "secret-name", ...}
apiVersion: v1
kind: Secret
metadata:
  name: anything
  labels:
    airflow.apache.org/secret-type: connection
    airflow.apache.org/secret-name: <conn_name>
data:
  value: <conn_value>

---
# labels (options 2)
# backend_kwargs = {"connections_secret_lable_name": "connection-name", ... }
apiVersion: v1
kind: Secret
metadata:
  name: anything
  labels:
    airflow.apache.org/connection-name: <conn_name>
data:
  value: <conn_value>



---
# single secret to manage all connections (option 3)
# backend_kwargs = {"connections_secret_name": "airflow-connections", ... }
apiVersion: v1
kind: Secret
metadata:
  name: airflow-connections
data:
  <conn_name1>: <conn_value>
  <conn_name2>: <conn_value>
 

labels search could be equipped with same cacheing feature, if a secrets has been discovered on previous runs, it does not perform search (i knows that the secret/connection exists)

I would go with option 2. If there are duplicates for the same field, there’s less ambiguity about them being genuine duplicates vs them being 2 separate entities that share the same field as in option 1. Please avoid option 3 as that expands the blast radius for secrets misconfiguration, subverts RBAC and makes secrets rotation harder. I can’t see any genuine argument in favour of it.

[SameerMesiah97]

in opotion 2, i did not add it, but it would go like

• airflow.apache.org/connection-name
• airflow.apache.org/variable-name
• airflow.apache.org/config-name

as per maintainig uniquness on secrets/labels, this is not a backend role to maintain proper secrets deployment, secrets backend reads connections, if the connections are wrongly deployed to a namespace this is a totally different problem (this is how I see it )

I agree with you. I think it should be the user’s responsibility to enforce secrets/labels uniqueness. But is it possible for your new secrets backend to detect duplicate secret configurations (by name) and log a warning?

[piotrlinski]

in opotion 2, i did not add it, but it would go like

• airflow.apache.org/connection-name
• airflow.apache.org/variable-name
• airflow.apache.org/config-name

as per maintainig uniquness on secrets/labels, this is not a backend role to maintain proper secrets deployment, secrets backend reads connections, if the connections are wrongly deployed to a namespace this is a totally different problem (this is how I see it )

So it would have a hardcoded label prefix plus a name for the connection/variable/config, alright, seems reasonable and probably a good Idea, yet I still don't understand what is a 'config'? is it an airflow configuration? as I do not know about any airflow object of type config other than the core airflow configuration, which from what I remember, cannot be pulled at runtime.

Option 1 still seems like the better option, as in my opinion it will look cleaner in code rather than using formatted strings in a bunch of places, but it is not a must, nor is it a blocker from my side, just personal preference.

Any of the options, 1 or 2 will suffice

secrets backend and config , what you could do:

apiVersion: v1
kind: Secret
metadata:
  name: secretname
  labels:
    airflow.apache.org/config-name: sql_alchemy_conn_value
data:
  value: postgresql://secretuser:secretpassword@postgres:5432)

AIRFLOW__DATABASE__SQL_ALCHEMY_CONN_SECRET=sql_alchemy_conn_value

the metadata database connection string with secrets (password and user) is pulled from secrets backend in runtime

[Nataneljpwd]

Looks great, one last minor change (and a test for the change) and I think that the pr is ready

[Nataneljpwd]

Looks great!
Thank you for the addition of the secrets backend!
I think it is a great QOL improvement for cluster administrators!

[jscheffl]

Nit in CI for spellcheck, if this is fixed then LGTM!

[jscheffl]

Nit in CI for spellcheck, if this is fixed then LGTM!

Ah and small nit in static checks, provider_info.py needs to be re-generated. Best would be to install prek and running pre-commit checks, then it is auto-fixed.

[Nataneljpwd]

Looks amazing, can't wait to try this!

Fun fact, from k8s 1.31 reads from cache can be consistent, I think this can be leveraged in a lot of places when airflow drops support for 1.30

[jscheffl]

Hope CI is getting green now, then LGTM!

[anleib]

@piotrlinski Fantastic work! super useful, been waiting for this!