Context
apache-airflow-providers-cncf-kubernetes 10.18.0 widened the kubernetes client bound to allow 36.x (#68041). Client 36.x renamed the bearer-token auth key, which breaks the google (GKE) provider's synchronous credentials path with a 401 Unauthorized. The bound is capped back to <36.0.0 (35.x, last known-good) as a workaround in #69025 until the provider auth paths support 36.x.
Root cause (verified against kubernetes-client/python 36.x)
kubernetes-client/python 36.0.0 changed the generated Configuration auth handling (kubernetes-client/python#2582): the bearer scheme key was renamed from authorization to BearerToken. In 36.x:
def auth_settings(self):
auth = {}
if 'BearerToken' in self.api_key or 'authorization' in self.api_key:
auth['BearerToken'] = {
'type': 'api_key', 'in': 'header', 'key': 'authorization',
'value': self.get_api_key_with_prefix('BearerToken', alias='authorization'),
}
return auth
def get_api_key_with_prefix(self, identifier, alias=None):
if self.refresh_api_key_hook is not None:
self.refresh_api_key_hook(self)
key = self.api_key.get(identifier, self.api_key.get(alias) if alias is not None else None)
if key:
prefix = self.api_key_prefix.get(identifier) # looked up by 'BearerToken', NOT the alias
...36.0.1 (kubernetes-client/python#2585) fixed this only for load_incluster_config() / load_kube_config() (sync + async) by writing the token under the new BearerToken key. It did not fix code that hand-builds a Configuration.
The google GKE sync hook hand-builds the config and registers the bearer token/prefix under the old authorization key (GKEKubernetesHook.get_conn / _get_config, providers/google/src/airflow/providers/google/cloud/hooks/kubernetes_engine.py:75-98):
client.Configuration(
api_key_prefix={"authorization": "Bearer"},
api_key={"authorization": token},
)
configuration.refresh_api_key_hook = self._refresh_api_key_hook # also sets api_key["authorization"]On 36.x, get_api_key_with_prefix('BearerToken', alias='authorization') finds the token via the authorization alias, but looks up the prefix by the primary identifier BearerToken — which is absent — so the Bearer prefix is dropped. The request goes out as Authorization: <raw token> instead of Authorization: Bearer <token>, and the API server rejects it with 401. On 35.x the scheme used identifier authorization for both the key and the prefix, so it worked. This is why downgrading the provider to 10.17.1 (client 35.x) fixes it, and why the failure is immediate (not token-rotation related).
The async GKE path is unaffected — it sets the header directly via ApiClient(header_name=..., header_value="Bearer <token>"), bypassing auth_settings(). cncf.kubernetes's own hooks go through load_incluster_config/load_kube_config, which 36.0.1 already fixed.
The same 36.x Configuration change is also behind the in-cluster pod_override PicklingError (#68827), partially addressed by #68848.
Follow-up work to lift the cap
Acceptance criteria
References
Drafted-by: Claude Code (Opus 4.8) (no human review before posting)
Context
apache-airflow-providers-cncf-kubernetes10.18.0 widened the kubernetes client bound to allow 36.x (#68041). Client 36.x renamed the bearer-token auth key, which breaks the google (GKE) provider's synchronous credentials path with a 401 Unauthorized. The bound is capped back to<36.0.0(35.x, last known-good) as a workaround in #69025 until the provider auth paths support 36.x.Root cause (verified against kubernetes-client/python 36.x)
kubernetes-client/python 36.0.0 changed the generated
Configurationauth handling (kubernetes-client/python#2582): the bearer scheme key was renamed fromauthorizationtoBearerToken. In 36.x:36.0.1 (kubernetes-client/python#2585) fixed this only for
load_incluster_config()/load_kube_config()(sync + async) by writing the token under the newBearerTokenkey. It did not fix code that hand-builds aConfiguration.The google GKE sync hook hand-builds the config and registers the bearer token/prefix under the old
authorizationkey (GKEKubernetesHook.get_conn/_get_config,providers/google/src/airflow/providers/google/cloud/hooks/kubernetes_engine.py:75-98):On 36.x,
get_api_key_with_prefix('BearerToken', alias='authorization')finds the token via theauthorizationalias, but looks up the prefix by the primary identifierBearerToken— which is absent — so theBearerprefix is dropped. The request goes out asAuthorization: <raw token>instead ofAuthorization: Bearer <token>, and the API server rejects it with 401. On 35.x the scheme used identifierauthorizationfor both the key and the prefix, so it worked. This is why downgrading the provider to 10.17.1 (client 35.x) fixes it, and why the failure is immediate (not token-rotation related).The async GKE path is unaffected — it sets the header directly via
ApiClient(header_name=..., header_value="Bearer <token>"), bypassingauth_settings(). cncf.kubernetes's own hooks go throughload_incluster_config/load_kube_config, which 36.0.1 already fixed.The same 36.x
Configurationchange is also behind the in-clusterpod_overridePicklingError(#68827), partially addressed by #68848.Follow-up work to lift the cap
BearerToken(keepingauthorizationfor client-35.x back-compat), or set theAuthorizationheader directly on theApiClientas the async path already does (kubernetes_engine.py:75-98).Configurationwithapi_key={'authorization': ...}and apply the same fix.Authorization: Bearer <token>on client 36.x (it must fail on the pre-fix code).Acceptance criteria
Authorization: Bearer …header.References
Configuration.auth_settings()returns empty dict in v36.0.0 - all BearerToken-authed requests go anonymous kubernetes-client/python#2582 (auth_settings BearerToken key rename / 401), config: write api_key['BearerToken'] so v36+ SDK auth works kubernetes-client/python#2585 (36.0.1 loader-only fix)kubernetesandkubernetes_asyncioclient 36.x in cncf.kubernetes #68041 (the widening), KubernetesExecutor scheduler crashes with PicklingError on pod_override with kubernetes client 36.x #68827 + Make cncf.kubernetes model deserialization picklable in-cluster #68848 (pickling), Cap kubernetes client to <36 to fix cluster auth 401 regression #69025 (the cap), 10.17.1 (last known-good)Drafted-by: Claude Code (Opus 4.8) (no human review before posting)