[v3-2-test] KPO: treat registry 5xx errors as transient during pod startup (#65490)#66170
Merged
Merged
Conversation
…e#65490) When a pod is starting, kubelet automatically retries failed image pulls with exponential backoff. `KubernetesPodOperator` monitors the pod via `detect_pod_terminate_early_issues()` and aborts early on what it deems a fatal pull error — but the transient-error pattern list did not cover registry 5xx / gateway outages. A short Docker Hub outage (502/503/504 from `auth.docker.io` or the registry) was being classified as fatal and would abort the pod before kubelet's next retry, even though the pull would have succeeded once upstream recovered. Add `bad gateway`, `service unavailable`, `gateway timeout` to `TRANSIENT_ERROR_PATTERNS` so these conditions let kubelet keep retrying within the caller's `startup_timeout` budget. On kubelet >= 1.32 the `ImagePullBackOff` message is `Back-off pulling image "X": <prev_error>`, so these patterns also match during the backoff state. On older kubelets the bare message carries no detail and we keep the existing fail-fast behaviour — matching `back-off pulling` unconditionally would have caused a 120s wait for a genuinely missing image instead of an immediate error (thanks Jens for catching this). `startup_timeout` still bounds the overall wait, and genuinely fatal pull errors (`InvalidImageName`, `ErrImageNeverPull`, `manifest unknown`, `unauthorized`, …) remain fatal. (cherry picked from commit 938ccf3)
potiuk
requested review from
hussein-awala,
jedcunningham and
jscheffl
as code owners
April 30, 2026 18:31
Contributor
|
Interesting case that you back-proted provider code to maintenance branch for CI test - so maybe we need to progress on this and install latest released providers and drop the soruce tree in future in the maintenance branch? |
vatsrahul1001
pushed a commit
that referenced
this pull request
May 20, 2026
… (#66170) When a pod is starting, kubelet automatically retries failed image pulls with exponential backoff. `KubernetesPodOperator` monitors the pod via `detect_pod_terminate_early_issues()` and aborts early on what it deems a fatal pull error — but the transient-error pattern list did not cover registry 5xx / gateway outages. A short Docker Hub outage (502/503/504 from `auth.docker.io` or the registry) was being classified as fatal and would abort the pod before kubelet's next retry, even though the pull would have succeeded once upstream recovered. Add `bad gateway`, `service unavailable`, `gateway timeout` to `TRANSIENT_ERROR_PATTERNS` so these conditions let kubelet keep retrying within the caller's `startup_timeout` budget. On kubelet >= 1.32 the `ImagePullBackOff` message is `Back-off pulling image "X": <prev_error>`, so these patterns also match during the backoff state. On older kubelets the bare message carries no detail and we keep the existing fail-fast behaviour — matching `back-off pulling` unconditionally would have caused a 120s wait for a genuinely missing image instead of an immediate error (thanks Jens for catching this). `startup_timeout` still bounds the overall wait, and genuinely fatal pull errors (`InvalidImageName`, `ErrImageNeverPull`, `manifest unknown`, `unauthorized`, …) remain fatal. (cherry picked from commit 938ccf3)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #65490 to
v3-2-test.The fix landed on
mainon April 19 but was never backported. It wasneeded in the wild today: the
v3-2-testK8s system-test jobK8S System:CeleryExecutor-3.10-v1.30.13-false
hit
test_pod_affinity[backcompat]failing with exactly this class oferror — Docker Hub returned a 502 Bad Gateway pulling
docker.io/library/ubuntu:latest, anddetect_pod_terminate_early_issues()on
v3-2-testdoesn't yet recognisebad gatewayas transient, so itaborted the pod before kubelet's next retry.
Original commit message:
Cherry-pick conflict was a single trivial import (
_parse_log_leveldoesn't exist on
v3-2-test), resolved by keeping only thedetect_pod_terminate_early_issuesimport that the new tests need. All11 affected unit tests pass locally, including the
[ErrImagePull-rpc error: 502 Bad Gateway from auth.docker.io]casethat mirrors the failing CI run.
(cherry picked from commit 938ccf3)
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.7) following the guidelines