Skip to content

fix: confirm NotFound against the API server before dropping status writes#9566

Open
zhaohuabing wants to merge 2 commits into
envoyproxy:mainfrom
zhaohuabing:fix/status-updater-confirm-notfound-uncached
Open

fix: confirm NotFound against the API server before dropping status writes#9566
zhaohuabing wants to merge 2 commits into
envoyproxy:mainfrom
zhaohuabing:fix/status-updater-confirm-notfound-uncached

Conversation

@zhaohuabing

@zhaohuabing zhaohuabing commented Jul 23, 2026

Copy link
Copy Markdown
Member

Description

Fixes #9536: under high watch-event churn (e.g. ~300+ HTTPRoutes on a single Gateway with ArgoCD auto-sync), some objects end up with an empty status ({"parents":[]}) forever, even though the data plane is programmed correctly. A controller restart flushes all of them at once.

Root cause

The status updater's apply() reads the object with the manager's cache-backed client before writing:

if err := u.client.Get(ctx, update.NamespacedName, obj); err != nil {
    if kerrors.IsNotFound(err) {
        return nil // silently skipped, and counted as a success
    }
    return err
}

Under churn the informer cache lags behind the API server, so this Get can return NotFound for a freshly-created object the cache hasn't caught up to yet. apply() then silently returns, so the status is never written.

Fix

When the cache-backed Get returns NotFound, confirm against an uncached APIReader (mgr.GetAPIReader()) before giving up. If the object is genuinely gone, skip as before; otherwise proceed with the fresh object and write the status.

The extra uncached read happens only on a cache miss — never for stable, already-cached objects — so it is bounded by create/delete churn rather than the status-event rate and adds no meaningful API-server load in steady state.

…rites

Under high watch-event churn the status updater's cache-backed Get can
return NotFound for a freshly-created object that the informer cache has
not caught up to yet. apply() then silently skipped the write (and even
counted it as a success), so the object was left with an empty status
until a controller restart rebuilt the cache from a fresh LIST.

Confirm the NotFound against an uncached APIReader before giving up. The
extra uncached read only happens on a cache miss, so it adds no meaningful
API-server load in steady state.

Addresses envoyproxy#9536

Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com>
@zhaohuabing
zhaohuabing requested a review from a team as a code owner July 23, 2026 05:40
@netlify

netlify Bot commented Jul 23, 2026

Copy link
Copy Markdown

Deploy Preview for cerulean-figolla-1f9435 ready!

Name Link
🔨 Latest commit 77ea943
🔍 Latest deploy log https://app.netlify.com/projects/cerulean-figolla-1f9435/deploys/6a61adda6078b10008cd30e2
😎 Deploy Preview https://deploy-preview-9566--cerulean-figolla-1f9435.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@codecov

codecov Bot commented Jul 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 66.66667% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.60%. Comparing base (bae64fd) to head (77ea943).

Files with missing lines Patch % Lines
internal/provider/kubernetes/status_updater.go 62.50% 2 Missing and 1 partial ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #9566   +/-   ##
=======================================
  Coverage   75.59%   75.60%           
=======================================
  Files         252      252           
  Lines       41747    41751    +4     
=======================================
+ Hits        31559    31564    +5     
+ Misses       8061     8060    -1     
  Partials     2127     2127           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Addresses envoyproxy#9536

Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

HTTPRoute status.parents never populated under high watch-event churn (~300+ routes), despite correct data-plane routing

1 participant