Skip to content

feat: support EnvoyProxyModule Wasm code source via EnvoyProxy allowlist#9530

Open
daanvinken wants to merge 8 commits into
envoyproxy:mainfrom
daanvinken:feat/wasm-filesystem-source-9448
Open

feat: support EnvoyProxyModule Wasm code source via EnvoyProxy allowlist#9530
daanvinken wants to merge 8 commits into
envoyproxy:mainfrom
daanvinken:feat/wasm-filesystem-source-9448

Conversation

@daanvinken

@daanvinken daanvinken commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Description

Adds an EnvoyProxyModule Wasm code source so Envoy can load modules from a
path registered on EnvoyProxy, without the control-plane fetch/cache path
used by HTTP and Image sources.

Design matches #9448 and review discussion (DynamicModule-style allowlist):

  • Register modules on EnvoyProxy.spec.wasmModules with
    source.type: Local and source.local.path
  • Reference them from EnvoyExtensionPolicy with
    code.type: EnvoyProxyModule and envoyProxyModule.name (no path on the policy)
  • Translator emits AsyncDataSource.local for Local modules
  • HTTP/Image behavior is unchanged
  • Control-plane wasm_cluster is only created when a remote Wasm source is present

The Wasm code type enum is now HTTP, Image, and EnvoyProxyModule.
The unused ConfigMap enum value (never implemented) was removed.

Remote/OCI registration on EnvoyProxy is intentionally not implemented yet;
the source shape leaves room for that later.

Fixes #9448

Testing

  • Unit test for local vs remote Wasm code source in the xDS translator
  • Translator fixtures for EnvoyProxyModule (registered module resolves to a path;
    unknown module name is rejected)
  • xDS fixture emits code.local.filename and no unused wasm_cluster
  • CEL cases for EnvoyProxyModule and wasmModules Local source
  • e2e: custom Envoy image with the example Wasm baked in
    (examples/wasm-module-test), asserts x-wasm-custom: FOO

@daanvinken
daanvinken requested a review from a team as a code owner July 20, 2026 12:30
@netlify

netlify Bot commented Jul 20, 2026

Copy link
Copy Markdown

Deploy Preview for cerulean-figolla-1f9435 ready!

Name Link
🔨 Latest commit bf94ea5
🔍 Latest deploy log https://app.netlify.com/projects/cerulean-figolla-1f9435/deploys/6a604015792378000849a7cb
😎 Deploy Preview https://deploy-preview-9530--cerulean-figolla-1f9435.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@daanvinken

Copy link
Copy Markdown
Contributor Author

@zhaohuabing if you have some time 🙏

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 53e53febd3

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

wasms []ir.Wasm
wasmFailOpen bool
)
if wasms, wasmError, wasmFailOpen = t.buildWasms(policy, resources, gtwCtx.envoyProxy); wasmError != nil {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Scope filesystem Wasm errors per ancestor

When an HTTPRoute is attached to multiple managed Gateways, this call now resolves the filesystem module against each Gateway's EnvoyProxy but appends any failure to one errs value returned to processEnvoyExtensionPolicyForRoute, which then sets the translation error on all ancestor refs. If one Gateway allowlists the module and another does not, the successful ancestor is still marked translation-failed even though its routes get the filter; please report/apply the error per parentRef instead of aggregating it across all ancestors.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pre-existing bug with ancestor refs. This should be fixed but is separate from your changes.

@daanvinken
daanvinken force-pushed the feat/wasm-filesystem-source-9448 branch from 53e53fe to 089e532 Compare July 20, 2026 12:43
@codecov

codecov Bot commented Jul 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 84.78261% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.57%. Comparing base (8b2e277) to head (bf94ea5).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
internal/gatewayapi/envoyextensionpolicy.go 72.72% 7 Missing and 5 partials ⚠️
internal/xds/translator/wasm.go 94.73% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9530      +/-   ##
==========================================
+ Coverage   75.54%   75.57%   +0.02%     
==========================================
  Files         252      252              
  Lines       41733    41785      +52     
==========================================
+ Hits        31529    31580      +51     
  Misses       8072     8072              
- Partials     2132     2133       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@daanvinken
daanvinken force-pushed the feat/wasm-filesystem-source-9448 branch 2 times, most recently from f4fe92e to fddff27 Compare July 20, 2026 14:21
@zhaohuabing

zhaohuabing commented Jul 20, 2026

Copy link
Copy Markdown
Member
  1. Unnecessary wasm_cluster emitted for filesystem-only configs (minor, worth fixing)

containsWasm (internal/xds/translator/globalresources.go:144) returns true for any Wasm, so createWasmHTTPServiceCluster still emits the wasm_cluster (the EG HTTP serving cluster, plus its upstream TLS transportSocket) even when every Wasm is a Filesystem source that never contacts it. This is visible in the PR's own fixture wasm-filesystem.clusters.yaml, which contains a full wasm_cluster that no filter references.

Suggest gating on the presence of a remote source:

if route.EnvoyExtensions != nil {
    for _, w := range route.EnvoyExtensions.Wasms {
        if w.Code != nil { // HTTP/Image only
            return true
        }
    }
}

This avoids shipping a dead cluster (and its client-cert secret dependency) to proxies that only use filesystem modules — which is precisely the "skip the control-plane path" benefit the PR advertises.

  1. buildWasms moved inside the per-parentRef loop — cache side effect now repeats (minor)

In translateEnvoyExtensionPolicyForRoute, buildWasms moved from once-per-policy to once-per-parentRef (necessary, since the filesystem path resolves against gtwCtx.envoyProxy, which can differ per Gateway). This matches the existing buildLuas/buildDynamicModules placement, so structurally it's fine.

The wrinkle unique to Wasm: for HTTP/Image sources this now calls t.WasmCache.Get() N times for a route with N parentRefs, where it previously ran once. For IfNotPresent this is a cheap cache hit, but with pullPolicy: Always it can trigger repeated fetches. Not a correctness bug, but a behavioral change worth a comment or a second look — a small memoization keyed by resolved config would eliminate it if it matters.

  1. Enum value ConfigMap silently removed (call out in PR description)

The field-level annotation changed from Enum=HTTP;Image;ConfigMap to Enum=HTTP;Image;Filesystem (wasm_types.go:41). ConfigMap was never implemented (no code path, and the type alias only listed HTTP;Image), so this is stale cleanup rather than a real removal. Still, it's technically a narrowing of the accepted API surface — worth an explicit mention in the PR description/release note in case anyone had it set (it would have been non-functional regardless).

Register Wasm modules on EnvoyProxy.spec.wasmModules and reference them
by name from EnvoyExtensionPolicy with type Filesystem. Envoy loads the
local file via AsyncDataSource.local and skips the control-plane
fetch/cache path used by HTTP and Image sources.

Includes generated CRD/helm fixtures, API docs, and the jwt DnsCluster
fixture update needed for gen-check.

Fixes envoyproxy#9448

Signed-off-by: daanvinken <daanvinken@tythus.com>
@daanvinken
daanvinken force-pushed the feat/wasm-filesystem-source-9448 branch from fddff27 to 3e206c5 Compare July 20, 2026 14:56
Only create the control-plane wasm HTTP service cluster (and gate the
client certificate) when a route uses a remote HTTP/Image code source.
Filesystem modules load a local path and never use that cluster.

Signed-off-by: daanvinken <daanvinken@tythus.com>
Filesystem resolution is Gateway-scoped, so buildWasms runs per parentRef
like Lua and DynamicModules. HTTP/Image may call WasmCache.Get once per
parent; IfNotPresent hits the cache, Always may re-fetch.

Signed-off-by: daanvinken <daanvinken@tythus.com>
ConfigMap was never implemented; the code source enum is now HTTP, Image,
and Filesystem only.

Signed-off-by: daanvinken <daanvinken@tythus.com>
@daanvinken

Copy link
Copy Markdown
Contributor Author

Super helpful, thanks @zhaohuabing

I've addressed your comments. I'll see when I can carve out some time to deploy this on an actual cluster and test.
Feel free to leave any comments in the meantime.

@zhaohuabing

zhaohuabing commented Jul 20, 2026

Copy link
Copy Markdown
Member

Should we mirror DynamicModuleSource to leave room for defining HTTP/OCI WASM extensions in the EnvoyProxy(not now)? cc @envoyproxy/gateway-maintainers

type WasmModuleEntry struct {
    Name   string           `json:"name"`
    Source WasmModuleSource `json:"source"`
}

type WasmModuleSource struct {
    // +kubebuilder:default=Local
    // +unionDiscriminator
    Type  *WasmModuleSourceType  `json:"type,omitempty"`   // Local (+ Remote/Image later)
    Local *LocalWasmModuleSource `json:"local,omitempty"`  // { path }
}

wasmModules:
- name: security-filter
  source:
    type: Local          # defaulted; can be omitted
    local:
      path: /var/lib/envoy/security-filter.wasm
const EnvoyProxyModuleWasmCodeSourceType WasmCodeSourceType = "EnvoyProxyModule"

type EnvoyProxyModuleWasmCodeSource struct {
    // Name references a module in EnvoyProxy.spec.wasmModules.
    Name string `json:"name"`
}

wasm:
- code:
    type: EnvoyProxyModule
    envoyProxyModule:
      name: security-filter        # local or remote — resolved from EnvoyProxy

@zhaohuabing

Copy link
Copy Markdown
Member

I'll see when I can carve out some time to deploy this on an actual cluster and test

You should be able to add an e2e test for this, following the dynamic module example. You can use this WASM file. for the test.

@jukie

jukie commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

for defining HTTP/OCI WASM extensions in the EnvoyProxy(not now)?

I agree with the suggested shape for filesystem WASM modules but long-term handling of image/http sources might get awkward so we should discuss that as well.
It doesn't need to be wired up here but maybe we should deprecate the existing image/http source fields and offer operators a RuntimeFlag to control whether EnvoyProxy registration is required or optional?

Register wasmModules with source.type Local and source.local.path, and
reference them from EnvoyExtensionPolicy as type EnvoyProxyModule. Leaves
room for additional EnvoyProxy module sources later without implementing
HTTP/OCI registration yet.

Signed-off-by: daanvinken <daanvinken@tythus.com>
Add a custom Envoy image with the example Wasm module baked in, wire it
into EXAMPLE_APPS, and add an e2e that registers the module on EnvoyProxy
and asserts the filter response header.

Signed-off-by: daanvinken <daanvinken@tythus.com>
The example Wasm filter appends text to the response body, which breaks
JSON extraction of request properties. Match the HTTP Wasm e2e workaround.

Signed-off-by: daanvinken <daanvinken@tythus.com>
@daanvinken

Copy link
Copy Markdown
Contributor Author

Ready for another review, @zhaohuabing
E2E failures seem unrelated?

@daanvinken daanvinken changed the title feat: support Filesystem Wasm code source via EnvoyProxy allowlist feat: support EnvoyProxyModule Wasm code source via EnvoyProxy allowlist Jul 21, 2026
@zhaohuabing

Copy link
Copy Markdown
Member

Ready for another review, @zhaohuabing E2E failures seem unrelated?

Some of the tests are flaky. You can comment /retest to run them again.

@zhaohuabing

Copy link
Copy Markdown
Member

Hi @daanvinken Thanks for working on this. I'll bring up the API design at this week's community meeting so we can reach a decision and unblock you.

@jukie

jukie commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Pending discussion at community sync but LGTM overall, thanks!

@daanvinken

Copy link
Copy Markdown
Contributor Author

/retest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support a local/filesystem Wasm code source in EnvoyExtensionPolicy

3 participants