Kubernetes Gateway API: Replacing Ingress with Envoy Gateway and Cilium

Limitations of the Ingress Resource

The Ingress resource was intentionally minimal. It handled HTTP routing (host + path -> backend) and TLS termination. Everything else, rate limiting, authentication, rewriting, CORS, WebSocket support, gRPC, TCP/UDP routing, canary deployments, was pushed to controller-specific annotations.

# A typical production Ingress with controller-specific annotations
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: api-gateway
 annotations:
 # nginx-specific - won't work with traefik or any other controller
 nginx.ingress.kubernetes.io/ssl-redirect: "true"
 nginx.ingress.kubernetes.io/proxy-body-size: "50m"
 nginx.ingress.kubernetes.io/proxy-read-timeout: "120"
 nginx.ingress.kubernetes.io/cors-allow-origin: "https://app.example.com"
 nginx.ingress.kubernetes.io/cors-allow-methods: "GET, POST, PUT, DELETE"
 nginx.ingress.kubernetes.io/rate-limit-rps: "100"
 nginx.ingress.kubernetes.io/configuration-snippet: |
 more_set_headers "X-Request-ID: $req_id";
 if ($http_upgrade = "websocket") {
 proxy_pass http://upstream;
 }
 nginx.ingress.kubernetes.io/server-snippet: |
 location /health {
 return 200 'ok';
 add_header Content-Type text/plain;
 }
spec:
 tls:
 - hosts:
 - api.example.com
 secretName: api-tls
 rules:
 - host: api.example.com
 http:
 paths:
 - path: /v1
 pathType: Prefix
 backend:
 service:
 name: api-v1
 port:
 number: 8080

This configuration is problematic. The annotations are unvalidated strings. The configuration-snippet is raw nginx config that gets injected verbatim, a security vulnerability if any annotation can be set by untrusted users (CVE-2021-25742 was exactly this). And none of it is portable across controllers.

Gateway API: The Three-Resource Model

Gateway API replaces the single Ingress resource with a three-layer model that maps to organizational roles:

┌─────────────────────────────────────────────────┐
│ GatewayClass (Cluster admin) │
│ "Which controller implementation to use" │
│ Created once per cluster │
├─────────────────────────────────────────────────┤
│ Gateway (Platform team / SRE) │
│ "Where to listen, TLS config, allowed routes" │
│ One per entry point (external, internal, etc.) │
├─────────────────────────────────────────────────┤
│ HTTPRoute / TLSRoute / GRPCRoute (App team) │
│ "How to route traffic to my service" │
│ One per service or group of services │
└─────────────────────────────────────────────────┘

This separation is the most significant architectural advantage. The platform team controls the Gateway, ports, TLS certificates, allowed namespaces. App teams create Routes in their own namespaces. No more "I need cluster-admin to add an annotation to my Ingress."

The following is the Gateway API equivalent of the nginx-ingress configuration above:

# GatewayClass - cluster admin sets this up once
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
 name: envoy
spec:
 controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
# Gateway - platform team manages this
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: external
 namespace: gateway-infra
spec:
 gatewayClassName: envoy
 listeners:
 - name: https
 port: 443
 protocol: HTTPS
 tls:
 mode: Terminate
 certificateRefs:
 - name: wildcard-tls
 namespace: cert-manager
 allowedRoutes:
 namespaces:
 from: Selector
 selector:
 matchLabels:
 gateway-access: "true"
---
# HTTPRoute - app team creates this in their namespace
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: api-v1
 namespace: api-team
spec:
 parentRefs:
 - name: external
 namespace: gateway-infra
 hostnames:
 - "api.example.com"
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /v1
 backendRefs:
 - name: api-v1
 port: 8080
 filters:
 - type: RequestHeaderModifier
 requestHeaderModifier:
 set:
 - name: X-Request-ID
 value: "%REQ(X-REQUEST-ID)%"

Every field is validated by the Kubernetes API server. Typos are rejected at apply time, not silently ignored. The route references its parent Gateway explicitly. And the same HTTPRoute works with Envoy Gateway, Cilium, Istio, or any conformant implementation.

Traffic Splitting: Canary Without the Tooling

This is where Gateway API provides immediate value. Using Flagger + nginx-ingress for canary deployments meant running a custom controller that manipulated annotations to shift traffic. With Gateway API, traffic splitting is native:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: api-canary
 namespace: api-team
spec:
 parentRefs:
 - name: external
 namespace: gateway-infra
 hostnames:
 - "api.example.com"
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /v2
 backendRefs:
 - name: api-v2-stable
 port: 8080
 weight: 95
 - name: api-v2-canary
 port: 8080
 weight: 5

Five percent of traffic goes to the canary. Adjust the weights and reapply. No custom controllers, no annotation gymnastics. This can be automated with a simple script that adjusts weights based on error rates from Prometheus:

#!/bin/bash
# Progressive canary rollout
CANARY_ERRORS=$(curl -s "http://prometheus:9090/api/v1/query?query=rate(http_requests_total{service='api-v2-canary',status=~'5..'}[5m])" | jq '.data.result[0].value[1]' -r)
CANARY_TOTAL=$(curl -s "http://prometheus:9090/api/v1/query?query=rate(http_requests_total{service='api-v2-canary'}[5m])" | jq '.data.result[0].value[1]' -r)

ERROR_RATE=$(echo "$CANARY_ERRORS / $CANARY_TOTAL" | bc -l)

if (( $(echo "$ERROR_RATE > 0.01" | bc -l) )); then
 echo "Error rate ${ERROR_RATE} exceeds 1%, rolling back"
 kubectl patch httproute api-canary -n api-team --type=merge \
 -p '{"spec":{"rules":[{"matches":[{"path":{"type":"PathPrefix","value":"/v2"}}],"backendRefs":[{"name":"api-v2-stable","port":8080,"weight":100},{"name":"api-v2-canary","port":8080,"weight":0}]}]}}'
else
 CURRENT_WEIGHT=$(kubectl get httproute api-canary -n api-team -o jsonpath='{.spec.rules[0].backendRefs[1].weight}')
 NEW_WEIGHT=$((CURRENT_WEIGHT + 10))
 if [ $NEW_WEIGHT -gt 100 ]; then NEW_WEIGHT=100; fi
 echo "Increasing canary weight to ${NEW_WEIGHT}%"
 # Update weights...
fi

TLS Passthrough: The Thing Ingress Never Did Right

TLS passthrough, forwarding encrypted traffic without terminating TLS at the gateway, was a constant headache with Ingress. nginx-ingress required nginx.ingress.kubernetes.io/ssl-passthrough: "true", which used a separate TCP listener and couldn't coexist with TLS termination on the same port.

Gateway API handles this with TLSRoute and Gateway listener modes:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: external
 namespace: gateway-infra
spec:
 gatewayClassName: envoy
 listeners:
 # Regular HTTPS with TLS termination
 - name: https-terminate
 port: 443
 protocol: HTTPS
 tls:
 mode: Terminate
 certificateRefs:
 - name: wildcard-tls
 allowedRoutes:
 kinds:
 - kind: HTTPRoute

 # TLS passthrough for services that handle their own TLS
 - name: tls-passthrough
 port: 443
 protocol: TLS
 tls:
 mode: Passthrough
 hostname: "vault.example.com"
 allowedRoutes:
 kinds:
 - kind: TLSRoute
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TLSRoute
metadata:
 name: vault-passthrough
 namespace: vault
spec:
 parentRefs:
 - name: external
 namespace: gateway-infra
 sectionName: tls-passthrough
 rules:
 - backendRefs:
 - name: vault
 port: 8200

Both listeners share port 443. The Gateway inspects the SNI (Server Name Indication) in the TLS ClientHello to decide whether to terminate or pass through. This is handled at the protocol level, no annotation hacks.

Envoy Gateway as Primary Implementation

Envoy Gateway serves well for external traffic because of its Envoy Proxy foundation and the ExtensionPolicy CRD for advanced features:

# Rate limiting per route
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
 name: api-rate-limit
 namespace: api-team
spec:
 targetRefs:
 - group: gateway.networking.k8s.io
 kind: HTTPRoute
 name: api-v1
 rateLimit:
 type: Global
 global:
 rules:
 - clientSelectors:
 - headers:
 - name: X-API-Key
 type: Distinct
 limit:
 requests: 1000
 unit: Minute
---
# Circuit breaking
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
 name: api-circuit-breaker
 namespace: api-team
spec:
 targetRefs:
 - group: gateway.networking.k8s.io
 kind: HTTPRoute
 name: api-v1
 circuitBreaker:
 maxConnections: 1024
 maxPendingRequests: 128
 maxRequests: 1024
 timeout:
 http:
 connectionIdleTimeout: 60s
 maxConnectionDuration: 300s
 retry:
 numRetries: 3
 retryOn:
 - "5xx"
 - "reset"
 - "connect-failure"
 perRetry:
 timeout: 2s

Cilium: Service Mesh Without the Sidecar Tax

For east-west (service-to-service) traffic, Cilium with Gateway API is an effective choice. Cilium's eBPF-based approach means no sidecar proxies, the networking logic runs in the kernel:

# Cilium Gateway for internal traffic
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: internal
 namespace: gateway-infra
 annotations:
 # Cilium-specific: use eBPF for L7 filtering
 io.cilium/lb-ipam-ips: "10.0.100.50"
spec:
 gatewayClassName: cilium
 listeners:
 - name: http
 port: 80
 protocol: HTTP
 allowedRoutes:
 namespaces:
 from: All
---
# Internal route with header-based routing
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: user-service
 namespace: user-team
spec:
 parentRefs:
 - name: internal
 namespace: gateway-infra
 hostnames:
 - "user-service.internal"
 rules:
 # Route beta traffic to v2
 - matches:
 - headers:
 - name: X-Beta-User
 value: "true"
 backendRefs:
 - name: user-service-v2
 port: 8080
 # Default to stable
 - backendRefs:
 - name: user-service-v1
 port: 8080

The performance difference between Cilium (eBPF) and sidecar-based meshes (Istio with Envoy sidecars) is measurable. On a latency-sensitive service handling 50K RPS:

Metric	No mesh	Istio sidecar	Cilium eBPF
P50 latency	2.1ms	3.4ms	2.3ms
P99 latency	8.7ms	14.2ms	9.1ms
Memory per pod	baseline	+50MB (sidecar)	+0MB
CPU per pod	baseline	+0.1 core (sidecar)	+~0

The sidecar overhead is 50MB RAM and 0.1 CPU core per pod. Across 200 services with 3 replicas each, that's 30GB of RAM and 60 CPU cores just for the mesh. Cilium eliminates this entirely.

Migration Approach: 200 Services in 6 Weeks

Incremental migration works well. The key insight: Gateway API and Ingress can coexist in the same cluster because they use different resources.

Week 1-2: Deploy Envoy Gateway alongside nginx-ingress. Create the GatewayClass and Gateway resources. Migrate 5 non-critical services.

Week 3-4: Migrate all stateless API services. Validate with synthetic traffic before switching DNS.

Week 5: Migrate services with WebSocket and gRPC requirements. These were actually easier with Gateway API because HTTPRoute natively supports WebSocket upgrades and GRPCRoute exists.

Week 6: Migrate remaining services, decommission nginx-ingress.

A notable finding: 23 Ingress resources were discovered with broken annotations that nobody had noticed. Typos, deprecated annotation names, annotations for the wrong controller. All silently doing nothing. Gateway API would have rejected them at creation time.

What Gateway API Still Lacks

Several gaps remain:

Rate limiting isn't standardized. Each implementation has its own CRDs (Envoy Gateway's BackendTrafficPolicy, Cilium's CiliumNetworkPolicy). The GEP-1731 proposal for policy attachment is still evolving.

Authentication/authorization at the gateway level is implementation-specific. Envoy Gateway has SecurityPolicy, Istio has AuthorizationPolicy, Cilium uses its own CRDs. There's no standard HTTPRoute filter for JWT validation or OAuth2.

TCP/UDP routing works (TCPRoute, UDPRoute) but is still in the experimental channel. If you're routing non-HTTP traffic, check your implementation's support level.

Migration tooling is sparse. Custom scripts may be needed to convert 200+ Ingress resources to HTTPRoutes. A community tool for this would be valuable, but the annotation-to-feature mapping is too controller-specific for a generic converter.

Despite these gaps, Gateway API represents the right direction. The role-based security model, typed configuration, cross-implementation portability, and native traffic management features make it a clear upgrade from the Ingress approach that has been in use for seven years.

For clusters still running Ingress with annotation-heavy configurations, planning a migration is worthwhile.