<!-- Canonical: https://lucioduran.com/blog/kubernetes-gateway-api-envoy-cilium
Language: en
Content-SHA256: fcc8c4eaf7a9f6709cb52c08b752aa9975a8e1910c04130f0647c84a1dd1a7d7
-->

Infrastructure

# Kubernetes Gateway API: Replacing Ingress with Envoy Gateway and Cilium

2026-02-068 min read

Lucio Durán
Engineering Manager & AI Solutions Architect

Also available in: [Español](<https://lucioduran.com/es/blog/kubernetes-gateway-api-envoy-cilium>)

## Limitations of the Ingress Resource

The Ingress resource was intentionally minimal. It handled HTTP routing (host + path -\> backend) and TLS termination. Everything else, rate limiting, authentication, rewriting, CORS, WebSocket support, gRPC, TCP/UDP routing, canary deployments, was pushed to controller-specific annotations.

```yaml
# A typical production Ingress with controller-specific annotations
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: api-gateway
 annotations:
 # nginx-specific - won't work with traefik or any other controller
 nginx.ingress.kubernetes.io/ssl-redirect: "true"
 nginx.ingress.kubernetes.io/proxy-body-size: "50m"
 nginx.ingress.kubernetes.io/proxy-read-timeout: "120"
 nginx.ingress.kubernetes.io/cors-allow-origin: "https://app.example.com"
 nginx.ingress.kubernetes.io/cors-allow-methods: "GET, POST, PUT, DELETE"
 nginx.ingress.kubernetes.io/rate-limit-rps: "100"
 nginx.ingress.kubernetes.io/configuration-snippet: |
 more_set_headers "X-Request-ID: $req_id";
 if ($http_upgrade = "websocket") {
 proxy_pass http://upstream;
 }
 nginx.ingress.kubernetes.io/server-snippet: |
 location /health {
 return 200 'ok';
 add_header Content-Type text/plain;
 }
spec:
 tls:
 - hosts:
 - api.example.com
 secretName: api-tls
 rules:
 - host: api.example.com
 http:
 paths:
 - path: /v1
 pathType: Prefix
 backend:
 service:
 name: api-v1
 port:
 number: 8080
```

This configuration is problematic. The annotations are unvalidated strings. The ` configuration-snippet ` is raw nginx config that gets injected verbatim, a security vulnerability if any annotation can be set by untrusted users (CVE-2021-25742 was exactly this). And none of it is portable across controllers.

## Gateway API: The Three-Resource Model

[Gateway API](<https://gateway-api.sigs.k8s.io/>) replaces the single Ingress resource with a three-layer model that maps to organizational roles:

```
┌─────────────────────────────────────────────────┐
│ GatewayClass (Cluster admin) │
│ "Which controller implementation to use" │
│ Created once per cluster │
├─────────────────────────────────────────────────┤
│ Gateway (Platform team / SRE) │
│ "Where to listen, TLS config, allowed routes" │
│ One per entry point (external, internal, etc.) │
├─────────────────────────────────────────────────┤
│ HTTPRoute / TLSRoute / GRPCRoute (App team) │
│ "How to route traffic to my service" │
│ One per service or group of services │
└─────────────────────────────────────────────────┘
```

This separation is the most significant architectural advantage. The platform team controls the Gateway, ports, TLS certificates, allowed namespaces. App teams create Routes in their own namespaces. No more "I need cluster-admin to add an annotation to my Ingress."

The following is the Gateway API equivalent of the nginx-ingress configuration above:

```yaml
# GatewayClass - cluster admin sets this up once
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
 name: envoy
spec:
 controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
# Gateway - platform team manages this
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: external
 namespace: gateway-infra
spec:
 gatewayClassName: envoy
 listeners:
 - name: https
 port: 443
 protocol: HTTPS
 tls:
 mode: Terminate
 certificateRefs:
 - name: wildcard-tls
 namespace: cert-manager
 allowedRoutes:
 namespaces:
 from: Selector
 selector:
 matchLabels:
 gateway-access: "true"
---
# HTTPRoute - app team creates this in their namespace
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: api-v1
 namespace: api-team
spec:
 parentRefs:
 - name: external
 namespace: gateway-infra
 hostnames:
 - "api.example.com"
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /v1
 backendRefs:
 - name: api-v1
 port: 8080
 filters:
 - type: RequestHeaderModifier
 requestHeaderModifier:
 set:
 - name: X-Request-ID
 value: "%REQ(X-REQUEST-ID)%"
```

Every field is validated by the Kubernetes API server. Typos are rejected at apply time, not silently ignored. The route references its parent Gateway explicitly. And the same HTTPRoute works with Envoy Gateway, Cilium, Istio, or any conformant implementation.

## Traffic Splitting: Canary Without the Tooling

This is where Gateway API provides immediate value. Using Flagger + nginx-ingress for canary deployments meant running a custom controller that manipulated annotations to shift traffic. With Gateway API, traffic splitting is native:

```yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: api-canary
 namespace: api-team
spec:
 parentRefs:
 - name: external
 namespace: gateway-infra
 hostnames:
 - "api.example.com"
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /v2
 backendRefs:
 - name: api-v2-stable
 port: 8080
 weight: 95
 - name: api-v2-canary
 port: 8080
 weight: 5
```

Five percent of traffic goes to the canary. Adjust the weights and reapply. No custom controllers, no annotation gymnastics. This can be automated with a simple script that adjusts weights based on error rates from Prometheus:

```bash
#!/bin/bash
# Progressive canary rollout
CANARY_ERRORS=$(curl -s "http://prometheus:9090/api/v1/query?query=rate(http_requests_total{service='api-v2-canary',status=~'5..'}[5m])" | jq '.data.result[0].value[1]' -r)
CANARY_TOTAL=$(curl -s "http://prometheus:9090/api/v1/query?query=rate(http_requests_total{service='api-v2-canary'}[5m])" | jq '.data.result[0].value[1]' -r)

ERROR_RATE=$(echo "$CANARY_ERRORS / $CANARY_TOTAL" | bc -l)

if (( $(echo "$ERROR_RATE > 0.01" | bc -l) )); then
 echo "Error rate ${ERROR_RATE} exceeds 1%, rolling back"
 kubectl patch httproute api-canary -n api-team --type=merge \
 -p '{"spec":{"rules":[{"matches":[{"path":{"type":"PathPrefix","value":"/v2"}}],"backendRefs":[{"name":"api-v2-stable","port":8080,"weight":100},{"name":"api-v2-canary","port":8080,"weight":0}]}]}}'
else
 CURRENT_WEIGHT=$(kubectl get httproute api-canary -n api-team -o jsonpath='{.spec.rules[0].backendRefs[1].weight}')
 NEW_WEIGHT=$((CURRENT_WEIGHT + 10))
 if [ $NEW_WEIGHT -gt 100 ]; then NEW_WEIGHT=100; fi
 echo "Increasing canary weight to ${NEW_WEIGHT}%"
 # Update weights...
fi
```

## TLS Passthrough: The Thing Ingress Never Did Right

TLS passthrough, forwarding encrypted traffic without terminating TLS at the gateway, was a constant headache with Ingress. nginx-ingress required ` nginx.ingress.kubernetes.io/ssl-passthrough: "true" `, which used a separate TCP listener and couldn't coexist with TLS termination on the same port.

Gateway API handles this with TLSRoute and Gateway listener modes:

```yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: external
 namespace: gateway-infra
spec:
 gatewayClassName: envoy
 listeners:
 # Regular HTTPS with TLS termination
 - name: https-terminate
 port: 443
 protocol: HTTPS
 tls:
 mode: Terminate
 certificateRefs:
 - name: wildcard-tls
 allowedRoutes:
 kinds:
 - kind: HTTPRoute

 # TLS passthrough for services that handle their own TLS
 - name: tls-passthrough
 port: 443
 protocol: TLS
 tls:
 mode: Passthrough
 hostname: "vault.example.com"
 allowedRoutes:
 kinds:
 - kind: TLSRoute
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TLSRoute
metadata:
 name: vault-passthrough
 namespace: vault
spec:
 parentRefs:
 - name: external
 namespace: gateway-infra
 sectionName: tls-passthrough
 rules:
 - backendRefs:
 - name: vault
 port: 8200
```

Both listeners share port 443. The Gateway inspects the SNI (Server Name Indication) in the TLS ClientHello to decide whether to terminate or pass through. This is handled at the protocol level, no annotation hacks.

## Envoy Gateway as Primary Implementation

[Envoy Gateway](<https://gateway.envoyproxy.io/>) serves well for external traffic because of its Envoy Proxy foundation and the ExtensionPolicy CRD for advanced features:

```yaml
# Rate limiting per route
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
 name: api-rate-limit
 namespace: api-team
spec:
 targetRefs:
 - group: gateway.networking.k8s.io
 kind: HTTPRoute
 name: api-v1
 rateLimit:
 type: Global
 global:
 rules:
 - clientSelectors:
 - headers:
 - name: X-API-Key
 type: Distinct
 limit:
 requests: 1000
 unit: Minute
---
# Circuit breaking
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
 name: api-circuit-breaker
 namespace: api-team
spec:
 targetRefs:
 - group: gateway.networking.k8s.io
 kind: HTTPRoute
 name: api-v1
 circuitBreaker:
 maxConnections: 1024
 maxPendingRequests: 128
 maxRequests: 1024
 timeout:
 http:
 connectionIdleTimeout: 60s
 maxConnectionDuration: 300s
 retry:
 numRetries: 3
 retryOn:
 - "5xx"
 - "reset"
 - "connect-failure"
 perRetry:
 timeout: 2s
```

## Cilium: Service Mesh Without the Sidecar Tax

For east-west (service-to-service) traffic, [Cilium](<https://cilium.io/>) with Gateway API is an effective choice. Cilium's eBPF-based approach means no sidecar proxies, the networking logic runs in the kernel:

```yaml
# Cilium Gateway for internal traffic
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: internal
 namespace: gateway-infra
 annotations:
 # Cilium-specific: use eBPF for L7 filtering
 io.cilium/lb-ipam-ips: "10.0.100.50"
spec:
 gatewayClassName: cilium
 listeners:
 - name: http
 port: 80
 protocol: HTTP
 allowedRoutes:
 namespaces:
 from: All
---
# Internal route with header-based routing
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: user-service
 namespace: user-team
spec:
 parentRefs:
 - name: internal
 namespace: gateway-infra
 hostnames:
 - "user-service.internal"
 rules:
 # Route beta traffic to v2
 - matches:
 - headers:
 - name: X-Beta-User
 value: "true"
 backendRefs:
 - name: user-service-v2
 port: 8080
 # Default to stable
 - backendRefs:
 - name: user-service-v1
 port: 8080
```

The performance difference between Cilium (eBPF) and sidecar-based meshes (Istio with Envoy sidecars) is measurable. On a latency-sensitive service handling 50K RPS:

| Metric | No mesh | Istio sidecar | Cilium eBPF |
| --- | --- | --- | --- |
| P50 latency | 2.1ms | 3.4ms | 2.3ms |
| P99 latency | 8.7ms | 14.2ms | 9.1ms |
| Memory per pod | baseline | +50MB (sidecar) | +0MB |
| CPU per pod | baseline | +0.1 core (sidecar) | +~0 |

The sidecar overhead is 50MB RAM and 0.1 CPU core per pod. Across 200 services with 3 replicas each, that's 30GB of RAM and 60 CPU cores just for the mesh. Cilium eliminates this entirely.

## Migration Approach: 200 Services in 6 Weeks

Incremental migration works well. The key insight: Gateway API and Ingress can coexist in the same cluster because they use different resources.

Week 1-2: Deploy Envoy Gateway alongside nginx-ingress. Create the GatewayClass and Gateway resources. Migrate 5 non-critical services.

Week 3-4: Migrate all stateless API services. Validate with synthetic traffic before switching DNS.

Week 5: Migrate services with WebSocket and gRPC requirements. These were actually easier with Gateway API because HTTPRoute natively supports WebSocket upgrades and GRPCRoute exists.

Week 6: Migrate remaining services, decommission nginx-ingress.

A notable finding: 23 Ingress resources were discovered with broken annotations that nobody had noticed. Typos, deprecated annotation names, annotations for the wrong controller. All silently doing nothing. Gateway API would have rejected them at creation time.

## What Gateway API Still Lacks

Several gaps remain:

**Rate limiting isn't standardized.** Each implementation has its own CRDs (Envoy Gateway's BackendTrafficPolicy, Cilium's CiliumNetworkPolicy). The GEP-1731 proposal for policy attachment is still evolving.

**Authentication/authorization at the gateway level** is implementation-specific. Envoy Gateway has SecurityPolicy, Istio has AuthorizationPolicy, Cilium uses its own CRDs. There's no standard HTTPRoute filter for JWT validation or OAuth2.

**TCP/UDP routing** works (TCPRoute, UDPRoute) but is still in the experimental channel. If you're routing non-HTTP traffic, check your implementation's support level.

**Migration tooling is sparse.** Custom scripts may be needed to convert 200+ Ingress resources to HTTPRoutes. A community tool for this would be valuable, but the annotation-to-feature mapping is too controller-specific for a generic converter.

Despite these gaps, Gateway API represents the right direction. The role-based security model, typed configuration, cross-implementation portability, and native traffic management features make it a clear upgrade from the Ingress approach that has been in use for seven years.

For clusters still running Ingress with annotation-heavy configurations, planning a migration is worthwhile.

## Frequently asked questions

### Is Gateway API ready for production?

Yes. Gateway API reached GA (v1.0) in October 2023 and v1.2 is current as of early 2026. Envoy Gateway, Cilium, Istio, Traefik, and Kong all have production-grade implementations. The spec is stable for core features (Gateway, HTTPRoute, TLSRoute).

### Should I migrate from nginx-ingress to Gateway API?

If you're managing more than 20-30 services, yes. The role-based model (platform team manages Gateways, app teams manage Routes) alone justifies the migration. If you have 3-5 services and nginx-ingress works, the migration cost probably isn't worth it yet.

### Envoy Gateway or Cilium for Gateway API?

Envoy Gateway if you need advanced traffic management (rate limiting, authentication, circuit breaking) and don't want a full service mesh. Cilium if you're already using it for CNI/networking and want Gateway API integrated with eBPF-based observability and network policies.

## Tips

1. **Tip:** When migrating from Ingress to Gateway API, don't convert all resources at once. Use ` parentRefs ` in HTTPRoute to bind routes to specific Gateways, you can run both Ingress and Gateway API simultaneously during migration.
2. **Tip:** For TLS passthrough (not termination), use ` TLSRoute ` with ` mode: Passthrough ` on the Gateway listener. This is the one thing Ingress annotations never handled cleanly across controllers.
3. **Tip:** Set ` backendRefs ` weights for traffic splitting: ` weight: 90 ` and ` weight: 10 ` gives you 90/10 canary deployment without any external tooling. The Gateway controller handles the weighted routing natively.
4. **Tip:** With Cilium Gateway API, enable ` hubble ` for L7 observability. You get per-route latency histograms and error rates without deploying a separate observability stack.

- kubernetes
- gateway-api
- envoy
- cilium
- ingress
- service-mesh
- httproute
- tls
- traffic-splitting
- infrastructure

[Back to Blog](<https://lucioduran.com/blog>)