Infrastructure # eBPF in Production: Building Kernel-Level Observability for Kubernetes Without the Overhead 2025-10-258 min read Lucio Durán Engineering Manager & AI Solutions Architect Also available in: [Español]() ## What eBPF Is: Technical Overview At its core, eBPF is straightforward: it's a virtual machine inside the Linux kernel that lets you run sandboxed programs at specific hook points without modifying kernel source or loading kernel modules. The "extended" in eBPF (originally Berkeley Packet Filter) refers to the fact that it's been expanded far beyond packet filtering into a general-purpose kernel instrumentation framework. The key constraints that make it safe: ``` 1. Programs must pass the BPF verifier (proves termination, memory safety) 2. No unbounded loops (the verifier must prove all loops terminate) 3. Limited stack size (512 bytes) 4. No sleeping or blocking in most program types 5. Helper functions are the only way to interact with kernel state ``` These constraints sound limiting, but they guarantee something crucial: **an eBPF program cannot crash your kernel**. Period. The verifier rejects anything that could cause undefined behavior. This is what makes eBPF viable for production, unlike kprobes or SystemTap scripts that can bring down a machine. ## The Observability Stack in 2026 The following shows what a modern eBPF-based observability stack looks like in production Kubernetes: ``` ┌─────────────────────────────────────┐ │ Grafana / UI │ ├─────────────────────────────────────┤ │ Prometheus / OpenTelemetry │ ├──────────┬──────────┬───────────────┤ │ Hubble │ Tetragon │ Custom Probes │ ├──────────┴──────────┴───────────────┤ │ Cilium (CNI + eBPF) │ ├─────────────────────────────────────┤ │ Linux Kernel (5.10+) │ └─────────────────────────────────────┘ ``` ### Cilium: The Network Layer [Cilium]() replaces kube-proxy and iptables with eBPF programs attached to TC (traffic control) hooks. Instead of traversing a chain of iptables rules for every packet, Cilium does a single BPF map lookup. The performance difference is dramatic: ``` # iptables-based kube-proxy (5,000 services): # ~20ms per connection setup (rule traversal) # ~15% CPU overhead on proxy node # Cilium eBPF (5,000 services): # ~0.2ms per connection setup (hash map lookup) # ~2% CPU overhead on proxy node ``` But the real win isn't raw throughput, it's observability. Cilium's Hubble component gives you L3/L4/L7 visibility into every packet flowing through the cluster, with zero instrumentation in your application code. ```bash # Watch all HTTP requests between services in real-time hubble observe --namespace production \ --protocol http \ --verdict FORWARDED \ -o json | jq '{src: .source.labels, dst: .destination.labels, http: .l7.http}' ``` This is observing actual kernel-level packet flow, not sampling. Every single packet. And because it's eBPF, the overhead is measured in microseconds per packet. ### Tetragon: Security Observability [Tetragon]() is where things get really interesting from a security monitoring perspective. It attaches eBPF programs to kernel functions (kprobes, tracepoints, LSM hooks) and gives you a declarative YAML interface to define what you want to observe. The following is a TracingPolicy that monitors for suspicious file access: ```yaml apiVersion: cilium.io/v1alpha1 kind: TracingPolicy metadata: name: sensitive-file-access spec: kprobes: - call: "security_file_open" syscall: false args: - index: 0 type: "file" selectors: - matchArgs: - index: 0 operator: "Prefix" values: - "/etc/shadow" - "/etc/passwd" - "/root/.ssh" - "/var/run/secrets/kubernetes.io" matchActions: - action: Sigkill # Kill the process immediately - action: Post # Also send event to userspace ``` This policy runs entirely in kernel space. When a process opens ` /etc/shadow `, the eBPF program fires, checks the path, and can **kill the process before the ` open() ` syscall returns**. No userspace roundtrip. No TOCTOU race condition. The enforcement happens at the kernel level. In one deployment scenario, a cluster running a third-party vendor's container suspected of exfiltrating secrets was monitored with this policy. Within an hour, Tetragon caught the container trying to read Kubernetes service account tokens from the mounted secret volume. The process was killed before it could read the first byte. ## Writing Custom Probes Sometimes the off-the-shelf tools are not enough. Here is how to build custom eBPF probes for specific observability needs. ### Example: TCP Connection Latency Histogram This probe measures the time between ` connect() ` being called and the TCP handshake completing: ```c // tcp_connect_latency.bpf.c #include "vmlinux.h" #include #include #include struct { __uint(type, BPF_MAP_TYPE_HASH); __uint(max_entries, 65536); __type(key, u64); // pid_tgid __type(value, u64); // start timestamp } connect_start SEC(".maps"); struct { __uint(type, BPF_MAP_TYPE_RINGBUF); __uint(max_entries, 256 * 1024); } events SEC(".maps"); struct event { u32 pid; u32 dport; u64 latency_ns; u32 daddr; }; SEC("kprobe/tcp_v4_connect") int BPF_KPROBE(tcp_v4_connect_entry, struct sock *sk) { u64 pid_tgid = bpf_get_current_pid_tgid(); u64 ts = bpf_ktime_get_ns(); bpf_map_update_elem(&connect_start, &pid_tgid, &ts, BPF_ANY); return 0; } SEC("kprobe/tcp_finish_connect") int BPF_KPROBE(tcp_finish_connect_entry, struct sock *sk) { u64 pid_tgid = bpf_get_current_pid_tgid(); u64 *start_ts = bpf_map_lookup_elem(&connect_start, &pid_tgid); if (!start_ts) return 0; u64 latency = bpf_ktime_get_ns() - *start_ts; bpf_map_delete_elem(&connect_start, &pid_tgid); struct event *e = bpf_ringbuf_reserve(&events, sizeof(*e), 0); if (!e) return 0; e->pid = pid_tgid >> 32; e->latency_ns = latency; e->dport = BPF_CORE_READ(sk, __sk_common.skc_dport); e->daddr = BPF_CORE_READ(sk, __sk_common.skc_daddr); bpf_ringbuf_submit(e, 0); return 0; } char LICENSE[] SEC("license") = "GPL"; ``` The Go userspace component that consumes these events: ```go package main import ( "bytes" "encoding/binary" "fmt" "log" "net" "github.com/cilium/ebpf/link" "github.com/cilium/ebpf/ringbuf" "github.com/cilium/ebpf/rlimit" ) //go:generate go run github.com/cilium/ebpf/cmd/bpf2go -cc clang tcpLatency tcp_connect_latency.bpf.c type Event struct { Pid uint32 Dport uint32 LatencyNs uint64 Daddr uint32 } func main() { if err := rlimit.RemoveMemlock(); err != nil { log.Fatal(err) } objs := tcpLatencyObjects{} if err := loadTcpLatencyObjects(&objs, nil); err != nil { log.Fatalf("loading objects: %v", err) } defer objs.Close() kp1, err := link.Kprobe("tcp_v4_connect", objs.TcpV4ConnectEntry, nil) if err != nil { log.Fatalf("opening kprobe: %v", err) } defer kp1.Close() kp2, err := link.Kprobe("tcp_finish_connect", objs.TcpFinishConnectEntry, nil) if err != nil { log.Fatalf("opening kprobe: %v", err) } defer kp2.Close() rd, err := ringbuf.NewReader(objs.Events) if err != nil { log.Fatalf("opening ringbuf reader: %v", err) } defer rd.Close() fmt.Println("Tracing TCP connect latency... Ctrl+C to stop") for { record, err := rd.Read() if err != nil { log.Printf("reading ringbuf: %v", err) continue } var event Event if err := binary.Read(bytes.NewBuffer(record.RawSample), binary.LittleEndian, &event); err != nil { continue } addr := make(net.IP, 4) binary.LittleEndian.PutUint32(addr, event.Daddr) fmt.Printf("pid=%d dst=%s:%d latency=%.2fms\n", event.Pid, addr, ntohs(uint16(event.Dport)), float64(event.LatencyNs)/1e6) } } ``` ### Deploying Custom Probes in Kubernetes The tricky part is not writing the probe, it is deploying it safely across a fleet. Here is a proven deployment pattern: ```yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: tcp-latency-probe namespace: observability spec: selector: matchLabels: app: tcp-latency-probe template: metadata: labels: app: tcp-latency-probe spec: hostPID: true hostNetwork: true containers: - name: probe image: ghcr.io/myorg/tcp-latency-probe:v1.2.0 securityContext: privileged: false capabilities: add: - BPF - PERFMON - SYS_RESOURCE volumeMounts: - name: bpf-fs mountPath: /sys/fs/bpf - name: debugfs mountPath: /sys/kernel/debug readOnly: true volumes: - name: bpf-fs hostPath: path: /sys/fs/bpf - name: debugfs hostPath: path: /sys/kernel/debug ``` Notice we're not using ` privileged: true `. The ` BPF `, ` PERFMON `, and ` SYS_RESOURCE ` capabilities are sufficient. Running eBPF probes as privileged containers is a common anti-pattern that gives the probe unnecessary access. ## Performance Characteristics Here are real numbers from a production cluster (150 nodes, ~3000 pods): ``` Baseline (no eBPF observability): Node CPU utilization: 62% average p99 request latency: 45ms With Cilium + Hubble (L7 visibility): Node CPU utilization: 63.8% average (+1.8%) p99 request latency: 45.2ms (+0.4%) With Tetragon (security monitoring, 12 TracingPolicies): Node CPU utilization: 64.1% average (+0.3% additional) p99 request latency: 45.2ms (no change) With custom TCP latency probe (all connections): Node CPU utilization: 64.3% average (+0.2% additional) p99 request latency: 45.3ms (+0.1%) Total overhead: ~2.3% CPU, ~0.3ms p99 latency ``` Compare this to a traditional sidecar-based service mesh (Istio with Envoy): ``` With Istio sidecar (L7 visibility): Node CPU utilization: 71% average (+9%) p99 request latency: 52ms (+7ms) Memory overhead: ~50MB per pod (sidecar) ``` The difference is staggering. eBPF operates at the kernel level, avoiding the context switches and memory copies that sidecar proxies require. You get better observability with an order of magnitude less overhead. ## Common Pitfalls After deploying eBPF across multiple production environments, these are the mistakes teams make repeatedly: **1. Copying too much data to userspace.** Every ` bpf_ringbuf_submit() ` or ` bpf_perf_event_output() ` involves a memory copy. If you're tracing every syscall and copying full argument buffers, you'll burn CPU. Filter and aggregate in-kernel whenever possible. **2. Not using CO-RE.** If you're compiling eBPF programs against specific kernel headers and shipping different binaries per kernel version, you're doing it wrong. BPF CO-RE (Compile Once, Run Everywhere) with BTF eliminates this entirely. **3. Ignoring map sizing.** BPF maps are allocated at load time. A ` BPF_MAP_TYPE_HASH ` with ` max_entries=1000000 ` allocates memory upfront on each CPU. Size your maps based on actual cardinality, not worst case. **4. Running as privileged.** As noted above, privileged containers are not needed. ` CAP_BPF ` + ` CAP_PERFMON ` is sufficient for observability probes. ` CAP_NET_ADMIN ` if you're doing network stuff. That's it. ## Where This Is Heading The eBPF ecosystem is maturing fast. The major shifts to watch: - **eBPF on Windows.** Microsoft has been building eBPF support for Windows, and it's reaching production readiness. Cross-platform eBPF probes are becoming viable. - **BPF Arena.** New map type that gives eBPF programs a shared memory arena, enabling more complex data structures (trees, graphs) in kernel space. - **Scheduling class programs.** BPF programs that can influence the kernel's CPU scheduler. Google's sched\_ext is already using this in production. For platform teams, the message is clear: if you're still running sidecar-based service meshes primarily for observability, you're paying a 5-10x overhead tax. eBPF gives you deeper visibility at a fraction of the cost. The tooling has matured to the point where you don't need kernel developers on staff to deploy it. Cilium and Tetragon handle 90% of use cases out of the box. The remaining 10%, custom probes for your specific domain, is where the real competitive advantage lies. And writing eBPF is not that hard once the constraints are internalized. The verifier is your friend, not your enemy. It's the reason you can deploy this stuff in production without fear. ## Frequently asked questions ### Does eBPF add significant overhead to production workloads? When implemented correctly, eBPF adds negligible overhead, typically under 1% CPU. The key is filtering events in-kernel rather than shipping everything to userspace. A poorly written probe that copies large buffers on every syscall can add 5-10% overhead. ### Can I use eBPF without writing C code? Yes. Tools like Cilium, Tetragon, and bpftrace provide high-level abstractions. Tetragon uses YAML TracingPolicies, and bpftrace has an awk-like scripting language. For custom needs, libbpf-rs (Rust) and cilium/ebpf (Go) provide safe wrappers. ### What kernel version do I need for eBPF observability? Minimum kernel 5.4 for BTF support (CO-RE portability). Kernel 5.8+ for ring buffers. Kernel 5.11+ for bpf\_snprintf and timer support. Most managed Kubernetes offerings (EKS, GKE, AKS) run 5.10+ as of 2025. ## Tips 1. **BPF ring buffer vs perf buffer:** Always use ` BPF_MAP_TYPE_RINGBUF ` over ` BPF_MAP_TYPE_PERF_EVENT_ARRAY ` for new programs. Ring buffers share memory across CPUs, reduce memory overhead by ~50%, and support variable-length records. The only exception is per-CPU event ordering where you need strict per-CPU semantics. 2. **Tail call chaining for complex probes:** Single eBPF programs have a 1M instruction limit (raised from 4096). For complex observability logic, use ` bpf_tail_call() ` to chain up to 33 programs via ` BPF_MAP_TYPE_PROG_ARRAY `. Each tail call resets the instruction counter, giving you effectively unlimited program size. 3. **BTF-enabled CO-RE portability:** Compile your eBPF probes with ` -g -target bpf ` and use ` __builtin_preserve_access_index() ` wrappers on struct accesses. This embeds BTF relocations so the same compiled .o file runs across kernel versions 5.4+ without recompilation. Test with ` bpftool btf dump file /sys/kernel/btf/vmlinux `. 4. **Tetragon TracingPolicy selectors:** Don't attach to every syscall. Use Tetragon's ` matchArgs ` and ` matchReturnArgs ` selectors to filter in-kernel: ` matchArgs: [{index: 0, operator: Equal, values: ["/etc/shadow"]}] `. This drops irrelevant events before they hit userspace, reducing CPU overhead from ~5% to \<0.1%. - ebpf - kubernetes - cilium - tetragon - observability - linux-kernel - security-monitoring [Back to Blog]()