eBPF in Production: Building Kernel-Level Observability for Kubernetes Without the Overhead

What eBPF Is: Technical Overview

At its core, eBPF is straightforward: it's a virtual machine inside the Linux kernel that lets you run sandboxed programs at specific hook points without modifying kernel source or loading kernel modules. The "extended" in eBPF (originally Berkeley Packet Filter) refers to the fact that it's been expanded far beyond packet filtering into a general-purpose kernel instrumentation framework.

The key constraints that make it safe:

1. Programs must pass the BPF verifier (proves termination, memory safety)
2. No unbounded loops (the verifier must prove all loops terminate)
3. Limited stack size (512 bytes)
4. No sleeping or blocking in most program types
5. Helper functions are the only way to interact with kernel state

These constraints sound limiting, but they guarantee something crucial: an eBPF program cannot crash your kernel. Period. The verifier rejects anything that could cause undefined behavior. This is what makes eBPF viable for production, unlike kprobes or SystemTap scripts that can bring down a machine.

The Observability Stack in 2026

The following shows what a modern eBPF-based observability stack looks like in production Kubernetes:

┌─────────────────────────────────────┐
│ Grafana / UI │
├─────────────────────────────────────┤
│ Prometheus / OpenTelemetry │
├──────────┬──────────┬───────────────┤
│ Hubble │ Tetragon │ Custom Probes │
├──────────┴──────────┴───────────────┤
│ Cilium (CNI + eBPF) │
├─────────────────────────────────────┤
│ Linux Kernel (5.10+) │
└─────────────────────────────────────┘

Cilium: The Network Layer

Cilium replaces kube-proxy and iptables with eBPF programs attached to TC (traffic control) hooks. Instead of traversing a chain of iptables rules for every packet, Cilium does a single BPF map lookup. The performance difference is dramatic:

# iptables-based kube-proxy (5,000 services):
# ~20ms per connection setup (rule traversal)
# ~15% CPU overhead on proxy node

# Cilium eBPF (5,000 services):
# ~0.2ms per connection setup (hash map lookup)
# ~2% CPU overhead on proxy node

But the real win isn't raw throughput, it's observability. Cilium's Hubble component gives you L3/L4/L7 visibility into every packet flowing through the cluster, with zero instrumentation in your application code.

# Watch all HTTP requests between services in real-time
hubble observe --namespace production \
 --protocol http \
 --verdict FORWARDED \
 -o json | jq '{src: .source.labels, dst: .destination.labels, http: .l7.http}'

This is observing actual kernel-level packet flow, not sampling. Every single packet. And because it's eBPF, the overhead is measured in microseconds per packet.

Tetragon: Security Observability

Tetragon is where things get really interesting from a security monitoring perspective. It attaches eBPF programs to kernel functions (kprobes, tracepoints, LSM hooks) and gives you a declarative YAML interface to define what you want to observe.

The following is a TracingPolicy that monitors for suspicious file access:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: sensitive-file-access
spec:
 kprobes:
 - call: "security_file_open"
 syscall: false
 args:
 - index: 0
 type: "file"
 selectors:
 - matchArgs:
 - index: 0
 operator: "Prefix"
 values:
 - "/etc/shadow"
 - "/etc/passwd"
 - "/root/.ssh"
 - "/var/run/secrets/kubernetes.io"
 matchActions:
 - action: Sigkill # Kill the process immediately
 - action: Post # Also send event to userspace

This policy runs entirely in kernel space. When a process opens /etc/shadow, the eBPF program fires, checks the path, and can kill the process before the open() syscall returns. No userspace roundtrip. No TOCTOU race condition. The enforcement happens at the kernel level.

In one deployment scenario, a cluster running a third-party vendor's container suspected of exfiltrating secrets was monitored with this policy. Within an hour, Tetragon caught the container trying to read Kubernetes service account tokens from the mounted secret volume. The process was killed before it could read the first byte.

Writing Custom Probes

Sometimes the off-the-shelf tools are not enough. Here is how to build custom eBPF probes for specific observability needs.

Example: TCP Connection Latency Histogram

This probe measures the time between connect() being called and the TCP handshake completing:

// tcp_connect_latency.bpf.c
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>

struct {
 __uint(type, BPF_MAP_TYPE_HASH);
 __uint(max_entries, 65536);
 __type(key, u64); // pid_tgid
 __type(value, u64); // start timestamp
} connect_start SEC(".maps");

struct {
 __uint(type, BPF_MAP_TYPE_RINGBUF);
 __uint(max_entries, 256 * 1024);
} events SEC(".maps");

struct event {
 u32 pid;
 u32 dport;
 u64 latency_ns;
 u32 daddr;
};

SEC("kprobe/tcp_v4_connect")
int BPF_KPROBE(tcp_v4_connect_entry, struct sock *sk)
{
 u64 pid_tgid = bpf_get_current_pid_tgid();
 u64 ts = bpf_ktime_get_ns();
 bpf_map_update_elem(&connect_start, &pid_tgid, &ts, BPF_ANY);
 return 0;
}

SEC("kprobe/tcp_finish_connect")
int BPF_KPROBE(tcp_finish_connect_entry, struct sock *sk)
{
 u64 pid_tgid = bpf_get_current_pid_tgid();
 u64 *start_ts = bpf_map_lookup_elem(&connect_start, &pid_tgid);
 if (!start_ts)
 return 0;

 u64 latency = bpf_ktime_get_ns() - *start_ts;
 bpf_map_delete_elem(&connect_start, &pid_tgid);

 struct event *e = bpf_ringbuf_reserve(&events, sizeof(*e), 0);
 if (!e)
 return 0;

 e->pid = pid_tgid >> 32;
 e->latency_ns = latency;
 e->dport = BPF_CORE_READ(sk, __sk_common.skc_dport);
 e->daddr = BPF_CORE_READ(sk, __sk_common.skc_daddr);

 bpf_ringbuf_submit(e, 0);
 return 0;
}

char LICENSE[] SEC("license") = "GPL";

The Go userspace component that consumes these events:

package main

import (
 "bytes"
 "encoding/binary"
 "fmt"
 "log"
 "net"

 "github.com/cilium/ebpf/link"
 "github.com/cilium/ebpf/ringbuf"
 "github.com/cilium/ebpf/rlimit"
)

//go:generate go run github.com/cilium/ebpf/cmd/bpf2go -cc clang tcpLatency tcp_connect_latency.bpf.c

type Event struct {
 Pid uint32
 Dport uint32
 LatencyNs uint64
 Daddr uint32
}

func main() {
 if err := rlimit.RemoveMemlock(); err != nil {
 log.Fatal(err)
 }

 objs := tcpLatencyObjects{}
 if err := loadTcpLatencyObjects(&objs, nil); err != nil {
 log.Fatalf("loading objects: %v", err)
 }
 defer objs.Close()

 kp1, err := link.Kprobe("tcp_v4_connect", objs.TcpV4ConnectEntry, nil)
 if err != nil {
 log.Fatalf("opening kprobe: %v", err)
 }
 defer kp1.Close()

 kp2, err := link.Kprobe("tcp_finish_connect", objs.TcpFinishConnectEntry, nil)
 if err != nil {
 log.Fatalf("opening kprobe: %v", err)
 }
 defer kp2.Close()

 rd, err := ringbuf.NewReader(objs.Events)
 if err != nil {
 log.Fatalf("opening ringbuf reader: %v", err)
 }
 defer rd.Close()

 fmt.Println("Tracing TCP connect latency... Ctrl+C to stop")

 for {
 record, err := rd.Read()
 if err != nil {
 log.Printf("reading ringbuf: %v", err)
 continue
 }

 var event Event
 if err := binary.Read(bytes.NewBuffer(record.RawSample), binary.LittleEndian, &event); err != nil {
 continue
 }

 addr := make(net.IP, 4)
 binary.LittleEndian.PutUint32(addr, event.Daddr)
 fmt.Printf("pid=%d dst=%s:%d latency=%.2fms\n",
 event.Pid, addr, ntohs(uint16(event.Dport)),
 float64(event.LatencyNs)/1e6)
 }
}

Deploying Custom Probes in Kubernetes

The tricky part is not writing the probe, it is deploying it safely across a fleet. Here is a proven deployment pattern:

apiVersion: apps/v1
kind: DaemonSet
metadata:
 name: tcp-latency-probe
 namespace: observability
spec:
 selector:
 matchLabels:
 app: tcp-latency-probe
 template:
 metadata:
 labels:
 app: tcp-latency-probe
 spec:
 hostPID: true
 hostNetwork: true
 containers:
 - name: probe
 image: ghcr.io/myorg/tcp-latency-probe:v1.2.0
 securityContext:
 privileged: false
 capabilities:
 add:
 - BPF
 - PERFMON
 - SYS_RESOURCE
 volumeMounts:
 - name: bpf-fs
 mountPath: /sys/fs/bpf
 - name: debugfs
 mountPath: /sys/kernel/debug
 readOnly: true
 volumes:
 - name: bpf-fs
 hostPath:
 path: /sys/fs/bpf
 - name: debugfs
 hostPath:
 path: /sys/kernel/debug

Notice we're not using privileged: true. The BPF, PERFMON, and SYS_RESOURCE capabilities are sufficient. Running eBPF probes as privileged containers is a common anti-pattern that gives the probe unnecessary access.

Performance Characteristics

Here are real numbers from a production cluster (150 nodes, ~3000 pods):

Baseline (no eBPF observability):
 Node CPU utilization: 62% average
 p99 request latency: 45ms

With Cilium + Hubble (L7 visibility):
 Node CPU utilization: 63.8% average (+1.8%)
 p99 request latency: 45.2ms (+0.4%)

With Tetragon (security monitoring, 12 TracingPolicies):
 Node CPU utilization: 64.1% average (+0.3% additional)
 p99 request latency: 45.2ms (no change)

With custom TCP latency probe (all connections):
 Node CPU utilization: 64.3% average (+0.2% additional)
 p99 request latency: 45.3ms (+0.1%)

Total overhead: ~2.3% CPU, ~0.3ms p99 latency

Compare this to a traditional sidecar-based service mesh (Istio with Envoy):

With Istio sidecar (L7 visibility):
 Node CPU utilization: 71% average (+9%)
 p99 request latency: 52ms (+7ms)
 Memory overhead: ~50MB per pod (sidecar)

The difference is staggering. eBPF operates at the kernel level, avoiding the context switches and memory copies that sidecar proxies require. You get better observability with an order of magnitude less overhead.

Common Pitfalls

After deploying eBPF across multiple production environments, these are the mistakes teams make repeatedly:

1. Copying too much data to userspace. Every bpf_ringbuf_submit() or bpf_perf_event_output() involves a memory copy. If you're tracing every syscall and copying full argument buffers, you'll burn CPU. Filter and aggregate in-kernel whenever possible.

2. Not using CO-RE. If you're compiling eBPF programs against specific kernel headers and shipping different binaries per kernel version, you're doing it wrong. BPF CO-RE (Compile Once, Run Everywhere) with BTF eliminates this entirely.

3. Ignoring map sizing. BPF maps are allocated at load time. A BPF_MAP_TYPE_HASH with max_entries=1000000 allocates memory upfront on each CPU. Size your maps based on actual cardinality, not worst case.

4. Running as privileged. As noted above, privileged containers are not needed. CAP_BPF + CAP_PERFMON is sufficient for observability probes. CAP_NET_ADMIN if you're doing network stuff. That's it.

Where This Is Heading

The eBPF ecosystem is maturing fast. The major shifts to watch:

• eBPF on Windows. Microsoft has been building eBPF support for Windows, and it's reaching production readiness. Cross-platform eBPF probes are becoming viable.
• BPF Arena. New map type that gives eBPF programs a shared memory arena, enabling more complex data structures (trees, graphs) in kernel space.
• Scheduling class programs. BPF programs that can influence the kernel's CPU scheduler. Google's sched_ext is already using this in production.

For platform teams, the message is clear: if you're still running sidecar-based service meshes primarily for observability, you're paying a 5-10x overhead tax. eBPF gives you deeper visibility at a fraction of the cost. The tooling has matured to the point where you don't need kernel developers on staff to deploy it. Cilium and Tetragon handle 90% of use cases out of the box.

The remaining 10%, custom probes for your specific domain, is where the real competitive advantage lies. And writing eBPF is not that hard once the constraints are internalized. The verifier is your friend, not your enemy. It's the reason you can deploy this stuff in production without fear.