{"schemaVersion":1,"id":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes","url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes","title":"eBPF in Production: Building Kernel-Level Observability for Kubernetes Without the Overhead","description":"How to use eBPF for production-grade observability in Kubernetes clusters. Cilium, Tetragon, custom kprobes, and why traditional monitoring is leaving performance on the table.","language":"en","datePublished":"2025-10-25","dateModified":null,"contentFormat":"text/markdown","content":"Infrastructure\n\n# eBPF in Production: Building Kernel-Level Observability for Kubernetes Without the Overhead\n\n2025-10-258 min read\n\nLucio Durán\nEngineering Manager & AI Solutions Architect\n\nAlso available in: [Español](<https://lucioduran.com/es/blog/ebpf-observability-production-kubernetes>)\n\n## What eBPF Is: Technical Overview\n\nAt its core, eBPF is straightforward: it's a virtual machine inside the Linux kernel that lets you run sandboxed programs at specific hook points without modifying kernel source or loading kernel modules. The \"extended\" in eBPF (originally Berkeley Packet Filter) refers to the fact that it's been expanded far beyond packet filtering into a general-purpose kernel instrumentation framework.\n\nThe key constraints that make it safe:\n\n```\n1. Programs must pass the BPF verifier (proves termination, memory safety)\n2. No unbounded loops (the verifier must prove all loops terminate)\n3. Limited stack size (512 bytes)\n4. No sleeping or blocking in most program types\n5. Helper functions are the only way to interact with kernel state\n```\n\nThese constraints sound limiting, but they guarantee something crucial: **an eBPF program cannot crash your kernel**. Period. The verifier rejects anything that could cause undefined behavior. This is what makes eBPF viable for production, unlike kprobes or SystemTap scripts that can bring down a machine.\n\n## The Observability Stack in 2026\n\nThe following shows what a modern eBPF-based observability stack looks like in production Kubernetes:\n\n```\n┌─────────────────────────────────────┐\n│ Grafana / UI │\n├─────────────────────────────────────┤\n│ Prometheus / OpenTelemetry │\n├──────────┬──────────┬───────────────┤\n│ Hubble │ Tetragon │ Custom Probes │\n├──────────┴──────────┴───────────────┤\n│ Cilium (CNI + eBPF) │\n├─────────────────────────────────────┤\n│ Linux Kernel (5.10+) │\n└─────────────────────────────────────┘\n```\n\n### Cilium: The Network Layer\n\n[Cilium](<https://cilium.io/>) replaces kube-proxy and iptables with eBPF programs attached to TC (traffic control) hooks. Instead of traversing a chain of iptables rules for every packet, Cilium does a single BPF map lookup. The performance difference is dramatic:\n\n```\n# iptables-based kube-proxy (5,000 services):\n# ~20ms per connection setup (rule traversal)\n# ~15% CPU overhead on proxy node\n\n# Cilium eBPF (5,000 services):\n# ~0.2ms per connection setup (hash map lookup)\n# ~2% CPU overhead on proxy node\n```\n\nBut the real win isn't raw throughput, it's observability. Cilium's Hubble component gives you L3/L4/L7 visibility into every packet flowing through the cluster, with zero instrumentation in your application code.\n\n```bash\n# Watch all HTTP requests between services in real-time\nhubble observe --namespace production \\\n --protocol http \\\n --verdict FORWARDED \\\n -o json | jq '{src: .source.labels, dst: .destination.labels, http: .l7.http}'\n```\n\nThis is observing actual kernel-level packet flow, not sampling. Every single packet. And because it's eBPF, the overhead is measured in microseconds per packet.\n\n### Tetragon: Security Observability\n\n[Tetragon](<https://tetragon.io/>) is where things get really interesting from a security monitoring perspective. It attaches eBPF programs to kernel functions (kprobes, tracepoints, LSM hooks) and gives you a declarative YAML interface to define what you want to observe.\n\nThe following is a TracingPolicy that monitors for suspicious file access:\n\n```yaml\napiVersion: cilium.io/v1alpha1\nkind: TracingPolicy\nmetadata:\n name: sensitive-file-access\nspec:\n kprobes:\n - call: \"security_file_open\"\n syscall: false\n args:\n - index: 0\n type: \"file\"\n selectors:\n - matchArgs:\n - index: 0\n operator: \"Prefix\"\n values:\n - \"/etc/shadow\"\n - \"/etc/passwd\"\n - \"/root/.ssh\"\n - \"/var/run/secrets/kubernetes.io\"\n matchActions:\n - action: Sigkill # Kill the process immediately\n - action: Post # Also send event to userspace\n```\n\nThis policy runs entirely in kernel space. When a process opens ` /etc/shadow `, the eBPF program fires, checks the path, and can **kill the process before the ` open() ` syscall returns**. No userspace roundtrip. No TOCTOU race condition. The enforcement happens at the kernel level.\n\nIn one deployment scenario, a cluster running a third-party vendor's container suspected of exfiltrating secrets was monitored with this policy. Within an hour, Tetragon caught the container trying to read Kubernetes service account tokens from the mounted secret volume. The process was killed before it could read the first byte.\n\n## Writing Custom Probes\n\nSometimes the off-the-shelf tools are not enough. Here is how to build custom eBPF probes for specific observability needs.\n\n### Example: TCP Connection Latency Histogram\n\nThis probe measures the time between ` connect() ` being called and the TCP handshake completing:\n\n```c\n// tcp_connect_latency.bpf.c\n#include \"vmlinux.h\"\n#include <bpf/bpf_helpers.h>\n#include <bpf/bpf_tracing.h>\n#include <bpf/bpf_core_read.h>\n\nstruct {\n __uint(type, BPF_MAP_TYPE_HASH);\n __uint(max_entries, 65536);\n __type(key, u64); // pid_tgid\n __type(value, u64); // start timestamp\n} connect_start SEC(\".maps\");\n\nstruct {\n __uint(type, BPF_MAP_TYPE_RINGBUF);\n __uint(max_entries, 256 * 1024);\n} events SEC(\".maps\");\n\nstruct event {\n u32 pid;\n u32 dport;\n u64 latency_ns;\n u32 daddr;\n};\n\nSEC(\"kprobe/tcp_v4_connect\")\nint BPF_KPROBE(tcp_v4_connect_entry, struct sock *sk)\n{\n u64 pid_tgid = bpf_get_current_pid_tgid();\n u64 ts = bpf_ktime_get_ns();\n bpf_map_update_elem(&connect_start, &pid_tgid, &ts, BPF_ANY);\n return 0;\n}\n\nSEC(\"kprobe/tcp_finish_connect\")\nint BPF_KPROBE(tcp_finish_connect_entry, struct sock *sk)\n{\n u64 pid_tgid = bpf_get_current_pid_tgid();\n u64 *start_ts = bpf_map_lookup_elem(&connect_start, &pid_tgid);\n if (!start_ts)\n return 0;\n\n u64 latency = bpf_ktime_get_ns() - *start_ts;\n bpf_map_delete_elem(&connect_start, &pid_tgid);\n\n struct event *e = bpf_ringbuf_reserve(&events, sizeof(*e), 0);\n if (!e)\n return 0;\n\n e->pid = pid_tgid >> 32;\n e->latency_ns = latency;\n e->dport = BPF_CORE_READ(sk, __sk_common.skc_dport);\n e->daddr = BPF_CORE_READ(sk, __sk_common.skc_daddr);\n\n bpf_ringbuf_submit(e, 0);\n return 0;\n}\n\nchar LICENSE[] SEC(\"license\") = \"GPL\";\n```\n\nThe Go userspace component that consumes these events:\n\n```go\npackage main\n\nimport (\n \"bytes\"\n \"encoding/binary\"\n \"fmt\"\n \"log\"\n \"net\"\n\n \"github.com/cilium/ebpf/link\"\n \"github.com/cilium/ebpf/ringbuf\"\n \"github.com/cilium/ebpf/rlimit\"\n)\n\n//go:generate go run github.com/cilium/ebpf/cmd/bpf2go -cc clang tcpLatency tcp_connect_latency.bpf.c\n\ntype Event struct {\n Pid uint32\n Dport uint32\n LatencyNs uint64\n Daddr uint32\n}\n\nfunc main() {\n if err := rlimit.RemoveMemlock(); err != nil {\n log.Fatal(err)\n }\n\n objs := tcpLatencyObjects{}\n if err := loadTcpLatencyObjects(&objs, nil); err != nil {\n log.Fatalf(\"loading objects: %v\", err)\n }\n defer objs.Close()\n\n kp1, err := link.Kprobe(\"tcp_v4_connect\", objs.TcpV4ConnectEntry, nil)\n if err != nil {\n log.Fatalf(\"opening kprobe: %v\", err)\n }\n defer kp1.Close()\n\n kp2, err := link.Kprobe(\"tcp_finish_connect\", objs.TcpFinishConnectEntry, nil)\n if err != nil {\n log.Fatalf(\"opening kprobe: %v\", err)\n }\n defer kp2.Close()\n\n rd, err := ringbuf.NewReader(objs.Events)\n if err != nil {\n log.Fatalf(\"opening ringbuf reader: %v\", err)\n }\n defer rd.Close()\n\n fmt.Println(\"Tracing TCP connect latency... Ctrl+C to stop\")\n\n for {\n record, err := rd.Read()\n if err != nil {\n log.Printf(\"reading ringbuf: %v\", err)\n continue\n }\n\n var event Event\n if err := binary.Read(bytes.NewBuffer(record.RawSample), binary.LittleEndian, &event); err != nil {\n continue\n }\n\n addr := make(net.IP, 4)\n binary.LittleEndian.PutUint32(addr, event.Daddr)\n fmt.Printf(\"pid=%d dst=%s:%d latency=%.2fms\\n\",\n event.Pid, addr, ntohs(uint16(event.Dport)),\n float64(event.LatencyNs)/1e6)\n }\n}\n```\n\n### Deploying Custom Probes in Kubernetes\n\nThe tricky part is not writing the probe, it is deploying it safely across a fleet. Here is a proven deployment pattern:\n\n```yaml\napiVersion: apps/v1\nkind: DaemonSet\nmetadata:\n name: tcp-latency-probe\n namespace: observability\nspec:\n selector:\n matchLabels:\n app: tcp-latency-probe\n template:\n metadata:\n labels:\n app: tcp-latency-probe\n spec:\n hostPID: true\n hostNetwork: true\n containers:\n - name: probe\n image: ghcr.io/myorg/tcp-latency-probe:v1.2.0\n securityContext:\n privileged: false\n capabilities:\n add:\n - BPF\n - PERFMON\n - SYS_RESOURCE\n volumeMounts:\n - name: bpf-fs\n mountPath: /sys/fs/bpf\n - name: debugfs\n mountPath: /sys/kernel/debug\n readOnly: true\n volumes:\n - name: bpf-fs\n hostPath:\n path: /sys/fs/bpf\n - name: debugfs\n hostPath:\n path: /sys/kernel/debug\n```\n\nNotice we're not using ` privileged: true `. The ` BPF `, ` PERFMON `, and ` SYS_RESOURCE ` capabilities are sufficient. Running eBPF probes as privileged containers is a common anti-pattern that gives the probe unnecessary access.\n\n## Performance Characteristics\n\nHere are real numbers from a production cluster (150 nodes, ~3000 pods):\n\n```\nBaseline (no eBPF observability):\n Node CPU utilization: 62% average\n p99 request latency: 45ms\n\nWith Cilium + Hubble (L7 visibility):\n Node CPU utilization: 63.8% average (+1.8%)\n p99 request latency: 45.2ms (+0.4%)\n\nWith Tetragon (security monitoring, 12 TracingPolicies):\n Node CPU utilization: 64.1% average (+0.3% additional)\n p99 request latency: 45.2ms (no change)\n\nWith custom TCP latency probe (all connections):\n Node CPU utilization: 64.3% average (+0.2% additional)\n p99 request latency: 45.3ms (+0.1%)\n\nTotal overhead: ~2.3% CPU, ~0.3ms p99 latency\n```\n\nCompare this to a traditional sidecar-based service mesh (Istio with Envoy):\n\n```\nWith Istio sidecar (L7 visibility):\n Node CPU utilization: 71% average (+9%)\n p99 request latency: 52ms (+7ms)\n Memory overhead: ~50MB per pod (sidecar)\n```\n\nThe difference is staggering. eBPF operates at the kernel level, avoiding the context switches and memory copies that sidecar proxies require. You get better observability with an order of magnitude less overhead.\n\n## Common Pitfalls\n\nAfter deploying eBPF across multiple production environments, these are the mistakes teams make repeatedly:\n\n**1. Copying too much data to userspace.** Every ` bpf_ringbuf_submit() ` or ` bpf_perf_event_output() ` involves a memory copy. If you're tracing every syscall and copying full argument buffers, you'll burn CPU. Filter and aggregate in-kernel whenever possible.\n\n**2. Not using CO-RE.** If you're compiling eBPF programs against specific kernel headers and shipping different binaries per kernel version, you're doing it wrong. BPF CO-RE (Compile Once, Run Everywhere) with BTF eliminates this entirely.\n\n**3. Ignoring map sizing.** BPF maps are allocated at load time. A ` BPF_MAP_TYPE_HASH ` with ` max_entries=1000000 ` allocates memory upfront on each CPU. Size your maps based on actual cardinality, not worst case.\n\n**4. Running as privileged.** As noted above, privileged containers are not needed. ` CAP_BPF ` + ` CAP_PERFMON ` is sufficient for observability probes. ` CAP_NET_ADMIN ` if you're doing network stuff. That's it.\n\n## Where This Is Heading\n\nThe eBPF ecosystem is maturing fast. The major shifts to watch:\n\n- **eBPF on Windows.** Microsoft has been building eBPF support for Windows, and it's reaching production readiness. Cross-platform eBPF probes are becoming viable.\n- **BPF Arena.** New map type that gives eBPF programs a shared memory arena, enabling more complex data structures (trees, graphs) in kernel space.\n- **Scheduling class programs.** BPF programs that can influence the kernel's CPU scheduler. Google's sched\\_ext is already using this in production.\n\nFor platform teams, the message is clear: if you're still running sidecar-based service meshes primarily for observability, you're paying a 5-10x overhead tax. eBPF gives you deeper visibility at a fraction of the cost. The tooling has matured to the point where you don't need kernel developers on staff to deploy it. Cilium and Tetragon handle 90% of use cases out of the box.\n\nThe remaining 10%, custom probes for your specific domain, is where the real competitive advantage lies. And writing eBPF is not that hard once the constraints are internalized. The verifier is your friend, not your enemy. It's the reason you can deploy this stuff in production without fear.\n\n## Frequently asked questions\n\n### Does eBPF add significant overhead to production workloads?\n\nWhen implemented correctly, eBPF adds negligible overhead, typically under 1% CPU. The key is filtering events in-kernel rather than shipping everything to userspace. A poorly written probe that copies large buffers on every syscall can add 5-10% overhead.\n\n### Can I use eBPF without writing C code?\n\nYes. Tools like Cilium, Tetragon, and bpftrace provide high-level abstractions. Tetragon uses YAML TracingPolicies, and bpftrace has an awk-like scripting language. For custom needs, libbpf-rs (Rust) and cilium/ebpf (Go) provide safe wrappers.\n\n### What kernel version do I need for eBPF observability?\n\nMinimum kernel 5.4 for BTF support (CO-RE portability). Kernel 5.8+ for ring buffers. Kernel 5.11+ for bpf\\_snprintf and timer support. Most managed Kubernetes offerings (EKS, GKE, AKS) run 5.10+ as of 2025.\n\n## Tips\n\n1. **BPF ring buffer vs perf buffer:** Always use ` BPF_MAP_TYPE_RINGBUF ` over ` BPF_MAP_TYPE_PERF_EVENT_ARRAY ` for new programs. Ring buffers share memory across CPUs, reduce memory overhead by ~50%, and support variable-length records. The only exception is per-CPU event ordering where you need strict per-CPU semantics.\n2. **Tail call chaining for complex probes:** Single eBPF programs have a 1M instruction limit (raised from 4096). For complex observability logic, use ` bpf_tail_call() ` to chain up to 33 programs via ` BPF_MAP_TYPE_PROG_ARRAY `. Each tail call resets the instruction counter, giving you effectively unlimited program size.\n3. **BTF-enabled CO-RE portability:** Compile your eBPF probes with ` -g -target bpf ` and use ` __builtin_preserve_access_index() ` wrappers on struct accesses. This embeds BTF relocations so the same compiled .o file runs across kernel versions 5.4+ without recompilation. Test with ` bpftool btf dump file /sys/kernel/btf/vmlinux `.\n4. **Tetragon TracingPolicy selectors:** Don't attach to every syscall. Use Tetragon's ` matchArgs ` and ` matchReturnArgs ` selectors to filter in-kernel: ` matchArgs: [{index: 0, operator: Equal, values: [\"/etc/shadow\"]}] `. This drops irrelevant events before they hit userspace, reducing CPU overhead from ~5% to \\<0.1%.\n\n- ebpf\n- kubernetes\n- cilium\n- tetragon\n- observability\n- linux-kernel\n- security-monitoring\n\n[Back to Blog](<https://lucioduran.com/blog>)","headings":[{"level":1,"text":"eBPF in Production: Building Kernel-Level Observability for Kubernetes Without the Overhead","url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#title"},{"level":2,"text":"What eBPF Is: Technical Overview","url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#what-ebpf-is-technical-overview"},{"level":2,"text":"The Observability Stack in 2026","url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#the-observability-stack-in-2026"},{"level":3,"text":"Cilium: The Network Layer","url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#cilium-the-network-layer"},{"level":3,"text":"Tetragon: Security Observability","url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#tetragon-security-observability"},{"level":2,"text":"Writing Custom Probes","url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#writing-custom-probes"},{"level":3,"text":"Example: TCP Connection Latency Histogram","url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#example-tcp-connection-latency-histogram"},{"level":3,"text":"Deploying Custom Probes in Kubernetes","url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#deploying-custom-probes-in-kubernetes"},{"level":2,"text":"Performance Characteristics","url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#performance-characteristics"},{"level":2,"text":"Common Pitfalls","url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#common-pitfalls"},{"level":2,"text":"Where This Is Heading","url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#where-this-is-heading"},{"level":2,"text":"Frequently asked questions","url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#questions"},{"level":3,"text":"Does eBPF add significant overhead to production workloads?","url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#question-1"},{"level":3,"text":"Can I use eBPF without writing C code?","url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#question-2"},{"level":3,"text":"What kernel version do I need for eBPF observability?","url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#question-3"},{"level":2,"text":"Tips","url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#article-tips"}],"sections":[{"heading":"eBPF in Production: Building Kernel-Level Observability for Kubernetes Without the Overhead","level":1,"url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes","content":"Infrastructure"},{"heading":"eBPF in Production: Building Kernel-Level Observability for Kubernetes Without the Overhead","level":1,"url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#title","content":"2025-10-258 min read\n\nLucio Durán\nEngineering Manager & AI Solutions Architect\n\nAlso available in: [Español](<https://lucioduran.com/es/blog/ebpf-observability-production-kubernetes>)"},{"heading":"What eBPF Is: Technical Overview","level":2,"url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#what-ebpf-is-technical-overview","content":"At its core, eBPF is straightforward: it's a virtual machine inside the Linux kernel that lets you run sandboxed programs at specific hook points without modifying kernel source or loading kernel modules. The \"extended\" in eBPF (originally Berkeley Packet Filter) refers to the fact that it's been expanded far beyond packet filtering into a general-purpose kernel instrumentation framework.\n\nThe key constraints that make it safe:\n\n```\n1. Programs must pass the BPF verifier (proves termination, memory safety)\n2. No unbounded loops (the verifier must prove all loops terminate)\n3. Limited stack size (512 bytes)\n4. No sleeping or blocking in most program types\n5. Helper functions are the only way to interact with kernel state\n```\n\nThese constraints sound limiting, but they guarantee something crucial: **an eBPF program cannot crash your kernel**. Period. The verifier rejects anything that could cause undefined behavior. This is what makes eBPF viable for production, unlike kprobes or SystemTap scripts that can bring down a machine."},{"heading":"The Observability Stack in 2026","level":2,"url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#the-observability-stack-in-2026","content":"The following shows what a modern eBPF-based observability stack looks like in production Kubernetes:\n\n```\n┌─────────────────────────────────────┐\n│ Grafana / UI │\n├─────────────────────────────────────┤\n│ Prometheus / OpenTelemetry │\n├──────────┬──────────┬───────────────┤\n│ Hubble │ Tetragon │ Custom Probes │\n├──────────┴──────────┴───────────────┤\n│ Cilium (CNI + eBPF) │\n├─────────────────────────────────────┤\n│ Linux Kernel (5.10+) │\n└─────────────────────────────────────┘\n```"},{"heading":"Cilium: The Network Layer","level":3,"url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#cilium-the-network-layer","content":"[Cilium](<https://cilium.io/>) replaces kube-proxy and iptables with eBPF programs attached to TC (traffic control) hooks. Instead of traversing a chain of iptables rules for every packet, Cilium does a single BPF map lookup. The performance difference is dramatic:\n\n```\n# iptables-based kube-proxy (5,000 services):\n# ~20ms per connection setup (rule traversal)\n# ~15% CPU overhead on proxy node\n\n# Cilium eBPF (5,000 services):\n# ~0.2ms per connection setup (hash map lookup)\n# ~2% CPU overhead on proxy node\n```\n\nBut the real win isn't raw throughput, it's observability. Cilium's Hubble component gives you L3/L4/L7 visibility into every packet flowing through the cluster, with zero instrumentation in your application code.\n\n```bash\n# Watch all HTTP requests between services in real-time\nhubble observe --namespace production \\\n --protocol http \\\n --verdict FORWARDED \\\n -o json | jq '{src: .source.labels, dst: .destination.labels, http: .l7.http}'\n```\n\nThis is observing actual kernel-level packet flow, not sampling. Every single packet. And because it's eBPF, the overhead is measured in microseconds per packet."},{"heading":"Tetragon: Security Observability","level":3,"url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#tetragon-security-observability","content":"[Tetragon](<https://tetragon.io/>) is where things get really interesting from a security monitoring perspective. It attaches eBPF programs to kernel functions (kprobes, tracepoints, LSM hooks) and gives you a declarative YAML interface to define what you want to observe.\n\nThe following is a TracingPolicy that monitors for suspicious file access:\n\n```yaml\napiVersion: cilium.io/v1alpha1\nkind: TracingPolicy\nmetadata:\n name: sensitive-file-access\nspec:\n kprobes:\n - call: \"security_file_open\"\n syscall: false\n args:\n - index: 0\n type: \"file\"\n selectors:\n - matchArgs:\n - index: 0\n operator: \"Prefix\"\n values:\n - \"/etc/shadow\"\n - \"/etc/passwd\"\n - \"/root/.ssh\"\n - \"/var/run/secrets/kubernetes.io\"\n matchActions:\n - action: Sigkill # Kill the process immediately\n - action: Post # Also send event to userspace\n```\n\nThis policy runs entirely in kernel space. When a process opens ` /etc/shadow `, the eBPF program fires, checks the path, and can **kill the process before the ` open() ` syscall returns**. No userspace roundtrip. No TOCTOU race condition. The enforcement happens at the kernel level.\n\nIn one deployment scenario, a cluster running a third-party vendor's container suspected of exfiltrating secrets was monitored with this policy. Within an hour, Tetragon caught the container trying to read Kubernetes service account tokens from the mounted secret volume. The process was killed before it could read the first byte."},{"heading":"Writing Custom Probes","level":2,"url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#writing-custom-probes","content":"Sometimes the off-the-shelf tools are not enough. Here is how to build custom eBPF probes for specific observability needs."},{"heading":"Example: TCP Connection Latency Histogram","level":3,"url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#example-tcp-connection-latency-histogram","content":"This probe measures the time between ` connect() ` being called and the TCP handshake completing:\n\n```c\n// tcp_connect_latency.bpf.c\n#include \"vmlinux.h\"\n#include <bpf/bpf_helpers.h>\n#include <bpf/bpf_tracing.h>\n#include <bpf/bpf_core_read.h>\n\nstruct {\n __uint(type, BPF_MAP_TYPE_HASH);\n __uint(max_entries, 65536);\n __type(key, u64); // pid_tgid\n __type(value, u64); // start timestamp\n} connect_start SEC(\".maps\");\n\nstruct {\n __uint(type, BPF_MAP_TYPE_RINGBUF);\n __uint(max_entries, 256 * 1024);\n} events SEC(\".maps\");\n\nstruct event {\n u32 pid;\n u32 dport;\n u64 latency_ns;\n u32 daddr;\n};\n\nSEC(\"kprobe/tcp_v4_connect\")\nint BPF_KPROBE(tcp_v4_connect_entry, struct sock *sk)\n{\n u64 pid_tgid = bpf_get_current_pid_tgid();\n u64 ts = bpf_ktime_get_ns();\n bpf_map_update_elem(&connect_start, &pid_tgid, &ts, BPF_ANY);\n return 0;\n}\n\nSEC(\"kprobe/tcp_finish_connect\")\nint BPF_KPROBE(tcp_finish_connect_entry, struct sock *sk)\n{\n u64 pid_tgid = bpf_get_current_pid_tgid();\n u64 *start_ts = bpf_map_lookup_elem(&connect_start, &pid_tgid);\n if (!start_ts)\n return 0;\n\n u64 latency = bpf_ktime_get_ns() - *start_ts;\n bpf_map_delete_elem(&connect_start, &pid_tgid);\n\n struct event *e = bpf_ringbuf_reserve(&events, sizeof(*e), 0);\n if (!e)\n return 0;\n\n e->pid = pid_tgid >> 32;\n e->latency_ns = latency;\n e->dport = BPF_CORE_READ(sk, __sk_common.skc_dport);\n e->daddr = BPF_CORE_READ(sk, __sk_common.skc_daddr);\n\n bpf_ringbuf_submit(e, 0);\n return 0;\n}\n\nchar LICENSE[] SEC(\"license\") = \"GPL\";\n```\n\nThe Go userspace component that consumes these events:\n\n```go\npackage main\n\nimport (\n \"bytes\"\n \"encoding/binary\"\n \"fmt\"\n \"log\"\n \"net\"\n\n \"github.com/cilium/ebpf/link\"\n \"github.com/cilium/ebpf/ringbuf\"\n \"github.com/cilium/ebpf/rlimit\"\n)\n\n//go:generate go run github.com/cilium/ebpf/cmd/bpf2go -cc clang tcpLatency tcp_connect_latency.bpf.c\n\ntype Event struct {\n Pid uint32\n Dport uint32\n LatencyNs uint64\n Daddr uint32\n}\n\nfunc main() {\n if err := rlimit.RemoveMemlock(); err != nil {\n log.Fatal(err)\n }\n\n objs := tcpLatencyObjects{}\n if err := loadTcpLatencyObjects(&objs, nil); err != nil {\n log.Fatalf(\"loading objects: %v\", err)\n }\n defer objs.Close()\n\n kp1, err := link.Kprobe(\"tcp_v4_connect\", objs.TcpV4ConnectEntry, nil)\n if err != nil {\n log.Fatalf(\"opening kprobe: %v\", err)\n }\n defer kp1.Close()\n\n kp2, err := link.Kprobe(\"tcp_finish_connect\", objs.TcpFinishConnectEntry, nil)\n if err != nil {\n log.Fatalf(\"opening kprobe: %v\", err)\n }\n defer kp2.Close()\n\n rd, err := ringbuf.NewReader(objs.Events)\n if err != nil {\n log.Fatalf(\"opening ringbuf reader: %v\", err)\n }\n defer rd.Close()\n\n fmt.Println(\"Tracing TCP connect latency... Ctrl+C to stop\")\n\n for {\n record, err := rd.Read()\n if err != nil {\n log.Printf(\"reading ringbuf: %v\", err)\n continue\n }\n\n var event Event\n if err := binary.Read(bytes.NewBuffer(record.RawSample), binary.LittleEndian, &event); err != nil {\n continue\n }\n\n addr := make(net.IP, 4)\n binary.LittleEndian.PutUint32(addr, event.Daddr)\n fmt.Printf(\"pid=%d dst=%s:%d latency=%.2fms\\n\",\n event.Pid, addr, ntohs(uint16(event.Dport)),\n float64(event.LatencyNs)/1e6)\n }\n}\n```"},{"heading":"Deploying Custom Probes in Kubernetes","level":3,"url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#deploying-custom-probes-in-kubernetes","content":"The tricky part is not writing the probe, it is deploying it safely across a fleet. Here is a proven deployment pattern:\n\n```yaml\napiVersion: apps/v1\nkind: DaemonSet\nmetadata:\n name: tcp-latency-probe\n namespace: observability\nspec:\n selector:\n matchLabels:\n app: tcp-latency-probe\n template:\n metadata:\n labels:\n app: tcp-latency-probe\n spec:\n hostPID: true\n hostNetwork: true\n containers:\n - name: probe\n image: ghcr.io/myorg/tcp-latency-probe:v1.2.0\n securityContext:\n privileged: false\n capabilities:\n add:\n - BPF\n - PERFMON\n - SYS_RESOURCE\n volumeMounts:\n - name: bpf-fs\n mountPath: /sys/fs/bpf\n - name: debugfs\n mountPath: /sys/kernel/debug\n readOnly: true\n volumes:\n - name: bpf-fs\n hostPath:\n path: /sys/fs/bpf\n - name: debugfs\n hostPath:\n path: /sys/kernel/debug\n```\n\nNotice we're not using ` privileged: true `. The ` BPF `, ` PERFMON `, and ` SYS_RESOURCE ` capabilities are sufficient. Running eBPF probes as privileged containers is a common anti-pattern that gives the probe unnecessary access."},{"heading":"Performance Characteristics","level":2,"url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#performance-characteristics","content":"Here are real numbers from a production cluster (150 nodes, ~3000 pods):\n\n```\nBaseline (no eBPF observability):\n Node CPU utilization: 62% average\n p99 request latency: 45ms\n\nWith Cilium + Hubble (L7 visibility):\n Node CPU utilization: 63.8% average (+1.8%)\n p99 request latency: 45.2ms (+0.4%)\n\nWith Tetragon (security monitoring, 12 TracingPolicies):\n Node CPU utilization: 64.1% average (+0.3% additional)\n p99 request latency: 45.2ms (no change)\n\nWith custom TCP latency probe (all connections):\n Node CPU utilization: 64.3% average (+0.2% additional)\n p99 request latency: 45.3ms (+0.1%)\n\nTotal overhead: ~2.3% CPU, ~0.3ms p99 latency\n```\n\nCompare this to a traditional sidecar-based service mesh (Istio with Envoy):\n\n```\nWith Istio sidecar (L7 visibility):\n Node CPU utilization: 71% average (+9%)\n p99 request latency: 52ms (+7ms)\n Memory overhead: ~50MB per pod (sidecar)\n```\n\nThe difference is staggering. eBPF operates at the kernel level, avoiding the context switches and memory copies that sidecar proxies require. You get better observability with an order of magnitude less overhead."},{"heading":"Common Pitfalls","level":2,"url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#common-pitfalls","content":"After deploying eBPF across multiple production environments, these are the mistakes teams make repeatedly:\n\n**1. Copying too much data to userspace.** Every ` bpf_ringbuf_submit() ` or ` bpf_perf_event_output() ` involves a memory copy. If you're tracing every syscall and copying full argument buffers, you'll burn CPU. Filter and aggregate in-kernel whenever possible.\n\n**2. Not using CO-RE.** If you're compiling eBPF programs against specific kernel headers and shipping different binaries per kernel version, you're doing it wrong. BPF CO-RE (Compile Once, Run Everywhere) with BTF eliminates this entirely.\n\n**3. Ignoring map sizing.** BPF maps are allocated at load time. A ` BPF_MAP_TYPE_HASH ` with ` max_entries=1000000 ` allocates memory upfront on each CPU. Size your maps based on actual cardinality, not worst case.\n\n**4. Running as privileged.** As noted above, privileged containers are not needed. ` CAP_BPF ` + ` CAP_PERFMON ` is sufficient for observability probes. ` CAP_NET_ADMIN ` if you're doing network stuff. That's it."},{"heading":"Where This Is Heading","level":2,"url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#where-this-is-heading","content":"The eBPF ecosystem is maturing fast. The major shifts to watch:\n\n- **eBPF on Windows.** Microsoft has been building eBPF support for Windows, and it's reaching production readiness. Cross-platform eBPF probes are becoming viable.\n- **BPF Arena.** New map type that gives eBPF programs a shared memory arena, enabling more complex data structures (trees, graphs) in kernel space.\n- **Scheduling class programs.** BPF programs that can influence the kernel's CPU scheduler. Google's sched\\_ext is already using this in production.\n\nFor platform teams, the message is clear: if you're still running sidecar-based service meshes primarily for observability, you're paying a 5-10x overhead tax. eBPF gives you deeper visibility at a fraction of the cost. The tooling has matured to the point where you don't need kernel developers on staff to deploy it. Cilium and Tetragon handle 90% of use cases out of the box.\n\nThe remaining 10%, custom probes for your specific domain, is where the real competitive advantage lies. And writing eBPF is not that hard once the constraints are internalized. The verifier is your friend, not your enemy. It's the reason you can deploy this stuff in production without fear."},{"heading":"Frequently asked questions","level":2,"url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#questions","content":""},{"heading":"Does eBPF add significant overhead to production workloads?","level":3,"url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#question-1","content":"When implemented correctly, eBPF adds negligible overhead, typically under 1% CPU. The key is filtering events in-kernel rather than shipping everything to userspace. A poorly written probe that copies large buffers on every syscall can add 5-10% overhead."},{"heading":"Can I use eBPF without writing C code?","level":3,"url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#question-2","content":"Yes. Tools like Cilium, Tetragon, and bpftrace provide high-level abstractions. Tetragon uses YAML TracingPolicies, and bpftrace has an awk-like scripting language. For custom needs, libbpf-rs (Rust) and cilium/ebpf (Go) provide safe wrappers."},{"heading":"What kernel version do I need for eBPF observability?","level":3,"url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#question-3","content":"Minimum kernel 5.4 for BTF support (CO-RE portability). Kernel 5.8+ for ring buffers. Kernel 5.11+ for bpf\\_snprintf and timer support. Most managed Kubernetes offerings (EKS, GKE, AKS) run 5.10+ as of 2025."},{"heading":"Tips","level":2,"url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#article-tips","content":"1. **BPF ring buffer vs perf buffer:** Always use ` BPF_MAP_TYPE_RINGBUF ` over ` BPF_MAP_TYPE_PERF_EVENT_ARRAY ` for new programs. Ring buffers share memory across CPUs, reduce memory overhead by ~50%, and support variable-length records. The only exception is per-CPU event ordering where you need strict per-CPU semantics.\n2. **Tail call chaining for complex probes:** Single eBPF programs have a 1M instruction limit (raised from 4096). For complex observability logic, use ` bpf_tail_call() ` to chain up to 33 programs via ` BPF_MAP_TYPE_PROG_ARRAY `. Each tail call resets the instruction counter, giving you effectively unlimited program size.\n3. **BTF-enabled CO-RE portability:** Compile your eBPF probes with ` -g -target bpf ` and use ` __builtin_preserve_access_index() ` wrappers on struct accesses. This embeds BTF relocations so the same compiled .o file runs across kernel versions 5.4+ without recompilation. Test with ` bpftool btf dump file /sys/kernel/btf/vmlinux `.\n4. **Tetragon TracingPolicy selectors:** Don't attach to every syscall. Use Tetragon's ` matchArgs ` and ` matchReturnArgs ` selectors to filter in-kernel: ` matchArgs: [{index: 0, operator: Equal, values: [\"/etc/shadow\"]}] `. This drops irrelevant events before they hit userspace, reducing CPU overhead from ~5% to \\<0.1%.\n\n- ebpf\n- kubernetes\n- cilium\n- tetragon\n- observability\n- linux-kernel\n- security-monitoring\n\n[Back to Blog](<https://lucioduran.com/blog>)"}],"links":[{"text":"Español","url":"https://lucioduran.com/es/blog/ebpf-observability-production-kubernetes"},{"text":"Cilium","url":"https://cilium.io/"},{"text":"Tetragon","url":"https://tetragon.io/"},{"text":"Back to Blog","url":"https://lucioduran.com/blog"}],"structuredData":[{"@context":"https://schema.org","@graph":[{"@type":"BlogPosting","@id":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes#article","mainEntityOfPage":{"@type":"WebPage","@id":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes"},"headline":"eBPF in Production: Building Kernel-Level Observability for Kubernetes Without the Overhead","description":"How to use eBPF for production-grade observability in Kubernetes clusters. Cilium, Tetragon, custom kprobes, and why traditional monitoring is leaving performance on the table.","datePublished":"2025-10-25","author":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"publisher":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"image":"https://statics.lucioduran.com/images/lucioduran-social-card.jpg","url":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes","inLanguage":"en","isPartOf":{"@id":"https://lucioduran.com/blog#blog"},"articleSection":"Infrastructure","keywords":"ebpf, kubernetes, cilium, tetragon, observability, linux-kernel, security-monitoring","wordCount":1556,"timeRequired":"PT8M","workTranslation":[{"@type":"BlogPosting","inLanguage":"es","url":"https://lucioduran.com/es/blog/ebpf-observability-production-kubernetes"}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://lucioduran.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://lucioduran.com/blog"},{"@type":"ListItem","position":3,"name":"eBPF in Production: Building Kernel-Level Observability for Kubernetes Without the Overhead","item":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes"}]},{"@type":"FAQPage","mainEntity":[{"@type":"Question","name":"Does eBPF add significant overhead to production workloads?","acceptedAnswer":{"@type":"Answer","text":"When implemented correctly, eBPF adds negligible overhead, typically under 1% CPU. The key is filtering events in-kernel rather than shipping everything to userspace. A poorly written probe that copies large buffers on every syscall can add 5-10% overhead."}},{"@type":"Question","name":"Can I use eBPF without writing C code?","acceptedAnswer":{"@type":"Answer","text":"Yes. Tools like Cilium, Tetragon, and bpftrace provide high-level abstractions. Tetragon uses YAML TracingPolicies, and bpftrace has an awk-like scripting language. For custom needs, libbpf-rs (Rust) and cilium/ebpf (Go) provide safe wrappers."}},{"@type":"Question","name":"What kernel version do I need for eBPF observability?","acceptedAnswer":{"@type":"Answer","text":"Minimum kernel 5.4 for BTF support (CO-RE portability). Kernel 5.8+ for ring buffers. Kernel 5.11+ for bpf_snprintf and timer support. Most managed Kubernetes offerings (EKS, GKE, AKS) run 5.10+ as of 2025."}}]}]}],"contentHash":"5620b968f1a6800486f74233970f5f27a880f9570c78f9407d8979e4b48459d3","provenance":{"method":"Deterministic serialization of the server-rendered main element","source":"https://lucioduran.com/blog/ebpf-observability-production-kubernetes","hashAlgorithm":"sha256","hashScope":"JSON document fields before contentHash and provenance","sourceHTMLBytes":70922,"contentBytes":15161}}