Knowledge Hub

How to Collect Runtime Evidence from Kubernetes Pods Using eBPF

What this article covers

Understanding eBPF and why it's effective for Kubernetes runtime monitoring
Types of runtime evidence you can collect (system calls, network activity, file operations, process execution)
Deploying and configuring popular eBPF tools (Falco, Cilium/Hubble, Tetragon)
Writing custom eBPF programs for specific monitoring needs
Security considerations and performance optimization
Real-world use cases with complete implementations
Troubleshooting common eBPF deployment issues

Understanding eBPF for Kubernetes Monitoring

eBPF (extended Berkeley Packet Filter) runs sandboxed programs in the Linux kernel without modifying kernel source code or loading kernel modules. This provides kernel-level visibility into container behavior with minimal performance overhead.

Why eBPF for Kubernetes:

Container-aware monitoring distinguishes between pods automatically
No sidecar containers required (single agent per node)
Access to events applications cannot hide (system calls, network, file I/O)
Real-time detection and response capabilities
Verified safe execution that cannot crash the kernel

Performance characteristics:

Less than 1% CPU overhead for most workloads
100-300 MB memory per node
In-kernel data aggregation reduces network transfer

Prerequisites

Before implementing eBPF monitoring, verify your environment meets these requirements:

Kubernetes requirements:

Cluster admin access
Ability to deploy DaemonSets
Nodes must have /sys/kernel/btf/vmlinux (kernel 5.4+)

Types of Runtime Evidence You Can Collect

System Call Evidence

Captures low-level system interactions:

Valuable system calls to monitor:

• execve - Process execution• open, openat - File access

• connect, bind - Network operations

• ptrace - Debugging/injection attempts

• setuid, setgid - Privilege changes

Example evidence:

Network Evidence

Tracks connections, data transfer, DNS queries:

Network metrics to capture:

• Connection establishment and termination

• Data transfer volume

• DNS queries and responses

• Protocol anomalies

• Lateral movement patterns

File I/O Evidence

Monitors filesystem interactions:

File operations to track:

• Sensitive file access (credentials, keys, configuration)

• Binary modifications

• Unusual read/write patterns

• Configuration changes

Step 1: Deploy Falco for Security Monitoring

Falco provides runtime threat detection with pre-built security rules.

Install Falco via Helm

Create Custom Security Rules

Apply Custom Rules

Verify Falco is Running

Step 2: Deploy Cilium/Hubble for Network Visibility

Cilium provides network observability and policy enforcement using eBPF.

Install Cilium

Enable Hubble UI

Observe Network Flows

Step 3: Deploy Tetragon for Process Monitoring

Tetragon provides flexible security observability with custom tracing policies.

Install Tetragon

Create Tracing Policy

Apply Tracing Policy

Step 4: Store and Analyze Evidence

Set Up Elasticsearch for Event Storage

Configure Falco to Send Events

Query Stored Evidence

Step 5: Create Custom eBPF Program (Advanced)

For specialized monitoring needs, write custom eBPF programs.

Example: Monitor File Access

C eBPF Program:

Python Userspace Program:

Security Considerations

Required Privileges

eBPF tools require elevated privileges:

Capabilities needed:

CAP_BPF (kernel 5.8+)
CAP_SYS_ADMIN (older kernels)
CAP_SYS_RESOURCE
CAP_PERFMON (for performance monitoring)

Host access required:

/sys filesystem
/proc filesystem
Kernel BTF
Kernel modules directory

Minimize Privileges

Data Privacy

Sanitize sensitive data before storage:

RBAC for Evidence Access

Restrict who can view collected evidence:

Performance Optimization

Overhead Analysis

Optimization Techniques

1. Filter events in kernel (most efficient):

2. Aggregate metrics in kernel:

3. Use sampling for high-frequency events:

Configure sampling rate to reduce overhead:

4. Monitor tool performance:

Real-World Use Cases

Use Case 1: Detecting Cryptocurrency Mining

Falco rule:

Automated remediation:

Use Case 2: PCI-DSS Compliance Monitoring

Monitor access to cardholder data:

Generate compliance reports:

Use Case 3: Network Policy Validation

Verify network policies work correctly:

Automated validation:

Troubleshooting

Issue: eBPF Programs Won't Load

Symptoms:

Solutions:

Issue: Missing Events

Symptoms:Events are not captured or stored.Solutions:

Issue: High CPU Usage

Symptoms:eBPF tools consuming excessive CPU.Solutions:

Issue: Permission Denied