Detecting Runtime Threats with eBPF and Falco
eBPF (extended Berkeley Packet Filter) is a Linux kernel technology that enables safe, user-space access to kernel data. Falco is a runtime security tool that uses eBPF to detect anomalies, policy violations, and suspicious behavior in running containers.
CleanStart integrates Falco for continuous runtime verification, detecting attacks and policy violations that static analysis can't see.
What Falco Does
Falco monitors container runtime behavior through eBPF kernel hooks, which operate transparently without requiring an agent in the container. These hooks capture system calls such as open, exec, network, and others. They track file access patterns, process creation and termination events, network connections, and privilege escalation attempts.
The captured events flow through Falco's Rule Engine, which matches events against security rules, calculates anomaly scores, cross-correlates related events to identify complex attack patterns, and suppresses false positives. When threats are detected, Falco generates Alerts and Audit Logs with real-time notifications, preservation of forensic evidence for investigation, and compliance reporting.
Installation and Setup
Install Falco on Kubernetes
# Add Falco Helm repositoryhelm repo add falcosecurity https://falcosecurity.github.io/chartshelm repo update # Install Falco in your clusterhelm install falco falcosecurity/falco \ --namespace falco --create-namespace \ --set falco.grpc.enabled=true \ --set falco.grpcOutput.enabled=true # Verify Falco is runningkubectl get pods -n falco# falco-xxxxx: RunningEnable eBPF Module
# Install eBPF driver (required for observability)helm install falco falcosecurity/falco \ --set ebpf.enabled=true \ --set falco.grpc.enabled=true # Verify eBPF is loadedkubectl exec -n falco <falco-pod> -- dmesg | grep -i ebpf# Output: eBPF module loaded successfullyFalco Rules for CleanStart Verification
Security-Focused Rules
# falco-cleanstart-rules.yaml # 1. Detect unauthorized privilege escalation- rule: Suspicious Privilege Escalation desc: Detect sudo/su usage or capability escalation condition: > spawned_process and (proc.name in (sudo, su) or container.privileged = true) and user.uid != 0 output: > Privilege escalation attempt (user=%user.name container=%container.info cmd=%proc.cmdline) priority: WARNING tags: [privilege_escalation, container_security] # 2. Detect cryptographic operations (zero-day check)- rule: Suspicious Cryptographic Library Usage desc: Detect unusual cryptography patterns condition: > spawned_process and fd.name glob "/proc/*/exe" and container.image.repository != "trusted-source" output: > Suspicious crypto operation (image=%container.image.repository process=%proc.name) priority: WARNING tags: [cryptography, zero_day_detection] # 3. Detect network anomalies- rule: Unexpected Network Connection desc: Container making outbound connection to unknown destination condition: > outbound and container.privileged = false and fd.sip not in (allowed_ips) and fd.sport not in (allowed_ports) output: > Unexpected outbound connection (container=%container.name dest=%fd.sip port=%fd.sport) priority: NOTICE tags: [network_security, anomaly] # 4. Detect ransomware patterns- rule: Potential Ransomware Activity desc: Detect file encryption or mass deletion condition: > write and container.privileged = false and (fd.name glob "*.locked" or fd.name glob "*.encrypted") and (number_files_written > 100 in 60 seconds) output: > Potential ransomware (container=%container.name files=%number_files_written) priority: CRITICAL tags: [ransomware, malware] # 5. Detect reverse shells- rule: Potential Reverse Shell desc: Detect suspicious shell spawning network connection condition: > spawned_process and container.privileged = false and (proc.name in (sh, bash) or proc.name contains /bin/) and (preceding_process.name not in (expected_shells)) and outbound output: > Potential reverse shell (container=%container.name shell=%proc.name dest=%fd.sip) priority: CRITICAL tags: [reverse_shell, intrusion] # 6. Detect supply chain attacks- rule: Suspicious Package Installation desc: Detect unauthorized package manager activity condition: > spawned_process and (proc.name in (apt, yum, npm, pip)) and user != expected_user and proc.cmdline contains "install" output: > Unauthorized package installation (container=%container.name cmd=%proc.cmdline) priority: WARNING tags: [supply_chain, package_manager]Integration with CleanStart
Falco Output Integration
# Send Falco alerts to CleanStart IntelligenceapiVersion: v1kind: ConfigMapmetadata: name: falco-custom-output namespace: falcodata: cleanstart_output: | - http: url: "https://intelligence.cleanstart.io/api/v1/runtime-alerts" method: "POST" headers: Authorization: "Bearer YOUR_API_KEY" Content-Type: "application/json" body_template: | { "alert_type": "runtime_anomaly", "container": "${container.name}", "image": "${container.image.repository}", "rule": "${rule}", "priority": "${priority}", "evidence": { "process": "${proc.name}", "user": "${user.name}", "command": "${proc.cmdline}", "network": { "src_ip": "${fd.sip}", "dst_ip": "${fd.dip}", "port": "${fd.sport}" } }, "timestamp": "${time}" }Audit Log Integration
# Export Falco alerts to audit systemkubectl logs -n falco -f deployment/falco | \ jq -r '.alert_type, .container, .rule, .priority' | \ tee -a /var/log/runtime-security-audit.log # Also send to SIEM (Splunk, ELK, etc.)kubectl logs -n falco -f deployment/falco | \ curl -X POST \ -H "Content-Type: application/json" \ -d @- \ https://siem.example.com:8088/services/collectorRuntime Compliance Verification
Verify Container Doesn't Violate Security Policy
# Monitor running container against policyfalco \ --rules /etc/falco/cleanstart-security-rules.yaml \ --rules /etc/falco/k8s-policy-rules.yaml # Example output:# 08:14:23.123456: WARNING: Suspicious Privilege Escalation# user=appuser container=myapp-xyz proc=sudo## 08:15:45.654321: CRITICAL: Potential Reverse Shell# container=myapp-xyz shell=/bin/bash dest=10.0.0.5:4444## 08:16:12.987654: NOTICE: Unexpected Network Connection# container=myapp-xyz dest=mining.attacker.com port=3333Real-Time Forensics
# During incident, capture full forensic evidencefalco \ --rules /etc/falco/forensics-rules.yaml \ --capture-file /var/log/forensics/$(date +%s).pcap \ --event-trace /var/log/forensics/events.json \ -o file:///var/log/forensics/falco.log # Output contains:# - All system calls made# - File access patterns# - Network traffic# - Process creation chains# - Environment variables at time of execution # Later analyze:cat /var/log/forensics/events.json | jq ' .[] | select(.container.name == "suspect-container") | {time, rule, process: .proc.name, user: .user.name, action: .proc.cmdline}'Anomaly Scoring
Calculate Risk Score from Runtime Behavior
# Anomaly scoring based on Falco alertsimport jsonfrom datetime import datetime, timedelta def calculate_runtime_risk_score(falco_events, time_window_minutes=60): """ Score container based on Falco alerts Higher score = higher risk of compromise """ risk_scores = { "CRITICAL": 10.0, "WARNING": 5.0, "NOTICE": 2.0, "INFO": 0.5 } now = datetime.now() cutoff = now - timedelta(minutes=time_window_minutes) relevant_events = [ e for e in falco_events if datetime.fromisoformat(e['timestamp']) > cutoff ] total_score = sum( risk_scores.get(e['priority'], 0) for e in relevant_events ) # Normalize to 0-1 scale max_possible = 10.0 * len(relevant_events) normalized_score = min(total_score / max_possible, 1.0) if max_possible > 0 else 0 return { "risk_score": normalized_score, "events": len(relevant_events), "critical_count": sum(1 for e in relevant_events if e['priority'] == 'CRITICAL'), "recommendation": ( "BLOCK" if normalized_score > 0.8 else "QUARANTINE" if normalized_score > 0.5 else "MONITOR" if normalized_score > 0.2 else "OK" ) } # Usageevents = json.load(open('/var/log/falco-events.json'))score = calculate_runtime_risk_score(events)print(f"Risk Score: {score['risk_score']:.2f}")print(f"Recommendation: {score['recommendation']}")Suspicious Behavior Examples
Cryptominer Detection
Event: Spawned process with suspicious patternContainer: worker-pod-xyzProcess: /usr/bin/xmrigParent: /bin/bash (PID 1234)User: appuserEnvironment: - POOL=mining.attacker.com:3333 - WALLET=attacker-wallet - THREADS=4 Falco Analysis: The process name matches known miners (xmrig, monero, etc.). Environment variables indicate mining pool connection. A CPU usage spike is detected. Network connection to mining pool is confirmed through another rule. Risk Score: 0.98 (CRITICAL)Action: Container terminated immediatelyPrivilege Escalation Detection
Event: Process attempted privilege escalationContainer: api-server-xyzProcess: sudo /bin/bashUser: appuser (UID 1000)Original Command: /app/bin/start-server.sh Falco Analysis:sudo was called from a non-privileged userUser is not in sudoers file (authorization failed)Shell was spawned with elevated privileges (escalation attempt)This violates the pod security policy Risk Score: 0.92 (CRITICAL)Action: Pod terminated, incident loggedReverse Shell Detection
Event: Suspicious shell spawning network connectionContainer: web-app-xyzProcess: /bin/bashParent: /bin/sh (background process)Network: Outbound connection to 10.0.0.5:4444 Falco Analysis:Shell process was spawned from an unexpected parentReverse shell pattern detected (bash listening/connecting)Destination IP is not in the allowlistPort 4444 matches common C2 communication ports Risk Score: 0.95 (CRITICAL)Action: Network connection blocked, container isolatedBest Practices
- Test Falco Rules: Verify rules don't produce excessive false positives
- Use Allowlists: Configure expected processes, users, and network destinations
- Monitor Baselines: Establish normal behavior baseline before alerting
- Retain Forensics: Keep Falco logs for 90 days minimum
- Integrate with SIEM: Send all alerts to central security platform
- Regular Tuning: Monthly review and update of rules
- Incident Response: Have runbook for each critical rule
Compliance Benefits
Control | How Falco Helps |
|---|---|
AU-2 (Audit) | Records all container activities |
IR-4 (Incident Response) | Captures forensic evidence |
SI-7 (Software Integrity) | Detects unauthorized changes |
SI-4 (Monitoring) | Real-time anomaly detection |
AC-6 (Privilege) | Detects privilege escalation |
SC-7 (Boundaries) | Monitors network violations |
See Also
Zero-Day Detection: ../intelligence/zero-day-detection.md — Catch unknowns. FIPS-Traces: ../fips/fips-traces.md — Cryptographic operations audit. Incident Response: ../regulatory/fedramp-high.md — IR-4 controls.
