document version: 1.0 audience: DevOps, Security Engineers, SRE reading time: 20 minutes updated: 2026-03-22.
Overview
Falco is an open-source runtime security tool that detects anomalous behavior in containers using eBPF (extended Berkeley Packet Filter). This guide provides custom Falco rules tuned for CleanStart containers and explains how to integrate runtime detection with CleanStart's supply chain security.
key integration points Detect shell spawning in distroless containers (should never happen) Monitor for file modifications in read-only filesystems (escalation attempts) Track network anomalies (unexpected connections) Correlate runtime alerts with CleanStart image attestations for forensics.
Section 1: Custom Falco Rules for CleanStart Images
Rule 1: Shell Process in Distroless Container (CRITICAL)
purpose: Distroless containers have no shell. Spawning /bin/bash, /bin/sh, or similar in a CleanStart container indicates compromise or misconfiguration.
rule yaml.
- rule: Shell in Distroless Container desc: Detects shell process spawned in CleanStart distroless container condition: > spawned_process and shell_procs and container and container.image.repository in ( "registry.cleanstart.com/cleanstart/*", "cleanstart/*" ) and not container.privileged output: > CRITICAL: Shell spawned in distroless container (user=%user.name command=%proc.cmdline container.id=%container.id image=%container.image.repository:%container.image.tag) priority: CRITICAL tags: [distroless, shell, privilege_escalation]detection logic spawned_process: Process creation system call (fork/exec) shell_procs: Known shell binaries (/bin/bash, /bin/sh, /bin/zsh, etc.) container: Process is in a container (cgroup-based detection) container.image.repository in ("registry.cleanstart.com/cleanstart/*", "cleanstart/*"): Image is from CleanStart registry. expected behavior CleanStart containers: Rule should never trigger If rule triggers: Immediate investigation required (possible breach or misconfigured image). example alert.
2026-03-22 14:32:15 CRITICAL: Shell spawned in distroless containeruser=nobody command=/bin/bash -i container.id=a1b2c3d4e5f6image=registry.cleanstart.com/cleanstart/python:3.11response.
- Correlate timestamp with CleanStart image deployment logs
- Verify image signature:
cosign verify registry.cleanstart.com/cleanstart/python:3.11 - Check SBOM for suspicious changes
- Kill container; do not allow process to continue
Rule 2: File Modifications in Read-Only Filesystem (HIGH)
purpose: CleanStart images have read-only root filesystem. Attempts to write to / indicate escalation attack or kernel exploit.
rule yaml.
- rule: Write to Read-Only Filesystem desc: Detects write attempt to read-only root filesystem in CleanStart container condition: > write and fd.name glob ("/etc/*", "/bin/*", "/lib/*", "/usr/*") and container and container.image.repository in ( "registry.cleanstart.com/cleanstart/*", "cleanstart/*" ) and not proc.name in (rootkit_processes) output: > HIGH: Write attempt to read-only filesystem (user=%user.name process=%proc.name file=%fd.name container.id=%container.id image=%container.image.repository:%container.image.tag) priority: HIGH tags: [filesystem, integrity, privilege_escalation]detection logic write: System call (write, writev, pwrite64) fd.name glob (...): Attempt to write to system directories container: Process is containerized not proc.name in (rootkit_processes): Exclude known problematic processes. expected behavior Should never trigger in running CleanStart container If triggered: Kernel exploit or misconfigured image. example alert.
2026-03-22 15:45:22 HIGH: Write attempt to read-only filesystemuser=nobody process=exploit_binary file=/etc/ld.so.preload container.id=x9y8z7w6v5u4image=registry.cleanstart.com/cleanstart/nodejs:18rootkit detection.
rootkit_processes: - "init-setup" - "cloud-init" # Note: Very few processes should write to /etc at runtimeRule 3: Network Anomaly Detection (MEDIUM/HIGH)
purpose: Detect unexpected outbound connections that might indicate data exfiltration or C2 (command-and-control) communication.
rule yaml.
- rule: Unexpected Outbound Connection from Container desc: Detects outbound network connection not in baseline for this container image condition: > outbound and container and container.image.repository in ( "registry.cleanstart.com/cleanstart/*", "cleanstart/*" ) and not fd.sip in (allowed_internal_ips) and not fd.sport in (dns_ports) and not fd.dport in (allowed_outbound_ports) and not (fd.saddr = container.ip and fd.daddr = "8.8.8.8") output: > MEDIUM: Unexpected outbound connection (user=%user.name process=%proc.name container.image=%container.image.repository src=%fd.sip:%fd.sport dst=%fd.dip:%fd.dport) priority: MEDIUM tags: [network, exfiltration, c2]configuration (baseline).
allowed_internal_ips: - "10.0.0.0/8" # Private network - "172.16.0.0/12" # Private network - "192.168.0.0/16" # Private network dns_ports: - 53 # DNS allowed_outbound_ports: - 443 # HTTPS (common for APIs) - 80 # HTTP (if needed) - 5432 # PostgreSQL (database) - 6379 # Redis (cache)expected behavior Applications connecting to internal services: Expected Unexpected connections to external IPs or suspicious ports: Alert. example alert.
2026-03-22 16:12:33 MEDIUM: Unexpected outbound connectionuser=appuser process=python container.image=registry.cleanstart.com/cleanstart/python:3.11src=10.0.1.42:50293 dst=185.92.75.30:4444investigation.
- Verify if connection is legitimate (check application config)
- Check if IP
185.92.75.30is on threat intelligence lists - If confirmed malicious: Kill container, rotate secrets, check SBOM for supply chain compromise
Rule 4: Privilege Escalation Attempts (CRITICAL)
purpose: Detect attempts to escalate from non-root user to root (via sudo, setuid, etc.).
rule yaml.
- rule: Privilege Escalation Attempt in Container desc: Detects privilege escalation attempts in CleanStart containers condition: > (proc_name in (sudo, su, capset) or file_write and fd.name glob "*setuid*") and container and user.uid > 0 and not proc.name in (whitelist_procs) output: > CRITICAL: Privilege escalation attempt detected (user=%user.name process=%proc.name target_uid=%user.target_uid container.id=%container.id image=%container.image.repository:%container.image.tag) priority: CRITICAL tags: [privilege_escalation, container_escape]Whitelist (processes allowed to escalate):
whitelist_procs: # Most CleanStart containers should have empty whitelist # (no process should escalate privilege)expected behavior Should never trigger in CleanStart container (non-root only, no sudo/su) If triggered: Compromised container or misconfigured image.
Rule 5: File Integrity Monitoring (MEDIUM)
purpose: Monitor for unauthorized modifications to critical application files.
rule yaml.
- rule: Critical File Modified in Container desc: Detects modification of critical files in CleanStart containers condition: > (write or rename or delete) and fd.name glob ( "/app/*", "/srv/*", "/opt/*" ) and container and user.uid = 65532 # Non-root UID output: > MEDIUM: Critical file modified (user=%user.name process=%proc.name file=%fd.name action=%syscall.type container.id=%container.id image=%container.image.repository:%container.image.tag) priority: MEDIUM tags: [file_integrity, tampering]configuration.
# Define protected directories per imageprotected_paths: cleanstart/python: ["/app/", "/usr/local/lib/python3.11/"] cleanstart/nodejs: ["/app/", "/usr/local/lib/node_modules/"] cleanstart/go: ["/app/", "/usr/local/lib/go/"]example alert.
2026-03-22 17:05:15 MEDIUM: Critical file modifieduser=appuser process=curl file=/app/config.json action=writecontainer.id=abc123 image=registry.cleanstart.com/cleanstart/python:3.11interpretation Expected: Configuration changes via application (e.g., app updating config) Unexpected: Manual curl/wget downloading and executing code (supply chain compromise?).
Section 2: Shell Detection and Debugging
How to Allow Debug Containers While Maintaining Security
problem: Distroless containers have no shell; debugging is hard. But spawning a shell means relaxing security.
solution: Use sidecar debug container with full shell access, while keeping production container secured.
configuration.
apiVersion: v1kind: Podmetadata: name: app-with-debugspec: containers: # Production container: distroless, no shell - name: app image: registry.cleanstart.com/cleanstart/python:3.11 securityContext: runAsNonRoot: true runAsUser: 65532 allowPrivilegeEscalation: false readOnlyRootFilesystem: true # Debug container: full shell, isolated namespace - name: debug image: registry.cleanstart.com/cleanstart/python:3.11-debug securityContext: runAsNonRoot: true runAsUser: 65532 volumeMounts: - name: app-temp mountPath: /tmp/app # Shared temp for file inspection volumes: - name: app-temp emptyDir: {}Falco Configuration (allow debug container to have shell):
- rule: Shell in Debug Container desc: Allow shell in debug-tagged containers only condition: > spawned_process and shell_procs and container and container.image.repository endswith "-debug" output: > INFO: Shell in debug container (expected) (user=%user.name command=%proc.cmdline) priority: INFORMATIONAL tags: [debug, expected]Section 3: Integration with CleanStart Attestations
Correlation Workflow: Runtime Alert + Forensics
scenario: Falco detects shell process in container. Correlate with CleanStart evidence.
workflow.
#!/bin/bash # 1. Falco alert provides:CONTAINER_ID="a1b2c3d4e5f6"IMAGE="registry.cleanstart.com/cleanstart/python:3.11" # 2. Query Kubernetes for pod detailsPOD_NAME=$(docker inspect $CONTAINER_ID | jq -r '.[0].Name')kubectl get pod $POD_NAME -o json > pod-metadata.json # 3. Download image attestationcosign download attestation $IMAGE > image-attestation.json # 4. Extract SBOMcosign download sbom $IMAGE > image-sbom.json # 5. Verify image signaturecosign verify --key https://keys.cleanstart.com/prod.pub $IMAGE# Expected: Signature valid (proves image was built by CleanStart) # 6. Check SBOM for suspicious packagesjq '.components[] | select(.name | contains("bash")) | .name' image-sbom.json# Expected: No bash found in CleanStart distroless image # 7. If bash found: Image was tampered with post-build# Investigation: How did unsigned/modified image get deployed? # 8. Check pod creation timestamp vs. image build timestampPOD_CREATED=$(jq -r '.metadata.creationTimestamp' pod-metadata.json)SBOM_BUILD_TIME=$(jq -r '.components[] | select(.name=="build-metadata") | .properties[0].value' image-sbom.json) # 9. Generate incident reportcat > incident-report.md <<EOF# Runtime Security Incident Report ## Alert Summary- **Time**: $(date -u)- **Container ID**: $CONTAINER_ID- **Image**: $IMAGE- **Alert Type**: Shell in distroless container (CRITICAL) ## Forensic Analysis- **Image Build Time**: $SBOM_BUILD_TIME- **Pod Created**: $POD_CREATED- **Image Signature**: VALID- **SBOM Packages**: $(jq '.components | length' image-sbom.json)- **Packages with CVEs**: $(jq '[.components[] | select(.vulnerabilities != null)] | length' image-sbom.json) ## Conclusions[Investigation findings]EOFSection 4: Deploying Falco with CleanStart
Installation and Configuration
helm deploy.
helm repo add falcosecurity https://falcosecurity.github.io/chartshelm repo update helm install falco falcosecurity/falco \ --values falco-values.yaml \ --namespace falco \ --create-namespacevalues.yaml (CleanStart-specific):
falco: rules: # Load custom CleanStart rules - /etc/falco/rules.d/cleanstart-rules.yaml # Falco configuration json_output: true http_output: enabled: true url: http://siem.company.com:8080/falco # Alert only on CleanStart images to reduce noise base_syscall_rules: enabled: false # Disable verbose syscall rules grpc: enabled: true bind_address: "unix:///run/falco.sock" ebpf: enabled: true # eBPF for better performance serviceAccount: create: true name: falco podSecurityPolicy: create: false falcoctl: enabled: true indexes: - name: falcosecurity url: https://falcosecurity.github.io/falcoctl/index.yamlSIEM Integration (Datadog / Splunk / ELK)
syslog output.
syslog_output: enabled: true endpoint: "syslog.company.com:514" format: "RFC5424" facility: "LOG_LOCAL0" priority: "LOG_ALERT"datadog integration.
# Configure Falco to send alerts to Datadogfile_output: enabled: false network_output: enabled: true type: "HTTP" url: "https://api.datadoghq.com/v1/input/event" headers: "DD-API-KEY": "${DATADOG_API_KEY}" "Content-Type": "application/json"Section 5: Performance and Tuning
CPU/Memory Overhead
Falco using eBPF has minimal overhead: CPU: <2% per node Memory: 50-100MB resident Syscall Capture: ~100μs per syscall (eBPF performance).
Rule Tuning for Production
start conservative.
# Week 1: Detect obvious issues only- rule: Shell in Distroless Container (CRITICAL)- rule: File Modifications in Read-Only Filesystem (HIGH) # Week 2-3: Add network and privilege escalation- rule: Privilege Escalation Attempts (CRITICAL)- rule: Unexpected Outbound Connections (MEDIUM) # Week 4+: Add file integrity monitoring- rule: Critical File Modified in Container (MEDIUM)whitelist false positives.
- macro: known_good_processes condition: (proc.name in (systemd, docker, containerd)) - rule: Shell in Distroless Container condition: > ... and not known_good_processesSection 6: Alerting and Response
Alert Routing
# Critical alerts -> PagerDuty immediately- rule: Shell in Distroless Container priority: CRITICAL tags: [escalate_pagerduty] - rule: Privilege Escalation Attempt priority: CRITICAL tags: [escalate_pagerduty] # High alerts -> Slack (security channel)- rule: File Modifications in Read-Only Filesystem priority: HIGH tags: [escalate_slack] # Medium alerts -> SIEM (searchable, not immediate alert)- rule: Unexpected Outbound Connection priority: MEDIUM tags: [log_siem_only]Playbook: How to Respond to Falco Alert
When Falco triggers CRITICAL alert: 1. [Immediate] Acknowledge alert in PagerDuty2. [0-5 min] Kill the container (kubectl delete pod ...)3. [5-15 min] Correlate with CleanStart image attestations4. [15-30 min] Check SBOM for package modifications5. [30-60 min] Determine if supply chain compromise or exploitation6. [1-24 hrs] Forensic investigation + root cause analysis7. [Follow-up] Patch or rebuild image if neededfurther reading Falco Official Documentation eBPF and Runtime Security Container Runtime Security Best Practices.
maintained by: Security Engineering Team last updated: 2026-03-22.