Document Version: 1.0 Audience: DevOps, Security Architects, SRE Reading Time: 20 minutes Updated: 2026-03-22

Overview

This document describes CleanStart's recommended runtime security monitoring architecture, integrating eBPF-based tools (Falco) with container supply chain evidence (SBOM, attestations) for continuous compliance and threat detection.

Key Concepts: eBPF is kernel-space monitoring with minimal overhead. Falco is open-source runtime security using eBPF. CleanStart Attestations are cryptographically-signed proof of image integrity. Compliance Evidence is continuous collection of audit-ready evidence.

The following diagram illustrates the runtime monitoring architecture from image deployment through threat response. A signed CleanStart image is deployed to a Kubernetes cluster with eBPF probes in the kernel. System calls, context switches, and network activity are captured and fed to Falco's rules engine. The rules engine evaluates three types of events: unexpected processes, suspicious file access, and network connections. When threats are detected, alerts are generated by severity level and sent to SIEM systems like Splunk/Datadog. Image SBOMs, attestations, and provenance data are correlated with alerts to determine if an incident represents a supply chain compromise or runtime exploit. Supply chain compromises result in blocking the registry image and revoking signatures. Runtime exploits trigger pod isolation and forensic snapshot capture. All incidents flow to incident response with audit-ready evidence for compliance.

Section 1: Architecture Overview

1.1 Full Stack Diagram

Development Phase documents that source code flows through the build pipeline to create CleanStart images. Evidence is automatically generated at build time, including an SBOM in SPDX 3.0 format, SLSA Provenance Attestation documenting the build process, and a cryptographic signature from Cosign.

Registry Storage includes signed images with attached attestations stored in the registry (registry.cleanstart.com). The trust root is established by the cosign public key, allowing later verification that images have not been tampered with.

Admission Control Layer ensures that when images are deployed to Kubernetes, the admission control layer performs signature verification using the cosign-webhook and enforces policies through Kyverno. Only signed CleanStart images are permitted to be deployed; unsigned images are rejected at this layer.

Runtime Execution enables containers to run in Kubernetes with continuous monitoring. eBPF probes collect all system call data without performance overhead. Falco's rules engine evaluates events in real-time, comparing observed behavior against expected patterns. Any anomalies trigger immediate alerts.

Response and Correlation means when threats are detected, alerts trigger automated forensic collection including memory dumps and filesystem snapshots. The system cross-correlates findings with image SBOMs and attestations to determine whether the incident is a supply chain compromise (requiring image revocation) or a runtime exploit (requiring pod isolation and remediation). Events are ingested into SIEM systems like Splunk, Datadog, or ELK for centralized analysis and incident response.

1.2 Architecture Components

The runtime monitoring architecture consists of interconnected components that provide defense in depth. The Supply Chain component handles image creation with cryptographic proof through CleanStart, SLSA framework, and Cosign signatures. The Registry component stores and distributes signed images using OCI v1.2 specifications with attached attestations. The Admission component prevents deployment of unsigned images through cosign-webhook and Kyverno policy enforcement. The Runtime component provides continuous monitoring through eBPF kernel instrumentation and Falco rule evaluation. The Alerting component detects incidents and routes them from Falco through SIEM systems to PagerDuty for immediate response. The Forensics component preserves evidence for investigation through container debugging and comprehensive log collection. The Compliance component maintains audit-ready evidence through SBOMs, attestations, and structured audit logs for regulatory review.

Section 2: eBPF-Based Monitoring

2.1 What is eBPF?

eBPF (extended Berkeley Packet Filter) is a kernel technology that enables safe, sandboxed execution of user-supplied programs in the kernel without requiring kernel recompilation or privilege escalation.

Advantages for Container Security: eBPF sees all syscalls (system calls) without kernel modules. Performance is approximately 100 microseconds per syscall (negligible overhead). Safety is achieved as the kernel verifies program validity before execution. Flexibility allows rules to be updated at runtime without restart.

The eBPF program intercepts write syscalls by checking if a process is in a container and if writing to a protected path, then sending an alert to user-space.

2.2 Syscall Monitoring Points

Falco monitors key syscalls relevant to container security:

Section 3: Falco + CleanStart Integration

3.1 Deployment Configuration

Helm Install with CleanStart Rules:

The Helm configuration enables eBPF probes, configures JSON output for SIEM integration, and sends alerts to HTTP endpoints. The configuration loads CleanStart-specific rules including distroless-rules.yaml for detecting shells in distroless containers, network-anomaly-rules.yaml for unexpected outbound connections, and supply-chain-rules.yaml for unsigned images being deployed. Performance tuning enables eBPF, and Kubernetes-specific settings create service accounts and configure resource limits. Resource requests and limits are configured to keep eBPF lightweight.

3.2 Rule Organization

Structure CleanStart rules by signal type in the /etc/falco/rules.d/ directory. The cleanstart-distroless-rules.yaml file contains rules for detecting shells in distroless containers, writes to read-only filesystems, and package manager invocations in distroless images. The network-anomaly-rules.yaml file includes rules for detecting unexpected outbound connections, suspicious DNS queries, and connections to known malicious IP addresses. The supply-chain-rules.yaml file contains rules for unsigned images being deployed when they shouldn't be, images with missing or invalid SBOMs, and containers sourced from non-registry sources.

3.3 Real-Time Alerting

Alert routing by severity:

High-priority alerts (CRITICAL/HIGH) include rules for shells in distroless containers, with escalation to PagerDuty. Medium-priority alerts (normal logging) include rules for unexpected outbound connections. Informational alerts include rules for normal container activity for statistics.

Falco Output Plugin Configuration:

Send to multiple destinations including syslog, JSON output, and Slack through configured plugins with initialization parameters.

Section 4: Compliance Evidence Collection

4.1 Continuous Audit Trail

Falco generates audit trail for compliance:

Every event is logged, timestamped, and archived as an immutable record.

What Gets Logged (HIPAA/PCI/SOC 2 compliant):

Complete audit events include timestamp in UTC, priority level, rule name, output details, syscall information with type and arguments, process details including PID/parent, container information including ID/name/image/image ID, user information with UID/GID/name, and Kubernetes information with pod name/namespace/labels.

Archival Pipeline:

Events flow from Falco to JSON output, then to Kafka for stream processing, then to SIEM (Splunk/DataDog), and finally to long-term storage (immutable GCS bucket) with 7-year retention for compliance.

4.2 SBOM-Aligned Monitoring

Track runtime behavior against known packages in SBOM:

Download SBOM from deployed image, have Falco monitor process execution against SBOM packages, check if executing processes are in the SBOM, and generate compliance reports linking Falco events to SBOMs. The compliance evidence includes timestamps, events, expected status, package presence in SBOM, and compliance status.

Section 5: Integration with Supply Chain Security

5.1 Admission Control + Runtime Monitoring

Multi-layer security:

Layer 1 (Admission): Only signed CleanStart images allowed, verified using cosign public keys.

Layer 2 (Runtime): Falco monitors for deviation from expected behavior using SBOM expectations.

Layer 3 (Forensics): Alerts trigger automatic evidence collection including memory dumps, filesystem snapshots, and attestation correlation.

Layer 4 (Analysis): System determines compromise type and appropriate response.

Kubernetes Admission Controller:

A ValidatingWebhookConfiguration verifies image signatures using cosign, rejecting unsigned images.

Cosign Webhook Logic:

Extract image reference from pod spec, verify image signature using cosign, allow pod creation if signature is valid, reject pod creation if signature is invalid, and log all decisions to an immutable audit log.

Section 6: Real-Time Alerting to Response

6.1 Alert-to-Remediation Workflow

End-to-end incident response automation:

The incident response workflow begins with a Falco Alert. The Alert Enrichment stage (SIEM) adds context including the image name and version, pod name and namespace, and process/file details. The Severity Assessment stage classifies the alert as CRITICAL (immediate response), HIGH (within 4 hours), or MEDIUM (within 24 hours). The Incident Commander Assignment stage sends a PagerDuty notification and auto-pages the on-call security engineer. The Automated Evidence Collection stage captures a memory dump of the container, a filesystem snapshot, correlation with the SBOM, and verification of the image signature.

The Decision Tree then evaluates whether the incident represents a supply chain compromise (in which case the image is revoked and all deployments are audited), a runtime exploit (kill the container and patch the app), or a misconfiguration (update the pod security policy).

This flows to Resolution and Post-Incident Review, which includes generating an RCA report, implementing improvements to detection rules, and documenting lessons learned.

6.2 Example: Shell Detection Response

Scenario: Falco detects shell in distroless container

An automated response script performs immediate containment by killing the compromised pod. It captures evidence already collected by forensics sidecar. It correlates with image attestation by verifying the image signature. If valid, it concludes runtime compromise (not supply chain). It extracts SBOM and checks for bash in the SBOM. If bash is not in SBOM, it confirms runtime compromise (not supply chain). It notifies incident response team via PagerDuty. Finally, it triggers forensic analysis using a security toolkit pod.

Section 7: Compliance Reporting

7.1 Monthly Audit Report

Auto-generated compliance report (exportable for auditors):

The report includes reporting period, organization name, compliance frameworks (SOC2 Type II, HIPAA, PCI DSS), container security metrics showing monitored pods, CleanStart image usage, unsigned images, distroless images, and standard images, runtime security events including totals and severity breakdowns, critical events detail with timestamps/rules/pods/status/resolution times, image integrity metrics including deployed images with valid signatures/SBOMs/SLSA attestations, and compliance status. The audit log status shows logs collected/archived, archival location, retention policy (7 years), and log accessibility. The conclusion states that all monitored containers run CleanStart images with zero incidents and full audit trail preservation, resulting in compliance with SOC2/HIPAA/PCI standards.

Section 8: Architecture Decisions and Trade-Offs

Section 9: Troubleshooting

Common Issues:

Further Reading: For deeper technical understanding, consult the eBPF Kernel Documentation, the Falco Official Docs, the SLSA Framework, and the Cosign Documentation.

Maintained by: Security Architecture, DevOps Teams Last Updated: 2026-03-22