Knowledge Hub

Security Testing Playbook for QA Teams

You're a QA engineer, not a security specialist. You don't have a background in cryptography, Linux kernel internals, or supply chain security. But you still…

Purpose

You're a QA engineer, not a security specialist. You don't have a background in cryptography, Linux kernel internals, or supply chain security. But you still need to verify that CleanStart's security properties actually work in practice.

This playbook gives you concrete, step-by-step tests you can run without needing deep security expertise. Each test has: What it verifies. Step-by-step instructions. Expected output. Pass/fail criteria. How to interpret results. By the end, you'll have confidence that your CleanStart images have the security properties they claim to have.

Security Testing vs Security Auditing

Security Testing (This Guide)

You're verifying that specific, measurable security properties exist and function correctly.

Does the image have a valid cosign signature? Does the image actually have a shell-less filesystem? Can I write to the root filesystem? (I shouldn't be able to.). Does the SBOM include all expected components? Scope: Technical verification, repeatable tests, objective pass/fail.

Security Auditing

Security teams perform deeper analysis: threat modeling, compliance mapping, policy alignment.

Is our security posture sufficient for our risk profile? Do we meet regulatory requirements? Are we following industry best practices? What's the business impact if this property fails? Scope: Risk assessment, policy review, subjective judgment calls.

Your role: QA. Run the tests in this guide. Let the security team interpret the results.

Tools You'll Need

You need to install these tools on your workstation. All are free and cross-platform.

Tool 1: cosign (Signature Verification)

# macOSbrew install sigstore/sigstore/cosign # Linux (Ubuntu/Debian)wget https://github.com/sigstore/cosign/releases/download/v2.2.0/cosign-linux-amd64chmod +x cosign-linux-amd64sudo mv cosign-linux-amd64 /usr/local/bin/cosign # Verify installationcosign version# Output: cosign version 2.2.0

Tool 2: trivy (Vulnerability Scanning)

# macOSbrew install trivy # Linux (Ubuntu/Debian)wget https://github.com/aquasecurity/trivy/releases/download/v0.48.0/trivy_0.48.0_Linux-64bit.debsudo dpkg -i trivy_0.48.0_Linux-64bit.deb # Verify installationtrivy version# Output: Version: 0.48.0

Tool 3: grype (Vulnerability Scanning, Alternative)

# macOSbrew install grype # Linux (Ubuntu/Debian)curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b /usr/local/bin # Verify installationgrype version# Output: 0.76.0

Tool 4: crane (Image Inspection)

# macOSbrew install crane # Linux (Ubuntu/Debian)wget https://github.com/google/go-containerregistry/releases/download/v0.17.0/go-containerregistry_Linux_x86_64.tar.gztar xzf go-containerregistry_Linux_x86_64.tar.gzsudo mv crane /usr/local/bin/ # Verify installationcrane version# Output: v0.17.0

Tool 5: jq (JSON Parsing)

# macOSbrew install jq # Linux (Ubuntu/Debian)sudo apt-get install jq # Verify installationjq --version# Output: jq-1.7

Tool 6: kubectl (Kubernetes Testing Only)

# macOSbrew install kubectl # Linux (Ubuntu/Debian)curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"chmod +x kubectlsudo mv kubectl /usr/local/bin/ # Verify installationkubectl version --client# Output: Client Version: v1.28.0

Test 1: Image Signature Verification

What This Tests

CleanStart images are signed with cosign using ECDSA keys. This test verifies that: The image has a valid signature. The signature was created by CleanStart's known key (not a forgery). The signature hasn't been tampered with.

Setup: Get the CleanStart Public Key

CleanStart publishes its cosign public key at:

https://registry.cleanstart.com/.cosign/cosign.pub

Save it locally:

curl -s https://registry.cleanstart.com/.cosign/cosign.pub > /tmp/cosign.pubcat /tmp/cosign.pub# Output starts with:# -----BEGIN PUBLIC KEY-----# MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...# -----END PUBLIC KEY-----

Step-by-Step Test

# Test image: python:3.12-stable from CleanStartIMAGE="registry.cleanstart.com/python:3.12-stable" # Verify the signaturecosign verify --key /tmp/cosign.pub "$IMAGE"

Expected Output (Pass)

Verification succeeds and the signing certificates are valid. The issuer is identified as https://token.actions.githubusercontent.com, the subject is https://github.com/cleanstart/container-images, and the certificate is valid until 2025-03-22. The image is confirmed to be valid and signed by CleanStart.

Expected Output (Fail)

Verification fails with an error indicating no signature is found for the image. Possible causes include: the image being corrupted, the image being from an untrusted source, or signature verification being misconfigured.

What to Do If It Fails

Check the image name is spelled correctly (case-sensitive)
Check the image exists: crane ls registry.cleanstart.com | grep python
Check network access to registry.cleanstart.com
If this is a custom golden image built internally, cosign verification may not apply

Pass/Fail Criteria

PASS: Verification successful, issuer is github.com/cleanstart/container-images. FAIL: No signature, tampered signature, or wrong issuer.

Test 2: SBOM Validation

What This Tests

CleanStart images include Software Bill of Materials (SBOM) in two formats: SPDX 3.0 (detailed component graph). CycloneDX 1.4 (dependency analysis). This test verifies: SBOMs exist and are well-formed. Component count is reasonable (not suspiciously low or high). Known-vulnerable components are noted.

Step-by-Step Test

Part A: Extract SBOMs from Image

IMAGE="registry.cleanstart.com/python:3.12-stable" # Extract SBOM from image (cosign convention)cosign download sbom "$IMAGE" > /tmp/sbom.json # Check if SBOM existsif [ -f /tmp/sbom.json ]; then  echo "✓ SBOM found"else  echo "✗ SBOM not found"  exit 1fi

Part B: Validate SBOM Format

# Check if it's valid JSONjq . /tmp/sbom.json > /dev/null 2>&1if [ $? -eq 0 ]; then  echo "✓ SBOM is valid JSON"else  echo "✗ SBOM is malformed"  exit 1fi # Check SBOM typeSBOM_TYPE=$(jq -r '.spdxVersion // .specVersion' /tmp/sbom.json)echo "SBOM format: $SBOM_TYPE" # For SPDX 3.0if [[ "$SBOM_TYPE" == "SPDX-3"* ]]; then  echo "✓ SPDX 3.0 format detected"fi # For CycloneDX 1.4if [[ "$SBOM_TYPE" == "1.4" ]]; then  echo "✓ CycloneDX 1.4 format detected"fi

Part C: Count Components

# For SPDX formatCOMPONENT_COUNT=$(jq '.packages | length' /tmp/sbom.json 2>/dev/null)if [ -n "$COMPONENT_COUNT" ]; then  echo "Components (SPDX): $COMPONENT_COUNT"fi # For CycloneDX formatCOMPONENT_COUNT=$(jq '.components | length' /tmp/sbom.json 2>/dev/null)if [ -n "$COMPONENT_COUNT" ]; then  echo "Components (CycloneDX): $COMPONENT_COUNT"fi

Part D: Check for Known Vulnerabilities

# Grype can consume SBOMs and report vulnerabilitiesgrype sbom:/tmp/sbom.json # Output shows:# Vulnerabilities by Severity# ────────────────────────────# Critical: 0# High:     0# Medium:   2# Low:      5# ...

Expected Output (Pass)

The SBOM is successfully found and contains valid JSON. The format is detected as SPDX 3.1.0, containing 347 components. The vulnerability summary shows zero critical vulnerabilities, zero high-severity issues, 2 medium-severity items (documented as known false positives in release notes), and 5 low-severity items.

Expected Output (Fail)

The SBOM is not found, or if present, is malformed. Alternatively, the vulnerability summary shows unexpected critical issues (3 critical vulnerabilities) or high-severity items (8 high-severity issues)

### What to Do If It Fails 1. **No SBOM**: Check if this image supports SBOM (all CleanStart images should)2. **Malformed SBOM**: Contact CleanStart support, image may be corrupted3. **Unexpected vulnerabilities**:   - Check release notes for documented false positives   - Search CVE database for the specific CVE   - May be a scanner error, consult security team ### Pass/Fail Criteria - **PASS**: SBOM exists, is well-formed (valid JSON), format is SPDX 3.0 or CycloneDX 1.4, component count >100 (reasonable), no critical or unexpected vulnerabilities- **FAIL**: SBOM missing, malformed, wrong format, suspiciously low component count (<50), or critical vulnerabilities present ## Test 3: Shell-Less Verification ### What This Tests CleanStart images are shell-less: they don't include bash, sh, dash, or other shells. This reduces attack surface and prevents shell-based escape vectors. This test verifies that no shell binary exists in the image. ### Step-by-Step Test #### Method 1: Try to Execute a Shell (Most Obvious Test) ```bash# Try to run bash in a containerdocker run --rm registry.cleanstart.com/python:3.12-stable /bin/bash -c "echo hello"

Expected Output (Pass)

The shell execution attempt fails with an OCI runtime error, indicating that /bin/bash does not exist. This is the expected behavior for a shell-less image.

Expected Output (Fail)

The shell command executes successfully and prints "hello". This is unexpected and indicates the image contains shell binaries when it should be shell-less.

Method 2: Inspect Filesystem for Shell Binaries

# Extract image filesystem and search for shell binariescrane export registry.cleanstart.com/python:3.12-stable | tar -xzf - -O \  bin/bash bin/sh bin/dash usr/bin/bash usr/bin/sh 2>/dev/null | wc -l # If any shells are found, output will be >0SHELL_COUNT=$?if [ $SHELL_COUNT -eq 0 ]; then  echo "✓ No shell binaries found in filesystem"else  echo "✗ Found shell binaries (count: $SHELL_COUNT)"fi

Method 3: Check Busybox (Common Shell Vector)

# Busybox often provides a shell; verify it's not presentdocker run --rm registry.cleanstart.com/python:3.12-stable \  ls -la /bin/busybox 2>&1 | grep -q "cannot access" if [ $? -eq 0 ]; then  echo "✓ Busybox (shell provider) not found"else  echo "✗ Busybox found (could provide shell)"fi

Method 4: Check PATH for Shell References

# Get image config and check PATHcrane config registry.cleanstart.com/python:3.12-stable | jq -r '.config.Env[] | select(. | contains("PATH"))' # Output should be something like:# PATH=/usr/local/bin:/usr/bin:/bin## Then verify these directories don't contain shells:docker run --rm registry.cleanstart.com/python:3.12-stable \  find /usr/local/bin /usr/bin /bin -name "bash" -o -name "sh" -o -name "dash" 2>/dev/null # Should output nothing if shell-less

What to Do If Shells Are Found

Verify the image is actually from CleanStart (check registry URL)
Check if this is a custom image you built (might have shells)
Report to CleanStart support if shells found in official image
For custom images: rebuild from golden base, don't add shells

Pass/Fail Criteria

PASS: No shell binaries found, shell execution attempt fails, busybox not present
FAIL: Shell binaries found, shell execution succeeds, busybox present

Test 4: Read-Only Filesystem Verification

What This Tests

CleanStart images have read-only root filesystems. Applications can write to /tmp and other writable mounts, but cannot modify system files.

This test verifies that the filesystem is actually read-only.

Step-by-Step Test

Method 1: Try to Write to System Directory

# Attempt to write to /etc (should fail)docker run --rm registry.cleanstart.com/python:3.12-stable \  touch /etc/test-file 2>&1

Expected Output (Pass)

The write attempt fails with a "Read-only file system" error. This is the expected behavior for a properly configured read-only root filesystem.

Expected Output (Fail)

The file is created without error, which is unexpected because the filesystem should be read-only.

Method 2: Check Writable Mounts

# List mounted filesystems inside containerdocker run --rm registry.cleanstart.com/python:3.12-stable \  mount | grep -E "rw" | grep -v "devtmpfs" # Output should show only tmpfs mounts:tmpfs on /tmp type tmpfs (rw, ...)tmpfs on /run type tmpfs (rw, ...)tmpfs on /dev/shm type tmpfs (rw, ...) # Root filesystem should be read-only:# /dev/... on / type ... (ro, ...)

Method 3: Verify in Kubernetes

# Create a test pod with read-only root filesystemkubectl apply -f - <<EOFapiVersion: v1kind: Podmetadata:  name: readonly-testspec:  containers:Name: app.    image: registry.cleanstart.com/python:3.12-stable    securityContext:      readOnlyRootFilesystem: true  restartPolicy: NeverEOF # Wait for pod to startsleep 3 # Try to write a filekubectl exec readonly-test -- touch /test-file 2>&1 # Should fail:# Read-only file system# command terminated with exit code 1

What to Do If Writes Succeed

This is a critical security issue
Report immediately to CleanStart support
Do not use this image in production until resolved
Use a previously-tested golden image version instead

Pass/Fail Criteria

PASS: Cannot write to system directories, writable mounts are tmpfs only, Kubernetes pod respects readOnlyRootFilesystem
FAIL: Can write to system directories, root filesystem is writable, Kubernetes pod can write to root

Test 5: Non-Root User Verification

What This Tests

CleanStart images run as non-root (UID 65532, a reserved unprivileged user). This prevents container escapes from leading to host root access.

This test verifies the running user is not root.

Step-by-Step Test

Method 1: Check Running UID

# Start container and check UIDdocker run --rm registry.cleanstart.com/python:3.12-stable \  id # Expected output:# uid=65532 gid=65532 groups=65532

Expected Output (Pass)

The user ID is 65532 (nobody), group ID is 65532 (nogroup). The container successfully runs as non-root as expected.

Expected Output (Fail)

The user ID is 0 (root), group ID is 0 (root). This is unexpected because the container should run as a non-root user.

Method 2: Verify No SUID Binaries

SUID binaries allow non-root users to temporarily gain root privileges for specific operations. Removing them reduces privilege escalation vectors.

# Search for SUID binaries (should be none)docker run --rm registry.cleanstart.com/python:3.12-stable \  find / -perm /4000 2>/dev/null # Output should be empty (no SUID binaries)

Expected Output (Pass)

The search returns no output, indicating no SUID binaries are present in the image. This is the expected behavior.

Expected Output (Fail)

The search finds SUID binaries such as /usr/bin/sudo and /usr/bin/passwd. This is unexpected and represents a potential privilege escalation vector.

Method 3: Check Capability Drops

Linux capabilities are granular permissions. Even if a non-root user runs the container, if they have the CAP_SYS_ADMIN capability, they can still escape.

# Check capabilities in running containerdocker run --rm registry.cleanstart.com/python:3.12-stable \  grep Cap /proc/self/status # Output should show all capabilities dropped:# CapInh:	0000000000000000# CapPrm:	0000000000000000# CapEff:	0000000000000000# CapBnd:	0000000000000000

Expected Output (Pass)

All capability fields (inherited, permitted, effective, bounding, and ambient) show all zeros (0000000000000000), indicating that all Linux capabilities have been dropped as expected.

Expected Output (Fail)

The effective capabilities field shows a non-zero value (00000000a80425fb), indicating that capabilities are still present. This is unexpected because all capabilities should be fully dropped.

Method 4: Verify in Kubernetes

# Pod with runAsNonRoot enforcedkubectl apply -f - <<EOFapiVersion: v1kind: Podmetadata:  name: nonroot-testspec:  securityContext:    runAsNonRoot: true    runAsUser: 65532  containers:Name: app.    image: registry.cleanstart.com/python:3.12-stable    securityContext:      allowPrivilegeEscalation: false      capabilities:        drop:ALL.  restartPolicy: NeverEOF # Pod should start successfullykubectl get pod nonroot-test# STATUS: Running

What to Do If Running as Root

This is a critical security issue
Report to CleanStart support
Do not use this image in production
Verify you're using the correct image (not a custom variant)

Pass/Fail Criteria

PASS: Running UID is 65532, no SUID binaries, all capabilities dropped, Kubernetes nonroot enforcement works
FAIL: Running UID is 0, SUID binaries present, capabilities present, Kubernetes nonroot enforcement fails

Test 6: FIPS Mode Verification (FIPS Images Only)

What This Tests

CleanStart offers FIPS 140-3 certified images. This test verifies cryptographic algorithms are restricted to FIPS-approved ones.

Note: Only relevant for registry.cleanstart.com/*/fips images. Skip this test for standard images.

Step-by-Step Test

Method 1: Check OpenSSL FIPS Mode

# Check if OpenSSL FIPS module is enableddocker run --rm registry.cleanstart.com/python:3.12-fips \  python3 -c "import sslprint('FIPS mode:', ssl.OPENSSL_VERSION)" # Output:# FIPS mode: OpenSSL 3.0.x (FIPS enabled)

Expected Output (Pass)

FIPS mode: OpenSSL 3.0.8 (FIPS) ✓ FIPS mode enabled

Method 2: Test Restricted Algorithms

FIPS mode rejects certain weak algorithms. MD5 and DES are not allowed.

# Try MD5 (should fail in FIPS mode)docker run --rm registry.cleanstart.com/python:3.12-fips \  python3 -c "import hashlibtry:  md5 = hashlib.md5()  print('ERROR: MD5 allowed (FIPS mode not active)')except ValueError as e:  print('MD5 blocked:', str(e))  print('✓ FIPS mode enforcing')" # Expected output:# MD5 blocked: unsupported hash type md5. ...# ✓ FIPS mode enforcing

Method 3: Verify TLS Cipher Suites

# Check available TLS ciphersdocker run --rm registry.cleanstart.com/python:3.12-fips \  python3 -c "import sslctx = ssl.create_default_context()ciphers = ctx.get_ciphers()for c in ciphers:  print(c['name'])" # Should only show FIPS-approved ciphers (TLS_AES_256_GCM_SHA384, etc.)# Should NOT include: TLS_RSA_*, DES-*, MD5, RC4, etc.

What to Do If FIPS Tests Fail

Verify you're using a FIPS image (-fips suffix)
Check OpenSSL version (must be 3.0+)
Report to CleanStart support if tests fail on FIPS image

Pass/Fail Criteria (FIPS Images Only)

PASS: FIPS mode enabled in OpenSSL, weak algorithms blocked, only FIPS ciphers available
FAIL: FIPS mode disabled, weak algorithms allowed, non-FIPS ciphers present

Test 7: Network Security Verification

What This Tests

CleanStart images expose only necessary network ports. Services listening on unexpected ports could be security risks.

This test verifies network exposure is minimal.

Step-by-Step Test

Method 1: Port Scanning

# Start container in backgroundCONTAINER_ID=$(docker run -d registry.cleanstart.com/python:3.12-stable sleep 1000) # Get container IPCONTAINER_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $CONTAINER_ID) # Scan for open portsnmap -p 1-65535 $CONTAINER_IP 2>/dev/null | grep "open" # Clean updocker rm -f $CONTAINER_ID

Expected Output (Pass)

No output is returned, or only ephemeral ports are shown. No unexpected ports are listening.

Expected Output (Fail)

The scan shows open ports for SSH (22/tcp) and Telnet (23/tcp). These are unexpected services that should not be listening.

Method 2: Check Listening Services

# List all listening sockets inside containerdocker run --rm registry.cleanstart.com/python:3.12-stable \  netstat -tlnp 2>/dev/null || ss -tlnp # Should output only application-specific ports, not sshd, telnet, etc.

Pass/Fail Criteria

PASS: No unexpected ports listening, only application services exposed
FAIL: SSH, Telnet, other management services listening

Test 8: Vulnerability Scan Verification

What This Tests

Vulnerability scanners (Trivy, Grype) identify known vulnerable components. This test runs scanners and verifies results are consistent with CleanStart's published vulnerability reports.

Step-by-Step Test

Method 1: Scan with Trivy

# Scan image for vulnerabilitiestrivy image registry.cleanstart.com/python:3.12-stable # Output:# Total: 347 (CRITICAL: 0, HIGH: 0, MEDIUM: 2, LOW: 5)

Method 2: Scan with Grype

# Grype provides alternative scangrype registry.cleanstart.com/python:3.12-stable # Output:# Vulnerabilities By Severity# ──────────────────────────# Critical: 0# High:     0# Medium:   2# Low:      5

Method 3: Compare to CleanStart Release Notes

# CleanStart publishes official scan results# Compare your results to: https://registry.cleanstart.com/releases/python/3.12-stable/scan-results.json # Expected:# Your scan: CRITICAL=0, HIGH=0, MEDIUM=2, LOW=5# Official: CRITICAL=0, HIGH=0, MEDIUM=2, LOW=5 (match!) # If counts differ, investigate:# - Scanner version difference# - Database update timing (scanners auto-update vulnerability database)# - False positives (documented in release notes)

Understanding Scanner Differences

Trivy and Grype may report different numbers due to:

Different vulnerability databases: Trivy uses trivy.dev, Grype uses Grype DB
Different severity scoring: CVE severity != CVSS severity
False positives: Some findings are known to be incorrect

Example:

Image: python:3.12-stableTrivy report: 2 medium-severity CVEsGrype report: 5 medium-severity CVEsDifference: Grype may be more conservative, Trivy may filter some false positives Solution: Check release notes, understand why Grype found more

What to Do If Scans Show Unexpected Vulnerabilities

Check release notes: Is this documented as a false positive?
Check CVE details: Search NVD.NIST.gov for the CVE, is it applicable?
Check VEX (Vulnerability Exploitability Exchange): CleanStart publishes VEX statements explaining false positives
Consult security team: Determine if it's an actual risk

Pass/Fail Criteria

PASS: Scan succeeds, results match CleanStart release notes, no critical/high findings, medium/low findings are documented
FAIL: Scan fails, results differ unexpectedly from release notes, critical findings present

Test 9: Container Isolation Verification

What This Tests

Kubernetes and container runtimes provide multiple isolation layers. This test verifies they're configured correctly in CleanStart images.

Step-by-Step Test

Method 1: Verify PID Namespace Isolation

# Two containers should not see each other's processesCONTAINER1=$(docker run -d registry.cleanstart.com/python:3.12-stable sleep 1000)CONTAINER2=$(docker run -d registry.cleanstart.com/python:3.12-stable sleep 1000) # In container 2, list processesdocker exec $CONTAINER2 ps aux # Should only show container 2's processes, not container 1# If you see container 1's processes, namespace isolation failed

Expected Output (Pass)

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMANDnobody       1  0.1  0.0   5000   500 ?        Ss   10:00   0:00 sleep 1000 ✓ Only local processes visible (good isolation)

Expected Output (Fail)

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMANDnobody       1  0.1  0.0   5000   500 ?        Ss   10:00   0:00 sleep 1000nobody       7  0.1  0.0   5000   500 ?        Ss   10:00   0:00 sleep 1000 (OTHER CONTAINER!) ✗ Can see other container's processes (isolation failure)

Method 2: Check seccomp Profile

# Check if seccomp is enableddocker inspect registry.cleanstart.com/python:3.12-stable | jq '.[0].HostConfig.SecurityOpt' # Output:# ["seccomp=unconfined"] ← seccomp NOT configured (default for base image)## In Kubernetes, you should configure:# securityContext:#   seccompProfile:#     type: RuntimeDefault

Method 3: Verify AppArmor/SELinux Profile (Kubernetes)

# Pod with AppArmorkubectl apply -f - <<EOFapiVersion: v1kind: Podmetadata:  name: isolation-test  annotations:    container.apparmor.security.beta.kubernetes.io/app: runtime/defaultspec:  containers:Name: app.    image: registry.cleanstart.com/python:3.12-stable  restartPolicy: NeverEOF # Pod should start and AppArmor should be appliedkubectl get pod isolation-test

Pass/Fail Criteria

PASS: PID namespace isolation verified, seccomp/AppArmor available and configurable
FAIL: Can see other containers' processes, isolation disabled

Test 10: Supply Chain Verification

What This Tests

CleanStart publishes SLSA Level 4 provenance, proving images are built from known source code. This test verifies supply chain integrity.

Step-by-Step Test

Method 1: Download Provenance Attestation

# SLSA provenance is stored as an OCI attestation# Use cosign to retrieve itcosign download attestation registry.cleanstart.com/python:3.12-stable > /tmp/provenance.json # Verify it's valid JSONjq . /tmp/provenance.json > /dev/nullif [ $? -eq 0 ]; then  echo "✓ Provenance attestation found and valid"else  echo "✗ Provenance attestation invalid"  exit 1fi

Method 2: Verify Build Source

# Extract build source from provenancejq -r '.payload.buildConfig.source.repository' /tmp/provenance.json # Output should be CleanStart's source repository:# https://github.com/cleanstart/container-images # Verify build timestamp is recentBUILD_TIME=$(jq -r '.payload.buildStartedOn' /tmp/provenance.json)echo "Built: $BUILD_TIME" # Calculate ageNOW=$(date +%s)BUILD_TIMESTAMP=$(date -d "$BUILD_TIME" +%s)AGE_DAYS=$(( ($NOW - $BUILD_TIMESTAMP) / 86400 ))echo "Image age: $AGE_DAYS days" if [ $AGE_DAYS -lt 365 ]; then  echo "✓ Image built within last year (good freshness)"else  echo "⚠ Image is older than 1 year (consider updating)"fi

Method 3: Verify Source-to-Image Mapping

# Check that image digest matches build outputIMAGE_DIGEST=$(crane digest registry.cleanstart.com/python:3.12-stable)BUILD_OUTPUT=$(jq -r '.payload.buildOutputs[0].digest' /tmp/provenance.json) if [ "$IMAGE_DIGEST" = "$BUILD_OUTPUT" ]; then  echo "✓ Image digest matches provenance (supply chain unbroken)"else  echo "✗ Image digest doesn't match provenance (possible tampering)"fi

Expected Output (Pass)

✓ Provenance attestation found and validBuilt: 2025-03-15T10:30:00ZImage age: 7 days✓ Image built within last year (good freshness)Source repository: https://github.com/cleanstart/container-images✓ Image digest matches provenance (supply chain unbroken)

Expected Output (Fail)

✗ Provenance attestation invalid  (or)✗ Image digest doesn't match provenance (possible tampering)

What to Do If Provenance Verification Fails

Image may be corrupted in transit
Image may be from an untrusted source
Report to CleanStart support immediately
Do not use in production

Pass/Fail Criteria

PASS: Provenance exists and is valid, source is CleanStart repo, image age reasonable, digest matches
FAIL: Provenance missing or invalid, source is wrong, digest mismatch

Automation: Run All 10 Tests

Here's a bash script that runs all 10 tests and produces a summary report:

#!/bin/bash# test-cleanstart-security.sh - Run all security tests on CleanStart image set -e IMAGE="${1:-registry.cleanstart.com/python:3.12-stable}"REPORT_FILE="cleanstart-security-report-$(date +%Y%m%d-%H%M%S).txt" echo "╔════════════════════════════════════════════════════════════╗"echo "║    CleanStart Security Testing Report                       ║"echo "╚════════════════════════════════════════════════════════════╝"echo ""echo "Image: $IMAGE"echo "Report: $REPORT_FILE"echo "" # Color codesGREEN='\033[0;32m'RED='\033[0;31m'YELLOW='\033[1;33m'NC='\033[0m' # No Color PASS_COUNT=0FAIL_COUNT=0 # Helper functionspass_test() {  echo -e "${GREEN}✓ PASS${NC}: $1"  ((PASS_COUNT++))} fail_test() {  echo -e "${RED}✗ FAIL${NC}: $1"  ((FAIL_COUNT++))} warn_test() {  echo -e "${YELLOW}⚠ WARN${NC}: $1"} # Test 1: Signature Verificationecho ""echo "Test 1: Signature Verification"cosign verify --key /tmp/cosign.pub "$IMAGE" > /dev/null 2>&1 && pass_test "Image signature valid" || fail_test "Image signature invalid" # Test 2: SBOM Validationecho ""echo "Test 2: SBOM Validation"cosign download sbom "$IMAGE" > /tmp/sbom.json 2>/dev/nulljq . /tmp/sbom.json > /dev/null 2>&1 && pass_test "SBOM format valid" || fail_test "SBOM format invalid" # Test 3: Shell-Less Verificationecho ""echo "Test 3: Shell-Less Verification"docker run --rm "$IMAGE" /bin/bash -c "echo" 2>&1 | grep -q "no such file" && pass_test "No shell binary found" || fail_test "Shell binary found" # Test 4: Read-Only Filesystemecho ""echo "Test 4: Read-Only Filesystem"docker run --rm "$IMAGE" touch /etc/test 2>&1 | grep -q "Read-only" && pass_test "Filesystem is read-only" || fail_test "Filesystem is writable" # Test 5: Non-Root Userecho ""echo "Test 5: Non-Root User"docker run --rm "$IMAGE" id | grep -q "uid=65532" && pass_test "Running as non-root (UID 65532)" || fail_test "Not running as expected non-root user" # Test 6: No SUID Binariesecho ""echo "Test 6: SUID Binary Check"docker run --rm "$IMAGE" find / -perm /4000 2>/dev/null | wc -l | grep -q "^0$" && pass_test "No SUID binaries found" || warn_test "SUID binaries found (check if expected)" # Test 7: Vulnerability Scanecho ""echo "Test 7: Vulnerability Scan (Trivy)"trivy image "$IMAGE" 2>/dev/null | grep -q "CRITICAL: 0" && pass_test "No critical vulnerabilities" || fail_test "Critical vulnerabilities found" # Test 8: Network Securityecho ""echo "Test 8: Network Services"EXPOSED_PORTS=$(docker run --rm "$IMAGE" netstat -tlnp 2>/dev/null | grep LISTEN | wc -l)[ "$EXPOSED_PORTS" -le 2 ] && pass_test "Minimal network exposure" || warn_test "Multiple services listening (verify if expected)" # Test 9: Provenanceecho ""echo "Test 9: Supply Chain Provenance"cosign download attestation "$IMAGE" > /tmp/provenance.json 2>/dev/nulljq . /tmp/provenance.json > /dev/null 2>&1 && pass_test "SLSA provenance valid" || fail_test "Provenance missing or invalid" # Summaryecho ""echo "╔════════════════════════════════════════════════════════════╗"echo "║ Summary                                                     ║"echo "╚════════════════════════════════════════════════════════════╝"echo ""echo -e "${GREEN}Passed: $PASS_COUNT${NC}"echo -e "${RED}Failed: $FAIL_COUNT${NC}"echo "" if [ $FAIL_COUNT -eq 0 ]; then  echo -e "${GREEN}✓ All tests passed! Image is secure.${NC}"  exit 0else  echo -e "${RED}✗ Some tests failed. Review failures above.${NC}"  exit 1fi

Usage:

chmod +x test-cleanstart-security.sh./test-cleanstart-security.sh registry.cleanstart.com/python:3.12-stable

Interpreting Common Results

Scenario 1: Trivy Found CVEs That CleanStart Says Are Fixed

Situation:

Trivy scan shows: CVE-2025-1234 (Medium) in opensslCleanStart release notes show: Fixed in v2.1.0

Explanation: Trivy's vulnerability database may not be fully up-to-date. CleanStart may have built an image with patched openssl, but Trivy's database hasn't refreshed yet.

Resolution:

# Update Trivy's databasetrivy image --severity HIGH,CRITICAL registry.cleanstart.com/python:3.12-stable # If still shows after update, check VEX statement:cosign download sbom registry.cleanstart.com/python:3.12-stable | jq '.vex'

Scenario 2: Shell Test Passed but App Needs Shell for Debugging

Situation:

Container runs fine in testingBut in production, developer needs to debug: "I need to run /bin/bash"

Explanation: CleanStart is shell-less by design. For debugging, use:

kubectl exec → ephemeral container with busybox (added at debug time, not in image)
kubectl debug → temporary sidecar with shell
Application logs → best debugging source

Solution:

# Debug with ephemeral container (Kubernetes 1.23+)kubectl debug -it <pod-name> --image=busybox # Application logging (better approach)# App outputs structured logs to stderr# Operator tails logs: kubectl logs <pod-name>

Scenario 3: FIPS Test Shows Non-FIPS Algorithm

Situation:

Testing FIPS image, but seeing TLS_RSA_* ciphers (non-FIPS)

Explanation: Application may be using older TLS configuration that allows non-FIPS ciphers. The system supports them, but you should restrict application config to FIPS-only.

Solution:

# In your Python/Node/etc applicationimport ssl # Force FIPS-only cipherscontext = ssl.create_default_context()context.set_ciphers(':'.join([    'ECDHE-ECDSA-AES256-GCM-SHA384',    'ECDHE-RSA-AES256-GCM-SHA384',]))

Scenario 4: Container Isolation Test Sees Other Container's Processes

Situation:

Running two containers, both can see each other's processes

Explanation: You may be running in a non-isolated mode (--pid=host). Check your docker run command or Kubernetes pod spec.

Solution:

# ❌ WRONGdocker run --pid=host registry.cleanstart.com/python:3.12-stable # ✅ CORRECT (default isolation)docker run registry.cleanstart.com/python:3.12-stable

Knowledge Hub

Security Testing Playbook for QA Teams

You're a QA engineer, not a security specialist. You don't have a background in cryptography, Linux kernel internals, or supply chain security. But you still…

Purpose

Security Testing vs Security Auditing

Security Testing (This Guide)

You're verifying that specific, measurable security properties exist and function correctly.

Security Auditing

Security teams perform deeper analysis: threat modeling, compliance mapping, policy alignment.

Your role: QA. Run the tests in this guide. Let the security team interpret the results.

Tools You'll Need

You need to install these tools on your workstation. All are free and cross-platform.

Tool 1: cosign (Signature Verification)

# macOSbrew install sigstore/sigstore/cosign # Linux (Ubuntu/Debian)wget https://github.com/sigstore/cosign/releases/download/v2.2.0/cosign-linux-amd64chmod +x cosign-linux-amd64sudo mv cosign-linux-amd64 /usr/local/bin/cosign # Verify installationcosign version# Output: cosign version 2.2.0

Tool 2: trivy (Vulnerability Scanning)

# macOSbrew install trivy # Linux (Ubuntu/Debian)wget https://github.com/aquasecurity/trivy/releases/download/v0.48.0/trivy_0.48.0_Linux-64bit.debsudo dpkg -i trivy_0.48.0_Linux-64bit.deb # Verify installationtrivy version# Output: Version: 0.48.0

Tool 3: grype (Vulnerability Scanning, Alternative)

# macOSbrew install grype # Linux (Ubuntu/Debian)curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b /usr/local/bin # Verify installationgrype version# Output: 0.76.0

Tool 4: crane (Image Inspection)

# macOSbrew install crane # Linux (Ubuntu/Debian)wget https://github.com/google/go-containerregistry/releases/download/v0.17.0/go-containerregistry_Linux_x86_64.tar.gztar xzf go-containerregistry_Linux_x86_64.tar.gzsudo mv crane /usr/local/bin/ # Verify installationcrane version# Output: v0.17.0

Tool 5: jq (JSON Parsing)

# macOSbrew install jq # Linux (Ubuntu/Debian)sudo apt-get install jq # Verify installationjq --version# Output: jq-1.7

Tool 6: kubectl (Kubernetes Testing Only)

# macOSbrew install kubectl # Linux (Ubuntu/Debian)curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"chmod +x kubectlsudo mv kubectl /usr/local/bin/ # Verify installationkubectl version --client# Output: Client Version: v1.28.0

Test 1: Image Signature Verification

What This Tests

Setup: Get the CleanStart Public Key

CleanStart publishes its cosign public key at:

https://registry.cleanstart.com/.cosign/cosign.pub

Save it locally:

curl -s https://registry.cleanstart.com/.cosign/cosign.pub > /tmp/cosign.pubcat /tmp/cosign.pub# Output starts with:# -----BEGIN PUBLIC KEY-----# MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...# -----END PUBLIC KEY-----

Step-by-Step Test

# Test image: python:3.12-stable from CleanStartIMAGE="registry.cleanstart.com/python:3.12-stable" # Verify the signaturecosign verify --key /tmp/cosign.pub "$IMAGE"

Expected Output (Pass)

Expected Output (Fail)

What to Do If It Fails

Check the image name is spelled correctly (case-sensitive)
Check the image exists: crane ls registry.cleanstart.com | grep python
Check network access to registry.cleanstart.com
If this is a custom golden image built internally, cosign verification may not apply

Pass/Fail Criteria

PASS: Verification successful, issuer is github.com/cleanstart/container-images. FAIL: No signature, tampered signature, or wrong issuer.

Test 2: SBOM Validation

What This Tests

Step-by-Step Test

Part A: Extract SBOMs from Image

IMAGE="registry.cleanstart.com/python:3.12-stable" # Extract SBOM from image (cosign convention)cosign download sbom "$IMAGE" > /tmp/sbom.json # Check if SBOM existsif [ -f /tmp/sbom.json ]; then  echo "✓ SBOM found"else  echo "✗ SBOM not found"  exit 1fi

Part B: Validate SBOM Format

# Check if it's valid JSONjq . /tmp/sbom.json > /dev/null 2>&1if [ $? -eq 0 ]; then  echo "✓ SBOM is valid JSON"else  echo "✗ SBOM is malformed"  exit 1fi # Check SBOM typeSBOM_TYPE=$(jq -r '.spdxVersion // .specVersion' /tmp/sbom.json)echo "SBOM format: $SBOM_TYPE" # For SPDX 3.0if [[ "$SBOM_TYPE" == "SPDX-3"* ]]; then  echo "✓ SPDX 3.0 format detected"fi # For CycloneDX 1.4if [[ "$SBOM_TYPE" == "1.4" ]]; then  echo "✓ CycloneDX 1.4 format detected"fi

Part C: Count Components

# For SPDX formatCOMPONENT_COUNT=$(jq '.packages | length' /tmp/sbom.json 2>/dev/null)if [ -n "$COMPONENT_COUNT" ]; then  echo "Components (SPDX): $COMPONENT_COUNT"fi # For CycloneDX formatCOMPONENT_COUNT=$(jq '.components | length' /tmp/sbom.json 2>/dev/null)if [ -n "$COMPONENT_COUNT" ]; then  echo "Components (CycloneDX): $COMPONENT_COUNT"fi

Part D: Check for Known Vulnerabilities

# Grype can consume SBOMs and report vulnerabilitiesgrype sbom:/tmp/sbom.json # Output shows:# Vulnerabilities by Severity# ────────────────────────────# Critical: 0# High:     0# Medium:   2# Low:      5# ...

Expected Output (Pass)

Expected Output (Fail)

### What to Do If It Fails 1. **No SBOM**: Check if this image supports SBOM (all CleanStart images should)2. **Malformed SBOM**: Contact CleanStart support, image may be corrupted3. **Unexpected vulnerabilities**:   - Check release notes for documented false positives   - Search CVE database for the specific CVE   - May be a scanner error, consult security team ### Pass/Fail Criteria - **PASS**: SBOM exists, is well-formed (valid JSON), format is SPDX 3.0 or CycloneDX 1.4, component count >100 (reasonable), no critical or unexpected vulnerabilities- **FAIL**: SBOM missing, malformed, wrong format, suspiciously low component count (<50), or critical vulnerabilities present ## Test 3: Shell-Less Verification ### What This Tests CleanStart images are shell-less: they don't include bash, sh, dash, or other shells. This reduces attack surface and prevents shell-based escape vectors. This test verifies that no shell binary exists in the image. ### Step-by-Step Test #### Method 1: Try to Execute a Shell (Most Obvious Test) ```bash# Try to run bash in a containerdocker run --rm registry.cleanstart.com/python:3.12-stable /bin/bash -c "echo hello"

Expected Output (Pass)

The shell execution attempt fails with an OCI runtime error, indicating that /bin/bash does not exist. This is the expected behavior for a shell-less image.

Expected Output (Fail)

The shell command executes successfully and prints "hello". This is unexpected and indicates the image contains shell binaries when it should be shell-less.

Method 2: Inspect Filesystem for Shell Binaries

# Extract image filesystem and search for shell binariescrane export registry.cleanstart.com/python:3.12-stable | tar -xzf - -O \  bin/bash bin/sh bin/dash usr/bin/bash usr/bin/sh 2>/dev/null | wc -l # If any shells are found, output will be >0SHELL_COUNT=$?if [ $SHELL_COUNT -eq 0 ]; then  echo "✓ No shell binaries found in filesystem"else  echo "✗ Found shell binaries (count: $SHELL_COUNT)"fi

Method 3: Check Busybox (Common Shell Vector)

# Busybox often provides a shell; verify it's not presentdocker run --rm registry.cleanstart.com/python:3.12-stable \  ls -la /bin/busybox 2>&1 | grep -q "cannot access" if [ $? -eq 0 ]; then  echo "✓ Busybox (shell provider) not found"else  echo "✗ Busybox found (could provide shell)"fi

Method 4: Check PATH for Shell References

# Get image config and check PATHcrane config registry.cleanstart.com/python:3.12-stable | jq -r '.config.Env[] | select(. | contains("PATH"))' # Output should be something like:# PATH=/usr/local/bin:/usr/bin:/bin## Then verify these directories don't contain shells:docker run --rm registry.cleanstart.com/python:3.12-stable \  find /usr/local/bin /usr/bin /bin -name "bash" -o -name "sh" -o -name "dash" 2>/dev/null # Should output nothing if shell-less

What to Do If Shells Are Found

Verify the image is actually from CleanStart (check registry URL)
Check if this is a custom image you built (might have shells)
Report to CleanStart support if shells found in official image
For custom images: rebuild from golden base, don't add shells

Pass/Fail Criteria

PASS: No shell binaries found, shell execution attempt fails, busybox not present
FAIL: Shell binaries found, shell execution succeeds, busybox present

Test 4: Read-Only Filesystem Verification

What This Tests

CleanStart images have read-only root filesystems. Applications can write to /tmp and other writable mounts, but cannot modify system files.

This test verifies that the filesystem is actually read-only.

Step-by-Step Test

Method 1: Try to Write to System Directory

# Attempt to write to /etc (should fail)docker run --rm registry.cleanstart.com/python:3.12-stable \  touch /etc/test-file 2>&1

Expected Output (Pass)

The write attempt fails with a "Read-only file system" error. This is the expected behavior for a properly configured read-only root filesystem.

Expected Output (Fail)

The file is created without error, which is unexpected because the filesystem should be read-only.

Method 2: Check Writable Mounts

# List mounted filesystems inside containerdocker run --rm registry.cleanstart.com/python:3.12-stable \  mount | grep -E "rw" | grep -v "devtmpfs" # Output should show only tmpfs mounts:tmpfs on /tmp type tmpfs (rw, ...)tmpfs on /run type tmpfs (rw, ...)tmpfs on /dev/shm type tmpfs (rw, ...) # Root filesystem should be read-only:# /dev/... on / type ... (ro, ...)

Method 3: Verify in Kubernetes

# Create a test pod with read-only root filesystemkubectl apply -f - <<EOFapiVersion: v1kind: Podmetadata:  name: readonly-testspec:  containers:Name: app.    image: registry.cleanstart.com/python:3.12-stable    securityContext:      readOnlyRootFilesystem: true  restartPolicy: NeverEOF # Wait for pod to startsleep 3 # Try to write a filekubectl exec readonly-test -- touch /test-file 2>&1 # Should fail:# Read-only file system# command terminated with exit code 1

What to Do If Writes Succeed

This is a critical security issue
Report immediately to CleanStart support
Do not use this image in production until resolved
Use a previously-tested golden image version instead

Pass/Fail Criteria

PASS: Cannot write to system directories, writable mounts are tmpfs only, Kubernetes pod respects readOnlyRootFilesystem
FAIL: Can write to system directories, root filesystem is writable, Kubernetes pod can write to root

Test 5: Non-Root User Verification

What This Tests

CleanStart images run as non-root (UID 65532, a reserved unprivileged user). This prevents container escapes from leading to host root access.

This test verifies the running user is not root.

Step-by-Step Test

Method 1: Check Running UID

# Start container and check UIDdocker run --rm registry.cleanstart.com/python:3.12-stable \  id # Expected output:# uid=65532 gid=65532 groups=65532

Expected Output (Pass)

The user ID is 65532 (nobody), group ID is 65532 (nogroup). The container successfully runs as non-root as expected.

Expected Output (Fail)

The user ID is 0 (root), group ID is 0 (root). This is unexpected because the container should run as a non-root user.

Method 2: Verify No SUID Binaries

SUID binaries allow non-root users to temporarily gain root privileges for specific operations. Removing them reduces privilege escalation vectors.

# Search for SUID binaries (should be none)docker run --rm registry.cleanstart.com/python:3.12-stable \  find / -perm /4000 2>/dev/null # Output should be empty (no SUID binaries)

Expected Output (Pass)

The search returns no output, indicating no SUID binaries are present in the image. This is the expected behavior.

Expected Output (Fail)

The search finds SUID binaries such as /usr/bin/sudo and /usr/bin/passwd. This is unexpected and represents a potential privilege escalation vector.

Method 3: Check Capability Drops

Linux capabilities are granular permissions. Even if a non-root user runs the container, if they have the CAP_SYS_ADMIN capability, they can still escape.

# Check capabilities in running containerdocker run --rm registry.cleanstart.com/python:3.12-stable \  grep Cap /proc/self/status # Output should show all capabilities dropped:# CapInh:	0000000000000000# CapPrm:	0000000000000000# CapEff:	0000000000000000# CapBnd:	0000000000000000

Expected Output (Pass)

All capability fields (inherited, permitted, effective, bounding, and ambient) show all zeros (0000000000000000), indicating that all Linux capabilities have been dropped as expected.

Expected Output (Fail)

The effective capabilities field shows a non-zero value (00000000a80425fb), indicating that capabilities are still present. This is unexpected because all capabilities should be fully dropped.

Method 4: Verify in Kubernetes

# Pod with runAsNonRoot enforcedkubectl apply -f - <<EOFapiVersion: v1kind: Podmetadata:  name: nonroot-testspec:  securityContext:    runAsNonRoot: true    runAsUser: 65532  containers:Name: app.    image: registry.cleanstart.com/python:3.12-stable    securityContext:      allowPrivilegeEscalation: false      capabilities:        drop:ALL.  restartPolicy: NeverEOF # Pod should start successfullykubectl get pod nonroot-test# STATUS: Running

What to Do If Running as Root

This is a critical security issue
Report to CleanStart support
Do not use this image in production
Verify you're using the correct image (not a custom variant)

Pass/Fail Criteria

PASS: Running UID is 65532, no SUID binaries, all capabilities dropped, Kubernetes nonroot enforcement works
FAIL: Running UID is 0, SUID binaries present, capabilities present, Kubernetes nonroot enforcement fails

Test 6: FIPS Mode Verification (FIPS Images Only)

What This Tests

CleanStart offers FIPS 140-3 certified images. This test verifies cryptographic algorithms are restricted to FIPS-approved ones.

Note: Only relevant for registry.cleanstart.com/*/fips images. Skip this test for standard images.

Step-by-Step Test

Method 1: Check OpenSSL FIPS Mode

# Check if OpenSSL FIPS module is enableddocker run --rm registry.cleanstart.com/python:3.12-fips \  python3 -c "import sslprint('FIPS mode:', ssl.OPENSSL_VERSION)" # Output:# FIPS mode: OpenSSL 3.0.x (FIPS enabled)

Expected Output (Pass)

FIPS mode: OpenSSL 3.0.8 (FIPS) ✓ FIPS mode enabled

Method 2: Test Restricted Algorithms

FIPS mode rejects certain weak algorithms. MD5 and DES are not allowed.

# Try MD5 (should fail in FIPS mode)docker run --rm registry.cleanstart.com/python:3.12-fips \  python3 -c "import hashlibtry:  md5 = hashlib.md5()  print('ERROR: MD5 allowed (FIPS mode not active)')except ValueError as e:  print('MD5 blocked:', str(e))  print('✓ FIPS mode enforcing')" # Expected output:# MD5 blocked: unsupported hash type md5. ...# ✓ FIPS mode enforcing

Method 3: Verify TLS Cipher Suites

# Check available TLS ciphersdocker run --rm registry.cleanstart.com/python:3.12-fips \  python3 -c "import sslctx = ssl.create_default_context()ciphers = ctx.get_ciphers()for c in ciphers:  print(c['name'])" # Should only show FIPS-approved ciphers (TLS_AES_256_GCM_SHA384, etc.)# Should NOT include: TLS_RSA_*, DES-*, MD5, RC4, etc.

What to Do If FIPS Tests Fail

Verify you're using a FIPS image (-fips suffix)
Check OpenSSL version (must be 3.0+)
Report to CleanStart support if tests fail on FIPS image

Pass/Fail Criteria (FIPS Images Only)

PASS: FIPS mode enabled in OpenSSL, weak algorithms blocked, only FIPS ciphers available
FAIL: FIPS mode disabled, weak algorithms allowed, non-FIPS ciphers present

Test 7: Network Security Verification

What This Tests

CleanStart images expose only necessary network ports. Services listening on unexpected ports could be security risks.

This test verifies network exposure is minimal.

Step-by-Step Test

Method 1: Port Scanning

# Start container in backgroundCONTAINER_ID=$(docker run -d registry.cleanstart.com/python:3.12-stable sleep 1000) # Get container IPCONTAINER_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $CONTAINER_ID) # Scan for open portsnmap -p 1-65535 $CONTAINER_IP 2>/dev/null | grep "open" # Clean updocker rm -f $CONTAINER_ID

Expected Output (Pass)

No output is returned, or only ephemeral ports are shown. No unexpected ports are listening.

Expected Output (Fail)

The scan shows open ports for SSH (22/tcp) and Telnet (23/tcp). These are unexpected services that should not be listening.

Method 2: Check Listening Services

# List all listening sockets inside containerdocker run --rm registry.cleanstart.com/python:3.12-stable \  netstat -tlnp 2>/dev/null || ss -tlnp # Should output only application-specific ports, not sshd, telnet, etc.

Pass/Fail Criteria

PASS: No unexpected ports listening, only application services exposed
FAIL: SSH, Telnet, other management services listening

Test 8: Vulnerability Scan Verification

What This Tests

Vulnerability scanners (Trivy, Grype) identify known vulnerable components. This test runs scanners and verifies results are consistent with CleanStart's published vulnerability reports.

Step-by-Step Test

Method 1: Scan with Trivy

# Scan image for vulnerabilitiestrivy image registry.cleanstart.com/python:3.12-stable # Output:# Total: 347 (CRITICAL: 0, HIGH: 0, MEDIUM: 2, LOW: 5)

Method 2: Scan with Grype

# Grype provides alternative scangrype registry.cleanstart.com/python:3.12-stable # Output:# Vulnerabilities By Severity# ──────────────────────────# Critical: 0# High:     0# Medium:   2# Low:      5

Method 3: Compare to CleanStart Release Notes

# CleanStart publishes official scan results# Compare your results to: https://registry.cleanstart.com/releases/python/3.12-stable/scan-results.json # Expected:# Your scan: CRITICAL=0, HIGH=0, MEDIUM=2, LOW=5# Official: CRITICAL=0, HIGH=0, MEDIUM=2, LOW=5 (match!) # If counts differ, investigate:# - Scanner version difference# - Database update timing (scanners auto-update vulnerability database)# - False positives (documented in release notes)

Understanding Scanner Differences

Trivy and Grype may report different numbers due to:

Different vulnerability databases: Trivy uses trivy.dev, Grype uses Grype DB
Different severity scoring: CVE severity != CVSS severity
False positives: Some findings are known to be incorrect

Example:

Image: python:3.12-stableTrivy report: 2 medium-severity CVEsGrype report: 5 medium-severity CVEsDifference: Grype may be more conservative, Trivy may filter some false positives Solution: Check release notes, understand why Grype found more

What to Do If Scans Show Unexpected Vulnerabilities

Check release notes: Is this documented as a false positive?
Check CVE details: Search NVD.NIST.gov for the CVE, is it applicable?
Check VEX (Vulnerability Exploitability Exchange): CleanStart publishes VEX statements explaining false positives
Consult security team: Determine if it's an actual risk

Pass/Fail Criteria

PASS: Scan succeeds, results match CleanStart release notes, no critical/high findings, medium/low findings are documented
FAIL: Scan fails, results differ unexpectedly from release notes, critical findings present

Test 9: Container Isolation Verification

What This Tests

Kubernetes and container runtimes provide multiple isolation layers. This test verifies they're configured correctly in CleanStart images.

Step-by-Step Test

Method 1: Verify PID Namespace Isolation

# Two containers should not see each other's processesCONTAINER1=$(docker run -d registry.cleanstart.com/python:3.12-stable sleep 1000)CONTAINER2=$(docker run -d registry.cleanstart.com/python:3.12-stable sleep 1000) # In container 2, list processesdocker exec $CONTAINER2 ps aux # Should only show container 2's processes, not container 1# If you see container 1's processes, namespace isolation failed

Expected Output (Pass)

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMANDnobody       1  0.1  0.0   5000   500 ?        Ss   10:00   0:00 sleep 1000 ✓ Only local processes visible (good isolation)

Expected Output (Fail)

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMANDnobody       1  0.1  0.0   5000   500 ?        Ss   10:00   0:00 sleep 1000nobody       7  0.1  0.0   5000   500 ?        Ss   10:00   0:00 sleep 1000 (OTHER CONTAINER!) ✗ Can see other container's processes (isolation failure)

Method 2: Check seccomp Profile

# Check if seccomp is enableddocker inspect registry.cleanstart.com/python:3.12-stable | jq '.[0].HostConfig.SecurityOpt' # Output:# ["seccomp=unconfined"] ← seccomp NOT configured (default for base image)## In Kubernetes, you should configure:# securityContext:#   seccompProfile:#     type: RuntimeDefault

Method 3: Verify AppArmor/SELinux Profile (Kubernetes)

# Pod with AppArmorkubectl apply -f - <<EOFapiVersion: v1kind: Podmetadata:  name: isolation-test  annotations:    container.apparmor.security.beta.kubernetes.io/app: runtime/defaultspec:  containers:Name: app.    image: registry.cleanstart.com/python:3.12-stable  restartPolicy: NeverEOF # Pod should start and AppArmor should be appliedkubectl get pod isolation-test

Pass/Fail Criteria

PASS: PID namespace isolation verified, seccomp/AppArmor available and configurable
FAIL: Can see other containers' processes, isolation disabled

Test 10: Supply Chain Verification

What This Tests

CleanStart publishes SLSA Level 4 provenance, proving images are built from known source code. This test verifies supply chain integrity.

Step-by-Step Test

Method 1: Download Provenance Attestation

# SLSA provenance is stored as an OCI attestation# Use cosign to retrieve itcosign download attestation registry.cleanstart.com/python:3.12-stable > /tmp/provenance.json # Verify it's valid JSONjq . /tmp/provenance.json > /dev/nullif [ $? -eq 0 ]; then  echo "✓ Provenance attestation found and valid"else  echo "✗ Provenance attestation invalid"  exit 1fi

Method 2: Verify Build Source

# Extract build source from provenancejq -r '.payload.buildConfig.source.repository' /tmp/provenance.json # Output should be CleanStart's source repository:# https://github.com/cleanstart/container-images # Verify build timestamp is recentBUILD_TIME=$(jq -r '.payload.buildStartedOn' /tmp/provenance.json)echo "Built: $BUILD_TIME" # Calculate ageNOW=$(date +%s)BUILD_TIMESTAMP=$(date -d "$BUILD_TIME" +%s)AGE_DAYS=$(( ($NOW - $BUILD_TIMESTAMP) / 86400 ))echo "Image age: $AGE_DAYS days" if [ $AGE_DAYS -lt 365 ]; then  echo "✓ Image built within last year (good freshness)"else  echo "⚠ Image is older than 1 year (consider updating)"fi

Method 3: Verify Source-to-Image Mapping

# Check that image digest matches build outputIMAGE_DIGEST=$(crane digest registry.cleanstart.com/python:3.12-stable)BUILD_OUTPUT=$(jq -r '.payload.buildOutputs[0].digest' /tmp/provenance.json) if [ "$IMAGE_DIGEST" = "$BUILD_OUTPUT" ]; then  echo "✓ Image digest matches provenance (supply chain unbroken)"else  echo "✗ Image digest doesn't match provenance (possible tampering)"fi

Expected Output (Pass)

✓ Provenance attestation found and validBuilt: 2025-03-15T10:30:00ZImage age: 7 days✓ Image built within last year (good freshness)Source repository: https://github.com/cleanstart/container-images✓ Image digest matches provenance (supply chain unbroken)

Expected Output (Fail)

✗ Provenance attestation invalid  (or)✗ Image digest doesn't match provenance (possible tampering)

What to Do If Provenance Verification Fails

Image may be corrupted in transit
Image may be from an untrusted source
Report to CleanStart support immediately
Do not use in production

Pass/Fail Criteria

PASS: Provenance exists and is valid, source is CleanStart repo, image age reasonable, digest matches
FAIL: Provenance missing or invalid, source is wrong, digest mismatch

Automation: Run All 10 Tests

Here's a bash script that runs all 10 tests and produces a summary report:

#!/bin/bash# test-cleanstart-security.sh - Run all security tests on CleanStart image set -e IMAGE="${1:-registry.cleanstart.com/python:3.12-stable}"REPORT_FILE="cleanstart-security-report-$(date +%Y%m%d-%H%M%S).txt" echo "╔════════════════════════════════════════════════════════════╗"echo "║    CleanStart Security Testing Report                       ║"echo "╚════════════════════════════════════════════════════════════╝"echo ""echo "Image: $IMAGE"echo "Report: $REPORT_FILE"echo "" # Color codesGREEN='\033[0;32m'RED='\033[0;31m'YELLOW='\033[1;33m'NC='\033[0m' # No Color PASS_COUNT=0FAIL_COUNT=0 # Helper functionspass_test() {  echo -e "${GREEN}✓ PASS${NC}: $1"  ((PASS_COUNT++))} fail_test() {  echo -e "${RED}✗ FAIL${NC}: $1"  ((FAIL_COUNT++))} warn_test() {  echo -e "${YELLOW}⚠ WARN${NC}: $1"} # Test 1: Signature Verificationecho ""echo "Test 1: Signature Verification"cosign verify --key /tmp/cosign.pub "$IMAGE" > /dev/null 2>&1 && pass_test "Image signature valid" || fail_test "Image signature invalid" # Test 2: SBOM Validationecho ""echo "Test 2: SBOM Validation"cosign download sbom "$IMAGE" > /tmp/sbom.json 2>/dev/nulljq . /tmp/sbom.json > /dev/null 2>&1 && pass_test "SBOM format valid" || fail_test "SBOM format invalid" # Test 3: Shell-Less Verificationecho ""echo "Test 3: Shell-Less Verification"docker run --rm "$IMAGE" /bin/bash -c "echo" 2>&1 | grep -q "no such file" && pass_test "No shell binary found" || fail_test "Shell binary found" # Test 4: Read-Only Filesystemecho ""echo "Test 4: Read-Only Filesystem"docker run --rm "$IMAGE" touch /etc/test 2>&1 | grep -q "Read-only" && pass_test "Filesystem is read-only" || fail_test "Filesystem is writable" # Test 5: Non-Root Userecho ""echo "Test 5: Non-Root User"docker run --rm "$IMAGE" id | grep -q "uid=65532" && pass_test "Running as non-root (UID 65532)" || fail_test "Not running as expected non-root user" # Test 6: No SUID Binariesecho ""echo "Test 6: SUID Binary Check"docker run --rm "$IMAGE" find / -perm /4000 2>/dev/null | wc -l | grep -q "^0$" && pass_test "No SUID binaries found" || warn_test "SUID binaries found (check if expected)" # Test 7: Vulnerability Scanecho ""echo "Test 7: Vulnerability Scan (Trivy)"trivy image "$IMAGE" 2>/dev/null | grep -q "CRITICAL: 0" && pass_test "No critical vulnerabilities" || fail_test "Critical vulnerabilities found" # Test 8: Network Securityecho ""echo "Test 8: Network Services"EXPOSED_PORTS=$(docker run --rm "$IMAGE" netstat -tlnp 2>/dev/null | grep LISTEN | wc -l)[ "$EXPOSED_PORTS" -le 2 ] && pass_test "Minimal network exposure" || warn_test "Multiple services listening (verify if expected)" # Test 9: Provenanceecho ""echo "Test 9: Supply Chain Provenance"cosign download attestation "$IMAGE" > /tmp/provenance.json 2>/dev/nulljq . /tmp/provenance.json > /dev/null 2>&1 && pass_test "SLSA provenance valid" || fail_test "Provenance missing or invalid" # Summaryecho ""echo "╔════════════════════════════════════════════════════════════╗"echo "║ Summary                                                     ║"echo "╚════════════════════════════════════════════════════════════╝"echo ""echo -e "${GREEN}Passed: $PASS_COUNT${NC}"echo -e "${RED}Failed: $FAIL_COUNT${NC}"echo "" if [ $FAIL_COUNT -eq 0 ]; then  echo -e "${GREEN}✓ All tests passed! Image is secure.${NC}"  exit 0else  echo -e "${RED}✗ Some tests failed. Review failures above.${NC}"  exit 1fi

Usage:

chmod +x test-cleanstart-security.sh./test-cleanstart-security.sh registry.cleanstart.com/python:3.12-stable

Interpreting Common Results

Scenario 1: Trivy Found CVEs That CleanStart Says Are Fixed

Situation:

Trivy scan shows: CVE-2025-1234 (Medium) in opensslCleanStart release notes show: Fixed in v2.1.0

Explanation: Trivy's vulnerability database may not be fully up-to-date. CleanStart may have built an image with patched openssl, but Trivy's database hasn't refreshed yet.

Resolution:

# Update Trivy's databasetrivy image --severity HIGH,CRITICAL registry.cleanstart.com/python:3.12-stable # If still shows after update, check VEX statement:cosign download sbom registry.cleanstart.com/python:3.12-stable | jq '.vex'

Scenario 2: Shell Test Passed but App Needs Shell for Debugging

Situation:

Container runs fine in testingBut in production, developer needs to debug: "I need to run /bin/bash"

Explanation: CleanStart is shell-less by design. For debugging, use:

kubectl exec → ephemeral container with busybox (added at debug time, not in image)
kubectl debug → temporary sidecar with shell
Application logs → best debugging source

Solution:

# Debug with ephemeral container (Kubernetes 1.23+)kubectl debug -it <pod-name> --image=busybox # Application logging (better approach)# App outputs structured logs to stderr# Operator tails logs: kubectl logs <pod-name>

Scenario 3: FIPS Test Shows Non-FIPS Algorithm

Situation:

Testing FIPS image, but seeing TLS_RSA_* ciphers (non-FIPS)

Explanation: Application may be using older TLS configuration that allows non-FIPS ciphers. The system supports them, but you should restrict application config to FIPS-only.

Solution:

# In your Python/Node/etc applicationimport ssl # Force FIPS-only cipherscontext = ssl.create_default_context()context.set_ciphers(':'.join([    'ECDHE-ECDSA-AES256-GCM-SHA384',    'ECDHE-RSA-AES256-GCM-SHA384',]))

Scenario 4: Container Isolation Test Sees Other Container's Processes

Situation:

Running two containers, both can see each other's processes

Explanation: You may be running in a non-isolated mode (--pid=host). Check your docker run command or Kubernetes pod spec.

Solution:

# ❌ WRONGdocker run --pid=host registry.cleanstart.com/python:3.12-stable # ✅ CORRECT (default isolation)docker run registry.cleanstart.com/python:3.12-stable