Container security often focuses on the image: scanning it, verifying signatures, checking policies. But the real risks are deeper — in the code before the build, in the build process itself, and in the low-level system libraries that everything depends on. Most security programs miss these entirely.

The Problem: Security Defenders Focus on the Wrong Layer

What Defenders Focus On          What Attackers Target─────────────────────────────────────────────────────────────────Image signing                    Source code compromiseRegistry scanning                Build system poisoningPolicy enforcement               Compiler manipulationNetwork policies                 System library vulnerabilitiesRuntime monitoring               Memory-unsafe code paths

Defenders focus on Layers 3 and 4 (registry and runtime) because those are visible and measurable. Attackers focus on Layers 1 and 2 (source and build) because those are less defended.

A vulnerability scanner at Layer 3 (registry) sees a container image and looks for known CVEs in packages. It sees openssl-3.0.11 and checks if that version has a known CVE. But it cannot see how the build system compiled OpenSSL, what patches were applied at compile time, whether compiler optimizations introduced vulnerabilities, whether the source code itself (not the library) contains exploitable patterns, or whether build artifacts were tampered with between build and deployment.

An attacker who compromises the build system can inject code that no registry scanner will detect because the resulting binary is valid. The source code in the repository is clean. The compiled artifact matches the build configuration. But the actual code executed is malicious.

Code-Level Vulnerabilities: Memory-Unsafe Languages and Unsafe Patterns

Most container vulnerabilities trace back to code-level issues in memory-unsafe languages (C, C++) or unsafe programming patterns.

Memory Safety Issues in C/C++

System-level libraries (glibc, OpenSSL, curl, zlib, and thousands of others) are written in C because performance matters at that level. C gives you raw memory access and responsibility to manage that memory safely. When you fail at memory management, attackers exploit the failure.

Buffer overflows are the classic vulnerability where a function reads data into a fixed-size buffer without checking the input length. An attacker sends data longer than the buffer, overwriting memory adjacent to it.

// Vulnerable code (simplified)void process_user_input(char *input) {    char buffer[64];    strcpy(buffer, input);  // No length check!    // If input is longer than 64 bytes,    // it overwrites memory after buffer}

The attacker sends 100 bytes to a 64-byte buffer. The extra 36 bytes overflow into memory containing the return address or function pointers. By crafting the input carefully, the attacker can overwrite the return address and redirect code execution to their payload.

This vulnerability class has been understood for 30+ years, yet it persists because C has no bounds checking—the language itself allows unsafe memory access. Many developers aren't trained in memory safety. Performance-critical code often sacrifices safety for speed. Legacy code was written before modern security practices were established.

Modern C can be written safely (checking buffer lengths, using safer functions), but legacy code in critical libraries was not written with safety in mind.

Use-after-free is a vulnerability where code accesses memory after it's been freed.

// Vulnerable code (simplified)void process_data() {    char *data = malloc(100);    // ... use data ...    free(data);    // ... later ...    strcpy(result, data);  // data was freed! Use-after-free}

When the memory is freed, it's returned to the heap and might be reallocated for something else. The code is now reading from memory that's been reused. If an attacker controls the memory allocation that reuses that space, they can make the code read or write to attacker-controlled data.

Use-after-free is often exploitable for code execution because the freed memory might contain function pointers or other control data.

Heap corruption occurs when an attacker writes to heap memory (through a buffer overflow or other vulnerability) and corrupts heap metadata. This can lead to arbitrary code execution when the heap is managed later. Stack smashing happens when an attacker overflows the stack (local variables, return addresses, etc.) to overwrite the return address and redirect code execution.

All of these vulnerabilities trace back to the same root cause: memory-unsafe languages allow programs to write to arbitrary memory addresses if you make a mistake or if an attacker can trigger a mistake.

Vulnerable Patterns in High-Level Languages

Even memory-safe languages can have exploitable patterns. Unsafe deserialization occurs when code that deserializes untrusted data (JSON, XML, pickle, etc.) can be tricked into instantiating arbitrary objects or executing code.

# Vulnerable code (simplified)import pickledata = request.get_json()obj = pickle.loads(data)  # If data is untrusted, can execute code

An attacker sends a specially-crafted pickle payload that, when deserialized, instantiates objects that execute code in their __init__ method or other magic methods.

Command injection occurs when code constructs shell commands from user input without proper escaping.

# Vulnerable codecommand = f"ls -la {user_provided_path}"os.system(command)  # If user_provided_path is "; rm -rf /", this executes that

An attacker provides a path like "; cat /etc/passwd", and the shell executes both the intended command and the attacker's command.

SQL injection follows a similar pattern but with SQL queries.

-- Vulnerable codequery = f"SELECT * FROM users WHERE id = {user_input}"-- If user_input is "1 OR 1=1", this returns all users

XML External Entity (XXE) attacks occur when XML parsers that resolve external entities can be exploited to read arbitrary files or cause denial of service. Regular expression denial of service (ReDoS) happens when a carefully-crafted input causes a regular expression to backtrack exponentially, consuming CPU until the server is unresponsive.

These vulnerabilities are code-level issues. Scanning the package versions won't detect them. You need static analysis (SAST) that can recognize unsafe patterns in your source code.

Build-Time Vulnerabilities: Trust, Reproducibility, and Secrets

The build process transforms source code into runnable artifacts. This is where multiple attack surfaces open up.

Non-Hermetic Builds: The Reproducibility Problem

A hermetic build means the output depends only on the source code and explicit inputs. The same source code, given to the same build system, produces identical output (byte-for-byte).

Most container builds are not hermetic. They depend on the state of the build machine (installed packages, environment variables, etc.), network access to pull dependencies, the current time and date (which might be baked into timestamps), build tool versions and configurations, and cached build artifacts.

Because builds are not hermetic, the same source code produces different binaries at different times or on different machines. This breaks reproducibility.

Reproducibility matters for security in several ways. Verification requires that if you rebuild your application and get a different binary, you can verify that the binary matches the source. An attacker could have tampered with the binary between the original build and your rebuild, and without reproducibility, you'd have no way to detect it. Patch isolation is important if you want to apply a security patch to one library without rebuilding your entire application. Non-hermetic builds make this impossible because rebuilding triggers changes throughout the dependency tree, and you don't know which changes are from your patch and which are from environment differences. Supply chain verification is critical if you want to verify that your container image came from your source code and your build process (not from a compromised registry or intermediate storage). Non-reproducible builds break this verification chain entirely.

Example of non-hermetic build problem:

FROM ubuntu:latestRUN apt-get update && apt-get install -y curlRUN curl https://example.com/download-and-compile.sh | bash

This Dockerfile pulls a shell script from the internet and executes it. Each time you build, the script might be different (if it's generated dynamically or if the upstream has updated). You have no way to know what script is being executed.

Even without downloading code, simple non-hermetic patterns exist:

FROM ubuntu:latestRUN apt-get install -y openssl  # Gets whatever version apt decidesRUN npm install                  # Gets dependencies from npm registry

Without pinning versions, the same Dockerfile produces different artifacts at different times. If a security patch is released, your next build might automatically include it (good), but your previous builds remain unpatched (bad). You have no control over the timeline.

Secrets in Build Artifacts

Secrets leak into build artifacts in multiple ways. Credentials in source code occur when a developer hardcodes a database password or API key in source code. They later delete it in a commit, but it remains in git history and anyone with repository access can extract it.

# Committed to gitDB_PASSWORD = "super-secret-password-123"

Even if deleted, it's still in git log. Attackers can clone the repository and find it.

Secrets in Docker layers happen when a developer uses a secret to authenticate with a private package registry, then forgets to clean up the build layer.

RUN apt-get install -y my-private-package --auth-token=secret-token-xyz# The secret is baked into the layer history

Even if you use a multi-stage build to avoid including the secret in the final image, the secret is still in intermediate layers and build history.

Environment variables used for secrets during build might be logged in CI/CD systems, build logs, or container history.

docker build --build-arg API_KEY=secret-key-xyz .# The argument is visible in docker history, build logs, etc.

Credentials in artifact metadata occur when build systems like Jenkins log all environment variables. If a secret is passed as an environment variable, it ends up in build logs that are stored permanently.

Secrets that leak during build are often rotated slowly or not at all. An attacker who gains access to old build artifacts can extract secrets that were used months or years ago. If those credentials haven't been rotated, they're still valid.

Base Image Supply Chain

When you start a Dockerfile with FROM ubuntu:22.04, you're implicitly trusting that the image came from Canonical (the Ubuntu maintainer) and hasn't been tampered with.

This trust is not enforced. Container registries have no mandatory signature verification. You can pull a base image, and if your network connection is intercepted (Man-in-the-Middle attack), the registry is compromised, your pull credentials are stolen, or a naming collision occurs (you pull from a different registry than intended), then you get a compromised base image, and every container built from it is compromised.

The xz/liblzma backdoor (CVE-2024-3094) is relevant here. A backdoor was introduced into the xz compression library source. While this particular backdoor was in the source (not the build), if a base image maintainer had been compromised, the backdoor would have been in every base image built from that time forward.

Mitigations exist (container image signatures via Cosign or similar tools), but they're not enabled by default. Most organizations don't verify base image signatures.

Build System Compromise

If the CI/CD system that builds your containers is compromised, every artifact it produces is compromised.

A compromised build system can inject malicious code into binaries while compiling, modify source code before building (undetectable without comparing with git), intercept dependencies and poison them, exfiltrate secrets stored in the build system, or modify the final artifact after building.

Examples include the SolarWinds supply chain attack (2020), where attackers compromised the build system of a widely-used network management software and the official builds were injected with malicious code, resulting in thousands of organizations installing what they thought was legitimate software. The XCodeGhost (2015) incident occurred when developers in China downloaded Xcode from a modified source (not Apple's official source), and the modified version contained malicious code that was then present in applications built with this modified Xcode.

A build system compromise is often undetectable for months because the build artifacts look legitimate. The source code is clean. The build logs look normal. But the final binary is malicious.

Low-Level Security: System Libraries and Kernel Interfaces

Below your application code and its libraries, everything runs on top of system libraries and the kernel. Vulnerabilities at this level affect every container.

System Library Vulnerabilities

Every dynamically-linked binary in your container depends on system libraries. glibc (GNU C Library) implements standard C library functions (malloc, strcmp, strlen, file I/O, etc.), and every C/C++ program links to glibc. A vulnerability in glibc affects every binary that uses it. If glibc has a buffer overflow, every program using that function is potentially exploitable. OpenSSL (or other TLS libraries) implements TLS/SSL encryption, and every binary that makes HTTPS connections depends on it. A vulnerability in OpenSSL might not affect your application code directly, but if your ORM uses it for database connections or if your web server uses it for HTTPS, you're affected. libc++ (C++ standard library) plays a similar role for C++ programs. The Linux kernel serves as the boundary between containers and the host through system calls.

A vulnerability in the kernel is exploitable from any container. If a kernel vulnerability allows privilege escalation, a container can escape and compromise the host.

The Kernel Interface: System Calls and Capabilities

Containers use Linux system calls to interact with the kernel. The system call interface is the boundary between container and host.

By default, containers have access to many system calls including mount() for mounting filesystems (which can mount the host root filesystem), reboot() for rebooting the system, sysctl() for modifying kernel parameters, ioctl() for interacting with devices, ptrace() for attaching to processes (allowing debugging), and key*() for managing kernel keyrings.

Some of these system calls, if available to a container, allow privilege escalation or container escape. A container with mount() can mount the host filesystem and modify it. A container with ptrace() can debug the host's processes. A container with access to /dev/mem can read and write host memory. A container with certain capabilities can manipulate cgroups and break isolation.

This is why capabilities and seccomp (secure computing) exist: to restrict what system calls a container can make and what capabilities it has.

A default container often has more capabilities than it needs. If the container is compromised, the attacker has access to dangerous system calls that enable further exploitation.

Glibc Vulnerabilities and Implications

glibc has had multiple vulnerabilities that, while patched, affect many systems. GLIBC_2.x vulnerabilities include stack-based buffer overflows in functions like gets(), sprintf(), and others that were used in real exploits years ago. DNS resolution vulnerabilities affect the glibc resolver which has had issues with cache poisoning and denial of service. Locale handling vulnerabilities include buffer overflows in locale-handling functions.

When a glibc vulnerability is discovered, the patch is released, Linux distributions rebuild glibc, base image maintainers rebuild base images, your CI/CD rebuilds containers using the new base image, and you deploy the new containers. During this entire timeline, all containers using the vulnerable glibc version are at risk.

The speed of this timeline depends on multiple factors, as discussed earlier. A critical glibc vulnerability might take 30+ days to propagate through all layers.

Why Scanning Misses These Issues

Registry-level vulnerability scanning looks at package versions and matches them against CVE databases. It misses code-level vulnerabilities where your application code might have unsafe deserialization or command injection—these aren't CVEs in known packages but vulnerabilities in your own code requiring static analysis (SAST) to find. It misses build-time vulnerabilities like non-hermetic builds, secret leaks, and build system compromises that don't show up in the final image as vulnerabilities—the artifact is valid and scanning can't detect the build process problems. It misses compiler-level vulnerabilities where a compromised or buggy compiler could make the resulting binary vulnerable even though the source code is secure (though this is extremely rare). It misses configuration vulnerabilities where the image runs as root, includes shells, or has writable filesystems—these aren't CVEs but configuration issues that make the container more exploitable. Finally, it misses transitive code behavior issues where a vulnerability in a library might require specific conditions to be exploitable (certain input types, race conditions, specific compiler optimizations), and scanning can't determine if those conditions are met in your usage.

What Code, Build-Time, and Low-Level Security Requires

Securing code, build, and low-level aspects requires different tools and approaches. For code-level security, use SAST (Static Application Security Testing) to analyze source code for unsafe patterns including buffer overflows and use-after-free in C/C++, unsafe deserialization in Python and Java, command injection and SQL injection patterns, hardcoded secrets, and unvalidated inputs. Tools include SonarQube, Checkmarx, Fortify, and Snyk Code. Use SCA (Software Composition Analysis) to identify vulnerabilities in dependencies and transitive libraries, with tools like Snyk, WhiteSource, Black Duck, and Dependabot. Employ fuzzing by providing random or semi-random inputs to your code to find crashes and vulnerabilities, using tools like AFL, libFuzzer, and ClusterFuzz. Include code review where humans reading code can catch vulnerabilities that tools miss.

Build-Time Security

For build-time security, use supply chain security tools to verify that build artifacts came from your build system and weren't tampered with, including tools like Cosign, Sigstore, in-toto, and Grafeas. Implement reproducible builds to ensure the same source code produces the same artifact, enabling verification through build system configuration, version pinning, and containerized build environments. Perform build log analysis to scan build logs for signs of compromise. Use secrets scanning to detect credentials in source code before they're committed, with tools like truffleHog, GitGuardian, and TruffleHog. Implement base image verification to verify base images are signed and haven't been tampered with, using tools like Cosign and Docker Content Trust.

Low-Level Security

For low-level security, implement security hardening by removing unnecessary capabilities, enabling seccomp, and using read-only filesystems. Drop unnecessary Linux capabilities to restrict what the container can do. Use seccomp profiles to restrict system calls to only those the container needs. Keep system library updates current by patching glibc, OpenSSL, and other system libraries. Deploy kernel security modules like AppArmor or SELinux to enforce mandatory access control.

The Source-Built Advantage

One approach to Layer 2 and low-level security is source-based distribution and compile-time recompilation.

Instead of distributing pre-compiled binaries (which have opaque build provenance), you distribute source code and compile it yourself. This allows you to implement patch isolation by applying a security patch to OpenSSL without rebuilding your entire application, which means you can recompile the patched OpenSSL and relink your application so only the necessary components are rebuilt. You gain build verification because you control the build environment, can ensure hermetic builds, verify no secrets are leaked, and trace every step. You achieve patch flexibility in the event that a maintainer is slow to patch by applying a patch to your copy of the source and recompiling immediately. You enable supply chain verification more easily because compiling from source is easier to verify than pre-compiled binaries.

The tradeoff is build time. Compiling from source is slower than pulling pre-compiled images. But the security benefits might be worth the build time cost for critical applications.

This is one approach among several. Other organizations use different strategies (dependency pinning, internal binary mirrors, vendor SLAs), but the core principle is the same: take control of the components you depend on rather than blindly accepting pre-built artifacts.

Bridging the Gap: Defense in Depth

Code, build-time, and low-level security require integration across your entire pipeline. At the code level, apply SAST, SCA, and code review. At build time, implement reproducible builds, secrets scanning, and supply chain verification. At the registry level, use image scanning, signature verification, and policy enforcement. At runtime, enforce capability dropping, seccomp, and behavioral monitoring. Each level catches different vulnerabilities, and together they provide defense in depth.

Most organizations focus on registry level (image scanning) because it's visible and measurable. But attackers target code, build, and low-level security because those are less defended. True security requires addressing all four levels, with particular attention to Layers 1 and 2 that most organizations neglect.