What this article covers

• Architectural mechanics of Rekor, Trillian, and the Merkle Tree structure.
• Installation and configuration of rekor-cli and cosign.
• Publishing procedures using keyless (OIDC) and long-lived key workflows.
• Ephemeral Certificate Issuance: Technical breakdown of the x.509 certificate lifecycle, subject alternative name (SAN) construction, and validity periods.
• Signing Execution: Step-by-step commands for executing cosign sign with ephemeral keys and attaching signatures to OCI registries.
• Verification and Enforcement: Constructing precise cosign verify commands using identity flags and implementing Kubernetes admission controllers.
• Forensics and Auditing: Procedures for querying the Rekor transparency log to audit signing events and detect identity spoofing.

Step 1: Establish the Trusted Build Environment

The foundation of a secure supply chain lies not merely in the act of signing but in the integrity of the environment where the signing occurs and the quality of the artifact being signed. Keyless signing shifts the root of trust from a protected file (a private key) to a protected process (the CI/CD pipeline). Therefore, the initial configuration of the build environment is critical.

1.1 Analysis of the Keyless Architecture

Traditional signing relies on asymmetric cryptography where a private key is generated, stored, and protected by the signer. This model fails at scale in containerized environments due to the "secret management problem." If a key is stored in a CI secret manager, it is vulnerable to exfiltration. If it is lost, the artifact cannot be verified. If it is compromised, revocation is complex and rarely enforced effectively. Sigstore solves this through ephemeral signing.

• Identity Provider (OIDC): The CI system (e.g., GitHub Actions) proves its identity via an OpenID Connect token.
• Fulcio (CA): Sigstore’s Certificate Authority verifies the OIDC token and issues a shortlived x.509 certificate (valid for 10 minutes) bound to the public key generated by the client. The "Subject" of the certificate is the identity from the OIDC token (e.g., the repository URL).
• Rekor (Transparency Log): The signature and certificate are logged in an immutable ledger. Verification proves that the certificate was valid at the time of signing, eliminating the need for checking revocation lists for expired certificates.

1.2 CleanStart Base Image Selection

Signing provides provenance (who built it) and integrity (it hasn't changed), but it does not guarantee quality (it is safe). A signed image containing critical vulnerabilities is merely a verified threat. The CleanStart methodology dictates "upstream thinking," where security is embedded in the foundation.1 Before implementing signing, ensure the Dockerfile references a CleanStart hardened base. These images are stripped of unnecessary components, reducing the attack surface by up to 80% compared to standard Docker Hub images.2

Correct Dockerfile Structure:

The use of the SHA256 digest (@sha256:...) is mandatory for high-assurance signing. Tags are mutable pointers; signing a tag introduces a race condition (Time-of-Check to Time-ofUse) where the tag could be moved to a malicious image between the time of build and the time of signing.3 CleanStart images are essentially "distroless" or "minimal," lacking shells and package managers in production, which aligns with the philosophy of minimal trusted computing bases.4

1.3 Tooling Installation (Cosign)

The cosign CLI is the primary interface for Sigstore. It manages the OIDC handshake, keygeneration, and registry interaction.Installation for Linux (AMD64):

Verification:Verify the installation and ensure version consistency (v2.0+ is required for the flags detailedin this guide).

Note: Cosign v2 introduced breaking changes to the command-line flags, specifically regarding how identity verification is handled (--certificate-identity vs the older CS_ environment variables). This guide utilizes the v2.0+ syntax.5

Step 2: Configure OIDC Identity Federation

The critical mechanism in keyless signing is the exchange of a workload identity token for a signing certificate. This requires configuring the Continuous Integration (CI) provider to mint OIDC tokens with the specific audience (aud) required by Sigstore.

2.1 GitHub Actions OIDC Configuration

GitHub Actions functions as an OIDC provider. When a workflow runs, GitHub can issue a JSON Web Token (JWT) that attests to the workflow's identity. However, for security reasons, this capability is disabled by default and must be explicitly granted via permission scopes.The id-token: write Permission:To enable the OIDC projection, the workflow YAML must include the permissions block. This allows the GitHub runner to interact with the GitHub OIDC provider (token.actions.githubusercontent.com).Configuration (.github/workflows/release.yaml):

Mechanism of Action:

When id-token: write is set, the GitHub Actions runner injects two environment variables:

• ACTIONS_ID_TOKEN_REQUEST_URL: The endpoint to request the token.
• ACTIONS_ID_TOKEN_REQUEST_TOKEN: The bearer token used to authenticate the request.6

Cosign detects these variables automatically. When cosign sign is invoked, it calls the REQUEST_URL specifying the audience as sigstore. GitHub responds with a JWT signed by its private key.

Token Payload Structure (GitHub):

The JWT issued by GitHub contains specific claims that Fulcio will use to construct the certificate subject:

• iss (Issuer): https://token.actions.githubusercontent.com
• aud (Audience): sigstore
• job_workflow_ref: octo-org/octo-repo/.github/workflows/oidc.yml@refs/heads/main
• sha: The commit SHA of the build.
• run_id: The unique ID of the workflow run.

Fulcio uses the job_workflow_ref to create the Subject Alternative Name (SAN) in the certificate, which allows for precise verification later.8

2.2 GitLab CI/CD OIDC Configuration

GitLab CI/CD utilizes a specific keyword, id_tokens, to generate OIDC tokens. Unlike GitHub, which makes the token available via an internal endpoint, GitLab injects the token directly as an environment variable. Configuration (.gitlab-ci.yml):

Mechanism of Action: The id_tokens block instructs GitLab to generate a JWT with the audience sigstore and bind it to the environment variable SIGSTORE_ID_TOKEN.10 Cosign looks for this specific variable name by default. If you name it differently, you must pass it to Cosign manually, but standardizing on SIGSTORE_ID_TOKEN reduces configuration drift.

Token Payload Structure (GitLab):

• iss: https://gitlab.com (or self-hosted URL).
• sub: project_path:my-group/my-project:ref_type:branch:ref:main
• aud: sigstore
• user_email: The email of the user triggering the pipeline (if applicable).

Fulcio maps these claims to the certificate extensions, allowing downstream users to verify that the image was built by a specific project and branch on GitLab.10

Step 3: Execute Keyless Signing

Once the environment is configured and the OIDC token is accessible, the signing process can be executed. This step involves generating the image, pushing it to the registry, and then attaching the cryptographic signature.

3.1 Build and Push the Artifact

The artifact must exist in the registry before it can be signed because Cosign attaches the signature as a related artifact (stored via the OCI Reference specification).GitHub Actions Example:

The Importance of Digest Capture: The docker/build-push-action outputs the specific digest of the image built. This digest (steps.build-push.outputs.digest) is the immutable identifier. Signing the tag is considered a security anti-pattern because the tag can be moved to a different digest after verification but before deployment.3

3.2 The cosign sign Command

The signing command interacts with the Fulcio CA and Rekor transparency log.Command Syntax:

Detailed GitHub Actions Implementation:

Deconstruction of the Process:

• Key Generation: Cosign generates an ephemeral ECDSA P-256 key pair locally in the runner's memory. This key pair is never stored on disk.
• Identity Proof: Cosign retrieves the OIDC token from the environment (ACTIONS_ID_TOKEN_REQUEST_TOKEN or SIGSTORE_ID_TOKEN).
• Certification: Cosign sends the public key component and the OIDC token to Fulcio.
• Validation: Fulcio validates the OIDC token against the issuer's public keys (OpenID Discovery). If valid, it extracts the identity (subject) and issues a code-signing certificate for the provided public key.
• Signature: Cosign uses the private key (in memory) to sign the image digest.
• Transparency: The certificate and the signature are uploaded to the Rekor transparency log. Rekor returns a Signed Certificate Timestamp (SCT).
• Artifact Upload: Cosign packages the signature, the certificate, and the SCT into a JSON blob and uploads it to the registry as a new tag (e.g., sha256-<digest>.sig).
• Key Destruction: The private key is discarded.

The Role of --yes: In non-interactive environments (CI/CD), the --yes flag is mandatory. Without it, Cosign may attempt to prompt for confirmation to upload to the transparency log, causing the pipeline to hang.11

3.3 Annotations for Traceability

The -a key=value flags in the command above add annotations to the signature payload.12 These are not part of the OIDC identity but are signed metadata. By including the git SHA (ref), workflow name, and repository, you create a semantic link between the binary artifact and the source code. While the OIDC identity proves the authorization, the annotations provide context for debugging and auditing.

Step 4: Verification and Policy Enforcement

Signing is a means to an end; the goal is verification. Without a verification step that enforces the signature's validity and identity, the signing process provides no security benefit. Verification should occur at the point of consumption, typically a Kubernetes cluster or a downstream deployment pipeline.aSigning is a means to an end; the goal is verification. Without a verification step that enforces the signature's validity and identity, the signing process provides no security benefit. Verification should occur at the point of consumption, typically a Kubernetes cluster or a downstream deployment pipeline.

4.1 The Verification Logic

Keyless verification is more complex than key-based verification because there is no static public key to distribute. Instead, the verifier must check three things:

• Signature Validity: Does the signature match the artifact?
• Certificate Validity: Does the certificate chain up to the Fulcio Root CA?
• Identity Policy: Does the Subject in the certificate match the Expected Identity?

This third step is crucial. If you only verify the signature, you are confirming that someone with a valid OIDC token signed it. That "someone" could be a developer in a personal fork of the repository. You must verify that the specific trusted workflow signed it.

4.2 Constructing the Verification Command

The cosign verify command requires specific flags to define the trust policy.Verification Command Template:

Example for GitHub Actions:

Identity String Precision:

The certificate-identity must match the SAN in the certificate exactly. For GitHub, this includes the repository path, the workflow filename, and the git reference (branch or tag).

• Correct: .../workflows/build.yml@refs/heads/main
• Incorrect: .../workflows/build.yml (Missing ref)
• Incorrect: .../workflows/build.yml@main (Missing refs/heads/)

Regex Matching:For flexibility (e.g., allowing signing from any tag starting with v), use the regex flags:

Warning: Regex usage requires careful construction to avoid "wildcard" vulnerabilities where an attacker could bypass the check using a similarly named branch.13

4.3 Policy Enforcement in Kubernetes

To enforce CleanStart standards, verification should be automated via an Admission Controller. Kyverno is a policy engine designed for Kubernetes that integrates natively with Cosign.

Kyverno Policy for CleanStart Images:

This policy ensures that any Pod attempting to run an image from the cleanstart-corp repository must be signed by the specific GitHub Action workflow.

This policy acts as a firewall. If a developer attempts to deploy an image that was built locally (and thus unsigned) or built on a feature branch (if the policy enforces main), the cluster will reject the deployment request.

Step 5: Advanced Considerations and CleanStart Integration

5.1 CleanStart "Upstream" Security

CleanStart differentiates itself by providing base images that are "secure by design." When implementing signing, it is critical to understand that you are signing the delta—the application layer you added on top of the CleanStart base. However, the security of the final artifact depends on the integrity of the base.

• Base Image Verification: In high-security environments, your build pipeline should verify the signature of the CleanStart base image before building the application layer.
• Verification Command:

By verifying the base image, you establish a "chain of trust" extending from the OS vendor (CleanStart) to your application logic.

5.2 Offline and Air-Gapped Verification

CleanStart environments often operate in air-gapped networks where contacting the public Rekor log (rekor.sigstore.dev) is impossible. Sigstore supports "offline" verification using the Signed Certificate Timestamp (SCT) and the signature bundle.Procedure:

• Bundle Download: In a connected environment, use cosign download signature to retrieve the signature bundle.
• Transport: Move the image and the bundle to the air-gapped network.
• Verification: Use the --offline flag.

In offline mode, Cosign verifies the cryptographic signature and the validity of the certificate chain against the embedded Fulcio root, but it cannot query Rekor to see if the certificate was transparently logged. To mitigate this, Cosign relies on the SCT embedded in the certificate, which acts as a "promise" of logging.15

Identity String Precision:

The public nature of Rekor allows for powerful auditing capabilities. You can query the log to see all artifacts signed by your identity. Audit Command:

If you detect a signing event that does not correlate with your known CI/CD build timestamps, it indicates a potential compromise of your OIDC provider or CI system.

Notes / Tips

• Registry Support: Not all container registries support the OCI artifact specification required to store signatures alongside images. GitHub (GHCR), AWS ECR, and Docker Hub support it. If using Artifactory or Nexus, ensure you are on a recent version.
• Tag Mutability: A common error is signing a tag (e.g., v1.0) and then overwriting that tag with a new build. The signature for the previous v1.0 will become "detached" or invalid for the new content. Always sign and verify digests.
• Private Rekor: Organizations with strict privacy requirements regarding their build metadata (e.g., internal repo names appearing in public logs) may deploy a private instance of Rekor and Fulcio. However, this increases operational complexity significantly.16
• Debugging OIDC: If you receive error: no identity token found, verify that the id-token: write permission is correctly indented in your YAML and applied to the specific job running the signing command.

Requirements

To implement the solution described in this article, the following prerequisites must be met:

● Cosign CLI: Version 2.2.0 or later installed on build agents.

● CI/CD Platform: GitHub Actions, GitLab CI, or Google Cloud Build with OIDC enabled.

● Container Registry: OCI v1.0 compliant registry with write access.

● Network Access: Outbound HTTPS (443) access to:

○ fulcio.sigstore.dev

○ rekor.sigstore.dev○ oauth2.sigstore.dev

○ The OIDC provider URL (e.g., token.actions.githubusercontent.com).

Related articles

• Configuring Kyverno for CleanStart Image Enforcement
• Deep Dive: The Security Architecture of CleanStart Base Images
• Troubleshooting OIDC Token Claims in GitHub Actions
• Migrating from GPG Keys to Keyless Signing

Best Real-World Example: Financial Services Implementation

The Scenario:

"FinTechGlobal," a provider of payment gateway infrastructure, faced a critical security challenge. Their compliance requirements (PCI-DSS and SOC 2) mandated strict provenance for all container workloads. They previously used long-lived GPG keys stored in Jenkins secrets. However, a near-miss incident where a key was inadvertently logged in a build artifact forced a re-evaluation of their strategy.

The Implementation:

FinTechGlobal adopted CleanStart base images to eliminate OS-level CVE remediation work and implemented Sigstore keyless signing to secure the build pipeline.The Workflow:

• Base Layer Security: The Dockerfile pins cleanstart/java:17-hardened@sha256:.... During the build, a script verifies the CleanStart signature on this base image using the CleanStart public key. This ensures the foundation is trustworthy.
• Automated Building: Developers commit code to the release branch. GitHub Actions triggers the build.
• Keyless Signing: The GitHub Action authenticates via OIDC. Fulcio issues a certificate bound to https://github.com/FinTechGlobal/payment- gateway/.github/workflows/prod.yml@refs/heads/release. The image is signed and logged to Rekor.
• Annotated Provenance: The signature is annotated with the specific Jira ticket ID linked to the release, bridging the gap between code and project management.
• Admission Control: Their Kubernetes clusters run Gatekeeper. A constraint template is configured to reject any image in the payments namespace that does not possess a valid signature from the specific prod.yml workflow identity.

The Outcome:

• Key Elimination: The DevSecOps team retired the management of 50+ static signing keys.
• Audit Velocity: During a SOC 2 audit, the team demonstrated chain-of-custody by querying the Rekor log, instantly proving that every running container was built by the authorized pipeline.
• Incident Response: When a vulnerability was disclosed in a Java library, they used the SBOMs (stored alongside the signatures) to identify affected services in minutes rather than days.

Table 1: Comparison of Legacy vs. CleanStart/Sigstore Approach

Feature

Legacy GPG Signing

Trust Root

Static Private Key file

Key Management

Revocation

Verification

Base Image

Often unverified/bloated

Verified, Zero-CVE CleanStart Base

Failure Mode

Deep Search Analysis: Technical Nuances of OIDC Tokens

A deeper analysis of the research data reveals critical nuances in how different providers handle OIDC tokens, which directly impacts the implementation of certificate-identity policies. GitHub Actions vs. GitLab CI Claims: Research indicates a divergence in token structure. GitHub Actions uses a custom claim job_workflow_ref 8 which provides a robust, immutable reference to the specific workflow file and git reference. This allows for extremely granular policies (e.g., "only the security-scan.yml workflow can sign"). In contrast, GitLab CI relies heavily on the sub (subject) claim which concatenates project path and branch: project_path:my-group/my-project:ref_type:branch:ref:main.10 While effective, it lacks the distinct separation of "workflow identity" vs "branch identity" found in GitHub. When writing verification policies for GitLab, engineers must be precise with regex to ensure they are not inadvertently trusting any pipeline running on the main branch, but rather the specific protected pipeline intended for releases. The "Audience" Requirement: A recurring theme in troubleshooting logs 18 is the failure of the OIDC handshake due to incorrect aud (audience) claims. Fulcio strictly requires the audience to be sigstore. While GitHub allows configuring this via the audiences configuration in the workflow, some custom OIDC implementations (like those used in enterprise onpremise solutions) may default to the URL of the service itself. Explicitly overriding the audience to sigstore in the token generation step is a non-negotiable requirement for Fulcio integration. Implications for CleanStart Users: For users of CleanStart, this identity-based approach aligns perfectly with the "Secure by Design" philosophy. Just as CleanStart removes the burden of patching OS vulnerabilities, Sigstore removes the burden of protecting signing keys. The synergy creates a supply chain where the only variable left to manage is the policy itself, rather than the cryptographic plumbing.