Production-ready pipelines for Jenkins and Azure DevOps, both using the headless clnstrt-cli for secure builds, signing, and verification. These tools excel in enterprise environments with complex approval workflows and legacy system integration.
Part 1: Jenkins Pipeline
Prerequisites
Before building CleanStart images in Jenkins, your Jenkins instance must meet several requirements. Your Jenkins installation should be version 2.300 or higher with the Blue Ocean plugin installed for modern pipeline visualization. You need either the Docker plugin or Docker-in-Docker agent capability to build container images within Jenkins jobs. Both the Pipeline plugin and Credentials plugin are required for managing secure credentials and orchestrating multi-stage builds.
Your Jenkins instance must have several plugins installed to support the full CI/CD workflow. The pipeline-stage-view plugin provides visualization of pipeline stages, pipeline-model-definition enables declarative pipeline syntax, docker-plugin allows container interaction, kubernetes enables cloud-native agent provisioning, hashicorp-vault-plugin manages secrets, and github-branch-source integrates with GitHub.
Your Jenkins agent pool should include Linux agents with Docker installed so that container builds can execute. You may optionally configure Kubernetes agents for distributed builds, allowing Jenkins to spin up ephemeral agents in Kubernetes clusters for scalable parallel builds.
Jenkins Declarative Pipeline
Create Jenkinsfile in repository root:
pipeline { agent any options { buildDiscarder(logRotator(numToKeepStr: '30', daysToKeepStr: '90')) timeout(time: 1, unit: 'HOURS') timestamps() disableConcurrentBuilds() } environment { // Credentials from Jenkins Credentials Store GCP_PROJECT_ID = credentials('gcp-project-id') GCP_SERVICE_ACCOUNT = credentials('gcp-service-account') REGISTRY_URL = "us-docker.pkg.dev/${GCP_PROJECT_ID}" IMAGE_NAME = sh(script: "basename $GIT_URL .git", returnStdout: true).trim().toLowerCase() IMAGE_TAG = "${GIT_COMMIT.substring(0,7)}" FULL_IMAGE = "${REGISTRY_URL}/dev/${IMAGE_NAME}:${IMAGE_TAG}" // CleanStart CLI CLNSTRT_CLI = "${WORKSPACE}/clnstrt-cli" } stages { stage('Initialize') { steps { script { echo "===== Pipeline Initialization =====" echo "Repository: ${GIT_URL}" echo "Branch: ${GIT_BRANCH}" echo "Commit: ${GIT_COMMIT}" echo "Image: ${FULL_IMAGE}" // Download clnstrt-cli sh ''' curl -sSL https://releases.cleanstart.dev/clnstrt-cli-latest.tar.gz | tar xz chmod +x ./clnstrt-cli ./clnstrt-cli version ''' } } } stage('Analyze') { parallel { stage('SAST Scan') { steps { script { echo "Running SAST analysis..." sh ''' docker run --rm \ -v ${WORKSPACE}:/workspace \ aquasec/trivy:latest \ fs /workspace \ --format sarif \ --output /workspace/sast-report.sarif ''' } } } stage('Dependency Check') { steps { script { echo "Checking dependencies..." sh ''' docker run --rm \ -v ${WORKSPACE}:/workspace \ owasp/dependency-check:latest \ --scan /workspace \ --format JSON \ --project jenkins-build ''' } } } stage('Generate SBOM') { steps { script { echo "Generating SBOM..." sh ''' docker run --rm \ -v ${WORKSPACE}:/workspace \ ghcr.io/cyclonedx/cyclonedx-linux:latest \ -o json \ -output /workspace/sbom.json \ /workspace ''' // Convert to SPDX sh ''' cat > sbom.spdx <<EOF SPDXVersion: SPDX-3.0 DataLicense: CC0-1.0 SPDXID: SPDXRef-DOCUMENT DocumentName: ${IMAGE_NAME} Creator: Tool: Jenkins-Build Created: $(date -u +'%Y-%m-%dT%H:%M:%SZ') EOF ''' } } } } } stage('Build') { steps { script { echo "Building hermetic container image..." sh ''' docker build \ --build-arg BUILD_DATE="$(date -u +'%Y-%m-%dT%H:%M:%SZ')" \ --build-arg VCS_REF="${GIT_COMMIT}" \ --build-arg VERSION="${IMAGE_TAG}" \ -t ${FULL_IMAGE} \ -t ${REGISTRY_URL}/dev/${IMAGE_NAME}:latest \ . ''' } } } stage('Test') { steps { script { echo "Running container tests..." sh ''' docker run --rm \ -v ${WORKSPACE}/test-results:/results \ ${FULL_IMAGE} \ /test.sh || exit 1 echo "✅ Container tests passed" ''' } } post { always { junit testResults: 'test-results/*.xml', allowEmptyResults: true } } } stage('Generate Image SBOM') { steps { script { echo "Generating image SBOM..." sh ''' docker run --rm \ -v ${WORKSPACE}:/workspace \ ${FULL_IMAGE} \ clnstrt-cli generate-sbom \ --image ${FULL_IMAGE} \ --output /workspace/image-sbom.spdx ''' } } } stage('Verify Compliance') { steps { script { echo "Verifying security compliance..." sh ''' # FIPS algorithm check if grep -iE "md5|sha1|des" Dockerfile; then echo "ERROR: Non-FIPS algorithms detected" exit 1 fi # SBOM validation ${CLNSTRT_CLI} verify \ --sbom sbom.spdx \ --min-components 50 echo "✅ Compliance checks passed" ''' } } } stage('Authenticate GCP') { steps { script { echo "Authenticating with Google Cloud..." withCredentials([file(credentialsId: 'gcp-service-account-json', variable: 'GCP_KEY')]) { sh ''' gcloud auth activate-service-account --key-file=${GCP_KEY} gcloud config set project ${GCP_PROJECT_ID} gcloud auth configure-docker us-docker.pkg.dev ''' } } } } stage('Sign Image') { steps { script { echo "Signing image with clnstrt-cli..." withCredentials([string(credentialsId: 'cosign-kms-key', variable: 'COSIGN_KEY')]) { sh ''' ${CLNSTRT_CLI} sign \ --image ${FULL_IMAGE} \ --key "${COSIGN_KEY}" \ --attestations sbom.spdx,image-sbom.spdx ''' } } } } stage('Push to Registry') { when { branch 'main' } steps { script { echo "Pushing image to Artifact Registry..." sh ''' docker push ${FULL_IMAGE} docker push ${REGISTRY_URL}/dev/${IMAGE_NAME}:latest # Also push to prod if main branch if [[ "${GIT_BRANCH}" == "main" ]]; then PROD_IMAGE="${REGISTRY_URL}/prod/${IMAGE_NAME}:${IMAGE_TAG}" docker tag ${FULL_IMAGE} ${PROD_IMAGE} docker push ${PROD_IMAGE} echo "✅ Image promoted to production" fi ''' } } } stage('Verify Deployment') { when { branch 'main' } steps { script { echo "Verifying image in registry..." sh ''' # Verify image exists and is signed gcloud container images describe ${FULL_IMAGE} # List attestations gcloud container images list-tags \ ${REGISTRY_URL}/dev/${IMAGE_NAME} \ --limit=5 \ --format='table(tags, digest, timestamp)' ''' } } } } post { always { script { echo "===== Pipeline Cleanup =====" // Publish test results publishHTML([ reportDir: 'test-results', reportFiles: 'index.html', reportName: 'Test Results' ]) // Archive SBOM and attestations archiveArtifacts artifacts: '*.spdx,*.json', allowEmptyArchive: true // Clean workspace cleanWs deleteDirs: true, patterns: [[pattern: '.git/**', type: 'EXCLUDE']] } } failure { script { echo "Pipeline failed - sending notifications" // Send email, Slack, or Teams notification // emailext( // subject: "Build ${BUILD_NUMBER} failed", // body: "See ${BUILD_URL} for details", // to: "${CHANGE_AUTHOR_EMAIL}" // ) } } success { script { echo "Pipeline succeeded - image ready for deployment" // Post to Slack sh ''' curl -X POST -H 'Content-type: application/json' \ --data '{"text":"✅ Build succeeded: ${FULL_IMAGE}"}' \ ${SLACK_WEBHOOK_URL} ''' } } }}Jenkins Shared Library Setup
Create vars/secureDockerBuild.groovy for reusable pipeline stages:
def call(Map config) { pipeline { agent any stages { stage('Secure Build') { steps { script { // Download CLI sh ''' curl -sSL https://releases.cleanstart.dev/clnstrt-cli-latest.tar.gz | tar xz chmod +x ./clnstrt-cli ''' // Execute build with security checks sh ''' ./clnstrt-cli sign \ --image ${config.image} \ --key ${config.kmsKey} \ --verify-fips \ --require-sbom ''' } } } } }}Part 2: Azure DevOps Pipeline
Prerequisites
Azure DevOps Setup: Azure DevOps Services or Server, Docker task installed, Artifact Registry connection configured, and Service connection to Google Cloud.
Agent Configuration: Hosted Ubuntu agents or self-hosted agents and Docker support enabled.
Azure DevOps YAML Pipeline
Create azure-pipelines.yml:
trigger: branches: include: - main - develop paths: exclude: - docs/ - '*.md' pr: branches: include: - main - develop pool: vmImage: 'ubuntu-latest' variables: GCP_PROJECT_ID: 'your-project-id' REGISTRY_URL: 'us-docker.pkg.dev/$(GCP_PROJECT_ID)' IMAGE_NAME: 'myapp' IMAGE_TAG: '$(Build.SourceVersionShort)' FULL_IMAGE: '$(REGISTRY_URL)/dev/$(IMAGE_NAME):$(IMAGE_TAG)' stages: - stage: Analyze displayName: 'Security Analysis' jobs: - job: SAST displayName: 'SAST Scanning' steps: - task: UsePythonVersion@0 inputs: versionSpec: '3.11' displayName: 'Use Python' - script: | curl -sSL https://releases.cleanstart.dev/clnstrt-cli-latest.tar.gz | tar xz chmod +x ./clnstrt-cli ./clnstrt-cli version displayName: 'Install clnstrt-cli' - script: | docker run --rm \ -v $(Build.SourcesDirectory):/workspace \ aquasec/trivy:latest \ fs /workspace \ --format sarif \ --output $(Build.ArtifactStagingDirectory)/sast-report.sarif displayName: 'Trivy SAST Scan' - task: PublishSecurityAnalysisLogs@3 inputs: ArtifactType: 'CodeAnalysisLogs' ArtifactName: 'CodeAnalysisLogs' PatternToSearch: | **/sast-report.sarif - job: SBOM displayName: 'SBOM Generation' steps: - script: | docker run --rm \ -v $(Build.SourcesDirectory):/workspace \ ghcr.io/cyclonedx/cyclonedx-linux:latest \ -o json \ -output $(Build.ArtifactStagingDirectory)/sbom.json \ /workspace displayName: 'Generate CycloneDX SBOM' - script: | cat > $(Build.ArtifactStagingDirectory)/sbom.spdx <<EOF SPDXVersion: SPDX-3.0 DataLicense: CC0-1.0 SPDXID: SPDXRef-DOCUMENT DocumentName: $(IMAGE_NAME) Creator: Tool: Azure-Pipelines Created: $(date -u +'%Y-%m-%dT%H:%M:%SZ') EOF displayName: 'Generate SPDX SBOM' - task: PublishBuildArtifacts@1 inputs: PathtoPublish: '$(Build.ArtifactStagingDirectory)' ArtifactName: 'sbom-artifacts' - stage: Build displayName: 'Hermetic Build' dependsOn: Analyze jobs: - job: BuildContainer displayName: 'Build Container Image' steps: - task: Docker@2 displayName: 'Build Docker image' inputs: containerRegistry: 'gcr-connection' repository: '$(REGISTRY_URL)/dev/$(IMAGE_NAME)' command: 'build' Dockerfile: 'Dockerfile' tags: | $(IMAGE_TAG) latest arguments: | --build-arg BUILD_DATE=$(date -u +'%Y-%m-%dT%H:%M:%SZ') --build-arg VCS_REF=$(Build.SourceVersion) --build-arg VERSION=$(IMAGE_TAG) - script: | docker run --rm \ -v $(Build.ArtifactStagingDirectory):/results \ $(FULL_IMAGE) \ /test.sh displayName: 'Run Container Tests' - task: PublishTestResults@2 displayName: 'Publish Test Results' inputs: testResultsFormat: 'JUnit' testResultsFiles: '$(Build.ArtifactStagingDirectory)/test-results.xml' mergeTestResults: true - stage: Verify displayName: 'Security Verification' dependsOn: Build jobs: - job: ComplianceCheck displayName: 'Compliance Verification' steps: - script: | curl -sSL https://releases.cleanstart.dev/clnstrt-cli-latest.tar.gz | tar xz chmod +x ./clnstrt-cli displayName: 'Install clnstrt-cli' - script: | ./clnstrt-cli verify \ --image $(FULL_IMAGE) \ --require-fips \ --require-sbom \ --fail-on-high-severity displayName: 'Verify Security Compliance' - script: | if grep -iE "md5|sha1|des" Dockerfile; then echo "ERROR: Non-FIPS algorithms in Dockerfile" exit 1 fi echo "✅ FIPS compliance verified" displayName: 'FIPS Algorithm Check' - stage: SignAndPush displayName: 'Sign and Push to Registry' dependsOn: Verify condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main')) jobs: - job: SignAndPush displayName: 'Sign Image and Push' steps: - task: GoogleCloudSDK@0 inputs: ServiceAccountKeyFile: '$(GCP_SERVICE_ACCOUNT_KEY)' displayName: 'Authenticate Google Cloud' - script: | curl -sSL https://releases.cleanstart.dev/clnstrt-cli-latest.tar.gz | tar xz chmod +x ./clnstrt-cli displayName: 'Install clnstrt-cli' - script: | gcloud auth configure-docker us-docker.pkg.dev docker push $(FULL_IMAGE) docker push $(REGISTRY_URL)/dev/$(IMAGE_NAME):latest displayName: 'Push Image to Registry' - script: | ./clnstrt-cli sign \ --image $(FULL_IMAGE) \ --key kms://projects/$(GCP_PROJECT_ID)/locations/us-central1/keyRings/cosign-keys/cryptoKeys/azure-ci \ --attestations sbom.spdx displayName: 'Sign Image with Cosign' - script: | # Promote to production on main branch if [[ "$(Build.SourceBranch)" == "refs/heads/main" ]]; then PROD_IMAGE="$(REGISTRY_URL)/prod/$(IMAGE_NAME):$(IMAGE_TAG)" docker tag $(FULL_IMAGE) ${PROD_IMAGE} docker push ${PROD_IMAGE} echo "✅ Image promoted to production" fi displayName: 'Promote to Production' - stage: Notify displayName: 'Notifications' dependsOn: SignAndPush condition: always() jobs: - job: NotifyStatus displayName: 'Send Notifications' steps: - script: | if [[ "$(System.PipelineResult)" == "Succeeded" ]]; then MESSAGE="✅ Build succeeded: $(FULL_IMAGE)" else MESSAGE="❌ Build failed - see $(System.TeamFoundationCollectionUri)$(System.TeamProject)/_build/results?buildId=$(Build.BuildId)" fi curl -X POST \ -H 'Content-type: application/json' \ --data "{\"text\":\"$MESSAGE\"}" \ $(SLACK_WEBHOOK_URL) displayName: 'Send Slack Notification' continueOnError: trueDockerfile for Enterprise Builds
FROM cleanstart-gcc:14-builder AS builderLABEL stage=builderLABEL org.opencontainers.image.vendor="Enterprise" RUN apk add --no-cache \ git \ make \ autoconf \ automake \ libtool \ curl COPY . /srcWORKDIR /src # Enterprise build with security scanningRUN ./configure \ --prefix=/usr/local \ --with-security-checks \ && make -j$(nproc) \ && make security-verify \ && make test # Generate SBOMRUN clnstrt-cli generate-sbom \ --output /build-artifacts/sbom.spdx FROM cleanstart-gcc:14-prod # Non-root user (Enterprise standard: UID 1000)RUN groupadd -r appuser -g 1000 && \ useradd -r -g appuser -u 1000 appuser # Labels for complianceARG BUILD_DATEARG VCS_REFARG VERSION LABEL \ org.opencontainers.image.created="$BUILD_DATE" \ org.opencontainers.image.revision="$VCS_REF" \ org.opencontainers.image.version="$VERSION" \ org.opencontainers.image.vendor="Enterprise" \ com.example.compliance="scanned" \ com.example.security="fips-verified" COPY --from=builder /usr/local /usr/localCOPY --from=builder /build-artifacts /attestations USER appuser HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \ CMD /usr/local/bin/healthcheck || exit 1 ENTRYPOINT ["/usr/local/bin/myapp"]Environment Variables Setup
Jenkins Credentials Mapping
// Jenkins Credentials ManagerwithCredentials([ string(credentialsId: 'gcp-project-id', variable: 'GCP_PROJECT_ID'), file(credentialsId: 'gcp-service-account-json', variable: 'GCP_KEY'), string(credentialsId: 'cosign-kms-key', variable: 'COSIGN_KEY'), string(credentialsId: 'slack-webhook', variable: 'SLACK_WEBHOOK_URL')]) { sh 'echo credentials loaded'}Azure DevOps Service Connections
- Go to Project Settings → Service connections → New service connection
- Select Google Cloud Platform
- Enter service account credentials
- Name it
gcr-connection
Troubleshooting
Jenkins: Credentials Not Found
// List available credentialssh ''' curl -s http://localhost:8080/credentials/store/system/domain/_/api/json | jq '.credentials[] | .id''''Azure DevOps: GCP Authentication Fails
- task: GoogleCloudSDK@0 inputs: ServiceAccountKeyFile: '$(GCP_SERVICE_ACCOUNT_KEY)' env: GOOGLE_APPLICATION_CREDENTIALS: '$(GCP_SERVICE_ACCOUNT_KEY)'clnstrt-cli Not Found
Ensure download completes:
curl -sSL https://releases.cleanstart.dev/clnstrt-cli-latest.tar.gz | tar xzchmod +x ./clnstrt-cli./clnstrt-cli --help