DevSecOps Pipeline Security: Integrating Security into CI/CD
DevSecOps represents a fundamental shift in how organizations approach security - moving from a final gate before deployment to an integrated practice throughout the software development lifecycle. The CI/CD pipeline is where this integration takes concrete form.
This guide provides practical approaches to building secure pipelines.
The Secure Pipeline Architecture
┌─────────────────────────────────────────────────────────────────────────────┐
│ Secure CI/CD Pipeline │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Code │ → │ Build │ → │ Test │ → │ Deploy │ → │ Monitor │ │
│ │ Commit │ │ │ │ │ │ │ │ │ │
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ │
│ │ │ │ │ │ │
│ ▼ ▼ ▼ ▼ ▼ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Pre- │ │ SAST │ │ DAST │ │ Config │ │ Runtime │ │
│ │ commit │ │ SCA │ │ Security │ │ Audit │ │ Security │ │
│ │ Hooks │ │ Secrets │ │ Tests │ │ CSPM │ │ RASP │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Stage 1: Pre-Commit Security
Git Hooks Configuration
# .pre-commit-config.yaml
repos:
# Secrets detection
- repo: https://github.com/Yelp/detect-secrets
rev: v1.4.0
hooks:
- id: detect-secrets
args: ['--baseline', '.secrets.baseline']
# Credential scanning
- repo: https://github.com/trufflesecurity/trufflehog
rev: v3.63.0
hooks:
- id: trufflehog
args: ['--only-verified']
# Security linting
- repo: https://github.com/PyCQA/bandit
rev: 1.7.5
hooks:
- id: bandit
args: ['-r', 'src/', '-ll']
# Dockerfile linting
- repo: https://github.com/hadolint/hadolint
rev: v2.12.0
hooks:
- id: hadolint-docker
args: ['--ignore', 'DL3008']
# Terraform security
- repo: https://github.com/antonbabenko/pre-commit-terraform
rev: v1.83.5
hooks:
- id: terraform_tfsec
- id: terraform_checkov
# YAML linting (for K8s manifests)
- repo: https://github.com/adrienverge/yamllint
rev: v1.32.0
hooks:
- id: yamllint
args: ['-c', '.yamllint.yml']Custom Security Hooks
#!/bin/bash
# .git/hooks/pre-commit
echo "Running security checks..."
# Check for hardcoded secrets patterns
SECRET_PATTERNS=(
'password\s*=\s*["\047][^"\047]+'
'api[_-]?key\s*=\s*["\047][^"\047]+'
'secret\s*=\s*["\047][^"\047]+'
'AWS[A-Z0-9]{16,}'
'AKIA[A-Z0-9]{16}'
)
for pattern in "${SECRET_PATTERNS[@]}"; do
if git diff --cached --name-only | xargs grep -lE "$pattern" 2>/dev/null; then
echo "ERROR: Potential secret detected matching pattern: $pattern"
echo "Please remove secrets before committing"
exit 1
fi
done
# Check for sensitive file types
SENSITIVE_EXTENSIONS=(.pem .key .p12 .pfx .env .env.local)
for ext in "${SENSITIVE_EXTENSIONS[@]}"; do
if git diff --cached --name-only | grep -E "\\${ext}$"; then
echo "ERROR: Sensitive file type detected: $ext"
echo "Consider using secrets management instead"
exit 1
fi
done
echo "Pre-commit security checks passed"Stage 2: Build Security
GitHub Actions Secure Pipeline
# .github/workflows/secure-pipeline.yml
name: Secure CI/CD Pipeline
on:
push:
branches: [main, develop]
pull_request:
branches: [main]
permissions:
contents: read
security-events: write
actions: read
jobs:
security-scan:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0
# SAST - Static Application Security Testing
- name: Run SAST (Semgrep)
uses: returntocorp/semgrep-action@v1
with:
config: >-
p/security-audit
p/secrets
p/owasp-top-ten
generateSarif: true
- name: Upload SAST results
uses: github/codeql-action/upload-sarif@v2
with:
sarif_file: semgrep.sarif
# SCA - Software Composition Analysis
- name: Run SCA (Trivy)
uses: aquasecurity/trivy-action@master
with:
scan-type: 'fs'
scan-ref: '.'
format: 'sarif'
output: 'trivy-sca.sarif'
severity: 'CRITICAL,HIGH'
- name: Upload SCA results
uses: github/codeql-action/upload-sarif@v2
with:
sarif_file: trivy-sca.sarif
# Secret scanning
- name: Scan for secrets
uses: trufflesecurity/trufflehog@main
with:
path: ./
base: ${{ github.event.repository.default_branch }}
head: HEAD
# License compliance
- name: Check licenses
run: |
npm install -g license-checker
license-checker --failOn 'GPL;AGPL' --summary
build-and-scan:
needs: security-scan
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build container image
uses: docker/build-push-action@v5
with:
context: .
push: false
load: true
tags: app:${{ github.sha }}
cache-from: type=gha
cache-to: type=gha,mode=max
# Container image scanning
- name: Scan container image
uses: aquasecurity/trivy-action@master
with:
image-ref: 'app:${{ github.sha }}'
format: 'sarif'
output: 'trivy-container.sarif'
severity: 'CRITICAL,HIGH'
exit-code: '1'
- name: Upload container scan results
uses: github/codeql-action/upload-sarif@v2
if: always()
with:
sarif_file: trivy-container.sarif
# SBOM generation
- name: Generate SBOM
uses: anchore/sbom-action@v0
with:
image: app:${{ github.sha }}
format: spdx-json
output-file: sbom.spdx.json
- name: Upload SBOM
uses: actions/upload-artifact@v3
with:
name: sbom
path: sbom.spdx.json
security-tests:
needs: build-and-scan
runs-on: ubuntu-latest
services:
app:
image: app:${{ github.sha }}
ports:
- 8080:8080
steps:
# DAST - Dynamic Application Security Testing
- name: Run DAST (OWASP ZAP)
uses: zaproxy/action-baseline@v0.9.0
with:
target: 'http://localhost:8080'
rules_file_name: '.zap/rules.tsv'
cmd_options: '-a'
- name: Upload DAST results
uses: actions/upload-artifact@v3
with:
name: zap-report
path: report_html.html
# API Security Testing
- name: Run API security tests
run: |
npm install -g @stoplight/spectral-cli
spectral lint openapi.yaml --ruleset .spectral.yml
deploy-staging:
needs: security-tests
runs-on: ubuntu-latest
environment: staging
steps:
- name: Deploy to staging
run: |
# Deployment commands here
echo "Deploying to staging..."
# Infrastructure security scan
- name: Scan infrastructure config
uses: bridgecrewio/checkov-action@v12
with:
directory: ./terraform
framework: terraform
output_format: sarif
output_file_path: checkov.sarif
- name: Upload infrastructure scan
uses: github/codeql-action/upload-sarif@v2
with:
sarif_file: checkov.sarifGitLab CI Secure Pipeline
# .gitlab-ci.yml
stages:
- security-scan
- build
- test
- deploy
variables:
DOCKER_DRIVER: overlay2
SECURE_LOG_LEVEL: info
# Security scanning templates
include:
- template: Security/SAST.gitlab-ci.yml
- template: Security/Secret-Detection.gitlab-ci.yml
- template: Security/Dependency-Scanning.gitlab-ci.yml
- template: Security/Container-Scanning.gitlab-ci.yml
- template: Security/License-Scanning.gitlab-ci.yml
# Override SAST job
sast:
stage: security-scan
variables:
SAST_EXCLUDED_PATHS: "spec, test, tests, tmp, vendor"
SEARCH_MAX_DEPTH: 10
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
# Custom security job
security-audit:
stage: security-scan
image: node:18-alpine
script:
- npm audit --audit-level=high
- npm run security:check
allow_failure: false
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
# Container scanning with Trivy
container-security:
stage: security-scan
image:
name: aquasec/trivy:latest
entrypoint: [""]
script:
- trivy image --exit-code 1 --severity HIGH,CRITICAL $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
# Infrastructure scanning
infra-security:
stage: security-scan
image:
name: bridgecrew/checkov:latest
entrypoint: [""]
script:
- checkov -d ./terraform --framework terraform --output cli --output junitxml --output-file-path ./checkov-results
artifacts:
reports:
junit: checkov-results/results_junitxml.xml
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
changes:
- "terraform/**/*"
# Build with security context
build:
stage: build
image: docker:24
services:
- docker:24-dind
script:
- docker build
--build-arg BUILD_DATE=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
--build-arg VCS_REF=$CI_COMMIT_SHA
--label "org.opencontainers.image.created=$(date -u +'%Y-%m-%dT%H:%M:%SZ')"
--label "org.opencontainers.image.revision=$CI_COMMIT_SHA"
-t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
# DAST testing
dast:
stage: test
image: owasp/zap2docker-stable
script:
- zap-baseline.py -t $STAGING_URL -r zap-report.html
artifacts:
paths:
- zap-report.html
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCHStage 3: Security Testing Integration
SAST Configuration (Semgrep)
# .semgrep.yml
rules:
- id: hardcoded-secret
patterns:
- pattern-either:
- pattern: $X = "..."
- pattern: $X = '...'
pattern-regex: (password|secret|api_key|token)\s*=\s*['"][^'"]+['"]
message: Hardcoded secret detected
languages: [python, javascript, typescript, java]
severity: ERROR
- id: sql-injection
patterns:
- pattern: |
$QUERY = f"SELECT ... {$INPUT} ..."
...
cursor.execute($QUERY)
message: Potential SQL injection vulnerability
languages: [python]
severity: ERROR
- id: unsafe-deserialization
pattern: pickle.loads($X)
message: Unsafe deserialization with pickle
languages: [python]
severity: ERROR
- id: command-injection
patterns:
- pattern: subprocess.call($CMD, shell=True)
- pattern: os.system($CMD)
message: Potential command injection
languages: [python]
severity: ERROR
# Custom rule for AI/LLM security
- id: prompt-injection-risk
patterns:
- pattern: |
prompt = f"... {$USER_INPUT} ..."
- pattern: |
prompt = "..." + $USER_INPUT + "..."
message: User input directly in prompt - potential prompt injection risk
languages: [python]
severity: WARNINGSCA Configuration (Dependabot + Renovate)
# .github/dependabot.yml
version: 2
updates:
- package-ecosystem: "npm"
directory: "/"
schedule:
interval: "daily"
open-pull-requests-limit: 10
labels:
- "dependencies"
- "security"
groups:
security-patches:
applies-to: security-updates
patterns:
- "*"
- package-ecosystem: "pip"
directory: "/"
schedule:
interval: "daily"
groups:
security-patches:
applies-to: security-updates
- package-ecosystem: "docker"
directory: "/"
schedule:
interval: "weekly"
- package-ecosystem: "terraform"
directory: "/terraform"
schedule:
interval: "weekly"// renovate.json
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": [
"config:base",
":semanticCommits",
"security:openssf-scorecard"
],
"vulnerabilityAlerts": {
"enabled": true,
"labels": ["security"]
},
"packageRules": [
{
"matchUpdateTypes": ["patch", "minor"],
"matchCurrentVersion": "!/^0/",
"automerge": true,
"automergeType": "pr",
"automergeStrategy": "squash"
},
{
"matchDepTypes": ["devDependencies"],
"automerge": true
},
{
"matchPackagePatterns": ["*"],
"matchUpdateTypes": ["major"],
"labels": ["major-update"]
}
],
"osvVulnerabilityAlerts": true
}Stage 4: Container Security
Secure Dockerfile
# Dockerfile with security best practices
# Use specific version, not latest
FROM node:20.10.0-alpine3.18 AS builder
# Create non-root user
RUN addgroup -g 1001 -S appgroup && \
adduser -u 1001 -S appuser -G appgroup
WORKDIR /app
# Copy only package files first for better caching
COPY package*.json ./
# Install dependencies with security audit
RUN npm ci --only=production && \
npm audit --audit-level=high
# Copy application code
COPY --chown=appuser:appgroup . .
# Build application
RUN npm run build
# Production stage
FROM node:20.10.0-alpine3.18 AS production
# Security updates
RUN apk update && \
apk upgrade --no-cache && \
apk add --no-cache dumb-init
# Create non-root user
RUN addgroup -g 1001 -S appgroup && \
adduser -u 1001 -S appuser -G appgroup
WORKDIR /app
# Copy built application
COPY --from=builder --chown=appuser:appgroup /app/dist ./dist
COPY --from=builder --chown=appuser:appgroup /app/node_modules ./node_modules
COPY --from=builder --chown=appuser:appgroup /app/package.json ./
# Remove unnecessary files
RUN rm -rf /var/cache/apk/* /tmp/* /root/.npm
# Security configurations
ENV NODE_ENV=production
ENV NPM_CONFIG_LOGLEVEL=warn
# Run as non-root user
USER appuser
# Health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
CMD node healthcheck.js || exit 1
# Use dumb-init to handle signals properly
ENTRYPOINT ["dumb-init", "--"]
CMD ["node", "dist/server.js"]
# Labels
LABEL org.opencontainers.image.source="https://github.com/org/repo"
LABEL org.opencontainers.image.description="Application description"
LABEL org.opencontainers.image.licenses="MIT"Kubernetes Security Policies
# k8s/security-policies.yaml
# Pod Security Policy (for older clusters)
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: restricted
spec:
privileged: false
allowPrivilegeEscalation: false
requiredDropCapabilities:
- ALL
volumes:
- 'configMap'
- 'emptyDir'
- 'projected'
- 'secret'
- 'downwardAPI'
- 'persistentVolumeClaim'
hostNetwork: false
hostIPC: false
hostPID: false
runAsUser:
rule: 'MustRunAsNonRoot'
seLinux:
rule: 'RunAsAny'
fsGroup:
rule: 'RunAsAny'
readOnlyRootFilesystem: true
---
# Network Policy
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: app-network-policy
namespace: production
spec:
podSelector:
matchLabels:
app: myapp
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app: api-gateway
ports:
- protocol: TCP
port: 8080
egress:
- to:
- podSelector:
matchLabels:
app: database
ports:
- protocol: TCP
port: 5432
- to:
- namespaceSelector: {}
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
---
# Resource Quotas
apiVersion: v1
kind: ResourceQuota
metadata:
name: security-quota
namespace: production
spec:
hard:
pods: "20"
requests.cpu: "10"
requests.memory: 20Gi
limits.cpu: "20"
limits.memory: 40GiStage 5: Compliance as Code
Policy as Code (OPA/Rego)
# policies/kubernetes.rego
package kubernetes.admission
# Deny containers running as root
deny[msg] {
input.request.kind.kind == "Pod"
container := input.request.object.spec.containers[_]
not container.securityContext.runAsNonRoot
msg := sprintf("Container %v must not run as root", [container.name])
}
# Require resource limits
deny[msg] {
input.request.kind.kind == "Pod"
container := input.request.object.spec.containers[_]
not container.resources.limits
msg := sprintf("Container %v must have resource limits", [container.name])
}
# Deny privileged containers
deny[msg] {
input.request.kind.kind == "Pod"
container := input.request.object.spec.containers[_]
container.securityContext.privileged
msg := sprintf("Container %v must not be privileged", [container.name])
}
# Require approved registries
deny[msg] {
input.request.kind.kind == "Pod"
container := input.request.object.spec.containers[_]
not startswith(container.image, "gcr.io/my-project/")
not startswith(container.image, "docker.io/library/")
msg := sprintf("Container %v uses unapproved registry", [container.name])
}
# Require security scanning label
deny[msg] {
input.request.kind.kind == "Deployment"
not input.request.object.metadata.labels["security-scanned"]
msg := "Deployment must have security-scanned label"
}Monitoring and Alerting
# prometheus-alerts.yml
groups:
- name: security-alerts
rules:
- alert: SecurityScanFailed
expr: security_scan_status{result="failed"} > 0
for: 5m
labels:
severity: critical
annotations:
summary: Security scan failed
description: "Security scan failed for {{ $labels.project }}"
- alert: HighVulnerabilityDetected
expr: security_vulnerabilities{severity="critical"} > 0
for: 1m
labels:
severity: critical
annotations:
summary: Critical vulnerability detected
description: "Critical vulnerability in {{ $labels.image }}"
- alert: UnauthorizedDeployment
expr: deployment_without_approval > 0
for: 1m
labels:
severity: warning
annotations:
summary: Deployment without security approval
- alert: SecretExposed
expr: secret_scan_detections > 0
for: 1m
labels:
severity: critical
annotations:
summary: Secret exposed in codeConclusion
A secure CI/CD pipeline requires security integration at every stage - from pre-commit hooks to runtime monitoring. The key is automation: security checks that run automatically, consistently, and without slowing down development.
Key takeaways:
- Shift left - Catch issues as early as possible
- Automate everything - Manual security checks don't scale
- Fail fast - Block insecure code from progressing
- Measure and improve - Track security metrics over time
- Balance security and velocity - Security shouldn't be a bottleneck
At DeviDevs, we help organizations build secure CI/CD pipelines that enable fast, safe deployments. Contact us to discuss your DevSecOps needs.
Related Resources
- ML CI/CD: Continuous Integration and Deployment for Machine Learning — Apply CI/CD security to ML pipelines
- MLOps Security: Securing Your ML Pipeline — ML-specific pipeline security
- MLOps Best Practices: Building Production-Ready ML Pipelines — Production pipeline architecture
- Common MLOps Mistakes and How to Avoid Them — Including manual deployment pitfalls
- What is MLOps? — Complete MLOps overview