DevSecOps

Container and Kubernetes Security: Production Best Practices

DeviDevs Team
••9 min read
#kubernetes#container-security#docker#pod-security#cloud-native

Container and Kubernetes Security: Production Best Practices

Container security is a multi-layered challenge that spans the entire container lifecycle - from base images to runtime. Kubernetes adds another dimension with its own security model for orchestration.

This guide covers practical security practices for production container environments.

Container Image Security

Secure Base Images

# Use distroless or minimal base images
FROM gcr.io/distroless/nodejs20-debian12:nonroot
 
# Or use specific Alpine version with security updates
FROM node:20.10.0-alpine3.18
 
# Apply security updates
RUN apk update && apk upgrade --no-cache
 
# Create non-root user
RUN addgroup -g 1001 -S appgroup && \
    adduser -u 1001 -S appuser -G appgroup
 
# Set working directory
WORKDIR /app
 
# Copy only necessary files
COPY --chown=appuser:appgroup package*.json ./
RUN npm ci --only=production
 
COPY --chown=appuser:appgroup . .
 
# Remove unnecessary packages
RUN rm -rf /var/cache/apk/* /root/.npm
 
# Run as non-root
USER appuser
 
# Use specific UID
USER 1001
 
# Don't run as root even if image allows it
CMD ["node", "server.js"]

Multi-Stage Build for Security

# Build stage
FROM node:20-alpine AS builder
 
WORKDIR /build
 
COPY package*.json ./
RUN npm ci
 
COPY . .
RUN npm run build && npm prune --production
 
# Security scan stage
FROM aquasec/trivy:latest AS scanner
COPY --from=builder /build /scan
RUN trivy filesystem --exit-code 1 --severity HIGH,CRITICAL /scan
 
# Production stage
FROM gcr.io/distroless/nodejs20-debian12:nonroot
 
WORKDIR /app
 
# Copy only built artifacts
COPY --from=builder /build/dist ./dist
COPY --from=builder /build/node_modules ./node_modules
COPY --from=builder /build/package.json ./
 
USER nonroot
 
CMD ["dist/server.js"]

Image Scanning Pipeline

# GitHub Actions workflow for image scanning
name: Container Security Scan
 
on:
  push:
    branches: [main]
  pull_request:
 
jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
 
      - name: Build image
        run: docker build -t app:${{ github.sha }} .
 
      - name: Trivy vulnerability scan
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: 'app:${{ github.sha }}'
          format: 'sarif'
          output: 'trivy-results.sarif'
          severity: 'CRITICAL,HIGH'
          exit-code: '1'
 
      - name: Grype vulnerability scan
        uses: anchore/scan-action@v3
        with:
          image: 'app:${{ github.sha }}'
          fail-build: true
          severity-cutoff: high
 
      - name: Dockle lint
        uses: erzz/dockle-action@v1
        with:
          image: 'app:${{ github.sha }}'
          exit-code: '1'
          failure-threshold: high
 
      - name: Upload results to GitHub Security
        uses: github/codeql-action/upload-sarif@v2
        with:
          sarif_file: 'trivy-results.sarif'

Kubernetes Pod Security

Pod Security Standards

# Restricted Pod Security Standard
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted
 
---
# Secure Pod specification
apiVersion: v1
kind: Pod
metadata:
  name: secure-app
  namespace: production
spec:
  # Security context at pod level
  securityContext:
    runAsNonRoot: true
    runAsUser: 1001
    runAsGroup: 1001
    fsGroup: 1001
    seccompProfile:
      type: RuntimeDefault
 
  # Service account
  serviceAccountName: app-service-account
  automountServiceAccountToken: false
 
  containers:
    - name: app
      image: myregistry/app:v1.0.0@sha256:abc123...
      imagePullPolicy: Always
 
      # Container security context
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        runAsNonRoot: true
        runAsUser: 1001
        capabilities:
          drop:
            - ALL
 
      # Resource limits
      resources:
        limits:
          cpu: "1"
          memory: "512Mi"
          ephemeral-storage: "1Gi"
        requests:
          cpu: "100m"
          memory: "128Mi"
 
      # Health probes
      livenessProbe:
        httpGet:
          path: /health
          port: 8080
        initialDelaySeconds: 30
        periodSeconds: 10
 
      readinessProbe:
        httpGet:
          path: /ready
          port: 8080
        initialDelaySeconds: 5
        periodSeconds: 5
 
      # Volume mounts
      volumeMounts:
        - name: tmp
          mountPath: /tmp
        - name: secrets
          mountPath: /secrets
          readOnly: true
 
  # Volumes
  volumes:
    - name: tmp
      emptyDir: {}
    - name: secrets
      secret:
        secretName: app-secrets
        defaultMode: 0400
 
  # Node affinity
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: node-type
                operator: In
                values:
                  - production
 
  # Tolerations
  tolerations: []
 
  # DNS policy
  dnsPolicy: ClusterFirst

Deployment Security

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-app
  namespace: production
spec:
  replicas: 3
  revisionHistoryLimit: 5
 
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
 
  selector:
    matchLabels:
      app: secure-app
 
  template:
    metadata:
      labels:
        app: secure-app
      annotations:
        # Enable security scanning
        container.apparmor.security.beta.kubernetes.io/app: runtime/default
 
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1001
        fsGroup: 1001
        seccompProfile:
          type: RuntimeDefault
 
      containers:
        - name: app
          image: myregistry/app:v1.0.0
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL
 
          ports:
            - containerPort: 8080
              protocol: TCP
 
          env:
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: password
 
          resources:
            limits:
              cpu: "1"
              memory: "512Mi"
            requests:
              cpu: "100m"
              memory: "128Mi"
 
---
# Pod Disruption Budget
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: secure-app-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: secure-app

Network Security

Network Policies

# Default deny all traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
 
---
# Allow specific ingress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-app-ingress
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: secure-app
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              name: ingress-nginx
          podSelector:
            matchLabels:
              app.kubernetes.io/name: ingress-nginx
      ports:
        - protocol: TCP
          port: 8080
 
---
# Allow app to database
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-app-to-db
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: secure-app
  policyTypes:
    - Egress
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: postgresql
      ports:
        - protocol: TCP
          port: 5432
    # Allow DNS
    - to:
        - namespaceSelector: {}
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
 
---
# Deny egress to metadata service (AWS/GCP)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-metadata-access
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 169.254.169.254/32  # AWS metadata
              - 169.254.0.0/16       # Link-local
              - 10.0.0.0/8           # Internal (adjust as needed)

Secrets Management

External Secrets Operator

# External Secrets configuration
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager
  namespace: production
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
 
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-secrets
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
 
  target:
    name: app-secrets
    creationPolicy: Owner
    template:
      type: Opaque
      data:
        db-password: "{{ .db_password }}"
        api-key: "{{ .api_key }}"
 
  data:
    - secretKey: db_password
      remoteRef:
        key: production/app/database
        property: password
 
    - secretKey: api_key
      remoteRef:
        key: production/app/api
        property: key

Sealed Secrets

# Sealed Secret (encrypted, safe for git)
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: app-secrets
  namespace: production
spec:
  encryptedData:
    db-password: AgBY8s9j3K...encrypted...
    api-key: AgDJ2k4m8N...encrypted...
  template:
    metadata:
      name: app-secrets
      namespace: production
    type: Opaque

RBAC Security

Least Privilege Service Account

# Service Account
apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-service-account
  namespace: production
automountServiceAccountToken: false
 
---
# Role with minimal permissions
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: app-role
  namespace: production
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["app-config"]
    verbs: ["get"]
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["app-secrets"]
    verbs: ["get"]
 
---
# Role Binding
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: app-role-binding
  namespace: production
subjects:
  - kind: ServiceAccount
    name: app-service-account
    namespace: production
roleRef:
  kind: Role
  name: app-role
  apiGroup: rbac.authorization.k8s.io

Cluster-Wide RBAC Restrictions

# Deny privilege escalation cluster-wide
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: deny-privilege-escalation
rules:
  - apiGroups: [""]
    resources: ["pods/exec", "pods/attach"]
    verbs: ["create"]
  - apiGroups: [""]
    resources: ["nodes/proxy", "pods/proxy", "services/proxy"]
    verbs: ["*"]
 
---
# Audit role for security team
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: security-auditor
rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: []  # No access to secrets

Runtime Security

Falco Rules

# Falco rules for runtime detection
- rule: Terminal shell in container
  desc: Detect shell execution in container
  condition: >
    spawned_process and container
    and shell_procs
    and proc.tty != 0
    and container_entrypoint
  output: >
    Shell spawned in container
    (user=%user.name container=%container.name
    shell=%proc.name parent=%proc.pname)
  priority: WARNING
  tags: [container, shell]
 
- rule: Sensitive file access
  desc: Detect access to sensitive files
  condition: >
    open_read and container
    and sensitive_files
  output: >
    Sensitive file read
    (user=%user.name file=%fd.name container=%container.name)
  priority: WARNING
  tags: [container, filesystem]
 
- rule: Crypto mining detection
  desc: Detect potential crypto mining
  condition: >
    spawned_process and container
    and (proc.name in (crypto_miners) or
         proc.args contains "stratum" or
         proc.args contains "xmr")
  output: >
    Crypto mining detected
    (user=%user.name command=%proc.cmdline container=%container.name)
  priority: CRITICAL
  tags: [container, crypto]
 
- rule: Outbound connection to unusual port
  desc: Detect outbound connections to unusual ports
  condition: >
    outbound and container
    and not fd.port in (allowed_outbound_ports)
  output: >
    Unusual outbound connection
    (container=%container.name connection=%fd.name)
  priority: WARNING
  tags: [container, network]

Admission Controller Policies (Kyverno)

# Require resource limits
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: require-limits
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: "Resource limits are required"
        pattern:
          spec:
            containers:
              - resources:
                  limits:
                    memory: "?*"
                    cpu: "?*"
 
---
# Require non-root
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-non-root
spec:
  validationFailureAction: Enforce
  rules:
    - name: check-non-root
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: "Containers must not run as root"
        pattern:
          spec:
            securityContext:
              runAsNonRoot: true
            containers:
              - securityContext:
                  runAsNonRoot: true
 
---
# Require approved registries
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-approved-registries
spec:
  validationFailureAction: Enforce
  rules:
    - name: check-registry
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: "Images must come from approved registries"
        pattern:
          spec:
            containers:
              - image: "gcr.io/myproject/* | myregistry.azurecr.io/*"

Security Monitoring

Prometheus Security Metrics

# ServiceMonitor for security metrics
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: security-metrics
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: security-metrics
  endpoints:
    - port: metrics
      interval: 30s
      path: /metrics
 
---
# PrometheusRule for security alerts
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: security-alerts
  namespace: monitoring
spec:
  groups:
    - name: container-security
      rules:
        - alert: PrivilegedContainerDetected
          expr: sum(kube_pod_container_info{container_privileged="true"}) > 0
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: Privileged container detected
 
        - alert: ContainerRunningAsRoot
          expr: sum(kube_pod_container_info{container_run_as_user="0"}) > 0
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: Container running as root
 
        - alert: HighCVEVulnerabilities
          expr: trivy_vulnerability_count{severity="CRITICAL"} > 0
          for: 1m
          labels:
            severity: critical
          annotations:
            summary: Critical CVE detected in image

Conclusion

Container and Kubernetes security requires a defense-in-depth approach across the entire lifecycle. From secure base images to runtime protection, each layer adds to the overall security posture.

Key takeaways:

  1. Secure images - Use minimal bases, scan regularly
  2. Least privilege - Non-root, minimal capabilities, read-only filesystem
  3. Network segmentation - Default deny, explicit allow
  4. Secrets management - External secrets, encryption at rest
  5. Runtime protection - Monitor, detect, respond

At DeviDevs, we help organizations secure their Kubernetes deployments with comprehensive security implementations. Contact us to discuss your container security needs.

Weekly AI Security & Automation Digest

Get the latest on AI Security, workflow automation, secure integrations, and custom platform development delivered weekly.

No spam. Unsubscribe anytime.

Related Articles

DevSecOps

Container Security Best Practices for Kubernetes Environments

Implement comprehensive container security controls for Kubernetes deployments including image scanning, runtime protection, network policies, and pod security standards.

•13 min read
DevSecOps

Container Security in Kubernetes: A DevSecOps Implementation Guide

Comprehensive guide to securing containerized workloads in Kubernetes including image scanning, runtime security, network policies, and admission controllers.

•9 min read
DevSecOps

Kubernetes CrashLoopBackOff: Complete Troubleshooting Guide

Fix Kubernetes CrashLoopBackOff errors including OOMKilled, failed health checks, missing secrets, image pull errors, and configuration issues with step-by-step solutions.

•6 min read
← Back to Blog