Image Security

Container security at runtime is only as strong as the image it originates from. An image built from a bloated base image containing hundreds of unused packages, carrying secrets inadvertently baked into its layers, or pulled from untrusted sources introduces risks long before a single line of application code is executed. Developers writing the Dockerfile and Sysadmins/DevOps Engineers managing the build pipeline share the responsibility of closing these gaps, as decisions made during the build process directly impact the attack surface faced by potential attackers. This chapter covers four key areas of image security: selecting minimal base images, avoiding secret leaks caused by layer caching mechanisms, building a supply chain chain of trust from base images to registries, and treating vulnerability scanning as a continuous cycle rather than a one-time check.

23.1 Minimal Base Images

Every package, binary, and shell included in a base image expands the attack surface that can be exploited by an attacker, even if those components are never actually used by the application. This section explains why bloated base images pose a risk and how to select leaner alternatives.

23.1.1 The Problem with Bloated Base Images

Full-featured base images like ubuntu or debian include package managers, interactive shells, and hundreds of system utilities designed for general-purpose use rather than specifically running a single application. From a security perspective, every additional package introduces potential CVEs that require patching. Furthermore, if an attacker successfully gains access to a container via an application vulnerability, the presence of shells and networking tools such as curl or wget inside the image simplifies lateral exploration and downloading additional payloads. A minimal base image reverses this advantage: without a shell or package manager inside the image, an attacker who manages to execute arbitrary code loses access to many common tools used to escalate an attack.

23.1.2 Alpine, Distroless, and Scratch

The three most common minimal base image options used in practice exhibit distinct characteristics and trade-offs.

Alpine Linux uses musl libc and busybox in place of glibc and GNU coreutils, yielding a base image significantly smaller than traditional distributions while still offering the apk package manager and a sh shell for debugging purposes.

FROM alpine:3.20

RUN apk add --no-cache curl

CMD ["sh"]

Distroless, provided by the gcr.io/distroless project, takes minimality a step further by removing package managers, shells, and almost all system utilities, retaining only the minimal runtime required by specific programming languages. For instance, the distroless/nodejs variant contains only the Node.js runtime without npm or any shell access.

FROM node:20 AS build
WORKDIR /app
COPY . .
RUN npm ci --omit=dev

FROM gcr.io/distroless/nodejs20-debian12
COPY --from=build /app /app
WORKDIR /app
CMD ["server.js"]

Scratch is an entirely empty base image containing no filesystem components whatsoever. It is ideal for applications compiled into static binaries with no external dependencies, such as Go applications built with CGO_ENABLED=0.

FROM golang:1.23 AS build
WORKDIR /app
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o server .

FROM scratch
COPY --from=build /app/server /server
ENTRYPOINT ["/server"]

23.1.3 Choosing the Right Base Image

The absence of a shell in distroless and scratch improves security, but it also means traditional debugging techniques like running docker exec directly inside the container are no longer viable; troubleshooting must instead rely on application logs or ephemeral debug containers attached to the target container's namespace. In practice, this choice involves a trade-off that should be discussed across teams: stateless, production-ready applications benefit from distroless or scratch to minimize attack surfaces, whereas applications under active development requiring frequent on-container debugging may temporarily benefit from alpine. Compare image sizes and contents prior to making a decision using the following command:

docker images --format "table {{.Repository}}\t{{.Tag}}\t{{.Size}}"

23.2 Layer Caching Risks During Build

BuildKit caches each Dockerfile instruction as a layer to accelerate subsequent builds. However, this caching mechanism poses distinct security risks if sensitive data is written into these cached layers.

23.2.1 Build Caches Can Leak Secrets

A common mistake identified by Sysadmins/DevOps Engineers during image audits involves passing credentials, API keys, or private keys via standard ARG or ENV instructions, or copying them temporarily into an image and attempting to delete them in subsequent instructions. Because every layer in an image is immutable, files remain fully stored within historical layers even if they are deleted in a later step; a deletion command merely hides the file from the unified filesystem view rather than purging it from the underlying layer. Anyone with access to the image, whether via docker pull or direct layer inspection, can extract these secrets from earlier layers retained within the image payload.

docker history --no-trunc payment-service:1.4.2

The docker history command shown above displays the commands executed for each layer, including build arguments passed via docker build --build-arg, allowing secrets supplied during the build process to be recovered.

23.2.2 BuildKit Secret Mounts as a Solution

BuildKit's Secret Mounts feature provides a mechanism for passing sensitive data to build processes temporarily during execution of specific RUN instructions without persisting that data into final image layers. Secrets are mounted using the --mount=type=secret flag within a RUN instruction, while the secret payload itself is supplied via the --secret flag when executing docker build. This functionality requires BuildKit as the builder, which has been enabled by default since Docker Engine 23.0; for older Docker Engine versions, BuildKit must be manually enabled by setting the environment variable DOCKER_BUILDKIT=1 prior to invoking docker build.

# syntax=docker/dockerfile:1
FROM alpine:3.20

RUN --mount=type=secret,id=npm_token \
    NPM_TOKEN=$(cat /run/secrets/npm_token) && \
    echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > .npmrc && \
    npm install && \
    rm .npmrc

docker build --secret id=npm_token,src=./npm_token.txt -t app:1.0 .

Note the syntax directive on the first line of the Dockerfile (# syntax=docker/dockerfile:1), which is required for BuildKit to enable support for the --mount option in RUN instructions. Secret files mounted to /run/secrets/npm_token exist exclusively while that specific RUN instruction executes and are never permanently saved to any layer, unlike standard COPY operations whose contents remain recorded even if deleted in subsequent steps.

To verify that secrets have not leaked into image layers, inspect the image using docker history to confirm no trace of sensitive content exists in the output. For a comprehensive check, extract individual layer filesystems using docker save for deeper inspection.

docker save app:1.0 -o app.tar
tar -tvf app.tar

23.3 Supply Chain Security

Images running in production environments represent the output of an extended supply chain: third-party base images, dependencies fetched from public package registries, and build steps executed on CI/CD runners shared across multiple projects. Attackers compromising any single point in this pipeline rather than directly targeting production infrastructure represent a growing supply chain threat vectors across modern software deployments.

23.3.1 Chain of Trust from Base Images to Registries

Every FROM line in a Dockerfile represents an implicit decision to trust the maintainer of that base image. Base images sourced from Docker Official Images on Docker Hub undergo ongoing security scanning and review processes managed by Docker, offering higher trustworthiness than unverified public account images with unclear maintenance practices or build provenance. Sysadmins/DevOps Engineers must audit all base images used across their organizations to ensure they originate from clearly identified, actively maintained publishers, as unmaintained base images tend to accumulate unpatched vulnerabilities over time.

23.3.2 Image Verification with Sigstore Cosign

Docker previously provided Docker Content Trust (DCT) based on Notary for signing and verifying images using the DOCKER_CONTENT_TRUST environment variable. However, official Docker documentation indicates DCT has entered a retirement phase, with the Notary v1 service at notary.docker.io scheduled for deprecation on December 8, 2026. As a result, DCT is no longer recommended for new signing implementations. A widely adopted alternative in the container ecosystem is Sigstore using the cosign tool.

Cosign stores digital signatures alongside images inside the registry itself and supports two signing models: self-managed key pairs or keyless signing, which uses OIDC identities (such as GitHub Actions accounts) to request short-lived certificates from Sigstore's Fulcio certificate authority, eliminating the need to manage private keys manually.

cosign generate-key-pair

cosign sign --key cosign.key registry.example.com/payment-service@sha256:3f2b...

Signing is performed against the image's immutable digest rather than its tag, because tags can be reassigned to different images at any time while digests remain unique and permanent for specific content. Verification at the consumer end, such as within a deployment pipeline, relies on matching public keys to verify image integrity and origin prior to deployment.

cosign verify --key cosign.pub registry.example.com/payment-service:1.4.2

For keyless signing executed within CI/CD workflows, verification relies on validating the signer's OIDC identity rather than checking static public keys.

cosign verify registry.example.com/payment-service:1.4.2 \
  --certificate-identity="https://github.com/org/repo/.github/workflows/build.yml@refs/heads/main" \
  --certificate-oidc-issuer="https://token.actions.githubusercontent.com"

23.3.3 Software Bill of Materials (SBOM)

A Software Bill of Materials (SBOM) is a comprehensive inventory detailing all components, packages, and dependencies composing an image, recording their respective names, versions, and licenses in standardized formats like SPDX. An SBOM gives Sysadmins/DevOps Engineers visibility into the precise components bundled inside an image, allowing teams to instantly identify affected images when new vulnerabilities are disclosed without needing to unpack or inspect images individually.

BuildKit supports automated SBOM generation during build processes using the --sbom flag within docker buildx build, using Anchore's Syft scanner by default. This attestation capability requires Buildx version 0.10+ alongside a BuildKit builder version 0.11+, and is not supported by the legacy default builder shipped with docker unless a new builder is explicitly initialized; create and activate a supported builder using docker buildx create --use if the --sbom flag fails with attestation errors.

docker buildx build --sbom=true -t payment-service:1.4.2 --push .

Generated SBOMs are stored as attestations attached to the target image in the registry and can be inspected at any time via docker buildx imagetools inspect without requiring image rebuilds.

docker buildx imagetools inspect payment-service:1.4.2 --format "{{ json .SBOM.SPDX }}"

23.4 Vulnerability Scanning as a Continuous Lifecycle

Vulnerability scanning tools such as Docker Scout and Trivy are routinely used to scan images for known CVEs prior to release. Rather than repeating basic usage commands, this section addresses how scan findings should be managed as part of an ongoing, continuous security lifecycle rather than a single pass-or-fail build gate.

23.4.1 Establishing Vulnerability Threshold Policies

Without clear policy definitions, teams risk falling into opposite extremes: ignoring scan outputs entirely due to alert fatigue or blocking all deployments over low-severity issues that carry minimal real-world risk. Practical operational policies typically dictate that critical findings block production promotion entirely, high findings require manual review prior to approval, while medium and low findings are tracked in remediation backlogs without stopping delivery pipelines. Documenting these thresholds explicitly ensures release gate decisions remain objective rather than subject to individual team member discretion.

23.4.2 Managing False Positives with Exemptions

Not every CVE flagged by automated scanners impacts an application in practice; a package might contain a vulnerability within a function never invoked by the application code, or the risk may be mitigated through external controls like network isolation. Trivy supports a .trivyignore file to explicitly exclude specified CVEs from scan reports, allowing teams to document justification notes and set optional expiration dates so exemptions are reviewed periodically.

# CVE-2024-XXXX: vulnerable function is not invoked within this codebase
CVE-2024-XXXX

# Temporary mitigation applied via network policy; review before expiration date
CVE-2024-YYYY exp:2026-12-31

trivy image --ignorefile .trivyignore payment-service:1.4.2

Entries added to .trivyignore should undergo formal peer review rather than unilateral additions to bypass failed pipeline checks, as exemption files can obscure findings that warrant remediation.

23.4.3 Pre-build Dockerfile Linting with Hadolint

Hadolint is a dedicated Dockerfile linter that checks build instructions against best practices before an image is built, distinguishing it from post-build scanners like Docker Scout or Trivy. Catching configuration flaws at this initial stage identifies potential security issues early in the delivery process, well before image scanning or deployment phases.

docker run --rm -i hadolint/hadolint < Dockerfile

Relevant security rules enforced by Hadolint include rule DL3007, which warns against using the latest tag in FROM instructions because base image contents can change without notice, and rule DL3008, which recommends pinning package versions explicitly when using apt-get install to ensure builds remain reproducible and protected against unvetted package updates. Executing Hadolint as an early CI/CD pipeline step catches Dockerfile issues faster and at lower remediation cost than identifying them during later image scanning stages.