Runtime Security

An image that is free of vulnerabilities can still be exploited if the container running it is given full freedom over the host kernel, system resources, networking, or carelessly attached credentials. Runtime security governs what a container is allowed and not allowed to do while it is running, regardless of how secure its source image is. Sysadmins or DevOps Engineers managing containers in production need to apply multiple control layers simultaneously: kernel security modules such as AppArmor and SELinux to restrict access to the file system and syscalls, resource limits so that a single rogue container does not exhaust the memory or CPU of the entire host, network isolation so that one container cannot freely reach another container, and secret management that does not leak credentials through environment variables or image layers. This chapter covers these four control layers practically.

24.1 AppArmor and SELinux

AppArmor and SELinux are two Linux Security Modules (LSM) that restrict what a process can access, including processes inside containers, far beyond standard user permissions. Both operate on similar principles but differ in their configuration methods, and Linux distributions usually define one of them as the default.

24.1.1 Default and Custom AppArmor Profiles

AppArmor is active by default in distributions like Ubuntu and Debian, and Docker Engine automatically applies a profile named docker-default to every container executed without requiring any additional configuration. According to official Docker documentation, this docker-default profile is moderate, strict enough to block many dangerous actions while remaining lenient to maintain compatibility with the majority of general applications. Verify that AppArmor is active and the default profile is applied using the following command.

docker info --format '{{.SecurityOptions}}'

An output containing name=apparmor indicates that AppArmor is active on the host. The default profile can be explicitly stated or replaced with a custom profile via the --security-opt option during docker run.

docker run --rm -it --security-opt apparmor=docker-default alpine sh

For more specific requirements, such as an application needing tighter restrictions than the default profile, Sysadmins or DevOps Engineers can write a custom AppArmor profile as a text file, load it into the kernel using apparmor_parser, and then apply it to specific containers.

sudo apparmor_parser -r -W /etc/apparmor.d/nginx-restricted

docker run --rm -d --security-opt apparmor=nginx-restricted nginx:1.27

Conversely, disabling AppArmor using --security-opt apparmor=unconfined allows the container to run without any LSM restrictions. This option is sometimes required for debugging or for specific applications that genuinely require broader access, but an attacker who successfully exploits a vulnerability inside an unconfined container gains significantly more maneuverability compared to a container using the default profile.

24.1.2 SELinux Labels and Contexts

SELinux is the default in Red Hat-derived distributions such as RHEL, Fedora, and CentOS. It works by attaching security labels to every process and system resource, then determining access permissions based on label matching rather than relying solely on standard Unix permissions. When SELinux is active in enforcing mode, processes inside a container automatically receive a label of type container_t, while files bind-mounted from the host retain their original labels unless explicitly relabeled.

getenforce

The most common issue Sysadmins or DevOps Engineers encounter when using bind mounts on a host with active SELinux is a permission denied error despite correct Unix permissions on the files. This occurs because the SELinux label on the host directory does not match the label expected by the process inside the container. The solution is to append the :z or :Z suffix to the volume option during docker run so Docker automatically relabels the directory.

docker run --rm -d -v /opt/app/config:/config:z nginx:1.27

The lowercase :z suffix assigns a shared label that allows access by multiple containers simultaneously, whereas the uppercase :Z suffix assigns a private label that restricts access exclusively to that container using a unique Multi-Category Security (MCS) category. For more granular control, SELinux labels on containers can also be set directly via --security-opt label.

docker run --rm -it --security-opt label=level:s0:c100,c200 alpine sh

Just like AppArmor, SELinux can be disabled per container via --security-opt label=disable, but this decision should be a documented exception rather than a default habit to avoid label troubleshooting.

24.1.3 Choosing the Module Based on the Host Distro

AppArmor and SELinux do not run simultaneously on a single Linux host; the system only uses one according to the default support of the installed distribution. Sysadmins or DevOps Engineers do not need to choose between them manually, as this choice is predetermined by the host distro: Ubuntu and Debian ship with AppArmor, whereas RHEL, Fedora, and CentOS ship with SELinux. What must be ensured is that the distribution's built-in security module is active and not accidentally disabled for convenience, because disabling an LSM removes an important defense layer against container escape, especially when a new vulnerability in the container runtime is discovered before a patch is applied.

24.2 Restricting Container Resources

Containers running without resource limits can exhaust all host memory or CPU, whether due to application bugs like memory leaks or due to an attacker deliberately triggering a denial of service from within a compromised container. Docker leverages cgroups in the Linux kernel to limit these resources per container.

24.2.1 Memory Limits and OOM Killer Behavior

The --memory (or -m) option in docker run sets the maximum memory limit a container can consume. If a container attempts to use memory beyond this limit, the Linux kernel's OOM killer will forcibly terminate the process inside that container.

docker run -d --memory=512m --memory-swap=512m --name payment-api payment-service:1.4.2

Setting --memory-swap to the same value as --memory in the example above disables additional swap usage, ensuring the total memory usable by the container is strictly limited to the --memory value. If --memory-swap is left unset, Docker by default permits additional swap equal to the --memory value, allowing total memory plus swap to reach double the visible limit.

The behavior of the OOM killer can be adjusted using two additional options. --oom-score-adj adjusts the container's priority for selection or avoidance by the OOM killer relative to other processes on the host, ranging from -1000 (rarely selected) to 1000 (highest priority for termination). Meanwhile, --oom-kill-disable prevents the OOM killer from terminating the container entirely. However, this option carries high risk: a container out of memory that cannot be terminated can render the entire host unresponsive. Use this option only for critical containers configured with thoroughly safe --memory limits.

docker stats payment-api --no-stream

The docker stats command above displays the actual memory and CPU consumption of the container relative to its defined limits, useful for verifying whether the assigned --memory value is realistic or overly restrictive to the point of triggering the OOM killer during normal application operation.

24.2.2 CPU Limits

Unlike memory limits that can lead to forced container termination, CPU limits function through throttling: a container reaching its CPU limit is not killed, but merely slowed down using the Completely Fair Scheduler (CFS) mechanism in the Linux kernel. The --cpus option sets the maximum number of CPU cores a container can use, and its value can be a fraction.

docker run -d --cpus=1.5 --name payment-api payment-service:1.4.2

The command above limits the payment-api container to a maximum of 1.5 CPU cores even if the host has more cores available. For relative priority needs among containers instead of absolute limits, the --cpu-shares option determines the CPU allocation weight during contention; containers with higher --cpu-shares receive a larger portion of CPU time compared to other containers when the host CPU is heavily utilized, but it imposes no restriction when the CPU is not being used by other containers.

docker run -d --cpu-shares=512 --name batch-worker worker-service:2.1.0

24.2.3 PID Limits and Ulimits

In addition to memory and CPU, the number of processes allowed to run inside a single container must also be restricted. Applications that spawn processes or threads indefinitely, whether due to fork bomb bugs or exploitation, can exhaust process slots across the entire host and disrupt other containers. The --pids-limit option sets the maximum number of processes and threads allowed inside the container's cgroup.

docker run -d --pids-limit=100 --name payment-api payment-service:1.4.2

Besides --pids-limit, the --ulimit option configures process-level resource limits, such as the maximum number of open file descriptors (nofile) or the maximum file size that can be written (fsize), similar to the standard Linux shell ulimit command but applied specifically to processes inside the container.

docker run -d --ulimit nofile=1024:2048 --name payment-api payment-service:1.4.2

In production, resource constraints like these are most effectively implemented as cluster-wide standards via docker-compose.yml or deployment templates, rather than being added manually one by one each time a container is launched, ensuring no container runs without resource limits.

24.3 Network Policies and Network Isolation

Containers on the same Docker network can communicate with each other by default without restrictions. While convenient during development, this presents a risk in production if a container is compromised by an attacker and used as a stepping stone to attack other containers. Docker does not have a declarative network policy object like Kubernetes, but it provides equivalent mechanisms through a combination of custom networks, default isolation options, and firewall rules.

24.3.1 Isolation via Custom Bridge Networks

The most fundamental practice for limiting communication scope between containers is isolating services into multiple custom bridge networks based on their communication requirements, rather than placing all containers on a single network. Containers that do not need to communicate are placed on different networks, rendering them unreachable from one another by default.

docker network create frontend-net
docker network create backend-net

docker run -d --network frontend-net --name web nginx:1.27
docker run -d --network backend-net --name db postgres:16
docker run -d --network frontend-net --network-alias api --name api payment-service:1.4.2
docker network connect backend-net api

In the example above, the web container is connected only to frontend-net and cannot reach db at all, whereas api is intentionally connected to both networks so it can be accessed from frontend-net while also accessing db on backend-net. Verify this isolation by attempting to ping from the web container to the db container; a failed connection indicates that network isolation is functioning as intended.

docker exec web ping -c 2 db

For networks that must not have outbound external access, such as networks used strictly for internal databases, the --internal option during network creation prevents containers within it from acquiring a default gateway to the external host network.

docker network create --internal backend-net

Containers connected to an --internal network cannot initiate outbound connections to the internet or to other external Docker networks, making it ideal for databases or internal services that should only be reached from within the cluster.

24.3.2 Disabling Inter-Container Communication

In addition to network segregation, Docker offers an Inter-Container Communication (ICC) setting that controls whether containers on the same bridge network can communicate directly via container IP addresses. Disabling ICC on the default bridge network is done by modifying the daemon configuration in /etc/docker/daemon.json.

{
  "icc": false
}

After updating daemon.json, restart the Docker daemon to apply the configuration.

sudo systemctl restart docker

The icc configuration in daemon.json applies only to the default bridge network. For custom networks, an equivalent option is supplied via driver options when the network is created.

docker network create -o com.docker.network.bridge.enable_icc=false isolated-net

Disabling ICC causes Docker to insert iptables rules that block all direct traffic between containers on that network. Containers that still require inter-communication must be linked explicitly using --link on legacy bridges or isolated using separate custom networks as discussed in the previous section. Disabling ICC by default and selectively opening communication for required container pairs is significantly safer than leaving all containers exposed to each other.

24.3.3 Firewalls via the DOCKER-USER Chain

Docker automatically manipulates iptables rules to handle port mapping and container forwarding. This manipulation can override custom firewall rules added by Sysadmins or DevOps Engineers directly to the standard FORWARD chain. According to official Docker documentation, the DOCKER-USER chain is provided specifically so custom firewall rules persist even when the Docker daemon restarts, as this chain is processed prior to Docker's own DOCKER and DOCKER-FORWARD chains.

sudo iptables -I DOCKER-USER -s 203.0.113.0/24 -j ACCEPT
sudo iptables -I DOCKER-USER -j DROP

The example above allows traffic only from the 203.0.113.0/24 IP block to all containers, rejecting any traffic that does not match preceding rules. Rule order within DOCKER-USER is critical because iptables processes rules top-down and stops upon finding a match; therefore, more specific ACCEPT rules must precede generic DROP rules. It is also important to note that packets reaching the DOCKER-USER chain have already undergone Docker's Destination NAT process, meaning rules written here match internal container IP addresses rather than the original source host address before NAT. Test firewall rules in a staging environment prior to production deployment, as incorrect rule sequencing can inadvertently block legitimate traffic or expose unauthorized access.

24.4 Secrets Management

Credentials such as database passwords, API keys, or private keys require special handling compared to ordinary configuration files, as a secret leak can directly grant an attacker access to external systems outside the container itself. This section explains why common methods of passing secrets into containers carry security risks, along with safer mechanisms to replace them.

24.4.1 Risks of Environment Variables for Secrets

Passing secrets via environment variables using the -e option in docker run or the environment section in docker-compose.yml is a widespread pattern that carries several risks. Environment variables of a process can be read via docker inspect by anyone with access to the Docker daemon, appear in /proc/<pid>/environ accessible by other processes with sufficient privileges in the same container, and are frequently printed to application logs during environment dumps for debugging.

docker inspect --format '{{.Config.Env}}' payment-api

The command above demonstrates how easily anyone with access to docker inspect can view all environment variables of a container, including secrets passed via -e. A safer alternative is passing secrets as mounted files inside the container, as file permissions are much easier to restrict and file contents are not exposed through docker inspect.

24.4.2 Docker Compose Secrets

Docker Compose secrets provide a standardized method for managing file-based secrets using the top-level secrets section in docker-compose.yml. According to the official Compose Specification, the secrets declaration can be used in standard Compose without Swarm; the difference outside Swarm is that secrets act as standard file bind mounts with standardized paths and permissions, rather than encrypted cluster-managed secrets as in Swarm.

services:
  payment-api:
    image: payment-service:1.4.2
    secrets:
      - db_password

secrets:
  db_password:
    file: ./secrets/db_password.txt

The configuration above exposes the contents of db_password.txt inside the container at path /run/secrets/db_password, allowing the application to read the secret file directly instead of relying on environment variables.

DB_PASSWORD=$(cat /run/secrets/db_password)

Ensure that secret files like db_password.txt are never committed to a git repository; add them to .gitignore and distribute them through secure secondary channels, such as external secret managers or encrypted deployment pipelines. Verify that the secret is mounted correctly using the following command without revealing its contents in the terminal.

docker compose exec payment-api ls -l /run/secrets/

24.4.3 Docker Secrets in Swarm Mode

When applications are deployed in Swarm mode, the docker secret mechanism offers stronger security guarantees than file-based secrets in standard Compose: secret data is stored encrypted in the Swarm manager's Raft log, distributed strictly to nodes running the targeted service, and mounted into the container as an in-memory tmpfs so it is never written to disk.

printf "SuperSecretPassword123" | docker secret create db_password -

docker service create \
  --name payment-api \
  --secret db_password \
  payment-service:1.4.2

As with Compose, secrets created via docker secret create are available inside the container at /run/secrets/db_password. When secret contents need updating—such as during periodic database password rotation—existing secrets cannot be edited in place. Create a new secret with a distinct name, update the service to use the new secret, and remove the old secret once it is no longer referenced by any service.

docker service update \
  --secret-rm db_password \
  --secret-add source=db_password_v2,target=db_password \
  payment-api

docker secret rm db_password

Verify that no other service is actively using an old secret before attempting to remove it; the docker secret rm command will fail and report an error if the secret remains in use, serving as a safety mechanism in Swarm to prevent active services from losing access unexpectedly.

docker secret ls

24.4.4 Integration with External Secret Managers

For organizations maintaining centralized credential management policies, secrets should not be stored as static files on servers. Instead, they should be retrieved dynamically from an external secret manager such as HashiCorp Vault, AWS Secrets Manager, or Google Secret Manager when a container starts up. A common pattern involves using an init container or sidecar to fetch secrets from the external manager via authenticated APIs and write them to a shared tmpfs volume accessible to the primary application container. This allows the application to read secrets from local files without needing embedded integration logic for the secret manager.

This approach offloads authentication, rotation, and access audit logging entirely to a platform designed specifically for secret management, replacing static secret files that are prone to stale rotation or exposure in backups and volume snapshots. Deciding to implement an external secret manager is typically made in coordination with organizational security teams, as implementation impacts broader identity and access management policies beyond Docker configuration alone.