Docker in Production

Containers that have been successfully built and run smoothly on a developer's laptop are not necessarily ready to be deployed directly to production. In production, different questions arise: if an application crashes in the middle of the night, where can its logs be traced? If resource utilization silently spikes, who will know before users start complaining? If a process inside a container hangs but the container itself is still "running", how does the system know it needs to take action? And if workloads are distributed across multiple hosts, what manages scheduling and automated recovery when a node fails? This chapter discusses the four pillars that answer these questions: logging strategies to maintain application traceability, monitoring and observability to provide visibility into system health before incidents occur, health checks so Docker knows when a container is genuinely healthy, and an overview of container orchestration for managing workloads across multiple hosts.

25.1 Logging Strategies

Logs are the primary source of information Sysadmins and DevOps Engineers look for when troubleshooting production incidents. However, improperly managed logs can create new problems: consuming all host disk space or vanishing entirely when a container is removed. This section covers how Docker handles container logs and how to manage them so they remain useful without overwhelming the system.

25.1.1 Docker Logging Drivers

Docker does not store container logs in a single fixed way; each container uses a logging driver that determines the storage format and the final destination of the process's stdout and stderr outputs. According to official Docker documentation, the json-file driver is the default used if no other configuration is specified, storing each log line as a JSON object on the host disk. Beyond json-file, Docker also provides the local driver, which uses a more compact format and performs automatic log rotation, syslog and journald for integration with native Linux logging systems, as well as several drivers for external logging platforms such as gelf, fluentd, awslogs, and splunk. Check the driver currently used by the Docker Engine on the host using the following command.

docker info --format '{{.LoggingDriver}}'

The logging driver can be specified per container using the --log-driver option during docker run, without altering global daemon configurations.

docker run -d --log-driver=journald --name payment-api payment-service:1.4.2

The journald driver above only works if the host runs systemd with active journald, as this driver forwards container logs directly to the operating system journal. To inspect logs sent to journald, use the journalctl command with a container identifier filter.

sudo journalctl CONTAINER_NAME=payment-api

25.1.2 Log Rotation and Log Size

The default json-file driver does not rotate logs automatically unless explicitly configured, meaning applications emitting high log volumes can gradually exhaust host disk space. Limit the size and number of log files per container using --log-opt options during docker run.

docker run -d --log-driver=json-file --log-opt max-size=10m --log-opt max-file=3 --name payment-api payment-service:1.4.2

The configuration above limits each log file to a maximum of 10 MB. Docker creates a new file once this threshold is reached and removes the oldest log file when the total exceeds the max-file value of 3. To apply these limits automatically to all new containers without specifying them in every docker run command, set default values in /etc/docker/daemon.json at the daemon level.

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

sudo systemctl restart docker

Note that all values inside log-opts must be written as strings, including max-file, which is logically a numeric value. As a more efficient alternative to json-file, official Docker documentation recommends the local driver, which uses a more compact storage format and automatically handles log rotation even without log-opt parameters, making it a suitable default for production hosts where log rotation has not been configured manually.

docker run -d --log-driver=local --name payment-api payment-service:1.4.2

In practice, full disk incidents caused by unrotated container logs are one of the most easily preventable yet frequently overlooked causes of downtime, as symptoms often surface only after months of stable container execution. Include log-opts in daemon.json as part of standard provisioning for every new Docker host, rather than an afterthought applied after an incident.

25.1.3 Centralized Logging

Logs stored only on individual host disks become difficult to trace once containers move across hosts, are deleted, or when workloads are distributed across many nodes. Centralized logging aggregates logs from all containers into a single location, enabling Sysadmins and DevOps Engineers to search and correlate logs across containers without logging into each host individually. One direct approach uses the syslog driver to send logs to a remote syslog server over the network.

docker run -d --log-driver=syslog --log-opt syslog-address=udp://logs.internal:514 --name payment-api payment-service:1.4.2

For more structured logging ecosystems like Elasticsearch or Graylog, the gelf driver transmits logs in Graylog Extended Log Format, carrying structured metadata alongside raw log text.

docker run -d --log-driver=gelf --log-opt gelf-address=udp://logs.internal:12201 --name payment-api payment-service:1.4.2

Another common pattern, especially when multiple containers need to ship logs to the same destination, is running a logging agent such as Fluentd as a separate container on the same host, then configuring application containers to use the fluentd driver so logs pass through the agent before processing and transmission to the storage backend.

docker run -d --log-driver=fluentd --log-opt fluentd-address=127.0.0.1:24224 --log-opt tag=payment-api --name payment-api payment-service:1.4.2

Before changing the default logging driver across production hosts, test thoroughly in staging environments, as drivers like gelf, fluentd, and awslogs can block container processes when log endpoints are unreachable, unlike json-file, which always succeeds in writing locally. Verify log flow after changing drivers using docker logs for supported drivers, or directly inspect centralized logging dashboards for drivers that do not support docker logs.

25.2 Monitoring and Observability

Logging reports what occurred after an event is recorded, whereas monitoring and observability offer continuous insights into system health, including resource usage trends that serve as early warnings before incidents escalate. This section discusses container monitoring approaches ranging from built-in Docker commands to Prometheus integration.

25.2.1 Docker Stats for Quick Monitoring

The docker stats command displays real-time CPU, memory, network, and disk I/O utilization for running containers, making it ideal for quick checks without installing extra tools.

docker stats

For automation needs, such as basic monitoring scripts or scheduled logging, the --no-stream flag causes docker stats to output a single snapshot and exit, rather than continuously updating like its default interactive mode.

docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}"

While docker stats suffices for manual inspections or quick debugging, it does not store historical metrics nor trigger automated alerts when resource thresholds are breached, rendering it unsuitable as the sole monitoring solution in production environments.

25.2.2 Container Metrics with cAdvisor and Prometheus

cAdvisor (Container Advisor) is an open-source tool by Google that collects detailed resource metrics for every container and exposes them in a Prometheus-compatible format. cAdvisor runs as a container itself, mounting several host directories as read-only to read container information on the host system.

docker run \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:rw \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --publish=8080:8080 \
  --detach=true \
  --name=cadvisor \
  ghcr.io/google/cadvisor:v0.60.5

Once active, cAdvisor's built-in web dashboard can be accessed via browser at http://localhost:8080, and Prometheus metrics endpoints are available at the /metrics path on the same port.

curl http://localhost:8080/metrics

To ingest these metrics continuously, add the cAdvisor target to the Prometheus configuration file for regular scraping.

scrape_configs:
  - job_name: cadvisor
    scrape_interval: 15s
    static_configs:
      - targets: ["cadvisor:8080"]

In addition to container metrics via cAdvisor, Docker Engine can expose its own operational metrics, such as running container counts or daemon operation latencies, using the metrics-addr setting in daemon.json. According to official Docker documentation, this feature remains under active development and metric names may change, making this endpoint better suited for monitoring Docker daemon health rather than replacing cAdvisor for per-container metrics.

{
  "metrics-addr": "127.0.0.1:9323"
}

sudo systemctl restart docker
curl http://127.0.0.1:9323/metrics

Restrict metrics-addr to loopback addresses like 127.0.0.1 unless access from external hosts is strictly required, as this endpoint lacks built-in authentication; binding it to 0.0.0.0 allows anyone reachable on that network port to read Docker daemon operational data.

25.2.3 Container Events for Real-Time Observability

Beyond numerical metrics, Docker emits events whenever state changes occur on containers, images, networks, or volumes, such as when containers are created, started, stopped, or experience health status updates. The docker events command streams these events in real-time, helping identify patterns like containers continuously restarting within short intervals.

docker events --filter type=container

More specific filters can target individual event types, such as capturing health state changes for containers with active health checks.

docker events --filter event=health_status

To review past events within a specific timeframe rather than relying solely on real-time streams, the --since and --until flags restrict the output window.

docker events --since '2026-09-12T00:00:00' --until '2026-09-12T23:59:59'

In production environments, docker events is most useful when combined with external tools that listen to the event stream and forward alerts to platforms like Slack, alerting DevOps Engineers to failed or unhealthy containers without requiring constant manual terminal observation.

25.3 Health Checks

An Up status in docker ps simply indicates that a container's main process is running, not that the application inside is capable of serving requests. A web server hung due to a deadlock, for example, still displays as Up despite failing to respond to incoming traffic. Health checks bridge this gap by running periodic verification commands inside the container to confirm the application is operational, not just that its process exists.

25.3.1 HEALTHCHECK in Dockerfile

The HEALTHCHECK instruction in a Dockerfile defines the command Docker runs periodically inside the container to test application health. According to official Docker documentation, the command's exit code determines status: 0 indicates healthy, 1 indicates unhealthy, and 2 is reserved and must not be used.

HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
  CMD curl -f http://localhost/health || exit 1

The four options in the example above have default values if omitted: --interval defaults to 30s, --timeout to 30s, --start-period to 0s, and --retries to 3. The --start-period provides initialization time before health check failures count toward --retries, which is essential for applications requiring warm-up time upon startup so they are not flagged as unhealthy during normal initialization. During this start period, Docker Engine 25.0 and later supports the --start-interval option, which sets a custom check interval specifically for startup, independent of the regular --interval used after initialization completes.

Containers configured with HEALTHCHECK display additional status details under the STATUS column in docker ps, progressing from starting during the start period to healthy upon successful checks, or unhealthy if failures reach the specified --retries count.

docker ps --format "table {{.Names}}\t{{.Status}}"

Detailed health check history, including recent command output snippets, can be inspected using docker inspect.

docker inspect --format '{{json .State.Health}}' payment-api

If a base image contains an inherited HEALTHCHECK that is irrelevant to the built application, the HEALTHCHECK NONE instruction disables it.

HEALTHCHECK NONE

25.3.2 Health Checks in Docker Compose

Health checks can also be configured directly in docker-compose.yml using the healthcheck key per service, allowing check adjustments without rebuilding images.

services:
  payment-api:
    image: payment-service:1.4.2
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 10s

The test field accepts array syntax where the first element is CMD, CMD-SHELL, or NONE. The CMD form executes commands directly without a shell, while CMD-SHELL runs the command through the container's default shell, supporting operators like pipes and redirects.

healthcheck:
  test: ["CMD-SHELL", "curl -f http://localhost/health || exit 1"]

The primary power of healthcheck in Compose emerges when paired with the expanded depends_on syntax using the service_healthy condition, forcing dependent services to wait until underlying dependencies become fully healthy, rather than just waiting for containers to start.

services:
  payment-api:
    image: payment-service:1.4.2
    depends_on:
      db:
        condition: service_healthy

  db:
    image: postgres:16
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5

Without the service_healthy condition, payment-api might attempt startup immediately after the db container is created, before PostgreSQL completes initialization and accepts connections, causing initial connection attempts to fail. Verify startup order execution using docker compose ps, which reflects healthy or starting states for services with health checks configured.

docker compose ps

25.3.3 Health Checks and Automated Restarts

A common misconception is that Docker automatically restarts containers marked as unhealthy. In reality, standalone Docker Engine only records and emits health updates via health_status events; it takes no action against failing standalone containers launched via docker run, even when configured with --restart policies, because restart policies evaluate main process exit codes rather than unhealthy status.

To automatically restart unhealthy containers, an auxiliary mechanism must monitor health events and act accordingly. A common solution for standalone containers or Compose stacks is an external tool like autoheal, which listens to health_status events via the Docker API and executes docker restart when a container is flagged unhealthy.

docker run -d \
  --name autoheal \
  --restart=always \
  -e AUTOHEAL_CONTAINER_LABEL=all \
  -v /var/run/docker.sock:/var/run/docker.sock \
  willfarrell/autoheal

The AUTOHEAL_CONTAINER_LABEL=all environment variable instructs autoheal to monitor all containers with a HEALTHCHECK on the host. Because autoheal requires access to docker.sock to listen to events and issue restarts, mount this socket with caution, as socket access provides elevated administrative capabilities over the Docker daemon. Avoid deploying autoheal on hosts where containers might be managed by untrusted parties. For workloads managed under orchestrators like Swarm or Kubernetes, self-healing is natively integrated, automatically rescheduling unhealthy containers or pods without requiring third-party tools.

25.4 Overview of Container Orchestration

All previously discussed practices, including logging, monitoring, and health checks, require manual overhead per container when executing workloads via standalone docker run commands on a single host. Container orchestration automates cluster-wide container management: scheduling containers onto available nodes, maintaining target replica counts, replacing unhealthy containers automatically, and load balancing traffic across operational instances.

25.4.1 Why Orchestration is Needed

Managing containers manually is manageable when applications fit on a single server. Complexity increases as scaling requirements grow: applications must scale across multiple instances to absorb traffic, instances need distribution across host machines so single-server failures do not cause total outages, and crashed or unhealthy instances must be replaced immediately without waiting for Sysadmins to manually log in and re-run deployment commands. Container orchestration addresses these challenges through core capabilities shared across modern platforms: automated scheduling based on resource requests, self-healing that replaces failed or unhealthy containers, traffic load balancing, service discovery enabling containers to locate each other dynamically, and rolling updates that deploy application updates incrementally with zero downtime.

25.4.2 Docker Swarm and Kubernetes

Docker Swarm is a native orchestration mode built directly into Docker Engine, activated via a single command without installing supplementary components.

docker swarm init

Swarm's primary advantage lies in its simplicity; teams familiar with docker and docker compose commands will find Swarm concepts and syntax intuitive, as Swarm service definitions leverage standard Compose file formats via docker stack deploy. Kubernetes, by contrast, is a broader orchestration platform originally created by Google and now maintained by the Cloud Native Computing Foundation (CNCF). It introduces distinct abstractions such as Pods, Deployments, and Services, backed by a more complex control plane architecture that provides extensive flexibility for large-scale production requirements.

Choosing between the two depends on team expertise and operational scale. Smaller teams or projects already standardized on the Docker ecosystem often adopt Swarm quickly due to its low learning curve, whereas organizations with complex scaling needs, broad ecosystem tooling requirements, or managed Kubernetes offerings from cloud providers typically select Kubernetes despite its steeper learning curve. Both platforms run standard Docker-based container images under the hood, ensuring built images and Dockerfiles remain portable regardless of the chosen production orchestration platform.