Operations & Maintenance

Containers running stably in production are not the end of a Sysadmin/DevOps Engineer's job; in fact, that is precisely where daily operational work begins. Application images need to be updated without disrupting active users, data inside volumes needs to be backed up to prevent loss in case of incidents, CPU and memory usage must be managed so that a single rogue container does not exhaust resources for others, and all activities at the Docker daemon level must be monitored so problems can be detected before turning into major incidents. This chapter covers those daily operation and maintenance practices, ranging from container update strategies, data backup and recovery, resource management, to monitoring the Docker infrastructure itself.

43.1 Zero-Downtime Container Update Strategies

Replacing an image version running in production requires extra care, as a small mistake can render services suddenly inaccessible to users. This sub-chapter discusses how to update containers on a single standard Docker host, outside of orchestrators like Docker Swarm or Kubernetes that already feature built-in rolling update mechanisms.

43.1.1 Updating Images with Docker Compose

For applications managed via docker compose, the most common update workflow involves pulling the new image and then instructing Compose to recreate the containers using it. First, prepare the following simple docker-compose.yml example as practice material.

services:
  web:
    image: nginx:1.27-alpine
    ports:
      - "8080:80"
    restart: unless-stopped

Run this container first, then simulate the update process to a new version tag.

docker compose up -d
docker compose pull
docker compose up -d

The docker compose pull command fetches the latest image corresponding to the tag specified in docker-compose.yml, while the second docker compose up -d compares the running container configuration with the newly pulled image; if there are differences, Compose stops the old container and creates a new one from that image. This process still causes brief downtime, as the old container is stopped before the new container is ready to accept traffic. For services that strictly cannot tolerate any downtime, a blue-green strategy or rolling update via orchestrators like Docker Swarm becomes the more appropriate choice.

43.1.2 Rolling Back to a Previous Version

A failed update, such as a new container crashing immediately or its HEALTHCHECK continuously reporting unhealthy, must be reverted as quickly as possible. The key is to always use specific, non-overwritten image tags rather than the latest tag (whose content can change at any time), allowing a rollback by simply pointing back to the older tag still stored in the system.

docker inspect --format='{{.Config.Image}}' web-app
docker compose stop web
docker run -d --name web-rollback -p 8080:80 registry.example.com/web-app:1.3.1

When using docker-compose.yml, a cleaner method is changing the tag value under image: back to the previous version in the configuration file, then re-running docker compose up -d so that the entire service definition remains consistent with what is written in the file, rather than running a single container manually via docker run. In practice, disciplined teams store image tag histories in version control systems (via committed docker-compose.yml files) so rollbacks can be clearly tracked by who changed the version and when, rather than relying on memory.

43.1.3 Simple Blue-Green Deployment with a Reverse Proxy

Blue-green deployment runs two sets of containers simultaneously, the old version (blue) and the new version (green), and switches traffic from blue to green only after green is proven healthy. This pattern avoids downtime because the old container continues serving traffic until the new container is fully ready. On a single Docker host without an orchestrator, a reverse proxy like Nginx can act as the traffic router.

docker run -d --name app-blue --network app-net registry.example.com/app:1.3.1
docker run -d --name app-green --network app-net registry.example.com/app:1.4.0
docker exec app-green wget -qO- http://localhost:3000/health

Once the app-green container is confirmed healthy via its health endpoint check, update the upstream configuration in Nginx to point to app-green, and reload its configuration without interrupting active connections.

docker exec nginx-proxy nginx -s reload

The app-blue container is stopped and removed only after traffic has completely shifted and no issues arise during monitoring, ensuring that rolling back to app-blue remains possible without re-deploying from scratch if the new version encounters problems.

docker rm -f app-blue

43.2 Container Data Backup and Recovery

Containers themselves are disposable and can be removed at any time, but data stored in volumes or inside databases is often the most valuable asset that must not be lost. Sysadmins and DevOps Engineers must have tested backup procedures, rather than assuming data is safe simply because "no issues have occurred before."

43.2.1 Backing Up Volumes to Tar Archives

The most portable way to back up the contents of a named volume is running a temporary container that mounts both the target volume and the host directory storing the backup output, then archiving it using tar. Create a sample volume and populate a file inside it for practice.

docker volume create app-data
docker run --rm -v app-data:/data alpine sh -c "echo 'sample data' > /data/notes.txt"

Run the backup using a temporary Alpine container that is automatically removed upon completion (--rm), using a minimal image to ensure fast backup execution without adding unnecessary resource load.

docker run --rm -v app-data:/data -v "$(pwd)":/backup alpine \
  tar czf /backup/app-data-backup.tar.gz -C /data .

The command above archives the full contents of the /data directory (which is mapped to the app-data volume) into the app-data-backup.tar.gz file in the host working directory. The -C /data option changes the directory inside tar before archiving, ensuring paths inside the archive remain relative without dragging along absolute /data directory structures. Verify the archive contents without extracting it first.

tar tzf app-data-backup.tar.gz

43.2.2 Restoring Volumes from Backups

The restoration process follows a symmetric pattern to backups: create a target volume, then extract the archive into it using the same temporary container approach.

docker volume create app-data-restored
docker run --rm -v app-data-restored:/data -v "$(pwd)":/backup alpine \
  tar xzf /backup/app-data-backup.tar.gz -C /data

Ensure the target volume is empty prior to restoration if you want to avoid old files being mixed with extracted files. Verify the restore results by inspecting the contents of the new volume.

docker run --rm -v app-data-restored:/data alpine cat /data/notes.txt

In production environments, a backup that has never been test-restored is equivalent to having no backup at all, as issues like corrupted archives or incorrect permissions are only discovered when an actual restore is needed, usually during critical moments. Schedule regular restore drills instead of just running backups and assuming success.

43.2.3 Backing Up Databases Inside Containers

For databases like PostgreSQL or MySQL/MariaDB running inside containers, file-level volume backups risk producing inconsistent data if the database is actively writing transactions while files are snapshotted. A safer approach is using official database dump tools via docker exec, operating at the logical data level rather than making raw file copies.

docker exec db-postgres pg_dump -U postgres appdb > appdb-backup.sql

This command executes pg_dump inside the db-postgres container and streams the output to the appdb-backup.sql file on the host via standard shell redirection. The -t flag is intentionally omitted, as allocating a pseudo-TTY while output is redirected to a file risks injecting extra carriage return characters that corrupt dump contents. For MySQL/MariaDB, the pattern is similar using mysqldump.

docker exec db-mysql mysqldump -u root -p appdb > appdb-backup.sql

Restoring data from a dump file is performed in reverse, streaming file contents via stdin into the database container using docker exec -i (interactive mode without TTY allocation, preserving raw binary/text streams intact).

cat appdb-backup.sql | docker exec -i db-postgres psql -U postgres -d appdb

Commands like psql, mysqldump, and similar utilities can overwrite existing data in the target database without additional prompt confirmations, so always verify that the target restore database is correct before running these commands, especially when operating against production databases.

43.3 Resource Management in Production

Containers without resource constraints can consume all available CPU or memory on the host, causing neighboring containers on the same machine to suffer even if logically unrelated. Maintaining explicit resource limits is a core operational responsibility for containerized production environments.

43.3.1 Restricting CPU and Memory

Resource limits are defined via flags when running containers, or via the deploy.resources key in docker-compose.yml for modern Compose versions.

docker run -d --name app-limited \
  --cpus="1.5" --memory="512m" --memory-swap="512m" \
  nginx:1.27-alpine

The --cpus option limits CPU allocation in core count units (a value of 1.5 means the container can consume up to the equivalent of 1.5 cores), while --memory caps the maximum RAM usable. Setting --memory-swap equal to --memory disables additional swap space, ensuring the container terminates directly if it exceeds memory limits instead of silently slowing down on swap space. If a container exceeds its --memory limit without available swap space, the Linux kernel terminates processes inside it via the OOM killer (Out of Memory killer), which is logged in the host dmesg as well as in the container exit code associated with OOM.

docker inspect --format='{{.State.OOMKilled}}' app-limited

The equivalent configuration in docker-compose.yml is written under the deploy.resources.limits key.

services:
  app:
    image: nginx:1.27-alpine
    deploy:
      resources:
        limits:
          cpus: "1.5"
          memory: 512M

Note that the deploy block was originally designed for Docker Swarm; when executed using standard docker compose up (rather than docker stack deploy), only select sub-keys including resources.limits are honored, while other sub-keys under deploy such as replicas are ignored outside Swarm mode.

43.3.2 Updating Resource Limits for Running Containers

Resource limits do not need to be fixed at container creation time; docker update modifies CPU and memory limits on active running containers without requiring them to be stopped or recreated.

docker update --cpus="2" --memory="1g" --memory-swap="1g" app-limited

This feature is practical when Sysadmins or DevOps Engineers need to scale resource allocations dynamically during emergencies, such as sudden traffic spikes, without disrupting running container processes. Be mindful of risks when reducing memory limits using docker update: if the new value falls below the current active memory consumption of the container, the OOM killer may immediately trigger upon limit application. Always check actual usage via docker stats before lowering limits.

docker stats app-limited --no-stream

The --no-stream flag outputs a single data snapshot and exits immediately, making it ideal for automation scripts compared to default live streaming mode.

43.3.3 Restart Policies and Automatic Recovery

A restart policy defines Docker daemon behavior when a container exits, whether due to a crash or because the Docker daemon itself restarted. Choosing the right policy directly impacts how quickly services recover from failures without manual intervention.

docker run -d --name app-resilient --restart unless-stopped nginx:1.27-alpine

In production environments, unless-stopped is the most popular choice for single-host Docker services managed manually, as it provides full control when intentionally stopping containers for maintenance without risks of containers spontaneously starting up after host reboots. The on-failure policy is better suited for one-off jobs or tasks that should exit cleanly upon success, but require retries if failing from transient errors. Update restart policies on running containers using docker update without recreating containers.

docker update --restart on-failure:5 app-resilient

43.4 Docker Infrastructure Maintenance Routines

Knowing update, backup, and resource limiting commands is insufficient if every step is handled manually when needed. This sub-chapter combines these practices into automated operational routines, while tackling two frequently overlooked pitfalls: transient Docker event histories and log rotation policies that fail to retroactively apply to existing containers.

43.4.1 Persisting Docker Events History

The docker events command streams real-time event logs from the Docker daemon whenever container, image, volume, or network states change. However, the Docker daemon only maintains event history in an in-memory buffer capped at the last 1000 entries and never writes them to disk; once the daemon restarts or the buffer fills up, older uncaptured events are lost permanently. For audit trail requirements, stream docker events output to a log file on the host via a continuous background process.

nohup docker events --format json >> /var/log/docker-events.log 2>&1 &

The command above runs docker events in the background via nohup, appending JSON-formatted events to /var/log/docker-events.log, keeping the process running even after terminating the shell session. For production hosts requiring persistence across reboots, wrap the command in a systemd service to ensure automatic restart handling independent of Docker container restart policies.

[Unit]
Description=Docker Events Logger
After=docker.service
Requires=docker.service

[Service]
ExecStart=/usr/bin/docker events --format json
StandardOutput=append:/var/log/docker-events.log
Restart=always

[Install]
WantedBy=multi-user.target

Save the unit file above as /etc/systemd/system/docker-events-logger.service (applicable to Linux distributions using systemd; Docker Desktop on macOS/Windows does not use equivalent systemd mechanisms), then enable and start it.

sudo systemctl enable --now docker-events-logger.service
sudo systemctl status docker-events-logger.service

With this service running, /var/log/docker-events.log serves as a persistent history source no longer limited by the 1000-entry daemon buffer, making it searchable via grep or forwardable to centralized observability stacks during incident investigations. Note that this log file size must still be managed via host OS tools like logrotate, as it is not automatically bounded like container logs managed by the json-file driver.

43.4.2 Scheduled Maintenance Routines via Cron

Volume backups and unused resource pruning are most effective when automated periodically rather than relying on manual execution. Combine backup and maintenance tasks into a single shell script scheduled using cron.

#!/bin/sh
set -e
BACKUP_DIR="/backup/$(date +%Y%m%d)"
mkdir -p "$BACKUP_DIR"

docker run --rm -v app-data:/data -v "$BACKUP_DIR":/backup alpine \
  tar czf /backup/app-data.tar.gz -C /data .

docker exec db-postgres pg_dump -U postgres appdb > "$BACKUP_DIR/appdb.sql"

docker system prune -f
find /backup -maxdepth 1 -type d -mtime +30 -exec rm -rf {} \;

Save the script above as /usr/local/bin/docker-maintenance.sh, grant execution permissions, and schedule it via crontab to execute automatically during off-peak hours.

chmod +x /usr/local/bin/docker-maintenance.sh
crontab -e

0 3 * * * /usr/local/bin/docker-maintenance.sh >> /var/log/docker-maintenance.log 2>&1

The find ... -mtime +30 line in the script cleans up backup directories older than 30 days to prevent backup stores from consuming host disk space indefinitely; adjust retention windows according to organizational data retention policies. Redirecting cron output to a dedicated log file as shown ensures script failures (such as missing volumes or changed database credentials) are captured for inspection rather than failing silently.

The docker system prune -f command inside the script is destructive toward stopped containers, unused networks, and dangling images (untagged images not used by any container), though by default it does not touch volumes. Manually review potential impact using docker system df -v, which summarizes disk consumption per Docker resource category alongside a RECLAIMABLE column indicating reclaimable space, prior to adding --volumes flags into automated scripts to prevent essential volumes from accidental removal.

43.4.3 Applying Log Rotation to Existing Containers

Configuring log rotation via log-opts in daemon.json applies only as default values for containers created after the configuration takes effect. Previously running containers retain their original logging configurations, as log-driver and log-opt settings are evaluated once at container creation time and cannot be updated dynamically via docker update. Inspect active logging configurations on target containers to verify whether rotation is active.

docker inspect --format='{{json .HostConfig.LogConfig}}' app-lama

If the result displays an empty Config block or lacks the max-size key, the container continues using unconstrained default logging despite host daemon.json reconfigurations. The sole method to apply updated log rotation options to existing containers is recreating them with explicit --log-opt flags, following update patterns outlined earlier in this chapter, rather than executing standard container restarts.

docker stop app-lama
docker run -d --name app-lama \
  --log-opt max-size=10m --log-opt max-file=3 \
  registry.example.com/app-lama:1.0

For json-file log files that expanded excessively prior to rotation enforcement, Docker provides no native commands to truncate active files. Zeroing out the file at the OS level serves as the standard emergency mitigation using truncate rather than rm.

sudo truncate -s 0 $(docker inspect --format='{{.LogPath}}' app-lama)

truncate -s 0 empties file contents without unlinking the file descriptor, preserving container logging process file descriptors without disruption. Removing log files directly via rm can cause container logging processes to lose file references while continuing to write to deleted filesystem space that remains unreleased until container processes terminate, a scenario where df continues reporting full disks despite missing log files on disk.