Docker Compose and Multi-Container Applications

Chapter 26 closed with a rather inconvenient scenario for Sysadmins: running the web-test and db-demo containers using two separate docker run commands, and then connecting them manually via docker network create app-net. This approach is tolerable for two experimental containers. However, developers in the field rarely build applications using only a single service. A realistic production application typically consists of a web server, a database, and an administration tool simultaneously, each with different ports, volumes, and environment variables. Rewriting all those docker run flags every time the server needs to be restarted is not only exhausting, but also prone to typos that lead to inconsistent configurations between one deployment and the next. Chapter 27 resolves this problem through Docker Compose, the official tool from Docker for defining an entire multi-container application within a single declarative YAML file. This chapter begins with the concepts and structure of the compose.yaml file, builds a real three-service stack containing an Nginx web server, a PostgreSQL 18 database, and Adminer as its web administration UI, manages environment variables using a .env file, and concludes with restart policies and basic scaling to keep services running and duplicated according to load requirements.

27.1 Docker Compose Concepts and compose.yaml File Structure

Before writing the first configuration file, we need to understand the actual problem solved by Docker Compose, as well as the three main sections that make up a compose file.

27.1.1 From Repeated docker run Commands to a Single Declarative File

Docker Compose is an orchestration tool for single-host environments that reads a single YAML file containing definitions for all services, networks, and volumes of an application, and then aligns the state of the actually running containers with those definitions using just a single command. Instead of retyping -p, -v, -e, and --network flags every time a container needs to be run as in Section 26.5.3, we simply write them once inside the file, and then execute docker compose up whenever needed. This approach is called declarative: we describe the desired end state, rather than the sequence of commands that must be executed one by one.

Docker Compose is not a separate application that needs to be installed manually on Ubuntu Server 26.04 LTS. Since Section 26.2.1, Docker's official installer script automatically installs the docker-compose-plugin package as part of the Docker Engine installation, introducing the docker compose command (two words, without a hyphen) as a built-in CLI plugin for Docker. This differs from the older generation docker-compose (one word with a hyphen), which was a separate Python application and has been declared deprecated by Docker since Compose V2 was released.

Hands-on Steps

• Confirm that the Compose plugin is active as part of the Docker CLI installed since Chapter 26.

  docker compose version

Verification and Troubleshooting

• A healthy output starts with Docker Compose version v2.x.x followed by a build hash, indicating that the plugin is recognized as part of the Docker CLI rather than a separate command.
• If the error docker: 'compose' is not a docker command appears, the docker-compose-plugin package is not installed. Install it manually using sudo apt install docker-compose-plugin, or rerun the official installer script from Section 26.2.1.

27.1.2 Anatomy of compose.yaml: services, networks, and volumes

A compose file is basically composed of three main top-level blocks.

services:
  service_name_1:
    image: image_name:tag
    ports:
      - "host_port:container_port"
  service_name_2:
    image: another_image_name:tag

volumes:
  volume_name:

The services block is the mandatory section that contains the most information. Each key beneath it, such as service_name_1 in the example above, becomes a single container when executed, complete with its own image, ports, volumes, environment, and restart policy. The top-level volumes block serves to declare named volumes managed by Docker, equivalent to the docker volume create command that we practiced manually in Section 26.5.1, except now it only needs to be declared once inside the file.

The third section, networks, often does not need to be written at all. When docker compose up is executed, Compose automatically creates a dedicated bridge network for that project, and connects all services to it without needing to run docker network create app-net manually as in Section 26.5.3. Each service can automatically reach others using its own service name as a hostname, powered by Docker's built-in embedded DNS server, identical to what we discussed previously. The networks block only needs to be explicitly written if we require more than one isolated network within a single application, for example, separating a public frontend network from a backend database network, a pattern that will be more relevant when we discuss network hardening in Part VIII.

There are two important notes regarding naming before we start writing an actual file. First, the version field that previously had to be written on the very top line of the file (e.g., version: "3.8") is now deprecated since Docker fully adopted the Compose Specification. This field can be omitted entirely because modern Compose always validates files using the latest schema, regardless of the version value written. Second, regarding the filename itself: the Compose CLI searches for compose.yaml first as the standard recommended name, followed by compose.yml, docker-compose.yaml, and docker-compose.yml as legacy names for backward compatibility. This series uses compose.yaml as the filename across all practice examples, but docker-compose.yml, which might be more familiar from older tutorials, remains fully supported and works identically.

27.2 Running a Multi-Service Application: Web, Database, and Adminer

This section provides hands-on practice with a three-service stack: an Nginx web server, a PostgreSQL 18 database as discussed in Chapter 18, and Adminer as a containerized database management web UI, serving as a comparison to pgAdmin 4 installed natively in Chapter 22.

27.2.1 Setting Up the Project Structure

Hands-on Steps

1. Create the project directory alongside the static web content folder.

   mkdir -p ~/app-compose/html
   cd ~/app-compose

2. Create a simple HTML page as the web content.

   echo "<h1>Sample application running via Docker Compose</h1>" > html/index.html

27.2.2 Writing a Complete compose.yaml for Three Services

Hands-on Steps

1. Create a .env file containing the database credential values and ports to be used. Full details regarding this file are discussed in Section 27.3; for now, simply understand that this file holds values not directly hardcoded into compose.yaml.

   nano .env

   POSTGRES_USER=notes_user
   POSTGRES_PASSWORD=change_this_password
   POSTGRES_DB=notes_db
   WEB_PORT=8090
   ADMINER_PORT=8091

2. Create compose.yaml in the same directory.

   nano compose.yaml

   services:
     web:
       image: nginx:stable
       restart: unless-stopped
       ports:
         - "${WEB_PORT}:80"
       volumes:
         - ./html:/usr/share/nginx/html:ro
   
     db:
       image: postgres:18
       restart: unless-stopped
       environment:
         POSTGRES_USER: ${POSTGRES_USER}
         POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
         POSTGRES_DB: ${POSTGRES_DB}
       volumes:
         - db-data:/var/lib/postgresql/data
       healthcheck:
         test: ["CMD-SHELL", "pg_isready -U notes_user -d notes_db"]
         interval: 5s
         timeout: 5s
         retries: 5
   
     adminer:
       image: adminer
       restart: unless-stopped
       ports:
         - "${ADMINER_PORT}:8080"
       depends_on:
         db:
           condition: service_healthy
   
   volumes:
     db-data:

   Here are several key sections to understand. The web service uses volumes in the form of a bind mount as in Section 26.5.2, mapping the host's ./html folder to Nginx's document root, appended with :ro (read-only) so the container cannot write back to the host folder. The db service uses a healthcheck based on pg_isready, a built-in PostgreSQL utility used to check whether the database server is truly ready to accept connections, rather than merely checking if the postgres process is running. The adminer service uses depends_on with condition: service_healthy, meaning the Adminer container will only start after the db service's health check reports a healthy status, rather than just waiting for the db container to reach a running state, which could occur before PostgreSQL finishes initializing.

27.2.3 Running the Stack with docker compose up

Hands-on Steps

1. Run the entire stack in the background from the project directory.

   docker compose up -d

   Compose creates a new bridge network named app-compose_default (using the project directory name as the project name), a volume named app-compose_db-data, and then starts all three containers sequentially according to their dependencies: db first, followed by web which has no dependencies, and adminer last after db reaches a healthy status. Note that the containers themselves are named using hyphens, such as app-compose-db-1, unlike networks and volumes which use underscores as shown above. This is not a typo, but rather a deliberate resource-naming convention internal to Compose: containers follow the project-service-index format, whereas networks and volumes follow the project_name format.

Verification and Troubleshooting

• View the status of all three containers simultaneously, including the health check column.

  docker compose ps

• Access the static web page that was just started.

  curl http://localhost:8090

  The output should display the line <h1>Sample application running via Docker Compose</h1> that we created in Section 27.2.1.
• Open http://server-ip-address:8091 in a browser to log in to Adminer. Fill out the login form using System PostgreSQL, Server db (the service name, not localhost or an IP address, because both are connected via embedded DNS on the same Compose network), Username notes_user, Password matching the .env file content, and Database notes_db. A successful login will display the Adminer administration page with an empty notes_db database.
• If port 8090 or 8091 is already in use by another process, change its value in .env and rerun docker compose up -d. If UFW is active as discussed later in Chapter 31, ensure both ports are allowed before attempting to access them from another computer.
• If adminer under docker compose ps stays stuck for a long time in the STATUS column without becoming Started, check first whether db has become healthy.

  docker compose logs db

  The log line database system is ready to accept connections indicates that PostgreSQL is fully ready. If this line has not appeared yet, the health check will continue holding adminer back from starting, according to the design of depends_on in Section 27.2.2.

27.2.4 Everyday Operational Commands

The following commands will be frequently used by Sysadmins after the stack is running, offering far greater convenience than managing each container individually as done in Chapter 26.

Verification and Troubleshooting

• Follow logs for the db service directly, which is useful when debugging failed queries or connections from Adminer.

  docker compose logs -f db

  Press Ctrl+C to exit follow mode without stopping the container itself.
• A candid note on risk: the -v flag in docker compose down -v permanently deletes the db-data volume along with all database contents inside it. Never run this command on a production server without taking a backup first, following the same principles as the database backup strategy in Chapter 23.

27.3 Environment Variables and the .env File

This section breaks down two distinct ways Compose utilizes .env files, a distinction that frequently causes confusion because both methods use files with identical names but operate at entirely different stages.

27.3.1 Value Interpolation from .env to compose.yaml

Compose automatically reads a .env file located in the same directory as compose.yaml, and replaces every ${VARIABLE_NAME} placeholder inside that file with its actual value. This process is called interpolation, and it occurs purely at the Compose CLI level while reading the file, long before a single container is actually created. This pattern was used for ${WEB_PORT}, ${ADMINER_PORT}, and the three PostgreSQL variables in Section 27.2.2.

Hands-on Steps

1. View the final compiled compose.yaml after all interpolation processes finish, without actually executing anything.

   docker compose config

Verification and Troubleshooting

• The output of docker compose config displays the complete YAML file with every ${...} placeholder replaced by its actual value, including database passwords in plain text as-is. This command is very useful for debugging when a variable is not being read as expected, because we can directly view the final value sent to the Docker Engine.
• If a ${VARIABLE} appears empty in the output of docker compose config, it is highly probable that the variable name in .env has a typo, or the command was executed from a directory different from the location of the .env file itself, because Compose only looks for it in the current working directory.

27.3.2 env_file to Pass Variables Directly to Containers

The second method serves a completely different purpose. Instead of inserting values into compose.yaml via interpolation, the env_file directive passes all lines inside a file directly as environment variables inside the container process, exactly like the -e flag in docker run. Since the three variables POSTGRES_USER, POSTGRES_PASSWORD, and POSTGRES_DB in our .env happen to use the exact names recognized by the official postgres image, the environment block in the db service can actually be simplified.

Hands-on Steps

1. Modify the db service in compose.yaml, replacing the environment block with env_file.

     db:
       image: postgres:18
       restart: unless-stopped
       env_file: .env
       volumes:
         - db-data:/var/lib/postgresql/data
       healthcheck:
         test: ["CMD-SHELL", "pg_isready -U notes_user -d notes_db"]
         interval: 5s
         timeout: 5s
         retries: 5

2. Apply the changes. Compose automatically detects that only the db definition changed, so only that container is recreated, while web and adminer remain untouched.

   docker compose up -d

Verification and Troubleshooting

• Inspect the environment variables actually received by the db container.

  docker compose exec db env | grep POSTGRES

  All three variables should still appear with the exact same values as before, even though they are now passed via env_file instead of through ${...} interpolation.
• Field observation note: env_file sends all lines inside the file, including WEB_PORT and ADMINER_PORT which are completely irrelevant to the database container. This slightly litters the container environment compared to the explicit environment approach in Section 27.2.2, which only passes required variables. For applications with many mixed sensitive and non-sensitive variables, consider splitting configurations into multiple service-specific .env files.
• If both environment and env_file directives are used together on a single service with matching variable names, values from environment always take precedence, adhering to Compose's official priority order.

27.3.3 Securing .env from Being Committed to Git

The .env file stores passwords in plain text, so it must be handled with the same care as other credential files protected throughout this series.

Hands-on Steps

1. Restrict permissions on the .env file so it can only be read by its owner.

   chmod 600 .env

2. If this project directory is later managed via Git, ensure .env is never committed.

   echo ".env" >> .gitignore

3. In its place, create a .env.example file containing variable names without actual values, serving as documentation for other developers on the team who need to know which variables are required, without leaking real credentials.

   cat > .env.example << 'EOF'
   POSTGRES_USER=
   POSTGRES_PASSWORD=
   POSTGRES_DB=
   WEB_PORT=
   ADMINER_PORT=
   EOF

   The .env.example file is safe to commit to Git, while the actual .env file remains stored locally on each respective server.

Verification and Troubleshooting

• A candid note on security: whether using .env, environment, or env_file, all three still store and transmit values in plain text, which are visible via docker compose config or docker inspect to anyone with shell access to the server. For truly sensitive production needs, such as third-party API keys or production database passwords, consider a more mature secret management mechanism, such as Docker secrets in Docker Swarm or external tools like HashiCorp Vault, which fall outside the scope of this chapter.

27.4 Restart Policies and Safe Startup Order

Production servers will sooner or later undergo a restart, whether scheduled or unexpected. This section ensures our three services recover automatically and in the correct sequence whenever that happens.

27.4.1 Four Restart Policy Options

The restart directive at the service level determines Docker Engine behavior when a container stops, whether due to a crash or other reasons.

Our three services in Section 27.2.2 used unless-stopped, which is the most common choice for long-running services like web servers and databases. This policy ensures containers start up again after dockerd or the entire host reboots, while respecting Sysadmin decisions when a service is intentionally stopped via docker compose stop for maintenance.

Hands-on Steps

1. Simulate a sudden crash of the db container by sending a force kill signal.

   docker kill app-compose-db-1

2. Wait a few seconds, then check its status again.

   docker compose ps

3. For comparison, stop db officially via Compose, rather than using kill.

   docker compose stop db

Verification and Troubleshooting

• After step 2, db should return to a running status without requiring any docker compose up command, because dockerd itself monitors and applies the restart policy as soon as it detects an unexpected container exit.
• After running docker compose stop db in step 3, run docker compose ps again: this time db remains in a stopped state and will not automatically recover, because unless-stopped intentionally excludes manual stops from automatic restart mechanisms. Start it again using docker compose start db.
• An important note that is frequently misunderstood: neither always nor unless-stopped prevents docker compose down from removing containers. Both restart policies only govern behavior after crashes or host reboots; they do not protect containers from explicit removal commands.

27.4.2 Safe Startup Order with Health Checks and depends_on

This section was actually practiced back in Section 27.2.2 using a combination of healthcheck on db and depends_on: condition: service_healthy on adminer. Without this combination, a basic version of depends_on only guarantees the sequence in which containers are created, not whether the internal applications are actually ready. This is a classic pitfall causing dependent services to occasionally fail connecting on initial attempts because the database is still in its initialization phase.

Hands-on Steps

1. Tear down the entire stack and rerun it from a clean state to observe the startup sequence directly.

   docker compose down
   docker compose up -d

2. Immediately after the command above completes, check the raw health status of db.

   docker inspect --format='{{json .State.Health.Status}}' app-compose-db-1

Verification and Troubleshooting

• The returned value transitions from "starting" to "healthy" within a few seconds, matching the interval and retries parameters defined in Section 27.2.2. The adminer container enters a running state only after this value becomes "healthy".
• For applications with startup times much longer than PostgreSQL, such as Java applications requiring tens of seconds to warm up, add the start_period parameter to healthcheck so initial check failures during boot are not immediately counted as true failures.

27.5 Basic Scaling with docker compose up --scale

This closing section demonstrates running multiple instances of a single service simultaneously, which is a common requirement when a single Nginx container can no longer handle incoming traffic.

27.5.1 Running Multiple Service Instances Simultaneously

Hands-on Steps

1. Try scaling the web service to three instances using the existing compose.yaml configuration from Section 27.2.2.

   docker compose up -d --scale web=3

   This command fails with an error similar to Bind for 0.0.0.0:8090 failed: port is already allocated. The cause is that all three instances attempt to bind to the exact same host port 8090 via ports: "${WEB_PORT}:80", whereas a single host port can only be used by one process at a time.
2. Modify the ports section of the web service in compose.yaml, removing the host port side and keeping only the container port.

     web:
       image: nginx:stable
       restart: unless-stopped
       ports:
         - "80"
       volumes:
         - ./html:/usr/share/nginx/html:ro

   By specifying only the container port, Docker Engine assigns a host port automatically from the kernel's ephemeral port range (typically 32768 to 60999 on Linux) for each instance, preventing port collisions between replicas.
3. Repeat the scaling step after applying the changes.

   docker compose up -d --scale web=3

Verification and Troubleshooting

• Inspect the random host ports assigned to each instance.

  docker compose ps

  The PORTS column displays three web rows with distinct host ports.
• Retrieve a specific replica's port programmatically, which is useful for automation scenarios or monitoring scripts.

  docker compose port web 80
  docker compose port --index 2 web 80

  The first command returns the port belonging to the first replica (index 0, default if --index is omitted), while the second command returns the port belonging to the third replica.
• Access one of the replicas using the newly assigned port, for example, if the result returned 32771.

  curl http://localhost:32771

27.5.2 Limitations of Built-in Compose Load Balancing

Scaling using docker compose up --scale purely duplicates the number of running containers without providing a single access entry point that automatically distributes traffic across all replicas. Clients must know precisely which port to target, as demonstrated using docker compose port. This is clearly insufficient for true production scenarios where visitors should access a single domain or IP without needing to know how many replicas exist behind it.

The most common solution in the field is adding a load balancer in front of these replicas, such as using Nginx upstream blocks as discussed in Section 16.2, registering each randomly scaled port as a backend. However, this approach remains cumbersome because random ports change every time the stack is restarted. For automated scaling needs with built-in service discovery, without manually registering ports one by one, dedicated orchestrators like Docker Swarm or Kubernetes are far better suited than plain Docker Compose. A candid note to understand upfront is that Docker Compose is designed for single-host environments like development, testing, or small-to-medium scale applications, not as a replacement for large-scale production orchestrators.

Hands-on Steps

1. Clean up the entire stack before proceeding to Chapter 28, including database volumes to prevent leftover experimental data.

   docker compose down -v

Verification and Troubleshooting

• Confirm that no containers or volumes remain from this project.

  docker compose ps -a
  docker volume ls | grep app-compose

  Both commands should return no output related to the app-compose project.

Up to this point, we can define a complete multi-container application inside a single compose.yaml file, safely manage credentials using .env, ensure services recover automatically using appropriate restart policies, and duplicate services using basic scaling. Docker and Docker Compose solve container packaging and orchestration problems on a single host, but containers still share the underlying host kernel as discussed in Section 26.1. Chapter 28 transitions to a completely different isolation approach using full virtualization with KVM and QEMU, where every virtual machine runs its own dedicated kernel, suitable for workloads requiring stricter security isolation or running operating systems different from the host.