Troubleshooting and Debugging

Containers that fail to start, applications that suddenly become unreachable, or services that unexpectedly slow down are daily occurrences once Docker is used beyond local experimentation. The difference between a swift Sysadmin/DevOps Engineer and one who panics during a production incident is usually not about who remembers Docker commands better, but rather who knows the proper inspection sequence: from container status, log entries, and network configurations, to resource utilization. This chapter discusses diagnostic workflows for four of the most common categories of issues encountered in the field: containers that fail to start, processes inside containers that require direct debugging, inter-container network disruptions, and performance issues caused by resource limits.

40.1 Containers Failing to Start or Immediately Stopping

The most fundamental issue almost every Developer and Sysadmin/DevOps Engineer faces is a container entering the Exited status immediately upon execution, even when the same image previously ran without issue. This section explains how to interpret container exit codes and trace root causes through logs and container inspection results.

40.1.1 Reading Exit Codes and Container Status

Every stopped container leaves an exit code, a numerical value indicating why its main process terminated. According to the official Docker Engine documentation, several exit codes hold specific meanings beyond those returned by the application process itself: 125 indicates that the Docker daemon failed to run the docker run command (for instance, due to invalid flags), 126 means the specified command was found but could not be executed (permission issues or malformed binaries), and 127 signifies that the specified command was not found within the image at all.

First, check the exit code of a stopped container using docker ps -a to inspect its summary status.

docker ps -a --filter "name=web-app"

CONTAINER ID   IMAGE      STATUS                     NAMES
a1b2c3d4e5f6   web-app    Exited (1) 2 minutes ago   web-app

For more precise exit code extraction within scripts or automation, retrieve it directly via docker inspect.

docker inspect web-app --format '{{.State.ExitCode}}'

An exit code of 0 indicates that the container's main process stopped normally (not an error), while other values are generally passed directly from the application's exit code inside the container. Two values that frequently cause confusion because they do not reflect application errors are 137 and 143: both are the result of 128 + signal number, where 137 means the process received SIGKILL (signal 9, typically due to forced termination via docker kill or running out of memory), and 143 means the process received SIGTERM (signal 15, the standard termination signal sent by docker stop).

40.1.2 Tracing Causes via Logs and Inspect

While exit codes offer clues about the issue category, detailed root causes are almost always located within the container logs. Display the full logs of a stopped container by adding the --tail option so the output does not overwhelm the terminal for containers that ran for a long time previously.

docker logs --tail 50 web-app

If logs do not yield sufficient information, such as when a container stops before writing any logs, inspect the detailed configuration and state history of the container using docker inspect, focusing particularly on the State and Config sections.

docker inspect web-app

Pay close attention to the State.Error field, which sometimes contains error messages originating from the Docker daemon itself (such as volume mounting failures), as well as the Config.Cmd and Config.Entrypoint fields to ensure the commands executed by the container match expectations. In practice, the most common reasons for a container entering Exited status immediately are: a main process designed to run once and terminate (rather than a long-running process like a web server), required environment variables that were not set, causing the application to fail during startup, or configuration files mounted via bind mount that cannot be found at the expected path inside the container.

40.2 Debugging Processes Inside Containers

Some issues do not appear in logs and only become visible after directly examining running processes inside the container, such as malformed configuration files or a process that hangs without logging errors. This section explains how to access a running container and how to debug containers based on minimal images that lack a shell entirely.

40.2.1 Accessing a Running Container

The most direct way to inspect conditions inside a running container is by opening an interactive shell session using docker exec.

docker exec -it web-app sh

If the image provides Bash, replace sh with bash for a richer shell experience. Once inside, check fundamental items that frequently cause issues: configuration file contents using cat, active environment variables using env, and running processes using ps aux if the procps package is available in the image.

docker exec web-app env

Compare this environment variable output against expected application values. In practice, simple errors like misspelled variable names or swapped values between environments (development vs. production) are far more common causes of issues than bugs in the application code itself.

40.2.2 Debugging Shell-less Minimal Containers

Minimal images such as distroless or scratch-based images intentionally omit shells and package managers to minimize attack surface and image size, aligning with image security practices covered in a dedicated chapter. Consequently, running docker exec -it CONTAINER sh will fail with an error like OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH: unknown.

The most universal solution for this scenario is running a separate debug container that shares process and network namespaces with the target container, without altering the original image in any way.

docker run -it --rm --pid=container:web-app --network=container:web-app nicolaka/netshoot sh

The command above starts a new container from the nicolaka/netshoot image (a popular community image for troubleshooting containing network and diagnostic tools) that shares PID namespace and network namespace with the web-app container. From inside this debug container, the target container's processes remain visible via ps aux even though the shell runs in a separate container, and observed network traffic (such as via tcpdump or curl) is identical to executing directly inside the target container.

Recent versions of Docker Desktop also offer the docker debug command, which automates similar techniques through a single command complete with a built-in debugging toolbox. However, this feature requires Docker Desktop with an active Docker account subscribed to a Pro, Team, or Business plan, and is unavailable on standalone Docker Engine installations without Docker Desktop. For production Linux servers running standalone Docker Engine without Docker Desktop—the most common setup in the field—the namespace-sharing technique using --pid=container: and --network=container: remains the most portable approach as it does not rely on additional licensing or tools beyond Docker Engine itself.

40.3 Troubleshooting Container Networking

Containers unable to communicate with each other or services unreachable from the host represent a category of issues that frequently causes frustration because symptoms appear simple ("connection refused") while root causes vary widely. This section covers inspection procedures for three prevalent network scenarios: containers failing to interconnect, ports unreachable from the host, and inter-container DNS resolution failures.

40.3.1 Inter-Container Connection Failures

The most common cause of two containers failing to connect is that they do not reside on the same Docker network. First, inspect which networks are attached to each container.

docker inspect web-app --format '{{range $net, $conf := .NetworkSettings.Networks}}{{$net}} {{end}}'

Run the same command for the target container and compare the results. If both containers use Docker's default bridge network, note that this network does not support automatic container name resolution; only user-defined networks (created via docker network create or automatically via Docker Compose) provide inter-container DNS resolution based on service names. To resolve this, attach both containers to the same network.

docker network connect app-network web-app

Once confirmed to be on the same network, verify connectivity by attempting a direct connection from one container to the other using the container name as the host.

docker exec web-app ping -c 3 db

If ping is unavailable in the image, use a temporary debug container from the nicolaka/netshoot image attached to the same network to test connectivity without adding tools to production images.

40.3.2 Ports Unreachable from the Host

If an application inside a container is accessible via docker exec but inaccessible from the host, the most frequent cause is confusion between EXPOSE in the Dockerfile and the -p/--publish option during docker run. EXPOSE serves only as documentation for ports used by the application and facilitates inter-container linking; it does not publish the port to the host. Publishing ports to the host requires explicit configuration using -p.

docker ps --filter "name=web-app" --format "table {{.Names}}\t{{.Ports}}"

NAMES      PORTS
web-app    0.0.0.0:8080->80/tcp

If the PORTS column is completely empty, the container was executed without the -p option, meaning no ports were published to the host. If the column displays 127.0.0.1:8080->80/tcp instead of 0.0.0.0:8080->80/tcp, the port is published but accessible only from the host's localhost, not external networks; this usually results explicitly from -p 127.0.0.1:8080:80. If port publishing is correct but external host access still fails, check OS-level firewall rules on the host (such as ufw or firewalld on Linux), as Docker modifies its own iptables rules for port mapping but can still be blocked by additional firewalls running above it.

40.3.3 DNS Resolution Issues Inside Containers

Docker Engine runs an internal embedded DNS server at address 127.0.0.11 for every user-defined network, responsible for resolving container names or Compose service names to IP addresses within the same network. If a container fails to resolve another container's name despite being on the same network, check /etc/resolv.conf inside the container to ensure the DNS server points to Docker's internal DNS.

docker exec web-app cat /etc/resolv.conf

If the file contents are correct but resolution still fails, test directly using nslookup or dig from inside the container (or via a nicolaka/netshoot debug container if the target image lacks these tools).

docker exec web-app nslookup db

DNS resolution failures for an active container on the same network typically stem from two possibilities: a typo in the requested name (Docker DNS resolution is case-sensitive and relies on exact container names or network aliases), or the target container recently restarted, obtaining a new IP while the client application cached the old DNS result. For the latter case, ensure the application re-resolves DNS whenever opening new connections rather than permanently caching IP resolution results in process memory.

40.4 Debugging Container Performance

Containers that run without errors but feel slow, or unexpectedly terminate without clear error logs from the application, usually point to host resource constraints or resource limits applied to the container itself. This section discusses real-time resource monitoring, diagnosing containers killed by out-of-memory events, and tracing disk I/O bottlenecks.

40.4.1 Monitoring Resource Usage with docker stats

The docker stats command displays CPU, memory, network, and disk I/O usage across all running containers in real time, similar to top but tailored to the container level.

docker stats --no-stream

CONTAINER ID   NAME      CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O       PIDS
a1b2c3d4e5f6   web-app   145.32%   890.5MiB / 1GiB       86.96%    1.2MB / 850kB     12MB / 4MB      24

The --no-stream flag takes a single snapshot rather than continuously updating the view, making it suitable for monitoring scripts or quick checks. Note that the CPU % column can exceed 100% on multi-core hosts because the calculation is relative to a single core; a value of 145.32% in the example above indicates the container is utilizing the equivalent of 1.45 CPU cores. The MEM USAGE / LIMIT column should be evaluated alongside MEM %: if MEM % remains consistently close to 100%, the container risks being forcefully terminated by the kernel OOM killer. The PIDS column on the far right is also worth monitoring, as sudden spikes in process/thread counts usually indicate that the application failed to clean up child processes (process leak) rather than a standard traffic surge.

40.4.2 Containers Forcefully Terminated by OOM

When a container exceeds the memory limit set via --memory, the Linux kernel activates the OOM killer (Out Of Memory killer) to forcefully stop processes inside the container via a SIGKILL signal, resulting in exit code 137. Verify whether exit code 137 was caused by an OOM event rather than a manual docker kill by checking the OOMKilled field in docker inspect results.

docker inspect web-app --format '{{.State.OOMKilled}}'

A output of true confirms the container was terminated due to exceeding allocated memory limits. The short-term fix is increasing the container's memory limit via --memory to accommodate peak application load. However, the long-term solution involves determining whether application memory consumption is normal or if a memory leak causes usage to increase over time without returning to baseline. Monitor container memory usage trends across normal operational cycles using docker stats to differentiate these scenarios: memory usage that continuously increases without stabilizing points to a memory leak, whereas usage that fluctuates alongside traffic patterns typically indicates a need for higher memory limits.

40.4.3 Diagnosing Disk I/O Bottlenecks

Containers displaying slow read/write performance despite low CPU and memory usage usually indicate disk I/O bottlenecks. The BLOCK I/O column in docker stats provides a high-level view of data volumes read and written by the container. For detailed host-level I/O metrics, use standard Linux tools such as iostat (part of the sysstat package, installed via apt install sysstat on Debian/Ubuntu or dnf install sysstat on RHEL-based distributions) executed directly on the host rather than inside the container.

iostat -x 2

Monitor the %util column in the iostat output; values consistently near 100% on a specific disk indicate that the disk has become a bottleneck, regardless of which container generates the load. Note that iostat assumes a native Linux host. When running Docker via Docker Desktop on macOS or Windows, all containers execute inside an internal Linux virtual machine managed by Docker Desktop. Running iostat directly in a macOS/Windows terminal will not reflect container disk I/O; performance monitoring on these operating systems is more accurately performed using Docker Desktop's built-in resource dashboard rather than Linux command-line tools.

The most common cause of container-specific I/O bottlenecks in the field is applications writing large amounts of data to the container's writable layer rather than a dedicated volume. Storage drivers like overlay2 introduce additional overhead compared to writing directly to mounted volumes. Move directories handling high-volume data (application logs, uploaded files, database data directories) to named volumes instead of letting them accumulate on the container's writable layer, following volume management practices covered in the dedicated volume chapter, ensuring stable I/O performance and preventing data loss upon container deletion.

40.5 Summary of Common Symptoms and Causes

The following table summarizes symptoms, potential causes, and initial diagnostic steps for issues covered in this chapter, serving as a quick reference during incident response.

Most production Docker incidents boil down to a small combination of items from the list above rather than exotic problems requiring deep research. Establishing a systematic inspection sequence—checking exit codes, logs, network configurations, and resource usage before suspecting application code bugs—is typically sufficient to narrow down most Docker issues to clear root causes and concrete solutions.