Applications running stably in production still need fast answers to three daily questions: how healthy the resources currently used by the container are, why a specific request is suddenly slow, and at which exact line of log an error occurred. Relying on docker stats or docker logs one by one is still usable for quick debugging, but it is clearly impractical once the number of containers grows into dozens or when an incident requires correlation between metrics and logs within the same timeframe. This chapter builds a complete observability stack based on Docker Compose, which is a combination of tooling to observe system conditions comprehensively, using Prometheus to collect metrics, Grafana for visualization and dashboards, and Loki for centralized logs, then connecting the three with container applications discussed in previous case studies.
34.1 Setting Up Prometheus for Metrics Collection
Prometheus is an open-source monitoring system and time series database that collects metrics by pulling data periodically from registered targets, a pattern called pull-based scraping. Each target simply needs to provide an HTTP endpoint containing metrics in a standardized text format, and Prometheus visits that endpoint according to the configured interval, as opposed to the push-based pattern that requires applications to actively send data to the monitoring server.
34.1.1 Observability Stack Architecture
The stack in this chapter consists of six complementary services. The prometheus service is responsible for storing and providing metrics via queries. The cadvisor service (short for Container Advisor) exposes CPU, memory, network, and disk usage metrics per container. The node-exporter service exposes host-level metrics, such as overall machine CPU and memory usage. The grafana service displays all these metrics in the form of visual dashboards. The loki service stores centralized logs from all containers, while alloy collects logs from each container and sends them to Loki. Sysadmins/DevOps Engineers usually place all these services in a separate compose.yaml from the main application, so the observability stack can be managed and upgraded independently of the deployment cycle of the application it monitors.
Structure the compose.yaml framework with prometheus, cadvisor, and node-exporter services first as the foundation for metric collection.
services:
prometheus:
image: prom/prometheus:v3.13.3
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
- prometheus-data:/prometheus
ports:
- "9090:9090"
networks:
- observability-net
cadvisor:
image: gcr.io/cadvisor/cadvisor:v0.56.2
volumes:
- /:/rootfs:ro
- /var/run:/var/run:ro
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
networks:
- observability-net
node-exporter:
image: prom/node-exporter:v1.12.1
volumes:
- /proc:/host/proc:ro
- /sys:/host/sys:ro
- /:/rootfs:ro
command:
- "--path.procfs=/host/proc"
- "--path.sysfs=/host/sys"
- "--path.rootfs=/rootfs"
networks:
- observability-net
networks:
observability-net:
volumes:
prometheus-data:The configuration for cadvisor and node-exporter both borrow several paths from the host filesystem via bind mounts in ro (read-only) mode, because both need to read information from the kernel and Docker runtime at the host level to accurately measure resource usage. This read-only mode is important as an additional security layer, so that these two containers cannot write anything to the host filesystem despite having extensive read access.
34.1.2 prometheus.yml Configuration and Scrape Targets
The prometheus.yml file defines which targets Prometheus should scrape along with their intervals, following the official Prometheus configuration format.
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
- job_name: "cadvisor"
static_configs:
- targets: ["cadvisor:8080"]
- job_name: "node-exporter"
static_configs:
- targets: ["node-exporter:9100"]
- job_name: "api"
metrics_path: /metrics
static_configs:
- targets: ["api:3000"]The job_name here can be freely used as a label to distinguish metric origins when querying, while targets use Compose service names because Prometheus runs within the same internal network and can resolve service names via Docker's built-in DNS, the same pattern as inter-container communication in previous case studies. The api job in this example assumes the backend application already exposes its own /metrics endpoint via Prometheus client library instrumentation on the application code side.
Restart the prometheus service after modifying prometheus.yml, then open the Prometheus web interface to verify the status of each target.
docker compose up -d prometheus
curl -s http://localhost:9090/-/healthyAccess http://localhost:9090/targets from a browser to view the UP or DOWN status of each defined target. A target with DOWN status usually indicates that Prometheus cannot reach that endpoint, whether due to a misspelled service name, mismatched port, or the target container not being fully ready to accept connections.
34.2 Grafana for Visualization and Dashboards
Grafana is an open-source visualization platform that reads data from various data sources, including Prometheus and Loki, and displays it as interactive dashboards containing graphs, tables, and other panels. Add the grafana service to the same compose.yaml.
34.2.1 Setting Up Grafana and Data Source Connections
services:
grafana:
image: grafana/grafana:13.2.1
ports:
- "3001:3000"
environment:
GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD}
volumes:
- grafana-data:/var/lib/grafana
- ./grafana/provisioning:/etc/grafana/provisioning:ro
networks:
- observability-netThe GF_SECURITY_ADMIN_PASSWORD variable overrides Grafana's default admin password, following official documentation for Grafana configuration via environment variables, and should ideally be populated via a .env file that is not committed to version control, similar to credential management patterns in other case studies. The grafana-data volume keeps dashboards, users, and other settings persistent even if the Grafana container is removed and recreated.
Rather than adding Prometheus data sources manually via the web interface every time the stack is rebuilt, leverage provisioning, a mechanism where Grafana reads data source and dashboard configurations from YAML files during startup. Create grafana/provisioning/datasources/prometheus.yaml.
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: trueAlso create grafana/provisioning/datasources/loki.yaml for the Loki data source discussed in the logging section.
apiVersion: 1
datasources:
- name: Loki
type: loki
access: proxy
url: http://loki:310034.2.2 Automated Dashboard Provisioning
Grafana can also load JSON dashboard files automatically via provisioning, ensuring dashboards are not lost when the Grafana volume is reset. Create grafana/provisioning/dashboards/default.yaml to register the folder where dashboard files are stored.
apiVersion: 1
providers:
- name: "default"
folder: ""
type: file
options:
path: /etc/grafana/provisioning/dashboards/jsonPlace JSON dashboard files exported from Grafana, or community dashboards downloaded from the Grafana Dashboards Marketplace (such as the official cAdvisor or Node Exporter Full dashboard), into the grafana/provisioning/dashboards/json folder. Restart the grafana service so the dashboards automatically appear.
docker compose up -d grafanaLog in to Grafana via http://localhost:3001 using the admin user and the password defined in GRAFANA_ADMIN_PASSWORD, then check the Connections > Data sources menu to verify that Prometheus and Loki are registered with a successful connection status without requiring any manual configuration.
34.3 Loki for Centralized Logging
Loki is a log aggregation system from Grafana Labs specifically designed to efficiently store large volumes of logs, using an approach that indexes only label metadata instead of indexing the full log text content like Elasticsearch. This approach makes Loki much lighter in terms of resource usage, suitable for Sysadmins/DevOps Engineers who want centralized logging without having to operate a heavy text search cluster.
34.3.1 Setting Up Loki and Grafana Alloy
Collecting logs from containers to Loki is performed via Grafana Alloy, the official collector from Grafana Labs that replaces Promtail as the log collection agent. Promtail was declared deprecated starting from Loki version 3.0 and its code was fully merged into Alloy, according to Loki's official release notes, making Alloy the right choice for stacks built today. Add loki and alloy services to compose.yaml.
services:
loki:
image: grafana/loki:3.7.7
ports:
- "3100:3100"
volumes:
- ./loki-config.yaml:/etc/loki/local-config.yaml:ro
- loki-data:/loki
networks:
- observability-net
alloy:
image: grafana/alloy:1.19.2
ports:
- "12345:12345"
volumes:
- ./config.alloy:/etc/alloy/config.alloy:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
command:
- run
- --server.http.listen-addr=0.0.0.0:12345
- --storage.path=/var/lib/alloy/data
- /etc/alloy/config.alloy
networks:
- observability-netAlloy needs access to docker.sock so it can read container metadata via the Docker API, while pulling log streams from each container directly from the Docker daemon, without having to read raw JSON log files from the host filesystem like older approaches. Port 12345 mapped to the host provides Alloy's built-in debugging web interface, where you can monitor the real-time status of each pipeline component.
Create config.alloy containing the log collection pipeline definition, written in Alloy's component-based declarative syntax.
discovery.docker "containers" {
host = "unix:///var/run/docker.sock"
refresh_interval = "5s"
}
discovery.relabel "containers" {
targets = discovery.docker.containers.targets
rule {
source_labels = ["__meta_docker_container_name"]
regex = "/(.*)"
target_label = "container"
}
}
loki.source.docker "containers" {
host = "unix:///var/run/docker.sock"
targets = discovery.docker.containers.targets
relabel_rules = discovery.relabel.containers.rules
forward_to = [loki.process.containers.receiver]
}
loki.process "containers" {
stage.docker {}
forward_to = [loki.write.default.receiver]
}
loki.write "default" {
endpoint {
url = "http://loki:3100/loki/api/v1/push"
}
}The discovery.docker component detects all running containers on the host via the Docker API, automatically discovering new containers as soon as they run without needing to manually register targets one by one. The discovery.relabel component extracts the container name from the __meta_docker_container_name metadata label and assigns it to the container label, facilitating log filtering by container name during queries in Grafana later. The loki.source.docker component pulls log streams from each detected container, then forwards them to loki.process which applies stage.docker to parse Docker's native JSON log format into clean log text lines. The final component, loki.write, sends the final output to Loki's push endpoint.
Also create loki-config.yaml with basic local storage configuration, sufficient for single-node needs.
auth_enabled: false
server:
http_listen_port: 3100
common:
path_prefix: /loki
storage:
filesystem:
chunks_directory: /loki/chunks
rules_directory: /loki/rules
replication_factor: 1
ring:
instance_addr: 127.0.0.1
kvstore:
store: inmemory
schema_config:
configs:
- from: 2024-01-01
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h34.3.2 Docker Logging Driver to Loki (Optional)
Besides using Alloy to pull logs centrally from the Docker daemon, Docker also provides an official Loki logging driver plugin that sends logs from a single container directly to Loki without an intermediate Docker API call, suitable when you want to reduce one component (Alloy) from the stack. Install the plugin first on the Docker host.
docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissionsThe latest tag is intentionally used here, unlike the habit of avoiding latest on standard images, because plugin releases are distributed per architecture (such as amd64 and arm64) instead of a single uniform version number tag, according to its official tag list on Docker Hub. The latest tag remains the safest way to ensure the installed plugin matches the host architecture without guessing architecture-specific tag names manually.
Once the plugin is installed, configure the logging driver on the application service whose logs should be sent directly to Loki.
services:
api:
build: ./api
logging:
driver: loki
options:
loki-url: "http://localhost:3100/loki/api/v1/push"
loki-retries: "3"
loki-batch-size: "400"It is worth noting that this logging driver approach only captures logs from containers explicitly configured to use it, unlike Alloy which automatically captures logs from all containers via service discovery. Both approaches are equally valid; choose Alloy if you want a single centralized configuration point for all containers, or the logging driver if only specific containers need to be sent to Loki.
34.4 Integrating Monitoring with Container Applications
Infrastructure metrics from cAdvisor and node-exporter are useful, but Developers and Sysadmins/DevOps Engineers in the field typically need application-level metrics, such as request rates per second, endpoint latency, or error counts, because these metrics reveal the root cause much faster than simply knowing that container CPU usage is high.
34.4.1 Instrumenting Applications with Metrics Endpoints
Add the official Prometheus client library according to the backend language used, such as prom-client for Node.js, to expose the /metrics endpoint configured in prometheus.yml in the previous section.
const client = require("prom-client");
const express = require("express");
const app = express();
const register = new client.Registry();
client.collectDefaultMetrics({ register });
const httpRequestDuration = new client.Histogram({
name: "http_request_duration_seconds",
help: "HTTP request duration in seconds",
labelNames: ["method", "route", "status_code"],
});
register.registerMetric(httpRequestDuration);
app.get("/metrics", async (req, res) => {
res.set("Content-Type", register.contentType);
res.end(await register.metrics());
});
app.listen(3000);collectDefaultMetrics automatically exposes standard metrics such as Node.js process memory usage and garbage collection duration, while the http_request_duration_seconds metric is created specifically to measure request duration based on method, route, and status_code labels, adhering to metric naming conventions recommended by official Prometheus documentation. Connect the api container port to the observability-net network so Prometheus can reach it, without needing to expose that port to the host.
34.4.2 Alerting with Alertmanager
Simply looking at a dashboard is insufficient if no one is watching it continuously. Alertmanager handles notifications based on alert rules evaluated by Prometheus, dispatching them to channels like email, Slack, or webhooks. Add the alertmanager service to the stack.
services:
alertmanager:
image: prom/alertmanager:v0.34.0
volumes:
- ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
ports:
- "9093:9093"
networks:
- observability-netDefine alert rules in a separate file, such as alert-rules.yml, then register them via the rule_files option in prometheus.yml.
groups:
- name: container-alerts
rules:
- alert: ContainerHighCPU
expr: rate(container_cpu_usage_seconds_total{name!=""}[5m]) > 0.8
for: 5m
labels:
severity: warning
annotations:
summary: "Container {{ $labels.name }} high CPU usage"
description: "Average container CPU usage above 80% for the last 5 minutes."rule_files:
- "alert-rules.yml"
alerting:
alertmanagers:
- static_configs:
- targets: ["alertmanager:9093"]This ContainerHighCPU rule leverages the container_cpu_usage_seconds_total metric from cAdvisor, triggering an alert if average container CPU usage stays above 80% for at least 5 consecutive minutes, rather than triggering on a momentary spike. The for: 5m clause is important so alerts are not overly sensitive to normal fluctuations, a common practice applied by Sysadmins/DevOps Engineers to reduce alert fatigue, a condition where teams become desensitized or ignore notifications due to receiving frequent insignificant alerts.
34.4.3 Verification and Troubleshooting the Observability Stack
Run the entire stack at once, then verify the status of each container to ensure everything is operating normally.
docker compose up -d
docker compose psOpen Grafana at http://localhost:3001 and create a new panel with the query rate(container_cpu_usage_seconds_total[5m]) to verify that metrics from cAdvisor are arriving. To inspect logs, navigate to the Explore menu in Grafana, select the Loki data source, and execute a simple LogQL query like {container="api"} to view application container logs directly without running docker logs manually.
If a Prometheus target shows a DOWN status on the /targets page, the most common root cause in practice is that the target container has not joined the same network as Prometheus, or the port registered in prometheus.yml does not match the port the application actually listens on inside the container. If logs do not appear in Loki via Alloy, check whether Alloy has proper access permissions to docker.sock, as socket permission errors are the most frequent cause when the discovery.docker component fails to find running containers. Open Alloy's debugging interface at http://localhost:12345 to inspect real-time status across pipeline components, including whether loki.write successfully pushes data to the Loki endpoint or fails due to an incorrect url address. Also ensure that the prometheus-data, grafana-data, and loki-data volumes are preserved when executing docker compose down without the -v flag, because collected metrics history, dashboards, and logs will be permanently lost if those volumes are removed.