System Monitoring

Chapter 37 closed with a logical consequence: once cloud-init makes provisioning new VMs as simple as writing a single user-data file, the number of managed servers tends to grow rapidly, and the old habit of checking server health one by one via SSH quickly begins to fall behind. We have actually been practicing this reactive workflow unconsciously since the beginning of this series: logging in via SSH, running top briefly, and logging out once the server looks fine. This approach still makes sense for one or two servers, but it is impossible to scale once the server count grows to dozens, especially if you only inspect them after a problem is reported. Monitoring is here to transform this workflow from reactive to proactive: collecting server condition data continuously so problems can be detected before impacting the user, not after.

This scenario is very familiar to Sysadmins managing production servers. Developers report that the application feels slow since morning, while Users and Clients have already started complaining via support tickets. Without historical data, a Sysadmin can only guess whether the CPU is maxed out, memory is exhausted, or disk I/O is becoming the bottleneck. Proper monitoring answers that question within seconds through charts that recorded server conditions long before the first complaint came in. Ideally, the Sysadmin would already know and act before Developers even have the chance to report anything.

This chapter starts from classic command line tools like htop, vmstat, and iostat, which remain relevant for fast investigation via SSH, then moves up to Netdata as an example of modern monitoring presenting a real-time dashboard with almost zero configuration. From there, we introduce the basic concepts of Prometheus and Grafana: a monitoring stack combination that has become the de facto industry standard for supervising many servers at once from a single centralized place, a real necessity as server fleets grow via cloud-init in Chapter 37. The chapter wraps up with simple alerting: the final step that turns passive monitoring data into active notifications, ensuring Sysadmins are notified first instead of waiting for complaints from Developers or Users.

38.1 Basic Monitoring: htop, vmstat, and iostat

38.1.1 Why Command Line Monitoring Remains Relevant

Before moving into modern dashboards in Sections 38.2 and 38.3, it is important to master three command line tools that have served as standard rapid investigation tools for sysadmins for decades. All three are lightweight, put almost no load on a server that might already be struggling, and can still run over a standard SSH session without relying on additional services that might not be installed or might be failing while the server is critical. In practice, the most common moment these tools are used is right after an alert from Grafana in Section 38.4 arrives: the Sysadmin's first step remains the same as ever, SSHing into the affected server to inspect its condition directly.

38.1.2 Monitoring Real-Time CPU and Memory with htop

htop is an interactive version of Linux's built-in top command, displaying a process list that can be scrolled, sorted, and killed directly from its own interface. It is far more convenient than plain top, which has a much more rigid interface.

Practical Steps

1. Install htop, as this package is not included by default in the base Ubuntu Server installation since Chapter 2.

   sudo apt install -y htop

2. Run htop without any arguments.

   htop

3. Observe the top section of the screen displaying CPU meters per core, memory usage (Mem), and swap usage (Swp), while the lower section displays the full process list with %CPU, %MEM, and COMMAND columns.
4. Press F6 to sort the process list based on a specific column, for example %MEM when suspecting a memory leak in one of the PHP-FPM processes from Chapter 14.
5. Press t to toggle the tree view, showing parent-child relationships between processes, useful to see which worker processes originated from a single Nginx master process from Chapter 12.
6. Press q to exit.

Verification and Troubleshooting

• The RES column is much more accurate than VIRT for assessing the actual memory usage of a process. VIRT includes all virtual memory addresses mapped by the process, including shared libraries shared with other processes, so its figure often appears much larger than the physical memory actually consumed.
• Processes with a persistent D state in the S (state) column are usually waiting for disk I/O operations, an early indicator to switch and inspect iostat in Section 38.1.4.
• The Load average line at the top right of the screen displays three numbers representing average queued processes over the last 1, 5, and 15 minutes. A common rule of thumb in the field is comparing the first number with the server's CPU core count: a value of 4.00 on a 4-core server is considered heavy but acceptable, while the same value on a 2-core server indicates a fairly serious CPU queue.

38.1.3 Reading System Statistics with vmstat

vmstat summarizes system-wide CPU, memory, and swap conditions in a single concise line per interval, ideal for viewing load trends without having to scroll through process lists one by one like htop. Unlike htop, vmstat is part of the essential procps package, making it available by default since the initial Ubuntu Server setup in Chapter 2 without separate installation.

Practical Steps

1. Run vmstat with a 2-second interval, repeated 5 times.

   vmstat 2 5

Verification and Troubleshooting

• The r column under the procs section shows the number of processes queued waiting for CPU time. An r value consistently higher than the server's CPU core count indicates that the CPU is indeed a bottleneck, not merely experiencing a temporary spike.
• The si and so columns under the swap section are most critical to monitor. Non-zero values in both columns mean the kernel is actively swapping data between RAM and swap space: a sign that physical memory is exhausted. This is far more convincing than merely looking at a low free column, as Linux intentionally uses unused memory for page caching under normal conditions.
• The wa column under the cpu section indicates the percentage of CPU time spent waiting for I/O operations to complete. High wa values point suspicion toward the disks: time to proceed to iostat in Section 38.1.4 to confirm which disk is responsible.

38.1.4 Monitoring Disk I/O with iostat

iostat dissects I/O load per individual disk, extending the wa summary seen in vmstat with details on which disk is actually causing the bottleneck.

Practical Steps

1. Install the sysstat package, the source of the iostat command, which is not installed by default.

   sudo apt install -y sysstat

2. Run iostat with the -x option for extended statistics and -z to omit devices with zero activity, set to a 2-second interval repeated 5 times.

   iostat -xz 2 5

Verification and Troubleshooting

• A %util column hovering near 100 continuously indicates that the target disk is nearly always busy serving requests, the most direct indicator of an I/O bottleneck.
• The await column displays the average wait time (in milliseconds) for an I/O request to be served, including queue time. An await figure significantly higher than normal, even if %util has not hit 100, often serves as an early sign of physical disk degradation, especially on HDD-based storage whose characteristics were discussed in Chapter 7.
• The first line of output from iostat always represents system averages since the last boot, not current conditions. Ignore this first line and focus on subsequent lines that accurately reflect the 2-second interval that just passed.

38.2 Introduction to Modern Monitoring: Netdata

38.2.1 Why Modern Monitoring is Needed Beyond CLI Tools

All three tools in Section 38.1 share a common limitation: they only display system conditions at the exact moment the command is run, and their data disappears as soon as the SSH session closes. To see trends over the night, such as investigating a 3 AM CPU spike after it happened, these three tools offer little help without someone watching directly. Netdata fills this gap as an example of an open-source modern monitoring tool running as a background service, gathering thousands of metrics per second, and serving them via a web dashboard accessible anytime without maintaining an active SSH connection during data collection.

38.2.2 Netdata Installation

Netdata provides an official kickstart script as the fastest installation method, automatically detecting the running distribution and installing all components at once, similar to the Docker convenience script pattern practiced in Section 26.1.2.

Practical Steps

1. Download the script separately first to inspect its contents prior to execution: a good habit identical to the Docker setup in Section 26.1.2.

   curl https://get.netdata.cloud/kickstart.sh -o /tmp/netdata-kickstart.sh
   less /tmp/netdata-kickstart.sh

2. Run the script with the --non-interactive option to perform an unattended installation, skipping the prompt to claim the node to Netdata Cloud, which is unnecessary for this lab.

   sudo sh /tmp/netdata-kickstart.sh --non-interactive

3. Ensure the netdata service is active and set to start automatically on boot.

   systemctl status netdata

Verification and Troubleshooting

• A candid security note: executing an installation script via sh as root means entrusting execution completely to script contents downloaded from the internet. Inspecting its contents beforehand, as in step one above, remains recommended, especially for production servers, aligning with notes regarding get.docker.com in Section 26.1.2.
• Test from the server itself without needing a browser, simply verifying that the Netdata API responds on its default port, 19999.

  curl -s http://localhost:19999/api/v1/info

  An output of JSON data displaying the Netdata version and total active chart count indicates the service is running normally.

38.2.3 Exploring the Netdata Dashboard

The Netdata dashboard can be accessed via port 19999, but exposing it directly to external networks creates a new attack surface without built-in authentication mechanisms, violating the principle of minimizing open ports discussed in Chapters 30 and 31. The safest approach for this lab is accessing it via SSH local port forwarding, leveraging the SSH connection trusted since Chapter 3 without opening additional ports in UFW.

Practical Steps

1. From the local workstation (not inside the server), establish an SSH tunnel forwarding local port 19999 to port 19999 on the server.

   ssh -L 19999:localhost:19999 [email protected]

2. While the SSH session remains open, access the dashboard via a web browser on the local machine.

   http://localhost:19999

3. Explore the System Overview section for a unified summary of CPU, memory, disk, and network, then scroll down to the Disks section to view the %util chart per disk: equivalent to iostat output in Section 38.1.4, but presented as historical graphs that continue recording even when unmonitored.

Verification and Troubleshooting

• All charts on the Netdata dashboard update every second without requiring a page refresh, using per-second data resolution by default. This is far more detailed than the typical 15 to 60-second intervals common to other monitoring stacks like Prometheus in Section 38.3.
• If connection via http://localhost:19999 fails, verify that the SSH session from step one remains open in a separate terminal. Closing the SSH session automatically tears down the tunnel, even if the netdata service on the server itself continues running normally.
• For multi-computer access on an office LAN without per-user tunneling, an alternative is opening the port via UFW limited to the local subnet, following the same subnet restriction pattern as PostgreSQL in Section 18.4.2.

  sudo ufw allow from 192.168.1.0/24 to any port 19999 proto tcp

  The SSH tunnel approach remains preferred for internet-exposed servers, while this UFW pattern fits servers strictly accessed from internal office networks.

38.3 Introduction to Prometheus and Grafana (Basic Concepts)

38.3.1 Prometheus Architecture: Pull Model, Exporters, and Time Series

Netdata in Section 38.2 excels at deep monitoring of a single server, but as server counts scale via cloud-init in Chapter 37, opening dashboards individually per server is impractical. Prometheus answers this need as an open-source monitoring system and time series database designed specifically to scrape metrics from multiple targets into a centralized location.

The core distinction in Netdata's approach lies in data retrieval direction. Prometheus uses a pull model: the Prometheus server actively reaches out to each target to scrape data over HTTP at configured intervals, opposite to push models where targets push data to a central server. To be scraped, targets must expose a /metrics endpoint supplying text-formatted metric data recognized by Prometheus. Components providing this endpoint are called exporters; for operating system metrics like CPU, memory, and disk, the official exporter is node_exporter. It essentially presents the same conceptual data read manually via vmstat and iostat in Section 38.1, formatted for automated remote retrieval.

38.3.2 Prometheus and Node Exporter Installation

Practical Steps

1. Install Prometheus server and node_exporter from official Ubuntu repositories.

   sudo apt update
   sudo apt install -y prometheus prometheus-node-exporter

2. Verify both services are active automatically post-installation.

   systemctl status prometheus prometheus-node-exporter

3. Open the main Prometheus configuration file.

   sudo nano /etc/prometheus/prometheus.yml

4. Add a new job_name block under scrape_configs so Prometheus scrapes metrics from node_exporter running on port 9100, beneath the default prometheus self-monitoring job on port 9090.

   scrape_configs:
     - job_name: prometheus
       static_configs:
         - targets: ['localhost:9090']
   
     - job_name: node
       static_configs:
         - targets: ['localhost:9100']

5. Reload configuration by restarting the Prometheus service.

   sudo systemctl restart prometheus

Verification and Troubleshooting

• Ensure the Prometheus server itself is healthy via its built-in health check endpoint.

  curl http://localhost:9090/-/healthy

  Output reading Prometheus Server is Healthy. confirms the service is running normally.
• Open http://192.168.1.10:9090/targets in a browser (using an SSH tunnel or UFW rules per Section 38.2.3 if accessing remotely) to verify both prometheus and node jobs show an UP status. A DOWN status on the node job typically indicates the prometheus-node-exporter service is inactive or port 9100 is unreachable.
• YAML indentation errors in prometheus.yml cause the service to fail to start entirely. Run sudo systemctl status prometheus after restarting and check sudo journalctl -u prometheus -n 50 if the service fails to reach an active state: the same troubleshooting procedure used for systemd services since Chapter 5.

38.3.3 Grafana Installation and Connecting Prometheus Data Source

While Prometheus in Section 38.3.2 includes a native web interface, its visualization capabilities are basic. Grafana complements Prometheus as a feature-rich open-source visualization tool, capable of building dashboards across multiple data sources simultaneously, including Prometheus. Grafana is not in standard Ubuntu repositories, so it must be added via its official APT repository, following GPG key and repository patterns identical to MongoDB in Section 21.2.

Practical Steps

1. Install required dependencies, then add the official Grafana GPG key.

   sudo apt install -y apt-transport-https software-properties-common wget
   sudo mkdir -p /etc/apt/keyrings/
   wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null

2. Register the official Grafana repository.

   echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

3. Update package indexes and install Grafana.

   sudo apt update
   sudo apt install -y grafana

4. Enable and start the service.

   sudo systemctl enable --now grafana-server

5. Access the Grafana dashboard via browser on port 3000, utilizing SSH tunneling or UFW rules per Section 38.2.3 patterns, then log in using default credentials admin/admin. Grafana prompts for an immediate password change upon initial login.
6. After login, navigate to Connections > Data sources > Add data source, select Prometheus, set the URL field to http://localhost:9090 because Grafana and Prometheus run on the same server, then click Save & test.

Verification and Troubleshooting

• A Successfully queried the Prometheus API message after clicking Save & test verifies data source connectivity. Connection error messages typically mean an incorrect URL or an unstarted Prometheus service; re-check using curl http://localhost:9090/-/healthy per Section 38.3.2.
• In true production environments, Prometheus and Grafana are usually separated onto dedicated servers outside the application servers being monitored, ensuring monitoring overhead does not impact primary application resources. Running both on a single server in this chapter is strictly for lab simplicity, not a recommended architecture for multi-server production environments.

38.3.4 Creating a Simple Dashboard in Grafana

Practical Steps

1. Navigate to Dashboards > New > Import.
2. Enter ID 1860 into the Import via grafana.com field: the ID for the community Node Exporter Full dashboard, one of the most popular community dashboards for comprehensive node_exporter metrics without building panels manually from scratch.
3. Select the Prometheus data source created in Section 38.3.3 when prompted, then click Import.

Verification and Troubleshooting

• The imported dashboard immediately displays dozens of panels showing CPU, memory, disk, and network graphics from node_exporter, populating with live data within moments based on the scrape interval set in prometheus.yml in Section 38.3.2.
• Panels displaying blank states or No data usually indicate an incorrect data source choice during import, or that the node job in Prometheus has not reached UP status as checked in Section 38.3.2.
• Community dashboards serve as exceptional starting points covering almost all core node_exporter metrics, but should be customized further as monitoring demands grow specific, such as adding custom panels for application-specific metrics like Nginx or PostgreSQL not captured by node_exporter.

38.4 Simple Alerting

38.4.1 Alerting Concepts: Thresholds, Conditions, and Notification Channels

Dashboards, no matter how advanced, share one weakness: they are only useful when someone actively views them. Alerting closes this gap by evaluating specified conditions against metrics automatically and continuously, dispatching notifications when conditions match, without requiring a Sysadmin to monitor dashboards manually. The three core components of alerting are thresholds (the reference baseline, e.g., disk usage above 90%), evaluation conditions (how long a threshold must be breached before being flagged as an issue), and notification channels or contact points (where notifications are delivered, such as email, Slack, or Telegram).

38.4.2 Built-in Netdata Alerts and Inspection Methods

An advantage of Netdata from Section 38.2 is its collection of pre-configured health checks active upon installation. These cover standard scenarios like prolonged high CPU utilization, low available memory, or near-full disk space, requiring zero configuration lines.

Practical Steps

1. From the target server, query all active Netdata alarms via its API, including those in a normal status. Append the ?all parameter, as omitting it returns only active non-normal alarms (WARNING or CRITICAL).

   curl -s http://localhost:19999/api/v1/alarms?all | less

2. Open the Netdata dashboard per Section 38.2.3, then open the Alarms tab on the right side to inspect the identical list visually, including color-coded alarm statuses.

Verification and Troubleshooting

• A CLEAR status across alarms indicates normal conditions. Statuses shift to WARNING or CRITICAL once default thresholds are crossed, changing color to yellow or red on the dashboard without extra configuration because they are pre-defined by Netdata in the /etc/netdata/health.d/ directory.
• By default, alarm notifications appear on the dashboard only and are not automatically forwarded to email or Slack. Enabling external notifications requires extra configuration in /etc/netdata/health_alarm_notify.conf, which falls outside this chapter's scope, though threshold evaluation runs natively out of the box.

38.4.3 Building Grafana Alert Rules for Prometheus Metrics

Unlike Netdata's automatically active alarms, Grafana Alerting requires user-defined alert rules matched to relevant metrics. This section demonstrates the most common and vital scenario: detecting when a target stops sending metrics altogether, serving as an immediate indicator that the server is failing or offline.

Practical Steps

1. Navigate to Alerting > Alert rules > New alert rule.
2. In the query section, choose the Prometheus data source, then input the following PromQL expression to detect targets with a down status for the node job:

   up{job="node"} == 0

3. Configure the Evaluation group and evaluation interval to 1m, setting the Pending period to 1m as well. This requires conditions to persist for at least 1 minute before changing to Firing state, preventing false positives caused by brief network blips.
4. Assign a name to the alert rule, such as Node Exporter Down, then save.

Verification and Troubleshooting

• Test this alert rule by temporarily stopping node_exporter.

  sudo systemctl stop prometheus-node-exporter

• Wait out the configured Pending period, then inspect the Alerting > Alert rules page. The status for Node Exporter Down should transition from Normal to Pending, and finally to Firing once sustained.
• Restart node_exporter, then verify the alert status returns to Normal after Prometheus successfully scrapes the target again.

  sudo systemctl start prometheus-node-exporter

• A candid note on demonstration limits: while this alert rule state changes correctly, it does not send external notifications anywhere yet. Grafana's default contact point requires SMTP server configuration in grafana.ini, which has not been prepared. In production environments, next steps involve connecting contact points to email via SMTP relays or Slack/Telegram webhooks: topics intentionally left out here as they rely on individual organizational infrastructure.

At this point, we have covered the full spectrum of monitoring: from classic command-line tools for fast SSH investigations, to Netdata as an out-of-the-box modern dashboard for single servers, to Prometheus and Grafana as a centralized stack for managing multiple servers at once, concluding with alerting mechanisms that convert passive data into active notifications. However, numerical metrics gathered throughout this chapter have a foundational limit: they only answer what happened (such as CPU spikes or targets going down), without explaining why it happened. Answering why is the role of logs, and managing logs centrally across multiple servers becomes the primary focus of Chapter 39.