Networking Fundamentals for Sysadmins

Chapter 19.3.3 concluded the discussion of ping with a promise that other networking tools such as curl and ss would be "discussed in much greater depth in Chapter 22", and Chapter 21.4.1 also postponed a more comprehensive discussion of network health checks to this chapter. Now is the time to deliver on both promises. Sysadmins spend a significant amount of troubleshooting time answering one simple question: is this server reachable or not, and if not, at which layer does the issue stop? Is the host completely unresponsive, is its application port closed, or is the service active but responding with an error? This chapter builds the capability to answer that question systematically, starting with basic network commands (22.1), moving on to automated health checks that combine multiple inspection layers simultaneously (22.2), testing API connections to validate not just the HTTP status but also the response body (22.3), and concluding with a case study script that checks the connectivity of multiple servers at once (22.4). Chapter 23 will later resume using some of the tools introduced here (specifically ss) from a different perspective, namely open port security audits, so the focus of this chapter is purely on connectivity diagnosis and monitoring, not hardening.

22.1 Basic Network Commands

These four sub-sections examine the networking tools most frequently used by Sysadmins on a daily basis, along with details that are often overlooked until they become confusing sources of bugs.

22.1.1 Testing HTTP Endpoints with curl

Chapter 20.2.4 used curl -s -o /dev/null -w '%{http_code}' to check webhook endpoints, and Chapter 21.4.2 used curl -sf for deployment health checks. This sub-section examines these curl options in greater detail, as both previously utilized only a small portion of curl's capabilities for network diagnosis.

curl -I https://api.example.com/status

The -I/--head option sends a HEAD request instead of GET, according to the official curl documentation, fetching only response headers without any body, making it ideal for quickly checking if an endpoint is alive without downloading its entire content. A note from field experience: some strictly configured web servers or reverse proxies may reject the HEAD method even when a GET request to the same endpoint works normally. Therefore, if -I returns an unexpected result (such as 405 Method Not Allowed) while the endpoint itself is healthy, do not rush to conclude that the endpoint is malfunctioning; test it first using -s -o /dev/null -w '%{http_code}\n' which uses a standard GET request.

curl -sS -L --connect-timeout 5 --max-time 15 -o /dev/null -w '%{http_code}\n' https://api.example.com/status

-L/--location enables curl to automatically follow HTTP redirects (3xx codes). This is important because, without this option, an endpoint redirecting from http:// to https:// would be recorded as a 301/302 code rather than the status code of the actual target endpoint. --connect-timeout limits the maximum duration specifically for the TCP connection establishment phase, whereas --max-time limits the overall request duration from start to finish, according to the official curl documentation. Both operate at different layers and should ideally be used together, as --connect-timeout alone will not help if the connection opens successfully but the server hangs indefinitely before sending a response.

22.1.2 wget as an Alternative to curl

curl and wget are often considered interchangeable, yet they possess different design emphases: curl was designed as a versatile data transfer client whose output is easily processed further in scripts (via stdout or the -w option), whereas wget excels specifically at downloading operations, including recursively downloading entire website structures and resuming interrupted downloads.

wget --spider -q https://api.example.com/status; echo "Exit status: $?"

The --spider option makes wget behave like a web spider, checking page availability without actually downloading the contents, per official GNU Wget documentation. Combining it with -q (quiet, suppressing output) makes this pattern practical in scripts purely to check its exit status, similar to the use of curl -sf in 21.4.2. For resuming interrupted large file downloads midway, such as fetching large backup archives from another server, the -c/--continue option resumes from the last byte received instead of re-downloading from the beginning; this is a key advantage of wget that is not as easily accomplished with curl. For routine API endpoint health checks throughout the rest of this chapter, curl remains the primary choice because its output is far easier to process via the -w option and it is more commonly pre-installed in minimal container images.

22.1.3 Viewing Active Connections with ss

netstat was once the standard tool for inspecting active network connections and listening ports, but many modern distributions have deprecated the net-tools package from which netstat originates, replacing it with ss from the iproute2 package. Official Red Hat documentation for RHEL 7 explicitly notes that net-tools was deprecated in favor of iproute2 tools, a pattern followed by most Debian- and Arch-derived distributions around the same period. As a result, ss is now pre-installed by default across most modern Linux distributions, while netstat often requires manual installation via a separate net-tools package. ss retrieves connection data directly from the kernel via netlink sockets, whereas netstat parses the /proc/net/tcp text file, making ss significantly faster, especially on servers with very high connection counts.

ss -tuln

These four options, per man ss, function as follows: -t displays TCP sockets, -u displays UDP sockets, -l displays only listening sockets (which are hidden by default), and -n displays raw numeric values (ports and addresses) without attempting to resolve them into human-readable service names that require additional lookup time. This combination provides the fastest way to answer the question "which services are currently opening ports on this server", the exact same initial question required prior to auditing open ports in Chapter 23.3, although its usage in this chapter is strictly for connectivity diagnosis rather than hardening.

sudo ss -tnp state established '( dport = :443 or dport = :80 )'

The -p option appends process information (name and PID) for the owner of each socket, according to man ss, but this information appears completely only when executed with root privileges due to process detail access limits across users; running without sudo typically displays sockets belonging only to the currently logged-in user. The filter state established narrows the display to fully established connections (rather than mere listening sockets), and the expression in parentheses filters by destination port, which is useful when confirming whether an outbound connection to a third-party API on port 443 is actively established rather than hanging during initiation.

22.1.4 Connectivity Diagnosis with ping

The basic pattern ping -c 1 -W 2 host was introduced in Chapter 19.3.3 as a component of the server status dashboard. This sub-section covers details not previously discussed: reading exit statuses and understanding their limitations as an indicator of whether a host is up or down.

ping -c 3 -W 2 10.0.0.5
echo "Exit status: $?"

According to official documentation for ping (from the iputils package commonly used in modern Linux distributions), an exit status of 0 indicates that packets were successfully received back, 1 indicates no response was received (or fewer replies than requested via -c before timeout expiration), and 2 indicates another execution error occurred (such as host name resolution failure). The documentation highlights this design purpose: the exit status is explicitly structured so scripts can check a host's availability directly without parsing text output.

What should be candidly stated based on field experience: a failed ping does not automatically mean the service on that host is down. ping utilizes the ICMP protocol, and many firewalls (including cloud provider configurations such as security groups) intentionally block incoming ICMP traffic as a standard hardening measure, while the actual application port (such as 443 for HTTPS) remains open and operational. Conversely, a successful ping does not guarantee that the application running on the host is healthy; it merely confirms that the host responds at the fundamental network layer. Consequently, ping is best utilized as a rapid preliminary check (whether the host is powered on and reachable), rather than a single health signal for a service; section 22.2 builds multi-layered checks that extend beyond ping alone.

22.2 Automated Endpoint Health Checks

The summary table in 22.1 illustrates that each tool answers only part of the question. The following two sub-sections construct health checks evaluating multiple layers at once: whether the port is open, and whether the underlying application responds properly.

22.2.1 Checking TCP Ports Without External Tools

Bash itself, without requiring nc (netcat) or other external tools, can open raw TCP connections using the special redirection feature /dev/tcp/host/port. According to the GNU Bash Reference Manual, if host is a valid hostname or IP address and port is a valid port number, Bash attempts to open the corresponding TCP socket. This feature is internal to Bash (not a real device file on most systems), so it does not work in other shells like pure POSIX sh or dash.

An important condition often overlooked: /dev/tcp support depends on the compilation option --enable-net-redirections when Bash itself is built. While enabled in most modern Bash builds, some distributions (Debian and its derivatives were historically known to compile Bash without this feature in older releases) may disable it, producing a No such file or directory error on lines attempting to open /dev/tcp. Because availability can vary between systems and across releases, do not assume this feature is present on production servers; test a simple one-liner such as timeout 2 bash -c 'echo > /dev/tcp/example.com/80' && echo available before relying on it in automated scripts, and keep nc -z host port ready as an alternative if netcat is installed on the target machine.

check_port() {
    local host="$1" port="$2" timeout="${3:-3}"
    if timeout "$timeout" bash -c "echo > /dev/tcp/$host/$port" 2>/dev/null; then
        echo "Port $port on $host is open"
        return 0
    else
        echo "Port $port on $host is closed or unresponsive"
        return 1
    fi
}

check_port api.example.com 443

echo > /dev/tcp/$host/$port attempts to establish a connection and immediately close it once echo completes, sufficient to test if a port accepts connections without transmitting or receiving application data. Placing the timeout command at the front is crucial because connection attempts to ports silently blocked by a firewall (where no packet response is sent back) can hang while waiting for system default socket connection timeouts, which often span tens of seconds or more; wrapping it with timeout establishes an explicit threshold controllable by the script. This technique works well for quickly checking whether a database or API service port is reachable from a given server without needing nc pre-installed everywhere.

22.2.2 Generic HTTP Health Check Function

The curl -sf -o /dev/null pattern from 21.4.2 and HTTP_CODE=$(curl -s -o /dev/null -w '%{http_code}' ...) pattern from 20.2.4 can be combined into a single generic function accepting any URL, configured with timeout and retry mechanisms following the --retry approach discussed in 20.2.4.

check_endpoint() {
    local url="$1" timeout="${2:-10}"
    local code
    code=$(curl -s -o /dev/null -w '%{http_code}' \
        --connect-timeout 5 --max-time "$timeout" --retry 2 --retry-delay 2 \
        "$url")
    if [[ "$code" =~ ^2 ]]; then
        echo "OK ($code)"
        return 0
    else
        echo "FAILED ($code)"
        return 1
    fi
}

check_endpoint "https://api.example.com/health"

The condition [[ "$code" =~ ^2 ]] considers an endpoint healthy if the status code begins with the digit 2 (range 2xx, successful), which is more flexible than strictly matching 200, as certain endpoints intentionally return 201 or 204 under normal conditions. If curl fails to connect entirely (host unreachable, DNS resolution failed, timeout reached), the variable code remains an empty string rather than a numeric value, safely falling into the else branch because the regex ^2 does not match empty strings.

22.3 Testing Simple API Connections via Script

The health check in 22.2 only ensures that the endpoint returns a valid status code. For real-world APIs, that is insufficient: an endpoint can return 200 OK while the response payload indicates application-level errors (a common pattern in REST APIs), or it may require authentication before it can be accurately evaluated.

22.3.1 Requesting APIs with Authentication Headers

curl -s -o /dev/null -w '%{http_code}\n' \
    -H "Authorization: Bearer $API_TOKEN" \
    -H "Accept: application/json" \
    "https://api.example.com/v1/ping"

The -H option adds custom HTTP headers and can be repeated for multiple headers as shown above. $API_TOKEN should never be written directly as a literal inside scripts (especially if committed to Git repositories), but should instead be loaded from environment variables or secured configuration files with restricted file permissions, following the same pattern as source /etc/deploy-pipeline/alert-notifikasi.env in 21.4.2 for Telegram tokens. Chapter 23.1 explores security risks regarding credential exposure in scripts and shell history in greater detail.

22.3.2 Validating Response Bodies with jq

Section 20.2.3 introduced jq for safely constructing JSON payloads. Here, jq is applied in reverse to read and validate incoming JSON response bodies returned by APIs.

check_api_status() {
    local url="$1"
    local response status_field

    response=$(curl -s --max-time 10 "$url")
    status_field=$(echo "$response" | jq -r '.status // "unknown"' 2>/dev/null)

    if [[ "$status_field" == "ok" ]]; then
        echo "API healthy: status=$status_field"
        return 0
    else
        echo "API unhealthy: status=$status_field"
        return 1
    fi
}

check_api_status "https://api.example.com/v1/health"

The // operator in jq supplies a fallback value ("unknown") if the .status field is missing or null, preventing status_field from evaluating to an empty string that could complicate log analysis. 2>/dev/null on the jq line suppresses error messages if the response body is not valid JSON at all (e.g., when an API is down and an upstream load balancer returns an HTML error page), ensuring the script does not terminate due to a parsing error that itself indicates API failure. This approach complements check_endpoint() from 22.2.2: HTTP status code 200 confirms the server responded, whereas validating the status field within the body confirms the application reports itself as healthy, two outcomes that do not always align.

22.4 Case Study: Multi-Server Connectivity Check

This closing case study integrates every verification layer covered in this chapter, ping, TCP ports, and HTTP health checks, into a unified script that checks multiple servers concurrently and identifies the specific layer where failure occurs.

22.4.1 Script Design

The script check-connectivity.sh reads target servers from an associative array (Chapter 4.2). Each server is checked sequentially across three layers: (1) ping to verify basic network host availability, (2) TCP port check via the check_port() function from 22.2.1 to confirm the service is listening, and (3) check_endpoint() from 22.2.2 to verify the application returns a healthy response. If an earlier layer fails, subsequent checks for that host are skipped to save execution time rather than running remaining checks unnecessarily.

22.4.2 Implementation of check-connectivity.sh

#!/bin/bash
set -uo pipefail

declare -A SERVERS=(
    [app-01]="10.0.0.11|443|https://10.0.0.11/health"
    [app-02]="10.0.0.12|443|https://10.0.0.12/health"
    [db-01]="10.0.0.21|5432|"
)

LOG_FILE="/var/log/check-connectivity.log"
FAILED=0

log() {
    printf '[%s] %s\n' "$(date '+%Y-%m-%d %H:%M:%S')" "$*" | tee -a "$LOG_FILE"
}

check_port() {
    local host="$1" port="$2" timeout="${3:-3}"
    timeout "$timeout" bash -c "echo > /dev/tcp/$host/$port" 2>/dev/null
}

check_endpoint() {
    local url="$1"
    local code
    code=$(curl -s -o /dev/null -w '%{http_code}' \
        --connect-timeout 5 --max-time 10 --retry 2 --retry-delay 2 "$url")
    [[ "$code" =~ ^2 ]]
}

for name in "${!SERVERS[@]}"; do
    IFS='|' read -r host port url <<< "${SERVERS[$name]}"

    if ! ping -c 1 -W 2 "$host" &> /dev/null; then
        log "FAILED $name ($host): no ping response"
        FAILED=$((FAILED + 1))
        continue
    fi

    if ! check_port "$host" "$port"; then
        log "FAILED $name ($host:$port): port closed"
        FAILED=$((FAILED + 1))
        continue
    fi

    if [[ -n "$url" ]]; then
        if ! check_endpoint "$url"; then
            log "FAILED $name ($host): HTTP endpoint unhealthy"
            FAILED=$((FAILED + 1))
            continue
        fi
    fi

    log "OK $name ($host:$port)"
done

if (( FAILED > 0 )); then
    log "Finished, $FAILED server(s) reported issues"
    exit 1
fi

log "Finished, all servers healthy"

The SERVERS entry format is structured as a pipe-delimited string (host, port, health check URL) because Bash lacks native nested array structures; IFS='|' read -r host port url <<< "..." parses it back into separate variables, utilizing the same underlying parsing strategy applied to CSV processing in Chapter 11.4. The db-01 entry leaves the URL section blank because database engines do not provide HTTP health endpoints, stopping checks at the port validation layer; the block if [[ -n "$url" ]] skips layer three when irrelevant for that target. set -uo pipefail is used without -e for the same reasons outlined in 21.4.2 and 20.4.2: the script must continue inspecting remaining servers even if one server in the middle fails, rather than terminating immediately.

22.4.3 Verification and Result Interpretation

chmod +x check-connectivity.sh
./check-connectivity.sh
echo "Exit status: $?"

Inspect the log entries for each server to pinpoint the precise layer where failure occurred, rather than relying on a generic failure message. A failure at the ping layer points to basic networking issues or host downtime; failure at the port layer suggests a stopped service or a firewall blocking that specific port; failure at the HTTP endpoint layer indicates an active application experiencing internal issues (e.g., an underlying database outage). To test failure pathways, modify a target port value in SERVERS to a known closed port (e.g., 9999) and re-run, verifying that the log records "port closed" for that host and exits with status 1. For production deployment, integrate the final FAILED count variable with send_alert() from 20.4.2 to send notifications when failures occur, and schedule execution via cron or systemd timer (Chapter 18) to automate checks at regular intervals.

This chapter completes the Sysadmin toolset for multi-layered connectivity diagnosis, ranging from low-level network commands to API health checks validating payload contents. Chapter 23 shifts focus to security topics, reusing ss from 22.1.3 to audit unauthorized open public ports, while covering strategies to mitigate command injection and credential leaks introduced in 22.3.1.