Alerting & Notifications from Scripts

The dashboard in the Chapter 19.4 case study can already display server status in real time and record every anomaly via the log() function to a file. However, that workflow remains passive: [WARN] entries merely sit inside the log file until a Sysadmin happens to open the dashboard or run tail -f. If an incident occurs at three in the morning while everyone is sleeping, a full disk or a dead service could be left unattended for hours before anyone notices. This chapter transforms that passive pattern into an active one through two complementary concepts: alerting, the mechanism of detecting conditions that require attention and deciding when an alert is worthy of being sent, and notifications, the method by which that message reaches the right people through communication channels they actively monitor. The following four sub-chapters cover the three notification channels most commonly used by Sysadmins: email via mail/sendmail/SMTP, generic webhooks that serve as the foundational format for almost all modern integrations, as well as the Telegram Bot API and Slack Incoming Webhook which are popular because their messages appear directly on mobile phones. The concluding case study connects everything to the disk and service checks from Chapter 19.3 into a single automated alert system ready to be scheduled.

20.1 Email Notifications

Email remains the most common alert channel because almost every server comes with a built-in email delivery path, eliminating the need to sign up for a new third-party service. The following three sub-chapters discuss three ways to send emails, from the simplest approach to sending directly to an SMTP server.

20.1.1 MTA, MUA, and the Email Delivery Flow

It is important to understand these two terms first so that troubleshooting failed email deliveries does not turn into guesswork. MUA (Mail User Agent) is the program used to compose messages; in the context of scripts, this refers to commands like mail or sendmail. MTA (Mail Transfer Agent), such as Postfix, Exim, or original Sendmail, is the service actually tasked with routing and delivering that message to its destination over the network, whether directly to the internet or via a third-party SMTP provider smarthost. When a script calls mail or sendmail, that command is actually just handing off the message to the local MTA already running on the server, not directly sending it to the internet itself. If the MTA is not yet installed or configured for outbound relaying, the mail command might still return an exit status of 0 as if successful, even though the message is merely stuck in the local queue and never delivered. mailq (or postqueue -p specifically for Postfix) displays the contents of that queue, making it the first place to check if an alert is "successfully sent" but never appears in the inbox.

20.1.2 Quick Sending with mail/mailx

For short single-line alerts, the mail command (often also available as mailx) is sufficient without needing to construct manual headers. The name of the providing package varies across distributions: Debian/Ubuntu use mailutils (a GNU Mailutils implementation) or the bsd-mailx alternative, while RHEL/Fedora/Rocky Linux version 9 and above use s-nail as a replacement for the old mailx. Install one according to your distribution before proceeding.

sudo apt install mailutils        # Debian/Ubuntu
sudo dnf install s-nail           # RHEL/Fedora/Rocky Linux 9+

echo "Root disk is 92% full" | mail -s "[ALERT] Disk Full - $(hostname)" [email protected]

The -s option populates the subject and is consistently available across all implementations above, while the message body is read from stdin via a pipe. Options for file attachments vary between implementations (some use -a, others use -A), so check via man mail on the relevant server before relying on them; for the simple alerting needs in this chapter, file attachments are intentionally omitted because a short text message is sufficient.

20.1.3 Manual Header Formatting via sendmail -t

sendmail is the compatible command name provided by almost all modern MTAs (Postfix, Exim, and original Sendmail itself), usually installed at /usr/sbin/sendmail. Without additional options, recipients must be specified via command-line arguments; the -t option instructs it to extract the recipient list from the To:, Cc:, and Bcc: headers written directly inside the message via stdin, with the Bcc: header automatically stripped before the message is actually sent so that other recipients cannot see it. The Subject: header itself is always parsed as part of the message regardless of the -t option; what this option affects is purely where the recipient list is gathered from. This combination allows a complete message, headers and body combined, to be constructed in a single command using printf without needing to specify recipient addresses twice.

printf 'To: [email protected]\nSubject: [ALERT] Service Down - %s\n\n%s\n' "$(hostname)" "$PESAN" | sendmail -t

A blank line between headers and body (resulting from \n\n in the middle of the printf format) is mandatory, strictly following standard email message format; without that blank line, some MTAs will treat the entire body text as part of the header and fail to process it.

20.1.4 Direct SMTP Without a Local MTA

Sometimes a server does not have a full MTA installed, such as inside a minimal container, but still needs to send alerts. msmtp, a lightweight SMTP client designed as a replacement for sendmail, is a lighter choice compared to installing full Postfix. Its configuration is stored in ~/.msmtprc, and because it contains credentials, its permissions must be restricted to be readable only by its owner.

cat > ~/.msmtprc <<'EOF'
defaults
tls on
tls_trust_file /etc/ssl/certs/ca-certificates.crt

account alert
host smtp.example.com
port 587
auth on
user [email protected]
password isi_password_atau_app_password
from [email protected]

account default : alert
EOF
chmod 600 ~/.msmtprc

printf 'To: [email protected]\nSubject: [ALERT] Disk Full\n\nRoot disk is 92%%.\n' | msmtp [email protected]

If you prefer not to install msmtp either, curl itself supports built-in SMTP/SMTPS protocols, requiring only uploading the message content as a file (or via stdin using the - symbol) to the target SMTP server.

printf 'From: [email protected]\nTo: [email protected]\nSubject: [ALERT] Disk Full\n\nRoot disk is 92%%.\n' | curl --ssl-reqd "smtp://smtp.example.com:587" \
    --mail-from "[email protected]" \
    --mail-rcpt "[email protected]" \
    --user "[email protected]:$SMTP_PASSWORD" \
    --upload-file -

--mail-from and --mail-rcpt define the sender and recipient at the SMTP protocol level, --upload-file - reads the complete message body from stdin, and --ssl-reqd requires the connection to be upgraded to TLS (STARTTLS) on port 587; for servers using port 465, change the scheme to smtps:// which is encrypted from the start of the connection without needing --ssl-reqd. Placing plain $SMTP_PASSWORD inside a script is clearly not a safe habit; 20.4.3 discusses how to store credentials like this in a separate file with restricted permissions.

20.2 Generic Webhooks for Alerting

This sub-chapter discusses the most flexible integration pattern: sending data via HTTP POST to an endpoint, a format understood by almost all modern monitoring tools, automation suites, and collaboration platforms.

20.2.1 The Concept of Webhooks for Alerts

A webhook is an HTTP endpoint provided by a service to receive push notifications from another system as soon as an event occurs, opposite to standard APIs that must be queried repeatedly via polling. A script simply sends a single POST request containing a JSON payload when a problematic condition is detected, without requiring the destination service to constantly ask "is there anything new?". Many incident management platforms (PagerDuty, Opsgenie), automation tools (n8n, Zapier, Make), as well as custom in-house team systems provide generic webhook endpoints that accept payloads with similar formats: a single JSON object containing status, alert source, and message.

20.2.2 POST Payload JSON with curl

curl -X POST "https://webhook.example.com/alert" \
    -H "Content-Type: application/json" \
    -d '{"status":"critical","host":"web01","message":"Disk root 92% penuh"}'

-X POST explicitly sets the HTTP method used, -H adds a header so the endpoint knows the body payload is JSON, and -d sends that data as the request body.

20.2.3 Safe Escaping with jq

Constructing JSON by concatenating strings manually like the example above easily breaks whenever inserted variables contain double quotes, newlines, or Unicode characters, especially if part of that message body comes from log outputs that cannot be fully controlled. If the variable content happens to be influenced by external parties (for example, a suspicious filename recorded in logs and inserted raw into an alert message), an Attacker could potentially break the JSON structure or inject unintended additional fields. jq -n (null input mode, constructing output from scratch without requiring existing JSON input) combined with the --arg option becomes a safe way to assemble payloads because every value is automatically escaped according to JSON rules.

PAYLOAD=$(jq -n \
    --arg host "$(hostname)" \
    --arg pesan "$PESAN" \
    '{status: "critical", host: $host, message: $pesan}')

curl -X POST "https://webhook.example.com/alert" \
    -H "Content-Type: application/json" \
    -d "$PAYLOAD"

--arg host "$(hostname)" registers a variable named $host inside the jq filter with content from the shell side, then the filter '{status: ..., host: $host, ...}' uses it as a string value guaranteed to be valid JSON, regardless of any unusual characters inside it.

20.2.4 Retries and Response Validation

An alert script that does not verify its own transmission output easily fails silently, which is precisely the problem discussed regarding scheduled jobs in Chapter 19.2. curl provides several built-in options to handle this without requiring complex extra logic.

HTTP_CODE=$(curl -s -o /dev/null -w '%{http_code}' \
    --retry 3 --retry-delay 2 --max-time 10 \
    -X POST "https://webhook.example.com/alert" \
    -H "Content-Type: application/json" \
    -d "$PAYLOAD")

if [[ "$HTTP_CODE" -ge 200 && "$HTTP_CODE" -lt 300 ]]; then
    echo "Notification sent"
else
    echo "Failed to send notification, HTTP $HTTP_CODE" >&2
fi

-s suppresses the progress bar, -o /dev/null discards the response body since only the status code is needed, and -w '%{http_code}' prints just the HTTP status code to stdout. --retry 3 makes curl automatically retry up to 3 times specifically for temporary errors, such as timeouts or HTTP codes 408/429/5xx, with delays between attempts configured via --retry-delay. --max-time 10 caps the total wait time for a single request so that the alert script itself does not hang indefinitely if the target endpoint is slow to respond.

20.3 Telegram Bots and Slack Webhooks

These two platforms are popular for alerts because their messages trigger direct phone notifications, noticed far faster than emails that are opened only periodically.

20.3.1 Telegram Bots and Chat IDs

Telegram bots are created via @BotFather, Telegram's official bot: send the command /newbot then follow the prompts to assign a name; the end result is a token formatted as numbers:letters-numbers serving as that bot's authentication key. Treat this token strictly like a password; anyone possessing it can control the bot entirely. In addition to the token, notifications require a target chat ID; the most practical way to get it is to send any message to the bot first, then call the getUpdates method via curl to view the chat.id in the JSON response.

curl "https://api.telegram.org/bot$TELEGRAM_BOT_TOKEN/getUpdates"

20.3.2 Sending Messages via sendMessage

The sendMessage method on the Telegram Bot API requires only two mandatory parameters: the target chat_id and message body text, sent via an endpoint that embeds the bot token directly in the URL.

curl -s -X POST "https://api.telegram.org/bot$TELEGRAM_BOT_TOKEN/sendMessage" \
    -d chat_id="$TELEGRAM_CHAT_ID" \
    -d text="[ALERT] Root disk $(hostname) is 92% full" \
    -d parse_mode="HTML" > /dev/null

The parse_mode parameter is optional, allowing simple text formatting like bold or italic using basic HTML tags (or Markdown/MarkdownV2 syntax) inside the text body. The -d option in curl submits data as application/x-www-form-urlencoded, one of four formats supported by Bot API endpoints alongside JSON, URL query strings, and multipart/form-data.

Every Bot API method, including sendMessage, always responds with a JSON object featuring an ok field: holding a value of true on success, or false accompanied by a description field containing failure details when it fails. Simply pipe the curl output to jq -r '.ok, .description' (the awk pattern-action model from Chapter 11 could be used similarly, but jq is more suitable for JSON) if the script needs to programmatically determine whether delivery succeeded. One common pitfall encountered in production when first integrating a bot with alerting: Telegram does not allow bots to initiate messages to a user who has never started a conversation with them first, a default platform privacy policy. If the chat_id is correct but the response comes back as {"ok":false,...,"description":"Forbidden: bot can't initiate conversation with a user"}, the cause is not a script bug, but rather that the target account has never clicked Start or sent a message to the bot before.

20.3.3 Creating a Slack Incoming Webhook

Slack Incoming Webhooks are created via the Slack App dashboard: create a new App on the Slack developer page, choose the target workspace, enable the Incoming Webhooks feature, then click "Add New Webhook to Workspace" and select the channel that will receive messages. The result is a unique URL formatted as https://hooks.slack.com/services/T000.../B000.../xxxx.... This URL is self-authenticating, meaning anyone holding it can directly send messages to the selected channel without extra authentication, so treat it like a password and never commit it to a public repository.

20.3.4 Sending Messages to Slack

PAYLOAD=$(jq -n --arg teks "[ALERT] Root disk $(hostname) is 92% full" '{text: $teks}')
curl -X POST "$SLACK_WEBHOOK_URL" -H "Content-Type: application/json" -d "$PAYLOAD"

The minimal payload required is just a single field, text, matching the message you wish to display in that channel. The pattern of constructing payloads via jq -n --arg is intentionally reused here just like in 20.2.3, because alert message contents similarly come from variables that could contain characters breaking JSON if concatenated manually.

Successful requests are answered by Slack with an HTTP 200 code and a plain text body reading ok, not JSON. Failed requests reply with a 400 code alongside a brief error message in the body, such as invalid_payload if the submitted JSON is invalid, channel_not_found if the target channel was deleted, or no_service if the webhook URL was revoked via App settings. The checking pattern via HTTP_CODE in 20.2.4 applies identically for validating delivery to Slack, requiring only updating the target URL to $SLACK_WEBHOOK_URL.

20.4 Case Study: Automated Service Down/Disk Full Alerts

This concluding case study ties together the cek_disk and cek_service checks from Chapter 19.3 with the three notification channels just covered into a single alert-notifikasi.sh script ready to be scheduled via cron.

20.4.1 Notification Dispatcher Design

Credentials (bot tokens, webhook URLs, target email addresses) are stored in a separate configuration file, alert-notifikasi.env, rather than being hardcoded into the script. The kirim_alert() function acts as a dispatcher: receiving a single message, then passing it to every delivery function (kirim_telegram, kirim_slack, kirim_email) which each check whether their credential variables are populated before actually sending. This design allows Sysadmins to freely activate a single channel or all of them simultaneously simply by populating or clearing the configuration file, without modifying script code.

20.4.2 alert-notifikasi.sh Implementation

Create the credential configuration file first, then restrict its permissions just like ~/.msmtprc in 20.1.4.

TELEGRAM_BOT_TOKEN="123456789:ABCdefGhIJKlmNoPQRsTUVwxyZ"
TELEGRAM_CHAT_ID="987654321"
SLACK_WEBHOOK_URL="https://hooks.slack.com/services/T000/B000/xxxxxxxxxxxxxxxxxxxxxxxx"
EMAIL_TUJUAN="[email protected]"

chmod 600 alert-notifikasi.env

#!/bin/bash
set -uo pipefail

DIR_SCRIPT=$(dirname "$(readlink -f "$0")")
source "$DIR_SCRIPT/alert-notifikasi.env"

SERVICES=("nginx" "cron")
LOG_FILE="alert-notifikasi.log"
STATE_DIR="/tmp/alert-notifikasi-state"
COOLDOWN_DETIK=1800
mkdir -p "$STATE_DIR"

log() {
    local level="$1"
    shift
    local waktu
    waktu=$(date '+%Y-%m-%d %H:%M:%S')
    printf '[%s] [%s] %s\n' "$waktu" "$level" "$*" >> "$LOG_FILE"
}

kirim_telegram() {
    local pesan="$1"
    [[ -z "${TELEGRAM_BOT_TOKEN:-}" ]] && return 0
    curl -s -X POST "https://api.telegram.org/bot$TELEGRAM_BOT_TOKEN/sendMessage" \
        -d chat_id="$TELEGRAM_CHAT_ID" \
        --data-urlencode "text=$pesan" > /dev/null
}

kirim_slack() {
    local pesan="$1"
    [[ -z "${SLACK_WEBHOOK_URL:-}" ]] && return 0
    local payload
    payload=$(jq -n --arg teks "$pesan" '{text: $teks}')
    curl -s -X POST "$SLACK_WEBHOOK_URL" -H "Content-Type: application/json" -d "$payload" > /dev/null
}

kirim_email() {
    local pesan="$1"
    [[ -z "${EMAIL_TUJUAN:-}" ]] && return 0
    echo "$pesan" | mail -s "[ALERT] $(hostname)" "$EMAIL_TUJUAN"
}

sudah_cooldown() {
    local kunci="$1"
    local file_state="$STATE_DIR/$kunci"
    local sekarang
    sekarang=$(date +%s)
    if [[ -f "$file_state" ]]; then
        local terakhir
        terakhir=$(cat "$file_state")
        (( sekarang - terakhir < COOLDOWN_DETIK )) && return 1
    fi
    echo "$sekarang" > "$file_state"
    return 0
}

kirim_alert() {
    local kunci="$1"
    local pesan="$2"
    log WARN "$pesan"
    if sudah_cooldown "$kunci"; then
        kirim_telegram "$pesan"
        kirim_slack "$pesan"
        kirim_email "$pesan"
    else
        log INFO "Alert '$kunci' still in cooldown, notification skipped"
    fi
}

cek_disk() {
    local persen
    persen=$(df -h / --output=pcent | tail -n 1 | tr -d ' %')
    if (( persen >= 80 )); then
        kirim_alert "disk_root" "Root disk $(hostname) is ${persen}% full"
    fi
}

cek_service() {
    local svc
    for svc in "${SERVICES[@]}"; do
        if ! systemctl is-active --quiet "$svc"; then
            kirim_alert "service_$svc" "Service $svc on $(hostname) is inactive"
        fi
    done
}

cek_disk
cek_service

DIR_SCRIPT uses readlink -f "$0" to resolve the absolute path of the script itself, regardless of the working directory when executed, ensuring alert-notifikasi.env is always found in the same folder even when invoked via relative paths from cron. Each kirim_* function first evaluates its variable using [[ -z "${VAR:-}" ]] (Chapter 6.2), using ${VAR:-} so it does not error out if the variable is left unset in the configuration file. kirim_telegram intentionally uses --data-urlencode rather than plain -d as seen in 20.3.2, because this option automatically encodes special characters inside $pesan per URL standards, making it safer for dynamic message payloads. set -uo pipefail at the top enables a subset of debug modes covered thoroughly in Chapter 25.1; -e is intentionally omitted here because if cek_disk encounters an error mid-way, cek_service below it must still execute rather than aborting the entire script.

20.4.3 Cooldown: Preventing Alert Spam

If this script is scheduled to run every 5 minutes via cron and a full disk state persists for an hour, without an extra guard mechanism a Sysadmin could be flooded with 12 identical notifications in the same channel, a common pitfall when connecting monitoring to alerts. The sudah_cooldown() function saves a timestamp of the last delivery for each alert type (distinguished by a kunci key, such as disk_root or service_nginx) in a small file inside STATE_DIR. As long as the time difference between now and that timestamp remains under COOLDOWN_DETIK (1800 seconds or 30 minutes in the example above), notification delivery is skipped even though the condition is still logged via log WARN. This COOLDOWN_DETIK figure is not a hard rule, but rather a trade-off to adjust: setting it too short causes annoying repeating alerts, while setting it too long risks silencing a secondary alert that is actually a distinct event (such as a disk filling up again after briefly clearing).

20.4.4 Verification and Scheduling

Grant execute permissions to both files, then run manually to confirm there are no errors before scheduling.

chmod +x alert-notifikasi.sh
chmod 600 alert-notifikasi.env
./alert-notifikasi.sh
tail -f alert-notifikasi.log

Test problematic scenarios intentionally, such as temporarily stopping one of the services listed in SERVICES (remember to start it again after testing, just like the reminder in Chapter 19.4.3), then verify that notifications actively arrive in Telegram, Slack, or configured email inboxes, rather than merely logging to a file. Once verified, schedule via crontab -e following Chapter 18.1 patterns, using absolute paths so cron correctly locates the configuration file and script.

*/5 * * * * /full/path/to/alert-notifikasi.sh

The alerting system built in this chapter complements the logging and monitoring foundation from Chapter 19: problematic conditions are now not only recorded, but actively reported. Chapter 21 continues to another side of server administration: automated backups, secure file transfers, and deployments; the same kirim_alert() function can be reused directly there to notify Sysadmins if backup or deployment processes fail mid-way.