Centralized Logging

Chapter 38 concluded with an important limitation regarding metrics: spiking CPU numbers or a status marked as down only answer the question of what happened, not why it happened. Answering the why question is the job of logs. This scenario is very real for Sysadmins managing more than one server. When the application on app01 suddenly throws a 500 error, the Grafana dashboard from Chapter 38 immediately shows a spike in the error rate, but the application log lines explaining the underlying stack trace remain stored locally on the server in question. If there are ten such servers, the Sysadmin must SSH into each one individually to find the exact line, a process that is clearly inefficient during an ongoing incident.

There is a second reason that is no less important, namely security. An attacker who successfully breaches a server almost always takes a follow-up step: deleting or editing local logs so that their tracks are erased. If logs are stored only on that same server, that deletion proceeds smoothly without hindrance. Logs that have been sent and stored on a separate server are far more difficult to manipulate, because the attacker would need to breach two systems simultaneously instead of one. This chapter covers four complementary topics to address both needs: reviewing journalctl in greater depth than what was touched upon in Chapter 5, configuring rsyslog to forward logs to a central server, setting up log rotation with logrotate so the disk does not fill up, and devising a log retention strategy that balances investigation needs with storage capacity.

39.1 Review journalctl: Reading and Managing Systemd Logs in Depth

39.1.1 Journald as the Primary Log Source in Ubuntu Server

systemd-journald is a daemon that collects logs from all systemd services, the kernel, and other processes into one structured place, active by default since the basic Ubuntu Server installation in Chapter 2. Chapter 5 introduced journalctl -u service_name.service to read logs for a single service and the -f option to follow logs in real-time, two commands used most frequently in daily operations. This section goes further: filtering logs based on a combination of units, time ranges, and severity levels (priority) all at once, and then configuring how long and how much local data the journal may retain before older entries are forwarded to the central server via rsyslog in Section 39.2.

39.1.2 Filtering Logs Based on Unit, Time, and Priority

Raw logs accumulating from dozens of services overnight are practically useless without the ability to filter them. journalctl provides combination filters that narrow down searches directly from the command line, much faster than opening log files one by one as was traditional practice in /var/log.

Hands-on Steps

1. Display Nginx service logs from Chapter 12 for the last hour only, using the --since option with a relative time format.

   journalctl -u nginx.service --since "1 hour ago"

2. Narrow down to a specific time range using --since and --until together, useful when investigating an incident with a known time of occurrence, such as an error spike recorded by Grafana at 02:00 AM.

   journalctl --since "2026-08-28 02:00:00" --until "2026-08-28 02:30:00"

3. Display only messages with a severity level of err and above (covering err, crit, alert, and emerg) since the last boot, using the -p and -b options. The eight syslog priority levels in order from lightest to most critical are debug, info, notice, warning, err, crit, alert, and emerg, the same sequence as Nginx error_log levels discussed in Section 12.4.2.

   journalctl -p err -b

4. Combine unit and priority filters simultaneously to narrow down further, for example searching for SSH warnings that could indicate potential brute force attempt activity by an attacker.

   journalctl -u ssh.service -p warning --since today

Verification and Troubleshooting

• The --since format accepts many variations, ranging from relative keywords like "yesterday" and "1 hour ago" to absolute formats like "YYYY-MM-DD HH:MM:SS". Use absolute formats when time precision is critical, such as matching alert timestamps from Grafana in Section 38.4.
• If the results of a -p err filter are empty even though the Sysadmin is certain errors exist, check first whether the application actually writes logs to stderr with the appropriate priority. Many web applications write errors to stdout with a default info priority, causing them never to be caught by high-priority filters.
• Run journalctl -u nginx.service -p err -b | wc -l to count the number of error lines without reading them one by one, a fast way to assess severity before diving into detailed investigation.

39.1.3 Configuring Persistence and Local Journal Size Limits

The journald configuration resides in /etc/systemd/journald.conf, and the Storage= parameter inside determines whether logs are saved permanently or kept only in memory. On Ubuntu Server 26.04, the default value for Storage= is persistent, which automatically creates the /var/log/journal directory from the start and saves logs there, surviving reboots without requiring manual intervention. This differs from older tutorials that taught manual steps for creating the /var/log/journal directory to activate persistence, a step no longer required in this version of Ubuntu Server.

Hands-on Steps

1. Check how much disk space the journal currently consumes first.

   journalctl --disk-usage

2. Open the journald configuration file.

   sudo nano /etc/systemd/journald.conf

3. Add or modify the SystemMaxUse line inside the [Journal] block to limit the maximum size of the persistent journal, for example to 500 MB. Without this explicit limit, journald defaults to using 10% of the /var filesystem capacity (capped at a maximum of 4 GB), which can become overly large for servers with small disks.

   [Journal]
   SystemMaxUse=500M

4. Restart systemd-journald so the new configuration takes effect.

   sudo systemctl restart systemd-journald

Verification and Troubleshooting

• Confirm the new limit is active by repeating journalctl --disk-usage. The figure will not drop immediately below the new limit if the previous journal size was already below it. This limit only prevents further growth, rather than immediately truncating older data.
• To forcefully trim the journal without waiting for automatic limits to kick in, use --vacuum-size or --vacuum-time. The following command purges journal data older than two weeks.

  sudo journalctl --vacuum-time=2weeks

• A candid field note: restricting the local journal too tightly risks losing investigative tracks if the server has not yet forwarded logs to the central server in Section 39.2 when an incident occurs. Consider setting a reasonably generous SystemMaxUse locally, as the true long-term retention strategy is covered thoroughly in Section 39.4.

39.2 rsyslog: Forwarding Logs to a Centralized Server

39.2.1 Why Logs Need to Be Centralized

rsyslog is a modern syslog implementation installed by default on Ubuntu Server, responsible for capturing logs from journald and writing them to classic files like /var/log/syslog and /var/log/auth.log through rules defined in /etc/rsyslog.d/50-default.conf. The capability that makes rsyslog relevant for this chapter is forwarding, which sends copies of those logs to another server over the network, transforming rsyslog from a mere local recorder into the backbone of centralized logging. This section puts a real-world scenario into practice: server app01 (192.168.1.40), introduced in Chapter 36 for Ansible practice, now forwards all its logs to the main server (192.168.1.10), which acts as the log collection center.

39.2.2 Setting Up the Log Receiver Server

Hands-on Steps

1. From the main server (192.168.1.10), create the directory where client logs will be stored, then assign its ownership to the syslog user and adm group, matching the $PrivDropToUser and $PrivDropToGroup configurations set by default in /etc/rsyslog.conf.

   sudo mkdir -p /var/log/remote
   sudo chown syslog:adm /var/log/remote

2. Create a new configuration file specifically for receiving client connections.

   sudo nano /etc/rsyslog.d/10-remote-server.conf

3. Populate it with the imtcp module to enable the TCP listener, then define a template that saves each client log to a separate file based on the sender hostname. This listener is deliberately bound to a dedicated ruleset named remote-ruleset, rather than being left to use the default ruleset.

   module(load="imtcp")
   
   $template RemoteLogs,"/var/log/remote/%HOSTNAME%.log"
   
   ruleset(name="remote-ruleset") {
   	action(type="omfile" dynaFile="RemoteLogs")
   }
   
   input(type="imtcp" port="514" ruleset="remote-ruleset")

   Binding to a separate ruleset is not merely a formatting style, but a crucial detail that frequently becomes a pitfall in production. Without the ruleset="remote-ruleset" parameter, logs entering via port 514 are processed using the default ruleset alongside the server's own local logs, causing the main server's own logs to be written into /var/log/remote/ and potentially failing to reach its local /var/log/syslog and /var/log/auth.log versions. This separate ruleset ensures that incoming logs from clients via imtcp are processed independently without touching the local log pipeline, removing the need for an extra stop directive.
4. Restart rsyslog so the new configuration is loaded.

   sudo systemctl restart rsyslog

5. Open port 514/tcp in UFW, restricted exclusively to the local subnet following the same restriction pattern as PostgreSQL in Section 18.4.2 and Netdata in Section 38.2.3.

   sudo ufw allow from 192.168.1.0/24 to any port 514 proto tcp

Verification and Troubleshooting

• Ensure rsyslog is actively listening on port 514.

  sudo ss -tlnp | grep 514

• If the service fails to restart, check the configuration syntax first before attempting to guess the cause.

  sudo rsyslogd -N1

• Confirm that local logs on the main server itself remain normal after this change, and not just the incoming client logs. Send a test message from the main server itself, then ensure that message still appears in /var/log/syslog rather than straying into /var/log/remote/.

  logger "Main server local log test message"
  sudo tail -n 5 /var/log/syslog

  If this message appears in /var/log/remote/ instead, it indicates that the ruleset parameter on the input() line was not configured correctly.

39.2.3 Configuring Clients to Forward Logs

Hands-on Steps

1. From app01 (192.168.1.40), create the forwarding configuration file.

   sudo nano /etc/rsyslog.d/60-forward.conf

2. Add a single line that forwards all facilities and priorities (*.*) to the main server over TCP. The double @@ symbol denotes the more reliable TCP protocol, whereas a single @ represents UDP, which is lighter but offers no delivery guarantees.

   *.* @@192.168.1.10:514

3. Restart rsyslog on app01.

   sudo systemctl restart rsyslog

4. Send a test message from app01 using logger, a built-in utility for writing messages directly to syslog from the command line.

   logger -t forwarding-test "Test message for forwarding to central log server"

Verification and Troubleshooting

• Return to the main server and check if the app01 log file has appeared and contains the test message.

  sudo tail -f /var/log/remote/app01.log

  The output should display a line containing the forwarding-test tag along with the message just sent.
• If the app01.log file never appears, the most effective troubleshooting sequence is: verify first that UFW on the main server permits port 514 (Section 39.2.2), then test raw network connectivity from app01 without involving rsyslog at all.

  nc -zv 192.168.1.10 514

• An honest note on security: the setup above transmits logs in unencrypted plain text, which is sufficiently secure for this lab environment since it operates on a trusted local network like 192.168.1.0/24. For forwarding across public networks or separate data centers, rsyslog supports TLS transport via the gtls module, an advanced topic not covered in depth here as it requires dedicated certificates outside of the Let's Encrypt setup used for web services in Chapter 17.
• Logs stored on the main server are now significantly more resilient against tampering or deletion attempts by an attacker compromising app01, as copies are secured on a separate system as soon as the log lines are transmitted, even if app01 itself is later fully compromised.

39.3 Log Rotation with logrotate

39.3.1 How logrotate Works and Its Execution Schedule

logrotate is a built-in Ubuntu tool that truncates, compresses, and deletes old log files on a schedule, preventing files like /var/log/syslog from growing indefinitely and consuming all disk space. Its global configuration resides in /etc/logrotate.conf, containing default rules such as weekly rotation and retaining rotate 4 generations of backlog, while individual packages needing custom rules place their own configuration files in the /etc/logrotate.d/ directory, exactly like the Nginx setup mentioned in Section 12.4.2 and the default rsyslog configuration managing rotation for /var/log/syslog and /var/log/auth.log. Execution no longer relies on cron.daily as in older practices, but is instead triggered by the systemd timer logrotate.timer, which runs daily with a randomized delay (RandomizedDelaySec) so that multiple servers on the same network do not perform rotation at the exact same second.

Verification and Troubleshooting

• Confirm that this timer is active and check when the next execution is scheduled.

  systemctl list-timers logrotate.timer

39.3.2 Creating Custom logrotate Configurations

New logs appearing in /var/log/remote/*.log from forwarding in Section 39.2 do not yet have rotation rules, because these files are not part of any system package that ships with default logrotate configurations. Without additional rules, these files will grow continuously alongside the log volume from app01 without ever being rotated.

Hands-on Steps

1. Create a new configuration file in /etc/logrotate.d/.

   sudo nano /etc/logrotate.d/remote-logs

2. Add rules for weekly rotation, retaining 12 generations (roughly three months), and compressing older files to save disk space.

   /var/log/remote/*.log {
   	weekly
   	rotate 12
   	missingok
   	notifempty
   	compress
   	delaycompress
   	create 0640 syslog adm
   	sharedscripts
   	postrotate
   		/usr/lib/rsyslog/rsyslog-rotate
   	endscript
   }

   The postrotate line executes the same default script used by the package's /etc/logrotate.d/rsyslog configuration, sending a HUP signal to the rsyslog service so that it stops writing to the old renamed file and begins writing to the new file. Without this step, rsyslog would continue writing to the old file handle that no longer has a name on the filesystem, while the new file would remain empty despite rotation being marked complete.

Verification and Troubleshooting

• The delaycompress option intentionally postpones compression of the newly rotated file until the next rotation cycle. This avoids a race condition where rsyslog might still be writing to the old renamed file before its write process has completely finished.
• The permission 0640 syslog adm on the create line must match the original permissions of the /var/log/remote/*.log files. New files created with overly permissive file modes after rotation indicate that the create rule is incorrect and must be adjusted.

39.3.3 Testing Rotation Without Waiting for Schedules

Waiting a full week just to verify if a new configuration works is impractical. logrotate provides two options to test configurations at any time without altering the actual schedule.

Hands-on Steps

1. Run logrotate in debug mode (-d) first, which only displays a simulation of actions without executing them, making it safe to run at any time.

   sudo logrotate -d /etc/logrotate.d/remote-logs

2. Once the simulation output matches expectations, force an actual rotation using the -f option, ignoring file size and schedule requirements that are normally mandatory.

   sudo logrotate -f /etc/logrotate.d/remote-logs

Verification and Troubleshooting

• Verify that the newly compressed file has appeared in the directory.

  ls -la /var/log/remote/

  The output should display new files like app01.log.1.gz alongside an empty app01.log ready to accept new entries.
• An important note when repeating tests: logrotate records the last rotation timestamp for each file in the state file /var/lib/logrotate/status. Forcing rotation with -f updates this timestamp, ensuring that subsequent weekly rotations via logrotate.timer remain properly scheduled rather than running immediately upon the next timer trigger.

  grep remote /var/lib/logrotate/status

39.4 Log Retention Strategy

39.4.1 Determining Retention Duration Based on Requirements

Log retention is a policy determining how long logs are kept before being permanently removed, a decision that rarely has a single right answer because different log types serve different needs. Storing logs for too short a period risks losing crucial evidence when investigating incidents, while storing them indefinitely inflates storage costs without proportional benefits. The three main factors determining appropriate retention duration are security investigation requirements (how long it typically takes for an incident to be discovered from its initial occurrence), compliance obligations (standards like PCI DSS mandate keeping audit logs for at least 12 months, with recent months immediately accessible without restoring from archives), and available storage capacity.

The numbers in the table above represent standard baseline practices, not rigid rules. Sysadmins must confirm actual retention requirements with their organization's legal or compliance teams, especially for logs directly containing user or client data, before establishing them as official policy.

39.4.2 Archiving Older Logs to Separate Storage

Extended retention on the main server is not the only option. A more efficient pattern involves retaining recent logs on the main server for quick investigation, while archiving rotated and compressed logs to separate storage for long-term retention. This reduces disk load on production servers while adding another layer of defense against log deletion by an attacker.

Hands-on Steps

1. Copy compressed log files resulting from rotation in Section 39.3.3 to a separate archive server using rsync, a tool that will be covered comprehensively as the foundation for backup strategies in Chapter 40.

   sudo rsync -az /var/log/remote/*.gz [email protected]:/archive/logs/

Verification and Troubleshooting

• Schedule the rsync command above using a .timer and .service pair, following the same pattern as the backup-harian.timer unit practiced in Section 5.3.1, so the archival process runs automatically without manual weekly execution.
• Ensure permissions and SSH access to the archive server rely on key-based authentication rather than passwords, following the SSH key setup discussed in Section 3.2, enabling the process to run automatically via script without manual prompts.

At this point, logs that were previously scattered and vulnerable on individual servers are now centralized in one place via rsyslog, automatically rotated by logrotate to prevent disk exhaustion, and managed under a clear retention policy regarding how long each log type should be stored. Nevertheless, these organized central logs remain data on a single server, equally vulnerable to hardware failures or disasters as any other system. Chapter 40 builds on this foundation with a comprehensive server backup strategy, including backing up the centralized logs established in this chapter, ensuring that all critical data, not just logs, remains protected from permanent loss.