Basic Audit and Compliance

Chapter 33 concludes with a candid admission that AppArmor has one fundamental limitation: its profile only records access violations from a technical standpoint, without ever answering the very question most frequently asked by auditors or incident response teams, which is who logged in, what commands were executed, and which files were modified within a specific timeframe. This chapter concludes Part VIII through the fifth and final layer of the defense in depth map that we built in Section 30.1.2, namely auditing and least privilege, a layer that operates under a different assumption than the previous four layers: not to prevent or limit an Attacker, but to ensure that every activity on the server is genuinely recorded and every account holds only the exact permissions required for its role.

The scenario is also concrete. One morning, a Sysadmin discovers an unfamiliar line in /etc/sudoers.d/ that was never created by anyone on the team, or an Nginx configuration file from Chapter 12 that suddenly changed without any Developer claiming responsibility for it. Without adequate audit logs, the question of "who and when" is nearly impossible to answer, because standard application logs like the ones we will discuss in depth in Chapter 39 only record what happened from the application's own perspective, not which Linux user triggered it through another process. auditd closes this gap through kernel-level recording that binds every system call to the identity of the user that triggered it, while the principle of least privilege ensures that the number of accounts capable of triggering suspicious activity is as small as possible from the outset.

We will begin with an introduction to auditd and the reasons why this kernel log differs from standard application logs, followed by writing audit rules to monitor sensitive files such as /etc/passwd and /etc/sudoers, practicing how to read the results using ausearch and aureport, reviewing the principle of least privilege that we have actually practiced since Chapter 4 from an audit perspective, auditing accounts and SUID binaries that could potentially become vulnerabilities, until finally concluding Part VIII as a whole through a basic security checklist that summarizes all layers since Chapter 30.

34.1 Audit Logs with auditd: An Introduction

Before diving into the practice of writing rules, it is helpful to first align our understanding of auditd's position relative to the logs we already know, because the two are often confused even though they answer different questions.

34.1.1 Why Standard Application Logs Are Not Enough for Audit Needs

journalctl and application logs like /var/log/nginx/access.log record what happens from the perspective of each individual application: which HTTP requests came in, which service failed to start. Logs like these are very useful for troubleshooting, as we have practiced repeatedly since Chapter 12, but they are not designed to answer audit- or compliance-style questions: precisely which Linux user read the /etc/shadow file last night, or whose process modified the contents of /etc/sudoers.d/10-developer from Section 4.2.2.

auditd (Linux Audit Daemon) operates at a much deeper layer, directly within the kernel via the Linux Audit Framework subsystem. Every time a process calls a monitored system call, such as opening a specific file or executing execve, the kernel records that event along with the UID, PID, and precise timestamp, regardless of which application triggered it. Because recording occurs in the kernel rather than in application space, audit logs are much harder to tamper with than standard application logs, a characteristic that often makes auditd a mandatory requirement in compliance standards like PCI DSS or CIS Benchmarks for servers storing sensitive Client data.

One important point to be honest about from the start: auditd is purely responsible for recording, not for preventing or blocking anything. Its role complements, rather than replaces, the four layers we built from Chapter 30 through Chapter 33. The firewall from Chapter 31 filters traffic, Fail2ban from Chapter 32 blocks suspicious IPs, AppArmor from Chapter 33 restricts process actions, while auditd merely ensures that all of that activity, including activity that escapes the previous four layers, remains neatly logged for future investigations or audit reports.

34.1.2 Installation and Components of auditd

The auditd package is not installed by default in Ubuntu Server 26.04 LTS. This package includes three main components that we will frequently use: the auditd daemon itself which writes events to logs, auditctl for managing rules directly, and a pair of search tools, ausearch and aureport, for reading recorded output without manually parsing raw logs.

Practical Steps

1. Install the auditd package.

   sudo apt update
   sudo apt install auditd

2. Check the service status after installation completes.

   sudo systemctl status auditd

3. Confirm that the audit module is indeed active at the kernel level.

   sudo auditctl -s

Verification and Troubleshooting

• Installing the auditd package automatically enables and starts its service, so the second step should immediately display active (running) without needing additional enable or start commands.
• Healthy output from auditctl -s displays the line enabled 1, indicating that the kernel is currently receiving and recording audit events. If this line displays enabled 0, there is likely a kernel parameter audit=0 intentionally added to GRUB, a condition that rarely has a valid reason on a production server that must meet compliance requirements.
• Unlike AppArmor in Section 33.1.1 which displays active (exited) because its job finishes as soon as profiles are loaded, auditd must indeed remain active (running) indefinitely, as this daemon continuously receives and writes events from the kernel to disk for as long as the server is running.

34.1.3 Writing Audit Rules: Monitoring Sensitive Files

Without rules, auditd is installed but records nothing specific. Audit rules define which files, directories, or syscalls are monitored. File-based rules are called watches, written using the -w flag for the watched path, -p for the types of access logged (a combination of r read, w write, x execute, a attribute change), and -k to assign a search label to the rule for easy querying later via ausearch.

Rules added directly via auditctl only persist until the next reboot. For rules that need to survive permanently, such as audit requirements on a production server, we write them to files in the /etc/audit/rules.d/ directory, following the same split-directory pattern as sudoers.d/ in Section 4.2.2 and jail.local in Section 32.2.2, and then load them using augenrules.

Practical Steps

1. Add a temporary rule to try monitoring changes to /etc/passwd.

   sudo auditctl -w /etc/passwd -p wa -k identity_changes

2. Create a permanent rule file monitoring several sensitive targets at once: account files (/etc/passwd, /etc/shadow, /etc/group), sudoers rules from Section 4.2.2, and SSH configuration from Chapter 3.

   sudo nano /etc/audit/rules.d/50-identity-access.rules

   Populate the file with the following lines.

   -w /etc/passwd -p wa -k identity_changes
   -w /etc/shadow -p wa -k identity_changes
   -w /etc/group -p wa -k identity_changes
   -w /etc/sudoers -p wa -k privilege_changes
   -w /etc/sudoers.d/ -p wa -k privilege_changes
   -w /etc/ssh/sshd_config -p wa -k sshd_changes

3. Reload all rules from /etc/audit/rules.d/ so they become permanent and persist across reboots.

   sudo augenrules --load

4. Confirm that all rules, both temporary and permanent, are loaded in the kernel.

   sudo auditctl -l

Verification and Troubleshooting

• The output of auditctl -l should display each rule line exactly as written in the .rules file, marked with its respective -k label.
• Test a rule directly by triggering a small change on a monitored file, such as adding a comment in /etc/ssh/sshd_config via sudo nano /etc/ssh/sshd_config and saving it, then ensure this event appears when searching logs by the key sshd_changes in Section 34.1.4 next.
• If augenrules --load displays syntax error messages, re-check every line in the .rules file: the -p flag only accepts combinations of the letters r, w, x, a without spaces, and each line may only contain a single rule.
• An excessive number of watch rules, especially on highly active directories like /var/www/, can degrade server performance because every access to those paths triggers kernel logging. Focus watches only on files and directories that are strictly crucial for audit compliance, not the entire filesystem.

34.1.4 Reading Audit Logs with ausearch and aureport

Raw auditd logs are stored in /var/log/audit/audit.log in a human-readable but fairly dense format, containing key-value pairs like uid=, exe=, and key= on every line. ausearch and aureport parse these logs into a format that is much easier to navigate, without requiring us to use manual grep commands on a continuously growing file.

Practical Steps

1. Search for all events recorded under the privilege_changes key from the rules created in Section 34.1.3.

   sudo ausearch -k privilege_changes

2. Search for events based on a time frame, such as all activity within the last hour.

   sudo ausearch -k identity_changes --start recent

3. Display an authentication summary in a concise tabular format compared to raw ausearch output.

   sudo aureport -au

4. Display a summary of all files logged as accessed according to our watch rules.

   sudo aureport -f

Verification and Troubleshooting

• Every event from ausearch displays paired type=PATH and type=SYSCALL lines; the SYSCALL line includes the actor's uid as well as success=yes or success=no, while the PATH line includes the touched file path. To translate numeric UIDs into more readable usernames, append the -i flag to ausearch.

  sudo ausearch -k privilege_changes -i

• If neither ausearch nor aureport returns any results even though the rule is active, confirm first that the monitored file has actually been touched since the rule was loaded. Rules added after a file has been modified cannot record past events.
• A honest note from the field: on high-traffic servers, the audit.log file can grow very rapidly. Log rotation and size limit configurations are located in /etc/audit/auditd.conf via the max_log_file and max_log_file_action parameters, separate from the logrotate mechanism we will discuss for standard application logs in Chapter 39, because auditd handles its own log rotation internally.
• A practical side effect Sysadmins should know: once auditd is installed and active, this daemon typically takes over ownership of the kernel audit socket from systemd-journald. This means AppArmor denials from Chapter 33, which we previously inspected via journalctl -k | grep -i apparmor in Section 33.3.2, might no longer appear there after this chapter, and should be queried via sudo ausearch -i | grep -i apparmor instead. If on your server those events still appear in both places, it is not a sign of an error, but simply a difference in default behaviors across kernel and package versions.

34.2 The Principle of Least Privilege on Multi-User Servers

Clean audit logs only answer questions after an incident occurs. This section moves one step earlier: shrinking as much as possible the set of accounts capable of triggering suspicious events in the first place, a principle we have actually been practicing piecemeal since Chapter 4 without explicitly naming it.

34.2.1 Least Privilege as a Principle, Not Just a Tool

Least privilege means that every account, whether belonging to a human or a service, is granted only the absolute minimum access rights required to perform its tasks, nothing more. This principle is not a single tool installed once and forgotten, but a habit spanning multiple chapters: role-based access via groups in Section 4.3.1, granular per-user sudoers policies in Section 4.2.2, up to AppArmor profiles restricting process kernel capabilities in Chapter 33. This section unites those habits under a single audit question: of all the permissions granted throughout this series, which ones turned out to be broader than strictly necessary?

34.2.2 Auditing Previously Granted Access Rights

Servers that have been running for a long time, especially those managed by multiple Sysadmins over time, tend to accumulate access permissions that were never revoked. A Developer who switched teams but remains in the sudo group, or an old line in sudoers.d/ whose purpose no one remembers, are common findings when access right audits are conducted for the first time on production servers.

Practical Steps

1. List all users belonging to the sudo group, the broadest source of administrative access on the server.

   getent group sudo

2. Review all custom files in sudoers.d/ created since Section 4.2.2, one by one.

   sudo ls -la /etc/sudoers.d/
   sudo cat /etc/sudoers.d/*

3. Search for user accounts with UID 0 other than root, a sign of root-equivalent access that often escapes notice.

   awk -F: '$3 == 0 {print $1}' /etc/passwd

4. Find accounts that have never logged in at all, a key indicator of accounts whose presence warrants review.

   sudo lastlog | grep -i "never logged in"

Verification and Troubleshooting

• Step three should only list root. If any other username appears, it is a serious sign that the account has full root-equivalent privileges without needing sudo at all, a condition that almost never has a valid justification and must be fixed immediately via usermod.
• The output of step four naturally displays quite a few service accounts such as www-data or postgres, because these accounts were never designed for interactive logins. Ignore those lines as long as their login shell is set to nologin as discussed in Section 34.2.3, and focus your attention on human accounts that appear in this list while also being listed in the sudo group from step one, as that combination makes them prime candidates for review or access revocation.
• For any account that is no longer active but remains listed in the sudo group, revoke its membership using sudo deluser username sudo rather than deleting the account outright, especially if the account still owns files or processes that need to be traced first via the audit logs from Section 34.1.4.
• Schedule access privilege audits like this regularly, not just once before go-live. Team transitions, completed projects, and Developers changing roles mean that an access list valid today could easily be outdated a few months later.

34.2.3 Restricting Service Accounts and SUID Binaries

Least privilege does not apply only to human accounts. Service accounts such as www-data for Nginx from Chapter 12 or postgres for PostgreSQL from Chapter 18 should never be able to log in interactively to a shell, because these accounts exist solely to run service processes, not for anyone to log in directly. Beyond accounts, binaries with the SUID bit set are also worth auditing because such binaries execute with the permissions of the file owner, usually root, regardless of which user executes them, making them a favorite target for Attackers once initial access to the server is gained.

Practical Steps

1. Check the configured login shell for the primary service accounts on your server.

   grep -E "^(www-data|postgres|mysql):" /etc/passwd

2. If a service account is found with its shell still set to /bin/bash or /bin/sh, replace it with nologin to prevent interactive logins.

   sudo usermod -s /usr/sbin/nologin www-data

3. Find all binaries with an active SUID bit across the system.

   sudo find / -xdev -perm -4000 -type f 2>/dev/null

Verification and Troubleshooting

• Default service accounts in Ubuntu Server like www-data on a fresh installation typically use /usr/sbin/nologin by default from the start, so the second step is generally relevant only if someone previously changed it manually, for instance, for temporary debugging that was forgotten.
• The find output in step three usually lists a dozen standard binaries such as /usr/bin/sudo, /usr/bin/passwd, or /usr/bin/su, because these binaries legitimately require root privileges for their functionality. Vigilance should be directed toward unknown SUID binaries or those located outside standard system directories like /usr/bin/ and /usr/sbin/, such as in /home/ or /tmp/, as the combination of an SUID bit and an unusual location is a common pattern used by Attackers to plant backdoors after compromising a server.
• Do not strip the SUID bit from standard system binaries without understanding the impact. Stripping SUID from /usr/bin/sudo, for example, instantly breaks the entire sudo mechanism from Chapter 4 for regular users.

34.3 Basic Pre-Go-Live Security Checklist

This concluding section pulls together all layers of defense in depth from Section 30.1.2 into a single checklist suitable for checking off item by item before a server is declared fully ready to handle production traffic.

34.3.1 Checklist Based on Defense in Depth Layers

The following table summarizes the minimum verification points for each layer built since Chapter 3, ordered by their defense layer. This checklist is not an exhaustive list of formal compliance standards like CIS Benchmarks, but rather a practical starting point tying back every chapter completed so far.

34.3.2 Self-Audit: Checklist Verification Script

Reviewing the seven items above manually every time a server is prepared for go-live is time-consuming and prone to omissions. The simple bash script below automates checks for most table points, serving as a stepping stone before we dive deeper into shell scripting for server automation in Chapter 35.

Practical Steps

1. Create a new script file.

   nano ~/golive_check.sh

2. Populate it with the following checks.

   #!/bin/bash
   echo "=== SSH ==="
   grep -E "^PermitRootLogin" /etc/ssh/sshd_config.d/*.conf /etc/ssh/sshd_config 2>/dev/null
   
   echo "=== Firewall (UFW) ==="
   sudo ufw status | head -1
   
   echo "=== Fail2ban ==="
   sudo systemctl is-active fail2ban
   
   echo "=== AppArmor ==="
   sudo aa-status --enabled && echo "AppArmor active"
   
   echo "=== auditd ==="
   sudo systemctl is-active auditd
   sudo auditctl -l | wc -l
   
   echo "=== unattended-upgrades ==="
   sudo systemctl is-active unattended-upgrades.service 2>/dev/null || echo "check timer via systemctl list-timers"
   
   echo "=== Non-root accounts with UID 0 ==="
   awk -F: '$3 == 0 {print $1}' /etc/passwd

3. Grant execution permissions and run it.

   chmod +x ~/golive_check.sh
   ./golive_check.sh

Verification and Troubleshooting

• A healthy PermitRootLogin line displays a value of no, matching the configuration applied in Section 3.2.1. The ufw status line should show Status: active, and both fail2ban and auditd services should display active.
• If any output displays inactive or unknown, it does not mean the script failed, but rather serves as an honest sign that a defense layer from that chapter has not been configured on this server. Return to the relevant chapter before proceeding with the go-live process.
• This script is intentionally simple and only checks status, rather than inspecting detailed rule contents. For formal, standardized compliance audit needs, consider tools like lynis or official CIS (Center for Internet Security) benchmarks, which cover hundreds of inspection points well beyond the scope of this basic checklist.

With this, Part VIII is officially complete. The five defense in depth layers from Section 30.1.2 have been systematically constructed: attack surface reduction and routine patching in Chapter 30, network traffic filtering via firewalls in Chapter 31, automated detection and response through Fail2ban in Chapter 32, mandatory access control using AppArmor in Chapter 33, and auditing alongside least privilege in this chapter via auditd to log kernel-level activity traces, reviewing permissions accumulated since Chapter 4, and establishing a go-live checklist that binds all layers into a verifiable unit. A server that has passed all five layers is not invulnerable, as no system is completely unhackable, but it is vastly better prepared to withstand and detect Attacker attempts compared to a server fresh out of basic installation in Chapter 2. The upcoming Part IX shifts focus from security to daily Sysadmin operational efficiency, starting with Chapter 35 on shell scripting for server automation, including fulfilling our promise in Section 34.3.2 by writing cleaner scripts with proper error handling and logging, before moving on to Ansible and cloud-init as Infrastructure as Code in the following two chapters.