Advanced Storage

Servers running out of disk space in the middle of the night, databases requiring additional capacity without a single second of downtime, or a sudden disk failure that takes down all production data with it, all these scenarios are daily realities faced by Sysadmins once a server enters production. Desktop-style static partitioning that is set up once and forgotten is no longer adequate. Production servers require storage that can grow, withstand hardware failures, and have its health monitored before issues actually occur.

This chapter discusses four advanced storage pillars that every Sysadmin must master: LVM for volume flexibility, software RAID for redundancy and performance, procedures for adding new disks to a running server without downtime, and the habit of monitoring physical disk health using smartctl before failures actually happen. We will also address a default change in Ubuntu that often goes unnoticed: /tmp now running on top of tmpfs.

7.1 Partition and Filesystem Review

Every disk recognized by Linux needs to be divided into one or more partitions before data can be stored, and each partition requires a filesystem that defines how data is organized, read, and written within it. Modern partition tables use the GPT (GUID Partition Table) scheme, which has been the default for Ubuntu Server installations for several recent releases because it supports large-capacity disks and offers far more flexible partition limits compared to the older MBR scheme, which was restricted to only four primary partitions.

For filesystems, ext4 remains the default choice for Ubuntu Server due to its maturity, stability, and recovery tooling that has been proven over many years. XFS is a frequently chosen alternative for workloads involving large file sizes and high parallel I/O, such as database or media servers, owing to its superior performance in those scenarios. One important limitation to keep in mind: XFS supports online grow operations (enlarging), but does not support shrinking at all. If there is any possibility that a volume may need to be reduced in size in the future, ext4 is the safer choice.

Quick Verification

lsblk -f
df -hT
sudo blkid

lsblk -f displays the structure of disks, partitions, and filesystems along with their UUIDs in a single concise tree view. df -hT shows the used and available space on each mount point alongside its filesystem type. blkid is useful when we need specific partition UUIDs, for example, when writing entries for /etc/fstab, which we will practice in Section 7.4.

7.2 In-Depth Logical Volume Manager (LVM)

Standard partitions are rigid: once created with a specific size, expanding them requires dealing directly with the physical disk, and shrinking them almost always carries a risk of data loss. LVM (Logical Volume Manager) adds an abstraction layer on top of physical disks so that volume sizes become flexible, can be aggregated from multiple disks, and can be modified at any time without repartitioning. As touched upon in Chapter 2, the Ubuntu Server Subiquity installer uses LVM by default under guided partitioning options, so it is highly likely that the server you manage has been running on top of LVM from day one.

7.2.1 Physical Volume, Volume Group, Logical Volume

LVM operates across three conceptual layers. A Physical Volume (PV) is a physical disk or partition that is registered to LVM. Multiple PVs are then combined into a single Volume Group (VG), which represents a pooled storage space treated as one large pool. From this VG, we carve out space into one or more Logical Volumes (LV), which are then formatted with a filesystem and mounted like regular partitions.

Practical Steps

Ensure LVM tools are installed. On installations set up with LVM from the start, this package is present by default.
```
sudo apt update
sudo apt install lvm2
```
Register a new disk (such as an unused /dev/sdb) as a Physical Volume.
```
sudo pvcreate /dev/sdb
sudo pvs
```
Create a Volume Group from that Physical Volume.
```
sudo vgcreate vg_data /dev/sdb
sudo vgs
```
Carve the Volume Group into a Logical Volume of your specified size.
```
sudo lvcreate -L 20G -n lv_data vg_data
sudo lvs
```

Format the Logical Volume with a filesystem, then mount it like a regular partition.

sudo mkfs.ext4 /dev/vg_data/lv_data
sudo mkdir -p /mnt/data
sudo mount /dev/vg_data/lv_data /mnt/data

In production environments, a far more flexible practice is to leave a portion of the VG unallocated when first creating LVs. This unallocated space serves as a buffer that can be immediately used for emergency resizing without needing to add new physical disks first, a practice whose benefits will become apparent in Section 7.2.2 below.

7.2.2 Resizing Volumes without Downtime

The primary advantage of LVM over traditional partitioning is the ability to expand a Logical Volume while the filesystem remains mounted and applications continue writing data to it, without requiring an unmount or a reboot. This makes LVM a mandatory choice for application or database data volumes where downtime cannot be tolerated.

Practical Steps

If the Volume Group still has free space, immediately extend the Logical Volume and its filesystem simultaneously using the -r flag, which automatically invokes resize2fs (for ext4) or xfs_growfs (for XFS) in the background.
```
sudo lvextend -L +10G -r /dev/vg_data/lv_data
```
If free space in the VG is exhausted, add a new Physical Volume to the VG first before extending the LV.
```
sudo pvcreate /dev/sdc
sudo vgextend vg_data /dev/sdc
sudo lvextend -L +20G -r /dev/vg_data/lv_data
```

Verification and Troubleshooting

Confirm that the new size is reflected at the filesystem level, not just at the LVM level.
```
df -h /mnt/data
sudo lvs vg_data
```
Shrinking an LV is a significantly riskier operation than expanding it, especially for ext4, which requires the filesystem to be unmounted first, checked for integrity with e2fsck -f, and then shrunk via resize2fs before lvreduce reduces the underlying LV. This sequence must never be reversed, as shrinking the LV before the filesystem finish shrinking will truncate actively used data. For XFS, shrinking is not supported at all, as mentioned in Section 7.1. Always back up critical data before attempting a shrink operation, regardless of how low the risk appears.
The safest real-world practice: never allocate 100 percent of VG space to a single LV initially. Always leave a few percent as a buffer, as LVM resizing is far simpler than hastily adding a new physical disk during a capacity emergency.

7.2.3 LVM Snapshots

An LVM snapshot is a frozen copy of a Logical Volume at a specific point in time, created almost instantaneously using a copy-on-write mechanism, meaning original data is only copied to the snapshot space when those specific blocks change after the snapshot is taken. Snapshots are extremely useful for taking consistent backups of volumes undergoing active writes, such as PostgreSQL data directories, without stopping the database service, a technique that will be relevant again in Chapters 23 and 40.

Practical Steps

Create a snapshot from an active Logical Volume. The allocated size is not intended to hold a full copy of all data, but rather to serve as space for tracking changes (deltas) that occur while the snapshot is active.
```
sudo lvcreate -L 5G -s -n lv_data_snap /dev/vg_data/lv_data
```
Mount the snapshot separately, for example, to run backups via rsync or tar, without disturbing the original volume used by live applications.
```
sudo mkdir -p /mnt/snapshot
sudo mount -o ro /dev/vg_data/lv_data_snap /mnt/snapshot
```
Once the backup process completes, unmount and remove the snapshot so it does not continue consuming VG space.
```
sudo umount /mnt/snapshot
sudo lvremove /dev/vg_data/lv_data_snap
```

Verification and Troubleshooting

Monitor how full the snapshot space gets, as a snapshot that reaches 100 percent capacity will become invalid and unusable.
```
sudo lvs -o +snap_percent vg_data
```
An LVM snapshot is not a substitute for an off-site backup. The snapshot resides on the exact same physical disk as the original volume, so if that physical disk fails completely, the snapshot is lost alongside it. Treat snapshots as a tool for capturing consistent backups, not as the backup itself, a principle we will explore further under the 3-2-1 backup strategy in Chapter 40.

7.3 Software RAID with mdadm

RAID (Redundant Array of Independent Disks) combines multiple physical disks into a single logical unit to achieve combinations of redundancy, performance, or both. Ubuntu Server provides mdadm as the standard tool for managing software-based RAID, an approach far more affordable than dedicated hardware RAID controllers, offering performance that in many modern use cases is virtually comparable.

7.3.1 RAID 0, 1, 5, 10: Concepts and Use Cases

Every RAID level offers a distinct trade-off between capacity, redundancy, and performance. Selecting the wrong level for a given workload can be disastrous, such as using RAID 0 for critical production data that cannot afford to be lost.

Level	Minimum Disks	Redundancy	Effective Capacity	Use Case
RAID 0 (striping)	2	None	100% (n * disk)	Cache or scratch space requiring high throughput where data loss is acceptable, not for production data
RAID 1 (mirroring)	2	High (survives 1 disk failure)	50% (1 * disk)	OS/boot partitions and critical small-to-medium volumes prioritizing reliability
RAID 5 (single parity)	3	Medium (survives 1 disk failure)	(n-1) * disk	Large capacity storage on a limited budget, less ideal for very large drives due to long rebuild times
RAID 10 (mirror + stripe)	4	High (survives specific multi-disk failures)	50% (n/2 * disk)	Production databases and high-I/O workloads requiring both performance and redundancy

Sysadmins must be honest about the often-overlooked risks of RAID 5: when a single disk fails and the array enters a rebuild phase, all remaining disks are read completely to reconstruct the missing data. On drive capacities spanning tens of terabytes, this rebuild process can take one to two days, during which the array remains in a degraded state with zero redundancy. If a second disk fails before the rebuild finishes, all data on the array is lost. This is why RAID 10 is preferred for modern critical data despite being capacity-expensive: its rebuild time is significantly shorter because it only copies from a single mirror pair rather than reading the entire array.

7.3.2 Building RAID with mdadm

This section demonstrates setting up RAID 1 as an example, as RAID 1 is most commonly used to protect OS volumes or small-to-medium critical data. The same principles apply to other RAID levels, varying primarily in the --level parameter and disk count.

Practical Steps

Install mdadm.
```
sudo apt update
sudo apt install mdadm
```
Create a RAID 1 array from two empty disks, such as /dev/sdb and /dev/sdc. This command destroys all existing data on both disks; ensure the disks are empty or backed up.
```
sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb /dev/sdc
```
Monitor the initial background synchronization process, which may take time depending on disk sizes.
```
cat /proc/mdstat
```

Format and mount the array like a standard disk.

sudo mkfs.ext4 /dev/md0
sudo mkdir -p /mnt/raid1
sudo mount /dev/md0 /mnt/raid1

Save the array configuration so it is automatically recognized after rebooting, then update initramfs so the array can assemble during early boot steps.
```
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf
sudo update-initramfs -u
```

Verification and Troubleshooting

Inspect array health status in detail, including active disks and any flagged as faulty.
```
sudo mdadm --detail /dev/md0
```
If a disk in the array fails, mark it as failed and remove it before replacing it with a new disk.
```
sudo mdadm --manage /dev/md0 --fail /dev/sdb
sudo mdadm --manage /dev/md0 --remove /dev/sdb
```
After attaching the new physical disk, add it to the array to trigger automatic rebuilding.
```
sudo mdadm --manage /dev/md0 --add /dev/sdd
```
In real-world setups, do not wait for manual mdadm --detail checks to find out a disk has failed. Enable the mdadm monitoring daemon so email alerts are dispatched immediately upon status changes, and integrate it with centralized monitoring discussed in Chapter 38.
```
sudo systemctl enable --now mdmonitor
```
RAID and LVM are not mutually exclusive. A common production pattern is stacking both: an mdadm array (such as RAID 10 for database storage) serves as the Physical Volume for LVM, rather than using raw disks directly.
```
sudo pvcreate /dev/md0
sudo vgcreate vg_data /dev/md0
```
This layer structure provides physical redundancy from RAID alongside flexible resizing and snapshot features from LVM covered in Section 7.2, a combination far more resilient than using either on its own.

7.4 Adding New Disks to a Running Server

Expanding storage capacity on a production server should ideally never require downtime. Whether in virtual machine environments (KVM, VMware) or cloud platforms, new disks can be attached virtually to a running server, and modern Linux kernels can detect them without a reboot provided you know how to trigger a rescan.

7.4.1 Partitioning, Formatting, and Permanent Mounts via `/etc/fstab`

Practical Steps

After attaching a new disk from the hypervisor or cloud console, check if the kernel has recognized it.
```
lsblk
sudo dmesg | grep -i sd
```
If the disk is not detected, particularly on virtual SCSI hypervisors, force the kernel to rescan the SCSI bus without rebooting. Check available SCSI hosts first, as numbers are not always host0.
```
ls /sys/class/scsi_host/
echo "- - -" | sudo tee /sys/class/scsi_host/host0/scan
```
Create a partition using parted, which is safer for automation scripts than interactive fdisk.
```
sudo parted /dev/sdb --script mklabel gpt mkpart primary ext4 0% 100%
```
Format the partition with your chosen filesystem.
```
sudo mkfs.ext4 /dev/sdb1
```
Obtain the partition UUID, which is far more reliable for /etc/fstab entries than device paths like /dev/sdb1. Device naming orders can change across reboots on multi-disk servers, whereas UUIDs remain static.
```
sudo blkid /dev/sdb1
```

Create a mount point, then add a permanent entry in /etc/fstab using the acquired UUID.

sudo mkdir -p /mnt/data
sudo nano /etc/fstab

UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /mnt/data ext4 defaults,nofail 0 2

Test the fstab entry without rebooting.
```
sudo mount -a
df -h /mnt/data
```

Verification and Troubleshooting

Always execute sudo mount -a after editing /etc/fstab before rebooting. If there are syntax errors or incorrect UUIDs, this command reports them instantly, which is vastly safer than discovering boot failures and emergency mode prompts in the middle of the night.
The nofail option in the example above is essential. Without it, if the disk is ever undetected during boot (such as a detached cloud volume or unready external storage), systemd halts the boot process to await manual console intervention. The nofail flag allows the system to boot even if mounting fails, a critical safeguard for secondary data volumes.
If the storage will be shared with Docker containers, note that volumes mounted here can serve directly as bind mounts for containers, a topic we will explore in Chapters 26 and 27.

7.5 `/tmp` as `tmpfs` by Default: Application Implications

Starting with Ubuntu 21.04, the /tmp directory runs on tmpfs by default, a filesystem stored entirely in RAM (with overflow into swap if RAM fills up), rather than residing on a physical disk as in older Ubuntu releases. This change follows systemd upstream defaults via the tmp.mount unit and continues as default behavior in Ubuntu Server 26.04 LTS. The consequences are significant: files written to /tmp consume server RAM, and all contents disappear upon reboot.

Quick Verification

findmnt /tmp
df -h /tmp

The command findmnt /tmp verifies whether /tmp is actively mounted as tmpfs on your server. Some cloud base images or specific providers modify this default, so always verify on the target server rather than assuming.

By default, tmp.mount size is capped at 50 percent of total physical RAM. On servers with limited RAM, such as small VPS instances with 1 to 2 GB of RAM, this can present a serious trap. Build processes, large file compression, or applications creating large temporary files in /tmp (such as database dumps during migration) risk triggering the Out of Memory (OOM) killer to forcefully terminate processes, even while the primary disk shows ample free space.

Practical Steps: Adjusting /tmp Size Limits

To increase or decrease the tmpfs size limit for /tmp, create an override file for the tmp.mount unit rather than modifying the original unit file.
```
sudo systemctl edit tmp.mount
```
Add the following lines between the comments provided by the editor.
```
[Mount]
Options=mode=1777,strictatime,nosuid,nodev,size=2G
```
Reload systemd configuration and remount /tmp to apply changes without restarting the server.
```
sudo systemctl daemon-reload
sudo systemctl restart tmp.mount
```

Verification and Troubleshooting

If an application requires large and sustained scratch space on disk, do not force it to use /tmp. A safer approach is directing the application's TMPDIR variable to a directory located on a physical disk, or disabling tmp.mount entirely to return /tmp to a standard disk partition via an explicit /etc/fstab entry.
```
sudo systemctl mask tmp.mount
```
In production, the wisest approach is not blindly enabling or disabling tmpfs, but aligning it with server workload profiles. Stateless web servers with ample RAM benefit from tmpfs due to faster /tmp I/O. Conversely, database servers or heavy data-processing units with limited memory are safer using disk-backed /tmp storage.

7.6 Disk Health Monitoring with smartctl

RAID and LVM protect against disk failures after they occur, but neither provides early warning that a drive is degrading toward failure. This is where S.M.A.R.T. (Self-Monitoring, Analysis, and Reporting Technology) plays its role, a feature built into almost all modern drives (HDDs, SSDs, and NVMe drives) that logs internal health metrics like bad sector counts, temperature, and operating hours. The smartmontools package supplies smartctl as an interface to read this telemetry under Linux.

Practical Steps

Install smartmontools.

sudo apt update
sudo apt install smartmontools

Verify S.M.A.R.T. support and ensure it is enabled on the drive.
```
sudo smartctl -i /dev/sda
```
Check a quick summary of the drive's health status.
```
sudo smartctl -H /dev/sda
```
To view complete details for all S.M.A.R.T. attributes (bad sector count, temperature, reallocated sector count, etc.), use the following option.
```
sudo smartctl -a /dev/sda
```
For NVMe drives, pass the NVMe device path; smartctl handles NVMe protocols natively in modern versions.
```
sudo smartctl -a /dev/nvme0n1
```
Run a self-test for a comprehensive diagnostic. The short test takes a few minutes, while the long test scans the entire disk surface and may take hours depending on capacity.
```
sudo smartctl -t short /dev/sda
```
Enable smartd so background health checks execute automatically and dispatch alerts when failure indicators appear.
```
sudo systemctl enable --now smartmontools
```

Verification and Troubleshooting

A PASSED result under smartctl -H means the drive considers itself healthy internally, but this is not an absolute guarantee. Track attribute trends such as Reallocated_Sector_Ct and Current_Pending_Sector in smartctl -a output. Numbers that increase over time, even with an overall PASSED status, serve as an early warning that drive degradation is underway and replacement should be scheduled.
Critical attributes differ between HDDs and SSDs. On HDDs, Reallocated_Sector_Ct and Current_Pending_Sector indicate mechanical degradation. On SSDs, monitor Media_Wearout_Indicator or Percentage_Used (depending on vendor), which measures consumed flash endurance. SSDs lack mechanical components to wear out, but still have defined write endurance limits that require tracking.
A common trap when transitioning to virtual environments: virtual disks in VMs or cloud instances typically do not expose actual underlying S.M.A.R.T. data, as the hypervisor abstracts physical storage. Running smartctl against such virtual disks often yields messages like Unavailable - device lacks SMART capability. Physical disk monitoring in cloud infrastructure rests entirely with the provider, rather than from within the guest VM.
For bare-metal servers with multiple physical disks, route smartd check results into centralized alerting systems rather than relying solely on local system mail that may go unread. This integration will be unified with overall system monitoring in Chapter 38.

At this point, we have established a resilient storage foundation: LVM for volume flexibility, RAID for redundancy, safe procedures for adding live drives, awareness of RAM-backed /tmp implications, and proactive drive health monitoring routines. Chapter 8 will launch Part III of this series, covering server network configuration via Netplan as the essential preparation before servers accept and process external network traffic.

Advanced Storage

7.1 Partition and Filesystem Review

Quick Verification

7.2 In-Depth Logical Volume Manager (LVM)

7.2.1 Physical Volume, Volume Group, Logical Volume

Practical Steps

7.2.2 Resizing Volumes without Downtime

Practical Steps

Verification and Troubleshooting

7.2.3 LVM Snapshots

Practical Steps

Verification and Troubleshooting

7.3 Software RAID with mdadm

7.3.1 RAID 0, 1, 5, 10: Concepts and Use Cases

7.3.2 Building RAID with mdadm

Practical Steps

Verification and Troubleshooting

7.4 Adding New Disks to a Running Server

7.4.1 Partitioning, Formatting, and Permanent Mounts via /etc/fstab

Practical Steps

Verification and Troubleshooting

7.5 /tmp as tmpfs by Default: Application Implications

Quick Verification

Practical Steps: Adjusting /tmp Size Limits

Verification and Troubleshooting

7.6 Disk Health Monitoring with smartctl

Practical Steps

Verification and Troubleshooting

Share

7.4.1 Partitioning, Formatting, and Permanent Mounts via `/etc/fstab`

7.5 `/tmp` as `tmpfs` by Default: Application Implications