TLS/SSL and Let's Encrypt

All virtual hosts built since Chapter 12, namely example.local, www.example.local, blog.example.local in Nginx and Apache, and app.example.local in Chapter 16, still serve traffic over plain HTTP. Every request and response between Visitor and server runs as plain text that can be read by anyone who eavesdrops on the network path between them. On shared office WiFi networks, public hotspots, or even wired networks where a device has been compromised, an Attacker simply runs a packet sniffer like Wireshark or tcpdump to read request contents as-is, including login passwords sent via HTML forms or session cookies that should remain secret.

This chapter closes that gap by adding a TLS (Transport Layer Security) layer to all virtual hosts we have built. We will start from basic concepts of HTTPS and digital certificates, followed by Certbot and Let's Encrypt as the most common way to obtain free certificates for real public domains, then practice TLS configuration in Nginx and Apache using lab certificates, and wrap up with auto-renewal strategies so certificates never expire unnoticed.

17.1 HTTPS Concepts and Digital Certificates

HTTPS is essentially the exact same HTTP protocol used since Chapter 12, except all its data is wrapped in TLS before being sent over the network. The old name for this protocol is SSL (Secure Sockets Layer), but all SSL versions (2.0 and 3.0) have been declared obsolete and insecure for years. The term "SSL" persists in everyday conversation including this chapter title, but actual implementations running on any modern server, including those configured in this chapter, are TLS versions 1.2 and 1.3.

17.1.1 Encryption and the TLS Handshake in Brief

TLS combines two types of encryption simultaneously. Asymmetric encryption, using a pair of public and private keys, is used at connection startup to mutually verify identity and agree on a shared secret key, whereas symmetric encryption, which is computationally faster, encrypts actual data throughout the connection session. This initial negotiation process is called the TLS handshake: the client sends a list of supported ciphers via a ClientHello message, the server responds by selecting a cipher and sending its certificate via a ServerHello message, both parties calculate the same session key without transmitting private keys over the network, and only then does actual application data flow encrypted.

Full handshake details changed significantly between TLS 1.2 and TLS 1.3, especially regarding the number of round trips required before data transmission begins, but for Sysadmins managing web servers, these details are handled behind the scenes by TLS libraries such as OpenSSL. Our task is simply providing a valid certificate and enabling secure protocol versions, which will be practiced directly in Section 17.3.

17.1.2 Anatomy of X.509 Certificates and the Chain of Trust

Digital certificates used by HTTPS follow the X.509 standard, a format binding a domain identity to a public key, digitally signed by the issuer. The most important fields in this certificate include Subject (certificate owner identity, usually a domain name), Issuer (the party issuing and signing the certificate), Validity Period (Not Before and Not After ranges determining validity start and end times), Public Key, and the Issuer's digital signature at the end. In recent years, modern browsers like Chrome and Firefox no longer trust the Common Name (CN) field alone, requiring registered domain names inside the Subject Alternative Name (SAN) extension. This capability allows one certificate to cover multiple domains simultaneously, a feature utilized in Section 17.3.1 to cover all lab domains in a single certificate.

The party issuing certificates is called a Certificate Authority (CA). Browsers and operating systems store a list of trusted root CAs natively in their trust stores. Root CAs rarely sign domain certificates directly; instead, they delegate this task to intermediate CAs while keeping root keys offline for security. This sequence of signatures from domain certificate to intermediate CA up to the trusted root CA is called the chain of trust. Once a browser receives a certificate from a server, it traces this chain until finding a root CA present in its trust store; if the chain is broken or ends at an unknown CA, the browser displays a security warning instead of connecting directly.

Certificates signed by oneself, without a third-party CA, are called self-signed certificates. Cryptographically, their encryption strength equals certificates issued by public CAs, but because no chain of trust leads to a root CA recognized by browsers, self-signed certificates trigger security warnings for Visitors. Such certificates suit internal needs like labs, staging environments, or server-to-server communications fully controlled by Sysadmins, but are never suitable for public services accessed by general Visitors, a point discussed further in Section 17.2.3 when issuing certificates for our lab domain.

17.2 Certbot and Let's Encrypt

Let's Encrypt is a non-profit Certificate Authority managed by the Internet Security Research Group (ISRG) since 2015, issuing free and fully automated TLS certificates. Its presence became a major driver for massive HTTPS adoption across the public web, as commercial CA TLS certificates were previously paid and manually issued. Certbot, developed by the Electronic Frontier Foundation (EFF), is the most popular official client for communicating with Let's Encrypt and will be used in this chapter.

17.2.1 ACME: How Let's Encrypt Verifies Domain Ownership

Let's Encrypt issues certificates via a standardized protocol called ACME (Automated Certificate Management Environment, RFC 8555). Unlike legacy commercial CAs that manually verify applicant identities, ACME proves domain ownership automatically through challenges completed by clients like Certbot before certificate issuance. The two most common challenge types are:

• HTTP-01: Certbot places a file containing a unique token at /.well-known/acme-challenge/ on the web server, and Let's Encrypt servers attempt to retrieve this file via HTTP on port 80 of the requested domain. If retrieved successfully with matching content, domain ownership is proven.
• DNS-01: Certbot is requested to create a specific TXT record in the domain's public DNS zone, and Let's Encrypt servers verify its existence via standard DNS queries. This challenge type is required for wildcard certificates (e.g., *.example.com) and useful when port 80 cannot be reached directly from the internet.

A prerequisite applying to both challenge types often overlooked by beginners: validation servers owned by Let's Encrypt must reach the requested domain over the public internet, either via HTTP in HTTP-01 or public DNS in DNS-01. This point will be demonstrated in Section 17.2.3.

17.2.2 Installing Certbot

Ubuntu Server provides Certbot directly from official repositories, allowing installation using standard APT without adding third-party repositories, consistent with package installation practices established since Chapter 6. Certbot in APT is split into core packages and separate plugin packages for each web server, ensuring Nginx-only servers avoid Apache dependencies and vice versa.

Hands-on Steps

1. Update package lists and install the core Certbot package.

   sudo apt update
   sudo apt install certbot

   This package suffices for issuing certificates via standalone or webroot plugins, but cannot yet be used with --nginx or --apache flags tested in Section 17.2.3.
2. Install python3-certbot-nginx so Certbot can read and edit Nginx configurations automatically via the --nginx flag. Also install python3-certbot-apache since our lab includes Apache since Chapter 13, though this package becomes relevant in Section 17.2.4.

   sudo apt install python3-certbot-nginx python3-certbot-apache

3. Verify installation success by checking its version.

   certbot --version

Verification and Troubleshooting

• Displayed version numbers may vary depending on when Ubuntu repositories were last synced, so do not be surprised if numbers differ from module examples, similar to PHP version notes in Section 14.1.1.
• To ensure both plugins are installed properly, verify via dpkg.

  dpkg -l | grep certbot

  Output should show three package lines with status ii in the first column: certbot, python3-certbot-nginx, and python3-certbot-apache.

17.2.3 Attempting Certificate Issuance for Lab Domains

Let us directly demonstrate what happens when attempting to issue a Let's Encrypt certificate for domain www.example.local built in Chapter 12. Nginx from Chapter 16 should remain active on this server, so we use the --nginx plugin installed earlier to auto-edit Nginx configuration.

Hands-on Steps

1. Run Certbot with the Nginx plugin, targeting domains example.local and www.example.local simultaneously.

   sudo certbot --nginx -d example.local -d www.example.local

2. Certbot prompts for an email address for expiration notifications, followed by Terms of Service agreement. Fill out as needed and proceed.

Shortly after, this process fails with an error similar to the following:

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
  Domain: www.example.local
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up A for www.example.local - check that a DNS record exists for this domain

Hint: The Certificate Authority failed to download the challenge files from the temporary standalone webserver started by Certbot on port 80. Ensure that the listed domains point to this machine and that it can accept inbound connections from the internet.

This failure is not a bug, but proof that the HTTP-01 mechanism in Section 17.2.1 operates as intended. Domain example.local resolves perfectly within our lab network thanks to BIND9 from Chapter 9, but TLD .local is entirely unregistered in public root DNS used worldwide. Therefore, Let's Encrypt servers attempting external lookups will find the domain unknown. Beyond Certbot or Let's Encrypt, any public CA complying with CA/Browser Forum rules will never issue certificates for domains whose ownership cannot be verified from the public internet. This represents a fundamental limitation of public CA trust systems, not a mere tooling limitation.

Verification and Troubleshooting

• The same error occurs for newly registered public domains not yet pointing to server public IPs via A/AAAA records, or servers where port 80 remains blocked by firewalls/NAT. The message DNS problem: NXDOMAIN indicates the domain itself was not found, while Fetching ... Connection refused or Connection timed out indicates the domain resolved but the server was unreachable on port 80.
• In practice, assuming internal domains like .local, .internal, or .lan can obtain Let's Encrypt certificates is a common pitfall for new Sysadmins. Sections 17.2.4 and 17.3 address two different solutions depending on whether the domain is genuinely public or purely internal.

17.2.4 Certbot Workflow for Real Public Domains

For Sysadmins managing production servers with registered public domains pointing via public DNS to server IPs (not private IPs like 192.168.1.20 used in this lab), the Certbot workflow above executes smoothly. Commands are identical to Section 17.2.3, replacing lab domains with actual public domains.

sudo certbot --nginx -d real-domain.com -d www.real-domain.com

For Apache, use the --apache plugin installed in Section 17.2.2. It operates similarly by directly reading and editing relevant VirtualHost files.

sudo certbot --apache -d real-domain.com -d www.real-domain.com

Both plugins automatically add ssl_certificate (Nginx) or SSLCertificateFile (Apache) directives to existing virtual hosts, offering options to configure automatic HTTP-to-HTTPS redirects: two tasks we will manually configure in Section 17.3 using lab certificates.

Certbot never places certificates directly into web server configuration directories. Issued certificates are stored in /etc/letsencrypt/live/<domain-name>/ as four symlinks pointing to the latest certificate revisions in /etc/letsencrypt/archive/<domain-name>/:

• cert.pem: domain certificate itself.
• chain.pem: intermediate CA certificate.
• fullchain.pem: combination of cert.pem and chain.pem, referenced by ssl_certificate in Nginx or SSLCertificateFile in Apache.
• privkey.pem: private key, referenced by ssl_certificate_key in Nginx or SSLCertificateKeyFile in Apache.

The --nginx and --apache plugins automatically populate directives with paths pointing to fullchain.pem and privkey.pem inside the live directory rather than archive. This detail is crucial for auto-renewal in Section 17.4: each time Certbot renews a certificate, new revisions are added to archive and live symlinks update automatically, requiring no web server configuration changes after renewal.

For servers not using Nginx or Apache as reverse proxies (e.g., mail or database servers requiring dedicated TLS certificates), Certbot provides a --standalone plugin running a temporary web server on port 80 to complete challenges, as well as a --webroot plugin placing challenge files directly into existing web server document roots without configuration edits.

17.3 Configuring TLS in Nginx and Apache with Self-Signed Certificates

Because our lab domain is purely internal, this section uses self-signed certificates to practice actual TLS configuration mechanics in Nginx and Apache. Directives used, such as ssl_certificate in Nginx and SSLCertificateFile in Apache, match those automatically added by Certbot for real public domains in Section 17.2.4. Differences lie only in certificate source and storage locations, not web server configuration methodology.

17.3.1 Creating Multi-Domain Self-Signed Certificates with OpenSSL

Thanks to the SAN extension discussed in Section 17.1.2, we can generate a single certificate covering all lab domains: example.local, www.example.local, blog.example.local, and app.example.local, avoiding repetitive generation per virtual host. Unlike Let's Encrypt certificates standardized in /etc/letsencrypt/live/, self-signed certificates can be stored anywhere; we use /etc/ssl/example.local/ to distinguish them from system default certificates in /etc/ssl/certs/.

Hands-on Steps

1. Create a dedicated directory to store lab certificates and private keys.

   sudo mkdir -p /etc/ssl/example.local

2. Create an OpenSSL configuration file defining certificate identity and SAN entries.

   sudo nano /etc/ssl/example.local/san.cnf

   [req]
   distinguished_name = req_distinguished_name
   x509_extensions = v3_req
   prompt = no
   
   [req_distinguished_name]
   C = ID
   ST = DKI Jakarta
   L = Jakarta
   O = Linux Server 101 Lab
   CN = example.local
   
   [v3_req]
   subjectAltName = @alt_names
   
   [alt_names]
   DNS.1 = example.local
   DNS.2 = www.example.local
   DNS.3 = blog.example.local
   DNS.4 = app.example.local

3. Generate private key and self-signed certificate simultaneously in a single command, valid for 365 days.

   sudo openssl req -x509 -nodes -newkey rsa:2048 \
     -keyout /etc/ssl/example.local/example.local.key \
     -out /etc/ssl/example.local/example.local.crt \
     -days 365 \
     -config /etc/ssl/example.local/san.cnf

   The -nodes flag stores private keys without passphrases, standard for web server certificates as passphrases prevent Nginx or Apache from starting automatically upon service restarts without manual intervention.
4. Restrict private key permissions so only root can read it, following standard security practices for cryptographic key files.

   sudo chmod 600 /etc/ssl/example.local/example.local.key

Verification and Troubleshooting

• Inspect generated certificate contents to ensure all four domains exist in the SAN extension.

  openssl x509 -in /etc/ssl/example.local/example.local.crt -noout -text | grep -A1 "Subject Alternative Name"

  Output should display all four domains as configured in san.cnf.
• If openssl req fails with errors regarding missing alt_names, recheck section header formatting (brackets) and indentation in san.cnf, as OpenSSL strictly enforces configuration syntax.

17.3.2 Enabling HTTPS in Nginx for All Virtual Hosts

With certificates ready, next add a new server block listening on port 443 for each Nginx virtual host, while redirecting all HTTP traffic on port 80 to HTTPS so Visitors cannot access unencrypted versions.

Hands-on Steps

1. Allow port 443 in the firewall, as Chapter 12 only opened port 80.

   sudo ufw allow 443/tcp

2. Edit the example.local virtual host configuration, modifying the existing server block to HTTPS and adding a new block for HTTP redirection.

   sudo nano /etc/nginx/sites-available/example.local

   server {
       listen 443 ssl;
       listen [::]:443 ssl;
   
       server_name example.local www.example.local;
       root /var/www/example.local/html;
   
       ssl_certificate /etc/ssl/example.local/example.local.crt;
       ssl_certificate_key /etc/ssl/example.local/example.local.key;
       ssl_protocols TLSv1.2 TLSv1.3;
   
       location / {
           try_files $uri $uri/ =404;
       }
   
       error_page 404 /404.html;
       location = /404.html {
           internal;
       }
   }
   
   server {
       listen 80;
       listen [::]:80;
   
       server_name example.local www.example.local;
   
       return 301 https://$host$request_uri;
   }

   The ssl_protocols directive restricts Nginx to accept TLS 1.2 and TLS 1.3 only, rejecting insecure older versions like TLS 1.0 and 1.1. The second block serves no content, enforcing HTTPS redirection via 301 Moved Permanently status.
3. Repeat the pattern for blog.example.local.

   sudo nano /etc/nginx/sites-available/blog.example.local

   server {
       listen 443 ssl;
       listen [::]:443 ssl;
   
       server_name blog.example.local;
       root /var/www/blog.example.local/html;
   
       ssl_certificate /etc/ssl/example.local/example.local.crt;
       ssl_certificate_key /etc/ssl/example.local/example.local.key;
       ssl_protocols TLSv1.2 TLSv1.3;
   
       location / {
           try_files $uri $uri/ =404;
       }
   }
   
   server {
       listen 80;
       listen [::]:80;
   
       server_name blog.example.local;
   
       return 301 https://$host$request_uri;
   }

4. Apply the pattern to app.example.local from Chapter 16 by adding ssl_certificate, ssl_certificate_key, and ssl_protocols directives to the existing server block, updating listen directives to 443 ssl, and adding a port 80 redirect block. The upstream block and proxy_pass directive remain unchanged.
5. Test configuration syntax and reload.

   sudo nginx -t
   sudo systemctl reload nginx

6. Test from another machine on the same network. The -k flag prevents curl from failing due to self-signed certificates.

   curl -k https://www.example.local
   curl -k https://blog.example.local
   curl -k https://app.example.local

7. Verify HTTP to HTTPS redirection works properly.

   curl -I http://www.example.local

   Response should show a 301 status with a Location header pointing to https://.

Verification and Troubleshooting

• Without the -k flag, curl rejects connections with curl: (60) SSL certificate problem: self-signed certificate. This behavior is expected when CAs are untrusted by curl, matching browser warning behaviors.
• If nginx: [emerg] cannot load certificate appears during nginx -t, verify file paths in ssl_certificate and check directory access permissions for /etc/ssl/example.local/.
• To inspect certificates served during handshakes, execute openssl s_client directly in the terminal:

  echo | openssl s_client -connect www.example.local:443 -servername www.example.local 2>/dev/null | openssl x509 -noout -dates -subject

  This command displays subject details along with notBefore and notAfter dates.

17.3.3 Enabling HTTPS in Apache and Redirecting HTTP to HTTPS

Apache virtual hosts for example.local and blog.example.local built in Chapter 13 remain unmodified. This section enables HTTPS using the same SAN certificate generated in Section 17.3.1.

Hands-on Steps

1. Stop Nginx and start Apache, matching the port 80 toggle pattern established in Chapter 13 and Chapter 14.

   sudo systemctl stop nginx
   sudo systemctl start apache2

2. Enable the ssl module providing SSL* directives required by Apache.

   sudo a2enmod ssl

3. Edit virtual host example.local.conf, adding a new VirtualHost block for port 443 below the existing block, and converting port 80 to redirect.

   sudo nano /etc/apache2/sites-available/example.local.conf

   <VirtualHost *:80>
       ServerName example.local
       ServerAlias www.example.local
   
       Redirect permanent / https://example.local/
   </VirtualHost>
   
   <VirtualHost *:443>
       ServerName example.local
       ServerAlias www.example.local
       DocumentRoot /var/www/example.local/html
   
       SSLEngine on
       SSLCertificateFile /etc/ssl/example.local/example.local.crt
       SSLCertificateKeyFile /etc/ssl/example.local/example.local.key
       SSLProtocol -all +TLSv1.2 +TLSv1.3
   
       <Directory /var/www/example.local/html>
           Options -Indexes
           AllowOverride None
           Require all granted
       </Directory>
   
       ErrorLog ${APACHE_LOG_DIR}/example.local.error.log
       CustomLog ${APACHE_LOG_DIR}/example.local.access.log combined
   </VirtualHost>

   The SSLProtocol -all +TLSv1.2 +TLSv1.3 directive matches Nginx's ssl_protocols TLSv1.2 TLSv1.3; functionality by disabling old protocols before activating secure ones.
4. Repeat the pattern for blog.example.local.conf.

   sudo nano /etc/apache2/sites-available/blog.example.local.conf

   <VirtualHost *:80>
       ServerName blog.example.local
   
       Redirect permanent / https://blog.example.local/
   </VirtualHost>
   
   <VirtualHost *:443>
       ServerName blog.example.local
       DocumentRoot /var/www/blog.example.local/html
   
       SSLEngine on
       SSLCertificateFile /etc/ssl/example.local/example.local.crt
       SSLCertificateKeyFile /etc/ssl/example.local/example.local.key
       SSLProtocol -all +TLSv1.2 +TLSv1.3
   
       <Directory /var/www/blog.example.local/html>
           Options -Indexes
           AllowOverride None
           Require all granted
       </Directory>
   
       ErrorLog ${APACHE_LOG_DIR}/blog.example.local.error.log
       CustomLog ${APACHE_LOG_DIR}/blog.example.local.access.log combined
   </VirtualHost>

5. Test configuration syntax and restart Apache. Module changes require full restarts rather than simple reloads.

   sudo apache2ctl configtest
   sudo systemctl restart apache2

6. Test from another machine on the network.

   curl -k https://example.local
   curl -I http://example.local

7. Revert active services back to Nginx to match ending state conditions from Chapter 14 and Chapter 17 practices.

   sudo systemctl stop apache2
   sudo systemctl start nginx

Verification and Troubleshooting

• The UFW profile Apache Full enabled in Chapter 13 includes port 443, eliminating additional firewall rule requirements.
• If apache2ctl configtest throws SSL Library Error related to certificate files, check permissions on /etc/ssl/example.local/.
• The Redirect permanent directive from module mod_alias comes enabled by default in Apache, requiring no extra a2enmod setup.

17.4 Certificate Auto-renewal

Let's Encrypt certificates carry short 90-day lifespans, driving ecosystem automation over error-prone manual renewal processes. This section covers automated renewals for Let's Encrypt certificates alongside manual expiration tracking for self-signed lab certificates.

17.4.1 Let's Encrypt Auto-renewal via Certbot

The certbot APT package installed in Section 17.2.2 configures periodic renewal background jobs automatically. Scheduled via systemd timer (covered in Chapter 5), checks run twice daily across issued certificates. Certbot only renews certificates expiring within 30 days or fewer, keeping Let's Encrypt servers free from unnecessary requests.

Hands-on Steps

1. Check active renewal timers configured upon installation.

   systemctl list-timers | grep certbot

   Output indicates the next scheduled execution of certbot.timer.

For servers running active Let's Encrypt certificates from Section 17.2.4, these two management commands are essential:

sudo certbot renew --dry-run
sudo certbot certificates

The --dry-run flag simulates renewal processes and ACME challenges without overwriting active production certificates. Running certbot certificates lists managed certificates, expiration windows, and active paths in /etc/letsencrypt/live/.

Verification and Troubleshooting

• Because validation failed in Section 17.2.3, certbot renew --dry-run reports no managed certificates on this lab instance.
• If certbot.timer is missing from systemctl list-timers, verify timer service status via:

  systemctl status certbot.timer

17.4.2 Monitoring Self-Signed Certificate Expiration

Self-signed certificates created in Section 17.3.1 are unmanaged by Certbot, placing expiration tracking entirely on Sysadmins.

Hands-on Steps

1. Check expiration dates directly from certificate files.

   openssl x509 -enddate -noout -in /etc/ssl/example.local/example.local.crt

   Output displays a single notAfter= line matching 365 days from creation.
2. To track upcoming expirations programmatically, evaluate remaining valid days using OpenSSL commands wrapped in automated checks:

   openssl x509 -checkend 2592000 -noout -in /etc/ssl/example.local/example.local.crt && echo "Certificate valid for over 30 days" || echo "Certificate expires within 30 days"

   The -checkend flag evaluates seconds (2592000 seconds equals 30 days), matching Certbot's internal threshold.

Verification and Troubleshooting

• When lab certificates near expiration, rerun the openssl req command from Section 17.3.1 using original san.cnf configurations, followed by web server service reloads.
• For larger infrastructure demands, deploying private CA solutions like step-ca allows internal root certificate distribution across networks, streamlining local certificate issuance.

All virtual hosts built since Chapter 12 (example.local, www.example.local, blog.example.local, and app.example.local) now serve HTTPS with HTTP auto-redirection. Section IV covering web services concludes here. Chapter 18 opens Section V, shifting focus toward database administration with PostgreSQL 18.