Reverse Proxy and Load Balancing

In Chapter 12, the application backend behind location /app/ on domain example.local relied on only a single Python process running on port 8080. This configuration is sufficient for demonstration, but fragile if deployed in a production environment. Once the backend process crashes, restarts to deploy a new version, or gets overwhelmed with traffic, the entire domain immediately becomes inaccessible even though Nginx itself is still running normally in front of it. A single backend means a single point of failure, and growing traffic will sooner or later exceed the capacity of a single process.

This chapter continues the topic of reverse proxy discussed in Section 12.3.2, this time using more than one backend simultaneously. We will build an upstream containing several backend instances, distribute requests to all instances via load balancing, choose the algorithm that suits our needs, and add a simple health check so that Nginx knows when to stop sending traffic to a backend that is having issues. All practices use a new domain, app.example.local, following the pattern of adding subdomains to the example.local zone practiced for blog.example.local in Section 12.2.2.

16.1 Reverse Proxy Concepts and Load Balancing

As a reverse proxy, Nginx receives requests from Visitors and forwards them to the backend that processes the application logic, exactly as practiced in Chapter 12. Load balancing is an additional layer on top of this capability, which is Nginx's ability to distribute incoming requests to more than one backend running identical applications, rather than simply forwarding them to a single fixed destination. The group of backends ready to receive this distribution is called an upstream or backend pool, and the component that decides which backend receives each request is called a load balancer.

16.1.1 From One Backend to a Backend Pool

Two main reasons Sysadmins use load balancing in the field are scalability and availability. In terms of scalability, a single application server has a limited capacity due to CPU, memory, and the number of connections it can handle simultaneously. Adding new backends to the upstream is much cheaper and faster than upgrading the specifications of a single server, a pattern known as horizontal scaling. From an availability perspective, as long as there is at least one healthy backend in the pool, Nginx can still serve Visitors even if one of the other backends is down or undergoing deployment, something impossible to achieve with only one backend.

Open source Nginx, used throughout this series, natively provides load balancing and passive health check features via the ngx_http_upstream_module module without requiring special compilation or extra packages. Advanced capabilities like active health checks that inspect backends periodically without waiting for real Visitor requests will be discussed in Section 16.4.2, but those features are exclusive to Nginx Plus, the commercial product from the same company.

16.2 Nginx as a Reverse Proxy to Backend Applications

This section sets up two dummy backend instances and connects both to Nginx using a single upstream block and a new virtual host named app.example.local.

16.2.1 Setting Up Two Backend Instances for Simulation

Just like the demo in Section 12.3.2, we use Python's built-in HTTP server to avoid deploying a real application just to learn load balancing mechanisms. The difference is that this time there are two instances running on different ports, and each is intentionally given a distinct identity on its page so they are easy to differentiate via the response received by curl.

Practical Steps

1. Prepare the directory and page for the first backend.

   mkdir -p ~/demo-backend-1
   echo "Response from backend 1 (port 8080)" > ~/demo-backend-1/index.html

2. Prepare the directory and page for the second backend.

   mkdir -p ~/demo-backend-2
   echo "Response from backend 2 (port 8081)" > ~/demo-backend-2/index.html

3. Run both backends in separate tmux sessions (Chapter 3) so they remain active even when switching terminals. In the first session:

   cd ~/demo-backend-1
   python3 -m http.server 8080 --bind 127.0.0.1

   In the second session:

   cd ~/demo-backend-2
   python3 -m http.server 8081 --bind 127.0.0.1

4. From a third session, ensure both are truly listening before proceeding to the Nginx configuration.

   ss -tlnp | grep -E ':(8080|8081)'

Verification and Troubleshooting

• If one of the ports fails to bind because it is already used by another process, such as a leftover demo process from Chapter 12 still running on port 8080, terminate that old process first before running the new one. Find the process PID using the output of ss -tlnp above, then stop it using the kill command.
• Both backends are intentionally set to listen only on 127.0.0.1 rather than 0.0.0.0 because Visitors should never access the backend directly. The only proper entry path is through Nginx on port 80, in accordance with the reverse proxy principles discussed in Section 12.3.2.

16.2.2 Configuring upstream Block and app.example.local Virtual Host

The next step is to connect both backends to Nginx via an upstream block, then create a new virtual host that forwards all requests to that pool.

Practical Steps

1. Add a new DNS record for app.example.local in the example.local zone file managed since Chapter 9, following the same pattern as the blog record in Section 12.2.2.

   sudo nano /etc/bind/db.example.local

   app	IN	A	192.168.1.20

   Increment the Serial number in the SOA, save, and validate syntax before reloading, a habit established since Chapter 9.

   sudo named-checkzone example.local /etc/bind/db.example.local
   sudo systemctl reload bind9

2. Create a new virtual host configuration file.

   sudo nano /etc/nginx/sites-available/app.example.local

3. Fill it with the following upstream block and server block.

   upstream backend_app {
       server 127.0.0.1:8080;
       server 127.0.0.1:8081;
   }
   
   server {
       listen 80;
       listen [::]:80;
   
       server_name app.example.local;
   
       location / {
           proxy_pass http://backend_app;
           proxy_set_header Host $host;
           proxy_set_header X-Real-IP $remote_addr;
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_set_header X-Forwarded-Proto $scheme;
       }
   }

   The upstream block defines the pool name, backend_app, along with its members. This name is then used directly as the destination in proxy_pass, replacing a single backend address like http://127.0.0.1:8080 used in Chapter 12. The four proxy_set_header directives are retained exactly as in Section 12.3.2 for the same reason: the backend still needs to know the actual domain and real Visitor IP, rather than seeing Nginx as the source of all requests.
4. Enable this virtual host, test syntax, and apply.

   sudo ln -s /etc/nginx/sites-available/app.example.local /etc/nginx/sites-enabled/
   sudo nginx -t
   sudo systemctl reload nginx

5. Test from another computer on the same network.

   curl http://app.example.local

Verification and Troubleshooting

• If nginx -t fails with the message "upstream" directive is not allowed here, the upstream block was likely written inside the server { } block. The upstream block must sit parallel to the server block, not nested inside it.
• For clients that have not synchronized their DNS cache, use the --resolve flag as in Section 12.2.2 so testing does not depend on DNS propagation.

  curl --resolve app.example.local:80:192.168.1.20 http://app.example.local/

• A 502 Bad Gateway response at this stage usually indicates that both backends are not running. Repeat the ss -tlnp check from Section 16.2.1.

16.3 Basic Load Balancing: Round Robin and Least Connection

The configuration in Section 16.2.2 automatically performs load balancing, but we have not observed it directly or compared it to other algorithms. This section tests the behavior of the two most common algorithms: round robin as default, and least connection as an alternative for uneven workloads.

16.3.1 Round Robin as Default Algorithm

When the upstream block does not include an algorithm directive, as in the configuration just created, Nginx uses weighted round-robin by default. Each upstream member has a default weight of 1 unless specified otherwise, and requests are distributed sequentially and proportionally to that weight value. Since both backends have a weight of 1, requests should alternate between backend 1 and backend 2.

Practical Steps

1. Send several consecutive requests and observe the content of each response.

   for i in {1..6}; do curl -s http://app.example.local; echo; done

The output will alternate roughly as follows, though Nginx does not guarantee a rigid alternating pattern under all conditions, such as when one backend has just failed or recovered from down status.

Response from backend 1 (port 8080)
Response from backend 2 (port 8081)
Response from backend 1 (port 8080)
Response from backend 2 (port 8081)
Response from backend 1 (port 8080)
Response from backend 2 (port 8081)

Verification and Troubleshooting

• If all responses keep coming from the same backend, check via ss -tlnp to see if the other backend is truly still active. Nginx will not send requests to a backend whose connection is refused, so an apparently unbalanced result often happens because one backend is dead.
• Weights can be set explicitly using the weight parameter on the server line, for example server 127.0.0.1:8080 weight=3; for a backend with higher capacity. This scheme is useful when both backends run on unequal hardware, a scenario commonly encountered during gradual migrations to new servers.

16.3.2 Least Connection for Uneven Request Workloads

Round robin works well as long as every request takes relatively equal processing time. Issues arise when request durations vary significantly, for instance when some requests only read light data while others run heavy database queries. Round robin still divides request counts equally regardless of which backend is busy, meaning a backend receiving multiple heavy requests can continuously get overwhelmed with new requests even if its queue is full.

Least connection, enabled via the least_conn directive, addresses this problem by directing each new request to the backend with the fewest active connections at that moment, rather than simply taking turns sequentially. Let us prove the difference using an intentionally slowed backend.

Practical Steps

1. Stop backend 1 (Ctrl+C in its tmux session), then replace it with a small Python script that delays every response by 3 seconds to simulate a backend processing heavy requests.

   nano ~/demo-backend-1/slow_backend.py

   from http.server import BaseHTTPRequestHandler, HTTPServer
   import time
   
   class SlowHandler(BaseHTTPRequestHandler):
       def do_GET(self):
           time.sleep(3)
           body = b"Response from backend 1 (slow, port 8080)"
           self.send_response(200)
           self.send_header("Content-Length", str(len(body)))
           self.end_headers()
           self.wfile.write(body)
   
   HTTPServer(("127.0.0.1", 8080), SlowHandler).serve_forever()

   python3 ~/demo-backend-1/slow_backend.py

   The built-in http.server, including the HTTPServer class used by this script, processes one request at a time sequentially. Requests piling up on this backend will truly queue up, reproducing the exact effect we want to observe.
2. First, use the default round robin configuration from Section 16.3.1, send six parallel requests simultaneously, and measure total time.

   time (for i in {1..6}; do curl -s -o /dev/null http://app.example.local & done; wait)

   Because round robin still sends half of those six requests to the slow backend 1 and processes them one by one, total time will be dominated by the queue on that backend, roughly three times 3 seconds for requests going to backend 1.
3. Add least_conn; to the first line of the upstream block.

   sudo nano /etc/nginx/sites-available/app.example.local

   upstream backend_app {
       least_conn;
       server 127.0.0.1:8080;
       server 127.0.0.1:8081;
   }

4. Apply changes and repeat the same parallel test.

   sudo nginx -t
   sudo systemctl reload nginx
   time (for i in {1..6}; do curl -s -o /dev/null http://app.example.local & done; wait)

   Once the first request is directed to backend 1 and holds an active connection for 3 seconds, Nginx will favor backend 2 (which has fewer active connections) for subsequent requests. The total execution time should be significantly shorter than the round robin result in step two.

Verification and Troubleshooting

• Active connection tracking for least_conn is done per Nginx worker process rather than across all workers, unless the upstream block uses the zone directive to share state across workers. On production servers with worker_processes greater than one (the default in nginx.conf is auto, matching CPU core counts), least_conn effects may feel less precise than in this small single-VM demo because each worker makes decisions based on connections it monitors.
• After experimenting, stop slow_backend.py and restart the normal version of backend 1 from Section 16.2.1 before proceeding to the next section, ensuring health check tests do not mix with this slow backend simulation.

  cd ~/demo-backend-1
  python3 -m http.server 8080 --bind 127.0.0.1

The following quick summary helps decide which algorithm is more relevant to your application characteristics behind Nginx.

16.4 Simple Backend Health Checks

Load balancing alone is insufficient if Nginx keeps sending requests to a dead backend. This section adds a mechanism for Nginx to temporarily stop sending traffic to a backend that fails to respond until it is proven healthy again.

16.4.1 Passive Health Check with max_fails and fail_timeout

Open source Nginx provides passive health checks through two optional parameters on the server lines inside an upstream block: max_fails and fail_timeout. It is called passive because Nginx does not send probe requests to backends on its own; it merely observes the results of passing Visitor requests. The max_fails parameter sets how many consecutive failed attempts must occur before a backend is considered unavailable, while fail_timeout serves a dual purpose: a time window for counting failures and the duration a backend remains marked unavailable before being retested.

Practical Steps

1. Add max_fails and fail_timeout parameters to both server lines inside the upstream.

   sudo nano /etc/nginx/sites-available/app.example.local

   upstream backend_app {
       least_conn;
       server 127.0.0.1:8080 max_fails=2 fail_timeout=10s;
       server 127.0.0.1:8081 max_fails=2 fail_timeout=10s;
   }

2. Apply changes.

   sudo nginx -t
   sudo systemctl reload nginx

3. Stop backend 1 (Ctrl+C in its tmux session) to simulate a completely down backend, then send several consecutive requests.

   for i in {1..5}; do curl -s http://app.example.local; echo; done

   The first few requests likely succeed and display responses from backend 2, thanks to the proxy_next_upstream directive active by default with values error timeout. This means as soon as Nginx fails to connect to backend 1 for a request, it automatically tries the next backend in the pool without the Visitor seeing an error. The max_fails and fail_timeout parameters work on top of this mechanism, causing Nginx to stop trying backend 1 once its failure threshold is reached, so subsequent requests do not wait on connection attempts destined to fail.
4. Check the error log to see both messages directly.

   sudo tail -n 20 /var/log/nginx/error.log

   Look for lines mentioning connect() failed during connection attempts to backend 1, as well as lines stating upstream server temporarily disabled after reaching the max_fails threshold.
5. Wait longer than 10 seconds according to the configured fail_timeout, restart backend 1 normally, and send a few more requests to verify Nginx resumes sending traffic to it.

   cd ~/demo-backend-1
   python3 -m http.server 8080 --bind 127.0.0.1

   for i in {1..6}; do curl -s http://app.example.local; echo; done

Verification and Troubleshooting

• If an upstream contains only a single backend, max_fails and fail_timeout are automatically ignored by Nginx, and that single backend will never be marked unavailable despite continuous failures. This is a frequent trap in practice, particularly when Sysadmins add these parameters to an upstream containing only one server, assuming the feature is active when it actually has no effect.
• If both backends go down simultaneously, Visitors still receive a 502 Bad Gateway error because no healthy backend is available. Health checks only help Nginx select healthy backends from available choices; they do not revive dead backends.

16.4.2 Limitations of Passive Health Checks and Active Health Check Alternatives

The passive health check practiced above has one fundamental limitation: Nginx only discovers a backend issue after a real Visitor request fails through it. This means at least a few real requests act as "victims" before Nginx marks the backend as unavailable, though thanks to proxy_next_upstream, most Visitors still receive proper responses through automatic retries to another backend.

Active health checks work differently. Nginx sends probe requests to backends periodically on its own, independent of Visitor traffic, detecting problematic backends and removing them from the pool before affecting Visitors. Unfortunately, this feature is exclusive to Nginx Plus, the commercial product from F5, and is absent in open source Nginx used throughout this series. For production environments requiring active health checks without subscribing to Nginx Plus, common field alternatives include using orchestrators like Docker or Kubernetes (which have built-in liveness probes outside Nginx), or writing custom health check scripts executed via systemd timers (Chapter 35) to monitor backends and dynamically update upstream configurations as needed.

For small to medium workloads focused on in this series, passive health checks via max_fails and fail_timeout are usually sufficient, especially since the configuration is simple and adds no probe overhead to the backends.

At this stage, domain app.example.local is served by two backends simultaneously using a single upstream block, utilizing least connection load balancing for fairer distribution of uneven workloads, and passive health checks ensuring Nginx automatically avoids down backends. Chapter 17 completes all virtual hosts built since Chapter 12, including app.example.local, by adding an HTTPS layer via Let's Encrypt so communication between Visitors and the server no longer runs in plain HTTP.