Server Troubleshooting

Chapter 42 closed with a candid admission: the demo-node deployment pipeline we just built does not yet include automated health checks. The final step of the GitHub Actions workflow, "Deliver release via SSH", was deemed successful as soon as tar finished extracting and systemctl restart finished executing, not when the application was actually proven to remain running afterward. The scenario referenced in the closing of that chapter is now actually happening. The Actions tab on the demo-node repository displays a green checkmark on all steps, but a User attempting to open http://192.168.1.40:3000 finds their connection refused. The Sysadmin receiving this report does not yet know whether the problem lies in the application code, the systemd service, the network, or even the database server used by another application in the same infrastructure.

Moments like this are the core of troubleshooting, which is the process of finding the exact cause of a problem and fixing it, strictly distinguished from merely changing configurations one by one while hoping one of them happens to work. All previous chapters equipped us with specific tools and services, ranging from networking in Part III, web services in Part IV, databases in Part V, to automation in Part IX. This chapter does not introduce new tools. This chapter unites everything through the same mindset, used repeatedly by experienced Sysadmins whenever something in production stops working as expected.

The discussion begins with a methodological framework so that the root cause analysis process does not rely on guesswork, followed by a complete map of logs as the primary source of truth, then dives into four of the most commonly encountered field scenarios: services failing to start, suddenly full disks, network or DNS connection issues, and inaccessible databases. The chapter concludes with how to seek help effectively for cases that are genuinely beyond the reach of our own knowledge.

43.1 Systematic Troubleshooting Methodology in Production Environments

43.1.1 Why Guessing Becomes the Most Expensive Habit on Production Servers

The most common reaction when a service suddenly encounters issues is to change several things at once in rapid succession: restart the service, then restart the server, then change a line of configuration that looks suspicious, all within minutes due to the pressure to resolve the issue quickly. This approach sometimes happens to work, but leaves behind a question far more dangerous than the original issue itself: which change actually fixed the situation. Without a definitive answer, the exact same issue has the potential to reappear at an unexpected time, and the Sysadmin will repeat the entire guessing process from scratch because they never truly knew the root cause.

In practice, this habit of guessing worsens precisely when it is most dangerous: when an incident occurs outside working hours and the pressure to finish quickly is far higher than during calm periods. Panic drives Sysadmins to alter multiple variables simultaneously, whereas the actual need is quite the opposite: a calmer and more structured approach, rather than a more hurried one. This section offers a five-step framework that keeps the process systematic, even when the clock reads two in the morning.

43.1.2 Five-Step Framework: Define, Collect Data, Hypothesize, Test, Verify

The following framework is not a rigid procedure to be followed word for word, but rather a thought process that prevents the two most common errors in troubleshooting: jumping to conclusions before gathering sufficient data, and changing multiple things at once so that the true cause is never actually known.

1. Define the symptoms precisely. "Application error" is not a traceable definition. "User cannot open http://192.168.1.40:3000, connection refused, started occurring after deployment at 23:40" is far more actionable because it already specifies the target, the exact symptom, and the onset time.
2. Collect data before theorizing. Read the logs and service status first (Section 43.2), then form a hypothesis. The reverse habit, assuming the cause first and then looking for logs to support that assumption, tends to make Sysadmins ignore evidence that actually points to another cause.
3. Form one hypothesis, then test it. A good hypothesis can be answered true or false through a single concrete check, for example, "the service failed to start because port 3000 is already in use by another process" can be tested directly via ss -tulnp, rather than "it seems something is wrong on the server".
4. Change one variable at a time, document every change. The greatest temptation under pressure is to modify multiple configurations at once. Resisting that temptation is the only way to ensure that a successful fix can truly be repeated in the future, rather than being a coincidence.
5. Verify the fix, then document the root cause. Having the service turn back on is not the end of the process. Record the true cause (root cause) and preventive measures so that the same incident does not escape twice, precisely like the evaluation touched upon in Section 42.5.3 regarding the importance of the automated health check that we had not yet built.

Before diving deep into logs, a quick habit worth performing at the start of an incident is checking from the outside in: whether the server is reachable over the network, whether the service itself is running, and only then reading the application logs. This quick inspection sequence is immediately put into practice during the demo-node incident that opened this chapter.

ping -c 3 192.168.1.40
curl -v http://192.168.1.40:3000
ssh [email protected] "systemctl status demo-node.service"

A successful ping indicates that the basic network and host are still alive, so the problem is unlikely to be at the network layer (Section 43.3.3). A curl -v failing with Connection refused on this attempt means no process is listening on port 3000, pointing suspicion directly at the service itself (Section 43.3.1), distinct from a curl failing with a timeout without any rejection message at all, which typically points to firewalls or routing instead.

43.2 Essential Logs as the First Source of Truth

43.2.1 journalctl -b: Reading Logs Since the Last Boot

Section 39.1 already covered journalctl in depth for filtering by unit, time, and priority. One flag that has not been widely mentioned but is extremely useful when troubleshooting post-reboot or post-crash is -b, which limits the output strictly to logs since a specific boot.

journalctl -b

Without additional arguments, -b displays logs since the currently running boot. Negative arguments point to previous boots, which is very useful when a server has rebooted on its own without the Sysadmin's knowledge, a symptom that often indicates hardware issues, running out of memory triggering the OOM killer, or a kernel panic.

journalctl --list-boots
journalctl -b -1
journalctl -b -1 -p err

journalctl --list-boots displays a list of all boots still stored in the journal, complete with boot IDs and their time ranges. journalctl -b -1 opens the log of the boot prior to the current one, and adding -p err narrows it down to messages with severity levels of error and above, a reasonable first step when suspecting an unclean server reboot overnight.

43.2.2 /var/log/ Map: System Logs vs. Third-Party Application Logs

Section 39.1.1 explained that systemd-journald captures logs from all systemd services centrally. However, not all software writes logs solely through journald. Some popular tools, especially those created before systemd became standard or deliberately designed cross-platform, continue writing their own plaintext logs to /var/log/, independently of journald. Understanding this map is important so that incident response time is not wasted looking for logs in the wrong place.

The safest approach when unsure where a service writes its logs is to check via journalctl -u service_name.service first. If the result is empty or contains only start/stop messages without details, only then investigate possible separate log files in /var/log/, usually located in a subdirectory named after the package.

43.2.3 Filtering Logs Quickly During an Ongoing Incident

Time is the most limited resource while an incident is ongoing. The following filter combinations accelerate the process of finding relevant lines compared to reading logs sequentially from the beginning.

journalctl -u demo-node.service --since "-15min" --no-pager
journalctl -u demo-node.service -u nginx.service --since "-15min"
journalctl -u demo-node.service -p err -b

--since "-15min" narrows the log to just the last 15 minutes, far more relevant when investigating a newly occurred incident than reading the entire history since boot. Specifying -u more than once, as in the second line above, merges logs from two different services into a single chronological timeline, a technique that is extremely helpful when looking at event sequences across layers, such as a request entering Nginx right before the backend application behind it records an error.

An equally important technique is correlating timestamps across data sources. In the demo-node incident at the opening of this chapter, the release directory name in /opt/demo-node/releases/ itself has been a YYYYMMDDHHMMSS timestamp since Section 42.3.3. Thus, comparing the timestamp of the latest release with the timestamp of the first error line in the journal immediately shows whether that error indeed appeared right after deployment, or had already existed previously due to another cause.

ls -1 /opt/demo-node/releases/ | tail -n1
journalctl -u demo-node.service --since "2026-08-28 23:40:00"

43.3 Common Production Server Troubleshooting Scenarios

43.3.1 Service Fails to Start: Case Study of demo-node on app01

This section resolves the incident that opened the chapter: curl to demo-node refuses connections right after the latest deployment, even though the CI/CD pipeline in Section 42.4.2 reported complete success.

Practical Steps

1. Log in to app01, then inspect the service status. The Active line is the most important item to read first.

   ssh [email protected]
   sudo systemctl status demo-node.service

2. The output shows the status repeatedly transitioning between activating (auto-restart) and failed, with the Main PID line stating code=exited, status=1/FAILURE. Status 1/FAILURE indicates that the Node.js process exited on its own due to an unhandled exception, distinct from 203/EXEC which means the binary path in ExecStart is incorrect, or 127 which means the command was not found at all.
3. Read the exception details via journal, restricted to recent lines so as not to get buried in repetitive restart loops.

   journalctl -u demo-node.service -n 30 --no-pager

4. The log shows lines such as the following, repeating every time systemd attempts a restart.

   SyntaxError: Unexpected token ')'
       at wrapSafe (node:internal/modules/cjs/loader:1378:18)
       at Module._compile (node:internal/modules/cjs/loader:1428:20)

5. This is a bug in the newly pushed code that bypassed the CI test stage because node --test in Section 42.2.3 only tested the buildResponseText function, while this SyntaxError originates from another line untouched by that test. Rather than attempting to patch code directly on the production server, use rollback.sh, which was prepared specifically for this situation in Section 42.5.2.

   sudo /opt/demo-node/rollback.sh

Verification and Troubleshooting

• Ensure the service becomes stable again and the application responds normally.

  sudo systemctl status demo-node.service
  curl http://192.168.1.40:3000

  The line Active: active (running) remaining stable without shifting status for several minutes indicates the rollback was successful.
• Units without explicit StartLimitIntervalSec or StartLimitBurst, such as demo-node.service since Section 15.6.2, inherit systemd's default values of a maximum of five start attempts within ten seconds. If this limit is exceeded, systemd stops trying, and the status changes permanently to failed with a start-limit-hit message, a situation requiring sudo systemctl reset-failed demo-node.service to be run before the service can be restarted, even if the primary cause has already been fixed.
• Two other equally common causes for the "service failed to start" symptom, although not occurring in this specific incident, remain worth checking first in similar cases. A port already occupied by another process produces Error: listen EADDRINUSE in the log, checked via sudo ss -tulnp | grep 3000. Incorrect permissions on application files produce EACCES, generally because a new release was extracted with wrong ownership, checked via ls -l /opt/demo-node/current and compared against the nodeapp user that should be running the service since Section 15.6.2.
• An honest next step that needs to be mentioned here is addressing the root cause according to Step 5 of the framework in Section 43.1.2: adding tests that cover this failed scenario in the demo-node repository, and considering an automated health check before moving the symlink as touched upon in Section 42.5.3, so that subsequent rollbacks happen automatically without waiting for User reports.

43.3.2 Disk Full on a Production Server

A full disk is unlike high CPU usage, which typically recovers on its own once the load decreases. A full disk tends to progressively worsen until handled manually, and once completely exhausted, many services cease operating with confusing symptoms because they can no longer write their own log or temporary files. The scenario in this section inspects the main server (192.168.1.10) after the Netdata dashboard from Section 38.2.3 showed disk utilization exceeding 90 percent.

Practical Steps

1. First, confirm which filesystem is actually full, as a single server can have several separate mount points with different capacities.

   df -h

2. After identifying the problematic mount point, such as /, trace which directories consume the most space, starting with /var because that is where most continuously growing data is typically stored.

   sudo du -sh /var/* 2>/dev/null | sort -rh | head -10

3. The four most common causes found via the command above, ordered from most frequently encountered in the field, are APT package cache, accumulated old kernels, Docker images and build cache, and custom application logs not yet included in logrotate rules. Clear the APT cache first, as it is the safest and quickest way to gain breathing room.

   sudo apt clean

4. Check if old kernels have accumulated in /boot, a partition that is often intentionally made small, making it one of the first points to fill up.

   df -h /boot
   dpkg -l 'linux-image-*' | grep ^ii
   uname -r

5. Remove old, unused kernels via autoremove, which is far safer than manually removing them one by one with dpkg -r because APT knows which packages are safe to discard without breaking dependencies.

   sudo apt autoremove --purge

6. If this server also runs Docker since Chapter 26 or 27, check how much space is consumed by unused images, stopped containers, and build cache.

   docker system df

7. Clean up idle Docker resources. Add -a only if completely certain that images not currently used by any container are also safe to delete.

   docker system prune -a

8. The fourth cause, custom application logs not yet included in logrotate rules, is most frequently missed because it is not visible via du -sh /var/* in Step 2 if those logs are written to subdirectories that are rarely checked. Directly inspect large log files across all of /var/log/, not just in suspected directories.

   sudo find /var/log -type f -size +200M -exec ls -lh {} \;

9. Files as large as /var/log/remote/*.log resulting from forwarding since Section 39.2 are the most relevant example in this infrastructure, because those logs only received rotation rules via custom configuration in Section 39.3.2. If similar files are found without any rotation rules, check whether an entry already exists in /etc/logrotate.d/ before adding a new one.

   ls /etc/logrotate.d/
   sudo logrotate -d /etc/logrotate.d/remote-logs

   The -d flag runs logrotate in dry run mode, showing what would be done without actually rotating log files, a safe way to ensure existing rules run as scheduled before blaming their configuration.

Verification and Troubleshooting

• Repeat df -h after each cleanup step to observe the impact incrementally, rather than performing all steps at once and only then checking the result, aligning with the "change one variable at a time" principle in Section 43.1.2.
• Never delete the currently running kernel. Always compare the output of uname -r with the linux-image package list before manually removing anything. apt autoremove in Step 5 automatically excludes the active kernel and at least one previous kernel as a backup, but the habit of manual verification remains important before typing a dpkg -r command yourself.
• docker system prune -a is destructive toward images not tagged as used by any container. On a server with a CI/CD pipeline like app01 in Chapter 42, this means the next build will need to re-download base images from the registry, slowing down the first post-cleanup deployment, a reasonable trade-off for disk space that should nevertheless be communicated honestly to the team before running during peak hours.
• If all cleanup steps above have been performed and the disk remains near full because application needs are growing naturally, the correct solution is not continuous cleaning, but expanding capacity. LVM-based volumes discussed in Section 7.2 can be expanded without downtime as long as the volume group has remaining space, which is far more practical than migrating data to a new disk.

43.3.3 Network Connection and DNS Issues

Network issues are often misleading because symptoms on the client side look identical even when causes stem from vastly different layers, ranging from severed cables, incorrect IPs, firewall blocks, to DNS failing to resolve names to addresses. This section covers two real scenarios: a connection suddenly refused due to self-inflicted actions, and failed DNS resolution.

Practical Steps

1. First scenario: A Sysadmin in a hurry mistypes their SSH password to app01 multiple times, after which subsequent connections suddenly timeout completely, even after typing the correct password. Test first whether the host is still alive at the most basic network layer.

   ping -c 3 192.168.1.40

2. ping succeeds, meaning the host is alive and ICMP is not blocked, yet SSH continues to fail. This indicates a specific issue on port 22, rather than a routing issue or a dead host. Test the connection to that port separately from ping.

   nc -zv 192.168.1.40 22

3. Since multiple failed login attempts are the most suspicious factor, inspect Fail2ban's SSH jail status from another server that can still connect to app01, such as via the fallback console access in Section 3.4 or from the main server whose IP is not banned.

   sudo fail2ban-client status sshd

4. If our own IP appears on the Banned IP list, that is the root cause: Fail2ban since Chapter 32 is designed to block any IP after several failed login attempts within a certain timeframe, regardless of whether that IP belongs to a real Attacker or an unlucky Sysadmin who mistyped. Unban this IP using the same command practiced in Section 32.5.

   sudo fail2ban-client set sshd unbanip 192.168.1.40

   Replace the IP value in this command with the IP of the Sysadmin's own workstation, not the IP of app01.
5. Second scenario: An application on another server fails to download dependencies from the internet with a message mentioning domain name resolution failure. First, test DNS resolution against the internal DNS server built in Chapter 9.

   dig @192.168.1.10 registry.npmjs.org +short

6. If the command above returns no address, check whether BIND9 on that server is still running, then check whether the allow-recursion configuration in Section 9.4 permits recursive queries from the request's source subnet.

   sudo systemctl status bind9.service
   sudo journalctl -u bind9.service -n 30 --no-pager

7. Also check the client side, ensuring that the resolver actually used by the system points to the internal DNS server, rather than remaining set to an old resolver or ISP default.

   resolvectl status

Verification and Troubleshooting

• After unbanning, test SSH normally again, and ensure passwords are not mistyped in short succession, as the same jail will immediately re-block once the threshold is reached again.

  ssh [email protected]

• Repeated password mistyping leading to self-lockout happens far more frequently than it appears, especially in automation scripts storing old credentials after password updates. A practice more resilient against this issue is switching entirely to key-based authentication as recommended since Section 3.2.2, because connection attempts with incorrect keys are rejected instantly without repeated retries triggering Fail2ban jails.
• If dig in Step 5 returns a SERVFAIL response instead of empty results, that typically points to an issue in the zone file itself, such as a misconfigured or unreachable forwarder to public DNS, distinct from dig timing out completely, which leans toward a dead BIND9 service or port 53 blocked by a firewall.
• For internal domains (unlike public domains in Step 5), add suspicion toward the zone file's serial number. Record changes in the zone file of Section 9.2.1 that forget to increment the serial will never be seen by secondary DNS servers, as DNS replication mechanisms rely on serial numbers to detect new changes rather than inspecting file contents directly.

43.3.4 Database Inaccessible

A Developer reports that their web application failed to connect to the PostgreSQL database at 192.168.1.20 configured since Section 18.3. The reported error message heavily determines the direction of investigation, as three different error types point to three completely different problem layers: connection refused means nothing is listening on the destination port, a timeout without rejection means something is silently blocking along the route, and messages mentioning passwords or host not allowed mean network connectivity succeeded but was rejected at the PostgreSQL authentication layer.

Practical Steps

1. Ask the Developer to copy the exact error message rather than paraphrasing. In this case, the message is psql: error: connection to server at "192.168.1.20", port 5432 failed: Connection refused, which immediately points suspicion to the network layer or service, not authentication.
2. Log in to the database server, then verify whether the PostgreSQL service is actually running.

   ssh [email protected]
   sudo systemctl status postgresql
   pg_isready

3. If the service is active and operating normally, connection refused from the outside typically means PostgreSQL is listening only on the loopback address and has not been opened to network connections as configured in Section 18.4.1. Check which addresses are currently being listened on.

   sudo ss -tulnp | grep 5432

4. Compare the result with the contents of postgresql.conf. If the address listed is only 127.0.0.1:5432 without the actual server IP, it indicates that the listen_addresses line was not changed or reverted to default after a PostgreSQL upgrade.

   sudo grep listen_addresses /etc/postgresql/*/main/postgresql.conf

5. If listen_addresses is correct but connections are still rejected from specific IPs, check pg_hba.conf, where per-host authorization rules are determined separately from listening addresses.

   sudo cat /etc/postgresql/*/main/pg_hba.conf

   PostgreSQL evaluates pg_hba.conf lines top-down and stops at the first matching line, making line order as important as line content. Rules allowing an old subnet but not updated after deploying a new application server outside that subnet are the most common cause found at this step.
6. Finally, ensure the firewall on the database server permits port 5432 from the relevant IP, following the same pattern as UFW rules for other services since Chapter 31.

   sudo ufw status numbered

Verification and Troubleshooting

• After applying fixes to listen_addresses, pg_hba.conf, or firewall rules, ask the Developer to repeat the exact connection attempt as performed in Section 18.4.2.

  psql -h 192.168.1.20 -U webapp_user -d webapp_db

• Changes to postgresql.conf or pg_hba.conf require a reload to take effect, without needing a full restart that would disconnect active connections.

  sudo systemctl reload postgresql

• If the error message from the beginning was not connection refused but rather FATAL: sorry, too many clients already, the root cause is entirely different from all steps above: the number of connections has reached the max_connections limit. Check via the following query, and suspect application-side connection leaks (connections opened but never closed) as the most common cause, rather than actual traffic being that high.

  SELECT count(*) FROM pg_stat_activity;
  SHOW max_connections;

• Another possibility that honestly needs checking before blaming network configuration is a full disk on the database server itself, per Section 43.3.2. A PostgreSQL server unable to write its write-ahead log due to disk exhaustion will reject new connections with messages that can be confusing and do not explicitly mention the word "disk", making df -h on the database server one of the earliest checks worth performing rather than the last.

43.4 Seeking Help Effectively

43.4.1 Formulating Questions That Others Can Answer

No matter how powerful the framework in Section 43.1 is, there are times when a problem is genuinely beyond our own experience, and asking the community or consulting external references becomes a reasonable step, not a sign of failure. What separates questions that get answered quickly from those left without responses for days is how complete and precise the information provided is from the outset.

Questions like "my server has an error, please help" are nearly impossible for anyone to answer because they do not mention anything actionable. Contrast this with questions that include the following components, the exact same pattern used by open-source communities when asking bug reporters to complete their reports:

• Relevant OS and software versions, retrieved from commands that can be run directly and copied, rather than typed from memory.

  cat /etc/os-release
  dpkg -l | grep nginx

• The exact command executed, copied as-is without paraphrasing.
• Complete error messages, copied directly from the terminal, not summarized in your own words which might strip vital details like line numbers or error codes.
• Relevant log snippets from mapped sources in Section 43.2.2, trimmed sufficiently so as not to overwhelm readers with irrelevant lines.
• Everything already attempted, along with their respective results, so that helpers do not suggest steps that have already been tried and failed.

An equally vital habit before pasting logs or configuration files to public forums is sanitizing sensitive data beforehand, such as passwords, actual public IPs, or internal company domain names. Even seemingly harmless configuration lines, such as a complete pg_hba.conf file, can leak internal network schemas to an Attacker reading the same forum.

43.4.2 Trusted Sources: Ask Ubuntu, Server Fault, and Official Documentation

Not all online help sources are created equal. Before asking humans directly, a habit that is far faster and almost always worth trying first is searching for the exact error message (enclosed in quotation marks so search engines do not break it into separate words), as chances are another Sysadmin has encountered and discussed the exact same issue before.

Man pages remain worthy of mention as the most practical primary source during an ongoing incident, as they are always available locally without depending on internet connectivity, which might itself be part of the issue being investigated.

man systemctl
man pg_hba.conf

Chapter 44 discusses further communities and advanced learning sources beyond daily troubleshooting, including the official Ubuntu Server Discourse and communities like r/linuxadmin, as part of the overall series recap.

At this point, we have seen how all the skills built since Chapter 1 connect precisely when needed most: when something stops working. The systematic framework in Section 43.1 prevents panic from turning into guesswork, the log map in Section 43.2 indicates where to look for evidence, the four scenarios in Section 43.3 demonstrate how that framework is applied to real-world issues, and Section 43.4 ensures we know when and how to ask for help without wasting others' time or our own. Troubleshooting is not a skill fully mastered from a single chapter, but a continuous habit honed every time it is practiced on a new problem. Chapter 44 concludes this entire series by reflecting on the journey completed, as well as the pathways available to advance further as a Sysadmin.