System troubleshooting is a fundamental skill for system administrators, DevOps engineers, and hobbyists alike. When a service crashes, a server hangs, or network packets drop into a black hole, having a structured, logical methodology makes the difference between a minor hiccup and a major outage. Every dedicated Linux Dork knows that diagnosing system issues relies on knowing which diagnostic tools to run and how to interpret their output.
This comprehensive guide breaks system troubleshooting into four primary diagnostic pillars: CPU/Memory, Storage I/O, Networking, and Process Management.
Contents
Pillar 1: Diagnosing CPU and Memory Bottlenecks
When system responsiveness slows down, the first suspect is usually resource exhaustion.
1. Real-Time Resource Monitoring
Start with top or the modern, colorful alternative htop. Look at the Load Average at the top of the output. Load average represents the average number of processes in a runnable or uninterruptible state over 1, 5, and 15-minute intervals.
- If the load average exceeds the total number of CPU cores, your system is experiencing queuing delays.
- Look at the %wa (IO Wait) value. High CPU usage combined with high IO wait indicates that the CPU is idling while waiting for disk operations to complete, pointing to a storage bottleneck rather than a CPU limitation.
2. Investigating RAM Exhaustion and OOM Kills
Check available system memory with free -h. Pay attention to the available column rather than free, as Linux aggressively uses unused RAM for file caching. If your system suddenly terminates a process (like a database or web server), check the kernel ring buffer for Out-Of-Memory (OOM) killer events:
dmesg -T | grep -i oom
If the kernel killed a process due to low memory, you will see explicit records here showing which process was terminated.
Pillar 2: Storage and Disk I/O Bottlenecks
A full disk or a bottlenecked disk subsystem can bring an entire operating system to a halt.
1. Checking Disk Usage
Run df -h to verify space across mounted filesystems. If a partition shows 100% usage, applications will fail to write logs, temporary files, or database records.
2. The Hidden Culprit: Inode Exhaustion
Sometimes df -h shows plenty of available space, but you still receive “No space left on device” errors. This happens when you run out of inodes (index nodes that track file metadata). Check inode usage with:
df -i
If inode usage is at 100%, you must locate and remove directories containing millions of tiny files (such as orphaned session files or uncleaned mail queues).
3. Analyzing I/O Operations
To pinpoint which process is thrashing the hard drives, install sysstat and run iostat or use iotop:
sudo iotop -o
This displays real-time disk read/write throughput per process, making it easy to spot rogue logging scripts or unindexed database queries.
Pillar 3: Resolving Network Connectivity Issues
When network services become unreachable, follow the standard OSI model approach to identify where the failure occurs.
1. Physical and Interface Layer
Check if your network interface is UP and assigned an IP address:
ip a
Verify local route tables with ip route to ensure the default gateway is properly configured.
2. Testing Connectivity and Routing
Test basic connectivity with ping. If hostnames fail to ping but public IP addresses respond, you are facing a DNS resolution issue. Inspect /etc/resolv.conf or test DNS resolution explicitly using dig or nslookup:
dig example.com
If traffic drops along the route, run traceroute (or mtr for continuous real-time trace statistics) to identify the exact router or hop where packet loss begins.
3. Inspecting Listening Ports
To see if a local daemon is listening on the expected port, use ss (which replaces the deprecated netstat):
sudo ss -tulpn
This command lists all listening TCP/UDP ports alongside their associated process names and PIDs.
Pillar 4: Inspecting System Logs with Journalctl
Modern Linux distributions rely on systemd to manage daemons and collect logs. The journalctl utility is your primary diagnostic tool when services fail to start.
- View logs for a specific service:
sudo journalctl -u nginx.service -n 50 --no-pager - Follow live logs in real time:
sudo journalctl -f -u mysql - Filter logs by boot session or priority:
sudo journalctl -p err..emerg -b
System Troubleshooting Checklist
|
| Problem Area | Primary Command | Secondary Diagnostic | | High Load / Slow System | htop | `dmesg -T | | Disk Operations | df -h & df -i | iotop -o | | Network Failure | ip a / ping | ss -tulpn / dig | | Service Failure | systemctl status <service> | journalctl -u <service> -e |
By approaching system errors systematically—checking resource utilization, disk health, network paths, and service logs—you can resolve complex technical issues calmly and efficiently.