How do you find out why a Linux program is slow or stuck?
Short answer: First look at what the process is doing and waiting for: top/htop (CPU, state), ps -o stat,wchan (where it sleeps), strace -p PID (system calls), cat /proc/PID/stack (kernel stack, as root), and gdb -p PID / thread apply all bt (user stacks). Then measure the right resource: perf top/perf record for CPU, vmstat/iostat for memory and I/O, ss for sockets, and PSI files for contention.
Step by step
- Is it using CPU?
top -H -p PID(per thread). High CPU → profile withperf record -g -p PIDthenperf report, orperf top. - If not, what is it waiting for? State
SorDinps.strace -f -p PIDshows the blocking call (readon a socket,futex= waiting on a lock,pollwith a timeout)./proc/PID/wchannames the kernel function it sleeps in. - Deadlock?
gdb -p PID,thread apply all bt: look for threads waiting inpthread_mutex_lockon each other's locks. - Memory?
/proc/PID/status(VmRSS,VmSwap),smem,valgrind --tool=massif, orheaptrackfor growth;vmstat 1for swapping. - I/O?
iostat -x 1(device utilisation and latency),iotop,pidstat -d. - Network?
ss -tinp(per-connection queues, retransmits, RTT),tcpdump/Wireshark. - System-wide pressure:
/proc/pressure/{cpu,memory,io}shows how much time tasks were stalled waiting for each resource.
Tracing tools for deeper problems: ftrace (trace-cmd, function graph, scheduler events), perf sched latency (scheduling delays), bpftrace/BCC tools (offcputime, biolatency, runqlat), and ltrace for library calls.
Embedded tip: many of these tools are missing from a minimal image. Build a debug image variant with perf, strace, gdbserver and trace-cmd, and use gdbserver + a cross gdb on your PC with the unstripped binaries from the build system.