</>EmbeddedInterview

How do you find out why a Linux program is slow or stuck?

MediumLinux System Programming

Short answer: First look at what the process is doing and waiting for: top/htop (CPU, state), ps -o stat,wchan (where it sleeps), strace -p PID (system calls), cat /proc/PID/stack (kernel stack, as root), and gdb -p PID / thread apply all bt (user stacks). Then measure the right resource: perf top/perf record for CPU, vmstat/iostat for memory and I/O, ss for sockets, and PSI files for contention.

Step by step

  1. Is it using CPU? top -H -p PID (per thread). High CPU → profile with perf record -g -p PID then perf report, or perf top.
  2. If not, what is it waiting for? State S or D in ps. strace -f -p PID shows the blocking call (read on a socket, futex = waiting on a lock, poll with a timeout). /proc/PID/wchan names the kernel function it sleeps in.
  3. Deadlock? gdb -p PID, thread apply all bt: look for threads waiting in pthread_mutex_lock on each other's locks.
  4. Memory? /proc/PID/status (VmRSS, VmSwap), smem, valgrind --tool=massif, or heaptrack for growth; vmstat 1 for swapping.
  5. I/O? iostat -x 1 (device utilisation and latency), iotop, pidstat -d.
  6. Network? ss -tinp (per-connection queues, retransmits, RTT), tcpdump/Wireshark.
  7. System-wide pressure: /proc/pressure/{cpu,memory,io} shows how much time tasks were stalled waiting for each resource.

Tracing tools for deeper problems: ftrace (trace-cmd, function graph, scheduler events), perf sched latency (scheduling delays), bpftrace/BCC tools (offcputime, biolatency, runqlat), and ltrace for library calls.

Embedded tip: many of these tools are missing from a minimal image. Build a debug image variant with perf, strace, gdbserver and trace-cmd, and use gdbserver + a cross gdb on your PC with the unstripped binaries from the build system.