</>EmbeddedInterview

How do you debug a kernel oops or panic?

MediumLinux Kernel & Device Drivers

Short answer: Read the oops message: it gives the reason (e.g. "Unable to handle kernel NULL pointer dereference at virtual address 0000000000000010"), the faulting instruction (pc : my_func+0x48/0x120 [mymod]), registers and the call trace. Translate the offset to a source line with scripts/faddr2line or gdb on a debug build, reproduce with debugging options enabled (KASAN, lockdep, KCSAN, DEBUG_ATOMIC_SLEEP), and use tracing (ftrace, dynamic debug) to see what led up to it. For crashes that stop the system, capture logs over serial, with pstore/ramoops, or with kdump.

Reading an oops

Unable to handle kernel NULL pointer dereference at virtual address 0000000000000010
pc : mydev_irq+0x24/0x80 [mydev]
lr : __handle_irq_event_percpu+0x5c/0x1e0
...
Call trace:
 mydev_irq+0x24/0x80 [mydev]
 handle_irq_event+0x4c/0xa8
 ...
  • Address 0x10 → dereferencing a field at offset 16 of a NULL struct pointer.
  • pc gives function + offset/size; lr the caller.
  • "Tainted: G O" flags show out-of-tree (O) or proprietary (P) modules are loaded.
./scripts/faddr2line drivers/misc/mydev.ko mydev_irq+0x24/0x80
# or
gdb drivers/misc/mydev.ko -ex 'list *(mydev_irq+0x24)'
./scripts/decode_stacktrace.sh vmlinux < oops.txt      # decodes a whole trace

Toolbox

Problem Tool
use-after-free, out-of-bounds KASAN (CONFIG_KASAN), KFENCE (low-overhead, for production)
locking errors, deadlocks lockdep (CONFIG_PROVE_LOCKING)
data races KCSAN
sleeping in atomic context CONFIG_DEBUG_ATOMIC_SLEEP
memory leaks kmemleak
hangs soft/hard lockup and hung task detectors, echo t > /proc/sysrq-trigger, echo l > ... (backtraces of all CPUs)
understanding flow ftrace (function_graph), tracepoints, trace-cmd/KernelShark, pr_debug with dynamic debug, bpftrace
crash dumps kdump + crash utility, pstore/ramoops (logs survive reboot)
live debugging KGDB over serial, JTAG/OpenOCD with vmlinux symbols, QEMU + gdb (-s -S)

Method: make it reproducible, simplify the test case, check recent changes (git bisect), and check the obvious: reference counting, missing locking, use after remove(), IRQ firing before initialisation completes.