How do you debug a kernel oops or panic?
Short answer: Read the oops message: it gives the reason (e.g. "Unable to handle kernel NULL pointer dereference at virtual address 0000000000000010"), the faulting instruction (pc : my_func+0x48/0x120 [mymod]), registers and the call trace. Translate the offset to a source line with scripts/faddr2line or gdb on a debug build, reproduce with debugging options enabled (KASAN, lockdep, KCSAN, DEBUG_ATOMIC_SLEEP), and use tracing (ftrace, dynamic debug) to see what led up to it. For crashes that stop the system, capture logs over serial, with pstore/ramoops, or with kdump.
Reading an oops
Unable to handle kernel NULL pointer dereference at virtual address 0000000000000010
pc : mydev_irq+0x24/0x80 [mydev]
lr : __handle_irq_event_percpu+0x5c/0x1e0
...
Call trace:
mydev_irq+0x24/0x80 [mydev]
handle_irq_event+0x4c/0xa8
...
- Address 0x10 → dereferencing a field at offset 16 of a NULL struct pointer.
pcgives function + offset/size;lrthe caller.- "Tainted: G O" flags show out-of-tree (O) or proprietary (P) modules are loaded.
./scripts/faddr2line drivers/misc/mydev.ko mydev_irq+0x24/0x80
# or
gdb drivers/misc/mydev.ko -ex 'list *(mydev_irq+0x24)'
./scripts/decode_stacktrace.sh vmlinux < oops.txt # decodes a whole trace
Toolbox
| Problem | Tool |
|---|---|
| use-after-free, out-of-bounds | KASAN (CONFIG_KASAN), KFENCE (low-overhead, for production) |
| locking errors, deadlocks | lockdep (CONFIG_PROVE_LOCKING) |
| data races | KCSAN |
| sleeping in atomic context | CONFIG_DEBUG_ATOMIC_SLEEP |
| memory leaks | kmemleak |
| hangs | soft/hard lockup and hung task detectors, echo t > /proc/sysrq-trigger, echo l > ... (backtraces of all CPUs) |
| understanding flow | ftrace (function_graph), tracepoints, trace-cmd/KernelShark, pr_debug with dynamic debug, bpftrace |
| crash dumps | kdump + crash utility, pstore/ramoops (logs survive reboot) |
| live debugging | KGDB over serial, JTAG/OpenOCD with vmlinux symbols, QEMU + gdb (-s -S) |
Method: make it reproducible, simplify the test case, check recent changes (git bisect), and check the obvious: reference counting, missing locking, use after remove(), IRQ firing before initialisation completes.