What happens when a program executes an illegal instruction or accesses invalid memory?
Short answer: The CPU raises a synchronous exception into the kernel (an undefined-instruction exception, or a data/instruction abort for a bad memory access). The kernel decides whether the fault can be fixed (for example, a page fault on a valid but not-yet-mapped page). If not, it sends the faulting thread a signal: SIGILL for an illegal instruction, SIGSEGV for an invalid address or permission violation, SIGBUS for some alignment or hardware errors, SIGFPE for arithmetic errors. The default action terminates the process and may write a core dump.
Example (ARM, from user space)
int main(void)
{
__asm__ volatile(".word 0xe7f000f0"); /* permanently undefined encoding (UDF) in A32 */
return 0;
}
/* Shell prints: Illegal instruction (core dumped) */
On AArch64, .inst 0x00000000 (a UDF #0) has the same effect.
Path through the kernel on ARM64
- The CPU takes a synchronous exception to EL1.
ESR_EL1says why: exception class "unknown/undefined instruction", "data abort from a lower EL" (with the fault type), etc. - The vector code saves the user registers and calls the handler (
do_el0_undef,do_mem_abort). - For memory aborts, the page fault handler checks the VMA list: if the address is valid, it maps the page and the instruction is retried transparently. Otherwise...
- ...the kernel calls
force_sig_fault(SIGSEGV, SEGV_MAPERR or SEGV_ACCERR, addr). - On return to user mode the signal is delivered: the default action kills the process, and the kernel log may show
unhandled level 3 translation faultwith the PC and address.
Kernel vs user faults: if the same thing happens in kernel mode (outside a copy_from_user fixup), the result is an oops, and possibly a panic. That is why kernel bugs are so much more serious.
Debugging the user-space case
- Enable core dumps (
ulimit -c unlimited, check/proc/sys/kernel/core_pattern), thengdb ./prog coreandbt. si_addrin thesiginfo_tgives the faulting address;si_codetellsSEGV_MAPERR(no mapping) fromSEGV_ACCERR(permission).- AddressSanitizer (
-fsanitize=address) catches many memory errors before they become segfaults.
Why SIGILL happens in practice: code compiled for a newer CPU (e.g. ARMv8.2 atomics or SVE) running on an older core, a corrupted function pointer jumping into data, or a mismatched hard-float/soft-float build.