</>EmbeddedInterview

What happens when a program executes an illegal instruction or accesses invalid memory?

MediumLinux System Programming

Short answer: The CPU raises a synchronous exception into the kernel (an undefined-instruction exception, or a data/instruction abort for a bad memory access). The kernel decides whether the fault can be fixed (for example, a page fault on a valid but not-yet-mapped page). If not, it sends the faulting thread a signal: SIGILL for an illegal instruction, SIGSEGV for an invalid address or permission violation, SIGBUS for some alignment or hardware errors, SIGFPE for arithmetic errors. The default action terminates the process and may write a core dump.

Example (ARM, from user space)

int main(void)
{
    __asm__ volatile(".word 0xe7f000f0");   /* permanently undefined encoding (UDF) in A32 */
    return 0;
}
/* Shell prints: Illegal instruction (core dumped) */

On AArch64, .inst 0x00000000 (a UDF #0) has the same effect.

Path through the kernel on ARM64

  1. The CPU takes a synchronous exception to EL1. ESR_EL1 says why: exception class "unknown/undefined instruction", "data abort from a lower EL" (with the fault type), etc.
  2. The vector code saves the user registers and calls the handler (do_el0_undef, do_mem_abort).
  3. For memory aborts, the page fault handler checks the VMA list: if the address is valid, it maps the page and the instruction is retried transparently. Otherwise...
  4. ...the kernel calls force_sig_fault(SIGSEGV, SEGV_MAPERR or SEGV_ACCERR, addr).
  5. On return to user mode the signal is delivered: the default action kills the process, and the kernel log may show unhandled level 3 translation fault with the PC and address.

Kernel vs user faults: if the same thing happens in kernel mode (outside a copy_from_user fixup), the result is an oops, and possibly a panic. That is why kernel bugs are so much more serious.

Debugging the user-space case

  • Enable core dumps (ulimit -c unlimited, check /proc/sys/kernel/core_pattern), then gdb ./prog core and bt.
  • si_addr in the siginfo_t gives the faulting address; si_code tells SEGV_MAPERR (no mapping) from SEGV_ACCERR (permission).
  • AddressSanitizer (-fsanitize=address) catches many memory errors before they become segfaults.

Why SIGILL happens in practice: code compiled for a newer CPU (e.g. ARMv8.2 atomics or SVE) running on an older core, a corrupted function pointer jumping into data, or a mismatched hard-float/soft-float build.