What happens during an OS context switch, and why is it expensive?
Short answer: The kernel saves the CPU state of the current task (registers, stack pointer, program counter, FP/SIMD state as needed) into its PCB, chooses the next task, switches the address space if it belongs to a different process (page table base register, ASID), restores the new task's registers, and returns into it. The direct cost is a few microseconds; the indirect cost (cold caches, TLB misses, branch predictors) is often larger.
When a context switch happens
- Voluntary: the task blocks (I/O, sleep, mutex) or yields.
- Involuntary (pre-emption): a timer interrupt ends its time slice, or an interrupt wakes a higher-priority task.
Linux path (simplified)
interrupt or syscall return → need_resched set → schedule()
pick_next_task() scheduler class chooses the next task
context_switch(rq, prev, next)
switch_mm() new page tables (TTBR0_EL1 + ASID on ARM64; CR3 + PCID on x86)
switch_to(prev, next) cpu_switch_to: save callee-saved regs, SP; load next's
... next task continues from where it called schedule() last time
Registers that the C calling convention says the caller saves are already on the stack, so switch_to only saves the callee-saved registers and the stack pointer. FP/SIMD registers are saved lazily on some architectures (only when the next task uses them).
Thread vs process switch: switching between two threads of the same process skips switch_mm, so there is no address-space change and TLB entries remain valid.
Why it is expensive
- Direct: entering the kernel, the scheduler's work, saving and restoring registers (ARM64 has 31 general registers plus 32 128-bit SIMD registers).
- Indirect: the new task's data is not in the L1/L2 caches; TLB entries may not match (ASIDs/PCIDs avoid full flushes); branch predictors are trained for the previous task. Security mitigations (Spectre/Meltdown) added extra flushes on some CPUs.
Measuring: perf stat -e context-switches,cpu-migrations, vmstat 1 (cs column), /proc/<pid>/status (voluntary_ctxt_switches).
Reducing switches: fewer threads than cores for CPU-bound work, batching, avoiding lock contention, CPU affinity, and event-driven I/O (one thread handling many sockets with epoll).