π¦Embedded Linux & Boot
Embedded Linux roles care about the whole system: how the board boots, how the image is built, and how it is updated in the field. Interviewers love "walk me through the boot process" because it shows whether you have actually brought up a board. Each answer starts with the short version, then goes into the detail you would add if the interviewer keeps asking.
01Walk through the complete boot process of an embedded Linux system, from power-on to the login prompt.Mediumβ
Short answer: Boot ROM β first-stage loader (SPL/FSBL) in on-chip SRAM, which initialises DRAM β full bootloader (U-Boot) in DRAM β kernel + device tree (+ initramfs) loaded and started β kernel initialises the CPU, memory, interrupts, scheduler and drivers β mounts the root filesystem β starts /sbin/init as PID 1 β init starts services and a getty/login or the product application.
Stage by stage (typical ARM SoC)
| Stage | Runs from | Main job |
|---|---|---|
| 1. Boot ROM | on-chip ROM (mask ROM) | read boot pins/fuses, pick boot device (eMMC, SD, SPI NOR, USB, UART), load first stage into SRAM, optionally verify its signature |
| 2. SPL / FSBL | on-chip SRAM (tensβhundreds of KB) | clocks, PMIC, DRAM controller init and training, load the next stage into DRAM |
| 3. (ARM64) TF-A BL31 | secure DRAM/SRAM | EL3 secure monitor, PSCI (CPU on/off, suspend), sometimes OP-TEE (BL32) |
| 4. U-Boot proper | DRAM | console, storage, network, environment, bootcmd; loads kernel, DTB, initramfs |
| 5. Kernel | DRAM | decompress (zImage) or run directly (Image), head.S β start_kernel() β drivers β mount root |
| 6. init (PID 1) | user space | systemd / BusyBox init / SysVinit starts services, mounts other filesystems, starts getty or the app |
Why so many stages? At power-on DRAM does not work yet: its controller must be configured and trained, which needs code. The boot ROM is tiny and fixed, and on-chip SRAM is small, so a small stage that fits in SRAM brings up DRAM and then loads the large bootloader.
What gets passed along
- Boot ROM β SPL: nothing much; the SPL image header tells the ROM where to load it.
- U-Boot β kernel: the kernel command line (
bootargs), the device tree blob address and, optionally, the initramfs address. On ARM64 the DTB address is in registerx0. - Kernel β init: the command line options it did not consume, and the environment.
What to say if asked "where would you look if it hangs?"
- No output at all: boot ROM did not find a valid image (check boot pins, image header, signature).
- SPL prints then stops: DRAM init or the next-stage load.
- U-Boot works but "Starting kernel ..." is the last line: wrong load address, wrong DTB, wrong
console=, or the kernel crashed before the console driver started. Enableearlyconon the command line. - Kernel panics with "VFS: Unable to mount root fs": wrong
root=, missing storage driver or filesystem driver. - "No working init found":
/sbin/initmissing, wrong architecture binary, or missing shared libraries.
02What happens inside U-Boot from reset until it boots the kernel?Mediumβ
Short answer: U-Boot starts in architecture assembly (start.S), sets up the CPU and a temporary stack, runs board_init_f() (the pre-relocation init list: clocks, serial, DRAM), relocates itself to the top of DRAM, then runs board_init_r() (post-relocation init: MMC, flash, environment, network, console) and finally main_loop(), which either waits for commands or runs bootcmd after the bootdelay countdown.
In more detail
start.S/_start(e.g.arch/arm/cpu/armv8/start.S): exception vectors, disable interrupts, set the CPU mode/exception level, invalidate caches and TLBs, call low-levellowlevel_init, set up a stack in SRAM or cache.board_init_f()incommon/board_f.c: runs theinit_sequence_f[]list: timer, serial (so you see output early), console, CPU and board info, DRAM init (dram_init()), and calculates where to relocate. The global data structuregdlives in a reserved area because there is no.data/.bssyet.- Relocation (
relocate_code): U-Boot copies itself to the top of DRAM, fixes up addresses, clears BSS, and continues there. Now the full RAM is available. board_init_r()incommon/board_r.c: runsinit_sequence_r[]: malloc area, driver model (initr_dm), MMC/NAND/SPI flash, environment (env_relocateloadsbootcmd,bootargsetc. from storage), stdio and console, network, interrupts, and finally callsrun_main_loop().main_loop()incommon/main.c: prints the "Hit any key to stop autoboot" countdown. If interrupted, it gives the command prompt (=>); otherwise it runsbootcmd.
A common mix-up: board_init_f() runs before board_init_r(). The _f part runs from flash/SRAM with very little memory (originally "flash"), and the _r part runs after relocation to RAM.
What bootcmd typically does
load mmc 0:1 ${kernel_addr_r} Image
load mmc 0:1 ${fdt_addr_r} board.dtb
setenv bootargs console=ttyS0,115200 root=/dev/mmcblk0p2 rootwait
booti ${kernel_addr_r} - ${fdt_addr_r}
Modern U-Boot often uses distro boot / bootstd instead: it scans boot devices for an extlinux.conf or boot.scr and follows it.
SPL is a cut-down build of U-Boot (same source, CONFIG_SPL) that fits in SRAM; it runs board_init_f() style init, brings up DRAM, and loads U-Boot proper (or the kernel directly in Falcon mode).
03How does U-Boot pass control to the Linux kernel, and what state must the CPU be in?Hardβ
Short answer: The bootm/bootz/booti command loads and checks the images, fixes up the device tree (memory size, bootargs into /chosen, initrd location), shuts down U-Boot's drivers, disables caches and the MMU, and jumps to the kernel entry point. On ARM64 it passes the DTB physical address in x0 (with x1βx3 zero); on 32-bit ARM it passes r0 = 0, r1 = machine type, r2 = DTB (or ATAGS) address.
Which command
| Command | Image format |
|---|---|
bootm |
legacy uImage (64-byte header) or FIT image (kernel + DTBs + initramfs + signatures in one file) |
bootz |
32-bit ARM zImage (self-decompressing) |
booti |
ARM64 / RISC-V Image (uncompressed, or compressed with U-Boot's help) |
Arguments are kernel_addr initrd_addr fdt_addr; - means "none" for the initrd.
What the kernel's boot protocol requires (ARM64, Documentation/arch/arm64/booting.rst)
- MMU off, data cache off (or cleaned), instruction cache may be on.
- All interrupts masked (DAIF set).
- CPU in EL2 (preferred, so the kernel can run KVM) or EL1.
x0= physical address of the DTB,x1βx3= 0.- The DTB must be 8-byte aligned and not overlap the kernel image.
- Secondary CPUs are held in firmware (PSCI) until the kernel asks for them.
32-bit ARM: jump with r0 = 0, r1 = machine ID (only used by old non-DT boards), r2 = DTB address. Conceptually:
void (*kernel_entry)(int zero, int arch, unsigned long params);
kernel_entry = (void *)kernel_addr;
kernel_entry(0, machid, (unsigned long)fdt_addr); /* never returns */
Device tree fix-ups U-Boot does before jumping: writes the real DRAM size into /memory, bootargs into /chosen/bootargs, initrd start/end into /chosen, MAC addresses, and reserved-memory regions.
Why caches and MMU are off: the kernel builds its own page tables in head.S. Stale cache lines from U-Boot or a mapping the kernel does not know about could corrupt memory.
Debugging tip: if nothing appears after "Starting kernel ...", add earlycon (with the right UART type and address) to bootargs; it prints from the earliest point in head.S/setup_arch before the real console driver probes.
04What does the Linux kernel do from head.S to starting the init process?Hardβ
Short answer: head.S sets up the early page tables, enables the MMU and jumps to start_kernel() in init/main.c. start_kernel() initialises the architecture (setup_arch), memory management, the scheduler, interrupts, timers and the console, then calls rest_init(), which creates kernel_init (it becomes PID 1) and kthreadd (PID 2) and turns the boot thread into the idle task (PID 0). kernel_init runs all driver initcalls, mounts the root filesystem and execs /sbin/init.
1. Early assembly (arch/arm64/kernel/head.S)
- Record the boot arguments (DTB address in
x0). - Check the exception level; set up EL2 for KVM if entered there.
- Create identity-mapped and kernel page tables, enable the MMU.
- Set up the initial stack and jump to
start_kernel()(via__primary_switched).
On 32-bit ARM with a zImage, a decompressor runs first and then jumps to the real kernel head.S.
2. start_kernel() (key calls, in roughly this order)
| Call | What it does |
|---|---|
setup_arch(&command_line) |
parse the device tree, find memory, early page tables, CPU features |
setup_command_line, parse_args |
process bootargs |
mm_core_init() (older: mm_init()) |
buddy allocator, slab, vmalloc |
sched_init() |
runqueues, the idle task |
early_irq_init(), init_IRQ() |
interrupt descriptors, the GIC driver |
rcu_init() |
read-copy-update |
time_init(), timers, timekeeping_init() |
clocksource and clockevents (arch timer) |
console_init() |
early console so printk output appears |
calibrate_delay() |
BogoMIPS for delay loops |
rest_init() |
hand over to the threads below |
3. rest_init()
user_mode_thread(kernel_init, ...)creates PID 1 (it starts as a kernel thread and later becomes the user-space init).kernel_thread(kthreadd, ...)creates PID 2, the parent of all other kernel threads.- The boot CPU then calls
cpu_startup_entry()and becomes the idle task (PID 0), which runs when nothing else is runnable.
4. kernel_init()
kernel_init_freeable(): brings up the other CPUs (SMP), runsdo_basic_setup()βdo_initcalls(): every driver and subsystem*_initcall()in level order, which is when platform drivers probe against device tree nodes.- Unpacks the initramfs (if any); if it has
/init, runs it. Otherwiseprepare_namespace()mounts the root device fromroot=. - Frees the
__initmemory ("Freeing unused kernel memory"). - Tries
init=from the command line, then/sbin/init,/etc/init,/bin/init,/bin/sh. If none work: "Kernel panic - not syncing: No working init found."
Follow-up: "Is PID 1 a kernel thread?" It begins as one and becomes a normal user process when kernel_execve() loads the init binary, so it keeps PID 1.
05What does the init process (PID 1) do, and what special rules apply to it?Mediumβ
Short answer: PID 1 is the first user-space process. It brings the system to a working state (mounts filesystems, starts daemons, starts login terminals or the main application) and then supervises it. It must never exit (the kernel panics if it does), it becomes the parent of every orphaned process so it must reap zombies, and the kernel does not deliver signals to it that it has not installed a handler for.
Typical jobs of init
- Mount
/proc,/sys,/dev(devtmpfs),/tmp,/run, and remount the root read-write if needed. - Set the hostname, load kernel modules, start
udev/mdevto create device nodes and load firmware. - Start services in the right order (logging, networking, the product application).
- Start
gettyon the console/serial lines so users can log in. - Watch services and restart them, handle shutdown/reboot (
SIGTERMto everything, unmount, sync, thenreboot()syscall).
Special requirements
- Must not die. If PID 1 exits or crashes, the kernel reports "Attempted to kill init!" and panics. Embedded products often rely on this plus
panic=5to reboot automatically. - Reaps orphans. When a parent dies, its children are re-parented to init (or to a "subreaper"). Init must call
wait()on them, otherwise zombies accumulate. - Signal protection. The kernel ignores signals sent to PID 1 unless init has a handler for them, so even
kill -9 1from root does nothing. This also matters in containers: a shell script as PID 1 will not stop onSIGTERMunless it traps it, which is why tools liketiniexist.
Common inits
| Init | Configuration | Typical use |
|---|---|---|
| BusyBox init | /etc/inittab + shell scripts |
tiny images, Buildroot |
| SysVinit | runlevels, /etc/init.d/ scripts in order |
older systems |
| systemd | unit files, dependency graph, parallel start, journald, watchdog | larger devices, Yocto default for many BSPs |
| Your application | init=/usr/bin/app |
single-purpose appliances |
Running your application directly as init is possible but you then take on all of these duties (mounting, reaping zombies, handling shutdown), which is why most products still use a small init.
06How does a Raspberry Pi boot, and how is it different from a typical SoC?Mediumβ
Short answer: On the Raspberry Pi the GPU (VideoCore) boots first, not the ARM cores. The GPU's ROM loads the boot firmware (on Pi 1β3 bootcode.bin and start.elf from the SD card's FAT partition; on Pi 4/5 a bootloader stored in an on-board SPI EEPROM), reads config.txt, loads the device tree, the kernel (kernel8.img for 64-bit) and cmdline.txt, and only then releases the ARM cores to run the kernel.
Raspberry Pi 3 sequence
- Power on: the ARM cores are held in reset; the VideoCore GPU runs its ROM.
- GPU ROM reads
bootcode.binfrom the first FAT partition of the SD card (USB/network boot is possible on later revisions) and runs it from the GPU's L2 cache. bootcode.binenables SDRAM and loadsstart.elf(the main GPU firmware).start.elfreadsconfig.txt(memory split between GPU and ARM, overlays,arm_64bit, clock settings), loads the DTB for the board plus any overlays (dtoverlay=), loads the kernel image andcmdline.txt, and puts them in RAM.- It releases the ARM cores, which start executing the kernel (optionally via U-Boot if
kernel=u-boot.bin). From here it is the normal Linux boot:start_kernel()β mount root (usually the second partition, ext4) β init.
Pi 4 and Pi 5: the first-stage bootloader lives in an SPI EEPROM on the board, can be updated, and supports USB and network boot. bootcode.bin is no longer used.
Differences from a typical industrial SoC
| Raspberry Pi | Typical i.MX / AM62x / STM32MP |
|---|---|
| GPU boots first, closed firmware | ARM core runs the boot ROM directly |
| Boot files on a FAT partition | boot ROM reads a fixed offset or partition (raw image) |
config.txt replaces most bootloader setup |
SPL + U-Boot (+ TF-A, OP-TEE) do the setup |
| Secure boot only on newer models (signed boot images) | Secure boot with fused keys is standard |
Why it matters in an interview: the Pi is the most common board people have used. Being able to say what is different shows you understand which parts are Pi-specific and which are general Linux.
07What happens when a user logs in on a Linux terminal?Mediumβ
Short answer: init starts getty on each terminal; getty opens the TTY, prints the login prompt and reads the user name, then execs login. login checks the password (via PAM against /etc/shadow), sets up the session, changes to the user's UID/GID with setgid()/setuid(), and execs the user's shell from /etc/passwd. Over SSH, sshd does the same job instead of getty/login.
Step by step on a serial console
- init starts
getty(agetty, or BusyBoxgetty) forttyS0/ttyAMA0, as configured in/etc/inittabor a systemdserial-getty@.service. - getty opens the terminal device, sets the baud rate and line settings (
termios), makes it the controlling terminal of a new session (setsid()), prints/etc/issueandlogin:, reads the name. - getty
execslogin username(the same process, PID unchanged). - login reads the password with echo disabled, verifies it through PAM (which hashes the input and compares with
/etc/shadow), checks account expiry and/etc/nologin. - login records the session (
utmp/wtmp,lastlog), sets the TTY owner to the user, then drops privileges:initgroups(),setgid(),setuid(). This order matters: aftersetuid()to a normal user it can no longer change groups. - It sets
HOME,SHELL,USER,PATH, changes to the home directory andexecs the shell with a leading-inargv[0](e.g.-bash), which tells the shell it is a login shell and should read/etc/profileand~/.profile.
Where the kernel is involved
- The TTY layer and line discipline handle echo, line editing (canonical mode), and turning Ctrl-C into
SIGINTfor the foreground process group. setuid()/setgid()are system calls; the kernel checks that only a privileged process (root, or withCAP_SETUID) may change to an arbitrary UID.- Sessions and process groups (
setsid,tcsetpgrp) let the shell do job control. - When you log out, the shell exits, init sees its child died and starts a fresh
getty.
Embedded note: many products disable login on the serial console in production, or use autologin during development. Leaving a root shell on the debug UART of a shipped device is a classic security finding.
08What is a bootloader and why does an embedded Linux system need one?Easyβ
Short answer: A bootloader is the program that runs before the operating system. It initialises the minimum hardware the kernel needs (clocks, DRAM, storage, console), loads the kernel, device tree and optional initramfs into RAM, prepares the boot arguments and jumps to the kernel. The kernel cannot do this itself because, at power-on, nothing (not even DRAM) is ready to hold it.
What only the bootloader can do
- DRAM bring-up: the memory controller needs board-specific timings and training before DRAM is usable.
- Find and load the kernel from eMMC, SD, NAND, SPI NOR, USB or the network.
- Hand over information: the device tree,
bootargs, the initramfs address. - Choose what to boot: A/B slot selection, recovery mode, factory mode.
- Verify signatures as part of secure boot.
- Recovery and production: flash an image over USB/TFTP, read hardware IDs, run memory tests.
Multi-stage boot is normal. The on-chip boot ROM loads a small first stage (U-Boot SPL, a vendor FSBL, or TF-A BL2) into internal SRAM. That stage initialises DRAM and loads the full bootloader. On ARM64 the chain often includes TF-A (BL31, the EL3 secure monitor) and optionally OP-TEE.
Common bootloaders
| Bootloader | Where |
|---|---|
| U-Boot | most ARM/RISC-V embedded boards |
| barebox | an alternative with a cleaner driver model, used by some industrial vendors |
| GRUB / systemd-boot | x86 and ARM servers with UEFI |
| UEFI (EDK II) | ARM servers, some high-end SoCs |
| Vendor (e.g. Qualcomm XBL/ABL, Android bootloaders) | phones and some SoCs |
Follow-up: "Can you boot Linux without U-Boot?" Yes: Falcon mode (SPL loads the kernel directly), EFI stub (UEFI loads the kernel as an EFI application), or the Raspberry Pi firmware loading the kernel itself. You lose U-Boot's flexibility (interactive recovery, network loading), but boot is faster.
09What are the common U-Boot commands?Easyβ
Short answer: Environment commands (printenv, setenv, saveenv), loading commands (load/fatload/ext4load, tftp, mmc, sf), boot commands (bootm, bootz, booti, run bootcmd), and hardware debug commands (md, mw, i2c, gpio).
Environment
=> printenv bootcmd
=> setenv bootargs console=ttyS0,115200 root=/dev/mmcblk0p2 rootwait
=> saveenv # write the environment to its storage
=> run bootcmd # execute the variable as a script
=> env default -a # restore defaults
Storage and loading
=> mmc list # list MMC/SD controllers
=> mmc dev 0 # select device
=> mmc part # show partitions
=> ls mmc 0:1 # list files on partition 1
=> load mmc 0:1 ${kernel_addr_r} Image
=> sf probe; sf read ${loadaddr} 0x100000 0x400000 # SPI NOR flash
Network (fast development loop)
=> setenv serverip 192.168.1.10; setenv ipaddr 192.168.1.50
=> tftp ${kernel_addr_r} Image
=> tftp ${fdt_addr_r} board.dtb
=> booti ${kernel_addr_r} - ${fdt_addr_r}
Loading over TFTP and using an NFS root filesystem means you can rebuild and test without re-flashing the board.
Booting
bootiβ ARM64Image.bootzβ ARM32zImage.bootmβ uImage or FIT (can verify signatures).bootefiβ boot an EFI application (GRUB or the kernel's EFI stub).
Hardware debugging
md.l 0x40000000 10β dump 16 words of memory (or a peripheral register).mw.l 0x40000000 0xdeadbeefβ write memory.i2c dev 0; i2c probe; i2c md 0x50 0 16β scan the bus and read an EEPROM.gpio status; gpio set 42β check and drive pins.mtestβ simple DRAM test.bdinfo,coninfo,versionβ board information.
Follow-up: "How do you stop autoboot?" Press a key during bootdelay (set bootdelay=-2 to disable the prompt entirely in production, and consider disabling the console or protecting it with a password, since an open U-Boot prompt can boot anything).
10What are kernel command line (bootargs) parameters?Easyβ
Short answer: The kernel command line is a string of parameters passed by the bootloader (or stored in the device tree's /chosen/bootargs, or built into the kernel) that configures the kernel at boot: which console to use, where the root filesystem is, log level, and settings for built-in drivers. The running values are in /proc/cmdline.
console=ttymxc0,115200 earlycon root=/dev/mmcblk0p2 rootfstype=ext4 rootwait ro quiet loglevel=4 panic=5
Common parameters
| Parameter | Meaning |
|---|---|
console=ttyS0,115200 |
kernel console device and baud rate (can be given more than once) |
earlycon |
very early console from the DT stdout-path, before the UART driver probes |
root=/dev/mmcblk0p2 |
root filesystem device (also PARTUUID=..., /dev/nfs, ubi0:rootfs) |
rootfstype=ext4 |
skip filesystem probing |
rootwait |
wait until the root device appears (needed for slow eMMC/SD/USB) |
ro / rw |
mount root read-only or read-write |
init=/bin/sh |
run a different first program (recovery trick) |
quiet, loglevel=N |
reduce console output (printing to a slow UART can add seconds to boot) |
panic=5 |
reboot 5 s after a kernel panic |
mem=512M |
limit usable memory |
module.param=value |
parameter for a built-in driver, e.g. usbcore.autosuspend=-1 |
nfsroot=, ip=dhcp |
network root filesystem for development |
Where it comes from (priority depends on kernel config): U-Boot's bootargs variable, which U-Boot writes into the DTB's /chosen node; the DTB itself; or CONFIG_CMDLINE built into the kernel (can be set to append, extend or force).
How the kernel parses it: parameters registered with __setup() or early_param(), and module_param() for built-in modules. Anything the kernel does not recognise is passed to init as arguments or environment variables.
Security note: if users can change bootargs (open U-Boot console), they can add init=/bin/sh and get a root shell. Locking the environment is part of hardening.
11What is an initramfs and when is it needed?Mediumβ
Short answer: An initramfs is a small root filesystem packed as a cpio archive, which the kernel unpacks into a RAM-based filesystem (tmpfs/ramfs) and runs /init from before the real root filesystem is mounted. It is needed when mounting the real root needs work that the kernel alone cannot do: loading storage drivers built as modules, decrypting, assembling RAID/LVM, picking an A/B slot, or checking integrity.
How it works
- The bootloader loads the initramfs into RAM and passes its address (or it is built into the kernel image with
CONFIG_INITRAMFS_SOURCE). - The kernel unpacks it into
rootfs(a tmpfs) early in boot. - If
/initexists, the kernel runs it as PID 1. /initdoes the setup, mounts the real root at e.g./new_root, then callsswitch_root(which deletes the initramfs contents to free RAM andexecs the real/sbin/init).
Typical reasons to use one
- Root is on a device whose driver is a module (common on generic distro kernels).
- Encrypted root (LUKS/dm-crypt): unlock with a key from a TPM, OP-TEE or the user.
- dm-verity for a verified read-only root.
- A/B slot selection, factory reset, or a recovery shell.
- Root over the network, iSCSI, or an overlay of a read-only image plus a writable partition.
initramfs vs the older initrd
| initrd (old) | initramfs |
|---|---|
| a block-device image (ext2) in a RAM disk | a cpio archive unpacked into tmpfs |
| needs a filesystem driver and a fixed-size RAM disk | no block layer, grows as needed |
uses pivot_root |
uses switch_root |
Embedded uses: some small products run entirely from an initramfs (the whole system in RAM, nothing writable on flash to get corrupted). Recovery/update images are often a kernel plus initramfs.
Cost: extra boot time to load and unpack it, and RAM usage. If you do not need it, booting straight to a root partition is faster.
12What is the difference between Buildroot and Yocto?Mediumβ
Short answer: Both build a complete embedded Linux image (toolchain, bootloader, kernel, root filesystem) from source. Buildroot is simple Makefiles plus Kconfig and produces one firmware image; it is quick to learn and build. Yocto (OpenEmbedded with BitBake) builds a full distribution from layered recipes with package management; it is harder to learn but scales to many products and long maintenance, and SoC vendors ship their BSPs as Yocto layers.
| Buildroot | Yocto Project / OpenEmbedded | |
|---|---|---|
| Model | Makefiles + Kconfig (make menuconfig) |
BitBake recipes in layers |
| Output | one root filesystem image | packages (ipk/deb/rpm) + images + SDK |
| Learning curve | gentle (hours) | steep (weeks) |
| Full build time | short (often < 1 hour) | long (hours), but shared-state cache reuses results |
| Incremental changes | often needs a clean rebuild of affected packages | precise task-level rebuilds |
| Package management on target | no | yes, optional package feeds |
| Customisation | defconfig, rootfs overlay, post-build scripts, BR2_EXTERNAL |
layers, .bbappend, machine/distro/image configs |
| License compliance | make legal-info |
built-in license manifests, SPDX/SBOM output |
| Typical use | single, fixed-function product; small team | product families, vendor BSPs, long-term maintained products |
How to answer "which would you choose?"
- Small team, one board, a fixed image, fast iteration: Buildroot.
- Several boards or product variants, a vendor BSP only in Yocto (meta-imx, meta-ti, meta-st-stm32mp, meta-qcom), need for SDKs, SBOMs, package updates, or 10-year maintenance: Yocto.
Related tools: Debian-based images (debos, isar, mkosi) when you want a full Debian userland; and Android/AOSP for Android devices.
Interview follow-up: "How do you add your own application?" Buildroot: a package directory with Config.in and *.mk, ideally in a BR2_EXTERNAL tree. Yocto: a recipe in your own layer, added to the image with IMAGE_INSTALL:append = " myapp".
13What are Yocto layers, recipes and bbappend files?Mediumβ
Short answer: A recipe (.bb) tells BitBake how to build one piece of software. A layer (meta-*) groups recipes and configuration so that the vendor BSP, the distribution policy and your product stay separate. A .bbappend modifies a recipe from another layer without editing it.
Recipe: describes the source, license and build steps through BitBake tasks.
SUMMARY = "Sensor daemon"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://LICENSE;md5=..."
SRC_URI = "git://github.com/acme/sensord.git;branch=main;protocol=https"
SRCREV = "1a2b3c..."
S = "${WORKDIR}/git"
inherit cmake systemd
SYSTEMD_SERVICE:${PN} = "sensord.service"
Tasks run in order: do_fetch β do_unpack β do_patch β do_configure β do_compile β do_install β do_package, and the image tasks assemble the root filesystem. Classes such as cmake, autotools, meson, systemd supply the standard steps.
Layer: a directory with conf/layer.conf, listed in build/conf/bblayers.conf. Each layer has a priority that decides which version of a recipe or file wins. A typical stack:
poky/meta core recipes (openembedded-core)
meta-openembedded/... extra recipes
meta-vendor-bsp SoC/board support (kernel, U-Boot, firmware)
meta-mycompany your distro, images, applications, bbappends
.bbappend: for example linux-imx_%.bbappend in your layer adds a patch and a kernel config fragment to the vendor's kernel recipe:
FILESEXTRAPATHS:prepend := "${THISDIR}/files:"
SRC_URI += "file://0001-add-my-panel.patch file://panel.cfg"
This keeps your changes out of the vendor layer, so you can upgrade it later.
Configuration levels
- MACHINE (
conf/machine/myboard.conf): CPU tune, kernel and bootloader providers, DTB names, serial console. - DISTRO (
conf/distro/mydistro.conf): policy: init system, C library,DISTRO_FEATURES(wifi, bluetooth, systemd). - Image recipe (
my-image.bb): what goes into the root filesystem (IMAGE_INSTALL). local.conf: per-build settings (machine choice, download and sstate directories). Do not put product settings only here; they are not version controlled with the layers.
Useful commands: bitbake -c devshell virtual/kernel, bitbake -e recipe | grep ^SRC_URI, devtool modify, bitbake-layers show-appends.
14What is cross-compilation and what is a toolchain?Easyβ
Short answer: Cross-compiling means building on one machine (the host, e.g. an x86-64 PC) a program that runs on a different architecture (the target, e.g. an ARM64 board). A toolchain is the set of tools and libraries that make this possible: cross compiler, assembler, linker, binutils, the C library and kernel headers for the target.
Why cross-compile: the target is often too slow, has too little RAM/storage, or has no compiler at all. Building a kernel on a quad-core Cortex-A53 takes hours; on a PC it takes minutes.
Toolchain triplets (the prefix tells you the target)
| Prefix | Target |
|---|---|
aarch64-linux-gnu- |
64-bit Arm, Linux, glibc |
arm-linux-gnueabihf- |
32-bit Arm, Linux, hard-float ABI |
arm-none-eabi- |
bare-metal Arm (Cortex-M), no OS, newlib |
riscv64-linux-gnu- |
64-bit RISC-V, Linux |
x86_64-linux-gnu- |
native PC |
aarch64-linux-gnu-gcc -O2 -o hello hello.c
file hello # ELF 64-bit LSB executable, ARM aarch64
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- defconfig Image dtbs
What is in a toolchain
gcc/clangβ compiler for the target.as,ld,objcopy,objdump,readelf,nm,strip,addr2lineβ binutils.- C library (glibc/musl/newlib) and kernel headers that match the target kernel's userspace API.
gdb(cross) for remote debugging withgdbserver.
Sysroot: a directory with the target's headers and libraries (--sysroot=...). Linking against the sysroot ensures the binary uses the same library versions as the target, not the host's.
Getting a toolchain: distro packages, Arm GNU Toolchain releases, Bootlin toolchains, or the SDK generated by Yocto (bitbake -c populate_sdk) or Buildroot (make sdk). Using the build system's SDK guarantees it matches the image.
Classic mistakes: linking against host libraries (missing sysroot), mismatched glibc version ("GLIBC_2.34 not found" on the target), wrong float ABI on 32-bit Arm (hard-float vs soft-float), and forgetting ARCH= when building the kernel.
15What is the difference between glibc, musl and uClibc-ng?Mediumβ
Short answer: All three are C standard libraries for Linux. glibc is the most complete and compatible but the largest. musl is small, clean and easy to link statically, with strict standards behaviour. uClibc-ng targets very small systems, including MMU-less (noMMU) Linux. The choice affects image size, compatibility with existing software, and licensing.
| glibc | musl | uClibc-ng | |
|---|---|---|---|
| Size | large (~2 MB+ shared) | small (~1 MB, tiny static binaries) | small, very configurable |
| Compatibility | reference; almost all software assumes it | high, but some glibc extensions missing or different | good for common software |
| Static linking | possible but awkward (NSS, dlopen) |
designed for it | supported |
| noMMU support | no | limited | yes |
| Locale/i18n | full | basic (UTF-8 only) | configurable |
| License | LGPL-2.1 | MIT | LGPL-2.1 |
| Used by | most distros, Yocto default | Alpine Linux, many Buildroot/OpenWrt images | old OpenWrt, very small devices |
Practical points
- Binary compatibility: a program built against glibc will not run on a musl system (different dynamic loader and symbol versions), and vice versa. Pick one per image.
- Behaviour differences seen in practice: DNS resolver details, smaller default thread stack size in musl (128 KB),
dlclose()not unloading, some GNU-only functions missing. - Licensing: LGPL requires that users can relink your application with a modified library. With dynamic linking that is easy; with static linking against glibc you must provide object files. musl's MIT license avoids this, which is one reason for its use in products.
- Bare-metal is different: Cortex-M firmware uses newlib or newlib-nano (or picolibc), not these Linux libraries.
How to answer "which would you choose": glibc for compatibility with third-party and vendor software (and when a vendor SDK requires it); musl for small, statically linked, security-focused or container images; uClibc-ng only for very constrained or noMMU targets.
16Which filesystems are used on embedded flash?Mediumβ
Short answer: It depends on the flash type. On raw NAND/NOR (an MTD device without its own controller) use UBIFS on top of UBI (or JFFS2 on small NOR), because the filesystem must handle wear levelling and bad blocks. On managed flash (eMMC, SD, UFS, SSD) the controller does that, so use ext4 or F2FS. For a read-only root use SquashFS or EROFS.
Raw flash vs managed flash
| Raw NAND/NOR (MTD) | Managed (eMMC, SD, UFS) | |
|---|---|---|
| Interface | /dev/mtdN, erase blocks, pages, OOB area |
/dev/mmcblkN, normal block device |
| Bad blocks, ECC, wear levelling | software (UBI, MTD, NAND driver) | controller firmware (FTL) |
| Filesystems | UBIFS (on UBI), JFFS2, YAFFS2 | ext4, F2FS, (btrfs) |
Filesystem guide
- UBIFS: journaling, compression, scales to large NAND, fast mount. UBI below it spreads erases across the whole flash and manages bad blocks and volumes.
- JFFS2: older log-structured filesystem for small NOR; mount time grows with size.
- ext4: the default on eMMC/SD; mature, journaling. Use
data=ordered(the default) and avoid unnecessary writes. - F2FS: designed for flash-based block devices; can reduce write amplification on cheap SD/eMMC.
- SquashFS / EROFS: compressed and read-only; the root image cannot be corrupted by power loss. EROFS is newer, faster to read, and used by Android.
- tmpfs: RAM-based, for
/tmp,/runand logs that do not need to survive reboot. - OverlayFS: a writable layer (on a data partition or tmpfs) over a read-only root.
A robust layout used by many products
boot partition (FAT or raw) bootloader, kernel, DTB
rootfs A / rootfs B (SquashFS/EROFS or ext4 mounted ro) A/B update slots
data (ext4/F2FS or UBIFS) config, logs, user data
Protecting against power loss and wear: keep the root read-only, write data with fsync() and the write-temp-then-rename() pattern, limit logging to flash (journald with Storage=volatile, or rate limits), avoid atime updates (noatime), and monitor eMMC health (mmc extcsd read gives the life-time estimate).
17How do A/B (dual-copy) OTA updates work?Mediumβ
Short answer: The device has two copies (slots) of the system. While running from slot A, the update client writes the new image into slot B, verifies it, and tells the bootloader to try B on the next boot. If B boots and the application confirms it is healthy, B becomes the active slot. If B fails to boot after a few attempts (detected by the bootloader's boot counter, often helped by a watchdog), the bootloader rolls back to A.
Step by step
- Download the update (full image or delta) over HTTPS; verify its signature.
- Write it to the inactive slot (B) while the device keeps working on A.
- Verify what was written (hash).
- Set bootloader variables:
boot_slot=B,boot_count=0,upgrade_available=1. - Reboot. The bootloader increments
boot_countand boots B. - User space (after the application is up and passes self-tests) marks the slot good:
upgrade_available=0. - If the system reboots before that and
boot_countexceeds the limit (e.g. 3), the bootloader switches back to A (rollback).
What makes it robust
- Atomic: the switch is a single variable change; a power cut during writing only affects the inactive slot.
- Automatic rollback: a bad kernel, a missing library or a crashing application does not brick the device.
- Watchdog: if the new system hangs instead of crashing, the hardware watchdog resets it so the boot counter can work.
Costs and variations
- Double storage for the system partitions. Alternatives: a single system plus a small recovery partition (cheaper, but the device is offline during the update).
- Delta updates (binary diffs, casync, zchunk) reduce download size, which matters for cellular devices.
- Data, configuration and the bootloader itself are usually not A/B; bootloader updates need extra care.
Tools to name: RAUC, SWUpdate, Mender, OSTree/Aktualizr (file-level, not block-level), and Android's update_engine (A/B "seamless updates").
Security: sign updates, check versions to prevent rollback attacks (installing an old vulnerable image; anti-rollback counters in fuses or RPMB), and combine with secure boot so a modified slot cannot boot.
18What is secure boot and a chain of trust?Hardβ
Short answer: Secure boot means each boot stage cryptographically verifies the next stage before running it, starting from code that cannot be changed (the boot ROM) and a key that cannot be changed (a hash of the public key burned into one-time-programmable fuses). If any check fails, the boot stops. The chain of trust extends from the ROM through the bootloaders and kernel to the root filesystem.
A typical ARM chain
- Boot ROM (immutable) reads the public key hash from OTP fuses and verifies the signature on the first-stage loader (SPL / BL2).
- The first stage verifies TF-A BL31, OP-TEE and U-Boot.
- U-Boot verifies a signed FIT image containing the kernel, DTB and initramfs (
CONFIG_FIT_SIGNATURE, key embedded in U-Boot's own DTB). - The kernel verifies the root filesystem with dm-verity (a hash tree checked on every block read; the root hash is signed or embedded in the signed initramfs/command line), and signed kernel modules.
- Optionally IMA/EVM or signed application packages extend trust into user space.
Signing vs encryption: secure boot is about authenticity (signatures). Encrypting images (for IP protection) is a separate feature that uses device-unique keys.
Important practical points
- Fuses are irreversible. Burning the key hash and then the "secure boot closed" fuse makes the device reject unsigned images forever. Test on development parts, and follow a manufacturing procedure.
- Key protection: the private signing keys live in an HSM or a secure signing server, never on a developer laptop or in the repository. Plan key revocation (most SoCs allow several key slots).
- Close the side doors: disable or lock JTAG, the U-Boot console and environment (or make the environment read-only), USB recovery modes, and
init=/bootargschanges. Otherwise an attacker bypasses the checks rather than breaking them. - Anti-rollback: version counters in fuses or eMMC RPMB stop signed but old, vulnerable images from being installed.
Related terms: "verified boot" (Android, Chrome OS), "measured boot" (each stage is hashed into a TPM so a remote server can check what booted, without blocking boot), and UEFI Secure Boot on PCs and servers.
19What is the difference between systemd, SysVinit and BusyBox init?Easyβ
Short answer: All three are implementations of PID 1, the first user-space process that starts and supervises the rest of the system. BusyBox init is tiny and reads a simple /etc/inittab. SysVinit runs ordered shell scripts per runlevel. systemd starts services in parallel from a dependency graph and also supervises, logs and manages them, at the cost of size and complexity.
| BusyBox init | SysVinit | systemd | |
|---|---|---|---|
| Config | /etc/inittab + rcS script |
/etc/inittab, /etc/init.d/*, runlevel links |
unit files (.service, .socket, .timer, .mount) |
| Start order | sequential script | sequential, numbered scripts | parallel, based on dependencies (After=, Requires=) |
| Service supervision | respawn entries only |
none built in | restart policies, cgroups track every process |
| Logging | syslogd (BusyBox) | syslog | journald |
| Extra features | β | β | watchdog support, socket activation, timers, device and mount management, resource limits |
| Size | tens of KB | small | several MB plus dependencies |
| Typical use | small Buildroot images | legacy systems | Yocto/Debian-based products, larger devices |
A systemd service for a product application
[Unit]
Description=Sensor daemon
After=network-online.target
Wants=network-online.target
[Service]
ExecStart=/usr/bin/sensord
Restart=on-failure
WatchdogSec=10s
[Install]
WantedBy=multi-user.target
WatchdogSec means the app must call sd_notify("WATCHDOG=1") regularly or systemd restarts it; systemd itself can pet the hardware watchdog (RuntimeWatchdogSec= in system.conf).
How to choose: BusyBox init when flash is tiny and the system only runs a few processes; systemd when you need robust supervision, dependency handling, timers and watchdog integration, and you can afford a few MB. Mention boot time: systemd's parallel start helps, and systemd-analyze blame shows what is slow.
20How do you reduce boot time on an embedded Linux device?Mediumβ
Short answer: Measure first, then remove or defer work at every stage: shorten the bootloader (no countdown, minimal init, or Falcon mode), make the kernel smaller and quieter (only needed drivers built in, quiet, fast decompression, deferred probing), and start the critical application early in user space (fewer services, read-only compressed root, no unnecessary waits).
Measure
- Timestamp the serial output:
grabserial -tor the terminal's timestamp option. - Kernel:
printk.time=1,initcall_debug(shows how long each driver init takes),bootgraph.pl. - User space:
systemd-analyze,systemd-analyze blame,systemd-analyze critical-chain, bootchart.
Bootloader
bootdelay=0(or -2) to remove the autoboot countdown.- Remove unused drivers and commands; skip USB/network init unless needed.
- Load only what is needed, from the fastest storage; use DMA and the highest eMMC bus speed.
- Falcon mode: SPL loads the kernel directly, skipping U-Boot proper.
Kernel
- Build in only the drivers needed for the boot path; make the rest modules loaded later, or remove them.
quiet(printing to a 115200-baud UART can cost seconds).- Use LZ4 or an uncompressed image if storage is fast (decompression is CPU time).
CONFIG_PRINTK_TIMEfor measurement, then disable debugging options.- Async probe and deferred initcalls for slow, non-critical devices.
- Use
rootwaitonly where needed and avoid unnecessary delays such asrootdelay=.
User space
- Start the critical application first (a dedicated early systemd unit, or directly from init).
- Remove unused services; avoid network-wait targets unless needed.
- Use a read-only SquashFS/EROFS root (no filesystem check), prelink or statically link the main application, reduce shared library count.
- Show a splash screen or "early camera" output from the bootloader or kernel to improve perceived boot time (automotive rear-view cameras must show within about 2 seconds).
Results: with this approach, 1β3 seconds to the application is achievable on modern Cortex-A SoCs. Hibernation-based "snapshot boot" can be faster still for complex systems.
21How do you access GPIO from Linux user space?Mediumβ
Short answer: Use the GPIO character device (/dev/gpiochipN) through libgpiod and its command-line tools (gpiodetect, gpioinfo, gpioget, gpioset, gpiomon). The old sysfs interface (/sys/class/gpio) is deprecated. For production, describe fixed-function pins in the device tree and let a kernel driver own them.
Command-line examples (libgpiod v2 syntax)
gpiodetect # list GPIO chips
gpioinfo gpiochip0 # line names, direction, consumers
gpioget -c gpiochip0 17 # read line 17
gpioset -c gpiochip0 18=1 # drive line 18 high (holds it while running)
gpiomon -c gpiochip0 -e rising 23 # wait for rising edges
C with libgpiod v2 (sketch)
#include <gpiod.h>
struct gpiod_chip *chip = gpiod_chip_open("/dev/gpiochip0");
struct gpiod_line_settings *s = gpiod_line_settings_new();
gpiod_line_settings_set_direction(s, GPIOD_LINE_DIRECTION_OUTPUT);
struct gpiod_line_config *lc = gpiod_line_config_new();
unsigned int offset = 18;
gpiod_line_config_add_line_settings(lc, &offset, 1, s);
struct gpiod_line_request *req = gpiod_chip_request_lines(chip, NULL, lc);
gpiod_line_request_set_value(req, 18, GPIOD_LINE_VALUE_ACTIVE);
Why sysfs was deprecated
- Global GPIO numbers that change between kernel versions and boards.
- No ownership: any process could change any pin, and an exported pin stayed configured after the program crashed.
- Edge events through
poll()on a value file were awkward and could lose events. - The character device gives named lines, per-process ownership (released automatically when the process exits), timestamps on events, bias (pull-up/down) and debounce settings.
Production advice
- LEDs:
gpio-ledsdriver β/sys/class/leds/...(with triggers like heartbeat). - Buttons:
gpio-keysβ a standard input device (/dev/input/eventN). - Resets and enables for chips: let the chip's kernel driver request them via
reset-gpios/enable-gpiosin the device tree. - Use user-space GPIO for prototyping, test equipment and truly application-specific signals.
Timing limits: user-space GPIO toggling has jitter (scheduling, syscalls). For precise waveforms use PWM, SPI or a timer peripheral, a kernel driver, or a coprocessor (e.g. the Cortex-M core on an i.MX8M or STM32MP1).
22What is the difference between a kernel driver and a user-space driver (UIO, spidev)?Mediumβ
Short answer: A kernel driver runs in kernel space, handles interrupts directly and plugs into a subsystem (network, input, IIO, DRM), so any application can use the device through a standard interface. A user-space driver talks to the hardware through a generic kernel helper (spidev, i2c-dev, UIO, VFIO, libgpiod); it is faster to write and debug and can stay proprietary, but it has higher latency, cannot be used by other kernel subsystems, and is harder to make safe.
| Kernel driver | User-space driver | |
|---|---|---|
| Interrupt latency | microseconds, top half runs immediately | scheduling latency of a process (tens of Β΅s to ms) |
| Crash impact | can panic the whole system | only the process dies |
| Debugging | printk, ftrace, kgdb; reboot on mistakes | gdb, valgrind, normal tools |
| Integration | standard interfaces (/dev/input, net, iio, v4l2), used by any app and by other drivers |
custom API for your application |
| License | must be GPL-compatible to use GPL-only kernel symbols | can be proprietary |
| Maintenance | must track kernel API changes (unless upstreamed) | stable user-space ABI |
Generic helpers for user-space drivers
- spidev (
/dev/spidevB.C): full-duplex SPI transfers withioctl(SPI_IOC_MESSAGE). - i2c-dev (
/dev/i2c-N):ioctl(I2C_SLAVE, addr)thenread/write, orI2C_RDWRfor combined transactions. - UIO (
/dev/uioN):mmap()the device registers; a blockingread()returns when an interrupt arrives (the small kernel part only acknowledges/masks it). - VFIO: like UIO but with IOMMU protection so a user-space driver can do DMA safely (used by DPDK and for PCIe devices).
When to use which
- Prototype or bring-up: user space (quick to iterate).
- A sensor read once a second by one application:
i2c-dev/spidevis fine. - Anything other applications or subsystems need (network interface, input device, display, audio), anything with tight interrupt timing, or DMA without an IOMMU: write a kernel driver, and consider upstreaming it.
Follow-up: "spidev in the device tree?" Bind a specific compatible that the spidev driver lists (or use driver_override); a plain compatible = "spidev" is rejected because the device tree should describe hardware, not which driver to use.