How do you design logging and diagnostics for devices in the field?
Short answer: Make failures reportable after the fact: store a persistent crash record (fault registers, PC/LR, stack snippet, task name, firmware version and reset reason) in no-init RAM or flash, keep a compact ring-buffer event log with timestamps and severity levels, maintain counters and health metrics (resets by cause, communication errors, watchdog events, minimum free stack/heap, battery), and upload them when connectivity is available or expose them over a service interface. Keep logging cheap (binary records or log IDs instead of formatted strings), rate-limited and safe to call from any context.
Design elements
| Element | Detail |
|---|---|
| Log levels | ERROR / WARN / INFO / DEBUG, configurable at run time per module |
| Format | binary records: {timestamp, module, event ID, args}; decoded on the host using the build's string table (keeps flash small and logging fast) |
| Storage | RAM ring buffer, flushed to flash in batches (wear levelling, power-fail safe), with a bounded size |
| Context safety | ISR-safe enqueue (lock-free SPSC or a short critical section); formatting and I/O in a low-priority task |
| Crash path | minimal code that does not depend on the heap, RTOS or peripherals that may have failed |
| Versioning | firmware version, build ID (hash of the ELF) in every report so addresses can be symbolised |
| Privacy/security | no secrets or personal data in logs; authenticate the upload channel |
Reset-cause handling at boot: read the reset-cause register (POR, BOR, watchdog, software, pin), log it, check for a crash record from the previous run, then clear both.
Tools/services: Memfault, Zephyr's logging and coredump subsystems, AWS IoT Device Defender, or a custom backend; on embedded Linux, journald, pstore/ramoops, and kdump.
Interview point: "Without field diagnostics, an intermittent bug that happens once a month on 1 in 1,000 devices is nearly impossible to fix; with a crash record and the matching ELF file, it is often a 10-minute job."