Low-Cost Ex-Vivo Record-Replay-Diagnosis for IoT Devices

Postmortem program analysis is a daunting task. This is especially true for deeply embedded systems which lack basic facilities to record the context at crash points, not to mention advanced mechanisms such as program tracing that can record the problematic instruction trace ahead of the crash. Worse, without basic hardware features to promptly capture faulty states, the device can execute for days after a memory corruption happens, making fault diagnosis even harder. This paper presents μ RnR, a lightweight ex-vivo record-and-replay architecture for deeply embedded IoT devices, with a primary application on effortless postmortem fault diagnosis. We target a popular IoT architecture in which IoT devices are locally managed by a centralized hub or edge, which further connects to the cloud for remote functions. In these systems, the attack payload carried on external inputs also passes through the IoT hub. Therefore, our system leverages the IoT hub to collect these suspicious external inputs, with which we try to recover the execution trace before the crash. Although edge observable inputs are not sufficient to fully replay previous execution, we found that the rest device inputs including internal hardware signals can be inferred via program analysis techniques. To demonstrate the application of μRnR, we also developed a bug diagnosis system named μARCUS, which uses the recovered execution trace to find the root cause of crashed execution. Our prototype shows promising results in terms of both trace reconstruction and bug diagnosis.