Sysnif: A log-based workflow construction method and performance measurement in intelligent IoT system

The massive smart devices in intelligent IoT system can break down due to malicious attacks and system failures. As a non-invasive method, mining workflow from system log can help administrators locate and diagnose anomalies in time and quickly, so as to realize remote monitoring of the system. However, the system logs are usually interleaved, because there are many concurrent and asynchronous operations on a large number of devices in the IoT system. Consequently, it is challenging to construct an adaptive workflow from these logs and realize real-time anomaly detection. To meet this challenge, we propose a two-stage workflow construction approach named Sysnifin this paper, which includes offline construction and online adjustment. First, the window-based dependence computing method is employed to obtain the context of execution paths. Second, a weight-greedy algorithm is designed to denoise the interleaved system logs effectively. Third, in order to fit system mechanism variation, the online micro-iteration adjusting algorithm is designed to update the workflow model. Extensive measurements have been taken to verify the accuracy and efficiency of SysnifṪhe results highlight that Sysnif can outperform the state-of-the-art method named Logsed on the data set of OpenStack virtual machine logs by 22.5% on recall, meanwhile maintaining the same precision roughly. The precision and recall of Sysnif can achieve at least 92.0% and 93.2%, respectively.