Sysnif: A log-based workflow construction method and performance measurement in intelligent IoT system

Zhaolong Jian, Zongming Jin, Xue Shuo Xie, Ye Lu, Guangying Li, Xinwei Chen, Thar Baker

Research output: Contribution to journalArticlepeer-review

Abstract

The massive smart devices in intelligent IoT system can break down due to malicious attacks and system failures. As a non-invasive method, mining workflow from system log can help administrators locate and diagnose anomalies in time and quickly, so as to realize remote monitoring of the system. However, the system logs are usually interleaved, because there are many concurrent and asynchronous operations on a large number of devices in the IoT system. Consequently, it is challenging to construct an adaptive workflow from these logs and realize real-time anomaly detection. To meet this challenge, we propose a two-stage workflow construction approach named Sysnifin this paper, which includes offline construction and online adjustment. First, the window-based dependence computing method is employed to obtain the context of execution paths. Second, a weight-greedy algorithm is designed to denoise the interleaved system logs effectively. Third, in order to fit system mechanism variation, the online micro-iteration adjusting algorithm is designed to update the workflow model. Extensive measurements have been taken to verify the accuracy and efficiency of SysnifṪhe results highlight that Sysnif can outperform the state-of-the-art method named Logsed on the data set of OpenStack virtual machine logs by 22.5% on recall, meanwhile maintaining the same precision roughly. The precision and recall of Sysnif can achieve at least 92.0% and 93.2%, respectively.

Original languageEnglish
Article number110175
JournalMeasurement: Journal of the International Measurement Confederation
Volume186
DOIs
Publication statusPublished - Dec 2021

Bibliographical note

Funding Information:
This work is partially supported by the National Key Research and Development Program of China ( 2018YFB2100300 ), the National Natural Science Foundation, China ( 62002175 ), the Natural Science Foundation of Tianjin, China ( 19JCQNJC00600 and 20JCZDJC00610 ), the Open Project Fund of State Key Laboratory of Computer Architecture, Institute of Computing Technology, Chinese Academy of Sciences ( CARCHB202016 ), Zhejiang Lab, China ( 2021KF0AB04 ), and China University Industry-University-Research Innovation Fund ( 2020HYA01003 ).

Publisher Copyright:
© 2021 Elsevier Ltd

Keywords

  • Dependence computing
  • Interleaved logs
  • Micro-iteration adjusting
  • Workflow

Cite this