TY - JOUR
T1 - Learning latent events from network message logs
AU - Satpathi, Siddhartha
AU - Deb, Supratim
AU - Srikant, R.
AU - Yan, He
N1 - Manuscript received November 6, 2018; revised April 30, 2019; accepted July 1, 2019; approved by IEEE/ACM TRANSACTIONS ON NETWORKING Editor T. He. Date of publication July 31, 2019; date of current version August 16, 2019. The work of S. Satpathi and R. Srikant was supported in part by AT&T, and in part by the NSF under Grant NeTS 1718203 and Grant CPS ECCS 1739189. (Corresponding author: Siddhartha Satpathi.) S. Satpathi and R. Srikant are with University of Illinois at Urbana-Champaign (UIUC), Champaign, IL 61820-5783 USA (e-mail: [email protected]; [email protected]).
PY - 2019/8
Y1 - 2019/8
N2 - We consider the problem of separating error messages generated in large distributed data center networks into error events. In such networks, each error event leads to a stream of messages generated by hardware and software components affected by the event. These messages are stored in a giant message log. We consider the unsupervised learning problem of identifying the signatures of events that generated these messages; here, the signature of an error event refers to the mixture of messages generated by the event. One of the main contributions of the paper is a novel mapping of our problem which transforms it into a problem of topic discovery in documents. Events in our problem correspond to topics and messages in our problem correspond to words in the topic discovery problem. However, there is no direct analog of documents. Therefore, we use a non-parametric change-point detection algorithm, which has linear computational complexity in the number of messages, to divide the message log into smaller subsets called episodes, which serve as the equivalents of documents. After this mapping has been done, we use a well-known algorithm for topic discovery, called LDA, to solve our problem. We theoretically analyze the change-point detection algorithm, and show that it is consistent and has low sample complexity. We also demonstrate the scalability of our algorithm on a real data set consisting of 97 million messages collected over a period of 15 days, from a distributed data center network which supports the operations of a large wireless service provider.
AB - We consider the problem of separating error messages generated in large distributed data center networks into error events. In such networks, each error event leads to a stream of messages generated by hardware and software components affected by the event. These messages are stored in a giant message log. We consider the unsupervised learning problem of identifying the signatures of events that generated these messages; here, the signature of an error event refers to the mixture of messages generated by the event. One of the main contributions of the paper is a novel mapping of our problem which transforms it into a problem of topic discovery in documents. Events in our problem correspond to topics and messages in our problem correspond to words in the topic discovery problem. However, there is no direct analog of documents. Therefore, we use a non-parametric change-point detection algorithm, which has linear computational complexity in the number of messages, to divide the message log into smaller subsets called episodes, which serve as the equivalents of documents. After this mapping has been done, we use a well-known algorithm for topic discovery, called LDA, to solve our problem. We theoretically analyze the change-point detection algorithm, and show that it is consistent and has low sample complexity. We also demonstrate the scalability of our algorithm on a real data set consisting of 97 million messages collected over a period of 15 days, from a distributed data center network which supports the operations of a large wireless service provider.
KW - Bayesian inference
KW - Change point detection
KW - Data center networks
KW - Data mining
KW - Event message log
KW - Time series mixture
KW - Unsupervised learning
UR - http://www.scopus.com/inward/record.url?scp=85074969667&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85074969667&partnerID=8YFLogxK
U2 - 10.1109/TNET.2019.2930040
DO - 10.1109/TNET.2019.2930040
M3 - Article
AN - SCOPUS:85074969667
SN - 1063-6692
VL - 27
SP - 1728
EP - 1741
JO - IEEE/ACM Transactions on Networking
JF - IEEE/ACM Transactions on Networking
IS - 4
M1 - 3370621
ER -