Learning latent events from network message logs

Siddhartha Satpathi, Supratim Deb, R. Srikant, He Yan

Research output: Contribution to journalArticle

Abstract

We consider the problem of separating error messages generated in large distributed data center networks into error events. In such networks, each error event leads to a stream of messages generated by hardware and software components affected by the event. These messages are stored in a giant message log. We consider the unsupervised learning problem of identifying the signatures of events that generated these messages; here, the signature of an error event refers to the mixture of messages generated by the event. One of the main contributions of the paper is a novel mapping of our problem which transforms it into a problem of topic discovery in documents. Events in our problem correspond to topics and messages in our problem correspond to words in the topic discovery problem. However, there is no direct analog of documents. Therefore, we use a non-parametric change-point detection algorithm, which has linear computational complexity in the number of messages, to divide the message log into smaller subsets called episodes, which serve as the equivalents of documents. After this mapping has been done, we use a well-known algorithm for topic discovery, called LDA, to solve our problem. We theoretically analyze the change-point detection algorithm, and show that it is consistent and has low sample complexity. We also demonstrate the scalability of our algorithm on a real data set consisting of 97 million messages collected over a period of 15 days, from a distributed data center network which supports the operations of a large wireless service provider.

Original languageEnglish (US)
Article number3370621
Pages (from-to)1728-1741
Number of pages14
JournalIEEE/ACM Transactions on Networking
Volume27
Issue number4
DOIs
StatePublished - Aug 2019

Fingerprint

Direct analogs
Unsupervised learning
Scalability
Computational complexity
Hardware

Keywords

  • Bayesian inference
  • Change point detection
  • Data center networks
  • Data mining
  • Event message log
  • Time series mixture
  • Unsupervised learning

ASJC Scopus subject areas

  • Software
  • Computer Science Applications
  • Computer Networks and Communications
  • Electrical and Electronic Engineering

Cite this

Learning latent events from network message logs. / Satpathi, Siddhartha; Deb, Supratim; Srikant, R.; Yan, He.

In: IEEE/ACM Transactions on Networking, Vol. 27, No. 4, 3370621, 08.2019, p. 1728-1741.

Research output: Contribution to journalArticle

Satpathi, Siddhartha ; Deb, Supratim ; Srikant, R. ; Yan, He. / Learning latent events from network message logs. In: IEEE/ACM Transactions on Networking. 2019 ; Vol. 27, No. 4. pp. 1728-1741.
@article{696d608c2db24f7bb6e61f49a164e359,
title = "Learning latent events from network message logs",
abstract = "We consider the problem of separating error messages generated in large distributed data center networks into error events. In such networks, each error event leads to a stream of messages generated by hardware and software components affected by the event. These messages are stored in a giant message log. We consider the unsupervised learning problem of identifying the signatures of events that generated these messages; here, the signature of an error event refers to the mixture of messages generated by the event. One of the main contributions of the paper is a novel mapping of our problem which transforms it into a problem of topic discovery in documents. Events in our problem correspond to topics and messages in our problem correspond to words in the topic discovery problem. However, there is no direct analog of documents. Therefore, we use a non-parametric change-point detection algorithm, which has linear computational complexity in the number of messages, to divide the message log into smaller subsets called episodes, which serve as the equivalents of documents. After this mapping has been done, we use a well-known algorithm for topic discovery, called LDA, to solve our problem. We theoretically analyze the change-point detection algorithm, and show that it is consistent and has low sample complexity. We also demonstrate the scalability of our algorithm on a real data set consisting of 97 million messages collected over a period of 15 days, from a distributed data center network which supports the operations of a large wireless service provider.",
keywords = "Bayesian inference, Change point detection, Data center networks, Data mining, Event message log, Time series mixture, Unsupervised learning",
author = "Siddhartha Satpathi and Supratim Deb and R. Srikant and He Yan",
year = "2019",
month = "8",
doi = "10.1109/TNET.2019.2930040",
language = "English (US)",
volume = "27",
pages = "1728--1741",
journal = "IEEE/ACM Transactions on Networking",
issn = "1063-6692",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "4",

}

TY - JOUR

T1 - Learning latent events from network message logs

AU - Satpathi, Siddhartha

AU - Deb, Supratim

AU - Srikant, R.

AU - Yan, He

PY - 2019/8

Y1 - 2019/8

N2 - We consider the problem of separating error messages generated in large distributed data center networks into error events. In such networks, each error event leads to a stream of messages generated by hardware and software components affected by the event. These messages are stored in a giant message log. We consider the unsupervised learning problem of identifying the signatures of events that generated these messages; here, the signature of an error event refers to the mixture of messages generated by the event. One of the main contributions of the paper is a novel mapping of our problem which transforms it into a problem of topic discovery in documents. Events in our problem correspond to topics and messages in our problem correspond to words in the topic discovery problem. However, there is no direct analog of documents. Therefore, we use a non-parametric change-point detection algorithm, which has linear computational complexity in the number of messages, to divide the message log into smaller subsets called episodes, which serve as the equivalents of documents. After this mapping has been done, we use a well-known algorithm for topic discovery, called LDA, to solve our problem. We theoretically analyze the change-point detection algorithm, and show that it is consistent and has low sample complexity. We also demonstrate the scalability of our algorithm on a real data set consisting of 97 million messages collected over a period of 15 days, from a distributed data center network which supports the operations of a large wireless service provider.

AB - We consider the problem of separating error messages generated in large distributed data center networks into error events. In such networks, each error event leads to a stream of messages generated by hardware and software components affected by the event. These messages are stored in a giant message log. We consider the unsupervised learning problem of identifying the signatures of events that generated these messages; here, the signature of an error event refers to the mixture of messages generated by the event. One of the main contributions of the paper is a novel mapping of our problem which transforms it into a problem of topic discovery in documents. Events in our problem correspond to topics and messages in our problem correspond to words in the topic discovery problem. However, there is no direct analog of documents. Therefore, we use a non-parametric change-point detection algorithm, which has linear computational complexity in the number of messages, to divide the message log into smaller subsets called episodes, which serve as the equivalents of documents. After this mapping has been done, we use a well-known algorithm for topic discovery, called LDA, to solve our problem. We theoretically analyze the change-point detection algorithm, and show that it is consistent and has low sample complexity. We also demonstrate the scalability of our algorithm on a real data set consisting of 97 million messages collected over a period of 15 days, from a distributed data center network which supports the operations of a large wireless service provider.

KW - Bayesian inference

KW - Change point detection

KW - Data center networks

KW - Data mining

KW - Event message log

KW - Time series mixture

KW - Unsupervised learning

UR - http://www.scopus.com/inward/record.url?scp=85074969667&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85074969667&partnerID=8YFLogxK

U2 - 10.1109/TNET.2019.2930040

DO - 10.1109/TNET.2019.2930040

M3 - Article

AN - SCOPUS:85074969667

VL - 27

SP - 1728

EP - 1741

JO - IEEE/ACM Transactions on Networking

JF - IEEE/ACM Transactions on Networking

SN - 1063-6692

IS - 4

M1 - 3370621

ER -