Real-time Bayesian anomaly detection in streaming environmental data

David J. Hill, Barbara S. Minsker, Eyal Amir

Research output: Contribution to journalArticlepeer-review

Abstract

With large volumes of data arriving in near real time from environmental sensors, there is a need for automated detection of anomalous data caused by sensor or transmission errors or by infrequent system behaviors. This study develops and evaluates three automated anomaly detection methods using dynamic Bayesian networks (DBNs), which perform fast, incremental evaluation of data as they become available, scale to large quantities of data, and require no a priori information regarding process variables or types of anomalies that may be encountered. This study investigates these methods' abilities to identify anomalies in eight meteorological data streams from Corpus Christi, Texas. The results indicate that DBN-based detectors, using either robust Kalman filtering or Rao-Blackwellized particle filtering, outperform a DBN-based detector using Kalman filtering, with the former having false positive/negative rates of less than 2%. These methods were successful at identifying data anomalies caused by two real events: a sensor failure and a large storm.

Original languageEnglish (US)
Article numberW00D28
JournalWater Resources Research
Volume46
Issue number4
DOIs
StatePublished - Apr 1 2009
Externally publishedYes

ASJC Scopus subject areas

  • Water Science and Technology

Fingerprint

Dive into the research topics of 'Real-time Bayesian anomaly detection in streaming environmental data'. Together they form a unique fingerprint.

Cite this