Mixed-membership naive bayes models

Hanhuai Shan, Arindam Banerjee

Research output: Contribution to journalArticlepeer-review


In recent years, mixture models have found widespread usage in discovering latent cluster structure from data. A popular special case of finite mixturemodels is the family of naive Bayes (NB) models, where the probability of a feature vector factorizes over the features for any given component of the mixture. Despite their popularity, naive Bayes models do not allow data points to belong to different component clusters with varying degrees, i.e., mixed memberships, which puts a restriction on their modeling ability. In this paper, we propose mixed-membership naïve Bayes (MMNB) models. On one hand, MMNB can be viewed as a generalization of NB by putting a Dirichlet prior on top to allow mixed memberships. On the other hand, MMNB can also be viewed as a generalization of latent Dirichlet allocation (LDA) with the ability to handle heterogeneous feature vectors with different types of features, e.g., real, categorical, etc.. We propose two variational inference algorithms to learnMMNBmodels. The first one is based on ideas originally used in LDA, and the second one uses substantially fewer variational parameters, leading to a significantly faster algorithm. Further, we extend MMNB/LDA to discriminative mixed-membership models for classification by suitably combining MMNB/LDA with multi-class logistic regression. The efficacy of the proposed mixed-membership models is demonstrated by extensive experiments on several datasets, including UCI benchmarks, recommendation systems, and text datasets.

Original languageEnglish (US)
Pages (from-to)1-62
Number of pages62
JournalData Mining and Knowledge Discovery
Issue number1
StatePublished - Jul 2011
Externally publishedYes


  • Generative models
  • Latent Dirichlet allocation
  • Logistic regression
  • Mixed-membership
  • Naive Bayes
  • Variational inference

ASJC Scopus subject areas

  • Information Systems
  • Computer Science Applications
  • Computer Networks and Communications


Dive into the research topics of 'Mixed-membership naive bayes models'. Together they form a unique fingerprint.

Cite this