Bayesian regression tree ensembles that adapt to smoothness and sparsity

Antonio R. Linero, Yun Yang

Research output: Contribution to journalArticlepeer-review

Abstract

Ensembles of decision trees are a useful tool for obtaining flexible estimates of regression functions. Examples of these methods include gradient-boosted decision trees, random forests and Bayesian classification and regression trees. Two potential shortcomings of tree ensembles are their lack of smoothness and their vulnerability to the curse of dimensionality. We show that these issues can be overcome by instead considering sparsity inducing soft decision trees in which the decisions are treated as probabilistic. We implement this in the context of the Bayesian additive regression trees framework and illustrate its promising performance through testing on benchmark data sets. We provide strong theoretical support for our methodology by showing that the posterior distribution concentrates at the minimax rate (up to a logarithmic factor) for sparse functions and functions with additive structures in the high dimensional regime where the dimensionality of the covariate space is allowed to grow nearly exponentially in the sample size. Our method also adapts to the unknown smoothness and sparsity levels, and can be implemented by making minimal modifications to existing Bayesian additive regression tree algorithms.

Original languageEnglish (US)
Pages (from-to)1087-1110
Number of pages24
JournalJournal of the Royal Statistical Society. Series B: Statistical Methodology
Volume80
Issue number5
DOIs
StatePublished - Nov 2018

Keywords

  • Bayesian additive regression trees
  • Bayesian non-parametrics
  • High dimensional regimes
  • Model averaging
  • Posterior consistency

ASJC Scopus subject areas

  • Statistics and Probability
  • Statistics, Probability and Uncertainty

Fingerprint Dive into the research topics of 'Bayesian regression tree ensembles that adapt to smoothness and sparsity'. Together they form a unique fingerprint.

Cite this