Multilocus phylogenetic analysis with gene tree clustering

Ruriko Yoshida, Kenji Fukumizu, Chrysafis Vogiatzis

Research output: Contribution to journalArticlepeer-review


Both theoretical and empirical evidence point to the fact that phylogenetic trees of different genes (loci) do not display precisely matched topologies. Nonetheless, most genes do display related phylogenies; this implies they form cohesive subsets (clusters). In this work, we discuss gene tree clustering, focusing on the normalized cut (Ncut) framework as a suitable method for phylogenetics. We proceed to show that this framework is both efficient and statistically accurate when clustering gene trees using the geodesic distance between them over the Billera–Holmes–Vogtmann tree space. We also conduct a computational study on the performance of different clustering methods, with and without preprocessing, under different distance metrics, and using a series of dimensionality reduction techniques. Our results with simulated data reveal that Ncut accurately clusters the set of gene trees, given a species tree under the coalescent process. Other observations from our computational study include the similar performance displayed by Ncut and k-means under most dimensionality reduction schemes, the worse performance of hierarchical clustering, and the significantly better performance of the neighbor-joining method with the p-distance compared to the maximum-likelihood estimation method. Supplementary material, all codes, and the data used in this work are freely available at online.

Original languageEnglish (US)
Pages (from-to)293-313
Number of pages21
JournalAnnals of Operations Research
Issue number1-2
StatePublished - May 1 2019
Externally publishedYes


  • Clustering
  • Normalized cut
  • Phylogenetics

ASJC Scopus subject areas

  • Decision Sciences(all)
  • Management Science and Operations Research


Dive into the research topics of 'Multilocus phylogenetic analysis with gene tree clustering'. Together they form a unique fingerprint.

Cite this