AutoHOOT: Automatic high-order optimization for tensors

Linjian Ma, Jiayu Ye, Edgar Solomonik

Research output: Chapter in Book/Report/Conference proceedingConference contribution


High-order optimization methods, including Newton's method andits variants as well as alternating minimization methods, dominatethe optimization algorithms for tensor decompositions and tensornetworks. These tensor methods are used for data analysis andsimulation of quantum systems. In this work, we introduce AutoHOOT, the first automatic differentiation (AD) framework targetingat high-order optimization for tensor computations. AutoHOOTtakes input tensor computation expressions and generates optimized derivative expressions. In particular, AutoHOOT contains anew explicit Jacobian / Hessian expression generation kernel whoseoutputs maintain the input tensors' granularity and are easy to optimize. The expressions are then optimized by both the traditionalcompiler optimization techniques and specific tensor algebra transformations. Experimental results show that AutoHOOT achievescompetitive CPU and GPU performance for both tensor decomposition and tensor network applications compared to existing ADsoftware and other tensor computation libraries with manuallywritten kernels. The tensor methods generated by AutoHOOT arealso well-parallelizable, and we demonstrate good scalability on adistributed memory supercomputer.

Original languageEnglish (US)
Title of host publicationPACT 2020 - Proceedings of the ACM International Conference on Parallel Architectures and Compilation Techniques
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages13
ISBN (Electronic)9781450380751
StatePublished - Sep 30 2020
Externally publishedYes
Event2020 ACM International Conference on Parallel Architectures and Compilation Techniques, PACT 2020 - Virtual, Online, United States
Duration: Oct 3 2020Oct 7 2020

Publication series

NameParallel Architectures and Compilation Techniques - Conference Proceedings, PACT
ISSN (Print)1089-795X


Conference2020 ACM International Conference on Parallel Architectures and Compilation Techniques, PACT 2020
Country/TerritoryUnited States
CityVirtual, Online


  • Automatic differentiation
  • Computational graph optimization
  • Tensor computation
  • Tensor decomposition
  • Tensor network

ASJC Scopus subject areas

  • Software
  • Theoretical Computer Science
  • Hardware and Architecture


Dive into the research topics of 'AutoHOOT: Automatic high-order optimization for tensors'. Together they form a unique fingerprint.

Cite this