Enabling Transformers to Understand Low-Level Programs

Zifan Carl Guo, William S. Moses

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Unlike prior approaches to machine learning, Transformer models can first be trained on a large corpus of unlabeled data with a generic objective and then on a smaller task-specific dataset. This versatility has led to both larger models and datasets. Consequently, Transformers have led to breakthroughs in the field of natural language processing. Generic program optimization presently operates on low-level programs such as LLVM. Unlike the high-level languages (e.g. C, Python, Java), which have seen initial success in machine-learning analyses, lower-level languages tend to be more verbose and repetitive to precisely specify program behavior, provide more details about microarchitecture, and derive properties necessary for optimization, all of which makes it difficult for machine learning. In this work, we apply transfer learning to low-level (LLVM) programs and study how low-level programs can be made more amenable to Transformer models through various techniques, including preprocessing, infix/prefix operators, and information deduplication. We evaluate the effectiveness of these techniques through a series of ablation studies on the task of translating C to both unoptimized (-O0) and optimized (-01) LLVM IR. On the AnghaBench dataset, our model achieves a 49.57% verbatim match and BLEU score of 87.68 against Clang -O0 and 38.73% verbatim match and BLEU score of 77.03 against Clang -O1.

Original languageEnglish (US)
Title of host publication2022 IEEE High Performance Extreme Computing Conference, HPEC 2022
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781665497862
StatePublished - 2022
Externally publishedYes
Event2022 IEEE High Performance Extreme Computing Conference, HPEC 2022 - Virtual, Online, United States
Duration: Sep 19 2022Sep 23 2022

Publication series

Name2022 IEEE High Performance Extreme Computing Conference, HPEC 2022


Conference2022 IEEE High Performance Extreme Computing Conference, HPEC 2022
Country/TerritoryUnited States
CityVirtual, Online


  • compilers
  • LLVM
  • machine learning
  • machine translation
  • NLP

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Science Applications
  • Hardware and Architecture
  • Software
  • Computational Mathematics
  • Numerical Analysis


Dive into the research topics of 'Enabling Transformers to Understand Low-Level Programs'. Together they form a unique fingerprint.

Cite this