Software/Hardware Co-design for Multi-modal Multi-task Learning in Autonomous Systems

Cong Hao, Deming Chen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Optimizing the quality of result (QoR) and the quality of service (QoS) of AI-empowered autonomous systems simultaneously is very challenging. First, there are multiple input sources, e.g., multimodal data from different sensors, requiring diverse data preprocessing, sensor fusion, and feature aggregation. Second, there are multiple tasks that require various AI models to run simultaneously, e.g., perception, localization, and control. Third, the computing and control system is heterogeneous, composed of hardware components with varied features, such as embedded CPUs, GPUs, FPGAs, and dedicated accelerators. Therefore, autonomous systems essentially require multi-modal multitask (MMMT) learning which must be aware of hardware performance and implementation strategies. While MMMT learning has been attracting intensive research interests, its applications in autonomous systems are still underexplored. In this paper, we first discuss the opportunities of applying MMMT techniques in autonomous systems, and then discuss the unique challenges that must be solved. In addition, we discuss the necessity and opportunities of MMMT model and hardware co-design, which is critical for autonomous systems especially with power/resource-limited or heterogeneous platforms. We formulate the MMMT model and heterogeneous hardware implementation co-design as a differentiable optimization problem, with the objective of improving the solution quality and reducing the overall power consumption and critical path latency. We advocate for further explorations of MMMT in autonomous systems and software/hardware co-design solutions.

Original languageEnglish (US)
Title of host publication2021 IEEE 3rd International Conference on Artificial Intelligence Circuits and Systems, AICAS 2021
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781665419130
DOIs
StatePublished - Jun 6 2021
Event3rd IEEE International Conference on Artificial Intelligence Circuits and Systems, AICAS 2021 - Washington, United States
Duration: Jun 6 2021Jun 9 2021

Publication series

Name2021 IEEE 3rd International Conference on Artificial Intelligence Circuits and Systems, AICAS 2021

Conference

Conference3rd IEEE International Conference on Artificial Intelligence Circuits and Systems, AICAS 2021
Country/TerritoryUnited States
CityWashington
Period6/6/216/9/21

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Networks and Communications
  • Computer Vision and Pattern Recognition
  • Hardware and Architecture
  • Electrical and Electronic Engineering

Cite this