DeepRT: A Soft Real Time Scheduler for Computer Vision Applications on the Edge

Zhe Yang, Klara Nahrstedt, Hongpeng Guo, Qian Zhou

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The ubiquity of smartphone cameras and IoT cameras, together with the recent boom of deep learning and deep neural networks, proliferate various computer vision driven mobile and IoT applications deployed on the edge. This paper focuses on applications which make soft real time requests to perform inference on their data - they desire prompt responses within designated deadlines, but occasional deadline misses are acceptable. Supporting soft real time applications on a multi-tenant edge server is not easy, since the requests sharing the limited GPU computing resources of an edge server interfere with each other. In order to tackle this problem, we comprehensively evaluate how latency and throughput respond to different GPU execution plans. Based on this analysis, we propose a GPU scheduler, DeepRT, which provides latency guarantee to the requests while maintaining high overall system throughput. The key component of DeepRT, DisBatcher, batches data from different requests as much as possible while it is proven to provide latency guarantee for requests admitted by an Admission Control Module. DeepRT also includes an Adaptation Module which tackles overruns. Our evaluation results show that DeepRT outperforms state-of-the-art works in terms of the number of deadline misses and throughput.

Original languageEnglish (US)
Title of host publication6th ACM/IEEE Symposium on Edge Computing, SEC 2021
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages271-284
Number of pages14
ISBN (Electronic)9781450383905
DOIs
StatePublished - 2021
Externally publishedYes
Event6th ACM/IEEE Symposium on Edge Computing, SEC 2021 - San Jose, United States
Duration: Dec 14 2021Dec 17 2021

Publication series

Name6th ACM/IEEE Symposium on Edge Computing, SEC 2021

Conference

Conference6th ACM/IEEE Symposium on Edge Computing, SEC 2021
Country/TerritoryUnited States
CitySan Jose
Period12/14/2112/17/21

Keywords

  • Edge Computing
  • computer vision
  • real time scheduler

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Hardware and Architecture

Fingerprint

Dive into the research topics of 'DeepRT: A Soft Real Time Scheduler for Computer Vision Applications on the Edge'. Together they form a unique fingerprint.

Cite this