Criticality-based data segmentation and resource allocation in machine inference pipelines

This chapter introduces a criticality-aware data segmentation and resource allocation framework for real-time machine perception pipelines at the edge, for running DNN-based perception models in real time on resource-constraint edge platforms to process the sensing data stream (i.e., sequence of image frames). Mainstream machine inference frameworks commonly adopt a simple First-in-First-out (FIFO) policy to process the perceived images in a holistic manner without differentiating the data criticality, which results in a significant form of algorithmic priority inversion issue. Priority inversion happens when data of lower priority are processed ahead of or together with data of higher priority. The proposed framework first segments the input data into fine-grained subframe regions with different criticality, and processes them in a priority-based manner with differentiated deadlines and computation resource allocation. We design the general architecture in a modularized way and implement multiple alternative algorithms for data segmentation, prioritization, and resource allocation respectively for different edge scenarios. Experimental results on autonomous driving applications show that the framework is able to provide more timely responses to critical regions with only negligible degradation in overall perception quality. We also extend the idea into two generalized edge AI scenarios: collaborative multi-camera surveillance and edge-assisted live video analytics.