MASK: Redesigning the GPU memory hierarchy to support multi-application concurrency

Rachata Ausavarungnirun, Vance Miller, Joshua Landgraf, Saugata Ghose, Jayneel Gandhi, Adwait Jog, Christopher J. Rossbach, Onur Mutlu

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Graphics Processing Units (GPUs) exploit large amounts of thread-level parallelism to provide high instruction throughput and to efficiently hide long-latency stalls. The resulting high throughput, along with continued programmability improvements, have made GPUs an essential computational resource in many domains. Applications from different domains can have vastly different compute and memory demands on the GPU. In a large-scale computing environment, to efficiently accommodate such wide-ranging demands without leaving GPU resources underutilized, multiple applications can share a single GPU, akin to how multiple applications execute concurrently on a CPU. Multi-application concurrency requires several support mechanisms in both hardware and software. One such key mechanism is virtual memory, which manages and protects the address space of each application. However, modern GPUs lack the extensive support for multi-application concurrency available in CPUs, and as a result suffer from high performance overheads when shared by multiple applications, as we demonstrate. We perform a detailed analysis of which multi-application concurrency support limitations hurt GPU performance the most. We find that the poor performance is largely a result of the virtual memory mechanisms employed in modern GPUs. In particular, poor address translation performance is a key obstacle to efficient GPU sharing. State-of-the-art address translation mechanisms, which were designed for single-application execution, experience significant inter-application interference when multiple applications spatially share the GPU. This contention leads to frequent misses in the shared translation lookaside buffer (TLB), where a single miss can induce long-latency stalls for hundreds of threads. As a result, the GPU often cannot schedule enough threads to successfully hide the stalls, which diminishes system throughput and becomes a first-order performance concern. Based on our analysis, we propose MASK, a new GPU framework that provides low-overhead virtual memory support for the concurrent execution of multiple applications. MASK consists of three novel address-translation-aware cache and memory management mechanisms that work together to largely reduce the overhead of address translation: (1) a token-based technique to reduce TLB contention, (2) a bypassing mechanism to improve the effectiveness of cached address translations, and (3) an application-aware memory scheduling scheme to reduce the interference between address translation and data requests. Our evaluations show that MASK restores much of the throughput lost to TLB contention. Relative to a state-of-the-art GPU TLB, MASK improves system throughput by 57.8%, improves IPC throughput by 43.4%, and reduces application-level unfairness by 22.4%. MASK's system throughput is within 23.2% of an ideal GPU system with no address translation overhead.

Original languageEnglish (US)
Title of host publicationProceedings of the 23rd International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS 2018
PublisherAssociation for Computing Machinery
Number of pages16
ISBN (Electronic)9781450349116
StatePublished - Mar 19 2018
Externally publishedYes
Event23rd International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS 2018 - Williamsburg, United States
Duration: Mar 24 2018Mar 28 2018


Conference23rd International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS 2018
Country/TerritoryUnited States


  • Address translation
  • GPGPU applications
  • Graphics processing units
  • Memory interference
  • Memory protection
  • Memory systems
  • Performance
  • Virtual memory management

ASJC Scopus subject areas

  • General Computer Science


Dive into the research topics of 'MASK: Redesigning the GPU memory hierarchy to support multi-application concurrency'. Together they form a unique fingerprint.

Cite this