Nested kernel: An operating system architecture for intra-kernel privilege separation

Nathan Dautenhahn, Theodoros Kasampalis, Will Dietz, John Criswell, Vikram Adve

Research output: Contribution to journalArticlepeer-review


Monolithic operating system designs undermine the security of computing systems by allowing single exploits anywhere in the kernel to enjoy full supervisor privilege. The nested kernel operating system architecture addresses this problem by "nesting" a small isolated kernel within a traditional monolithic kernel. The "nested kernel" interposes on all updates to virtual memory translations to assert protections on physical memory, thus significantly reducing the trusted computing base for memory access control enforcement. We incorporated the nested kernel architecture into FreeBSD on x86-64 hardware while allowing the entire operating system, including untrusted components, to operate at the highest hardware privilege level by write-protecting MMU translations and de-privileging the untrusted part of the kernel. Our implementation inherently enforces kernel code integrity while still allowing dynamically loaded kernel modules, thus defending against code injection attacks. We also demonstrate that the nested kernel architecture allows kernel developers to isolate memory in ways not possible in monolithic kernels by introducing write-mediation and write-logging services to protect critical system data structures. Performance of the nested kernel prototype shows modest overheads: < 1% average for Apache and 2.7% for kernel compile. Overall, our results and experience show that the nested kernel design can be retrofitted to existing monolithic kernels, providing important security benefits.

Original languageEnglish (US)
Pages (from-to)191-206
Number of pages16
JournalACM SIGPLAN Notices
Issue number4
StatePublished - Apr 2015


  • Intra-kernel isolation
  • Malicious operating systems
  • Operating system architecture
  • Virtual memory

ASJC Scopus subject areas

  • General Computer Science


Dive into the research topics of 'Nested kernel: An operating system architecture for intra-kernel privilege separation'. Together they form a unique fingerprint.

Cite this