RTAEval: A Framework for Evaluating Runtime Assurance Logic

Kristina Miller, Christopher K. Zeitler, William Shen, Mahesh Viswanathan, Sayan Mitra

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Runtime assurance (RTA) addresses the problem of keeping an autonomous system safe while using an untrusted (or experimental) controller. This can be done via logic that explicitly switches between the untrusted controller and a safety controller, or logic that filters the input provided by the untrusted controller. While several tools implement specific instances of RTAs, there is currently no framework for evaluating different approaches. Given the importance of the RTA problem in building safe autonomous systems, an evalutation tool is needed. In this paper, we present the RTAEval framework as a low code framework that can be used to quickly evaluate different RTA logics for different types of agents in a variety of scenarios. RTAEval is designed to quickly create scenarios, run different RTA logics, and collect data that can be used to evaluate and visualize performance. In this paper, we describe different components of RTAEval and show how it can be used to create and evaluate scenarios involving multiple aircraft models.

Original languageEnglish (US)
Title of host publicationAutomated Technology for Verification and Analysis - 21st International Symposium, ATVA 2023, Proceedings
EditorsÉtienne André, Jun Sun
Number of pages12
ISBN (Print)9783031453311
StatePublished - 2023
Event21st International Symposium on Automated Technology for Verification and Analysis, ATVA 2023 - Singapore, Singapore
Duration: Oct 24 2023Oct 27 2023

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume14216 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference21st International Symposium on Automated Technology for Verification and Analysis, ATVA 2023


  • Autonomous systems
  • Runtime assurance

ASJC Scopus subject areas

  • Theoretical Computer Science
  • General Computer Science


Dive into the research topics of 'RTAEval: A Framework for Evaluating Runtime Assurance Logic'. Together they form a unique fingerprint.

Cite this