Entropy Maximization for Markov Decision Processes under Temporal Logic Constraints

Yagiz Savas, Melkior Ornik, Murat Cubuktepe, Mustafa O. Karabag, Ufuk Topcu

Research output: Contribution to journalArticle

Abstract

We study the problem of synthesizing a policy that maximizes the entropy of a Markov decision process (MDP) subject to a temporal logic constraint. Such a policy minimizes the predictability of the paths it generates, or dually, maximizes the exploration of different paths in an MDP while ensuring the satisfaction of a temporal logic specification. We first show that the maximum entropy of an MDP can be finite, infinite, or unbounded. We provide necessary and sufficient conditions under which the maximum entropy of an MDP is finite, infinite, or unbounded. We then present an algorithm which is based on a convex optimization problem to synthesize a policy that maximizes the entropy of an MDP. We also show that maximizing the entropy of an MDP is equivalent to maximizing the entropy of the paths that reach a certain set of states in the MDP. Finally, we extend the algorithm to an MDP subject to a temporal logic specification. In numerical examples, we demonstrate the proposed method on different motion planning scenarios and illustrate the relation between the restrictions imposed on the paths by a specification, the maximum entropy, and the predictability of paths.

Original languageEnglish (US)
Article number8735817
Pages (from-to)1552-1567
Number of pages16
JournalIEEE Transactions on Automatic Control
Volume65
Issue number4
DOIs
StatePublished - Apr 2020

Keywords

  • Markov processes
  • convexity
  • entropy
  • temporal logic

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Computer Science Applications
  • Electrical and Electronic Engineering

Fingerprint Dive into the research topics of 'Entropy Maximization for Markov Decision Processes under Temporal Logic Constraints'. Together they form a unique fingerprint.

  • Cite this