TY - JOUR
T1 - Maximum-Entropy Multi-Agent Dynamic Games
T2 - Forward and Inverse Solutions
AU - Mehr, Negar
AU - Wang, Mingyu
AU - Bhatt, Maulik
AU - Schwager, Mac
N1 - This work was supported in part by ONR Grant N00014-18-1-2830, Toyota Research Institute, and National Science Foundation, under Grant ECCS-2145134 CAREER Award, Grant CNS-2218759, and Grant CCF-2211542. Toyota Research Institute (\"TRI\") provided funds to assist the authors with their research but this article solely reflects the opinions and conclusions of its authors and not TRI or any other Toyota entity. This article was recommended for publication by Associate Editor P. Tokekar and Editor P. Robuffo Giordano upon evaluation of the reviewers' comments.
PY - 2023/6
Y1 - 2023/6
N2 - In this article, we study the problem of multiple stochastic agents interacting in a dynamic game scenario with continuous state and action spaces. We define a new notion of stochastic Nash equilibrium for boundedly rational agents, which we call the entropic cost equilibrium (ECE). We show that ECE is a natural extension to multiple agents of maximum entropy optimality for a single agent. We solve both the "forward" and "inverse" problems for the multi-agent ECE game. For the forward problem, we provide a Riccati algorithm to compute closedform ECE feedback policies for the agents, which are exact in the linear-quadratic-gaussian case. We give an iterative variant to find locally ECE feedback policies for the nonlinear case. For the inverse problem, we present an algorithm to infer the cost functions of the multiple interacting agents given noisy, boundedly rational input and state trajectory examples from agents acting in an ECE. The effectiveness of our algorithms is demonstrated in a simulated multi-agent collision avoidance scenario, and with data from the INTERACTION traffic dataset. In both cases, we show that, by taking into account the agents' game theoretic interactions using our algorithm, a more accurate model of agents' costs can be learned, compared with standard inverse optimal control methods.
AB - In this article, we study the problem of multiple stochastic agents interacting in a dynamic game scenario with continuous state and action spaces. We define a new notion of stochastic Nash equilibrium for boundedly rational agents, which we call the entropic cost equilibrium (ECE). We show that ECE is a natural extension to multiple agents of maximum entropy optimality for a single agent. We solve both the "forward" and "inverse" problems for the multi-agent ECE game. For the forward problem, we provide a Riccati algorithm to compute closedform ECE feedback policies for the agents, which are exact in the linear-quadratic-gaussian case. We give an iterative variant to find locally ECE feedback policies for the nonlinear case. For the inverse problem, we present an algorithm to infer the cost functions of the multiple interacting agents given noisy, boundedly rational input and state trajectory examples from agents acting in an ECE. The effectiveness of our algorithms is demonstrated in a simulated multi-agent collision avoidance scenario, and with data from the INTERACTION traffic dataset. In both cases, we show that, by taking into account the agents' game theoretic interactions using our algorithm, a more accurate model of agents' costs can be learned, compared with standard inverse optimal control methods.
KW - Game-theoretic interactions
KW - inverse reinforcement learning (IRL)
KW - learning from demonstration
KW - multi-agent systems
UR - http://www.scopus.com/inward/record.url?scp=85147267974&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85147267974&partnerID=8YFLogxK
U2 - 10.1109/TRO.2022.3232300
DO - 10.1109/TRO.2022.3232300
M3 - Article
AN - SCOPUS:85147267974
SN - 1552-3098
VL - 39
SP - 1801
EP - 1815
JO - IEEE Transactions on Robotics
JF - IEEE Transactions on Robotics
IS - 3
ER -