TY - GEN
T1 - Controlling a Markov Decision Process with an Abrupt Change in the Transition Kernel
AU - Dahlin, Nathan
AU - Bose, Subhonmesh
AU - Veeravalli, Venugopal V.
N1 - Funding Information:
All authors are with the Department of Electrical and Computer Engineering at the University of Illinois Urbana-Champaign, Urbana, IL 61801. Emails: {dahlin,boses,vvv}@illinois.edu. This work was partly supported by a grant from C3.ai Digital Transformation Institute.
Publisher Copyright:
© 2023 American Automatic Control Council.
PY - 2023
Y1 - 2023
N2 - We consider the control of a Markov decision process (MDP) that undergoes an abrupt change in its transition kernel (mode). We formulate the problem of minimizing regret under control switching based on mode change detection, compared to a mode-observing controller, as an optimal stopping problem. Using a sequence of approximations, we reduce it to a quickest change detection (QCD) problem with Markovian data, for which we characterize a state-dependent threshold-type optimal change detection policy. Numerical experiments illustrate various properties of our control-switching policy.
AB - We consider the control of a Markov decision process (MDP) that undergoes an abrupt change in its transition kernel (mode). We formulate the problem of minimizing regret under control switching based on mode change detection, compared to a mode-observing controller, as an optimal stopping problem. Using a sequence of approximations, we reduce it to a quickest change detection (QCD) problem with Markovian data, for which we characterize a state-dependent threshold-type optimal change detection policy. Numerical experiments illustrate various properties of our control-switching policy.
UR - http://www.scopus.com/inward/record.url?scp=85167820999&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85167820999&partnerID=8YFLogxK
U2 - 10.23919/ACC55779.2023.10156034
DO - 10.23919/ACC55779.2023.10156034
M3 - Conference contribution
AN - SCOPUS:85167820999
T3 - Proceedings of the American Control Conference
SP - 3401
EP - 3408
BT - 2023 American Control Conference, ACC 2023
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 2023 American Control Conference, ACC 2023
Y2 - 31 May 2023 through 2 June 2023
ER -