Controlling a Markov Decision Process with an Abrupt Change in the Transition Kernel

Nathan Dahlin; Subhonmesh Bose; Venugopal V. Veeravalli

doi:10.23919/ACC55779.2023.10156034

Controlling a Markov Decision Process with an Abrupt Change in the Transition Kernel

Nathan Dahlin, Subhonmesh Bose, Venugopal V. Veeravalli

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

We consider the control of a Markov decision process (MDP) that undergoes an abrupt change in its transition kernel (mode). We formulate the problem of minimizing regret under control switching based on mode change detection, compared to a mode-observing controller, as an optimal stopping problem. Using a sequence of approximations, we reduce it to a quickest change detection (QCD) problem with Markovian data, for which we characterize a state-dependent threshold-type optimal change detection policy. Numerical experiments illustrate various properties of our control-switching policy.

Original language	English (US)
Title of host publication	2023 American Control Conference, ACC 2023
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	3401-3408
Number of pages	8
ISBN (Electronic)	9798350328066
DOIs	https://doi.org/10.23919/ACC55779.2023.10156034
State	Published - 2023
Event	2023 American Control Conference, ACC 2023 - San Diego, United States Duration: May 31 2023 → Jun 2 2023

Publication series

Name	Proceedings of the American Control Conference
Volume	2023-May
ISSN (Print)	0743-1619

Conference

Conference	2023 American Control Conference, ACC 2023
Country/Territory	United States
City	San Diego
Period	5/31/23 → 6/2/23

ASJC Scopus subject areas

Electrical and Electronic Engineering

Online availability

10.23919/ACC55779.2023.10156034

Library availability

Discover UIUC Full Text

Cite this

Controlling a Markov Decision Process with an Abrupt Change in the Transition Kernel. / Dahlin, Nathan; Bose, Subhonmesh ; Veeravalli, Venugopal V.
2023 American Control Conference, ACC 2023. Institute of Electrical and Electronics Engineers Inc., 2023. p. 3401-3408 (Proceedings of the American Control Conference; Vol. 2023-May).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Dahlin, N, Bose, S & Veeravalli, VV 2023, Controlling a Markov Decision Process with an Abrupt Change in the Transition Kernel. in 2023 American Control Conference, ACC 2023. Proceedings of the American Control Conference, vol. 2023-May, Institute of Electrical and Electronics Engineers Inc., pp. 3401-3408, 2023 American Control Conference, ACC 2023, San Diego, United States, 5/31/23. https://doi.org/10.23919/ACC55779.2023.10156034

@inproceedings{3ce50aa5b01a466395b49cdbc7deee2d,

title = "Controlling a Markov Decision Process with an Abrupt Change in the Transition Kernel",

abstract = "We consider the control of a Markov decision process (MDP) that undergoes an abrupt change in its transition kernel (mode). We formulate the problem of minimizing regret under control switching based on mode change detection, compared to a mode-observing controller, as an optimal stopping problem. Using a sequence of approximations, we reduce it to a quickest change detection (QCD) problem with Markovian data, for which we characterize a state-dependent threshold-type optimal change detection policy. Numerical experiments illustrate various properties of our control-switching policy.",

author = "Nathan Dahlin and Subhonmesh Bose and Veeravalli, {Venugopal V.}",

note = "All authors are with the Department of Electrical and Computer Engineering at the University of Illinois Urbana-Champaign, Urbana, IL 61801. Emails: {dahlin,boses,vvv}@illinois.edu. This work was partly supported by a grant from C3.ai Digital Transformation Institute.; 2023 American Control Conference, ACC 2023 ; Conference date: 31-05-2023 Through 02-06-2023",

year = "2023",

doi = "10.23919/ACC55779.2023.10156034",

language = "English (US)",

series = "Proceedings of the American Control Conference",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "3401--3408",

booktitle = "2023 American Control Conference, ACC 2023",

address = "United States",

}

TY - GEN

T1 - Controlling a Markov Decision Process with an Abrupt Change in the Transition Kernel

AU - Dahlin, Nathan

AU - Bose, Subhonmesh

AU - Veeravalli, Venugopal V.

N1 - All authors are with the Department of Electrical and Computer Engineering at the University of Illinois Urbana-Champaign, Urbana, IL 61801. Emails: {dahlin,boses,vvv}@illinois.edu. This work was partly supported by a grant from C3.ai Digital Transformation Institute.

PY - 2023

Y1 - 2023

N2 - We consider the control of a Markov decision process (MDP) that undergoes an abrupt change in its transition kernel (mode). We formulate the problem of minimizing regret under control switching based on mode change detection, compared to a mode-observing controller, as an optimal stopping problem. Using a sequence of approximations, we reduce it to a quickest change detection (QCD) problem with Markovian data, for which we characterize a state-dependent threshold-type optimal change detection policy. Numerical experiments illustrate various properties of our control-switching policy.

AB - We consider the control of a Markov decision process (MDP) that undergoes an abrupt change in its transition kernel (mode). We formulate the problem of minimizing regret under control switching based on mode change detection, compared to a mode-observing controller, as an optimal stopping problem. Using a sequence of approximations, we reduce it to a quickest change detection (QCD) problem with Markovian data, for which we characterize a state-dependent threshold-type optimal change detection policy. Numerical experiments illustrate various properties of our control-switching policy.

UR - http://www.scopus.com/inward/record.url?scp=85167820999&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85167820999&partnerID=8YFLogxK

U2 - 10.23919/ACC55779.2023.10156034

DO - 10.23919/ACC55779.2023.10156034

M3 - Conference contribution

AN - SCOPUS:85167820999

T3 - Proceedings of the American Control Conference

SP - 3401

EP - 3408

BT - 2023 American Control Conference, ACC 2023

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2023 American Control Conference, ACC 2023

Y2 - 31 May 2023 through 2 June 2023

ER -

Controlling a Markov Decision Process with an Abrupt Change in the Transition Kernel

Abstract

Publication series

Conference

ASJC Scopus subject areas

Online availability

Library availability

Related links

Fingerprint

Cite this