Robust sustainability management and maintenance using markov decision processes

The quantitative sustainability assessment of urban systems relies on available predictive models for these complex systems of systems. Markov chains and Markov Decision Processes are among the most used tools to integrate the stochasticity in decisions related to infrastructure management. In the present paper, we investigate the effects of uncertainties in the characterization of these decision models and make the case for the development of rationales that can differentiate between them. We will explain the extension of the theory of random transition matrices developed by the authors to Markov Decision Processes and illustrate, using a numerical example, that the policies obtained by solving the deterministic transitions is not necessarily robust to the potential variabilities of transition rates. The resulting probabilistic framework for Markov Decision Processes will also enhance the maintenance of urban systems by rigorously quantifying the confidence in the sustainability assessment metrics.