Random Markov decision processes for sustainable infrastructure systems

The quantitative assessment of the life-cycle performance of infrastructure systems has seen rapid progress using methods from systems dynamics. Markov chains and Markov decision processes are some of the more useful modelling tools that permit the integration of stochasticity in decisions related to sustainable infrastructure management. In this paper, we explore the effect of uncertainties in the characterisation of these decision models. In particular, we consider the uncertainty in the transition parameters of these models and quantitatively discuss the implication of this kind of uncertainty on the optimal decisions as the solution of these models. We also make the case for the development of rationales that can differentiate between these decision models. In addition, a computationally efficient solution algorithm is introduced for these systems under transition uncertainties. Finally, we demonstrate the sensitivity of the optimal decisions to transition uncertainties and also the computational advantage of the discussed solution algorithm using the problem of optimal maintenance of pavement.