On Generating Explanations for Reinforcement Learning Policies: An Empirical Study

Explaining reinforcement learning policies is important for deploying them in real-world scenarios. We introduce a set of linear temporal logic formulae designed to provide such explanations, and an algorithm for searching through those formulae for the one that best explains a given policy. Our key idea is to compare action distributions from the target policy with those from policies optimized for candidate explanations. This comparison provides more insight into the target policy than existing methods and avoids inference of "catch-all" explanations. We demonstrate our method in a simulated game of capture-the-flag, a car-parking environment, and a robot navigation task.