Simulating emergent properties of human driving behavior using multi-agent reward augmented imitation learning

Raunak P. Bhattacharyya, Derek J. Phillips, Changliu Liu, Jayesh K. Gupta, Katherine Driggs-Campbell, Mykel J. Kochenderfer

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Recent developments in multi-agent imitation learning have shown promising results for modeling the behavior of human drivers. However, it is challenging to capture emergent traffic behaviors that are observed in real-world datasets. Such behaviors arise due to the many local interactions between agents that are not commonly accounted for in imitation learning. This paper proposes Reward Augmented Imitation Learning (RAIL), which integrates reward augmentation into the multi-agent imitation learning framework and allows the designer to specify prior knowledge in a principled fashion. We prove that convergence guarantees for the imitation learning process are preserved under the application of reward augmentation. This method is validated in a driving scenario, where an entire traffic scene is controlled by driving policies learned using our proposed algorithm. Further, we demonstrate improved performance in comparison to traditional imitation learning algorithms both in terms of the local actions of a single agent and the behavior of emergent properties in complex, multi-agent settings.

Original languageEnglish (US)
Title of host publication2019 International Conference on Robotics and Automation, ICRA 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages789-795
Number of pages7
ISBN (Electronic)9781538660263
DOIs
StatePublished - May 2019
Event2019 International Conference on Robotics and Automation, ICRA 2019 - Montreal, Canada
Duration: May 20 2019May 24 2019

Publication series

NameProceedings - IEEE International Conference on Robotics and Automation
Volume2019-May
ISSN (Print)1050-4729

Conference

Conference2019 International Conference on Robotics and Automation, ICRA 2019
CountryCanada
CityMontreal
Period5/20/195/24/19

Fingerprint

Learning algorithms

ASJC Scopus subject areas

  • Software
  • Control and Systems Engineering
  • Artificial Intelligence
  • Electrical and Electronic Engineering

Cite this

Bhattacharyya, R. P., Phillips, D. J., Liu, C., Gupta, J. K., Driggs-Campbell, K., & Kochenderfer, M. J. (2019). Simulating emergent properties of human driving behavior using multi-agent reward augmented imitation learning. In 2019 International Conference on Robotics and Automation, ICRA 2019 (pp. 789-795). [8793750] (Proceedings - IEEE International Conference on Robotics and Automation; Vol. 2019-May). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICRA.2019.8793750

Simulating emergent properties of human driving behavior using multi-agent reward augmented imitation learning. / Bhattacharyya, Raunak P.; Phillips, Derek J.; Liu, Changliu; Gupta, Jayesh K.; Driggs-Campbell, Katherine; Kochenderfer, Mykel J.

2019 International Conference on Robotics and Automation, ICRA 2019. Institute of Electrical and Electronics Engineers Inc., 2019. p. 789-795 8793750 (Proceedings - IEEE International Conference on Robotics and Automation; Vol. 2019-May).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Bhattacharyya, RP, Phillips, DJ, Liu, C, Gupta, JK, Driggs-Campbell, K & Kochenderfer, MJ 2019, Simulating emergent properties of human driving behavior using multi-agent reward augmented imitation learning. in 2019 International Conference on Robotics and Automation, ICRA 2019., 8793750, Proceedings - IEEE International Conference on Robotics and Automation, vol. 2019-May, Institute of Electrical and Electronics Engineers Inc., pp. 789-795, 2019 International Conference on Robotics and Automation, ICRA 2019, Montreal, Canada, 5/20/19. https://doi.org/10.1109/ICRA.2019.8793750
Bhattacharyya RP, Phillips DJ, Liu C, Gupta JK, Driggs-Campbell K, Kochenderfer MJ. Simulating emergent properties of human driving behavior using multi-agent reward augmented imitation learning. In 2019 International Conference on Robotics and Automation, ICRA 2019. Institute of Electrical and Electronics Engineers Inc. 2019. p. 789-795. 8793750. (Proceedings - IEEE International Conference on Robotics and Automation). https://doi.org/10.1109/ICRA.2019.8793750
Bhattacharyya, Raunak P. ; Phillips, Derek J. ; Liu, Changliu ; Gupta, Jayesh K. ; Driggs-Campbell, Katherine ; Kochenderfer, Mykel J. / Simulating emergent properties of human driving behavior using multi-agent reward augmented imitation learning. 2019 International Conference on Robotics and Automation, ICRA 2019. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 789-795 (Proceedings - IEEE International Conference on Robotics and Automation).
@inproceedings{f1d8db2d8a0d45cb865b9356a28d04fd,
title = "Simulating emergent properties of human driving behavior using multi-agent reward augmented imitation learning",
abstract = "Recent developments in multi-agent imitation learning have shown promising results for modeling the behavior of human drivers. However, it is challenging to capture emergent traffic behaviors that are observed in real-world datasets. Such behaviors arise due to the many local interactions between agents that are not commonly accounted for in imitation learning. This paper proposes Reward Augmented Imitation Learning (RAIL), which integrates reward augmentation into the multi-agent imitation learning framework and allows the designer to specify prior knowledge in a principled fashion. We prove that convergence guarantees for the imitation learning process are preserved under the application of reward augmentation. This method is validated in a driving scenario, where an entire traffic scene is controlled by driving policies learned using our proposed algorithm. Further, we demonstrate improved performance in comparison to traditional imitation learning algorithms both in terms of the local actions of a single agent and the behavior of emergent properties in complex, multi-agent settings.",
author = "Bhattacharyya, {Raunak P.} and Phillips, {Derek J.} and Changliu Liu and Gupta, {Jayesh K.} and Katherine Driggs-Campbell and Kochenderfer, {Mykel J.}",
year = "2019",
month = "5",
doi = "10.1109/ICRA.2019.8793750",
language = "English (US)",
series = "Proceedings - IEEE International Conference on Robotics and Automation",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "789--795",
booktitle = "2019 International Conference on Robotics and Automation, ICRA 2019",
address = "United States",

}

TY - GEN

T1 - Simulating emergent properties of human driving behavior using multi-agent reward augmented imitation learning

AU - Bhattacharyya, Raunak P.

AU - Phillips, Derek J.

AU - Liu, Changliu

AU - Gupta, Jayesh K.

AU - Driggs-Campbell, Katherine

AU - Kochenderfer, Mykel J.

PY - 2019/5

Y1 - 2019/5

N2 - Recent developments in multi-agent imitation learning have shown promising results for modeling the behavior of human drivers. However, it is challenging to capture emergent traffic behaviors that are observed in real-world datasets. Such behaviors arise due to the many local interactions between agents that are not commonly accounted for in imitation learning. This paper proposes Reward Augmented Imitation Learning (RAIL), which integrates reward augmentation into the multi-agent imitation learning framework and allows the designer to specify prior knowledge in a principled fashion. We prove that convergence guarantees for the imitation learning process are preserved under the application of reward augmentation. This method is validated in a driving scenario, where an entire traffic scene is controlled by driving policies learned using our proposed algorithm. Further, we demonstrate improved performance in comparison to traditional imitation learning algorithms both in terms of the local actions of a single agent and the behavior of emergent properties in complex, multi-agent settings.

AB - Recent developments in multi-agent imitation learning have shown promising results for modeling the behavior of human drivers. However, it is challenging to capture emergent traffic behaviors that are observed in real-world datasets. Such behaviors arise due to the many local interactions between agents that are not commonly accounted for in imitation learning. This paper proposes Reward Augmented Imitation Learning (RAIL), which integrates reward augmentation into the multi-agent imitation learning framework and allows the designer to specify prior knowledge in a principled fashion. We prove that convergence guarantees for the imitation learning process are preserved under the application of reward augmentation. This method is validated in a driving scenario, where an entire traffic scene is controlled by driving policies learned using our proposed algorithm. Further, we demonstrate improved performance in comparison to traditional imitation learning algorithms both in terms of the local actions of a single agent and the behavior of emergent properties in complex, multi-agent settings.

UR - http://www.scopus.com/inward/record.url?scp=85071423261&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85071423261&partnerID=8YFLogxK

U2 - 10.1109/ICRA.2019.8793750

DO - 10.1109/ICRA.2019.8793750

M3 - Conference contribution

AN - SCOPUS:85071423261

T3 - Proceedings - IEEE International Conference on Robotics and Automation

SP - 789

EP - 795

BT - 2019 International Conference on Robotics and Automation, ICRA 2019

PB - Institute of Electrical and Electronics Engineers Inc.

ER -