Combining Optimal Control and Learning for Visual Navigation in Novel Environments

Somil Bansal, Varun Tolani, Saurabh Gupta, Jitendra Malik, Claire Tomlin

Research output: Contribution to journalConference articlepeer-review


Model-based control is a popular paradigm for robot navigation because it can leverage a known dynamics model to efficiently plan robust robot trajectories. However, it is challenging to use model-based methods in settings where the environment is a priori unknown and can only be observed partially through onboard sensors on the robot. In this work, we address this short-coming by coupling model-based control with learning-based perception. The learning-based perception module produces a series of waypoints that guide the robot to the goal via a collision-free path. These waypoints are used by a model-based planner to generate a smooth and dynamically feasible trajectory that is executed on the physical system using feedback control. Our experiments in simulated real-world cluttered environments and on an actual ground vehicle demonstrate that the proposed approach can reach goal locations more reliably and efficiently in novel environments as compared to purely geometric mapping-based or end-to-end learning-based alternatives. Our approach does not rely on detailed explicit 3D maps of the environment, works well with low frame rates, and generalizes well from simulation to the real world.

Original languageEnglish (US)
Pages (from-to)420-429
Number of pages10
JournalProceedings of Machine Learning Research
StatePublished - 2019
Event3rd Conference on Robot Learning, CoRL 2019 - Osaka, Japan
Duration: Oct 30 2019Nov 1 2019


  • Learning for Control
  • Learning for Visual Navigation
  • Optimal Control

ASJC Scopus subject areas

  • Artificial Intelligence
  • Software
  • Control and Systems Engineering
  • Statistics and Probability


Dive into the research topics of 'Combining Optimal Control and Learning for Visual Navigation in Novel Environments'. Together they form a unique fingerprint.

Cite this