Introduction
Robot navigation is one of the most studied problems in robotics and the key capability for robot autonomy. Navigation techniques have become more and more reliable, but evaluation mainly focused on individual navigation components (i.e., mapping, localization, and planning) using datasets or simulations. The goal of BRIN is to define an experimental protocol to evaluate the whole navigation system, deployed in a real environment. To ensure repeatability and reproducibility of experiments, BRIN includes detailed definitions and controls of the environment dynamics. BRIN defines standardized environments and includes the use of a reference robot to allow comparison between different navigation systems at different experimentation sites.
Contents
“Navigation Benchmark Instructions” provides a detailed overview of BRIN and the benchmarking process from choosing your environments to acquiring the right props. It includes the complete testing procedure.
“Navigation Benchmark” is the technical specification for the benchmark itself and the NavBenchmark software that enables “inexpensive and easy to deploy” ground truth measurement and benchmark result generation. It describes how the navigation application being tested will communicate progress with the NavBenchmark software during the test.
“Reference Robot Setup and Execution” explains how to use the Adept MobileRobots Pioneer 3 DX robot with the provided sample navigation application to generate “reference” results enabling cross-environment comparisons.
Getting Started
We recognize that the procedure is quite involved and detailed. We invite you to open a dialog with us before you start using it to discuss any questions you might have about choosing the best environments and how to implement the challenges and conduct the interactions. We would be happy to work together with you to ensure a successful outcome. Contact us at brin@microsoft.com.
When you are ready to start benchmarking, please download the NavBenchmarkBeta2.zip file. In it you will find the latest readme.txt with instructions on installation and getting started.
