{"id":624726,"date":"2019-12-05T09:59:46","date_gmt":"2019-12-05T17:59:46","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=624726"},"modified":"2019-12-23T16:21:03","modified_gmt":"2019-12-24T00:21:03","slug":"game-of-drones-at-neurips-2019-simulation-based-drone-racing-competition-built-on-airsim","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/game-of-drones-at-neurips-2019-simulation-based-drone-racing-competition-built-on-airsim\/","title":{"rendered":"Game of Drones at NeurIPS 2019: Simulation-based drone-racing competition built on AirSim"},"content":{"rendered":"<p><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/12\/Research_20191202_NeurIPS_GameOfDrones_Site_1400x788.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-624957\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/12\/Research_20191202_NeurIPS_GameOfDrones_Site_1400x788.png\" alt=\"Image from Game of Drones simulation\" width=\"1400\" height=\"788\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/12\/Research_20191202_NeurIPS_GameOfDrones_Site_1400x788.png 1400w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/12\/Research_20191202_NeurIPS_GameOfDrones_Site_1400x788-300x169.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/12\/Research_20191202_NeurIPS_GameOfDrones_Site_1400x788-1024x576.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/12\/Research_20191202_NeurIPS_GameOfDrones_Site_1400x788-768x432.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/12\/Research_20191202_NeurIPS_GameOfDrones_Site_1400x788-1066x600.png 1066w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/12\/Research_20191202_NeurIPS_GameOfDrones_Site_1400x788-655x368.png 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/12\/Research_20191202_NeurIPS_GameOfDrones_Site_1400x788-343x193.png 343w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/12\/Research_20191202_NeurIPS_GameOfDrones_Site_1400x788-640x360.png 640w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/12\/Research_20191202_NeurIPS_GameOfDrones_Site_1400x788-960x540.png 960w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/12\/Research_20191202_NeurIPS_GameOfDrones_Site_1400x788-1280x720.png 1280w\" sizes=\"(max-width: 1400px) 100vw, 1400px\" \/><span class=\"sr-only\"> (opens in new tab)<\/span><\/a><\/p>\n<p>Drone racing has transformed from a niche activity sparked by enthusiastic hobbyists to an internationally televised sport. In parallel, computer vision and machine learning are making rapid progress, along with advances in agile trajectory planning, control, and state estimation for quadcopters. These advances enable increased autonomy and reliability for drones. More recently, the unmanned aerial vehicle (UAV) research community has begun to tackle the drone-racing problem. This has given rise to competitions, with the goal of beating human performance in drone racing.<\/p>\n<p>At <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/neurips.cc\/Conferences\/2019\">the thirty-third Conference on Neural Information Processing Systems<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> (NeurIPS 2019), the AirSim research team is working together with Stanford University and University of Zurich to further democratize drone-racing research by hosting a simulation-based competition, <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/microsoft.github.io\/AirSim-NeurIPS2019-Drone-Racing\/\">Game of Drones<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. We are hosting the competition on <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/github.com\/Microsoft\/Airsim\">Microsoft AirSim<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, our Unreal Engine-based simulator for multirotors. The competition focuses on trajectory planning and control, computer vision, and opponent drone avoidance. This is achieved via three tiers:<\/p>\n<ul>\n<li><strong>Tier 1 <\/strong><strong>\u2013<\/strong> Planning only: The participant\u2019s drone races t\u00eate-\u00e0-t\u00eate with a Microsoft Research opponent racer. The goal is to go through all gates in the minimum possible time, without hitting the opponent drone. Ground truth for gate poses, the opponent drone pose, and the participant drone are provided. These are accessible via our <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/github.com\/microsoft\/AirSim-NeurIPS2019-Drone-Racing#quick-api-overview\">application-programming interfaces<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> (APIs). The opponent racer follows a minimum jerk trajectory, which goes through randomized waypoints selected in each gate\u2019s cross section.<\/li>\n<li><strong>Tier 2 <\/strong><strong>\u2013<\/strong> Perception only: This is a time trial format where the participants are provided with noisy gate poses. There&#8217;s no opponent drone. The next gate will not always be in view, but the noisy pose returned by our API will steer the drone roughly in the right direction, after which vision-based control would be necessary.<\/li>\n<li><strong>Tier 3 \u2013<\/strong> Perception and Planning: This combines Tier 1 and 2. Given the ground truth state estimate for participant drone and noisy estimate for gates, the goal is to race against the opponent racer without colliding with it.<\/li>\n<\/ul>\n<p>The animation on the left below shows the ground truth gate poses (Tier 1), while the animation on the right shows the noisy gate poses (Tier 2 and Tier 3). In each animation, the drone is tracking a minimum jerk trajectory using one of our competition APIs.<\/p>\n<table style=\"width: 100%; border-collapse: collapse; border-spacing: inherit;\" border=\"1\">\n<tbody>\n<tr>\n<td style=\"padding: inherit; border: 1px solid; width: 50%;\"><u><span style=\"background-color: #bfe6ff; color: #000120;\"><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/11\/drones1.gif\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-624738\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/11\/drones1.gif\" alt=\"Image shows the ground truth gate poses\" width=\"600\" height=\"338\" \/><span class=\"sr-only\"> (opens in new tab)<\/span><\/a><\/span><\/u><\/td>\n<td style=\"padding: inherit; border: 1px solid; width: 50%;\"><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/11\/drones2.gif\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-624741\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/11\/drones2.gif\" alt=\"\" width=\"600\" height=\"338\" \/><span class=\"sr-only\"> (opens in new tab)<\/span><\/a><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>&nbsp;<\/p>\n<p>The following animation shows a segment of one of our racing tracks with two drones racing against each other. Here \u201cdrone_2\u201d (pink spline) is the opponent racer going through randomized waypoints in each gate cross section, while \u201cdrone_1\u201d (yellow spline) is a representative competitor going through the gate centers.<\/p>\n<p><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/11\/drones3.gif\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-624747\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/11\/drones3.gif\" alt=\"This animation shows a segment of one of our racing tracks with two drones racing against each other\" width=\"600\" height=\"338\" \/><span class=\"sr-only\"> (opens in new tab)<\/span><\/a><\/p>\n<p>The competition is being run in two stages\u2014an initial qualification round and a final round. A set of training binaries with configurable racetracks was made available to the participants initially, for prototyping and verification of algorithms on arbitrary racetracks. In the qualification stage (Oct 15<sup>th<\/sup> to Nov 21<sup>st<\/sup>), teams were asked to submit their entries for a subset or all of the three competition tiers.\u00a0 <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/microsoft.github.io\/AirSim-NeurIPS2019-Drone-Racing\/registered_teams_heatmap_neurips_2019.html\">117 teams registered for the competition worldwide<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, with 16 unique entries that have shown up on <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/nam06.safelinks.protection.outlook.com\/?url=https%3A%2F%2Fmicrosoft.github.io%2FAirSim-NeurIPS2019-Drone-Racing%2Fleaderboard.html&data=02%7C01%7Cv-erjunt%40microsoft.com%7C853db037758d439c6a7f08d7790c49e4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637110966000776178&sdata=i9%2FZ6ffUkY2r6Q5f5YG6lP%2B6jA04DoT4WE74WjrcFKs%3D&reserved=0\">the qualification leaderboard<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>.<\/p>\n<p>We are now running the final round of the competition and the corresponding leaderboard is <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/microsoft.github.io\/AirSim-NeurIPS2019-Drone-Racing\/leaderboard_final.html\">available here<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. All of the information for the competition is available at our <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/github.com\/microsoft\/AirSim-NeurIPS2019-Drone-Racing\">GitHub repository<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, along with the training, qualification, and final race environments.<\/p>\n<p>Engineering-wise, we introduced some new APIs in AirSim specifically for the competition, and we\u2019re continually adding more features as we get feedback. We highlight the main components below:<\/p>\n<ul>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/github.com\/microsoft\/AirSim-NeurIPS2019-Drone-Racing#loading-unreal-engine-environments\">Changing Unreal Engine environments<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> at runtime<\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/github.com\/microsoft\/AirSim-NeurIPS2019-Drone-Racing#race-apis\">Race APIs<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> to start and reset a race, get last gate passed, and get racer status<\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/github.com\/microsoft\/AirSim-NeurIPS2019-Drone-Racing#lower-level-control-apis\">Low level control APIs<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> for multiple combinations of angle and angle rate setpoints, with and without altitude stabilization<\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/github.com\/microsoft\/AirSim-NeurIPS2019-Drone-Racing#high-level-control-apis\">High level black box trajectory planning and tracking<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> APIs for users who want to focus on the perception aspect of the competition<\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/github.com\/microsoft\/AirSim-NeurIPS2019-Drone-Racing#gain-setter-apis\">Control gain setter APIs<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> for full customization of low level, medium level, and high level controllers<\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/github.com\/microsoft\/AirSim-NeurIPS2019-Drone-Racing#apis-to-help-generate-gate-detection-datasets\">Object APIs to facilitate drone gate dataset generation<span class=\"sr-only\"> (opens in new tab)<\/span><\/a><\/li>\n<\/ul>\n<p>In the long term, we intend to keep the competition open, and we will be adding more racing environments after NeurIPS 2019. While the first iteration brought an array of new features to AirSim, there are still many essential ingredients for trustable autonomy in real-world scenarios and effective simulation-to-reality transfer of learned policies. These include reliable state estimation; camera sensor models and motion blur; robustness to environmental conditions like weather, brightness, and diversity in texture and shape of the drone racing gates; and robustness against dynamics of the quadcopter. Over the next iterations, we aim to extend the competition to focus on these components of autonomy as well.<\/p>\n<p>For more of the exciting work Microsoft is doing with AirSim, see <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/blogs.microsoft.com\/ai\/ignite-2019-autonomous-systems\/\">our blog post<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> on Ignite 2019.<\/p>\n<p><strong>Acknowledgements:<\/strong> This work would not have been possible without the substantial team effort behind the scenes by all members of the organizing team\u2014Ratnesh Madaan, Nicholas Gyde, Keiko Nagami, Matthew Brown, Sai Vemprala, Tim Taubner, Eric Cristofalo, Paul Stubbs, Jim Piavis, Guada Casuso, Mac Schwager, Davide Scaramuzza, and Ashish Kapoor.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Drone racing has transformed from a niche activity sparked by enthusiastic hobbyists to an internationally televised sport. In parallel, computer vision and machine learning are making rapid progress, along with advances in agile trajectory planning, control, and state estimation for quadcopters. These advances enable increased autonomy and reliability for drones. More recently, the unmanned aerial [&hellip;]<\/p>\n","protected":false},"author":38679,"featured_media":625692,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"footnotes":""},"categories":[194467],"tags":[],"research-area":[13556],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-624726","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-artifical-intelligence","msr-research-area-artificial-intelligence","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[199565],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[237595,867219],"related-projects":[],"related-events":[609480],"related-researchers":[{"type":"guest","value":"ratnesh-madaan","user_id":"624729","display_name":"Ratnesh Madaan","author_link":"<a href=\"https:\/\/www.linkedin.com\/in\/ratneshmadaan\/\" aria-label=\"Visit the profile page for Ratnesh Madaan\">Ratnesh Madaan<\/a>","is_active":true,"last_first":"Madaan, Ratnesh","people_section":0,"alias":"ratnesh-madaan"}],"msr_type":"Post","featured_image_thumbnail":"<img width=\"960\" height=\"540\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/12\/Research_20191202_NeurIPS_GameOfDrones_Site_1400x788-5de946d379682-960x540.png\" class=\"img-object-cover\" alt=\"Game of Drones simulation\" decoding=\"async\" loading=\"lazy\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/12\/Research_20191202_NeurIPS_GameOfDrones_Site_1400x788-5de946d379682-960x540.png 960w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/12\/Research_20191202_NeurIPS_GameOfDrones_Site_1400x788-5de946d379682-300x169.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/12\/Research_20191202_NeurIPS_GameOfDrones_Site_1400x788-5de946d379682-1024x576.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/12\/Research_20191202_NeurIPS_GameOfDrones_Site_1400x788-5de946d379682-768x432.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/12\/Research_20191202_NeurIPS_GameOfDrones_Site_1400x788-5de946d379682-1066x600.png 1066w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/12\/Research_20191202_NeurIPS_GameOfDrones_Site_1400x788-5de946d379682-655x368.png 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/12\/Research_20191202_NeurIPS_GameOfDrones_Site_1400x788-5de946d379682-343x193.png 343w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/12\/Research_20191202_NeurIPS_GameOfDrones_Site_1400x788-5de946d379682-640x360.png 640w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/12\/Research_20191202_NeurIPS_GameOfDrones_Site_1400x788-5de946d379682-1280x720.png 1280w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/12\/Research_20191202_NeurIPS_GameOfDrones_Site_1400x788-5de946d379682.png 1400w\" sizes=\"(max-width: 960px) 100vw, 960px\" \/>","byline":"<a href=\"https:\/\/www.linkedin.com\/in\/ratneshmadaan\/\" title=\"Go to researcher profile for Ratnesh Madaan\" aria-label=\"Go to researcher profile for Ratnesh Madaan\" data-bi-type=\"byline author\" data-bi-cN=\"Ratnesh Madaan\">Ratnesh Madaan<\/a> and Ashish Kapoor","formattedDate":"December 5, 2019","formattedExcerpt":"Drone racing has transformed from a niche activity sparked by enthusiastic hobbyists to an internationally televised sport. In parallel, computer vision and machine learning are making rapid progress, along with advances in agile trajectory planning, control, and state estimation for quadcopters. These advances enable increased&hellip;","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/624726"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/38679"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=624726"}],"version-history":[{"count":14,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/624726\/revisions"}],"predecessor-version":[{"id":625536,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/624726\/revisions\/625536"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/625692"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=624726"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=624726"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=624726"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=624726"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=624726"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=624726"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=624726"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=624726"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=624726"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=624726"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=624726"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}