{"id":502862,"date":"2018-08-27T23:27:11","date_gmt":"2018-08-28T06:27:11","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-project&#038;p=502862"},"modified":"2024-04-04T10:26:39","modified_gmt":"2024-04-04T17:26:39","slug":"project-frigatebird-ai-for-autonomous-soaring","status":"publish","type":"msr-project","link":"https:\/\/www.microsoft.com\/en-us\/research\/project\/project-frigatebird-ai-for-autonomous-soaring\/","title":{"rendered":"Project Frigatebird: AI for Autonomous Soaring"},"content":{"rendered":"<h2>Autonomous Soaring: an Open-World Challenge for AI<\/h2>\n<p>Techniques for automatic decision making under uncertainty have been making great strides in their ability to learn complex policies from streams of observations. However, this progress is happening mostly in &#8212; and has a bias towards &#8212; settings with abundant data or readily available high-fidelity simulators, such as games. Learning algorithms in these environments enjoy luxuries unavailable to AI agents in the open world, including resettable training episodes, many attempts at completing a task, and negligible costs of making mistakes.<\/p>\n<p>Enabling fixed-wing small uninhabited aerial vehicles (sUAVs) to travel hundreds of miles without using a motor by taking advantage of regions of rising air the way some bird species and sailplane pilots do, challenges modern reinforcement learning and related approaches at all their weak spots. It requires building an AI agent that, given limited computational resources onboard, can plan far in advance for a wide variety of highly uncertain atmospheric conditions only crudely modeled by existing simulators, deal with these conditions&#8217; non-stationarity, and learn literally on the fly, with mistakes potentially resulting in drone loss.<\/p>\n<div>\n<div style=\"width: 90%;margin: 0 auto\">\n<div style=\"float: left;padding: 5px\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-large wp-image-503252\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/08\/F3J_Shadow-1024x635.jpg\" alt=\"\" width=\"362\" height=\"635\" \/><\/div>\n<div style=\"float: right;padding: 5px\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-large wp-image-503255\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/08\/frigate-1024x683.jpg\" alt=\"\" width=\"335\" height=\"683\" \/><\/div>\n<p style=\"clear: both\" align=\"center\">\n<\/div>\n<p align=\"center\"><small>A frigatebird and one of its sUAV counterparts from our fleet, an F3J Shadow.<br \/>\n<em><span style=\"font-size: xx-small\">Frigatebird photo credit: Benjamint444, <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/commons.wikimedia.org\/w\/index.php?curid=13487246\">Wikimedia Commons<\/a>. Licensed under <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/www.gnu.org\/licenses\/old-licenses\/fdl-1.2.html\">GFDL 1.2<\/a>.<\/span><\/em><\/small><\/p>\n<\/div>\n<h2>Project Goals<\/h2>\n<p>We aim to push the state of the of the art in AI decision making under uncertainty through enabling sailplane (a.k.a. &#8220;glider&#8221;) sUAVs to do what the most adept soarers &#8212; frigatebirds &#8212; can: <strong>autonomously fly long distances using energy extracted from the atmosphere<\/strong>. The problems we are interested in solving with AI include:<\/p>\n<ul>\n<li><strong>Thermal, ridge, and wave soaring.\u00a0<\/strong> Air can move upwards, carrying a sailplane with it, for a number of reasons. It may be heated up by the Earth&#8217;s surface beneath it, giving rise to thermals. It may be pushed up by the wind blowing against an obstacle such as a cliff, creating ridge soaring opportunities. Or, its flow over a mountain top can create standing waves that can take a sUAV as high as the stratosphere. In all cases, the sUAV&#8217;s AI needs to identify the rising air pattern and exploit it immediately, while ensuring that the sUAV remains at a safe distance from ground structures and within a safe operating envelope.<\/li>\n<li><strong>Long-range motorless navigation with atmospheric energy harvesting.<\/strong> Navigating from A to B for a sailplane sUAV, which is at best equipped with a motor that can run a few minutes, is very much unlike path-finding for aircraft with stable active propulsion. Even equipped with a soaring capability, the UAV&#8217;s AI needs to route the aircraft through areas where opportunities for soaring exist in the first place. Their locations are never known with certainty, and the AI must constantly update its belief about them.<\/li>\n<li><strong>Dynamic soaring.<\/strong> Frigatebirds spend much of their time meters flying over seas only meters from wave crests while hardly producing a wing flap. They extract energy from a wind gradient &#8212; an activity that requires immaculate situational awareness and control. Presently, it is beyond the capabilities of robotic flying machines.<\/li>\n<\/ul>\n<div id=\"attachment_503123\" style=\"width: 335px\" class=\"wp-caption alignright\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-503123\" class=\"wp-image-503123\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/08\/Radian_Hawk.jpg\" alt=\"\" width=\"325\" height=\"211\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/08\/Radian_Hawk.jpg 723w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/08\/Radian_Hawk-300x195.jpg 300w\" sizes=\"(max-width: 325px) 100vw, 325px\" \/><p id=\"caption-attachment-503123\" class=\"wp-caption-text\">A Radian Pro sUAV (left) and a hawk sharing a thermal.<\/p><\/div>\n<ul>\n<li><strong>Incorporating visual cues into decision making.<\/strong> Although our sUAVs have sensors to detect a rising air mass once they are <em>in<\/em> it, they can only guess where to find it. Visual cues such as circling birds or certain cloud types can greatly help pinpoint the location of a thermal. Incorporating them into path planning and\u00a0balancing energy demands of onboard video processing against its benefits present interesting research problems.<\/li>\n<\/ul>\n<h2>Methods and First Steps<\/h2>\n<p>Building an AI for soaring differs from designing controllers for other drones and ground vehicles in a fundamental way: <strong>transition dynamics crucial for continuing flight are nonstationary and a-priori unknown in many operating regions<\/strong>.\u00a0 Indeed, a sailplane sUAV is usually very uncertain where to find areas of rising air, and even when it flies through a thermal, initially doesn&#8217;t know how strong of an updraft the thermal provides, where this updraft begins and ends, and how turbulent it is. Other kinds of autonomous vehicles also occasionally face situations with uncertain or unexpected dynamics\u00a0(imagine a car driving into an oil slick). However, for them these situations are usually undesirable, and they\u00a0try to escape them using reactive controllers. On the contrary, <strong>AI for soaring aims to exploit this uncertain dynamics for extending flight time, and needs to learn it and plan for it deliberatively.<\/strong><\/p>\n<p>Following this intuition, in the first stage of this project we focused on approaches for using thermals autonomously to gain altitude.<\/p>\n<div id=\"attachment_503516\" style=\"width: 307px\" class=\"wp-caption alignright\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-503516\" class=\"wp-image-503516\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/08\/POMDSoar-1024x483.png\" alt=\"\" width=\"297\" height=\"140\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/08\/POMDSoar-1024x483.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/08\/POMDSoar-300x142.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/08\/POMDSoar-768x362.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/08\/POMDSoar.png 1462w\" sizes=\"(max-width: 297px) 100vw, 297px\" \/><p id=\"caption-attachment-503516\" class=\"wp-caption-text\">POMDSoar illustration<\/p><\/div>\n<ul>\n<li><strong>In the <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/arxiv.org\/pdf\/1805.09875.pdf\"><strong>RSS-2018 paper<\/strong><\/a> <\/strong> we modeled thermalling as a Bayesian reinforcement learning problem and proposed a thermalling algorithm, POMDSoar that deliberatively plans thermal exploration and exploitation. POMDSoar outperformed another strong thermalling controller, ArduSoar, which does not do explicit exploration, in live experiments in adverse thermalling conditions.<\/li>\n<li><strong>In the <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/arxiv.org\/abs\/1802.08215\"><strong>IROS-2018 paper<\/strong><\/a><\/strong> we studied ArduSoar in more detail to better understand its strengths and limitations. We have found it to perform very well given its approach that plans based on the current most likely dynamics model, regularly reestimating this model but not taking deliberate steps to gather information for it.<\/li>\n<\/ul>\n<div>\n\t<iframe\n\t\tsrc=\"https:\/\/ayvri.com\/embed\/eyj8r7ny5g\/cjl746v4r00023b6a4p5i4gez\"\n\t\twidth=\"800\"\n\t\theight=\"450\"\n\t\taria-label=\"\"\n\t\tallowfullscreen=\"true\">\n\t<\/iframe>\n\t\n<p align=\"center\"><small>3D replay of a simultaneous live test flight of two identical Radian Pros running ArduSoar and POMDSoar. Zoom into the scene and rotate it with the mouse to get different views. The Radians flew the same default waypoint sequence, repeatedly climbing to a specified altitude using motors, turning off the motors, and gliding down. Whenever they encountered a thermal in a glide, they would interrupt waypoint following and try to soar. Thus, their thermalling ability extended the sUAVs&#8217; flight duration. Replays of many more of our test flights are viewable on <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" style=\"vertical-align: middle\" href=\"https:\/\/ayvri.com\/profile\/eyj8r7ny5g\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/08\/ayvri_ai.png\" width=\"60\" height=\"60\" \/><\/a><\/small><\/p>\n<\/div>\n<p>Research literature describes special-purpose controllers partly based on human intuitions, such as ArduSoar, for several aspects of soaring. Despite their lack of explicit information gathering, some of them prove challenging to outperform for more general algorithms.\u00a0Our current work focuses on (a) <strong>theoretically understanding when these pure control\/replanning-based approaches are near-optimal<\/strong> (b) exploring <strong>reinforcement and imitation learning-based approaches<\/strong> that relax existing baselines&#8217; assumptions and (c) researching <strong>long-range navigation algorithms that produce robust routing policies in spite of high uncertainty<\/strong> about the locations of lift sources.<\/p>\n<h2>Equipment<\/h2>\n<p>Our sUAV fleet currently consists of 1 Thermik XXXL (5m wingspan), 1 F3J Shadow (3.5m wingspan), several Radian Pros (2m wingspan), and 1 Snipe-2 (2.15m wingspan). Onbard equipment varies slightly depending on the airframe type, but always includes a GPS, compass, telemetry radio, RC receiver, pitot\/static sensor, barometer, IMU, and a computer for fully autonomous flight.<\/p>\n<div style=\"width: 90%;margin: 0 auto\">\n<div style=\"float: left;padding: 5px\">\n<div id=\"attachment_503213\" style=\"width: 258px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-503213\" class=\" wp-image-503213\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/08\/equipment_diagram-1024x768.jpg\" alt=\"\" width=\"248\" height=\"305\" \/><p id=\"caption-attachment-503213\" class=\"wp-caption-text\">Equipment onboard a Radian Pro sUAV.<\/p><\/div>\n<\/div>\n<div style=\"float: right;padding: 5px\">\n<div id=\"attachment_503189\" style=\"width: 445px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-503189\" class=\"wp-image-503189\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/08\/fleet2.jpg\" alt=\"\" width=\"435\" height=\"355\" \/><p id=\"caption-attachment-503189\" class=\"wp-caption-text\">Our fleet: Thermik XXXL, F3J Shadow, a few Radian Pros, and Snipe-2.<\/p><\/div>\n<\/div>\n<p style=\"clear: both\">\n<\/div>\n<h2>Code, Parameters, Data<\/h2>\n<p>The project&#8217;s code is on GitHub:<\/p>\n<p><a class=\"button-solid no-margin-bottom\" style=\"margin-top: 10px\" href=\"https:\/\/github.com\/Microsoft\/Frigatebird\" target=\"_blank\" rel=\"noopener noreferrer\">Frigatebird on GitHub<\/a><\/p>\n<p>Currently, this repository contains an implementation of POMDPSoar, a POMDP-based thermalling controller from our <a href=\"https:\/\/arxiv.org\/pdf\/1805.09875.pdf\">RSS-2018 paper<\/a>. We have implemented POMDSoar in a fork of <a href=\"https:\/\/github.com\/ArduPilot\/ardupilot\/tree\/master\/ArduPlane\">ArduPlane<\/a>, a flavor of <a href=\"http:\/\/ardupilot.org\/\">ArduPilot<\/a> open-source drone autopilot suite for fixed-wing sUAVs. Please refer to the README.md for build instructions. The parameter files for Radian Pros that we have been using for experiments are in the DATA subdirectory.<\/p>\n<p>We also intend to contribute implementations of existing soaring controllers such as ArduSoar (see the <a href=\"https:\/\/arxiv.org\/abs\/1802.08215\">IROS-2018 paper<\/a>), which serve as benchmarks in our experiments, to ArduPilot directly.<\/p>\n<p>3D replays of GPS traces from many of our live test flights are available on\u00a0<a style=\"vertical-align: middle\" href=\"https:\/\/ayvri.com\/profile\/eyj8r7ny5g\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/08\/ayvri_ai.png\" width=\"100\" height=\"10\" \/><\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Autonomous Soaring: an Open-World Challenge for AI Techniques for automatic decision making under uncertainty have been making great strides in their ability to learn complex policies from streams of observations. However, this progress is happening mostly in &#8212; and has a bias towards &#8212; settings with abundant data or readily available high-fidelity simulators, such as [&hellip;]<\/p>\n","protected":false},"featured_media":503003,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"footnotes":""},"research-area":[13556],"msr-locale":[268875],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-502862","msr-project","type-msr-project","status-publish","has-post-thumbnail","hentry","msr-research-area-artificial-intelligence","msr-locale-en_us","msr-archive-status-active"],"msr_project_start":"","related-publications":[502712,502700,713527,1002690,1022382],"related-downloads":[],"related-videos":[],"related-groups":[],"related-events":[],"related-opportunities":[],"related-posts":[],"related-articles":[],"tab-content":[],"slides":[],"related-researchers":[{"type":"user_nicename","display_name":"Andrey Kolobov","user_id":30910,"people_section":"Section name 1","alias":"akolobov"},{"type":"guest","display_name":"Iain Guilliard","user_id":502964,"people_section":"Section name 1","alias":""},{"type":"user_nicename","display_name":"Rick Rogahn","user_id":33458,"people_section":"Section name 1","alias":"rrogahn"}],"msr_research_lab":[199565],"msr_impact_theme":[],"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/502862"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-project"}],"version-history":[{"count":137,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/502862\/revisions"}],"predecessor-version":[{"id":1022541,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/502862\/revisions\/1022541"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/503003"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=502862"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=502862"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=502862"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=502862"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=502862"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}