{"id":867693,"date":"2022-08-09T16:39:47","date_gmt":"2022-08-09T23:39:47","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-blog-post&p=867693"},"modified":"2022-08-10T10:21:06","modified_gmt":"2022-08-10T17:21:06","slug":"robot-language","status":"publish","type":"msr-blog-post","link":"https:\/\/www.microsoft.com\/en-us\/research\/articles\/robot-language\/","title":{"rendered":"Just say the magic word: using language to program robots"},"content":{"rendered":"\n

LaTTe paper (opens in new tab)<\/span><\/a> and video (opens in new tab)<\/span><\/a> | Trajectory Transformer paper (opens in new tab)<\/span><\/a> and video (opens in new tab)<\/span><\/a> | Github code (opens in new tab)<\/span><\/a><\/p>\n\n\n\n

Language is the most intuitive way for us to express how we feel and what we want. However, despite recent advancements in artificial intelligence, it is still very hard to control a robot using natural language instructions. Free-form commands such as \u201cRobot, please go a little slower when you pass close to my TV\u201d or \u201cStay far away from the swimming pool!\u201d are hard to parse into actionable robot behaviors, and most human-robot interfaces today still rely on complex strategies such directly programming cost functions which define the desired behavior. <\/p>\n\n\n\n

With our latest work, we attempt to change this reality through the introduction of \u201cLaTTe: Language Trajectory Transformer\u201d (opens in new tab)<\/span><\/a>. LaTTe is a deep machine learning model that lets us send language commands to robots in an intuitive way with ease. When given an input sentence by the user, the model fuses it with camera images of objects that the robot observes in its surroundings, and outputs the desired robot behavior.  <\/p>\n\n\n\n

As an example, think of a user trying to control a robot barista that\u2019s moving a wine bottle. Our method allows a non-technical user to control the robot\u2019s behavior only using words, in a natural and simple interface. We will explain how we can achieve this in detail through this post. <\/p>\n\n\n\n