{"id":584854,"date":"2019-05-08T12:44:12","date_gmt":"2019-05-08T19:44:12","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=584854"},"modified":"2019-06-17T11:23:07","modified_gmt":"2019-06-17T18:23:07","slug":"reinforcement-learning-for-the-real-world-with-dr-john-langford-and-rafah-hosn","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/podcast\/reinforcement-learning-for-the-real-world-with-dr-john-langford-and-rafah-hosn\/","title":{"rendered":"Reinforcement learning for the real world with Dr. John Langford and Rafah Hosn"},"content":{"rendered":"<p><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/05\/John-Langford-And-Rafah-Hosn_Podcast_Site_04_2019_1400x788.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-584869 size-large\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/05\/John-Langford-And-Rafah-Hosn_Podcast_Site_04_2019_1400x788-1024x576.png\" alt=\"Dr. John Langford and Rafah Hosn\" width=\"1024\" height=\"576\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/05\/John-Langford-And-Rafah-Hosn_Podcast_Site_04_2019_1400x788-1024x576.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/05\/John-Langford-And-Rafah-Hosn_Podcast_Site_04_2019_1400x788-300x169.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/05\/John-Langford-And-Rafah-Hosn_Podcast_Site_04_2019_1400x788-768x432.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/05\/John-Langford-And-Rafah-Hosn_Podcast_Site_04_2019_1400x788-1066x600.png 1066w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/05\/John-Langford-And-Rafah-Hosn_Podcast_Site_04_2019_1400x788-655x368.png 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/05\/John-Langford-And-Rafah-Hosn_Podcast_Site_04_2019_1400x788-343x193.png 343w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/05\/John-Langford-And-Rafah-Hosn_Podcast_Site_04_2019_1400x788.png 1400w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/a><\/p>\n<h3>Episode 75, May 8, 2019<\/h3>\n<p>Dr. <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/jcl\/\">John Langford<\/a>, a partner researcher in the Machine Learning group at Microsoft Research New York City, is a reinforcement learning expert who is working, in his own words, to solve machine learning. <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/raaboulh\/\">Rafah Hosn<\/a>, also of MSR New York, is a principal program manager who\u2019s working to take that work to the world. If that sounds like big thinking in the Big Apple, well, New York City has always been a \u201cgo big, or go home\u201d kind of town, and MSR NYC is a \u201cgo big, or go home\u201d kind of lab.<\/p>\n<p>Today, Dr. Langford explains why online reinforcement learning is critical to solving machine learning and how moving from the current foundation of a Markov decision process toward a <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/contextual-bandits-with-continuous-actions-smoothing-zooming-and-adapting\/\">contextual bandit<\/a> future might be part of the solution. Rafah Hosn talks about why it\u2019s important, from a business perspective, to move RL agents out of simulated environments and into the open world, and gives us an under-the-hood look at the product side of MSR\u2019s \u201cresearch, incubate, transfer\u201d process, focusing on real world reinforcement learning which, at Microsoft, is now called <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/azure.microsoft.com\/en-us\/services\/cognitive-services\/personalizer\/\">Azure Cognitive Services Personalizer<\/a>.<\/p>\n<h3>Related:<\/h3>\n<ul type=\"disc\">\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/podcast\">Microsoft Research Podcast<\/a>: View more podcasts on Microsoft.com<\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/itunes.apple.com\/us\/podcast\/microsoft-research-a-podcast\/id1318021537?mt=2\">iTunes<\/a>: Subscribe and listen to new podcasts each week on iTunes<\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/subscribebyemail.com\/www.blubrry.com\/feeds\/microsoftresearch.xml\">Email<\/a>: Subscribe and listen by email<\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/subscribeonandroid.com\/www.blubrry.com\/feeds\/microsoftresearch.xml\">Android<\/a>: Subscribe and listen on Android<\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/open.spotify.com\/show\/4ndjUXyL0hH1FXHgwIiTWU\">Spotify<\/a>: Listen on Spotify<\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/www.blubrry.com\/feeds\/microsoftresearch.xml\">RSS feed<\/a><\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/note.microsoft.com\/ww-registration-microsoft-research-newsletter-s.html?wt.mc_id=S-webpage_podcast\">Microsoft Research Newsletter<\/a>: Sign up to receive the latest news from Microsoft Research<\/li>\n<\/ul>\n<hr \/>\n<h3>Transcript<\/h3>\n<p><strong>Host: Welcome to another two-chair, two-mic episode of the Microsoft Research Podcast. Today we bring you the perspectives of two guests on the topic of reinforcement learning for online applications. Since most research wants to be a product when it grows up, we\u2019ve brought in a brilliant researcher\/program manager duo to illuminate the classic \u201cresearch, incubate, transfer\u201d process in the context of real-world reinforcement learning.<\/strong><\/p>\n<p><strong>Host: You\u2019re listening to the Microsoft Research Podcast, a show that brings you closer to the cutting-edge of technology research and the scientists behind it. I\u2019m your host, Gretchen Huizinga.<\/strong><\/p>\n<p><strong>Host: Dr. John Langford, a partner researcher in the Machine Learning group at Microsoft Research New York City, is a reinforcement learning expert who is working, in his own words, to solve machine learning. Rafah Hosn, also of MSR New York, is a principal program manager who\u2019s working to take that work to the world. If that sounds like big thinking in the Big Apple, well, New York City has always been a \u201cgo big, or go home\u201d kind of town, and MSR NYC is a \u201cgo big, or go home\u201d kind of lab.<\/strong><\/p>\n<p>Today, Dr. Langford explains why online reinforcement learning is critical to solving machine learning and how moving from the current foundation of a Markov decision process toward a contextual bandit future might be part of the solution. Rafah Hosn talks about why it\u2019s important, from a business perspective, to move RL agents out of simulated environments and into the open world, and gives us an under-the-hood look at the product side of MSR\u2019s \u201cresearch, incubate, transfer\u201d process, focusing on real world reinforcement learning which, at Microsoft, is now called Azure Cognitive Services Personalizer. That and much more on this episode of the Microsoft Research Podcast.<\/p>\n<p><strong>Host: I\u2019ve got two guests in the both today, both working on some big research problems in the Big Apple. John Langford is a partner researcher in the Machine Learning group at MSR NYC, and Rafah Hosn, also at the New York lab, is the principal program manager for personalization service, also known as real-work reinforcement learning. John and Rafah, welcome to the podcast.<\/strong><\/p>\n<p>John Langford: Thank you.<\/p>\n<p>Rafah Hosn: Thank you.<\/p>\n<p><strong>Host: Microsoft Research\u2019s New York lab is relatively small, in the constellation of MSR labs, but there\u2019s some really important work going on there. So, to get us started, tell us what each of you does for a living and how you work together. What gets you up in the morning? Rafah, why don\u2019t you start?<\/strong><\/p>\n<p>Rafah Hosn: Okay, I\u2019ll start. So I wake up every day and think about all the great things that the reinforcement learning researchers are doing and first I map what they\u2019re working on, something that could be useful for customers, and then I think to myself, how can we now take this great research, which typically comes in the form of a paper, to a prototype, to an incubation, to something that Microsoft can make money out of?<\/p>\n<p><strong>Host: That\u2019s a big thread, starting with a little seed, and ending up with a big plant at the end.<\/strong><\/p>\n<p>Rafah Hosn: Yes, we have to think big.<\/p>\n<p><strong>Host: That\u2019s right. How about you, John?<\/strong><\/p>\n<p>John Langford: I want to solve machine learning! And that\u2019s ambitious, but one of the things that you really need to do if you want to solve machine learning is you need to solve reinforcement learning, which is kind of the common basis for learning algorithms to learn from interaction with the real world. And so, figuring out new ways to do this, or trying to expand the scope of where we can actually apply these techniques, is what really drives me.<\/p>\n<p><strong>Host: Can you go a little deeper into \u201csolve machine learning?\u201d What would solving machine learning look like?<\/strong><\/p>\n<p>John Langford: It would look like anything that you can pose to a machine learning problem you can solve, right? So, I became interested in machine learning back when I was an undergrad, actually.<\/p>\n<p><strong>Host: Yeah.<\/strong><\/p>\n<p>John Langford: I went to a machine learning class and I was like, oh, this is what I want to do for my life! And I\u2019ve been pursuing it ever since.<\/p>\n<p><strong>Host: And here you are.<\/strong><\/p>\n<p>John Langford: Yeah.<\/p>\n<p><strong>Host: So, we\u2019re going to spend the bulk of our time today talking about the specific work you\u2019re doing in reinforcement learning. But John, before we get into it, give us a little context as a level set. From your perspective, what\u2019s unique about reinforcement learning within the machine learning universe, and why is it an important part of MSR\u2019s research portfolio?<\/strong><\/p>\n<p>John Langford: So, most of the machine learning that\u2019s actually deployed is of the supervised learning variety. And supervised learning is fundamentally about taking expertise from people and making that into some sort of learned function that you can then use to do some task. Reinforcement learning is different because it\u2019s about taking information from the world and learning a policy for interacting with the world so that you perform better in one way or another. So, that different source of information can be incredibly powerful, because you can imagine a future where, every time you type on the keyboard, the keyboard learns to understand you better, right? Or every time you interact with some website, it understands better what your preferences are, so the world just starts working better and better in interacting with people.<\/p>\n<p><strong>Host: And so, reinforcement learning, as a method within the machine learning world, is different from other methods because you deploy it in less-known circumstances, or how would you define that?<\/strong><\/p>\n<p>John Langford: So, it\u2019s different in many ways, but the key difference is the information source. The consequence of that is that reinforcement learning can be surprising. It can actually surprise you. It can find solutions you might have not thought of to solve problems that you posed to it. That\u2019s one of the key things. Another thing is, it requires substantially more skill to apply than supervised learning. Supervised learning is pretty straightforward as far as the statistics go, while reinforcement learning, there\u2019s some real traps out there, and you want to think carefully about what you\u2019re doing. Let me go into a little more detail there.<\/p>\n<p><strong>Host: Please do.<\/strong><\/p>\n<p>John Langford: Let\u2019s suppose you need to make a sequence of ten steps, and you want to maximize the rewards you get in those ten steps, right? So, it might be the case that going left gives you a small reward immediately, and then you get no more rewards. While if you go right, you get no reward, and then you go left, and then right, and then right, and then left, and then right, so on, ten times\u2026 do it just the right way, you get a big reward, right? So many reinforcement learning algorithms just learn to go left, because that gave the small reward immediately. And that gap is not like a little gap. It\u2019s like, you may require exponentially many more samples to learn unless you actually gather the information in an intelligible, conscious way.<\/p>\n<p><strong>Host: Yeah. I\u2019m grinning, and no one can see it, because I\u2019m thinking, that\u2019s how people operate generally, you know? If I&#8230;<\/strong><\/p>\n<p>Rafah Hosn: Actually, yeah. I mean, the way I explain reinforcement learning is the way you teach a puppy how to do a trick. And the puppy may surprise you and do something else, but the reward that John speaks of is the treat that you give the puppy when the puppy does what you are trying to teach it to do, and sometimes they just surprise you and do something different. And actually, reinforcement learning has a very great affinity to Pavlovian psychology.<\/p>\n<p><strong>Host: Well, back to your example, John, you\u2019re saying if you turn left you get the reward immediately.<\/strong><\/p>\n<p>John Langford: Yeah, a small reward immediately.<\/p>\n<p><strong>Host: A small reward. So, the agent would have to go through many, many steps of this to figure out, don\u2019t go left, because you\u2019ll get more later.<\/strong><\/p>\n<p>John Langford: You\u2019ll get more later if you go right and you take the right actions after you go right.<\/p>\n<p>Rafah Hosn: Now, imagine explaining this to a customer.<\/p>\n<p><strong>Host: And we will get there, and I\u2019ll have you explain it. Rafah, let\u2019s talk for a second about the personalization service, which is an instantiation of what you call real-world reinforcement learning, yeah?<\/strong><\/p>\n<p>Rafah Hosn: That\u2019s right.<\/p>\n<p><strong>Host: So, you characterize it as a general framework for reinforcement learning algorithms that are suitable for real-world applications. Unpack that a bit. Give us a short primer on real-world reinforcement learning and why it\u2019s an important direction for reinforcement learning in general.<\/strong><\/p>\n<p>Rafah Hosn: Yeah, I\u2019ll give you my version, and I\u2019m sure John will chime in. But, you know, many of the reinforcement learning that people hear about are almost always done in a simulated environment, where you can be creative as to what you simulate, and you can generate, you know, gazillions of samples to make your agents work. Our type of reinforcement\u2026 John\u2019s type of reinforcement learning is something that we deploy online, and what drives us, John and I, is to create or use this methodology to solve real-world problems. And our goal is really to advance the science in order to help enterprises maximize their business objective through the usage of real-world reinforcement learning. So, when I say real world, these are models that we deploy, in production with real users, getting real feedback, and they learn on the job.<\/p>\n<p><strong>Host: Well, John, talk a little bit about what Rafah has alluded to. There\u2019s an online, real-world element to it, but prior to this, reinforcement learning has had some big investments in the gaming space. Tell us the difference and what happens when you move from a very closed environment to a very open environment from a technical perspective.<\/strong><\/p>\n<p>John Langford: Yeah, so I guess the first thing to understand is why you\u2019d want to do this, because if reinforcement learning in simulators works great, then why do you need to do something else? And I guess the answer is, there are many things you just can\u2019t simulate. So, an example that I often give in talks is, would I be interested in a news article about Ukraine? The answer is, yes. Because my wife is from Ukraine. But you would never know this. Your simulator would never know this. There would be no way for the policy to actually learn that if you\u2019re learning in the simulator.<\/p>\n<p><strong>Host: Right.<\/strong><\/p>\n<p>John Langford: So, there are many problems where there are no good simulators. And in those simulators, you don\u2019t have a choice. So, given that you don\u2019t have a choice, you need to embrace the difficulties of the problem. So, what are the difficulties of the real-world reinforcement learning problems? Well, you don\u2019t have zillions of examples which are typically required for many of the existing deep reinforcement learning algorithms. You need to be careful about how you use your samples. You need to use them to maximum and utmost efficiency in trying to do the learning. Another element that happens is often, when people have simulators, those simulators are kind of effectively stationary. They stay the same throughout the process of training. But in real-world problems, many of them that we encounter, we run into all kinds of non-stationarities, these exogenous events, the algorithms need to be very robust, so the combination of using samples very efficiently in great robustness in these algorithms are kind of key offsetting elements from what you might see in other places.<\/p>\n<p><strong>Host: Which is challenging Alpha Go or Ms. Pacman or the other games that have been sort of flags waved about our progress in reinforcement learning?<\/strong><\/p>\n<p>John Langford: I think those are fun applications. I really enjoy reading about them and learning about them. I think it\u2019s a great demonstration of where the field has gotten, but I feel like there\u2019s this issue of AI winter, right? So, there was once a time when AI crashed. That may happen again, because AI is now a buzzword. But I think it\u2019s important that we actually do things that have some real value in the world which actually affect peoples\u2019 lives, because that\u2019s what creates a lasting wave of innovation and puts civilization into a new place.<\/p>\n<p><strong>Host: Right.<\/strong><\/p>\n<p>John Langford: So that\u2019s what I\u2019m really seeking.<\/p>\n<p><strong>Host: What season are we in now? I\u2019ve heard there has been more than one AI winter, and some people are saying it\u2019s AI spring. I don&#8217;t know. Where do you see us in terms of that progress?<\/strong><\/p>\n<p>John Langford: I think it\u2019s fair to say that there\u2019s a lot of froth in terms of people claiming things that are not going to come to pass. At the same time, there is real value being created. Suddenly we can do things and things work better through some of these techniques, right? So, it\u2019s kind of this mishmash of overpromised things that are going to fail, and there are things that are not overpromised, and they will succeed, and so if there\u2019s enough of those that succeed, then maybe we don\u2019t have a winter. Maybe it just becomes a long summer.<\/p>\n<p><strong>Host: Like San Diego all the time\u2026.<\/strong><\/p>\n<p>Rafah Hosn: Yeah, but I think, to comment on John\u2019s point here, I think reinforcement learning is a nascent technique compared to supervised learning. And what\u2019s important is to do the crawl, walk, run, right? So, yeah, it\u2019s sexy now and people are talking about it, but we need to rein it in from a business perspective as to, you know, what are the classes of problems that we can satisfy the business leader with? And satisfy them effectively, right? And I think, from a reinforcement learning, John, correct me, we are very much at the crawl phase in solving generic business problems.<\/p>\n<p>John Langford: I mean, we have solved some generic business problems. But we don\u2019t have widely deployed, or deployable, platforms for reusing those solutions over and over again. And it\u2019s so easy to imagine many more applications than people have even tried. So, we\u2019re nowhere near a mature phase in terms of even simple kinds of reinforcement learning. We are ramping up in our ability to solve real-world reinforcement learning problems\u2026<\/p>\n<p><strong>Host: Heading towards\u2026<\/strong><\/p>\n<p>John Langford: \u2026and there\u2019s a huge ramp still to happen.<\/p>\n<p><strong>Host: Heading towards your goal of solving machine learning?<\/strong><\/p>\n<p>John Langford: Yes.<\/p>\n<p>Rafah Hosn: But I mean to be fair though, we can actually satisfy some classes of problems really well with nascent technology.<\/p>\n<p><strong>Host: Yes.<\/strong><\/p>\n<p>Rafah Hosn: So yes, we are nascent, and the world is out there for us to conquer, but I think we do have techniques that can solve a whole swath of problems and it\u2019s up to us to harvest that.<\/p>\n<p>(music plays)<\/p>\n<p><strong>Host: Well, let\u2019s continue the thread a little bit on the research areas of reinforcement learning. And there\u2019re several that seem to be gaining traction. Let\u2019s go sort of high level and talk about this one area that you\u2019re saying is basically creating a new foundation for reinforcement learning. What\u2019s wrong with the current foundation, what do we need the new foundation for, and what are you doing?<\/strong><\/p>\n<p>John Langford: The current foundation of reinforcement learning is called a Markov decision process. The idea in a Markov decision process is that you have states and actions and, given a state, you take an action, then you have some distribution over the next state. So that\u2019s kind of what the foundation is, is to help everybody describe your solutions. And the core issue with this is that there are no good solutions when you have a large number of states. All solutions kind of scale with the number of states, and so, if you have a small number of possible observations about the world, then you can employ these theoretically motivated reinforcement learning algorithms, which are <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/provably-efficient-rl-with-rich-observations-via-latent-state-decoding\/\">provably efficient<\/a>, and they will work well. But in the real world, you have a megapixel camera which has two to the one million or sixteen to the one million possible inputs. And so, you never encounter the same thing twice, and so you just can\u2019t even apply these algorithms. It doesn\u2019t even make sense. It\u2019s ridiculous. So, when I was a young graduate student, I was, of course, learning about Markov decision processes and trying to figure out how to solve reinforcement learning better with them. And then at some point, after we had a breakthrough, I realized that the breakthrough was meaningless, because it was all about these Markov decision processes. And no matter what, it just never was going to get to the point where you could actually do something useful. So around 2007, I decided to start working on contextual bandits. This is an expansion of what reinforcement learning means, in one sense, but a restriction in another sense. So instead of caring about the reward of a long sequence of actions, we\u2019re going to care about the reward of the next action. Right? So that\u2019s a big simplification. On the other hand, instead of caring about the state, we\u2019re going to care about an observation and we\u2019re going to demand that our algorithms don\u2019t depend on the number of possible observations, just like they do in supervised learning. So, we studied this for several years. We discovered how to create statistically efficient algorithms for these kinds of problems. So that\u2019s kind of the foundation of the systems that we\u2019ve been working on. And then, more recently, after cracking these contextual bandit problems, we wanted to address a larger piece of reinforcement learning. So now we\u2019re thinking about contextual decision processes where you have a sequence of rounds, and on each round, you see some observation, you choose some action, and then you do that again and again and again. And then at the end of an episode, maybe ten steps, maybe a hundred, you get a reward. Right? So, there\u2019s some long, delayed reward dependent upon all the actions you\u2019ve taken and all the observations you\u2019ve made. And now it turns out that when these observations are generated by some small, underlying state space which you do not know in advance and which is never told to you, you can still learn. You can still do reinforcement learning. You can efficiently discover what a good policy is, globally. So, the new foundation of reinforcement learning is about creating a foundation for reinforcement learning algorithms that can cope with a megapixel camera as an observation rather than having like ten discrete or a hundred discrete states.<\/p>\n<p><strong>Host: And you\u2019re getting some good traction with this approach?<\/strong><\/p>\n<p>John Langford: Yeah. I mean, contextual bandits are deployed in the real world and being used in many places at this point.<\/p>\n<p><strong>Host: Okay.<\/strong><\/p>\n<p>John Langford: There\u2019s every reason to believe that if we can crack contextual decision processes, which is our current agenda, that will be of great use as well.<\/p>\n<p><strong>Host: Rafah, at its core, reinforcement learning systems are designed to be self-improving systems, and kind of learn from the real world like humans do.<\/strong><\/p>\n<p>Rafah Hosn: Yes.<\/p>\n<p><strong>Host: Or puppies.<\/strong><\/p>\n<p>Rafah Hosn: Or puppies.<\/p>\n<p><strong>Host: And the real world is uncertain and risky.<\/strong><\/p>\n<p>Rafah Hosn: Yes.<\/p>\n<p><strong>Host: So how do you, from your perspective or from your angle, build trust with the customers that you interact with, both third-party and first-party customers who are giving you access to their own real-life traffic online?<\/strong><\/p>\n<p>Rafah Hosn: Yeah, this is an important topic when we start looking at how we do incubations in our team. And we have a specific challenge, as you were saying, because if we were a supervised learning model, we would go to a customer and say, hey, you know, give me a data set, I\u2019ll run my algorithm, if it improves, you deploy it, we deploy it in an A\/B test, and if we are good, you\u2019re good to go. Our system is deployed in production, so here we are with customers and talking to them about advanced machine learning techniques from research, and we want to deploy them in their online production system. So, as you can imagine, it becomes an interesting conversation. So, the way we approach this actually is by taking ideas from product teams. So, when we went and did our incubations, we did it with a hardened prototype, meaning this is a prototype that\u2019s not your typical stitched-up Python code, that, you know is hacky. We took a fair amount of time to harden it to the degree that if you run it in production, it\u2019s not going to crash your customers online productions system. So that\u2019s number one. And then when we approach customers, our system learns from the real world, and you do need a certain amount of traffic because our models are like newborn puppies. They don\u2019t know any tricks. So, you need to give them information in order to learn. But what we typically do is we have a conversation with our customer and say, hey, you know, yes, this is research, but it is hardened prototype. That\u2019s number one. And two, we use previous incubations as reference to newer ones. We borrow ideas from how products go sell their prototypes, right? And then we, as a methodology, say to customers, when they have large volumes of traffic, to give us a portion of their traffic which is good enough for us to learn and prove the ROI, but small enough for them to de-risk. And that methodology has worked very well for us.<\/p>\n<p><strong>Host: De-risk is such a good word. Let\u2019s go a little further on that thread. Talk a little bit about the cold start versus the warm start when you\u2019re deploying.<\/strong><\/p>\n<p>Rafah Hosn: Yes, so that\u2019s another interesting conversation with our customers, especially those that are used to supervised learning where you train your model, right, with a lot of data, and you deploy it, and it\u2019s already learned something. Our models and our personalization service start really cold, but the way John and the teams created those algorithms allows us to learn very fast. And the more traffic you give it, the faster it learns. So, I\u2019ll give you an example. We deployed a pilot with Xbox Top of Home where we were personalizing two of the three slots or four slots that they have on the Top of Home. And Xbox gets, you know, millions of events per day. So, with only 6 million events per day, which is a fraction of Xbox traffic, in about a couple of hours, we went from cold to very warm. So again, from a de-risking, with these conversations with our customers, first or third parties, we tend to say, yes, it\u2019s cold start. But these algorithms learn super-fast, and there\u2019s a certain amount of traffic flow that enables that efficient learning. So, we haven\u2019t had major problems. We start by making our customers understand how the system works, and we go from there.<\/p>\n<p><strong>Host: Are there instances where you\u2019re coming into a warm start where there\u2019s some existing data or infrastructure?<\/strong><\/p>\n<p>John Langford: Yeah, so that definitely happens. It\u2019s typically more trouble than it\u2019s worth to actually use pre-existing data, because, when you\u2019re training in a contextual bandit, you really need to capture four things: the features, the action, the reward for the action, and then the probability of taking the action. And almost always, the probability is not recorded in any kind of reliable way if it was even randomized previously. So, given that you lack one of those things, you can\u2026 there are ways to try to repair that. They kind of work, but they\u2019re kind of a pain, and not the kind of thing that you can do in an automatic fashion. So typically, we want to start with recording our own data so we can be sure that it is, in fact, good data. Now, with that said, there are many techniques for taking into account pre-existing models, right? So, we actually have a paper now in archive talking about how to combine an existing supervised data source with a contextual bandit data source. Another approach, which is commonly very helpful, is people may have an existing supervised system which may be very complex, and they may have built up a lot of features around that\u2026<\/p>\n<p><strong>Host: Right.<\/strong><\/p>\n<p>John Langford: \u2026which may not even be appropriate. Often, there\u2019s a process around any kind of real system where the learning algorithm and the features are kind of co-evolving, and so moving away from either of them causes a degradation in performance.<\/p>\n<p><strong>Host: Sure.<\/strong><\/p>\n<p>John Langford: So, in that kind of situation, what you want to do is, you want to tap the existing supervised models to extract features which are very powerful. And then, given those very powerful features, you can very quickly, get to a good solution. And then so the exact mechanism of that extraction is going to depend upon the representation that you\u2019re using. With a neural network, you kind of rip off the top layer and use that. With a decision tree or a boosted decision tree or a decision forest, you can use the leaf membership as a feature that you can then feed in for a very fast warm-up of a contextual bandit learner.<\/p>\n<p><strong>Host: John, talk about offline experimentations. What\u2019s going on there?<\/strong><\/p>\n<p>John Langford: Yeah, so this is one of the really cool things that\u2019s possible when you\u2019re doing shallow kinds of reinforcement learning, reinforcement learning with maybe one step or maybe two steps. So, if you record that quad of features, action, reward and the probability, then it becomes possible to evaluate any policy that chooses amongst the set of available actions. Okay? So, what that means is that, if you record this data, and then later you discover that maybe a different learning rate was helpful, or maybe you should be taking this feature and that feature and combining them to make a new feature. You can test to see exactly how that would have performed if you had deployed that policy at the time you were collecting data. So, this is amazing, because this means that you no longer need to use an A\/B test for the purpose of optimization. You still have reasons to use it for purposes of safety, but for optimization, you can do that offline in a minute rather than doing it online for two weeks waiting to get the data necessary to actually learn.<\/p>\n<p>Rafah Hosn: Yeah, just to pick up on, why is this a gold nugget? Data scientists spend a fair amount of time today designing models a priori and testing them in A\/B tests only to learn two weeks after that they failed, and they go back to ground zero. So here you\u2019re running hundreds, if not thousands, of A\/B tests on this pod. And when we talk about this to data scientists and enterprises, their eyes light up. I mean, that is one of the key features of our system that just brightens the day for many data scientists. It\u2019s a real pain for them to design models, run them in A\/B\u2026 it\u2019s very costly as well.<\/p>\n<p><strong>Host: Mm-hmm.<\/strong><\/p>\n<p>Rafah Hosn: So, talk about productivity gains. It\u2019s immense when you can run a hundred to two hundred A\/B tests in a minute versus running one A\/B test for two weeks.<\/p>\n<p><strong>Host: Rafah, you work as a program manager within a research organization.<\/strong><\/p>\n<p>Rafah Hosn: Yes.<\/p>\n<p><strong>Host: And it\u2019s your job to bring science to the people.<\/strong><\/p>\n<p>Rafah Hosn: Yes.<\/p>\n<p><strong>Host: Talk about your process, a little more in detail, of \u201cresearch, incubate, transfer\u201d in the context of how you develop RL prototypes and engineer them and how you test them, and specifically maybe you could explain a couple of examples of this process of deployments that are out there already. How are you living up to the code?<\/strong><\/p>\n<p>Rafah Hosn: We have a decent size engineering team that supports our RL efforts in MSR. And our job is twofold. One is to, from a program management perspective, it\u2019s to really drive what it means to go from an algorithm to a prototype, and then validate whether that prototype has any market potential. I take it upon me, as a program manager, when researchers are creating these wonderful academic papers with great algorithms, and some of them may have huge market potential. So, this market analysis happens actually in MSR. And we ask ourselves, great algorithm; what are the classes of problems we can solve for it? And would people like relate to these problems such that we could actually go and incubate them? And the incubation is a validation of this market hypothesis. So that\u2019s what we do in our incubations. We are actually trying to see whether this is something that we could potentially tech transfer to the product team. And we\u2019ve done this with contextual bandits in the context of personalization scenarios. So contextual bandits is a technique, right? And so, we ask ourselves, okay, with this technique, what classes of problems can we solve very efficiently? And personalization was one of them. And we went and incubated it first with MSN. Actually, John and the team incubated it with MSN first, and they got a twenty six percent lift. That\u2019s multi-million-dollar revenue potential. So, from a market potential, it really made sense. So, we went and said, okay, one customer is not statistically significant, so we need to do more. And we spent a fair amount of time actually validating this idea and validating the different types of personalization. So, MSN was a news article personalization. Recently, we did a page layout personalization with surface.com Japan where they had four boxes on surface.com Japan, and they were wondering how to present these boxes based on the user that was visiting that page. And guess what? We gave them two thousand five hundred events, so it was a short-run pilot that we did with them. We gave them an eighty percent lift. EIGHTY. They were flabbergasted. They couldn\u2019t believe \u2013 and this was run on an A\/B test. So they had their page layout that their designers had specified for them, for all users, running as the control, and they had our personalization engine running with our contextual bandit algorithm, and they ran it, and for us, you know, twenty-five hundred samples is not really a lot. But even with that, we gave them an eighty percent lift over their control. So, these are the kinds of incubation that, when we go to our sister product team in Redmond and tell the story, they get super excited that this could be a classes of application that could work for the masses.<\/p>\n<p>(music plays)<\/p>\n<p><strong>Host: John, there\u2019s a lot of talk today about diversity, and that often means having different people on the team, but there\u2019s other aspects, especially in reinforcement learning that include diversity of perspective and approach. How do you address this in the work you\u2019re doing and how do you practically manage it?<\/strong><\/p>\n<p>John Langford: One thing to understand is that research is an extreme sport in many ways. You\u2019re trying to do something which nobody has ever done before. And so, you need an environment that supports you in doing this in many ways. It\u2019s hard for a single researcher to have all the abilities that are needed to succeed. When you\u2019re learning to do research, you\u2019re typically learning a very narrow thing. And over time, maybe that gets a little bit broader, but it\u2019s still going to be the case that you just know a very narrow perspective on how to solve a problem. So, one of the things that we actually do is, on a weekly basis, we have an open problems discussion where a group of researchers gets together, and one of them talks about the problem that they\u2019re interested in. And then other people can chime in and say, oh, maybe we should look at it this way or think about it that way. That helps, I think, sharpen the problems. And then, in the process of solving problems, amazing things come up in discussion, but they can only come up if you can listen to each other. I guess the people that I prefer to work with are the ones who listen carefully. There\u2019s a process of bouncing ideas off each other, discovering the flaws in them, figuring out how to get around the flaws. This process can go on. It\u2019s indefinite.<\/p>\n<p><strong>Host: Yeah.<\/strong><\/p>\n<p>John Langford: But sometimes it lands. And when it lands, that moment when you discover something, that\u2019s really something!<\/p>\n<p><strong>Host: Rafah, do you have anything to add to that?<\/strong><\/p>\n<p>Rafah Hosn: So, when I think about diversity in our lab, I think that, to compliment what John\u2019s saying, I like to always also think about the diversity of disciplines. So, in our lab, we\u2019re not a big lab, but we have researchers, we have engineers, we have designers, and we have program managers. And I think these skillsets are diverse, and yet they complement each other so well. And I think that adds to the richness of what we have in our lab.<\/p>\n<p><strong>Host: In the context of the work you do, its applications and implications in the real world, is there anything that keeps you up at night? Any concerns you\u2019re working to mitigate even as you work to innovate?<\/strong><\/p>\n<p>John Langford: I think the answer is, yes. Uhhh\u2026 anybody who understands the potential of machine learning, AI, whatever you want to call it, understands that there are negative ways to use it\u2026<\/p>\n<p>Rafah Hosn: Right.<\/p>\n<p>John Langford: &#8230;right? It is a tool and\u2026<\/p>\n<p><strong>Host: Yeah.<\/strong><\/p>\n<p>John Langford: \u2026we need to try to use the tool responsibly, and we need to mitigate the downsides where we can see them in advance. So, I do wonder about this. But recently we had a paper on fair machine learning\u2026<\/p>\n<p><strong>Host: Mm hmm.<\/strong><\/p>\n<p>John Langford: \u2026and we showed that any supervised learning algorithm can, in a black box fashion, be turned into a fair supervised learning algorithm. We demonstrated this both theoretically and experimentally. So that\u2019s a promising paper that addresses a narrow piece of ethics around AI, I guess I would say.<\/p>\n<p><strong>Host: Yeah.<\/strong><\/p>\n<p>John Langford: As we see more opportunities along these lines, we will solve them.<\/p>\n<p>Rafah Hosn: Yeah, also use these techniques for the social good, right? I mean, as we are trying to use them to monetize, also we should use them for the social good.<\/p>\n<p><strong>Host: How did each of you end up at Microsoft Research in New York City?<\/strong><\/p>\n<p>John Langford: This is actually quite a story. So, I used to be at Yahoo Research. One day, right about now, seven years ago, the head of Yahoo Research quit. So, we decided to essentially sell the New York lab. So, we created a portfolio of everybody in the New York lab. There were fifteen researchers there. We sent it around to various companies. Microsoft ended up getting thirteen out of fifteen people! And that was the beginning of Microsoft Research New York.<\/p>\n<p><strong>Host: Rafah, how did you come to Microsoft Research New York City?<\/strong><\/p>\n<p>Rafah Hosn: He told me I was going to revolutionize the world. That\u2019s why I came over from IBM! So, I actually had a wonderful job at IBM, applying Watson Technologies for children\u2019s education. And one day, a Microsoft recruiter called me, and they said, \u201cJohn Langford, renowned RL researcher, is looking for a program manager. You should interview with him!\u201d And I\u2019m like, okay! So, I interviewed at Microsoft Research New York, spoke to many people, and at the time I, you know, I was comfortable in my job and I had other opportunities but, in his selling pitch to me, John Langford calls me one day at home and he says, \u201cYou should choose to come and work for Microsoft Research because we\u2019re going to revolutionize the world.\u201d And I think it sunk in that we can be at the cusp of something really big, and that got me really excited to join, and that\u2019s how I ended up at Microsoft Research.<\/p>\n<p><strong>Host: As we close, I\u2019d like each of you to address a big statement that you\u2019ve made. John, you started out our interview with \u201cI want to solve machine learning.\u201d Rafah, you have said that your ultimate goes is \u201creal-world reinforcement learning for everyone.\u201d What does the world look like if each of you is wildly successful?<\/strong><\/p>\n<p>John Langford: Yeah, so there\u2019s a lot of things that are easy to imagine being a part of the future world that just aren\u2019t around now. You should imagine that every computer interface learns to adapt to you, rather than you needing to adapt to the user interface. You could imagine lots of companies just working better. You could imagine a digital avatar that, over time, learns to help you book the flights that you want to book or things like that, right? Often there\u2019s a lot of mundane tasks that people do over and over again, and if you have a system that can record and learn from all the interactions that you make with computers or with the internet, it can happen on your behalf. That could really ease the lives of people in many different ways. Lots of things where there\u2019s an immediate sense of, oh, that was the right outcome or, oh, that was the wrong outcome, can be addressed with just the technology that we have already. And then there\u2019s technologies beyond that, like the contextual decision processes that I was talking about that may open up even more possibilities in the future.<\/p>\n<p><strong>Host: Rafah.<\/strong><\/p>\n<p>Rafah Hosn: To me, what a bright future would look like is when we can cast a lot of issues that we see today, problems enterprises and, at the personal level as a reinforcement learning problem, that we can actually solve. And more importantly for me, you know, as we work in technology and we develop all these techniques, the question is, are we making the world a better world, right? And can we actually solve some hard problems like famine and diseases with reinforcement learning? And maybe not now, but can it be the bright future that we look out for? I hope so.<\/p>\n<p><strong>Host: I do too. John Langford, Rafah Hosn, thank you for joining us today.<\/strong><\/p>\n<p>Rafah Hosn: Thank you.<\/p>\n<p>John Langford: Thank you.<\/p>\n<p>(music plays)<\/p>\n<p><strong>Host: To learn more about Dr. John Langford and Rafah Hosn and the quest to bring reinforcement learning to the real world, visit <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/\">Microsoft.com\/research<\/a><\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Episode 75, May 8, 2019- Dr. John Langford, a partner researcher in the Machine Learning group at Microsoft Research New York City, is a reinforcement learning expert who is working, in his own words, to solve machine learning. Rafah Hosn, also of MSR New York, is a principal program manager who\u2019s working to take that work to the world. If that sounds like big thinking in the Big Apple, well, New York City has always been a \u201cgo big, or go home\u201d kind of town, and MSR NYC is a \u201cgo big, or go home\u201d kind of lab.<\/p>\n<p>Today, Dr. Langford explains why online reinforcement learning is critical to solving machine learning and how moving from the current foundation of a Markov decision process toward a contextual bandit future might be part of the solution. Rafah Hosn talks about why it\u2019s important, from a business perspective, to move RL agents out of simulated environments and into the open world, and gives us an under-the-hood look at the product side of MSR\u2019s \u201cresearch, incubate, transfer\u201d process, focusing on real world reinforcement learning which, at Microsoft, is now called Azure Cognitive Services Personalizer.<\/p>\n","protected":false},"author":38022,"featured_media":584869,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"https:\/\/player.blubrry.com\/id\/43783585\/","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"footnotes":""},"categories":[240054],"tags":[],"research-area":[13556],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-584854","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-msr-podcast","msr-research-area-artificial-intelligence","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"https:\/\/player.blubrry.com\/id\/43783585\/","podcast_episode":"","msr_research_lab":[199571],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[144902,395930],"related-projects":[669597],"related-events":[],"related-researchers":[],"msr_type":"Post","featured_image_thumbnail":"<img width=\"960\" height=\"540\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/05\/John-Langford-And-Rafah-Hosn_Podcast_Site_04_2019_1400x788.png\" class=\"img-object-cover\" alt=\"John Langford and Rafah Hosn\" decoding=\"async\" loading=\"lazy\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/05\/John-Langford-And-Rafah-Hosn_Podcast_Site_04_2019_1400x788.png 1400w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/05\/John-Langford-And-Rafah-Hosn_Podcast_Site_04_2019_1400x788-300x169.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/05\/John-Langford-And-Rafah-Hosn_Podcast_Site_04_2019_1400x788-768x432.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/05\/John-Langford-And-Rafah-Hosn_Podcast_Site_04_2019_1400x788-1024x576.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/05\/John-Langford-And-Rafah-Hosn_Podcast_Site_04_2019_1400x788-1066x600.png 1066w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/05\/John-Langford-And-Rafah-Hosn_Podcast_Site_04_2019_1400x788-655x368.png 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/05\/John-Langford-And-Rafah-Hosn_Podcast_Site_04_2019_1400x788-343x193.png 343w\" sizes=\"(max-width: 960px) 100vw, 960px\" \/>","byline":"","formattedDate":"May 8, 2019","formattedExcerpt":"Episode 75, May 8, 2019- Dr. John Langford, a partner researcher in the Machine Learning group at Microsoft Research New York City, is a reinforcement learning expert who is working, in his own words, to solve machine learning. Rafah Hosn, also of MSR New York,&hellip;","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/584854"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/38022"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=584854"}],"version-history":[{"count":11,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/584854\/revisions"}],"predecessor-version":[{"id":593539,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/584854\/revisions\/593539"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/584869"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=584854"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=584854"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=584854"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=584854"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=584854"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=584854"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=584854"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=584854"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=584854"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=584854"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=584854"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}