{"id":607842,"date":"2019-09-11T08:05:35","date_gmt":"2019-09-11T15:05:35","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=607842"},"modified":"2022-11-07T11:45:03","modified_gmt":"2022-11-07T19:45:03","slug":"inside-the-microsoft-ai-residency-program-with-dr-brian-broll","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/podcast\/inside-the-microsoft-ai-residency-program-with-dr-brian-broll\/","title":{"rendered":"Inside the Microsoft AI Residency Program with Dr. Brian Broll"},"content":{"rendered":"<h3><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2019\/09\/Brian-Broll-_Podcast__Site_11_2019_1400x788.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-607848 size-large\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2019\/09\/Brian-Broll-_Podcast__Site_11_2019_1400x788-1024x576.png\" alt=\"Dr. Brian Broll\" width=\"1024\" height=\"576\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2019\/09\/Brian-Broll-_Podcast__Site_11_2019_1400x788-1024x576.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2019\/09\/Brian-Broll-_Podcast__Site_11_2019_1400x788-300x169.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2019\/09\/Brian-Broll-_Podcast__Site_11_2019_1400x788-768x432.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2019\/09\/Brian-Broll-_Podcast__Site_11_2019_1400x788-1066x600.png 1066w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2019\/09\/Brian-Broll-_Podcast__Site_11_2019_1400x788-655x368.png 655w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2019\/09\/Brian-Broll-_Podcast__Site_11_2019_1400x788-343x193.png 343w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2019\/09\/Brian-Broll-_Podcast__Site_11_2019_1400x788-640x360.png 640w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2019\/09\/Brian-Broll-_Podcast__Site_11_2019_1400x788-960x540.png 960w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2019\/09\/Brian-Broll-_Podcast__Site_11_2019_1400x788-1280x720.png 1280w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2019\/09\/Brian-Broll-_Podcast__Site_11_2019_1400x788.png 1400w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/a><\/h3>\n<h3>Episode 89, September 11, 2019<\/h3>\n<p>In 2018, Microsoft launched the <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/academic-program\/microsoft-ai-residency-program\/\">Microsoft AI Residency Program<\/a>, a year-long, expanded research experience designed to give recent graduates in a variety of fields the opportunity to work alongside prominent researchers at MSR on cutting edge AI technologies to solve real-world problems. <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/github.com\/brollb\">Dr. Brian Broll<\/a> was one of them. A newly minted PhD in Computer Science from Vanderbilt University, Dr. Broll was among the inaugural cohort of AI residents who spent a year working on machine learning in game environments and is on the pod to talk about it!<\/p>\n<p>Today, Dr. Broll gives us an overview of the work he did and the experience he had as a Microsoft AI Resident, talks about his passion for making complex concepts easier and more accessible to novices and young learners, and tells us how growing up on a dairy farm in rural Minnesota helped prepare him for a life in computer science solving some of the toughest problems in AI.<\/p>\n<h3>Related:<\/h3>\n<ul type=\"disc\">\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/podcast\">Microsoft Research Podcast<\/a>: View more podcasts on Microsoft.com<\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/itunes.apple.com\/us\/podcast\/microsoft-research-a-podcast\/id1318021537?mt=2\">iTunes<\/a>: Subscribe and listen to new podcasts each week on iTunes<\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/subscribebyemail.com\/www.blubrry.com\/feeds\/microsoftresearch.xml\">Email<\/a>: Subscribe and listen by email<\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/subscribeonandroid.com\/www.blubrry.com\/feeds\/microsoftresearch.xml\">Android<\/a>: Subscribe and listen on Android<\/li>\n<li class=\"x-hidden-focus\"><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/open.spotify.com\/show\/4ndjUXyL0hH1FXHgwIiTWU\">Spotify<\/a>: Listen on Spotify<\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/www.blubrry.com\/feeds\/microsoftresearch.xml\">RSS feed<\/a><\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/note.microsoft.com\/ww-registration-microsoft-research-newsletter-s.html?wt.mc_id=S-webpage_podcast\">Microsoft Research Newsletter<\/a>: Sign up to receive the latest news from Microsoft Research<\/li>\n<\/ul>\n<hr>\n<h3>Transcript<\/h3>\n<p>Brian Broll: My first project was focused on actually trying to get more human-like behavior in Minecraft, which was leveraging a scripted agent and very few human demonstrations. So, given like thirty-three human demonstrations and an existing scripted agent, the question was, how can we incorporate some sort of learning into the agent, not to necessarily make it perform better but to make it more engaging and interesting, and hopefully more human-like?<\/p>\n<p><strong>Host: You\u2019re listening to the Microsoft Research Podcast, a show that brings you closer to the cutting-edge of technology research and the scientists behind it. I\u2019m your host, Gretchen Huizinga.<\/strong><\/p>\n<p><strong>Host: In 2018, Microsoft launched the Microsoft AI Residency Program, a year-long, expanded research experience designed to give recent graduates in a variety of fields the opportunity to work alongside prominent researchers at MSR on cutting edge AI technologies to solve real-world problems. Dr. Brian Broll was one of them. A newly minted PhD in Computer Science from Vanderbilt University, Dr. Broll was among the inaugural cohort of AI residents who spent a year working on machine learning in game environments and is on the pod to talk about it!<\/strong><\/p>\n<p><strong>Today, Dr. Broll gives us an overview of the work he did and the experience he had as a Microsoft AI resident, talks about his passion for making complex concepts easier and more accessible to novices and young learners, and tells us how growing up on a dairy farm in rural Minnesota helped prepare him for a life in computer science solving some of the toughest problems in AI. That and much more on this episode of the Microsoft Research Podcast.<\/strong><\/p>\n<p>(music plays)<\/p>\n<p><strong>Host: Brian Broll, welcome to the podcast!<\/strong><\/p>\n<p>Brian Broll: Thanks, happy to be here.<\/p>\n<p><strong>Host: So, this one\u2019s going to be different. You\u2019re not actually in Microsoft Research proper, you\u2019re part of a really interesting new program. It\u2019s called Microsoft\u2019s AI Residency Program and you\u2019re part of the very first cohort, right? So, 2018 it started the first year. You\u2019re just finishing up?<\/strong><\/p>\n<p>Brian Broll: Yep.<\/p>\n<p><strong>Host: Before we begin our \u201cregularly scheduled programming,\u201d give us a bit of an overview about the residency program. Tell us what it is, what\u2019s cool about it, and actually, why you did it.<\/strong><\/p>\n<p>Brian Broll: Sure, so the residency program is a year-long, fixed-term position where you come in and you work on two different six-month projects and the applicants come from a variety of different backgrounds. A lot of them with a computer science background, but some even from like quantum physics and some people with more of an NLP background, some people with ethics and psychology and cognitive psych. So, there\u2019s a variety of different backgrounds and a variety of different degree levels and people come in and get to work on all sorts of different projects for, again, two six-month periods.<\/p>\n<p><strong>Host: Right. It\u2019s its first instantiation of a residency, when I hear that I think of medical residencies, right, but you\u2019re a doctor, but a different kind!<\/strong><\/p>\n<p>Brian Broll: Yeah, yeah, I don\u2019t\u2019 usually operate, or at least I try not to.<\/p>\n<p><strong>Host: Only on data.<\/strong><\/p>\n<p>Brian Broll: Yes, exactly, exactly. I mean there are some things that make a lot of sense about doing a residency. I would suspect that one of the benefits is, certainly, that if they have people here working on research projects with a lot of people at MSR and making a lot of these really great connections, and then they move to, like, some other product team or stick around at Microsoft or even if they returned back to like academia to, uh, in some cases pursue a PhD, in other cases come on directly as faculty, it certainly facilitates a lot of these relationships and collaborations either within, again, product teams and MSR, or potentially within other academic institutions and MSR.<\/p>\n<p><strong>Host: Well at the beginning of every podcast I ask my guests, \u201cWhat gets you up in the morning?\u201d You\u2019re more the beginning of your career than most of the people that I have in the booth, so I\u2019m going to tweak the question a bit and ask you what\u2019s on your research roadmap. What kinds of questions excite you and, if you could encapsulate it succinctly, what\u2019s the big goal of your research?<\/strong><\/p>\n<p>Brian Broll: Yeah, that\u2019s a good question. In the past I\u2019ve done things with like educational and visual programming environments and trying to make more advanced and complex topics more easily accessible by younger users, or young learners in some cases. And I\u2019ve also done some work in, like, related spaces around, like, developer tools and again just trying to see how you can take advanced concepts and make them something that people can, not just, like, interact with at a high level, but hopefully have a deep and rich engagement in. So that they can really leverage it for whatever sort of task or problem they really care about, you know\u2026<\/p>\n<p><strong>Host: Yeah.<\/strong><\/p>\n<p>Brian Broll: Ummm! So, that\u2019s definitely a recurring theme. And I have found that, when I work on different projects, a lot of times I end up coming back to this question. Like, even if I\u2019m not trying to target some, like, end user or developer or make something accessible, I always find myself wondering, first of all, how could I solve whatever problem I\u2019m working on, and then one step back, like, how could I get to that information that I need to be able to figure out or solve whatever problem I\u2019m working on, and then, like, how could I do this easier? So, then I often find myself falling into the same train of thought, even when I\u2019m the only person doing it, of like, I really want to be confident that things are working the way that I think they are, or\u2026 The questions come in a variety of different forms, depending upon specifically what I\u2019m working on. But it can vary from, how can I really better understand this problem so I can figure out a solution, to, like, how can I really be sure that what I think is happening is actually happening? And when I start going to those types of questions, I always find myself naturally wondering, like, is there a better way that I can, like, fundamentally approach this so that it\u2019s an easier task to solve or, or to be confident in my solution?<\/p>\n<p><strong>Host: We\u2019ll get to the laziness business later because I think that\u2019s an underrated quality for people\u2026 Well, let\u2019s talk for a minute about the graduate work you did since you are a, what I would call, \u201cnewly minted\u201d PhD and your work is really fresh. So, give us an overview of the work you did to get your degree and tell us a little of how that\u2019s informed what you\u2019re doing now.<\/strong><\/p>\n<p>Brian Broll: Sure. So, I finished up my PhD a year ago at Vanderbilt and I was working on making distributed computing concepts more accessible to kids. So, basically, we wanted to extend programming environments that are meant to have a really low, like, threshold so that kids can get started programming really easily. So, like, more specifically, I was focused on, like, block space programming environments. So, you might be familiar with like Scratch or Snap. Now, my work was trying to empower the students using these environments with the abilities to build distributed applications. So, this would include potentially fun and, hopefully, more engaging things like chat applications or multiplayer games, or, like, it\u2019s really the sky\u2019s the limit. Hopefully you can give them the tools where they can really express their creativity and learn something cool in the process. So, I think it\u2019s cool when you can appeal to other demographics by making it much more grounded, so it feels applicable and relevant\u2026<\/p>\n<p><strong>Host: Yeah.<\/strong><\/p>\n<p>Brian Broll: \u2026while also trying to make it more social by giving them the power to build network-enabled applications so they can actually make these games multiplayer or make chat applications, and then start to reason about the natural things that often come up. I\u2019ve been in classes before where I\u2019ve introduced things like client-server architecture. One that comes to mind is, I was working with a group of eighth graders and I introduced client-server architecture as we made a chat application during class. And it\u2019s funny how concepts come up super naturally that are really huge. Like, they were doing a distributed denial of service, unintentionally, immediately.<\/p>\n<p><strong>Host: What?<\/strong><\/p>\n<p>Brian Broll: Like, I mean it makes sense, you have a classroom of students and I set up a server and they can all see things and naturally you get some trolls in the audience who just want\u2026 who find out that, well, they can use this loop that\u2019s just an infinite loop and just keep sending messages to the server and then we have to step back and talk about, how do we detect spam and how do we start filtering this stuff out? But the thing that I want to emphasize about this example is not necessarily that these kids were learning to DDoS but that these concepts come up organically\u2026<\/p>\n<p><strong>Host: Right.<\/strong><\/p>\n<p>Brian Broll: \u2026in a way that like they can get their hands on it and then start reasoning about how to solve this problem and different techniques and then hopefully then evaluating the different types of solutions that they might come up with.<\/p>\n<p><strong>Host: So, did they even know that what they were doing had a name, an acronym?<\/strong><\/p>\n<p>Brian Broll: No, no they didn\u2019t!<\/p>\n<p><strong>Host: Did you tell them?<\/strong><\/p>\n<p>Brian Broll: Yes, I did, yeah.<\/p>\n<p><strong>Host: You just did a DDoS.<\/strong><\/p>\n<p>Brian Broll: Yeah, it\u2019s funny you have to be pretty flexible when you\u2019re doing things like this because the class can go a lot of different ways.<\/p>\n<p><strong>Host: Oh sure, I\u2019ve taught eighth graders, I know, that\u2019s too funny. Well, how did that work play out and do you find it has legs and you\u2019re building on it?<\/strong><\/p>\n<p>Brian Broll: Yeah. So that work is a continuing project at Vanderbilt. There are a lot of a different projects and efforts that are building on top of it. For example, a year ago, we had two one-week long summer camps that were sponsored by NSA, that were focusing on cybersecurity in the context of cyber-physical systems. So, more concretely, they were given robots and then we started introducing them to, like, programming, because we couldn\u2019t assume prior programming experience\u2026<\/p>\n<p><strong>Host: Right.<\/strong><\/p>\n<p>Brian Broll: \u2026so, we first tried to do some basic some introduction to programming, and then I started introducing them to programming on, like, robots and then, after that, started getting into cybersecurity questions. So essentially, grounding this a little bit more, we had a classroom of students and we had, like, a collection of robots and rather than a lot of the existing robotic programming platforms, we didn\u2019t, like, have a single computer have to be like physically connected or associated with this specific robot, it was more like we had a sandbox of students who could interact with a group of robots and they were all assigned a specific one, but this means that kids will start trolling each other and trying to control each other\u2019s robots and things like that. And I think it\u2019s great because this means that, in the way that we design and set up the environment, we can do it in a way that facilitates the, I guess, natural development of some of these sorts of concepts.<\/p>\n<p><strong>Host: Right.<\/strong><\/p>\n<p>Brian Broll: So, like, encryption comes up pretty naturally then if we\u2019re,like, well, people keep trying to control my robot and I want it to drive but\u2026<\/p>\n<p><strong>Host: How are you going to stop that?<\/strong><\/p>\n<p>Brian Broll: Exactly. And then we start talking about codebreaking and this adversarial nature of cybersecurity lends itself very nicely to a curriculum too, in the sense that you can introduce one sort of very natural initial fix and then how you can counter that and then how you counter the counter and you know. And you can continue developing along that route when you have students who are changing the encryption on the robot really frequently and trying to prevent replay attacks and all sorts of fun topics.<\/p>\n<p>(music plays)<\/p>\n<p><strong>Host: Well, we talked a little bit about the residency program in general, but I want to talk about what you did while you were here, let\u2019s plunge right into your year at the Microsoft AI Residency Program.<\/strong><\/p>\n<p>Brian Broll: Yep, so I worked with the <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/group\/reinforcement-learning-group\/\">reinforcement learning team<\/a>, and I was working on machine learning in games. So, rather than some of the machine learning where they\u2019re trying to essentially replace a human with an AI, we were focused more on how can we leverage machine learning to be, like, complementary to the game development process? So, not necessarily just trying to use them as a sandbox to show that we can learn complex tasks but say, like, how can we actually benefit games from incorporating machine learning either into, like, the development process or some sort of aspect of the game that complements it later? So, I was working mostly with a couple of people on the RL team here in Redmond but also collaborated a bit with a bunch of researchers out in <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/lab\/microsoft-research-cambridge\/\">Cambridge<\/a>, who are also working on this\u2026<\/p>\n<p><strong>Host: Cambridge, UK\u2026<\/strong><\/p>\n<p>Brian Broll: Yeah, Cambridge, UK, yes, who were working on a similar effort. So, my first project was focused on actually trying to get more human-like behavior in Minecraft, which was leveraging a scripted agent and very few human demonstrations. So, given, like, thirty-three human demonstrations and an existing scripted agent, the question was, how can we incorporate some sort of learning into the agent, not to necessarily make it perform better but to make it more engaging and interesting, and hopefully more human-like?<\/p>\n<p><strong>Host: Interesting, all right. How did that go?<\/strong><\/p>\n<p>Brian Broll: We did find some pretty positive results, and we were able to create a hybrid agent that did demonstrate more of the behaviors that we thought were interesting and engaging from the human perspective. So, like, contextual decision-making as well as, you saw similar high-level strategies exhibited by this hybrid agent where it was able to learn the strategies just from the human demonstrations. So those aspects of it were very positive. So there is a lot of flexibility in taking some of these data-driven approaches and we were hoping that we could be able to, when defining a new agent in this game, use scripting for the parts that make sense to code for and then use data for the parts that make sense to use demonstrations for.<\/p>\n<p><strong>Host: Right.<\/strong><\/p>\n<p>Brian Broll: There are other aspects, or ways to view it, that are less practical and more trying to think about some of the fundamental challenges, and some of the fundamental challenges in this context would be that the scripted agent gave us a hierarchical policy where we could replace, in our context, a meta-controller, so basically the part of the agent that was picking the high-level, like, goals of the agent in this game. And you can assume that humans have a similar type of implicit hierarchical policy. In other words, they\u2019re not just thinking, like, whether or not they should move forward and left, like, without any concern for a high-level plan\u2026<\/p>\n<p><strong>Host: Right.<\/strong><\/p>\n<p>Brian Broll: \u2026like they\u2019re thinking, like, ah I should go and grab this thing or interact with\u2026<\/p>\n<p><strong>Host: More strategically.<\/strong><\/p>\n<p>Brian Broll: Exactly, there\u2019s some sort of high-level strategy. Now, one of the challenges in this case was that we were just trying to learn the meta-controller via imitation from the human demonstrations, and the lower-level policy of the scripted agent was, well, of course, scripted. So that meant that that wasn\u2019t able to learn at all. So now the challenging part is that we know that the lower-level policy, things like pathfinding and such, might not match with what the humans are doing, but we still want to be able to imitate, which makes it challenging because you don\u2019t want to try to imitate at a lower level then, like you can\u2019t just look at, like, different states of the game and just try to compare them between, like, the hybrid agent, which you\u2019re trying to, again, train the meta-controller, and the human demonstrations because, given the same goal, they might have different ways of achieving that goal. So really, you want to try to, essentially, predict the human\u2019s intention and then try to incorporate the human\u2019s intention into the meta-controller. So it tries to predict what a human would try to do, given the state that it\u2019s seen so far, and then reuse the scripted part to actually, like, achieve this intention or this sub-goal that it thinks that the human would have. So, it\u2019s a tricky problem\u2026<\/p>\n<p><strong>Host: Just a little!<\/strong><\/p>\n<p>Brian Broll: Yes.<\/p>\n<p><strong>Host: Well, what kinds of research thinking does that inspire? I mean you\u2019ve got to, then, figure out, okay, how I am going to do this? How are we going to take what\u2019s natural up here and put it in the software?<\/strong><\/p>\n<p>Brian Broll: So one of the biggest questions, if we\u2019re thinking about trying to, like, approach or tackle this question, is a bit of like, how can you actually find some sort of signal in the data that we can actually use to train the meta-controller? And in this case, it\u2019s not just some signal, I guess it\u2019s really the right signal. So, I gave the example already of, if you just tried to look at states, we could try to imagine just trying all sorts of different possibilities in the meta-controller and seeing which one gets closest to what the human was doing. But the problem there is that you start to, like, encapsulate, I guess, the things like pathfinding and stuff, which will just introduce noise, which isn\u2019t really what you\u2019re trying to measure, because you could imagine that \u2013 now this is usually easier to explain with a whiteboard! \u2013 but you could imagine that you see some human taking some arc to whatever its goal is. And there\u2019s all sorts of different, like, entities or sub-goals that are possible on the screen, and you could imagine that you have some sort of scripted agent that just goes directly to its sub-goal. Now, if you knew what the person\u2019s intention was, the expected behavior would be an arc in the human demonstration and a direct path in the resulting hybrid agent. But if you\u2019re not careful, and you\u2019re only looking at the low-level states, you would try to force the hybrid agent to take an arc. And you might be able to do this by picking all sorts of garbage sub-goals, really quickly, to make it move directly\u2026 I don\u2019t know, a little bit to the right, and a little bit forward, then a little bit back to the left, and kind of create this arc shape, but that\u2019s really not at all what you want. And it will be especially problematic when we start looking at different humans. It\u2019s not that all humans share the same pathfinding logic or low-level sub-policies, so it will get even more complicated and difficult to try to learn. So the type of thinking for this project really required you to kind of step back and try to really understand, fundamentally, what sort of approaches could be used to try to learn, really, what we care about here, which isn\u2019t immediately obvious, how to isolate the aspects of the human demonstrations that we care about, in the context of this hybrid agent. And those parts got a little hairy!<\/p>\n<p><strong>Host: So, another thing you did during the residency was extend the results of the work that you did at the outset. So, what questions were left unanswered, what prompts did you, sort, of gather from what you learned, and how did you go about trying to answer them with follow-up research?<\/strong><\/p>\n<p>Brian Broll: Sure, so, I mean I think this is a pretty standard process in a lot of research where you want to try to make something work in a complex environment and you might make some compromises on the way, and then you\u2019ll want to step back and see how many of those you can fix or resolve or weaken or loosen. So, one of the things that I\u2019ve been working on during the course of the second project has been trying to, I guess, relax some of the requirements that we had from the first project. Now, specifically, one of the ways that we derived a signal in the first project was to look at the human demonstrations and leverage a labelling function, so a concept that\u2019s been used to drive supervision for large unlabeled data sets, to drive like weak supervision. So, you might get like these noisy labels and things like that, but hopefully, given enough data, you can still learn something meaningful in this context. Now, for the first project, I wrote a labeling function, and just took the most, I guess, direct approach to try and get signal. Like, in this case, I tried to encode the heuristic that, regardless if it\u2019s a human or an AI, we can probably infer what its goal is based on what it\u2019s going to move most toward in the future. So I had a labeling function that would look ahead into the future and then look at what the human moves most directly toward, and then it would say that this is probably a decent approximation of what their sub-goal was, so we can do this for every state, and then, even though there might be some noise in the labels, we can hopefully learn something meaningful enough that reasonably approximates what the human\u2019s intention was.<\/p>\n<p><strong>Host: Sure.<\/strong><\/p>\n<p>Brian Broll: The thing that I didn\u2019t like about it was that that means that our labeling function has to have a good heuristic and a good idea of what the human was going toward, and especially if we\u2019re going to apply this into other domains, it might be harder for any arbitrary state to give me a good approximation of what the human\u2019s trying to do now. And even though it doesn\u2019t have to be a perfect approximation, there are some sort of domains where this could be really challenging. So, what I\u2019ve been trying to do during this second portion of the project was relax that to the point where, rather than provide an approximate signal or an approximate label for all states, see if we can just go through and, if there\u2019s a state we really are confident about, we label that one. In other words, this is really just trying to ground the human\u2019s intentions, in the context of Minecraft, based on some interaction. So, it\u2019s reasonable to say that if the player picks up, like, some item, they were probably trying to. If they attacked a zombie, or some sort of entity, that was probably their goal. So, instead of worrying about, over the course of five seconds, while it\u2019s doing battle with a bunch of different enemies and interacting with things, like picking up weapons and attacking other entities in the game, rather than trying to predict the label at each point and trying to like disambiguate when exactly, or what exact frame does it start moving from one enemy to another or things like that, or when exactly does it start to retreat or when it is just collecting itself to attack again, rather than trying to disambiguate the labels for those states using a labeling function, so directly, we just tried to relax it to the point where, when we see something that we\u2019re confident about, like again, an interaction in the game, we\u2019ll just label those, and then see if, using this much more sparse set of labels, we can still get meaningful results for predicting a human\u2019s intention. The idea here is that, if we\u2019re able to do some sort of, like, pretraining, and bias the network on some related task, then maybe, using these sparser labels, we can just kind of fine-tune the original weights which were, again, trained on some similar task, like, that could be predicting, like, distances or just something that\u2019s learning more of what the heuristic that the labeling function was encoding, we could, instead, just bias some of these suspicions that we have through the use of training it on related tasks, and then fine-tune it on this smaller set of labels that we can trust with much higher confidence because we actually see some sort of game interaction.<\/p>\n<p><strong>Host: Okay. Well, let\u2019s circle back to the laziness issue. In some way, laziness is the great motivator for invention and we talked about this as sort of a recurring theme in the work you\u2019re choosing to tackle, making complex things easier for people, especially for novices and non-experts, and more accessible in the process. So, can you expand on that just a little bit, why it\u2019s important to you?<\/strong><\/p>\n<p>Brian Broll: Sure, I mean I can never help the irony, when I find myself sitting at the computer and doing repeated tasks, especially when a huge part of programming is about automation.<\/p>\n<p><strong>Host: Right?<\/strong><\/p>\n<p>Brian Broll: I mean, I guess a little bit more, maybe, philosophically I don\u2019t really like when the tools that I\u2019m working with get in the way of the problem I\u2019m solving. Like, they should really be complementary, and I shouldn\u2019t be spending time trying to, like, I don\u2019t know, wrangle or fight with the tools themselves. I guess that ideology kind of pushes me towards, like, umm\u2026 or gives me very little patience for when I\u2019m doing something, some repeated task manually, over and over, and definitely, like, pushes me towards finding some way to not only remove or reduce any repetition, but also see if there\u2019s a way that the tools can really hopefully get out of the way so that the person can actually try to reason about and understand the problem that they\u2019re actually trying to solve. I don\u2019t want to confuse this with trying to find, like, a silver bullet. I\u2019ve run into this a bit, actually, in grad school when people sometimes would debate about, like, visual programming versus, like, textural programming languages, or, like, whether or not block space programming is like a real programming language. Which, which I don\u2019t know, I mean, personally, I feel like there\u2019s syntax, there\u2019s semantics, it falls under the category of a programming language, even though most of the time the point that they\u2019re making is really, do you use it for like industrial grade applications, which, no. The ones that I\u2019ve used, and the ones that I\u2019ve written, they\u2019re\u2026 I think their strength is in, making it easy to access some of these complex topics and concepts, and be able to learn really valuable things, but I\u2019m not recommending that we go and write performance-critical applications in these languages. But I do think that it\u2019s funny when people kind of get this idea of it having to be one or the other because I think really the best set of tools are ones that play well with each other. I guess what I should say, it\u2019s not like every carpenter starts with just, like, a hammer, you know, like, and then tries to build a house entirely with a hammer.<\/p>\n<p><strong>Host: You need a whole full set of tools\u2026<\/strong><\/p>\n<p>Brian Broll: Exactly. You need a huge tool set and you need to make sure these all, like, complement each other and work together nicely.<\/p>\n<p>(music plays)<\/p>\n<p><strong>Host: Well, I can\u2019t let any podcast go by without asking what could possibly go wrong. And this is actually my thing because I\u2019m that person in eighth grade who would have tried to hack someone\u2019s else\u2019s robot just because, you know\u2026 Do we all use\u2026<\/strong><\/p>\n<p>Brian Broll: It\u2019s fun??<\/p>\n<p><strong>Host: \u2026our powers for good? Some of us use them for evil. Just saying. Umm. Not full-on evil but just, you know\u2026<\/strong><\/p>\n<p>Brian Broll: Yeah, yeah, certainly\u2026<\/p>\n<p><strong>Host: Anyway. So, is there anything about the work that you\u2019re doing that keeps you up at night? Any potential downside? And if so, how could you, from the get-go, build in safeguards to mitigate that?<\/strong><\/p>\n<p>Brian Broll: Yeah, so it\u2019s a really good question. I mean a lot of things I really liked about the work that I\u2019ve done in the past, and this gets a little bit more into my, I don\u2019t know, maybe preferences and ideologies about, like, software and ecosystems and community and all that good stuff, but I really like when these sorts of tools and concepts can be really accessible to people across the board. You know, like, in undergrad, I was a math major and I got a teaching licensure, and my first teaching placement was at a school in need of assistance, so, like, a very large minority demographic. Some of them come from, or would come from, like really rough childhoods and really unsafe home environments and things like that. One of the things that I really enjoy about a lot of the work that I did at Vandy in trying to make these more complex topics accessible, and some of the things that I really like about open source software in general, is this idea of, basically, being able to give people the resources, or the abilities, to be able to get involved and to learn a lot without any regard to, like, a lot of the social issues that can get in the way. Now, that being said, a lot of the work that I\u2019ve been doing has been, like, publicly available. It\u2019s free. If people want to use it, they can just use it. But this does go hand-in-hand with potential misuse. Now, in my case, since I\u2019m mostly trying to empower people to be able to do things that they couldn\u2019t do before, it\u2019s not necessarily a social platform so there are different types of risks and issues, but it is challenging. I mean, just because you teach people some of these like cyber-security concepts, it doesn\u2019t really guarantee that they won\u2019t try to be on the hacker side rather than you know, like, the white hat side. I haven\u2019t really built in any safeguards right now. I\u2019m concerned about taking away from the people who could really benefit from it, and who already have enough challenges. So that it makes it really challenging. I always hope that these types of things can be addressed with, ideally, just more of developing, like, a positive culture and community and people buying into that. Because you hope that people aren\u2019t only doing the right thing just because it\u2019s not against the law, or it\u2019s the only thing not against the law\u2026 you know? I mean, like, I\u2019d rather people are motivated to help the community and get involved, and when doing things like that, I really don\u2019t want to have anything get in the way, you know. Those are the kinds of scenarios where I feel like I just want to, like, if we can pave the road as much and as far as possible, and then hope we can build a community around this, and an ecosystem where people want to do good to the people around them. But I realize that there will be cases where people might fall through the cracks and this might be a little bit more ideological, but that\u2019s what I shoot for or strive for.<\/p>\n<p><strong>Host: Right. All right. Well I always ask my guests to get us a little bit of their personal story and you\u2019ve talked a little bit about what you\u2019ve done prior to this with your doctoral work and a little bit about the residency program. But give us a little more context on your background, where you\u2019re from, and how you ended up in the Microsoft AI Residency Program.<\/strong><\/p>\n<p>Brian Broll: So, sometimes it\u2019s funny looking back, things can seem a little bit arbitrary. I mean, I grew up on a dairy farm in Minnesota. I wasn\u2019t sure if I wanted to go to college or not, but other people seemed to be, and then another wrestling coach was recruiting me so that made it easy. Then I got an academic full ride which also made it easier. So, I decided to go for my bachelor\u2019s in math. Then I was planning on going back and teaching high school math and coaching wrestling\u2026 and I had taken a couple of computer science classes. I took, like, the intro course and a data structures course and a programming languages course\u2026 and then found out\u2026 there had been a few professors in undergrad who had asked, like, why I wasn\u2019t planning on going for, like, my PhD or going to grad school or anything like that, and I always said I didn\u2019t really want to invest any more money in an education if I didn\u2019t see how it would pay out. So, I was a bit more, I guess, pragmatic perhaps, given my background and lack of exposure to those kind of things before. But then my senior year, one of the professors was asking in more detail about why I don\u2019t want to go and stuff. And then when I mentioned that I didn\u2019t want to invest more money in an education if I didn\u2019t see, necessarily, the payout, she said that a lot of PhD programs are covered, and you can get like a stipend on top of it and, and then I figured I might as well apply.<\/p>\n<p><strong>Host: Yeah.<\/strong><\/p>\n<p>Brian Broll: So, I had enjoyed doing stuff with computer science so I thought I would apply to computer science PhD programs even though I didn\u2019t major or minor it in. And then, I actually heard about the Microsoft AI Residency while interviewing on site for the Google AI Residency\u2026<\/p>\n<p><strong>Host: Oh, really!<\/strong><\/p>\n<p>Brian Broll: \u2026and then came back and applied to this one! So, sometimes it\u2019s funny looking back because sometimes the path to get here can seem pretty arbitrary but\u2026<\/p>\n<p><strong>Host: Non-intuitive\u2026<\/strong><\/p>\n<p>Brian Broll: Yeah, definitely, but I\u2019m excited to be here.<\/p>\n<p><strong>Host: Well, on the heels of that question, my new favorite one is the one I\u2019m going to ask you next. What\u2019s one interesting thing about you that we couldn\u2019t find on a web search, maybe, that has influenced your career?<\/strong><\/p>\n<p>Brian Broll: Hmmm. So, I mean, I bring up wrestling, but when I was student teaching and running, like, studies, the students liked to Google me and find pictures of me wrestling and then set it as like their background or incorporate into their games. So, you can definitely find that on a web search. But it\u2019s hard to really give enough credit to some of the experiences that you can have that really play a role in your career development. Like, although being a dairy farmer and growing up on a dairy farm doesn\u2019t really seem to be, maybe, the most closely related in doing research in AI and machine learning, I think there are certainly a lot of different attitudes and perspectives that can be really positive. And I think that, in some ways, it can keep me very pragmatic in the sense that I really like seeing some, like, outcome or the benefit of what I\u2019m working on, and I think, in those ways, like, being able to work with kids and just trying to help give them all the tools to be able to succeed, regardless of background, is something that can have some sort of positive impact. And I think that some of the experiences and the pragmatic nature growing up on a farm played a little bit of an influence in keeping me grounded that way.<\/p>\n<p><strong>Host: My husband\u2019s father used to say, cows never go on vacation.<\/strong><\/p>\n<p>Brian Broll: That\u2019s true, that\u2019s true! My wife and I have been together for twelve years. Basically, I got a driver\u2019s license and a girlfriend. So, like when we were dating, a lot of times it was hard for me to get out of chores and things like that, so we spent a lot of dates milking cows and feeding cows and doing all sorts of things like that!<\/p>\n<p><strong>Host: Brian, this has been so much fun. As we close, you\u2019re much closer to the beginning of your career than many of the guests I\u2019ve had on this show, so, I want to put a little different spin on the \u201cwhat advice would you give to emerging researchers\u201d because you\u2019re kind of one\u2026<\/strong><\/p>\n<p>Brian Broll: Yeah!<\/p>\n<p><strong>Host: \u2026and I\u2019ll simply ask you, what\u2019s next for Dr. Brian Broll?<\/strong><\/p>\n<p>Brian Broll: Yeah, so I\u2019m going to be a research scientist at Vanderbilt starting in September and I\u2019ll be continuing some of the work, try to increase the impact of some of the work that I did during my PhD, as well as explore ways that we can try to make deep learning more accessible, especially to people in the natural sciences. So, I\u2019ll be working on those immediate projects, but I\u2019m definitely interested in seeing other ways that we can try to combine some of my experience here. We\u2019re getting more involved with AI and machine learning with some of the perspectives and kind of driving ideas that I\u2019ve had throughout grad school and about trying to make these tools more powerful and accessible so that hopefully it can have a bigger impact more broadly. So\u2026<\/p>\n<p><strong>Host: Let me ask you a question I was going to ask before, but I really do want to hear what you think. What was your favorite thing about the residency program and what would you advise people that might be interested in ending up at Microsoft\u2019s AI Residency Program?<\/strong><\/p>\n<p>Brian Broll: So, my favorite thing is probably the, like, diversity of research projects that are out there and being able to work so closely with so many very impressive researchers in so many different areas. I think it\u2019s really great to be able to have that sort of exposure, especially when you\u2019re trying to learn more about different aspects of AI and machine learning. It\u2019s really hard to replace being able to work so closely with so many different researchers here at MSR.<\/p>\n<p><strong>Host: Brian Broll, thank you for joining us today. It\u2019s been illuminating\u2026 and delightful!<\/strong><\/p>\n<p>Brian Broll: Thanks, it\u2019s been a pleasure.<\/p>\n<p>(music plays)<\/p>\n<p>To learn more about academic programs with Microsoft Research, visit <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/\">Microsoft.com\/research<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>In 2018, Microsoft launched the Microsoft AI Residency Program, a year-long, expanded research experience designed to give recent graduates in a variety of fields the opportunity to work alongside prominent researchers at MSR on cutting edge AI technologies to solve real-world problems. Dr. Brian Broll was one of them. A newly minted PhD in Computer Science from Vanderbilt University, Dr. Broll was among the inaugural cohort of AI residents who spent a year working on machine learning in game environments and is on the pod to talk about it!<\/p>\n<p>Today, Dr. Broll gives us an overview of the work he did and the experience he had as a Microsoft AI Resident, talks about his passion for making complex concepts easier and more accessible to novices and young learners, and tells us how growing up on a dairy farm in rural Minnesota helped prepare him for a life in computer science solving some of the toughest problems in AI.<\/p>\n","protected":false},"author":38022,"featured_media":607848,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"https:\/\/player.blubrry.com\/id\/48739654\/","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"categories":[240054],"tags":[],"research-area":[13556],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-607842","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-msr-podcast","msr-research-area-artificial-intelligence","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"https:\/\/player.blubrry.com\/id\/48739654\/","podcast_episode":"","msr_research_lab":[199565],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[395930],"related-projects":[],"related-events":[],"related-researchers":[],"msr_type":"Post","featured_image_thumbnail":"<img width=\"960\" height=\"540\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2019\/09\/Brian-Broll-_Podcast__Site_11_2019_1400x788-960x540.png\" class=\"img-object-cover\" alt=\"a man wearing glasses and smiling at the camera\" decoding=\"async\" loading=\"lazy\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2019\/09\/Brian-Broll-_Podcast__Site_11_2019_1400x788-960x540.png 960w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2019\/09\/Brian-Broll-_Podcast__Site_11_2019_1400x788-300x169.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2019\/09\/Brian-Broll-_Podcast__Site_11_2019_1400x788-768x432.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2019\/09\/Brian-Broll-_Podcast__Site_11_2019_1400x788-1024x576.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2019\/09\/Brian-Broll-_Podcast__Site_11_2019_1400x788-1066x600.png 1066w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2019\/09\/Brian-Broll-_Podcast__Site_11_2019_1400x788-655x368.png 655w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2019\/09\/Brian-Broll-_Podcast__Site_11_2019_1400x788-343x193.png 343w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2019\/09\/Brian-Broll-_Podcast__Site_11_2019_1400x788-640x360.png 640w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2019\/09\/Brian-Broll-_Podcast__Site_11_2019_1400x788-1280x720.png 1280w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2019\/09\/Brian-Broll-_Podcast__Site_11_2019_1400x788.png 1400w\" sizes=\"auto, (max-width: 960px) 100vw, 960px\" \/>","byline":"","formattedDate":"September 11, 2019","formattedExcerpt":"In 2018, Microsoft launched the Microsoft AI Residency Program, a year-long, expanded research experience designed to give recent graduates in a variety of fields the opportunity to work alongside prominent researchers at MSR on cutting edge AI technologies to solve real-world problems. Dr. Brian Broll&hellip;","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/607842","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/38022"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=607842"}],"version-history":[{"count":5,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/607842\/revisions"}],"predecessor-version":[{"id":896334,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/607842\/revisions\/896334"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/607848"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=607842"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=607842"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=607842"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=607842"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=607842"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=607842"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=607842"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=607842"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=607842"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=607842"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=607842"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}