{"id":507143,"date":"2018-09-26T08:00:13","date_gmt":"2018-09-26T15:00:13","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=507143"},"modified":"2018-10-17T08:11:02","modified_gmt":"2018-10-17T15:11:02","slug":"all-about-automated-machine-learning-with-dr-nicolo-fusi","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/podcast\/all-about-automated-machine-learning-with-dr-nicolo-fusi\/","title":{"rendered":"All about automated machine learning with Dr. Nicolo Fusi"},"content":{"rendered":"<div id=\"attachment_507149\" style=\"width: 1034px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-507149\" class=\"wp-image-507149 size-large\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/09\/NicoloFusi_Pod_Site_09_2018_1400x788-1024x576.png\" alt=\"Dr. Nicolo Fusi Podcast headshot\" width=\"1024\" height=\"576\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/09\/NicoloFusi_Pod_Site_09_2018_1400x788-1024x576.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/09\/NicoloFusi_Pod_Site_09_2018_1400x788-300x169.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/09\/NicoloFusi_Pod_Site_09_2018_1400x788-768x432.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/09\/NicoloFusi_Pod_Site_09_2018_1400x788-1066x600.png 1066w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/09\/NicoloFusi_Pod_Site_09_2018_1400x788-655x368.png 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/09\/NicoloFusi_Pod_Site_09_2018_1400x788-343x193.png 343w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/09\/NicoloFusi_Pod_Site_09_2018_1400x788.png 1400w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><p id=\"caption-attachment-507149\" class=\"wp-caption-text\">Dr. Nicolo Fusi, researcher at the Microsoft Research New England lab in Cambridge, Massachusetts<\/p><\/div>\n<h3>Episode 43, September 26, 2018<\/h3>\n<p>You may have heard the phrase, necessity is the mother of invention, but for <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/fusi\/\">Dr. Nicolo Fusi<\/a>, a researcher at the Microsoft Research <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/lab\/microsoft-research-new-england\/\">lab in Cambridge, Massachusetts<\/a>, the mother of his invention wasn\u2019t so much necessity as it was boredom: the special machine learning boredom of manually fine-tuning models and hyper-parameters that can eat up tons of human and computational resources, but bring no guarantee of a good result. His solution? Automate machine learning with a meta-model that figures out what other models are doing, and then predicts how they\u2019ll work on a given dataset.<\/p>\n<p>On today\u2019s podcast, Dr. Fusi gives us an inside look at Automated Machine Learning \u2013 Microsoft\u2019s version of the industry\u2019s AutoML technology \u2013 and shares the story of how an idea he had while working on a gene editing problem with <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/project\/crispr\/\">CRISPR\/Cas9<\/a> turned into a bit of a machine learning side quest and, ultimately, a surprisingly useful instantiation of Automated Machine Learning &#8211; now a feature of <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/azure.microsoft.com\/en-us\/services\/machine-learning-studio\/\" target=\"_blank\" rel=\"noopener\">Azure Machine Learning<\/a> &#8211; that reduces dependence on intuition and takes some of the tedium out of data science at the same time.<\/p>\n<h3>Related:<\/h3>\n<ul type=\"disc\">\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/podcast\">Microsoft Research Podcast<\/a>: View more podcasts on Microsoft.com<\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/itunes.apple.com\/us\/podcast\/microsoft-research-a-podcast\/id1318021537?mt=2\">iTunes<\/a>: Subscribe and listen to new podcasts each week on iTunes<\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/subscribebyemail.com\/www.blubrry.com\/feeds\/microsoftresearch.xml\">Email<\/a>: Subscribe and listen by email<\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/subscribeonandroid.com\/www.blubrry.com\/feeds\/microsoftresearch.xml\">Android<\/a>: Subscribe and listen on Android<\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/open.spotify.com\/show\/4ndjUXyL0hH1FXHgwIiTWU\">Spotify<\/a>: Listen on Spotify<\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/www.blubrry.com\/feeds\/microsoftresearch.xml\">RSS feed<\/a><\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/note.microsoft.com\/ww-registration-microsoft-research-newsletter-s.html?wt.mc_id=S-webpage_podcast\">Microsoft Research Newsletter<\/a>: Sign up to receive the latest news from Microsoft Research<\/li>\n<\/ul>\n<hr \/>\n<h3>TRANSCRIPT<\/h3>\n<p>Nicolo Fusi: So, we cast it, again, as a machine-learning problem. But we had multiple models interacting, and tuning each model separately was a complete nightmare. And during that process, I decided, surely somebody must have thought about something. And you know, they had. But the problem is that a lot of the state-of-the-art was working only for tuning a few hyper-parameters at a time. What we are trying to do was really tune thousands.<\/p>\n<p><strong>(music plays)<\/strong><\/p>\n<p><strong>Host: You\u2019re listening to the Microsoft Research Podcast, a show that brings you closer to the cutting-edge of technology research and the scientists behind it. I\u2019m your host, Gretchen Huizinga.<\/strong><\/p>\n<p><strong>Host: You may have heard the phrase, necessity is the mother of invention, but for Dr. Nicolo Fusi, a researcher at the Microsoft Research lab in Cambridge, Massachusetts, the mother of his invention wasn\u2019t so much necessity as it was boredom: the special machine learning boredom of manually fine-tuning models and hyper-parameters that can eat up tons of human and computational resources, but bring no guarantee of a good result. His solution? Automate machine learning with a meta-model that figures out what other models are doing, and then predicts how they\u2019ll work on a given dataset.<\/strong><\/p>\n<p><strong>On today\u2019s podcast, Dr. Fusi gives us an inside look at Automated Machine Learning \u2013 Microsoft\u2019s version of the industry\u2019s AutoML technology \u2013 and shares the story of how an idea he had while working on a gene editing problem with CRISPR-Cas9, turned into a bit of a machine learning side quest, and, ultimately, a surprisingly useful instantiation of Automated Machine Learning \u2013 now a feature of Azure Machine Learning \u2013 that reduces dependence on intuition and takes some of the tedium out of data science at the same time. That and much more on this episode of the Microsoft Research Podcast.<\/strong><\/p>\n<p><strong>(music plays)<\/strong><\/p>\n<p><strong>Host: Nicolo Fusi, welcome to the podcast.<\/strong><\/p>\n<p>Nicolo Fusi: Thank you, it\u2019s great to be here.<\/p>\n<p><strong>Host: So, you lead the Automated Machine Learning efforts at Microsoft Research in Cambridge, Massachusetts. I really want to wade into the technical weeds with you in a bit, but for right now, in broad strokes, tell us about your work. What gets you up in the morning?<\/strong><\/p>\n<p>Nicolo Fusi: Yeah, it\u2019s interesting, because my background is in machine learning and I got very excited about problems in computational biology. And so I did a lot of work in computational biology and then during that work, I kind of figured, oh, there are so many machine learning problems that you can solve that are interesting and apply to a wide range of things. And so, I kind of went back a little bit in machine learning. So, most recently, as you said, I\u2019m working on automated machine learning which is a field where the goal is to kind of automate as much of the machine learning process as possible. That goes from data preparation to model criticism, for instance, once we come up with a model.<\/p>\n<p><strong>Host: Ok. So, drill in a little bit on this idea of computational biology.<\/strong><\/p>\n<p>Nicolo Fusi: So, computational biology is an enormous field. There are many kind of different people doing different things from proteomics to genomics. Some of them are using mathematical tools. Some of them are more probabilistic, or statistical, in nature. So, my slice of this world was using machine learning and statistics to kind of investigate molecular mechanism. And in particular I was working on genetics, mostly. I also worked on functional genomics, but genetics was the most formative part of my training.<\/p>\n<p><strong>Host: So, we talked a little bit about your interest in machine learning, computational biology and medicine, in fact. So those are three sort of divergent paths \u2013 well there\u2019s some overlaps on Venn diagram \u2013 but how did those all come together for you?<\/strong><\/p>\n<p>Nicolo Fusi: I started in machine learning, and you know machine learning you can do either applied work \u2013 uh, you pick a problem, you apply machine learning to it, it always comes with its own set of challenges \u2013 or you can pick something more theoretical and maybe advance the way people do modeling or infer the parameters of a model, for instance. And when it came to starting my PhD, I had the choice of problems from both fields and, due to personal circumstances, I felt like I needed to do something that had an effect on human health. But I also thought that human health was too far from machine learning. And in some sense, to this day, I still think that if you want to do medicine with machine learning, I think you need to stop somewhere in between first. Like kind of break up your journey. And I think you probably should break up your journey at the molecular level, which is where computational biology comes in. So, I started working on solving questions in computational biology using machine learning with the goal of later, kind of going from computational biology to medicine, again using machine learning.<\/p>\n<p><strong>Host: That\u2019s fascinating. Before we launch into your specific work in the field, let\u2019s talk a little more generally about automated machine learning. Forbes Magazine had an article where the author claimed that it was set to become \u201cthe future of AI.\u201d Is that overstatement?<\/strong><\/p>\n<p>Nicolo Fusi: Well, in general in AI right now, there is a lot of \u201cThis is the future of AI! That is the future of AI!\u201d And it would be great, as somebody who does a lot of AutoML research, if AutoML will single-handedly be the future of all of AI. But I think it\u2019s going to be a huge component. And I think, more than AutoML, it\u2019s probably going to be meta-learning, if one has to put names on fields.<\/p>\n<p><strong>Host: Interesting, yeah.<\/strong><\/p>\n<p>Nicolo Fusi: Because meta-learning is learning about learning. So, in some sense, I think we have developed, over time, a good set of kind of base models or base algorithms. And now we are starting to move up the hierarchy and kind of combine classes and families of models into meta-models, and that kind of incorporates all that is going on underneath them. So, in some sense I agree with the Forbes article in the sense that we need to move one level up the hierarchy. Ummm\u2026 But there is a lot more work to be done at the base.<\/p>\n<p><strong>Host: Let\u2019s drill in a little bit on why automated machine learning is such a big deal.<\/strong><\/p>\n<p>Nicolo Fusi: Yes.<\/p>\n<p><strong>Host: Ok? And perhaps by way of comparison. So, you alluded earlier to the traditional machine learning workflow? Tell us what that looks like and what an automated machine learning workflow looks like, and why it\u2019s different and why it matters.<\/strong><\/p>\n<p>Nicolo Fusi: In my mind, the traditional machine learning workflow, which is also the data science workflow \u2013 people use different names \u2013 you start with some question and you define what kind of data do I need to answer that question, what kind of metrics measure my success and what\u2019s the closest numerically computable metric that I can pair and I can measure to see whether my model is doing well? And then there is a lot of data cleaning, and then, eventually, you start the modeling phase. And the modeling phase involves transforming features, changing different models, tuning different parameters. And every time you go down one path, you pick a way to transform your features, you pick one model, you pick a set of parameters and then you test it and then you go back. And you maybe try different hypothesis, you gather more data. You keep doing this loop, over and over again, and then, at the end, you basically produce one model that maybe you deploy, maybe you inspect to see whether the predictions are correct or fair or stuff like that. The goal of automated machine learning is to automate as much of this as possible. I don\u2019t think we\u2019ll ever be able to automate the \u201cphrasing the question\u201d or \u201cdeciding the metric,\u201d because that\u2019s what the human should be doing, really. But the goal is to kind of remove as much of the high-dimensional thinking with many options that are not always clear, that really kind of slows down the process for humans.<\/p>\n<p><strong>Host: I\u2019ve heard it described as the drudge work of data science, the fine-tuning of the models and the parameters. Explain that a little bit more about how\u2026 is it basically a trial and error?<\/strong><\/p>\n<p>Nicolo Fusi: It is a lot of a trial and error, because it\u2019s really high-dimensional space. So, depending on which value you set one hyper-parameter to, all the values for another hyper-parameter completely change meaning or change the scale at which they are relevant. And so, it becomes a really difficult problem because\u2026 I\u2019ve done it, right? It\u2019s extremely boring. Not a good use of time. If you do kind of parameter sweeps, which is a lot of what the industry is doing, you kind of waste a ton of computational resources, and you have no guarantee of finding anything good.<\/p>\n<p><strong>Host: (laughs) That\u2019s sad\u2026<\/strong><\/p>\n<p>Nicolo Fusi: So, it\u2019s bleak.<\/p>\n<p>Yeah. Uh, yeah\u2026 So, how are you tackling that?<\/p>\n<p>Nicolo Fusi: So there are different techniques. And AutoML is a very exciting research area. There has been like AutoML competitions, AutoML workshops, symposiums at NIPS. It\u2019s really very exciting. The broad idea is, let\u2019s try to use a model to kind of figure out what other models are doing. So in that sense it\u2019s kind of meta-learning, because you\u2019re trying to predict how different models react when you change their parameters. And you use that model to guide you through a series of experiments.<\/p>\n<p><strong>Host: Alright, so you would have to have a base of model experiments for this uber model to judge, right?<\/strong><\/p>\n<p>Nicolo Fusi: Exactly, to learn form, yes.<\/p>\n<p><strong>Host: To learn from. Better word.<\/strong><\/p>\n<p>Nicolo Fusi: Yes.<\/p>\n<p><strong>(music plays)<\/strong><\/p>\n<p><strong>Host: According to legend \u2013 I use that term loosely \u2013 you were using machine learning and getting mired in the drudgework of data science and, basically, thinking there\u2019s got to be an app for this, or something.<\/strong><\/p>\n<p>Nicolo Fusi: Yeah.<\/p>\n<p><strong>Host: And you set out to fix the problem for yourself. It wasn\u2019t like, \u201cI\u2019m going to go create this AutoML thing for the world.\u201d It\u2019s like, I got to solve my own problem. And you kind of did it covertly. Tell us that story.<\/strong><\/p>\n<p>Nicolo Fusi: Yeah, so I was working on CRISPR gene editing.<\/p>\n<p><strong>Host: OK.<\/strong><\/p>\n<p>Nicolo Fusi: It was a joint collaboration between Microsoft Research and the Broad Institute. At the Broad Institute, the lead investigator was John Doench, and Microsoft Research was Jennifer Listgarten, who\u2019s now at Berkeley, and me. And basically, we got this data. We figured out the question. We did all the deciding which metric we want to optimize, and then we spent six months, maybe, of our own time, almost full-time, trying different ways to slice and dice the model space, the parameter space. It was just exhausting.<\/p>\n<p><strong>Host: What question were you trying to answer?<\/strong><\/p>\n<p>Nicolo Fusi: In this work, we were trying to investigate and build a predictive model of the off-target activity in CRISPR-Cas9. So, CRISPR is a gene editing system. It allows you to mute a gene that you don\u2019t want to be expressed, for instance. This gene is causing a disease. I want to shut it down. So, you can do that. In previous work, we basically figured out, again, a machine-learning model to predict, given the many ways you can edit the gene, because you can do it in different ways, what\u2019s the most successful edit? And in this follow-up work, we were investigating the issue of, given that I want to perform this edit, what\u2019s the likelihood that I mess up something else in the genome that I didn\u2019t want to touch? You can imagine if I want to remove a gene, or silence a gene that was causing a disease, I don\u2019t want to suddenly give you a different disease because an unintended edit happened somewhere else.<\/p>\n<p><strong>Host: Right.<\/strong><\/p>\n<p>Nicolo Fusi: And so, we cast it, again, as a machine-learning problem. But we had multiple models interacting, and tuning each model separately was a complete nightmare. And during that process, I decided, surely somebody must have thought about something. And you know, they had. But the problem is that a lot of the state-of-the-art was working only for tuning a few hyper-parameters at a time. What we are trying to do was really tune thousands.<\/p>\n<p><strong>Host: Right.<\/strong><\/p>\n<p>Nicolo Fusi: And so, it would\u2019ve taken ages. And so, I kind of had an idea, while working on CRISPR, so it was kind of like a side project. It kind of worked, and I was suspicious because it was a weird approach that was not supposed to work. It was really a hack. And so, I kind of kept it quiet. I kind of used it to inform my own experiments but I didn\u2019t\u2026<\/p>\n<p><strong>Host: Advertise\u2026.<\/strong><\/p>\n<p>Nicolo Fusi: \u2026 advertise it.<\/p>\n<p><strong>Host: Ok. So, the best and brightest minds all over the world are trying to tackle this problem. Like you say, there\u2019s other companies working on it. They\u2019ve got symposiums, they\u2019ve got competitions. And here you are, in your lab, working on a CRISPR-Cas9 gene problem, and you come up with this. Tell us what it is.<\/strong><\/p>\n<p>Nicolo Fusi: So, it\u2019s something that\u2019s actually is used already by you know, Netflix, Amazon, we probably use it somewhere in the company to recommend things.<\/p>\n<p><strong>Host: Right.<\/strong><\/p>\n<p>Nicolo Fusi: So, the idea was ultimately deciding which algorithm, which set of hyper-parameters to use for a given problem, you\u2019re kind of trying to recommend a series of things and then I evaluate them and I tell you how well they work. And then you kind of update your beliefs about what\u2019s going to work and what\u2019s not going to work. And that is similar to movies, right? You watch a movie, you rate it and then they learn more about you, your tastes and so on. The good news for us is that we don\u2019t actually rely on the human watching the movie. We can force the execution of a machine learning pipeline. We can just tell, execute this thing. And they perform, let\u2019s say, well or not so well depending on which data set they\u2019re exposed to. So, you can now gather a corpus of experiments that help you guide the selection of what to do in the new data set. So that\u2019s the meta-learning aspect. You have this meta-model that knows, and can predict, how individual base models will perform when shown some given data.<\/p>\n<p><strong>Host: Okay. So, implementing this\u2026<\/strong><\/p>\n<p>Nicolo Fusi: Yeah\u2026 So because I was the first user, I didn\u2019t want just something that you could you know academically show, \u201cOh, you know, we beat random,\u201d because random is a strong baseline for AutoML, surprisingly. Like picking a model at random. I wanted to encapsulate this work into something I could use into a library.<\/p>\n<p><strong>Host: Right.<\/strong><\/p>\n<p>Nicolo Fusi: So, we worked on the first version of a toolkit you could just deploy in your data. And we started using it for our own stuff. And then we were working on this, and in the summer, at Microsoft, you have the hackathon, which is this huge initiative, everybody kind of takes part. And a lot of teams were looking for data scientists. And it was crazy, because on the machine learning mailing list was, \u201cOh I\u2019m working on, you know, this accessibility problem. Is there anybody who knows machine learning who can help me out with this data analysis question?\u201d And so, we thought, ok, so if nobody\u2019s responding to that email, maybe we should just blast out an email to the entire mailing list saying, \u201cWe have fifty spots. So, we can give you an API key that we designed in a bad way. And if you want to use something that kind of finds a model automatically, you phrase the question and we kind of search your model space for you and give you a pipeline you can use at the end. Just give you a Python object. You can ask for predictions.\u201d<\/p>\n<p><strong>Host: OK. What was the response?<\/strong><\/p>\n<p>Nicolo Fusi: The response was crazy. So, we were a research team of two people, specifically, so, we didn\u2019t have a lot of systems \u201cumph\u201d behind us. So, it was a single machine serving this meta-brain model that kind of figures out what to do. And by our calculations, we could only accommodate fifty teams. So, we got a hundred and fifty requests within, you know, the first week.<\/p>\n<p><strong>Host: Unbelievable.<\/strong><\/p>\n<p>Nicolo Fusi: And so, we stretched our resources a bit thin to let people use it.<\/p>\n<p><strong>Host: So, you did accommodate the hundred and fifty?<\/strong><\/p>\n<p>Nicolo Fusi: We did accommodate the hundred and fifty in the end.<\/p>\n<p><strong>Host: OK.<\/strong><\/p>\n<p>Nicolo Fusi: It was a lot of CPU time\u2026<\/p>\n<p><strong>Host: Yeah.<\/strong><\/p>\n<p>Nicolo Fusi: And a lot of like, last minute, \u201cOh, the service is down. Can you reboot?\u201d Something that we had never experienced.<\/p>\n<p><strong>Host: Well, ok, so this is the hackathon. And it sounds to me like the scenario you\u2019re painting is something that would help validate your research and actually help the people that are doing projects within the hackathon itself.<\/strong><\/p>\n<p>Nicolo Fusi: Yes. It was kind of a win\/win, because we saw some real-world usage of our tool. And it was very limited in the beginning. We could only do classification, not regression problems, just because we started with that. And people came and they started saying \u201cOh, could you add this base learner or this processing method?\u201d and it was very useful to us.<\/p>\n<p><strong>Host: What did you say, \u201cNo I can\u2019t? Not yet?\u201d<\/strong><\/p>\n<p>Nicolo Fusi: You know, our answer was always, \u201cOh, it\u2019s just, you know, it\u2019s just us, but one day\u2026.\u201d<\/p>\n<p><strong>Host: It\u2019s just research.<\/strong><\/p>\n<p>Nicolo Fusi: It\u2019s just research, yes.<\/p>\n<p><strong>Host: Well, let\u2019s go there for a minute. There\u2019s been a big announcement at Ignite.<\/strong><\/p>\n<p>Nicolo Fusi: Very exciting.<\/p>\n<p><strong>Host: Can you talk about that?<\/strong><\/p>\n<p>Nicolo Fusi: So, yes, it\u2019s a very exciting announcement. It took a ton of work from a lot of very smart people. So, it\u2019s a joint collaboration at this point between MSR, you know we did the original kind for proof-of-concept, and a huge amount of work went in from people within Azure. So, it\u2019s being released as kind of like an Automated ML library, or SDK, that you can use on your data. And it\u2019s in public preview.<\/p>\n<p><strong>Host: That\u2019s super exciting.<\/strong><\/p>\n<p>Nicolo Fusi: You know, we have a very good collaboration, and a very good ability to now transfer what are technically complex ideas. You know this technology transfer was not like a small, simple model that you could just write quickly. And we had to think about, is the probability distribution calibrated? Are the choices that we make based on that information correct in most cases for most datasets? What\u2019s the cost on runtime on our servers to satisfy the demand that this thing will likely have? So, it was a lot of engineering work. And I think we figured a lot of that out and so, we were able to transfer a lot of research into product much more quickly.<\/p>\n<p><strong>Host: Well, who\u2019s the customer for this, right now?<\/strong><\/p>\n<p>Nicolo Fusi: Uhhh, we struggled a little bit, because in the early validation phase, let\u2019s call it, the research prototype, \u201clet\u2019s give it away\u201d phase, we got a lot of different kind of people approaching us. So, data scientists, they are interested in it because they want to save time. They don\u2019t care, in a lot of cases, what the final model is, they just want a good model, and they don\u2019t want to spend ages just running parameter sweeps. So, data scientists are one set of individuals who could be interested. Developers. Sometimes developers now are tasked with including intelligence in their applications. And if you don\u2019t know what to do, this kind of solution gives you a good model in a short amount of time, relative to the size of your data so you can just use it. And then there is a lot of kind of business analysts, buyers from companies. They have to make data-driven decisions and they would benefit from good predictions and this tool would give them good predictions.<\/p>\n<p><strong>Host: So, going back to your comment about data scientists being a customer, why wouldn\u2019t a data scientist be a little bit worried that this AutoML might be taking over their job?<\/strong><\/p>\n<p>Nicolo Fusi: Yeah, I get asked that question a lot. I created for myself, not intending to replace myself, but just kind of as a tool for me to use. The metaphor I use for it is, it\u2019s kind of like using a word editor. It doesn\u2019t replace the role\u2026<\/p>\n<p><strong>Host: Of a writer\u2026<\/strong><\/p>\n<p>Nicolo Fusi: \u2026of a writer. It just makes the writer that much more effective, because you don\u2019t have to cancel, with a pencil, your old text and just rewrite it from scratch. You can just erase one letter, for instance. And that\u2019s what AutoML does. Like, if you change, let\u2019s say, the way you featurize your data, you don\u2019t have to start from scratch with the tuning, you just set an AutoML run going and you just move on.<\/p>\n<p><strong>Host: Would it democratize it to the point where non-data scientists could become data scientists?<\/strong><\/p>\n<p>Nicolo Fusi: I think it\u2019s a possibility. Because in some sense you observe this meta-model kind of reasoning about your data, and you see the thinking process, if you can call it thinking. You see which models it starts out with. And then it sees how it evolves. And so, you can actually learn things about your data by observing the process, observing how different metrics are, you know, sometimes you want to maximize accuracy. But maybe looking at something like an area under the ROC curve is informative because you can now see how the probabilities are changing. So, I think you can learn from AutoML, and I think it can become kind of like some training wheels if you are starting out.<\/p>\n<p><strong>Host: Yeah, yeah. Let\u2019s go back to CRISPR for a minute, since this all started with how to make it easier to decide where to edit a gene.<\/strong><\/p>\n<p>Nicolo Fusi: Yeah.<\/p>\n<p><strong>Host: You made a website called CRISPR.ML that provides bioscientists with free tools they can use to make CRISPR gene edits?<\/strong><\/p>\n<p>Nicolo Fusi: Basically, yes.<\/p>\n<p><strong>Host: Tell us about the site. Why did you start it and how\u2019s it going?<\/strong><\/p>\n<p>Nicolo Fusi: That\u2019s a great question. So, it\u2019s CRISPR.ML, that\u2019s the URL. We had to really, really resist the crazy hype to not call it CRISPR.AI, which was available. And we just said no. It\u2019s ML. It\u2019s not AI. We decided not to feed the hype around AI.<\/p>\n<p><strong>Host: That was self-restraint on steroids.<\/strong><\/p>\n<p>Nicolo Fusi: I know, I know. It took a lot out of us. Um\u2026 So, we had done work on the on-target problem, which was how do you find the optimal, for some notion of optimality, how do you find the best edit to perform to make sure you disable a gene you want to disable? And that was giving you predictions. It was a tool that could use in Python. You could just download it from GitHub. We just give it away, liberally licensed and so on. And a lot of startups incorporated that in their tools, selling it. A lot of institutions were using it. The off-target stuff, which was, how do you make sure that you don\u2019t have unintended edits somewhere else in the genome? That was much more computationally intensive to run as something you could download. So, if you were interested in running it for your own problem of interest, you would have to wait a long time. And so, we decided, why don\u2019t we just pre-compute everything for the human genome which took an exorbitant amount of CPU time on Azure, but we could just pre-populate a giant database table and then search it almost instantly. And that\u2019s what the website does. You can put in the gene you want to edit, and you get a \u201cleast of possible\u201d guide with a score that tells you how likely the edit is to be successful and how likely each off-target is\u2026<\/p>\n<p><strong>Host: To happen?<\/strong><\/p>\n<p>Nicolo Fusi: \u2026to happen. And a global score that tells you, broadly speaking, \u201cBad, bad, bad off-targets here. Don\u2019t touch it.\u201d<\/p>\n<p><strong>Host: Don\u2019t do it.<\/strong><\/p>\n<p>Nicolo Fusi: Yeah.<\/p>\n<p><strong>Host: So, it both tells you what\u2019s a good place to go and tells you, avoid these places because lots of bad stuff could happen?<\/strong><\/p>\n<p>Nicolo Fusi: In some sense. It tells you which place to edit and if you choose to edit this, these other spots on the genome, might be edited. And maybe you don\u2019t care. So maybe your experiment is very narrowly focused on a given gene. So maybe you don\u2019t care, but in therapeutic applications, you want zero off-targets, pretty much, all the time.<\/p>\n<p><strong>Host: Yeah. That leads me into a question that I ask all of the guests on the podcast. Is there anything that keeps you up at night about what you\u2019re doing?<\/strong><\/p>\n<p>Nicolo Fusi: Uh\u2026 yeah, gene editing, AutoML, and AI\u2026 what can go wrong?<\/p>\n<p><strong>Host: Right?<\/strong><\/p>\n<p>Nicolo Fusi: No, I\u2019m not easily kept awake at night. I can sleep anywhere. But you know, there are things that concern me. And I\u2019ve tried to move my work towards addressing them as a first priority. Maybe the main thing on my mind right now is, I know that AutoML, or what we call AutoML, is a very good way to predict. So, it\u2019s a very strong, supervised machine learning method. And it can be applied to all kinds of data. And I want to make sure that, as we build a capability to generate better and better predictors, we are also thinking of ways to make sure that the predictions are well-explained, that the biases are auditable and visible to the person who\u2019s deploying these systems. So, we are spending a lot of time now thinking how fairness and all these themes that are mentioned a lot in this podcast are addressed.<\/p>\n<p><strong>Host: Right.<\/strong><\/p>\n<p>Nicolo Fusi: Because it\u2019s a very powerful tool and if you apply it in the wrong way, you\u2019re going to have exorbitant amounts of bias.<\/p>\n<p><strong>(music plays)<\/strong><\/p>\n<p><strong>Host: Tell us a bit about yourself, Nicolo. What\u2019s your background? How did you get interested in what you\u2019re doing, and how did you end up at Microsoft Research?<\/strong><\/p>\n<p>Nicolo Fusi: It\u2019s a good story. So I\u2026 I will not start from when I was a baby. I will start directly from university. I did my university in Milan close to home\u2026 well, at home, basically. And then I attended some advanced courses in statistics and I thought it was fascinating. But I was always kind of like more of a computer science person. And so, I figured computer science plus statistics? At the time, the answer was machine learning. I think to this day, probably, there is a lot\u2026<\/p>\n<p><strong>Host: Same answer.<\/strong><\/p>\n<p>Nicolo Fusi: Same answer for most people. And so, I decided to kind of do a summer internship somewhere. Anybody who would take me. You know, these kind of \u201chashtag rejection\u201d stories? I must have sent thirty or forty emails to everybody in Europe to say can you please, like, I will come for free, like, for a summer. It was kind of like my summer vacation that year. And I think the response I got was from Neil Lawrence, who\u2019s now also a podcast host as part of other things. Talking Machines. And he wrote this email in broken Italian because he speaks a little bit of Italian, but he can speak it, but he cannot write it. If he hears this I hope he\u2019s OK with that. And he says sure, come over. And I went there for a summer. The goal was to kind of build up my CV to do grad school somewhere. So, I wanted to do some research, you know, get a feeling. And after that summer, I basically decided no, I\u2019m staying here. I want to do a PhD right here. I\u2019m coming back in four months. And I basically kind of closed up all my stuff. Like finished my exams at home. Just like my thesis and just packed up and went to the UK. So, Neil had a choice of projects. And because I wanted to have an impact on health, I kind of chose the molecular biology-inspired projects and I started working on that. And it was one of the best, you know, three years of my life. It was a lot of fun. I learned a lot.<\/p>\n<p><strong>Host: Yeah. And you got your PhD\u2026<\/strong><\/p>\n<p>Nicolo Fusi: I got my PhD\u2026 Well, in the last year, I traveled a lot during my PhD, because I spent some time at Max Planck Institute in T\u00fcbingen. I spent some times at UCLA at the Institute of Pure Applied Mathematics. There was a program where the idea, I think, of to represent it correctly, was to kind of combine some mathematically-minded people and some biology\/medicine-minded people to see what kind of collaborations arise. It was an incredible program for like, I think three or four months, during which I met this group of Microsoftees in LA who were doing statistical genomics. And that included my long-term collaborator Jennifer Listgarten. And I started an internship there the year after. And then at the end of the internship they said, \u201cHey, do you want to join?\u201d So again, once I more I went back. I packed everything up, and I said sure! And I joined Microsoft Research in LA, which was this remote site. It was five of us, all kind of working in health.<\/p>\n<p><strong>Host: So, there\u2019s not actually a lab in LA?<\/strong><\/p>\n<p>Nicolo Fusi: It\u2019s not a lab. It was a rented office that used to be a steakhouse on the UCLA campus. Very, very unofficial. If you\u2019ve ever been to a Microsoft building, you see all these you know machines that include beverages, like sodas and so on. We had to have a standing order from supermarkets, delivering us the sodas.<\/p>\n<p><strong>Host: Because you didn\u2019t have a place to put your machine?<\/strong><\/p>\n<p>Nicolo Fusi: We didn\u2019t really have facilities. And it was just us. And our network was ethernet cables running everywhere.<\/p>\n<p><strong>Host: That\u2019s hilarious, did you at least have some signage?<\/strong><\/p>\n<p>Nicolo Fusi: I don\u2019t think so. It was probably, you know those plastic signs that you can kind of get at any office store?<\/p>\n<p><strong>Host: Yeah, yeah.<\/strong><\/p>\n<p>Nicolo Fusi: We had those. But we didn\u2019t have a sign that said Microsoft. But we were in the address book so sometimes Microsoft sales people would come to our office intending to access the corporate network, but they didn\u2019t understand that we were not on the corporate network even.<\/p>\n<p><strong>Host: Oh, you were that off-target.<\/strong><\/p>\n<p>Nicolo Fusi: Off, off the grid.<\/p>\n<p><strong>Host: Listen, so then how did you come to be at Cambridge? Did you go straight from LA to the Cambridge lab?<\/strong><\/p>\n<p>Nicolo Fusi: Yes, so\u2026 I think after a couple of years in LA, different people moved in different places, and Jen Listgarten came to Boston and you know, I heard this Boston lab is incredible. And so, at NIPS I met with Jennifer Chayes, who was the lab director, and I was like oh I need visit there and check it out and I joined, I think, three years ago now.<\/p>\n<p><strong>Host: So, you moved up all your stuff again?<\/strong><\/p>\n<p>Nicolo Fusi: I moved up all my stuff again. I said I love it here. I\u2019m just moving.<\/p>\n<p><strong>Host: As we close, I like to ask all my guests to look to the future a little. And it\u2019s not like predictions of the future, but more just sort of as you look at the landscape, what are the exciting challenges, hard problems, that are still out there for young researchers who might be, yourself, a few years ago, trying to decide where do I want to land? Where do I want to pack all my stuff up and move to?<\/strong><\/p>\n<p>Nicolo Fusi: Ah! That\u2019s a great question, and one I spend a lot of time thinking about because we have a lot of interns. The quality of the students right now is exceptional. So even maybe a first-year PhD student has an incredible amount of experience, very often. And they\u2019re asking you where should I direct my career? And it\u2019s hard to give advice. But I think the area where I expect the most improvement, and interesting work to be done, is probably the area of making decisions, given predictions. So, I think a lot of machine learning is focused, correctly so, on giving good predictions. We are now kind of topped out performance on a lot of tasks that were considered very hard. Image recognition\u2026 in 2012 I think it was\u2026 or 2013 it was a really hard task. And now we are kind of like, we can achieve great performance. Get top one, top five percent performance. But I think the gap now is, ok, we got good predictions, how do we make decisions with those predictions? And I think with that, you need to have a notion of uncertainty, and well-calibrated uncertainty. So, you need to be certain the correct percentage of the time, and then uncertain the rest. And, you know, self-driving cars and all these things will need the notion of raising the red flag and saying, \u201cI don\u2019t know what\u2019s going on. I need to not make a decision right now. Please intervene.\u201d You need a notion of a low confidence prediction, \u201cPlease do something else. Don\u2019t use me for your decision.\u201d And beyond. But in general, there is this notion that you need uncertainty. And we are in a decent spot for quantifying uncertainty, but there is a lot more work that needs to be done to have safe, robust, machine learning systems.<\/p>\n<p><strong>Host: So that\u2019s a fruitful line of inquiry for somebody who\u2019s interested in this?<\/strong><\/p>\n<p>Nicolo Fusi: Yes. I think, you know, as a student you need to kind of imagine like skeet shooting. You need to shoot ahead of your target. If you\u2019re entering the game now, and you\u2019re trying to just maximize predictive accuracy, where predictive accuracy is basically like a root mean square. Minimizing a root mean and maximizing accuracy. I think it\u2019s about time to do machine learning if that\u2019s your objective. But I think, thinking more end-to-end, \u201cwhat is the end goal of this machine learning system\u201d is going to be a much more interesting area in the future.<\/p>\n<p><strong>(music plays)<\/strong><\/p>\n<p><strong>Host: Nicolo Fusi, thank you so much for joining us today on the podcast. I\u2019m enlightened, and it was so much fun.<\/strong><\/p>\n<p>Nicolo Fusi: Thanks for having me. It was fun.<\/p>\n<p><strong>To learn more about Dr. Nicolo Fusi and the latest research in Automated Machine Learning, visit <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/\">Microsoft.com\/research<\/a>.<\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Episode 43, September 26, 2018 &#8211; Dr. Nicolo Fusi gives us an inside look at Automated Machine Learning \u2013 Microsoft\u2019s version of the industry\u2019s AutoML technology \u2013 and shares the story of how an idea he had while working on a gene editing problem with CRISPR\/Cas9 turned into a bit of a machine learning side quest and, ultimately, a surprisingly useful instantiation of Automated Machine Learning &#8211; now a feature of Azure Machine Learning &#8211; that reduces dependence on intuition and takes some of the tedium out of data science at the same time.<\/p>\n","protected":false},"author":37583,"featured_media":507149,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"https:\/\/player.blubrry.com\/id\/38000600\/","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"categories":[240054],"tags":[],"research-area":[13556],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-507143","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-msr-podcast","msr-research-area-artificial-intelligence","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"https:\/\/player.blubrry.com\/id\/38000600\/","podcast_episode":"","msr_research_lab":[199563],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[330695],"related-projects":[545241,292757],"related-events":[],"related-researchers":[],"msr_type":"Post","featured_image_thumbnail":"<img width=\"960\" height=\"540\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/09\/NicoloFusi_Pod_Site_09_2018_1400x788.png\" class=\"img-object-cover\" alt=\"Dr. Nicolo Fusi Podcast headshot\" decoding=\"async\" loading=\"lazy\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/09\/NicoloFusi_Pod_Site_09_2018_1400x788.png 1400w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/09\/NicoloFusi_Pod_Site_09_2018_1400x788-300x169.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/09\/NicoloFusi_Pod_Site_09_2018_1400x788-768x432.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/09\/NicoloFusi_Pod_Site_09_2018_1400x788-1024x576.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/09\/NicoloFusi_Pod_Site_09_2018_1400x788-1066x600.png 1066w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/09\/NicoloFusi_Pod_Site_09_2018_1400x788-655x368.png 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/09\/NicoloFusi_Pod_Site_09_2018_1400x788-343x193.png 343w\" sizes=\"(max-width: 960px) 100vw, 960px\" \/>","byline":"","formattedDate":"September 26, 2018","formattedExcerpt":"Episode 43, September 26, 2018 - Dr. Nicolo Fusi gives us an inside look at Automated Machine Learning \u2013 Microsoft\u2019s version of the industry\u2019s AutoML technology \u2013 and shares the story of how an idea he had while working on a gene editing problem with&hellip;","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/507143"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/37583"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=507143"}],"version-history":[{"count":12,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/507143\/revisions"}],"predecessor-version":[{"id":543396,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/507143\/revisions\/543396"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/507149"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=507143"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=507143"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=507143"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=507143"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=507143"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=507143"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=507143"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=507143"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=507143"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=507143"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=507143"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}