{"id":591532,"date":"2019-06-12T07:59:49","date_gmt":"2019-06-12T14:59:49","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=591532"},"modified":"2022-11-07T11:52:49","modified_gmt":"2022-11-07T19:52:49","slug":"all-data-ai-with-dr-andrew-fitzgibbon","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/podcast\/all-data-ai-with-dr-andrew-fitzgibbon\/","title":{"rendered":"All Data AI with Dr. Andrew Fitzgibbon"},"content":{"rendered":"<h3><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/06\/Andrew-Fitzgibbon_Podcast_Site_06_2019_1400x788.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-591619 size-large\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/06\/Andrew-Fitzgibbon_Podcast_Site_06_2019_1400x788-1024x576.png\" alt=\"All Data AI with Dr. Andrew Fitzgibbon\" width=\"1024\" height=\"576\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/06\/Andrew-Fitzgibbon_Podcast_Site_06_2019_1400x788-1024x576.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/06\/Andrew-Fitzgibbon_Podcast_Site_06_2019_1400x788-300x169.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/06\/Andrew-Fitzgibbon_Podcast_Site_06_2019_1400x788-768x432.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/06\/Andrew-Fitzgibbon_Podcast_Site_06_2019_1400x788-1066x600.png 1066w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/06\/Andrew-Fitzgibbon_Podcast_Site_06_2019_1400x788-655x368.png 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/06\/Andrew-Fitzgibbon_Podcast_Site_06_2019_1400x788-343x193.png 343w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/06\/Andrew-Fitzgibbon_Podcast_Site_06_2019_1400x788.png 1400w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/a><\/h3>\n<h3>Episode 80, June 12, 2019<\/h3>\n<p>You may not know who <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/awf\/\">Dr. Andrew Fitzgibbon<\/a> is, but if you\u2019ve watched a TV show or movie in the last two decades, you\u2019ve probably seen some of his work. An expert in 3D computer vision and graphics, and head of the new <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/group\/ada\/\">All Data AI group<\/a> at <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/lab\/microsoft-research-cambridge\/\">Microsoft Research Cambridge<\/a>, Dr. Fitzgibbon was instrumental in the development of Boujou, an Emmy Award-winning 3D camera tracker that lets filmmakers place virtual props, like the floating candles in Hogwarts School for Witchcraft and Wizardry, into live-action footage. But that was just his warm-up act.<\/p>\n<p>On today\u2019s podcast, Dr. Fitzgibbon tells us what he\u2019s been working on since the Emmys in 2002, including body- and hand-tracking for powerhouse Microsoft technologies like <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/azure.microsoft.com\/en-us\/services\/kinect-dk\/\">Kinect<\/a> for <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/www.xbox.com\/en-US\/\">Xbox 360<\/a> and <a href=\"https:\/\/www.microsoft.com\/en-us\/hololens\">HoloLens<\/a>, explains how research on dolphins helped build mathematical models for the human hand, and reminds us, once again, that the \u201csecret sauce\u201d to most innovation is often just good, old-fashioned hard work.<\/p>\n<h3>Related:<\/h3>\n<ul type=\"disc\">\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/podcast\">Microsoft Research Podcast<\/a>: View more podcasts on Microsoft.com<\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/itunes.apple.com\/us\/podcast\/microsoft-research-a-podcast\/id1318021537?mt=2\">iTunes<\/a>: Subscribe and listen to new podcasts each week on iTunes<\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/subscribebyemail.com\/www.blubrry.com\/feeds\/microsoftresearch.xml\">Email<\/a>: Subscribe and listen by email<\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/subscribeonandroid.com\/www.blubrry.com\/feeds\/microsoftresearch.xml\">Android<\/a>: Subscribe and listen on Android<\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/open.spotify.com\/show\/4ndjUXyL0hH1FXHgwIiTWU\">Spotify<\/a>: Listen on Spotify<\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/www.blubrry.com\/feeds\/microsoftresearch.xml\">RSS feed<\/a><\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/note.microsoft.com\/ww-registration-microsoft-research-newsletter-s.html?wt.mc_id=S-webpage_podcast\">Microsoft Research Newsletter<\/a>: Sign up to receive the latest news from Microsoft Research<\/li>\n<\/ul>\n<hr>\n<h3>Transcript<\/h3>\n<p>Andrew Fitzgibbon: I do believe that there will be a future where we find it weird that we used to carry a flat screen in our pocket and pull out this flat screen to look at it in order to do our digital work. I think that sometime soon, when I refit my office, instead of setting up a bunch of LCD panels, I\u2019ll just put a large black curved piece of plywood in front of me, and I\u2019ll wear a HoloLens, and all my documents will appear in the real world in front of me. I absolutely believe that the screen in the pocket, or the screen attached to the desk, is going to be as weird as the phone attached to the building.<\/p>\n<p><strong>Host: You\u2019re listening to the Microsoft Research Podcast, a show that brings you closer to the cutting-edge of technology research and the scientists behind it. I\u2019m your host, Gretchen Huizinga.<\/strong><\/p>\n<p><strong>Host: You may not know who Dr. Andrew Fitzgibbon is, but if you\u2019ve watched a TV show or movie in the last two decades, you\u2019ve probably seen some of his work. An expert in 3D computer vision and graphics, and head of the new All Data AI group at Microsoft Research Cambridge, Dr. Fitzgibbon was instrumental in the development of Boujou, an Emmy Award-winning 3D camera tracker that lets filmmakers place virtual props, like the floating candles in Hogwarts School for Witchcraft and Wizardry, into live-action footage. But that was just his warm-up act.<\/strong><\/p>\n<p><strong>On today\u2019s podcast, Dr. Fitzgibbon tells us what he\u2019s been working on since the Emmys in 2002, including body- and hand-tracking for powerhouse Microsoft technologies like Kinect for Xbox 360 and HoloLens, explains how research on dolphins helped build mathematical models for the human hand, and reminds us, once again, that the \u201csecret sauce\u201d to most innovation is often just good, old-fashioned hard work. That and much more on this episode of the Microsoft Research Podcast.<\/strong><\/p>\n<p><strong>Host: Andrew Fitzgibbon, welcome to the podcast.<\/strong><\/p>\n<p>Andrew Fitzgibbon: Hi, Gretchen, great to be here.<\/p>\n<p><strong>Host: So, I usually start my podcasts by way of introduction, but I want to go a little off script with you because you\u2019re funny. You\u2019ve said that you situate your research at the intersection of computer vision and computer graphics with excursions into neuroscience. And at a more subatomic level, you characterize your work as, at its core, extracting information about the world from photons. So, give us a little more context about these excursions and extractions. How, in general, would you describe what you do, Andrew? What gets you up in the morning?<\/strong><\/p>\n<p>Andrew Fitzgibbon: So, what I want to do is make computers help us change the world. And we can change the world in a bunch of ways that are useful to humans. I\u2019d like to think that I work on the technologies that underpin the ways we can make computers make the world better for us. A lot of what we do in what we now call AI \u2013 AI, the combination of machine learning, computer vision, natural language processing \u2013 it\u2019s all about taking information from the world, and that information comes through sensors somehow, whether the sensors are the pixels in a camera, or the sensors in a microphone, or even the sensors in your keyboard when you type a tweet. These are all sources of information from the real world that we would like to do something good with. And for me, something good might be improving the computer graphics in Harry Potter. And I consider that good because it makes lots of humans happy. And in some sense, everything we do in life is about making humans happy.<\/p>\n<p><strong>Host: Well, tell me a little more about the excursions into neuroscience. When you\u2019re taking about computer vision and computer graphics, it might not seem a natural leap, but I think it kind of is. Can you unpack that a little?<\/strong><\/p>\n<p>Andrew Fitzgibbon: Yes, I started collaborating when I was in Oxford with some of the neuroscientists who worked there. And we were interested in a fairly simple question, which is roughly, can humans point at things? Now, you think, obviously humans can point at things, but we don\u2019t have a good theory for how the brain integrates 3D information. So, I was interested in purely providing the neuroscientists with a sort of a mathematical backup. I was saying, no one thinks the brain does it by multiplying matrices and vectors, but if it did, you would see these sort of error patterns. And then we went to the real world and looked at people wearing virtual reality headsets and the kind of pointing mistakes they make. And those are different from the patterns that a computer would make using today\u2019s technology. So, we still don\u2019t know what the brain does, but we have more evidence that it\u2019s not the same as what the computer does.<\/p>\n<p><strong>Host: Hmm. All right, so let\u2019s talk a little more about extracting information about the world from photons. That\u2019s a very granular way of describing the computer vision work that you do. Could you explain that, technically, a little better?<\/strong><\/p>\n<p>Andrew Fitzgibbon: So, regular listeners to your podcast will know that computer vision is one of those problems that\u2019s sort of easy to state, you know, does this picture contain a cat or not? But has turned out to be incredibly hard. And even in this age of deep learning successes with computer vision, we still know that it\u2019s incredibly hard. So, my abstraction of \u201cgather information from photons\u201d is just to kind of stand back and say, why are we so excited about computer vision? Why are we so excited about these capabilities? And one of the things that computer vision does is it allows us to acquire information from far away. And of course, one can hear information from far away. But it\u2019s a capability that allows us to do things like recognize people far away or to drive a car. And I want to think about it at that level of abstraction, because I want to always understand that my end goal is to do something real. If we think always about the end goal, I think we do a much better job of making progress on the fundamental research. The alternative is to say my interest is in understanding deep neural networks. I absolutely love trying to understand deep neural networks, and I do so at a theoretical level. But also, I want to know what the practical consequence of that understanding will be.<\/p>\n<p><strong>Host: Well, now that we\u2019ve situated your research, let\u2019s situate you. Until this week, I would have introduced you as a partner scientist in HoloLens, but you\u2019ve recently been tapped to lead a group at MSR Cambridge called All Data AI, or ADA, which turns out to be an acronym that references Ada Lovelace. Was that intentional at all, and why, if it was?<\/strong><\/p>\n<p>Andrew Fitzgibbon: So yes, it was intentional in the sense that Ada Lovelace is sometimes considered the world\u2019s first computer programmer. Whether or not she programmed computers, she certainly was the first person to observe that the computational powers of the analytical engine, or the difference engine, could be applied to quantities that were not numbers. Now, of course since the dawn of computers, we\u2019ve represented quantities in the computer, like strings of characters, like words. They\u2019re all numbers. So, what\u2019s new about saying that it operates on something beyond numbers? Well, the thing that\u2019s new is not so much that the computer understands a sequence of numbers, but that it can understand the interconnections between them and that is represented, fundamentally, by a computer science concept called a graph. And a graph is something that we can use to represent a wide variety of computer science concepts.<\/p>\n<p><strong>Host: Okay. So why now with this group at MSR Cambridge? Has there been a confluence of things that exist now that didn\u2019t before?<\/strong><\/p>\n<p>Andrew Fitzgibbon: I think, of course, we have all observed the advances in AI due to the advances in machine learning, due to the advances in deep learning. And I think now, there\u2019s a golden opportunity to apply that to a broader range of areas. It\u2019s also a fantastic opportunity, here and now, for us to think about a new generation of AI programming. Until today, AI programming has been the domain of high priests and priestesses who have PhDs in machine learning and who understand linear algebra. And yet, we believe that, actually, a lot of AI programming could be a lot simpler. So, we\u2019re looking at how we might think about a third generation of AI programming. The first generation would have been the raw work of Hinton and LeCun who just wrote the code, you know, it was difficult code to write. The second generation is this set of tools which go by the name of TensorFlow and PyTorch, again, which lots of listeners will have used or heard of.<\/p>\n<p><strong>Host: Right.<\/strong><\/p>\n<p>Andrew Fitzgibbon: And these tools have been, again, fantastic for democratizing AI. But they make it hard for somebody who\u2019s just a great computer programmer to understand.<\/p>\n<p><strong>Host: Hmm.<\/strong><\/p>\n<p>Andrew Fitzgibbon: In my view, they somewhat hide the beauty of the AI models that are underneath. Neural network models are actually relatively simple. They\u2019re simple models with complex consequences, and the research world is increasing its understanding of these complex consequences, but it\u2019s also nice to just see the code, unadorned, and see how simple these models are.<\/p>\n<p>(music plays)<\/p>\n<p><strong>Host: Well, let\u2019s go back a ways, before we talk about more current research, because you have some, what I would call \u201cgreatest hits\u201d in your earlier work. One is called Boujou, a camera tracker that actually won an Emmy. And it\u2019s been used in computer graphics and live-action footage in, what I\u2019ve heard is pretty much every movie made since it was released in 2002. Tell us more about Boujou. How and where did it come about, what does it do, and how does it work technically?<\/strong><\/p>\n<p>Andrew Fitzgibbon: Boujou was a great fun project. I moved from the University of Edinburgh, where I did my PhD, to Oxford University in the mid-nineties. And I was working there with some amazing people who were interested in the question of how a robot might navigate its way around some building or around some environment. So, we worked hard on the \u201chow does a robot navigate?\u201d problem. And we discovered that one of the things the robot has to do in order to know where it\u2019s been in three dimensions, is to build a three-dimensional model of the world. And we worked hard on making a beautiful 3D model, because we figured this would be useful, maybe, for those industries where, even today, it\u2019s common for example to sculpt a car out of clay before building the computer model. Certainly, in those days, if you were going to have an alien in the movie \u2013 I think Pitch Black was one of the first aliens we looked at \u2013 they would make the alien out of clay and then try to scan it into the computer. So, we thought we had a great product. We were going to use our robot navigator, spin the camera around the alien model, pull that into the computer, and then give people a computer model of the 3D object. It turned out nobody in the movie industry was interested.<\/p>\n<p><strong>Host: Dang it.<\/strong><\/p>\n<p>Andrew Fitzgibbon: You had to \u2013 yeah, yeah! You had to spray paint the model with a toothbrush to get texture on it. It didn\u2019t work in little corners. It was terrible. And they could just do better by using, you know, existing artists to create the models. But somebody at one of the effects companies kind of thought through how we must be getting this model, and said, how do you know where the cameras are? And we said, you know, well, it works it out. That\u2019s part of how we get the 3D model. We work out how the cameras are. So, it turns out, something you really need in movies that\u2019s really hard to do is to figure out where the cameras are. Now, what does that mean? That means maybe I have a camera mounted in a boat. And the boat contains some hobbits and is sailing down a river towards, you know, some impressive mountain. It would be just great if the impressive mountain had two huge statues pointing at the hobbits, but it turns out that nobody built those 400-meter statues, so what we have to do is build some statues back in the studio \u2013 and we built some 40-centimeter statues back in the studio \u2013 and if only we knew what motion the camera had undergone while it rocked in that boat moving down the river, we\u2019d be able to make a robot do that same motion and then we could superimpose the images.<\/p>\n<p><strong>Host: Hmm.<\/strong><\/p>\n<p>Andrew Fitzgibbon: So, in those days, the only way to find out where the camera was, was \u2013 you could imagine trying to use markers or trying to use GPS\u2026 None of these things work. So, you would have to just sort of manually position the camera for every single shot. It was incredibly expensive. So, somebody who had worked on that kind of footage realized that our algorithms were producing this as a kind of unwanted side effect. We flipped \u2013 I guess nowadays, we would say we pivoted \u2013 uh, there was a startup, and Boujou was launched. And I\u2019m super proud of it because it was one of the first products which did kind of 3D computer vision. There were products available then that would do character recognition, that would do number plate recognition, but this really did 3D vision. And we had to go beyond the academic state-of-the-art to deliver it. And we learned about delivering computer vision to the real world. Some of what we learned was, you just have to type in lots of code. People would ask me, you know, what\u2019s the secret sauce in Boujou? And I would say, well, you know, have you read all the papers on 3D structure and motion? They would say, yeah, I\u2019m pretty sure of them. I would say, yeah, that\u2019s the secret sauce. We implemented all of them. And, you know, it\u2019s an attitude that helps us today. Sometimes in research, you\u2019re assessed by how beautiful and clean your algorithm is. Sometimes in the real world, you have to implement all the dirty algorithms until you find the beautiful one. But with Boujou, it just worked, and we had something that was actually useful.<\/p>\n<p><strong>Host: You know, on that note, I was going to ask you a question further down in the podcast, but I\u2019m going to bump it up because it\u2019s sort of tied in. I imagine you said it about this. You once said, \u201cIf I had to nominate one key to success, it\u2019s a focus on everything.\u201d Is that what you\u2019re talking about?<\/strong><\/p>\n<p>Andrew Fitzgibbon: That\u2019s exactly what I\u2019m talking about. Yeah, exactly. There is no silver bullet. You know, you really have to focus on building a real thing. And that just means something that other humans would be happy to use. So, remember, in Boujou, we didn\u2019t focus on the right thing, but the real thing we wanted to build was this 3D modeler. And we knew we would make a sort of a, what\u2019s called a Wizard of Oz demo. We\u2019d say, if this thing worked, it would make you a model that looks like this. Would you like it? And then, when the humans agree that that\u2019s what they would like, then you have a target. And then of course, you may have to pivot, or you may deliver something that\u2019s only half as good and discover that hey, that\u2019s still useful. Or you may achieve everything you thought you needed to achieve and discover that actually, it needs to be twice as good. But I really like having a concrete goal to aim for.<\/p>\n<p><strong>Host: Well that\u2019s a beautiful segue into the next topic I want to talk about, because Microsoft\u2019s Kinect technology has been dubbed a failure that became a great success, and the science behind it has a fascinating history. You were there early on, and I\u2019d love you to share some stories about Kinect with us. What\u2019s your particular perspective on this technology? How did it come about, how has it evolved, and how has it impacted other areas of research you\u2019re involved in?<\/strong><\/p>\n<p>Andrew Fitzgibbon: I love the Kinect story, because, in some sense, it\u2019s a classic example of when academic style researchers meet engineers who really want to change the world. So, at Microsoft Research, we were looking at whether we could make computer vision algorithms that would be able to follow the movements of the human body. And we are working with the academic research field and we were doing pretty well, and we had good results. So one day, the Xbox people, Alex Kipman, who was working in Xbox then, came to us and said, we\u2019ve had this great idea for a video game where you\u2019re going to like, uh, recognize the motion of a human in a camera, and then it\u2019s going to control the games, and it\u2019s going to be amazing. And we said, I\u2019m glad you asked us that, because we\u2019re actually the world experts on this, and I can tell you it\u2019s not going to work. So, then they said, oh, yeah, it\u2019s funny you say that, because look at this program we wrote. And they showed us their version of it, and their version was better than anything in the academic literature.<\/p>\n<p><strong>Host: Oh my gosh.<\/strong><\/p>\n<p>Andrew Fitzgibbon: Some genius programmers had put together an amazing demo of how it would work. What was amazing was, it was using an idea from the academic literature, but they had engineered it so well and made it so effective and really worked hard on it.<\/p>\n<p><strong>Host: Hmm.<\/strong><\/p>\n<p>Andrew Fitzgibbon: The reason that idea wasn\u2019t very popular with academics at the time was that it\u2019s pretty hard to take a single image from a camera, identify the human in the image, and then list off where the hands are, where the elbows are\u2026 But supposing you already had an image from, let\u2019s say, 30 milliseconds ago because it\u2019s a video sequence, and you already knew, in the image from 30 milliseconds ago, where everything was, then you could just simply say to yourself, well, they can\u2019t have gone far; they were there 30 milliseconds ago, you know, and find where everything was. So, we knew how to do that in academia, but what no one had done is really, really worked hard on that, really kind of tried to moonshot that and make it really work. And our contribution was to observe that, okay, you\u2019ve got a system that works 99 percent of the time, assuming it was right 30 milliseconds ago. There\u2019s a calculation you can do which tells you that system will definitely fail after five minutes. And this is what they observed, and they knew that that would happen. And you could design, again, around that, you know, ran in sort of three-minute sections and then reset itself. But, ideally, you would have the system just not make mistakes like that. So, our first contribution was to say, what we need to do here is just basically have the system every couple of seconds, kind of reset itself. And we were using machine learning, and this was kind of an early instance of machine learning really being applied to one of these hard computer vision problems. So, we said to them, okay, we could maybe do it, maybe\u2026 But, you know, we would need real-world examples of this thing running in ten different living rooms across the planet in order to even know if we\u2019re doing well, not to mind train our machine learning algorithm. And then the horrifying moment two weeks later when we were on a call and they said, yup, we\u2019ve got ten people in living rooms across the planet! Our Japan people are finishing up there tomorrow, and then they\u2019re moving over to China\u2026 so suddenly we realized, okay, these people are really serious. And then, when we needed to hire a Hollywood studio to generate training data, they hired a Hollywood studio to generate training data. So, there was just a huge amount of vision there which\u2026 we were saying this stuff, because it seemed right, but really, no one had done it to that level before, and the reason was, they understood what the machine learning was doing. They figured it works from examples. If you don\u2019t have enough examples, no brainer, let\u2019s get the examples. Whereas, academics would always be, I\u2019m happy to spend a year building a better theory. Whereas with Kinect, the partnership with people who really wanted to get stuff done \u2013 maybe that\u2019s where I\u2019ve inherited some of this \u2013 allowed us to make really fantastic progress.<\/p>\n<p>(music plays)<\/p>\n<p><strong>Host: Microsoft\u2019s HoloLens is another computer vision technology that you\u2019ve had a lot to do with. So, give us the \u201cAndrew Fitzgibbon take\u201d on HoloLens and its journey from birth until now, with the release of HoloLens 2. What have you discovered about its capabilities over the years, and what do you think HoloLens has contributed to the computer vision research community?<\/strong><\/p>\n<p>Andrew Fitzgibbon: HoloLens is an amazing device. It came out of the same team that we worked with on Kinect, Alex Kipman\u2019s team. And, at one level, HoloLens is exactly what Kipman said it was when they announced it first, three, maybe four years ago. He said, this is the future of the PC. And in one sense, I do believe that there will be a future where we find it weird that we used to carry a flat screen in our pocket and pull out this flat screen to look at it in order to do our digital work. I think that sometime soon, when I refit my office, instead of setting up a bunch of LCD panels, I\u2019ll just put a large black curved piece of plywood in front of me, and I\u2019ll wear a HoloLens, and all my documents will appear in the real world in front of me. I absolutely believe that the screen in the pocket, or the screen attached to the desk, is going to be as weird as the phone attached to the building. So that\u2019s far future. That\u2019s when HoloLens has the form factor of a small set of glasses. But towards that future, HoloLens today is amazingly valuable for real people doing real work. Because HoloLens lives in a 3D world, it has lots of 3D vision in it. One of the pieces that I\u2019m incredibly impressed by in HoloLens because I didn\u2019t work on it, does the job of figuring out where your head is in the 3D world. This is related to work I did, you know, on Boujou many years ago. But on HoloLens, it does it all the time, in real time, on an incredibly low-power device, so that\u2019s a beautiful piece of technology that again, I think very few people other than Microsoft could have put together. My work on HoloLens stemmed from a piece of very blue skies research we did almost ten years ago now. We \u2013 that\u2019s me and a friend called Tom Cashman, who arrived as an intern \u2013 we decided we wanted to learn about the 3D structure of stuff that moves. What\u2019s stuff that moves? Well, the human body is something that moves, and we knew a bit about the 3D structure of the human body from Kinect. But we wanted to learn about this 3D structure just from still images. So, we had to think of something that\u2019s kind of bendy and movey, but that is somehow not too bendy and not too movey. So, we realized that dolphins were the ideal thing to work on. So, we decided to write a paper called, What Shape are Dolphins? Now, we didn\u2019t really care about dolphins. But we cared about bendy, movey 3D stuff. And we worked on that paper in order to build mathematical models of 3D. We thought, well, why did we want to know about 3D? Well, when you\u2019re interacting with the virtual world, or the mixed reality world and the HoloLens, one of the important 3D objects is the human hand. If the system can look at your hands and fully accurately determine the position of every bone and knuckle in the hand, then you can use your hands in the virtual world to pick up virtual objects. And the virtual objects behave exactly as you would expect from real-world physics. So, the research on dolphins became a research on the slightly more bendy object that is the human hand. And then we knew that this technology was useful for something. HoloLens was being developed. So, we thought, let\u2019s see if we can deliver this dream of real-world physics to the HoloLens. And we were extremely happy to see announced recently at Mobile World Congress in Barcelona, HoloLens 2 with fully articulated hand-tracking.<\/p>\n<p><strong>Host: Let\u2019s shift over and talk about ADA a little bit more deeply and talk about specific problems you want to tackle and what technical ground you hope to break in the projects you want to take on. What\u2019s on your roadmap, Andrew?<\/strong><\/p>\n<p>Andrew Fitzgibbon: So, ADA, is about permeating AI into the parts of the world where we almost don\u2019t yet know we need it. Today, we still can\u2019t ask questions of the internet like, find me all the ski chalets within a hundred meters of the slopes, right? That\u2019s a hard question. Why is it a hard question? Because all the ski chalets have their own little website. It\u2019s not necessarily aggregated. And the way to answer that question is very easy: you should just read and understand all the web pages in the world and be able to answer any questions about them. So, a computer that can really read and understand all the web pages in the world is clearly, in some sense, you know, sort of infinitely far away. So, the idea is that throughout computer science, there are areas where we can permeate, using AI and machine learning, to deliver systems that work better. So, for example, on the HoloLens, we had to take our hand-tracking code which we wrote as standard, you know, computer vision researcher code, and make it maybe five hundred times more efficient in order to run on that low-power device.<\/p>\n<p><strong>Host: Right.<\/strong><\/p>\n<p>Andrew Fitzgibbon: Now, if we could achieve that for general code, then maybe we could make, you know, five hundred times more efficient the code we run in datacenters around the world. Maybe we could achieve five hundred times as much and maybe we could save energy. If we had capabilities in user interfaces that allowed us to adapt to what the user is doing, but not be annoying \u2013 and this is a crucial combination \u2013 then of course, we would have much better capabilities, of course we would have happier users, happier humans. If we think about the challenge of maybe adapting user interfaces so that my user interface works better for me and yours works better for you, one of the things that we want to do is learn from a very small number of signals. So maybe, every time you switch on your computer, you know, you arrange the windows in a certain order. We already have systems that will try to learn your preferences and try to, uh, do that for you. But what we know today is that a lot of the times, the systems that do that are a bit annoying. Why are they a bit annoying? Because they don\u2019t have the human-level understanding. Or this person\u2019s in a hurry. This person\u2019s using a different machine. They\u2019re in a different context. They\u2019re at home, not work. How could a system really understand, from small amounts of training data, what it should do and when it should do nothing? And these are areas which I think we\u2019re sort of touching on today. There are whole areas of AI research that our team looks at, and they\u2019re different questions. In all of them, again, does this tradeoff between a fundamental research angle and how are we actually going to demonstrate our AI to the world?<\/p>\n<p>(music plays)<\/p>\n<p><strong>Host: So, Andrew, I\u2019ve heard you use the phrase, change the world, or a better world. And, and these are the sort of end goals. So, there are a lot of things that could go right if you\u2019re successful. But we have to talk, at least a little bit, about what could possibly go wrong. So, with all of the possibilities that you\u2019re describing, they seem to pose new kinds of social protocols that we\u2019ll have to develop and adapt with things like advanced 3D computer vision and head-mounted computers and cameras galore\u2026 Is there anything that keeps you up at night?<\/strong><\/p>\n<p>Andrew Fitzgibbon: There are things where I think I have some idea what the answer\u2019s going to be, and then there are things where I hope I have some idea what the answer\u2019s going to be! Let me do one of the first ones. When I describe a world where everybody\u2019s glasses have the possibility of projecting 3D content into the world in front of them, you can immediately think, well, this is terrible. People are going to just, you know, spend their entire time looking at the content and not interacting with the other person in front of them. But I think we already have social protocols that solve that today. So, if you and I are talking and there\u2019s a TV over to the right, maybe I\u2019ll turn to look at the TV. It\u2019s rude, but maybe you understand something interesting must have happened on the TV and we both turn and maybe it\u2019s something we\u2019ll discuss, and we\u2019ll get back to chatting. We also have sort of protocols, it\u2019s rude for me to take out my phone and look at it when I\u2019m talking with you. But you understand if I tell you, well, I\u2019m just going to look that up or I\u2019m just going to figure out what\u2019s happening in that meeting.<\/p>\n<p><strong>Host: Mm-hmm.<\/strong><\/p>\n<p>Andrew Fitzgibbon: So, I\u2019m not worried, with something like HoloLens, that we won\u2019t easily develop these protocols. We might need some technical solutions like, when there\u2019s something in my 3D world, maybe it\u2019s, you know, my email client is maybe sitting on the desk in front of me. You can see that it\u2019s my email client, but of course you can\u2019t read the content within it. So&#8230; But these are technical problems we can solve. The social protocols, I\u2019m confident, there, we will develop. And I\u2019m not concerned about that aspect. But you might well say, okay, but you should be concerned about maybe a world where the cameras on my device are revealing information that I wouldn\u2019t like revealed to other people in the world. And I\u2019m happy that we, the AI community, essentially, are talking about this and thinking about ways in which we can ensure the security of our data so that any person can have a concrete understanding about what information\u2019s leaving the device, how well encrypted information is used, and that we can all have protocols for how we dispose of information when we don\u2019t need it. Another exciting thing about the HoloLens is that, for example, in hand-tracking, none of the images that the cameras take leave the device. And today, we are looking at a number of ways in which I can securely send your information to the cloud, securely do machine learning on it and send the answer back, and then delete the information and guarantee that to the customer. So that\u2019s one aspect where we, as Microsoft, have just devoted an awful lot of effort to thinking about security and privacy of that information. Another side of security is trust. We earn that trust obviously by, you know, having behaved well in the past, by having strong statements about what we\u2019re going to keep and what we\u2019re not going to keep. And also, by the strength of our research in aspects like secure computation and secure machine learning.<\/p>\n<p><strong>Host: It\u2019s story time, and you\u2019ve got a good one. Tell us what got a young Andrew Fitzgibbon interested in computer science, and what was your path to Microsoft Research?<\/strong><\/p>\n<p>Andrew Fitzgibbon: Gosh. It\u2019s a long one\u2026 Um\u2026 I grew up in the eighties. I liked mathematics. I liked messing around with electronics, because we didn\u2019t really have computers then. We had computers at school. One of my summer jobs was as a water taxi driver. And being a water taxi driver means you\u2019re incredibly busy from about 9:00 a.m. to 11:00 a.m. when all the boats go out to the sea, and then you\u2019re incredibly busy from about 3:00 p.m. to about 6:00 p.m. or maybe 7:00 when everyone comes back. So, you\u2019ve got a huge amount of time sitting out in the sun, or maybe the rain, during the day where you\u2019re just kind of killing time. And one of the things you can do is read. And of course, if many of your listeners will love mathematics, you can play around with mathematics. But what was also great back then was, because you didn\u2019t have a computer beside you, I would write programs that I would then try and type in the following day when I got to school. And that was just a great way to sort of, in a very casual way, learn about computer programming. So, I would scratch out a program on paper, line ten, do this, line twenty, do that, or later I would write in assembly language the little bits that needed doing. And it\u2019s kind of therapeutic. It\u2019s like solving mathematical puzzles, right? Mathematical puzzles are great, but they\u2019re hard to invent for yourself. So, you might buy a book of mathematical puzzles and then solve them all after a few weeks. Whereas with a computer program, you\u2019re of course continually inventing the puzzles for yourself.<\/p>\n<p><strong>Host: Right. So, from that to job as a water taxi driver, then what, then where?<\/strong><\/p>\n<p>Andrew Fitzgibbon: So, I did mathematics and computer science at university. I wanted to do physics, but the physics department told me it would be much too hard to do physics with computer science. And someone told me later that also, they weren\u2019t that keen on people with blue hair. So, I ended up doing mathematics and computer science. The mathematics department were much more enlightened. And I was sitting in a topology lecture one day and thinking, what would this stuff be useful for? And I thought, oh, maybe it will be useful for recognizing the shapes of letters. And basically, I went and found myself a master\u2019s course that had something to do with computer vision. And I went to Heriot-Watt University in Edinburgh, which was running a fantastic course on sort of \u2013 they called it knowledge-based systems. Nowadays, you would think of it as an introductory AI course. So, I finished my master\u2019s, and then I got a job as a programmer in Edinburgh University. I got that job in 1989. But then, somehow by 1997, somewhat accidentally, I ended up with a PhD. I remember now why that happened. I was sitting in some meeting as the programmer, and they were talking about their research. And I said something like, well, has anybody tried just like rendering five hundred views of the thing and then using that? And someone said, oh, you should write a paper on that. And I thought it was a joke, and of course it turned out that that was my first paper. It was one of these things where, let\u2019s use brute force instead of mathematics and see what happens. And from then on, you know, I hope I\u2019ve managed to do elegant mathematics as well as brute force, but I think a practical mindset was there even from then.<\/p>\n<p><strong>Host: Well, sometimes you need tweezers and sometimes you need a hammer.<\/strong><\/p>\n<p>Andrew Fitzgibbon: Right, exactly, exactly.<\/p>\n<p><strong>Host: All right, so is that your accidental PhD? And then what?<\/strong><\/p>\n<p>Andrew Fitzgibbon: So, the PhD was part-time, with a research assistant job at Edinburgh. And that job was coming to an end, or at least I thought it was coming to an end because the contract was due to end at a certain date, and of course, like a very clever person, I forgot to ask anybody whether that was true, so I started looking for another job. I ended up in Oxford working with the great computer vision researcher of the UK, Andrew Zisserman, and a bunch of other amazing people. And there again, I learned the interaction of mathematics and code. I worked there. That led to Boujou, which we have mentioned earlier. And then, about mid-2000s, I moved to Microsoft Research.<\/p>\n<p><strong>Host: Right. Uh, you did a bit of a drive-by on a reference to blue hair. It was the eighties, but did you, indeed, have blue hair?<\/strong><\/p>\n<p>Andrew Fitzgibbon: Oh, yes, I indeed had blue hair, for much of my undergraduate career. It was varying degrees of blue and, and then sometimes would turn into green when the bleach wore out.<\/p>\n<p><strong>Host: Was it ever Smurf blue?<\/strong><\/p>\n<p>Andrew Fitzgibbon: No, it was kind of a dark, electric blue.<\/p>\n<p><strong>Host: Oh.<\/strong><\/p>\n<p>Andrew Fitzgibbon: At least that was the aim.<\/p>\n<p><strong>Host: Well, at the end of every podcast, I give my guests a chance to say anything they want to our listeners. And sometimes it\u2019s general advice or wisdom or inspiration. Other times, it\u2019s specific challenges or open problems in the field. So, what would you like to say to emerging researchers?<\/strong><\/p>\n<p>Andrew Fitzgibbon: People sometimes ask, what problem should I research? And, you know, there\u2019s sort of a simple, general answer: solve important problems that will change the world. But sometimes that\u2019s too big, and you don\u2019t really have an idea what the big, important problems are. A signal that I found valuable is, when you\u2019re listening to a talk or reading a paper, find something that annoys you. Find something where you think, really? That can\u2019t be the right way to do this. Then \u2013 and this is crucial \u2013 ask yourself really, why is this annoying me? And then find a real-world example where the thing that\u2019s annoying you is going to go wrong. A real-world example just needs to be something that you know we should be able to do, none of the existing technologies can do it, and you\u2019ve got an idea. And I think that has been a very valuable source of inspiration for the times when you don\u2019t have a big idea. Sometimes you just have a big idea, and that\u2019s great. Go for it.<\/p>\n<p><strong>Host: And don\u2019t forget to focus on everything.<\/strong><\/p>\n<p>Andrew Fitzgibbon: Focus on everything. That\u2019s \u2013 that\u2019s the best idea!<\/p>\n<p><strong>Host: Andrew Fitzgibbon, thank you for joining us from Cambridge today.<\/strong><\/p>\n<p>Andrew Fitzgibbon: Thank you. It\u2019s an honor to have been here.<\/p>\n<p>(music plays)<\/p>\n<p>To learn more about Dr. Andrew Fitzgibbon and the latest in 3D computer vision and All Data AI, visit <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/\">Microsoft.com\/research<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>You may not know who Dr. Andrew Fitzgibbon is, but if you\u2019ve watched a TV show or movie in the last two decades, you\u2019ve probably seen some of his work. An expert in 3D computer vision and graphics, and head of the new All Data AI group at Microsoft Research Cambridge, Dr. Fitzgibbon was instrumental in the development of Boujou, an Emmy Award-winning 3D camera tracker that lets filmmakers place virtual props, like the floating candles in Hogwarts School for Witchcraft and Wizardry, into live-action footage. But that was just his warm-up act.<br \/>\nOn today\u2019s podcast, Dr. Fitzgibbon tells us what he\u2019s been working on since the Emmys in 2002, including body- and hand-tracking for powerhouse Microsoft technologies like Kinect for Xbox 360 and HoloLens, explains how research on dolphins helped build mathematical models for the human hand, and reminds us, once again, that the \u201csecret sauce\u201d to most innovation is often just good, old-fashioned hard work.<\/p>\n","protected":false},"author":38022,"featured_media":591619,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"https:\/\/player.blubrry.com\/id\/44602199","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"footnotes":""},"categories":[240054],"tags":[],"research-area":[13561,13556,13562],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-591532","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-msr-podcast","msr-research-area-algorithms","msr-research-area-artificial-intelligence","msr-research-area-computer-vision","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"https:\/\/player.blubrry.com\/id\/44602199","podcast_episode":"","msr_research_lab":[199561],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[606351],"related-projects":[],"related-events":[593398],"related-researchers":[],"msr_type":"Post","featured_image_thumbnail":"<img width=\"960\" height=\"540\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/06\/Andrew-Fitzgibbon_Podcast_Site_06_2019_1400x788.png\" class=\"img-object-cover\" alt=\"Photo of Andrew Fitgibbon\" decoding=\"async\" loading=\"lazy\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/06\/Andrew-Fitzgibbon_Podcast_Site_06_2019_1400x788.png 1400w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/06\/Andrew-Fitzgibbon_Podcast_Site_06_2019_1400x788-300x169.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/06\/Andrew-Fitzgibbon_Podcast_Site_06_2019_1400x788-768x432.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/06\/Andrew-Fitzgibbon_Podcast_Site_06_2019_1400x788-1024x576.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/06\/Andrew-Fitzgibbon_Podcast_Site_06_2019_1400x788-1066x600.png 1066w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/06\/Andrew-Fitzgibbon_Podcast_Site_06_2019_1400x788-655x368.png 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/06\/Andrew-Fitzgibbon_Podcast_Site_06_2019_1400x788-343x193.png 343w\" sizes=\"(max-width: 960px) 100vw, 960px\" \/>","byline":"","formattedDate":"June 12, 2019","formattedExcerpt":"You may not know who Dr. Andrew Fitzgibbon is, but if you\u2019ve watched a TV show or movie in the last two decades, you\u2019ve probably seen some of his work. An expert in 3D computer vision and graphics, and head of the new All Data&hellip;","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/591532"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/38022"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=591532"}],"version-history":[{"count":6,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/591532\/revisions"}],"predecessor-version":[{"id":896355,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/591532\/revisions\/896355"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/591619"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=591532"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=591532"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=591532"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=591532"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=591532"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=591532"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=591532"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=591532"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=591532"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=591532"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=591532"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}