{"id":564210,"date":"2019-01-30T07:58:35","date_gmt":"2019-01-30T15:58:35","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=564210"},"modified":"2019-02-11T14:14:34","modified_gmt":"2019-02-11T22:14:34","slug":"enableing-people-to-do-more-with-dr-rico-malvar","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/podcast\/enableing-people-to-do-more-with-dr-rico-malvar\/","title":{"rendered":"Enable(ing) people to do more with Dr. Rico Malvar"},"content":{"rendered":"<div id=\"attachment_564216\" style=\"width: 1034px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-564216\" class=\"size-large wp-image-564216\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/01\/Rico-Malvar_POD_Site_01_2019_1400x788-1024x576.png\" alt=\"\" width=\"1024\" height=\"576\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/01\/Rico-Malvar_POD_Site_01_2019_1400x788-1024x576.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/01\/Rico-Malvar_POD_Site_01_2019_1400x788-300x169.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/01\/Rico-Malvar_POD_Site_01_2019_1400x788-768x432.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/01\/Rico-Malvar_POD_Site_01_2019_1400x788-1066x600.png 1066w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/01\/Rico-Malvar_POD_Site_01_2019_1400x788-655x368.png 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/01\/Rico-Malvar_POD_Site_01_2019_1400x788-343x193.png 343w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/01\/Rico-Malvar_POD_Site_01_2019_1400x788.png 1400w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><p id=\"caption-attachment-564216\" class=\"wp-caption-text\">Rico Malvar, Chief Scientist and Distinguished Engineer<\/p><\/div>\n<h3>Episode 61, January 30, 2019<\/h3>\n<p>From his deep technical roots as a principal researcher and founder of the Communications, Collaboration and Signal Processing group at MSR, through his tenure as Managing Director of the lab in Redmond, to his current role as Distinguished Engineer, Chief Scientist for Microsoft Research and manager of the MSR NExT <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/group\/enable\/\">Enable group<\/a>, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/malvar\/\">Dr. Rico Malvar<\/a> has seen \u2013 and pretty well done \u2013 it all.<\/p>\n<p>Today, Dr. Malvar recalls his early years at a fledgling Microsoft Research, talks about the exciting work he oversees now, explains why designing with the user is as important as designing for the user, and tells us how a challenge from an ex-football player with ALS led to a prize winning hackathon project and produced the core technology that allows you to type on a keyboard without your hands and drive a wheelchair with your eyes.<\/p>\n<h3>Related:<\/h3>\n<ul type=\"disc\">\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/podcast\">Microsoft Research Podcast<\/a>: View more podcasts on Microsoft.com<\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/itunes.apple.com\/us\/podcast\/microsoft-research-a-podcast\/id1318021537?mt=2\">iTunes<\/a>: Subscribe and listen to new podcasts each week on iTunes<\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/subscribebyemail.com\/www.blubrry.com\/feeds\/microsoftresearch.xml\">Email<\/a>: Subscribe and listen by email<\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/subscribeonandroid.com\/www.blubrry.com\/feeds\/microsoftresearch.xml\">Android<\/a>: Subscribe and listen on Android<\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/open.spotify.com\/show\/4ndjUXyL0hH1FXHgwIiTWU\">Spotify<\/a>: Listen on Spotify<\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/www.blubrry.com\/feeds\/microsoftresearch.xml\">RSS feed<\/a><\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/note.microsoft.com\/ww-registration-microsoft-research-newsletter-s.html?wt.mc_id=S-webpage_podcast\">Microsoft Research Newsletter<\/a>: Sign up to receive the latest news from Microsoft Research<\/li>\n<\/ul>\n<hr \/>\n<h3>Episode Transcript<\/h3>\n<p>Rico Malvar: At some point, the leader of the team, Alex Kipman, came to us and says, oh, we want to do a new controller. What if you just spoke to the machine, made gestures and we could recognize everything? You say, that sounds like sci-fi. And then we said, no, wait a second, but to detect gestures, we need specialized computer vision. We\u2019ve been doing computer vision for 15 years. To identify your voice, we need speech recognition. We\u2019ve also been doing speech recognition for 15 years. Oh, but now there maybe be other sounds and multiple people\u2026 oh, but just a little over 10 years ago, we started these microphone arrays. They are acoustic antennas. And I said, wait a second, we actually have all the core elements, we could actually do this thing!<\/p>\n<p><strong>Host: You\u2019re listening to the Microsoft Research Podcast, a show that brings you closer to the cutting-edge of technology research and the scientists behind it. I\u2019m your host, Gretchen Huizinga.<\/strong><\/p>\n<p><strong>Host: From his deep technical roots as a principal researcher and founder of the Communications, Collaboration and Signal Processing group at MSR, through his tenure as Managing Director of the lab in Redmond, to his current role as Distinguished Engineer, Chief Scientist for Microsoft Research and manager of the MSR NExT Enable group, Dr. Rico Malvar has seen \u2013 and pretty well done \u2013 it all.<\/strong><\/p>\n<p><strong>Today, Dr. Malvar recalls his early years at a fledgling Microsoft Research, talks about the exciting work he oversees now, explains why designing with the user is as important as designing for the user, and tells us how a challenge from an ex-football player with ALS led to a prize winning hackathon project and produced the core technology that allows you to type on a keyboard without your hands and drive a wheelchair with your eyes. That and much more on this episode of the Microsoft Research Podcast.<\/strong><\/p>\n<p><strong>Host: Rico Malvar, welcome to the podcast.<\/strong><\/p>\n<p>Rico Malvar: It\u2019s a pleasure to be with you, Gretchen.<\/p>\n<p><strong>Host: You\u2019re a Distinguished Engineer and Chief Scientist at Microsoft Research. How would you define your current role? What gets you up in the morning?<\/strong><\/p>\n<p>Rico Malvar: Ha ha! Uh, yeah, by chief scientist, it means I tell everybody what to do, very simple. (laughing) Yeah\u2026 Not really, but Chief Scientist is basically a way for me to have my fingers and eyes, in particular, on everything going on at Microsoft Research. So, I have an opportunity to interact with, essentially, all the labs, many of the groups, and find opportunities to do collaborative projects. And that is really super-exciting. And it\u2019s really hard to be on top of what everybody is doing. It\u2019s quite the opposite of telling people what to do, it\u2019s like trying follow-up what they are doing.<\/p>\n<p><strong>Host: It\u2019s um \u2013 on some level herding cats?<\/strong><\/p>\n<p>Rico Malvar: It\u2019s not even herding. It\u2019s where are they??<\/p>\n<p><strong>Host: You got to find the cats.<\/strong><\/p>\n<p>Rico Malvar: Find the cats, yeah.<\/p>\n<p><strong>Host: Well, talk a little bit about your role as Distinguished Engineer. What does that entail, what does that mean?<\/strong><\/p>\n<p>Rico Malvar: That\u2019s basically\u2026 there\u2019s a whole set of us. We have Distinguished Engineers and Technical Fellows which are at the top of our technical ladder. And the idea is a little bit recognition of some of the contributions we\u2019ve done in the technical area, but it\u2019s mostly our responsibility to go after big technical problems and don\u2019t think just about the group you\u2019re in, but think about the company, what the company needs, what the technology in that particular area should be evolving. My area, in particular, on the technical side, is signal processing, data compression, media compression. And these days, with audio and video entering the internet, that matters a lot. But also a few other areas, but that\u2019s the idea. The idea is that what are the big problems in technology, how can we drive new things, how can we watch out for new things coming up at the company level?<\/p>\n<p><strong>Host: You know, those two things that you mentioned, drive things and anticipate things, are two kind of different gears and two different, I won\u2019t say skillsets, but maybe it\u2019s having your brain in two places.<\/strong><\/p>\n<p>Rico Malvar: You are right. It\u2019s not completely different skillsets but driving and following are both important and one helps the other. And it\u2019s very important for us to do both.<\/p>\n<p><strong>Host: Let\u2019s go back to your roots a little bit. When you started here at Microsoft Research, you were a principle researcher and the founder and manager of what was called the Communications, Collaboration and Signal Processing group at MSR. So, tell us a little bit about the work you used to do and give us a short \u201cwhere are they now?\u201d snapshot of that group.<\/strong><\/p>\n<p>Rico Malvar: Yeah, that name is funny. That name was a bad example when you get too democratic about choosing names, and then we got everybody in the team to give ideas and then it got all complicated and we end up with a little bit of everything and came up with a boring name instead of a cool one. But it was a very descriptive name which was good. It was just called Signal Processing when we started, and then it evolved to Communication, Collaboration and Signal Processing because of the new things we were doing. For example, we had a big project on the collaboration area which is the prototype of a system which later evolved to become the <a href=\"https:\/\/www.microsoft.com\/en-us\/download\/details.aspx?id=6204\">RoundTable<\/a> product. And that\u2019s just not signal processing, it\u2019s collaboration. Well, we have put collaboration. But people use it to communicate so it\u2019s also communication, saying okay, put it all in the name. So, it\u2019s just like that. And on your question of where people are, a cool thing is that we had a combination of expertise in the team to be able to do things like RoundTable. So, we had computer vision experts, we had distributed systems experts, we had streaming media experts and we had audio experts, on the last one for example, in audio. Then later, we actually evolved a new group doing specifically audio signal processing which is now led by Ivan Tashev who was a member of my team and now has his own team. He already participated in your podcast, so it\u2019s nice to see the interesting challenges in those areas continue. And we keep evolving, as you know. The groups are always changing, modifying, renewing.<\/p>\n<p><strong>Host: In fact, that leads into my next question. Microsoft Research, as an entity, has evolved quite a bit since it was formed in 1991. And you were Managing Director in the mid-2000\u2019s from like 2007 to 2010?<\/strong><\/p>\n<p>Rico Malvar: \u201810. Of the lab here in Redmond, yeah.<\/p>\n<p><strong>Host: Yeah. So, tell us a little bit about the history of the organization in the time you\u2019ve been here.<\/strong><\/p>\n<p>Rico Malvar: Yeah. It\u2019s great. One thing I really like about Microsoft Research is first, is that it started early with the top leaders in the company always believing in the concept. So, Bill Gates started Microsoft Research, driven by Nathan Myhrvold who was the CTO at the time, and it was a no-brainer for them to start Microsoft Research. They found Rick Rashid, who was our first leader of MSR. And I had the pleasure of reporting to Rick for many years. And the vision he put in, it is still to this day, is let\u2019s really push the limits of technology. We don\u2019t start by thinking how this is going to help Microsoft, we start by thinking how we push the technology, how it helps people. Later, we will figure out how it\u2019s going to help Microsoft. And to this date, that\u2019s how we operate. With the difference being, maybe, is that in the old days, the lab was more of a classical research lab. Almost everything was pivoted on research projects.<\/p>\n<p><strong>Host: Sure.<\/strong><\/p>\n<p>Rico Malvar: Which is great, and many, many of them generated good technology or even new products to the company. I was just talking about RoundTable as one example, and we have several. Of course, the vast majority fail because research is a business of failure and we all know that! We submit ten papers for publication, two or three get accepted. That is totally fine, and we keep playing the game. And we do the papers as a validation and also as a way to interact with the community. And both are extremely of value to us so we can have a better understanding we are pushing the state-of-the-art. And today, the new Microsoft Research puts even a little more emphasis on the impact side. We still want to push the state-of-the-art, we still do innovative things, but we want to spend a little more effort on making those things real.<\/p>\n<p><strong>Host: Yeah.<\/strong><\/p>\n<p>Rico Malvar: On helping the company. And even the company, itself, evolved to a point where that has even a higher value from Satya, our CEO, down. It is the mission of the company to empower people to do more. But empowering is not just developing the technology, it\u2019s packaging it, shipping it in the right way, making products that actually leverage that. So, I would say the new MSR gets even more into, okay, what it takes to make this real.<\/p>\n<p><strong>Host: Well, let\u2019s talk a little bit about Microsoft Research NExT. Give our listeners what I would call your elevator pitch of Microsoft NExT. What does it stand for, how does it fit in the portfolio of Microsoft Research? I kind of liken it to pick-up basketball, only with scientists and more money, but you do it more justice than I do!<\/strong><\/p>\n<p>Rico Malvar: That\u2019s funny. Yeah, NExT is actually a great idea. As I said, we\u2019re always evolving. And then, when Peter Lee came in, and also Harry Shum is our new leader, they thought hard about diversifying the approaches in which we do research. So, we still have the Microsoft Research labs, the part that is a bit more traditional in the sense that the research is mostly pivoted by areas. We have a graphics team, natural language processing group, human computer interaction, systems, and so forth. Many, many of them. When you go to NExT, the idea is different. One way to achieve potentially even more impact is pivot some of those activities, not by area, but by project, by impact goal. Oh, because of this technology and that technology, maybe we have an opportunity to do X, where X is this new project. Oh, but we\u2019re going to have the first technology is computer vision, the other one is hardware architecture. Oops, we\u2019re going to have to need people in all those areas together in a project team and then Peter Lee has been driving that, always trying to find disruptive, high impact things so that we can take new challenges. And lots of things are coming up from this new model which we call NExT, which is New Experiences in Technology.<\/p>\n<p><strong>Host: I actually didn\u2019t know that, what the acronym stood for. I just thought it was, what\u2019s NExT, right?<\/strong><\/p>\n<p>Rico Malvar: Of course, that is a cool acronym. Peter did a much better job than we did on the CCSB thing.<\/p>\n<p><strong>Host: I love it.<\/strong><\/p>\n<p><strong>(music plays)<\/strong><\/p>\n<p><strong>Host: Well, let\u2019s talk about Enable, the group. There\u2019s a fascinating story of how this all got started and it involves a former football player and what\u2019s now called the yearly hackathon. Tell us the story.<\/strong><\/p>\n<p>Rico Malvar: That is exactly right. It all started when that famous football player, ex-football player, Steve Gleason, still a good partner of ours, is still a consultant to my team\u2026 Steve is a totally impressive person. He got diagnosed with ALS, and ALS is a very difficult disease because you basically lose mobility. And at some point in life, your organs may lose their ability to function, so, most people actually don\u2019t survive ALS. But with some mitigations you can prolong, a little bit, and technology can help. Steve, actually, we quote him saying, \u201cUntil there is a cure for ALS, technology is the cure.\u201d This is very inspiring. And he created a foundation, Team Gleason, that really does a wonderful job of securing resources and distributing resources to people with ALS. They really, really make a difference in the community. And he came to us almost five years ago, and we were toying with the idea of creating this hackathon, which is a company-wide effort to create hack-projects. And then in one of those, which actually the first time we did, which is in 2014, Steve told us, \u201cYou know what guys, I want to be able to do more. In particular, I want to be able to argue with my wife and play with my son. So, I need to communicate, and I need to move. My eyes still work, this eye tracking thing might be the way to go. Do you want to do something with that?\u201d The hackathon team really got inspired by the challenge and within a very short period of time, they created an eye tracking system where you look at the computer and then there\u2019s a keyboard and you can look at the keys and type at the keys by looking. And there is a play button so you can compose sentences and then speak out with your eyes.<\/p>\n<p><strong>Host: That\u2019s amazing.<\/strong><\/p>\n<p>Rico Malvar: And they also created an interface where they put buttons, similar to a joy stick, on the screen. You look at those, and the wheelchair moves in the direction of where you are selecting. They did a nice overlay between the buttons and the video, so it\u2019s almost like they put the computer, mount it on the wheelchair, you look through the computer, the camera shows what\u2019s in front of you, and then the wheelchair goes. With lots of safety things like a stop button. And it was very successful, that project. In fact, it won the first prize.<\/p>\n<p><strong>Host: The hackathon prize?<\/strong><\/p>\n<p>Rico Malvar: On the hackathon prize. And then, a little bit later, Peter and I were thinking about where to go on new projects. And then Peter really suggested, Rico, what about that hackathon thing? That seems to be quite impactful, so maybe we want to develop that technology further. What do you think? I said, well if I had a team\u2026 (laughs) we could do that\u2026<\/p>\n<p><strong>Host: (sings) If I only had a team\u2026<\/strong><\/p>\n<p>Rico Malvar: (sings) If I only had had a team\u2026 And then Peter said, ehh, how many people you need? I don\u2019t know, six, seven to start. I said, okay, let\u2019s go do it. It was as easy as that.<\/p>\n<p><strong>Host: Well, let\u2019s talk a little bit more about the hackathon. Like you said, it\u2019s about in its fifth year. And, as I understand it, it\u2019s kind of a ground-up approach. Satya replaced the annual \u201cexecutive-inspirational-talk-top-down\u201d kind of summer event with, hey, let\u2019s get the whole company involved in invention. I would imagine it\u2019s had a huge impact on the company at large. But how would you describe the role of the hackathon for people in Microsoft Research now? It seems like a lot of really interesting things have come out of that summer event.<\/strong><\/p>\n<p>Rico Malvar: You know, for us, it was a clear thing, because Microsoft Research was always bottom-up. I mean, we literally don\u2019t tell researchers what to do. People, researchers, engineers, designers, managers, they all have great ideas, right? And they come up with those great ideas. When they click enough, they start developing something and we look from the top and say, that sounds good, keep going, right? So, we try to foster the most promising ones. But the idea of bottom-up was already there.<\/p>\n<p><strong>Host: Yeah.<\/strong><\/p>\n<p>Rico Malvar: When we look at the hackathon, we say, hey, thanks to Satya and the new leadership of Microsoft, the company\u2019s embracing this concept of moving bottom-up. There\u2019s The Garage. The Garage has been involved with many of those hackathons. Garage has been a driver and supporter of the hackathon. So, to us, it was like, hey, great, that\u2019s how we work! And now we\u2019re going to do more collaboration with the rest of the company.<\/p>\n<p><strong>Host: You have a fantastic and diverse group of researchers working with you, many of whom have been on the podcast already and been delightful. Who and what does it take to tackle big issues, huge ideas like hands-free keyboards and eye tracking and 3-D sound?<\/strong><\/p>\n<p>Rico Malvar: Right. One important concept, and it\u2019s particularly important for Enable, is that we really need to pay attention to the user. Terms such as \u201cuser-centric\u201d \u2013 yeah, they sound like clich\u00e9 \u2013 but especially in accessibility, this is super important. For example, in our Enable team, the area working with eye tracking, our main intended user were people with ALS since the motivation from Steve Gleason. And then, in our team, Ann Paradiso, who is our user experience manager, she created what we call the PALS program. PALS means Person with ALS. And we actually brought people with ALS in their wheelchairs and everything to our lab and discussed ideas with them. So, they were not just testers, they were brainstorming with us on the design and technologies\u2026<\/p>\n<p><strong>Host: Collaborators.<\/strong><\/p>\n<p>Rico Malvar: Collaborators. They loved doing it. They really felt, wow, I\u2019m in this condition but I can contribute to something meaningful and we will make it better for the next generation\u2026<\/p>\n<p><strong>Host: Sure.<\/strong><\/p>\n<p>Rico Malvar: \u2026of people with this. So, this concept of strong user understanding through user design and user research, particularly on accessibility, makes a big difference.<\/p>\n<p><strong>Host: Mmm hmm. Talk a little bit about the technical side of things. What kinds of technical lines of inquiry are you really focusing on right now? I think our listeners are really curious about what they\u2019re studying and how that might translate over here if they wanted to\u2026<\/strong><\/p>\n<p>Rico Malvar: That\u2019s a great question. Many of the advancements today are associated with artificial intelligence, AI, because of all the applications of AI, including in our projects. AI is typically a bunch of algorithms and data manipulation in finding patterns in data and so forth. But AI, itself, doesn\u2019t talk to the user. You still need the last mile of the interfaces, the new interface. Is the AI going to appear to the user as a voice? Or as something on the screen? How is the user going to interact with the AI? So, we need new interfaces. And then, with the evolution of technology, we can develop novel interfaces. Eye tracking being an example. If I tell you that you\u2019re going to control your computer with your eyes, you\u2019re going to say, what? What does that mean? If I tell you, you\u2019re going to control the computer with your voice, you say, oh yeah, I\u2019ve been doing that for a while. With the eye tracking for a person with a disability, they immediately get it and say, a-ha! I know what it means, and I want to use that. For everybody, suppose, for example, that you are having your lunch break and you want to browse the news on the internet, get up to date on a topic of interest. But you\u2019re eating a sandwich. Your hands are busy, your mouth is busy, but your eyes are free. You could actually flip around pages, do a lot of things, just with your eyes and you don\u2019t need to worry about cleaning your hands and touching the computer because you don\u2019t need to touch the computer. And you can think, in the future, where you may not even need your eyes. I may read your thoughts directly. And, at some point, it\u2019s just a matter of time. It\u2019s not that far away. We are going to read your thoughts directly.<\/p>\n<p><strong>Host: That\u2019s both exciting and scary. Ummmm\u2026<\/strong><\/p>\n<p>Rico Malvar: Yes.<\/p>\n<p><strong>Host: What does it take to say, all right, we\u2019re going to make a machine be able to look at your eyes and tell you back what you are doing?<\/strong><\/p>\n<p>Rico Malvar: Yeah, you see, it\u2019s a specialized version of computer vision. It\u2019s basically cameras that look at your eyes. In fact, the sensor works by first illuminating your eyes with bright IR lights, infrared, so it doesn\u2019t bother you because you can\u2019t see. But now you have this bright image that the camera is looking at, IR can see, and then models in a little bit of AI and a little bit of just graphics and computer vision and signal modeling, that then make an estimate of the position of your eyes and associate that with elements on the screen. So, it\u2019s almost as if you have a cursor on the screen.<\/p>\n<p><strong>Host: Okay.<\/strong><\/p>\n<p>Rico Malvar: That is controlled with your eyes, very similar to a mouse, with the difference that the eye control works better if we don\u2019t display the cursor. With the mouse, you actually should display the cursor\u2026<\/p>\n<p><strong>Host: Ooohhh, interesting\u2026.<\/strong><\/p>\n<p>Rico Malvar: \u2026with eye control, the cursor works better if it is invisible. But you see the idea there is that you do need specialists, you need folks who understand that. And sometimes you do a combination of some of that understanding being in the group, so we need to be the top leaders in that technology, or we partner with partners that have a piece of the technology. For example, for the eye tracking, we put much more emphasis on designing the proper user interfaces and user experiences, because there are companies that do a good job introducing eye tracking devices. So, we leverage the eye tracking devices that these companies produce.<\/p>\n<p><strong>Host: And behind that, you are building on machine learning technologies, on computer vision technologies and\u2026 um\u2026 so\u2026<\/strong><\/p>\n<p>Rico Malvar: Correct. For example, a typical one is that the keyboard driven by your eyes. You still want to have a predictive keyboard.<\/p>\n<p><strong>Host: Sure.<\/strong><\/p>\n<p>Rico Malvar: So, as you are typing the letters, it guesses. But how you interface on the guess, it\u2019s very interesting, because when you are typically using a keyboard, your eye is looking at the letters, your fingers are typing on the keys. When you\u2019re doing an eye control keyboard, your eye has to do everything. So, how you design the interface should be different.<\/p>\n<p><strong>Host: Yeah.<\/strong><\/p>\n<p>Rico Malvar: And we\u2019ve learned and designed good ways to make that different.<\/p>\n<p><strong>Host: If I\u2019m looking at the screen and I\u2019m moving my eyes, how does it know when I\u2019m done, you know, like that\u2019s the letter I want? Do I just laser beam the\u2026??<\/strong><\/p>\n<p>Rico Malvar: You said you would be asking deep technical questions and you are. That one, we use the concept that we call \u201cdwelling.\u201d As you look around the keyboard, remember that I told you we don\u2019t display the cursor?<\/p>\n<p><strong>Host: Right.<\/strong><\/p>\n<p>Rico Malvar: So, but as you \u2013 the position where you look in your eyes, the focus of your eye, is in a particular letter, we highlight that letter. It can be a different color, it can be a lighter shade of grey\u2026<\/p>\n<p><strong>Host: Gotcha.<\/strong><\/p>\n<p>Rico Malvar: So, as you move around, you see the letters moving around. If you want to type a particular letter, once you get to that letter, you stop moving for a little bit, let\u2019s say half a second. That\u2019s a dwell. You dwell on that letter a little bit and we measure the dwell. And there\u2019s a little bit of AI to learn what is the proper dwell time based on the user.<\/p>\n<p><strong>(music plays)<\/strong><\/p>\n<p><strong>Host: One thing I\u2019m fascinated by, not just here, but in scientific ventures everywhere, is the research \u201csuccess story.\u201d The one that chronicles the path of a blue-sky research thing to instantiation in a product. And, I know, over and over, researchers have told me, research is generally a slow business, so it\u2019s not like, oh, the overnight success story, but there\u2019s a lot of hard-won success stories or stories that sort of blossomed over multiple years of serendipitous discovery. Do you have any stories that you could share about things that you\u2019ve seen that started out like a hair-brained idea and now millions of people are using?<\/strong><\/p>\n<p>Rico Malvar: You know, there\u2019s so many examples. I particularly like the story of Kinect, which was actually not a product developed by Microsoft Research, but in close collaboration with Microsoft Research. It was the Kinect team, at the time, in Windows. Because at some point, the leader of the team, Alex Kipman, came to us and says, oh, we want to do a new controller. What if you just spoke to the machine, made gestures and we could recognize everything? You say, that sounds like sci-fi. So, naahhh, that doesn\u2019t work. But then Alex was very insistent. And then we said, no, wait a second, but to detect gestures, we need specialized computer vision. We\u2019ve been doing computer vision for 15 years. To identify your voice, we need speech recognition. We\u2019ve also been doing speech recognition for 15 years. Oh, but now there maybe be other sounds and there are maybe multiple people\u2026 oh, but just a little over 10 years ago, we started these microphone arrays. They are acoustic antennas. They can tune to the sound of whoever is speaking all of that.<\/p>\n<p><strong>Host: Directional.<\/strong><\/p>\n<p>Rico Malvar: The directional sound input. And I said, wait a second, we actually have all the core elements, we could actually do this thing. So, after the third or fourth meeting, I said, okay Alex, I think we can do that. And he said, great, you have two years to do it. What??? Yeah, because we need to ship at this particular date. And it all worked. I doubt there\u2019s some other institution or company that could have produced that because we\u2019ve been doing what was, apparently, \u201cblue-sky\u201d for many years, but then we created all those technologies and when then need arose, I say, a-ha, we can put them altogether.<\/p>\n<p><strong>Host: Where is Kinect today?<\/strong><\/p>\n<p>Rico Malvar: Kinect used to be a peripheral device for Xbox. We changed it into an IoT device. So, there\u2019s a new Kinect kit, connects to Azure so people can do Kinect-like things, not just for games but for everything. And all the technology that supports that is now in Azure.<\/p>\n<p><strong>Host: So, Rico, you have a reputation for being an optimist. You\u2019ve actually said as much yourself.<\/strong><\/p>\n<p>Rico Malvar: (laughs) Yes, I am!<\/p>\n<p><strong>Host: Plus, you work with teams on projects that are actually making the lives of people with disabilities, and others, profoundly better. But I know some of the projects that you worked on fall somewhere in the bounds of medical interventions.<\/strong><\/p>\n<p>Rico Malvar: Mmm-hmm.<\/p>\n<p><strong>Host: So, is there anything about what you do that keeps you up at night, anything we should be concerned about?<\/strong><\/p>\n<p>Rico Malvar: Yeah, you know, when you are helping a person with disability, sometimes what you are doing can be seen as, is that a treatment, is that a medical device? In most cases, they are not. But the answer to those questions can be complicated and there can be regulations. And of course, Microsoft is a super-responsible company, and if anything is regulated, of course, we are going to pay attention to the regulations. But some of those are complex. So, doing it right by the regulations can take significant amount of work. So, we have to do this extra work. So, my team has to spend time, sometimes in collaboration with our legal team, to make sure we do the right things. And I hope also that we will help evolve those regulations, potentially by working with the regulatory bodies, educating them on the evolution of the technology. Because in all areas, not just this area, but almost all areas of technology, regulations tend to be behind. It\u2019s hard to move, and understandably so. So, the fact that we have to spend significant effort dealing with that does keep me up at night a little bit. But we do our best.<\/p>\n<p><strong>Host: You know, there\u2019s a bit of a Wild West mentality where you have to, like you say, educate. And so, in a sense what I hear you saying is that, as you take responsibility for what you are doing, you are helping to shape and inform the way the culture onboards these things.<\/strong><\/p>\n<p>Rico Malvar: Exactly right, yes. Exactly right.<\/p>\n<p><strong>Host: So, how would you sort of frame that for people out there? How do we, you, help move the culture into a space that more understands what\u2019s going on and can onboard it with responsibility themselves?<\/strong><\/p>\n<p>Rico Malvar: That is a great question. And you see for example, in areas such as AI, artificial intelligence, people are naturally afraid of how far can AI go? What are the kinds of things it could do?<\/p>\n<p><strong>Host: Yeah.<\/strong><\/p>\n<p>Rico Malvar: Can we regulate so that there will be some control in how it\u2019s developed? And Microsoft has taken the stance that we have to be very serious about AI. We have to be ethical, we have to preserve privacy and all of those things. So, instead of waiting for regulation and regulatory aspects to develop, let\u2019s help them. So, we were founders of \u2013 not just me, but the company and especially the Microsoft Research AI team \u2013 founders of the Partnership for AI, in partnership with other companies to actually say no, let\u2019s be proactive about that.<\/p>\n<p><strong>(music plays)<\/strong><\/p>\n<p><strong>Host: Tell us a bit about Rico Malvar. Let\u2019s go further back than your time here at MSR and tell us how you got interested in technology, technology research. How did you end up here at Microsoft Research?<\/strong><\/p>\n<p>Rico Malvar: Okay, on the first question, how I got interested in technology? It took me a long time. I think I was 8 years old when my dad gave me an electronics kit and I start playing with that thing and I said, a-ha! That\u2019s what I want to do when I grow up. So, then I went through high school taking courses in electronics and then I went to college to become an electrical engineer and I loved the academic environment, I loved doing research. So, I knew I wanted to do grad school. I got lucky enough to be accepted at MIT and when I arrived there, I was like, boy, this place is tough! And it was tough! But then when I finished and I went back to my home country, I created the signal processing group at the school there, which was\u2026 I was lucky to get fair amounts of funding, so we did lots of cool things. And then, one day, some colleagues in a company here in the US called me back in Brazil and they say, hey, our director of research decided to do something else. Do you want to apply for the position? And then I told my wife, hey, there\u2019s a job opening in the US, what about that? I said, well go talk to them. And I came, talked to them. They make me an offer. And then it took us about a whole month discussing, are we going to move our whole family to another country? Hey, we lived there before, it\u2019s not so bad, because I studied here. And maybe it\u2019s going to be good for the kids. Let\u2019s go. If something doesn\u2019t work, we move back. I say, okay. So, and\u2026 here we are. But that was not Microsoft. That was for another company at the time, a company called PictureTel which was actually the leading company in professional video conferencing systems.<\/p>\n<p><strong>Host: Oh, okay.<\/strong><\/p>\n<p>Rico Malvar: So, we were pushing the state-of-the-art on how do you compress video and audio and these other things? And I was working happily there for about four years and then one day I see Microsoft and I say, wow, Microsoft Research is growing fast. Then one afternoon, I said, ah, okay, I think about it and I send an email to the CTO of Microsoft saying, you guys are great, you are developing all these groups. You don\u2019t have yet a group on signal processing. And signal processing is important because one day we\u2019re going to be watching video on your computers via the internet and all of that, so you should be investing more on that. And I see you already have Windows Media Player. Anyways, if you want to do research in signal processing, here\u2019s my CV. I could build and lead a group for you doing that. And then I tell my wife and she goes, you did what?? You sent an email to the CTO of Microsoft??<\/p>\n<p><strong>Host: Who was it at the time?<\/strong><\/p>\n<p>Rico Malvar: It was Nathan Myhrvold.<\/p>\n<p><strong>Host: Nathan.<\/strong><\/p>\n<p>Rico Malvar: And she said, nah. I say, what do I have to lose? The worst case, they don\u2019t respond, and life is good. I have a good job here. It\u2019s all good. And that was on a Sunday afternoon. Monday morning, I get an email from Microsoft. Hey, my name is Suzanne. I work on recruiting. I\u2019m coordinating your interview trip. I said, alright! And then I show the email to my wife and she was like, what? It worked? Whoa! And then it actually was a great time. The environment here, from day one, since the interviews, the openness of everybody, of management, the possibilities and the desire of Microsoft to, yeah, let\u2019s explore this area, this area. One big word here is diversity. Diversity of people, diversity of areas. It is so broad. And that\u2019s super exciting. So, I was almost saying, whatever offer they make me, I\u2019ll take it! Fortunately, they made a reasonable one, so it wasn\u2019t too hard to make that decision.<\/p>\n<p><strong>Host: Well, two things I take away from what you\u2019ve just told me. You keep using the word lucky and I think that has less to do with it than you are making it out to be. Um, because there\u2019s a lot of really smart people here that say, I was so lucky that they offered me this. It\u2019s like, no, they\u2019re lucky to have you, actually. But also, the idea that if you don\u2019t ask, you are never going to know whether you could have or not. I think that\u2019s a wonderful story of boldness and saying why not?<\/strong><\/p>\n<p>Rico Malvar: Yeah. And in fact, boldness is very characteristic of Microsoft Research. We\u2019re not afraid. We have an idea, we just go and execute. And we\u2019re fortunate, and I\u2019m not going to say lucky, I\u2019m going to say fortunate, that we\u2019re in a company that sees that and gives us the resources to do so.<\/p>\n<p><strong>Host: Rico, I like to ask all my guests, as we come to the end of our conversation, to offer some parting thoughts to our listeners. I think what you just said is a fantastic parting thought. But maybe there\u2019s more. So, what advice or wisdom would you pass on to what we might call the next generation of technical researchers? What\u2019s important for them to know? What qualities should they be cultivating in their lives and work in order to be successful in this arena?<\/strong><\/p>\n<p>Rico Malvar: I would go back on boldness and diversity. Boldness, you\u2019ve already highlighted Gretchen, that, you know, if you have an idea but it\u2019s not just too rough an idea, you know a thing or two why that actually could work, go after it! Give it a try. Especially if you are young. Don\u2019t worry if you fail many things. I failed many things in my life. But what matters is not the failures. You learn from the failures and you do it again. And the other one is diversity. Always think diversity in all the dimensions. All kind of people, everywhere in the world. It doesn\u2019t matter gender, race, ethnicity, upbringing, rich, poor, whatever they come from, everybody can have cool ideas. The person whom you least expect to invent something might be the one inventing. So, listen to everybody because that diversity is great. And remember, the diversity of users. Don\u2019t assume that all users are the same. Go learn what users really think. If you are not sure if Idea A or Idea B is the better, go talk to them. Try them out, test, get their opinion, test things with them. So, push diversity on both sides, diversity on the creation and diversity on who is going to use your technology. And don\u2019t assume you know. In fact, Satya has been pushing the whole company towards that. Put us in a growth mindset which basically means keep learning, right? Because then if you do that, that diversity will expand and then we\u2019ll be able to do more.<\/p>\n<p><strong>Host: Rico Malvar, I\u2019m so glad that I finally got you on the podcast. It\u2019s been delightful. Thanks for joining us today.<\/strong><\/p>\n<p>Rico Malvar: It has been a pleasure. Thanks for inviting me.<\/p>\n<p><strong>(music plays)<\/strong><\/p>\n<p><strong>To learn more about Dr. Rico Malvar and how research for people with disabilities is enabling people of all abilities, visit<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/\"> Microsoft.com\/research<\/a>.<\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Episode 61, January 30, 2019 &#8211; Dr. Rico Malvar recalls his early years at a fledgling Microsoft Research, talks about the exciting work he oversees now, explains why designing with the user is as important as designing for the user, and tells us how a challenge from an ex-football player with ALS led to a prize winning hackathon project and produced the core technology that allows you to type on a keyboard without your hands and drive a wheelchair with your eyes.<\/p>\n","protected":false},"author":37074,"featured_media":564216,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"https:\/\/player.blubrry.com\/id\/41333560\/","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"categories":[240054],"tags":[],"research-area":[13556,13554],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-564210","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-msr-podcast","msr-research-area-artificial-intelligence","msr-research-area-human-computer-interaction","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"https:\/\/player.blubrry.com\/id\/41333560\/","podcast_episode":"","msr_research_lab":[],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[144928],"related-projects":[],"related-events":[],"related-researchers":[],"msr_type":"Post","featured_image_thumbnail":"<img width=\"960\" height=\"540\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/01\/Rico-Malvar_POD_Site_01_2019_1400x788.png\" class=\"img-object-cover\" alt=\"Rico Malvar, Chief Scientist and Distinguished Engineer standing in front of bushes\" decoding=\"async\" loading=\"lazy\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/01\/Rico-Malvar_POD_Site_01_2019_1400x788.png 1400w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/01\/Rico-Malvar_POD_Site_01_2019_1400x788-300x169.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/01\/Rico-Malvar_POD_Site_01_2019_1400x788-768x432.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/01\/Rico-Malvar_POD_Site_01_2019_1400x788-1024x576.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/01\/Rico-Malvar_POD_Site_01_2019_1400x788-1066x600.png 1066w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/01\/Rico-Malvar_POD_Site_01_2019_1400x788-655x368.png 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/01\/Rico-Malvar_POD_Site_01_2019_1400x788-343x193.png 343w\" sizes=\"(max-width: 960px) 100vw, 960px\" \/>","byline":"","formattedDate":"January 30, 2019","formattedExcerpt":"Episode 61, January 30, 2019 - Dr. Rico Malvar recalls his early years at a fledgling Microsoft Research, talks about the exciting work he oversees now, explains why designing with the user is as important as designing for the user, and tells us how a&hellip;","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/564210"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/37074"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=564210"}],"version-history":[{"count":4,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/564210\/revisions"}],"predecessor-version":[{"id":564234,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/564210\/revisions\/564234"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/564216"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=564210"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=564210"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=564210"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=564210"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=564210"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=564210"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=564210"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=564210"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=564210"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=564210"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=564210"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}