[Music]

Sunayana Sitaram: Our ultimate goal is to build evaluation systems and also other kinds of systems in general where humans and LLMs can work together. We’re really trying to get humans to do the evaluation, get LLM’s to do the evaluation, use the human data in order to improve the LLM. And then just this continues in a cycle. And the ultimate goal is, send the things to the LLM that it’s good at doing and send the rest of the things that the LLM can’t do to humans who are like the ultimate authority on the evaluation.

Sridhar Vedantham: Welcome to the Microsoft Research India podcast, where we explore cutting-edge research that’s impacting technology and society. I’m your host, Sridhar Vedantham.

[Music]

Sridhar Vedantham: LLM’s are perhaps the hottest topic of discussion in the tech world today. And they’re being deployed across domains, geographies, industries and applications. I have an extremely interesting conversation with Sunayana Sitaram, principal researcher at Microsoft Research about LLMs, where they work really well and also challenges that arise when trying to build models with languages that may be under resourced. We also talk about the critical work she and her team are doing in creating state-of-the-art methods to evaluate the performance of LLMs, including those LLMs that are based on Indic languages.

Related

• Microsoft Research India Podcast: More podcasts from MSR India
• iTunes (opens in new tab): Subscribe and listen to new podcasts on iTunes
• Android (opens in new tab)
• RSS Feed (opens in new tab)
• Spotify (opens in new tab)
• Google Podcasts (opens in new tab)
• Email (opens in new tab)

[Music]

Sridhar Vedantham: Sunayana, welcome to the podcast.

Sunayana Sitaram: Thank you.

Sridhar Vedantham: And I’m very excited to have you here because we get to talk about a subject that seems to be top of mind for everybody right now. Which is obviously LLMs.  And what excites me even more is I think, we’re going to be talking about LLMs in a way that’s slightly different from what the common discourse is today, right?

Sunayana Sitaram: That’s right.

Sridhar Vedantham: OK. So before we jump into it, why don’t you give us a little bit of background about yourself and how you came to be at MSR?

Sunayana Sitaram: Sure. So it’s been eight years now since I came to MSR. I came here as a postdoc after finishing my PhD at Carnegie Mellon. And so yeah, it’s been around 15 years now for me in the field, and it’s been super exciting, especially the last few years.

Sridhar Vedantham: So, I’m guessing that these eight years have been interesting, otherwise we won’t be having this conversation. What areas of research, I mean, have you changed course over the years and how is that progressed?

Sunayana Sitaram: Yeah, actually, I’ve been working pretty much on the same thing for the last 15 years or so. So I’ll describe how I got started. When I was an undergrad, I actually met the principal of a blind children’s school who himself was visually impaired. And he was talking about some of the technologies that he uses in order to be independent. And one of those was using optical character recognition and text to speech in order to take documents or letters that people sent him and have them read out without having to depend on somebody. And he was in Ahmedabad, which is where I grew up. And his native language was Gujarati.  And he was not able to do this for that language. Whereas for English, the tools that he required to be independent were available. And so, he told me like it would be really great if somebody could actually build this kind of system in Gujarati. And that is when it sort of it was like a, you know, aha moment for me. And I decided to take that up as my undergrad project. And ever since then, I’ve been trying to work on technologies trying to bridge that gap between English and other languages- under resourced languages. And so, since then, I’ve worked on very related areas. So, my PhD thesis was on text to speech systems for low resource languages. And after I came to MSR I started working on what is called code switching, which is a very common thing that multilinguals all over the world do. So they use multiple languages in the same conversation or sometimes even in the same sentence. And so you know, this was a project called Project Melange that was started here and that really pioneered the code switching work in the research community in NLP. And after that it’s been about LLMs and evaluation but again from a multilingual under resource languages standpoint.

Sridhar Vedantham: Right. So I have been here for quite a while at MSR myself and one thing that I always heard is that there is this in general, a wide gulf in terms of the resources available for a certain set of languages to do say NLP type work. And the other languages is just the tail, it’s a long tail, but the tail just falls off dramatically. So, I wanted you to answer me in a couple of ways. One is, what is the impact that this generally has in the field of NLP itself and in the field of research into language technologies, and what’s the resultant impact on LLMs?

Sunayana Sitaram: Yeah, that’s a great question. So, you know the paradigm has shifted a little bit after LLM’s have come into existence. Before this, so this was around say a few years ago, the paradigm would be that you would need what is called unlabeled data. So, that is raw text that you can find on the web, say Wikipedia or something like that, as well as labeled data. So, this is something that a human being has actually sat and labeled for some characteristic of that text, right? So these are the two different kinds of texts that you need if you want to build a text based language model for a particular language. And so there were languages where, you know, you would find quite a lot of data on the web because it was available in the form of documents or social media, etc. for certain languages. But nobody had actually created the labeled resources for those languages, right? So that was the situation a few years ago. And you know the paradigm at that time was to use both these kinds of data in order to build these models, and our lab actually wrote quite a well-regarded paper called, ‘The State and Fate of Linguistic Diversity and Inclusion’, where they grouped different languages into different classes based on how much data they had labeled, as well as unlabeled.

Sridhar Vedantham: Right.

Sunayana Sitaram: And it was very clear from that work that, you know only around 7 or 8 languages of the world actually can be considered to be high resource languages which have this kind of data. And most of the languages of the world spoken by millions and millions of speakers don’t have these resources. Now with LLMs, the paradigm changed slightly, so there was much less reliance on this labeled data and much more on the vast amount of unlabeled data that exists, say, on the web. And so, you know, we were wondering what would happen with the advent of LLMs now to all of the languages of the world, which ones would be well represented, which ones wouldn’t etc. And so that led us to do, you know, the work that we’ve been doing over the last couple of years. But the story is similar, that even on the web some of these languages dominate and so many of these models have, you know, quite a lot of data from only a small number of languages, while the other languages don’t have much representation.

Sridhar Vedantham: OK. So, in real terms, in this world of LLMs that we live in today, what kind of impact are we looking at? I mean, when you’re talking about inequities and LLMs and in this particular field, what’s the kind of impact that we’re seeing across society?

Sunayana Sitaram: Sure. So when it comes to LLMs and language coverage, what we found from our research is that there are a few languages that LLMs perform really well on. Those languages tend to be high resource languages for which there is a lot of data on the web and they also tend to be languages that are written in the Latin script because of the way the LLMs are designed currently with the tokenization. And the other languages, unfortunately there is a large gap between the performance in English and other languages, and we also see that a lot of capabilities that we see in LLMs in English don’t always hold in other languages. So a lot of capabilities, like really good reasoning skills, etc, may only be present in English and a few other languages, and they may not be seen in other languages. And this is also true when you go to smaller models that you see that their language capabilities fall off quite drastically compared to the really large models that we have, like the GPT 4 kind of models. So when it comes to real world impact of this, you know, if you’re trying to actually integrate one of these language models into an application and you’re trying to use it in a particular language, chances are that you may not get as good performance in many languages compared to English. And this is especially true if you’re already used to using these systems in English and you want to use them in a second language. You expect them to have certain capabilities which you’ve seen in English, and then when you use them in another language, you may not find the same capabilities. So in that sense, I think there’s a lot of catching up to do for many languages. And the other issue also is that we don’t even know how well these systems perform for most languages of the world because we’ve only been able to evaluate them on around 50 to 60 or maybe 100 languages. So for the rest of the 6000ish languages of the world, many of which don’t even have a written form, most of which are not there on the web. We don’t even know whether these language models are, you know, able to do anything in them at all. So I think that is another, you know, big problem that is there currently.

Sridhar Vedantham: So, if you want to change the situation where we say that you know even if you’re a speaker of a language that might be small, maybe say only two million speakers as opposed to a larger language that might have 100 million or 200 million speakers. How do we even go about addressing inequities like that because at some level it just seems unfair, that for no fault of their own, you know, large sections of population could be excluded from the benefits of LLM’s, right? Because there could be any number of languages in which the number of speakers might be, say, 1,000,000 or 100,000.

Sunayana Sitaram: Right. I think that’s a very hard question. How to actually involve language communities into our efforts, but do that at scale, so that we can actually involve all language communities, all cultures, etc. into the whole building process. So we’ve had some success with doing this with some language communities. So there is a project called ELLORA in MSR India that you know Kalika leads where you know they work with specific language communities, try to understand what the language communities actually need and then try to co-build those resources with them. And so you know in that sense, you know, working directly with these language communities, especially those that have a desire to build this technology and can contribute to some of the data aspects, etc. That’s definitely one way of doing things. We’ve also done some work recently where we’ve engaged many people in India in trying to contribute resources in terms of cultural artifacts and also evaluation. And so you know, we’re trying to do that with the community itself, with the language community that is underrepresented in these LLMs, but doing that at scale is the challenge to try and really bring everyone together. Another way of course, is just raising awareness about the fact that this issue exists, and so I think our work over the last couple of years has really, you know, moved the needle on that. So we’ve done the most comprehensive multilingual evaluation effort that exists both within the large models as well as across different sized models which we call Mega and Megaverse respectively.

Sridhar Vedantham: So if I can just interrupt here, what I’d like is if you could, you know, spend a couple of minutes maybe talking about what evaluating an LLM actually means and how do you go about that?

Sunayana Sitaram: Sure. So when we talk about evaluating LLM’s right, there are multiple capabilities that we expect LLMs to possess. And so, our evaluation should ideally try to test for all of those different kinds of capabilities. So, this could be the ability to reason, this could be the ability to produce output that actually sounds natural to a native speaker of the language. It could be completing some particular task, it could be not hallucinating or not making up things. And also of course, responsible AI, you know, metrics. So things like, you know, being safe and fair, no bias, etc. Only if all of those things work in a particular language, can you say that, that LLM actually works for that language. And so there are several dimensions that we need to consider when we are evaluating these LLMs.

[Music]

Sridhar Vedantham: Before I interrupted you, you were talking about certain projects that you were working on, which are to do with evaluating LLMs, right? I think there’s something called Mega there’s something called Megaverse. Could you tell us a little bit about those and what exactly they do?

Sunayana Sitaram: Sure. So Mega project we started when ChatGPT came out basically. And the question that we were trying to answer was how well these kinds of LLMs perform on languages of the world. So with Mega what we did was, we took already existing open source benchmarks that tested for different kinds of capabilities. So some of them were question-answering benchmarks. Some of them were testing for whether it can summarize text properly or not. Some of them were testing for other capabilities like reasoning etc. And we tested a bunch of models across all of these benchmarks and we covered something like 80 different languages across all these benchmarks. And our aim with Mega was to figure out what the gap was between English and other languages for all of these different tasks, but also what the gap was between the older models, so the models pre LLM and LLMs. Whether we’ve become better or worse in terms of linguistic diversity and performance on different languages in the new era of LLMs or not. And that was the aim with Mega.

Sridhar Vedantham: Sorry, but what was the result of it? Have we become better or not?

Sunayana Sitaram: Yeah. So we have mixed results. So for certain languages, we are doing quite well, but for some languages, unfortunately the larger models don’t do as well as some of the older models used to do. And the older models used to be specialized and trained on labeled data as I said in the beginning, right? And that would help them also be better at all the languages under consideration, whereas with the LLMs we were not really using labeled data in a particular language to train them. And so, we found that, you know, in some cases, the performance of English had shot up drastically and so the gap between English and other languages had also increased in the LLM’s case.

Sridhar Vedantham: Ok. So, the performance of English, I mean the LLMs, that’s much better than what there was earlier, but the other languages didn’t manage to show the same performance increase.

Sunayana Sitaram: That’s right. They didn’t always show. Some of them did, some of the higher resource languages written in the Latin script, for example, did perform quite well, but some of the others did not.

Sridhar Vedantham: OK. And after Mega, then what happened?

Sunayana Sitaram: Yes. So with Mega, we were primarily trying to evaluate the GPT family of models with the older generation of models as I mentioned. But then we realized by the time we finished the work on Mega, there was a plethora of models that came out. So there’s Llama and, you know other models by competitors as well as smaller models, the SLMs, you know, like the Llama sized models, Mistral etc, right. So, there were all of these different models. And then we wanted to see across different models, especially when you’re trying to compare larger models with smaller models, how do these trends look? And that is what we call Megaverse, where we do all of these different evaluations, but not just for the GPT family, but across different models. And what we found in Megaverse were the trends were similar that there were some languages that were doing well, some of the other lower resource languages, especially the ones written in other scripts, were not doing so well. So, for example, the Indian languages were not doing very well across the board. But we also found that the larger frontier models, like the GPT model, they were doing much better than the smaller models for multilingual. And this is again something that you know was shown for the first time in our work that there is this additional gap when you have this large model and small model and there are important practical implications of this. So, say you’re trying to integrate the small model into your workflow as a startup or something like that in a particular language then because it is cheaper it is much more cost efficient, etc, you may not get the same performance in non-English languages as you would get with the larger model right? So that has an impact in the real world.

Sridhar Vedantham: Interesting. And how do you draw this line between what constitutes a large language model and what constitutes a small language model? And I’m also increasingly hearing of this thing called a tiny language model.

Sunayana Sitaram: That’s right. Yeah. So the large language models are the GPTs, the Geminis, you know, those kinds of models. Everything else, we just club as a smaller language model. We don’t really draw a line there. I haven’t actually seen any tiny models that do very well on multilingual. They’re tiny, because they are, you know, trained on a smaller set of data, they have fewer parameters and typically we haven’t seen too many multilingual tiny models so we haven’t really evaluated those. Although there is a new class of models that have started coming up, which are language specific models. So, for example a lot of the Indic model developers have started building specialized model for one language or a small family of languages.

Sridhar Vedantham: OK, so going back to something you said earlier, how do these you know kind of models that people are building for specific Indian languages actually work or perform, given that, I think we established quite early in this podcast that, these are languages that are highly under resourced in terms of data to build models.

Sunayana Sitaram: That’s right. So I think it’s, it’s not just a problem of them being under resourced, it’s also that the proportion of data in the model for a particular language that is not English, say Hindi or Malayalam or Kannada, is very tiny compared to English. And so there are ways to actually change this by doing things with the model after it has been trained. So this is called fine tuning. So what you could do is you could take, say, an open source model which is like a medium sized or a small model and then you could fine tune it or specialize it with data in a particular language, and that actually makes it work much better for that particular language because the distributions shift towards the language that you’re targeting. And so, it’s not just, you know, about the amount of data, but also the proportion of data and how the model has been trained in these giant models that cover hundreds of languages in a single model versus, you know, having a model that is specialized to just one language which makes it do much better. So these Indic models we have found actually do better than the open source models that they were built on top of, because now they have been specialized to a particular language.

Sridhar Vedantham: Ok. I know that your work focuses primarily on the evaluation of LLM’s, right? There must be a lot of other people who are also doing similar work in terms of evaluating performance of LLM on different parameters. How do you differentiate your work from what others are doing?

Sunayana Sitaram: Yeah, that’s a great question. So we’ve been doing evaluation work pre LLM actually. We started this a few years ago. And so we’ve actually done several evaluation projects. The previous one was called LITMUS where we were trying to evaluate without even having a benchmark in a particular language, right? And so we’ve built up a lot of expertise in how to do evaluation, and this has actually become a very hard problem in the LLM world because it’s becoming increasingly difficult to figure out what the strengths and weaknesses are of these LLMs because of how they’re built and how they behave, right. And so I think we bring in so much rich evaluation expertise that we’ve been able to do these kinds of, you know, Mega evaluations in a very systematic way where we’ve taken care of all of the we’ve taken care of all of the hanging loose threads that otherwise others don’t take care of. And that is why we managed to do these comprehensive giant exercises of Mega and Megaverse and also got these clear trends from them. So in that sense I would say that our evaluation research is very mature and we’ve been spending a lot of time thinking about how to evaluate which is unique in our group.

Sridhar Vedantham: OK, so one thing I’ve been curious about for a while is there seems to be a lot of cultural and social bias that creeps into these models, right? How does one even try to address these issues?

Sunayana Sitaram: That’s right. So, I think over the last few months, building culture, specific language models or even evaluating whether language models are appropriate for a particular culture, etc, that has become a really hot topic. Because people have started seeing that, you know,   most of these language models are a little tilted towards western protestant and, rich, industrialized kind of worldviews and the values that they encode may not be appropriate for all cultures. And so there have been some techniques that we’ve been working on in order to again shift the balance back into other target cultures that we want to fine tune the model for, so again, you know, you could take data that has characteristics of a particular culture, values of a particular culture, and then do some sort of fine tuning on a model in order to shift  its distribution more towards a target culture. There are techniques that are coming to be for these kinds of culture specific language models. However, I still think that we are far away from a comprehensive solution, because even defining what culture is and what constitutes, you know, say an Indic culture LLM, I think that’s a really hard problem.  Because culture is complex and there are so many factors that go into determining what culture is, and also it’s deeply personal. So, each individual has their own mix of factors that determine their own culture, right? So, generalizing that to an entire population is also quite hard, I think to do. So, I think we’re still in the very initial stages in terms of actually figuring out how well aligned these models are to different cultures and also trying to sort of align them to any specific target cultures. But it is a hot topic that a lot of people are currently working on.

Sridhar Vedantham: Yeah. You know, while you’re talking and giving me this answer, I was thinking that if you’re going to go culture by culture, first of all, you know, what is the culture, what are you doing about subcultures and how many cultures are there in the world, so I was just wondering how it’s going to even work in the long term? But I guess you answered the question by saying it’s just starting. Now let’s see how it goes.

Sunayana Sitaram: Absolutely.

Sridhar Vedantham:  It’s a very, very open canvas right now, I guess.

Sunayana Sitaram: Yeah.

Sridhar Vedantham: Sunayana, you know you’ve been speaking a lot about evaluation and so on and so forth and especially in the context of local languages and smaller languages and Indic languages and so on. Are these methods of evaluation that you talk about, are they applicable to different language groups and languages spoken in different geographies too?

Sunayana Sitaram: Absolutely. So in the Mega and Megaverse work, we covered 80 languages and many of them were not Indic languages. In fact, in the Megaverse work, we included a whole bunch of African languages as well. So the techniques, you know, would be applicable to all languages for which we have data for which data exists on the web. Where it is challenging is the languages that are only spoken, that are not written, and languages for which there is absolutely no data or representation available, on the web, for example. So, unfortunately, there aren’t benchmarks available for those languages, and so we would need to look at other techniques. But other than that, our evaluation techniques are for, you know,  all languages, all non-English languages.

[Music]

Sridhar Vedantham: There is something that I heard recently from you which again I found extremely interesting. It’s a project called Pariksha, which I know, in Hindi and derived from Sanskrit, basically means test or exam. And I remember this project because I’m very scared of tests and exams, and I’ve always been from school. But what is this?

Sunayana Sitaram: Yes, Pariksha is actually quite a new project. It’s under the project VeLLM that is on universal empowerment with Large Language Models and Pariksha is something that we are super excited about because it’s a collaboration with Karya, which is an ethical data company that was spun off from MSR India. So what we realized a few months ago is that you know there is just so much happening in the Indic LLM space and there are so many people building specialized models either for a single language or for a group of languages like Dravidian languages, for example. And of course, there are also the GPTs of the world which do support Indian languages as well, right. So now at last count, there are something like 30 different Indic LLMs available today. And if you’re a model builder, how do you know whether your Indic LLM is good or better than all of the other LLMs? If you’re somebody who wants to use these models, how do you know which ones to pick for your application? And if you’re a researcher, you know, how do you know what the big challenges are that still remain, right?

Sridhar Vedantham: Right

Sunayana Sitaram: And so to address this, of course, you know, one way to do this is to do evaluation, right. And try to figure out, you know, compare all these models on some standard benchmarks and then try to figure out which ones are the best. However, what we found from our work with Mega and Megaverse is that the Indian language benchmarks unfortunately are usually translations of already existing English benchmarks and also many of them are already present in training data of these large models, which means that we can’t use the already existing benchmarks to get a very good idea about whether these Indic LLMs are culturally appropriate, whether they capture linguistic nuances in the Indian languages or not, right. So we decided to sort of reinvent evaluation for these Indic LLMs and that’s where Pariksha came in. But then how do we scale if we want to actually, you know, get this kind of evaluation done and we were looking at human evaluation to do this, right. And so, we thought of partnering with Karya on this. Because Karya has reached in all the states in India and they have, you know, all of these workers who can actually do this kind of evaluation for different Indian languages. And so, what Pariksha is, it’s a combination of human evaluation as well as automated evaluation. And with this combination we can scale and we can do thousands and thousands of evaluations, which we have already done actually on all of these different models. And so this is the first time actually that all of the different Indic LLMs that are available are being compared to each other in a fair way. And we are able to come up with a leaderboard now of all of the Indic models for each Indic language that we are looking at. So that’s what Pariksha is. It’s quite a new project and we’ve already done thousands of evaluations and we are continuing to scale this up even further.

Sridhar Vedantham: So how does someone, you know, if I have a LLM of my own in any given Indic language, how do I sign up for Pariksha, or how do I get myself to be evaluated against the others?

Sunayana Sitaram: Yeah. So you can contact any of us for that, the Pariksha team. And we will basically include this model, the new model into the next round of evaluation. So what we do with Pariksha is we do several rounds. So we’ve already finished a pilot round and we’re currently doing the first round of evaluations. So we would include the new model in the next round of evaluations. And you know, as long as it’s an open source model or there is an API access available for that model, we can evaluate the model for you. We are also planning to release all the artifacts from Pariksha, including all the evaluation prompts. So even if it is a closed model, you can use these to do your own evaluation as well later to figure out how you compare with the other models on the leaderboard.

Sridhar Vedantham: Right. Quick question. When you say that you’re working with Karya, and you also say that you’re looking at human evaluation along with the regular methods of evaluation. Why do you need human evaluation at all in these situations? It’s just simpler to throw everything into a machine and let it do the work?

Sunayana Sitaram: Yeah, that’s a great question. So we did some work on, you know, making machines evaluators. So basically asking GPT itself to be the evaluator and it does a very good job at that. However, it has some blind spots. So we found that GPT is not a very good evaluator in languages other than English. Basically, it’s not a good evaluator in the languages that it doesn’t do well in otherwise and so using only automated techniques to do evaluation may actually give you the wrong picture. It may give you the wrong sort of trends, right? And so we need to be very careful. And so our ultimate goal is to build evaluation systems and also other kinds of systems in general where humans and LLMs can work together. And so the human evaluation part is to have checks and balances on the LLM evaluation part. Initially, what we are doing is we’re getting the same things evaluated by the human, and the LLM is doing the exact same evaluation. So we have a point by point comparison of what the humans are saying and what the LLM is saying so that we can really see where the LLM goes wrong, right. Where it doesn’t agree with humans. And then we use all of this information to improve the LLM evaluator itself. So we’re really trying to get humans to, you know, do the evaluation, get LLM’s to do the evaluation, use the human data in order to improve the LLM. And then just this continues in a cycle. And the ultimate goal is, send the things to the LLM that it’s good at doing and send the rest of the things that the LLM can’t do to humans who are like the ultimate authority on the evaluation. So it’s like this hybrid system that we are designing with Pariksha.

Sridhar Vedantham: Interesting. OK, so I know we are kind of running out of time. My last question to you would be, where do you see evaluation of LLM’s and where do you see your work going or progressing in the near future?

Sunayana Sitaram: So evaluation for me is a path to understanding what these systems can do and cannot do and then improving them, right. So our evaluations are always actionable. So we try to figure out why something is not working well. So even in the Mega paper, we had lots of analysis about what factors may lead to, you know, lower performance in certain languages, etc. So I see all of this as providing a lot of rich information to model developers in order to figure out what the next steps should be, how they should be designing the next generation of models and I think that has already happened. It’s, you know, systems have already improved from the time we started working on Mega and a lot of the issues that we pointed out in Mega, like tokenization etcetera, now they are well known in the field and people are actually taking steps in order to make those better in these language specific models etc. So I see the work as being, you know, first of all raising awareness about the problems that exist, but also providing actionable insights on how we could improve things. And with Pariksha also the idea is to release all the artifacts from our evaluation so that Indic model developers can use those in order to improve their systems. And so I see that you know better evaluation will lead to better quality models. That’s the aim of the work.

Sridhar Vedantham: Sunaya, thank you so much for your time. I really had a lot of fun during this conversation.

Sunayana Sitaram: Same here. Thank you so much.

Sridhar Vedantham: Thank you.

[Music ends]

The post Podcast: Evaluating LLMs using novel approaches. With Dr. Sunayana Sitaram appeared first on Microsoft Research.

[Music]

Ajay Manchepalli: One thing we have learned is that, you know as they say, necessity is the mother of invention. This is a great example of that because it’s not that we didn’t have remote people before. And it’s not that we didn’t have technology to support something like this. But we have had this Black Swan moment with COVID, which required us to be not in the same physical location at all time and that accelerated the adoption of digital technologies. You can build all the technology you want. But having it at the right time and right place matters the most.

Sridhar Vedantham: Welcome to the Microsoft Research India podcast, where we explore cutting-edge research that’s impacting technology and society. I’m your host, Sridhar Vedantham.

[Music]

Sridhar Vedantham:  The COVID pandemic forced most of us into a new paradigm of work from home and a number of tools to cater to remote work became popular. However, the post pandemic environment has seen interesting scenarios with some people preferring to continue to work from home, some people preferring to return full time to work and a number of people adopting something in between. This hybrid work environment exists today in the workplace as well as in other scenarios such as events. While tools such as Microsoft Teams do extremely well in supporting scheduled and agenda driven work meetings, there is need for a tool that supports a mix of virtual and in-person gatherings in an informal or semi-structured work environment, such as in hallways or at water coolers. In this edition of the podcast, I speak to Venkat Padmanabhan, Deputy MD (Deputy Managing Director) of MSR India and Ajay Manchepalli. Principal Research Program Manager, about a project called HyWay. HyWay’s a system to support unstructured and semi structured hybrid and informal interactions between groups of in-person and remote participants.

Venkat Padmanabhan is Deputy Managing Director at Microsoft Research India in Bengaluru. He was previously with Microsoft Research Redmond, USA for nearly 9 years. Venkat’s research interests are broadly in networked and mobile computing systems, and his work over the years has led to highly-cited papers and paper awards, technology transfers within Microsoft, and also industry impact. He has received several awards and recognitions, including the Shanti Swarup Bhatnagar Prize in 2016, four test-of-time paper awards from ACM SIGMOBILE, ACM SIGMM, and ACM SenSys, and several best paper awards. He was also among those recognized with the SIGCOMM Networking Systems Award 2020, for contributions to the ns family of network simulators. Venkat holds a B.Tech. from IIT Delhi (from where he received the Distinguished Alumnus award in 2018) and an M.S. and a Ph.D. from UC Berkeley, all in Computer Science, and has been elected a Fellow of the INAE, the IEEE, and the ACM. He is an adjunct professor at the Indian Institute of Science and was previously an affiliate faculty member at the University of Washington. He can be reached online at http://research.microsoft.com/~padmanab/ (opens in new tab).

Ajay Manchepalli, as a Research Program Manager, works with researchers across Microsoft Research India, bridging Research innovations to real-world scenarios. He received his Master’s degree in Computer Science from Temple University where he focused on Database Systems. After his Masters, Ajay spent his next 10 years shipping SQL Server products and managing their early adopter customer programs.

For more information about the HyWay project, click HyWay – Microsoft Research (opens in new tab).

For more information about the Microsoft Research India click here.

Related

• Microsoft Research India Podcast: More podcasts from MSR India
• iTunes (opens in new tab): Subscribe and listen to new podcasts on iTunes
• Android (opens in new tab)
• RSS Feed (opens in new tab)
• Spotify (opens in new tab)
• Google Podcasts (opens in new tab)
• Email (opens in new tab)

Transcript

[Music]

Sridhar Vedantham: So, Venkat and Ajay, welcome to the podcast.

Venkat Padmanabhan: Good to be here again.

Ajay Manchepalli: Yeah, likewise.

Sridhar Vedantham: Yeah, both of you guys have been here before, right?

Venkat Padmanabhan: Yeah, it’s my second time.

Sridhar Vedantham: OK.

Ajay Manchepalli: Same here.

Sridhar Vedantham: Great! So anyway, we wanted to talk today about this project called HyWay, which, unlike the way the name sounds, is not related to one of your earlier projects which was called HAMS, which actually had to do with road safety. So, tell us a bit about what HyWay is all about and especially where the name comes from?

Venkat Padmanabhan: Right. Yeah. So, HyWay, we spell it as H Y W A Y. It’s short for hybrid hallway. It’s really about hybrid interaction. What we mean by that is interaction between people who are physically present in a location- think of a conference venue or an office floor- and people who are remote. So that’s where hybrid comes from, and it’s really about sort of enabling informal mingling style, chitchat kind of interaction in such settings, which perhaps other platforms don’t quite support.

Sridhar Vedantham: OK. And why come up with this project at all? I mean there are plenty of other solutions and products and ways to talk to people that already are out there. So why do we really need something new?

Venkat Padmanabhan: Yeah, yeah. So maybe I can give you a little bit of background on this. I think in the very early days of the pandemic, I think in March or April of 2020, you know, all of us were locked up in our respective homes. And obviously there were tools like Teams at Microsoft and equivalent ones like Zoom and so on elsewhere, that allowed people to stay connected and participate in work meetings and so on. But it was very clear very soon that what’s missing is these informal interactions, bumping into someone in the hallway and just chatting with them. That kind of interaction was pretty much nonexistent because, you know, if you think of something like a Teams call or, you know, Zoom call, any of those, it’s a very sanitized environment, right? If, let’s say the three of us are on a Teams call, no one else in the world knows we are meeting, and no one else in the world can overhear us or be, you know, have an opportunity to join us unless they’re explicitly invited. So, we said, OK, you know, we want to sort of make these meeting porous, not have these hard boundaries. And that was the starting point. And then as the months went along, we realized that, hey, the world is not going to be just remote all the while. You know people are going to come back to the office and come back to having face-to-face meetings. And so how do you sort of marry the convenience of remote, with the richer experience of being in-person, and so that’s where hybrid comes in. And that’s something that in our experience, existing tools, including the new tools that came up in the pandemic, don’t support. There are tools that do all virtual experiences. But there is nothing that we have seen that does hybrid the way we are trying to do in HyWay.

Sridhar Vedantham: Right. So, I wanted to go back to something you just said earlier and basically, when you use the term porous, right, and what does that actually mean? Because like you said, the paradigm in which we are used to generally conducting meetings is that it’s a closed, sanitized environment. So, what exactly do we mean by porosity and if you are in a meeting environment, why do you even want porosity?

Venkat Padmanabhan: OK. Maybe I can give an initial answer then maybe Ajay can add. I think we’re not saying every meeting is going to be porous, just to be clear, right. You know when you have a closed-door meeting and you know, maybe you’re talking about sensitive things, you don’t want porosity, right? You want to sort of maintain the privacy and the sanctity of that environment, but when you are trying to enable mingling in a, say, conference setting where you’re sort of bumping into people, joining a conversation, and while you’re having the conversation, you overhear some other conversation or you see someone else and you want to go there. There we think something like porosity and other elements of the design of HyWay, which we can get to in a moment, allow for awareness, right? Essentially, allow you to be aware of what else is going on and give you that opportunity to potentially join other conversations. So that’s where we think porosity is really important. It’s not like it’s something that we are advocating for all meetings.

Ajay Manchepalli: One way to think about this is if you are in a physical space and you want to have a meeting with somebody on a specific topic. You pick a conference room, and you get together and it’s a closed-door conversation. However, when you’re at a workplace or any location for that matter, you tend to have informal conversations, right? So where you’re just standing by the water cooler or you’re standing in the hallway and you want to have discussions. And at that point in time, what you realize is that, even though you’re having conversations with people, there are people nearby that you can see, and you can overhear their conversations. It’s a very natural setting. However, if you’re remote and you’re missing out on those conversations, how do you bring them into play, right? And where it is not predefined or a planned conversation and you’re just gonna happen to see someone or happen to hear someone and join in. And what we talk about is a natural porous nature of air and we are trying to simulate something similar in our system.

Sridhar Vedantham: OK. So, it’s kind of trying to mimic an actual real life physical interaction kind of setting where you can kind of combine some degree of formality and informality.

Ajay Manchepalli: Correct! And many of these platforms like Teams or Zoom and things like that, it is built on this notion of virtual presence, so you could be anywhere, and you could join and have discussions. However, our concept is more aligned with, how do I get to participate in a physical space? So, our whole philosophy is anchored on a physical location, right? If I want to be in the office by a water cooler where we have interesting conversations, how do I get into those conversations? So that physical anchoring is a very critical aspect of our work.

Sridhar Vedantham: Interesting! Now, I’m assuming, given that you’re looking at quite a different paradigm from what existed earlier, you know Teams, for example, how do you go about figuring out what you want to do? What are the kind of design principles involved? How they actually figure out what people really want to participate in?

Venkat Padmanabhan: So, I think that’s a great question. I would say there’s not necessarily one right answer. But you know the way we thought about this, I think Ajay touched on this. You know HyWay is anchored in a physical space. So, at one level, we are just trying to recreate as best as possible that experience in a setting where some people are remote, right? So, what does that mean? Just imagine a physical hallway setting, right? You know, think of a conference, coffee break time, bunch of people out in the hallway in little groups, chatting and moving around. People are very comfortable in that setting because they have a very sort of surprise-free experience. What I mean by that is, I could be chatting with you and with the corner of my eye I can see someone else approaching, I can see Ajay approaching.  So, I’m not surprised by his appearance. And so, if I’m talking something sensitive with respect to him, I have the opportunity to change subjects by the time he comes, right. So, we decided that, OK, you know, that’s really the core of what makes informal mingling work in the physical world. And we sort of thought about how we can recreate that in a hybrid setting and so awareness is a key principle, right? So, enabling the participants, both the people are physically present in the location as well as the people coming in remotely. I guess we have not really talked about how the setup is. The setup is very simple. We just have large screens with speaker, microphone, camera that are installed, multiple of them, maybe in the location of interest. And so, the remote people are on these screens, right? So whether you’re remotely present on a screen or physically there, you have awareness of who else is around, who’s approaching. You can overhear bits of other conversations. I mean, not that you can eavesdrop on people, but you know you can just like in the physical world, you can overhear some conversation, you may not know exactly what’s being said, but you know that, hey, there’s Sridhar there. He’s talking to someone, let me go and join him, right? The second principle is this principle of reciprocity. You know, again with video conferencing tools, often in a business context or in a classroom lecture, it’s perfectly fine to keep your video off and audio off right and just be listening, because there’s a dominant speaker who’s sort of running the show often.

Sridhar Vedantham: Right.

Venkat Padmanabhan: But if you’re enabling chitchat and informal mingling, we believe that doesn’t fly, right. And if I’m going to chit chat with someone and I can’t even see them, and I can’t hear them, that’s not going to go very far. So, we insist on this idea, that of sort of see and be seen and listen and be heard, right. So you know, that sort of two-way street. The third is the notion of agency, right? So, it’s great to have awareness, but then you need to be able to do something with awareness, right? And what do you do in a physical world? You see someone interesting, or you overhear something interesting, you go there if you want to and then if you lose interest, you come back, right? So the ability to move between conversations in a very fluid and natural way, and we do that with a map interface in our system. And we can talk a bit more about the map and how it is a little different from what other systems have done with the map in a bit. But maybe, Ajay, if you want to add anything to this?

Ajay Manchepalli: Yeah. The design principle is trying to ensure that the remote person has the same sort of an experience had they been in-person. And some of the aspects that Venkat just talked about, which is awareness and the aspect of reciprocity, and the ability to change or move to different conversations is very natural in a physical setting and we are trying to mimic what you enjoy in a physical setup even for someone who is remotely trying to participate in that conversation. And so it’s inspired by that.

Sridhar Vedantham: Right, so Venkat just referred to this thing about a map, right? So how is it different from what already exists out there? I mean, what really, how do I put this? I mean, how does the map contribute to the experience of a user?

Venkat Padmanabhan: So, I think the key point is that the map that we have in HyWay, is a representation of the actual physical space where the in-person users are, the physical users are. This is in contrast to, there’s a lot of spatial conversation tools that came up, especially in the pandemic that also have a map-based interface. But the map that is completely synthetic, right? So, by anchoring the map to the physical space, we can enable very interesting mingling scenarios across the physical and virtual divide. What I mean by that is, you know, I could be in a conversation, I could be physically present and I could then walk over to a different location. The virtual user can see where I went, and they can follow along.  Just like they would be able to do if they were physically there, right. And the reverse can happen to, you know, as a physical user, I can say, hey, there’s this remote person I was talking to, now they have moved to a different location. Well, on the map, I’ll know where they’ve gone and I can sort of follow, right. So that correspondence between the virtual view and the physical reality is key, right? That’s what enables this kind of mingling across the physical and virtual divide.

Ajay Manchepalli: To follow up on that, the fact that we anchored to a physical space, I mean the whole premise is that we, a remote person, wants to participate in a particular location of interest implies there has to be some context, right? And so, if I want to go and participate in conversations happening in my office, and specifically, let’s say there is a book club that happens in a particular location. I can sort of contextualize myself, I mean, I have that spatial awareness to that space, right? So, if you give me a map-based experience, I can naturally navigate to the space that I’m interested in going to so that sort of one-on-one to one mapping between the physical space and the map helps us stay oriented.

Venkat Padmanabhan: I’ll add one more thing this. I just got reminded, as Ajay was talking. You know, this idea of creating a map that is anchored to the physical reality and tries to sort of portray that actually has worked out quite well. So, we’ve received feedback from users, but I think there was a user who’s actually a visitor to our lab. He’s professor at the University of Oregon, or, I think, Oregon State University, who had visited on HyWay remotely in 2022. And then when he came a year later, in 2023, in person his first remark was, “Hey, this place looks so familiar. I’ve already been here.” It was his first time, right.

Sridhar Vedantham: OK.

Venkat Padmanabhan: So, the fact that he was able to navigate around our space using this map that corresponded pretty well to the physical reality gave him familiarity in a way that a synthetic map will simply not give you, right. So, I think that’s really the point.

Sridhar Vedantham: Yeah.

[Music]

Sridhar Vedantham: It sounds like when you want to recreate the kind of a physical experience for someone who’s there virtually, and you’re saying that you’ll have multiple spaces where people might congregate or have informal meetings or even formal meetings and so on. I think it logically follows that you’re going to have multiple setups or instances of HyWay in various places right. Now, two questions that flow from that. One is, what is the kind of infrastructure that’s required to enable a HyWay kind of scenario? And I think equally important is how much does it actually cost? I mean, what kind of infrastructure is required for this?

Ajay Manchepalli: That is an extremely important question to think through before we deploy, and the way we thought about this is that, the world already has a lot of digital equipment, right? If you think in the context of an office, you have conference rooms which are set up with AV. And you have various, like if you think of cafeterias, you have screens and things of that nature. And so we wanted to use off the shelf equipment to flight HyWay, right. And so that means all we need is a screen so that people who are physically present can see the remote person participating. And we want an AV device, the camera and the microphone, and the speakers, so that the remote person can hear people as well as the in-person folks can hear the remote person. So, it’s a very standard AV setup that you can think of which is powered by a basic PC. This is a typical setup. The important thing here to note is that you need to place it in places where it doesn’t create blind spots and it’s a natural area for mingling.  Those are the things that we have to think through when we set up such equipment. And as we iterate through this, we are learning different ways to set things up and make it surprise free. And make it more comfortable for people to not feel like they’re being, you know, watched over or something of that nature.

Sridhar Vedantham: It’s interesting you bring that up because my next question was going to be about privacy and letting people know that this thing is on and, you know, there might be someone virtually hanging out there, but how are people actually made aware of this?

Ajay Manchepalli: Right. I mean, there are there are different ways we can approach it. One, when we set up the space, we provide sufficient visual cues to let people know that this is an area where your colleagues can participate remotely, right. In other words, they have a reason to be there because they know that it’s their own colleagues who are going to be participating. #2, we also made a very conscious decision that these experiences have to be ephemeral, right? We do not record, we do not store this information. And if you are in that moment, you get to hear and see people. If you weren’t, you have missed it, just as it would have happened if you were in a physical location.

 Sridhar Vedantham: In real life.

Ajay Manchepalli: In real life, right. So, these two are very important components. Now having said that, you still have cameras out there and you still have a microphone out there and the fact that we have now made it advertised, so to speak, makes them even more nervous, right. Even though people are living in a world with cameras everywhere, from a security standpoint. So, one of the things that we are continuing to work on is this notion of when do we have these systems up and running, right? We don’t just have it up and running all the time. You really need that system up and running only if there is a remote person joining in. And so, what we try to do is allow the physical users to have their natural conversations without a video feed of themselves on the screen, but at the same time, provide visual cues to the remote user saying that there are users present in that location so that, that motivates them to get to that point. And once they are in that location, we start the conversation or start the call, which enables the remote person to join in. But we also provide aural queues, right? We’ll give an audio chime, that gives an awareness to the people locally present saying that, uh, there’s a remote person joining in. This is a new experience. There will be reservations and there will be concerns and I think just as how we transition from, you know, having independent offices to an open desk environment, this is sort of another form of change that we are going through and so we have to be careful how we parse the feedback that we get and keep that in perspective.

Venkat Padmanabhan: I’ll also add just a few things to this, right. You know there are technical design elements in HyWay that try to address this. Ajay touched on no recording for example. We also, for example, don’t do any user recognition, right. So, in our map we represent users with face bubbles. But that’s just detecting a face. We don’t who it is. We don’t try to find out who it is, right we just detect the face, we put on the map at the appropriate location, right? So as when you glance at the map, you can see a face bubble of someone there. If you happen to know who that is, you’ll recognize them. Otherwise, you’ll see, there’s a person there, right? The other aspect is reciprocity, which I touched on earlier, so that also I think helps with privacy in the following sense. HyWay will not allow someone to hide themselves and be eavesdropping on a conversation, because they will not be able to hear anything unless they are seen and heard. So, the fact that they are on the screen in a big video tile will make the people who are physically present aware that, hey, there’s so and so on the screen. And so, you know, from a privacy viewpoint, it makes it more comfortable. The other thing I would say is, the deployment context, right? So, Ajay sort of talked about I think workspace setting where you have colleagues who know each other and so on. We’ve also deployed in events which are confined in space and time where people go to a trade show or a, you know, conference with the expectation that there’ll be other people around and they will see them, and they will hear them and so on. And you know, things like privacy haven’t really been, you know, front and center in those conversations because people are really about it’s really about.

Sridhar Vedantham: They are already in the public space anyway.

Venkat Padmanabhan: They’re in public space, there’s no expectation that they are going to be, you know, shielded from the view of others, right. And so, I think the context matters. I think, in a setting where this is running 24/7 and you know an interaction can happen at any time. Yes, you have to, you know, worry about some of the things that you know Ajay talked about but in other settings it’s a little easier.

Ajay Manchepalli: That’s an extremely important point. It really depends upon the purpose that people are there for, right, in an event you’re going there to learn about something or to meet certain people and you know in that context that there are many people that you’re not aware of and it’s OK.   And yeah, so the context is super important.

Sridhar Vedantham: So, what kind of events have you deployed in and what’s been the user reaction? Has it been all positive or mixed reactions? How’s it been?

Ajay Manchepalli: One of the ethos that we have followed is we didn’t want to build a system and then think of where do we deploy. We have actually sort of taken an opposite approach. That is we have picked a particular event and we have decided that event needs, when we realize that there is a need for remote participation, then we go deploy the system based on the requirements that have been called out. And we built our system based on that. So, we have been deploying in events where people want to share information about the work that they have done. And you know, things like, let’s say your colleagues, all the researchers, they are working on various topics, and we want to let people be aware of what the work is. In that context, we have set up things like poster sessions where it is very natural for us to have booths and have topical presentations that’s happening. In this context, initially we decided that, OK, it makes sense to have the ability for a remote person to join in and participate in such events. But through the events, we realized that, once we set up such a system, there are remote presenters who would like to also participate. And so, then we basically modified the system to support not just remote attendees, but to also enable remote presentations, where in-person attendees would actually interact with a remote presenter over the screen. So, these are the things that we have learned through the events. And the surprising results from this is, once we deployed in such events, people experienced it and based on the experience we had the pull of having more events. We didn’t have to go and advertise, so to speak. The very experience made people realize the value that we were bringing to the table. In terms of user feedback, it has been overwhelmingly positive. But at the same time, it’s a research prototype. So, most of the sort of feedback we have been receiving is more about the system, the audio quality and things like that, which is a systemic improvement that we can do over time. And we also have received feedback which is more in the context of, “Oh, I’m able to do this. Can you also add this” right? You know it’s additional capabilities that they would like to see. And especially in the context of events, we haven’t really seen any hesitations in terms of the privacy aspect, so to speak. Apart from such events, we have also tried to put it in informal mingling type of events where, OK, there is end of your party and people are getting together and just talking about all the great moments they’ve had through the year. And people who are remote were able to also participate and mingle with people. And so those are the other type of setups where we have deployed these things. All in all, I would say we have had deployments every couple of months. And each deployment we learn something new and we improve upon the system. Initially, we used to just have their name initials showing up on the map. Then there was feedback that visual cues of who the person is there would be useful because they can recognize that person rather than having their initials and things like that.

Sridhar Vedantham: So, that’s where the face bubble came in.

Ajay Manchepalli: Face bubble came in, yeah.

Venkat Padmanabhan: I’ll also add one other use case. So far, we have one user for it, but a pretty important user, which is what I would call as remote visits, right? So think of a busy exec who has a global organization and they don’t have the time to fly all the way to the other location to visit. You know, obviously you can do remote visit on tools like Teams and Zoom and so on. But when our CEO Satya Nadella came in January of this year (2023), he was remote and he wanted to visit our lab. We decided to do it on HyWay, right. And that worked very well. Essentially, he got on HyWay and walked around the floor of the lab, you know, met with various people. And you know that gave him a sense of the place in a way that just presentations on a call would not have given, right. And that was his reaction, was very positive, and that also gave some sort of momentum to our efforts and sort of finding more use cases for this.

Ajay Manchepalli: Yeah, got people excited. There is one other scenario that we have been exploring in the recent past and that is in the context of workspace. That is, you have an open office sort of a structure where you have a large table where you have 6 to 8 employees, our colleagues sitting together and working, right. In this sort of a setting, if, with the flexible come into work sort of arrangement that we see these days, you invariably end up seeing only maybe 50-70% of the people showing up in person, and the rest of them working remotely on any given day. And so in order to ensure at the end of the day, we are all about increasing productivity for our for each person, right, on this planet. So. From that context, if we are able to enable the remote employees who are sitting at their home by themselves to sort of feel part of the team, how do we bring them into the context of their workspace?  So here it’s not informal mingling, but it is more of being present along with your colleagues for a long duration of time where you have serendipitous moments of discussions that happens, which you would have missed if you were staying remote. And that is another scenario that is turning out to be an important scenario for us to look at. Because there is that need that exists and that requires a different way to enable the sort of interactions that we expect within the context of a team.

Sridhar Vedantham: OK.

 Venkat Padmanabhan: It’s really unscheduled interactions that can happen anytime with colleagues.

Sridhar Vedantham: Right. Much like we keep doing in office actually.

Ajay Manchepalli, Venkat Padmanabhan: Yes, exactly right.

Sridhar Vedantham: OK, so where do you see this going in the future? I mean, do you have a road map in mind? Do you really have a, say, a longer-term plan?

Venkat Padmanabhan: I think we are very much in the phase where we are you know augmenting the system as we go along, right. So Ajay said, you know we’ve been in this habit of deploying and learning through deployment. So you know we’ve done a bunch of deployments, we’ve got feedback, we’ve had ideas. And so, at this stage we are just looking to, you know, make the system better. And you know, we’ve deployed a bunch of internal events. One day we might actually take it to external events, external conferences, academic conferences, and so on, and hopefully have a larger audience experience this. And then we’ll see where it takes us, right. You know the feedback, as Ajay said has been very positive and what’s been particularly satisfying is that it is very clear from the feedback both from users but also from the event organizers, that there’s an unmet need here. There is something that people want to do, but existing tools are not doing, which is why they’re coming to us. And as I said, we are a small team, we have not advertised ourselves, but people are coming and asking for it, right, internally at Microsoft so far. So that tells us there is unmet need here. So, we think if we do the right things and make the system even better than it is, good things will happen.

Ajay Manchepalli: And the fact that we are able to deploy and there is more ask for it in spite of the fact that you have microphones and cameras everywhere, tells us that the value that we are bringing outweighs the risks that people perceive, right. And that’s a very strong case for the unmet need that Venkat was referring to.

Sridhar Vedantham: People want to meet people.

Ajay Manchepalli: Yeah.

Venkat Padmanabhan: (Laughs) Well said.

Sridhar Vedantham: All right. So we are kind of coming to the end of this. Is there any last thoughts you want to leave the listeners with?

Venkat Padmanabhan: Maybe I’ll just go back to the earlier point- this is like a more of a specific point. I think when we were discussing hardware requirements and you know deployment models and so on, one of the things we decided early on is to of course keep things simple- low-cost commodity hardware and so on. But you know related to that is this idea that we don’t want to impose on the people who are physically present in any way, right. So they just show up and talk and mingle just the way they’ve done. The only difference is there is an additional screen, maybe in their vicinity, that allows remote people to join. And you know, the unexpected benefit of that really has been that, you know, our system allows bootstrapping for free. So the physical people vote with their feet, they are there regardless of HyWay. They would have been there anyway in the hallway, chatting with each other. But because HyWay is really about instrumenting the environment with these screens and not really imposing on the users, they’re automatically, you know, represented in the system. And so now there’s a strong incentive for me as a remote user to get on HyWay. And it’s not even an app. You just go to the web page, you get on HyWay, because there are other people to interact with. This is in contrast to an approach where you need everyone to do something like install an app or get some hardware or you know, have a wearable computer or whatever to participate. Because then you have the network effects sort of biting youthe wrong way, which is, you know, you start with a user base of zero and then you have to really get to a critical mass. We’re getting critical mass for free.  And we didn’t really think of it that way on day one, but that realization has come as we have deployed and sort of learnt from that.

Sridhar Vedantham: OK.

Ajay Manchepalli: From HyWay one thing we have learned is that, you know as they say, necessity is the mother of invention. This is a great example of that because it’s not that we didn’t have remote people before. And it’s not that we didn’t have technology to support something like this. But we have had this Black Swan moment with COVID, which required us to be not in the same physical location at all time and that accelerated the adoption of digital technologies. That also accelerated the possibility of something like this. And it gave us that flexibility and that led to something that it almost became a no brainer to have such a system. And that has been a key aspect, is that you can build all the technology you want. But having it at the right time and right place matters the most.

Sridhar Vedantham: And you know, after seeing deployments of HyWay in the lab and stuff, right? I think one thing that kind of came home to me was the fact that you don’t have to invest in some extremely expensive infrastructure and fancy VR headsets. You don’t need an augmented reality, virtual reality and all that to actually create a kind of a hybrid workspace.

Venkat Padmanabhan: Yeah, there’s a spectrum here and you know, we are at the point where we are focusing on low cost so that we can scale. Obviously, you can do other things with more fancy hardware, but they may be harder to scale. And you know it’s sort of the philosophy of a lot of our work. If you remember the previous project, we talked about the HAMS project. Again just a smartphone, rather than instrumenting the vehicle, just put a smartphone and sort of it observes how you’re driving and gives you safety feedback, right? So it’s an approach that we think works well in many cases.

Ajay Manchepalli: And when you build something that you want to use every day, it’s super exciting to be in such a project. I think we have a bunch of motivated individuals to make this successful.

Venkat Padmanabhan: So I should say, yeah, you know, I think before we close, you know, I do want to give a huge shout out to a lot of people, but in particular to the team, the HyWay team, right from day one, they’ve really been the backbone of the project. Bunch of young people, you know, researchers,  Research Fellows, RSDEs, that is research software development engineers, all pulled together to do this, and it’s a small team. I mean, it’s a pretty small team that’s done amazing things. So, I really want to call them out.

Sridhar Vedantham: Great! Thank you for yet another fascinating conversation, Venkat and Ajay.

Venkat Padmanabhan: Thanks, Sridhar. It was a real pleasure again.

Ajay Manchepalli: Pleasure is all ours. Thank you.

[Music ends]

The post Podcast: HyWay- Enabling Mingling in the Hybrid World appeared first on Microsoft Research.

Road safety is a very serious public health issue across the world. Estimates put the traffic related death toll at approximately 1.35 million fatalities every year, and the World Health Organization ranks road injuries in the top 10 leading causes of death globally. This raises the question- can we do anything to improve road safety? In this podcast, I speak to Venkat Padmanabhan, Deputy Managing Director of Microsoft Research India and Akshay Nambi, Principal Researcher at MSR India. Venkat and Akshay talk about a research project called Harnessing Automobiles for Safety, or HAMS. The project seeks to use low-cost sensing devices to construct a virtual harness for vehicles that can help monitor the state of the driver and how the vehicle is being driven in the context of the road environment it is in. We talk about the motivation behind HAMS, its evolution, its deployment in the real world and the impact it is already having, as well as their future plans.

Venkat Padmanabhan is Deputy Managing Director at Microsoft Research India in Bengaluru. He was previously with Microsoft Research Redmond, USA for nearly 9 years. Venkat’s research interests are broadly in networked and mobile computing systems, and his work over the years has led to highly-cited papers and paper awards, technology transfers within Microsoft, and also industry impact. He has received several awards and recognitions, including the Shanti Swarup Bhatnagar Prize in 2016, four test-of-time paper awards from ACM SIGMOBILE, ACM SIGMM, and ACM SenSys, and several best paper awards. He was also among those recognized with the SIGCOMM Networking Systems Award 2020, for contributions to the ns family of network simulators. Venkat holds a B.Tech. from IIT Delhi (from where he received the Distinguished Alumnus award in 2018) and an M.S. and a Ph.D. from UC Berkeley, all in Computer Science, and has been elected a Fellow of the INAE, the IEEE, and the ACM. He is an adjunct professor at the Indian Institute of Science and was previously an affiliate faculty member at the University of Washington. He can be reached online at http://research.microsoft.com/~padmanab/ (opens in new tab).

Akshay Nambi is a Principal Researcher at Microsoft Research India. His research interests lie at the intersection of Systems and Technology for Emerging Markets broadly in the areas of AI, IoT, and Edge Computing. He is particularly interested in building affordable, reliable, and scalable IoT devices to address various societal challenges. His recent projects are focused on improving data quality in low-cost IoT sensors and enhancing performance of DNNs on resource-constrained edge devices. Previously, he spent two years at Microsoft Research as a post-doctoral scholar and he has completed his PhD from the Delft University of Technology (TUDelft) in the Netherlands.

More information on the HAMS project is here: HAMS: Harnessing AutoMobiles for Safety – Microsoft Research

For more information about the Microsoft Research India click here.

Related

• Microsoft Research India Podcast: More podcasts from MSR India (opens in new tab)
• iTunes (opens in new tab): Subscribe and listen to new podcasts on iTunes
• Android (opens in new tab)
• RSS Feed (opens in new tab)
• Spotify (opens in new tab)
• Google Podcasts (opens in new tab)
• Email (opens in new tab)

Transcript

Venkat Padmanabhan: There’s hundreds of thousands of deaths and many more injuries happening in the country every year because of road accidents. And of course it’s a global problem and the global problem is even bigger. The state of license testing is as that by some estimates of public reports, over 50% of license are issued without a test or a proper test. So we believe a system like HAMS that improves the integrity of the testing process has huge potential to make a positive difference.

[Music]

Sridhar Vedantham: Welcome to the Microsoft Research India podcast, where we explore cutting-edge research that’s impacting technology and society. I’m your host, Sridhar Vedantham.

[Music]

Sridhar Vedantham: Road safety is a very serious public health issue across the world. Estimates put the traffic related death toll at approximately 1.35 million fatalities every year, and the World Health Organization ranks road injuries in the top 10 leading causes of death globally. This raises the question- can we do anything to improve road safety? In this podcast, I speak to Venkat Padmanabhan, Deputy Managing Director of Microsoft Research India and Akshay Nambi, Principal Researcher at MSR India. Venkat and Akshay talk about a research project called Harnessing Automobiles for Safety, or HAMS. The project seeks to use low-cost sensing devices to construct a virtual harness for vehicles that can help monitor the state of the driver and how the vehicle is being driven in the context of the road environment it is in. We talk about the motivation behind HAMS, its evolution, its deployment in the real world and the impact it is already having, as well as their future plans.

[Music]

Sridhar Vedantham: Venkat and Akshay, welcome to the podcast. I think this is going to be quite an interesting one.

Venkat Padmanabhan: Hello Sridhar, nice to be here.

Akshay Nambi: Yeah, hello Sridhar, nice to be here.

Sridhar Vedantham: And Akshay is of course officially a veteran of the podcast now since it’s your second time.

Akshay Nambi: Yes, but the first time in person so looking forward to it.

Sridhar Vedantham: Yes, in fact I am looking forward to this too. It’s great to do these things in person instead of sitting virtually and not being able to connect physically at all.

Akshay Nambi: Definitely.

Sridhar Vedantham: Cool, so we’re going to be talking about a project that Venkat and you are working on, and this is something called HAMS. To start with, can you tell us what HAMS means or what it stands for, and a very brief introduction into the project itself?

Venkat Padmanabhan: Sure, I can take a crack at it. HAMS stands for Harnessing Automobiles for Safety. In a nutshell, it’s a system that uses a smartphone to monitor a driver and their driving, with a view to improving safety. So we look at things like the state of the driver, where they’re looking, whether they’re distracted, and so on. That’s sort of looking at the driver. But we also look at the driving environment, because we think, to truly attack the problem of safety, you need to have both the internal context inside the vehicle as well as the external context. So that’s the sort of brief description of what HAMS tries to do.

Sridhar Vedantham: Ok. So, you spoke about a couple of things here, right? One is the safety aspect of, you know, driving both internal and external. When you’re talking about this, can you be more concise? And especially, how did this kind of consideration feed into, say, the motivation or the inspiration behind HAMS?

Akshay Nambi: Yeah, so as you know, road safety is a major concern, not just in India globally, right?  And when you look at the factors affecting roads safety, there is the vehicle, there’s the infrastructure and the driver. And majority of the instance today focus on the driver. For instance, the key factors affecting road safety includes over speeding, driving without seatbelts, drowsy driving, drunken driving. All centering around the driver. And that kind of started that was motivating towards looking at the driver more carefully, which is where we build the system HAMS, which focuses on monitoring the driver and also how he’s driving.

Sridhar Vedantham: And India in particular has an extremely high rate of deaths per year, right, in terms of in terms of roads accidents.

Akshay Nambi: Yes, it’s on the top list. In fact, around 80,000 to 1.5 lakh people die every year according to the stats from the government. Yeah, it’s an alarming thing and hopefully we are doing baby steps to improve that.

Venkat Padmanabhan: In fact, if I may add to that, if you look at the causes of death, not just road accidents, diseases and so on, road accidents are in the top 10. And if you look at the younger population, you know people under 35 or 40, it’s perhaps in the top two or three. So it is a public health issue as well.

Sridhar Vedantham: And that’s scary. Ok, so how does this project actually work? I mean, the technology and the research that you guys developed and the research that’s gone into it. Talk to us a little bit about that.

Venkat Padmanabhan: Sure yeah, let me actually wind back, maybe 10-15 years to sort of when we first started on this journey, and then talk more specifically about HAMS and what’s happened more recently. Smartphones, as you know, have been around for maybe 15 years. A bit longer maybe. And when smartphones started emerging in the mid 2000s and late 2000s, we got quite interested in the possibility of using a smartphone as a sensor for, you know, road monitoring, driving monitoring and so on. And we built a system here at Microsoft Research India back in 2007-08, it’s called Nericell, where we used a leading-edge smartphone of that era to do sensing. But it turned out that the hardware then was quite limited in its capabilities in terms of sensors, even accelerometer was not there. We had to pair an external accelerometer and so on. And so the ability for us to scale that system and really have interesting things come out of it was quite limited. Fast forward, about 10 years, not only did smartphone hardware get much better, AI and machine learning models that could process this information became much better and among the new sensors in the newer edge smartphones or the cameras, the front camera and the back camera. And machine learning models for computer vision have made tremendous progress. So that combination allowed us to do far more interesting things than we were able to back then. Maybe Akshay can talk a bit more about the specific AI models and so on that we built.

Akshay Nambi: Yeah, so if you compare the systems in the past to HAMS, what was missing was the context. In the past, systems like what Venkat mentioned- Nericell, right, it was correcting the sensor data, but it was lacking context. For example, it could tell did the driver did this rash braking or not, but it could not tell, did he do it because somebody jumped in front of the vehicle, or was he distracted? These cameras new smartphones have can provide this context, which makes these systems much more capable and can provide valuable insights. And in terms of specific technology itself, we go with commodity smartphones, which have multiple cameras today. The front camera looking at the driver, the back camera looking at the road, and we have built numerous AI models to track the driver state, which includes driver fatigue and driver gaze, where the driver is actually looking. And also with the back camera we look at how the driver is driving with respect to the environment. That is, is he over speeding, is he driving on the wrong side of the road and so on.

Sridhar Vedantham: So, this is all happening in real time.

Akshay Nambi: The system can support both real time and also offline processing. And as you know smartphones are intelligent edge devices, but still they have limited processing power. So, we decide what some of the capabilities should run in real time, and some can be offloaded to the cloud. Or some could be for offline processing.

Sridhar Vedantham: OK.

Venkat Padmanabhan: I want to sort of make a distinction between our ability to run things in real time, which has Akshay said, you know, many of our, actually much of our research was in making the computation inexpensive enough so that you can run real time and the user interface. So, we explicitly decided early on in our journey that we did not want a system that intervened in real time, and you know provided alerts, because the bar for that, if you will, is very high. In the sense that you don’t want to make a mistake, like if you alert a driver to driver and that alert is actually a mistake, you might actually cause an accident, right? And since we were shooting for a low-cost system with just a smartphone and so on, it did not seem like a reasonable thing to sort of aim for that. What we really aim for is processing that is efficient, that actually doesn’t overheat the phone, you know, the processing can sometimes just cause a smartphone to meltdown. But at the same time, depending on the context and we’ll get to, I guess our driver testing application soon hopefully, we can offload computation to either a more capable edge device nearby or to the cloud as Akshay said. and we definitely want to leverage that. We’re not sort of bound to just the smart phone for compute.

Sridhar Vedantham: Right, so you know you spoke about the fact that you’re using commodity hardware to do all this right? And it’s always fascinated me that today’s consumer device, basically the commodity hardware that you get, is capable of doing a lot of stuff. But even then, there must have been challenges in taking you know, maybe a 25,000 Rupee or 20,000 Rupee phone just off the market and trying to get it to do things that, uh, sound like they should be running in the cloud and not on the phone, frankly. Did you have any of these challenges?

Akshay Nambi: Oh, numerous of them. So, to start with the low-cost smartphones, as you turn on cameras, most of you would have realized the phone gets heated up much quickly than in a normal setup. While the system setup itself is very simple, which is just a smartphone, to build a system on top of smartphone, there are numerous challenges. Starting with real world conditions, that is, there is different lighting as you drive in on roads, there is daytime, nighttime. How does your algorithm adapt to these conditions? There are different types of vehicles, hatchback, SUV. How does your algorithm adapt to these. Different driving seating positions, the way you mount the smartphone in the vehicle. All of these can change, right? So getting your AI models to work in such a dynamic setup is the biggest challenge. And one of the key research in HAMS is to address these challenges in a practical way. That’s been one of our key focus. Second, coming to the hardware itself, since we want to do some of these processing on the smartphone itself, you have to come up with algorithms that are efficient. Which means, today smartphone’s cameras can generate 30 frames per second. But the hardware, the compute power is not there to process all the 30 frames. So you have to come up with intelligent algorithms to decide, which frame do you want to process, which frame you want to discard, or which frame you want to apply a lower algorithm compared to a better algorithm. So there are lot of these divisions which have to go through to build this system.

Sridhar Vedantham: Right.

Venkat Padmanabhan: Just to add to what Akshay said, you know if I step back right, there are, I would say, two major pillars of our research. One is being adaptive. For example, Akshay talked about the processing being expensive. Let’s say you’re trying to do vehicle ranging. We’re trying to figure out whether the driver is tailgating, being too close to the vehicle in front. There are very good machine learning models that will do object detection, you know, find the vehicles in front of you in a in a frame and so on, but they’re expensive. Instead, we could combine that approach with a much less expensive tracking algorithm so that once you found an object, you just use the cheaper tracking algorithm for a while because the vehicle in front of you is not going to disappear. If it’s in front of you now chances are that for the next several seconds it’ll be there, so this is adaptivity. The other aspect is auto calibration or calibration in general. As Akshay said, you know, vehicle geometry, the mounting of the phone, the driver seating position, all that changes and it is not practical to recalibrate each time a driver mounts a phone and starts, obviously. So we need ways of automatically calibrating and learning system. So, I would say a lot of our technical work and all the research we’ve published falls in these two buckets.

[Music]

Sridhar Vedantham: One thing that either one of you mentioned I don’t remember exactly who it was, but you mentioned this thing about being able to detect if a driver is, say, drowsy. How does that actually work? Because you know it just sounds a little science fictiony to me, right? How a phone is going to be sitting there mounted on the dashboard or on the window or windshield of a car and if someone magically telling you whether the driver is sleepy or drowsy or whether the driver is doing the right thing while driving.

Venkat Padmanabhan: You’re right. I mean, knowing the true internal state of a driver is not easy, but there are outward manifestations of how tired someone is or how sleepy someone is. Obvious ones are, your eyes sort of drooping and also yawning and things like that. And so those are things that we can pick out by just having a camera looking at the driver’s face. We’re not claiming that this is 100% percent perfect in terms of knowing driver state, but these are good indicators. In fact, we have some interesting data from our work and maybe you know actually can talk about this long drive he went on and where these outward signs actually correlated with his own perception. 

Akshay Nambi: Yeah, that’s true. So there are several studies which has looked at eye blinking and yawning patterns to say the state of the driver and we developed this one and in fact we were going for an interstate travel where we deployed HAMS in our own cab. It was early morning, so the driver was just awake, and he was fresh where he was driving well and our algorithms were also detecting that he was active. And we stopped for breakfast and then the system started beeping, directing the eye blinking and yawning state was much higher and, we were in the cab and we did not notice it. And the system was able to detect. He was an experienced driver, yes, that made a lot of sense, but still he was drowsy. 

Venkat Padmanabhan: As a safety conscious person, Akshay should have stopped the cab right then, but he was part of the research project. He wanted the data so he kept going. He kept going, yeah.

Sridhar Vedantham: He is truly committed to the science. Cool, have there been other projects to do with traffic and road safety and so on anywhere in the world? And how does what you guys are doing differ from those things?

Venkat Padmanabhan: Yeah, so let me start and maybe Akshay can add to it. It’s a very rich space right. Around the same time we started Nericell in the mid-2000s, a bunch of other people started startups as well as university groups like at MIT and Princeton and so on. IIT Bombay had an active project that was looking at similar things. And then, as I said, HAMS, we did sort of about 10 years later. The biggest distinction I would say is what Akshay touched on in the beginning which is that compared to a lot of these existing systems that people use, including what insurance companies now use, at least in the western world, we have these camera sensors that allow us to get the context of a driver. I think Akshay gave an example of the driver being distracted. I’ll give you an even simpler example. Now think of something like speeding. You would imagine a speed is very easy to get. You have a GPS device, pretty accurate, that will give you speed. But when you talk about speeding and the safety issues related to that, it is not just your vehicle speed.

Sridhar Vedantham: It’s contextual.

Venkat Padmanabhan: It is how you’re doing relative to other vehicles. If you know the speed limit is 80 kilometers, others are going at 90 or 100. It is safer to go at 90 or 100 than to be 80, right?

Sridhar Vedantham: Yeah.

Venkat Padmanabhan: So that context is something that you get only with camera-based sensing and that for the most part the other work is not looking at. I would say we are perhaps among the earliest to look at that as part of the mix for this problem.

Sridhar Vedantham: Ok, and uh, I know this project is, I mean, you guys have been working on this project for about three years, four years now?

Venkat Padmanabhan: Oh, longer. We actually started in 2016.

Sridhar Vedantham: 2016.

Venkat Padmanabhan: So it’s a bit longer, yeah.

Sridhar Vedantham: Right, and I know that before we had this unfortunate pandemic, there used to be all these weird posts and signs in the basement of the office here, which I was told Akshay put up. What are those for and they told me that, I’m not supposed to run over those signs or move them and all that because there for a very important research project that Akshay is running called HAMS. What were you guys doing there in the basement?

Akshay Nambi: Right. While we spoke about various detectors in terms of understanding the driver state how he’s driving, one of the key things which we have developed Is how the driver drives. Specifically, we look at the trajectory of the driving itself. When we talk about trajectory, today you can use GPS to get the trajectory of how the vehicle is being driven. The accuracy of these GPS devices are in meters. Especially if we are now trying to understand how the driver is parking in a reverse parking position, or in a parallel parking position, you want to understand the trajectory in centimeter level. How many forwards he took, how many reverses he took. And to do that we have come up with a way where we use visual cues, which is basically the features in the environment plus some of these markers which you have seen in the basement, which provides you very accurate localization up to a few centimeters. And that’s how we were able to get the accurate trajectory. And now this trajectory can be used for various applications. One, as I said, for seeing how the driver is parking, it could be used for how the driver is driving in an open environment, and this was mainly to see how new drivers are actually driving the vehicle.

Sridhar Vedantham: OK.

Venkat Padmanabhan: You can think of these markers, I guess they’re called fiducial markers as you know, the beacons of a lighthouse. It gives you a reference point, right? So you can locate yourself accurately down to centimeter level using these as reference points.

Sridhar Vedantham: Ok. Now I also know that you’ve implemented HAMS at scale at various places. Uh, can we talk a bit about that, where it’s been implemented and what it’s being used for?

Venkat Padmanabhan: That’s a good question, Sridhar. Let me provide the broader context here, and then maybe Akshay can chime in with the details. As I said, we started HAMS in 2016. And in the in the year that followed, we looked at many different possible potential applications. For example, fleet monitoring, driver training and so on. Now, as luck would have it, we got connected with a very interesting organization called Institute for Driving and Traffic Research, which is run by Maruti Suzuki, which is India’s largest car manufacturer, and they are very much interested in issues of road safety and also being Maruti and such a big player in the vehicle business, they’re very well connected with the government. So, in late 2018 we went to the Ministry of Road Transport and Highways along with them and met with the senior bureaucrat and what was supposed to be a 15 minute meeting went on for over 2 hours because the person we met with really liked the potential of HAMS. In particular, in the context of driver testing. As you know, before you’re granted a license to drive, you’re supposed to be tested, but the reality in India is that because of the scale of the country and population, and so on, anecdotally, and even some studies have shown, that many licenses are issued without a proper test or without a test at all. Which obviously means untested and potentially unsafe drivers are on the road and they’re contributing to the scale of the problem. Now the government is alive to this problem, and they’re quite interested in technology for automation and so on. But the technology that people were using was quite expensive and therefore difficult to scale. So from this conversation we had in 2018, what emerged is that our simple smartphone based solution could be used as a very low cost, but at the same time, high coverage solution. When I say high coverage, it can actually monitor many more parameters of a drivers’ expertise in driving than the existing high-cost solutions. So that really got us started and maybe Akshay can talk about where that journey led us.

Akshay Nambi: This meeting what Venkat mentioned with the ministry in Delhi, led us to talking to the government of Uttarakhand who were looking to set up a greenfield automated license testing track in Dehradun. This was the first implementation of HAMS to provide automated license testing. And remember, this was a research project. Taking this research project to a actual deployment with the third-party organization and government being involved was a major task. It’s not just about technology transfer, it’s making the people on the ground understand the technology and run with it every day. We have spent several months with the government officials to translate the research project to a actual real-world deployment, which also included translating the parameters in their driver license testing guidelines to something which can be measured and monitored in real-world. For instance, this would mean, is the driver wearing a seat belt and the amount of time it took to complete a particular maneuver. All of these things need to be monitored by the system. And this translation was one of the biggest challenges taking their research project to a real-world deployment. This deployment went live in July 2019, where the entire test was completely automated. Automated here means that there is no inspector sitting in the vehicle. So, the inspector comes into the vehicle, deploys smartphone and the exits. You as a candidate drive within a confined area which has multiple maneuvers, and the smartphone monitors how you are driving. And has a bunch of parameters which is being defined based on which you would be marked. At the end of the test, basically, a report will be automatically generated which says which maneuvers you passed, which maneuvers you failed along with video evidence why you failed, will be provided to the candidate. And the final result will be uploaded to the central government for issuing the licenses.

Sridhar Vedantham: This is very interesting. What has been the reaction of people to this? I’m sure that when people suddenly saw that they’re going to be evaluated by a smartphone, it must have thrown them for a loop, at least in the initial stages.

Akshay Nambi: Much to our surprise, it was completely opposite. They were very much welcoming their smartphone than an inspector in the vehicle. (Laughs)

Venkat Padmanabhan: I think people trust a machine more than a person. Because they feel that person perhaps can be biased and so on, whereas the machine they just trust. In fact, the comments we got also said, you know, look, I failed the test, but I like the system.

Sridhar Vedantham: So people seem to be happy to remove the human error part of the thing. Out of the equation.

Venkat Padmanabhan: The subjectivity, right?

Sridhar Vedantham: Yeah, the subjectivity, yeah.

Venkat Padmanabhan: They feel the system is objective. The other thing I should mention, which obviously we didn’t plan for, and we didn’t anticipate. After COVID happened, this idea that you take the test without anyone else in the vehicle gained a new significance because you know things like physical distancing became the norm. You could take a test with just a smartphone and not have to worry about sitting next to an inspector or inspector worrying about sitting next to a driver.

Venkat Padmanabhan: And that was an unexpected benefit. Of our approach.

Sridhar Vedantham: Right? That’s very interesting. I never thought of this, although I’ve been tracking this project for a while.

Venkat Padmanabhan: Neither did we. It just happened and then we realize how you know, in retrospect, it was a good idea.

Sridhar Vedantham: Yeah. And what’s the road map? Where does the project go now?

Venkat Padmanabhan: Yep. Akshay talked about Dehradun deployment that happened in 2019. That really sort of caught the attention of several state governments. In fact, they sent their people to Dehradun and to see how the system was working and came back quite impressed. So there was deployments in Bihar, deployments happening in Andhra Pradesh, and some sites in Haryana and several other states that are in discussion to deploy the system. So at this point we have four RTOs that are live and with a couple of more than almost live, they’re pretty much ready to go. And about a dozen more that are in the works in various stages. But of course there are thousand RTOs in the country. So, there’s still a long way to go, and one of the challenges is that this has to proceed on a state by state basis, because it is…

Sridhar Vedantham: A state subject.

Venkat Padmanabhan: It’s a state subject, exactly. But we are working with external partners who we have enabled with the HAMS technology.

Sridhar Vedantham: Venkat, it sounds like this project has some serious potential for large societal impact.

Venkat Padmanabhan: That’s indeed the case Sridhar. In fact, we think there’s huge potential here for beneficial impact, and that’s what really been driving us. Just to give you context for the numbers. The scale of the problem we already talked about, there’s hundreds of thousands of deaths and many more injuries happening in the country every year because of road accidents. And of course, it’s a global problem and the global problem is even bigger. The state of license testing is as that by some estimates of public reports, over 50% of licenses are issued without a test or a proper test. So we believe a system like HAMS that improves the integrity of the testing process has huge potential to make a positive difference. Now, where we are in the journey today is that we have done about 28,000 automated license tests using HAMS across all these states where it’s been deployed. But an estimated 10 million or more license tests or licenses are issued in the country every year. So we think that by scaling to the 1000 plus RTOs that I talked about earlier, we can actually, potentially touch a majority, or perhaps even all of these license tests that are happening and license being issued, and thereby have much safer roads in the country by having, you know, drivers are well tested and really ready to drive. Making the only ones who are on the road.

Sridhar Vedantham: Fantastic, now we are coming towards another podcast. Are there any thoughts that you’d like to leave the listeners with before we wind up?

Venkat Padmanabhan: Sure, I can share something and then maybe Akshay can add to it as well. I would say you know stepping back from the specific use case of HAMS and you know road safety and so on, what this experience has taught us is that, if you take technology and mate it with a significant, a societally significant problem. In this case, road safety, but really understand the problem, work with partners like we did. You know, we worked with IDTR, the Maruti IDTR, we worked with other external partners, talked to the government, multiple governments at the center and the state, and so on. Really understand the problem, understand technology and bring it together in a meaningful way, we can make a huge difference, and that’s really quite inspirational for us because it tells us that there’s a lot of good we can do as technologists and researchers.

Akshay Nambi: Nothing much to add to Venkat, he nicely summed it up. But I think one just minor point would be that we don’t have to look for problems elsewhere. Problems are just right next to us. And picking up these societal impact problems, have lot of value.

Sridhar Vedantham: OK, fantastic thank you both for your time.

Venkat Padmanabhan: Thanks Sridhar. It’s been a pleasure.

Akshay Nambi: Thanks Sridhar, this was very good.

The post Podcast: HAMS- Using Smartphones to Make Roads Safer. With Dr. Venkat Padmanabhan and Dr. Akshay Nambi appeared first on Microsoft Research.

Episode 012 | May 30, 2022

Sridhar Vedantham: Welcome to the MSR India podcast. In this podcast, Ravishankar Krishnaswamy, a researcher at the MSR India lab, speaks to Neeraj Kayal. Neeraj is also a researcher at MSR India and works on problems related to or at the intersection of Computational Complexity and Algebra, Number Theory and Geometry. He has received multiple recognitions through his career, including the Distinguished Alumnus award from IIT Kanpur, the Gödel prize (opens in new tab) and the Fulkerson Prize (opens in new tab). Neeraj received the Young Scientist Award from the Indian National Science Academy (opens in new tab) (INSA) in 2012 and the Infosys Prize in Mathematical Sciences in 2021. Ravi talks to Neeraj about how he became interested in this area of computer science and his journey till now.

Neeraj Kayal: It’s just a matter of time before we figure out how computers can themselves learn like humans do. Just human babies, they have an amazing ability to learn by observing things around them. And currently, despite all the progress, computers don’t have that much ability. But I just think it’s a matter of time before we figure that out, some sort of general artificial intelligence.

For more information about the Microsoft Research India click here.

Related

• Microsoft Research India Podcast: More podcasts from MSR India (opens in new tab)
• iTunes (opens in new tab): Subscribe and listen to new podcasts on iTunes
• Android (opens in new tab)
• RSS Feed (opens in new tab)
• Spotify (opens in new tab)
• Google Podcasts (opens in new tab)
• Email (opens in new tab)

Transcript

[Music]

Ravi Krishnaswamy: Hi Neeraj, how are you doing? It’s great to see you after two years of working from home.

Neeraj Kayal: Hi Ravi, yeah thank you.

Thank you for having me here and it’s great to be back with all the colleagues in office.

Ravi Krishnaswamy: First of all, congratulations on the Infosys prize and it’s an amazing achievement.

And it’s a great privilege for all of us to have you as a colleague here. So, congratulations on that.

Neeraj Kayal: Thank you.

Ravi Krishnaswamy: Yeah, so maybe we can get started on the podcast. So, you work in complexity theory, which is I guess one extreme of, I mean, it’s very theoretical end of the spectrum in computer science almost bordering mathematics. So hopefully by the end of this podcast we can, uh, I mean, convince the audience that there’s more to it than intellectual curiosity. Before that right, let me ask you about how you got into theoretical computer science and the kind of problems that you work on. So, could you maybe tell us a bit about your background and how you got interested into this subject?

Neeraj Kayal: Yeah, so in high school I was doing well in maths in general and I also wrote some computer programs to play some board games, like a generalized version of Tic Tac Toe where you have a bigger board, say 20 by 20, and you try to place five things in the row, column, or diagonal continuously and then I started thinking about how could a computer learn to play or improve itself in such a game? So, I tried some things and didn’t get very far with that, but at that time I was pretty convinced that one day computers will be able to really learn like humans do. I didn’t see how that will happen, but I was sure of it and I just wanted to be in computer science to eventually work on such things. But in college in the second year of my undergrad, I enrolled for a course in cryptography taught by Manindra Agrawal at IIT Kanpur and then the course started off with some initial things which are like fairly predictable that something called symmetric key cryptosystems where, essentially it says that let’s say we two want to have a private conversation, but anyone else can listen to us. So how do we have a private conversation?  Well, if we knew a language, a secret language which no one else did, then we could easily just converse in that language, and no one will understand this.  And so, this is made a little more formal in this symmetric key cryptosystem. And then, one day, Manindra ended one of the lectures with the following problem: but now suppose we did not know a secret language. Then we just know English, and everyone knows English and then how do we talk privately when everyone can hear us? I thought about it for a few days. It seemed completely impossible. And then Manindra told us about these wonderful cryptosystems, called the Diffie Hellman cryptosystem and the RSA cryptosystem where they achieved this and it was very surprising. And the key thing that these cryptosystems use is something that lies at the heart of computer science, a big mystery still even to this day at the heart of computer science. There are these problems which we believe are hard for computers to solve in the following sense, that even if a computer takes a very long amount of time, if we give it a fairly long amount of time, a reasonable amount of time it cannot solve it. But if we give it time like till the end of the universe, it can in principle solve such problems. So that got me interested into which problems are hard and can we prove they are actually hard or not? And to this day, we don’t know that.

Ravi Krishnaswamy: So, I’m guessing that you’re talking about the factoring problem, right?

Neeraj Kayal: Yes, factoring is one of the big ones here. And the RSA cryptosystem uses factoring.

Ravi Krishnaswamy: So, it’s actually very interesting, right? You started out by trying to show that some of these problems are very, very hard, but I think, looking back, your first research paper, which happens to be a breakthrough work in itself, is in showing that a certain problem is actually easier to solve. Then we had originally thought right so, it is this seminal work on showing that primality testing can be solved in deterministic polynomial time. I mean, that’s an amazing feat and you had worked on this paper with your collaborators as an undergrad, right?

Neeraj Kayal: Yes.

Ravi Krishnaswamy: Yeah, that’s an incredible achievement. So maybe to motivate others who are in undergrad and who have an interest and inclination in such topics, could you maybe share us some story on how you got working in that problem and what sort of led you to this spark that eventually got you to this breakthrough result?

Neeraj Kayal: So, my advisor Manindra, who also was the professor who taught us cryptography – he had been working on this problem for a long time and there were already algorithms that existed which are very good in practice- very very fast in practice, but they had this small chance that they might give the wrong answer. The chance was so small that practically it did not matter, but still as a mathematical challenge, it remained whether we could remove that small chance of error, and that’s what the problem was about. So, Manindra had this approach and he had worked with other students also- some of our seniors- on it, and in that course, he came up with a conjecture. And then when we joined, me and my colleague Nitin, we joined this project , we came across this conjecture and my first reaction was that the conjecture is false. So, I tried to write a program which would find a counterexample and I thought we would be done in a few days-Just find that counterexample and the project would be over. So, I wrote a program- it will train for some time, didn’t find a counterexample, so I decided to parallelize it. A huge number of machines in the computer center in IIT Kanpur started looking for that counterexample. And then to my surprise, we still couldn’t find the counterexample. So there seemed to be something to it. Something seemed to be happening there which we didn’t understand, and in trying to sort of prove that conjecture, we managed to prove some sort of weaker statement which sufficed for obtaining the polynomial time algorithm to test if a number is prime or not. But it was not the original conjecture itself. Many days after this result came out, we met a mathematician called Hendrik Lenstra who had worked on primality testing, and we told him about this conjecture. And after a few days he got back to us and it showed that if you assume some number theoretic conjecture is true, which we really really believe, it’s true.

Ravi Krishnaswamy: Ok, I see. So, the original conjecture, which you hoped to prove true is false, but the weaker conjecture was actually true, you proved it to be true, and that was enough for your eventual application.

Neeraj Kayal: Yes, so in some sense we are very lucky that in trying to prove something false we managed to prove something useful.

Ravi Krishnaswamy: Yeah, I mean it’s a fascinating story, right? All the experiments that you ran pointed you towards proving it, and then you actually went and proved it. If you had found, I imagine what would have happened if you found a counterexample at that time, right?

Neeraj Kayal: So yeah, Hendrix proof was very interesting. He showed that modulo this number theory conjecture a counterexample existed. But it would have to be very, very large and that’s why you couldn’t find it. So, he explained it beautifully.

Ravi Krishnaswamy: Yeah, thanks for that story Neeraj. So. I guess from then on you’ve been working in complexity theory, right?

Neeraj Kayal: That’s right, yeah.

Ravi Krishnaswamy: So, for me at least, the Holy Grail in complexity theory that I’ve often encountered or seen is the P versus NP problem, which many of us might know. But you’ve been working on a very equally important, but a very close cousin of the problem, which is called the VP versus VNP problem, right? So, I’m going to take a stab at explaining what I understand of the problem. So, correct me whenever I’m wrong. So, you are interested in trying to understand the complexity of expressing polynomials using small circuits. So, for example, if you have a polynomial of the form X ^2 + Y ^2 + 2 XY, you could represent it as a small circuit which has a few addition operations and a few multiplication operations like you could express it as X ^2 + Y ^2 + 2 XY itself. Or you could express it as (X + Y)^2. Which may have a smaller representation in terms of a circuit. So, you have been working on trying to identify which polynomials have small representations and which polynomials are natural but still don’t have small representations.

Neeraj Kayal: That’s right.

Ravi Krishnaswamy: Is that a reasonable approximation of the problem you’re thinking about?

Neeraj Kayal: Yes, that’s right. So, another way to put the same thing is what is the power of computation when you do additions, multiplications, subtractions, all these arithmetic operations. You could include division, square roots also.

Ravi Krishnaswamy: So, I have seen this VP class and it makes a lot of sense to me. It’s the set of all the polynomials that can be captured by small sized circuits with the plus I mean addition and multiplication gates. I’ve also seen the VNP class, which seems to me at least to be a bit mysterious, right? So, these are all the polynomials whose coefficients of the individual monomials can be computed efficiently. Is that a reasonable definition, at least? Is my understanding correct?

Neeraj Kayal: Yeah, that’s the technical definition of this class, but there’s another natural sort of intuition why we want to look at it, and the intuition is that it relates to counting the number of solutions to a problem, and also therefore to computing probabilities of various things happening.

Ravi Krishnaswamy: I see. Ok, so that gives me a lot more understanding.  I guess when you’re able to estimate probabilities, you could also do sampling over those objects.

Neeraj Kayal: Yes exactly.

Ravi Krishnaswamy: Yeah, that’s a very nice connection. I did not know about this. Thanks for that. So, you have been working, you have an agenda on trying to show some sort of a separation between the two classes, right, VP and VNP, by constructing these low depth circuits. So, you’re able to show that all polynomials in VP have admit the low depth representation and your hope in this agenda is to find one polynomial in VNP which does not have a low depth representation, right?

Neeraj Kayal: That’s right.

Ravi Krishnaswamy: So, how far are you in this agenda and do you think we have all the tools needed to actually achieve success through this sort of a method?

Neeraj Kayal: Yeah, so just historically for converting a circuit or a program into a low depth program, this was done earlier. Most of this work was done by other people. So, we haven’t contributed much in that direction. We have been trying to prove certain polynomials don’t have small depth and small sized arithmetic circuits. So, it’s not clear to us whether the existing techniques are good enough to prove this or not. And like on Mondays, Wednesdays, and Fridays, I think they are capable maybe, and on the other days I think maybe not. And this is what researchers generally deal with. Especially in these areas where you don’t know whether your tools are good enough or not. And very recently, just last year, there was a big breakthrough by trio of complexity theorists who showed somewhat good lower bounds for all constant depth arithmetic formulas or circuits. And what was surprising also about this result is that, they use in a very clever way, techniques that were already known.

Ravi Krishnaswamy: So, they would have probably shown it on a Monday or Wednesday or Friday.

Neeraj Kayal: Yes, yes. [Laughs]

Ravi Krishnaswamy: OK, that’s very interesting. So, you still don’t know whether this will lead to success or not through this route.

Neeraj Kayal: Yes, yeah, we still don’t know that.

Ravi Krishnaswamy: Are there other people approaching this problem through other techniques?

Neeraj Kayal: So, there’s a program called the Geometric Complexity Theory program initiated independently by other people who basically try to understand symmetries. Because implicit in this question is a whole bunch of symmetry, then they try to exploit that. And there’s a field of mathematics called group theory and representation theory, which is all about understanding symmetries of objects. That area is beautiful, really beautiful, and a lot of advancement has been made there. So, people have been trying to also attack this problem through using those tools.

Ravi Krishnaswamy: Yeah, that’s very nice, I think. So basically, you’re saying a lot of like diverse techniques from math and computer science are at play here and trying to help you on your progress.

Neeraj Kayal: That’s right.

Ravi Krishnaswamy: I see. I mean, it’s very beautiful. I find it fascinating and beautiful that a lot of these different diverse techniques from mathematics and computer science come into play into establishing these lower bounds. And what’s more fascinating to me is that they are all not just from an intellectual curiosity standpoint.  They seem to be powering a lot of things that we take for granted, right, right from, like, as you said, messaging each other through social networks or whatever it is. They seem to be like at the foundation- the inherent hardness of certain problems seem to be at the foundation of a lot of things that we take for granted.

Neeraj Kayal: Yeah, that’s right, Ravi. So, for example, I do transactions using my mobile phone and anyone who is within a reasonable distance of my mobile phone can read all the signals that my phone is sending. So, they can see all the communication that I’m having with the bank. And the fact that despite that they are not able to infer my banking passwords relies on the fact that certain problems are very inherently hard to solve and that’s what we are trying to prove.

Ravi Krishnaswamy: OK, so that’s very interesting Neeraj. And in the last part of this podcast, I want to flip the topic around a little bit. So, you’ve been working a lot on showing lower bounds, and in lower bounds in arithmetic complexity. But lately in the last couple of years you have also been using those insights into showing some very nice algorithms for some learning problems. I find that also very cool, so maybe you can talk a little bit about that.

Neeraj Kayal: Yeah, so the algorithms that we are trying to devise are trying to solve the following problem. More general version of it is the following. Given a function or a polynomial, what’s the smallest number of operations that you need to do to be able to compute that function or polynomial? So, for Boolean functions this has a very long history. That essentially is like designing chips, and you can imagine it was naturally very useful to think about. But more recently, it turns out a lot of works have found another very surprising connection because of which this problem specifically for polynomials has also become very interesting. And the connection is this. Suppose you have some very big data set. For now, think of this data set as consisting of a bunch of points in high dimensional space.  For example, you can think of images as a point, every image as a point in the high dimensional space. Now it turns out that you can take statistics of this data. So, for example, you can take what’s the average value of the first coordinate, what’s the average value of the second coordinate? Or what’s the average value of the product of the first two coordinates in this data set and so on. So, you can take some of these statistics, encode them as the coefficients of a polynomial. And here’s the interesting part. When the data has some very nice structure, then this polynomial tends to have a small circuit.

Ravi Krishnaswamy: I see.

Neeraj Kayal: And so, when you want to understand the structure of data, so this general area is called unsupervised learning. Turns out that it’s useful to find small circuits for polynomials. So, this is the computational problem that we are looking at: given a polynomial, what’s the smallest number of operations, or what’s the smallest circuit representing this polynomial.

Ravi Krishnaswamy: So, if you’re able to find the smallest circuit representing this, then from that you will be able to infer the underlying distribution or the structure of the underlying data.

Neeraj Kayal: Yes, yes, that’s right. So, this is one connection, and it also turns out that the lower bounds that we are proving, showing that certain things are very hard to compute are also useful for now devising algorithms to find the circuits of polynomials which do have small circuits and maybe let me give you some very rough sense of how that comes about, and I find this a bit fascinating. Here’s how the lower bounds proofs work. So, underlying all those lower bounds for the various subclasses of circuits that we do have is a collection of linear maps and now it turns out that when you are given a polynomial which has a small circuit, using this polynomial and the collection of linear maps, which go into the lower bound proof you can form another big linear map, such that, very roughly, the eigen spaces of this new linear map correspond to the smallest circuit for F.

Ravi Krishnaswamy: I see.

Neeraj Kayal: And this was the connection that we discovered some time ago, which helped us find small circuits.

Ravi Krishnaswamy: So, you find small circuits by computing the eigen space of the of the map.

Neeraj Kayal: Yes, of this other linear map. That’s right Ravi.

Ravi Krishnaswamy: I see that’s very nice. Ok, so I think we covered a lot of the topics that I wanted to cover, so maybe I’ll end with two philosophical questions. So, one is you began the podcast by talking about how as a kid, you thought computers or machines could be able to do everything that human intelligence can do. So, I think it’s a vague question, but what’s your take on that now? And two is what advice would you give for budding theoreticians, whether they’re in school or college or grad school? What sort of advice would you give them?

Neeraj Kayal: So, for the first question, Ravi, I know a lot of other people also share this feeling, that it’s just a matter of time before we figure out how computers can themselves learn like humans do. Just human babies, they have an amazing ability to learn by observing things around them. And currently, despite all the progress, computers don’t have that much ability. But I just think it’s a matter of time before we figure that out, some sort of general artificial intelligence. To your second question, Ravi, I don’t have much to offer other than perhaps a banal comment that anyone looking to work in this area should really enjoy thinking about these kinds of problems. They tend to be rather abstract, sometimes the applications are not always apparent, but if you enjoy thinking about them, I’m sure you’ll do well.

Ravi Krishnaswamy: That’s great, Neeraj. It’s been a pleasure chatting with you. Thanks a lot for your time and hope you had fun.

Neeraj Kayal: Yeah, thanks Ravi. Thanks a lot for having me.

[Music]

The post Podcast: A Random Walk from Complexity Theory to Machine Learning. With Dr. Neeraj Kayal and Dr. Ravishankar Krishnaswamy appeared first on Microsoft Research.

Episode 011 | January 18, 2022

Keratoconus is a severe eye disease that affects the cornea, causing it to become thin and develop a conical bulge. Keratoconus, if undiagnosed and untreated, can lead to partial or complete blindness in people affected by it. However, the equipment needed to diagnose keratoconus is expensive and non-portable, which makes early detection of keratoconus inaccessible to large populations in low and middle income countries. This makes it a leading cause for partial or complete blindness amongst such populations. Doctors from Sankara Eye Hospital, Bengaluru and researchers from Microsoft Research India have been working together to develop SmartKC, a low-cost and portable diagnostic system that can enable early detection and mitigation of keratoconus. Join us as we speak to Dr. Kaushik Murali from Sankara Eye Hospital and Dr. Mohit Jain from Microsoft Research India.

Dr. Kaushik Murali, President Medical Administration, Quality & Education , Sankara Eye Foundation India (Sri Kanchi Kamakoti Medical Trust) which is among the largest structured community eye hospital network in India, (www.sankaraeye.com (opens in new tab)) with an objective of providing world class eye care with a social impact.

A paediatric ophthalmologist, Dr. Kaushik has completed a General Management Programme and is an alumnus of Insead. He has done a course on Strategic Management of Non Profits at the Harvard Business School. He has been certified in infection control, risk management for health care and digital disruption. He is a member of Scalabl, a global community promoting entrepreneurship.

Dr. Kaushik is a member of the Scientific Committee of Vision 2020, the Right to Sight India. He is currently involved in collaborative research projects among others with the University of Bonn & Microsoft.

Dr. Kaushik has received many recognitions, key among them being the Bernadotte Foundation for Children’s Eyecare Travel Grant, Mother Teresa Social Leadership Scholarship, International Eye Health Hero, All India Ophthalmological Society best research, International Association for Prevention of Blindness (IAPB) Eye Health Hero, Indian Journal of Ophthalmology Certificate of Merit.

Beyond the medical world, he is part of the National Management Team of Young Indians – Confederation of Indian Industry (CII). He represented India at G20 Young Entrepreneur Alliance 2018 at Argentina and led the Indian delegation for the Inaugural India-Israel Young Leaders Forum in 2019. More recently, he led the first citizen’s cohort for a workshop on Strategic Leadership at LBSNAA (Lal Bahadur Shastri National Academy of Administration).

Mohit Jain (opens in new tab) is a Senior Researcher in the Technology and Empowerment (TEM) (opens in new tab) group at Microsoft Research India. His research interests lie at the intersection of Human Computer Interaction and Artificial Intelligence. Currently, he focuses on developing end-to-end systems providing low-cost smartphone-based patient diagnostic solutions for critical diseases. Over the past decade, he has worked on technological solutions for the developing regions focusing on health, accessibility, education, sustainability, and agriculture.

He received his Ph.D. in Computer Science & Engineering from the University of Washington, focusing on extending interactivity, accessibility and security of conversational systems. While pursuing his Ph.D., he also worked as a Senior Research Engineer in the Cognitive IoT team at IBM Research India. Prior to that, he graduated with a Masters in Computer Science from the University of Toronto, and a Bachelors in Information and Communication Technology from DA-IICT.

For more information about the SmartKC project, and for project related code, click here (opens in new tab).

For more information about the Microsoft Research India click here (opens in new tab).

Related

• Microsoft Research India Podcast: More podcasts from MSR India (opens in new tab)
• iTunes (opens in new tab): Subscribe and listen to new podcasts on iTunes
• Android (opens in new tab)
• RSS Feed (opens in new tab)
• Spotify (opens in new tab)
• Google Podcasts (opens in new tab)
• Email (opens in new tab)

Transcript

Dr. Murali Kaushik: Sitting in an eye hospital, often we have ideas, but we have no clue whom to ask. But honestly, now we know that there is a team at MSR that we can reach out to saying that hey, here is a problem, we think this warrants attention. Do you think you guys can solve it? And we found that works really well. So, this kind of a collaboration is, I think, a phenomenal impact that this project has brought together, and we hope that together we will be able to come up with few more solutions that can align with our founders’ dream of eliminating needless blindness from India.

[Music]

Sridhar Vedantham: Welcome to the Microsoft Research India podcast, where we explore cutting-edge research that’s impacting technology and society. I’m your host, Sridhar Vedantham.

[Music]

Sridhar Vedantham: Keratoconus is a severe eye disease that affects the cornea, causing it to become thin and develop a conical bulge. Keratoconus, if undiagnosed and untreated, can lead to partial or complete blindness in people affected by it. However, the equipment needed to diagnose keratoconus is expensive and non-portable, which makes early detection of keratoconus inaccessible to large populations in low and middle income countries. This makes it a leading cause for partial or complete blindness amongst such populations. Doctors from Sankara Eye Hospital, Bengaluru and researchers from Microsoft Research India have been working together to develop SmartKC, a low-cost and portable diagnostic system that can enable early detection and mitigation of keratoconus. Join us as we speak to Dr. Kaushik Murali from Sankara Eye Hospital and Dr. Mohit Jain from Microsoft Research India.

[Music]

Sridhar Vedantham:  So, Dr. Kaushik and Mohit, welcome to the podcast. 

Mohit Jain: Hi, Sridhar.

Dr. Kaushik Murali: Hi Sridhar, pleasure to be here.

Sridhar Vedantham: It’s our pleasure to host you, Dr. Kaushik, and for me this is going to be a really interesting podcast for a couple of reasons. One is that the topic itself is kind of so far afield from what I normally hear at Microsoft Research and the second is I think you’re the first guest we are having on the podcast who’s actually not part of MSR, so basically a collaborator. So, this is really exciting for me. So let me jump right into this. We’re going to be talking about something called keratoconus, so could you educate us a little bit as to what keratoconus actually is and what its impact is?

Dr. Kaushik Murali: So, imagine that you were a 14-year-old who was essentially near sighted. You wore glasses and you were able to see. But with passing time, your vision became more distorted rather than being blurred, which is what you would have expected if just your minus power kept increasing, especially for distance. And to add to your misery, you started seeing more glare and more halos at nighttime. Words that you started to read had shadows around them or even started to look doubled. This essentially is the world of a person with Keratoconus. Literally it means cone shaped. Keratoconus is a condition of the cornea, which is the transparent front part of the eye, similar to your watch glass, where instead of it normally retaining its dome shape, it is characterized by progressive thinning and weakening of the central part, what we call as a stroma, and this makes the cornea take on a conical shape. In a few, this can actually even progress beyond what I describe, where the central cornea overtime becomes scarred and the person could no longer be corrected, with just optical devices like a glass or a contact lens but may actually end up requiring a corneal transplant.

Sridhar Vedantham: I see, and what are the causes for this?

Dr. Kaushik Murali: So there have been very many causes that have been attributed, so it’s thought to be multifactorial. So, this again makes it a little tricky in terms of us not being able to prevent the condition, so to speak. But multiple risk factors are known. Ultraviolet exposure, chronic allergies; habitual eye rubber is thought to be more prone for this. Essentially, you end up seeing it more during the pubertal age group, and more in men.

Sridhar Vedantham: I see. And how widespread is this problem, really? Because frankly, I’m of course as lay a person as you can get, and I hadn’t really heard of eye disease called keratoconus until I spoke to Mohit at some point and then of course after reading papers and so on. But what is the extent of the issue and is it really that widespread a problem?

Dr. Kaushik Murali: So, unlike most other conditions, there is no real population-based survey where we have screened every household to arrive at numbers. But largely, we base our estimation on small surveys that have been done across different parts of the world. Based on this, we estimate that it is approximately affecting about one in 2000 individuals. So, in the US, for example, it is thought to affect almost about 55 people in about 100,000, who had been diagnosed with keratoconus. But in countries like India, it is thought to be more widespread. So there was actually a survey in central India where they found almost 2300 people out of 100,000 people being affected with keratoconus. So, the numbers are quite large. And again, all of this could be underestimated simply because we don’t have enough ability to screen. And what makes this number even scarier is this is a disease that typically affects people between the age group of 10 and 25. So, once they’re affected and they’re progressively going to have their vision come down, they’re going to spend most of their protective years not being able to see clearly.

Sridhar Vedantham: OK, that is kind of scary.

Mohit Jain: I would just like to add to that is that there is actually a combination of demographics, genetic and weather condition which makes India a really good host for this disease. So, apparently Indian population tend to have a thinner and steeper cornea to begin with and moreover the hot and humid climate of India actually contribute towards keratoconus because it causes irritation which leads to frequent rubbing of the eye and that can actually start the process of distortion of the cornea.

Sridhar Vedantham: OK, so that just made the problem sound a little scarier because there are these conditions that cannot be altered, right? I mean climate and stuff like that we can’t really change. Uh, OK, doctor, so, this is basically a well-established and recognized disease, right? So, what are the barriers and hurdles actually to effective diagnosis and treatment of this disease?

Dr. Kaushik Murali: So, in any health intervention the barriers typically are availability, accessibility and affordability. And when you look at a condition like keratoconus, all these barriers actually again come into play. So, the clinical gold standard for diagnosing keratoconus essentially entails us being able to map the curvature of the corneal surface using a technique known as corneal topography. So, here we either use something called as a Placido base, where you project a disc kind of structure onto the cornea and capture an image using tomographers, or you use certain other technology to map out the surface of the cornea, both the anterior and the posterior surface. But all these devices, by and large are non-portable and are expensive. And these are typically available again at secondary or tertiary eye hospitals. India is a land of diversity. We have some of the best health care in cities like Bangalore where we are sitting and doing this recording while you move about 150 to 200 kilometers away, it’s a completely different world out there. The cost of each of these tests again, makes this diagnosis very inaccessible to large sections of population, not only in India but in similar middle and low income countries. And to add to this, you have the bogey of the pandemic. So, with COVID-19 in play the last two years with restrictions on travel, it has become very difficult for young children to actually undergo annual eye exams where we could have even proactively tried to pick up some of these conditions.

Sridhar Vedantham: OK, alright, Mohit, let me bring you in here. I’m curious as to what Microsoft Research is doing as part of this particular research collaboration, right? Because keratoconus sounds very, very far afield from Microsoft Research’s typical computer science focus.

Mohit Jain: So, I think one of the key pillars of MSR India is societal impact. So, if we can develop a diagnostic tool which can save even a handful of children getting blind, we think it has huge societal impact and Microsoft Research India completely, I think, supports such kind of technological solutions. With respect to the computer science research lab, we have developed a solution which actually uses cutting edge AI, especially in the field of image processing and then also we have developed a full end to end system. So hence there are like enough computer science research problem, really hard ones, that we actually solve as part of this research collaboration.

Sridhar Vedantham: OK, so that’s a good lead into me to ask you to talk a little bit about the solution that’s been developed here. 

Mohit Jain: So, I think the solution has actually 3 core components. So the first component is actually a Placido disc like what Dr. Kaushik said that like there has to be something which projects concentric rings over the cornea. So, we actually 3D printed a Placido disc. So, we actually attach that Placido to the back of a smartphone camera and then capture an image of the cornea with the Placido projected over the cornea of a human. And the second component we have is actually a smart phone app which has inbuilt AI in it. So, in real time it actually helps the person who is capturing the image to actually get a perfect image because I think one of the core design principles that we had while working on this was to make sure that anyone can use this app to actually diagnose. We don’t want like medical technician to be only be able to use it. So, there is some kind of AI assistance to help capture perfect image of the eye. And the third then we have like an image processing pipeline which takes this captured image as input and converts that into corneal topography heatmaps. So, basically it gives a curvature heatmap, which tells you like what is the curvature at each point of the cornea, and that is the kind of output that you also get from a medical grade corneal topographer.

Sridhar Vedantham: So, the way you explain it sounds very simple, but I’m sure there were a bunch of challenges, while you know, both Sankara and MSR was doing the research. Can you give me a sense of what kind of challenges you faced?

Mohit Jain: Yes, yes, I think the most trickiest part of this project was to actually do it in a portable setting. So right now the corneal topographer, that is like a $10,000 device which is there in Sankara hospital, there is actually a head-rest and a chin-rest. So, basically your whole face like the patient face is very very stable and hence the image capture process is fairly easy. Apart from that the topographer has a lot of specific hardware. For example, it has a specific hardware to determine how far is the eye from the camera, so which is actually called as a working distance, and getting that parameter right is very crucial. Like even a few millimeters of actually predicting that value wrong can completely change the generated heatmap. So, we have to actually design a very innovative solution to figure out this working distance directly just from the captured image and the second part was that, like we actually did a lot of iteration on the Placido disc and also on the AI assistance which was actually running on the smart phone to actually help capture the best image, even without any kind of a support system in place, like without any head-rest or chin-rest.

Sridhar Vedantham:  Dr. Kaushik was there anything you wanted to add to that in terms of challenges from a medical point of view?

Dr. Kaushik Murali: So, from the medical point of view, we are people of habit. When I say that, there is certain things that we are comfortable about and certain things that puts us off. So, one of the key challenges we gave to the MSR team was saying that the interface and reports had to be similar to what we were used to seeing. So, I think the challenge also came to the team in terms of ensuring that the heatmaps were similar to the heatmaps that we were used to seeing using a regular topographer and how we actually were able to match with it. So, the minute we were able to get that kind of familiarity built in, we found our doctors also being able to accept it much better. And once that was done then automatically the learning curve that came in in terms of using the device or interpreting the images came down very very fast. So, we were able to adapt to this much faster. We were even able to get some of our colleagues from other units to read these heatmaps that we generated, just to validate it.  Because we were also concerned saying that when you are putting it out as a screening device you should end up overestimating the disease. Because there is an indirect cost to a person and imagine the psychological impact to a parent if you tell him your young child may have a possible problem. So we didn’t want to do that, so we were very finicky about the validation. So, it went through multiple iterations almost to the effect that Mohit and his team could have lunch on a Thursday, only after they finished a call with us.

(Laughs)

Mohit Jain: To add to that, this is actually a very crucial point. Initially what we were thinking, so there are like these two values called as sensitivity and specificity. So just to give you some context here, sensitivity is actually if the person has keratoconus, are we able to diagnose it as keratoconus. So that’s the sensitivity. Specificity is that if the person does not have Keratoconus are we actually even diagnosing that correctly? So, we were thinking that we need to get really high sensitivity. We cannot leave anyone who has keratoconus undiagnosed. But we can have low specificity that even someone who does not have keratoconus, we can say that he or she has keratoconus and still it’s OK because then he or she will actually go to a doctor and get checked up, maybe for a second time and then it will be diagnosed that he or she does not have keratoconus. But Dr. Kaushik specifically was really against that. OK, so he actually made our problem statement harder, saying that we want both sensitivity and specificity to be above 92%, OK. Because he does not want like parents to be unnecessarily worrying that their kid who is like in his teens right now and he has like this very serious eye condition. So hence we actually have to like, as what Dr. Kaushik said, like you have to go through multiple iteration to even like get the specificity right. And right now, I think we are at a stage where we have like both the numbers which are above 94% for a small trial that we did with Sankara, with more than 100 patients, and in future we plan to actually extend that trial and like do it in many more patients.

[Music]

Sridhar Vedantham: A question for both you, Mohit, as well as Doctor Kaushik, so, whoever wants to answer it. You know, you just put out some numbers there in terms of percentages, right, to say that this solution works well, but typically these need to be tagged against some kind of industry benchmark or some kind of medical benchmark given the established machines and equipment already that are already being used, right? So how does this SmartKC stack up against those?

Dr. Kaushik Murali: So once the MSR team had come up with a complete solution and they had tested in their lab the reliability of the solution, so to speak, with the images that they had with them, we then apply to our Ethics Committee for a prospective study. So, we enrolled subjects coming to our hospital, and we had them get their cornea image with the SmartKC as well as with the Placido based topography system that we have in our hospital that we would typically have used in any case. One index that the device that we use in the hospital uses to identify keratoconus, is an index called as the percentage probability of keratoconus, or the PPK value. This if it is more than 45% is supposed to indicate the presence of keratoconus. So, what we found was with the SmartKC, the sensitivity of PPK value was 94.1% and the specificity was 100%, which was pretty comparable to what we had with our other device which stood at about 94.1% sensitivity and 95.5% specificity. More importantly, whenever you use any device as a screening tool is how repeatable is it, and how is the inter-rater agreement. If two different people use the same device on the same person, are they actually reading it out the same. So again, in this we found those indices to be really, really high. So, there’s something called as the Cohen’s Kappa value. This was about 0.94 for our tool, which compared to 0.96 for the device that we have in the hospital. This essentially indicates a fair agreement between two experts diagnosing keratoconus based on the images that we are generating using the SmartKC.

Sridhar Vedantham:  Wow, that’s impressive. Uh, I had another question. You know, whenever I went into an eye hospital, there always seemed to be somebody who is well trained as a technician operating various machines. How does it work with this? I mean do you need any specific training on SmartKC? How intensive is that and are there any particular type of people you look for or some kind of skill level in people who can operate a SmartKC unit? 

Dr. Kaushik Murali: So, currently most of the data that was collected has been done by our optometry interns at the Sankara College of Optometry, so the skill set that was required was not very high. Like Mohit mentioned earlier, there is some assistant in terms of even how you can capture an image. So, the eventual endpoint for us would be this being used at a peripheral vision center or at some Primary Health Center as a part of a school screening program where a lay person can pick up this device and actually capture an image and at least be able to counsel the person about them having to have a further examination. So, the training that would probably be required is largely in terms of what the condition that they are screening for and what the next action needs to be. It’s more counseling skill I would say rather than anything really technical. The machine does at all. It gives them a heatmap, it tells them what the value is, it kind of literally puts up a red flag saying “Boss, be careful, there is a problem here”.

Sridhar Vedantham: So, I’m guessing that’s all part of the technology that’s been built into it, say you know various guardrails to ensure that whoever is operating it is doing it in the right way, Mohit?  

Mohit Jain: Yes, yes Sridhar. So, basically in the latest version of the app, what we have done is that like the operator, whoever is actually capturing the image, he or she doesn’t even have to click the capture button. So, you just have to place the SmartKC on a patient eye and it automatically tries to align it appropriately with the eye and once the alignment is right, once the image does not have any kind of a blur and the focus is appropriate, the lighting is fine, it automatically captures three images. So, that actually makes the process really easy for the operator. And he or she needs to go through very minimal training to get up and running with the device.

Sridhar Vedantham: Nice. And you know we have referred earlier to the cost of current machines that are there that are used in eye hospitals and so on, right? And one of the guiding principles behind developing SmartKC was for something to be portable, inexpensive and easy to use in the field. One thing that I don’t think we’ve spoken about yet is actually how inexpensive SmartKC is, and I’m also curious to find out whether the equipment that you use needs to be high end. For example, do you need a high-end mobile phone to do it and how does the whole system work?

Mohit Jain: Yes, so currently for developing our research prototype, we actually end up spending like almost like $33 is the amount that we are end up spending making the device, apart from the smartphone. The smartphone that we are using is a slightly high end one, so it is around like a $400 device that we have used for the current prototype. But moving ahead we are actually for the next round of data collection with Sankara Hospital, we are actually trying out with like three different devices and starting from like $100 device, $200 device and the $300 device, so that we can actually see that with whether it works on all the devices. However, based on our current work, we hypothesize that, actually, it can work on any standard smartphone. It just needs a decent camera. By the way, nowadays even the $100 device have like a 20-megapixel camera. So, I think that’s already taken care by most of the latest smartphone. So, I think yeah, it should ideally work in all of them, but we will only know for sure once we have the second round of testing.

Sridhar Vedantham: Cool. Uh, so you know, given that you’ve put in so much work and you’ve come up with something that sounds like a fantastic solution, what kind of impact do you think a SmartKC can have or what kind of impact do you hope SmartKC will have?  

Mohit Jain: So right now, actually, we have discussed a few use cases with Dr. Kaushik, and I think the most interesting use case is what Dr. Kaushik initially referred to is the teacher use case. The biggest impact SmartKC could have is that, let’s say, all the teachers in India, even the rural one, urban one, semi urban or even like low-income community even in urban India they have access to SmartKC. And every year maybe twice a year or thrice a year they screen every children in their school for keratoconus. And because the numbers are really high, the numbers are like two to three children out of every one hundred children will have keratoconus, so with that in mind, we should be able to get like a few cases from every school. So, if these are diagnosed very early on, then there is a very very high likelihood that they could have been treated just by simple glasses or contact lens, and they don’t have to go through surgery or corneal transplant or blindness, which is the worst-case situation. By the way, I think earlier when we were talking about corneal transplant, so globally 27% of the corneal transplant happens to actually fix keratoconus. So it is, it is a very deadly disease.

Dr. Kaushik Murali:  So, I’m going to take off on a tangent. I’m actually going to look beyond just this particular solution. So, although I am here representing Sankara at this podcast, a couple of my colleagues actually did a deep dive into this entire solution. Dr. Anand and Dr. Pallavi were there, as well as some of our faculty from the College of Optometry. What this project has actually impacted us is to think of how we can actually leverage technology. Sitting in an eye hospital, often we have ideas, but we have no clue whom to ask. But honestly, now we know that there is a team at MSR that we can reach out to saying that hey, here is a problem, we think this warrants attention. Do you think you guys can solve it? And we found that works really well. So, this ability to collaborate between complete extremes of people, one end you have medical specialists who have no clue about what engineering is. Today I think Mohit knows more optometry than even I do.

(Laughs)

So they actually borrowed textbooks from us to read. So, this kind of a collaboration is, I think, a phenomenal impact that this project has brought together, and we hope that together we will be able to come up with few more solutions that can align with our founders’ dream of eliminating needless blindness from India.

Sridhar Vedantham: Nice, congratulations Mohit.

 Mohit Jain: Thank you. (Laughs).

Sridhar Vedantham: I’ll come to you for my next eye test. 

Dr. Kaushik Murali: You will soon start referring to him as Doctor Mohit.

(Laughs)

Sridhar Vedantham:  All right. I think I’ve got all the information that I really wanted. But one thing is, you know, if people want to adopt SmartKC and they want to take it out there in the field, what do they need to do? And are there any particular requirements or criteria that they need to fulfil, etc? I mean, how do people get hold of this? 

Mohit Jain: So right now, we have actually made everything open source. So, even the 3D print for the Placido disc, the STL file of that is open source. So, anyone can download it and just like you take a printout, you can actually go to a makerspace and get a 3D print of that. Even the app which is actually AI assisted app which is running on the smartphone, we have only written it for an Android phone, so you can actually download it and you can install it on your Android phone and connect that SmartKC Placido attachment and can click an image of eye. The image processing pipelines code is also completely open source, so the selected image you can then run on it to actually generate corneal topography heatmap. So, that is that is the current state. Going ahead we are actually putting everything on the cloud so that once you have the Placido disc and you capture an image, it automatically goes to the cloud and gives you the final output.

Sridhar Vedantham: Excellent. So I will add links to various things onto the podcast page once we’re done with this. Dr. Kaushik, any final thoughts before we close this podcast?

Dr. Kaushik Murali: So quite often we look at challenges as challenges, but I think essentially this collaboration has looked at an opportunity and how we can actually work on it. All it required was to put some method to the madness and what came up as one discussion is today, I think, churned out into three different projects that are currently underway. So, this is something that is a potential. It would be lovely if similar technology companies can collaborate with medical institutions across India. We don’t have to look elsewhere for solutions. It’s all here. It’s up to us to figure it out and run with it.

Sridhar Vedantham: That’s a great note to end the podcast on. So, once again thank you so much, Dr. Kaushik and Dr. Mohit. Thank you so much.  

(Laughs)

Mohit Jain: Thank you, Sridhar.

[Music ends]

 Opens in a new tab

The post Podcast: Collaborating to Develop a Low-cost Keratoconus Diagnostic Solution. With Dr. Kaushik Murali and Dr. Mohit Jain appeared first on Microsoft Research.

Episode 010 | September 28, 2021

Artificial intelligence, Machine Learning, Deep Learning, and Deep Neural Networks are today critical to the success of many industries. But they are also extremely compute intensive and expensive to run in terms of both time and cost, and resource constraints can even slow down the pace of innovation. Join us as we speak to Muthian Sivathanu, Partner Research Manager at Microsoft Research India, about the work he and his colleagues are doing to enable optimal utilization of existing infrastructure to significantly reduce the cost of AI.

Muthian’s interests lie broadly in the space of large-scale distributed systems,  storage, and systems for deep learning, blockchains, and information retrieval.

Prior to joining Microsoft Research, he worked at Google for about 10 years, with a large part of the work focused on building key infrastructure powering Google web search — in particular, the query engine for web search. Muthian obtained his Ph.D from University of Wisconsin Madison in 2005  in the area of file and storage systems, and a B.E. from CEG, Anna University, in 2000.

For more information about the Microsoft Research India click here (opens in new tab).

Related

• Microsoft Research India Podcast: More podcasts from MSR India (opens in new tab)
• iTunes (opens in new tab): Subscribe and listen to new podcasts on iTunes
• Android (opens in new tab)
• RSS Feed (opens in new tab)
• Spotify (opens in new tab)
• Google Podcasts (opens in new tab)
• Email (opens in new tab)

Transcript

Muthian Sivathanu: Continued innovation in systems and efficiency and costs are going to be crucial to drive the next generation of AI advances, right. And the last 10 years have been huge for deep learning and AI and primary reason for that has been the significant advance in both hardware in terms of emergence of GPUs and so on, as well as software infrastructure to actually parallelize jobs,  run large distributed jobs efficiently and so on. And if you think about the theory of deep learning, people knew about backpropagation about neural networks 25 years ago. And we largely use very similar techniques today. But why have they really taken off in the last 10 years? The main catalyst has been sort of advancement in systems. And if you look at the trajectory of current deep learning models, the rate at which they are growing larger and larger, systems innovation will continue to be the bottleneck in sort of determining the next generation of advancement in AI.

[Music]

Sridhar Vedantham: Welcome to the Microsoft Research India podcast, where we explore cutting-edge research that’s impacting technology and society. I’m your host, Sridhar Vedantham.

[Music]

Sridhar Vedantham: Artificial intelligence, Machine Learning, Deep Learning, and Deep Neural Networks are today critical to the success of many industries. But they are also extremely compute intensive and expensive to run in terms of both time and cost, and resource constraints can even slow down the pace of innovation. Join us as we speak to Muthian Sivathanu, Partner Research Manager at Microsoft Research India, about the work he and his colleagues are doing to enable optimal utilization of existing infrastructure to significantly reduce the cost of AI.

[Music]

Sridhar Vedantham: So Muthian, welcome to the podcast and thanks for making the time for this.

Muthian Sivathanu: Thanks Sridhar, pleasure to be here.

Sridhar Vedantham: And what I’m really looking forward to, given that we seem to be in some kind of final stages of the pandemic, is to actually be able to meet you face to face again after a long time.  Unfortunately, we’ve had to again do a remote podcast which isn’t all that much fun.

Muthian Sivathanu: Right, right. Yeah, I’m looking forward to the time when we can actually do this again in office.

Sridhar Vedantham: Yeah. Ok, so let me jump right into this. You know we keep hearing about things like AI and deep learning and deep neural networks and so on and so forth. What’s very interesting in all of this is that we kind of tend to hear about the end product of all this, which is kind of, you know, what actually impacts businesses, what impacts consumers, what impacts the health care industry, for example, right, in terms of AI. It’s a little bit of a mystery, I think to a lot of people as to how all this works, because… what goes on behind the scenes to actually make AI work is generally not talked about.

Muthian Sivathanu: Yeah.

Sridhar Vedantham: So, before we get into the meat of the podcast you just want to speak a little bit about what goes on in the background.

Muthian Sivathanu: Sure. So, machine learning, Sridhar, as you know, and deep learning in particular, is essentially about learning patterns from data, right, and deep learning system is fed a lot of training examples,  examples of input and output, and then it automatically learns a model that fits that data, right. And this is typically called the training phase. So, training phase is where it takes data builds a model how to fit. Now what is interesting is, once this model is built, which was really meant to fit the training data, the model is really good at answering queries on data that it had never seen before, and this is where it becomes useful. These models are built in various domains. It could be for recognizing an image for converting speech to text, and so on, right. And what has in particular happened over the last 10 or so years is that there has been significant advancement both on the theory side of machine learning, which is, new algorithms, new model structures that do a better job at fitting the input data to a generalizable model as well as rapid innovation in systems infrastructure which actually enable the model to sort of do its work, which is very compute intensive, in a way that’s actually scalable that’s actually feasible economically, cost effective and so on.

Sridhar Vedantham: OK, Muthian, so it sounds like there’s a lot of compute actually required to make things like AI and ML happen. Can you give me a sense of what kind of resources or how intensive the resource requirement is?

Muthian Sivathanu: Yeah. So the resource usage in a machine learning model is a direct function of how many parameters it has, so the more complex the data set, the larger the model gets, and correspondingly requires more compute resources, right. To give you an idea, the early machine learning models which perform simple tasks like recognizing digits and so on, they could run on a single server machine in a few hours, but models now, just over the last two years, for example, the size of the largest model that’s useful that state of the art, that achieves state of the art accuracy has grown by nearly three orders of magnitude, right. And what that means is today to train these models you need thousands and thousands of servers and that’s infeasible. Also, accelerators or GPUs have really taken over the last 6-7 years and GPUs. A single V-100 GPU today, a Volta GPU from NVIDIA can run about 140 trillion operations per second. And you need several hundreds of them to actually train a model like this. And they run for months together to train a 175 billion model, which is called GPT 3 recently, you need on the order of thousands of such GPUs and it still takes a month.

Sridhar Vedantham: A month, that’s sounds like a humongous amount of time.

 Muthian Sivathanu: Exactly, right? So that’s why I think just as I told you how the advance in the theory of machine learning in terms of new algorithms, new model structures, and so on have been crucial to the recent advance in the relevance in practical utility of deep learning.Equally important has been this advancement in systems, right, because given this huge explosion of compute demands that these workloads place, we need fundamental innovation in systems to actually keep pace, to actually make sure that you can train them in reasonable time, you can actually do that with reasonable cost.

Sridhar Vedantham: Right. Ok, so you know for a long time, I was generally under the impression that if you wanted to run bigger and bigger models and bigger jobs, essentially you had to throw more hardware at it because at one point hardware was cheap. But I guess that kind of applies only to the CPU kind of scenario, whereas the GPU scenario tends to become really expensive, right?

Muthian Sivathanu: Yep, yeah.

Sridhar Vedantham: Ok, so in which case, when there is basically some kind of a limit being imposed because of the cost of GPUs, how does one actually go about tackling this problem of scale?

Muthian Sivathanu: Yeah, so the high-level problem ends up being, you have limited resources, so let’s say you can view this in two perspectives, right. One is from the perspective of a machine learning developer or a machine learning researcher, who wants to build a model to accomplish a particular task right. So, from the perspective of the user, there are two things you need. A, you want to iterate really fast, right, because deep learning, incidentally, is this special category of machine learning, where the exploration is largely by trial and error. So, if you want to know which model actually works which parameters, or which hyperparameter set actually gives you the best accuracy, the only way to really know for sure is to train the model to completion, measure accuracy, and then you would know which model is better, right. So, as you can see, the iteration time, the time to train a model to run inference on it directly impacts the rate of progress you can achieve. The second aspect that the machine learning researcher cares about is cost. You want to do it without spending a lot of dollar cost.

Sridhar Vedantham: Right.

Muthian Sivathanu: Now from the perspective of let’s say a cloud provider who runs this, huge farm of GPUs and then offers this as a service for researchers, for users to run machine learning models, their objective function is cost, right. So, to support a given workload you need to support it with as minimal GPUs as possible. Or in other words, if you have a certain amount of GPU capacity, you want to maximize the utilization, the throughput you can get out of those GPUs, and that’s where a lot of the work we’ve been doing at MSR has focused on. How do you sort of multiplex lots and lots of jobs onto a finite set of GPUs, while maximizing the throughput that you can get from them?

Sridhar Vedantham: Right, so I know you and your team have been working on this problem for a while now. Do you want to share with us some of the key insights and some of the results that you’ve achieved so far, because it is interesting, right? Schedulers have been around for a while. It’s not that there aren’t schedulers, but essentially what you’re saying is that the schedulers that exist do not really cut it, given the, intensity of the compute requirements as well as the jobs, as the size of the jobs and models that are being run today in terms of deep learning or even machine learning models, right?

Muthian Sivathanu: That’s right.

Sridhar Vedantham: So, what are your, key insights and what are some of the results that you guys have achieved?

Muthian Sivathanu: So, you raise a good point. I mean, schedulers for distributed systems have been around for decades, right. But what makes deep learning somewhat special is that it turns out, in contrast to traditional schedulers, which have to view a job as a black box, because they’re meant to run arbitrary jobs. There is a limit to how efficient they can be. Whereas in deep learning, first of all because deep learning is such high impact area with lots, and I mean from an economic perspective, there are billions of dollars spent in these GPUs and so on. So, there is enough economic incentive to extract the last bit of performance out of these expensive GPUs, right. And that lends itself into this realm of- what if we co-design? What if we custom design a scheduler for the specific case of deep learning, right. And that’s what we did in the Gandiva project which we published at OSDI in 2018. What we said was, instead of viewing a deep learning job as just another distributed job which is opaque to us, let’s actually exploit some key characteristics that are unique to deep learning jobs, right? And one of those characteristics, is that  although, as I said, a single deep learning training job can run for days or even months, right, deep within it is actually composed of millions and millions of these what are called mini batches. So, what is a mini batch? A mini batch is an iteration in the training where it reads one set of input training examples, runs it through the model, and then back propagates the loss, and essentially, changes the parameters to fit that input. And this sequence this mini batch repeats over and over again across millions and millions of mini batches. And what makes it particularly interesting and relevant from a systems optimization viewpoint is that from a resource usage perspective and from a performance perspective, mini batches are identical. They may be operating on different data in each mini batch, but the computation they do is pretty much identical. And what that means is we can look at the job for a few mini batches and we can know what exactly is going to do for the rest of its life time, right. And that allows us to, for example, do things like, we can automatically decide which hardware generation is the best fit for this job, because you can just measure it in a whole bunch of hardware configurations. Or when you’re distributing the job, you can compare it across a whole bunch of parallelism configurations, and you can automatically figure out, this is the right configuration, right hardware assignment for this particular job, which you couldn’t do in an arbitrary job with a distributed scheduler because the job could be doing different things at different times. Like a MapReduce job for example, it would keep fluctuating across how we’d use a CPU, network, storage, and so on, right. Whereas with deep learning there is this remarkable repeatability and predictability, right. What it also allows us to do is, we can then look within a mini batch what happens, and it turns out, one of the things that happens is, if you look at the memory usage, how much GPU memory the training loop itself is consuming, somewhere at the middle of a mini batch, the memory peaks to almost fill the entire GPU memory, right. And then by the time the mini batch ends, the memory usage drops down by like a factor of anywhere between 10 to 50x. Right, and so there is this sawtooth pattern in the memory usage, and so one of the things we did in Gandiva was proposed this mechanism of transparently migrating a job, so you should be able to, on demand checkpoint a job. The scheduler should be able to do it and just move it to a different machine, maybe even essentially different GPU, different machine, and so on, right. And this is very powerful from load balancing. Lots of scheduling things become easy if you do this. Now, when you’re doing that, when you are actually moving a job from one machine to another, it helps if the amount of state you need to move is small, right. And so that’s where this awareness of mini batch boundaries and so on helps us, because now you can choose when exactly to move it so that you move 50x, smaller amount of state.

Sridhar Vedantham: Right. Very interesting, and another part of this whole thing about resources and compute and all that is, I think, the demands on storage itself, right?

Muthian Sivathanu: Yeah.

Sridhar Vedantham: Because if the models are that big, that you need some really high-powered GPUs to compute, how do you manage the storage requirements?

Muthian Sivathanu: Right, right. So, it turns out the biggest requirement from storage that deep learning poses is on the throughput that you need from storage, right. So, as I mentioned, because GPUs are the most expensive resource in this whole infrastructure stack, the single most important objective is to keep GPUs busy all the time, right. You don’t want them idling, at all. What that means is the input training data that the model needs in order to run its mini batches, that is to be fed to it at a rate that is sufficient to keep the GPUs busy. And GPUs process, I mean the amount of data that the GPU can process from a compute perspective has been growing at a very rapid pace, right. And so, what that means is, you know, when between Volta series and an Ampere series, for example, of GPUs there is like 3X improvement in compute speed, right. Now that means the storage bandwidth should keep up with that pace, otherwise faster GPU doesn’t help. It will be stalling on IO. So, in that context one of the systems we built was the system called Quiver, where we say a traditional remote storage system like the standard model for running this training is…the datasets are large- I mean the data sets can be in terabytes, so, you place it on some remote cloud storage system, like Azure blob or something like that, and you read it remotely from whichever machine does the training, right. And that bandwidth simply doesn’t cut it because it goes through network backbone switches and so on, and it becomes insanely expensive to sustain that level of bandwidth from a traditional cloud storage system, right. So what we need, to achieve here is hyper locality. So, ideally the data should reside on the exact machine that runs the training, then it’s a local read and it has to reside on SSD and so on, right. So, you need several gigabytes per second read bandwidth.

Sridhar Vedantham: And this is to reduce network latency?

Muthian Sivathanu: Yes, this is to reduce network latency and congestion, like when it goes through lots of back end, like T1 switches, T2 switches etc. The end-to-end throughput that you get across the network is not as much as what you can get locally, right?

Sridhar Vedantham: Right.

Muthian Sivathanu: So, ideally you want to keep the data local in the same machine, but as I said, for some of these models, the data set can be in tens of terabytes. So, what we really need is a distributed cache, so to speak, right, but a cache that is locality aware. So, what we have is a mechanism by which, within each locality domain like a rack for example, we have a copy of the entire training data, so, a rack could comprise maybe 20 or 30 machines, so across them you can still fit the training data and then you do peer to peer across machines in the rack for the access to the cache. And within a rack, network bandwidth is not a limitation. You can get nearly the same performance as you could from local SSD, so that’s what we did in Quiver and  there are a bunch of challenges here, because if every model wants the entire training data to be local to be within the rack, then there is just no cache space for keeping all of that.

Sridhar Vedantham: Right.

Muthian Sivathanu: Right. So we have this mechanism by which we can transparently share the cache across multiple jobs, or even multiple users without compromising security, right. And we do that by sort of intelligent content addressing of the cache entries so that even though two users may be accessing different copies of the same data internally in the cache, they will refer to the same instance.

Sridhar Vedantham: Right, I was actually just going to ask you that question about how do you maintain security of data, given that you’re talking about distributed caching, right? Because it’s very possible that multiuser jobs will be running simultaneously, but that’s good, you answered it yourself. So, you know I’ve heard you speak a lot about things like micro design and so on. How do you bring those principles to bear in these kind of projects here?

Muthian Sivathanu: Right, right. So, I alluded to this a little bit in one of my earlier points, which is the interface, I mean, if you look at a traditional scheduler which we use the job as a black box, right. That is an example of traditional philosophy to system design, where you build each layer independent of the layer above or below it, right, so that, there are good reasons to do it because you know, like multiple use cases can use the same underlying infrastructure, like if you look at an operating system, it’s built to run any process, whether it is Office or a browser or whatever, right.

Sridhar Vedantham: Right.

Muthian Sivathanu: But, in workloads like deep learning, which place particularly high demands on compute and that are super expensive and so on, there is benefit to sort of relaxing this tight layering to some extent, right. So that’s the philosophy we take in Gandiva, for example, where we say the scheduler no longer needs to think of it as a black box, it can make use of internal knowledge. It can know what mini batch boundaries are. It can know that mini batch times are repeatable and stuff like that, right. So, co-design is a philosophy that has been gaining traction over the last several years, and people typically refer to hardware, software co-design for example. What we do in micro co-design is sort of take a more pragmatic view to co-design where we say look, it’s not always possible to rebuild entire software layers from scratch to make them more tightly coupled, but the reality is in existing large systems we have these software stacks, infrastructure stacks, and what can we do without rocking the ship, without essentially throwing away everything in building everything from a clean slate. So, what we do is very surgical, carefully thought through interface changes, that allow us to expose more information from one layer to another, and then we also introduce some control points which allow one layer to control. For example, the scheduler can have a control point to ask a job to suspend. And it turns out by opening up those carefully thought through interface points, you leave the bulk of the infrastructure unchanged, but yet achieve these efficiencies that result from richer information and richer control, right. So, micro co-design is something we have been adopting, not only in Gandiva and Quiver, but in several other projects in MSR. And MICRO stands for Minimally Invasive Cheap and Retrofittable Co-design. So, it’s a more pragmatic view to co-design in the context of large cloud infrastructures.

Sridhar Vedantham: Right, where you can do the co-design with the minimum disruption to the existing systems.

Muthian Sivathanu: That’s right.

 Sridhar Vedantham: Excellent.

[Music]

Sridhar Vedantham: We have spoken a lot about the work that you’ve been doing and it’s quite impressive. Do you have some numbers in terms of you know, how jobs will run faster or savings of any nature, do you have any numbers that you can share with us?

Muthian Sivathanu: Yeah, sure. So the numbers, as always depend on the workload and several aspects. But I can give you some examples. So, in the Gandiva work that we did. We, introduce this ability to time slice jobs, right. So, the idea is, today when you launch a job in a GPU machine, that job essentially holds on to that machine until it completes, and until that time it has exclusive possession of that GPU, no other job can use it, right. And this is not ideal in several scenarios. You know, one classic example is hyperparameter tuning, where you have a model and you need to decide what exact hyperparameter values like learning rate, etc. actually are the best fit and give the best accuracy for this model. So, people typically do what is called the hyperparameter search where you run maybe 100 instances of the model, see how it’s doing, maybe kill some instances spawn of new instances, and so on, right. And hyperparameter exploration really benefits from parallelism. You want to run all these instances at the same time so that you have an apples-to-apples comparison of how they are doing. And if you want to run like 100 configurations and you have only 10 GPUs, that significantly slows down hyperparameter exploration- it serializes it, right. What Gandiva has is an ability to perform fine grained time slicing of the same GPU across multiple jobs, just like how an operating system time slices multiple processes, multiple programs on the same CPU, we do the same in GPU context, right. And because we make use of mini batch boundaries and so on, we can do this very efficiently. And with that we showed that for typical hyperparameter tuning, we can sort of speed up the end-to-end time to accuracy by nearly 5-6x, right. Uh, and so this is one example of  how time slicing can help. We also saw that from a cluster wide utilization perspective, some of the techniques that Gandiva adopted can improve overall cluster utilization by 20-30%. Right, and this directly translates to cost incurred to the cloud provider running those GPS because it means with the same GPU capacity, I can serve 30% more workload or vice versa, right, for a given workload I only need 30% lesser number of GPUs.

Sridhar Vedantham: Yeah, I mean those savings sound huge and I think you’re also therefore talking about reducing the cost of AI making the process of AI itself more efficient.

Muthian Sivathanu: That’s correct, that’s correct. So, the more we are able to extract performance out of the same infrastructure, the cost per model or the cost per user goes down and so the cost of AI reduces and for large companies like Microsoft or Google, which have first party products that require deep learning, like search and office and so on, it reduces the capital expenditure running such clusters to support those workloads.

Sridhar Vedantham

Right.

Muthian Sivathanu: And we’ve also been thinking about areas such as, today there is this limitation that large models need to run in really tightly coupled hyperclusters which are connected via InfiniBand and so on. And that brings up another dimension of cost escalation to the equation, because these are sparse, the networking itself is expensive, there is fragmentation across hyperclusters and so on. What we showed in some recent work is how can you actually run training of large models in just commodity VMs-these are just commodity GPU VMs- but without any requirement on them being part of the same InfiniBand cluster or hypercluster, but just they can be scattered anywhere in the data center, and more interestingly, we can actually run these off of spot VMs. So Azure, AWS, all cloud providers provide these bursty VMs or low priority VMs, which is away essentially for them to sell spare capacity, right. So, you get them at a significant discount. Maybe 5-10x cheaper price. And the disadvantage, I mean the downside of that is they can go away at any time. They can be preempted when real demand shows up. So, what we showed is it’s possible to train such massive models at the same performance, despite these being on spot VMs and spread over a commodity network without custom InfiniBand and so on. So that’s another example how you can bring down the cost of AI by reducing constraints on what hardware you need.

Sridhar Vedantham: Muthian, we’re kind of reaching the end of the podcast, and is there anything that you want to leave the listeners with, based on your insights and learning from the work that you’ve been doing?

Muthian Sivathanu: Yeah, so taking a step back, right? I think continued innovation in systems and efficiency and costs are going to be crucial to drive the next generation of AI advances, right. And the last 10 years have been huge for deep learning and AI and primary reason for that has been the significant advance in both hardware in terms of emergence of GPUs and so on, as well as software infrastructure to actually parallelize jobs,  run large distributed jobs efficiently and so on. And if you think about the theory of deep learning, people knew about backpropagation about neural networks 25 years ago. And we largely use very similar techniques today. But why have they really taken off in the last 10 years? The main catalyst has been sort of advancement in systems. And if you look at the trajectory of current deep learning models, the rate at which they are growing larger and larger, systems innovation will continue to be the bottleneck in sort of determining the next generation of advancement in AI.

Sridhar Vedantham: Ok Muthian, I know that we’re kind of running out of time now but thank you so much. This has been a fascinating conversation.

Muthian Sivathanu: Thanks Sridhar, it was a pleasure.

Sridhar Vedantham: Thank you.Opens in a new tab

The post Podcast: Accelerating AI Innovation by Optimizing Infrastructure. With Dr. Muthian Sivathanu appeared first on Microsoft Research.

Episode 009 | June 15, 2021

The Internet of Things has been around for a few years now and many businesses and organizations depend on data from these systems to make critical decisions. At the same time, it is also well recognized that this data- even up to 40% of it- can be spurious, and this obviously can have a tremendously negative impact on an organizations’ decision making. But is there a way to evaluate if the sensors in a network are actually working properly and that the data generated by them are above a defined quality threshold? Join us as we speak to Dr Akshay Nambi (opens in new tab) and Ajay Manchepalli (opens in new tab), both from Microsoft Research India, about their innovative work on making sure that IoT data is dependable and verified, truly enabling organizations to make the right decisions.

Akshay Nambi is a Senior Researcher at Microsoft Research India. His research interests lie at the intersection of Systems and Technology for Emerging Markets broadly in the areas of AI, IoT, and Edge Computing. He is particularly interested in building affordable, reliable, and scalable IoT devices to address various societal challenges. His recent projects are focused on improving data quality in low-cost IoT sensors and enhancing performance of DNNs on resource-constrained edge devices. Previously, he spent two years at Microsoft Research as a post-doctoral scholar and he has completed his PhD from the Delft University of Technology (TUDelft) in the Netherlands.

Ajay Manchepalli, as a Research Program Manager, works with researchers across Microsoft Research India, bridging Research innovations to real-world scenarios. He received his Master’s degree in Computer Science from Temple University where he focused on Database Systems. After his Masters, Ajay spent his next 10 years shipping SQL Server products and managing their early adopter customer programs.

More information on the Dependable IoT project is here: aka.ms/dependableIoT (opens in new tab)

The Verified Telemetry repository on GitHub is here: aka.ms/verifiedtelemetry (opens in new tab)

For more information about the Microsoft Research India click here (opens in new tab).

Related

• Microsoft Research India Podcast: More podcasts from MSR India (opens in new tab)
• iTunes (opens in new tab): Subscribe and listen to new podcasts on iTunes
• Android (opens in new tab)
• RSS Feed (opens in new tab)
• Spotify (opens in new tab)
• Google Podcasts (opens in new tab)
• Email (opens in new tab)

Transcript

Ajay Manchepalli: The interesting thing that we observed in all these scenarios is how the entire industry is trusting data, and using this data to make business decisions, and they don’t have a reliable way to say whether the data is valid or not. That was mind boggling. You’re calling data as the new oil, we are deploying these things, and we’re collecting the data and making business decisions, and you’re not even sure if that data that you’ve made your decision on is valid. To us it came as a surprise that there wasn’t enough already done to solve these challenges and that in some sense was the inspiration to go figure out what it is that we can do to empower these people, because at the end of the day, your decision is only as good as the data.

[Music]

Sridhar Vedantham: Welcome to the Microsoft Research India podcast, where we explore cutting-edge research that’s impacting technology and society. I’m your host, Sridhar Vedantham.

[Music]

The Internet of Things has been around for a few years now and many businesses and organizations depend on data from these systems to make critical decisions. At the same time, it is also well recognized that this data- even up to 40% of it- can be spurious, and this obviously can have a tremendously negative impact on an organizations’ decision making. But is there a way to evaluate if the sensors in a network are actually working properly and that the data generated by them are above a defined quality threshold? Join us as we speak to Dr Akshay Nambi and Ajay Manchepalli, both from Microsoft Research India, about their innovative work on making sure that IoT data is dependable and verified, truly enabling organizations to make the right decisions.

[Music]

Sridhar Vedantham: So, Akshay and Ajay, welcome to the podcast. It’s great to have you guys here.

Akshay Nambi:  Good evening Sridhar. Thank you for having me here.

Ajay Manchepalli: Oh, I’m excited as well.

Sridhar Vedantham: Cool, and I’m really keen to get this underway because this is a topic that’s quite interesting to everybody, you know.  When we talk about things like IoT in particular, this has been a term that’s been around for quite a while, for many years now and we’ve heard a lot about the benefits that IoT can bring to us as a society or as a community, or as people at an individual level. Now you guys have been talking about something called Dependable IoT. So, what exactly is Dependable IoT and what does it bring to the IoT space?

Ajay Manchepalli: Yeah, IoT is one area we have seen that is exponentially growing. I mean, if you look at the number of devices that are being deployed it’s going into the billions and most of the industries are now relying on this data to make their business decisions. And so, when they go about doing this, we have, with our own experience, we have seen that there are a lot of challenges that comes in play when you’re dealing with IoT devices. These are deployed in far off locations, remote locations and in harsh weather conditions, and all of these things can lead to reliability issues with these devices. In fact, the CTO of GE Digital mentioned that, you know, about 40% of all the data they see from these IoT devices are spurious, and even KPMG had a report saying that you know over 80% of CEOs are concerned about the quality of data that they’re basing their decisions on.

And we observed that in our own deployments early on, and that’s when we realized that there is, there is a fundamental requirement to ensure that the data that is being collected is actually good data, because all these decisions are being based on the data. And since data is the new oil, we are basically focusing on, ok, what is it that we can do to help these businesses know whether the data they’re consuming is valid or not and that starts at the source of the truth, which is the sensors and the sensor devices. And so Akshay has built this technology that enables you to understand whether the sensors are working fine or not.

Sridhar Vedantham: So, 40% of data coming from sensors being spurious sounds a little frightening, especially when we are saying that you know businesses and other organizations base a whole lot of the decisions on the data they’re getting, right?

Ajay Manchepalli: Absolutely.

Sridhar Vedantham: Akshay, was there anything you wanted to add to this?

Akshay Nambi: Yeah, so if you see, reliability and security are the two big barriers in limiting the true potential of IoT, right? And over the past few years you would have seen IoT community, including Microsoft, made significant progress to improve security aspects of IoT. However, techniques to determine data quality and sensor health remain quite limited. Like security, sensor reliability and data quality are fundamental to realize the true potential of IoT which is the focus of our project- Dependable IoT.

Sridhar Vedantham: Ok, so you know, once again, we’ve heard these terms like IoT for many years now. Just to kind of demonstrate what the two of you have been speaking about in terms of various aspects or various scenarios in which IoT can be deployed, could you give me a couple of examples where IoT use is widespread?

Akshay Nambi: Right, so let me give an example of air pollution monitoring. So, air pollution is a major concern worldwide, and governments are looking for ways to collect fine grained data to identify and curb pollution. So, to do this, low-cost sensors are being used to monitor pollution levels. There have been deployed in numerous places on moving vehicles to capture the pollution levels accurately. The challenge with these sensors are that these are prone to failures, mainly due to the harsh environments in which they are deployed.

For example, imagine a pollution sensor is measuring high pollution values right at a particular location. And given air pollution is such a local phenomenon, it’s impossible to tell if this sensor data is an anomaly or a valid data without having any additional contextual information or sensor redundancy.  And due to these reliability challenges the validity and viability of these low-cost sensors have been questioned by various users.

Sridhar Vedantham: Ok, so it sounds kind of strange to me that sensors are being deployed all over the place now and you know, frankly, we all carry sensors on ourselves, right, all the time. Our phones have multiple sensors built into them and so on. But when you talk about sensors breaking down or being faulty or not providing the right kind of data back to the users, what causes these kind of things? I mean, I know you said in the context of, say, air pollution type sensors, you know it could be harsh environments and so on, but what are other reasons for, because of which the sensors could fail or sensor data could be faulty?

Akshay Nambi: Great question, so sensors can go bad for numerous reasons, right? This could be due to sensor defect or damage. Think of a soil moisture sensor deployed in agricultural farm being run over by a tractor. Or it could be sensor drift due to wear and tear of sensing components, sensor calibration, human error and also environmental factors, like dust and humidity. And the challenge is, in all these cases, right, the sensors do not stop sending data but still continues to keep sending some data which is garbage or dirty, right? And the key challenge is it is nontrivial to detect if a remote sensor is working or faulty because of the following reasons. First a faulty sensor can mimic a non-faulty sensor data which is very hard to now distinguish. Second, to detect sensor faults, you can use sensor redundancy which becomes very expensive. Third, the cost and logistics to send a technician to figure out the fault is expensive and also very cumbersome. Finally, time series algorithms like anomaly detectors are not reliable because an anomaly need not imply it’s a faulty data.

Sridhar Vedantham: So, a quick question on one of the things that you’ve said. When you’re talking about sensor redundancy, this just means that deploy multiple sensors, so if one fails then you use the other one or do you use data from the other one. Is that what that means?

Akshay Nambi: Yeah, so sensor redundancy can be looked at both the ways. When one fails you could use the other, but it also be used to take the majority voting of multiple sensors in the same location. Going back to my air pollution example, if multiple sensors are giving very high values right, then you have high confidence in the data as opposed to thinking that is a faulty data. So that’s how sensory redundancy is typically used today.

Sridhar Vedantham: OK, and there you just have to take on faith that the data that you’re getting from multiple sensors is actually valid.

Akshay Nambi: Exactly, exactly. You never know that if all of them could have undergone the same fault.

Sridhar Vedantham: Right.

Ajay Manchepalli: It’s interesting that when we think about how the industry tries to figure out if the sensors are working or not. There are three distinct approaches that we always observe, right? One is you have the sense of working, but you also try to use additional surrounding data. For example, let’s say it’s raining heavily, but your moisture sensor is indicating that the moisture level is low. That data doesn’t align, right. The weather data indicates there’s rains, but the moisture sensor is not giving you the right reading, so that’s one way people can identify it’s working or not. The other is what we just talked about, which is sensor redundancy- just to increase the number of sensors in that area and try to poll among a bunch of sensors. That also makes sense. And the third one is what typically you really can trust and that is you deploy someone out there physically, go look at the sensor and then have it tested. And if you start thinking about the scenarios we are talking about, which is remote locations, far away locations- imagine deploying sensors across the country, having to send people out and validate and things like that. There is cost associated to sending people as well as you have that sort of a down time, and so being able to, you know, remotely and reliably be able to say that the sensor is at fault, is an extremely empowering scenario. And as we look at this, it’s not just sensor reliability, right? For example, if you think of a telephone, a landline, right, you have the dial tone which tells you if the phone is working or not, right? Similarly, we are trying to use certain characteristics in these sensors, that tells us if it’s working or not. But the beauty of this solution is it’s not just limited to being a dial tone for sensors, it is more than that. It not only tells you whether it is working or not, it can tell you if it is the sensor you intended to deploy.

I mean, think of it this way. A company could work with a vendor and procure certain class of sensors and they have an agreement for that. And when these sensors are deployed, the actual sensors that get deployed may or may not be in that class of devices, intentionally or unintentionally, right? How do you know that? If we understand the nature of the sensor, we can actually remotely identify the type of sensor that is deployed and help industries essentially figure out whether the sensor that’s deployed is the sensor you intended to. So it’s more than just whether the sensor is working, you can identify it, you can even figure out things like data drift. This is a pretty powerful scenario that we are going after.

Sridhar Vedantham: Right, and that’s a lovely teaser for me to ask my next question. What exactly are you guys talking about and how do you do this?

Akshay Nambi: Right, so our key value proposition right is basically a simple and easy way to remotely measure and observe the health of the sensor. The core technology behind this value proposition is the ability to automatically generate a fingerprint. When I say a fingerprint, what I’m referring to is the unique electrical characteristic, exhibited by these sensors, both analog and digital. Let me give an example. So, think of analog sensors which produce continuous output signal proportional to the quantity being measured. Our key insight here is that a sensor’s voltage response right after powering down exhibits a unique characteristic, what we refer to as fall curve? This fall curve is dependent upon the sensor circuitry and the parasitic elements present in the sensor, thereby making it unique for each sensor type. So think of this as basically as fall curve acts as a reference signature for the sensor when it is working. And when the sensor goes bad, this fall curve drastically changes, and now by just comparing this fingerprint, we can tell whether a sensor is working or faulty.

Ajay Manchepalli: The interesting part about the fingerprint that Akshay just mentioned is that it is all related to the physical characteristics of the sensors, right? You have a bunch of capacitors, resistors, all of those things put together to build the actual sensor device. And each manufacturer or each sensor type or each scenario would have a different circuit and because of that, when you power down this, because of its physical characteristics, you see different signatures. So this is a unique way of being able to identify not just what type of sensor, but even based on the manufacturer, because the circuitry for that particular manufacturer will be different.

Sridhar Vedantham: So, just to clarify, when you’re saying that sensors have unique fingerprints, are you talking about particular model of a sensor or a particular class of a sensor or a particular type of sensor?

Akshay Nambi: Right, great question again. So, these fingerprints are unique for that particular type of sensors. For example, take soil moisture sensor from Seed Studio, for that particular type of sensor from that manufacturer, this signature remains the same. So all you have to do is for that manufacturer and for that sensor type you collect the fingerprint once. And then you can use that to compare against the operational fingerprints. Similarly, in case of digital sensors we use current drawn as a reference fingerprint to detect whether the sensor is working or not, and the key hypothesis here behind these fingerprints, is that when a sensor accumulates damage, we believe its physical properties also change, leading to a distinct current profile compared to that of a working sensor? And that’s the key property behind developing these fingerprints and one of the key aspects of these fingerprints is also that this is unaffected by the external factors like environmental changes like temperature, humidity, right. So these fingerprints are unique for each sensor type and also are independent of the environmental changes. In that way, once you collect a fingerprint that should hold good irrespective of your scenario where you are deploying the sensor.

Ajay Manchepalli: One other thing that I want to call out there is the beauty of this electrical signatures is based on the physical characteristics, right? So, it’s not only when this device fails that the physical characteristics changes, and hence the signature changes, but also the beauty of this is that over time, when things degrade, that implies that the physical characteristics of that sensor or the device is also degrading, and when that happens, your electrical signature also shows that kind of degradation, and that is very powerful, because now you can actually identify or track the type of data drift that people are having. And when you observe such data drift, you can have calibration mechanisms to kind of recalibrate the data that you’re getting and continue to function while you deploy people out and get it rectified, and things like that. So, it almost gives you the ability to have a planned downtime because you’re not only seeing that the sensor has failed, but you are observing that the sensor will potentially fail down the line and you can take corrective actions.

Sridhar Vedantham: Right, so basically you’re getting a heads up that something bad is going to happen with the sensor.

Ajay Manchepalli: Exactly.

Sridhar Vedantham: Great. And have you guys actually deployed this out in the field in real world scenarios and so on to figure out whether it works or not?

Akshay Nambi: Yeah, so this technology is already deployed in hundreds of devices in the space of agricultural farms, water monitoring and air pollution monitoring. To give you a concrete example, we are working with a company called Respirer who is using dependable IoT technology to provide reliable high fidelity pollution data to its customers and also policymakers. So, for example, Respirer today is able to provide for every data point what they measure, they are able to provide the status of the sensor, whether a sensor is working or faulty. This way users can filter out faulty or drifted data before consuming them. This has significantly increased the credibility of such low-cost sensors and the data that it is generating. And the key novelty to highlight again here is that we do this without any human intervention or redundancy. And in fact, if you think about it, we are not even looking at the sensor data. We are looking at these electrical characteristics, which is completely orthogonal to data, to determine whether the sensor is working, faulty, or drifted.

Ajay Manchepalli: The interesting part of this work is that we observed in multiple real-world scenarios that there was a real need for reliability of such sensors, and it was really impacting their function. For example, there is a team that’s working on smart agriculture, and the project is called FarmBeats. And in that case, we observed that they had these sensors deployed out in the fields and out there in the farms, you have harsh conditions, and sensors could easily get damaged, and they had to actually deploy people to go and figure out what the issue is. And it became very evident and clear that it was important for us to be able to solve that challenge of helping them figure out if the sensor is working or not, and the ability to do that remotely. So that that was sort of the beginning and maybe Akshay, you can talk about the other two projects that led after that.

Akshay Nambi: Right, so another example is Respirer who is using dependable IoT technology to provide reliable, high-fidelity pollution data to its customers and policymakers. So they are now measuring the sensor status for every time they measure pollution data to determine whether the data which was measured, as from a working or a faulty or a drifted sensor. This way the users can filter out faulty or drifted data before they consume them. And this has significantly increased the credibility of low-cost sensors and the data it is measuring. To give another example, we’re also working with Microsoft for Startups and Accenture for a particular NGO called. Jaljeevika, which focus on improving livelihood of small-scale fish farmers. They have a IoT device that monitors temperature, TDS, pH of water bodies to provide advisories for fish farmers. Again, since these sensors are deployed in remote locations and farmers are relying on this data and the advice being generated, it is very critical to collect reliable data.  And today Jaljeevika is using dependable IoT technology to ensure the advices generated is based on reliable IoT data.

[Music]

Sridhar Vedantham: Right, so this is quite inspiring, that you’ve actually managed to deploy these things in, you know, real life scenarios and it’s already giving benefits to the people that you’re working with. You know, what always interests me with research, especially when you have research that’s deployed in the field- is there anything that came out of this that surprised you in terms of learning, in terms of outcome of the experiments that you conducted?

Akshay Nambi: Yeah, so I can give you one concrete learning, right, going back to air pollution sensors, so we have heard partners identifying these sensors going bad within just few weeks of deployment. And today they have no way to figure out what was wrong with these sensors.  Using out technology, in many cases they were able to pinpoint, yes, these are faulty sensor which needed replacement right? And there was also another interesting scenario where the sensor is working well- it’s just that because of dust, the sensor was showing wrong data.  And we were able to diagnose that and inform the partner that all you have to do is just clean the sensor, which should bring back to the normal state as opposed to discarding that. So that was a great learning in the field what we had.

Ajay Manchepalli: The interesting thing that we observed in all these scenarios is how the entire industry is trusting data, and using this data to make business decisions, and they don’t have a reliable way to say whether the data is valid or not. That was mind boggling. You’re calling data as the new oil, we are deploying these things, and we’re collecting the data and making business decisions, and you’re not even sure if that data that you’ve made your decision on is valid. To us it came as a surprise that there wasn’t enough already done to solve these challenges and that in some sense was the inspiration to go figure out what it is that we can do to empower these people, because at the end of the day, your decision is only as good as the data.

Sridhar Vedantham: Right. So, you know, one thing that I ask all my guests on the podcast is, you know, the kind of work that you guys do and you’re talking about is truly phenomenal. And is there any way for people outside of Microsoft Research or Microsoft to actually be able to use the research that you guys have done and to be able to deploy it themselves?

Akshay Nambi: Yeah. Yeah. So all our work right is in public domain. So we have published numerous top conference papers in the areas of IoT and sensors. And all of these are easily accessible from our project page aka.ms/dependableIoT (opens in new tab). And in fact, recently we also made our software code available through a SDK on GitHub, which we call as Verified Telemetry. So today IoT developers can now seamlessly integrate this SDK into their IoT device and get sensor status readily. We have also provided multiple samples on how do you integrate with the device, how do you use a solution sample and so on. So if you are interested, please visit aka.ms/verifiedtelemetry (opens in new tab) to access our code.

Sridhar Vedantham: Right, and it’s also very nice when a research project name clearly and concisely says what it is all about. Verified Telemetry- it’s a good name.

Akshay Nambi: Thank you.

Sridhar Vedantham: All right, so we’re kind of coming to the end of the podcast. But before we, you know, kind of wind this thing up- what are you looking at in terms of future work? I mean, where do you go with this?

Akshay Nambi: So, till now we are mostly focused on some specific scenarios in environmental monitoring and so on, right? So, one area which we are deeply thinking is towards autonomous and safety critical systems. Imagine a faulty sensor in a self-driving vehicle or an autonomous drone, right? Or in an automated factory floor, where data from these sensors are used to take decisions without human in the loop.  In such cases, bad data leads to catastrophic decisions. And recently we have explored one such safety critical sensor, which is smoke detectors. And as we all know, smoke detectors are being deployed in numerous scenarios right from hospitals to shopping malls to buildings, and the key question which we went after, right, is how do you know if your smoke detector is working or not, right? To address this, what today people do is, especially in hospitals, they do a manual routine maintenance check where a person uses a aerosol can, to trigger the smoke alarm and then turn them off in the back end.

Sridhar Vedantham: OK, that does not sound very efficient.

Akshay Nambi: Exactly, and it’s also a very laborious process and significantly limits the frequency of testing? And the key challenge, unlike other sensors here, is you cannot notice failures until unless there is a fire event or smoke.

Sridhar Vedantham: Right.

Akshay Nambi: Thus it is very imperative to know whether your detector is working or not in a non-smoke condition. We have again developed a novel fingerprint which can do this and this way we can detect if a sensor is working or faulty even before a fire event occurred and alert the operators in a timely manner. So for those who are interested to understand and curious of how would you do that, please visit our webpage and access the manuscript.

Sridhar Vedantham: Yeah, so I will add links to the web page as well as to the GitHub repository in the transcript of this podcast.

Akshay Nambi: Thank you.

Sridhar Vedantham: Ajay, was there something you wanted to add to that?

Ajay Manchepalli: Yeah, in all our early deployments that we have made, we have seen that sensor fault is one of the primary issues that comes in play and that’s what this has been addressing. But there are many other scenarios that come up that are very relevant and can empower the scenarios even further and that is things like, when you have the data drift or when you observe that the sensors are not connected correctly to the devices and so is some sort of sensor identification. These are some of the things that we can extend on top of what we already have. And while they are incremental changes in terms of the capability, the impact and the potential it can have for those scenarios is tremendous. And that’s what keeps it exciting is that all the work that we are doing is driven by the actual needs that we are seeing out there in the field.

Sridhar Vedantham: Excellent work. And Akshay and Ajay, thank you so much once again for your time.

Akshay Nambi: Thank you Sridhar. Great having this conversation with you.

Ajay Manchepalli: Yep, thanks Sridhar. This is exciting work, and we can’t wait to do more and share more with the world.

[Music]Opens in a new tab

The post Podcast: Dependable IoT- Making data from IoT devices dependable and trustworthy for good decision making appeared first on Microsoft Research.

Episode 008 | April 20, 2021

Podcast: Research @Microsoft Research India: interdisciplinary and impactful. With Dr. Sriram Rajamani

Microsoft Research India is constantly exploring how research can enable new technologies that positively impact the lives of people while also opening new frontiers in computer science and technology itself. In this podcast we speak to Dr. Sriram Rajamani, distinguished scientist and Managing Director of the Microsoft Research India Lab. We talk about some of the projects in the lab that are making fundamental changes to the computing at Internet scale, computing at the edge and the role he thinks technology should play in the future to ensure digital fairness and inclusion. Sriram also talks to us about a variety of things his own journey as a researcher, how the lab has changed from the time he joined it years ago, and his vision for the lab.

Sriram’s research interests are in designing, building and analyzing computer systems in a principled manner. Over the years he has worked on various topics including Hardware and Software Verification, Type Systems, Language Design, Distributed Systems, Security and Privacy. His current research interest is in combining Program Synthesis and Machine Learning.

Together with Tom Ball, he was awarded the CAV 2011 Award for “contributions to software model checking, specifically the development of the SLAM/SDV software model checker that successfully demonstrated computer-aided verification techniques on real programs.” Sriram was elected ACM Fellow in 2015 for contributions to software analysis and defect detection, and Fellow of  Indian National Academy of Engineering in 2016.

Sriram was general chair for POPL 2015 in India, and was program Co-Chair for CAV 2005. He co-founded the Mysore Park Series, and the ISEC conference series in India. He serves on the CACM editorial board  as co-chair for special regional sections, to bring computing innovations from around the world to CACM.

Sriram has a PhD from UC Berkeley, MS from University of Virginia and BEng from College of Engineering, Guindy, all with specialization in Computer Science. In 2020, he was named as a Distinguished Alumnus by College of Engineering, Guindy.

For more information about the Microsoft Research India click here.

Related

• Microsoft Research India Podcast: More podcasts from MSR India
• iTunes: Subscribe and listen to new podcasts on iTunes
• Android
• RSS Feed
• Spotify
• Google Podcasts
• Email

Transcript

Sriram Rajamani: We are not like an ivory tower lab. You know we are not a lab that just writes papers. We are a lab that has our hands and feet, dirty, we sort of get ourselves dirty sort of get in there, you know, we test our assumptions, see whether it works, learn from them and in that sense actually the problems that we work on are a lot more real than a purely academic environment.

[Music]

Sridhar Vedantham: Welcome to the Microsoft Research India podcast, where we explore cutting-edge research that’s impacting technology and society. I’m your host, Sridhar Vedantham.

[Music]

Sridhar Vedantham: Microsoft Research India is constantly exploring how research can enable new technologies that positively impact the lives of people while also opening new frontiers in computer science and technology itself. In this podcast we speak to Dr. Sriram Rajamani, distinguished scientist and Managing Director of the Microsoft Research India Lab. We talk about some of the projects in the lab that are making fundamental changes to computing at Internet scale, computing at the edge and the role he thinks technology should play in the future to ensure digital fairness and inclusion. Sriram also talks to us about a variety of things his own journey as a researcher, how the lab has changed from the time he joined it many years ago and his vision for the lab.

Sridhar Vedantham: So today we have a very special guest on the podcast, and he is none other than Dr. Sriram Rajamani, who is the Managing Director of the Microsoft Research Lab in India. So Sriram welcome to the podcast.

Sriram Rajamani: Yeah, thank you. Thank you for having me here, Sridhar.

Sridhar Vedantham: OK, you’ve been around in Microsoft Research for quite a while, right? Can you give me a brief background as to how you joined and when you join and what’s your journey been in MSR so far?

Sriram Rajamani: Yeah, so I joined in 1999. And , oh man, it’s now 22 years, I guess. I’ve been here for a while.

Sridhar Vedantham: That’s a long time.

Sriram Rajamani: I joined in Microsoft Research in Redmond right after I finished my PhD in Berkeley and then I, you know, my PhD was in formal verification. So, my initial work in Microsoft in Redmond was in the area of formal verification and then at some point I moved to India around 2006 or something like that. So I think I spent about six or seven years in Redmond and my remaining time- another 15 years- in India. So that’s been my journey, yeah.

Sridhar Vedantham: OK, so this is interesting, right, because, you know, we constantly hear about India as being this great talent pool for software engineers, but we certainly don’t hear as often that it is a great place for a computer science research lab. Why do you think a Microsoft Research lab in India works and what drew you to the lab here?

Sriram Rajamani: I’m a scientist and I joined MSR because I wanted to do high quality science work that is also applicable in the real world, you know.  That’s why I joined MSR and the reason why I moved to India was because at some point. I just wanted to live here – I wanted to live here because I have family here and so on and then Anandan started the lab and so somehow things came together, and that’s why I personally moved. But if you ask, you know, ask me why it makes sense for MSR to have a lab here, the reasons are quite clear.

I think we are such a big country, we have enormous talent. I think talent is the number one reason I think we are here. Particularly unique to India is that we have really strong undergraduate talent, which is why we have programs like our Research Fellow program. But over the past, many years, right, the PhD talent is also getting better and better. As you know, initially when we started, you know, we recruited many PHDs you know from abroad, who had their PhD from abroad and then return just like me. But over the years we’ve also recruited many PhDs from Indian institutions as well.

So, I think that talent is the number one reason.

The second reason is you know the local tech ecosystem is very different.  It started out as a service industry for the West- you know essentially all of the software we were doing, we were servicing companies in the western hemisphere. But over time, India has also become a local consumer of technology, right? Now, be it if you sort of think about, you know Ola or Flipkart, you know, the country is now using technology for its own local purposes. And because of the size and scale of the country, the amount the government and industry is pushing digitization, there’s a huge opportunity there as well.

And finally, I would say another reason to have a lab is in a place like India that it’s a very unique testbed. You know, cost is a huge concern in a place like India, technology has to be really low cost for it to be adopted here. There are very severe resource constraints. Be it bandwidth…you know if you think about NLP, you know many of our languages don’t have data resources. Very unreliable infrastructure- things fail all the time, and so you know, I’ve heard of saying that you know if you build something so that it works in India, it works anywhere. So it’s a test bed to actually build something.

If you can deploy it and make it work here, you can make it work anywhere. So in that sense actually it’s also another reason.

Sridhar Vedantham: OK, so basically it works here it’s a good certification that it’ll work anywhere in the world.

Sriram Rajamani: Yeah, yeah.

Sridhar Vedantham: All right. OK Sriram, so here’s something I’m very curious about. How does a research scientist end up becoming the managing director of a lab?

Sriram Rajamani:

So the short answer is that it was rather unplanned, but maybe I can give a more longer answer. You know, I started out, you know, being a researcher like anyone else who joins MSR. My initial projects were all in the area of, you know, formal verification, you know, I built together with Tom Ball something called static driver verifier that used formal methods to improve windows reliability. Then I worked on verifiable design- how can you do better design so that you produce better systems?

Then I worked on, you know, security, and now I work on machine learning and program synthesis. And you know, a common thread in my work has always been the use of programming languages and formal methods to sort of understand how to build various kinds of systems be it drivers, be it secure systems, be it machine learning systems. That has been sort of the theme underlying my research. But to answer your question as to how I sort of became lab director, you know, after some years after I moved back to MSR India, you know Anandan who was the lab director then, you know, he left. There was a leadership churn there, and at the time I was asked whether I would consider being the lab director. The first time I declined and because I had many other technical projects that are going on. But I got the opportunity the second time, you know, when Chandu and Jeanette really encouraged me when Chandu decided to move on. I had been in MSR maybe 15-16 years when that event happened. And one of the reasons why I decided to take this up was I felt very strongly for MSR, and I thought that MSR has given me a lot and I wanted to give back to MSR and MSR India.

And MSR India is easily one of the best CS, computer science industrial labs in this part of the world.

And, you know, it made sense that I actually devote my time to support my colleagues, grow their lab in ambition, impact and I sort of had a sense of purpose in that, and so I decided to take this on. So the real answer your question is I don’t think anyone plans to be a lab director, and sometimes you know you get an opportunity to become one, and sometimes you say yes.

Sridhar Vedantham: Great. OK, and you know, given that you’ve been in the lab here in India for quite a while, how do you see the lab having evolved over the years? I mean, I’m sure there are lots of things that have changed quite a bit. So, what do you think are those things that have changed quite a bit and what’s not changed and what are the kind of things that you’d like to preserve going forward?

Sriram Rajamani: Yeah, I think the number one thing that has not changed is quality. MSR… I’ve now been here for now 21 years and I’ve been with MSR India you know, from the very beginning. I came here after 6-7 months after the lab started. We’ve always had great people and the quality of the work we do has always been exceptional. But I think what has changed over the years is that we think much more end to end. When I joined, you know, ’99, we were sort of more academic in nature. We always used to publish in high quality conferences, which we still do. But I think what we do more now is that we think much more end to end. We are no longer satisfied with solving a particular piece of a problem, but we sort of think about how does that piece connect with many, many other pieces, some social, some technical and how do those things fit together broadly to solve problems end to end. We sort of think about that a lot more. As a result, I think we more often than not deploy what we build, you know, either in scale, solutions that actually are adopted by product groups, or actually in our communities to actually validate whether what we think about as something that creates a change, does it indeed create a change and learn from that and use that to even reframe our problems, test our assumptions.  And so, you know, I don’t think we are, we are not like an ivory tower lab. You know we are not a lab that just writes papers. We just we are a lab that has our hands and feet, dirty, we sort of get ourselves dirty, sort of get in there, you know, we test our assumptions, see whether it works, learn from them and in that sense actually the problems that we work on are a lot more real than a purely academic environment.

I think that that’s the way in which things have changed. And I think partly also, Sridhar, as you do that,

we have become a lot more interdisciplinary…

Sridhar Vedantham: Right.

Sriram Rajamani: …you know, if you look at our projects today, right? Because if you want to get something to work end to end, it is not just one piece you build, you know. You have to make it interdisciplinary and many of our projects are interdisciplinary. I think that’s the other way in which we’ve changed.

Sridhar Vedantham: Yeah, in fact this particular term, right, interdisciplinary research- is something that I’ve heard quite often coming from you. Do you want to just bring up a couple of examples of what you mean by interdisciplinary research through by using some projects as examples?

Sriram Rajamani: Yeah, I can give like you know, two or three you know. The first one that comes to mind is our EzPC or you know, our multiparty computation project. And if you look at actually how that project is working, the whole goal is to take computations, be it, DNN training or DNN inference and run it securely with multiple parties. And you know that’s a pretty complex problem. There’s compiler people, there’s programming languages people, and there’s cryptographers. All of them work together to build a solution where the programmer can express their computation in some language, and there’s a compiler that compiles it. And then there is, you know, a lot of cryptography smarts are there in order to make this you know, multiparty computation work and, and that’s very unique. You can’t do this without compiler people and the cryptographers getting together.

Another example, you know, is if you look at our Akupara semantic search work, that’s actually a combination of algorithms, work machine learning work and systems work so that we can index trillions of vector indices and look them up right in a reasonable amount of time with a reasonable number of machines. I mean, I can’t imagine you doing that without expertise in algorithms, machine learning and systems.

And if you look at our, more recent, you know societal impact projects like we have this project called HAMS which we are using to improve road safety. I mean that has actually quite a bit of tech like computer vision. You have to make that work on a smartphone. So you need to have systems innovation to actually make that work on a smartphone. And it has quite a bit of HCI. I mean it has to work in an environment where you go into a driver license testing RTO and it should just work there, right? You know it has to work with people, the feedback that is given should be consumable. You know if somebody fails a driving test, the feedback has to be given in such a way that it’s a positive experience for them even if they fail the test, right? So it has all these interdisciplinary aspects. And so I hope that those give you a little bit of a flavor for what it takes to solve things end to end.

[Music]

Sridhar Vedantham: A lot of the listeners of this podcast are not going to be really familiar with all the stuff that we do here at MSR right, at MSR India especially. In your mind, you know, how do you kind of categorize or bucket the different research work that goes on in the lab?

Sriram Rajamani: We now think about our work as being classified into themes and the themes are different from our expertise. If you look at our expertise, right, our expertise has always been, you know, from the beginnings of the lab, we have 4 broad areas of expertise.

We have, you know, people with expertise in algorithms, second in machine learning, and third in systems, very broadly interpreted in including programming languages, distributed systems, networking, security and so on.

And then we have people who do human computer interaction and social sciences, right?

Those are our four areas of expertise, but if you look at the way we organize our work now, it is in the  themes. We have 5 themes. One theme is around large-scale machine learning, you know, things like recommendation systems, search, large multilingual learning which spans an entire gamut, from algorithms to practical machine learning algorithms as well as systems, right, you know, in order to build them and scale them.

Then we have two systems related themes. One is data driven systems and networking where we are using telemetry and the enormous amount of data that we get from large scale cloud systems to do machine learning on them and improve those systems themselves. And then the second systems area we have is called co-designed systems, where we think about interdisciplinary systems work that spans distributed systems, security, privacy, programming languages, verifications. So we sort of think about systems much more holistically.

Another thing we have is edge computing, where we sort of think of about machine learning systems, usability in the edge, which is such an important topic from the perspective of India. And the last theme is socio-technical systems and inclusion where we really think about technology as an ally for an enabler for inclusion and empowerment.

And each of these five teams, right, draws on the expertise of people from these various disciplines.

Sridhar Vedantham: Great, so I’ve heard you many times talking about things like tech at scale, so I think you have a couple of things you know. There are a couple of things that you said that kind of stick in my mind, so there is one tech at scale and then there is one tech in minute form. I forget the exact terms you use and socio-technical computing is also quite big at MSR India right now. Could you give me a flavor of what exactly is happening in, say, the tech at scale area and also the social technical computing area?

Sriram Rajamani: Yeah, so I think the tech at scale is quite important because digital systems are very pervasive now. The pandemic has only accelerated the adoption of digital systems. Most interactions these days are online, and even when we come back from the pandemic, it’s going to be hybrid, right? The amount of information is just increasing and increasing and increasing and as a result, right, for any useful user experience we need to be able to sort through this huge amount of information and make the right information available at the right time, right. And that I think is in some sense the primary goal of AI and machine learning and systems at scale, and I think most of our systems at scale work are about how to build systems that use AI and machine learning to process huge, humongous amounts of information in billions and trillions of pages or documents, or vectors, understand them and make sure that their right information is available to you at the right time. And how do you do that reliably, how do you do that securely, how do you do that while preserving privacy? So that I think is the is the crux of our tech at scale.

I already mentioned Akupara, which is a trillion-scale index and serving system that we are building for semantic search. Another at-scale project we are doing is called extreme classification where we are trying to build classifiers that can take an object and classify it into hundreds of millions of categories, right? And just like when we think about machine learning, we think about a picture and classifying it into a cat or a dog or a small number of categories. But in extreme classification we take an object like a web page or a document, and we classify it into, potentially, millions or hundreds of millions of, for example, topics. I mean, what are these topics that this document is talking about, or if this object that I’m talking about is an advertisement, what are the keyword bid phrases that are actually relevant to this advertisement, right? So those kinds of classifications are significantly more complex, and our lab really originated this field and is a thought leader in this field.

Another at-scale work that we are doing is if you take the area of DNN training. Deep neural network training is an extremely resource intensive process. You know if you take, billions and billions of training points and train deep neural networks, that uses a huge number of GPU resources and other hardware resources. Can you do that more efficiently? And we have a project called Gandiva that improves the throughput of all of the infrastructure that we are using to train these kinds of DNNs.

And we want to give you one more example. We have a project called Sankie and what Sankie does is actually, to use all of this telemetry from huge software engineering processes, including coding, testing, development to really improve the productivity of the engineering itself. So, I would say you know those are the kinds of examples of at-scale AI and ML and Systems project that we do. And I think every CS lab has to do that because that is the real world today.

Sridhar Vedantham: And we’ve actually done a podcast earlier on Sankie, so I think I’ll link through to that when we publish the transcript of this podcast.

Sriram Rajamani: Wonderful.

Sridhar Vedantham: Right, and in sociotechnical computing, do you want to talk a little more about that? And this is something that personally I find quite fascinating, you know, because this lab has always had, from the very beginning, a focus on the ICTD space itself. But the kind of projects that are happening now in the lab seem to be taking that to a different level altogether in terms of actually going out there and figuring out the impact and deploying at scale.

Sriram Rajamani: You’re right, Sridhar, that Technology for Emerging Markets has always been a really interesting area in the lab from the inception. But one thing that has changed if you sort of think about it, is that…see, when the lab was started, right, the kind of technology that was available to everybody in rural India was very different from the technology that all of us use, right? You know they had maybe feature phones and everybody else, you know, had smartphones and so on. But now connectivity, smartphone penetration and so on has increased significantly, right? So in some sense, I think the smartphone and through 4G and so on, connectivity to the cloud, the cloud and the mobile, and with the smartphone is much more accessible, much more prevalent these days, right? But still the problems are there, you know, bandwidth is a problem, you know, they don’t work in local language, English works much better than local language. Those constraints are there, but the technology platform has up leveled throughout the country.

So as a result, right, if you take our own work on socio-technical computing, we are doing technologically more sophisticated things now than we did before because more technologically sophisticated things are accessible to a much broader population of the country. That I think is the way things have changed, which is why we are actually now able to do projects like you know, HAMS where you’re using driver license testing. Because even, uh, even an RTO in a rural area, right, they have access to smartphones, right? And you know, they are interested to see whether you know driver license testing can be streamlined. So I think that the high tide has lifted the technology everywhere. I think that’s one way in which things have changed. Another one where we are now using peer to peer connectivity, this project called BlendNet, where we are actually helping people share media and other bulky resources better. And even that actually, you know the reason why we are doing this because you know smartphone, the desire to view movies, entertainment, it’s very widespread throughout the country, right? So that’s actually another example of projects.

And even just this morning I was actually looking at a piece of news where they were talking about, uh, this company Respirer Living Sciences and we’re having a collaboration with them to measure air pollution, and they want to monitor pollution and democratize the data, right. I mean this is now such an important problem and but if you look at what is needed to do that right, we have to solve really hard technical problems. Like how do you make sure that the sensors that are sensing these are reliable? If there’s a way in which the sensors are calibrated, if it is erroneous, how do you re-calibrate them? But these are hardcore technology problems that I think are important to solve a societal problem like air pollution.

So another way I think things have changed is that maybe, perhaps, previously all our societal scale problems were sort of low tech- that’s no longer true. That doesn’t mean actually that the tech works as it is, right?

You know, we still work on projects like Karya where we are trying to do data collection and crowdsourcing for low resource Indian languages and that requires actually us to build user interfaces that work with semi-literate and illiterate users and you know, make sure that we are actually able to cater to the multilingual population in the country and so on, right? So the user centered design and the need to design to people on the other side of the digital divide is still important, right?

But you know at the same time the tech tidal wave has also lifted things up, so I think that’s sort of the dynamic here, I think.

Sridhar Vedantham: Right, and there’s a bit of a conundrum here, right? Because at one point of time it was assumed that technology itself is going to help people’s lives become better and so on. And we’ve obviously seen technology permeate to levels within society that it’s never permeated before. Now this brings about questions of digital inclusion and fairness and equitable access to information and to the benefits of technology. So, a couple of questions here. How do we actually ensure things like digital inclusion and fairness? And given very specific, unprecedented situations like the one that we find ourselves now, in the midst of a pandemic, how does this actually impact people or impact society at large?

Sriram Rajamani: I think in spite of the fact that digital technology has permeated right, it is very clear that technology is still very non inclusive, right? That is also, I think true at the same time. And so there is no silver bullet I think to the question that you’re asking. I think it’s extremely important for us to think about as scientists and technologists, think about underserved populations, underserved communities and see whether the technologies that we build you know are inclusive, whether they are useful, you know. I give an example of what Manohar Swaminathan is doing with his work on accessibility, where he has done quite a bit of study in schools for visually impaired children, thinking about even the curriculum that they have you know, in STEM and computing, computational thinking, I think, for this population. And seeing whether the tools that we have and even the curriculum that we have and the technologies that we have, are they actually reaching you know this demographic and the answer is no.

And then quite a bit of work needs to be done in order to make sure that you know people with vision impairment, children with vision impairment are getting educated in digital technologies and the technology is inclusive. And there’s a huge gap there, so his work is particularly inspiring in that sense. And you know, I think problems like awareness and literacy, they are very hard problems to solve. You know you make a smartphone cheaper. You can actually make you know 4G and 5G maybe more available. But you know things like literacy, cognition and understanding of actually what’s going on, those take many, many generations to resolve. So I think one has to think about people’s context, you know people preparedness, when thinking about inclusion.

Sridhar Vedantham: Great, so I’m going to be cognizant of your time. I know you’ve got a bunch of meetings every day all the time. So before we sign off are there any final thoughts?

Sriram Rajamani: Yeah, so I would say that I think that the pandemic in some way has really accelerated digital transformation, right? But at the same time the pandemic has also exacerbated the gap between the rich and the poor. That has also happened. So I would say that, I think this is a very interesting time, as scientists and technologists. On the one hand, actually, science is our, you know, an important hope for us to get out of the pandemic, you know, be it vaccines, you know, be it digital technology to help us communicate and collaborate even when we are at our homes, technology is such an important thing to do. And in order to actually serve, you know, the large number of people we have to build technology at scale. I think that’s such an important thing. At the same time, I think, you know, that the virus doesn’t discriminate between you know, rich or poor. It doesn’t discriminate based on race or gender, and so if we have to actually get out of the pandemic, you know, we have to actually make sure that the solutions you know be it vaccines, they reach everyone. If anything, the pandemic has taught us that you know, unless we serve everyone problems like the pandemic and same thing with climate change, those are not going to be solved. Those are universal problems and that by definition they are inclusive, right? So I think my closing comment would be for technologists to think about technologies in such a way, technology in such a way that it brings people together. You know, have empathy for people in every shape, size and form and make sure that what we build, serves the whole of the world.

Sridhar Vedantham: OK Siram, thank you so much for your time. This has been a fascinating conversation.

Sriram Rajamani: Yeah, thank you Sridhar, and I wish the listeners health and happiness in the rest of the year as well. Thank you.

[Music Ends]

 Opens in a new tab

The post Podcast: Research @Microsoft Research India: interdisciplinary and impactful. With Dr. Sriram Rajamani appeared first on Microsoft Research.

Episode 007 | December 22, 2020

One of Microsoft Research India’s goals is to help strengthen the research ecosystem and encourage young students to look at research as a career. But it is not always easy for students to understand what research is all about and how to figure out if research is the right career for them. The Research Fellow program at Microsoft Research India enables bright young students to work on real-world research problems with top notch researchers across the research lifecycle, including ideation, implementation, evaluation, and deployment. Many of the students who have been part of the program have gone on to become researchers, engineers and entrepreneurs.

Today, we speak to Shruti Rijhwani (opens in new tab), a graduate of MSR India’s Research Fellow program who is currently doing her PhD at the Carnegie Mellon University, and joining us is Dr. Vivek Seshadri, a researcher at MSR India who also heads the Research Fellow program at the lab.

Shruti was a research fellow at MSR India in 2016, working on natural language processing models for code-switched text.

She is currently PhD student at the Language Technologies Institute at Carnegie Mellon University. Stemming from her work at MSR India, she has continued research in multilingual NLP, with a focus on low-resource and endangered languages.

Vivek primarily works with the Technology for Emerging Markets group at Microsoft Research India. He received his bachelor’s degree in Computer Science from IIT Madras, and a Ph.D. in Computer Science from Carnegie Mellon University where he worked on problems related to Computer Architecture and Systems. After his Ph.D., Vivek decided to work on problems that directly impact people, particularly in developing economies like India. Vivek is also the Director for the Research Fellow program at MSR India.

For more information about the Research Fellow program, click here (opens in new tab).

Related

• Microsoft Research India Podcast: More podcasts from MSR India (opens in new tab)
• iTunes (opens in new tab): Subscribe and listen to new podcasts on iTunes
• Android (opens in new tab)
• RSS Feed (opens in new tab)
• Spotify (opens in new tab)
• Google Podcasts (opens in new tab)
• Email (opens in new tab)

Transcript

Shruti Rijhwani: I think I credit my whole graduate school decision-making process, the application process, and even the way I do research in grad school to my experience as a Research Fellow in MSR India. Of course, the first thing was that I wasn’t even sure whether I wanted to go to grad school, but after going through the Research Fellow program and with my amazing mentors and collaborators at MSR India, I took the decision to apply to grad school.

[Music]

Sridhar Vedantham: Welcome to the Microsoft Research India podcast, where we explore cutting-edge research that’s impacting technology and society. I’m your host, Sridhar Vedantham.

[Music]

One of Microsoft Research India’s goals is to help strengthen the research ecosystem and encourage young students to look at research as a career. But it is not always easy for students to understand what research is all about and how to figure out if research is the right career for them. The Research Fellow program at Microsoft Research India enables bright young students to work on real-world research problems with top notch researchers across the research lifecycle, including ideation, implementation, evaluation, and deployment. Many of the students who have been part of the program have gone on to become researchers, engineers and entrepreneurs.

Today, we speak to Shruti Rijhwani (opens in new tab), a graduate of MSR India’s Research Fellow program who is currently doing her PhD at the Carnegie Mellon University, and joining us is Dr. Vivek Seshadri, a researcher at MSR India who also heads the Research Fellow program at the lab.  

[Music]

Sridhar Vedantham: OK, so I’m looking forward to this podcast because it’s going to be a little different from what we’ve done in the past, in the sense that this is not a podcast about research projects or technologies, but it’s something much more human, and we’re going to be talking about the Research Fellow program that we have at MSR India.

And I’d like to welcome a special guest- Shruti, who used to be a Research Fellow at the lab and also Vivek Seshadri, who is a researcher at the lab and whom we’ve had on the podcast earlier in a different capacity. But today he’s wearing the hat of the czar or the director of the Research Fellow program here.

So welcome, Shruti and Vivek.

Vivek Seshadri: Good evening, Sridhar, and very good morning Shruti.

Shruti Rijhwani: Hi Sridhar and Vivek, it’s great to be here and great to be back to interacting with people from MSR India. It’s been about four years since I left the RF program, so I’m really looking forward to talking about it and remembering some of my experiences.

Sridhar Vedantham: Excellent, so let’s lay a little bit of a groundwork before we jump into the whole thing. Vivek can you give us a bit of an overview of what the Research Fellow program is.

Vivek Seshadri: Sridhar, the Research Fellow program has been around ever since the organization Microsoft Research India started itself. I think initially it was called the Assistant Researcher program and then it was called the Research Assistant Program and right now we’re calling it the Research Fellow program. But the core of the program has been to enable recent undergraduate and Master’s students to spend one or two years at MSR India and get a taste for what research in computer science looks like, especially in an industrial setting.

Sridhar Vedantham: So has the program evolved over time, or have there been any substantive changes or it’s still the same at its core and its essence?

Vivek Seshadri: I think the only thing that has changed significantly is the number of research fellows that we have had. I think in the program started in its first year, I think we had had three assistant researchers, and today as we speak, we have over 50 Research Fellows in the lab working on various projects. So, in that sense, the program has definitely grown in size along with the lab, but I think the core goal of the program has not changed at all. It is still to give Research Fellows a taste of what research looks like in computer science and enable them to build their profile and prepare them for a career in computer science research and engineering.

Sridhar Vedantham: Right, so one thing that I’ve seen personally is that the Research Fellows add a huge amount of energy and life to the lab. And on that note, Shruti, what motivated you to come into MSR India to join the Research Fellow program?

Shruti Rijhwani: That’s a great question and something that I think my experience is probably what a lot of Research Fellows who apply and join the program actually go through before deciding to join the program. So, I did an undergrad degree in computer science from BITS PILANI and during those four years I took classes on machine learning, information retrieval and so on, and also did two internships where I got a taste of like how machine learning applications can be applied to products in the real world. But both of those internships were kind of focused on the engineering side and I was really interested in what a career in doing machine learning research or using machine learning for research-based applications would look like. And I knew that if I wanted to pursue a career in this field, I would probably have to go to graduate school to get a Master’s and a PhD, but I wasn’t entirely sure whether this is what I wanted to do, so it was kind of like an exploratory phase for me to be in the Research Fellow program. I wanted to get some research experience, I wanted to see what established researchers in computer science do on a daily basis, and what the research process kind of is like when you’re working in machine learning, and more specifically in natural language processing, which is what I was interested in.

Sridhar Vedantham: Right, so, Vivek, what I’m getting from Shruti is that, uh, the Research Fellow program is something that she is looking at to form a base or a basis for a longer career in research itself. So do you have any specific insights or inputs into about what the program actually offers in a structured manner to the Research Fellows that we have?

Vivek Seshadri: Yeah, so Microsoft Research now at its core is a research organization. All researchers at MSR do research in computer science and they’re working on a variety of projects spanning all areas of computer science, you know, from theory, artificial Intelligence, machine learning, systems, security and MSR India also is known for its Technology for Emerging Markets (TEM) Group where we look at, you know, problems specifically affecting developing countries like India. So, essentially, Research Fellows join one of these projects and work with world class researchers on multiple phases of research, including ideations, building solutions, prototyping those solutions, deploying them in the field, working with large real industrial datasets to test out their solutions. So that experience essentially gives the perfect taste of what modern computer science research looks like for Research Fellows, and just like Shruti, most of our Research Fellows after their stint at MSR India apply for grad school and you know, go to one of the top grad schools across the world, but there are others who decide, you know, research is not for them. Many of them join Microsoft and continue to work at Microsoft in some, you know some other role. And a few of them actually, you know, have taken the entrepreneurial route. You know, we have had CEOs of many, many big companies, including Ola and some of our own projects which have been converted to startups like Digital Green and Everwell. So, you know, some of them take that route as well. But primarily it’s that experience of what computer science research looks like today. I think that’s the essence of what the research program offers students.

Sridhar Vedantham: Great and uh, in terms of numbers, how many Research Fellows do you think would have graduated so far from MSR India over the years?

Vivek Seshadri: Like I said, you know, in the initial years it was handful of Research Fellows that we have So overall, if you look at the numbers, it’s around 150 Research Fellows in a period of 15 years. But if you look at our run rate in the most recent years, we have been graduating close to 30 Research Fellows every year, and that number is just increasing.

Sridhar Vedantham: Great, so this is another contribution of MSR to the research field, I guess.

Vivek Seshadri: Absolutely, yeah. And you know these guys hit the ground running. You know, if they do choose to do a PhD, they hit the ground running. They know how research works. They know the different phases of, you know, research projects, how to go about tackling these problems. So, in that sense I’m assuming all the advisors of past Research Fellows are extremely happy advisors- little less work for them in actually guiding their students.

Sridhar Vedantham: [Laughing] OK, I think we need to speak to some of the advisors privately to validate that.

Vivek Seshadri: Yes.

Sridhar Vedantham: So Vivek, another question for you.  You know MSR India has also got this very large internship program. What’s the fundamental difference between an internship at MSR and a Research Fellow position?

Vivek Seshadri: MSR used to have a very vibrant three-month summer internship program and I think over the years what we have realized is that for any stint at MSR to benefit both the student and the research projects at MSR, you need that reasonable amount of time. So, if you see even the internship program, you know we have almost faded out three-month internships. We only offer six-month internships these days. So in that sense, the main difference between the traditional internship program and the Research Fellow program is that duration. You know when a Research Fellow comes in, we know that they’re going to spend at least one year with us. In most cases, they are going to spend two years with us, which means the researcher, with confidence, can set a really big goal for the Research Fellow and you know, really have that long shot instead of, a short internship where to have any meaningful output, you know, you cannot have a really large vision or a large goal. So in that sense, if you actually see the contributions that Research Fellows have made to some of our projects, they are extremely substantial, solving some fundamental problems in computer science, impacting, projects in major ways within Microsoft and also having impact on society. These contributions would not be possible in a six-month internship program.

[Music]

Sridhar Vedantham: Shruti, when you were doing your Research Fellowship at MSR India, what is the kind of work you were doing and who were your mentors out here?

Shruti Rijhwani: Right, so as I said, I was really interested in machine learning applications, more specifically as applied to natural language processing problems. So when I was a Research Fellow at MSR India, I was mentored by Monojit Choudhury and also worked closely with Kalika Bali on natural language processing problems. We were focusing on the problem of code switching, which is basically when multilingual people mix languages when they’re speaking. we were basically trying to create natural language processing tools to automatically understand such code-mixed language.

Sridhar Vedantham: And how do you think that helped you? I mean you are at CMU now, which is one of the top tier PhD schools in the world. But how do you think the time that you spent here in MSR India actually helped you get to where you are?

Shruti Rijhwani: I think I credit my whole graduate school decision-making process, the application process, and even the way I do research in grad school to my experience as a Research Fellow in MSR India. Of course, the first thing was that I wasn’t even sure whether I wanted to go to grad school, but after going through the Research Fellow program and with my amazing mentors and collaborators at MSR India, I took the decision to apply to grad school. And as I said, I was working on problems in multilingual natural language processing. And although I focused on code switching at MSR India, it kind of made me very interested in continuing in this field of just trying to be able to automatically process many, many languages at the same time and through my Masters and now my PhD, all of my research has been focused on doing multilingual NLP. So, in a way my experience at MSR sort of shaped my view of NLP research and taught me important research and technical skills, going all the way from like doing a literature survey to collecting data, doing experiments, and finally writing publications. I went through that process for the first time at MSR India, and it’s like really helped me go through graduate school as well.

Sridhar Vedantham: Great, so it sounds like you really worked hard while you were at MSR. Did you actually have some time to do something other than work, to kind of go out, have fun and enjoy yourself?

Shruti Rijhwani: Yeah, I tried to keep a great work life balance at MSR India and in fact I think MSR and the Research Fellow program in general quite encourages that. We had a strong community of Research Fellows. We were all really good friends and it kind of makes sense because all of us were in the same exploratory phase of doing research for the first time, for most of us. So there’s a lot we had in common, and all of us were good friends, enjoyed ourselves outside of work as well. So yeah, I really enjoyed my time at MSR.

Sridhar Vedantham: I believe MSR India’s also got this fantastic annual offsite which is a great deal of fun.

Shruti Rijhwani: Definitely. I was a Research Fellow for one year and the offsite that year was really good fun. It’s good to interact with people outside of work as well and learn about the people you collaborate with as people, not just colleagues. So I really enjoyed the MSR offsite and as Research Fellows, we also would often have our own outings like we would go out in the city and explore the city and so on. So it was – really fun.  I really appreciated the community of Research Fellows that we had.

Sridhar Vedantham: Super. Vivek, a question for you. What does the Research Fellow selection process look like? How do you actually go about it, and is there any particular way in which you look at matching candidates to mentors?

Vivek Seshadri: Absolutely. I think in in many ways the Research Fellow selection process is similar to how grad school applications work. how universities select PhD students and there are multiple projects going on inside MSR India in different areas. Applicants have specific interests. You know, these days, even in the four years of undergrad that they go through, people develop interest in various areas, so they come in with their own interests. So our goal is to essentially identify the best candidates for each project. And like I mentioned, you know, it’s not only the case that the number of Research Fellows slots have increased over the years from three in the first year to something close to 40 to 50 right now. It’s also that the number of applicants has increased significantly.  We receive close to 1000 applications and in some sense you know it’s a long review process where we ensure that each applicant at least gets one pair of eyes looking at the application and then determining whether there are projects within MSR India that suit that candidate, . we come up with extensive rankings and finally researchers go through that pool of applicants that we have ranked and figure out who the best candidate for their project is.

Sridhar Vedantham: So, Shruti, if I were to ask you what the main takeaway is for you from the Research Fellow program, what would you say?

Shruti Rijhwani: Well, there are a whole bunch, I really gained so much from this program. As I already said, you know it shaped my view of research and really gave me a good footing to start my journey through Graduate School with my Master’s and now my PhD. I would really recommend this program to anyone who is looking to get a little bit of experience in computer science research and if anyone wants to kind of explore this as a career option, I think this is a really good first step. Particularly because there are researchers in MSR India working on such broad fields in computer science. And just attending talks and having conversations about their research can really expand your view on computer science research and give you a really strong place to set your own career off in whatever field you’re interested in. So I think that’s my main takeaway. I really enjoy the community feeling in MSR India among the RFs as well as with the senior researchers in the lab. And just the fact that it helped me so much in Graduate School. I definitely recommend the program to anyone who has an inkling of interest in computer science research.

Sridhar Vedantham: That’s fantastic, and I know for a fact that the people who are involved with the Research Fellow program and in general people at the lab really feel proud when, you know, people like you go through the program and then join some of the top organizations in the world.  

So Vivek, how do students apply for an RF position? And what’s the kind of exposure that they can look at? I mean, I know we’ve been speaking to Shruti, who you know, kind of was focusing on one particular area, but in a more general sense, can you talk a bit about what students can expect when they come for a Research Fellow position here?

Vivek Seshadri: Sridhar, if you look at computer science as a field in the past few years, it has become increasingly interdisciplinary. And Microsoft Research India being a flat organization, enables researchers for experts in different areas to sort of collaborate with each other and, you know, solve big problems that require such collaboration. In fact, in the past few years, if you have noticed we have had major impact in multiple problems where researchers from machine learning, systems, compilers and HCI have come together and offered big solutions to important problems. So in that sense, given where computer science is today and where it is going, I think MSR India is sort of an ideal place for new budding researchers to come and gain experience in what that interdisciplinary research looks like.

In fact, as an RF, if I mean just like Shruti mentioned, even though they are working on a specific problem in a specific area, they will often not only have the opportunity to listen to people working in other areas through talks and lectures and whatnot, they may also have to actively collaborate with other Research Fellows working in these other areas. So that experience of interdisciplinary research is sort of essential for any budding scientists in computer science. So I think that’s one of the main reasons why I feel MSR India is a great place for students to come and test what research in computer science looks like today.

Sridhar Vedantham: And what’s the process for somebody applying for the RF position?

Vivek Seshadri: Yeah, like I mentioned, the application process is very similar to, you know, a grad school application. You know we do require students to upload their resume and a statement of purpose.  There’s a portal. People can just search for the Research Fellow program at MSR. In the Internet it will lead them directly to our page from where they can apply. There’s a single deadline this time on January 15^th (2021), which is right after you know grad school applications. So if students are applying to grad school, they already have all the material to apply for the Research Fellow program.

Sridhar Vedantham: Great, so what I’ll do is add links to the Research Fellow program. Will attach it to the podcast so that the listeners can actually go and check it out and see what the Research Fellow program offers. 

And Shruti and Vivek. Thank you so much for your time. I know that we’re doing this in multiple time zones, so thank you so much for being accommodative. And it’s been a great conversation.

Vivek Seshadri: Likewise, Sridhar, thanks a lot.

Shruti Rijhwani: Thanks, I had a great time.

Sridhar Vedantham: Thank you, stay safe, everybody.

[Music Ends]Opens in a new tab

The post Podcast: Helping young students build a career in research through the MSR India Research Fellow program. With Shruti Rijhwani and Dr. Vivek Seshadri appeared first on Microsoft Research.

Episode 006 | October 20, 2020

At Microsoft Research India, research focused on societal impact is typically a very interdisciplinary exercise that pulls together social scientists, technology experts and designers. But how does one evaluate or validate the actual impact of research in the real world? Today, we talk to Tanuja Ganu who manages the Societal Impact through Cloud and AI (or SCAI) group in MSR India. SCAI focuses on deploying research findings at scale in the real world to validate them, often working with a wide variety of collaborators including academia, social enterprises and startups.

Tanuja is a Research SDE Manager at Microsoft Research, India. She is currently part of MSR’s new center for Societal impact through Cloud and Artificial Intelligence (SCAI).

Prior to joining MSR, she was a Co-Founder and CTO of DataGlen Technologies, a B2B startup that focuses on AI for renewable energy and sustainability technologies. Prior to this, she has worked as Research Engineer at IBM Research, India.

Tanuja has completed MS in Computer Science (Machine Learning) from Indian Institute of Science (IISc, Bangalore). She has been recognized as MIT Technology Review’s Innovator Under 35 (MIT TR 35) in 2014 and IEEE Bangalore Woman Technologist of the Year in 2018. Her work was covered by top technical media (IEEE Spectrum, MIT Technology Review, CISCO Women Rock IT TV series, IBM Research blog and Innovation 26X26: 26 innovations by 26 IBM women).

Click here (opens in new tab) to go to the SCAI website.

Related

• Microsoft Research India Podcast: More podcasts from MSR India (opens in new tab)
• iTunes (opens in new tab): Subscribe and listen to new podcasts on iTunes
• Android (opens in new tab)
• RSS Feed (opens in new tab)
• Spotify (opens in new tab)
• Google Podcasts (opens in new tab)
• Email (opens in new tab)

Transcript

Tanuja Ganu: As the name suggests, SCAI, that is Societal Impact through Cloud and Artificial Intelligence, it is an incubation platform within MSR for us to ideate on such research ideas, work with our collaborators like academia, NGOs, social enterprises, startups, and to test or validate our hypothesis through very well defined real world deployments. At SCAI, it’s an interdisciplinary team of social scientists, computer scientists, software engineers, designers, and program managers from the lab who come together for creating, nurturing and evaluating our research ideas through real world deployments and validations.

[Music]

Sridhar: Welcome to the Microsoft Research India podcast, where we explore cutting-edge research that’s impacting technology and society. I’m your host, Sridhar Vedantham.

[Music]

At Microsoft Research India, research focused on societal impact is typically a very interdisciplinary exercise that pulls together social scientists, technology experts and designers. But how does one evaluate or validate the actual impact of research in the real world? Today, we talk to Tanuja Ganu who manages the Societal Impact through Cloud and AI (or SCAI) group in MSR India. SCAI focuses on deploying research findings at scale in the real world to validate them, often working with a wide variety of collaborators including academia, social enterprises and startups.

Tanuja has been recognized as one of MIT Technology Review’s Innovators Under 35 (MIT TR 35) in 2014 and by IEEE Bangalore as a Woman Technologist of the Year in 2018, and her work has been covered by top technical media. 

[Music]

Sridhar Vedantham: Tanuja, welcome to the podcast. I’m really looking forward to this particular edition of what we do here. Because, I know that you manage SCAI and it’s quite an intriguing part of the lab. Now before we get into that, tell us a little bit about yourself.

Tanuja Ganu: First of all, thanks Sridhar for having me on the podcast today. And uh, yes, uh, I’m not a full-time researcher, but I’m engineer by training and I have done my Master’s in Computer Science. Over the last decade or so, my work is primarily at the intersection of research and engineering, and it’s on the applied research side. So throughout my experience and journey, working at research labs and start up, I’m very much interested in taking a research idea through the entire incubation phase to validate its applicability in real world problem settings.

Sridhar Vedantham: So, Tanuja, I know you manage this thing called SCAI within the lab and I think it’s a very interesting part of the lab. Talk to us a little bit about that, and especially expand upon what SCAI- the term SCAI- itself stands for, because I myself keep tripping up on it whenever I try to explain it.

Tanuja Ganu: Yes, Sridhar. So since the inception of our lab, the lab has been doing very interesting work in the societal impact space. Additionally, with the advances in artificial intelligence and cloud-based technologies in recent years there are increased opportunities to address some of these societal problems through technology and amplify its positive effect. So as the name suggests, SCAI, that is Societal Impact through Cloud and Artificial Intelligence, it is an incubation platform within MSR for us to ideate on such research ideas, work with our collaborators like academia, NGOs, social enterprises, startups, and to test or validate our hypothesis through very well defined real world deployments. Also our location in India allows us to witness and carefully analyze various socio-economic challenges. So the solutions that we ideate are inspired by Indian settings and in many cases equally applicable to different parts of the world.

Sridhar Vedantham: Interesting, so it sounds like there’s a fair amount of difference between the kind of work that SCAI does and between what the rest of the lab actually does in terms of research.

Tanuja Ganu: So at MSR India, where research work is mainly along three different axes, firstly advancing the state of the art in science and technology, second is inspiring the direction for technology advances, and the third important axis is building the technology for driving societal impact. So SCAI is primarily focused on social impact access and many of our projects also do have very strong academic and technological impact. At SCAI, it’s an interdisciplinary team of social scientists, computer scientists, software engineers, designers, and program managers from the lab who come together for creating, nurturing and evaluating our research ideas through real world deployments and validations. So that’s really the difference in terms of the other type of research that we do at lab and what we do at SCAI.

Sridhar Vedantham: So when you decide to take up a project or accept it under the SCAI umbrella, what do you actually look for?

Tanuja Ganu: Yeah, we look for a few things for defining a SCAI project. So firstly, it should address a significant real-world problem and should have a potential to scale. The second thing is the problem should offer interesting research challenges for our team. The next thing is whether we have credible partners or collaborators with domain expertise to deploy, evaluate and validate of our research. We also look for how we can define rigorous impact evaluation plan for a project. And lastly, we look for what are the feasible graduation paths for the project within two to three years of time horizon.

Sridhar Vedantham: What do you mean by graduation?

Tanuja Ganu: So, um, there are different ways in which a particular project can complete its successful execution at SCAI center, and that’s what we’re really terming it as a graduation. And there could be really different types of graduation path depending upon each type of project.

Sridhar Vedantham: OK, let’s talk a little bit about some of the projects that you are currently doing under the SCAI umbrella. Because to me from what you’ve said so far, it sounds like there’s probably going to be a fairly wide spread of types of projects, and quite a large variety in the type of things that you’re doing there.

Tanuja Ganu: So yes, Sridhar, that’s very true. We are working on a very diverse set of projects right now. And, um, so to give a flavor of our work, I would discuss about two or three projects briefly. The first project is called HAMS that is Harnessing Automobiles for Safety. We all know that road safety is very critical issue and according to World Bank Report globally there are 1.25 million road traffic deaths every year. In India there is one death every 4 minutes. That happens due to road accidents. So, to understand and address this very critical issue of road safety, HAMS project was initiated by our team at MSR, including Venkat Padmanabhan, Akshay Nambi and Satish Sangameswaran. HAMS provides a low cost solution which is being evaluated for automated driver license testing. HAMS includes a smartphone with its associated sensors like camera, accelerometer, etc that is fitted inside a car. It monitors a driver and the driving environment and using AI and edge intelligence, it provides effective feedback on the safe driving practices. So at present, HAMS has been deployed at regional transport office in Dehradun, India for conducting dozens of driver license tests a day, and the feedback from this deployment is very encouraging, since it provides transparency and objectivity to the overall license testing and evaluation process. The second project is in the domain of natural language processing, called Interactive Neural Machine Translation, which was initiated by Kalika Bali and Monojit Choudhury in our NLP team. So, when we look at this problem, there are 7000 plus spoken languages worldwide, and for many many use cases, we often need to translate content from one language to another. Though there are many commercial machine translation tools available today, those are applicable to a very small subset of languages, say 100, which have sufficiently large digital datasets available to train machine learning models. So to aid human translation process as well as for creating digital data set for many low resource or underserved languages, we combine innovations from deep learning and human computer interactions and bring human in the loop. So when we talk about INMT, the initial translation model is bootstrapped using small data set that is available for these languages. And then INMT provides quick suggestions for human translators while they are performing translations. And over time it also helps in creating larger digital datasets which would help in increasing accuracy of translation for such underserved languages. So in INMT we’re currently working with three external collaborators called Pratham Books, Translators Without Borders and CGNet Swara to evaluate and enhance INMT. So just to give few examples, Pratham Books is a nonprofit publisher who would like to translate children story books in as many languages as possible. Translators Without Borders is a nonprofit who is working in the areas of crisis relief, health and education, and they would like to evaluate IN&MT for an Ethiopian language called Tigrinya. Our other collaborator CGNet Swara is working with INMT for collecting Hindi Gondi data set. And just to give you one last flavor of one more project…

Sridhar Vedantham: So I’m sorry, sorry to interrupt, but I was curious, how do you actually go around selecting or identifying partners and collaborators for these projects?

Tanuja Ganu: So when we started thinking about SCAI projects last year, we had initiated a call for proposals where we invited external partners and collaborators to submit various ideas that they do have and the process that they have in addressing some of the societal impact projects and we Interestingly received a huge pool of applications through this call for proposals we received more than 150 applications through that. And through careful evaluation process, as we discussed earlier, we finally selected a few projects to start under SCAI umbrella.

Sridhar Vedantham: OK, so I’m sorry I interrupted. You wanted to…you were speaking about another project.

Tanuja Ganu: Yeah, so just to give one more flavor of the project that we are currently doing which is addressing another important issue of air pollution. So air pollution is another major concern worldwide, with an estimated 7 million deaths every year, and when we look in India, it’s even more serious problem since 13 out of 20 most polluted cities in the world are in India. So to solve the air pollution problem, it is important to get correct monitoring of pollution levels, their timely and seasonal patterns in more granular manner, that is, from multiple locations inside the city. So apart from sophisticated and expensive air pollution monitoring stations feature already available, there are low-cost air pollution sensors which are being deployed for this purpose. But the local sensors tend to drift or develop fault overtime and the entire monitoring and analytical insights are dependent on reliability and correctness of this IoT data. So taking these things into account, we are now evaluating our research project called Dependable IoT for these low-cost air pollution sensors. Dependable IoT helps in automatically identifying and validating the drift or malfunction in the sensors and notifies for recalibration or replacement. So currently we are working with a few startups in this space to evaluate dependable IoT Technology and as the project name such as this is not only limited to air pollution sensing, but this technology is applicable for many other use cases for IoT sensing- in agriculture, food technology or in healthcare. So I guess this gives you a view on some of the diverse projects that now we are doing and working on at present in SCAI.

Sridhar Vedantham: Yeah, so this Dependable IoT thing sounds quite interesting. So correct me if I’m wrong, but essentially, what we’re saying is that we’re trying to figure out ways in which we can ensure that the data that we’re receiving in order to extract information from it and make decisions– we’re actually trying to figure out our trying to make sure that the data itself is solid.

Tanuja Ganu: Absolutely. That’s correct, Sridhar, and it’s like monitoring the monitor, right? So while we’re doing the IoT monitoring and sensing, we need to make sure that the technology that we’re putting in place is being monitored and it’s giving us reliable and correct data.

Sridhar Vedantham: Great. Now what’s also coming across to me throughout this conversation is that given the variety of projects and the variety of collaborators that you’re looking at in SCAI– would I be right in saying that the kind of people that you have in SCAI in addition to the researchers, obviously who are your internal collaborators, the people who are part of SCAI, are they a very diverse and varied set of people?

Tanuja Ganu: Yes, absolutely true, Sridhar. As we discussed earlier, SCAI’s an interdisciplinary team that consists of social scientists, CS researchers, solid software engineers and designers. And we also have a program called SCAI Fellows where fresh under graduates or the candidates who are already working in the industry can join on the specific SCAI project for a fixed time period and contribute towards the development of SCAI project. So particularly in SCAI, in addition to all these technical or academic skills, we’re also looking for people who have passion for societal impact and willingness to do the field work and deployment to scale a research idea.

Sridhar Vedantham: OK, and you know, you might at any point of time be working on say, four, five or six projects. Uh, what happens to these projects once they are completed?

Tanuja Ganu: Yeah, so I would say each project would have a different graduation plan. So whenever a project is complete from the SCAI perspective, we call it as a graduation plan where we would define how this project would then sustainably grow further internally or externally. And this graduation plan would be different for different projects depending upon the nature of the project. So for some of the projects, the graduation plan could be an independent entity that is spun off to take the journey of the project forward by scaling the initial idea to more people, more geographies, or for more use cases. A very good example of this type of graduation plan is a MSR project called 99 DOTS, where researchers like Bill Thies and others at Microsoft Research started this project to address medical adherence for tuberculosis. Over the years, this work has significantly grown and there is an independent entity spun off called Everwell to take the 99 DOTS journey forward. The other type of graduation plan can be putting up a work and technology in the open source wherein the external social enterprises, NGOs or our collaborators can build on top of it and take the solution forward at larger scale. The example of this is our work on interactive machine translation, where we have open sourced our initial work and various collaborators are now using, validating and building on top of this technology.

Sridhar Vedantham: OK, and does the work that you do in SCAI or say the validation that you’re looking for from research projects or the validation you’re looking at of research projects through SCAI– does that feed back further into the research itself, or does it kind of just stay at SCAI?

Tanuja Ganu: So, it has two or I would say it would have multiple pathways, but primarily the first thing is certainly the work that we’re doing is validating certain research hypothesis that we do have. So some of the output or outcome of these SCAI projects is feeding back into the research areas and validating or invalidating the hypothesis to say how the technology is helping to solve a particular research problem or not. But also if the intervention is successful, it would be useful for external collaborators internally, externally for them to take up this idea forward and utilize the technology that we have built at SCAI to taking it to larger scale.

Sridhar Vedantham: OK, so once again coming back to the fact that the projects that you do are of such different nature, how do you actually even define success metrics for SCAI projects?

Tanuja Ganu: Yeah, this is a very interesting question, Sridhar. So, the whole purpose of SCAI, as the name suggests, is about bringing social impact through technology innovations. So there is no one fixed set of metrics that would be applicable for each and every project at SCAI. But our success metrics for these projects are geared towards validating whether technological interventions can support the people and ecosystem and actually help address a specific problem or not. And if it does help solve the problem, then how can we amplify the positive effect using technology? So those are really the metrics that we’re defining on each of the project depending upon nature of the project.

Sridhar Vedantham: So Tanuja, thank you so much for your time. This has been a great conversation and all the best for going forward in SCAI.

Tanuja Ganu: Thank you, Sridhar, for having me here and I really enjoyed discussing these projects and ideas with you. Thank you.

[Music Ends]Opens in a new tab

The post Podcast: Evaluating and validating research that aspires to societal impact in real world scenarios. With Tanuja Ganu appeared first on Microsoft Research.