Putting the “human” in human computer interaction with Haiyan Zhang

Published

haiyan zhang standing in front of a wall

Haiyan Zhang, Innovation Director

Episode 62, February 6, 2019

Haiyan Zhang (opens in new tab) is a designer, technologist and maker of things (really cool technical things) who currently holds the unusual title of Innovation Director at the Microsoft Research lab in Cambridge, England (opens in new tab). There, she applies her unusual skillset to a wide range of unusual solutions to real-life problems, many of which draw on novel applications of gaming technology in serious areas like healthcare.

On today’s podcast, Haiyan talks about her unique “brain hack” approach to the human-centered design process, and discusses a wide range of projects, from the connected play experience of Zanzibar (opens in new tab), to Fizzyo, which turns laborious breathing exercises for children with cystic fibrosis into a video game, to Project Emma (opens in new tab), an application of haptic vibration technology that, somewhat curiously, offsets the effects of tremors caused by Parkinson’s disease.

Related:

Episode Transcript

Haiyan Zhang: We started out going very broad, and looking at lots of different solutions out there, not necessarily just for tremor, but across the spectrum to address different symptoms of Parkinson’s disease. And this is actually really part of this whole design thinking methodology which is to look at analogous experiences. So, taking your core problem and then looking at adjacent spaces where there might be solutions in a completely different area that can inform upon the challenge that you are tackling.

Host: You’re listening to the Microsoft Research Podcast, a show that brings you closer to the cutting-edge of technology research and the scientists behind it. I’m your host, Gretchen Huizinga.

Host: Haiyan Zhang is a designer, technologist and maker of things (really cool technical things) who currently holds the unusual title of Innovation Director at the Microsoft Research lab in Cambridge, England. There, she applies her unusual skillset to a wide range of unusual solutions to real-life problems, many of which draw on novel applications of gaming technology in serious areas like healthcare.

On today’s podcast, Haiyan talks about her unique “brain hack” approach to the human-centered design process, and discusses a wide range of projects, from the connected play experience of Zanzibar, to Fizzyo, which turns laborious breathing exercises for children with cystic fibrosis into a video game, to Project Emma, an application of haptic vibration technology that, somewhat curiously, offsets the effects of tremors caused by Parkinson’s disease. That and much more on this episode of the Microsoft Research Podcast.

(music plays)

Host: Haiyan Zhang, welcome to the podcast.

Haiyan Zhang: Hi, thanks Gretchen. Great to be here.

Host: You are the Innovation Director at MSR Cambridge in England, which is a super interesting title. What is an Innovation Director? What does an Innovation Director do? What gets an Innovation Director up in the morning?

Haiyan Zhang: I guess it is quite an unusual title. It’s a kind of a bespoke role, I would say, because of my quite unusual background, I guess. Part of what I do is look at how technology can be applied in real use cases in the world to create business impact, within Microsoft and outside of Microsoft, and to make those connections between our deeply technical research with applied product groups across the company.

Host: So, is this a job that existed at MSR in Cambridge or did you arrive with this unique set of talents and skills and background and ability, and bring the job with you?

Haiyan Zhang: I would say it’s something I brought with me and it’s evolving over time. (laughs)

Host: Well, unpack that a little bit. How has it evolved since you began? When did you begin?

Haiyan Zhang: So, I actually joined Microsoft about five and a half years ago and I actually initially joined as part of the Xbox organization, running an innovation team in Xbox in London and looking at new play experiences for kids, for teens, that were completely outside of the box. And then from that, I transitioned into Microsoft Research. And part of my team also continued on that research in terms of creating completely new technology experiences around entertainment. And more recently, I’m working across the lab with various projects to see how we can connect our sort of fundamental computer science work better with products across Microsoft in terms of Azure cloud infrastructure, in terms of Xbox and gaming, in terms of Office and productivity.

Host: You’ve been in high-tech for nearly twenty years and you’ve worked in engineering and user experience and research… R&D, hardware, service design, etc., and even out in the “blue-sky envisioning space.” So, that brings a lot to the party in the form of one person. (laughter) Quite frankly, I’m impressed. How has your experience in each, or all, of these areas informed how you approached the research you do today?

Haiyan Zhang: Well thanks, Gretchen. I’m really… I’m quite honored to be on the podcast actually because I’m so impressed with all the researchers that you’ve been interviewing across all the MSR labs. So, I would say that, in the research work that I do, I bring a very human-centered lens to looking at technology. So I undertake a full, human-centered design process starting from talking to people, getting empathy with people, trying to extract insight from what people really need and then going deeply into the technical research to develop prototypes, technology ideas to support those needs, and then deploying those prototypes in the field to understand how that can be improved and how we can evolve our technology thinking.

Host: Let’s talk about design thinking, then, for a minute. I don’t know if you’d call it discrete from computational thinking or any other kind of thinking, but it seems to be a buzz phrase right now. So, as a self-described designer, technologist and maker of things, how would you define design thinking?

Haiyan Zhang: So, I would say that design thinking is not separate from computational thinking, it’s a layer above. It’s just an approach to problem-solving, and it’s basically a tool kit that allows you to utilize different methods to really gain an understanding of people’s needs, to gain an understanding of insight into how people’s lives can be improved through technology, and then tools around prototyping and evaluating those prototypes. So, I would say that it is not, in itself, a scientific method, but it can be used to improve and augment your existing practice.

Host: Let’s get specific now and talk about some of those projects that you’ve been working on, starting with Project Zanzibar. What was the inspiration behind this project? How did you bring it to life and how does it embody your idea of connected play experiences that you’ve talked about?

Haiyan Zhang: I think there is a rich history in computer science of tangible user interfaces. You know, some of the early work at Xerox Park even or at the MIT Media Lab around how we can create these seamless interactions between people, between their physical environment and between a digital universe. And I think the approach we had to Zanzibar was that the most fruitful area for exploration in tangible user interfaces would be to enable kids to play and learn though physicality. Through interacting with physical objects that were augmented with virtual information, because we’re really trying to tap into this idea of multi-modal learning and learning through play. So, just coming from this initial approach, we dive very deeply into how would we invent a completely new technology platform to enable people to very seamlessly manipulate objects in a natural way using gestures, and then bring about some new digital experiences layered on top of that, that were games or education scenarios and then sort of bringing those together in terms of really fundamental technology invention, but also applications that could demonstrate what that technology could do.

Host: Right. Well, and it’s too bad that this is an audio-only experience here on the podcast because there’s a really cool overview of this project on the Microsoft Research website and it’s a very visual, artifact-based approach to playing with computers.

Haiyan Zhang: Yeah, yeah. And I encourage everyone to visit the project page and take a look at some of the videos and our prototypes that we have published.

Host: Right. So, what was the thinking behind tying in the artifact and the digital?

Haiyan Zhang: You know, there’s this rich history of research with physical objects and we’ve proven out that physical/digital interaction is a great way forward in terms of novel interactions between people and computing. But the pragmatics of these systems have not been ideal. You know, if you have to be sat at your desk and there has to be an overhead camera, usually a lot of research projects involve this or there’s occlusion in terms of where your hand can be and where the physical objects can be because the cameras won’t be able to track it. So, what we set out to do was think about well, how would you design a technology platform that overcomes a lot of these barriers to these platforms so that we can then be freed up to think about those scenarios, but we can also empower other researchers who are doing research in this space to think about those scenarios. So, our research group, we had to this idea of leveraging NFC, but leveraging it in terms of an NFC antenna array so that we could track objects in a 2-D space. And then the additional novelty was also layering that with a capacitive multi-touch layer so that we could track both the objects in terms of the physical IDs of the objects on top of this surface area. The capacitor’s multi-touch would enhance that tracking that the NFC provided, but also, we could track hand gestures, both in terms of multi-touch gestures on top of the surface and also some hover gestures just above the surface as well.

(music plays)

Host: Let’s talk a bit about another really cool project that you’re working on. I know Cambridge, your lab, is deeply, and maybe even uniquely, invested in researching and developing technologies to improve healthcare, and you have a couple projects going on in this area. One of them, Project Fizzyo. I’ll spell it. It’s F (as in Frank)-i-z-z-y-o. Tell us about this project. How did it come about? What’s the technology behind it and how does it work?

Haiyan Zhang: So, Fizzyo really started as a collaboration with the BBC and we were inspired by one family. The mom, Vicky, she has four kids and two of her boys have cystic fibrosis, they have a genetic condition where their internal organs are constantly secreting mucous. And so, every day, twice a day, the boys have to do this laborious breathing exercise to expel the mucous from their lungs, and it involves breathing into these plastic apparatus. And they basically apply pressure to your breath so that when you breathe, it creates and oscillating effect in your lungs and escalates the mucous and then it culminates in you coughing and trying to cough out the mucous from your lungs. They’re usually plastic devices, where as you blow, the air kind of enters a chamber and there might be some sort of mechanism that oscillates the air like a ball-bearing that bounces up and down and so they are very low-fi, so there’s no digital aspect to these devices. And you can imagine, these kids, they are having to do these exercises from a very early age, from as early as they can remember, twice a day for 30 minutes, for an hour at a time. It’s really intensive and it can be, you know, if not painful, at least really uncomfortable to do. And I actually tried to do this once and I felt really light-headed. I actually couldn’t do one session of it. And also, the kids, they want to be outside playing with their friends. You know, they don’t want to be stuck indoors doing this all the time. And there is no thread from doing the exercise and feeling an improvement because the activity is about maintenance, so you are trying to maintain your health because if you don’t clear the mucous from your lungs, infection can set in and that means going to the hospital, that means getting antibiotics. And so, it’s a very challenging thing for Vicky, their mom, to be jostling them, be harassing them to do this all the time. And she said that her role has really changed with her kids and that she’s no longer a mom, she’s sort of nagging them all the time. And so, we visited with the family to really understand their plight. And she asked, you know, can we create a piece of technology that can help us in getting the kids to do this kind of physio, the treatment is a type of physio. And so, we actually came up with this idea together where she said, you know, the boys really love to play video games so, what if we could create a way for the boys to be playing a video game as they are undertaking this exercise. So, we started this process of prototyping and developing a digital attachment, a sensor, that attaches to all these various different physio devices. And as the patient is expelling, is breathing out, the sensor actually senses the breath and transmits that digital signal to a tablet and we can translate that signal into controls for a video game. And we’re also able to upload that to the cloud, to do further analysis on that medical treatment.

Host: Wow. How is it working?

Haiyan Zhang: We started this project about two and a half years ago. It’s been a long process, but a really fruitful and rewarding one. So, we started out with just some early prototypes, just using off-the-shelf electronics to get the breath sensor working just right. We added a single button, because we realized if you were just using the breath to play video games, it’s actually really challenging. And then, within the team, our industrial designer, Greg Saul, designed the physical attachment. We developed our own sensor board and we had it manufactured along with the product design. And we partnered with University College London, their physiotherapy department, and the Great Ormond Street Hospital in London where they’ve deployed over a hundred of these units with kids across the country to do a long-term trial. So actually, when we first met with the University College London physiotherapy department, I mean, this is a department that they’ve spent their entire careers working with kids in this domain. And they had never had any contact with the computer science department. This was not a digital research area. When they first met us, and they saw, on the computer screen, someone breathing out, and a graph showing that breath, the peak of that breath, one of the heads of the department that we were working with, she started to cry because she said that in her entire career, she had never seen physio data visualized in this way. It was just incredible for her.

Host: Wow.

Haiyan Zhang: And so, we decided to partner, and they’ve been amazing because, through this journey, they’ve gone to meet people in the computer science department, they initiated masters’ degrees incorporating data science and digital understanding. They just hired their first data scientist in order to leverage the platform that we’ve built to do further analysis to improve the health of these kids. And they said that even though this kind of exercise has been around for decades, no one has actually done a definitive, long-term study to track the efficacy of this kind of exercise to health, to outcomes. You know, because I think past studies have really relied on keeping paper diaries, answering questionnaires, but no one has done that digital study, which is what the power of Internet of Things can really bring you, which is tracking in the background in a very precise way.

Host: Talk about the role of machine learning. How do any of the new methodologies in computer science like machine learning methods and techniques play into this?

Haiyan Zhang: You know, what’s really interesting with machine learning is the availability of data. And, you know, we understand that what has driven this AI revolution is now the availability of large data sets to actually be able to develop new ML algorithms and models. And in many cases, especially in healthcare, there is the lack of data. So, I think throughout different areas of computer science research, there’s a real need to kind of connect the dots and actually develop IoT solutions that can start at the beginning and capture the data, because it’s only through cleverly capturing valid data, that we can then do the machine learning in the back end once we’ve done the data collection. And so, I think the Fizzyo project is a really good proof point of that in that we started out with IoT in order to gather the information that track the health exercises. And we just sort of deployed in the UK, so as we’re collected this data, we’re now able to look at that and start to do some predictions around long-term health. So, you know, some of the questions that physiotherapy researchers are trying to answer, if kids are very adherent to this kind of exercise, if they are doing what they are being told, they are doing this this twice a day for the duration that they are supposed to be doing it, does that mean, in six months’ time or a year’s time, their number of days in hospital is going to be reduced? Does it actually impact how much time they are spending being ill? If we see a trailing-off of this exercise, does that mean that we’ll see an increase in infection rates? So, with the data that we’re collecting, we’re now working with a different part of Microsoft, they’re called the Microsoft Commercial Software Engineering team, who are actively delving into projects around AI for good and they are going to be working with UCL to do some of this clustering and developing models around health prediction. So, clustering the patients into different cohorts to understand if there is prediction factors around how they are doing the exercises and how much time they are going to be spending in hospital in the years to come.

Host: Well, it almost would be hard for me to get more excited about something than what you just described in Project Fizzyo, but there is another project to talk about which is Project Emma. This is so cool it’s even been featured on a documentary series there in the UK called The Big Life Fix. And it didn’t just start with a specific idea, but with a specific person. Tell us the story of Emma.

Haiyan Zhang: Yes! So, again, Project Emma started with a single person, with Emma Lawton, who, when she was 28 years old, she was diagnosed with early onset Parkinson’s disease. And, it had been five years since her diagnosis and some of her symptoms had progressed quite quickly and one of them was an active tremor. So, her tremor would get worse as she started to write or draw. And this really affected how she went about her day-to-day work because she was a creative director, a graphic designer and day-to-day she would be in client meetings, talking with people and trying to sketch out what they meant in terms of the ideas that they had. And she would not be able to do that. And when I first met with her, she would sit with a colleague and her colleague would actually draw on her behalf. So, she really was looking for some kind of technology intervention to help her. And, we started out going very broad, and looking at lots of different solutions out there, not necessarily just for tremor, but across the spectrum to address different symptoms of Parkinson’s disease. And this is actually really part of this whole design thinking methodology which is to look at analogous experiences. So, taking your core problem and then looking at adjacent spaces where there might be solutions in a completely different area that can inform upon the challenge that you are tackling. So, we looked at lots of different solutions for other kinds of symptoms and of course, there was a lot of desk research. It was reading research papers that had been published over the decades that looked at tremors specifically. So, I think the two aspects that really influenced our thinking, one was around going to visit with a local charity called Parkinson’s UK and we were asking them to show us their catalogue of widgets and devices that they sold to Parkinson’s patients that helped them in their every day. And on the table, there was a digital metronome. So, you know, when you’re playing the piano you see musicians, they have this ticking metronome. And I asked, you know, so why is there a metronome on the table? And the lady said, well, for some Parkinson’s patients, they have a symptom called freezing of gait and this is where when you are walking along, your legs suddenly freeze, and you lose control of your legs. And so, sometimes people find that if they take out this metronome and they turn it on and it makes this rhythmic ticking sound, it somehow distracts their brain into being able to walk again, which is really kind of odd. There’s been a little bit of literature around this. In the literature it’s called queuing, it’s a queuing effect, but it doesn’t apply to tremor. But, for me, it sort of signaled an interesting brain hack, and signaled kind of underlying what might be going on in your brain when you have Parkinson’s disease. At the same time, there had been a number of papers around using vibration on the muscles to try to ameliorate tremor, to try to address it, to various effect. And not specifically looking at Parkinson’s but looking at other kinds of tremor diseases like central tremor, dystonia. And so, we developed a hypothesis and in order to test out the hypothesis, we developed a prototype which was a wearable device for the wrist that had a number of vibrating motors on it. So, it would apply vibration to the wrist in a rhythmic fashion in order to somehow circumvent the mechanism that was causing the tremor. And of course, we had a number of other hypotheses, too. This was not the only hypothesis. We had other devices that worked in a completely different way that was more about mechanically stopping the tremor, mechanically countering the tremor. And this device actually worked really well. So, we were surprised, but very, very happy, and so this is the direction that we took in order to further develop this product.

Host: Right. So, drilling in, I do want to mention that there is a video on this, on the website as well. It’s a video that made me cry. I think it made you cry, and it made Emma cry. We’re all just puddles of tears, because it’s so fantastic. And so, this kind of circles back to research writ large, and experimenting with ideas that may not necessarily be super, what we would call high-tech, maybe they are kind of low-fi, you know, a vibration tool that can keep you from shaking. So, how did it play out? How did you prototype this? Give us a little overview of your process.

Haiyan Zhang: For us, it was a very simple prototyping exercise. We took some off-the-shelf coin cell motors and developed, basically, a haptic type bracelet that we then had an app that you could program the haptics on the bracelet. And that’s what we sort of experimented with. So, just research from the haptics area of computer science research which is really about a mechanism for sort of using in VR or sensing something about the digital world, now applied to this medical domain.

(music plays)

Host: You have a diverse slate of projects going on at any given time and your teams are really diverse. So, I want you to talk, specifically, about the composition of skills and expertise that are required to bring some of these really fascinating research projects to life, and ultimately to market. Who is on your team and what do they bring to the party?

Haiyan Zhang: Well, I think there’s just something really unique about Microsoft Research and Microsoft Research Cambridge, in particular, we have such a broad portfolio of projects, but also expertise in the different computer science fields, that we can sort of pull together these multidisciplinary teams to go after a single topic. So, within our lab we have social scientists doing user research, gaining real insight into how people behave, how people think about various technologies. We have designers that are exploring user interfaces, exploring products to bring these ideas to life. We have, you know, computer vision specialists. We have machine learning specialists. We have natural language processing people, systems researchers, and securities researchers and, obviously, healthcare researchers. So, it’s that broad outlook that I think can really push forward in terms of technology innovation and really emphasizing the applications for people, for improving society as a whole.

Host: I ask all my guests some form of the question is there anything that keeps you up at night. And I know that many people, mainly parents, are worried that their kids are too engaged with screens or not spending enough time in real life and so on. What would you say to them, and is there anything that keeps you up at night about sort of the broader swath of what you are working on?

Haiyan Zhang: You know, on the topic of screen time, obviously it’s something that we really wrestled with Zanzibar research specifically which is thinking about how you could interact with physical objects instead of a digital screen, and also bringing that kind of bigger interaction surface between family and between friends so they could interact together. You know, at the same time, I would say that culture is constantly changing and how we live our lives is constantly changing. We’ve only seen the internet be really embedded in our lives in the last, I’d say, twenty years, fifteen years, twenty years. When I think we were younger, we had television and there were no computers and so, I say culture is constantly evolving. How we’re growing, how we’re living is constantly evolving. It’s important for parents to evaluate this changing landscape of technology and to figure out what is the best thing to do with their kids. And maybe you don’t have to rely on how you grew up, but to kind of evaluate that our kids are getting the right kind of social interaction, getting the right amount of parental support and quality time with their family. I think that’s what is important, but to accept that how we’re growing is changing.

Host: What about the idea of the internet of things and privacy when we’re talking about toys and kids?

Haiyan Zhang: Mmm, yeah, it is something we really have to watch out for, and um you know, we’ve seen some bad examples of the toy industry jumping ahead too far and enabling toys to be connected 24/7 and conversing with kids and what does that really mean? I’ve seen some really great research out of the MIT Media Lab where there was a researcher really looking at how kids are conversing with AI, with different AI agents and their mental model of these AI agents. So, I think that’s a really great piece of research to look at, but also maybe to expand upon. As a research community, if we’re thinking about kids, to understand that how kids are interacting with AI is going to be more commonplace, and rather than trying to avoid it, to really tackle it head-on and see how we can improve the principles around designing AI, how we can inform companies in the market out there of what is the ethical approach to doing this so that kids really understand what AI is as they are growing up with it.

Host: We’re coming up on an event at Microsoft Research called I Chose STEM and it’s all about encouraging women to… well, choose STEM! As an area of study or a career.

Haiyan Zhang: Yeah.

Host: So, tell us the story of how you chose it? What got you interested in a career in high-tech in general, and maybe even high-tech research specifically? Who were your influences?

Haiyan Zhang: I have a I guess slightly unique background in that I was born in China and at the time it was very kind of Communist education that I had when I was growing up. And my family moved to Australia when I was 8 years old. And I was always very technical and very nerdy. But I never thought about technology as a career. I actually wanted to study law when I was in high school. And computing was just something where I was sort of, you know, it was kind of fun, but I never thought about it as a career. And I’d say in the last sort of year of high school, I decided to switch and do computer science and I realized that I was actually really good at computer science. I guess what led me to choose STEM is just the – I think the fun and creativity you can have with programming. You know, I would always come up with my own little creative exercises to write on the computer. It wasn’t the rote exercises, it was the ability to kind of be creative with this technical tool that really got me excited. I think at the same time, I love this huge effort within our industry to really focus on getting more women, more girls into technology, into STEM education, and we really want to increase representation, increase sort of equal representation. At the same time, I think I found it, at times, to be, you know, challenging to be the only woman in the room. You know, when I was in computer science, sometimes I’d be, you know, one of three women in the lecture theater or something. I think we need to adopt this kind of pioneer mindset so that we can go into these new areas, go into a room where you’re the only person, where you’re unique in that room and you have something to contribute and don’t be afraid to speak up. I think that’s a really important mindset and skill for anybody to have.

Host: No interview would be complete if I didn’t ask my guest to predict the future. No pressure, Haiyan. Seriously though, you are living on the cutting edge of technology research which is what this podcast is all about. And so what advice or encouragement – you’ve just kind of given some – would you give to any of our listeners across the board who might be interested or inspired by what you are doing? Who is a good fit for the research you do?

Haiyan Zhang: My advice would be, especially in the research domain, to develop that deep research expertise, but to keep a holistic outlook. I think the research landscape is changing in that we are going to be working in more multidisciplinary teams, working across departments. You know, sometimes it’s the healthcare department, the physiotherapy department, with the computer science department. It’s through the connection of these disparate fields that I think we’re going to see dramatic impact from technology. And I think for researchers to have that holistic outlook, to visit other departments, to understand what are the challenges beyond their own group, I think is really, really important. And develop collaboration skills and techniques.

Host: Haiyan Zhang, it’s been a delight. Thanks for joining us today.

Haiyan Zhang: Thanks so much, Gretchen. It’s been a real pleasure, thank you.

Continue reading

See all podcasts

Research Areas

Research Groups

Related projects

Related labs