{"id":287486,"date":"2013-10-02T09:00:38","date_gmt":"2013-10-02T16:00:38","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=287486"},"modified":"2016-09-12T09:41:16","modified_gmt":"2016-09-12T16:41:16","slug":"collaboration-expertise-produce-enhanced-sensing-xbox-one","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/collaboration-expertise-produce-enhanced-sensing-xbox-one\/","title":{"rendered":"Collaboration, Expertise Produce Enhanced Sensing in Xbox One"},"content":{"rendered":"

Cyrus Bamji had encountered a challenge. Luckily for him, Microsoft Research had just the solution.<\/p>\n

Bamji, Microsoft partner hardware architect for Microsoft\u2019s Silicon Valley-based Architecture and Silicon Management group, and members of his team were trying to incorporate a time-of-flight camera into Xbox One<\/a>, the successor to the wildly popular Xbox 360<\/a>.<\/p>\n

A time-of-flight camera emits light signals and then measures how long it takes them to return. That needs to be accurate to 1\/10,000,000,000 of a second\u2014remember, we\u2019re talking the speed of light here. With such measurements, the camera is able to differentiate light reflecting from objects in a room and the surrounding environment. That provides an accurate depth estimation that enables the shape of those objects to be computed.<\/p>\n

That speed-of-light capability would be a major advancement for the Kinect sensor portion of Xbox One, being released to 13 launch markets next month. The new Kinect, a key differentiator for Xbox One against its competition, needed to capture a larger field of view with greater accuracy and higher resolution. An infrared sensor will enable object identification requiring little to no light, and improved hand-pose recognition, giving gamers and more casual users the ability to control the console with their hands.<\/p>\n

But Cyrus Bamji had a challenge. The sensor was great, but it also left those working on it eager to do even more with it.<\/p>\n

\u201cWhen we take a relatively new technology, such as time-of-flight, and put it into a commercial product, there are a whole bunch of things that happen,\u201d he says. \u201cThere are things that we didn\u2019t know how important they were until the product was made. For example, we know theoretically that motion blur in time of flight is a big problem, but just how important is only discoverable when you\u2019re building a product with it and that product needs to deliver an excellent experience.\u201d<\/p>\n

Accurate depth measurement in diverse scenes with the new camera\u2019s high resolution and a wider field of view also pose user-experience issues, making it difficult to keep small objects, such as a finger, from fading into the background, for instance. While those features delivered more versatile device performance, they also created issues of their own in real-life scenarios, such as the need for accurate depth measurement in diverse, high-resolution scenes. That, as well as improving the wider field of view and the motion blur, required clean data\u2014quickly. Xbox One had to be ready for the 2013 holiday season.<\/p>\n

\u201cWe knew our time was limited,\u201d Bamji recalls. \u201cBut we also had the advantage of being able to tap into Microsoft Research\u2019s deep reservoir of technical expertise to get expert advice and help solve the various problems we encountered with new, cutting-edge solutions.\u201d<\/p>\n

\"Eyal

Eyal Krupka<\/p><\/div>\n

Eyal Krupka, principal applied researcher with Microsoft Research Advanced Technology Lab, was up to the challenge.<\/p>\n

\u201cI was in Redmond last summer, working on hand-pose-recognition research, also for Xbox One,\u201d Krupka says, \u201cand Mark Plagge, a principal program-manager lead on the Xbox One team, approached me about the ongoing work in solving some issues with the camera. They had made huge progress, but the progress had not come quickly enough, and there were not any clear solutions yet. He asked me to check to see if I could help.\u201d<\/p>\n

Travis Perry, a senior system architect lead with the Architecture and Silicon Management team, says that things took off from there.<\/p>\n

\u201cEyal and I had many meetings discussing the various tradeoffs of the sensor and discussing the problem statements,\u201d says Perry, who worked with Krupka on algorithm and parameter optimization. \u201cOur team supported Eyal and his team with data and software for the existing depth calculations, and Eyal and I worked together to achieve better edge and motion-blur performance.\u201d<\/p>\n

That sort of engagement and teamwork was key to the project\u2019s success.<\/p>\n

\u201cIt wasn\u2019t like consultant mode, where we asked something and the researchers gave us an opinion,\u201d Bamji says. \u201cThey really took charge of the project. They did all the tests. They built a whole infrastructure of software to deliver us a complete solution. Essentially, they took charge. We\u2019re really grateful for that.\u201d<\/p>\n

Krupka\u2014with a few Microsoft Research colleagues making contributions of their own\u2014worked well with their Xbox partners. Their combined domain knowledge meshed well.<\/p>\n

\u201cThe reason Eyal and I were successful,\u201d Perry says, \u201cwas because of his extensive knowledge of computer vision, signal processing, and machine learning, along with my knowledge of time-of-flight technology and the system tradeoffs, allowing us to make the right decisions in a short amount of time and keep to the tight schedule.\u201d<\/p>\n

Krupka also worked diligently to gain a deep understanding of how the system worked.<\/p>\n

\u201cEyal was curious from the beginning to understand how the technology works, the underlying mechanism, and the various noise models that go into the system,\u201d says Sunil Acharya, senior director of engineering for the Architecture and Silicon Management team. \u201cHis team was helping the software team with face- and hand-recognition algorithms when he found out about the time-of-flight challenges we were facing. He jumped in and worked with us very closely, and his team started working on solutions that mapped directly into the product timeline.\u201d<\/p>\n

For Bamji, it was an a-ha moment.<\/p>\n

\u201cWe had researchers who understood that time was of the essence,\u201d he says. \u201cWe could ask them about a problem, and they would get on it and essentially come up with solutions that were technically challenging, but not in a vacuum. And they delivered solutions in a timeframe that was something that could be of use to us.<\/p>\n

\"Kinect

Kinect sensor for Xbox One<\/p><\/div>\n

\u201cThe success story is a rapid response and the solving of difficult problems.\u201d<\/p>\n

That is a concise summation of the value of Microsoft Research, a unique asset to Microsoft developers of devices and services. For Krupka, this is significant.<\/p>\n

\u201cThe research aspects of what we delivered for Xbox One did not start on the day we start working with the product team,\u201d he says. \u201cIt starts years before we learn about any specific project or problem. It is based on accumulating a wide range of research expertise through exploration on multiple research projects, accumulating engineering and research tools and practices\u2014including rapid research methods.<\/p>\n

\u201cThis is achieved by rotating cycles of working on long-term research problems, then switching to short-term research tasks. This is critical to the success. If we did only short-term, on-demand research, we couldn\u2019t have the critical assets when we work on the product\u2019s problems. If we worked only on long-term research, we would have had a harder time switching gears to deliver solutions on a product group\u2019s timeline.\u201d<\/p>\n

The analog nature of the time-of-flight data posed challenges to delivering such a solution.<\/p>\n

\u201cThe time-of-flight data coming out of our sensor is per pixel, per frame, and there is a lot more analog information,\u201d Acharya says. \u201cAnother issue was that the foreground objects close to the background objects would melt into the background\u2014again, due to the analog nature of how our sensor provides the depth data for pixels that land on edges.\u201d<\/p>\n

\u201cThis resulted in a lot of information, and to make it easier for foreground\/background extraction and scene segmentation, use by software and game developers, the requirement was to clean up this data simultaneously by adding software algorithms in the pipe, yet without incurring a performance hit. This was crucial. We started with various work streams and, in the end, settled on making optimization to the parameters in the system to overcome the issue.\u201d<\/p>\n

The collaborators wanted to deliver a clear separation of foreground and background even if the objects are close to each other. That, too, proved difficult. And then there was motion blur.<\/p>\n

\"Travis

Travis Perry (left) and Sunil Acharya<\/p><\/div>\n

\u201cMotion blur,\u201d Acharya explains, \u201cis a parameter that needs to be minimized and is not technology-specific. The time-of-flight camera uses global shutter, which has helped reduce motion blur significantly\u2014from 65 milliseconds in the original Kinect to fewer than 14 milliseconds now.\u201d<\/p>\n

Other challenges presented themselves. For one thing, processing time became an issue. In the academic literature about time-of-flight systems, processing time wasn\u2019t an issue. In the laboratory environment, the technology worked fine. But Xbox One needs to process a whopping 6.5 million pixels per second. And only a small part of Xbox One\u2019s computing power could be harnessed for this task. The lion\u2019s share is reserved, understandably, for essentials such as gaming, skeleton tracking, face recognition, and audio.<\/p>\n

\u201cYou need to do very, very light computation for each pixel,\u201d Krupka says, \u201cand this is one of the things that made the problem challenging and different from the typical approach in the academic literature in this field.\u201d<\/p>\n

And then there was that tight timeframe for delivery. Benchmark numbers in multiple domains had to be attained to make the Xbox One perform at the highest level.<\/p>\n

\u201cThe fact that we needed to hit all the benchmark performance numbers in all these multiple, different domains was a challenge,\u201d Bamji confirms. \u201cMany of these things were heavy-duty theoretical stuff. That\u2019s why we reached out to Microsoft Research and asked for their help.\u201d<\/p>\n

Remarkably, it all came together, and that means that while entertainment lovers worldwide will soon find themselves delighted by the Xbox One experience, so, too, will those eager to develop for the platform. Reducing that edge-data noise makes the data developer-ready, and being able to segment clearly between the foreground and the background solves a complex computational problem. The data is clean, and it can be absorbed more easily by game developers.<\/p>\n

Another fascinating feature of the Kinect sensing device in Xbox One stems from its infrared sensor, which can identify objects in a completely darkened room. It can recognize people and track bodies even without any light visible to the naked eye. It can identify a hand pose from four meters away, see the fingers of a child, and remember your identity even minus room illumination.<\/p>\n

The wider field of view makes it possible for more players to play an Xbox One game at the same time. With the new console, as many as six players can crowd into one scene. A tall adult can play with a small child without either being squeezed out of the picture. Users get a better experience if they\u2019re standing close by, farther away, or on the periphery of the room.<\/p>\n

\"Xbox

The Xbox One hardware system, with console, sensor, and controller<\/p><\/div>\n

And the improved hand-pose recognition enables users to interact with the Xbox One just by using their hands\u2014no controller necessary. Thanks to the infrared camera, hand activities can be identified at any illumination, or with none at all. Prior hand-pose solutions were able to deliver speed or accuracy, but not both. The hand-pose solution jointly devised by the Xbox team and Microsoft Research can do both.<\/p>\n

Based on his experience with both hand-pose and facial recognition, Krupka was able to find solutions that take into account system-level concerns.<\/p>\n

\u201cWe ourselves were customers of the camera improvements, as part of our research for accurate hand pose,\u201d he says. This helped us to determine the necessary tradeoffs. This also is a good example of Microsoft Research interdisciplinary work to deliver system-level optimization, not just optimization of separate parts.\u201d<\/p>\n

The aforementioned Xbox One enhancements resulted from an intensive collaborative effort by a number of individuals and teams. On the hardware side, Mirko Schmidt worked closely with Krupka and his team, and Arrigo Benedetti did likewise during investigations of the time-of-flight camera conducted with Philip A. Chou<\/a> and Alex Acero of Microsoft Research Redmond<\/a>, along with Shahram Izadi<\/a> of Microsoft Research Cambridge<\/a>. Krupka says he found those investigations seminal to his crash-course study of how the camera functions, enabling him to gain a quick understanding of which were the more promising directions\u2014and which were not.<\/p>\n

Two other contributors came from Perry\u2019s team. Abdelrehim Ahmed worked with Simon Baker<\/a> and Rick Szeliski<\/a> of Microsoft Research Redmond on the camera\u2019s filter, which has proved instrumental in detecting erroneous pixels caused by edges and motion. And Vishali Mogallapu supported all the above efforts.<\/p>\n

Other Microsoft Research contributions to the Xbox One include Fang Wen<\/a> and her team at Microsoft Research Asia<\/a>, who contributed their state-of-the-art technologies for high-accuracy face recognition. Ivan Tashev<\/a> of Microsoft Research Redmond brought his audio expertise, demonstrated during the development of the Xbox 360, to optimize the sound experience enabled by the Xbox One. Ranveer Chandra<\/a> contributed a wireless-controller protocol. And, from Krupka\u2019s team, Ido Leichter<\/a> wrote valuable simulations of the camera and environment, and Daniel Freedman<\/a> provided innovative solutions to improve the robustness of depth images.<\/p>\n

\u201cWe found people eager to help us solve our problems,\u201d says Bamji of his collaboration with Microsoft Research. \u201cThe researchers were very interested in making an impact by working on products that were going to market.\u201d<\/p>\n

Krupka found the collaboration challenging\u2014and, ultimately, rewarding.<\/p>\n

\u201cFor me,\u201d he says, \u201cit was the most intensive three months of work in my whole career. It seemed like I was working 24 hours a day, and the response from the Silicon Valley Xbox team was similar.<\/p>\n

\u201cWe had a clear deadline, and we understood what was at stake. We understood that this was very critical for the project. We were aware of it from the beginning. Everyone was on board, and now, we\u2019re excited to see how Xbox One customers respond later this fall. It was a great example of teamwork that, hopefully, translates into a \u2018just works\u2019 experience that delights Xbox One customers and developers.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":"

Cyrus Bamji had encountered a challenge. Luckily for him, Microsoft Research had just the solution. Bamji, Microsoft partner hardware architect for Microsoft\u2019s Silicon Valley-based Architecture and Silicon Management group, and members of his team were trying to incorporate a time-of-flight camera into Xbox One, the successor to the wildly popular Xbox 360. A time-of-flight camera […]<\/p>\n","protected":false},"author":39507,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"footnotes":""},"categories":[194466,194471,194474],"tags":[212210,212216,212213,187348,212204,212219,212207,212201,204677],"research-area":[13561,13562,13563],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-287486","post","type-post","status-publish","format-standard","hentry","category-algorithms","category-computer-vision","category-data-visulalization","tag-depth-estimation","tag-hand-pose-recognition","tag-infrared-sensor","tag-kinect","tag-light-signals","tag-motion-blur","tag-speed-of-light","tag-time-of-flight-camera","tag-xbox-one","msr-research-area-algorithms","msr-research-area-computer-vision","msr-research-area-data-platform-analytics","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[199560,199561,199565],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[144899],"related-projects":[],"related-events":[],"related-researchers":[],"msr_type":"Post","byline":"","formattedDate":"October 2, 2013","formattedExcerpt":"Cyrus Bamji had encountered a challenge. Luckily for him, Microsoft Research had just the solution. Bamji, Microsoft partner hardware architect for Microsoft\u2019s Silicon Valley-based Architecture and Silicon Management group, and members of his team were trying to incorporate a time-of-flight camera into Xbox One, the…","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/287486"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/39507"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=287486"}],"version-history":[{"count":3,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/287486\/revisions"}],"predecessor-version":[{"id":291695,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/287486\/revisions\/291695"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=287486"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=287486"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=287486"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=287486"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=287486"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=287486"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=287486"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=287486"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=287486"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=287486"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=287486"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}