{"id":687813,"date":"2020-08-28T00:01:00","date_gmt":"2020-08-28T07:01:00","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=687813"},"modified":"2020-10-06T16:11:00","modified_gmt":"2020-10-06T23:11:00","slug":"microsoft-hololens-2-improved-research-mode-to-facilitate-computer-vision-research","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/microsoft-hololens-2-improved-research-mode-to-facilitate-computer-vision-research\/","title":{"rendered":"Microsoft HoloLens 2: Improved Research Mode to facilitate computer vision research"},"content":{"rendered":"\n<figure class=\"wp-block-image alignwide size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"683\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/Hololens-2-lifestyle-imagery-male-1024x683.png\" alt=\"Lifestyle image of male wearing a Hololens 2 device \" class=\"wp-image-687897\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/Hololens-2-lifestyle-imagery-male-1024x683.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/Hololens-2-lifestyle-imagery-male-300x200.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/Hololens-2-lifestyle-imagery-male-768x512.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/Hololens-2-lifestyle-imagery-male.png 1038w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>Since its launch in November 2019, Microsoft HoloLens 2 has helped enterprises in manufacturing, construction, healthcare, and retail onboard employees more quickly, complete tasks faster, and greatly reduce errors and waste. It sets the high-water mark for intelligent edge devices by leveraging a multitude of sensors and a dedicated ASIC (Application-Specific Integrated Circuit) to allow multiple real-time computer vision workloads to run continuously. In Research Mode, HoloLens 2 is also a potent computer vision research device. (Note: Research Mode is available today to Windows Insiders and soon in an upcoming release of Windows 10 for HoloLens .)<\/p>\n\n\n\n<p>Compared to the previous edition, Research Mode for HoloLens 2 has the following main advantages:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>In addition to sensors exposed in HoloLens 1 Research Mode, we now also provide IMU sensor access (these include an accelerometer, gyroscope, and magnetometer).<\/li><li>HoloLens 2 provides new capabilities that can be used in conjunction with Research Mode. Specifically, articulated hand-tracking and eye-tracking which can be accessed through APIs while using research mode, allowing for a richer set of experiments.<\/li><\/ul>\n\n\n\n<p>With Research Mode, application code can not only access video and audio streams, but can also simultaneously leverage the results of built-in computer vision algorithms such as SLAM (simultaneous localization and mapping) to obtain the motion of the device as well as the spatial-mapping algorithms to obtain 3D meshes of the environment. These capabilities are made possible by several built-in image sensors that complement the color video camera normally accessible to applications.<\/p>\n\n\n\n\t<div class=\"border-bottom border-top border-gray-300 mt-5 mb-5 msr-promo text-center text-md-left alignwide\" data-bi-aN=\"promo\" data-bi-id=\"931956\">\n\t\t\n\n\t\t<p class=\"msr-promo__label text-gray-800 text-center text-uppercase\">\n\t\t<span class=\"px-4 bg-white display-inline-block font-weight-semibold small\">Spotlight: On-demand video<\/span>\n\t<\/p>\n\t\n\t<div class=\"row pt-3 pb-4 align-items-center\">\n\t\t\t\t\t\t<div class=\"msr-promo__media col-12 col-md-5\">\n\t\t\t\t<a class=\"bg-gray-300\" href=\"https:\/\/www.microsoft.com\/en-us\/research\/blog\/ai-explainer-foundation-models-and-the-next-era-of-ai\/\" aria-label=\"AI Explainer: Foundation models \u200band the next era of AI\" data-bi-cN=\"AI Explainer: Foundation models \u200band the next era of AI\" target=\"_blank\">\n\t\t\t\t\t<img decoding=\"async\" class=\"w-100 display-block\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2023\/03\/AIEx01_blog_hero_1400x788.png\" alt=\"a screenshot of a computer screen shot of a man\" \/>\n\t\t\t\t<\/a>\n\t\t\t<\/div>\n\t\t\t\n\t\t\t<div class=\"msr-promo__content p-3 px-5 col-12 col-md\">\n\n\t\t\t\t\t\t\t\t\t<h2 class=\"h4\">AI Explainer: Foundation models \u200band the next era of AI<\/h2>\n\t\t\t\t\n\t\t\t\t\t\t\t\t<p class=\"large\">Explore how the transformer architecture, larger models and more data, and in-context learning have helped advance AI from perception to creation.<\/p>\n\t\t\t\t\n\t\t\t\t\t\t\t\t<div class=\"wp-block-buttons justify-content-center justify-content-md-start\">\n\t\t\t\t\t<div class=\"wp-block-button\">\n\t\t\t\t\t\t<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/blog\/ai-explainer-foundation-models-and-the-next-era-of-ai\/\" class=\"btn btn-brand glyph-append glyph-append-chevron-right\" aria-label=\"Watch video\" data-bi-cN=\"AI Explainer: Foundation models \u200band the next era of AI\" target=\"_blank\">\n\t\t\t\t\t\t\tWatch video\t\t\t\t\t\t<\/a>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div><!--\/.msr-promo__content-->\n\t<\/div><!--\/.msr-promo__inner-wrap-->\n\t<\/div><!--\/.msr-promo-->\n\t\n\n\n<p>HoloLens 2 has four grayscale head-tracking cameras and a depth camera to sense its environment and perform articulated hand tracking. It also has two additional infrared cameras and accompanying LEDs that are used for eye tracking and iris recognition. As shown in Figure 1, two of the grayscale cameras are configured as a stereo rig, capturing the area in front of the device so that the absolute depth of tracked visual features can be determined through triangulation. Meanwhile, the two additional grayscale cameras help provide a wider field of view to keep track of features. These synchronized global-shutter cameras are significantly more sensitive to light than the color camera and can be used to capture images at a rate of up to 30 frames per second (FPS).<\/p>\n\n\n\n<figure class=\"wp-block-image alignwide size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"576\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/1400x788_HoloLens_NoLogo-1-1024x576.png\" alt=\"\" class=\"wp-image-687909\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/1400x788_HoloLens_NoLogo-1-1024x576.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/1400x788_HoloLens_NoLogo-1-300x169.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/1400x788_HoloLens_NoLogo-1-768x432.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/1400x788_HoloLens_NoLogo-1-1536x864.png 1536w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/1400x788_HoloLens_NoLogo-1-2048x1152.png 2048w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/1400x788_HoloLens_NoLogo-1-1066x600.png 1066w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/1400x788_HoloLens_NoLogo-1-655x368.png 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/1400x788_HoloLens_NoLogo-1-343x193.png 343w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/1400x788_HoloLens_NoLogo-1-640x360.png 640w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/1400x788_HoloLens_NoLogo-1-960x540.png 960w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/1400x788_HoloLens_NoLogo-1-1280x720.png 1280w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/1400x788_HoloLens_NoLogo-1-1920x1080.png 1920w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><figcaption>Figure 1: Hololens 2 Research Mode enables access to the gray-scale, depth camera and IMU sensors on device. This complements the color camera normally available to applications. <\/figcaption><\/figure>\n\n\n\n<p>The depth camera uses active infrared (IR) illumination to determine depth through phase-based time-of-flight. The camera can operate in two modes. The first mode enables high-framerate (45 FPS) near-depth sensing, commonly used for hand tracking, while the other mode is used for lower-framerate (1-5 FPS) far-depth sensing, currently used by spatial mapping. As hands only need to be supported up to 1 meter from the device, HoloLens 2 saves power by reducing the number of illuminations, which results in the depth wrapping around beyond one meter . For example, something at 1.3 meters will appear at 0.3 meters in HoloLens 2 in this case. In addition to depth, this camera also delivers actively illuminated IR images (in both modes) that can be valuable in their own right because they are illuminated from the HoloLens and reasonably unaffected by ambient light. Azure Kinect uses the same sensor package, but with slightly different depth modes.<\/p>\n\n\n\n<p>With the newest Windows Insider release of Windows 10 for HoloLens, researchers now have the option to enable Research Mode on their HoloLens devices to gain access to all of these external facing raw image sensors streams. Research Mode for HoloLens 2 also provides researchers with access to the accelerometer, gyroscope, and magnetometer readings. To protect users\u2019 privacy, raw eye-tracking camera images are not available through Research Mode. Researchers can access eye-gaze direction through existing APIs.<\/p>\n\n\n\n<p>For other sensor streams, researchers can also still use the results of the built-in computer vision algorithms and can now also choose to use the raw sensor data for their own algorithms.<\/p>\n\n\n\n<p>The sensors\u2019 streams can either be processed or stored on device or transferred wirelessly to another PC or to the cloud for more computationally demanding tasks. This opens a wide range of new computer vision applications for HoloLens 2. HoloLens 2 is particularly well suited as a platform for egocentric vision research as it can be used to analyze the world from the perspective of a user wearing the device. For these applications, HoloLens devices\u2019 abilities to visualize results of the algorithms in the 3D world in front of the user can be a key advantage. HoloLens sensing capabilities can also be very valuable for robotics where these can, for example, enable a robot to navigate its environment.<\/p>\n\n\n\n<p>These new HoloLens capabilities will be demonstrated at <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/event\/hololens-2-kinect-for-azure-as-tools-for-computer-vision-research-tutorial-eccv-2020\/\">a tutorial <span class=\"sr-only\"> (opens in new tab)<\/span><\/a>on August 28th, 2020, at the <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/eccv2020.eu\/\">European Conference on Computer Vision (ECCV).<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> An initial set of sample apps is being made available showcasing <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/github.com\/microsoft\/HoloLens2ForCV\">computer vision use cases on GitHub<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, and you can check out the <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/docs.microsoft.com\/en-us\/windows\/mixed-reality\/research-mode\">Research Mode documentation<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> for further technical details.<\/p>\n\n\n\n<div class=\"annotations \" data-bi-aN=\"citation\">\n\t<ul class=\"annotations__list card depth-16 bg-body p-4 \">\n\t\t<li class=\"annotations__list-item\">\n\t\t\t\t\t\t<span class=\"annotations__type d-block text-uppercase font-weight-semibold text-neutral-300 small\">TUTORIAL<\/span>\n\t\t\t<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/event\/hololens-2-kinect-for-azure-as-tools-for-computer-vision-research-tutorial-eccv-2020\/\" target=\"_self\" class=\"annotations__link font-weight-semibold text-decoration-none\" data-bi-type=\"annotated-link\" aria-label=\"Microsoft HoloLens 2 and Kinect for Azure as tools for computer vision research \" data-bi-aN=\"citation\" data-bi-cN=\"Microsoft HoloLens 2 and Kinect for Azure as tools for computer vision research \">\n\t\t\t\tMicrosoft HoloLens 2 and Kinect for Azure as tools for computer vision research &nbsp;<span class=\"glyph-append glyph-append-chevron-right glyph-append-xsmall\"><\/span>\n\t\t\t<\/a>\n\t\t\t\t\t<\/li>\n\t<\/ul>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Since its launch in November 2019, Microsoft HoloLens 2 has helped enterprises in manufacturing, construction, healthcare, and retail onboard employees more quickly, complete tasks faster, and greatly reduce errors and waste. It sets the high-water mark for intelligent edge devices by leveraging a multitude of sensors and a dedicated ASIC (Application-Specific Integrated Circuit) to allow [&hellip;]<\/p>\n","protected":false},"author":38838,"featured_media":687909,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"footnotes":""},"categories":[1],"tags":[],"research-area":[13556],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-687813","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-research-blog","msr-research-area-artificial-intelligence","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[602418],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[],"related-projects":[708502],"related-events":[637092],"related-researchers":[{"type":"user_nicename","value":"Marc Pollefeys","user_id":36191,"display_name":"Marc Pollefeys","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/mapoll\/\" aria-label=\"Visit the profile page for Marc Pollefeys\">Marc Pollefeys<\/a>","is_active":false,"last_first":"Pollefeys, Marc","people_section":0,"alias":"mapoll"}],"msr_type":"Post","featured_image_thumbnail":"<img width=\"960\" height=\"540\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/1400x788_HoloLens_NoLogo-1-960x540.png\" class=\"img-object-cover\" alt=\"\" decoding=\"async\" loading=\"lazy\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/1400x788_HoloLens_NoLogo-1-960x540.png 960w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/1400x788_HoloLens_NoLogo-1-300x169.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/1400x788_HoloLens_NoLogo-1-1024x576.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/1400x788_HoloLens_NoLogo-1-768x432.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/1400x788_HoloLens_NoLogo-1-1536x864.png 1536w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/1400x788_HoloLens_NoLogo-1-2048x1152.png 2048w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/1400x788_HoloLens_NoLogo-1-1066x600.png 1066w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/1400x788_HoloLens_NoLogo-1-655x368.png 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/1400x788_HoloLens_NoLogo-1-343x193.png 343w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/1400x788_HoloLens_NoLogo-1-640x360.png 640w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/1400x788_HoloLens_NoLogo-1-1280x720.png 1280w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/08\/1400x788_HoloLens_NoLogo-1-1920x1080.png 1920w\" sizes=\"(max-width: 960px) 100vw, 960px\" \/>","byline":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/mapoll\/\" title=\"Go to researcher profile for Marc Pollefeys\" aria-label=\"Go to researcher profile for Marc Pollefeys\" data-bi-type=\"byline author\" data-bi-cN=\"Marc Pollefeys\">Marc Pollefeys<\/a>","formattedDate":"August 28, 2020","formattedExcerpt":"Since its launch in November 2019, Microsoft HoloLens 2 has helped enterprises in manufacturing, construction, healthcare, and retail onboard employees more quickly, complete tasks faster, and greatly reduce errors and waste. It sets the high-water mark for intelligent edge devices by leveraging a multitude of&hellip;","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/687813"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/38838"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=687813"}],"version-history":[{"count":12,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/687813\/revisions"}],"predecessor-version":[{"id":696465,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/687813\/revisions\/696465"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/687909"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=687813"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=687813"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=687813"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=687813"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=687813"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=687813"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=687813"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=687813"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=687813"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=687813"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=687813"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}