{"id":885738,"date":"2022-10-24T04:30:00","date_gmt":"2022-10-24T11:30:00","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=885738"},"modified":"2022-11-01T09:03:29","modified_gmt":"2022-11-01T16:03:29","slug":"eccv-2022-highlights-advancing-the-foundations-of-mixed-reality","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/eccv-2022-highlights-advancing-the-foundations-of-mixed-reality\/","title":{"rendered":"ECCV 2022 highlights: Advancing the foundations of mixed reality"},"content":{"rendered":"<aside id=accordion-f47c437f-679d-45e5-94de-89ffb8e2145a class=\"msr-table-of-contents-block accordion mb-5 pb-0\" data-bi-aN=\"table-of-contents\">\n\t<button class=\"btn btn-collapse bg-gray-100 mb-0 display-flex justify-content-between\" type=\"button\" data-mount=\"collapse\" data-target=\"#accordion-collapse-f47c437f-679d-45e5-94de-89ffb8e2145a\" aria-expanded=\"true\" aria-controls=\"accordion-collapse-f47c437f-679d-45e5-94de-89ffb8e2145a\">\n\t\t<span class=\"msr-table-of-contents-block__label subtitle\">In this article<\/span>\n\t\t<span class=\"msr-table-of-contents-block__current mr-4 text-gray-600 font-weight-normal\" aria-hidden=\"true\"><\/span>\n\t<\/button>\n\t<div id=\"accordion-collapse-f47c437f-679d-45e5-94de-89ffb8e2145a\" class=\"msr-table-of-contents-block__collapse-wrapper collapse show\" data-parent=\"#accordion-f47c437f-679d-45e5-94de-89ffb8e2145a\">\n\t\t<div class=\"accordion-body bg-gray-100 border-top pt-4\">\n\t\t\t<ol class=\"msr-table-of-contents-block__list\">\n\t\t\t\t\t\t\t\t\t<li class=\"msr-table-of-contents-block__list-item\">\n\t\t\t\t\t\t<a href=\"#3d-face-reconstruction-with-dense-landmarks\" class=\"msr-table-of-contents-block__list-item-link\">3D face reconstruction with dense landmarks<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t<li class=\"msr-table-of-contents-block__list-item\">\n\t\t\t\t\t\t<a href=\"#lamar-benchmarking-localization-and-mapping-for-augmented-reality\" class=\"msr-table-of-contents-block__list-item-link\">LaMAR: Benchmarking localization and mapping for augmented reality<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t<\/ul>\n\t\t<\/div>\n\t<\/div>\n\t<span class=\"msr-table-of-contents-block__progress-bar\"><\/span>\n<\/aside>\n\n\n\n<div style=\"height:10px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"2560\" height=\"1441\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/1400x788_ECCV_Blog_hero_v2-scaled.jpg\" alt=\"This image contains two panels. The panel on the left is titled \u201c3D face reconstruction\u201d and depicts two faces of real people and corresponding face models developed using the dense landmarks method with 703 facial landmarks. The panel on the right is titled \u201cBenchmarking localization and mapping for AR\u201d and shows the interior of a building with paths\u2014or sequences\u2014where people had captured the environment using Microsoft HoloLens.\" class=\"wp-image-890526\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/1400x788_ECCV_Blog_hero_v2-scaled.jpg 2560w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/1400x788_ECCV_Blog_hero_v2-300x169.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/1400x788_ECCV_Blog_hero_v2-1024x576.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/1400x788_ECCV_Blog_hero_v2-768x432.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/1400x788_ECCV_Blog_hero_v2-1536x865.jpg 1536w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/1400x788_ECCV_Blog_hero_v2-2048x1153.jpg 2048w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/1400x788_ECCV_Blog_hero_v2-1066x600.jpg 1066w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/1400x788_ECCV_Blog_hero_v2-655x368.jpg 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/1400x788_ECCV_Blog_hero_v2-343x193.jpg 343w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/1400x788_ECCV_Blog_hero_v2-240x135.jpg 240w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/1400x788_ECCV_Blog_hero_v2-640x360.jpg 640w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/1400x788_ECCV_Blog_hero_v2-scaled-960x540.jpg 960w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/1400x788_ECCV_Blog_hero_v2-1280x720.jpg 1280w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/1400x788_ECCV_Blog_hero_v2-1920x1080.jpg 1920w\" sizes=\"(max-width: 2560px) 100vw, 2560px\" \/><\/figure>\n\n\n\n<p><em>By Microsoft Mixed Reality & AI Labs in <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/lab\/mixed-reality-ai-lab-cambridge\/\">Cambridge<\/a> and <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/lab\/mixed-reality-ai-zurich\/\">Zurich<\/a><\/em><\/p>\n\n\n\n<p>Computer vision is one of the most remarkable developments to emerge from the field of computer science. It\u2019s among the most rapidly growing areas in the technology landscape and has the potential to significantly impact the way people live and work. Advances at the intersection of machine learning (ML) and computer vision have been accelerating in recent years, leading to significant progress in numerous fields, including healthcare, robotics, the automotive industry, and augmented reality (AR). Microsoft is proud to be a prominent contributor to computer vision research.<\/p>\n\n\n\n<p>Microsoft researchers have long been collaborating with academics and experts in the field on numerous computer vision projects with the goal of expanding what\u2019s possible and helping people achieve more. One example is <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/blog\/peoplelens-using-ai-to-support-social-interaction-between-children-who-are-blind-and-their-peers\/\" target=\"_blank\" rel=\"noreferrer noopener\">PeopleLens<\/a>, a head-worn device that helps children who are blind or have low vision more easily interact in social situations by identifying people around them through spatialized audio. Another example is <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/blog\/swin-transformer-supports-3-billion-parameter-vision-models-that-can-train-with-higher-resolution-images-for-greater-task-applicability\/\" target=\"_blank\" rel=\"noreferrer noopener\">Swin Transformer<\/a>. This computer vision architecture attains high accuracy in object detection and provides an opportunity to unify computer vision and natural language processing (NLP) architectures\u2014increasing the capacity and adaptability of computer vision models.&nbsp;<\/p>\n\n\n\n<p>Microsoft Research is excited to share some of its newest work in this space at the <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/eccv2022.ecva.net\/\" target=\"_blank\" rel=\"noreferrer noopener\">European Conference on Computer Vision<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> (ECCV) 2022, with 45 accepted papers that will be presented through live presentations, tutorials, and poster sessions. This post highlights two of these papers, which showcase the latest research from Microsoft and its collaborators. One involves increasing the number of facial landmarks for more accurate 3D face reconstruction, achieving state-of-the-art results while decreasing the required compute power. The other introduces a dataset that takes advantage of the capabilities of AR devices for visual localization and mapping driven by real-world AR scenarios.&nbsp;<\/p>\n\n\n\n<h2 id=\"3d-face-reconstruction-with-dense-landmarks\">3D face reconstruction with dense landmarks<\/h2>\n\n\n\n<p>Facial landmarks are points that correspond across all faces, and they often play a key role in face analysis. Researchers frequently rely on them when performing basic computer vision tasks, such as estimating head position and identifying gaze direction and more generally the position in space of all the details of the face. Facial landmarks include such areas as the tip of the nose, corners of the eyes, and points along the jawline. Typically, public datasets that practitioners use to train ML models contain annotations for 68 facial landmarks. However, numerous aspects of human faces are not precisely represented by 68 landmarks alone, and additional methods are often needed to supplement landmark detection, adding complexity to the training workflow and increasing the required compute power.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"556\" height=\"186\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/ECCV2022-Highlights_Fig1.png\" alt=\"Image depicting two head models. The one on the left has the 68 commonly used facial landmarks identified along the jawline, eyebrows, eyes, nose, and mouth. The one on the right has 703 facial landmarks applied, covering the entire head in great detail. \" class=\"wp-image-885774\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/ECCV2022-Highlights_Fig1.png 556w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/ECCV2022-Highlights_Fig1-300x100.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/ECCV2022-Highlights_Fig1-240x80.png 240w\" sizes=\"(max-width: 556px) 100vw, 556px\" \/><figcaption>Figure 1: Compared with a typical sparse set of 68 facial landmarks (a), dense landmarks&nbsp;(b) cover the entire head in detail, including ears, eyes, and teeth. These dense&nbsp;landmarks are better at encoding facial identity and subtle expressions.&nbsp;<\/figcaption><\/figure>\n\n\n\n<div class=\"annotations \" data-bi-aN=\"margin-callout\">\n\t<ul class=\"annotations__list card depth-16 bg-body p-4 annotations__list--left\">\n\t\t<li class=\"annotations__list-item\">\n\t\t\t\t\t\t<span class=\"annotations__type d-block text-uppercase font-weight-semibold text-neutral-300 small\">GitHub<\/span>\n\t\t\t<a href=\"https:\/\/microsoft.github.io\/DenseLandmarks\/\" target=\"_self\" class=\"annotations__link font-weight-semibold text-decoration-none\" data-bi-type=\"annotated-link\" aria-label=\"Dense Landmarks\" data-bi-aN=\"margin-callout\" data-bi-cN=\"Dense Landmarks\">\n\t\t\t\tDense Landmarks&nbsp;<span class=\"glyph-append glyph-append-share glyph-append-xsmall\"><\/span>\n\t\t\t<\/a>\n\t\t\t\t\t<\/li>\n\t<\/ul>\n<\/div>\n\n\n\n<p>With the goal of achieving accurate 3D face reconstruction, we propose increasing the number of facial landmarks. In our paper \u201c<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/microsoft.github.io\/DenseLandmarks\/\" target=\"_blank\" rel=\"noreferrer noopener\">3D Face Reconstruction with Dense Landmarks<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>,\u201d we introduce a method to accurately predict 703 facial landmarks, more than 10 times as many as are commonly used, covering the entire face in great detail, including the eyes, ears, and teeth, as shown in Figure 1. We show that the increased number of landmarks are very precise when visible, and when they are occluded, for example, when someone lifts a coffee mug to their lips, we can estimate the location of these landmarks and what the part of the face looks like behind the object blocking it. We can use these landmarks to constrain a model-fitting problem to efficiently and precisely estimate all aspects of a face model, shown in the right-most column in Figure 2. This includes the head pose, eye gaze, as well as the identity of the person whose face is being reconstructed, for example, the thickness of the lips and the shape of the nose.<\/p>\n\n\n\n<p>This simple pipeline is comprised only of dense landmarks and continuous mathematical optimization, allowing for extreme compute efficiency and enabling the entire system to run at over 150 frames per second on a single core of a laptop.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1010\" height=\"416\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-2.png\" alt=\"Image with three rows and seven columns. Real faces are in the first column. Different baselines are shown in columns two through six and depict models based on the real faces with varying degrees of distinction in terms of facial identity. The column on the right shows face models with our proposed system of 703 landmarks and depicts a great amount of facial identity. \" class=\"wp-image-888663\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-2.png 1010w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-2-300x124.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-2-768x316.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-2-240x99.png 240w\" sizes=\"(max-width: 1010px) 100vw, 1010px\" \/><figcaption>Figure 2: In this image, the original faces are on the left, the baselines are in the second through sixth columns, and our results are in the right-most column. Compared with previous recent monocular 3D face reconstruction methods, ours better captures gaze, expressions like winks and sneers, and the subtleties of facial identity. In addition, our method can run in real time with only a minor loss of fidelity.&nbsp;<\/figcaption><\/figure>\n\n\n\n<h3 id=\"increasing-privacy-fairness-and-efficiency-with-synthetic-data\">Increasing privacy, fairness, and efficiency with synthetic data&nbsp;<\/h3>\n\n\n\n<p>In computer vision, and particularly the area of face reconstruction, there are understandable concerns about anonymity when training ML models because training data often comes from real people. Our proposed method significantly reduces privacy concerns, as it uses only <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/microsoft.github.io\/FaceSynthetics\/\" target=\"_blank\" rel=\"noreferrer noopener\">synthetic data<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> to train ML models, compared with methods that use images of real people as part of their training datasets. That said, when we built the synthetic data pipeline, we needed to preserve the privacy of the people whose data we used, and we took care to acquire the consent of those several hundred subjects. This contrasts with the feasibility of acquiring consent from thousands (or even tens of thousands) of subjects, which would have been necessary if we were using real data. <\/p>\n\n\n\n<p>It\u2019s especially challenging, if not impossible, to preserve the privacy of people appearing in \u201cfound images\u201d online, where the subject is often unknown. Using synthetic data helps us protect the privacy of data subjects and the rights of photographers and content creators. It\u2019s another tool we can use in our mission to build technology in an ethical and responsible manner. Additionally, because people\u2019s private information is not included in our dataset, if the ML model were to be attacked, only synthetic data would be subject to compromise.&nbsp;<\/p>\n\n\n\n<p>Synthetic data also provides an opportunity to address inclusivity and fairness. Largely because the distribution of the data is fully controlled, ML practitioners can manage the fairness of representation by including diverse samples in their datasets, and all the data needed to do this would be perfectly labeled. For further details on how we build the synthetics model and training data and our approach to capturing the diversity of the human population, please see our face analysis <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/fake-it-till-you-make-it-face-analysis-in-the-wild-using-synthetic-data-alone\/\" target=\"_blank\" rel=\"noreferrer noopener\">paper<\/a>.&nbsp;<\/p>\n\n\n\n<p>There are other advantages to using synthetic data to train ML models, as well. For example, these models require a lot of data, giving rise to numerous difficulties that practitioners must navigate to obtain this data, such as the logistics of finding the number of people required, scheduling time in a lab, and situating multiple cameras to capture the various angles of a person\u2019s face. These concerns are greatly reduced with synthetic data.&nbsp;<\/p>\n\n\n\n<p>In addition, because data doesn\u2019t need to be sourced from a real person, the iteration speed to improve the quality of the 3D face reconstruction is remarkably high, creating a robust workflow. And it isn\u2019t necessary to apply quality assurance (QA) processes on each labeled image when using synthetic data\u2014another cost- and time-saving benefit. Another advantage is the increase in accuracy, speed, and cost-effectiveness in labeling data. It would be nearly impossible to ask someone to consistently label 703 landmarks in a set of images.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"684\" height=\"223\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/ECCV2022-Highlights_Fig3.png\" alt=\"Image showing 30 faces created using synthetic data. Each face varies in terms of age, race, hair, expression, gaze, and other identifying characteristics. The faces are both female and male and are depicted in a different environments. \" class=\"wp-image-885768\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/ECCV2022-Highlights_Fig3.png 684w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/ECCV2022-Highlights_Fig3-300x98.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/ECCV2022-Highlights_Fig3-240x78.png 240w\" sizes=\"(max-width: 684px) 100vw, 684px\" \/><figcaption>Figure 3: Examples of the synthetic training data used in this project. Without the perfectly consistent annotations provided by synthetic data, dense landmark prediction would not be possible.&nbsp;<\/figcaption><\/figure>\n\n\n\n<p>Face analysis is a foundational piece for many ML systems, such as facial recognition and controlling avatars, and using a method that provides both accuracy and efficiency while also addressing privacy and fairness concerns pushes the boundaries of the state of the art. Up until now, there hasn\u2019t been much work, if any, on methods that can yield this level of quality with only synthetic data. The ability to achieve 3D face reconstruction using dense landmarks and synthetic data has the potential to truly transform what\u2019s possible with ML.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"3D Face Reconstruction with Dense Landmarks\" width=\"500\" height=\"281\" src=\"https:\/\/www.youtube-nocookie.com\/embed\/M8NNytHysVg?feature=oembed&rel=0\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe>\n<\/div><\/figure>\n\n\n\n<h3 id=\"acknowledgments\">Acknowledgments&nbsp;<\/h3>\n\n\n\n<p>This research was conducted by Erroll Wood, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/tabaltru\/\" target=\"_blank\" rel=\"noreferrer noopener\">Tadas Baltru\u0161aitis<\/a>, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/chewitt\/\" target=\"_blank\" rel=\"noreferrer noopener\">Charlie Hewitt<\/a>, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/matjoh\/\" target=\"_blank\" rel=\"noreferrer noopener\">Matthew Johnson<\/a>, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/jinshen\/\" target=\"_blank\" rel=\"noreferrer noopener\">Jingjing Shen<\/a>, Nikola Milosavljevi\u0107, Daniel Wilde, Stephan Garbin, Chirag Raman, Jamie Shotton, Toby Sharp, Ivan Stojiljkovi\u0107, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/tcashman\/\" target=\"_blank\" rel=\"noreferrer noopener\">Tom Cashman<\/a>, and Julien Valentin.&nbsp;<\/p>\n\n\n\n<h2 id=\"lamar-benchmarking-localization-and-mapping-for-augmented-reality\">LaMAR: Benchmarking localization and mapping for augmented reality<\/h2>\n\n\n\n<p>To unlock the full potential of augmented reality (AR), anyone using a mixed reality headset should be able to place virtual content in the physical world, share it with others, and expect it to remain in place over time. However, before they can augment digital content in the real world in the form of holograms, AR devices need to build a <em>digital<\/em> <em>map<\/em> of the physical 3D world. These devices then position, or <em>re-localize<\/em>, themselves with respect to this map, as illustrated in Figure 4, which allows them to retrieve previously placed holograms and show them to the user at a designated location. The computer vision foundations enabling these capabilities are called<em> mapping <\/em>and<em> <em>visual localization<\/em><\/em>.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-video alignwide\"><video autoplay controls loop muted src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/169_ECCV_animation.mp4\"><\/video><figcaption><center><span style=\"color: initial;\">Figure 4: The mapping and localization process.<\/span><\/center><\/figcaption><\/figure>\n\n\n\n<p>In general, research in visual localization focuses on single images, usually carefully selected views of famous attractions, shown on the left in Figure 5. However, this doesn\u2019t reflect real AR scenarios\u2014the combination of AR devices and applications\u2014and the opportunity they provide. AR devices can locally map the environment and provide spatially registered sequences rather than single images, as shown in the image on the right in Figure 5. These sequences can also include additional data, like inertial or radio signals from sensors, which are typically available on modern AR devices, such as <a href=\"https:\/\/www.microsoft.com\/en-us\/hololens\" target=\"_blank\" rel=\"noreferrer noopener\">Microsoft HoloLens 2<\/a>. Yet it\u2019s challenging to use such sequences for localization because they are typically just collected during normal device usage and not generally aimed at facilitating localization.<\/p>\n\n\n\n<figure class=\"wp-block-image alignwide size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1126\" height=\"375\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-5.png\" alt=\"This graphic contains two images: The image on the left shows six different views of a single point of interest. These are large and expansive views that don\u2019t contain any details of the environment. In contrast, the image on the right shows numerous paths, each representing a different AR sequence, in a single location. These sequences are densely sampled and do not focus on any specific spot.\" class=\"wp-image-886887\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-5.png 1126w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-5-300x100.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-5-1024x341.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-5-768x256.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-5-240x80.png 240w\" sizes=\"(max-width: 1126px) 100vw, 1126px\" \/><figcaption><center>Figure 5: On the left, single views are spread throughout large spaces and concentrate at points of interest. In contrast, the AR sequences on the right are densely sampled through the environment and do not focus on specific spots.&nbsp;While our dataset (on the right) captures a much smaller spatial area than landmark datasets (on the left), they contain a similar number of images. This illustrates how much more densely typical AR data is sampled within the scene. Each path represents a different sequence.<\/center><\/figcaption><\/figure>\n\n\n\n<p>To close this gap, we introduce a new <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/lamar.ethz.ch\/\" target=\"_blank\" rel=\"noreferrer noopener\">benchmark<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, the first to focus on this more realistic setting for AR, with the understanding that visual re-localization is a key element for compelling, shared, and persistent AR experiences. Given the spatial scale of the environment for typical AR scenarios, such as navigating an airport or inspecting a factory, we had to design a pipeline that could automatically compute the ground-truth camera positions of real AR sequences captured by a variety of readily available AR devices, such as the HoloLens or iPhone. By evaluating state-of-the-art methods on our benchmark, we offer novel insights on current research and provide avenues for future work in the field of visual localization and mapping for AR.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image alignwide size-full\"><a data-bi-bhvr=\"14\"  data-bi-cn=\"Various images showing paths in the different environments that were captured. They illustrate how we revisited localization and mapping in the context of AR by introducing the LaMAR dataset. \" href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-6.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1265\" height=\"668\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-6.png\" alt=\"Various images showing paths in the different environments that were captured. They illustrate how we revisited localization and mapping in the context of AR by introducing the LaMAR dataset. \" class=\"wp-image-886893\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-6.png 1265w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-6-300x158.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-6-1024x541.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-6-768x406.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-6-240x127.png 240w\" sizes=\"(max-width: 1265px) 100vw, 1265px\" \/><\/a><figcaption><center>Figure 6: We revisited localization and mapping in the context of AR by introducing LaMAR, a large-scale dataset captured using AR devices (iPhone, iPad, HoloLens 2) and laser scanners.<\/center><\/figcaption><\/figure>\n\n\n\n<div class=\"annotations \" data-bi-aN=\"margin-callout\">\n\t<ul class=\"annotations__list card depth-16 bg-body p-4 annotations__list--right\">\n\t\t<li class=\"annotations__list-item\">\n\t\t\t\t\t\t<span class=\"annotations__type d-block text-uppercase font-weight-semibold text-neutral-300 small\">Publication<\/span>\n\t\t\t<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/lamar-benchmarking-localization-and-mapping-for-augmented-reality\/\" target=\"_self\" class=\"annotations__link font-weight-semibold text-decoration-none\" data-bi-type=\"annotated-link\" aria-label=\"LaMAR: Benchmarking Localization and Mapping for Augmented Reality\" data-bi-aN=\"margin-callout\" data-bi-cN=\"LaMAR: Benchmarking Localization and Mapping for Augmented Reality\">\n\t\t\t\tLaMAR: Benchmarking Localization and Mapping for Augmented Reality&nbsp;<span class=\"glyph-append glyph-append-chevron-right glyph-append-xsmall\"><\/span>\n\t\t\t<\/a>\n\t\t\t\t\t<\/li>\n\t<\/ul>\n<\/div>\n\n\n\n<p>This research is a result of a two-year collaboration between the <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/lab\/mixed-reality-ai-zurich\/\" target=\"_blank\" rel=\"noreferrer noopener\">Microsoft Mixed Reality & AI Lab<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> in Zurich and <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/ethz.ch\/en.html\" target=\"_blank\" rel=\"noreferrer noopener\">ETH Zurich<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> (Swiss Federal Institute of Technology) and will be published at <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/eccv2022.ecva.net\/\" target=\"_blank\" rel=\"noreferrer noopener\">ECCV 2022<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> in the paper, &#8220;<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/lamar-benchmarking-localization-and-mapping-for-augmented-reality\/\" target=\"_blank\" rel=\"noreferrer noopener\">LaMAR: Benchmarking Localization and Mapping for Augmented Reality<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>.\u201d We will also be giving a tutorial called <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/lamar.ethz.ch\/tutorial-eccv2022\/\" target=\"_blank\" rel=\"noreferrer noopener\">Localization and Mapping for AR<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> at ECCV.<\/p>\n\n\n\n<h3 id=\"developing-a-large-scale-ar-dataset\">Developing a large-scale AR dataset&nbsp;<\/h3>\n\n\n\n<p>To enable the research community to address the specifics of mapping and visual localization in the context of AR, we collected multi-sensor data streams from modern AR devices. These sensor streams come with camera poses (the camera\u2019s position and orientation) from the on-device tracker at each instant. These data streams also contain images, depth measurements, samples from inertial measurement units (IMUs), and radio signals. Exploiting these can lead to more efficient algorithms. For example, radio signals such as Wi-Fi or Bluetooth can simplify image retrieval. Similarly, sequence localization can exploit the temporal aspect of sensor streams to provide a more spatial context, which can lead to more accurate estimates of camera poses. This typifies the realistic use case of a user launching an AR application and streaming sensorial data to localize the camera with respect to a previously built map, and it reflects how AR applications built on mixed reality cloud services, like <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/azure.microsoft.com\/en-us\/products\/spatial-anchors\/\" target=\"_blank\" rel=\"noreferrer noopener\">Azure Spatial Anchors<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, work.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-video alignwide\"><video autoplay controls loop muted src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/figure7.mp4\"><\/video><figcaption><center>Figure 7: Sample sequences from the dataset.<\/center><\/figcaption><\/figure>\n\n\n\n<p>The initial release of the LaMAR dataset contains more than 100 hours of recordings covering 45,000 square meters (484,000 square feet) captured over the course of two years using the head-mounted HoloLens 2 and handheld iPhone\/iPad devices. The data was captured at various indoor and outdoor locations (a historical building, a multi-story office building, and part of a city center) and represents typical AR scenarios. It includes changes in illumination and the movement of objects\u2014either slowly, such as the placement of a book on a desk, or more quickly, like anonymized people walking down a sidewalk.&nbsp;<\/p>\n\n\n\n<h3 id=\"automatically-aligning-ar-sequences-to-establish-ground-truth\">Automatically aligning AR sequences to establish ground truth&nbsp;<\/h3>\n\n\n\n<p>To estimate the ground-truth camera poses, we aligned the captured data with reference 3D models of the locations, as shown in Figure 8. These reference models were captured using NavVis M6 and VLX mapping systems, both equipped with laser scanners (lidars) that generate dense, textured, and highly accurate 3D models of the locations. To align the data, we developed a robust pipeline that does not require manual labeling or setting custom infrastructure, such as fiducial markers, and this enabled us to robustly handle crowd-sourced data from a variety of AR devices captured over extended periods.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image alignwide size-full\"><a data-bi-bhvr=\"14\"  data-bi-cn=\"Four images, each divided into four parts. The top-right and bottom-left parts show the reference 3D model of the environment rendered from an estimated ground-truth pose and overlayed with the image captured using the AR device. The top-left and bottom-right show pixel-level accuracy of the alignment. \" href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-8.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1503\" height=\"481\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-8.png\" alt=\"Four images, each divided into four parts. The top-right and bottom-left parts show the reference 3D model of the environment rendered from an estimated ground-truth pose and overlayed with the image captured using the AR device. The top-left and bottom-right show pixel-level accuracy of the alignment. \" class=\"wp-image-886896\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-8.png 1503w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-8-300x96.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-8-1024x328.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-8-768x246.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-8-240x77.png 240w\" sizes=\"(max-width: 1503px) 100vw, 1503px\" \/><\/a><figcaption><center>Figure 8: The top-right and bottom-left of each square show the reference 3D model of the environment rendered from estimated ground-truth poses and overlayed with the image captured using the AR device. The top-left and bottom-right of each square show pixel-level accuracy of the alignment.<\/center><\/figcaption><\/figure>\n\n\n\n<p>The actual alignment process is fully automatic and utilizes the on-device real-time tracker of AR devices, which provides camera poses in their local coordinate system. We aligned each captured sequence individually with the dense ground truth reference model, as illustrated in Figure 9. Once completed, all camera poses were refined jointly by optimizing the visual constraints within and across sequences.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image alignwide size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1527\" height=\"440\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-9.png\" alt=\"Diagram showing the sequence-to-scan alignment process in developing the ground-truth reference model.\" class=\"wp-image-886899\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-9.png 1527w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-9-300x86.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-9-1024x295.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-9-768x221.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-9-240x69.png 240w\" sizes=\"(max-width: 1527px) 100vw, 1527px\" \/><figcaption><center>Figure 9: Sequence-to-scan alignment. We first estimated the absolute pose of each sequence frame using image retrieval and matching. This initial localization prior was used to obtain a single rigid alignment between the input trajectory and the reference 3D model via voting. This allows us to discard outliers, as shown in the bottom part of the figure. The alignment was then relaxed by optimizing the individual frame poses in a pose graph (PGO) based on both relative and absolute pose constraints. We bootstrapped this initialization by mining relevant image pairs and re-localizing the queries. Given these improved absolute priors, we optimized the pose graph again and finally included reprojection errors of the visual correspondences in a bundle adjustment (BA) problem, yielding a refined trajectory.<\/center><\/figcaption><\/figure>\n\n\n\n<h3 id=\"evaluating-localization-and-mapping-in-the-context-of-ar\">Evaluating localization and mapping in the context of AR&nbsp;<\/h3>\n\n\n\n<p>We evaluated current state-of-the-art approaches in the single-frame setting as localizing i) single images obtained from phones and ii) single images and full camera rigs from HoloLens 2. Then we adapted these state-of-the-art approaches to take advantage of radio signals. Finally, we designed baselines, building on these methods and utilizing the on-device real-time tracker in a multi-frame localization setting corresponding to a real-world AR application. The results show that performance of state-of-the-art methods can be significantly improved by including these additional data streams generally available in modern AR devices, as shown in Figure 10.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1017\" height=\"379\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-10.png\" alt=\"Two bar graphs that show localization recall for state-of-the-art methods on both HoloLens 2 and iPhone queries.  The results show that performance of state-of-the-art methods can be significantly improved by including additional data streams generally available in modern AR devices.\" class=\"wp-image-888666\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-10.png 1017w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-10-300x112.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-10-768x286.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-10-240x89.png 240w\" sizes=\"(max-width: 1017px) 100vw, 1017px\" \/><figcaption>Figure 10: These bar graphs show localization recall for state-of-the-art methods on both HoloLens 2 and iPhone queries. We considered several tracks: single-image and rig localization with and without radios, and similarly for sequence localization.&nbsp;In addition, we report a theoretical upper bound: the percentage of queries with at least 5 percent ground truth overlap with respect to the best database image.&nbsp;<\/figcaption><\/figure>\n\n\n\n<p>For a compelling user experience, AR applications should strive to retrieve and visualize content as quickly as possible after starting a session. To quantify this, we introduce a new metric called <em>time-to-recall<\/em>, which measures the sequence duration needed for successful localization. This encourages researchers to develop algorithms to accurately localize the camera as quickly as possible, as shown in Figure 11.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"431\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-11-1024x431.png\" alt=\"We show the time-to-recall at 80 percent for the HoloLens 2\u202fand at 70 percent for phone queries. Using radio signals reduces the time-to-recall from 10 seconds to 1.40 seconds and 3.58 seconds, respectively. \" class=\"wp-image-888669\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-11-1024x431.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-11-300x126.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-11-768x323.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-11-665x280.png 665w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-11-240x101.png 240w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/Figure-11.png 1173w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><figcaption>Figure 11: We show the time-to-recall at 80 percent for the HoloLens 2\u202fand at 70 percent for phone queries. Using radio signals reduces the time-to-recall from 10 seconds to 1.40 seconds and 3.58 seconds, respectively.&nbsp;<\/figcaption><\/figure>\n\n\n\n<h3 id=\"using-the-lamar-benchmark\">Using the LaMAR benchmark&nbsp;<\/h3>\n\n\n\n<p>LaMAR is the first benchmark that focuses on a realistic setup for visual localization and mapping using AR devices. The evaluation results show enormous potential for leveraging posed sequences instead of single frames and for leveraging other sensor modalities, like radio signals, to localize the camera and map the environment.&nbsp;<\/p>\n\n\n\n<p>Researchers can access the LaMAR benchmark, evaluation server, implementations of the ground-truth pipeline, as well as baselines with additional sensory data at the <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/lamar.ethz.ch\/\" target=\"_blank\" rel=\"noreferrer noopener\">LaMAR Benchmark<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> page. We hope this work inspires future research in developing localization and mapping algorithms tailored to real AR scenarios.&nbsp;<\/p>\n\n\n\n<h3 id=\"acknowledgments\">Acknowledgments&nbsp;<\/h3>\n\n\n\n<p>This research was conducted by Paul-Edouard Sarlin, <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/dsmn.ml\/\" target=\"_blank\" rel=\"noreferrer noopener\">Mihai Dusmanu<\/a>, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/joschonb\/\" target=\"_blank\" rel=\"noreferrer noopener\">Johannes L. Sch\u00f6nberger<\/a>, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/paspecia\/\" target=\"_blank\" rel=\"noreferrer noopener\">Pablo Speciale<\/a>, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/lugruber\/\" target=\"_blank\" rel=\"noreferrer noopener\">Lukas Gruber<\/a>, Viktor Larsson, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/onmiksik\/\" target=\"_blank\" rel=\"noreferrer noopener\">Ondrej Miksik<\/a>, and <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/mapoll\/\" target=\"_blank\" rel=\"noreferrer noopener\">Marc Pollefeys<\/a>. <\/p>\n","protected":false},"excerpt":{"rendered":"<p>Computer vision is one of the most remarkable developments to emerge from the field of computer science. It\u2019s among the most rapidly growing areas in the technology landscape and has the potential to significantly impact the way people live and work. Advances at the intersection of machine learning (ML) and computer vision have been accelerating in recent years, leading to significant progress in numerous fields, including healthcare, robotics, the automotive industry, and augmented reality (AR). Microsoft is proud to be a prominent contributor to computer vision research.<\/p>\n","protected":false},"author":42183,"featured_media":890526,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"categories":[1],"tags":[],"research-area":[13562],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[243984],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-885738","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-research-blog","msr-research-area-computer-vision","msr-locale-en_us","msr-post-option-blog-homepage-featured"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[734161,602418],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[],"related-projects":[],"related-events":[],"related-researchers":[],"msr_type":"Post","featured_image_thumbnail":"<img width=\"960\" height=\"540\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/1400x788_ECCV_Blog_hero_v2-scaled-960x540.jpg\" class=\"img-object-cover\" alt=\"This image contains two panels. The panel on the left is titled \u201c3D face reconstruction\u201d and depicts two faces of real people and corresponding face models developed using the dense landmarks method with 703 facial landmarks. The panel on the right is titled \u201cBenchmarking localization and mapping for AR\u201d and shows the interior of a building with paths\u2014or sequences\u2014where people had captured the environment using Microsoft HoloLens.\" decoding=\"async\" loading=\"lazy\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/1400x788_ECCV_Blog_hero_v2-scaled-960x540.jpg 960w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/1400x788_ECCV_Blog_hero_v2-300x169.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/1400x788_ECCV_Blog_hero_v2-1024x576.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/1400x788_ECCV_Blog_hero_v2-768x432.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/1400x788_ECCV_Blog_hero_v2-1536x865.jpg 1536w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/1400x788_ECCV_Blog_hero_v2-2048x1153.jpg 2048w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/1400x788_ECCV_Blog_hero_v2-1066x600.jpg 1066w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/1400x788_ECCV_Blog_hero_v2-655x368.jpg 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/1400x788_ECCV_Blog_hero_v2-343x193.jpg 343w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/1400x788_ECCV_Blog_hero_v2-240x135.jpg 240w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/1400x788_ECCV_Blog_hero_v2-640x360.jpg 640w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/1400x788_ECCV_Blog_hero_v2-1280x720.jpg 1280w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/10\/1400x788_ECCV_Blog_hero_v2-1920x1080.jpg 1920w\" sizes=\"(max-width: 960px) 100vw, 960px\" \/>","byline":"","formattedDate":"October 24, 2022","formattedExcerpt":"Computer vision is one of the most remarkable developments to emerge from the field of computer science. It\u2019s among the most rapidly growing areas in the technology landscape and has the potential to significantly impact the way people live and work. Advances at the intersection&hellip;","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/885738"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/42183"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=885738"}],"version-history":[{"count":71,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/885738\/revisions"}],"predecessor-version":[{"id":894531,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/885738\/revisions\/894531"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/890526"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=885738"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=885738"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=885738"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=885738"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=885738"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=885738"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=885738"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=885738"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=885738"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=885738"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=885738"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}