Tracking 3D-position of controllers is an important problem in AR and VR devices. Current state-of-the-art in Windows Mixed Reality (MXR) utilizes a constellation of LEDs on the controllers to track pose. The performance of this vision-based system suffers in sunlight and when the controller moves out of the camera’s field-of-view (Out-of-FOV). In this work, we employ sensor fusion within a learning-based framework to track the controller position. Specifically, we utilize ultrasound sensors on hand-held controllers and the head-mounted display to obtain ranging information. We then combine this information within the feedback loop of an auto-regressive forecasting model that is built with Recurrent Neural Networks (RNN). Finally, we fuse the RNN output with the default MXR tracking result via a Kalman Filter across different positional states (including Out-of-FOV). Thanks to the proposed approach, we demonstrate near-isotropic accuracy levels for estimating controller position, which was not possible to achieve before with the default MXR tracking system.<\/p>\n","protected":false},"excerpt":{"rendered":"
Tracking 3D-position of controllers is an important problem in AR and VR devices. Current state-of-the-art in Windows Mixed Reality (MXR) utilizes a constellation of LEDs on the controllers to track pose. The performance of this vision-based system suffers in sunlight and when the controller moves out of the camera’s field-of-view (Out-of-FOV). In this work, we […]<\/p>\n","protected":false},"featured_media":510770,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"footnotes":""},"research-area":[13556,13562,13552],"msr-video-type":[],"msr-locale":[268875],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-510761","msr-video","type-msr-video","status-publish","has-post-thumbnail","hentry","msr-research-area-artificial-intelligence","msr-research-area-computer-vision","msr-research-area-hardware-devices","msr-locale-en_us"],"msr_download_urls":"","msr_external_url":"https:\/\/youtu.be\/S_DNqU9iwVY","msr_secondary_video_url":"","msr_video_file":"","_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-video\/510761"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-video"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-video"}],"version-history":[{"count":1,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-video\/510761\/revisions"}],"predecessor-version":[{"id":510779,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-video\/510761\/revisions\/510779"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/510770"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=510761"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=510761"},{"taxonomy":"msr-video-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-video-type?post=510761"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=510761"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=510761"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=510761"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}