{"id":165138,"date":"2013-10-01T00:00:00","date_gmt":"2013-10-01T00:00:00","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/msr-research-item\/real-time-rgb-d-camera-relocalization\/"},"modified":"2018-10-16T21:36:15","modified_gmt":"2018-10-17T04:36:15","slug":"real-time-rgb-d-camera-relocalization","status":"publish","type":"msr-research-item","link":"https:\/\/www.microsoft.com\/en-us\/research\/publication\/real-time-rgb-d-camera-relocalization\/","title":{"rendered":"Real-Time RGB-D Camera Relocalization"},"content":{"rendered":"<div class=\"asset-content\">\n<p>We introduce an efficient camera relocalization approach which can be easily integrated into dense 3D reconstruction pipelines, such as KinectFusion. Our method is based on keyframes and makes use of randomized ferns which provide both compact encoding of frames and fast retrieval of pose proposals in case of tracking failure. During successful tracking, each frame\/pose pair is considered as a potential keyframe. Only frames which are sufficiently dissimilar in the space of appearance are added to the set of keyframes. This keeps the scene representation compact and at the same time the coverage sufficiently dense. Frame dissimilarity is defined via the block-wise hamming distance (BlockHD) between the codes generated by the ferns. Distances between incoming frames and keyframes are efficiently and simultaneously evaluated by simply traversing the nodes of the ferns and counting co-occurrences with keyframes having equal sub-codes. For tracking recovery, camera pose proposals are retrieved from keyframes with smallest BlockHDs which are then used to reinitialize the tracking algorithm.<\/p>\n<p>Both, online determination of keyframes and camera pose recovery are computationally efficient and have minimal impact on the run-time of the 3D reconstruction. Incorporating our method allows seamless continuation of reconstructions even when tracking is frequently lost. Additionally, we demonstrate how marker-free augmented reality can be realized by mesh-to-volume registration between an offline model and the online reconstruction. Our relocalization method is particularly appealing in such AR applications where the experience of steady pose tracking is essential.<\/p>\n<\/div>\n<p><!-- .asset-content --><\/p>\n","protected":false},"excerpt":{"rendered":"<p>We introduce an efficient camera relocalization approach which can be easily integrated into dense 3D reconstruction pipelines, such as KinectFusion. Our method is based on keyframes and makes use of randomized ferns which provide both compact encoding of frames and fast retrieval of pose proposals in case of tracking failure. During successful tracking, each frame\/pose [&hellip;]<\/p>\n","protected":false},"featured_media":0,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"msr-content-type":[3],"msr-research-highlight":[],"research-area":[13556,13562],"msr-publication-type":[193716],"msr-product-type":[],"msr-focus-area":[],"msr-platform":[],"msr-download-source":[],"msr-locale":[268875],"msr-post-option":[],"msr-field-of-study":[],"msr-conference":[],"msr-journal":[],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-165138","msr-research-item","type-msr-research-item","status-publish","hentry","msr-research-area-artificial-intelligence","msr-research-area-computer-vision","msr-locale-en_us"],"msr_publishername":"IEEE","msr_edition":"International Symposium on Mixed and Augmented Reality (ISMAR)","msr_affiliation":"","msr_published_date":"2013-10-01","msr_host":"","msr_duration":"","msr_version":"","msr_speaker":"","msr_other_contributors":"","msr_booktitle":"","msr_pages_string":"","msr_chapter":"","msr_isbn":"","msr_journal":"","msr_volume":"","msr_number":"","msr_editors":"","msr_series":"","msr_issue":"","msr_organization":"","msr_how_published":"","msr_notes":"","msr_highlight_text":"","msr_release_tracker_id":"","msr_original_fields_of_study":"","msr_download_urls":"","msr_external_url":"","msr_secondary_video_url":"","msr_longbiography":"","msr_microsoftintellectualproperty":1,"msr_main_download":"218155","msr_publicationurl":"","msr_doi":"","msr_publication_uploader":[{"type":"file","title":"glocker2013ismar.pdf","viewUrl":"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2013\/10\/glocker2013ismar.pdf","id":218155,"label_id":0}],"msr_related_uploader":"","msr_attachments":[{"id":218155,"url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2013\/10\/glocker2013ismar.pdf"}],"msr-author-ordering":[{"type":"user_nicename","value":"glocker","user_id":31887,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=glocker"},{"type":"user_nicename","value":"shahrami","user_id":33590,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=shahrami"},{"type":"user_nicename","value":"jamiesho","user_id":32162,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=jamiesho"},{"type":"user_nicename","value":"antcrim","user_id":31055,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=antcrim"}],"msr_impact_theme":[],"msr_research_lab":[],"msr_event":[],"msr_group":[],"msr_project":[171359,171165,170770,169659],"publication":[],"video":[],"download":[],"msr_publication_type":"inproceedings","related_content":{"projects":[{"ID":171359,"post_title":"Real-Time RGB-D Camera Relocalization","post_name":"real-time-rgb-d-camera-relocalization","post_type":"msr-project","post_date":"2014-05-27 08:54:30","post_modified":"2017-06-14 09:48:26","post_status":"publish","permalink":"https:\/\/www.microsoft.com\/en-us\/research\/project\/real-time-rgb-d-camera-relocalization\/","post_excerpt":"We introduce an efficient camera relocalization approach which can be easily integrated into real-time 3D reconstruction methods, such as KinectFusion. Our approach makes use of compact encoding of whole image frames which enables both online harvesting of keyframes in tracking mode, and fast retrieval of pose proposals when tracking is lost. The encoding scheme is based on randomized ferns and simple binary feature tests. Each fern generates a small block code, and the concatenation of&hellip;","_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/171359"}]}},{"ID":171165,"post_title":"RGB-D Dataset 7-Scenes","post_name":"rgb-d-dataset-7-scenes","post_type":"msr-project","post_date":"2013-07-03 01:46:14","post_modified":"2022-09-07 10:55:36","post_status":"publish","permalink":"https:\/\/www.microsoft.com\/en-us\/research\/project\/rgb-d-dataset-7-scenes\/","post_excerpt":"The 7-Scenes dataset is a collection of tracked RGB-D camera frames. The dataset may be used for evaluation of methods for different applications such as dense tracking and mapping and relocalization techniques. Overview All scenes were recorded from a handheld Kinect RGB-D camera at 640x480 resolution. We use an implementation of the KinectFusion system to obtain the 'ground truth' camera tracks, and a dense 3D model. Several sequences were recorded per scene by different users,&hellip;","_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/171165"}]}},{"ID":170770,"post_title":"KinectFusion Project Page","post_name":"kinectfusion-project-page","post_type":"msr-project","post_date":"2011-08-09 16:17:54","post_modified":"2023-11-28 10:50:38","post_status":"publish","permalink":"https:\/\/www.microsoft.com\/en-us\/research\/project\/kinectfusion-project-page\/","post_excerpt":"This project investigates techniques to track the 6DOF position of handheld depth sensing cameras, such as Kinect, as they move through space and perform high quality 3D surface reconstructions for interaction. Other collaborators (missing from the list below): Richard Newcombe (Imperial College London); David Kim (Newcastle University &amp; Microsoft Research); Andy Davison (Imperial College London) &nbsp; &nbsp;","_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/170770"}]}},{"ID":169659,"post_title":"Project InnerEye - Democratizing Medical Imaging AI","post_name":"medical-image-analysis","post_type":"msr-project","post_date":"2008-10-07 05:22:18","post_modified":"2023-07-28 05:51:32","post_status":"publish","permalink":"https:\/\/www.microsoft.com\/en-us\/research\/project\/medical-image-analysis\/","post_excerpt":"InnerEye is a research project that uses state of the art\u00a0machine learning\u00a0technology to build innovative tools for the automatic, quantitative analysis of three-dimensional medical images.","_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/169659"}]}}]},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/165138"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-research-item"}],"version-history":[{"count":2,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/165138\/revisions"}],"predecessor-version":[{"id":537295,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/165138\/revisions\/537295"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=165138"}],"wp:term":[{"taxonomy":"msr-content-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-content-type?post=165138"},{"taxonomy":"msr-research-highlight","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-highlight?post=165138"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=165138"},{"taxonomy":"msr-publication-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-publication-type?post=165138"},{"taxonomy":"msr-product-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-product-type?post=165138"},{"taxonomy":"msr-focus-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-focus-area?post=165138"},{"taxonomy":"msr-platform","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-platform?post=165138"},{"taxonomy":"msr-download-source","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-download-source?post=165138"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=165138"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=165138"},{"taxonomy":"msr-field-of-study","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-field-of-study?post=165138"},{"taxonomy":"msr-conference","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-conference?post=165138"},{"taxonomy":"msr-journal","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-journal?post=165138"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=165138"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=165138"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}