{"id":308246,"date":"2006-04-12T12:00:59","date_gmt":"2006-04-12T19:00:59","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=308246"},"modified":"2016-10-18T12:37:21","modified_gmt":"2016-10-18T19:37:21","slug":"photo2search-explore-real-world-via-camera-phone","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/photo2search-explore-real-world-via-camera-phone\/","title":{"rendered":"Photo2Search: Explore the Real World via Camera Phone"},"content":{"rendered":"

By Rob Knies, Managing Editor, Microsoft Research<\/em><\/p>\n

There\u2019s a new restaurant in town. Wonder what people are saying about it?<\/p>\n

Take a photo.<\/p>\n

That handy gadget you\u2019ve been coveting is on sale at the mall. How does its price compare to those offered elsewhere?<\/p>\n

Snap a picture.<\/p>\n

A new, blockbuster movie arrives at your local theater. Thumbs up, or thumbs down?<\/p>\n

Point and shoot.<\/p>\n

It can\u2019t be that easy, can it? Xing Xie<\/a> says that, yes, it can.<\/p>\n

Xie, a researcher for the Web Search and Mining<\/a> group within Microsoft Research Asia<\/a>, is working on technology called Photo2Search, which is designed to provide information on the go for users of camera phones.<\/p>\n

\u201cAs the old saying goes,\u201d Xie says, \u201ca picture is worth a thousand words.\u201d<\/p>\n

Maybe more, actually. Photo2Search gives users a way to search a Web-based database by using nothing more than an image captured by a cellphone equipped with a digital camera.<\/p>\n

\u201cThis technology,\u201d Xie says, \u201caims to solve the problem of mapping a physical-world object to a digital-world object. You see an object in the physical world, and you want to know the corresponding information in the digital world\u2014for example, its price on the Web, user comments, or Web sites. There are many different solutions. You can use a bar code or radio-frequency identification. But using a picture of the object is very convenient and very easy to deploy.\u201d<\/p>\n

The easy part is the key. Camera phones are simple to use, but the process of text-based search on them is not. That realization provided the late-2004 genesis for Phone2Search.<\/p>\n

\u201cAt that time,\u201d Xie recalls, \u201cthe idea was very simple: Use a camera phone to do a Web search. This is very interesting, because inputting images is much more convenient than inputting text queries on a small device.\u201d<\/p>\n

Photo2Search works like this: Seeking information about something seen, a user takes a photo of the object and sends the photo, via e-mail or Multimedia Messaging Service, to a Web-based server, which searches an image database for matches. The server then delivers database information\u2014whether it be a Web page featuring the object in the photo or information associated with the object\u2014to the user, who can act on the information received: read a menu, enter a gallery, book a hotel room, make a purchase.<\/p>\n

Sounds simple, right? The devil, as always, is in the details.<\/p>\n

When Xie, who collaborated on the Photo2Search technology with some Microsoft Research Asia colleagues and a handful of visiting students, first developed his concept of image-based Web searching using photos captured using camera phones, most prior work in this area had been based on Content Based Image Retrieval (CBIR) approaches, which index the content of images by features such as color, texture, shape, object layout, and edge direction. But while such approaches, which take considerable computational resources, are able to identify photos with some similar visual features, they don\u2019t necessarily excel at locating ones with the same prominent object or scene pictured in a query image.<\/p>\n

\u201cWe found,\u201d Xie says, \u201cthe precision of CBIR is not sufficient for practical use.\u201d<\/p>\n

Then he turned to computer-vision techniques, but again, challenges arose.<\/p>\n

\u201cWe found speed is a very big concern,\u201d he says. \u201cMost computer-vision algorithms are slow.\u201d<\/p>\n

In the second half of 2005, Xie and colleagues rebuilt the system, with image matching based on some well-known computer-vision algorithms that extract features from images. That choice proved productive, resulting in an efficient, high-dimensional index that can search through a large image database and return results quickly\u2014combing through a collection of 6,000 images and delivering matches in a mere three seconds using a common laptop. At that point, the process begins to enter the realm of the practical.<\/p>\n

The searchable database still needs to be a predefined collection of images, but they can be harvested from the Web. Manual annotation and organization are then employed to enhance performance.<\/p>\n

The promise of Xie\u2019s technology is significant. The burgeoning consumer adoption of camera phones offers the potential for much richer Web queries, using, in addition to text, the use of images, voices, even video.<\/p>\n

In a paper entitled \u201cPhoto-to-Search: Using Camera Phones to Inquire of the Surrounding World,\u201d to be delivered in Japan in May during the upcoming seventh International Conference on Mobile Data Management<\/a>, Xie and co-authors Mingjing Li and Wei-Ying Ma<\/a>, both of Microsoft Research Asia, and Menglei Jia and Xin Fan of the University of Science and Technology of China, underscore how important camera phones could become in searching via mobile devices.<\/p>\n

\u201cThe value of camera phones on daily information acquisition has not been sufficiently recognized by the wireless industry and researchers,\u201d the authors state. \u201cWith necessary technologies, they [could] become a powerful tool to acquire \u2026 information [about] the surrounding world on the go.\u201d<\/p>\n

In an increasingly device-filled world, that could prove a boon to millions.<\/p>\n

\u201cThe coolest thing,\u201d Xie says, \u201cis that you can use a pure image as a query, with no text. That is a totally new search experience.\u201d<\/p>\n

And further refinements are forthcoming.<\/p>\n

\u201cWe will continue to work on efficiency and to support larger databases,\u201d Xie says. \u201cWe hope, in the future, when a user submits a photo to, for example, MSN Spaces\u00ae, we can quickly figure out the latitude and the longitude of that photo by using our technology.\u201d<\/p>\n

It remains to be seen how Photo2Search might be incorporated into an upcoming product, but the possibilities are intriguing.<\/p>\n

\u201cThere is still a lot to do to make this technology into products,\u201d Xie cautions, adding, \u201cIf we can make it practical, then this is a big contribution for both local-search and mobile-search products.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":"

By Rob Knies, Managing Editor, Microsoft Research There\u2019s a new restaurant in town. Wonder what people are saying about it? Take a photo. That handy gadget you\u2019ve been coveting is on sale at the mall. How does its price compare to those offered elsewhere? Snap a picture. A new, blockbuster movie arrives at your local […]<\/p>\n","protected":false},"author":39507,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"footnotes":""},"categories":[194475,194460],"tags":[215411,215426,215423,203269,215417,215420,215414],"research-area":[13563,13555],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-308246","post","type-post","status-publish","format-standard","hentry","category-database-data-analytics-platforms","category-search-and-information-retrieval","tag-camera-phones","tag-cbir","tag-content-based-image-retrieval","tag-photo2search","tag-query","tag-search-experience","tag-web-based-database","msr-research-area-data-platform-analytics","msr-research-area-search-information-retrieval","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[199560],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[],"related-projects":[],"related-events":[],"related-researchers":[],"msr_type":"Post","byline":"","formattedDate":"April 12, 2006","formattedExcerpt":"By Rob Knies, Managing Editor, Microsoft Research There\u2019s a new restaurant in town. Wonder what people are saying about it? Take a photo. That handy gadget you\u2019ve been coveting is on sale at the mall. How does its price compare to those offered elsewhere? Snap…","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/308246"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/39507"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=308246"}],"version-history":[{"count":3,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/308246\/revisions"}],"predecessor-version":[{"id":308375,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/308246\/revisions\/308375"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=308246"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=308246"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=308246"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=308246"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=308246"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=308246"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=308246"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=308246"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=308246"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=308246"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=308246"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}