Sketch-based image search is a well-known problem difficult to solve. Little progress has been made in the past decade in developing a large-scale, practical, sketch-based search engine. We have revisited this problem and developed a scalable solution. Based on this solution, we have built a system called MindFinder that indexes more than 2 million web images, enabling efficient, sketch-based image retrieval. MindFinder enables users to sketch major curves of a target image as a query. Based on these sketches, relevant images can be returned in real time. The system also supports the addition of semantic tags and colors to enhance the relevance of search results.

Videos

Although searching images via keywords or an example image has been successfully launched in some commercial search engines of billions of images, it is still very challenging for both academia and industry to develop a sketch-based image retrieval system on a billion-level database. In this work, we systematically study this problem and try to build a system to support query-by-sketch for two billion images. The raw edge pixel and Chamfer matching are selected as the basic representation and matching in this system, owning to the superior performance compared with other methods in extensive experiments. To get a more compact feature and a faster matching, a vector-like Chamfer feature pair is introduced, based on which the complex matching is reformulated as the crossover dot-product of feature pairs. Based on this new formulation, a compact shape code is developed to represent each image/sketch by projecting the Chamfer features to a linear subspace followed by a non-linear source coding. Finally, the multi-probe Kmedoids-LSH is leveragedto index database images, and the compact shape codes are further used for fast reranking. Extensive experiments show the effectiveness of the proposed features and algorithms in building such a sketch-based image search system.

We develop the MagicBrush system, a novel painting-based image search engine. This system enables users to draw a color sketch as a query to find images. Different from existing works on sketch-based image retrieval, most of which focus on matching the shape structure without carefully considering other important visual modalities, MagicBrush takes into account the indispensable value of “color” related to “shape”, and explores to make use of both the shape and color expectations that users usually have when they’re imaging or searching for an image. To achieve this, we 1) develop a user-friendly interface to allow users to easily “paint out”their colorful visual expectations; 2) design a compact feature “color-edge word” to encode both shape and color information in a organic way; and 3) develop a novel matching and index structure to support a real-time response in 6.4 million images. By taking into account both shape and color information, the MagicBrush system helps users to vividly present what they are imagining, and retrieve images in a more natural way.

Sketch Match: On 2011-08-17, the app, Sketch Match, was released to WP7 marketplace! It is powered by the MindFinder technology (sketch-based image search), with a well-refined sketch matching game.

Sketch Match is powered by MindFinder research work. The app supports users to search for similar images by sketching. It also provides a sketch matching game for entertainment and learning.

Use your fingers to sketch what you’re imagining and find similar images:

In the matching game, sketch the images as closely as possible to score:

Find your sketching rank in the global leaderboard:

• Changcheng Xiao, Changhu Wang, Liqing Zhang, Lei Zhang. Sketch-based Image Retrieval via Shape Words. ICMR 2015.
• Changcheng Xiao, Changhu Wang, Liqing Zhang, Lei Zhang. IdeaPanel: A Large Scale Inter-active Sketch-based Image Search System (Demo). ICMR 2015.
• Changqing Zou, Changhu Wang, Yafei Wen, Lei Zhang, Jianzhuang Liu. Viewpoint-Aware Representation for Sketch-Based 3D Model Retrieval. IEEE Signal Processing Letters, Vol. 21, No. 8, pp. 966-970, 2014.
• Xinghai Sun, Changhu Wang, Avneesh Sud, Chao Xu, Lei Zhang. MagicBrush: Image Search by Color Sketch (Demo). ACM Multimedia 2013.
• Xinghai Sun, Changhu Wang, Chao Xu, Lei Zhang. Indexing Billions of Images for Sketch-based Retrieval. ACM Multimedia 2013.
• Jiangping Wang, Changhu Wang, Thomas Huang. Efficient Image Contour Detection using Edge Prior. ICME 2013.
• Changhu Wang, Lei Zhang. 草图搜索的魅力与挑战. (The Charms and Challenges of Sketch-based Image Search.) In Communications of China Computer Federation, vol. 8, no. 12, pp. 20-25, December 2012.
• Xianming Liu, Changhu Wang, Hongxun Yao, Lei Zhang, The Scale of Edges, in CVPR 2012.
• Changhu Wang, Jun Zhang, Bruce Yang, Lei Zhang, Sketch2Cartoon: Composing Cartoon Images by Sketching, in ACM Multimedia 2011.
• Wei Zheng, Changhu Wang, Xilin Chen, Shape-based Web Image Clustering for Unsupervised Object Detection, in ICME 2011.
• Changhu Wang, Yang Cao, Lei Zhang, MindFinder: A Sketch-based Image Search Engine based on Edgel Index (demo), in CVPR 2011.
• Yang Cao, Changhu Wang, Liqing Zhang, and Lei Zhang, Edgel Inverted Index for Large-Scale Sketch-based Image Search, in CVPR 2011.
• Yang Cao, Hai Wang, Changhu Wang, Zhiwei Li, Liqing Zhang, and Lei Zhang, MindFinder: Interactive Sketch-based Image Search on Millions of Images, in ACM Multimedia 2010 International Conference, 2010. (Best demo award)
• Changhu Wang, Zhiwei Li, and Lei Zhang, MindFinder: Image Search by Interactive Sketching and Tagging, in WWW’10: 19th International World Wide Web Conference, 2010.