MagicBrush: Image Search by Color Sketch

In this paper, we showcase the MagicBrush system, a nov-
el painting-based image search engine. This system enables
users to draw a color sketch as a query to find images. Dif-
ferent from existing works on sketch-based image retrieval,
most of which focus on matching the shape structure with-
out carefully considering other important visual modalities,
MagicBrush takes into account the indispensable value of
“color” related to “shape”, and explores to make use of both
the shape and color expectations that users usually have
when they’re imaging or searching for an image. To achieve
this, we 1) develop a user-friendly interface to allow users to
easily “paint out”their colorful visual expectations; 2) design
a compact feature “color-edge word” to encode both shape
and color information in a organic way; and 3) develop a
novel matching and index structure to support a real-time
response in 6.4 million images. By taking into account both
shape and color information, the MagicBrush system helps
users to vividly present what they are imagining, and re-
trieve images in a more natural way.