Lines, Shading, and the Perception of 3D Shape

Humans can easily see 3D shape from single 2D images, exploiting multiple kinds of information. There are several subfields (in both human vision and computer vision) devoted to the study of particular cues to 3D shape, such as shading, texture, and contours. However, the resulting algorithms remain specialized and fragile (in contrast with the flexibility and robustness of human vision). Recent work in graphics and psychophysics has demonstrated the importance of local orientation structure in conveying 3D shape. This information is fairly stable and reliable across rendering condition. We have developed an exemplar-based system (which we call Shape Collage) that learns to associate image patches with corresponding 3D shape patches.
We train it with synthetic images of “blobby” objects rendered in various ways, including solid texture, Phong shading, and line drawings. Given a new image, it finds the best candidate scene patches and assembles them into a coherent interpretation of the object shape.
Our system is the first that can retrieve the shape of naturalistic objects from line drawings. The same system, without modification, works for shape-from-texture and can also do shape-from-shading without requiring Lambertian surfaces. Thus disparate types of image information can be processed by a single mechanism to extract 3D shape.
(Collaborative work with Forrester Cole, Phillip Isola, Fredo Durand, and William Freeman.)