AMETHYST: Image Registration Engine for Multiframe Processing

Multi-frame processing is a technique in computational photography that combines several video frames to produce a specially-tailored, and typically enhanced, frame. Serial processing of frames in image signal processors (ISPs) leads to artifacts when objects in the scene move or the camera moves during capture. In order to mitigate movement issues, camera processing pipelines employ simultaneous capture and processing techniques such as the Nvidia Computational Photography Engine. Accelerated computing units such as the Chimera (see figure) bring down frame capture and processing delays from 2 sec. to 0.2 sec, which is a 10x speed up. Such fast capture limits object and camera movement between frames leading to high-quality image composition. Multi-frame processing computations are thus accelerated with GPUs. We propose to partition the compositing steps into capture and processing. In our approach, we rely on advances in optics to allow fast frame capture and accelerate feature-extraction computations via on-device image alignment. Furthermore, we follow up with image fusion through various application-specific enhancement techniques run on an ISP. Our approach targets a 100x speed up in this process by exploiting dedicated ASICs for feature extraction.