Real Time Image-based Rendering Using Sparsely Placed Video Cameras

In the recently five years, image-based rendering (IBR) has been an active research area and considered as a potential approach for photorealistic rendering of complex scenes that are difficult to model and render using the traditional polygon-based graphics pipeline. The basic idea of IBR is to combine images of the scene collected from different but fixed viewpoints to create new views. Early work in image-based rendering was based on the assumptions that a large number of cameras are densely placed around the scene or the geometry of the scene is known in advance. However, it remains unclear how to deal with the configuration that has the most potential commercial applications, i.e. IBR based on a small set of cameras and unknown scene geometry. In addition, even if the knowledge of the scene geometry is perfectly known, the quality of the output images of current IBR techniques is usually limited by the resolution of the input images or videos. The objectives of the proposed research are to advance the understanding of these problems and develop novel geometric representations and the associated estimation and resolution enhanced rendering algorithms for sparse image-based new view synthesis. The technology will enable many real world applications such as Immersive virtual tourism, 3D teleconferencing, IBR-based gaming and movie special effects, 3D immersive surveillance and monitoring. The proposed research approach consists of the following three related components: (1) Novel explicit scene representations for sparse IBR and the associate estimation and rendering algorithms. (2) Novel efficient direct sparse IBR techniques. (3)Resolution enhanced image-based rendering technique for rendering image with higher resolution than the original input images. The proposed research activities will be an integral part of the long-term effort to build a vigorous research and education program in computer vision and human computer interaction at the University of California, Santa Cruz.