Performance High Imaging Using Large Camera Arrays
Bennett Wilburn1 Neel Joshi2 Vaibhav Vaish2 Eino-Ville Talvala1 Emilio Antunez1 Adam Barth2 Andrew Adams2 Mark Horowitz1 Marc Levoy2
1 Electrical
Engineering Department Stanford University
2 Computer
Science Department Stanford University
(a)
(b)
(c)
Figure 1: Different configurations of our camera array. (a) Tightly packed cameras with telephoto lenses and splayed fields of view. This arrangement is used for high-resolution imaging (section 4.1). (b) Tightly packed cameras with wide-angle lenses, which are aimed to share the same field of view. We use this arrangement for high-speed video capture (section 4.2) and for hybrid aperture imaging (section 6.2). (c) Cameras in a widely spaced configuration. Also visible are cabinets with processing boards for each camera and the four host PCs needed to run the system.
Abstract
The advent of inexpensive digital image sensors and the ability to create photographs that combine information from a number of sensed images are changing the way we think about photography. In this paper, we describe a unique array of 100 custom video cameras that we have built, and we summarize our experiences using this array in a range of imaging applications. Our goal was to explore the capabilities of a system that would be inexpensive to produce in the future. With this in mind, we used simple cameras, lenses, and mountings, and we assumed that processing large numbers of images would eventually be easy and cheap. The applications we have explored include approximating a conventional single center of projection video camera with high performance along one or more axes, such as resolution, dynamic range, frame rate, and/or large aperture, and using multiple cameras to approximate a video camera with a large synthetic aperture. This permits us to capture a video light field, to which we can apply spatiotemporal view interpolation algorithms in order to digitally simulate time dilation and camera motion. It also permits us to create video sequences using custom non-uniform synthetic apertures.
email:[e-mail] Neel
CR Categories: I.4.1 [Image Processing and Computer Vision]: Digitization and Image Capture--imaging geometry, sampling; C.3 [Computer Systems Organization]: Special Purpose and Application-Based Systems--real-time and embedded systems Keywords: aperture camera arrays, spatiotemporal sampling, synthetic
1
Introduction
One of the economic tenets of the semiconductor industry is products that sell in large volumes are cheap, while products that sell in lower volumes are more expensive, almost independent of the complexity of the part. For computers, this relationship has changed the way people think about building high-end systems; rather than building a custom high-end processor, it is more cost effective to use a large number of commodity processors. We are now seeing similar trends in digital imaging. As the popularity of digital cameras grows, the performance of low-end imagers continues to improve, while the cost of the high-end cameras remains relatively constant. In addition, researchers have shown that multiple images of a static scene can be used to expand the performance envelope of these cameras. Examples include creating images with increased resolution [Szeliski 1994] or dynamic range [S.Mann and R.W.Picard 1994; Debevec and Malik 1997]. In other work, Schechner and Nayar used spatially varying filters on a rotating camera to create high-resolution panoramas that also had high dynamic range or high spectral resolution [Schechner and Nayar 2001]. Another use for multiple views is view interpolation to create the illusion of a smoothly moving virtual camera in a static
Joshi is now at the University of California, San Diego.
or dynamic scene [Levoy and Hanrahan 1996; Gortler et al. 1996; Rander et al. 1997; Matusik et al. 2000]. Most of these efforts employ a single moving high-quality camera viewing a static scene. To achieve similar results on dynamic scenes, multiple cameras are required. This motivated us in 1999 to think about designing a flexible array containing a large number of inexpensive video imagers. The multiple camera array that resulted consists of 100 video cameras, each connected to its own processing board. The processing boards are capable of local image computation, as well as MPEG2 compression. In section 2, we review prior work in building multiple video camera systems. While these systems are generally directed at specific applications, they provide valuable insights into the requirements for a flexible capture system. Section 3 gives an overview of our multiple camera array and explains in a little more depth the features we added to make it a general purpose research tool. The rest of this paper focuses on our recent results using the camera array in different imaging applications. We start by exploring ways of using multiple cameras to create an aggregate virtual camera whose performance exceeds the capability of an individual camera. Since these applications intend to approximate a camera with a single center of projection, they generally use densely packed cameras. In particular, section 4 explores the creation of a very high-resolution video camera in which the cameras are adjusted to have modestly overlapping fields of view. We then aim the cameras inward until their fields of view overlap completely, and we use our system's fine timing control to provide a virtual video camera with a very high frame-rate. In both of these applications, the large number of cameras provide some opportunity that would not be present in a single camera system. For the virtual high-resolution imager, one can perform exposure metering individually on each camera, which for scenes with spatially varying brightness allows us to form a mosaic with high dynamic range. For the virtual highspeed imager, one can integrate each frame for longer than one over the frame-rate, thereby capturing more light per unit time than is possible using a single high-speed camera. Sections 5 and 6 consider applications in which the cameras are spread out, thereby creating a multi-perspective video camera. One important application for this kind of data is view interpolation, whose goal is to move the virtual observer smoothly among the captured viewpoints. For video lightfields, the problem becomes one of spatiotemporal interpolation. Section 5 shows that the optimal sampling pattern to solve this problem uses cameras with staggered, not coincident, trigger times. It also describes a spatiotemporal interpolation method that uses a novel optical flow variant to smoothly interpolate data from...
CameraArray_Sig05