of Calibration a Multicamera Network
Patrick T. Baker Center for Automation Research University of Maryland College Park, MD 20742 Yiannis Aloimonos Center for Automation Research University of Maryland College Park, MD 20742
for a particular network and thus a camera configuration change could render the calibration object useless. If we do not use a 3D object, then we are forced to use some kind of structure from motion techniques where we simultaneously find the internal and external calibration parameters for all our cameras. This problem is well-studied for the case of two cameras [3]. The second problem could be solved with a technological solution such as having LEDs blink in a specified sequence. Unfortunately, the LEDs are difficult to measure more accurately than a pixel, because if the LED image is contained within a single pixel, there is no way to tell where it is. There are ways to get around this, but there are simpler ways to obtain calibration, as we shall see. We tried using a single corner as the source for our points [1], but we sometimes had trouble with spurious point detection due to the clothing of the data collector. Also, even when we obtained clean data, the optimization tended to be unstable due to the projective ambiguity. While this can be solved to some extent with the addition of some metric information [6, 4] (such as right angles, or a constant distance), the solution was not as stable as we might have wished, and was not accurate as accurate as we would have hoped with regard to nonlinear calibration [9]. The homography based methods [11, 10] combine the flexibility of an easily generated data set with the accuracy and stability given by having known distances between points. These methods are excellent for easy calibration of one or a few cameras. Since the point extraction is based on the intersection of two lines, this step is accurate, and was attractive to us for implementation. Unfortunately, the data collection requires the user to show a pattern to each camera, an O(n) requirement (n being the number of cameras), and also requires the user to click on points before the calibration, another O(n) requirement. We therefore could not use these methods. However, we did take away the valuable knowledge that the homography based methods can be very stable and accurate, because they are based on entire patterns for which the absolute position of the corners on the pattern is known. Picking an appropriate projective dis1
Abstract
With the advent of laboratories containing dozens of cameras, and the possibility of laboratories containing hundreds of cameras, the question of how to calibrate all the cameras has become pressing. While it is certainly possible to calibrate these networks in a labor intensive manner, a simple, stable, and accurate calibration method is still needed. This paper presents such a method, based on textures printable on a laser printer and mounted on a board. We will show what the problems with the current methods are, and show how these problems can be overcome with a novel use of a trilinear constraint related to the vanishing point constraint, which we call the primsatic line constraint. High accuracy with little user effort is achieved with this method.
1 Introduction
Calibration is an integral part of any multicamera lab and must be addressed before any data can be used. There are remarkably few papers on the calibration problem as applied to many cameras, because it has been thought that standard calibration techniques would prove adequate to calibrate the new laboratories, and that new techniques would not be necessary. We explain briefly why these techniques break down with large camera networks, then explain what is necessary for a good solution, then show our solution and why we believe it satisfies our requirements.
1.1
Possible Calibration Methods
The calibration method used in photogrammetry [8] has been to construct a 3D object which has been precisely measured. This method, while highly accurate and fast, can be expensive to construct (or purchase) and maintain. Second, an operator needs to look at the output from each camera, and this is infeasible for networks with more than a few cameras. Third, the 3D object would have to be constructed
tortion is then not a factor in this case, and even nonlinear calibration can be obtained accurately with a simple technique. Our conclusion was that the problem of calibrating networks of dozens (or more) of cameras is qualitatively different from the problem of calibration of just a few cameras. Many engineering issues, manageable when calibrating a few cameras, become overwhelming in the case of dozens (or more) of cameras. For these large networks of cameras, we need new methods of calibration so that we do not need to be so careful with each individual camera yet can still get high accuracy.
us to carve voxels precisely, or use stereo with high accuracy. We believe that our method addresses all these requirements.
2
Camera Geometry and Calibration
We describe in this section our camera model and the constraints we use to determine our camera calibration.
2.1 Cameras and Points
We define world points in three dimensions without homogeneous coordinates. Definition 1 A world point is an element P R3 . We may represent such a point in a particular coordinate system as: X (1) P = Y Z Our image points exist in the projective plane P2 . Definition 2 An image point is an element p P2 . We may represent such a point in a particular coordinate system as: x p = y (2) z These coordinates are homogeneous. A camera takes world points and projects them into image points. Usually this would be accomplished using a 3 4 projection matrix. However, since we are calibrating cameras, and we have nonlinear terms in our projection equations, we define our camera as follows. Definition 3 A camera C is a map C : R3 P2 from world points to image points. If our world points are coordinatized in a fiducial coordinate system, we may represent this map with a pair (B, T), where B : R3 R3 is a nonlinear function, and T is a 3-vector representing the camera center. The action of the map on a world point coordinate P is: C(P) = B(P - T) where C(P) is considered as a member of P2 . For every camera we would like to find this B and T, which will be the calibration parameters for our multicamera network. Here we have defined a camera as taking a world point and mapping it to an image point through a nonlinear transformation on the world coordinates. In the interest of simplicity, we will treat B as a linear function for most of the paper, although our method works well for nonlinear calibration. 2 (3)
1.2
Requirements for New Solution
From an engineering (as opposed to geometric) point of view, let us discuss what is necessary for a many camera calibration solution. Such considerations for the single camera calibration were discussed in [5]. First of all, the standard calibration parameters [2] are not as important as the ability to reconstruct the Euclidean geometry of the viewing space as accurately as possible. We use a reprojection error to determine whether we have calibrated our viewing space properly. The calibration should involve some object which is easy to create. A particularly cheap and accurate object then is some sort of printed sheet mounted on a rigid board, as in the homography methods. Our camera networks may be quite large, and depending on how much the fields of view of the cameras overlap, we may need a larger or smaller board. The larger the pattern the more likely that many cameras will have at least some view of the pattern, but the more difficult it is to keep the board rigid. Beyond noting this we do not delve into the mechanical properties of our boards. We are not concerned with minimizing the number of views of our texture, because we are using video cameras. Therefore we always assume in this paper that we have many views of the pattern, and that the pattern is positioned in multiple orientations everywhere in the space that we anticipate our objects of interest will be in. We don't want to have to confirm that the entire board is in any particular frame, an impossible task when you have many cameras viewing from different directions. Therefore we need a method which will be able to use frames in which only a part of the board is visible. Linked to this is a desire to make sure that the entirety projection of the viewing volume is well-calibrated for all cameras, no matter whether the pattern is in or out of the depth of field of the camera. The method should also be stable in the sense that if the input is reasonable then the calibration should proceed from just the image sequences to the calibration parameters with no other input necessary. Finally, the method should be highly accurate, enabling
2.2
Cameras and Lines
P2 . The
Definition 4 An image line is represented by line may be given coordinates = 2
3 1
If we identify cameras 2 and 1 by setting B2 = B1 , which corresponds to the case where both 1 and 2 are taken from the same camera. If these are different parallel lines, then we obtain the vanishing point constraint Proposition 3 We have two or three parallel world lines, and two cameras with rotation/calibration matrices Bi . If camera 1 views image lines ^1 and ^3 and camera 2 views image line ^2 we obtain the vanishing point constraint. ^T B2 B -1 (^1 ^3 ) = 0
2 1
(4)
The point p and line must satisfy pT = 0 (5)
(9) (10)
^ B2 B -1 p 2 1 ^
T
=0
We use the following to go between our calibration equations for points and those for lines. Proposition 1 If we linearly transform image points p as ^ p = Bp with a matrix B, then the image line segment is transformed by ^ = B -T . Also, in this paper we use both calibrated image points (and lines) and uncalibrated image points (and lines). A calibrated point p (line ) exists in a coordinate frame which is only a translation away from the fiducial frame, while ^ the uncalibrated point p (line ^) exists in the actual camera coordinate system, so that we have: ^ p = B -1 p = B T^ (6) (7)
^ The quantity p = ^1 ^3 is called a vanishing point, and it is the point through which all images of world lines of direction Ld will pass. The constraint says that if we have a vanishing point in one image and a line in another image which we know is parallel to the lines in the first camera, then we have a constraint on the Bi . If we further identify cameras 2 and 1, then given an image of a set of parallel lines in one camera, we know that we must still have a zero triple product. Proposition 4 We have three parallel world lines, and a camera with rotation/calibration nonlinear function B : R3 R3 . Given images of these three world lines ^i , i [1, . . . , 3]. We obtain the the vanishing point existence constraint. |B T ^1 B T ^2 B T ^3 | = 0 (11) This last constraint means nothing if B is a linear function, since the constraint would be trivally satisfied. However, in the case where there is some nonlinear distortion in the projection equation, there will be a constraint on B. We may say that the prismatic line constraint operates on 1, 2, or 3 cameras. We use the prismatic line constraint constraint to obtain an initial estimate for the internal and rotational calibration parameters. We use the prismatic line consraint to obtain an initial estimate of our functions Bi . We run a nonlinear optimization to find the best Bi for the data. Please note that it was not necessary for the cameras to share any parts of their field of view to obtain an initial estimate for these functions, if we can ensure that they view parallel lines. Having these functions Bi , we show how to obtain estimates for our camera positions Ti . Because we are using many cameras, we use a slightly different formulation of the epipolar constraint [7], designed so that we use the cameras' absolute position Ti rather than the epipoles between the cameras.
-1 -1 -1 -1 ^ ^ ^ ^ (B1 p1B2 p2 )T T1 +(B2 p2B1 p1 )T T2 = 0 (12)
2.3
Constraints
The vector representing a calibrated image line is perpendicular to all the calibrated image points on the line. This line vector is thus perpendicular to the plane containing the calibrated image points and the center of projection. This in turn means that our image line must be perpendicular to the direction of the world line, since that world line is wholly contained in the plane. We may use this observation to form a constraint which operates independently of translation. In particular, the cameras may be separate or identical. Also, the image lines can have been created from a single line or two parallel lines. Because all of our lines, when put in the fiducial coordinate system, must be perpendicular to the line direction, we obtain a type of vanishing point constraint which operates on three cameras. Proposition 2 If we have one, two, or three parallel world lines, and three cameras with rotation/calibration matrices Bi , then if these three cameras view images of one of our world lines as ^i , with the lines not necessarily the same in all cameras, then we obtain the prismatic line constraint. ^T B2 (B T ^1 B T ^3 ) = 0
2 1 3
Since we will have initial estimates for both B1 and B2 , we also note that by using calibrated image points: (p1 p2 )T T1 + (p2 p1 )T T2 = 0 3 (13)
(8)
For some camera networks we may not have a single point visible from multiple cameras, if the cameras do not share a field of view. In this case we may use a line to obtain initial translational estimates. Proposition 5 If we have three cameras with parameters (Bi , Ti ), and a world line which projects to ^i , then the prismatic line constraint holds. There is only one other independent constraint, and it is the line trifocal constraint: 0= (Ti1 )T
[i1 ..i3 ]P+ [1..3] i1 | i2 i3 |
(14)
where | | is the signed magnitude. These constraints are sufficient to fix the cameras, but we need an efficient mechanism for combining the measurements into a coherent whole.
where the terms mki are always in the ki column. In the case of the epipolar equation, we will have two columns in each row and in the case of the trifocal equation, we will have three columns in each row. Let us call this matrix on the LHS A. A has ( M ) rows n and n columns. Let us form the matrix C = AT A. It is well known that the eigenvector corresponding to the smallest eigenvalue of C will be the unit vector which solves AT = 0. Because the problem is translation invariant, we have three extra degrees of freedom, so that our matrix C will have rank M - 4 in the general case. If we set T1 = 0, then we fix the location of the first camera to be at the origin of the fiducial coordinate system. If we form C to be the bottom right 3(M - 1) square matrix of C, then we know that (T )T C T = 0 (19) That is, the zero eigenvector of C gives the solution for our Ti , if T1 = 0. We thus have a method for integrating either our epipolar or trifocal constraints over any number of cameras to obtain an initial translation translation. We only discuss the epipolar equation here.
2.4
Putting the Constraints Together
Let us assume that we have M cameras and that we want to extend our constraints to find the Ti considering all of the cameras. Note that our epipolar and trilinear constraints are of the form
n
2.5 Nonlinear Factors and Lines
Let us look at how to find radial calibration parameters with lines. Since we work with image measurements, we will parameterize our functions over B -1 . We do this because it is very difficult to invert the function r = B -1 (^)^. It is rr easy to invert the matrix function B(^), and we do use this r inverse. Note that this is the reverse of the way that most authors have defined it, but when the object is to use the images in order to reconstruct, it makes just as much sense to parametrize the inverse of the projection than the forward direction. In the linear case, this inverse mapping is a matrix B -1 which is the same over the whole image. In this case the function inverse is easy and corresponds to the matrix inverse. In the nonlinear case, we have a matrix function B -1 (r), which varies depending on the location of r. So to put an image point into the translated fiducial coordinate system, we use r = B -1 (^)^ rr (20) The function is defined in terms of points ^. If we have a r measurement on a line which we would like to put in that same coordinate system, we can use the same technique before as and form: = B T (^)^ r (21) Note that this means that we need to know the location ^ r where we made our measurement of ^. Because the location of our measurement is crucial our results, if we are to calibrate it is important to make a measurement of ^ as locally as possible for maximum accuracy. 4
ci mT Ti = 0 i
i=1
(15)
where n is 2 for the epipolar and 3 for the line trilinear, and mi is either a point p or a line . Let us also assume that n < M . We may form an equation for every choice {k1 ..kn } of n numbers from the set {1..M }. Each of our constraints is a linear equation over the mki , Tki . We can form this into a linear equation:
n
fi (mk1 , .., mkn )mTi Tki k
i=1
(16)
Let us now set
T1 . . . T = Ti . . . TM
T2 . . . T = Ti . . . TM
(17)
With each choice of n measurements from the M measurements, we obtain another constraint, so we get ( M ) n equations, which we put into matrix form: . . . fi (mk1 , .., mkn )mT T = 0 (18) ki . . .
We consider in this section radial distortion, which in high quality lenses accounts for most of the distortion. We express our matrix function as B -1 (^) = RT S(K -1 ^)K -1 r r where K is upper triangular, R is orthogonal, and S(^) = I +(^(^^))(1 +2 |^^|2 +3 |^^|4 )(^(^^))T r z r z z r z r z r z (23) This is the reverse of the usual method of describing radial calibration, and while the parameters are not the same, the result encodes just the same information. To map lines, then, we use the equation (B -1 (^))-T = B T (^) r r r = RT S -T (K -1 ^)K T But since S is symmetric (B -1 (^))-T = RT S -1 (K -1 ^)K T r r Next note that if we can invert a matrix I + vvT as (I + vv )
T
(22)
Figure 1: Parallel lines are on one side of the board
(24) (25)
(26) Figure 2: Boxes are on the other side of the board
-1
vvT =I- (27) 1 + vT v = I - vvT (1 - vT v + (vT v)2 - . . . ) (28)
-1 ^
3
Measurements on the Image
From here on we set r = K v = (^ (r ^)) z z then we can see that
r
1 + 2 |^ r |2 + 3 |^ r |4 z z
(29)
S(r ) = I + vvT so that S -1 (r ) = I - (^ (r ^))(^ (r ^))T z z z z (1 + 2 |^ r |2 + 3 |^ r |4 z z - |^ r |2 (1 + 2 |^ r |2 + 3 |^ r |4 )2 z z z + |^ r |4 (1 + 2 |^ r |2 + 3 |^ r |4 )3 . . . ) z z z
(30)
We now show how to use the constraints in a realistic engineering setting to solve the problem of multicamera calibration. While the geometry of multicamera calibration is mostly known, the image processing is just as important. We must make the signal processing easy to obtain stable initial estimates. We have only tested our solution in the case where cameras have highly overlapping fields of view, but we see no reason that nonoverlapping fields of view could not be calibrated with a common line.
3.1 Proposed New Solution
We can break up the problem into a two stage solution in order to obtain a stable solution. We work with textures printed on either side of a board. On one side of the board is printed a set of lines (as in figure 1). On the other side of the board is printed a set of boxes with one missing in the middle (as in figure 2). In the first stage the set of lines is used to find the internal and rotational calibrations to within an affine distortion. Because we use the prismatic line constraint, we do not have to actually extract particular lines from the set of parallel lines. It is enough to extract only the line direction at different places in the image, as in figure 3. In each local area centered at ri there is a texture which can be approximated by set of lines with no foreshortening, if the area is small enough. We can easily find the orientation in this area. Us5
(31)
We throw away all terms attached to |^ r |6 and higher to z obtain S -1 (r ) = I - (^ (r ^))(1 + (2 - 2 )|^ r |2 z z 1 z + (3 - 21 2 + 3 )|^ r |4 )(^ (r ^))T z z 1 z (32) We can have the tools to put the line ^ into the fiducial frame with radial distortion, using the parameters of radial distortion designed for points ^. We use this theory to find an r initial estimate for radial distortion by just using lines.
Figure 3: A projected board is measured locally ing this orientation di we can find the uncalibrated image lines as ^i = ri di . We then obtain a collection of these ^i for each image. Knowing the aspect ratio and skew (probably 1 and 0) allows us to find the radial distortion parameters using these line measurements and the vanishing point existence constraint. We use this method to obtain a reasonable initial estimate for the first radial parameter, which should contain most of the distortion [11]. We have found that this procedure overestimates the radial distortion somewhat, but it is good enough to give us starting conditions for a nonlinear minimization. Once we do this T can collect our measurements into a we matrix L = i ^i ^i . The null space of this matrix, which is the uncalibrated point at infinity, we call s. To get an initial estimate for the internal and rotational calibration to within an affine distortion, we must fix some arbitrary affine distortion. We can do this by selecting four video frames to fix our space (3 to define our three cardinal directions, and one more frame to define dot products). The first three frames define the directions e1 = e2 =
0 1 0 1 0 0
tion to rectify our entire space [6]. We have after this stage an initial estimate for the internal, nonlinear, and rotational parameters for all the cameras. At this point one may ask why we do not simply use the boxes themselves to obtain an initial estimate for the rotation, and internal calibration. The answer is that the correspondence problem militates against this. We wouldn't know which orientation would correspond to which in the images. We need to use the parallel lines first to get an initial orientation for all the cameras. Having that orientation, we can then use the orthogonal sets of parallel lines.
3.2 Finding the Translational Parameters
We mentioned at the beginning of this paper that precisely measuring features is an important component the calibration method. Unfortunately, in a multicamera system it can be difficult to have consistent features because if the texture is out of the depth of field there will be significant blurring. The scale differences can cause any features to be quite small with respect to the cameras. We therefore take our feature to be the entire pattern of boxes. Remember that at this point in the calibration, we have a good estimate for the internal and rotational parameters. We also know the principal directions of the box texture. With this information, it is an easy matter to unwarp the box texture. This unwarped pattern will be a frontoparallel set of boxes with one box missing in the middle. If a camera sees the boxes but the missing box is out of the field of view then that frame for that camera is not used. In any case, given a sufficient number of views of the box texture from each camera, we can extract corresponding points with pixel accuracy. Using these correspondences and the matrix eigenspace technique showed earlier, we can find estimates for the positions of all the cameras.
,
, and e3 =
0 0 1
. With these we can see easily
1 1 1
that the directions of the column of B will be s1 , s2 , and s3 . If we assume that the fourth frame is , then we can see that s4 = 1 s1 +2 s2 +3 s3 , which will solve for the i up to an entire factor, which is enough to fix B. This gives us our initial estimates for all cameras which have views of the fiducial four video frames. Once we have fixed this, we may then add camera by camera to our initial set of cameras, using a linear method followed by a nonlinear minimization. We have found this to be a stable method for finding our internal and rotational parameters to within an affine distortion. On the other side of the board we have boxes which are aligned in the same direction as the lines. Since these boxes define orthogonal directions, we may extract these orthogonal directions and use them to find the proper affine distor6
3.3 Final Optimization
At this point we have initial estimates for the camera parameters, and and estimate for the normal and position of the texture. The only point measurement we have made is the one for the missing box. In a single camera calibration system, it would be a simple matter to extract points from the boxes. However, due to the great variability in photometric properties, blurred images, and foreshortened images, we found we needed to hand-check each image to make sure that the points were extracted properly. This was unacceptable. We therefore decided to optimize over the images themselves. While this is computationally expensive, we found that it gave us high accuracy with little user effort. We can describe the position of a box at frame i with a point Pi at the missing box, and two perpendicular unit
vectors vi,1 , and vi,2 which represent the displacement between contiguous boxes in each direction. The cameras j are described with the function Bj and position Tj . We desire to minimize over the sum of all the pixel values which we expect would be inside of a box. Let us determine which pixels should be projections of the black box. We have a pixel r in a camera described by (B, T), at a frame where the box texture is described by (P, v1 , v2 ). If we consider the v1 and v2 as defining a coordinate system on the texture, then the coordinates of where the ray r intersects the texture are x= |B -1 r v1 P| + |B -1 r T v1 | |B -1 r v1 v2 | -1 |B r v2 P| + |B -1 r T v2 | y= |B -1 r v1 v2 | (33) (34)
1 Thus our pixel is in the box if x and y are both within 4 of an integer, and also x and y are within the boundaries of the number of boxes contained on the board. It may be the case that for many frames of a camera, there are no boxes. For some frames, there may be partial showing of a board. For some frames the full board may...