Inpainting Video Under Constrained Camera Motion
Kedar A. Patwardhan, Student Member, IEEE, Guillermo Sapiro, Senior Member, IEEE, and Marcelo Bertalmo,
A BSTRACT A framework for inpainting missing parts of a video sequence recorded with a moving or stationary camera is presented in this work. The region to be inpainted is general: it may be still or moving, in the background or in the foreground, it may occlude one object and be occluded by some other object. The algorithm consists of a simple pre-processing stage and two steps of video inpainting. In the pre-processing stage we roughly segment each frame into foreground and background. We use this segmentation to build three image mosaics that help to produce time consistent results and also improve the performance of the algorithm by reducing the search space. In the rst video inpainting step we reconstruct moving objects in the foreground that are occluded by the region to be inpainted. To this end we ll the gap as much as possible by copying information from the moving foreground in other frames, using a priority-based scheme. In the second step, we inpaint the remaining hole with the background. To accomplish this, we rst align the frames and directly copy when possible. The remaining pixels are lled-in by extending spatial texture synthesis techniques to the spatio-temporal domain. The proposed framework has several advantages over state of the art algorithms that deal with similar types of data and constraints. It permits some camera motion, is simple to implement, is fast, does not require statistical models of background nor foreground, works well in the presence of rich and cluttered background and the
This work was partially supported by the Ofce of Naval Research, the National Science Foundation, DARPA, the National Institutes of Health, the National Geospatial-Intelligence Agency, Ram n y Cajal Program, and IP-RACINE Project. o
Electrical and Computer Engineering, University of Minnesota, Minneapolis, MN 55455.
{kedar,guille}@umn.edu, Ph: (612)-625-1343, Fax: [phone].
University Pompeu Fabra, Barcelona, Spain. [e-mail]
results show that there is no visible blurring or motion artifacts. A number of real examples taken with a consumer hand-held camera are shown supporting these ndings.
1
Video Inpainting Under Constrained Camera Motion
2
I. I NTRODUCTION AND OVERVIEW A. Introduction to the Video Inpainting problem The problem of automatic video restoration in general, and automatic object removal and modication in particular, is beginning to attract the attention of many researchers. In this paper we address a constrained but important case of video inpainting. We assume that the camera motion is approximately parallel to the plane of image projection, and the scene essentially consists of stationary background with a moving foreground, both of which may require inpainting. The algorithm described in this paper is able to inpaint objects that move in any fashion but do not change size appreciably. As we will see below, these assumptions are implicitly or explicitly present in most state of the art algorithms for video inpainting, but they still leave a very challenging task and apply to numerous scenarios. For a detailed discussion about these assumptions, including how they are actually relaxed in the real examples here presented, please refer to Section II-A. A number of algorithms for automatic still image completion have been proposed in the literature [3], [5], [6], [11]. These cannot be generalized in a straight-forward manner to address the challenging problem of video completion reported in this paper. There has also been some preliminary work on frame-by-frame partial differential equations (PDEs) based video inpainting [4], following [5]. In [4], the PDE is applied spatially, and completes the video frame-by-frame. This does not take into account the temporal information that a video provides, and its application is thereby limited. Also, the PDEs based methods interpolate edges in a smooth manner, but temporal edges are often more abrupt than spatial edges. The authors in [24] recently proposed a method for space-time completion of damaged areas in a video sequence. They pose the problem of video completion as a global optimization problem, which is inherently computationally very expensive. The work extends to space+time the pioneering technique of non-parametric sampling developed for still images by Efros and Leung [13]. This implies the assumption that objects move
in a periodic manner and also they do not signicantly change scale, because otherwise the copy and paste approach of [13] would fail. Although the results are good, they suffer from several shortcomings. Only low
3
resolution videos are shown, and over-smoothing is often observed. This is due in part to the fact that pixels are synthesized by a weighted average of the best candidates, and this averaging produces blurring. Also, the camera is always static in all the examples in that paper. Though the reason for this is not discussed, it is probably due to the fact that the authors use a very simple motion estimation procedure involving the temporal derivative. We present results comparing with their approach in the experimental section. An interesting probabilistic video modelling technique has been proposed in [10], with application to video inpainting. They dene epitomes as patch based probability models that are learnt by compiling together a large number of examples of patches from input images. These epitomes are used to synthesize data in the areas of video damage or object removal. The video inpainting results are reported to be similar to those in [24], are primarily low resolution, and over-smoothing is also observed. Very interesting work for repairing damaged video has been recently reported in [15]. Their method involves a gamut of different techniques that make the process of inpainting very complicated. There is an important amount of user interaction: the user has to manually draw the boundaries of the different depth layers of the sequence. Also, the algorithm has to learn the statistics of the background. The motion of the objects in the background is restricted to be periodic, which implies that objects also do not change scale as they move, so movement is approximately on a plane parallel to the projection plane of the camera. All the examples shown involve either a static camera or a very smooth horizontal lateral dolly type of camera motion. The results are good, although not free from artifacts. Damaged moving objects are reconstructed by synthesizing a new un-damaged object, overlaying it on the sequence, and moving it along a new, interpolated trajectory. This approach produces very noticeable artifacts where objects move in an unrealistic way (for instance a walking person seems at some points to oat over the ground). We here present results for videos of the same type as those in [15]. A related approach, also combining motion layer estimation and segmentation with warping and
region lling-in, has been reported in [25].
4
In [16] the authors propose a video inpainting technique also based in the non-parametric sampling of Efros and Leung [13]. Again, as in [24], this implies the assumption that objects move in a periodic manner and also they do not change scale. The authors use tracking to reduce the search space, and graphic cuts to merge the synthesized blocks. This approach can only deal with scenes from a static camera. And although the authors do not provide video examples, they report that their results suffer from artifacts at hole boundaries, and the lling process may fail when tracking is lost.
B. Key contributions Our approach is fundamentally related to the non-parametric sampling method proposed in [13] for the problem of 2-D texture synthesis. This method was further improved upon by using a priority and condence based synthesis in [11]. We adapted and extended this technique for video inpainting for the static camera case in [20]. In this paper we present the extension of our work in [20], including addressing the case when the camera moves. For this we introduce foreground, background, and optical-ow mosaics (see Section II-B), which not only help to produce good quality results, but also reduce the search space and lead to a faster implementation. Although the copy and synthesis components of the proposed framework are basically two dimensional, the whole search and metric distances fully exploit the spatio-temporal information. Our key contribution is the following: we present a simple and fast (compared with the literature) method to automatically ll-in video holes which shares the same assumptions of the state of the art works on the subject while being free of the common visual artifacts (blurring, unrealistic motion) those works present, and at the same time relaxing the static camera constraint. Figure 1 gives an overview of our technique. The subsequent sections describe in detail each step in the proposed video inpainting process.
5
Pre-processing: Segment each frame into "Static Background + Moving Foreground " Spatially align all frames, compute optical flow and create background and foreground mosaic
Motion Inpainting: Compute the highest priority location for motion filling-in Search in the foreground mosaic , to find the possible candidate frames - Search each candidate frame for best match, - Copy moving object from match to damaged frame, and - Constrain priorities for next iteration
Continue until all pixels either are filled-in or have zero-priority
Repeat this block for all frames requiring motion inpainting
Background Filling-in: Temporal copying
Priority based texture synthesis to fill in the remaining hole
Fig. 1.
Overview of the proposed video inpainting algorithm.
II. A SSUMPTIONS AND P RE -P ROCESSING A. Basic Assumptions
6
In this work we make several assumptions on the kind of video sequences we are able to restore. As mentioned above, these assumptions are implicitly or explicitly shared by most state of the art works on the subject, often in an even more restrictive fashion. Our basic assumptions are the following:
The scene essentially consists of stationary background with some moving foreground. Camera motion is approximately parallel to the plane of image projection. This restriction ensures that background objects will not (signicantly) change size, allowing for texture synthesis in the spirit of [13], which can not deal with changes in size nor perspective.
Foreground objects move in a repetitive fashion. In order to recover occluded or damaged foreground, and without the use of probabilistic models or libraries (used for instance in [1]), the vital information must be present in the video itself. Hence this periodicity assumption.
Moving objects do not signicantly change size. Again, this restriction is imposed by the use of the non-parametric texture synthesis of [13]. This constraint can be removed by using a multi-scale matching algorithm which can address the change in size when the object moves away from or towards the camera.
All the examples in this paper are taken with a hand-held camera, thereby complying with these assumptions only partially, while still producing very satisfactory results.
B. Pre-processing The simple assumptions that we make allow us to compute a rough motion condence mask Mc for each frame just by comparing it with the following frame using block-matching1 . The median shift of all the blocks in the image gives a good estimate of the camera shift in this case. Any block that has considerable shift after
1
We only take into account blocks that have no information missing.
subtracting this median camera motion is assumed to belong to the moving foreground. Hence, given that the motion of the camera does not produce any transformation of the static background besides translation, Mc
7
can be easily computed by a simple thresholding of the block-matching result. We should note that we could use of course more advanced techniques to detect moving objects and to separate foreground from background, see for example [7], [19], [21], [23] and references therein, but all the examples in this paper were obtained with the very simple method of computation for Mc described above.2 Also, we must point out that not every scene can be decomposed into foreground and background, sometimes this simple model just does not apply and the framework here presented needs to be extended. As we mentioned earlier, we use image mosaics to be able to deal with some camera motion and to speed up the inpainting process. A mosaic is a panoramic image obtained by stitching a number of frames together. In the pre-processing stage we build three mosaics: a background mosaic, a foreground mosaic, and an optical ow mosaic. The computation of Mc gives us a segmentation of the sequence into foreground and background layers, as well as a good estimate of the camera shift for each frame. We use this camera shift estimation to align (register) the frames. Each mosaic is built from the set of aligned overlapping frames in the following way: each pixel of the mosaic is the average of the overlapping components. This is straightforward in the case of the foreground and background mosaics: in Figure 3 we can see the mosaics obtained from a video sequence shown in Figure 2. For the optical ow mosaic, which contains data used for the Sum of Squared Difference (SSD) computations as shown below, we use a 2-channel image to store the horizontal and vertical components of the residual optical ow, that is, the motion vectors from which we have subtracted the camera shift. In Figure 3 we use color coding to represent the direction of these 2D vectors: green tones indicate horizontal motion and red tones indicate vertical motion. We must mention that there are more sophisticated mosaic generation techniques in the literature to handle camera motion, like [2], [12] and [22], but our simple
2
We wish to clarify that the Mc motion masks are different from the masks that indicate the area of inpainting (the hole): the latter
are given by the user, as in every (image or video) inpainting procedure.
8
Fig. 2.
Some frames of a video sequence satisfying the assumptions stated in Section II-A.
Fig. 3.
The pre-processing step: background, foreground, and optical-ow mosaics, respectively, of the sequence shown in 2.
approach has been satisfactory for the results reported here. This mosaic generation step allows us to do a quick search for possible candidate frames from where to copy information when lling-in the moving object, thereby speeding up the implementation by limiting our search to only these candidate frames instead of the entire sequence. The next section discusses the moving foreground completion step in detail.
III. M OTION I NPAINTING
Damaged Frame ( F ) Highest Priority Patch ( P ) Corresponding Mosaic Location ( Pm ) Matching Patch ( Pcand ) Foreground Mosaic
9
( f1, f2 fn ) = Candidate Frames
Fig. 4. The foreground candidate search process. First, the highest priority patch is located in the damaged frame (top), and then the mosaic is used to nd the candidate frames (f1 ,f2 ,...,fn ) from where information can be copied into the damaged frame (F ).
The algorithm consists of a pre-processing stage and two steps of video inpainting. In the rst video inpainting step, that of Motion Inpainting, we reconstruct foreground (moving) objects that are occluded by the region to be inpainted. To this end we ll the gap as much as possible by copying information from the moving foreground in another frame, using a priority-based scheme and the above mentioned three mosaics. Here we are using the technique introduced for still images by Efros and Leung in [13] and rened in [11] by Criminisi et al., so let us start this section by briey reviewing that procedure.
Pseudo-code for completing moving object in one frame ( F ): P = getHighestPriorityPatch ( F , Mc ); While ( P exists ) : Pm = MosaicCoord ( P , Cam-Shift ); Pcand = mosaicSearch ( Pm ); (f1 fn) = getCandidates ( Pcand ); q = getMatch ( F, P , (f1 fn) ); copy ( q , P ); constrainPriority ( P ); updateConfidence ( P ); updateMosaics ( P ); Repeat the above procedure for each frame requiring moving object completion.
10
Fig. 5.
Pseudocode for the motion inpainting step.
A. Review of non-parametric sampling Given the problem of inpainting an image hole in a still image I , Efros and Leung proposed in [13] a simple yet extremely effective algorithm. For each pixel P in the boundary of , consider its surrounding patch P , a square centered in P . Compare this patch, using a simple metric such as the Sum of Squared Differences (SSD), with every possible patch in the image. There will be a set of patches with small SSD distance to P . Randomly choose a patch Q from this set, and copy its central pixel Q to the current pixel
P . We have lled P , so we next proceed to the following pixel in the boundary of . Criminisi et al. noted
that the order in which the pixels of are lled is crucial, so in [11] they proposed an inpainting procedure which is basically that of Efros and Leung with a new ordering scheme that allows to restore long structures occluded by the hole . The ordering scheme proposed by Criminisi et al. is as follows. They compute a priority value for each pixel in the boundary of , and at each step the pixel chosen for lling is the one with the highest priority. For any given pixel P , its priority P r(P ) is the product of two terms: a condence term
C(P ) and a data term D(P ): P r(P ) = C(P )D(P ). The condence term C(P ) is proportional to the number
of undamaged and reliable pixels surrounding P . The data term D(P ) is high if there is an image edge arriving
11
at P , and highest if the direction of this edge is orthogonal to the boundary of . We thus get higher priority values at signicant edges that need to be continued, as in [5]. At this point, we suggest the reader to take a close look at gures 4, 5, and 6, in order to have a further understanding of our algorithm. It is important to note that data from the mosaics is not used to ll-in the damaged frames. The mosaics are only used to search for the candidate-undamaged-frames, from where we copy into the damaged frames.
B. Initial Guess Search Coming back to our problem, we start by restoring moving objects occluded by the gap in our video sequence. We want this restoration to be done by copying suitable information from other frames. But searching the entire sequence for a good match would be computationally very inefcient. Hence, we need to rst have a small set of candidate frames which can provide information to complete those moving objects. To achieve this, we rst search in the foreground mosaic, since we are inpainting foreground objects, to nd the candidate frames,i.e. a small subset of frames where we will look for the best match. This initial guess search is implemented using the following steps (refer to gures 4-6): 1) In the current damaged frame under consideration,3 nd the highest priority location P and its surrounding patch P 2) Using the already available camera shifts computed during the pre-processing step, nd the corresponding location Pm for P and also its surrounding patch (P m ) in the foreground mosaic. 3) Using P m as a template perform a search in the foreground mosaic to nd the matching patch(es)
P cand .
3
We start our lling-in with the temporally outermost damaged frame (in the 3D-video-cube) and move towards the center of the
hole in the video-cube. This approximately gives more priority to the frames that have more un-damaged information in the temporal vicinity.
4) Now, using the camera shifts and the motion condence masks for each frame, identify the frames that have motion at the location corresponding to the mosaic area specied by the matching patch(es) P cand . These frames are the candidate frames for searching a matching patch for the highest priority location
P in the current frame.
12
Now some details on the above steps. Firstly, for the data term D(P ) in the priority computation, we use the following formula:
D(k) = |( Mc )k nk | ,
(1)
where is a normalizing constant (usually 255) and nk is the normal to the hole-boundary. The inner product of the rotated gradient of Mc ,
Mc ), and the normal nk is computed using central differences.
4
Secondly,
when looking for a (2D) match for the template patch P m , we follow the approach used in [24]: we use a SSD metric involving a 5D vector value composed by the color values R, G and B and the optical ow values Vx and Vy .5 The optical ow components are computed using the simple approximation Vx =
Vy =
It Iy , It Ix
and
where I is the gray-scale frame under consideration,6 and Ix , Iy , and It are its horizontal, vertical,
and temporal derivatives respectively (computed with a very simple numerical scheme, like central differences). The optical ow components can be computed with more recent, robust and fast state of the art techniques such as [8], [9], but all our results were obtained with the very simple approximation just described. Adding optical ow to the SSD vector helps us to ensure motion consistency. example, For if the moving person in a video has his right leg going forward and left going backward, there is no way to get a similar match without using optical ow, because in a 2D image this situation would look similar (in R,G,B) to the situation when the two legs are in the same position but moving in the opposite direction (i.e., left leg moving forward and right moving backward).
4 5 6
This is simply computing the change of Mc along the boundary, so central differences are a natural choice. For simplicity, the same weight is given to each of the dimensions, though different weights might produce better results. As luminance we use the average of the three color channels.
Original Damaged Frame
Green = High Priority
Patch To Be Filled-In
13
A
A
A
A
A
B
Stopping Criterion
Motion Copied. Red = Zero Priority
Matching Patch In Another Frame
Fig. 6.
The motion inpainting scheme. Green dots indicate highest priority. Red squares indicate the patch to be inpainted (frame A)
and the corresponding best match (frame B). Areas lled with red are constrained to have zero priority.
C. Copy Foreground and Constrain Priority Once the candidate frames are identied, we perform the main process of motion inpainting (refer to Figure 6). We search each candidate frame for a best matching patch Q , the patch with minimum distance to our target patch P . Again, following [24] we use the SSD metric for the distance computation, and a 5D vector value composed of the three color components and the two optical ow components. Once the matching patch Q is found, instead of fully copying it onto the target P , we do the following. We look at Mc and copy from Q only the pixels that correspond to the moving foreground. The remaining unlled pixels of P must correspond to the background, so we dont want to ll them at this Motion Inpainting
stage. For this reason we mark them to have zero priority (i.e., disable them from any future motion-lling-in).
14
This last one is a key point of our algorithm. The separation of background and foreground is essential if the background is rich and inhomogeneous. If we copied the whole patch Q instead of only its foreground pixels, we would be assuming that whenever foreground matches foreground, their surrounding background matches as well. Such an assumption would imply that the background is more or less the same all along the trajectory of the moving foreground object(s). This is an implicit limitation present in [24] for instance.
D. Update After inpainting P , the Mc values at P are updated to the Mc values at Q . Next we update the condence
C(p) at each newly inpainted pixel p as follows: C(p) =
qP (Mc \) C(q)
|P |
,
(2)
where |P | is the area of the patch and is the region of inpainting. Finally we update the foreground and the optical ow mosaics with the newly lled-in data.
E. Ending the foreground inpainting We repeat the above steps (sections III-B, III-C and III-D) until all the pixels in the inpainting area are either lled-in or have zero priority for motion inpainting (i.e., are disabled as explained above). This is precisely our indication that moving objects have been fully inpainted in the current frame. We now repeat this process for all the frames that require motion inpainting. This gives us a sequence with only moving objects lled-in, and the rest of the missing region needs to be lled-in with background.
IV. BACKGROUND I NPAINTING Once we have nished the stage of Motion Inpainting, we enter the stage where we inpaint the background. To accomplish this we rst align the frames and directly copy whenever possible, while the remaining pixels
15
Fig. 7. The missing part of the background (left) is lled-in (right) using a priority based texture synthesis scheme derived from [11].
are lled-in by extending spatial texture synthesis techniques to the spatio-temporal domain. Lets see this in a little more detail. When there is camera motion involved, often the background is less occluded in one frame than another (see [17], [18]). When lling-in the background, we align all the frames using the pre-computed shifts, and then look for background information available in nearby frames. We then copy this temporal information using a nearest neighbor rst rule, that is, copy available information from the temporally nearest frame (for more details refer to [20]). Note that this will of course be faster and of better quality than a simple block synthesizing procedure. In cases where the occluder is stationary (refer to Figure 7), there is a considerable part of the background that remains occluded in all of the frames. This shows up as a hole in the background mosaic. We ll-in this hole directly on the background mosaic using the priority based texture synthesis scheme in [11] (extended to use a 5D vector of colors and motion as explained in the previous section). The missing information in each frame is then copied from the the inpainted background mosaic, by spatially aligning the frame with the mosaic using the pre-computed shifts. This leads to a consistent looking background throughout the sequence.
V. E XAMPLES All the videos referred to in this section have been captured using a consumer hand-held camera, providing a video resolution of 640x480 pixels per frame. The natural motion of a hand-held camera is a very com-
mon lming scenario. These and other video examples may be seen at www.tc.umn.edu/patw0007/ video-inpainting, some at full resolution (640x480), some at half resolution (320x240.) Please note that even if we display the inpainted videos at full resolution, no blurring artifacts appear, Figure 8 shows at large scale a restored frame. Also, in the video results it can be observed that inpainted moving objects have a consistent, natural looking trajectory. These results are state of the art, lacking the visual artifacts present in recent works on the subject, and with a faster and generally simpler technique.
16
Fig. 8. Left: a frame from the video in Figure 11 shown in large scale. Right: its inpainted result. Resolution is 640x480. Notice how there is no blur in the inpainted region, even at this full resolution.
In Figure 9 we created an articial rectangular hole in each frame at the same location. This presents not only a challenging task but also models practical scenarios like a camera with a damaged set of CCDs, or a speckle in the camera lens or on the lm stock. Notice also that the camera is in motion throughout the sequence. The moving person has been successfully inpainted and the lled-in background is consistent along the sequence, thanks in part to the mosaic lling process. Figure 10 shows another real-life video sequence where a moving person is occluded by a stationary pole, which also occludes considerable amount of the static background in all the frames. Notice that the camera does not move exactly parallel to the plane of projection while tracking the person of interest, which shows that our method is not very sensitive to mild relaxations of
the assumptions stated in Section II-A. We have successfully removed the pole and the motion of the person is
17
seamlessly continued through the region of inpainting. Again, Figure 7 illustrates the hole in the background due to the static occluder, which is inpainted directly on the background mosaic, as described earlier. Figure 11 shows a challenging example where the region to inpaint is a moving object that changes shape constantly. Results in Figure 12 show that our algorithm works well even when the captured video does not strictly adhere to the assumptions mentioned in Section II-A. The moving car moves at an angle to the plane of projection, thereby changing size. The occluder is removed and the lled-in background is consistent throughout the video, in spite of appreciable hand-held camera motion and small parallax. Figures 13 and 14 show a comparison between the proposed approach and results shown in [24]. It should be observed that our technique compares favorably even in the presence of the moderate dynamic background in Figure 14, though our algorithm was not designed to specically address dynamic background7 . This is achieved by incorporating optical-ow in the SSD computation for synthesizing background also. Also note the better performance of our technique in restoring small moving objects such as the hat in the womans hand, or her left leg. The inpainted region in Figure 13 is sharp and no over-smoothing is observed. The complete video inpainting algorithm was implemented using C++, on a P-4 machine, with run-times of about 15 minutes (including pre-processing) for sequences with 50 frames at 320x240 resolution (with experimental non-optimized code). The table below gives more details about the accompanying result videos.
7
Since the background is dynamic, our simple segmentation technique did not always give us the correct boundaries for the moving
person in this video. We had to manually prune the Mc masks in such cases.
18
Video Fig. 9 Fig. 10 Fig. 11 Fig. 12 Fig. 14 Fig. 13
Frame size 640 x 480 320 x 240 640 x 480 320 x 240 170 x 180 300 x 100
Mosaic size 852 x 520 487 x 257 721 x 488 321 x 241 170 x 180 300 x 100
Damaged frames 50 50 77 18 48 240
Damaged pixels
2700/frame 3800/frame 10,781/frame 2500/frame 1800/frame 1600/frame
Pre-Process time
< 3 min < 2 min < 5 min << 1 min < 1 min < 3 min
VI. C ONCLUDING R EMARKS We have presented a simple framework for lling-in video sequences in the presence of camera motion. The technique is based on combining motion based inpainting with spatial inpainting, using three image mosaics that allow us to deal with camera motion and speed up the process as well. If there are moving objects to be restored, they are lled-in rst, independently of the changing background from one frame to another. Then the background is lled-in by extending spatial texture synthesis techniques to the spatio-temporal domain. Currently we are working on removing the assumptions stated in Section II-A, to be able to deal with arbitrary camera motion (including zooms), changes of scale in the moving objects, and dynamic backgrounds. Currently our algorithm does not address complete occlusion of the moving object as in [15]. We are working towards adapting our technique to such scenarios. Also to be addressed are the automated selection of parameters (such as patch size, mosaic size, etc.), and dealing with illumination changes along the sequence. Results towards adapting to illumination changes have recently appeared as an extension of [15], see [14].
ACKNOWLEDGEMENTS The authors are thankful to Dr. E. Shechtman and Prof. M. Irani for permitting the use of their original videos in [24] to generate the comparative results in Figure 14 and Figure 13. The presentation of this paper has tremendously beneted from the critical comments of the anonymous reviewers.
R EFERENCES
19
[1] D. Anguelov, P. Srinivasan, D. Koller, S. Thrun, J. Rodgers, and J. Davis, Scape: Shape completion and animation of people, ACM Transactions on Graphics (SIGGRAPH 2005), August 2005. [2] S. Baker, R. Szeliski, and P. Anandan, A layered approach to stereo reconstruction, cvpr, vol. 00, p. 434, 1998. [3] C. Ballester, V. Caselles, and J. Verdera, Dissoclusion by joint interpolation of vector elds and gray levels, SIAM Multiscale Modeling and Simulation, vol. 2, pp. 80123, 2003. [4] M. Bertalmio, A. L. Bertozzi, and G. Sapiro, Navier-stokes, uid dynamics, and image and video inpainting, Proc. IEEE Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 355362, 2001. [5] M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester, Image inpainting, Computer Graphics (SIGGRAPH 2000), pp. 417424, 2000. [6] M. Bertalmio, L. Vese, G. Sapiro, and S. Osher, Simultaneous texture and structure image inpainting, IEEE Transactions on Image Processing, vol. 12:8, pp. 882889, 2002. [7] M. J. Black and P. Anandan, The robust estimation of multiple motions: Parametric and piecewise-smooth ow elds, Computer Vision and Image Understanding, CVIU, vol. 63:1, pp. 75104, 1996. [8] A. Bruhn and J. Weickert, Towards ultimate motion estimation: Combining highest accuracy with real-time performance, in IEEE International Conference on Computer Vision (ICCV), 2005. [9] V. Caselles, L. Igual, and L. Garrido, A contrast invariant approach to motion estimation. in Scale Space, 2005. Springer-Verlag, 2005. [10] V. Cheung, B. J. Frey, and N. Jojic, Video epitomes, in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, 2005, pp. 4249. [11] A. Criminisi, P. Perez, and K. Toyama, Region lling and object removal by exemplar-based inpainting, IEEE Transactions on Image Processing, vol. 9, pp. 12001212, 2004. [12] J. Davis, Mosaics of scenes with moving o...