lecture11 - Robert Collins CSE454, PSU Lecture 11: Stereo...

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: Robert Collins CSE454, PSU Lecture 11: Stereo Algorithms Robert Collins CSE454, PSU Recall: Simple Stereo System Y Z (X,Y,Z) left camera ( , ) ( , ) right camera Image coords of point (X,Y,Z) Left Camera: Right Camera: Camps, PSU X Robert Collins CSE454, PSU Recall: Stereo Disparity Left camera Right camera Stereo Disparity depth baseline disparity Important equation! Robert Collins CSE454, PSU Recall: Stereo Disparity Left camera Right camera Note: Depth and stereo disparity are inversely proportional depth disparity Important equation! Robert Collins CSE454, PSU Stereo Example Left Image Right Image From Middlebury stereo evaluation page http://www.middlebury.edu/stereo/ Robert Collins CSE454, PSU Stereo Example Disparity values (0-64) Left image Right image Note how disparity is larger (brighter) for closer surfaces. Robert Collins CSE454, PSU Computing Disparity •  Correspondence Problem: –  Determining which pixel in the right image corresponds to each pixel in the left image. –  Disp = x_coord(left) - x_coord(right) Recall our discussion of computing correspondences of image patches ( Lecture 7 ). SSD - sum of squared difference measure NCC - normalized cross correlation measure Camps, PSU Robert Collins CSE454, PSU Epipolar Constraint Important Concept: For stereo matching, we don’t have to search the whole 2D right image for a corresponding point. The “epipolar constraint” reduces the search space to a one-dimensional line. Robert Collins CSE454, PSU Recall : Simple Stereo System P Left camera Right camera Epipolar line Same Y Coord! Robert Collins CSE454, PSU Matching using Epipolar Lines Left Image Right Image For a patch in left image Compare with patches along same row in right image Match Score Values Robert Collins CSE454, PSU Matching using Epipolar Lines Left Image Right Image Select patch with highest match score. Repeat for all pixels in left image. Match Score Values Robert Collins CSE454, PSU Example: 5x5 windows NCC match score Computed disparities Black pixels: bad disparity values, or no matching patch in right image Ground truth Robert Collins CSE454, PSU Occlusions: No matches Left image Right image Robert Collins CSE454, PSU Effects of Patch Size 5x5 patches 11x11patches Smoother in some areas Loss of finer details Robert Collins CSE454, PSU Adding Inter-Scanline Consistency So far, each left image patch has been matched independently along the right epipolar line. This can lead to errors. We would like to enforce some consistency among matches in the same row (scanline). Robert Collins CSE454, PSU Disparity Space Image First we introduce the concept of DSI. The DSI for one row represents pairwise match scores between patches along that row in the left and right image. Pixels along left scanline Pixel i Pixels along right scanline Pixel j C(i,j) = Match score for patch centered at left pixel i with patch centered at right pixel j. Robert Collins CSE454, PSU Disparity Space Image (DSI) Left Image Right Image Dissimilarity Values (1-NCC) or SSD Robert Collins CSE454, PSU Disparity Space Image (DSI) Left Image Right Image Dissimilarity Values (1-NCC) or SSD Robert Collins CSE454, PSU Disparity Space Image (DSI) Left Image Right Image Dissimilarity Values (1-NCC) or SSD Robert Collins CSE454, PSU Disparity Space Image (DSI) Left Image DSI Dissimilarity Values Enter each vector of match scores as a column in the DSI Robert Collins CSE454, PSU Disparity Space Image Left scanline Right scanline Robert Collins CSE454, PSU Disparity Space Image Left scanline Invalid entries due to constraint that disparity >= low value (0 in this case) Right scanline Invalid entries due to constraint that disparity <= high value 64 in this case) Robert Collins CSE454, PSU Disparity Space Image N cols in left scanline M cols in right scanline If we rearrange the diagonal band of valid values into a rectangular array (in this case of size 64 x N), that is what is traditionally known as the DSI However, I’m going to keep the full image around, including invalid values (I think it is easier to understand the pixel coordinates involved) coordinate in left scanline (e.g. N) Disparity (e.g. 64) Disparity Space Image Robert Collins CSE454, PSU DSI and Scanline Consistency Assigning disparities to all pixels in left scanline now amounts to finding a connected path through the DSI Start End Robert Collins CSE454, PSU Lowest Cost Path We would like to choose the “best” path. Want one with lowest “cost” (Lowest sum of dissimilarity scores along the path) ? ? ? Robert Collins CSE454, PSU Constraints on Path It is common to impose an ordering constraint on the path. Intuitively, the path is not allowed to “double back” on itself. Robert Collins CSE454, PSU Ordering Constraint A B C D A B C ABCD D ABCD ABCD DABC Ordering constraint… …and its failure Robert Collins CSE454, PSU Dealing with Occlusions Left scanline Right scanline … … Robert Collins CSE454, PSU Dealing with Occlusions Left scanline Right scanline … Match Match Occluded from right scanline Match … Occluded from left scanline Robert Collins CSE454, PSU An Optimal Scanline Strategy •  find best path, taking into account ordering constraint and the possibility of occlusions, refer to the following paper: Cox, Hingorani, Rao, Maggs, “A Maximum Likelihood Stereo Algorithm,” Computer Vision and Image Understanding, Vol 63(3), May 1996, pp.542-567. General idea: use dynamic programming on each row to find optimal disparities. ...
View Full Document

Ask a homework question - tutors are online