Lecture-18-h

Lecture-18-h - Lecture-18 Block-based & Token-based...

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: Lecture-18 Block-based & Token-based Optical Flow Block Matching Frame k-1 Frame k 8X8 16X16 Origin is at bottom right corner 1 Block Matching • For each 8X8 block, centered around pixel (x,y) in frame k, Bk – Obtain 16X16 block in frame k-1, enclosing (x,y), Bk-1 – Compute Sum of Squares Differences (SSD) between 8X8 block, Bk, and all possible 8X8 blocks in Bk-1 – The 8X8 block in Bk-1 centered around (x’,y’), which gives the least SSD is the match – The displacement vector (optical flow) is given by u=xx’; v=y-y’ Sum of Squares Differences (SSD) -7 7 2 (U ( x, y ),V ( x, y )) = arg min u= 0K- 8 Â Â ( f k ( x + i, y + j ) - f k -1 ( x + i + u, y + j + v )) v = 0K 8 Frame k-1 i= 0 j = 0 Frame k † 8X8 16X16 Origin is at bottom right corner 2 Minimum Absolute Difference (MAD) -7 7 (U ( x, y ),V ( x, y )) = arg min u= 0K- 8 Â Â | ( f k ( x + i, y + j ) - f k -1 ( x + i + u, y + j + v )) | v = 0K 8 i= 0 j = 0 † Maximum Matching Pixel Count (MPC) Ï1 if | f k ( x, y ) - f k -1( x + u, y + v ) |£ t T ( x, y; u, v ) = Ì Otherwise Ó0 -7 7 (U ( x, y ),V ( x, y )) = arg max u= 0K- 8 Â Â T ( x + i, y + j; u, v ) v = 0K 8 i= 0 j = 0 † 3 Cross Correlation -7 7 (U,V ) = arg maxu= 0K- 8 Â Â ( f k ( x + i, y + j )).( f k -1 ( x + i + u, y + j + v )) v = 0K 8 † i= 0 j = 0 Potential problem -7 2 7 (U ( x, y ),V ( x, y )) = arg min u= 0K- 8 Â Â ( f k ( x + i, y + j ) - f k -1 ( x + i + u, y + j + v )) v = 0K 8 i= 0 j = 0 -7 7 ( 2 2 (U ( x, y ),V ( x, y )) = arg min u= 0K- 8 Â Â f k - 2 f k f k -1 + f k -1 v = 0K 8 † i= 0 j = 0 ) † Normalized Correlation j = -7 7 Â Â ( f ( x + i, y + j)).( f k (U,V ) = arg maxu= 0K- 8 v = 0K 8 k -1 ( x + i + u, y + j + v )) i= 0 j = 0 -7 7 ÂÂ f k -1 ( x + i + u, y + j + v ). f k -1 ( x + i + u, y + j + v ) i= 0 j = 0 † 4 Mutual Correlation (U,V ) = arg maxu= 0K- 8 v = 0K 8 -7 7 1 Â Â ( f k ( x + i, y + j ) - m1).( f k-1( x + i + u, y + j + v ) - m2 ) 64s1s 2 i= 0 j = 0 † Sigma and mu are standard deviation and mean of patch-1 and patch-2 respectively Correlation Surface -7 7 C (u , y ) = Â Â ( f k ( x + i, y + j )).( f k -1 ( x + i + u , y + j + v) ) i =0 j =0 mission reference Correlation surface 5 Correlation Using FFT -7 7 C (u , y ) = Â Â ( f k ( x + i, y + j )).( f k -1 ( x + i + u , y + j + v) ) i =0 j =0 c(u , v) = f ( x, y ) ƒ g ( x, y ) Fourier transform convolution C ( w1 , w2 ) = F ( w1 , w2 ).G ( w1 , w2 ) Append smaller patch with zeros to make it equal to bigger patch Find FFT of reference (f(x,y) and mission (g(x,y)) patches multiply two FFTs Find inverse FFT of the product, this will give you a correlation surface Find the peak in the correlation surface This method is faster for a large patch size Phase Correlation c ( x, y ) = f ( x, y ) ƒ g(- x, - y ) C( w1,w 2 ) = F ( w1,w 2 ).G* ( w1,w 2 ) † † † † Now F ( w1,w 2 ).G* ( w1,w 2 ) ˜ C( w1,w 2 ) = | F ( w1,w 2 ).G* ( w1,w 2 ) | Assume f ( u, v ) = g( x + u, y + v ) Then F ( w1,w 2 ) = G( w1,w 2 )e-( i( w1 u + w2 v )) F ( w1,w 2 ).G* ( w1,w 2 ) G( w1,w 2 )e-( i( w1 u + w2 v )) .G* ( w1,w 2 ) ˜ C( w1,w 2 ) = = = e-( i( w1 u + w2 v )) * | F ( w1,w 2 ).G ( w1,w 2 ) | | G( w1,w 2 )e-( i( w1 u + w2 v )) .G* ( w1,w 2 ) | † c ( x, y ) = d( x - u, y - v ) So it is very easy to find the peak in the correlation surface † 6 Issues with Correlation • Patch Size • Search Area • How many peaks • Should use pyramids here too for large displacements Token-based Optical Flow • Find tokens – Moravec’s interest operator – Corners – Edges • Solve point correspondence 7 Point Correspondence • Given n video frames taken at different time instants and m points in each frame, the motion correspondence problem deals with a mapping of a point in one frame to another point in the second frame such that no two points map onto the same point. Key Points • Constraints‡ Cost Function • Algorithm‡ Minimize the cost function 8 Constraints Maximum Velocity Common Motion Consistent Match Minimum Velocity Model Proximal Uniformity Constraint • Most objects in the real world follow smooth paths and cover small distance in a small time. – Given a location of point in a frame, its location in the next fame lies in the proximity of its previous location. – The resulting trajectories are smooth and uniform. 9 Proximal Uniformity Constraint Xr Xq Xp E F Proximal Uniformity Constraint Establish correspondence by minimizing: k -1 p k q d( X , X , X k +1 r )= k k k || X p-1 X q - X q X rk +1 || m ÂÂ| X x =1 z =1 † + m k -1 x k y k y X -X X k +1 z || k || X q X rk +1 || m m ÂÂ| X k y X zk +1 || x =1 z =1 Initial correspondence is known, for each x in the denominator y is known. 10 Greedy Algorithm • For k=2 to n-1 do • Construct M, an mxm matrix, with the points from k-th frame along the rows and points from (k+1)-th frame along the columns. Let k k M [i, j ] = d ( X p -1 , X q , X rk +1 ) • for a=1 to m do – Identify the minimum element [i, li ] in each row i of M m m – Compute priority matrix, B, such that B[i, li ] = Â M [i, j ] + Â M [k , li ] j =1, j ≠ l k =1, k ≠ i for each i. – Select [i, l i ] pair with highest priority value and make f k (i ) = l i – Mask row i and column l from M. i i http://www.cs.ucf.edu/~vision/papers/PAPER3.PDF M[i,j] j È.6 .3˘ M =Í ˙ Î.7 .2˚ i * È B=Í Î .8˘ 1˙ ˚ 11 Tracking Finger Tips 12 ...
View Full Document

Ask a homework question - tutors are online