Unformatted text preview: Lecture18
Blockbased & Tokenbased Optical
Flow Block Matching
Frame k1 Frame k 8X8
16X16 Origin is at bottom right corner 1 Block Matching
• For each 8X8 block, centered around pixel (x,y) in
frame k, Bk
– Obtain 16X16 block in frame k1, enclosing (x,y), Bk1
– Compute Sum of Squares Differences (SSD) between
8X8 block, Bk, and all possible 8X8 blocks in Bk1
– The 8X8 block in Bk1 centered around (x’,y’), which
gives the least SSD is the match
– The displacement vector (optical flow) is given by u=xx’; v=yy’ Sum of Squares Differences
(SSD)
7 7 2 (U ( x, y ),V ( x, y )) = arg min u= 0K 8 Â Â ( f k ( x + i, y + j )  f k 1 ( x + i + u, y + j + v ))
v = 0K 8 Frame k1 i= 0 j = 0 Frame k †
8X8
16X16 Origin is at bottom right corner 2 Minimum Absolute Difference
(MAD) 7 7 (U ( x, y ),V ( x, y )) = arg min u= 0K 8 Â Â  ( f k ( x + i, y + j )  f k 1 ( x + i + u, y + j + v )) 
v = 0K 8 i= 0 j = 0 † Maximum Matching Pixel Count
(MPC)
Ï1 if  f k ( x, y )  f k 1( x + u, y + v ) £ t
T ( x, y; u, v ) = Ì
Otherwise
Ó0
7 7 (U ( x, y ),V ( x, y )) = arg max u= 0K 8 Â Â T ( x + i, y + j; u, v )
v = 0K 8 i= 0 j = 0 † 3 Cross Correlation
7 7 (U,V ) = arg maxu= 0K 8 Â Â ( f k ( x + i, y + j )).( f k 1 ( x + i + u, y + j + v ))
v = 0K 8 † i= 0 j = 0 Potential problem
7 2 7 (U ( x, y ),V ( x, y )) = arg min u= 0K 8 Â Â ( f k ( x + i, y + j )  f k 1 ( x + i + u, y + j + v ))
v = 0K 8 i= 0 j = 0
7 7 ( 2 2 (U ( x, y ),V ( x, y )) = arg min u= 0K 8 Â Â f k  2 f k f k 1 + f k 1
v = 0K 8 † i= 0 j = 0 ) † Normalized Correlation
j = 7 7 Â Â ( f ( x + i, y + j)).( f
k (U,V ) = arg maxu= 0K 8
v = 0K 8 k 1 ( x + i + u, y + j + v )) i= 0 j = 0
7 7 ÂÂ f k 1 ( x + i + u, y + j + v ). f k 1 ( x + i + u, y + j + v ) i= 0 j = 0 † 4 Mutual Correlation (U,V ) = arg maxu= 0K 8
v = 0K 8 7 7
1
Â Â ( f k ( x + i, y + j )  m1).( f k1( x + i + u, y + j + v )  m2 )
64s1s 2 i= 0 j = 0 † Sigma and mu are standard deviation and mean of patch1 and patch2
respectively Correlation Surface
7 7 C (u , y ) = Â Â ( f k ( x + i, y + j )).( f k 1 ( x + i + u , y + j + v) )
i =0 j =0 mission reference Correlation surface 5 Correlation Using FFT
7 7 C (u , y ) = Â Â ( f k ( x + i, y + j )).( f k 1 ( x + i + u , y + j + v) )
i =0 j =0 c(u , v) = f ( x, y ) ƒ g ( x, y )
Fourier transform convolution C ( w1 , w2 ) = F ( w1 , w2 ).G ( w1 , w2 ) Append smaller patch with zeros to make it equal to bigger patch
Find FFT of reference (f(x,y) and mission (g(x,y)) patches
multiply two FFTs
Find inverse FFT of the product, this will give you a correlation surface Find the peak in the correlation surface
This method is faster for a large patch size Phase Correlation
c ( x, y ) = f ( x, y ) ƒ g( x,  y ) C( w1,w 2 ) = F ( w1,w 2 ).G* ( w1,w 2 )
† † †
†
Now F ( w1,w 2 ).G* ( w1,w 2 )
˜
C( w1,w 2 ) =
 F ( w1,w 2 ).G* ( w1,w 2 ) 
Assume
f ( u, v ) = g( x + u, y + v )
Then
F ( w1,w 2 ) = G( w1,w 2 )e( i( w1 u + w2 v )) F ( w1,w 2 ).G* ( w1,w 2 )
G( w1,w 2 )e( i( w1 u + w2 v )) .G* ( w1,w 2 )
˜
C( w1,w 2 ) =
=
= e( i( w1 u + w2 v ))
*
 F ( w1,w 2 ).G ( w1,w 2 )   G( w1,w 2 )e( i( w1 u + w2 v )) .G* ( w1,w 2 ) 
†
c ( x, y ) = d( x  u, y  v ) So it is very easy to find the peak in the correlation surface
† 6 Issues with Correlation
• Patch Size
• Search Area
• How many peaks
• Should use pyramids here too for large
displacements Tokenbased Optical Flow
• Find tokens
– Moravec’s interest operator
– Corners
– Edges • Solve point correspondence 7 Point Correspondence
• Given n video frames taken at different time
instants and m points in each frame, the
motion correspondence problem deals with
a mapping of a point in one frame to
another point in the second frame such that
no two points map onto the same point. Key Points
• Constraints‡ Cost Function
• Algorithm‡ Minimize the cost function 8 Constraints Maximum Velocity Common Motion Consistent Match Minimum
Velocity Model Proximal Uniformity Constraint
• Most objects in the real world follow
smooth paths and cover small distance in a
small time.
– Given a location of point in a frame, its location
in the next fame lies in the proximity of its
previous location.
– The resulting trajectories are smooth and
uniform. 9 Proximal Uniformity Constraint
Xr
Xq
Xp
E
F Proximal Uniformity Constraint
Establish correspondence by minimizing:
k 1
p k
q d( X , X , X k +1
r )= k
k
k
 X p1 X q  X q X rk +1 
m ÂÂ X
x =1 z =1 † + m
k 1
x k
y k
y X X X k +1
z  k
 X q X rk +1 
m m ÂÂ X k
y X zk +1  x =1 z =1 Initial correspondence is known, for each x in the denominator
y is known. 10 Greedy Algorithm
• For k=2 to n1 do
• Construct M, an mxm matrix, with the points from kth frame along the
rows and points from (k+1)th frame along the columns. Let
k
k
M [i, j ] = d ( X p 1 , X q , X rk +1 ) • for a=1 to m do
– Identify the minimum element [i, li ] in each row i of M
m
m
– Compute priority matrix, B, such that B[i, li ] = Â M [i, j ] + Â M [k , li ]
j =1, j ≠ l
k =1, k ≠ i
for each i.
– Select [i, l i ] pair with highest priority value and make f k (i ) = l
i
– Mask row i and column l from M.
i i http://www.cs.ucf.edu/~vision/papers/PAPER3.PDF M[i,j]
j È.6 .3˘
M =Í
˙
Î.7 .2˚
i * È
B=Í
Î .8˘
1˙
˚ 11 Tracking Finger Tips 12 ...
View
Full
Document
This note was uploaded on 06/12/2011 for the course CAP 5415 taught by Professor Staff during the Fall '08 term at University of Central Florida.
 Fall '08
 Staff

Click to edit the document details