Lecture-19-h - Lecture-19 Structure from Motion Problem •...

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: Lecture-19 Structure from Motion Problem • Given optical flow or point correspondences, compute 3-D motion (translation and rotation) and depth. 1 Main Points • • • • • What projection? How many points? How many frames? Single Vs multiple motions. Mostly non-linear problem. Displacement Model Point Correspondences 2 3-D Rigid Motion (displacement) È X ¢˘ È X ˘ È 1 - a b ˘ È X ˘ ÈTX ˘ ÍY ¢ ˙ = R ÍY ˙ + T = Í a 1 - g ˙ ÍY ˙ + ÍT ˙ Í˙ Í˙ Í ˙Í ˙ Í Y ˙ ÍZ ¢ ˙ ÍZ ˙ Í- b g 1 ˙ ÍZ ˙ ÍTZ ˙ Î˚ Î˚ Î ˚Î ˚ Î ˚ X ¢ = X - aY + bZ + TX Y ¢ = aX + Y - gZ + TY Z ¢ = - bX + gY + Z + TZ Orthographic Projection (displacement model) X ¢ = X - aY + bZ + TX Y ¢ = aX + Y - gZ + TY Z ¢ = - bX + gY + Z + TZ x¢ = x - ay + bZ + TX y¢ = ax + y - gZ + TY 3 Perspective Projection (displacement) X ¢ = X - aY + bZ + TX Y ¢ = aX + Y - gZ + TY Z ¢ = - bX + gY + Z + TZ TX Z x¢ = T - bx + gy + 1 + Z Z T ax + y - g + Y Z y¢ = T - bx + gy + 1 + Z Z x - ay + b + Instantaneous Velocity Model Optical Flow 4 3-D Rigid Motion È X ¢ - X ˘ È 0 - a b ˘ È X ˘ ÈTX ˘ ÍY ¢ - Y ˙ = Í a 0 - g ˙ ÍY ˙ + ÍTY ˙ Í ˙Í ˙Í ˙ Í ˙ Í Z ¢ - Z ˙ Í- b g 0 ˙ Í Z ˙ ÍTZ ˙ Î ˚Î ˚Î ˚ Î ˚ È X ¢˘ È 1 - a b ˘ È X ˘ ÈTX ˘ ÍY ¢ ˙ = Í a 1 - g ˙ ÍY ˙ + ÍT ˙ Í˙Í ˙Í ˙ Í Y ˙ ÍZ ¢ ˙ Í- b g 1 ˙ ÍZ ˙ ÍTZ ˙ Î˚Î ˚Î ˚ Î ˚ È X ¢˘ Ê È 0 ÍY ¢ ˙ = Á Í a Í ˙ ÁÍ Í Z ¢ ˙ Á Í- b Î ˚ ËÎ -a 0 g & ÈX ˘ È 0 Í& ˙ Í ÍY ˙ = Í W 3 Í Z ˙ Í- W & 2 Î˚Î b ˘ È1 0 0˘ ˆ È X ˘ ÈTX ˘ ˜ - g ˙ + Í0 1 0˙ ˜ ÍY ˙ + ÍTY ˙ ˙Í ˙Í ˙ Í ˙ 0 ˙ Í0 0 1˙ ˜ Í Z ˙ ÍTZ ˙ ˚Î ˚ ¯Î ˚ Î ˚ - W3 0 W1 W 2 ˘ È X ˘ ÈV1 ˘ - W1 ˙ ÍY ˙ + ÍV2 ˙ ˙Í ˙ Í ˙ 0 ˙ Í Z ˙ ÍV3 ˙ ˚Î ˚ Î ˚ & X = W 2 Z - W 3Y + V1 & Y = W X - W Z +V 3 1 2 & Z = W1Y - W 2 X + V3 3-D Rigid Motion & X = W 2 Z - W 3Y + V1 & Y = W X - W Z +V 3 1 2 & Z = W1Y - W 2 X + V3 & X = W¥ X + V ÈX ˘ X = ÍY ˙, Í˙ ÍZ ˙ Î˚ ÈW1 ˘ W = ÍW 2 ˙ Í˙ ÍW 3 ˙ Î˚ Cross Product 5 Orthographic Projection & X = W 2 Z - W 3Y + V1 & Y = W X - W Z +V 3 1 y =Y x=X 2 & Z = W1Y - W 2 X + V3 & u = x = W 2 Z - W 3 y + V1 & v = y = W 3 x - W1Z + V2 (u,v) is optical flow Perspective Projection (arbitrary flow) fX Z fY y= Z & u=x= & X = W 2 Z - W 3Y + V1 & Y = W X - W Z +V 3 1 & & fZX - fXZ & & X Z -x Z2 Z Z & & & & fZY - fYZ Y Z & v= y= = f -y Z2 Z Z x= 2 & Z = W1Y - W 2 X + V3 =f & & W Z - W 3Y + V1 W Y - W 2 X + V3 X Z -x = f 2 -x 1 Z Z Z Z & & W X - W1Z + V2 W Y - W 2 X + V3 Y Z & v= y= f -y = f 3 -y 1 Z Z Z Z & u=x= f V V1 W W + W 2 ) - 3 x - W 3 y - 1 xy + 2 x 2 Z Z f f V V W W v = f ( 2 - W1 ) + W 3 x - 3 y + 2 xy - 1 y 2 Z Z f f u= f( 6 Perspective Projection (optical flow) V V1 W W + W 2 ) - 3 x - W 3 y - 1 xy + 2 x 2 Z Z f f V V W W v = f ( 2 - W1 ) + W 3 x - 3 y + 2 xy - 1 y 2 Z Z f f u= f( fV1 - V3 x W W + fW 2 - W 3 y - 1 xy + 2 x 2 Z f f fV - V y W W v = 2 3 - fW1 + W 3 x + 2 xy - 1 y 2 Z f f u= Pure Translation (FOE) fV1 - V3 x Z fV - V y =23 Z u (T ) = v (T ) x0 = f V1 V , y0 = f 2 V3 V3 V3 Z V = ( y0 - y ) 3 Z u (T ) = ( x0 - x) v (T ) 7 Pure Translation (FOE) fV1 - V3 x Z fV - V y v (T ) = 2 3 Z fV u (T ) = 1 Z fV v (T ) = 2 Z u (T ) = If V3=0 Pure Translation (FOE) • If V3 is not zero, the flow field is radial, and all vectors point towards (or away from) a single point. If V3=0, the flow field is parallel. • The length of flow vectors is inversely proportional to the depth, if V3 is not zero, then it is also proportional to the distance between p and p0. 8 Pure Translation (FOE) • p0 is the vanishing point of the direction of translation. • p0 is the intersection of the ray parallel to the translation vector with the image plane. 9 Structure From Motion ORTHOGRAPHIC PROJECTION Orthographic Projection (displacement) È X ¢˘ È X ˘ È 1 - a b ˘ È X ˘ ÈTX ˘ ÍY ¢ ˙ = R ÍY ˙ + T = Í a 1 - g ˙ ÍY ˙ + ÍTY ˙ Í˙ Í˙ Í ˙Í ˙ Í Í˙ Í˚ Í- b g 1 ˙ Í Z ˙ ÍTZ ˙ ÎZ ¢ ˚ ÎZ ˙ Î ˚Î ˚ Î ˚ x ¢ = x - ay + bZ + TX y ¢ = ax + y - gZ + TY 10 Simple Method • Two Steps Method x ¢ = x - ay + bZ + TX y ¢ = ax + y - gZ + TY -Assume depth is known, compute motion È˘ a Í Íb È x¢ - x ˘ È - y Z 0 1 0 ˘ Í Í y¢ - y ˙ = Í x 0 - Z 0 1˙ Íg Î ˚Î ˚T Íx Í ÍTy Î ˙ ˙ ˙ ˙ ˙ ˙ ˙ ˚ Simple Method -Assume motion is known, refine depth È x¢ - x - ay - Tx ˘ Èb ˘ Í- g ˙[Z ]= Í y¢ - y - ax - T ˙ y˚ Î˚ Î 11 Structure from Motion:Perspective Projection Heeger & Jepson sfm method Three Step Algorithm • Find Translation using search. • Find Rotation using least squares fit. • Find Depth. 12 Heeger & Jepson sfm method V V1 + W 2 ) + 3 x + W 3 y + W1 xy - W 2 x 2 Z Z V V v = -( 2 - W1 ) - W 3 x + 3 y - W 2 xy + W1 y 2 Z Z u = -( ÈV1 ˘ È W1 ˘ Èu ( x, y )˘ 1 È- 1 0 x ˘ Í ˙ È xy - (1 + x 2 ) y ˘ Í ˙ ˙ W2 Ív( x, y ) ˙ = Z ( x, y ) Í 0 - 1 y ˙ ÍV2 ˙ + Í 1 + y 2 - xy - x ˚ Í ˙ Î ˚ Î ˚ ÍV ˙ Î ÍW 3 ˙ Î 3˚ Î˚ Heeger & Jepson sfm method È xy - (1 + x 2 ) y ˘ Èu ( x, y )˘ 1 È- 1 0 x ˘ ˙W Ív ( x , y ) ˙ = z ( x , y ) Í 0 - 1 y ˙ V + Í 2 Î ˚ Î ˚ Î1 + y - xy - x ˚ Q(x, y ) = p(x,y)A( x, y )V + B( x, y )W One point (x,y) 13 Heeger & Jepson sfm method Q(x, y ) = p(x,y)A( x, y )V + B( x, y )W 0 È Q( x1 , y1 ) ˘ È A( x1 , y1 )V K K K ˘ È p ( x1 , y1 ) ˘ ÈB( x1 , y1 ) ˘ Í ˙Í ˙Í ˙ ˙Í M M M M M Í ˙Í ˙Í ˙ ˙Í Í ˙=Í ˙Í ˙W ˙+Í M M M M M Í ˙Í ˙Í ˙ ˙Í M M M Í ˙Í ˙ Í p ( xn , y n ) ˙ Í ˙ ÍQ( xn , y n ) ˙ Í ˙ Í B ( xn , y n ) ˙ 0 K K K A ( xn , y n ) V ˙ Í ˚Î Î ˚Î ˚Î ˚ n points Heeger & Jepson sfm method 0 B( x1 , y1 ) ˘ È p ( x1 , y1 ) ˘ È Q( x1 , y1 ) ˘ È A( x1 , y1 )V K K K Í ˙Í ˙Í ˙ M M M M Í ˙Í ˙Í ˙ Í ˙=Í ˙Í ˙ M M M M Í ˙Í ˙Í ˙ M M M Í ˙Í ˙ Í p ( xn , y n ) ˙ ÍQ( xn , y n ) ˙ Í 0 K K K A ( xn , y n ) V B ( xn , y n ) ˙ Í W ˙ ˚ Î ˚Î ˚Î Q = C(V )q 14 Heeger & Jepson sfm method E (V, q) = Q - C(V )q E(V ) = QT C ^ (V ) 2 2 C ^ (V ) Orthogonal complement to C(V ) Find translation by search. C(V ) = C (V )U(V ) QR decomposition Orthonormal & 2 Upper triangular E (V, q) = Q - C (V )U(V )q E (V, q) = Q - C (V ) q 2 minimize ˆ q = C T ( V )Q 15 E (V ) = Q - C (V ) C T (V )Q T 2 E (V ) = ( I - C (V ) C (V ))Q T E(V ) = Q C (V ) ^ 2 Null space 2 Translation Unit vector translation can be represented by spherical coordinates: V = (sin q cos f , sin q sin f , cosq ) 0 £ q £ 90 0 £ f £ 360 Slant Tilt Only half of sphere can be considered 16 Spherical Coordinates Z (x,y,z) q 1 X f Y Rotation Q(x, y ) = p(x,y)A( x, y )V + B( x, y )W d T ( x, y, V )Q(x, y ) = d T ( x, y, V )B( x, y )W A d T ( x, y, V ) is perpendicular to ( x, y )V 17 Rotation Find rotation by least squares fit d T ( x1 , y1 , V )Q(x1 , y1 ) = d T ( x1 , y1 , V )B( x1 , y1 )W M d T ( xn , y n , V ) Q( x n , y n ) = d T ( xn , y n , V ) B ( xn , y n ) W Depth Find depth for each pixel (x,y) from following eqs V V1 + W 2 ) + 3 x + W 3 y + W1 xy - W 2 x 2 Z Z V V v = -( 2 - W1 ) - W 3 x + 3 y - W 2 xy + W1 y 2 Z Z u = -( 18 Synthetic Image Optical Flow 19 Computed Depth Map Synthetic Image 20 Optical Flow Translation Search Space 21 Novel View Generated from Reconstructed Depth Another Novel View Generated from Reconstructed Depth 22 ...
View Full Document

This note was uploaded on 06/12/2011 for the course CAP 5415 taught by Professor Staff during the Fall '08 term at University of Central Florida.

Ask a homework question - tutors are online