Lecture-8 - Lecture-8 Structure from Motion Problem • Given optical flow or point correspondences compute 3-D motion(translation and rotation and

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: Lecture-8 Structure from Motion Problem • Given optical flow or point correspondences, compute 3-D motion (translation and rotation) and shape (depth). 1 3-D Rigid Motion (displacement) È X ¢˘ È X ˘ È 1 - a b ˘ È X ˘ ÈTX ˘ ÍY ¢ ˙ = R ÍY ˙ + T = Í a 1 - g ˙ ÍY ˙ + ÍT ˙ Í˙ Í˙ Í ˙Í ˙ Í Y ˙ ÍZ ¢ ˙ ÍZ ˙ Í- b g 1 ˙ ÍZ ˙ ÍTZ ˙ Î˚ Î˚ Î ˚Î ˚ Î ˚ X ¢ = X - aY + bZ + TX Y ¢ = aX + Y - gZ + TY Z ¢ = - bX + gY + Z + TZ Orthographic Projection (displacement model) X ¢ = X - aY + bZ + TX Y ¢ = aX + Y - gZ + TY Z ¢ = - bX + gY + Z + TZ x¢ = x - ay + bZ + TX y¢ = ax + y - gZ + TY 2 Perspective Projection (displacement) X ¢ = X - aY + bZ + TX Y ¢ = aX + Y - gZ + TY Z ¢ = - bX + gY + Z + TZ X - aY + bZ + TX -bX + gY + Z + TZ aX + Y - gZ + TY y¢ = -bX + gY + Z + TZ x¢ = TX Z x¢ = T - bx + gy + 1 + Z Z T ax + y - g + Y Z y¢ = T - bx + gy + 1 + Z Z x - ay + b + † Instantaneous Velocity Model Optical Flow 3 3-D Rigid Motion È X ¢ - X ˘ È 0 - a b ˘ È X ˘ ÈTX ˘ ÍY ¢ - Y ˙ = Í a 0 - g ˙ ÍY ˙ + ÍTY ˙ Í ˙Í ˙Í ˙ Í ˙ Í Z ¢ - Z ˙ Í- b g 0 ˙ Í Z ˙ ÍTZ ˙ Î ˚Î ˚Î ˚ Î ˚ È X ¢˘ È 1 - a b ˘ È X ˘ ÈTX ˘ ÍY ¢ ˙ = Í a 1 - g ˙ ÍY ˙ + ÍT ˙ Í˙Í ˙Í ˙ Í Y ˙ ÍZ ¢ ˙ Í- b g 1 ˙ ÍZ ˙ ÍTZ ˙ Î˚Î ˚Î ˚ Î ˚ È X ¢˘ Ê È 0 ÍY ¢ ˙ = Á Í a Í ˙ ÁÍ Í Z ¢ ˙ Á Í- b Î ˚ ËÎ -a 0 g & ÈX ˘ È 0 Í& ˙ Í ÍY ˙ = Í W 3 Í Z ˙ Í- W & 2 Î˚Î b ˘ È1 0 0˘ ˆ È X ˘ ÈTX ˘ ˜ - g ˙ + Í0 1 0˙ ˜ ÍY ˙ + ÍTY ˙ ˙Í ˙Í ˙ Í ˙ 0 ˙ Í0 0 1˙ ˜ Í Z ˙ ÍTZ ˙ ˚Î ˚ ¯Î ˚ Î ˚ - W3 0 W1 W 2 ˘ È X ˘ ÈV1 ˘ - W1 ˙ ÍY ˙ + ÍV2 ˙ ˙Í ˙ Í ˙ 0 ˙ Í Z ˙ ÍV3 ˙ ˚Î ˚ Î ˚ & X = W 2 Z - W 3Y + V1 & Y = W X - W Z +V 3 1 2 & Z = W1Y - W 2 X + V3 3-D Rigid Motion & X = W 2 Z - W 3Y + V1 & Y = W X - W Z +V 3 1 2 & Z = W1Y - W 2 X + V3 & X = W¥ X + V ÈX ˘ X = ÍY ˙, Í˙ ÍZ ˙ Î˚ ÈW1 ˘ W = ÍW 2 ˙ Í˙ ÍW 3 ˙ Î˚ Cross Product 4 Orthographic Projection & X = W 2 Z - W 3Y + V1 & Y = W X - W Z +V 3 1 y =Y x=X 2 & Z = W1Y - W 2 X + V3 & u = x = W 2 Z - W 3 y + V1 & v = y = W 3 x - W1Z + V2 (u,v) is optical flow Perspective Projection (arbitrary flow) fX Z fY y= Z & u=x= & X = W 2 Z - W 3Y + V1 & Y = W X - W Z +V 3 1 & & fZX - fXZ & & X Z -x Z2 Z Z & & & & fZY - fYZ Y Z & v= y= = f -y Z2 Z Z x= 2 & Z = W1Y - W 2 X + V3 =f & & W Z - W 3Y + V1 W Y - W 2 X + V3 X Z -x = f 2 -x 1 Z Z Z Z & & W X - W1Z + V2 W Y - W 2 X + V3 Y Z & v= y= f -y = f 3 -y 1 Z Z Z Z & u=x= f V V1 W W + W 2 ) - 3 x - W 3 y - 1 xy + 2 x 2 Z Z f f V V W W v = f ( 2 - W1 ) + W 3 x - 3 y + 2 xy - 1 y 2 Z Z f f u= f( 5 Perspective Projection (optical flow) V V1 W W + W 2 ) - 3 x - W 3 y - 1 xy + 2 x 2 Z Z f f V V W W v = f ( 2 - W1 ) + W 3 x - 3 y + 2 xy - 1 y 2 Z Z f f u= f( fV1 - V3 x W W + fW 2 - W 3 y - 1 xy + 2 x 2 Z f f fV - V y W W v = 2 3 - fW1 + W 3 x + 2 xy - 1 y 2 Z f f u= Pure Translation (FOE) fV1 - V3 x Z fV - V y =23 Z u (T ) = v (T ) V3 Z V = ( y0 - y ) 3 Z u (T ) = ( x0 - x) v (T ) x0 = f V1 V , y0 = f 2 V3 V3 p0 = ( x 0 , y 0 ) † 6 Pure Translation (FOE) • p0 is the vanishing point of the direction of translation. • p0 is the intersection of the ray parallel to the translation vector with the image plane. Pure Translation (FOE) • If V3 is not zero, the flow field is radial, and all vectors point towards (or away from) a single point. • The length of flow vectors is inversely proportional to the depth, if V3 is not zero, then it is also proportional to the distance between p and p0. 7 Pure Translation (FOE) fV1 - V3 x Z fV - V y v (T ) = 2 3 Z fV u (T ) = 1 Z •If V =0, the flow field is parallel. fV2 (T ) v= Z u (T ) = 3 8 Structure From Motion ORTHOGRAPHIC PROJECTION Orthographic Projection (displacement) È X ¢˘ È X ˘ È 1 - a b ˘ È X ˘ ÈTX ˘ ÍY ¢ ˙ = R ÍY ˙ + T = Í a 1 g ˙ ÍY ˙ + ÍTY ˙ Í˙ Í˙ Í ˙Í ˙ Í ˙ Í˙ Í˚ Í- b g 1 ˙ Í Z ˙ ÍTZ ˙ ÎZ ¢ ˚ ÎZ ˙ Î ˚Î ˚ Î ˚ x ¢ = x - ay + bZ + TX y ¢ = ax + y - gZ + TY 9 Simple Method • Two Steps Method -Assume depth is known, compute motion x ¢ = x - ay + bZ + T X y ¢ = ax + y - gZ + TY Èa Í Íb È x¢ - x ˘ È - y Z 0 1 0 ˘ Í Í y¢ - y ˙ = Í x 0 - Z 0 1˙ Íg Î ˚Î ˚T Íx Í ÍTy Î ˘ ˙ ˙ ˙ ˙ ˙ ˙ ˙ ˚ Simple Method -Assume motion is known, refine depth x ¢ = x - ay + bZ + TX y ¢ = ax + y - gZ + TY È x¢ - x - ay - Tx ˘ Èb ˘ Í- g ˙[Z ]= Í y¢ - y - ax - T ˙ y˚ Î˚ Î 10 Tomasi and Kanade Orthographic Projection Assumptions • The camera model is orthographic. • The positions of “p” points in “f” frames (f>=3), which are not all coplanar, have been tracked. • The entire sequence has been acquired before starting (batch mode). • Camera calibration not needed, if we accept 3D points up to a scale factor. 11 Tomasi & Kanade Image point {u ( fp , v fp ) | f = 1, K , F , p = 1, K , P} È u11 K u1P ˘ Í ˙ M Í ˙ ÍuF1 K uFP ˙ W =Í ˙ v11 K v1P ˙ Í Í ˙ M Í ˙ Îv F1 K v FP ˚ ÈU ˘ W = Í- ˙ Í˙ ÍV ˙ Î˚ † Tomasi & Kanade 1P af = Âup P p =1 1P bf =  vp P p =1 ~ u fP = u fP - a fP ~ v fP = v fP - b fP 12 s p = ( X p , YP , Z P ) u fP = i T ( sP - t f ) f v fP = j T ( sP - t f ) f 3D world point Orthographic projection kf = if ¥ jf † 13 ~ u fp = u fP - a f 1PT = i ( s p - t f ) -  i f ( sq - t f ) P q =1 T f È 1P ˘ = i Í s P -  sq ˙ P q =1 ˚ Î T f = iT sP f ~ = iT s u fP f P ~ v = jT s fP f P Origin of world is at the centroid of object points ~ ÈU ˘ ~ Í˙ W = Í- ˙ ~ ÍV ˙ Î˚ 14 ~ u fP = i T sP f ~ v = jT s fP f P ~ ÈU ˘ ~ Í˙ W = Í- ˙ ~ ÍV ˙ Î˚ È i1T ˘ Í˙ ÍM˙ T ~ Íif ˙ W = Í T ˙[s1 K sP ]= RS Í j1 ˙ 3XP ÍM˙ Í T˙ Rank of S is 3, because points in 3D space are not Í jf ˙ Î˚ 2FX3 Co-planar Rank Theorem Without noise, the registered ~ measurement matrix W is at most of rank three. È i1T ˘ Í˙ ÍM˙ T ~ Íi ˙ W = Í fT ˙[s1 K sP ]= RS Í j1 ˙ ÍM˙ 3XP Í T˙ Í jf ˙ Î˚ 2FX3 15 Translation ~ u fp = u fP - a f ~ u fp = u fP + a f ~ u fp = i T sP f u fp = i f s p + a f u fp = i T ( s p - t f ) f a f is projection of camera translation along x-axis Translation u fp = i f s p + a f v fp = j f s p + b f T W = RS + tep 2FX3 3XP 2FX1 1XP t = (a1 , K , a f , b1 , K , b f )T T e p = (1, K1) 16 Translation Projected camera translation can be computed: 1P T - if t f = af = Âup P p =1 1P - j t = bf =  vp P p =1 T ff Noisy Measurements ~ • Without noise, the matrix W must be at most of rank 3. When noise corrupts the ~ images, however, W will not be rank 3. Rank theorem can be extended to the case of noisy measurements. 17 Approximate Rank SVD ~ W = O1SO2 2FXP PXP PXP Singular Value Decomposition (SVD) • For some linear systems Ax=b, Gaussian Elimination or LU decomposition does not work, because matrix A is singular, or very close to singular. SVD will not only diagnose for you, but it will solve it. 18 Singular Value Decomposition (SVD) Theorem: Any m by n matrix A, for which m ≥ n ,can be written as A = O1SO2 mxn mxn nxn is diagonal S O1 , O2 nxn are orthogonal T O1T O1 = O2 O2 = I Singular Value Decomposition (SVD) If A is square, then O1 , S, O2 are all square. O1-1 = O1T T O2 1 = O2 S-1 = diag( 1 ) wj A = O1SO2 † † A-1 = O2 diag( 1 )O1 wj † 19 Singular Value Decomposition (SVD) The condition number of a matrix is the ratio of the largest of the w j to the smallest of w j . A matrix is singular if the condition number is infinite, it is ill-conditioned if the condition number is too large. Singular Value Decomposition (SVD) Ax = b • If A is singular, some subspace of “x” maps to zero; the dimension of the null space is called “nullity”. • Subspace of “b” which can be reached by “A” is called range of “A”, the dimension of range is called “rank” of A. 20 Range and Null Space Ax=b x b b=0 Null space of A Dimension of Null space is Nullity Range of A Dimension of range is rank of A Singular Value Decomposition (SVD) • If A is non-singular its rank is “n”. • If A is singular its rank <n. • Rank+nullity=n 21 Singular Value Decomposition (SVD) A = O1SO2 • SVD constructs orthonormal basses of null space and range. of O1 spans range. • Columns • Columns of O2 null space. with non-zero with zero wj w j spans Solution of Linear System • How to solve Ax=b, when A is singular? • If “b” is in the range of “A” then system has many solutions. • Replace 1 wj by zero if x = O2 [diag ( wj = 0 1 )]O1T b wj 22 Solution of Linear System If b is not in the range of A, above eq still gives the solution, which is the best possible solution, it minimizes: r ≡| Ax - b | Approximate Rank ~ W = O1SO2 ¢ ¢ O1SO2 = O1¢S¢O2 + O1¢¢S¢¢O2¢ P-3 3 O1 = [ 1¢ O1¢¢] O 3 2F P-3 È S¢ 0 ˘ S=Í ˙ Î 0 S¢¢˚ 3 P-3 ¢ ÈO2 ˘ 3 O2 = Í ˙ ¢ P-3 ÎO2¢˚ P 23 Approximate Rank ˆ ¢ W = O1¢S¢O2 The best rank 3 approximation to the ideal registered measurement matrix. Rank Theorem for noisy measurement The best possible shape and rotation estimate is obtained by considering only ~ 3 greatest singular values of W together with the corresponding left, right eigenvectors. 24 Approximate Rank ˆ = O¢[S¢]12 R 1 Approximate Rotation matrix ˆ = [S¢]12 O¢ Approximate Shape matrix S 2 ˆ ˆˆ W = RS This decomposition is not unique ˆ ˆ ˆ W = RQ Q-1S ( )( ) Q is any 3X3 invertable matrix † Approximate Rank ˆ R = RQ How to determine Q? ˆ ˆ i fT QQT i fT = 1 ˆ S =Q S -1 R and S are linear transformation of approximate Rotation and shape matrices ˆT QQT ˆT = 1 jf jf ˆ i fT QQT ˆT = 0 jf Rows of rotation matrix are unit vectors and orthogonal 25 How to determine Q: Newton’s Method ˆ ˆ f1 (q) = iiT QQT iiT - 1 = 0 f (q) = ˆ T QQT ˆ T - 1 = 0 j j MDq = e ˆ f 3 (q) = i QQ ˆ1T = 0 j Dq = [Dq1,K, Dq9 ] 2 1 1 T 1 T M M ij = ˆ ˆ f 3 f - 2 (q) = i fT QQT i fT - 1 = 0 f 3 f -1 (q) = ˆ T QQT ˆ T - 1 = 0 jf jf ∂f i ∂q j e is error ˆ f 3 f (q) = i fT QQT ˆ T = 0 jf † † Algorithm • Compute SVD of • define • Compute 1 ˆ R = O1¢[S¢] 2 ~ W = O1SO2 1 ˆ ¢ S = [S¢] 2 O2 Q • Compute R ˆ = RQ ˆ S = Q -1S 26 Hotel Sequence Results (rotations) 27 Selected Features Reconstructed Shape 28 Comparison House Sequence 29 Reconstructed Walls ../tomasiTr92Figures.pdf 30 Web Page • http://vision.stanford.edu/cgibin/svl/publication/publication1992.cgi 31 ...
View Full Document

This note was uploaded on 06/12/2011 for the course CAP 6411 taught by Professor Shah during the Spring '09 term at University of Central Florida.

Ask a homework question - tutors are online