Lecture-20 - 3/17/2008 LECTURE 20 Multi View Geometry of...

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: 3/17/2008 LECTURE 20 Multi View Geometry of Multi View Geometry of Moving Moving Cameras Alper Yilmaz and Mubarak Shah Computer Vision Lab. Vision Lab. Univ. of Central Florida http://www.cs.ucf.edu/~vision http://www.cs.ucf.edu/~vision 1 3/17/2008 Alper Yilmaz and Mubarak Shah, Recognizing Human Actions in Videos Acquired by Uncalibrated Moving Cameras, IEEE ICCV 2005, Beijing, China, October 15-21. http://www.cs.ucf.edu/~vision/papers/yilmaz_iccv_ iccv 2005.pdf Multi View Geometry ! Defined for two or more static cameras for two or more static cameras 3D World Right camera Left camera Left camera image plane Right camera image plane 2 3/17/2008 Epipolar Geometry P xr xl Cl P: Cl: Cr: e l: e r: Cr er el Epipolar line: set of set world points that project to the same point in left image, when projected to right right image forms a line. T world point left camera center right camera center left epipole right epipole Xl maps to line Xrer. maps Epipolar plane: plane defined by the camera centers centers and world point. Pl ! Cl P Pr ! Cr P T ! C r " Cl Essential Matrix P xr xl Cl Cr er el T Coplanarity Coplanarity constraint between vectors (Pl-T), T, Pl. ), ,0 * T % Pl ! * Tz *"T +y #P R $T % P ! 0 #R P $ T % P ! 0 Pr ! R #Pl " T $ & & r " Tz 0 Tx Ty ) ' " Tx 'Pl ! SPl 0' ( #Pl " T $& T % Pl ! 0 & r l & Pr RSPl ! 0 l & Pr !Pl ! 0 essential matrix 3 3/17/2008 Fundamental Matrix P Pr ! R #Pl " T $ Pl ! Rl P - Tl Pr ! Rr P - Tr xr xl Cl el R ! Rr Rl & er Cr T (A) T ! Tl " R &Tr (B) x ! M l Pl x' ! M r Pr & Pr !Pl ! 0 "& "1 x'& M r EM l x ! 0 x'& Fx ! 0 fundamental matrix Fundamental Matrix ,a b c ) * ' x ' Fx ! x ' * d e f 'x ! 0 * g h m' + ( & ! ! ! ! & Rank Rank 2 matrix (due to S) 3x3 matrix with 9 components 7 degrees of freedom Given Given a point in left camera x, epipolar line in epipolar right camera is: ur=Fx 4 3/17/2008 What Happens to Fundamental What Happens to Fundamental Matrix Matrix When Cameras Move? Observations ! At each time instant each time instant ! ! ! ! ! Different epipolar geometries Different epipoles Different and Different R and T (equations A and B) Different fundamental matrices Is Is there any relation between consecutive consecutive epipolar geometries? 5 3/17/2008 Epipolar Epipolar Geometry of Moving Moving Cameras P2 P3 P1 Cl3 3 2 1 Cr Cr Cr Cl2 Cl1 Theorems Theorems Governing Temporal Temporal Epipolar Geometry ! Theorem (temporal fundamental matrix): Theorem 1 (temporal fundamental matrix): ! Corresponding Corresponding points in two sequences captured by independently moving cameras are related to each other through a temporal fundamental matrix of of the form: ˆ x & (t ) F (t )x l (t ) ! 0 r 6 3/17/2008 Theorems Theorems Governing Temporal Temporal Epipolar Geometry ! Theorem (on the order of polynomials in the Theorem 2 (on the order of polynomials in the temporal fundamental matrix): ! Assume Assume that motion of cameras are approximated by polynomials in time variable. Then, the temporal the fundamental matrix is a 3x3 matrix whose components are polynomials order: polynomials of order: ˆ deg Fi , j (t ) ! max(nl , nr , ml , mr ) - 1 where refers where i,j refers to ith row jth column, and m. and n. refers to row column, and refers degree of polynomials for translational and rotational velocities. velocities. Computing TFM ! ! Find corresponding points in first frames corresponding points in first frames Normalize all trajectories in both views ! ! Mean normalize Isotropically scale: on the average a point scale: is (1,1,1) is 7 3/17/2008 Computing TFM (for order 2) ! Select the order of polynomials the order of polynomials /xr (t ) ! 6 a1 - a2t - a3t 2 4 yr (t ) 104 d1 - d 2t - d 3t 2 4 g1 - g 2t - g 3t 2 5 b1 - b2t - b3t 2 e1 - e2t - e3t 2 h1 - h2t - h3t 2 c1 - c2t - c3t 2 3 6 xl (t ) 3 1 f1 - f 2t - f 3t 2 1 4 yl (t )1 ! 0 4 1 m1 - m2t - m3t 2 1 4 1 1 2 25 Construct Construct a linear system to solve TFM unknowns: ANx27.f27x1=0Nx1 Ai ! ( xr xl , xr xl t , xr xl t 2 , xr yl , xr yl t , xr yl t 2 , xr , xr t , xr t 2 , yr xl , yr xl t , yr xl t 2 , yr yl , yr yl t , yr yl t 2 , yr , yr t , yr t 2 , xl , xl t , xl t 2 , yl , yl t , yl t 2 ,1, t , t 2 ) f ! #a1 , a2 , a3 , b1 , b2 , b3 , c1 , c2 , c3 , d1 , d 2 , d 3 , e1 , e2 , e3 , f1 , f 2 , f 3 , g1 , g2 , g3 , h1 , h2 , h3 , m1 , m2 , m3 $ & Computing TFM (for order 2) Af ! 0 A & Af ! 0 ! ! Compute Compute Singular Value Decomposition of ATA Select minimum eigenvalued eigenvector. 8 3/17/2008 Quality of Recovered Geometry ! Two measures measures ! ! Condition Condition Number of ATA: How well-conditioned is well-conditioned the the homogenous linear system? Symmetric Symmetric Epipolar Distance: How correct is the Distance: estimated estimated TFM? Condition Number ! Select rank of ATA of from singular values 7I rank of of from singular values : A) ! i 9 : i rank ( A & j !1 N 7j j !i -1 ! 7j 8 th Compute condition number of ATA by c! 71 7 rank ( A & A) 9 3/17/2008 Geometric Error in TFM Compute TFM unknowns TFM unknowns Construct TFM by enforcing constraints For each point on left and right cameras ! ! ! ! Compute epipolar line ˆ u r ! F ( t ) xl ! ˆ ul ! F & (t ) xr Compute distance of point from epipolar line di li & & 1 , ul xl u r xr g! * ur 2 * ul + ) ' ' ( Applications of This New Applications of This New Geometry Geometry ! ! Tracking Across Multiple Moving Cameras Action Recognition in Video Captured Using Uncalibrated Moving Cameras 10 3/17/2008 Tracking Objects Across Tracking Objects Across Multiple Multiple Moving Cameras Problem Definition ! ! Multiple cameras cameras Multiple objects 11 3/17/2008 Problem Definition ! Find corresponding objects corresponding objects ! ! Cameras move independently Objects move independently Tracking in Single Camera ! Point based based ! ! ! Region based ! ! Object Object detection (background subtraction) subtraction) Point (centroid) correspondence Rigid motion models (mean-shift motion models (mean tracker, Eigentracking, etc.) Contour based ! Non-rigid object deformations 12 3/17/2008 Tracking Across Multiple Cameras ! ! ! Should be more than one object in Should be more than one object in scene scene Object Object tracking in single camera is performed performed first Which Which object in one camera associates with the objects in the second camera? ith th th ! ! Object correspondence Trajectory correspondence Tracking Across Multiple Cameras Object Correspondence ! Appearance matching matching ! ! Same Same scene appears different in different cameras cameras (different camera gain) Different object views 13 3/17/2008 Tracking Across Multiple Cameras Trajectory Correspondence A B ! Trajectory correspondence correspondence ! ! ! A Advantage: Advantage: Cameras can be of different different modalities Registration based: Compute Compute transformation between two camera views (affine, projective) Epipolar geometry based B Registration Registration Based Trajectory Trajectory Matching ! (1) If you have stationary cameras you have stationary cameras ! ! ! (2) ! ! ! (3) Register one camera view onto the other Label closest trajectories OR… Use Use epipolar geometry constraints (applied in context context of action recognition) OR… Treat Treat trajectories as 3D objects (x,y,t), compute 3D transformation between them and compute reconstruction reconstruction error. 14 3/17/2008 Registration Registration Based Trajectory Trajectory Matching ! For moving cameras moving cameras ! Compensate camera motion ! (1) ! Apply method 2 or 3 for stationary camera case ! ! ! (2) Generates trajectory as if camera is stationary Yaser’s work OR… It It is like magic " ! ! Temporal epipolar geometry Complete perspective geometry Fundamental Fundamental Matrix in Context of of Trajectory Matching ! ! Let there be Let there be N corresponding points in left and right cameras points in left and right cameras Unknowns Unknowns of F can be found by least squares f ! &########### A ###########$ 6 a 3 % x '1 y1 x '1 y '1 x1 y '1 y1 y '1 x1 y1 13 4 1 6 x '1 x1 " 4" " " " " " " " " 14 1 ! 0 4 14 g 1 4 x ' N xN x ' N y N x ' N y ' N x N y ' N y N y ' N xN y N 11 4 1 5 2m 52 ! ! ! A & Af ! 0 Solution is found by SVD: Eigenvector with smallest eigenvalue SVD Ei Ideally 27th singular value is 0. Use 27th singular value as distance measure between two singular trajectories. trajectories. 15 3/17/2008 Finding Finding Corresponding Objects Objects Across Cameras ! ! ! Given and Given Nr and Nl trajectories in right and left cameras in right and left cameras Find Find correct correspondences that satisfy temporal fundamental fundamental matrix (TFM). What is TFM? ! ! We We know neither TFM nor correspondences # We We know the constraints used to compute TFM " ! . Finding Finding Corresponding Objects Objects Across Cameras ! Select correspondence hypothesis Select a correspondence hypothesis and and check its validity ! ! Total Total of NrxNl hypotheses How do we check validity? ! ! Algebraic error Geometric reconstruction error 16 3/17/2008 Computing Computing Correspondences Correspondences ! Graph theoretic matching theoretic matching Bipartite graph: No edges No between trajectories in the same same camera view. Edge weights: wi are are computed using either geometric or algebraic error. wi Correspondences: Computed by Computed left camera right camera maximum matching of weighted bipartite bipartite graph. Results 17 3/17/2008 18 3/17/2008 Algebraic Algebraic & Geometric Errors Errors in Matching 19 3/17/2008 Action Action Recognition in Video Captured Using Uncalibrated Captured Using Uncalibrated Moving Moving Cameras Representations Representations for Action Recognition Recognition ! Motion trajectories Motion trajectories (Rao (Rao et al. IJCV 2002) ! Actions Actions as Objects (Yilmaz&Shah (Yilmaz&Shah CVPR 2005) ! Set Set of Landmarks falling (Gritai, Sheikh & Shah, ICPR Gritai, 2004) 2004) 20 3/17/2008 Representing Human Body ! A set of landmark points on the body set of landmark points on the body Representing Action A collection of trajectories: # & U ! ;1& , ;2& ,' , ;13 $ 21 3/17/2008 What What happens to action trajectories trajectories when cameras move? ! ! ! Trajectories include camera motion include camera motion Viewpoint constantly changes Action Action trajectories appear different appear Example (Picking Up Action) ! ! ! Different actors actors Different camera motion Different viewpoints 22 3/17/2008 ApprpachApprpach-1 ! Compensate camera motion camera motion ! ! ! High computational cost Assumes Assumes planar scenes, actions are not planar planar Compensation Compensation distorts world to image perspective projection ! Introduces artificial deformations ApproachApproach-II ! Compute epipolar geometry of each Compute epipolar geometry of each frame frame individually ! Computational Computational cost ! ! Estimation of huge number of unknowns Temporal Temporal dependency of epipolar geometries geometries is not considered ! Additional constraints are required 23 3/17/2008 Proposed Approach ! ! Find the geometry between two actions Find the geometry between two actions views views using TFM Matching score: ! ! Condition number (CN) Symmetric epipolar distance (SED) # & U ! ;1& , ;2& ,', ;13 $ &~ U tone F (t )U two ! 0 t Form a homogenous system of equations Similarity of Two Actions ! Proposition: Given two action videos captured by Given uncalibrated moving cameras, there exists a unique temporal fundamental matrix which can be computed using landmark landmark points on the actors. U tone ! & #: k $ Fi (t ) k U ttwo ! 0 i !0 Similarity Similarity is defined in terms of quality of the linear system and the the quality of the recovered geometry. $ Condition number of ATA $ Symmetric epipolar distance g 24 3/17/2008 On the Matching Score ! Homogenous equation system has many Homogenous equation system has many solutions solutions ! ! ! High CN: does not mean good TFM estimate Low CN: ill-conditioned equation system illHigh CN & low SED indicate correct action match Database ! 19 action videos action videos ! ! ! Moving camera Different viewpoints Different actors 25 3/17/2008 Action Videos Action Videos 26 3/17/2008 Action Videos Action Videos 27 3/17/2008 Action Videos Recognition Performance ! Confusion matrices for various methods matrices for various methods Proposed method Use of static epipolar geometry Use of static epipolar geometry and our metric 28 3/17/2008 Recognition Performance ! Confusion matrices for various methods matrices for various methods Proposed method Use of static epipolar geometry Action Retrieval Exemplar Actions 29 3/17/2008 Actions 30 ...
View Full Document

Ask a homework question - tutors are online