This preview shows page 1. Sign up to view the full content.
Unformatted text preview: 3/17/2008 LECTURE 20 Multi View Geometry of
Multi View Geometry of
Moving
Moving Cameras
Alper Yilmaz and Mubarak Shah
Computer Vision Lab.
Vision Lab.
Univ. of Central Florida
http://www.cs.ucf.edu/~vision
http://www.cs.ucf.edu/~vision 1 3/17/2008 Alper Yilmaz and Mubarak Shah, Recognizing Human Actions
in Videos Acquired by Uncalibrated Moving Cameras, IEEE
ICCV 2005, Beijing, China, October 1521. http://www.cs.ucf.edu/~vision/papers/yilmaz_iccv_
iccv
2005.pdf Multi View Geometry
! Defined for two or more static cameras
for two or more static cameras 3D World
Right
camera Left
camera
Left camera
image plane Right camera
image plane 2 3/17/2008 Epipolar Geometry
P
xr xl Cl P:
Cl:
Cr:
e l:
e r: Cr er el Epipolar line: set of
set
world points that project
to the same point in left
image, when projected to
right
right image forms a line. T world point
left camera center
right camera center
left epipole
right epipole Xl maps to line Xrer.
maps Epipolar plane: plane
defined by the camera
centers
centers and world point. Pl ! Cl P Pr ! Cr P
T ! C r " Cl Essential Matrix
P
xr xl Cl Cr er el
T Coplanarity
Coplanarity constraint between vectors (PlT), T, Pl.
), ,0
*
T % Pl ! * Tz
*"T
+y #P R $T % P ! 0 #R P $ T % P ! 0 Pr ! R #Pl " T $ & & r " Tz
0
Tx Ty )
'
" Tx 'Pl ! SPl
0'
( #Pl " T $& T % Pl ! 0
& r l & Pr RSPl ! 0 l & Pr !Pl ! 0
essential matrix 3 3/17/2008 Fundamental Matrix
P Pr ! R #Pl " T $
Pl ! Rl P  Tl
Pr ! Rr P  Tr xr xl Cl el R ! Rr Rl & er Cr T (A) T ! Tl " R &Tr (B) x ! M l Pl x' ! M r Pr
& Pr !Pl ! 0 "& "1 x'& M r EM l x ! 0
x'& Fx ! 0
fundamental matrix Fundamental Matrix
,a b c )
*
'
x ' Fx ! x ' * d e f 'x ! 0
* g h m'
+
(
& !
!
!
! & Rank
Rank 2 matrix (due to S)
3x3 matrix with 9 components
7 degrees of freedom
Given
Given a point in left camera x, epipolar line in
epipolar
right camera is: ur=Fx 4 3/17/2008 What Happens to Fundamental
What Happens to Fundamental
Matrix
Matrix When Cameras Move? Observations
! At each time instant
each time instant
!
!
!
! ! Different epipolar geometries
Different epipoles
Different and
Different R and T (equations A and B)
Different fundamental matrices Is
Is there any relation between
consecutive
consecutive epipolar geometries? 5 3/17/2008 Epipolar
Epipolar Geometry of
Moving
Moving Cameras
P2 P3 P1 Cl3 3
2
1
Cr Cr Cr Cl2
Cl1 Theorems
Theorems Governing
Temporal
Temporal Epipolar Geometry
! Theorem (temporal fundamental matrix):
Theorem 1 (temporal fundamental matrix):
! Corresponding
Corresponding points in two sequences captured
by independently moving cameras are related to
each other through a temporal fundamental matrix
of
of the form: ˆ
x & (t ) F (t )x l (t ) ! 0
r 6 3/17/2008 Theorems
Theorems Governing
Temporal
Temporal Epipolar Geometry
! Theorem (on the order of polynomials in the
Theorem 2 (on the order of polynomials in the
temporal fundamental matrix):
! Assume
Assume that motion of cameras are approximated by
polynomials in time variable. Then, the temporal
the
fundamental matrix is a 3x3 matrix whose components are
polynomials order:
polynomials of order: ˆ
deg Fi , j (t ) ! max(nl , nr , ml , mr )  1
where refers
where i,j refers to ith row jth column, and m. and n. refers to
row column,
and refers
degree of polynomials for translational and rotational
velocities.
velocities. Computing TFM
!
! Find corresponding points in first frames
corresponding points in first frames
Normalize all trajectories in both views
!
! Mean normalize
Isotropically scale: on the average a point
scale:
is (1,1,1)
is 7 3/17/2008 Computing TFM (for order 2)
! Select the order of polynomials
the order of polynomials
/xr (t ) ! 6 a1  a2t  a3t 2
4
yr (t ) 104 d1  d 2t  d 3t 2
4 g1  g 2t  g 3t 2
5 b1  b2t  b3t 2
e1  e2t  e3t 2
h1  h2t  h3t 2 c1  c2t  c3t 2 3 6 xl (t ) 3
1
f1  f 2t  f 3t 2 1 4 yl (t )1 ! 0
4
1
m1  m2t  m3t 2 1 4 1 1
2
25 Construct
Construct a linear system to solve TFM
unknowns: ANx27.f27x1=0Nx1
Ai ! ( xr xl , xr xl t , xr xl t 2 , xr yl , xr yl t , xr yl t 2 , xr , xr t , xr t 2 , yr xl , yr xl t , yr xl t 2 ,
yr yl , yr yl t , yr yl t 2 , yr , yr t , yr t 2 , xl , xl t , xl t 2 , yl , yl t , yl t 2 ,1, t , t 2 ) f ! #a1 , a2 , a3 , b1 , b2 , b3 , c1 , c2 , c3 , d1 , d 2 , d 3 , e1 , e2 , e3 , f1 , f 2 , f 3 , g1 , g2 , g3 , h1 , h2 , h3 , m1 , m2 , m3 $ & Computing TFM (for order 2)
Af ! 0
A & Af ! 0
! ! Compute
Compute Singular Value Decomposition of
ATA
Select minimum eigenvalued eigenvector. 8 3/17/2008 Quality of Recovered Geometry
! Two measures
measures
! ! Condition
Condition Number of ATA: How wellconditioned is
wellconditioned
the
the homogenous linear system?
Symmetric
Symmetric Epipolar Distance: How correct is the
Distance:
estimated
estimated TFM? Condition Number
! Select rank of ATA of from singular values 7I
rank of
of from singular values :
A) ! i 9
: i rank ( A & j !1
N 7j j !i 1 ! 7j 8 th Compute condition number of ATA by
c! 71 7 rank ( A & A) 9 3/17/2008 Geometric Error in TFM
Compute TFM unknowns
TFM unknowns
Construct TFM by enforcing constraints
For each point on left and right cameras !
!
! ! Compute epipolar line
ˆ
u r ! F ( t ) xl ! ˆ
ul ! F & (t ) xr Compute distance of point from epipolar line
di
li
&
&
1 , ul xl u r xr
g! *
ur
2 * ul
+ )
'
'
( Applications of This New
Applications of This New
Geometry
Geometry
!
! Tracking Across Multiple Moving Cameras
Action Recognition in Video Captured Using
Uncalibrated Moving Cameras 10 3/17/2008 Tracking Objects Across
Tracking Objects Across
Multiple
Multiple Moving Cameras Problem Definition
!
! Multiple cameras
cameras
Multiple objects 11 3/17/2008 Problem Definition
! Find corresponding objects
corresponding objects
!
! Cameras move independently
Objects move independently Tracking in Single Camera
! Point based
based
! ! ! Region based
! ! Object
Object detection (background
subtraction)
subtraction)
Point (centroid) correspondence
Rigid motion models (meanshift
motion models (mean
tracker, Eigentracking, etc.) Contour based
! Nonrigid object deformations 12 3/17/2008 Tracking Across
Multiple Cameras
! ! ! Should be more than one object in
Should be more than one object in
scene
scene
Object
Object tracking in single camera is
performed
performed first
Which
Which object in one camera associates
with the objects in the second camera?
ith th
th
!
! Object correspondence
Trajectory correspondence Tracking Across
Multiple Cameras
Object Correspondence
! Appearance matching
matching
! ! Same
Same scene appears different in different
cameras
cameras (different camera gain)
Different object views 13 3/17/2008 Tracking Across
Multiple Cameras
Trajectory Correspondence
A B ! Trajectory correspondence
correspondence
! ! ! A Advantage:
Advantage: Cameras can be of
different
different modalities
Registration based: Compute
Compute
transformation between two
camera views (affine, projective)
Epipolar geometry based B Registration
Registration Based
Trajectory
Trajectory Matching
! (1) If you have stationary cameras
you have stationary cameras
!
!
! (2) ! !
! (3) Register one camera view onto the other
Label closest trajectories
OR…
Use
Use epipolar geometry constraints (applied in
context
context of action recognition)
OR…
Treat
Treat trajectories as 3D objects (x,y,t), compute
3D transformation between them and compute
reconstruction
reconstruction error. 14 3/17/2008 Registration
Registration Based
Trajectory
Trajectory Matching
! For moving cameras
moving cameras
! Compensate camera motion
! (1) ! Apply method 2 or 3 for stationary camera case
! !
! (2) Generates trajectory as if camera is stationary
Yaser’s work OR…
It
It is like magic "
!
! Temporal epipolar geometry
Complete perspective geometry Fundamental
Fundamental Matrix in Context
of
of Trajectory Matching
!
! Let there be
Let there be N corresponding points in left and right cameras
points in left and right cameras
Unknowns
Unknowns of F can be found by least squares f
!
&########### A ###########$ 6 a 3
%
x '1 y1
x '1 y '1 x1
y '1 y1
y '1 x1 y1 13 4 1
6 x '1 x1
"
4"
"
"
"
"
"
"
"
" 14 1 ! 0
4
14 g 1
4 x ' N xN x ' N y N x ' N y ' N x N y ' N y N y ' N xN y N 11 4 1
5
2m
52 !
!
! A & Af ! 0 Solution is found by SVD: Eigenvector with smallest eigenvalue
SVD Ei
Ideally 27th singular value is 0.
Use 27th singular value as distance measure between two
singular
trajectories.
trajectories. 15 3/17/2008 Finding
Finding Corresponding
Objects
Objects Across Cameras
!
! ! Given and
Given Nr and Nl trajectories in right and left cameras
in right and left cameras
Find
Find correct correspondences that satisfy temporal
fundamental
fundamental matrix (TFM).
What is TFM?
!
! We
We know neither TFM nor correspondences #
We
We know the constraints used to compute TFM "
! . Finding
Finding Corresponding
Objects
Objects Across Cameras
! Select correspondence hypothesis
Select a correspondence hypothesis
and
and check its validity
! ! Total
Total of NrxNl hypotheses How do we check validity?
!
! Algebraic error
Geometric reconstruction error 16 3/17/2008 Computing
Computing
Correspondences
Correspondences
! Graph theoretic matching
theoretic matching
Bipartite graph: No edges
No
between trajectories in the
same
same camera view.
Edge weights: wi are
are
computed using either
geometric or algebraic
error. wi Correspondences: Computed by
Computed left
camera right
camera maximum matching of weighted
bipartite
bipartite graph. Results 17 3/17/2008 18 3/17/2008 Algebraic
Algebraic & Geometric
Errors
Errors in Matching 19 3/17/2008 Action
Action Recognition in Video
Captured Using Uncalibrated
Captured Using Uncalibrated
Moving
Moving Cameras Representations
Representations for Action
Recognition
Recognition
! Motion trajectories
Motion trajectories
(Rao
(Rao et al. IJCV 2002) ! Actions
Actions as Objects
(Yilmaz&Shah
(Yilmaz&Shah CVPR 2005) ! Set
Set of Landmarks falling (Gritai, Sheikh & Shah, ICPR
Gritai,
2004)
2004) 20 3/17/2008 Representing Human Body
! A set of landmark points on the body
set of landmark points on the body Representing Action
A collection of trajectories: # &
U ! ;1& , ;2& ,' , ;13 $ 21 3/17/2008 What
What happens to action
trajectories
trajectories when cameras move?
!
!
! Trajectories include camera motion
include camera motion
Viewpoint constantly changes
Action
Action trajectories appear different
appear Example (Picking Up Action)
!
!
! Different actors
actors
Different camera motion
Different viewpoints 22 3/17/2008 ApprpachApprpach1
! Compensate camera motion
camera motion
!
! ! High computational cost
Assumes
Assumes planar scenes, actions are not
planar
planar
Compensation
Compensation distorts world to image
perspective projection
! Introduces artificial deformations ApproachApproachII
! Compute epipolar geometry of each
Compute epipolar geometry of each
frame
frame individually
! Computational
Computational cost
! ! Estimation of huge number of unknowns Temporal
Temporal dependency of epipolar
geometries
geometries is not considered
! Additional constraints are required 23 3/17/2008 Proposed Approach
! ! Find the geometry between two actions
Find the geometry between two actions
views
views using TFM
Matching score:
!
! Condition number (CN)
Symmetric epipolar distance (SED) # &
U ! ;1& , ;2& ,', ;13 $ &~
U tone F (t )U two ! 0
t Form a homogenous system of equations Similarity of Two Actions
! Proposition: Given two action videos captured by
Given
uncalibrated moving cameras, there exists a unique
temporal fundamental matrix which can be computed using
landmark
landmark points on the actors. U tone
! & #: k $ Fi (t ) k U ttwo ! 0
i !0 Similarity
Similarity is defined in terms of quality of the linear system and
the
the quality of the recovered geometry.
$ Condition number of ATA
$ Symmetric epipolar distance g 24 3/17/2008 On the Matching Score
! Homogenous equation system has many
Homogenous equation system has many
solutions
solutions
!
!
! High CN: does not mean good TFM estimate
Low CN: illconditioned equation system
illHigh CN & low SED indicate correct action match Database
! 19 action videos
action videos
!
!
! Moving camera
Different viewpoints
Different actors 25 3/17/2008 Action Videos Action Videos 26 3/17/2008 Action Videos Action Videos 27 3/17/2008 Action Videos Recognition Performance
! Confusion matrices for various methods
matrices for various methods Proposed method Use of static
epipolar geometry Use of static
epipolar geometry
and our metric 28 3/17/2008 Recognition Performance
! Confusion matrices for various methods
matrices for various methods Proposed method Use of static
epipolar geometry Action Retrieval Exemplar Actions 29 3/17/2008 Actions 30 ...
View Full
Document
 Spring '09
 Shah

Click to edit the document details