This preview shows page 1. Sign up to view the full content.
Unformatted text preview: Lecture8
Structure from Motion Problem
• Given optical flow or point
correspondences, compute 3D
motion (translation and rotation)
and shape (depth). 1 3D Rigid Motion (displacement)
È X ¢˘ È X ˘
È 1  a b ˘ È X ˘ ÈTX ˘
ÍY ¢ ˙ = R ÍY ˙ + T = Í a 1  g ˙ ÍY ˙ + ÍT ˙
Í˙ Í˙
Í
˙Í ˙ Í Y ˙
ÍZ ¢ ˙ ÍZ ˙
Í b g 1 ˙ ÍZ ˙ ÍTZ ˙
Î˚ Î˚
Î
˚Î ˚ Î ˚
X ¢ = X  aY + bZ + TX
Y ¢ = aX + Y  gZ + TY
Z ¢ =  bX + gY + Z + TZ Orthographic Projection
(displacement model)
X ¢ = X  aY + bZ + TX
Y ¢ = aX + Y  gZ + TY
Z ¢ =  bX + gY + Z + TZ
x¢ = x  ay + bZ + TX
y¢ = ax + y  gZ + TY 2 Perspective Projection (displacement)
X ¢ = X  aY + bZ + TX
Y ¢ = aX + Y  gZ + TY
Z ¢ =  bX + gY + Z + TZ X  aY + bZ + TX
bX + gY + Z + TZ
aX + Y  gZ + TY
y¢ =
bX + gY + Z + TZ
x¢ = TX
Z
x¢ =
T
 bx + gy + 1 + Z
Z
T
ax + y  g + Y
Z
y¢ =
T
 bx + gy + 1 + Z
Z
x  ay + b + † Instantaneous Velocity Model
Optical Flow 3 3D Rigid Motion
È X ¢  X ˘ È 0  a b ˘ È X ˘ ÈTX ˘
ÍY ¢  Y ˙ = Í a
0  g ˙ ÍY ˙ + ÍTY ˙
Í
˙Í
˙Í ˙ Í ˙
Í Z ¢  Z ˙ Í b g
0 ˙ Í Z ˙ ÍTZ ˙
Î
˚Î
˚Î ˚ Î ˚ È X ¢˘ È 1  a b ˘ È X ˘ ÈTX ˘
ÍY ¢ ˙ = Í a 1  g ˙ ÍY ˙ + ÍT ˙
Í˙Í
˙Í ˙ Í Y ˙
ÍZ ¢ ˙ Í b g 1 ˙ ÍZ ˙ ÍTZ ˙
Î˚Î
˚Î ˚ Î ˚ È X ¢˘ Ê È 0
ÍY ¢ ˙ = Á Í a
Í ˙ ÁÍ
Í Z ¢ ˙ Á Í b
Î ˚ ËÎ a
0
g &
ÈX ˘ È 0
Í& ˙ Í
ÍY ˙ = Í W 3
Í Z ˙ Í W
&
2
Î˚Î b ˘ È1 0 0˘ ˆ È X ˘ ÈTX ˘
˜
 g ˙ + Í0 1 0˙ ˜ ÍY ˙ + ÍTY ˙
˙Í
˙Í ˙ Í ˙
0 ˙ Í0 0 1˙ ˜ Í Z ˙ ÍTZ ˙
˚Î
˚ ¯Î ˚ Î ˚  W3
0
W1 W 2 ˘ È X ˘ ÈV1 ˘
 W1 ˙ ÍY ˙ + ÍV2 ˙
˙Í ˙ Í ˙
0 ˙ Í Z ˙ ÍV3 ˙
˚Î ˚ Î ˚ &
X = W 2 Z  W 3Y + V1
&
Y = W X  W Z +V
3 1 2 &
Z = W1Y  W 2 X + V3 3D Rigid Motion
&
X = W 2 Z  W 3Y + V1
&
Y = W X  W Z +V
3 1 2 &
Z = W1Y  W 2 X + V3 &
X = W¥ X + V ÈX ˘
X = ÍY ˙,
Í˙
ÍZ ˙
Î˚
ÈW1 ˘
W = ÍW 2 ˙
Í˙
ÍW 3 ˙
Î˚ Cross Product 4 Orthographic Projection
&
X = W 2 Z  W 3Y + V1
&
Y = W X  W Z +V
3 1 y =Y
x=X 2 &
Z = W1Y  W 2 X + V3
&
u = x = W 2 Z  W 3 y + V1
&
v = y = W 3 x  W1Z + V2 (u,v) is optical flow Perspective Projection (arbitrary flow)
fX
Z
fY
y=
Z &
u=x= &
X = W 2 Z  W 3Y + V1
&
Y = W X  W Z +V
3 1 &
&
fZX  fXZ &
&
X
Z
x
Z2
Z
Z
&
&
&
&
fZY  fYZ
Y
Z
&
v= y=
= f y
Z2
Z
Z x= 2 &
Z = W1Y  W 2 X + V3 =f &
&
W Z  W 3Y + V1
W Y  W 2 X + V3
X
Z
x = f 2
x 1
Z
Z
Z
Z
&
&
W X  W1Z + V2
W Y  W 2 X + V3
Y
Z
&
v= y= f y = f 3
y 1
Z
Z
Z
Z &
u=x= f V
V1
W
W
+ W 2 )  3 x  W 3 y  1 xy + 2 x 2
Z
Z
f
f
V
V
W
W
v = f ( 2  W1 ) + W 3 x  3 y + 2 xy  1 y 2
Z
Z
f
f u= f( 5 Perspective Projection (optical flow)
V
V1
W
W
+ W 2 )  3 x  W 3 y  1 xy + 2 x 2
Z
Z
f
f
V
V
W
W
v = f ( 2  W1 ) + W 3 x  3 y + 2 xy  1 y 2
Z
Z
f
f u= f( fV1  V3 x
W
W
+ fW 2  W 3 y  1 xy + 2 x 2
Z
f
f
fV  V y
W
W
v = 2 3  fW1 + W 3 x + 2 xy  1 y 2
Z
f
f u= Pure Translation (FOE)
fV1  V3 x
Z
fV  V y
=23
Z u (T ) =
v (T ) V3
Z
V
= ( y0  y ) 3
Z u (T ) = ( x0  x)
v (T ) x0 = f V1
V
, y0 = f 2
V3
V3 p0 = ( x 0 , y 0 ) † 6 Pure Translation (FOE)
• p0 is the vanishing point of the direction of
translation.
• p0 is the intersection of the ray parallel to
the translation vector with the image plane. Pure Translation (FOE)
• If V3 is not zero, the flow field is radial,
and all vectors point towards (or away
from) a single point.
• The length of flow vectors is inversely
proportional to the depth, if V3 is not zero,
then it is also proportional to the distance
between p and p0. 7 Pure Translation (FOE)
fV1  V3 x
Z
fV  V y
v (T ) = 2 3
Z
fV
u (T ) = 1
Z
•If V =0, the flow field is parallel.
fV2
(T )
v=
Z u (T ) = 3 8 Structure From Motion
ORTHOGRAPHIC PROJECTION Orthographic Projection (displacement)
È X ¢˘ È X ˘
È 1  a b ˘ È X ˘ ÈTX ˘
ÍY ¢ ˙ = R ÍY ˙ + T = Í a
1 g ˙ ÍY ˙ + ÍTY ˙
Í˙ Í˙
Í
˙Í ˙ Í ˙
Í˙ Í˚
Í b g 1 ˙ Í Z ˙ ÍTZ ˙
ÎZ ¢ ˚ ÎZ ˙
Î
˚Î ˚ Î ˚ x ¢ = x  ay + bZ + TX
y ¢ = ax + y  gZ + TY 9 Simple Method
• Two Steps Method
Assume depth is known, compute
motion x ¢ = x  ay + bZ + T
X y ¢ = ax + y  gZ + TY Èa
Í
Íb
È x¢  x ˘ È  y Z 0 1 0 ˘ Í
Í y¢  y ˙ = Í x 0  Z 0 1˙ Íg
Î
˚Î
˚T
Íx
Í
ÍTy
Î ˘
˙
˙
˙
˙
˙
˙
˙
˚ Simple Method
Assume motion is known, refine depth
x ¢ = x  ay + bZ + TX
y ¢ = ax + y  gZ + TY È x¢  x  ay  Tx ˘
Èb ˘
Í g ˙[Z ]= Í y¢  y  ax  T ˙
y˚
Î˚
Î 10 Tomasi and Kanade
Orthographic Projection Assumptions
• The camera model is orthographic.
• The positions of “p” points in “f” frames
(f>=3), which are not all coplanar, have
been tracked.
• The entire sequence has been acquired
before starting (batch mode).
• Camera calibration not needed, if we accept
3D points up to a scale factor. 11 Tomasi & Kanade
Image point {u
( fp , v fp )  f = 1, K , F , p = 1, K , P} È u11 K u1P ˘
Í
˙
M
Í
˙
ÍuF1 K uFP ˙
W =Í
˙
v11 K v1P ˙
Í
Í
˙
M
Í
˙
Îv F1 K v FP ˚ ÈU ˘
W = Í ˙
Í˙
ÍV ˙
Î˚ † Tomasi & Kanade
1P
af = Âup
P p =1 1P
bf = Â vp
P p =1 ~
u fP = u fP  a fP
~
v fP = v fP  b fP 12 s p = ( X p , YP , Z P )
u fP = i T ( sP  t f )
f
v fP = j T ( sP  t f )
f 3D world
point Orthographic
projection kf = if ¥ jf † 13 ~
u fp = u fP  a f 1PT
= i ( s p  t f )  Â i f ( sq  t f )
P q =1
T
f È
1P ˘
= i Í s P  Â sq ˙
P q =1 ˚
Î
T
f = iT sP
f ~ = iT s
u fP f P
~
v = jT s
fP f P Origin of world is at the
centroid of object points ~
ÈU ˘
~ Í˙
W = Í ˙
~
ÍV ˙
Î˚ 14 ~
u fP = i T sP
f
~
v = jT s
fP f P ~
ÈU ˘
~ Í˙
W = Í ˙
~
ÍV ˙
Î˚ È i1T ˘
Í˙
ÍM˙
T
~ Íif ˙
W = Í T ˙[s1 K sP ]= RS
Í j1 ˙
3XP
ÍM˙
Í T˙
Rank of S is 3, because points in 3D space are not
Í jf ˙
Î˚
2FX3 Coplanar Rank Theorem
Without noise, the registered
~
measurement matrix W is at
most of rank three.
È i1T ˘
Í˙
ÍM˙
T
~ Íi ˙
W = Í fT ˙[s1 K sP ]= RS
Í j1 ˙
ÍM˙
3XP
Í T˙
Í jf ˙
Î˚
2FX3 15 Translation ~
u fp = u fP  a f
~
u fp = u fP + a f ~
u fp = i T sP
f u fp = i f s p + a f u fp = i T ( s p  t f )
f a f is projection of camera translation along xaxis Translation u fp = i f s p + a f v fp = j f s p + b f
T
W = RS + tep
2FX3 3XP 2FX1 1XP t = (a1 , K , a f , b1 , K , b f )T
T
e p = (1, K1) 16 Translation
Projected camera translation can
be computed:
1P
T
 if t f = af = Âup
P p =1
1P
 j t = bf = Â vp
P p =1
T
ff Noisy Measurements
~
• Without noise, the matrix W must be at
most of rank 3. When noise corrupts the
~
images, however, W will not be rank 3.
Rank theorem can be extended to the case
of noisy measurements. 17 Approximate Rank SVD ~
W = O1SO2
2FXP PXP PXP Singular Value Decomposition (SVD)
• For some linear systems Ax=b,
Gaussian Elimination or LU
decomposition does not work, because
matrix A is singular, or very close to
singular. SVD will not only diagnose for
you, but it will solve it. 18 Singular Value Decomposition (SVD) Theorem: Any m by n matrix A, for which
m ≥ n ,can be written as A = O1SO2
mxn mxn nxn is diagonal S
O1 , O2 nxn are orthogonal T
O1T O1 = O2 O2 = I Singular Value Decomposition (SVD)
If A is square, then O1 , S, O2 are all square. O11 = O1T
T
O2 1 = O2 S1 = diag( 1
)
wj A = O1SO2
†
†
A1 = O2 diag( 1
)O1
wj †
19 Singular Value Decomposition (SVD)
The condition number of a matrix is the ratio
of the largest of the w j to the smallest of w j .
A matrix is singular if the condition number is
infinite, it is illconditioned if the condition
number is too large. Singular Value Decomposition (SVD) Ax = b
• If A is singular, some subspace of “x”
maps to zero; the dimension of the null
space is called “nullity”.
• Subspace of “b” which can be reached by “A”
is called range of “A”, the dimension of range is
called “rank” of A. 20 Range and Null Space
Ax=b x b b=0 Null space of A
Dimension of Null space is Nullity Range of A
Dimension of
range is rank of A Singular Value Decomposition (SVD) • If A is nonsingular its rank is “n”. • If A is singular its rank <n. • Rank+nullity=n 21 Singular Value Decomposition (SVD) A = O1SO2
• SVD constructs orthonormal basses of
null space and range.
of O1
spans range.
• Columns • Columns of O2
null space. with nonzero
with zero wj w j spans Solution of Linear System
• How to solve Ax=b, when A is singular?
• If “b” is in the range of “A” then system has
many solutions.
• Replace 1
wj by zero if x = O2 [diag ( wj = 0 1
)]O1T b
wj 22 Solution of Linear System
If b is not in the range of A, above eq still gives
the solution, which is the best possible solution,
it minimizes: r ≡ Ax  b  Approximate Rank
~
W = O1SO2
¢
¢
O1SO2 = O1¢S¢O2 + O1¢¢S¢¢O2¢ P3 3 O1 = [ 1¢ O1¢¢]
O
3 2F P3 È S¢ 0 ˘
S=Í
˙
Î 0 S¢¢˚ 3
P3 ¢
ÈO2 ˘ 3
O2 = Í ˙
¢ P3
ÎO2¢˚
P 23 Approximate Rank ˆ
¢
W = O1¢S¢O2
The best rank 3 approximation to the ideal
registered measurement matrix. Rank Theorem for noisy
measurement
The best possible shape and rotation
estimate is obtained by considering only
~
3 greatest singular values of W
together with the corresponding left,
right eigenvectors. 24 Approximate Rank ˆ = O¢[S¢]12
R
1 Approximate Rotation matrix ˆ = [S¢]12 O¢ Approximate Shape matrix
S
2
ˆ ˆˆ
W = RS This decomposition is not unique
ˆ
ˆ
ˆ
W = RQ Q1S ( )( ) Q is any 3X3 invertable matrix † Approximate Rank
ˆ
R = RQ How to determine Q? ˆ
ˆ
i fT QQT i fT = 1 ˆ
S =Q S
1 R and S are linear transformation of
approximate Rotation and shape matrices ˆT QQT ˆT = 1
jf
jf
ˆ
i fT QQT ˆT = 0
jf Rows of rotation matrix are unit vectors
and orthogonal 25 How to determine Q: Newton’s
Method
ˆ
ˆ
f1 (q) = iiT QQT iiT  1 = 0
f (q) = ˆ T QQT ˆ T  1 = 0
j
j MDq = e ˆ
f 3 (q) = i QQ ˆ1T = 0
j Dq = [Dq1,K, Dq9 ] 2 1 1 T
1 T M M ij = ˆ
ˆ
f 3 f  2 (q) = i fT QQT i fT  1 = 0
f 3 f 1 (q) = ˆ T QQT ˆ T  1 = 0
jf
jf ∂f i
∂q j e is error ˆ
f 3 f (q) = i fT QQT ˆ T = 0
jf † † Algorithm
• Compute SVD of
• define
• Compute 1
ˆ
R = O1¢[S¢] 2 ~
W = O1SO2
1
ˆ
¢
S = [S¢] 2 O2 Q • Compute R ˆ
= RQ ˆ
S = Q 1S 26 Hotel Sequence Results (rotations) 27 Selected Features Reconstructed Shape 28 Comparison House Sequence 29 Reconstructed Walls ../tomasiTr92Figures.pdf 30 Web Page
• http://vision.stanford.edu/cgibin/svl/publication/publication1992.cgi 31 ...
View
Full
Document
This note was uploaded on 06/12/2011 for the course CAP 6411 taught by Professor Shah during the Spring '09 term at University of Central Florida.
 Spring '09
 Shah

Click to edit the document details