Lecture-6.ppt - Lecture-6 Szeliski, Mann & Picard...

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: Lecture-6 Szeliski, Mann & Picard Szeliski Projective 1 Projective f ( X ¢, t - 1) x x f ( X , t) a1 x + a2 y + b1 c1 x + c 2 y + 1 a x + a4 y + b2 y¢ = 3 c1 x + c 2 y + 1 x¢ = † Szeliski x¢ = a1 x + a2 y + b1 c1 x + c2 y + 1 y¢ = a3 x + a4 y + b2 c1 x + c2 y + 1 Projective E = Â[ f ( x ¢, y ¢) - f ( x, y )]2 =  e 2 min † 2 Szeliski Motion Vector: m = [a1 a2 a3 a4 b1 b2 T c1 c2 ] Szeliski (Levenberg-Marquadet) ∂e ∂e a kl =  n n n ∂m k ∂ml bk = - e ∂en ∂m k gradient Dm = ( A + lI ) -1 b † † Approximation of Hessian (JTJ, Jacobian) 3 ∂e ∂e ∂x¢ ∂e ∂y¢ = + ¢ ∂a1 ∂y¢ ∂a1 ∂a1 ∂x E =  [ f ( x¢, y¢) - f ( x, y )]2 =  e 2 a x + a2 y + b1 x¢ = 1 c1 x + c2 y + 1 a x + a4 y + b2 y¢ = 3 c1 x + c2 y + 1 Approximation of Hessian È ∂e1 Í ∂m Í1 Í ∂e1 Í∂m 2 Í ∂e1 Í Í∂m 3 Í ∂e1 Í∂m T J =Í 4 ∂e1 Í Í∂m 5 Í ∂e1 Í∂m 6 Í ∂e1 Í Í∂m 7 Í ∂e1 Í Î∂m 8 ∂e 2 ∂m1 ∂e 2 ∂m 2 ∂e 2 ∂m 3 ∂e 2 ∂m 4 ∂e 2 ∂m 5 ∂e 2 ∂m 6 ∂e 2 ∂m 7 ∂e 2 ∂m 8 K K K K K K K K ∂en ˘ ∂m1 ˙ È ∂e1 ˙ ∂en ˙ Í∂m Í1 ∂m2 ˙ Í ∂e2 ∂en ˙ Í∂m1 ˙ ∂m3 ˙ Í ∂en ˙ Í Í ∂m4 ˙ ∂en ˙ J = Í Í ˙ ∂m5 ˙ ÍM ∂en ˙ Í Í ∂m6 ˙ Í ∂en ˙ Í ˙ ∂m7 ˙ Í ∂en ∂en ˙ Í Î∂m1 ˙ ∂m8 ˚ A = JT J † ∂e ∂e a kl =  n n n ∂m k ∂ml ∂e1 ∂m 2 ∂e 2 ∂m 2 ∂e1 ∂m3 ∂e2 ∂m3 ∂e1 ∂m 4 ∂e2 ∂m 4 ∂e1 ∂m5 ∂e2 ∂m5 ∂e1 ∂m 6 ∂e2 ∂m 6 ∂e1 ∂m 7 ∂e2 ∂m 7 M M M M M M ∂e n ∂m 2 ∂en ∂m3 ∂en ∂m 4 ∂en ∂m5 ∂en ∂m 6 ∂en ∂m 7 ∂e1 ˘ ∂m8 ˙ ˙ ∂e2 ˙ ∂m8 ˙ ˙ ˙ ˙ ˙ ˙ M˙ ˙ ˙ ˙ ˙ ∂en ˙ ˙ ∂m8 ˚ † † † 4 Gradient Vector È ∂e n ˘ Í - en ∂a ˙ 1˙ n Í Í- n  e ∂en ˙ Í n n ∂a2 ˙ Í ∂e n ˙ Í - en ˙ ∂a3 ˙ Ín Í - e ∂e n ˙ Í Â n ∂a4 ˙ b =Í n ∂e ˙ Í - e n ˙ Í n ∂b1 ˙ ∂e n ˙ Í Í - en ∂b ˙ 2 n Í ∂e ˙ Í - en n ˙ ∂c1 ˙ Ín Í ∂e n ˙ Í - en ∂c ˙ În 2˚ † Partial Derivatives wrt motion parameters ∂e ∂e ∂x¢ ∂e ∂y¢ = + ∂a1 ∂x¢ ∂a1 ∂y¢ ∂a1 x¢ = a1 x + a2 y + b1 , c1 x + c2 y + 1 y¢ = a3 x + a4 y + b2 c1 x + c2 y + 1 ∂x¢ x = ∂a1 c1 x + c2 y + 1 ∂x¢ y = ∂a2 c1 x + c2 y + 1 ∂x¢ =0 ∂a3 ∂x¢ =0 ∂a4 ∂x¢ 1 = ∂b1 c1 x + c2 y + 1 ∂x¢ =0 ∂b2 ∂x¢ - x(a1 x + a2 y + b1 ) = ∂c1 (c1 x + c2 y + 1)2 ∂y¢ =0 ∂a1 ∂y¢ =0 ∂a2 ∂y¢ x = ∂a3 c1 x + c2 y + 1 ∂y¢ y = ∂a4 c1 x + c2 y + 1 ∂y¢ =0 ∂b1 ∂y¢ 1 = ∂b2 c1 x + c2 y + 1 ∂y¢ - x(a3 x + a4 y + b2 ) = ∂c1 (c1 x + c2 y + 1)2 ∂x¢ - y (a1 x + a2 y + b1 ) ∂y¢ - y (a3 x + a4 y + b2 ) = = ∂c2 (c1 x + c2 y + 1)2 ∂c2 (c1 x + c2 y + 1)2 5 Partial derivatives wrt image coordinates E =  [ f ( x¢, y¢) - f ( x, y )]2 =  e 2 ∂e = fx' ∂x ∂e = fy' ∂y † Partial derivatives ∂e ∂e ∂x¢ ∂e ∂y¢ x = + = f x¢ ∂a1 ∂x¢ ∂a1 ∂y¢ ∂a1 c1 x + c2 y + 1 ¢ ∂e ∂y¢ ∂e ∂e ∂x y = + = f x¢ ∂a2 ∂x¢ ∂a2 ∂y¢ ∂a2 c1 x + c2 y + 1 ∂e ∂e ∂x¢ ∂e ∂y¢ x = + = f y¢ ∂a3 ∂x¢ ∂a3 ∂y¢ ∂a3 c1 x + c2 y + 1 ∂e ∂e ∂x¢ ∂e ∂y¢ y = + = f y¢ ∂a4 ∂x¢ ∂a4 ∂y¢ ∂a4 c1 x + c2 y + 1 ∂e ∂e ∂x¢ ∂e ∂y¢ 1 = + = f x¢ ∂b1 ∂x¢ ∂b1 ∂y¢ ∂b1 c1 x + c2 y + 1 ∂e ∂e ∂x¢ ∂e ∂y¢ 1 = + = f y¢ ∂b2 ∂x¢ ∂b2 ∂y¢ ∂b2 c1 x + c2 y + 1 - x(a3 x + a4 y + b2 ) ∂e ∂e ∂x¢ ∂e ∂y¢ - x(a1 x + a2 y + b1 ) = + = f x¢ + f y¢ ∂c1 ∂x¢ ∂c1 ∂y¢ ∂c1 (c1 x + c2 y + 1)2 (c1 x + c2 y + 1)2 - y (a3 x + a4 y + b2 ) ∂e ∂e ∂x¢ ∂e ∂y¢ - y (a1 x + a2 y + b1 ) = + = f x¢ + f y¢ ∂c2 ∂x¢ ∂c2 ∂y¢ ∂c2 (c1 x + c2 y + 1)2 (c1 x + c2 y + 1)2 6 Gradient Vector È ˘ x - ef x ¢ Í ˙ c1 x + c 2 y + 1 Í ˙ y Í ˙ - ef x ¢ c1 x + c 2 y + 1 Í ˙ Í ˙ x - ef y ¢ Í ˙ c1 x + c 2 y + 1 Í ˙ y Í ˙ - ef y ¢ Í ˙ c1 x + c 2 y + 1 ˙ 1 b=Í - ef x ¢ Í ˙ c1 x + c 2 y + 1 Í ˙ 1 Í ˙ - ef y ¢ Í ˙ c1 x + c 2 y + 1 Í È f (a x + a y + b ) + f (a x + a y + b ) ˘˙ 2 1 y¢ 3 4 2˙ Í exÍ x ¢ 1 ˙ 2 Í Í ˙ (c1 x + c 2 y + 1) Î ˚˙ Í È f (a x + a y + b ) + f (a x + a y + b ) ˘˙ 2 1 y¢ 3 4 2˙ Í eyÍ x ¢ 1 ˙ 2 ÍÂ Í ˙˙ (c1 x + c 2 y + 1) Î ˚˚ Î † Szeliski (Levenberg-Marquadet) • Start with some initial value of m, and l=.001 • For each pixel I at • Compute ( x¢, y¢) ( xi , yi ) using projective transform. • Compute e = f ( x¢, y¢) - f ( x, y ) • Compute ∂e ∂e ∂x¢ ∂e ∂y¢ = + ∂mk ∂x¢ ∂mk ∂y¢ ∂mk 7 Szeliski (Levenberg-Marquadet) -Compute A and b -Solve system ( A - lI )Dm = b -Update m t +1 = m t + Dm Szeliski (Levenberg-Marquadet) • check if error has decreased, if not increase l by a factor of 10 and compute a new Dm • If error has decreased, decrease l by a factor of 10 and compute a new Dm • Continue iteration until error is below threshold. 8 Mann & Picard Projective Projective Flow (weighted) u f x + v f y + ft = 0 Optical Flow const. equation uT f x + f t = 0 x¢ = Ax + b CT x + 1 u = x¢ - x = Projective transform Ax + b CT x + 1 9 Projective Flow (weighted) e flow =  ( u Tf X + f t ) 2 =  (( Ax + b - x)T f x + f t ) 2 T Cx + 1 =  ((Ax + b - (CT x + 1) x )T fx + (CT x + 1) f t ) 2 minimize † Projective Flow (weighted) • (b) Homework 2 Derive this equation Due Sept 25 T x t ( ff T )a =  (x f - f )f a = [a1, a2 , b1, a3 , a4 , b2 , c1, c 2 ]T f t = [ f x x, f x y, f x , f y x, f y y, f y , xf t - x 2 f x - xyf y , yf t - xyf x - y 2 f y ] † 10 Projective Flow (unweighted) Bilinear x¢ = Ax + b CT x + 1 Taylor Series & remove Square terms u + x = a1 + a2 x + a3 y + a4 xy v + y = a5 + a6 x + a7 y + a8 xy 11 Pseudo-Perspective x¢ = Ax + b CT x + 1 Taylor Series x + u = a1 + a2 x + a3 y + a4 x 2 + a5 xy y + v = a6 + a7 x + a8 y + a4 xy + a5 y 2 Projective Flow (unweighted) e flow =  ( u Tf X + f t ) 2 Minimize 12 Bilinear and Pseudo-Perspective T ( FF )q = - f t F FT = [ f x ( xy, x, y,1), FT = [ f x ( x, y,1) c1 = x 2 f x + xyf x ( c) homewwork Derive these eqs Sept 25 f y ( xy, x, y,1)] bilinear f y ( x, y,1) c1 c 2 ] Pseudo perspective c 2 = xyf x + †2 f y y † Algorithm-1 • Estimate “q” (using approximate model, e.g. bilinear model). • Relate “q” to “p” – select four points S1, S2, S3, S4 – apply approximate model using “q” to compute ( x k¢ , y k¢ ) – estimate exact “p”: 13 True Projective x¢ = a1 x + a2 y + b1 c1 x + c2 y + 1 y¢ = a3 x + a4 y + b1 c1 x + c2 y + 1 ¢ ¢ È xk ˘ È xk y k 1 0 0 0 - xk xk Í y ¢ ˙ = Í0 0 0 x y 1 - x y ¢ k k kk Î k˚ Î a = [a1 a2 b1 a3 a4 b2 ¢ - y k xk ˘ a ¢˚ - yk yk ˙ T c1 c1 ] 14 ¢ È x1 ˘ È x1 Í y¢ ˙ Í 0 Í 1˙ Í Í˙Í Í ˙=Í Í˙Í Í xk ˙ Í xk ¢ Í˙Í Í yk ˙ Í 0 Î ¢˚ Î y1 0 1 0 0 x1 0 y1 0 1 ¢ - x1 x1 ¢ - x1 y1 yk 0 10 0 xk 0 yk 0 - xk x ¢ ¢ 1 - xk y k ¢ - y1 x1 ˘ ¢ - y1 y1 ˙ ˙ ˙ ˙a ˙ ¢ - y k xk ˙ ˙ - yk y¢ ˙ ˚ P = Aa Perform least squares fit to compute a. Final Algorithm • A Gaussian pyramid of three or four levels is constructed for each frame in the sequence. • The parameters “p” are estimated at the top level of the pyramid, between the two lowest resolution images, “g” and “h”, using algorithm-1. 15 Final Algorithm • The estimated “p” is applied to the next higher resolution image in the pyramid, to make images at that level nearly congruent. • The process continues down the pyramid until the highest resolution image in the pyramid is reached. Video Mosaics • Mosaic aligns different pieces of a scene into a larger piece, and seamlessly blend them. – High resolution image from low resolution images – Increased filed of view 16 Steps in Generating A Mosaic • Take pictures • Pick reference image • Determine transformation between frames • Warp all images to the same reference view Applications of Mosaics • • • • Virtual Environments Computer Games Movie Special Effects Video Compression 17 Steve Mann Sequence of Images 18 Projective Mosaic Affine Mosaic 19 Building Wal-Mart 20 Scientific American Frontiers Scientific American Frontiers 21 Head-mounted Camera at Restaurant MIT Media Lab 22 Webpages • http://n1nlf1.eecg.toronto.edu/tip.ps.gz Video Orbits of the projective group, S. Mann and R. Picard. • http://wearcam.org/pencigraphy (C code for generating mosaics) 23 Webpages • http://ww-bcs.mit.edu/people/adelson/papers.html – The Laplacian Pyramid as a compact code, Burt and Adelson, IEEE Trans on Communication, 1983. • J. Bergen, P. Anandan, K. Hanna, and R. Hingorani, “Hierarchical Model-Based Motion Estimation”, ECCV-92, pp 237-22. Webpages • http://www.cs.cmu.edu/afs/cs/project/cil/ftp/html/vsource.html (c code for several optical flow algorithms) • ftp://csd.uwo.ca/pub/vision Performance of optical flow techniques (paper) Barron, Fleet and Beauchermin 24 Webpages • http://www.wisdom.weizmann.ac.il/~irani/abstract s/mosaics.html (“Efficient representations of video sequences and their applications”, Michal Irani, P. Anandan, Jim Bergen, Rakesh Kumar, and Steve Hsu) • R. Szeliski. “Video mosaics for virtual environments”, IEEE Computer Graphics and Applications, pages,22-30, March 1996. • M. Irani and P. Anandan, Video Indexing Based on Mosaic Representations. Proceedings of IEEE, May,1998. • http://www.wisdom.weizmann.ac.il/~irani/a bstracts/videoIndexing.html 25 Homework Due Sept 25 • (a) Derive linear system equation in Anandan’s method Lecture 5, page 14, top slide. • (b) Derive equations for Mann’s method (weighted) Lecture 6, page 10. • (c) Derive equations for Mann’s method (unweighted) Lecture 6, page 13. Program-1 Due Oct 2 • (a) Implement Anandan’s algorithm using affine transformation. To show the results generate a mosaic. • (b) Implement Szeliski’s algorithm using projective transformation. To show the results generate a mosaic. • (c ) Implement Mann’s algorithm using projective transformation. To show the results generate a mosaic. • Implement all four steps: – Pyramid construction – Motion estimation – Image warping – Coarse-to-fine refinement •. 26 ...
View Full Document

This note was uploaded on 06/12/2011 for the course CAP 6411 taught by Professor Shah during the Spring '09 term at University of Central Florida.

Ask a homework question - tutors are online