Lecture-9.ppt - Lecture-9 Model-base Video Compression Li...

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: Lecture-9 Model-base Video Compression Li, Teklap Model-Based Image Coding 1 Model-Based Image Coding Model-Based Image Coding • The transmitter and receiver both posses the same 3D face model and texture images. • During the session, at the transmitter the facial motion parameters: global and local, are extracted. • At the receiver the image is synthesized using estimated motion parameters. • The difference between synthesized and actual image can be transmitted as residuals. 2 Candide Model ../CANDIDE.HTM 3 Face Model • Candide model has 108 nodes, 184 polygons. • Candide is a generic head and shoulder model. It needs to be conformed to a particular person’s face. • Cyberware scan gives head model consisting of 460,000 polygons. Wireframe Model Fitting • Fit orthographic projection of wireframe to the frontal view of speaker using Affine transformation. – Locate three to four features in the image and the projection of a model. – Find parameters of Affine transformation using least squares fit. – Apply Affine to all vertices, and scale (a + a ) 2 depth. 2 1 22 4 4 5 Synthesis • Collapse initial wire frame onto the image to obtain a collection of triangles. • Map observed texture in the first frame into respective triangles. • Rotate and translate the initial wire frame according to global and local motion, and collapse onto the next frame. • Map texture within each triangle from first frame to the next frame by interpolation. 6 Texture Mapping Video Phones Motion Estimation 7 Perspective Projection (optical flow) V V1 W W + W 2 ) - 3 x - W 3 y - 1 xy + 2 x 2 Z Z f f V V W W v = f ( 2 - W1 ) + W 3 x - 3 y + 2 xy - 1 y 2 Z Z f f u= f( Optical Flow Constraint Eq f xu + f y v + f t = 0 8 V V1 W W + W 2 ) - 3 x - W 3 y - 1 xy + 2 x 2 ) + f y Z Z f f V V W W ( f ( 2 - W1 ) + W 3 x - 3 y + 2 xy - 1 y 2 ) + f t = 0 Z Z f f fx ( f ( f f f )V1 + ( f y )V2 + ( ( f x x - f y y )V3 + Z Z Z 2 xy y x2 xy (- f x + fy - f y f )W1 + ( f x f + f x + f y )W 2 + f f f f ( f x y + f y x )W 3 = - f t ( fx f f f )V1 + ( f y )V2 + ( ( f x x - f y y )V3 + Z Z Z 2 xy y x2 xy (- f x + fy - f y f )W1 + ( f x f + f x + f y )W 2 + f f f f ( f x y + f y x )W 3 = - f t ( fx Ax = b Solve by Least Squares x = (V1,V2 ,V3 , W1, W2 , W3 ) † 9 Ax = b È V1 ˘ È ˘ÍV ˙ È M ˘ Í ˙Í 2 ˙ Í ˙ M Í ˙ Í V3 ˙ Í ˙ 2 2 f xy y x xy Í ˙= f f Í + fy - fy f ) ( fx f + fx + f y ) ( f x y + f y x)˙ Í W1 ˙ Í f t ˙ ( f x ) ( f y ) ( ( f x x - f y y ) (- f x Í ˙ Z f f f f Z Z Í˙ Í ˙ ÍW 2 ˙ Î M ˚ M Î ˚Í ˙ ÍW 3 ˙ Î˚ Comments • This is a simpler (linear) problem than sfm because depth is assumed to be known. • Since no optical flow is computed, this is called “direct method”. • Only spatiotemporal derivatives are computed from the images. 10 Problem • We have used 3D rigid motion, but face is not purely rigid! • Facial expressions produce non-rigid motion. • Use global rigid motion and non-rigid deformations. 3-D Rigid Motion X¢ = RX + T È X ¢˘ È 1 - a b ˘ È X ˘ ÈTX ˘ ÍY ¢ ˙ = Í a 1 - g ˙ ÍY ˙ + ÍTY ˙ Í˙Í ˙Í ˙ Í ˙ Í Z ¢ ˙ Í- b g 1 ˙ Í Z ˙ ÍTZ ˙ Î˚Î ˚Î ˚ Î ˚ È X ¢˘ Ê È 0 ÍY ¢ ˙ = Á Í a Í ˙ ÁÍ Í Z ¢ ˙ Á Í- b Î ˚ ËÎ -a 0 g b ˘ È1 0 0˘ ˆ È X ˘ ÈTX ˘ ˜ - g ˙ + Í0 1 0˙ ˜ ÍY ˙ + ÍTY ˙ ˙Í ˙Í ˙ Í ˙ 0 ˙ Í0 0 1˙ ˜ Í Z ˙ ÍTZ ˙ ˚Î ˚ ¯Î ˚ Î ˚ 11 3-D Rigid Motion È X ¢ - X ˘ È 0 - a b ˘ È X ˘ ÈTX ˘ ÍY ¢ - Y ˙ = Í a 0 - g ˙ ÍY ˙ + ÍTY ˙ Í ˙Í ˙Í ˙ Í ˙ Í Z ¢ - Z ˙ Í- b g 0 ˙ Í Z ˙ ÍTZ ˙ Î ˚Î ˚Î ˚ Î ˚ & ÈX ˘ È 0 - W3 Í& ˙ Í 0 ÍY ˙ = Í W 3 Í Z ˙ Í- W & W1 2 Î˚Î & X = W¥ X + V W 2 ˘ È X ˘ ÈTX ˘ - W1 ˙ ÍY ˙ + ÍTY ˙ ˙Í ˙ Í ˙ 0 ˙ Í Z ˙ ÍTZ ˙ ˚Î ˚ Î ˚ 3-D Rigid+Non-rigid Motion X¢ = RX + T + E( X)F Facial expressions † Action Units: -opening of a mouth -closing of eyes -raising of eyebrows F = (f1 , f 2 , K , f m )T 12 3-D Rigid+Non-rigid Motion ÈX ¢˘ È 1 Í˙ ÍY ¢ ˙ = Í a ÍZ ¢ ˙ Í-b Î˚Î -a 1 g m È ˘ ÍTX +  E1i ( X)f i ˙ i= 1 ˙ b ˘ÈX ˘ Í m Í ˙ ˙Í ˙ -g ˙ÍY ˙ + ÍTY +  E 2 i ( X)f i ˙ ˙ i= 1 1 ˙ÍZ ˙ Í ˚Î ˚ Í m ˙ ÍTZ +  E 3 i ( X)f i˙ Î ˚ i= 1 m È ˘ ÍTX +  E1i ( X)f i ˙ i= 1 ˙ ÈX ¢˘ ÊÈ 0 -a b ˘ È1 0 0˘ˆÈX ˘ Í m Í ˙ Í ˙ ÁÍ † -g ˙ + Í0 1 0˙˜ÍY ˙ + ÍTY +  E 2 i ( X)f i ˙ 0 ÍY ¢ ˙ = ÁÍ a ˙Í ˙˜Í ˙ ˙ i= 1 Í ˙ ÁÍ ˙Í ˙¯Í ˙ 0 ˚ Î0 0 1˚˜ÎZ ˚ Í ÎZ ¢ ˚ ËÎ-b g m Í ˙ ÍTZ +  E 3 i ( X)f i˙ Î ˚ i= 1 † 3-D Rigid+Non-rigid Motion m È ˘ ÍTX +  E1i ( X)f i ˙ i=1 ˙ ÈX ¢ - X ˘ È 0 -a b ˘ÈX ˘ Í m Í ˙ Í ˙Í ˙Í ˙ Y ¢ - Y ˙ = Í a 0 -g ˙ÍY ˙ + ÍTY +  E 2 i ( X)f i ˙ Í ˙ i=1 Í ˙Í ˙Í ˙ 0 ˚ÎZ ˚ Í ÎZ ¢ - Z ˚ Î-b g m Í ˙ ÍTZ +  E 3 i ( X)f i˙ Î ˚ i=1m ˙ ÈX ˘ È 0 -W3 Í˙Í ˙ ˙ = W3 0 ÍY ͆ ÍZ ˙ Î-W Î ˙ ˚ Í 2 W1 È ˘ ÍTX +  E1i ( X)f i ˙ i= 1 ˙ W2 ˘ÈX ˘ Í m Í ˙ ˙Í ˙ -W1˙ÍY ˙ + ÍTY +  E 2 i ( X)f i ˙ ˙ i= 1 ˙Í ˙ 0 ˚ÎZ ˚ Í m Í ˙ ÍTZ +  E 3 i ( X)f i˙ Î ˚ i= 1 ˙ X = W¥ X+ D † 13 3-D Rigid+Non-rigid Motion m ˙ X = -W3Y + W2 Z + V1 +  E1if i i=1 m ˙ Y = W3 X - W1Z + V2 +  E 2if i i= 1 m ˙ Z = -W2 X + W1Z + V3 +  E 3 if i i=1 † Perspective Projection (arbitrary flow) x= y= fX Z fY Z & & fZX - fXZ & & X Z & u=x= =f -x Z2 Z Z & & & & fZY - fYZ Y Z & v= y= = f -y Z2 Z Z 14 Perspective Projection (arbitrary flow) m ˙ X = -W3Y + W2 Z + V1 +  E1if i i=1 m ˙ Y = W3 X - W1Z + V2 +  E 2if i i= 1 m ˙ Z = -W2 X + W1Z + V3 +  E 3 if i & & fZX - fXZ & & X Z =f -x Z2 Z Z & & & & fZY - fYZ Y Z & v= y= = f -y Z2 Z Z & u=x= i=1 m † u= f( m V1 +  E1if i V3 +  E 3if i i=1 Z + W2 ) - i= 1 x - W3 y - Z m v = f( m V2 +  E 2 if i V3 +  E 3if i W1 W x + 2 x2 f f i=1 Z - W1 ) + W3 x - i= 1 Z y+ W2 W xy - 1 y f f † Optical Flow Constraint Eq f xu + f y v + f t = 0 15 Ax = b x = (V1 , V2 , V3 , W1 , W 2 , W 3 , f1 , f 2 , K , f m ) 16 Lecture-10 Estimation Using Flexible Wireframe Model 17 Main Points • Model photometric effects • Simultaneously compute 3D motion and adapt the wireframe model. Generalized Optical Flow Constraint f ( x, y, t ) = rN (t ).L df ( x, y, t ) dN = rL. dt dt Lambertian Model Albedo Surface Normal (-p,-q,1) dN f x u + f y v + f t = rL. dt 18 Lambertian Model S=L, light source f ( x, y ) = n.L = (nx , n y , nz ).(l x , l y , l z ) f ( x, y ) = n.L = ( 1 2 2 p + q +1 (- p,-q,1)).(l x , l y , l z ) Sphere z= (R 2 - x2 - y2 ) ∂z x =∂x z ∂z y q= =∂y z p= (nx , n y , nz ) = 1 ( x, y , z ) R 19 Sphere Vase (1, 0, 1) (-1, 1, 1) (-1,-1, 1) 20 Orthographic Projection & u = x = W 2 Z - W 3 y + V1 & v = y = W 3 x - W1Z + V2 (u,v) is optical flow & X = W¥ X + V & X = W 2 Z - W 3Y + V1 & Y = W X - W Z +V 3 1 2 & Z = W1Y - W 2 X + V3 Optical flow equation f x (W 2 Z - W 3 y + V1 ) + f y (W 3 x - W1Z + V2 ) + f t = rL. dN dt f x (W 2 Z - W 3 y + V1 ) + f y (W 3 x - W1Z + V2 ) + f t = È (- p¢,-q¢,1)T (- p,-q,1)T ˘ rL.Í ˙ 2 2 ¢ 2 + q¢ 2 + 1 Íp p + q + 1˙ Î ˚ Homework 3.1 Show this. Equation 24.8, page 473. 21 Error Function E= i 2 i ( x , y ) eithpatch Âe ( ( pi x1(ij ) + qi x2ij ) + ci = p j x1(ij ) + q j x2ij ) + c j constraint ei ( x, y ) = f x (W 3 y - W 2 ( pi x + q1 y + ci ) + V1 ) + f y (-W 3 x + W1 ( pi x + q1 y + ci ) + V2 ) + f t - W 2 + pi - W1 + qi , ) 1 + W 2 pi 1 - W1qi - r ( L1 , L2 , L3 ).( - W 2 + pi 2 - W1 + qi 2 1/ 2 (( ) +( ) + 1) 1 + W 2 pi 1 - W1qi Homework 3.3 (- pi ,-qi ,1) Show this. ( pi2 + qi2 + 1)1/ 2 (- Equation of a Planar Patch P (i ) = ( X 1(i ) , Y1(i ) , Z1(i ) ) 1 (i ) 2 P (i ) 2 (i ) 2 P1 (i ) 2 = ( X ,Y , Z ) P3(i ) = ( X 3(i ) , Y3(i ) , Z 3(i ) ) (i ) (i ) 1 (i ) P3 P2 P (i ) = ( X (i ) , Y (i ) , Z (i ) ) (i ) 1 (i ) P P .( P2 P ¥ P3 P (i ) ) = 0 1 22 Equation of a Planar Patch Z (i ) = pi X (i ) + qiY (i ) + ci Homework 3.2 Show this. pi = - ( (Y2(i ) - Y1(i ) )( Z 3(i ) - Z1(i ) ) - ( Z 2i ) - Z1(i ) )(Y3(i ) - Y1(i ) ) ( ( X 2i ) - X 1(i ) )(Y3(i ) - Y1(i ) ) - (Y2(i ) - Y1(i ) )( X 3(i ) - X 1(i ) ) qi = - ( ( ( Z 2i ) - Z1(i ) )( X 3(i ) - X 1(i ) ) - ( X 2i ) - X 1(i ) )( Z 3(i ) - Z1(i ) ) ( ( X 2i ) - X 1(i ) )(Y3(i ) - Y1(i ) ) - (Y2(i ) - Y1(i ) )( X 3(i ) - X 1(i ) ) ci = Z1(i ) + X 1(i ) Y1(i ) ( (Y2(i ) - Y1(i ) )( Z 3(i ) - Z1(i ) ) - ( Z 2i ) - Z1(i ) )(Y3(i ) - Y1(i ) ) + ( ( X 2i ) - X 1(i ) )(Y3(i ) - Y1(i ) ) - (Y2(i ) - Y1(i ) )( X 3(i ) - X 1(i ) ) ( ( ( Z 2i ) - Z1(i ) )( X 3(i ) - X 1(i ) ) - ( X 2i ) - X 1(i ) )( Z 3(i ) - Z1(i ) ) ( ( X 2i ) - X 1(i ) )(Y3(i ) - Y1(i ) ) - (Y2(i ) - Y1(i ) )( X 3(i ) - X 1(i ) ) Structure of Wireframe Model • Each triangular patch is either surrounded by two (if it is on the boundary of the wireframe) or three other triangles. 23 Neighboring patches must intersect at a straight line. ( x ij , y ij ) ( pi , qi , ci ) Z (i ) = pi X (i ) + qiY (i ) + ci pi x (ij ) + qi y (ij ) + ci = p j x (ij ) + q j y (ij ) + c j ( pj,qj,cj ) Main Points of Algorithm • Stochastic relaxation. • Each iteration visit all patches in a sequential order. – If, at present iteration none of neighboring patches of i have been visited yet, then pi, qi, ci are all independently perturbed. – If, only one of the neighbor, j, has been visited, then two parameters, say pi, qi, are independent and perturbed. The dependent variable ci is calculated from the equation: ( ij ) ( ij ) ( ij ) ( ij ) ci = p j x + qj y + c j - pi x - qi y 24 Main Points of Algorithm – If two of the neighboring patches, say j and k, have already been visited, i.e., the variables pk, qk, cik and pj, qj, cj have been updated, then only one variable pi is independent, and is perturbed. qi, ci can be evaluated as ci = p j x (ij ) + q j y (ij ) + c j - pi x (ij ) - qi y (ij ) pk x (ik ) + qk y (ik ) + ck - pi x (ik ) - ci qi = y (ik ) Main Points of Algorithm • The perturbation of structure parameters (surface normal) for each patch, results in the change of coordinates (X,Y,Z) of the nodes of wireframe. 25 Updating of (X,Y,Z): Patches i, j, k intersect at node n. pi X ( n ) + qiY ( n ) + ci = p j X ( n ) + q jY ( n ) + c j pi X ( n ) + qiY ( n ) + ci = pk X ( n ) + qk Y ( n ) + ck È X ( n ) ˘ È pi - p j Í (n) ˙ = Í Î Y ˚ Î pi - pk -1 qi - q j ˘ È c j - ci ˘ qi - qk ˙ Í- ci + ck ˙ ˚Î ˚ Z (i ) = pi X (i ) + qiY (i ) + ci Algorithm • Estimate light source direction • Initialize coordinates of all nodes using approximately scaled wireframe model (program2). Determine initial values of surface normals for each triangle. • Determine initial motion parameters based on selected feature correspondences and their depth values from wireframe model. (Assume motion parameters.) • (A) Compute the value of error function E. 26 • If error E is less than some threshold, then stop • Else – Perturb motion parameters (3 rotations and 2 translations) by random amount (zero mean Gaussian, s.d. equal to error E) (you can use uniform distribution) – Perturb structure parameters (p,q,c): • Perturb p, q, and c of first patch by adding random amount (zero mean Gaussian, s.d. equal to error E) • Increment count for all neighbors of patch-1 by 1 Uniform Distribution f ( x) = a b 1 (b - a ) ( a + b) 2 ( a - b) 2 2 s = variance = 12 X = mean = Use rand() in C to generate random number between a range. 27 • For patch 2 to n – If the count==1 » Perturb p and q » Compute c using equation for ci » Increment the count ci = p j x (ij ) + q j y (ij ) + c j - pi x (ij ) - qi y (ij ) – If count==2 » Perturb pi » Compute ci and qi using equations ci = p j x (ij ) + q j y (ij ) + c j - pi x (ij ) - qi y (ij ) pk x (ik ) + qk y (ik ) + ck - pi x (ik ) - ci qi = y (ik ) » Increment the count – If p, q and c for at least three patches intersecting at node are updated, then update the coordinates of the node using equation. È X ( n ) ˘ È pi - p j Í (n) ˙ = Í q - q j ÎY ˚ Î i • Go to step (A) -1 p j - pk ˘ È c j - ci ˘ q j - q k ˙ Í - c j + ck ˙ ˚Î ˚ Z (i ) = pi X (i ) + qiY (i ) + ci 28 ...
View Full Document

This note was uploaded on 06/12/2011 for the course CAP 6411 taught by Professor Shah during the Spring '09 term at University of Central Florida.

Ask a homework question - tutors are online