# Lecture-11 - Lecture-11 Synthesizing Realistic Facial...

This preview shows page 1. Sign up to view the full content.

This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: Lecture-11 Synthesizing Realistic Facial Expressions from Photographs: Pighin et al SIGGRAPH’98 Synthesizing Realistic Facial Expressions from Photographs Pighin et al SIGGRAPH’98 1 The Artist’s Complete Guide to Facial Expression: Gary Faigin • There is no landscape that we know as well as the human face. The twenty-five-odd square inches containing the features is the most intimately scrutinized piece of territory in existence, examined constantly, and carefully, with far more than an intellectual interest. Every detail of the nose, eyes, and mouth, every regularity in proportion, every variation from one individual to the next, are matters about which we are all authorities. Main Points • One view is not enough. • Fitting of wire frame model to the image is a complex problem (pose estimation) • Texture mapping is an important problem 2 Synthesizing Realistic Facial Expressions • Select 13 feature points manually in face image corresponding to points in face model created with Alias. • Estimate camera poses and deformed 3d model points. • Use these deformed values to deform the remaining points on the mesh using interpolation. Synthesizing Realistic Facial Expressions • Introduce more feature points (99) manually, and compute deformations as before by keeping the camera poses fixed. • Use these deformed values to deform the remaining points on the mesh using interpolation as before. • Extract texture. • Create new expressions using morphing. 3 Synthesizing Realistic Facial Expressions 3D Rigid Transformation È X ¢˘ È X ˘ Èr11 r12 r13 ˘ È X ˘ ÈTX ˘ Í ˙ Í˙ Í ˙Í ˙ Í ˙ Y ¢ ˙ = R ÍY ˙ + T = Ír21 r22 r23 ˙ ÍY ˙ + ÍTY ˙ Í ÍZ ¢ ˙ ÍZ ˙ Ír31 r32 r33 ˙ ÍZ ˙ ÍTZ ˙ Î ˚ Î˚ Î ˚Î ˚ Î ˚ Camera coordinates Wireframe coordinates X i¢k Yi¢k xi¢k = f k k , yi¢k = f k k Z i¢ Z i¢ perspective K=camera no. 4 3D Rigid Transformation k X i¢k k k Yi¢ xi¢ = f , yi¢ = f k Z i¢ Z i¢k k k k k k r11 X i + r12Yi + r13 Z i + TXk xi¢ = f k k k k r31 X i + r32Yi + r33 Z i + TZk k k k k r21 X i + r22Yi + r23 Z i + TYk yi¢ = f k k k k r31 X i + r32Yi + r33 Z i + TZk k Model Fitting k k rx p i + TX xi¢ k = f k k rz p i + TZk yi¢ k = f k k ry p i + TYk rzk p i + TZk 5 Model Fitting k k rx p i + TX xi¢ = f k k rz p i + TZk k k yi¢ = f k k ry p i + TYk rzk p i + TZk hk = k k rx p i + TX xi¢ = sk 1 + h k rzk p i 1k , s = f kh k k TZ k k yi¢ = sk k ry p i + TYk 1 + h k rzk p i Model Fitting k k rx p i + TX xi¢ = sk 1 + h k rzk p i k k yi¢ = sk k ry p i + TYk 1 + h k rzk p i wik = (1 + h k (rzk .p i )) -1 k wik ( xi¢k + xi¢kh k (rzk .p i ) - s k (rX .p i + TXk )) = 0 wik ( yi¢k + yi¢kh k (rzk .p i ) - s k (rYk .p i + TYk )) = 0 6 Model Fitting • Solve for unknowns in five steps: s k ; p i ; R k ; TXk , TYk ;h k • Use linear least squares fit. • When solving for an unknown, assume other parameters are known. Least Squares Fit a j .x - b j = 0 k wik ( xi¢k + xi¢kh k (rzk .p i ) - s k (rX .p i + TXk )) = 0 Â (a j .x - b j ) 2 wik ( yi¢k + yi¢kh k (rzk .p i ) - s k (rYk .p i + TYk )) = 0 j Â (a a j j T j )x = Â bja j j Update for p k b 2k + 0 = wi ( s k Txk - xik ) k b 2k +1 = wi ( s k Tyk - yik ) a2 k + 0 = wi ( xikh k rzk - s k rxk ) a2 k +1 = wi ( yikh k rzk - s k ryk ) k k 7 a j .x - b j = 0 Â (a .x - b ) j k wik ( xi¢k + xi¢kh k (rzk .p i ) - s k (rX .p i + TXk )) = 0 2 j wik ( yi¢k + yi¢kh k (rzk .p i ) - s k (rYk .p i + TYk )) = 0 j Â (a a j j T j ) x = Âb j a j j Update for sk † 2 k +0 = wi k (rxk . pi + t xk ) a b 2k + 0 = wi ( xik + xikh k (rzk . pi )) k b 2k +1 = wi ( yik + yikh k (rzk . pi )) k a2 k +1 = wi (ryk . pi + t y ) k k xi¢ + xi¢h (rz .p i ) - s (rx .p i +Tx ) = 0 y i¢ + y i¢h (rz .p i ) - s (r y .p i +T y ) = 0 Solving for Tx and Ty sTx = xi¢ + xi¢h (rz .p i ) - s (rx .p i ) a0 = s, b0 = xi¢ + xi¢h (rz .p i ) - s (rx .p i ) sT y = y i¢ + y i¢h (rz .p i ) - s (r y .p i ) a0 = s, b0 = yi¢ + yi¢h (rz .p i ) - s (ry .p i ) 8 xi¢ + xi¢h (rz .p i ) - s (rx .p i +Tx ) = 0 y i¢ + y i¢h (rz .p i ) - s (r y .p i +T y ) = 0 Solving for h a0 = xi¢(rz .p i ), b0 = s (rx .p i +Tx ) - xi¢ a1 = yi¢(rz .p i ), b1 = s (ry .p i +Ty ) - y¢ Rotation Around an Arbitrary Axis (Rodriguez’s Formula) V = (V .n)n + (V - (V .n)n) HW 4.1 ¢ V^ = cosq (V - (V .n)n) + sin q (n ¥ (V - (V .n)n) ¢ VC = (V .n)n ¢ ¢ V ¢ = V^ + VC V ¢ = cosq V + sin q n ¥ V + (1 - cosq )(V .n)n V ¢ = V + sin q n ¥ V + (1 - cosq )n ¥ (n ¥ V ) n ¥ (n ¥ V ) = (V .n)n - V n ¥ (n ¥ V ) + V = (V .n)n 9 Rotation Around an Arbitrary Axis (Rodriguez’s Formula) V ¢ = V + sin q n ¥ V + (1 - cosq ) n ¥ (n ¥ V ) V ¢ = R(n,q )V R(n,q ) = I + sin q X (n) + (1 - cosq ) X 2 (n) È0 Í X ( n) = Í n z Í- n y Î - nz 0 nx HW 4.2 nx ˘ ˙ - nx ˙ 0˙ ˚ Rotation Around an Arbitrary Axis (Rodriguez’s Formula) r =|| r || r = qn || r | n= r || r | R(n,q ) = I + sin q X (n) + (1 - cosq ) X 2 (n) È0 Í X ( n) = Í n z Í- n y Î - nz 0 nx R(r ,q ) = I + sin q È0 Í X (r ) = Í rz Í- ry Î - rz 0 rx nx ˘ ˙ - nx ˙ 0˙ ˚ † X (r ) X 2 (r ) + (1 - cosq ) || r || || r ||2 rx ˘ ˙ - rx ˙ 0˙ ˚ 10 R it +1 ~ it ¨ RR R ( m) = I + sin q X ( m) q + X ~ R ª I + X ( m) 2 q ( m) 2 (1 - cos q ) m = qn = (mx , m y , mz ) ~ k = (1,-m , m ) r x z y ~ k = (m ,1,-m ) ry z x ~ k = (-m , m ,1) rz y x k wik ( xi¢k + xi¢kh k (rzk .p i ) - s k (rX .p i + TXk )) = 0 wik ( yi¢k + yi¢kh k (rzk .p i ) - s k (rYk .p i + TYk )) = 0 R it +1 ~ k = (1,-m , m ) rx z y ~ it ¨ RR ~ k = (m ,1,-m ) ry z x k wi ( x + x h (~ k .qi ) - s k (~ k .qi + t x )) = 0 rz rx k w ( y k + y k (~ k .q ) - s k (~ k .q + t k )) = 0 r r k i k i k i i i z i y i qi = R it pi ~ k = (-m , m ,1) rz y x y 11 Solving for rotation: k wik ( xi¢k + xi¢kh k (rzk .p i ) - s k (rX .p i + TXk )) = 0 k i k k k k z k k Y k Y w ( yi¢ + yi¢ h (r .p i ) - s (r .p i + T )) = 0 k wi ( xi¢k + xi¢kh (~ k .qi ) - s k (~ k .qi + Txk )) = 0 rz rx k w ( y¢k + y¢kh (~ k .q ) - s k (~ k .q + T k )) = 0 r r i i i z i y i y ~ R it +1 ¨ R R it qi = R it pi ~ k = (-m , m ,1) rz y x ~ k = (1,-m , m ) r x z y ~ k = (m ,1,-m ) ry z x xi¢ + xi¢h (-m y q x + m x q y + q z ) - s (q x - m z q y + m y q z + Tx ) = 0 y i¢ + y i¢h (-m y q x + m x q y + q z ) - s (m z q x + q y - m x q z + T y ) = 0 xi¢ + xi¢h (-m y q x + m x q y + q z ) - s (q x - m z q y + m y q z + Tx ) = 0 y i¢ + y i¢h (-m y q x + m x q y + q z ) - s (m z q x + q y - m x q z + T y ) = 0 È x¢hq y Í Î - x¢hq x - sq z È Í y¢hq + sq y z Î - y¢hq x È x¢hq y - x¢hq x + sq z a0 = Í Î È a1 = Í Î y¢hq y - sq z - y¢hq x Èm ˘ sq y ˘ Í x ˙ È sTx - x¢ - x¢hq z + sq x ˘ ˙ Ím y ˙ = Í ˙ ˚ ˚ Ím ˙ Î Î z˚ È mx ˘ ˘Í ˙ È ˘ m= - sq x ˙ Í y ˙ Í sTy - y¢ - y¢hq z + sq y ˙ ˚ Ím ˙ Î ˚ Î z˚ sq y ˘ È sTx - x¢ - x¢hq z + sq x ˘ ˙, b0 = Í ˙ Î ˚ ˚ ˘ È ˘ ,b = sq x ˙ 1 Í sTy - y¢ - y¢hq z + sq y ˙ ˚ Î ˚ Â (a a j j T j )x = Â bja j j 12 13 ...
View Full Document

{[ snackBarMessage ]}

Ask a homework question - tutors are online