MITMAS_531F09_lec04

MITMAS_531F09_lec04 - MIT Media Lab Computational Camera...

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: MIT Media Lab Computational Camera & Photography: Camera Culture Ramesh Raskar MIT Media Lab http:// CameraCulture . info/ Plan for Today Computational Illumination Introduction to Lightfields Assignment #2: Optics A: Virtual Optical Bench B: Lightfields Check Wiki for Reading Material Pl edit, add URLs, figures, comments etc Everyone on comp-cam mailing list? Listeners: Pl send a topic you would like to present (15-20 mins) Youtube videos on camera tutorial (DoF etc) http://www.youtube.com/user/MPTutor Final Project Ideas User interaction device Camera based Illumination based Photodetector or line-scan camera Capture the invisible Tomography for internals Structured light for 3D scanning Fluorescence for transparent materials Cameras in different EM/other spectrum Wifi, audio, magnetic, haptic, capacitive Visible Thermal IR segmentation Thermal IR (emotion detection, motion detector) Multispectral camera, discriminating (camelsand) Illumination Multi-flash with lighfield Schielren photography Strobing and Colored strobing External non-imaging sensor Camera with gyro movement sensors, find identity of user Cameras with GPS and online geo-tagged photo collections Interaction between two cameras (with lasers on-board) Optics Lightfield Coded aperture Bio-inspired vision Time Time-lapse photos Motion blur Sample Final Projects Schlieren Photography (Best project award + Prize in 2008) Camera array for Particle Image Velocimetry BiDirectional Screen Looking Around a Corner (theory) Tomography machine .. .. Auto Focus Contrast method compares contrast of images at three depths, if in focus, image will have high contrast, else not Phase methods compares two parts of lens at the sensor plane, if in focus, entire exit pupil sees a uniform color, else not - assumes object has diffuse BRDF Homeworks Submit to class website Commented Source code Input images and output images PLUS intermediate results CREATE a webpage and send me a link Ok to use online software HDRshop Update results on Flickr (group) page Second Homework: Option A Extending Andrew Adam's Virtual Optical Bench (a) Create new optical bench Ability to add new elements Courtesy of Andrew Adams. Used with permission. (b) Form images Integrate light from multiple rays Show image effects (DoF, LF etc) Synthetic aperture videography Vaish, V., et al. "Using Plane + Parallax for Calibrating Dense Camera Arrays." Proceedings of CVPR 2004. Courtesy of IEEE. Used with permission. 2004 IEEE. Second HW (b): Lightfield Photography Translate camera and take photos Show refocussing and see-thru effects http://lightfield.stanford.edu/lfs.html Part 1 Create images with plane of focus at different depth Create images with variable depth of field (just use fewer images) Create see-thru effects (just small depth of field) Find depth using max-contrast operator Part 2 Images with SLANTED plane of focus See-thru effect by elminating foreground color pixels Extra credit: Create new bokeh (point spread function) Use high depth complexity, colorful, point specular (sphere) objects Second HW (c): Lightfield Photography Do ALL in software using Virtual Optical Bench Translate camera and take photos Show refocussing and see-thru effects http://lightfield.stanford.edu/lfs.html Part 1 Create images with plane of focus at different depth Create images with variable depth of field (just use fewer images) Create see-thru effects (just small depth of field) Find depth using max-contrast operator Part 2 Images with SLANTED plane of focus See-thru effect by elminating foreground color pixels Extra credit: Create new bokeh (point spread function) Use high depth complexity, colorful, point specular (sphere) objects Computational Illumination, Part 2 What are annoyances in photography ? Why CCD camera behaves retroreflective? Youtube videos on camera tutorial (DoF etc) http://www.youtube.com/user/MPTutor Poll When: Google Earth Live? Why? Commercially exploited pervasive recording? http://www.youtube.com/watch?v=_J7qE6frzz8 http://www.newscientist.com/article/dn17854-live-videomakes-google-earth-cities-bustle.html Measuring Light I say that if the front of a building--or any open piazza or field-- which is illuminated by the sun has a dwelling opposite to it, and if, in the front which does not face that sun, you make a small round hole, all the illuminated objects will project their images through that hole and be visible inside the dwelling on the opposite wall which may be made white; and there, in fact, they will be upside down, and if you make similar openings in several places in the same wall you will have the same result from each. Hence the images of the illuminated objects are all everywhere on this wall and all in each minutest part of it. - (The notebooks of Leonardo da Vinci) Next several slides by Ashok Veeraraghavan What do we see? 3D world 2D image Point of observation Images courtesy of Stephen E. Palmer. Used with permission. Figures Stephen E. Palmer, 2002 What do we see? 3D world 2D image Painted backdrop Images courtesy of Stephen E. Palmer. Used with permission. The Plenoptic Function Figure removed due to copyright restrictions. Q: What is the set of all things that we can ever see? A: The Plenoptic Function (Adelson & Bergen) Let's start with a stationary person and try to parameterize everything that he can see... Grayscale snapshot Figure removed due to copyright restrictions. P(,) is intensity of light Seen from a single view point At a single time Averaged over the wavelengths of the visible spectrum (can also do P(x,y), but spherical coordinate are nicer) Color snapshot Figure removed due to copyright restrictions. P(,,) is intensity of light Seen from a single view point At a single time As a function of wavelength A movie Figure removed due to copyright restrictions. P(,,,t) is intensity of light Seen from a single view point Over time As a function of wavelength Holographic movie Figure removed due to copyright restrictions. P(,,,t,VX,VY,VZ) is intensity of light Seen from ANY viewpoint Over time As a function of wavelength The Plenoptic Function Figure removed due to copyright restrictions. Can reconstruct every possible view, at every moment, from every position, at every wavelength Contains every photograph, every movie, everything that anyone has ever seen. P(,,,t,VX,VY,VZ) Sampling Plenoptic Function (top view) Ray Let's not worry about time and color: P(,,VX,VY,VZ) 5D 3D position 2D direction Courtesy of Rick Szeliski and Michael Cohen. Used with permission. Slide by Rick Szeliski and Michael Cohen Ray No Occluding Objects P(,,VX,VY,VZ) 4D 2D position 2D direction The space of all lines in 3-D space is 4D. Courtesy of Rick Szeliski and Michael Cohen. Used with permission. Slide by Rick Szeliski and Michael Cohen Lumigraph/Lightfield - Organization 2D position 2D direction s Courtesy of Rick Szeliski and Michael Cohen. Used with permission. Slide by Rick Szeliski and Michael Cohen Lumigraph - Organization 2D position 2D position s 2 plane parameterization u Courtesy of Rick Szeliski and Michael Cohen. Used with permission. Slide by Rick Szeliski and Michael Cohen Lumigraph - Organization 2D position 2D position s,t t s,t u,v v u,v 2 plane parameterization s Courtesy of Rick Szeliski and Michael Cohen. Used with permission. u Slide by Rick Szeliski and Michael Cohen Various images removed due to copyright restrictions. http://graphics.stanford.edu 2007 Marc Levoy MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan Reducing Glare Conventional Photo After removing outliers Glare Reduced Image Raskar, R., et al. "Glare Aware Photography: 4D Ray Sampling for Reducing Glare Effects of Camera Lenses." Proceedings of SIGGRAPH 2008. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan Enhancing Glare Conventional Photo Glare Enhanced Image Raskar, R., et al. "Glare Aware Photography: 4D Ray Sampling for Reducing Glare Effects of Camera Lenses." Proceedings of SIGGRAPH 2008. Courtesy of Andrew Adams. Used with permission. Conventional versus plenoptic camera Courtesy of Ren Ng. Used with permission. 2007 Marc Levoy Conventional versus plenoptic camera uv-plane st-plane Courtesy of Ren Ng. Used with permission. 2007 Marc Levoy Prototype camera Contax medium format camera Kodak 16-megapixel sensor Adaptive Optics microlens array 125 square-sided microlenses Courtesy of Ren Ng. Used with permission. 4000 4000 pixels 292 292 lenses = 14 14 pixels per lens Courtesy of Ren Ng. Used with permission. Example of digital refocusing Courtesy of Ren Ng. Used with permission. 2007 Marc Levoy Adaptive Optics A deformable mirror can be used to correct wavefront errors in an astronomical telescope http://en.wikipedia.org/wiki/Image:Adaptive _optics_correct.png Shack Hartmann wavefront sensor (commonly used in Adaptive optics). http://en.wikipedia.org/wiki/Image:Shack_hartmann.png Measuring shape of wavefront = Lightfield Capture http://www.cvs.rochester.edu/williamslab/r_shackhartmann.html Courtesy of David Williams Lab @ the Center for Visual Science, University of Rochester. Used with permission. The spots formed on the CCD chip for the eye will be displaced because the wavefront will hit each lenslet at an angle rather than straight on. DoF Depends on aperture 30 ft (9 m) 25 ft (7.5 m) 20 ft (6 m) 15 ft (4.5 m) 10 ft (3 m) Depth of field (in focus) out of focus 5 ft (1.5 m) 0 ft (0 m) f/2.8 f/8 f/22 Figure by MIT OpenCourseWare. Depends on focusing distance Depth of field (in focus) 30 ft (9 m) 25 ft (7.5 m) 20 ft (6 m) 15 ft (4.5 m) 10 ft (3 m) out of focus 5 ft (1.5 m) 0 ft (0 m) 5 ft (1.5 m) 10 ft (3 m) 15 ft (4.5 m) Figure by MIT OpenCourseWare. Depends on focal length Remember definition of f stop = diameter/focus length 30 ft (9 m) 25 ft (7.5 m) 20 ft (6 m) 15 ft (4.5 m) 10 ft (3 m) Depth of field (in focus) out of focus 5 ft (1.5 m) 0 ft (0 m) 135 mm 50 mm 28 mm Figure by MIT OpenCourseWare. Courtesy of paul_goyette on Flickr. Courtesy of paul_goyette on Flickr. Scheimpflug principle Image: Wikipedia, public domain. How do we see the world? Sequence of 7 slides removed due to copyright restrictions. 1) Diagram of light rays from object to film 2) Put pinhole camera into the lightpath 3, 4) Images demonstrating the blurring effects of aperture size 5, 6) Diagrams showing how lens gathers light 7) Diagram showing the "in focus" and "circle of confusion" effects of a lens. See Figure 5.109 in Hecht, Eugene. Optics. Addison-Wesley, 2002, p.199. Slide by Steve Seitz Material not covered Lens basics Concave, convex, refraction, Snells law Appearance Depth of field, zoom etc Lens artifacts Radial distortion, coma, astigmatism Please refer to Siggraph course videos/slides Does an out of focus image get dark? Does a zoomed in image get dark? Why CCD camera behaves retroreflective? How does auto-focus work? Anti-Paparazzi Flash Image removed due to copyright restrictions. See Berzon, Alexandra. "The Anti-Paparazzi Flash." New York Times, December 11, 2005. The anti-paparazzi flash: 1. The celebrity prey. 2. The lurking photographer. 3. The offending camera is detected and then bombed with a beam of light. 4. Voila! A blurry image of nothing much. Anti-Paparazzi Flash Images removed due to copyright restrictions. See Truong, K. N., et al. "Preventing Camera Recording by Designing a Capture-Resistant Environment." Ubicomp 2005. Auto Focus Contrast method compares contrast of images at three depths, if in focus, image will have high contrast, else not Phase methods compares two parts of lens at the sensor plane, if in focus, entire exit pupil sees a uniform color, else not - assumes object has diffuse BRDF Lens = Pin-holes + Prisms Sum up images from different pin-holes Ray matrix operations Sub-Aperture = Pin-hole + Prism Ives, Herbert. E. JOSA 20, no. 6 (1930): 332-340. doi:10.1364/JOSA.20.000332 Optical Society of America and H. E. Ives. All rights reserved. This content is excluded from our Creative Commons license. For more information, see http://ocw.mit.edu/fairuse. Ives 1930 Optical Society of America and H. E. Ives. All rights reserved. This content is excluded from our Creative Commons license. For more information, see http://ocw.mit.edu/fairuse. Ives, Herbert. E. JOSA 20, no. 6 (1930): 332-340. doi:10.1364/JOSA.20.000332 Light Field Inside a Camera Lenslet-based Light Field camera [Adelson and Wang, 1992, Ng et al. 2005 ] Courtesy of Ren Ng. Used with permission. Light Fields What are they? What are the properties? How to capture? What are the applications? Focus stack .. object * PSF focus stack McNally, J. G., et al. "Three-Dimensional Imaging by Deconvolution Microscopy." Methods 19, no. 3 (Nov. 1999): 373-385. Courtesy of Elsevier, Inc., http://www.sciencedirect.com. Used with permission. LED In Focus Photo Out of Focus Photo: Open Aperture Coded Aperture Camera The aperture of a 100 mm lens is modified Insert a coded mask with chosen binary pattern Rest of the camera is unmodified Out of Focus Photo: Coded Aperture Modeling and Synthesis of Aperture Effects in Cameras Douglas Lanman, Ramesh Raskar, and Gabriel Taubin Computational Aesthetics 2008 20 June, 2008 69 Slides removed due to copyright restrictions. See this paper and associated presentation at http://mesh.brown.edu/dlanman/research.html Light field photography and microscopy Marc Levoy Computer Science Department Stanford University Slides removed due to copyright restrictions. A public version of this talk may be downloaded at http://graphics.stanford.edu/talks/microscopy-public-apr09.pdf [Raskar, Agrawal, Wilson, Veeraraghavan SIGGRAPH 2008] Lens Glare Reduction Glare/Flare due to camera lenses reduces contrast MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan Reducing Glare Conventional Photo After removing outliers Glare Reduced Image MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan Enhancing Glare Conventional Photo Glare Enhanced Image Glare = low frequency noise in 2D But is high frequency noise in 4D Remove via simple outlier rejection Sensor i j u x MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan Glare due to Lens Inter-Reflections Sensor b a MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan Effects of Glare on Image Hard to model, Low Frequency in 2D But reflection glare is outlier in 4D ray-space Sensor b a Lens Inter-reflections Angular Variation at pixel a Raskar, R., et al. "Glare Aware Photography: 4D Ray Sampling for Reducing Glare Effects of Camera Lenses." Proceedings of SIGGRAPH 2008. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan Key Idea Lens Glare manifests as low frequency in 2D Image But Glare is highly view dependent manifests as outliers in 4D ray-space Reducing Glare == Remove outliers among rays Sensor b a MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan Reducing Glare using a Light Field Camera MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan Captured Photo: LED off MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan Captured Photo: LED On MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare No Glare Raskar, Agrawal, Wilson & Veeraraghavan Each Disk: Angular Samples at that Spatial Location MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare With Glare Raskar, Agrawal, Wilson & Veeraraghavan v MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan y x u MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Sequence of Sub-Aperture Views Raskar, Agrawal, Wilson & Veeraraghavan Average of all the Light Field views Low Res Traditional Camera Photo One of the Light Field views Glare Reduced Image MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan Key Idea Reducing Glare == Remove outlier among angular samples Sensor b a Light Fields What are they? What are the properties? How to capture? What are the applications? Light Field Applications Lens effects Refocussing New aperture setting All in focus image Geometric Estimate depth (Create new views) Synthetic aperture (Foreground/background) (Insert objects) Statistical Lens glare Specular-diffuse Note: LF not required, 4D sampling sufficient Similar HD analysis also works for motion, wavelength, displays MIT OpenCourseWare http://ocw.mit.edu MAS.531 Computational Camera and Photography Fall 2009 For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms. ...
View Full Document

Ask a homework question - tutors are online