is What Computer Vision?
Deals with the development of the theoretical and algorithmic basis by which useful information about the 3D world can be automatically extracted and analyzed from a single or multiple 2D images of the world. That is: 2D Image 3D Geometric Data It is also known as:
Image Analysis Scene Analysis Image Understanding
Introduction and Overview
Xiaojun Qi
1
2
Some Related Disciplines
Image Processing: 2D Image 2D Image (Edge Detection, Fourier Transformation, Enhancement, Segmentation, etc.) Computer Graphics: 3D Geometric data 2D Image Pattern Recognition:
The recognition and classification of 2D objects mainly from 2D images. Many pattern recognition principles are used extensively in computer vision.
Some Related Disciplines (Cont.)
Robotics
Autonomous robot navigation Inspection and assembly
Artificial Intelligence:
Concerned with designing systems that are intelligent and with studying computational aspects of intelligence. It is used to analyze scenes by computing a symbolic representation of the scene contents after the images have been processed to obtain features. Computer vision is considered a sub-field of artificial intelligence.
3 4
Why is Computer Vision Difficult?
It is a many-to-one mapping
A variety of surfaces with different material and geometrical properties, possibly under different lighting conditions, could lead to identical images Inverse mapping has non unique solution (a lot of information is lost in the transformation from the 3D world to the 2D image)
Practical Considerations
Impose constraints to recover the scene
Gather more data (images) Make assumptions about the world
Computability and robustness
Is the solution computable using reasonable resources? Is the solution robust?
It is computationally intensive We do not understand the recognition problem
Industrial computer vision systems work very well
Make strong assumptions about lighting conditions Make strong assumptions about the position of objects Make strong assumptions about the type of objects
5
6
1
Three Processing Levels
Low-level processing
Standard procedures are applied to improve image quality Procedures are required to have no intelligent capabilities.
Recognition Cues
Scene interpretation, even of complex, cluttered scenes is a straightforward task for humans. How are we able to discern reality and an image of reality? What clues are present in the image? What knowledge do we use to process the image? How about color, texture, and shape?
7 8
Intermediate-level processing
Extract and characterize components in the image Some intelligent capabilities are required.
High-level processing
Recognition and interpretation. Procedures require high intelligent capabilities.
Mathematics in Computer Vision
In the early days of computer vision, vision systems employed simple heuristic methods. Today, the domain is heavily inclined towards theoretically, well-founded methods involving nontrivial mathematics.
9
Calculus Linear Algebra Probabilities and Statistics Signal Processing Projective Geometry Computational Geometry Optimization Theory Control Theory
10
Computer Vision Applications
Industrial inspection/quality control Surveillance and security Face recognition Gesture recognition Space applications Medical image analysis Autonomous vehicles Virtual reality and much more ......
11
Visual Faces
What information do our faces convey to others?
We use our faces to communicate by the production of audible speech. We are constantly reading one another's faces and such processes clearly play an important part in social interaction and communication
12
2
Facial Expression
Facial expressions produced by visible deformation of the face provide us with a guide to judging
Mood Character Intent
Visual Cue for Identification
More enduring visual cues provided by a person's face allow us to estimate such factors as:
Age Gender Ethnic origin Identity
However, all these changes in facial appearance due to expression and speech occur on a rather short time-scale.
13
14
Face Objects
Faces form a unique class of objects since they share a common structure (the facial features are always configured in a similar way.) Given their similarity, it is remarkable that we can discriminate between so many different people based upon facial appearance alone!
15
Dynamic Faces
In human perception and behavior, we use more than just static facial appearances in discriminating between different faces.
Visual context constrains our expectation! Body gestures and gait also provide useful cues.
It is important to realize that the perception of faces is a spatio-temporal, dynamic vision task.
16
Dynamic Faces
While it is true that we can often identify someone from a static photograph, in the real world we usually observe and identify faces in a dynamic setting.
Facial appearance alters due to relative motion of the face and the observer. Changes in lighting and the environment also affect appearance.
17
Dynamic Faces
Face perception involves rather more than the perception of static pictures of faces. In particular, it would seem to require the ability to detect and faces track as they move through cluttered scenes which are themselves often complex and dynamic.
18
3
Changing Faces
A face can change its appearance due to either intrinsic or extrinsic factors. Intrinsic variation takes place independently of any observer and is due purely to the physical nature of the face. Extrinsic variation arises when the face is observed via the interaction of light with the face and the observer.
19
Perception of Faces
It is the intrinsic variations that one must learn to understand and interpret. Perceptual tasks have to be performed consistently and robustly under all sorts of changes in external conditions characterized by the extrinsic sources of variation.
20
Intrinsic Sources
Source Possible Tasks Identity: Classification, known-unknown, verification, full identification. Facial Expression: Inference of emotion or intention Speech: Lip reading Sex: Decide whether male or female. Age: Estimating age
21
Extrinsic Sources
Source Effects Viewing geometry: Pose Illumination: Shading, color, self-shadowing, specular highlights Imaging Process: Resolution, focus, imaging noise, sampling of irradiant energy distribution, perspective effects Other objects: Occlusion, shadowing, indirect illumination
22
Pose
A face undergoes rigid motion when it changes its position and orientation in 3D space relative to the observer. A face can also undergo non-rigid motion when its 3D shape changes due to, for example, speech or facial expression. While these two types of motion usually occur together, it is more convenient to treat them separately.
23
Rigid vs. Non-rigid Motion
Rigid motion provides strong visual cues for understanding pose. Non-rigid facial motion is also likely to provide useful cues for identification. It also clearly plays an even more important role in communication and perception of expressions.
24
4
Computing Faces
Computer-based face perception is becoming increasingly desirable for many applications including:
Human-machine interfaces Multimedia Surveillance Security Teleconferencing Communication Animation Visually mediated interaction and anthropomorphic environment
Scope of the Class
Focus on the perception of faces for identification under extrinsic variations such as pose change caused by movement. That is, we assume that intrinsic variation is constrained so as to have little effect on facial appearance.
25 26
Two General Procedures
Segmentation: Divide images into regions that correspond to bodies of physical objects in the scene. Recognition: Label such bodies as instances of known objects.
Detailed Procedures
The visual perception of faces relies upon a range of computational processes including:
The measurement of visual cues such as motion and color Selective attention Face detection (no identification is involved) Pose estimation View alignment Face tracking Modeling of identity and identification
27
28
Four Tasks Related to Identification
Classification: Identify a face under the assumption that it is a member of a DB. Known-Unknown: Decide whether or not a face is a member of a DB. Ve...