Visual The System Optics System to control light and focus it on a sensitive surface Pinhole camera Small hole focuses light in correct location But small hole = not much light! Lens & Iris camera Iris allows more or less light Adjustable lens focuses light But depth of field problem Big hole = shallow area of focus The human eye is a lens & iris system But we rarely experience the depth of field of our own eye! The Retina Rods Black & white vision Luminance detection Cones Color vision 3 types of cones Red Blue Green Luminance detection for a particular wavelength Fovea Center of retina Cluster of cones Parafovea Rest of retina More rods than cones At night, better vision or of the corner of your eye Lateral Geniculate Nucleus (LGN) Center Surround Cells Luminance detection cells Color-opponent cells Magnocellular Neurons Low Spatial Resolution Cover big visual field Fast Cell Response Turn on fast, shut off fast Color-insensitive Parvocellular Neurons High Spatial Resolution Slow Cell Response Color-sensitive Primary Visual Cortex (Striate) (V1) Lots of layers Extrastriate Cortex (V2) 1
Kanizsa Triangle V2 responds to illusory contours V1 responds to contours
What and Where Systems After V1 and V2 split between magno & cellular paths to different areas Dorsal Pathway Ventral Pathway Magnocellular/Dorsal/Where/How path Good for detecting motion Bad for detecting particular objects Parvocellular/Ventral/What path Good for recognizing particular objects Bad for detecting motion Vision isnt a unitary process! In consciousness we have interleave magno & parvo information Vision isnt a veridical process We mark sense of the world We dont just store a veridical representation of the world
Object Recognition The what path Goal is to recognize objects What they are Not where they are Not how fast they are moving
Properties of the system Robust to variability Can deal with noise in the system Robust to lossy information Obscured Properties of the underlying visual system There are detectors for features of various levels Activation of a point on the retina Lines are made up of rows of points Lines at different orientations A theory of object recognition Synthesis by features We recognize object by recognizing features of the objects And combining these into larger and larger units In the case of words, we recognize features that make letters that then make words As a first step look for evidence that we have access to features in the visual system Serial vs. Parallel processing Recognition of features should be automatic Easy wherever the feature is in space Easy whatever else is in space Takes no conscious effort 2
pops out Test using visual search paradigm Present a display of objects Participant looks for a particular objects Vary number & type of distracters For features, we should see parallel search Number of distracters doesnt matter A red vertical bar pops out against green vertical bars Number of distracters doesnt matter We can think of color as a primary feature A red vertical bar pops out against red horizontal bars Number of distracters doesnt matter We can think of verticality as a primary feature A red vertical bar fails to pop out against red horizontal bars and green vertical bars together Number of distracters does matter Conjunctions of features require serial search For primary features we see parallel search For conjunctions of features we see serial search
We have evidence for some set of primary features Combining them takes work Do we have evidence for a system that works by doing combination of features? Can we use a similar system to handle other objects? Recognition By Components Irv Biederman Object analog of features based models of word recognition In RBC the basic alphabet of objects are geons Evidence for RBC Recognition of only appropriate partial images Present degraded images which either preserve or destroy the component geons If geons are used in recognition, those pictures in which the geons are preserved will still be recognizable, while those in which the geons are destroyed will not be Repetition priming of objects only if same geons are used Present pairs of degraded pictures In one case the geons are the same in both pictures In another case they are different Cases with the same geons prime each other
Word Recognition Lots of work into this field is done with visual word recognition We have a complex stimulus (the word) We have simple features (lines at different orientations) We have some middle stages (letters)
Basics of Word Recognition
3
Word frequency effect Words you see very often are easier to recognize Repetition Effect Words you have seen recently are easier to recognize Word Superiority Effect Words are easier to recognize than letters Words mask component letters Present a display with a word or non-word Ask if a particular letter was present Finding an individual letter is harder when it is presented in a word than in a nonsense string Recognition Errors Another form of the word superiority effect We impose wordness on strings of letters We often see CORN instead of CQRN in a rapid presentation
One last property of the system Sensitive to local context Higher level of organizing principles influence perception
Summary Object recognition as a network in which simple features are used to identify more complex features and eventually to recognize objects The system recognizes objects, not an individual neuron Recognition is a process not an end result
Vision Is Object Recognition Enough? Typical framework for vision We reconstruct a veridical three dimensional representation of the world that is available to consciousness Find lines and contours Find textures Find and recognize objects 4
We reason about our model and then act based on this reasoning Out in the world is an array of objects But is this really the best approach? Fits with our conscious experience of the world Does it fit with the neuroscience of vision?
Perception for Action
An alternative approach Perception for action The goal of vision (& all perception) is to act in the world in a useful way Avoid danger Find food Find a mate Most perception is being used to drive movement At least in evolutionary terms In these terms what does a perceptual system need to do? Do we need a veridical 3-D representation of the world? Do we always need object recognition? Do we always need veridical representations of sound and taste and smell? GOFAI (and most connectionism) Modeling perception Modeling mental representations of information about the environment No real modeling of action
Robotics Key features of autonomous robots 5
Sensors A way to perceive the world Effectors A way to manipulate the world Thinking & reasoning? Maybe Battlebots Not a robot Remote controlled cars
Roomba Really simple robot Thinking and Reasoning? Direct link from sensor to effector Go randomly & if you hit something Back up a bit, turn 90 degrees, and repeat Photovore How a photovore works Two light sensors, left and right Wants to go toward the light Direct drive system If left light sensor sees more light than right, it will turn left Turns toward the light Thinking and reasoning? Direct link from sensor to effector Perception directly manipulate action No cognition in between But people are more complex We can choose to do or not do stuff We have multiple possible actions Photophobe How a photophobe works Two light sensors, left and right Wants to go toward the light drive Direct system If left light sensor sees more light than right, it will turn right Turns away from the light A more complex photovore/photophobe 6
Typically hiding in the dark is good It protects you from other animals etc But theres no food there A more complex machine will hide unless it needs food and then it will go hunting till its satisfied and then hide again A decision making photo-bot Solar cell to a comparator to a photovore wiring if hungry or photophobe wiring if full Subsumption architecture Decision 1 (charge level) dictates decision 2 (light-based behavior) If the battery is charged it can act like a photophobe If the battery is low it will go seek light till it is charged and then hide It decides on the behavior based on an internal state But does it have thinking & reasoning? Crickets A real world example for a simple robot Female rickets find mates via chirping Localize sound source & go there (and mate?) Just a sound analog to the photovore Identify songs of their species as distinct from other species Phonotaxis A real decision How do crickets do it? Each ear has two sound sources External sound Internal sound via spiracles At ear nearest to sound source External sound reaches ear first Internal sound is delayed via longer travel time Two signals out of phase At far ear Less of a phase shift System turns to out of phase side Localization & movement to sound With a direct perception to action system! But there is still phonotaxis Deciding which sounds to pay attention to Another trick of cricket physiology The tracheal tubes connecting the spiracles to the ear only transmit certain frequencies Because of the length & diameter of the tube Only cricket chirps get sent down the internal pathway Other sounds are ignored because the system is physically structured to ignore them Humans but crickets still make decisions About which males to mate with About when to mate About what to eat All this can be described as action based on direct perception of either internal or external stimuli in a subsumption architecture but humans dont overtly show this kind of hardwired behavior We have free will We have control over our actions 7
We have a feeling of a reason for our behavior We feel like we perceive the world and reason about it based on a veridical representation But do we? Recall the macaque cortex map Recall change blindness If we had a veridical perceptual representation wed see the jet engine disappear Recall inattentional blindness We shouldnt miss the dancing bear The perception we have is based on the task were doing Reflexes Obvious hardwired behaviors Walking Do we consciously plan all the movements? Do we look at each step carefully before going down them? We use routines Automatic chunk-like behaviors Even in more complex behaviors Reaching for a pencil on a desk We dont individually plan the mechanics of the movements based on our 3-D map We dont individually plan the grasping movements for the individual fingers Making fists & playing guitar Think of humans as a set of highly specialized simple systems, potentially evolved from older systems (like cricket phonotaxis) Each has a task that if performs well Each is (mostly) directly linked from perception to action consciousness is part of a subsumption architecture Able to manipulate which behaviors we choose or dont chose at a particular time We have discussed direct control of action via perceptual information Subsumption architecture Coordinated movements with a goal in mind Pretty easy to see in a simple robot One goal, two (or three) possible movements How do we achieve something like this in a complex organism like a human Nearly infinite possible goals Lots of movements
Biology of Movement Moving Muscles Binary status of muscle fibers (two states) Relaxed Contracted Ballistic system Each fiber contracts or it doesnt Activation via acetylcholine and the muscle itself ACH receptors in the muscle cause contraction Trading off dexterity and power (sort of) To move very finely we need to be able to control how much a muscle moves Fast or slow? Big or small movements? 8
Separate axons Each synapses with a separate muscle fiber By only activating some of the fibers we can achieve graded movement Hands, face, tongue, eyes To move lots of matter we need to contract lots of muscle fibers at once A single motor neuron axon will synapse with lots of muscle fibers One neural signal will release ACH to all of them All contract Big movements Legs & arms Knowledge of movement Need knowledge of our own movements Kinesthesia: Knowledge of where our body is Proprioception: We can do this without looking Our muscles can act as (internal) sensors as well as effectors Sensory neurons detect when a muscle is contracting Out of the brain This provides reflexes and some damage control We cant contract antagonistic muscles at the same time Proprioception of the biceps directly inhibits the triceps In the brain Conscious feedback based control of movement Control of movement unconscious control Medulla (& spinal cord) Directly cnxn to muscles Firing in the medulla causes contractions of muscles Evolutionarily old All animals with brains have a medulla Directly controls basic bodily functions Heart rate, breathing, swallowing Directly controls conscious muscle movements Electrical stimulation in medulla causes individual muscles to move conscious control Cortex Mostly primary motor cortex (M1) Projections to medulla Not directly to muscles Controls motor plans Electrical stimulation in M1 Movement of muscle groups to perform an action NOT movement of individual muscles The motor homunculus Structure in the cortex roughly resembles the structure of a human body Amount of space Based on innervations Not on physical size Some stuff is missing E.g.: the heart isnt mapped Why? Used for conscious control of movement, dont want to have to consciously control your heart rate 9
What happens in M1? Motor program: the decision to make a sequence of movements based on perception In a sense ballistic Make the plan and let it go Anticipatory speech errors Upcoming information gets in the way of current information Journal article journicle Actions requiring lots of preparation take a long time We plan speech carefully Lots of work is done in M1 before the medulla take over Actions that dont require careful planning (or that are automatic) are faster Reflex responses Proprioceptive control of movement Control of attention The motor homunculus mate The somatosensory homunculus Receives projections from Proprioceptive neurons From some other sensory systems as well Sends projections to M1 Directly to medulla unconscious feedback The hierarchy of conscious action The cortex codes an action The medulla codes a particular movement that will achieve the action The muscle does the movements Little conscious control of movements per se Lots of conscious control of action plans How is cognition driven by the bodies we inhabit? Actions we are currently performing (or plan to perform) constrain our perception Recall the dancing bear Eye movements during mental imagery We look as directed by the story Eye-movements during recall We look where things were not where they are Association between motor response and sensory input Interference makes the task harder Sleep paralysis The brain has to actively shut-off the body during dreaming The entire body is involved in cognition Not just the brain Even some stuff external to the body is involved in cognition
10