EE106A Lab 6.pdf - EE106A: Lab 6 - Computer Vision ∗ Fall...

EE106A: Lab 6 - Computer Vision
*
Fall 2019
Goals
By the end of this lab you should be able to:
• Explain the concept behind pointclouds and what they represent
• Implement some basic techniques in image segmentation and explain your results
• Explain the concept of pinhole cameras and describe how they work
• Combine information from different sensors to isolate an object of interest. (Build a point cloud segmentation pipeline)
Relevant Tutorials and Documentation:
• OpenCV Python Tutorial
Contents
1 Introduction 2
2 Starter Code 2
3 Intel RealSense 2
4 Point-Clouds 2
4.1 An approach to point-cloud segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
5 Image Segmentation 3
5.1 Color and Grayscale Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
5.2 Thresholding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
5.3 Edge Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
5.4 Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
6 Projective Geometry and The Camera Matrix 7
6.1 Homogenous Co-ordinates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
6.2 The Pinhole Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
6.3 The Intrinsic Camera Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
6.4 Projecting the Point-cloud . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
7 Putting it all together 10
7.1 Time Synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
7.2 Queue Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
*
Developed by Amay Saxena and Grant Wang, Fall 2019.
1

1 Introduction
This lab will introduce you to various techniques used in processing data from advanced sensors like the Intel RealSense. In
particular, you will learn about image segmentation, point cloud processing, and how we can combine various data modalities
to leverage the strengths and weaknesses of each data form.
You will write your own point-cloud segmentation pipeline . This means that by the end of this lab, you will be able to
isolate an object of interest in a point-cloud, by combining information you get from RGB image sensors.
2 Starter Code
Create a new workspace called lab6 . Then, unzip the contents of the lab6_resources folder into the src directory of
this workspace.
The starter code includes the following packages:
1. realsense-ros : This is a directory that contains several ROS packages to work with the Intel Realsense sensor,
which we will be using for this lab.
2. ros_numpy : A very useful ROS package that can be used to convert between ROS message datatypes and numpy
datatypes. Check out the documentation at .
3. segmentation : This is the package that contains all the skeleton code. All files that you will need to edit are in this
package.
After creating your workspace and copying over the contents of lab6_resources into lab6/src , use catkin_make
to build your workspace. Then, go into segmentation/src and make all python files executable using
chmod +x
*
.py
3 Intel RealSense
In this lab, you will be using an Intel RealSense camera, which is a powerful sensor equipped with an RGB camera, a depth
camera, an Inertial Measurement Unit, and an IR sensor which is capable of producing a rich colored point-cloud. We will
concern ourselves with the RGB camera and the point-cloud.
The drivers for the RealSense have already been installed for you. The starter code includes a package realsense-ros
that will allow you to begin publishing information from the RealSense camera onto various topics. Ensure that you have the