points Fusing and lines for high performance real-time tracking Ed Rosten, Tom Drummond University of Cambridge
Model based tracking
Point tracking Edge tracking FAST features Robust optimize
Combine for robust tracking
Different failure modes Combine for extra robusteness Combination is difficult Statistics are non Gaussian
Edge based tracking
Start from position prior
Edge based tracking
Start from position prior Search along edge-normal lines
Edge based tracking
Start from position prior Search along edge-normal lines Adjust position to minimize errors
Edge based tracking
Start from position prior Search along edge-normal lines Adjust position to minimize errors Gives drift free measurements Model is static
Good prior needed
Edges are a step change in intensity Correspondence is hard--pick closest edge
Good prior needed
Edges are a step change in intensity Correspondence is hard--pick closest edge Prior must be good, or the wrong edge will be found Correct edge might be nowhere near
Non Gaussian posterior
Correct correspondences Tracking is accurate Incorrect correspondences Tracking is inaccurate--even if prior is good
Non Gaussian posterior
Correct correspondences Tracking is accurate Incorrect correspondences Tracking is inaccurate--even if prior is good
Point based tracking
Detect features
1
Point based tracking
Detect features
1
2
Project features on to model. Drift occurs here
Point based tracking
Detect features
1 3
Detect and match features in next frame
2
Project features on to model. Drift occurs here
Point based tracking
Detect features
1 3
Detect and match features in next frame
2
4
Project features on to model. Drift occurs here
Alter pose to minimize reprojection error
Point based tracking
Detect features
1 3
Detect and match features in next frame
2
4
Project features on to model. Drift occurs here
Alter pose to minimize reprojection error
The FAST feature detector
The FAST feature detector
16 1 2 15 14 13 12 11 10 9 8 p 7 3 4 5 6
The FAST feature detector
16 1 2 15 14 13 12 11 10 9 8 p 7 3 4 5 6
12 contiguous pixels brighter than p+threshold
The FAST feature detector
1
p
9
12 contiguous pixels brighter than p+threshold Rapid rejection by testing 1, 9
The FAST feature detector
1
p
5
9
12 contiguous pixels brighter than p+threshold Rapid rejection by testing 1, 9, 5
The FAST feature detector
1
13
p
5
9
12 contiguous pixels brighter than p+threshold Rapid rejection by testing 1, 9, 5 then 13
The FAST feature detector
1
13
p
5
9
12 contiguous pixels brighter than p+threshold Rapid rejection by testing 1, 9, 5 then 13 1.59ms (Opteron 2.6GHz) - 8% of available CPU time Source code available (see paper for URL)
The FAST feature detector
16 1 2 15 14 13 12 11 10 9 8 p 7 3 4 5 6
12 contiguous pixels brighter than p+threshold Rapid rejection by testing 1, 9, 5 then 13 1.59ms (Opteron 2.6GHz) - 8% of available CPU time Source code available (see paper for URL) 16 test pixels used for feature vector SSD used for matching between frames
Position optimization
Sometimes > 90% outliers (even with Robust SIFT!) optimize required Use EM Mixture model is Gaussian (inliers) + uniform (outliers) SSD has some information about inlier probability If only we knew the relationship...
Matching prior
Probability of correct match 1 Raw data Temporally smoothed Function Frames 100-760 2500 5000 Sum squared difference
0.5
0 0
EM provides probability that a match is correct SSD for each match is known Compute smooth function mapping SSD to probability Use function to compute priors for each match next frame
Matching prior
Probability of convergence 1
0.5
EM EM with prior
0 -2 10
10 Proportion of inliers
-1
10
0
EM provides probability that a match is correct SSD for each match is known Compute smooth function mapping SSD to probability Use function to compute priors for each match next frame
Measurement Properties
Edge based tracking Requires 3D geometric model Good pose prior Provides Drift free measurements Non Gaussian posterior Point based tracking Requires 3D point cloud Provides Robust differential measurements... ...with approximately Gaussian posterior
Sensor fusion
Either tracker can be wrong Edge tracker can get correspondence wrong Point based tracker can drift Posterior can be multimodal Simple solutions do not work
Sensor fusion
n-1 Corner tracking
?
Frame n -Corner tracking ? Pick best
-Corner 6
n+1
-
?
Edge tracking
tracking
Edge tracking
-
Either tracker can be wrong Edge tracker can get correspondence wrong Point based tracker can drift Posterior can be multimodal Evaluate modes next frame when more data arrives
Results - Strong unmodelled edges
Strong unmodelled edges frequently break the edge tracker
Results - Camera shake
Pick up camera and shake really hard Can you follow the video? I can't (but my tracker can)
Results - Camera shake
100 Angle / degrees 50 0 -50 0
0.5
1
1.5 Time / s
2
2.5
3
...