3D SSD Tracking



Dana Cobzas, Martin Jagersand, Peter Sturm


References

Cobzas, D., Jagersand, M. and P. Sturm, 3D SSD Tracking with Estimated 3D Planes , Journal of Image and Vision Computing, 27:69-79, 2009

Cobzas, D. and Sturm, P. 3D SSD Tracking with Estimated 3D Planes, 2nd Canadian Conference on Computer and Robot Vision (CRV) 2005, pp129-134

Cobzas, D. and Jagersand M. Tracking and Predictive Display for a Remote Operated Robot using Uncalibrated Video, IEEE Conference on Robotics and Automation (ICRA) 2005, pp1859-1864 Best vision paper award

Cobzas, D. and Jagersand M. 3D SSD Tracking from Uncalibrated Video, Workshop on Spatial Coherence for Visual Motion Analysis (SCVMA) 2003, in conjunction with ECCV 2004, pp18-31

Demos


Comparison for translational SSD tracking without model (left), with projective model (middle) and Euclidean model (right)
Comparison of homography based SSD tracking without (left) and with Euclidean model (right).

Comparison of classical homography based SSD tracking (red) with model based SSD tracking using 3D planes (green) and 3D constraint planes (blue).
Tracking with predictive display


Description



Overview of the 2D-3D tracking system. In standard SSD tracking 2D surface patches are related through a warp W between frames. In our system a 3D model is estimated (from video alone), and a global 3D pose change delta P is computed, and used to enforce a consistent update of all the surface warps.


In registration-based motion tracking precise pose between a reference template and the current image is determined by warping image patches into the template coordinates and matching pixel-wise intensities. Efficient such algorithms are based on relating spatial and temporal derivatives using numerical optimization algorithms. We extend this approach from planar patches into a formulation where the 3D geometry of a scene is both estimated from uncalibrated video and used to constrain tracking. As a consequence our system supports direct tracking of full camera position (rotation and translation). We have investigated two types of models: a point based model obtained using structure from motion (SFM) and a plane based model. The plane based formulation also allows additional constraints to be naturally added, e.g. perpendicularity between walls, floor and ceiling surfaces, co-planarity of wall surfaces etc.

Experimentally we compare convergence and accuracy of our uncalibrated 3D tracking to previous approaches. Notably, the 3D algorithm can successfully track over significantly larger pose changes than ones using only planar regions. It also detects occlusions and removes/inserts tracking regions as appropriate in response.

As an application in robotics, we have integrated the tracking with the dynamic texture model into a predictive display system.