A Highly Efficient and Extensible Library for Registration based Visual Tracking
Fast and high precision visual tracking is crucial to the success of several robotics and virtual reality applications like SLAM, autonomous navigation and visual servoing. In recent years, online learning and detection based trackers have been more popular in the vision community due to their robustness to changes in the object's appearance which makes them better suited to long term tracking. However, these are often unsuitable for the aforementioned applications for two reasons. Firstly, they are too slow to allow real time execution of tasks where multiple trackers have to be run simultaneously or tracking is only a small part of a larger system with more computationally intensive modules that use its result to make higher level deductions about the environment. Secondly, they are not precise enough to give the exact object pose with sub pixel alignment required for these tasks, being usually limited to the estimation of simple transformations of the target patch such as translation and scaling. As a result, registration based trackers are more suitable for these applications as being several times faster and capable of estimating more complex transformations like affine and homography.
Though several major advances have been made in this domain since the original Lucas Kanade tracker was introduced almost thirty five years ago, yet efficient open source implementations of recent trackers are surprisingly difficult to find. In fact, the only such tracker offered by the popular OpenCV library, uses a pyramidal implementation of the original algorithm. In the absence of good open source implementations of modern trackers, most robotics and VR research groups either use these out dated trackers or implement their own custom trackers. These, in turn, are often not made publicly available or are tailored to suit very specific needs and so require significant reprogramming to be useful for an unrelated project. To address this requirement, we introduce Modular Tracking Framework (MTF) - a generic system for registration based tracking that provides highly efficient implementations for a large subset of trackers introduced in literature to date and is designed to be easily extensible with additional methods.
Each tracker within this framework comprises the following 3 modules:
Please refer this paper for more details on the system design and this one for some preliminary results. The latter was published at CRV 2016 (slides used for the presentation are available here). The former has been submitted to ICRA 2017. There is also this newer paper with more results that has been accepted at WACV 2017. Finally, the complete thesis based on this framework is available here if even more details are needed. This can be used as the official documentation till parts of it are adapted for Doxygen. Following is the ICRA supplementary video showing two usage examples:
The library is implemented entirely in C++ though a Python interface called
pyMTF also exists and works seamlessly with our Python Tracking Framework.
A Matlab interface similar to Mexvision is currently under development too.
We also provide a simple interface for ROS called mtf_bridge for seamless integration with robotics applications. A ROS package that uses it to exemplify integration of MTF with ROS is present in the ROS sub folder. Finally, MTF comes bundled with several state of the art learning and detection based trackers whose C++ implementations are publicly available including DSST, KCF, CMT, TLD, RCT, MIL, Struck, FragTrack, GOTURN and DFT. As s result, combined with the datasets provided below, MTF can serve as a great test bed for general purpose tracking too. We are always looking to add more such trackers to MTF so please let us know if there is a tracker with open source C++ implementation that you would like to see integrated.
Currently, MTF only works under Unix though support for Windows is under active development and will be added soon. Though it has been tested comprehensively only under Linux, specifically Ubuntu 14.04, it should work on Macintosh systems too (see Compile/Runtime Notes section on the installation page for resolving possible issues).
MTF is provided under BSD license and so is free for research and commercial applications. We do request, however, that this paper be cited by any publications resulting from projects that use MTF so more people can get to know about and benefit from it.
Download: mtf.1.0.0.zip - might not be up to date so please use the Git repository for the latest code.
Git Repository: https://github.com/abhineet123/MTF
Several publicly available tracking datasets with full ground truth formatted to work with MTF out of the box are also made available here for convenience:
db_root_pathconfiguration parameter and you are good to go !