A Highly Efficient and Extensible Library for Registration based Visual Tracking
Fast and high precision visual tracking is crucial to the success of several robotics and virtual reality applications like SLAM, autonomous navigation and visual servoing. In recent years, online learning and detection based trackers have been more popular in the vision community due to their robustness to changes in the object's appearance which makes them better suited to long term tracking. However, these are often unsuitable for the aforementioned applications for two reasons. Firstly, they are too slow to allow real time execution of tasks where multiple trackers have to be run simultaneously or tracking is only a small part of a larger system with more computationally intensive modules that use its result to make higher level deductions about the environment. Secondly, they are not precise enough to give the exact object pose with sub pixel alignment required for these tasks, being usually limited to the estimation of simple transformations of the target patch such as translation and scaling. As a result, registration based trackers are more suitable for these applications as being several times faster and capable of estimating more complex transformations like affine and homography.
Though several major advances have been made in this domain since the original Lucas Kanade tracker was introduced almost thirty five years ago, yet efficient open source implementations of recent trackers are surprisingly difficult to find. In fact, the only such tracker offered by the popular OpenCV library, uses a pyramidal implementation of the original algorithm. In the absence of good open source implementations of modern trackers, most robotics and VR research groups either use these out dated trackers or implement their own custom trackers. These, in turn, are often not made publicly available or are tailored to suit very specific needs and so require significant reprogramming to be useful for an unrelated project. To address this requirement, we introduce Modular Tracking Framework (MTF) - a generic system for registration based tracking that provides highly efficient implementations for a large subset of trackers introduced in literature to date and is designed to be easily extensible with additional methods.
Each tracker within this framework comprises the following 3 modules:
Please refer this paper for more details on the system design and this one for some preliminary results. The latter was published at CRV 2016. The former has been submitted to IROS 2017. There is also this newer results paper that has been accepted at WACV 2017. Finally, the complete thesis based on this framework is available here if even more details are needed. This can also be used as the official documentation till the Doxygen version is completed. A complete list of related papers is also given below.
The library is implemented entirely in C++ though a Python interface called
pyMTF also exists and works seamlessly with our Python Tracking Framework.
A Matlab interface similar to Mexvision is currently under development too.
We also provide a simple interface for ROS called mtf_bridge (also present in the ROS sub folder) for seamless integration with robotics applications. Finally, MTF comes bundled with several state of the art learning and detection based trackers whose C++ implementations are publicly available including DSST, KCF, CMT, TLD, RCT, MIL, Struck, FragTrack, GOTURN and DFT. As s result, combined with the datasets provided below, MTF can serve as a great test bed for general purpose tracking too. We are always looking to add more such trackers to MTF so please let us know if there is a tracker with open source C++ implementation that you would like to see integrated.
MTF supports both Unix and Windows platforms. Though it has been tested comprehensively only under Linux, specifically Ubuntu 14.04, it should work on Macintosh systems too. The Windows build system is in its early stages and needs some manual setting of variables but it is quite usable and we are working on making it more user friendly.
MTF is provided under BSD license and so is free for research and commercial applications. We do request, however, that this paper be cited by any publications resulting from projects that use MTF so more people can get to know about and benefit from it.
Several publicly available tracking datasets with full ground truth formatted to work with MTF out of the box are also made available here for convenience:
db_root_pathconfiguration parameter and you are good to go !