Motivation

The result of the tracking procedure is sets of corresponding points from the frames in the image sequences. A general structure from motion algorithm uses these the points to obtain a 3-dimensional geometric model. Some of these algorithms can be very restrictive, requiring large numbers of points correspondences and calibrated cameras, while still being sensitive to error in the tracked points. In our case, since we are using uncalibrated video, we need a method that does not need calibrated cameras. Furthermore, since tracking errors are common, the structure from motion algorithm must be able to handle noisy input from the tracking.

Theory

First, consider the case where you have 3d coordinate points, and their corresponding 2d image coordinates and you wish to find out some properties of your camera. This problem is know as resection and there exists a solution provided you have enough input points. Now consider the problem of intersection, when you know the camera parameters and have corresponding 2d points, but wish to find the 3d points that project onto the 2d points in the image plane. Note that resection provides us with the calibration of the camera, but requires the location of the 3d points. Also note that intersection provides us with the structure of the 3d points, but requires calibrated cameras. So these two problems seem to be intimately tied together, each one providing the other with what it needs.

resection	intersection

Structure from motion algorithms attempt to solve the resection problem for each camera, as well as find the 3d structure (intersection) by utilizing many point correspondences in a sequence of images. Once an initial structure is obtained by factorization, both the structure and the motion are updated using bundle adjustment. Bundle adjustment refines the solution in an attempt to minimize the geometric error.

For more information on structure from motion based techniques read our IEEE VR 2003 Tutorial Jagersand (ed) "Recent Methods in Image-based Modeling and Rendering"

Implementation

We have implemented a structure and texture editor which allows the user to change the structure that was outputted from the structure from motion algorithm. This editor allows the user to correct any errors in the tracked points, or in the resulting 3d point structure.