Incremental 3D Line Segment Extraction for Surface Reconstruction from Semi-dense SLAM

Abstract

Semi-dense SLAM systems have become popular in the last few years. They can produce much denser point clouds than sparse SLAM while being computationally efficient (using only CPU). In previous works, the surface of the viewed scene was reconstructed in real-time by combining sparse SLAM system and incremental surface reconstruction method. However, it is challenging to utilize the large scale point clouds of semi-dense SLAM for real-time surface reconstruction. In this thesis, in order to obtain meaningful surfaces and reduce the number of points used in surface reconstruction, we propose to simplify the point clouds generated by semi-dense SLAM using 3D line segments. Specifically, we present a novel incremental approach for real-time 3D line segments extraction. This approach reduces a 3D line segment fitting problem into two 2D line segment fitting problems, which take advantage of both image edge segments and depth maps. We first detect edge segments from keyframes. Then we search 3D line segments along the detected edge pixel chains by minimizing the fitting error on both image plane and depth plane. By incrementally clustering the detected line segments, the resulting 3D representation for the scene achieves a good balance between compactness and completeness. Our experimental results show that the 3D line segments generated by our method are highly accurate compared to other methods. With the reconstructed surfaces, we demonstrate that using the extracted 3D line segments greatly improves the quality of 3D surface compared to using the 3D points directly from SLAM systems.

Download thesis draft

Source code at Github

Motivation

Combining a sparse SLAM system (e.g. PTAM, ORB-SLAM) and an incremetal surface reconstruction method (e.g. Free-space carving), one can reconstruct a surface of the scene incrementally in real-time, as shown in Figure 1. However, the statement does not hold true for semi-dense SLAM systems. Although a semi-dense SLAM system (e.g. LSD-SLAM) can extract much more points from the scene, as shown in Figure 2, it is chanllenging to utilize all of them in the task of real-time surface reconstruction. In order to utilize the information in the semi-dense points and improve the quality of the reconstructed surface, we attempt to simplify the generated semi-dense point cloud using 3D line segments. The heuristic here is that the points which contributes the most to the surface are more likely to be found on 3D line segments. By extracting line segments, we can greatly reduce the number of elements while maintaining the major structure of the scene, which enables real-time surface reconstruction using semi-dense SLAM.

Figure 1. Surface reconstructed using PTAM and Free-space carving.

Figure 2. Sample point cloud of sparse SLAM (PTAM) and semi-dense SLAM (LSD-SLAM).

Results

Our method operates on the result of semi-dense SLAM systems. We perform 2D edge extraction from keyframe images. Depth information of the extracted edge pixels can be obtained from the semi-dense SLAM. Then we search 3D line segments along these pixels chains by minimizing fitting error on both image plane and depth plane. In this way, the 3D line segment extraction problem is reduced into two 2D simple line segment fitting problem for each segment. We limit the error on both plane so that the extracted 3D line segments fit the point cloud accurately. Also, in order to furthur reduce the number of elements, we perform incremental clustering on the extracted 3D line segments to merge redundant line segments.

Figure 3. Sample original image from sequence Vicon Room 101 in EuRoC MAV dataset.

Figure 4. Semi-dense point cloud produced by ORB-SLAM with semi-dense module.

Figure 5. 3D line segments extracted using our method. Notice our method significantly reduces the amount of outliers on the right side of the image.

Figure 6. 3D line segments after incremental clustering, which reduces the number of elements while keeping the major structure of the scene.

Given the extracted line segments, we can reconstruct the surface of the scene. Our incremental surface reconstruction method takes the endpoints of the 3D line segments and triangulate them into tetrahedra by performing 3D Delaunay Triangulation. After dividing the tetrahedra into two categories: free space and occupied space, we can take the triangles in between the two kinds of tetrahedra as the surface. Compared with using the sparse points from sparse SLAM, using the endpoints of line segments extracted by our method can produce surfaces with much higher quality.