Synthetic Aperture Imaging

In this project, we have developed a novel multi-object detection method using multiple cameras. Unlike conventional multi-camera object detection methods, our method detects multiple objects using a linear camera array. The array can stream different views of the environment and can be easily reconfigured for a scene compared with the overhead surround configuration. Using the proposed method, the synthesized results can provide not only views of significantly occluded objects but also the ability of focusing on the target while blurring objects that are not of interest. Our method does not need to reconstruct the 3D structure of the scene, can accommodate dynamic background, is able to detect objects at any depth by using a new synthetic aperture imaging method based on a simple shift transformation, and can see through occluders. The experimental results show that the proposed method has a good performance and can synthesize objects located within any designated depth interval with much better clarity than that using an existing method. To our best knowledge, it is the first time that such a method using synthetic aperture imaging has been proposed and developed for multi-object detection in a complex scene with significant occlusion at different depths.

Reference

Z. Pei, Y. Zhang, T.Yang, X. Zhang, and Y.H. Yang, "A Novel Multi-Object Detection Method in Complex Scene Using Synthetic Aperture Imaging," Pattern Recognition, to appear.

Demo video

A Cluster-Based Camera Array System

In this project, we are devloping a cluster-based system for camera array application. The idea is to develop a framework such that the cluster will be as easy to use as a single machine. Our current cluster consists of 8 nodes, each of which consists of two dual-core Opteron CPUs, 2 Nvidia 8800 GTX GPUs, and one 10Gb network interface card. The nodes are connected by 10Gb high speed Ethernet. Each node controls two firewire cameras.

References

C. Lei, X. Chen, and Y.H. Yang, "A New Multiview Spacetime-Consistent Depth Recovery Framework for Free Viewpoint Video Rendering, International Conference on Computer Vision, Kyoto, Japan, September 27-October 2, 2009.

Supplemental materials (21MB)

C. Lei and Y.H. Yang, "Efficient geometric, photometric, and temporal calibration of an array of unsynchronized video cameras," Sixth Canadian Conference on Computer and Robot Vision, Kelowna, BC, May 25-27, 2009.

C. Lei and Y.H. Yang, "Design and implementation of a cluster-based smart camera array application framework," Second ACM/IEEE International Conference on Distributed Smart Cameras, Stanford, California, September 7-11, 2008. Poster Presentation.

Tri-Focal Tensor based Multiple Video Synchronization with Sub-Frame Optimization

In this project, we develop a novel method for synchronizing multiple (more than 2) un-calibrated video sequences recording the same event by free-moving full-perspective cameras. Unlike previous synchronization methods, our method takes advantage of tri-view geometry constraints instead of the commonly used two-view one for their better performance in measuring geometric alignment when video frames are synchronized. In particular, the tri-ocular geometric constraint of point/line features, which is evaluated by tri-focal transfer, is enforced when building the timeline maps for sequences to be synchronized. A hierarchical approach is used to reduce the computational complexity. To achieve sub-frame synchronization accuracy, the Levenberg-Marquardt method based optimization is performed. The experimental results on several synthetic and real video datasets demonstrate the effectiveness and robustness of our method over previous methods in synchronizing full-perspective videos.

Reference

C. Lei and Y.H. Yang, “Tri-Focal Tensor based Multiple Video Synchronization with Sub-Frame Optimization,” IEEE Trans. on Image Processing, Vol. 15, 2006, pp. 2473-2480.

Supplemental materials (23MB)

Real-time Backward Disparity-based Rendering for Dynamic Scenes using Programmable Graphics Hardware

This project develops a backward disparity-based rendering algorithm, which runs at real-time speed on programmable graphics
hardware. The algorithm requires only a handful of image samples of the scene and estimated (inaccurate) disparity maps, whereas
most existing techniques need either dense samples or accurate depth information. To color a given pixel in the novel view, a backward searching process is conducted to find the corresponding pixels from the closest four reference images. The use of backward searching process makes the algorithm more robust to errors in estimated disparity maps than existing forward warping-based
approaches. In addition, since the computations for different pixels are independent, they can be performed in parallel on the Graphics
Processing Units of modern graphics hardware. Experiment results demonstrate that our algorithm can synthesize accurate novel views
for dynamic real scenes at a high frame rate.

video 1 and video 2

Refererence

M. Gong, J. Selzer, C. Lei, and Y.H. Yang "Real-time Backward Disparity-based Rendering for Dynamic Scenes using Programmable Graphics Hardware," Graphics Interface,  Montreal, 2007, pp. 241-248.

Camera Field Rendering of Static and Dynamic Scenes

Most of the previously proposed image-based rendering approaches rely on a large number of samples, accurate depth information, or 3D geometric proxies. It is, therefore, a challenge to apply them to render novel views for dynamic scenes as the required information is difficult to obtain in real-time. The proposed approach requires only sparely sampled data. Two new interpolation techniques are presented in this paper, the combination of which can produce reasonable rendering results for sparsely sampled real scenes. The first one, which is a color-matching based interpolation, searches for a possible physical point along the testing ray using color information in nearby reference images. The second technique, which is a disparity-matching based interpolation, tries to find the closest intersection between the testing ray and the disparity surfaces defined by nearby reference images. Both approaches are designed as backward rendering techniques and can be combined to produce robust results. Our experiments suggest that the proposed approach is capable of handling complex dynamic real scenes offline.

video 1 and video 2

Reference

M. Gong and Y.H. Yang, "Camera field rendering of static and dynamic scenes," Graphical Models, Vol. 67, 2005, pp. 73-99.