3D reconstruction – Classical SfM vs. Deep Learning (NeRF)

Overview: This project explores 3D reconstruction techniques by comparing a classical Structure from Motion (SfM) approach with a deep learning-based Neural Radiance Fields (NeRF) model. We reconstructed a building’s 3D structure using feature matching, camera pose estimation, and triangulation, then compared the study by generating photorealistic novel views using NeRF, a neural network-based volumetric rendering technique.

GitHub Repository: View Source Code on GitHub

Key Features:

• Structure from Motion (SfM): Uses classical computer vision techniques like feature detection (SIFT), fundamental & essential matrices, PnP, and bundle adjustment. Generates a 3D point cloud representing the building’s geometry.
• Neural Radiance Fields (NeRF): Uses a deep neural network to synthesize realistic novel views from a sparse set of 2D images. Implements Tiny NeRF, an efficient version of NeRF, for volumetric scene representation.
• Dataset: A set of images captured with a GoPro Hero 3 of a building at Levine Hall, UPenn, corrected for fisheye lens distortion.
• Camera Pose Estimation: Extracted using Essential Matrix Decomposition and PnP with RANSAC. Triangulated 3D points were refined using Non-Linear Triangulation and Bundle Adjustment.

Results & Impact

SfM Approach: Successfully reconstructed a 3D point cloud of the building, though it remained sparse due to dataset limitations.

NeRF Approach: Generated high-quality novel views, offering a realistic visualization of the scene, but without explicit 3D coordinates.

Comparison:

• SfM excels in accurate spatial measurements and geometric scene reconstruction.
• NeRF provides photorealistic renderings but lacks geometric data.
• Combining both techniques can enhance 3D mapping and visualization for AR/VR and autonomous navigation.

Technologies Used:

• Classical Computer Vision: OpenCV, SIFT, RANSAC, PnP, Bundle Adjustment
• Deep Learning: Tiny NeRF, PyTorch, Positional Encoding
• 3D Reconstruction: Poisson Surface Reconstruction, Camera Pose Estimation
• Dataset: Custom GoPro Hero 3 dataset, Lego dataset (for NeRF)

This project evaluates classical vision-based 3D reconstruction and modern AI-driven rendering techniques, demonstrating the strengths and trade-offs of each approach in real-world applications.