Skip to content

Latest commit

 

History

History
123 lines (92 loc) · 4.55 KB

README.md

File metadata and controls

123 lines (92 loc) · 4.55 KB

GenNBV: Generalizable Next-Best-View Policy for Active 3D Reconstruction

CVPR 2024
Xiao Chen  Quanyi Li  Tai Wang  Tianfan Xue  Jiangmiao Pang 
Shanghai AI Laboratory The Chinese University of Hong Kong

arXiv

📋 Contents (Coming Soon)

  1. About
  2. Getting Started
  3. Model and Benchmark
  4. TODO List
  5. Citation
  6. License
  7. Acknowledgements

🏠 About

Dialogue_Teaser
While recent advances in neural radiance field enable realistic digitization for large-scale scenes, the image-capturing process is still time-consuming and labor-intensive. Previous works attempt to automate this process using the Next-Best-View (NBV) policy for active 3D reconstruction. However, the existing NBV policies heavily rely on hand-crafted criteria, limited action space, or per-scene optimized representations. These constraints limit their cross-dataset generalizability. To overcome them, we propose GenNBV, an end-to-end generalizable NBV policy. Our policy adopts a reinforcement learning (RL)-based framework and extends typical limited action space to 5D free space. It empowers our agent drone to scan from any viewpoint, and even interact with unseen geometries during training. To boost the cross-dataset generalizability, we also propose a novel multi-source state embedding, including geometric, semantic, and action representations. We establish a benchmark using the Isaac Gym simulator with the Houses3K and OmniObject3D datasets to evaluate this NBV policy. Experiments demonstrate that our policy achieves a 98.26% and 97.12% coverage ratio on unseen building-scale objects from these datasets, respectively, outperforming prior solutions.

📚 Getting Started

Installation

We test our codes under the following environment:

  • Ubuntu 20.04
  • NVIDIA Driver: 545.29.02
  • CUDA 11.3
  • Python 3.8.12
  • PyTorch 1.11.0+cu113
  • PyTorch3D 0.7.5
  1. Clone this repository.
git clone https://github.com/zjwzcx/GenNBV
cd GenNBV
  1. Create an environment and install PyTorch.
conda create -n gennbv python=3.8 -y  # pytorch3d needs python>3.7
conda activate gennbv
pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0 --extra-index-url https://download.pytorch.org/whl/cu113
  1. NVIDIA Isaac Gym Installation: https://developer.nvidia.com/isaac-gym/download
cd isaacgym/python
pip install -e .
  1. Install GenNBV.
pip install -r requirements.txt
pip install -e .

Data Preparation

Please refer to the guide for downloading and organization.

📦 Model and Benchmark

Model Overview

Benchmark Overview

📝 TODO List

🔗 Citation

If you find our work helpful, please cite:

@inproceedings{chen2024gennbv,
  title={GenNBV: Generalizable Next-Best-View Policy for Active 3D Reconstruction},
  author={Chen, Xiao and Li, Quanyi and Wang, Tai and Xue, Tianfan and Pang, Jiangmiao},
  year={2024}
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
}

📄 License

Creative Commons License
This work is under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

👏 Acknowledgements