Skip to content

Latest commit

 

History

History
113 lines (79 loc) · 6.4 KB

README.md

File metadata and controls

113 lines (79 loc) · 6.4 KB
Raw

PWC

PWC

PWC

PWC

Table of contents
  1. Features
  2. Datasets
  3. Installation
  4. Data Preparation
  5. Training and Testing
  6. Quick Demo
  7. Visualization
  8. Acknowledgments
  9. Contacts

ISBNet: a 3D Point Cloud Instance Segmentation Network with Instance-aware Sampling and Box-aware Dynamic Convolution

Tuan Duc Ngo, Binh-Son Hua, Khoi Nguyen
VinAI Research, Vietnam

Abstract: Existing 3D instance segmentation methods are predominant by a bottom-up design: a manually fine-tuned algorithm to group points into clusters followed by a refinement network. Relying on the quality of the clusters, these methods generate susceptible results when (1) nearby objects with the same semantic class are packed together, or (2) large objects with complex shapes. To address these shortcomings, we introduce ISBNet, a novel cluster-free method that represents instances as kernels and decodes instance masks via dynamic convolution. To efficiently generate a high-recall and discriminative kernel set, we propose a simple strategy, named Instance-aware Farthest Point Sampling, to sample candidates and leverage the point aggregation layer adopted from PointNet++ to encode candidate features. Moreover, we show that training 3D instance segmentation in a multi-task learning setting with an additional axis-aligned bounding box prediction head further boosts performance. Our method set new state-of-the-art results on ScanNetV2 (55.9), S3DIS (60.8), and STPLS3D (49.2) in terms of AP and retains fast inference time (237ms per scene on ScanNetV2).

overview

Details of the model architecture and experimental results can be found in our paper:

@inproceedings{ngo2023isbnet,
 author={Tuan Duc Ngo, Binh-Son Hua, Khoi Nguyen},
 booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
 title={ISBNet: a 3D Point Cloud Instance Segmentation Network with Instance-aware Sampling and Box-aware Dynamic Convolution},
 year= {2023}
}

Please CITE our paper whenever this repository is used to help produce published results or incorporated into other software.

Features 📣

  • State-of-the-art performance on ScanNetV2, S3DIS, and STPLS3D.
  • High speed of 237ms per scan on ScanNetV2 dataset.
  • Reproducibility code for both ScanNetV2, S3DIS and STPLS3D datasets.

Datasets 💾

  • ScanNetV2
  • ScanNetV2-200
  • S3DIS
  • STPLS3D

Installation 📝

Please refer to installation guide.

Data Preparation 🔨

Please refer to data preparation.

Training and Testing 🚆

Please refer to training guide.

Quick Demo 🔥

ScanNetv2

Dataset AP AP_50 Config Checkpoint
ScanNet test 55.9 76.3
ScanNet val (paper) 54.5 73.1
ScanNet val 56.8 73.3 config checkpoint
ScanNet val (lightweight) 50.1 68.9 config checkpoint

ScanNetv2-200

Dataset AP AP_50 Config Checkpoint
ScanNet200 val 24.5 32.7 config checkpoint

S3DIS

Dataset AP AP_50 Config Checkpoint
Area 5 56.3 67.5 config checkpoint

STPLS3D

Dataset AP AP_50 Config Checkpoint
STPLS3D val 51.2 66.7 config checkpoint

Run evaluation with pre-trained models:

python3 tools/test.py <path_to_config_file> <path_to_pretrain_weight>

Visualization 💻

Please refer to visualization guide. We provide the qualitative results of our method at here

Acknowledgements 👏

This repo is built upon SpConv, DyCo3D, SSTNet, and SoftGroup.

Contacts 📧

If you have any questions or suggestions about this repo, please feel free to contact me ([email protected]).