SPARF: Large-Scale Learning of 3D Sparse Radiance Fields from Few Input Images (ICCV 2023)

By Abdullah Hamdi, Bernard Ghanem, Matthias Nießner

Paper | Video | Website | Dataset (code: sparf) .

The official Pytroch code of the paper SPARF: Large-Scale Learning of 3D Sparse Radiance Fields from Few Input Images. SPARF is a large-scale sparse radiance field dataset consisting of ~ 1 million SRFs with multiple voxel resolutions (32, 128, and 512) and 17 million posed images with a resolution of 400 X 400. Furthermore, we propose SuRFNet, a pipline to generate SRFs conditioned on input images, achieving SOTA on ShapeNet novel views synthesis from one or few input images.

Environment setup

follow instructions in INTALL.md to setup the conda environment.

SPARF Posed Multi-View Image Dataset

The dataset is released in the link (code: sparf). Each of SPARF's classes has the same structure of NeRF-synthetic dataset and can be loaded similarly. Download all content in the link and place inside data/SPARF_images. Then you can run the notebook example.

SPARF Radiance Field Dataset

The dataset is released in the link. Each of SPARF's instances has (beside the posed images above) two directories: STF (RGB voxels) and SRF (Spherical Harmonics voxels). The full radiance fileds are available under <CLASS_LABEL>/<INSTANCE_ID>/SRF/vox_<VOXEL-RESOLUTION>/full, where <VOXEL-RESOLUTION> is the resolution (32, 128 or 512). The partial SRFs are stored in <CLASS_LABEL>/<INSTANCE_ID>/STF/vox_<VOXEL-RESOLUTION>/partial similarly. The partitioning (shards) and splits of the dataset is available on the file SNRL_splits.csv in the root of the dataset. The voxles information are stored as sparse voxels in data_0.npzas coords and values.

Download all content in the link and place inside data/SPARF_srf. Then you can run the main training code.

Script for rendering ShapeNet images used in creating SPARF

make sure that ShapeNetCore.v2 is downloaded and placed in data/ShapeNetCore.v2. Then run the following script to render the images used in creating SPARF.

python run_sparf.py --run render --data_dir data/SPARF_srf/ --nb_views 400 --object_class car 

Script for extracting SPARF Radiance Fields (full SRFs with voxel res=128 and SH dim=4)

make sure that SPARF_images is downloaded and placed in data/SPARF_images. Then run the following script to extract the SRFs.

python run_sparf.py --run extract --vox_res 128 --sh_dim 4 --object_class airplane --data_dir data/SPARF_images/ --visualize --evaluate 

Script for extracting SPARF Radiance Fields (partial SRFs with voxel res=512 and SH dim=1, nb_views=3)

make sure that SPARF_images is downloaded and placed in data/SPARF_images. Then run the following script to extract the SRFs.

python run_sparf.py --run preprocess --vox_res 512 --sh_dim 1 --rf_variant 0 --object_class airplane --nb_views 3 --data_dir data/SPARF_images/ --randomized_views

Training and Inference pipeline on SPARF Radiance Fields

make sure that SPARF_srf is downloaded and placed in data/SPARF_srf. Then run the following script to train on SRFs.

python run_sparf.py --vox_res 128 --nb_views 3 --nb_rf_variants 4 --input_quantization_size 1.0 --strides 2 --lr_decay 0.99 --batch_size 6 --lr 1e-2 --visualize --normalize_input const --lambda_cls 30.0 --lambda_main 2.0 --augment_type none --mask_type densepoints --ignore_loss_mask --nb_frames 200 --validate_training  --data_dir data/SPARF_srf/ --run train --object_class airplane 

Citation

If you find our work useful in your research, please consider citing:

@InProceedings{Hamdi_2023_ICCV,
    author    = {Hamdi, Abdullah and Ghanem, Bernard and Nie{\ss}sner, Matthias},
    title     = {SPARF: Large-Scale Learning of 3D Sparse Radiance Fields from Few Input Images},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops},
    month     = {October},
    year      = {2023},
    pages     = {2930-2940}
}