|
- - Graz University of Technology - |
-
- - Graz University of Technology - |
-
- - Graz University of Technology - |
-
- 
- |
-
- 
- |
-
- 
- |
-
| - Modern Neural Radiance Fields (NeRFs) learn a mapping from position to volumetric density via proposal network samplers. - In contrast to the coarse-to-fine sampling approach with two NeRFs, this offers significant potential for speedups using lower network capacity as the task of mapping spatial coordinates to volumetric density involves no view-dependent effects and is thus much easier to learn. - Given that most of the network capacity is utilized to estimate radiance, NeRFs could store valuable density information in their parameters or their deep features. - To this end, we take one step back and analyze large, trained ReLU-MLPs used in coarse-to-fine sampling. - We find that trained NeRFs, Mip-NeRFs and proposal network samplers map samples with high density to local minima along a ray in activation feature space. - We show how these large MLPs can be accelerated by transforming the intermediate activations to a weight estimate, without any modifications to the parameters post-optimization. - With our approach, we can reduce the computational requirements of trained NeRFs by up to 50% with only a slight hit in rendering quality and no changes to the training protocol or architecture. - We evaluate our approach on a variety of architectures and datasets, showing that our proposition holds in various settings. - | -
| - | -
- |
-  |
-
| - We analyze the histograms of trained NeRFs, Mip-NeRFs and proposal networks samples (as in Mip-NeRF 360) - and find that their intermediate activations are useful in finding a density estimate. - | -
| - Using our simple and efficient method, we can extract a density estimate from the activations of early - layers in coarse-to-fine sampling, which leads to a reduction in run-time with comparable visual quality. - | -
| - Our approach requires no pre-training or fine-tuning. To our knowledge, we are the first to use activations - from coordinate-based MLPs during inference. - | -
| - We evaluate our approach on real-world and synthetic data and obtain similar visual quality compared with NeRF, Mip-NeRf and a modified nerfacto model, - with lower inference cost. - | -
| - Our project is built upon Nerfstudio and our modules can be installed as a Python Package. - | -
- |
-  |
-
 |
- Lukas Radl, Andreas Kurz, Markus Steinberger. - Analyzing the Internals of Neural Radiance Fields. - arXiv preprint arXiv:2306.00696, 2023. - (hosted on arXiv) - |
-
.png)
+ + Modern Neural Radiance Fields (NeRFs) learn a mapping from position to volumetric density leveraging proposal network samplers. + In contrast to the coarse-to-fine sampling approach with two NeRFs, this offers significant potential for acceleration using lower network capacity. + Given that NeRFs utilize most of their network capacity to estimate radiance, they could store valuable density information in their parameters or their deep features. +
+ To investigate this proposition, we take one step back and analyze large, trained ReLU-MLPs used in coarse-to-fine sampling. + Building on our novel activation visualization method, we find that trained NeRFs, Mip-NeRFs and proposal network samplers map samples with high density to local minima along a ray in activation feature space. + We show how these large MLPs can be accelerated by transforming intermediate activations to a weight estimate, without any modifications to the training protocol or the network architecture. +
+ With our approach, we can reduce the computational requirements of trained NeRFs by up to 50% with only a slight hit in rendering quality. + Extensive experimental evaluation on a variety of datasets and architectures demonstrates the effectiveness of our approach. + Consequently, our methodology provides valuable insight into the inner workings of NeRFs. +
++ Here, provide comparisons of renderings from our method against the baseline. +
+
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ + Here, we show how different normalizations look for exemplary scenes. +
+| + | 0 | +1 | +2 | +3 | +4 | +5 | +6 | +7 | +
|---|
+ 
+ 
+
+ + Our approach for density estimation is driven by analyzing real-world-activations. + We observe that minima in activation feature space correspond to samples with high density. + Consequently, our approach does not require fine-tuning and can be applied to various NeRF-based methods. +
+.png)
+ + If you find our work useful, consider using a citation. +
+@inproceedings{radl2024nerfinternals,
+ author = {Radl, Lukas and Kurz, Andreas and Steiner, Michael and Steinberger, Markus},
+ title = {{Analyzing the Internals of Neural Radiance Fields}},
+ booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)},
+ year = {2024},
+}
-
-
-
- @article(Radl2023NerfInternals,
-
-
-
-
-
-
-
-
- title
-
-
- = {{Analyzing the Internals of Neural Radiance Fields}},
-
-
-
-
-
-
- author
-
-
- = {Radl, Lukas and Kurz, Andreas and Steinberger, Markus},
-
-
-
-
-
-
-
- journal
-
-
- = {arXiv preprint arXiv:2306.00696},
-
-
-
-
-
-
- year
-
-
- = {2023},
-
-
-
-
- }
-
-
-
-
-
-
-
-
-
-
-
- Acknowledgements |
-