content/3-2 NeRF神经辐射场.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# 3-2 NeRF神经辐射场\n",
    "  * [0 章节目标](#0-----)\n",
    "  * [1 外观和形状表征](#1--------)\n",
    "    + [1.1 形状表征](#11-----)\n",
    "    + [1.2 外观表征](#12-----)\n",
    "  * [2 神经辐射场NeRF](#2------nerf)\n",
    "  * [3 代码演示](#3-----)\n",
    "    + [3.1 依赖库安装](#31------)\n",
    "    + [3.2 下载并准备数据集](#32---------)\n",
    "    + [3.3 建立并启动viewer](#33------viewer)\n",
    "    + [3.4 开始训练](#34-----)"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 0 章节目标\n",
    "- 了解外观和形状表征，了解新视角合成任务\n",
    "- 理解神经渲染原理（NeRF）"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 1 外观和形状表征\n",
    "世界如此纷繁复杂？该以何种形式表示它们呢？我们可以将我们看到的世界解耦为两种形式，采用的是外观（appearance）和形状（shape/geometry）来表示一个物体或者场景。其中外观又细分为光照、材质等等。通过将组成世界的光照、几何、材质等元素进行解耦从而可以对场景进行参数化表示。计算机视觉中则是以简单粗暴的rgb色彩值来理解世界。\n",
    "\n",
    "他们之间通过渲染和反渲染的方式进行转换。如下图所示，[来源](https://www.bilibili.com/video/BV1d34y1n7fn/?spm_id_from=333.788&vd_source=f1a43cfabe61c271b4df05136a41d18c)\n",
    "\n",
    "<img alt=\"介绍\" height=\"360\" src=\"images/3-2_1-rendering-and-inverse-rendering.png\" width=\"800\"/>\n",
    "\n",
    "\n",
    "其中渲染依赖于形状和外观的表征，并且这个过程需要可微。而反渲染可以粗浅地理解为通过同一个物体的多张不同角度的rgb图片反推出物体的几何或者外观参数。渲染是一个确定的过程，而反渲染目前的很多方法都是通过神经网络来猜。\n",
    "\n",
    "<img alt=\"渲染过程\" height=\"360\" src=\"images/3-2_2-rendering.png\" width=\"500\"/>\n",
    "\n",
    "### 1.1 形状表征\n",
    "目前主流的如下图所示。其中，前三种方式称为显式的表征方式，它们一般便于编辑但消耗资源或者重建困难；后两种方式称为隐式的表征方式，它们一般重建方便但不易编辑，后两种方式都是通过运用神经网络来预测的，一个预测[符号距离函数](https://en.wikipedia.org/wiki/Signed_distance_function)的值，一个预测密度可见性的值。\n",
    "\n",
    "<img alt=\"渲染过程\" height=\"650\" src=\"images/3-1_1-shape-representation.png\" width=\"1200\"/>\n",
    "\n",
    "### 1.2 外观表征\n",
    "外观是个很大的概念，一般又分为外观又细分为光照、材质等等。在我们得到物体和场景的几何和形状信息之后，简单来说，之后的流程可以简化成我们是把物体材质的信息对应贴在几何体上面，如下图所示。而nerf的方式则是将这个过程统一起来得到RGB的颜色，在接下来我们将详细介绍。\n",
    "\n"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 2 神经辐射场NeRF\n",
    "\n",
    "### 2.1 简介\n",
    "NeRF (Neural Radiance Fields) [wiki](https://en.wikipedia.org/wiki/Neural_radiance_field)做的事情实质就是把形状和外观直接通过rgb值表示出来，不需要走传统图形学的那一套pipeline，这既是它的优点（可以很方便的表征场景）也是他的缺点（无法融入管线，使用不方便，编辑困难）。\n",
    "首先，我们先直观的感受下nerf做的事情。\n",
    "\n",
    "<img alt=\"介绍\" height=\"800\" src=\"./images/3-2_5-nerf-demo.gif\" width=\"1400\"/>\n",
    "\n",
    "然后来看nerf的framework。\n",
    "\n",
    "![nerf network](./images/3-2_3-nerf1.png)\n",
    "其实它做的就是输入一些视角的图片，然后丢到神经网络中去预测新视角的图片。\n",
    "\n",
    "具体而言就是输入一些空间中的坐标点的位置和方向角信息，之后通过神经网络来预测这些点的不透明度的值以及该点的颜色，最后通过rendering的方法来得到每个像素的颜色值，很多个像素最终组成整张图片的rgb值。\n",
    "![nerf network](./images/3-2_4-nerf2.png)\n",
    "\n",
    "知道这些之后基本就可以理解后面章节的内容了，如果想进一步了解原理的东西，可以参考以下资料：\n",
    "\n",
    "> [NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis](https://arxiv.org/pdf/2003.08934.pdf)\n",
    ">\n",
    "> [NeRF](https://www.matthewtancik.com/nerf)\n",
    ">\n",
    "> [都2022年了，我不允许你还不懂NeRF](https://zhuanlan.zhihu.com/p/569843149)\n",
    ">\n",
    "> [NeRF代码解读-相机参数与坐标系变换](https://zhuanlan.zhihu.com/p/593204605)\n",
    "\n",
    "\n",
    "### 2.2 对比\n",
    "NeRF的重建方式与工业界以及传统图形学的重建和渲染方式非常不一样，在传统图形学中，通常需要人工重建三维物体，然后再经过一些列渲染步骤才能得到渲染好的每一帧。目前，工业界主流的做饭还是用成熟的图形学渲染管线来实现，而NeRF在工业上的落地和应用相对较少，现在很多工作也尝试将NeRF的技术嵌入到现有的工作流程之中来实现更好的质量和效率。\n",
    "\n",
    "渲染管线包括顶点数据的输入、顶点着色器、曲面细分过程、几何着色器、图元组装、裁剪剔除、光栅化、片段着色器以及混合测试等步骤。更多图形学渲染细节参考[细说图形学渲染管线](https://zhuanlan.zhihu.com/p/79183044)。\n",
    "\n",
    "<img alt=\"渲染过程\" height=\"800\" src=\"images/3-2_6-graphic-pipeline.png\" width=\"1100\"/>\n",
    "\n",
    "\n",
    "NeRFs的工作流程我总结为采样（Sample）、存储和查找（Store/Look up）、渲染（Rendering）三个部分，其中最原始的NeRF在采样阶段用的是ray marching的单光线上采样多个点的方式，效率非常慢并且存在采样失真的现象，后续诸如[Mip-NeRF](https://github.com/google/mipnerf)、[Zip-NeRF](https://jonbarron.info/zipnerf/)、[point-nerf](https://github.com/Xharlie/pointnerf) 等很多工作对此方面进行了改进；在存储和查找阶段用的是MLP，神经网络本身就是具有一定的压缩性，所以最终的模型比较小，但是查找非常慢，后续诸如[DVGO](https://sunset1995.github.io/dvgo/)、[Plenoxels](https://alexyu.net/plenoxels/)、[TensoRF](https://apchenstu.github.io/TensoRF/)、[Instant-NGP](https://nvlabs.github.io/instant-ngp/)等很多工作对此方面进行了改进；在渲染阶段用的是volume rendering而这个阶段本身就比较依赖前两个阶段用的方法，而这个过程也是比较慢且需要多次查询MLP，所以后续诸如point-based的方法用传统的图形学渲染的方法快速提升了渲染效率。\n",
    "\n",
    "\n",
    "![nerf network](./images/3-2_7-nerf-pipeline.png)\n",
    "\n",
    "\n",
    "\n"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 3 代码演示\n",
    "> 代码参考：[nerfstudio-colab](https://colab.research.google.com/github/nerfstudio-project/nerfstudio/blob/main/colab/demo.ipynb)"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 3.1 依赖库安装"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "%cd /content/\n",
    "!pip install --upgrade pip\n",
    "!pip install torch==2.0.1+cu118 torchvision==0.15.2+cu118 --extra-index-url https://download.pytorch.org/whl/cu118\n",
    "\n",
    "# Installing TinyCuda\n",
    "%cd /content/\n",
    "!gdown \"https://drive.google.com/u/1/uc?id=1-7x7qQfB7bIw2zV4Lr6-yhvMpjXC84Q5&confirm=t\"\n",
    "!pip install tinycudann-1.7-cp310-cp310-linux_x86_64.whl\n",
    "\n",
    "# Installing COLMAP\n",
    "%cd /content/\n",
    "!gdown \"https://drive.google.com/u/0/uc?id=15WngFRNar_b8CaPR5R-hvQ3eAnlyk_SL&confirm=t\"\n",
    "!sudo apt-get install \\\n",
    "    build-essential \\\n",
    "    libboost-program-options-dev \\\n",
    "    libboost-filesystem-dev \\\n",
    "    libboost-graph-dev \\\n",
    "    libboost-system-dev \\\n",
    "    libboost-test-dev \\\n",
    "    libeigen3-dev \\\n",
    "    libflann-dev \\\n",
    "    libfreeimage-dev \\\n",
    "    libmetis-dev \\\n",
    "    libgoogle-glog-dev \\\n",
    "    libgflags-dev \\\n",
    "    libsqlite3-dev \\\n",
    "    libglew-dev \\\n",
    "    qtbase5-dev \\\n",
    "    libqt5opengl5-dev \\\n",
    "    libcgal-dev \\\n",
    "    libceres-dev\n",
    "!unzip local.zip -d /usr/\n",
    "!chmod +x /usr/local/bin/colmap\n",
    "\n",
    "# Install nerfstudio\n",
    "%cd /content/\n",
    "!pip install git+https://github.com/nerfstudio-project/nerfstudio.git"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 3.2 下载并准备数据集"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "#@markdown <h3>Pick the preset scene or upload your own images/video</h3>\n",
    "import os\n",
    "import glob\n",
    "from google.colab import files\n",
    "from IPython.core.display import display, HTML\n",
    "\n",
    "scene = '\\uD83D\\uDE9C dozer' #@param ['🖼 poster', '🚜 dozer', '🌄 desolation', '📤 upload your images' , '🎥 upload your own video', '🔺 upload Polycam data', '💽 upload your own Record3D data']\n",
    "scene = ' '.join(scene.split(' ')[1:])\n",
    "\n",
    "if scene == \"upload Polycam data\":\n",
    "    %cd /content/\n",
    "    !mkdir -p /content/data/nerfstudio/custom_data\n",
    "    %cd /content/data/nerfstudio/custom_data/\n",
    "    uploaded = files.upload()\n",
    "    dir = os.getcwd()\n",
    "    if len(uploaded.keys()) > 1:\n",
    "        print(\"ERROR, upload a single .zip file when processing Polycam data\")\n",
    "    dataset_dir = [os.path.join(dir, f) for f in uploaded.keys()][0]\n",
    "    !ns-process-data polycam --data $dataset_dir --output-dir /content/data/nerfstudio/custom_data/\n",
    "    scene = \"custom_data\"\n",
    "elif scene == 'upload your own Record3D data':\n",
    "    display(HTML('<h3>Zip your Record3D folder, and upload.</h3>'))\n",
    "    display(HTML('<h3>More information on Record3D can be found <a href=\"https://docs.nerf.studio/en/latest/quickstart/custom_dataset.html#record3d-capture\" target=\"_blank\">here</a>.</h3>'))\n",
    "    %cd /content/\n",
    "    !mkdir -p /content/data/nerfstudio/custom_data\n",
    "    %cd /content/data/nerfstudio/custom_data/\n",
    "    uploaded = files.upload()\n",
    "    dir = os.getcwd()\n",
    "    preupload_datasets = [os.path.join(dir, f) for f in uploaded.keys()]\n",
    "    record_3d_zipfile = preupload_datasets[0]\n",
    "    !unzip $record_3d_zipfile -d /content/data/nerfstudio/custom_data\n",
    "    custom_data_directory = glob.glob('/content/data/nerfstudio/custom_data/*')[0]\n",
    "    !ns-process-data record3d --data $custom_data_directory --output-dir /content/data/nerfstudio/custom_data/\n",
    "    scene = \"custom_data\"\n",
    "elif scene in ['upload your images', 'upload your own video']:\n",
    "    display(HTML('<h3>Select your custom data</h3>'))\n",
    "    display(HTML('<p/>You can select multiple images by pressing ctrl, cmd or shift and click.<p>'))\n",
    "    display(HTML('<p/>Note: This may take time, especially on higher resolution inputs, so we recommend to download dataset after creation.<p>'))\n",
    "    !mkdir -p /content/data/nerfstudio/custom_data\n",
    "    if scene == 'upload your images':\n",
    "        !mkdir -p /content/data/nerfstudio/custom_data/raw_images\n",
    "        %cd /content/data/nerfstudio/custom_data/raw_images\n",
    "        uploaded = files.upload()\n",
    "        dir = os.getcwd()\n",
    "    else:\n",
    "        %cd /content/data/nerfstudio/custom_data/\n",
    "        uploaded = files.upload()\n",
    "        dir = os.getcwd()\n",
    "    preupload_datasets = [os.path.join(dir, f) for f in uploaded.keys()]\n",
    "    del uploaded\n",
    "    %cd /content/\n",
    "\n",
    "    if scene == 'upload your images':\n",
    "        !ns-process-data images --data /content/data/nerfstudio/custom_data/raw_images --output-dir /content/data/nerfstudio/custom_data/\n",
    "    else:\n",
    "        video_path = preupload_datasets[0]\n",
    "        !ns-process-data video --data $video_path --output-dir /content/data/nerfstudio/custom_data/\n",
    "\n",
    "    scene = \"custom_data\"\n",
    "else:\n",
    "    %cd /content/\n",
    "    !ns-download-data nerfstudio --capture-name=$scene\n",
    "\n",
    "print(\"Data Processing Succeeded!\")"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 3.3 建立并启动viewer"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "%cd /content\n",
    "\n",
    "# Install localtunnel\n",
    "# We are using localtunnel https://github.com/localtunnel/localtunnel but ngrok could also be used\n",
    "!npm install -g localtunnel\n",
    "\n",
    "# Tunnel port 7007, the default for\n",
    "!rm url.txt 2> /dev/null\n",
    "get_ipython().system_raw('lt --port 7007 >> url.txt 2>&1 &')\n",
    "\n",
    "import time\n",
    "time.sleep(3) # the previous command needs time to write to url.txt\n",
    "\n",
    "\n",
    "with open('url.txt') as f:\n",
    "  lines = f.readlines()\n",
    "websocket_url = lines[0].split(\": \")[1].strip().replace(\"https\", \"wss\")\n",
    "# from nerfstudio.utils.io import load_from_json\n",
    "# from pathlib import Path\n",
    "# json_filename = \"nerfstudio/nerfstudio/viewer/app/package.json\"\n",
    "# version = load_from_json(Path(json_filename))[\"version\"]\n",
    "url = f\"https://viewer.nerf.studio/?websocket_url={websocket_url}\"\n",
    "print(url)\n",
    "print(\"You may need to click Refresh Page after you start training!\")\n",
    "from IPython import display\n",
    "display.IFrame(src=url, height=800, width=\"100%\")"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 3.4 开始训练"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "%cd /content\n",
    "if os.path.exists(f\"data/nerfstudio/{scene}/transforms.json\"):\n",
    "    !ns-train nerfacto --viewer.websocket-port 7007 nerfstudio-data --data data/nerfstudio/$scene --downscale-factor 4\n",
    "else:\n",
    "    from IPython.core.display import display, HTML\n",
    "    display(HTML('<h3 style=\"color:red\">Error: Data processing did not complete</h3>'))\n",
    "    display(HTML('<h3>Please re-run `Downloading and Processing Data`, or view the FAQ for more info.</h3>'))"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "\n"
   ],
   "metadata": {
    "collapsed": false
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}