fuji-sfm-and-pfuji-size-pcl-npy-preparator.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "### Point Cloud Dataset Preparation Task\n",
        "\n",
        "With this script we intend to create base segmentation dataset for RandLA-Net's\n",
        "training to segment out the apple class objects.\n",
        "For this task, we first prepare two base separate point cloud datasets with `Fuji-SfM` and `PFuji-Size` datasets. And, then we merge these two datasets to train our models.\n",
        "\n",
        "Also, by just using the `Fuji-SfM` dataset we have observed that the model is unable to train and generalize the point cloud segmentation learning.\n",
        "Second, we have also observed that at further upsampling is required even at patch level data PCs for making the learning process more easier for the segmentation model.\n",
        "\n",
        "Here, in our experimentation we will prepare two merged dataset zip files for training, validation and evaluation. First, will contain the base merged dataset with x, y, z-coordinates and dropped out r,g,b-color information of 30 %. Second, the other dataset will additionally the normal information as well with 30 % of the data being dropped out for making the learning more robust."
      ],
      "metadata": {
        "id": "uSPj-ymFxzUe"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 1. PFuji-Size Data Extraction and Base Point Cloud File Preparation"
      ],
      "metadata": {
        "id": "G0XHU2X5aLbL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# installing laspy & opend3d package, post installation restart this python environment\n",
        "!pip install \"laspy[lazrs,laszip]\"\n",
        "!pip install open3d"
      ],
      "metadata": {
        "id": "uwcwCVrxiwCi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0CyA1iEpKFMc",
        "outputId": "8d5a199a-e9fc-4622-bdf7-2d58073bbeba"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ],
      "source": [
        "# for loading the dataset zips into the runtime\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# import statements\n",
        "import laspy\n",
        "import random\n",
        "import numpy as np\n",
        "import open3d as o3d\n",
        "from tqdm import tqdm\n",
        "# for loading all the directory files\n",
        "from os import listdir\n",
        "from os.path import isfile, join"
      ],
      "metadata": {
        "id": "cI0PDx2UjDF3"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LKEdWvZhKNII"
      },
      "outputs": [],
      "source": [
        "# Fuji-SfM and PFuji-Size dataset files for generating datasets\n",
        "# TODO: update the below paths based on your project setup\n",
        "!ls drive/MyDrive/point-cloud-prototyping/datasets/*.zip"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### 1.a Data Extration and Preparation of `PFuji-Size` dataset"
      ],
      "metadata": {
        "id": "z9bKQPHmuL_x"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# unzipping the PFuji-Size dataset point cloud datasets for apple and tree class\n",
        "# TODO: update the below paths based on your project setup\n",
        "!unzip drive/MyDrive/point-cloud-prototyping/datasets/2-trees_point_clouds.zip -d .\n",
        "!unzip drive/MyDrive/point-cloud-prototyping/datasets/3-apple_point_clouds.zip -d ."
      ],
      "metadata": {
        "id": "7jsVOwEwd7WN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# for more color robustness we are going to use west side for 2018 orchard\n",
        "# and east side for 2020 point cloud dataset building, and build two seperate\n",
        "# '.pkl' point cloud files for the dataset\n",
        "!ls 2-trees_point_clouds/"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WRDohmgZlpgF",
        "outputId": "279e48e8-f8b7-44d5-bba1-18b0275bd331"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Orch2018_east.laz  Orch2018_west.laz  Orch2020-east.laz  Orch2020-west.laz\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# extract '.laz' data from the 'Orch2018_west.laz' and 'Orch18_east.laz' files\n",
        "# only normalization of rgb values but not the xyz coordinates\n",
        "# TODO: update the below paths based on your project setup\n",
        "with laspy.open('2-trees_point_clouds/Orch2018_west.laz') as orch_eghtn:\n",
        "    las = orch_eghtn.read()\n",
        "    x_coords = (las.X * las.header.scales[0]) + las.header.offsets[0]\n",
        "    y_coords = (las.Y * las.header.scales[1]) + las.header.offsets[1]\n",
        "    z_coords = (las.Z * las.header.scales[2]) + las.header.offsets[2]\n",
        "\n",
        "    r_vals = (las.red - np.min(las.red)) / (np.max(las.red) - np.min(las.red))\n",
        "    g_vals = (las.green - np.min(las.green)) / (np.max(las.green) - np.min(las.green))\n",
        "    b_vals = (las.blue - np.min(las.blue)) / (np.max(las.blue) - np.min(las.blue))\n",
        "    orch_eghtn_pcl_arr_west = np.column_stack((x_coords, y_coords, z_coords,\n",
        "                                          r_vals, g_vals, b_vals))"
      ],
      "metadata": {
        "id": "93W3szRRfvx6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# TODO: update the below paths based on your project setup\n",
        "with laspy.open('2-trees_point_clouds/Orch2018_east.laz') as orch_eghtn:\n",
        "    las = orch_eghtn.read()\n",
        "    x_coords = (las.X * las.header.scales[0]) + las.header.offsets[0]\n",
        "    y_coords = (las.Y * las.header.scales[1]) + las.header.offsets[1]\n",
        "    z_coords = (las.Z * las.header.scales[2]) + las.header.offsets[2]\n",
        "\n",
        "    r_vals = (las.red - np.min(las.red)) / (np.max(las.red) - np.min(las.red))\n",
        "    g_vals = (las.green - np.min(las.green)) / (np.max(las.green) - np.min(las.green))\n",
        "    b_vals = (las.blue - np.min(las.blue)) / (np.max(las.blue) - np.min(las.blue))\n",
        "    orch_eghtn_pcl_arr_east = np.column_stack((x_coords, y_coords, z_coords,\n",
        "                                          r_vals, g_vals, b_vals))"
      ],
      "metadata": {
        "id": "efJ9ZCSU9i8k"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# exploratory analysis for the pcl array for\n",
        "# the 'Orch2018_west.laz' & 'Orch2018_east.laz' files\n",
        "print(orch_eghtn_pcl_arr_west, orch_eghtn_pcl_arr_west.shape)\n",
        "print(orch_eghtn_pcl_arr_east, orch_eghtn_pcl_arr_east.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DSpDCcZ89k0e",
        "outputId": "ef8e9fdc-e163-4361-b6e2-e66e4910738d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[ 0.80662251  3.47955847 -0.18946247  0.61568627  0.52362205  0.40392157]\n",
            " [ 0.78669149  3.49546671 -0.1783798   0.55294118  0.45275591  0.3372549 ]\n",
            " [ 0.80395353  3.47906494 -0.18989898  0.59215686  0.49606299  0.37647059]\n",
            " ...\n",
            " [-0.78412449  1.18851221  3.78472638  0.65882353  0.73622047  0.89019608]\n",
            " [-0.78315645  1.18451405  3.78412104  0.65490196  0.73228346  0.87058824]\n",
            " [-0.77879816  1.15837657  3.78567338  0.63529412  0.70866142  0.8627451 ]] (8884179, 6)\n",
            "[[-1.53670645 -0.09859817 -0.25578997  0.62598425  0.52380952  0.38823529]\n",
            " [-1.53300416 -0.09175679 -0.26506999  0.52362205  0.48015873  0.36470588]\n",
            " [-1.5359515  -0.09223113 -0.26814881  0.57874016  0.55555556  0.4627451 ]\n",
            " ...\n",
            " [ 1.23130023  3.5960362  -0.20349368  0.56692913  0.51190476  0.45882353]\n",
            " [ 1.22598445  3.59545898 -0.20638813  0.48425197  0.44444444  0.41568627]\n",
            " [ 1.23562944  3.59649801 -0.2042381   0.57480315  0.52777778  0.47843137]] (8763029, 6)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# combine two numpy array pcl data for adding more color data info\n",
        "orch_eghtn_pcl_arr = np.concatenate((orch_eghtn_pcl_arr_west,\n",
        "                                     orch_eghtn_pcl_arr_east), axis=0)"
      ],
      "metadata": {
        "id": "fbkVLSvd--jK"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# exploratory analysis for the pcl array for the combined pcl dataset\n",
        "print(orch_eghtn_pcl_arr, orch_eghtn_pcl_arr.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "udGLpwUcvAYA",
        "outputId": "230da5e8-62f8-4e68-f99c-35dbd07499e3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[ 0.80662251  3.47955847 -0.18946247  0.61568627  0.52362205  0.40392157]\n",
            " [ 0.78669149  3.49546671 -0.1783798   0.55294118  0.45275591  0.3372549 ]\n",
            " [ 0.80395353  3.47906494 -0.18989898  0.59215686  0.49606299  0.37647059]\n",
            " ...\n",
            " [ 1.23130023  3.5960362  -0.20349368  0.56692913  0.51190476  0.45882353]\n",
            " [ 1.22598445  3.59545898 -0.20638813  0.48425197  0.44444444  0.41568627]\n",
            " [ 1.23562944  3.59649801 -0.2042381   0.57480315  0.52777778  0.47843137]] (17647208, 6)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# TODO: update the below paths based on your project setup\n",
        "with laspy.open('2-trees_point_clouds/Orch2020-east.laz') as orch_twnty:\n",
        "    las = orch_twnty.read()\n",
        "    x_coords = (las.X * las.header.scales[0]) + las.header.offsets[0]\n",
        "    y_coords = (las.Y * las.header.scales[1]) + las.header.offsets[1]\n",
        "    z_coords = (las.Z * las.header.scales[2]) + las.header.offsets[2]\n",
        "    r_vals = (las.red - np.min(las.red)) / (np.max(las.red) - np.min(las.red))\n",
        "    g_vals = (las.green - np.min(las.green)) / (np.max(las.green) - np.min(las.green))\n",
        "    b_vals = (las.blue - np.min(las.blue)) / (np.max(las.blue) - np.min(las.blue))\n",
        "    orch_twnty_pcl_arr_east = np.column_stack((x_coords, y_coords, z_coords,\n",
        "                                          r_vals, g_vals, b_vals))"
      ],
      "metadata": {
        "id": "Q6d_N1hIwEFU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# TODO: update the below paths based on your project setup\n",
        "with laspy.open('2-trees_point_clouds/Orch2020-west.laz') as orch_twnty:\n",
        "    las = orch_twnty.read()\n",
        "    x_coords = (las.X * las.header.scales[0]) + las.header.offsets[0]\n",
        "    y_coords = (las.Y * las.header.scales[1]) + las.header.offsets[1]\n",
        "    z_coords = (las.Z * las.header.scales[2]) + las.header.offsets[2]\n",
        "    r_vals = (las.red - np.min(las.red)) / (np.max(las.red) - np.min(las.red))\n",
        "    g_vals = (las.green - np.min(las.green)) / (np.max(las.green) - np.min(las.green))\n",
        "    b_vals = (las.blue - np.min(las.blue)) / (np.max(las.blue) - np.min(las.blue))\n",
        "    orch_twnty_pcl_arr_west = np.column_stack((x_coords, y_coords, z_coords,\n",
        "                                          r_vals, g_vals, b_vals))"
      ],
      "metadata": {
        "id": "rr0ZlpZrAG9b"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# exploratory analysis for the pcl array for\n",
        "# the 'Orch2020-west.laz' & 'Orch2020-east.laz' files\n",
        "print(orch_twnty_pcl_arr_west, orch_twnty_pcl_arr_west.shape)\n",
        "print(orch_twnty_pcl_arr_east, orch_twnty_pcl_arr_east.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dZB6GQbpALOV",
        "outputId": "3eea50a0-22e5-4368-9067-9370d6c91a19"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[-0.25839853  2.95192218 -0.11821502  0.16078431  0.14117647  0.10980392]\n",
            " [-0.40850973  3.02875566 -0.1349324   0.6627451   0.62352941  0.46666667]\n",
            " [-0.40811172  3.03178954 -0.13438649  0.75294118  0.70196078  0.56862745]\n",
            " ...\n",
            " [ 0.21464436 -0.56504387  3.99386597  0.81960784  0.86666667  0.95686275]\n",
            " [ 0.21260631 -0.57097203  3.99224329  0.63529412  0.68627451  0.82745098]\n",
            " [ 0.211767   -0.56696969  3.99352241  0.78039216  0.82745098  0.89803922]] (9043033, 6)\n",
            "[[-1.5013206  -0.4447605  -0.25654498  0.21960784  0.2         0.14117647]\n",
            " [-1.52162778 -0.44923183 -0.26363975  0.2         0.18823529  0.13333333]\n",
            " [-1.52707398 -0.45156905 -0.26595756  0.20784314  0.2         0.13333333]\n",
            " ...\n",
            " [ 0.06417678  3.3079443   3.59787273  0.72941176  0.8         0.94509804]\n",
            " [ 0.06322569  3.29847312  3.59692883  0.78823529  0.83921569  0.93333333]\n",
            " [ 0.06022703  3.31223679  3.596627    0.7372549   0.8         0.89411765]] (9609280, 6)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# combine two numpy array pcl data for adding more color data info\n",
        "orch_twnty_pcl_arr = np.concatenate((orch_twnty_pcl_arr_west,\n",
        "                                     orch_twnty_pcl_arr_east), axis=0)"
      ],
      "metadata": {
        "id": "8rFMAtW-AZFd"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# exploratory analysis for the pcl array for the combined pcl dataset\n",
        "print(orch_twnty_pcl_arr, orch_twnty_pcl_arr.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FDGlUFb0wE7q",
        "outputId": "e8534d8f-d8ad-4663-a1c3-e2f7710f743a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[-0.25839853  2.95192218 -0.11821502  0.16078431  0.14117647  0.10980392]\n",
            " [-0.40850973  3.02875566 -0.1349324   0.6627451   0.62352941  0.46666667]\n",
            " [-0.40811172  3.03178954 -0.13438649  0.75294118  0.70196078  0.56862745]\n",
            " ...\n",
            " [ 0.06417678  3.3079443   3.59787273  0.72941176  0.8         0.94509804]\n",
            " [ 0.06322569  3.29847312  3.59692883  0.78823529  0.83921569  0.93333333]\n",
            " [ 0.06022703  3.31223679  3.596627    0.7372549   0.8         0.89411765]] (18652313, 6)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# loading the apple point clouds from '.txt' files of 2018 and 2020 orchards data\n",
        "!ls 3-apple_point_clouds/2018_* | head -3"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "H6c8iTJqOZuX",
        "outputId": "b148b74c-23ce-46a8-acd3-44807e67ed87"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "3-apple_point_clouds/2018_01_001.txt\n",
            "3-apple_point_clouds/2018_01_002.txt\n",
            "3-apple_point_clouds/2018_01_003.txt\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!ls 3-apple_point_clouds/2018_* | tail -3"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "paDkrowyR-Ns",
        "outputId": "966c1cf2-2eeb-4176-ac5a-d347c7060cd0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "3-apple_point_clouds/2018_01_338.txt\n",
            "3-apple_point_clouds/2018_01_339.txt\n",
            "3-apple_point_clouds/2018_01_340.txt\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!ls 3-apple_point_clouds/2020_01* | head -3"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AShxQuE7xVtR",
        "outputId": "c1e7cc8b-f46c-468d-8132-ea0bb726a8ff"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "3-apple_point_clouds/2020_01_001.txt\n",
            "3-apple_point_clouds/2020_01_002.txt\n",
            "3-apple_point_clouds/2020_01_003.txt\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!ls 3-apple_point_clouds/2020_01* | tail -3"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xfGqP_gWUGfU",
        "outputId": "861d4e67-ab42-412b-f335-2ce1ed7b2750"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "3-apple_point_clouds/2020_01_335.txt\n",
            "3-apple_point_clouds/2020_01_336.txt\n",
            "3-apple_point_clouds/2020_01_337.txt\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# peek into the file structure of 2018 and 2020 orchards data\n",
        "!head -5 3-apple_point_clouds/2018_01_001.txt"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qDDY6k-ixV02",
        "outputId": "6f05b98d-7999-4be4-eec5-9ef435764e7e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "-0.22910452 -0.09944689 0.65560591 153 122 79 0.000000 1.000000 0.000000 -0.223879 -0.709210 -0.668505\n",
            "-0.22910452 -0.09944689 0.65560591 153 122 79 0.000000 1.000000 0.000000 -0.223879 -0.709210 -0.668505\n",
            "-0.22910452 -0.09944689 0.65560591 153 122 79 0.000000 1.000000 0.000000 -0.223879 -0.709210 -0.668505\n",
            "-0.22910452 -0.09944689 0.65560591 153 122 79 0.000000 1.000000 0.000000 -0.223879 -0.709210 -0.668505\n",
            "-0.22910452 -0.09944689 0.65560591 153 121 79 0.000000 1.000000 0.000000 -0.223879 -0.709210 -0.668505\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!head -5 3-apple_point_clouds/2018_01_338.txt"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aLqjwZheSJ78",
        "outputId": "fedc062e-fae7-4ba1-99f2-a38ffebaab7a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.60063934 2.21131873 0.23708095 144 122 52 0.000000 4.000000 1.000000 0.597226 0.246101 -0.763384\n",
            "0.61113667 2.20790339 0.24079944 150 125 62 0.000000 1.000000 1.000000 0.473407 0.225360 -0.851527\n",
            "0.58905882 2.21994638 0.22884357 148 125 49 0.000000 1.000000 1.000000 0.543262 -0.115061 -0.831642\n",
            "0.60526359 2.20557427 0.23708454 142 120 56 0.000000 1.000000 1.000000 0.509396 0.107619 -0.853776\n",
            "0.60024565 2.20175362 0.23327842 143 118 52 0.000000 1.000000 1.000000 0.536338 0.026062 -0.843601\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# verifying whether the '.laz' extraction was done correctly\n",
        "print(np.max(orch_eghtn_pcl_arr[:, 0]), np.min(orch_eghtn_pcl_arr[:, 0]))\n",
        "print(np.max(orch_eghtn_pcl_arr[:, 1]), np.min(orch_eghtn_pcl_arr[:, 1]))\n",
        "print(np.max(orch_eghtn_pcl_arr[:, 2]), np.min(orch_eghtn_pcl_arr[:, 2]))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oW9JuolESqvZ",
        "outputId": "23504fbd-4c3e-44cf-d819-c0edd823198e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2.47710729 -1.98462462\n",
            "3.74508071 -0.34017122\n",
            "3.98604536 -0.28742504\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!head -5 3-apple_point_clouds/2020_01_001.txt"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uOVIbvWlRHgn",
        "outputId": "2d525524-6167-4c58-8b13-1d924885761d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "-0.32185173 -0.33695343 0.36797228 137 129 57 0.000000 1.000000 0.000000 -0.517100 0.349566 -0.781288\n",
            "-0.32471815 -0.33817196 0.36393949 131 119 59 0.000000 1.000000 0.000000 -0.054135 -0.134761 0.989398\n",
            "-0.32538888 -0.34947816 0.36468124 122 111 46 0.000000 1.000000 0.000000 -0.019902 0.998259 0.055517\n",
            "-0.31823555 -0.33507586 0.36933643 124 118 56 0.000000 1.000000 0.000000 -0.797107 0.570673 -0.197364\n",
            "-0.32462609 -0.34697980 0.36600029 121 113 48 0.000000 1.000000 0.000000 -0.448421 0.822370 -0.350181\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!head -5 3-apple_point_clouds/2020_01_337.txt"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lmuQUF6YRHw6",
        "outputId": "5fc3e13d-8bef-484b-aa8b-00b3e172767a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "-0.15741143 1.37385952 2.87322736 41 47 21 0.000000 1.000000 0.000000 0.050819 -0.877652 0.476597\n",
            "-0.15816760 1.36673987 2.86788082 44 50 22 0.000000 1.000000 0.000000 -0.330476 -0.796654 0.506091\n",
            "-0.15923078 1.36683381 2.87112236 44 50 22 0.000000 1.000000 0.000000 -0.432213 -0.652491 0.622453\n",
            "-0.15587974 1.37375402 2.87360597 41 46 21 0.000000 1.000000 0.000000 -0.662164 0.391354 -0.639046\n",
            "-0.15897395 1.36760497 2.87156463 43 49 21 0.000000 1.000000 0.000000 -0.574453 -0.503699 0.645206\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# verifying whether the '.laz' extraction was done correctly\n",
        "print(np.max(orch_twnty_pcl_arr[:, 0]), np.min(orch_twnty_pcl_arr[:, 0]))\n",
        "print(np.max(orch_twnty_pcl_arr[:, 1]), np.min(orch_twnty_pcl_arr[:, 1]))\n",
        "print(np.max(orch_twnty_pcl_arr[:, 2]), np.min(orch_twnty_pcl_arr[:, 2]))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "17aozTU8UU6W",
        "outputId": "8d28bfc0-79ac-44f9-d10d-1b9ee62a4ddb"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1.3716284 -1.59532678\n",
            "3.56140113 -0.80763555\n",
            "3.73156381 -0.41045624\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# TODO: update the below paths based on your project setup\n",
        "PFUJI_ANN_PATH = '3-apple_point_clouds/'\n",
        "ann_data_files = [f for f in listdir(PFUJI_ANN_PATH)\n",
        "                    if isfile(join(PFUJI_ANN_PATH, f))]\n",
        "ann_2018_data_files, ann_2020_data_files = [], []\n",
        "ann_2018_data_files.append([f for f in ann_data_files if '2018_01_' in f])\n",
        "ann_2020_data_files.append([f for f in ann_data_files if '2020_01_' in f])\n",
        "ann_2018_data_files, ann_2020_data_files = ann_2018_data_files[0], ann_2020_data_files[0]"
      ],
      "metadata": {
        "id": "bChcjxOyfh6q"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# verifying the length of the created lists in the project\n",
        "print(len(ann_data_files), len(ann_2018_data_files), len(ann_2020_data_files))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pBNJ8AYMOc86",
        "outputId": "801e9ce2-148e-4bb6-ee7b-e838d72498a7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "640 303 312\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# loading all the files into a numpy array and appending those array values to list\n",
        "pcl_apple_data_list_2018, pcl_apple_data_list_2020 = [], []\n",
        "\n",
        "for files in tqdm(ann_2018_data_files):\n",
        "    pcl_temp_arr = np.loadtxt(PFUJI_ANN_PATH + files)\n",
        "    pcl_temp_arr = pcl_temp_arr[:, 0:6]\n",
        "    pcl_temp_arr[:, 3:6] = pcl_temp_arr[:,3:6] / 255\n",
        "    pcl_apple_data_list_2018 = pcl_apple_data_list_2018 + pcl_temp_arr.tolist()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "n_5wkmPwOc_g",
        "outputId": "0a7e6b40-0965-4e8e-bead-c2d0c059515a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 303/303 [01:23<00:00,  3.64it/s]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(len(pcl_apple_data_list_2018))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yOwSYZvrOdCB",
        "outputId": "28d2b709-3bc5-4d95-ecf9-171293bce0d2"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "4779434\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "for files in tqdm(ann_2020_data_files):\n",
        "    pcl_temp_arr = np.loadtxt(PFUJI_ANN_PATH + files)\n",
        "    pcl_temp_arr = pcl_temp_arr[:, 0:6]\n",
        "    pcl_temp_arr[:, 3:6] = pcl_temp_arr[:,3:6] / 255\n",
        "    pcl_apple_data_list_2020 = pcl_apple_data_list_2020 + pcl_temp_arr.tolist()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "A60_2TQcOdE1",
        "outputId": "2421fc47-a37e-4a61-e6ff-4004e0f36889"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 312/312 [00:54<00:00,  5.75it/s]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(len(pcl_apple_data_list_2020))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WevpsJ1Pm02V",
        "outputId": "ce30f3bb-6c68-44e8-f769-a832a8cd9eff"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "3215507\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# convert these lists into corresponding numpy arrays and append\n",
        "appl_eghtn_pcl_arr = np.asarray(pcl_apple_data_list_2018)\n",
        "appl_twnty_pcl_arr = np.asarray(pcl_apple_data_list_2020)\n",
        "print(appl_eghtn_pcl_arr.shape, appl_twnty_pcl_arr.shape)\n",
        "print(orch_eghtn_pcl_arr.shape, orch_twnty_pcl_arr.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Opg2o-Oom1H0",
        "outputId": "7083e2d3-463a-4373-8c0f-57bb5c6527ce"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(4779434, 6) (3215507, 6)\n",
            "(17647208, 6) (18652313, 6)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "appl_eghtn_arr_one = np.ones(4779434)\n",
        "appl_twnty_arr_one = np.ones(3215507)\n",
        "orch_eghtn_arr_zer = np.zeros(17647208)\n",
        "orch_twnty_arr_zer = np.zeros(18652313)"
      ],
      "metadata": {
        "id": "HQsTKQZenf_t"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "appl_eghtn_pcl_arr = np.column_stack((appl_eghtn_pcl_arr, appl_eghtn_arr_one))\n",
        "appl_twnty_pcl_arr = np.column_stack((appl_twnty_pcl_arr, appl_twnty_arr_one))\n",
        "orch_eghtn_pcl_arr = np.column_stack((orch_eghtn_pcl_arr, orch_eghtn_arr_zer))\n",
        "orch_twnty_pcl_arr = np.column_stack((orch_twnty_pcl_arr, orch_twnty_arr_zer))"
      ],
      "metadata": {
        "id": "kgtSZY33gR0Q"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# exploratory data analysis for further improving the segmentation model training\n",
        "# observing the point cloud limit of apple class arrays in the orchard 2018 PCLs\n",
        "print(np.min(appl_eghtn_pcl_arr[:, 0]), np.max(appl_eghtn_pcl_arr[:, 0]),\n",
        "      (np.max(appl_eghtn_pcl_arr[:, 0]) - np.min(appl_eghtn_pcl_arr[:, 0])))\n",
        "\n",
        "print(np.min(appl_eghtn_pcl_arr[:, 1]), np.max(appl_eghtn_pcl_arr[:, 1]),\n",
        "      (np.max(appl_eghtn_pcl_arr[:, 1]) - np.min(appl_eghtn_pcl_arr[:, 1])))\n",
        "\n",
        "print(np.min(appl_eghtn_pcl_arr[:, 2]), np.max(appl_eghtn_pcl_arr[:, 2]),\n",
        "      (np.max(appl_eghtn_pcl_arr[:, 2]) - np.min(appl_eghtn_pcl_arr[:, 2])))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hYpO9J2VuOgU",
        "outputId": "61f92b53-67c7-4d4d-8e12-f39260be121c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "-0.60293728 0.70358139 1.30651867\n",
            "-0.10622169 2.92781162 3.03403331\n",
            "0.18001336 3.22922134 3.0492079800000003\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(np.min(orch_eghtn_pcl_arr[:, 0]), np.max(orch_eghtn_pcl_arr[:, 0]),\n",
        "      (np.max(orch_eghtn_pcl_arr[:, 0]) - np.min(orch_eghtn_pcl_arr[:, 0])))\n",
        "\n",
        "print(np.min(orch_eghtn_pcl_arr[:, 1]), np.max(orch_eghtn_pcl_arr[:, 1]),\n",
        "      (np.max(orch_eghtn_pcl_arr[:, 1]) - np.min(orch_eghtn_pcl_arr[:, 1])))\n",
        "\n",
        "print(np.min(orch_eghtn_pcl_arr[:, 2]), np.max(orch_eghtn_pcl_arr[:, 2]),\n",
        "      (np.max(orch_eghtn_pcl_arr[:, 2]) - np.min(orch_eghtn_pcl_arr[:, 2])))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QaLeq6rguOjH",
        "outputId": "c8eea723-f505-445a-93b2-a14be70a4be6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "-1.98462462 2.47710729 4.46173191\n",
            "-0.36339155 3.74508071 4.10847226\n",
            "-0.41931444 3.98604536 4.4053598\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# reduction of array size of application of deep learning models on this dataset\n",
        "orch_eghtn_pcl_arr = orch_eghtn_pcl_arr[orch_eghtn_pcl_arr[:,0] >= -0.75]\n",
        "orch_eghtn_pcl_arr = orch_eghtn_pcl_arr[orch_eghtn_pcl_arr[:,0] <= 0.75]\n",
        "\n",
        "orch_eghtn_pcl_arr = orch_eghtn_pcl_arr[orch_eghtn_pcl_arr[:,1] >= -0.25]\n",
        "orch_eghtn_pcl_arr = orch_eghtn_pcl_arr[orch_eghtn_pcl_arr[:,1] <= 3.0]\n",
        "\n",
        "orch_eghtn_pcl_arr = orch_eghtn_pcl_arr[orch_eghtn_pcl_arr[:,2] >= 0.0]\n",
        "orch_eghtn_pcl_arr = orch_eghtn_pcl_arr[orch_eghtn_pcl_arr[:,2] <= 3.25]\n",
        "print(orch_eghtn_pcl_arr, orch_eghtn_pcl_arr.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qgtEHh-npK36",
        "outputId": "6c0b7e61-d535-4d45-8e4d-a560398db68d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[0.15063816 2.86242628 0.07597023 ... 0.12598425 0.07058824 0.        ]\n",
            " [0.15288696 2.86359286 0.07708284 ... 0.12992126 0.0627451  0.        ]\n",
            " [0.15185295 2.86449361 0.07853915 ... 0.13385827 0.0627451  0.        ]\n",
            " ...\n",
            " [0.7420758  2.66130161 1.85778499 ... 0.67063492 0.66666667 0.        ]\n",
            " [0.72970802 2.65539908 1.86040342 ... 0.43253968 0.2        0.        ]\n",
            " [0.73756379 2.67185354 1.85862207 ... 0.32539683 0.23529412 0.        ]] (11694528, 7)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# exploratory data analysis for further improving the segmentation model training\n",
        "# observing the point cloud limit of apple class arrays in the orchard 2020 PCLs\n",
        "print(np.min(appl_twnty_pcl_arr[:, 0]), np.max(appl_twnty_pcl_arr[:, 0]),\n",
        "      (np.max(appl_twnty_pcl_arr[:, 0]) - np.min(appl_twnty_pcl_arr[:, 0])))\n",
        "\n",
        "print(np.min(appl_twnty_pcl_arr[:, 1]), np.max(appl_twnty_pcl_arr[:, 1]),\n",
        "      (np.max(appl_twnty_pcl_arr[:, 1]) - np.min(appl_twnty_pcl_arr[:, 1])))\n",
        "\n",
        "print(np.min(appl_twnty_pcl_arr[:, 2]), np.max(appl_twnty_pcl_arr[:, 2]),\n",
        "      (np.max(appl_twnty_pcl_arr[:, 2]) - np.min(appl_twnty_pcl_arr[:, 2])))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Bx1NPBkFpK6l",
        "outputId": "5d14cafd-90a9-408b-9637-d83253100fa1"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "-0.83455122 0.82778805 1.66233927\n",
            "-0.38381717 2.58056998 2.96438715\n",
            "0.04616591 2.94004345 2.89387754\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(np.min(orch_twnty_pcl_arr[:, 0]), np.max(orch_twnty_pcl_arr[:, 0]),\n",
        "      (np.max(orch_twnty_pcl_arr[:, 0]) - np.min(orch_twnty_pcl_arr[:, 0])))\n",
        "\n",
        "print(np.min(orch_twnty_pcl_arr[:, 1]), np.max(orch_twnty_pcl_arr[:, 1]),\n",
        "      (np.max(orch_twnty_pcl_arr[:, 1]) - np.min(orch_twnty_pcl_arr[:, 1])))\n",
        "\n",
        "print(np.min(orch_twnty_pcl_arr[:, 2]), np.max(orch_twnty_pcl_arr[:, 2]),\n",
        "      (np.max(orch_twnty_pcl_arr[:, 2]) - np.min(orch_twnty_pcl_arr[:, 2])))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IgrZw3j6pK9Z",
        "outputId": "c8a575f9-1f72-4e0b-c11d-4c27c791b1c0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "-1.59532678 1.4571104000000001 3.05243718\n",
            "-0.8208921 3.56140113 4.38229323\n",
            "-0.41938722 3.99765754 4.4170447600000005\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# reduction of array size of application of deep learning models on this dataset\n",
        "orch_twnty_pcl_arr = orch_twnty_pcl_arr[orch_twnty_pcl_arr[:,0] >= -1.00]\n",
        "orch_twnty_pcl_arr = orch_twnty_pcl_arr[orch_twnty_pcl_arr[:,0] <= 1.00]\n",
        "\n",
        "orch_twnty_pcl_arr = orch_twnty_pcl_arr[orch_twnty_pcl_arr[:,1] >= -0.50]\n",
        "orch_twnty_pcl_arr = orch_twnty_pcl_arr[orch_twnty_pcl_arr[:,1] <= 2.75]\n",
        "\n",
        "orch_twnty_pcl_arr = orch_twnty_pcl_arr[orch_twnty_pcl_arr[:,2] >= 0.0]\n",
        "orch_twnty_pcl_arr = orch_twnty_pcl_arr[orch_twnty_pcl_arr[:,2] <= 3.0]\n",
        "print(orch_twnty_pcl_arr, orch_twnty_pcl_arr.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yoX-OSW1pLAU",
        "outputId": "d149dff1-2778-48e9-ca4f-912efc4a525f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[-0.40458879  2.00968909  0.01972301 ...  0.16078431  0.14117647\n",
            "   0.        ]\n",
            " [-0.41081741  2.00792456  0.02378946 ...  0.17647059  0.14509804\n",
            "   0.        ]\n",
            " [-0.41779482  2.00558281  0.03036218 ...  0.16078431  0.14901961\n",
            "   0.        ]\n",
            " ...\n",
            " [ 0.58761132  2.698668    2.95487523 ...  0.91372549  0.96862745\n",
            "   0.        ]\n",
            " [ 0.59226578  2.69258618  2.94578671 ...  0.91372549  0.97254902\n",
            "   0.        ]\n",
            " [ 0.59276831  2.69483328  2.95398736 ...  0.91764706  0.96862745\n",
            "   0.        ]] (10997997, 7)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# fraction of apple class present in comparison to the background tree class\n",
        "print(appl_eghtn_pcl_arr.shape[0] / orch_eghtn_pcl_arr.shape[0]) # double upsampling for apple class\n",
        "print(appl_twnty_pcl_arr.shape[0] / orch_twnty_pcl_arr.shape[0]) # triple upsampling for apple class\n",
        "# upsampling crashes the session for such large array, best to create patches and then upsample the pcl dataset"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9w9CVd0HpLED",
        "outputId": "2c29f2a1-a77b-4e84-f96c-6015bff91999"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.4086897735419506\n",
            "0.29237205647537456\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# merging apple and tree class for 2018 orchard pcl data\n",
        "comp_eghtn_pcl_arr = np.concatenate((appl_eghtn_pcl_arr,\n",
        "                                     orch_eghtn_pcl_arr), axis=0)\n",
        "\n",
        "print(comp_eghtn_pcl_arr, comp_eghtn_pcl_arr.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8uENwe-DpLJp",
        "outputId": "319d11de-3877-4333-e554-0dc28b74e0ec"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[-0.28477782  0.83724743  1.37119353 ...  0.38823529  0.2\n",
            "   1.        ]\n",
            " [-0.28477782  0.83724743  1.37119353 ...  0.38823529  0.2\n",
            "   1.        ]\n",
            " [-0.28534016  0.83995342  1.35774648 ...  0.45882353  0.23137255\n",
            "   1.        ]\n",
            " ...\n",
            " [ 0.7420758   2.66130161  1.85778499 ...  0.67063492  0.66666667\n",
            "   0.        ]\n",
            " [ 0.72970802  2.65539908  1.86040342 ...  0.43253968  0.2\n",
            "   0.        ]\n",
            " [ 0.73756379  2.67185354  1.85862207 ...  0.32539683  0.23529412\n",
            "   0.        ]] (16473962, 7)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# cleaning out the memory with deletion of extra arrays\n",
        "del appl_eghtn_pcl_arr\n",
        "del orch_eghtn_pcl_arr\n",
        "del orch_eghtn_pcl_arr_west\n",
        "del orch_eghtn_pcl_arr_east"
      ],
      "metadata": {
        "id": "By4qz49FpLL2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# save the numpy array in current working directory\n",
        "# TODO: update the below paths based on your project setup\n",
        "np.save('pfuji-size-pcl-processed-2018.npy', comp_eghtn_pcl_arr)"
      ],
      "metadata": {
        "id": "GyJx4aDKMb9d"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# merging apple and tree class for 2020 orchard pcl data\n",
        "comp_twnty_pcl_arr = np.concatenate((appl_twnty_pcl_arr,\n",
        "                                     orch_twnty_pcl_arr), axis=0)\n",
        "\n",
        "print(comp_twnty_pcl_arr, comp_twnty_pcl_arr.shape)"
      ],
      "metadata": {
        "id": "lNjcqxEZO53N"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# cleaning out the memory with deletion of extra arrays\n",
        "del appl_twnty_pcl_arr\n",
        "del orch_twnty_pcl_arr\n",
        "del orch_twnty_pcl_arr_west\n",
        "del orch_twnty_pcl_arr_east"
      ],
      "metadata": {
        "id": "M0DFhkQ-pLQd"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# save the numpy array in current working directory\n",
        "# TODO: update the below paths based on your project setup\n",
        "np.save('pfuji-size-pcl-processed-2020.npy', comp_twnty_pcl_arr)"
      ],
      "metadata": {
        "id": "LbkhRuTKpLVh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!ls"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pKUOMWFENYDH",
        "outputId": "1aacd897-e88b-4219-e174-89e1ad82a8c3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2-trees_point_clouds  pfuji-size-pcl-processed-2018.npy\n",
            "3-apple_point_clouds  pfuji-size-pcl-processed-2020.npy\n",
            "drive\t\t      sample_data\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# storing a zipped version of the dataset for further processing into a merged dataset creation\n",
        "# TODO: update the below paths based on your project setup\n",
        "!zip -r drive/MyDrive/point-cloud-prototyping/datasets/pfuji-size-pcl-processed-2018.zip pfuji-size-pcl-processed-2018.npy\n",
        "!zip -r drive/MyDrive/point-cloud-prototyping/datasets/pfuji-size-pcl-processed-2020.zip pfuji-size-pcl-processed-2020.npy"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_7VLZxzQpLYT",
        "outputId": "2b3adcfc-c304-4d3c-f422-cd5581b4afb4"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "  adding: pfuji-size-pcl-processed-2018.npy (deflated 52%)\n",
            "  adding: pfuji-size-pcl-processed-2020.npy (deflated 51%)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 2. Fuji-SfM Data Extraction and Base Point Cloud File Preparation"
      ],
      "metadata": {
        "id": "XSC9n8J0Sx1-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# installing open3d library for data upsampling task\n",
        "!pip install open3d"
      ],
      "metadata": {
        "id": "O2G6-U3oU3yQ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# listing all directory and data processing import statements\n",
        "import pickle\n",
        "import numpy as np\n",
        "import open3d as o3d\n",
        "from os import listdir\n",
        "from tqdm import tqdm\n",
        "from os.path import isfile, join"
      ],
      "metadata": {
        "id": "bF5U0QdZQikB"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b5A_cKbtKNJo"
      },
      "outputs": [],
      "source": [
        "# extracting 3D point cloud data for creating base point cloud npy file\n",
        "# TODO: update the below paths based on your project setup\n",
        "!unzip drive/MyDrive/point-cloud-prototyping/datasets/3-3D_data.zip -d ."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RgfMzkdqKNLo",
        "outputId": "0a0eb794-18a5-47c5-f46c-dea2ad76a5fb"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Fuji_apple_trees_point_cloud.txt  GroundTruth-3D_apples_locations  readme.txt\n"
          ]
        }
      ],
      "source": [
        "# structure of the extracted txt dataset file\n",
        "!ls 3-3D_data/"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1eVXPFL5KNN6"
      },
      "outputs": [],
      "source": [
        "# loading txt data file onto the pcl numpy array\n",
        "# TODO: update the below paths based on your project setup\n",
        "pc_arr = np.loadtxt('3-3D_data/Fuji_apple_trees_point_cloud.txt')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jwJA7kLiKNQk",
        "outputId": "ad88d8ae-8522-457d-8af4-d5492b4cffae"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[[ 11.50647926  75.69534302 312.91494751 206.         206.\n",
            "  202.        ]\n",
            " [ 11.52078819  75.62413788 312.97903442 205.         207.\n",
            "  203.        ]\n",
            " [ 11.52173042  75.6230011  312.97857666 144.         151.\n",
            "  137.        ]\n",
            " ...\n",
            " [ 10.69560528  79.81440735 308.80877686 114.          98.\n",
            "   60.        ]\n",
            " [ 10.70045471  79.81336975 308.80587769 107.         115.\n",
            "   64.        ]\n",
            " [ 10.69812202  79.81700134 308.80447388 109.         123.\n",
            "   79.        ]]\n",
            "(10853801, 6)\n"
          ]
        }
      ],
      "source": [
        "# exploratory data analysis for the general shape of the loaded point cloud\n",
        "print(pc_arr)\n",
        "print(pc_arr.shape)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c8coe2ukfr-o"
      },
      "outputs": [],
      "source": [
        "# list containing array coordinate values of the bounding boxes in apple class objects\n",
        "bb_box_coord_lst = [] "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "G_COBjE-gOf-",
        "outputId": "5babd729-702d-4062-e41b-cda3e4fac872"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "test_set  training_set\n"
          ]
        }
      ],
      "source": [
        "# listing training and testing directories for the 3D apple pcl annotations\n",
        "!ls 3-3D_data/GroundTruth-3D_apples_locations"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0yvZM1jXf4p5",
        "outputId": "a89d47df-8a19-4f77-a662-b9332c20e000"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "['Tree01-annotations_000097.txt',\n",
              " 'Tree06-annotations_000017.txt',\n",
              " 'Tree01-annotations_000011.txt',\n",
              " 'Tree05-annotations_000091.txt',\n",
              " 'Tree07-annotations_000075.txt',\n",
              " 'Tree04-annotations_000011.txt',\n",
              " 'Tree07-annotations_000073.txt',\n",
              " 'Tree03-annotations_000014.txt',\n",
              " 'Tree01-annotations_000082.txt',\n",
              " 'Tree04-annotations_000100.txt']"
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# test set apple bounding box coordinate list\n",
        "# TODO: update the below paths based on your project setup\n",
        "bb_coord_path_test = '3-3D_data/GroundTruth-3D_apples_locations/test_set'\n",
        "bb_coord_files_test = [f for f in listdir(bb_coord_path_test) if isfile(join(bb_coord_path_test, f))]\n",
        "bb_coord_files_test[:10]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HgVVZJPsg5Fy",
        "outputId": "83e5e82c-339a-4d35-9272-80d2f8249994"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "['Tree11-annotations_000092.txt',\n",
              " 'Tree09-annotations_000116.txt',\n",
              " 'Tree11-annotations_000031.txt',\n",
              " 'Tree09-annotations_000111.txt',\n",
              " 'Tree11-annotations_000051.txt',\n",
              " 'Tree10-annotations_000055.txt',\n",
              " 'Tree11-annotations_000117.txt',\n",
              " 'Tree10-annotations_000113.txt',\n",
              " 'Tree09-annotations_000046.txt',\n",
              " 'Tree11-annotations_000155.txt']"
            ]
          },
          "execution_count": 6,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# train set apple bounding box coordinate list\n",
        "# TODO: update the below paths based on your project setup\n",
        "bb_coord_path_train = '3-3D_data/GroundTruth-3D_apples_locations/training_set'\n",
        "bb_coord_files_train = [f for f in listdir(bb_coord_path_train) if isfile(join(bb_coord_path_train, f))]\n",
        "bb_coord_files_train[:10]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "59Nt0dj5f4rw"
      },
      "outputs": [],
      "source": [
        "# reader loop for test set bounding box annotation preparation\n",
        "for bb_test_file in bb_coord_files_test:\n",
        "    bb_test_arr = np.loadtxt(bb_coord_path_test+'/'+bb_test_file)\n",
        "    bb_test_arr = bb_test_arr[1:]\n",
        "    bb_coords = [\n",
        "            np.min(bb_test_arr[:,0]), np.max(bb_test_arr[:,0]),\n",
        "            np.min(bb_test_arr[:,1]), np.max(bb_test_arr[:,1]),\n",
        "            np.min(bb_test_arr[:,2]), np.max(bb_test_arr[:,2])\n",
        "        ]\n",
        "    \n",
        "    bb_box_coord_lst.append(bb_coords)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "L-GHCXIJf4tn"
      },
      "outputs": [],
      "source": [
        "# reader loop for train set bounding box annotation  preparation\n",
        "for bb_train_file in bb_coord_files_train:\n",
        "    bb_train_arr = np.loadtxt(bb_coord_path_train+'/'+bb_train_file)\n",
        "    bb_train_arr = bb_train_arr[1:]\n",
        "    bb_coords = [\n",
        "            np.min(bb_test_arr[:,0]), np.max(bb_test_arr[:,0]),\n",
        "            np.min(bb_test_arr[:,1]), np.max(bb_test_arr[:,1]),\n",
        "            np.min(bb_test_arr[:,2]), np.max(bb_test_arr[:,2])\n",
        "        ]\n",
        "    bb_box_coord_lst.append(bb_coords)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hI-XIWWBf4vK"
      },
      "outputs": [],
      "source": [
        "# converting array to list for effective pcl data editing operation\n",
        "pc_list = pc_arr.tolist()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "h9cYDI6r0FNp"
      },
      "outputs": [],
      "source": [
        "# preparation of apple pcl list data\n",
        "apple_pc_list = []"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "J6ewah3g0FPG",
        "outputId": "af72eaa6-9651-4e53-bd16-8e590acd6b8e"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            " 95%|█████████▌| 10352754/10853801 [2:04:52<06:02, 1381.69it/s]\n"
          ]
        }
      ],
      "source": [
        "# iterating over the bounding box coordinate limits of different apple class objects\n",
        "# and also iterating over the complete point cloud to separate out the apple point cloud\n",
        "pc_idx = 0\n",
        "for pc_coord in tqdm(pc_list):\n",
        "    for bb_coord_vals in bb_box_coord_lst:\n",
        "        if ( pc_coord[0] >= bb_coord_vals[0] and pc_coord[0] <= bb_coord_vals[1] ) \\\n",
        "        and ( pc_coord[1] >= bb_coord_vals[2] and pc_coord[1] <= bb_coord_vals[3] ) \\\n",
        "        and ( pc_coord[2] >= bb_coord_vals[4] and pc_coord[2] <= bb_coord_vals[5] ):\n",
        "            apple_pc_list.append(pc_list.pop(pc_idx))\n",
        "            # print(len(apple_pc_list))\n",
        "            break\n",
        "    pc_idx = pc_idx + 1\n",
        "# note: most time-consuming bottleneck step with approximately two hour processing time\n",
        "# limitation note: the apple class objects are annotated with a square bounding box\n",
        "# and might include some noisy tree point cloud also leading to noisy labels"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "d_ImLOOo01N6",
        "outputId": "88cfc9eb-ffa2-4b94-d471-b1f19c4c7b11"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(501047, 10352754, (10853801, 6))"
            ]
          },
          "execution_count": 21,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# listing the resultant point cloud size for the apple and the tree objects\n",
        "len(apple_pc_list), len(pc_list), pc_arr.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pgYjfJyX0FQ4"
      },
      "outputs": [],
      "source": [
        "# appending label to the resultant point cloud rows of apple class\n",
        "apple_pc_list_lbl = []\n",
        "for apple_pc in apple_pc_list:\n",
        "    apple_pc = apple_pc[:6]\n",
        "    apple_pc.append(1.0)\n",
        "    apple_pc_list_lbl.append(apple_pc)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9Ms_PWVY0FTO"
      },
      "outputs": [],
      "source": [
        "# appending label to the resultant point cloud rows of tree class\n",
        "bg_pc_list_lbl = []\n",
        "for bg_pc in pc_list:\n",
        "    bg_pc.append(0.0)\n",
        "    bg_pc_list_lbl.append(bg_pc)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wGoRtOZ8f4z6",
        "outputId": "66f41347-679b-4972-b826-3e5508a4b54d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "10853801 501047 10352754\n"
          ]
        }
      ],
      "source": [
        "# merging two point cloud lists for generating the resultant pcl list\n",
        "pc_total_list = apple_pc_list_lbl + bg_pc_list_lbl\n",
        "print( len(pc_total_list), len(apple_pc_list_lbl), len(bg_pc_list_lbl))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dT5nGv6xfsUt",
        "outputId": "646f8d5c-c2f6-4aec-eb56-12edfb031112"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(10853801, 7)"
            ]
          },
          "execution_count": 41,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# generating the final resultant array from the pcl list\n",
        "pc_arr_new = np.array(pc_total_list)\n",
        "pc_arr_new.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2dhryn0TmSlf",
        "outputId": "e4d04662-3c11-4017-aff2-9d321ef98d39"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[ 11.73235703,  74.63858032, 311.94485474, ...,  68.        ,\n",
              "         15.        ,   1.        ],\n",
              "       [ 11.73261738,  74.63868713, 311.94223022, ...,  59.        ,\n",
              "         19.        ,   1.        ],\n",
              "       [ 11.73064613,  74.6374588 , 311.94668579, ...,  82.        ,\n",
              "         16.        ,   1.        ],\n",
              "       ...,\n",
              "       [ 10.69560528,  79.81440735, 308.80877686, ...,  98.        ,\n",
              "         60.        ,   0.        ],\n",
              "       [ 10.70045471,  79.81336975, 308.80587769, ..., 115.        ,\n",
              "         64.        ,   0.        ],\n",
              "       [ 10.69812202,  79.81700134, 308.80447388, ..., 123.        ,\n",
              "         79.        ,   0.        ]])"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ],
      "source": [
        "# basic exploratory analysis to check the value\n",
        "pc_arr_new"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2xw852jqm2ff"
      },
      "outputs": [],
      "source": [
        "import pickle\n",
        "with open('drive/MyDrive/faro-prototyping/datasets/fuji_apple_data_preprocessed.pkl','wb') as f:\n",
        "    pickle.dump(pc_arr_new, f)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6N6TDRDBbaiy",
        "outputId": "669d7e48-8256-4d32-e643-107eaf1caab1"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(10853801, 7)\n"
          ]
        }
      ],
      "source": [
        "with open('drive/MyDrive/point-cloud-prototyping/datasets/fuji_apple_data_preprocessed.pkl','rb') as f:\n",
        "     pc_arr_new = pickle.load(f)\n",
        "     print(pc_arr_new.shape)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Lts3e6tTlJ0q",
        "outputId": "115afd58-0a97-45af-b3be-4f645848e353"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "64.82430267 79.90513611 15.080833440000006\n",
            "10.40850639 13.90547562 3.4969692300000013\n",
            "307.55532837 313.66433716 6.1090087900000185\n"
          ]
        }
      ],
      "source": [
        "# y-axis coordinate limits\n",
        "print(np.min(pc_arr_new[:,1]), np.max(pc_arr_new[:,1]), (np.max(pc_arr_new[:,1]) - np.min(pc_arr_new[:,1])) )\n",
        "# x-axis coordinate limits\n",
        "print(np.min(pc_arr_new[:,0]), np.max(pc_arr_new[:,0]), (np.max(pc_arr_new[:,0]) - np.min(pc_arr_new[:,0])) )\n",
        "# z-axis coordinate limits\n",
        "print(np.min(pc_arr_new[:,2]), np.max(pc_arr_new[:,2]), (np.max(pc_arr_new[:,2]) - np.min(pc_arr_new[:,2])) )"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# checking for the subset PC data with apple class label\n",
        "# y-axis coordinate limits\n",
        "print(np.min(pc_arr_new[pc_arr_new[:,6] == 1][:,1]), np.max(pc_arr_new[pc_arr_new[:,6] == 1][:,1]), (np.max(pc_arr_new[pc_arr_new[:,6] == 1][:,1]) - np.min(pc_arr_new[pc_arr_new[:,6] == 1][:,1])) )\n",
        "# x-axis coordinate limits\n",
        "print(np.min(pc_arr_new[pc_arr_new[:,6] == 1][:,0]), np.max(pc_arr_new[pc_arr_new[:,6] == 1][:,0]), (np.max(pc_arr_new[pc_arr_new[:,6] == 1][:,0]) - np.min(pc_arr_new[pc_arr_new[:,6] == 1][:,0])) )\n",
        "# z-axis coordinate limits\n",
        "print(np.min(pc_arr_new[pc_arr_new[:,6] == 1][:,2]), np.max(pc_arr_new[pc_arr_new[:,6] == 1][:,2]), (np.max(pc_arr_new[pc_arr_new[:,6] == 1][:,2]) - np.min(pc_arr_new[pc_arr_new[:,6] == 1][:,2])) )"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xb8-hRNiU946",
        "outputId": "760b448d-4bb3-4683-9ccc-87fdb7a2e683"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "67.28730011 74.84210968 7.554809569999989\n",
            "11.45102787 12.98692036 1.5358924899999984\n",
            "308.83605957 312.26205444 3.425994870000011\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# reduction of array size of application of deep learning models on this dataset\n",
        "pc_temp = pc_arr_new[pc_arr_new[:,0] >= 11.40]\n",
        "pc_temp = pc_temp[pc_temp[:,0] <= 13.15]\n",
        "\n",
        "pc_temp = pc_temp[pc_temp[:,1] >= 67.25]\n",
        "pc_temp = pc_temp[pc_temp[:,1] <= 75.00]\n",
        "\n",
        "pc_temp = pc_temp[pc_temp[:,2] >= 308.75]\n",
        "pc_temp = pc_temp[pc_temp[:,2] <= 312.50]"
      ],
      "metadata": {
        "id": "lmXR6BoPWNZM"
      },
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# y-axis coordinate limits\n",
        "print(np.min(pc_temp[:,1]), np.max(pc_temp[:,1]), (np.max(pc_temp[:,1]) - np.min(pc_temp[:,1])) )\n",
        "# x-axis coordinate limits\n",
        "print(np.min(pc_temp[:,0]), np.max(pc_temp[:,0]), (np.max(pc_temp[:,0]) - np.min(pc_temp[:,0])) )\n",
        "# z-axis coordinate limits\n",
        "print(np.min(pc_temp[:,2]), np.max(pc_temp[:,2]), (np.max(pc_temp[:,2]) - np.min(pc_temp[:,2])) )"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qEtfpqEZZs7q",
        "outputId": "29180627-59d3-47bc-c3e4-da00a59aeaf0"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "67.25000763 75.0 7.749992370000001\n",
            "11.4000082 13.14999771 1.749989509999999\n",
            "308.75 312.49996948 3.7499694800000043\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# checking the current shape of the reduced pcl dataset\n",
        "print(pc_temp.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kxEgrQ50cS0z",
        "outputId": "e87f7f56-4b56-491f-833d-27dfc3b2e25a"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(5149657, 7)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# procedural logic for the upsampling function\n",
        "\n",
        "# separate out apple class sub array with rgb values\n",
        "# build a KDTree of the separated out sub array\n",
        "# loop over upsampling_factor number of points\n",
        "# then random point selection based on index\n",
        "# post that, query the nearest neighbor to that random point\n",
        "# preserve rgb information, separate array for actual upsampling operation\n",
        "\n",
        "# also, update the whole PC to create new upsampling temporary PC\n",
        "# after certain number of upsampled new PC size chunk is created\n",
        "# for better upsampling consistency.\n",
        "\n",
        "def upsample_minor_class(pcl_arr, upsampling_factor):\n",
        "    assert upsampling_factor >= 1.0 and upsampling_factor <= 5.0\n",
        "    pcl_arr_cls = pcl_arr[pcl_arr[:,6] == 1]\n",
        "    pcl_arr_bg = pcl_arr[pcl_arr[:,6] == 0]\n",
        "    # open3d pcl processing\n",
        "    pcd = o3d.geometry.PointCloud()\n",
        "    pcd.points = o3d.utility.Vector3dVector(pcl_arr_cls[:,0:3])\n",
        "    # building KD-Tree for NN query\n",
        "    pcd_tree = o3d.geometry.KDTreeFlann(pcd)\n",
        "    upsample_count = int((upsampling_factor - 1.0)*pcl_arr_cls.shape[0])\n",
        "    # upsampled points nested list\n",
        "    upsampled_points = []\n",
        "    umsample_limit_flg = 0\n",
        "    for i in tqdm(range(upsample_count)):\n",
        "        rand_ind = random.randint(0,pcl_arr_cls.shape[0]-1) \n",
        "        [k, idx_vals, _] = pcd_tree.search_knn_vector_3d(pcd.points[rand_ind], 4)\n",
        "        idx_vals = idx_vals[1:]\n",
        "        rand_neighbor = random.choice(idx_vals)\n",
        "        # creating new upsampled points\n",
        "        arr_vals = (pcl_arr_cls[rand_ind] + pcl_arr_cls[rand_neighbor]) / 2\n",
        "        upsampled_points.append(list(arr_vals))\n",
        "        if len(upsampled_points) > 0.8 * pcl_arr_cls.shape[0] and umsample_limit_flg == 0:\n",
        "            pcl_arr_cls =  np.concatenate((pcl_arr_cls, np.array(upsampled_points)), axis=0)\n",
        "            pcd = o3d.geometry.PointCloud()\n",
        "            pcd.points = o3d.utility.Vector3dVector(pcl_arr_cls[:,0:3])\n",
        "            # building KD-Tree for NN query\n",
        "            pcd_tree = o3d.geometry.KDTreeFlann(pcd)\n",
        "            umsample_limit_flg = 1\n",
        "            upsampled_points = []\n",
        "\n",
        "    pcl_arr_cls =  np.concatenate((pcl_arr_cls, np.array(upsampled_points)), axis=0)\n",
        "    pcl_arr_final = np.concatenate((pcl_arr_cls, pcl_arr_bg), axis=0)\n",
        "    return pcl_arr_final"
      ],
      "metadata": {
        "id": "N9Fli53BoVJQ"
      },
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "pcl_arr_upsampled = upsample_minor_class(pc_temp, 5)\n",
        "print(pcl_arr_upsampled.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sEAwdJW9VnVD",
        "outputId": "805b12bb-11e8-45b6-e963-ae9c32e621df"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 2004188/2004188 [01:28<00:00, 22550.17it/s]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(7153845, 7)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "id": "QCi77OwyyB7O",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c9d4cc1c-9013-419e-ba88-f00fd2d0bf82"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(501047, 7) (4648610, 7)\n",
            "(2505235, 7) (4648610, 7)\n",
            "0.10778426239241408\n",
            "0.5389213119620704\n"
          ]
        }
      ],
      "source": [
        "# post upsampling operation point cloud size qualitative analysis for fuji-sfm effective dataset\n",
        "print(pc_temp[pc_temp[:, 6] == 1].shape, pc_temp[pc_temp[:, 6] == 0].shape)\n",
        "print(pcl_arr_upsampled[pcl_arr_upsampled[:, 6] == 1].shape, pcl_arr_upsampled[pcl_arr_upsampled[:, 6] == 0].shape)\n",
        "print(pc_temp[pc_temp[:, 6] == 1].shape[0] / pc_temp[pc_temp[:, 6] == 0].shape[0])\n",
        "print(pcl_arr_upsampled[pcl_arr_upsampled[:, 6] == 1].shape[0] / pcl_arr_upsampled[pcl_arr_upsampled[:, 6] == 0].shape[0])"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# saving the numpy array of the processed point cloud\n",
        "# TODO: update the below paths based on your project setup\n",
        "np.save('fuji-sfm-pcl-processed.npy', pcl_arr_upsampled)"
      ],
      "metadata": {
        "id": "C6Ag3OcfV_Ru"
      },
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "id": "8Mns-G9zjZ1S",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e363edd4-571a-43c1-de4d-473f224dc08c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "  adding: fuji-sfm-pcl-processed.npy (deflated 60%)\n"
          ]
        }
      ],
      "source": [
        "# moving the processed point cloud into \n",
        "# TODO: update the below paths based on your project setup\n",
        "!zip -r drive/MyDrive/point-cloud-prototyping/datasets/fuji-sfm-pcl-processed.zip fuji-sfm-pcl-processed.npy"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [
        "uSPj-ymFxzUe"
      ]
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}