codes/00_tensor.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "📝 **Author:** Amirhossein Heydari - 📧 **Email:** amirhosseinheydari78@gmail.com - 📍 **Linktree:** [linktr.ee/mr_pylin](https://linktr.ee/mr_pylin)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# array_like\n",
    "   - [Python list](https://docs.python.org/3/tutorial/datastructures.html#more-on-lists)\n",
    "      - Used for storing elements of different data types\n",
    "      - Flexible: there is no length & shape limit\n",
    "      - Not optimized for mathematical operations\n",
    "   - [numpy.ndarray](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html)\n",
    "      - Implemented in C\n",
    "      - Used for mathematical operations\n",
    "      - Arrays are homogeneous: they can store elements of the same data type\n",
    "   - [troch.Tensor](https://pytorch.org/docs/stable/tensors.html)\n",
    "      - PyTorch's core functionality is implemented in C++\n",
    "      - Optimized for deep learning operations e.g. auto gradient\n",
    "      - Support GPU acceleration [NVIDIA GPUs]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scalar_1: 2 | ndim: 0 | dtype: <class 'int'>\n",
      "scalar_2: 2 | ndim: 0 | dtype: numpy.int32\n",
      "scalar_3: 2 | ndim: 0 | dtype: torch.int64\n"
     ]
    }
   ],
   "source": [
    "# scalar : 0-dimensional array/tensor\n",
    "scalar_1 = 2\n",
    "scalar_2 = np.array(2)\n",
    "scalar_3 = torch.tensor(2)\n",
    "\n",
    "# log\n",
    "print(f\"scalar_1: {scalar_1} | ndim: 0 | dtype: {type(scalar_1)}\")\n",
    "print(f\"scalar_2: {scalar_2} | ndim: {scalar_2.ndim} | dtype: numpy.{scalar_2.dtype}\")\n",
    "print(f\"scalar_3: {scalar_3} | ndim: {scalar_3.ndim} | dtype: {scalar_3.dtype}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vector_1: [1, 2, 3]         | ndim: 1 | dtype: <class 'int'>\n",
      "vector_2: [1 2 3]           | ndim: 1 | dtype: numpy.int32\n",
      "vector_3: tensor([1, 2, 3]) | ndim: 1 | dtype: torch.int64\n"
     ]
    }
   ],
   "source": [
    "# vector : 1-dimensional list/array/tensor\n",
    "vector_1 = [1, 2, 3]\n",
    "vector_2 = np.array(vector_1)\n",
    "vector_3 = torch.tensor(vector_1)\n",
    "\n",
    "# log\n",
    "print(f\"vector_1: {str(vector_1):<17} | ndim: 1 | dtype: {type(vector_1[0])}\")\n",
    "print(f\"vector_2: {str(vector_2):<17} | ndim: {vector_2.ndim} | dtype: numpy.{vector_2.dtype}\")\n",
    "print(f\"vector_3: {vector_3} | ndim: {vector_3.ndim} | dtype: {vector_3.dtype}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "matrix_1:\n",
      "[[0, 1], [2, 3]]\n",
      "ndim : 2\n",
      "dtype: <class 'int'>\n",
      "--------------------------------------------------\n",
      "matrix_2:\n",
      "[[0 1]\n",
      " [2 3]]\n",
      "matrix_2.ndim : 2\n",
      "matrix_2.shape: (2, 2)\n",
      "matrix_2.dtype: numpy.int32\n",
      "--------------------------------------------------\n",
      "matrix_3:\n",
      "tensor([[0, 1],\n",
      "        [2, 3]])\n",
      "matrix_3.ndim : 2\n",
      "matrix_3.shape: torch.Size([2, 2])\n",
      "matrix_3.dtype: torch.int64\n"
     ]
    }
   ],
   "source": [
    "# matrix : 2-dimensional list/array/tensor\n",
    "matrix_1 = [[0, 1], [2, 3]]\n",
    "matrix_2 = np.array(matrix_1)\n",
    "matrix_3 = torch.tensor(matrix_1)\n",
    "\n",
    "# log\n",
    "print(f\"matrix_1:\\n{matrix_1}\\nndim : 2\\ndtype: {type(matrix_1[0][0])}\")\n",
    "print('-' * 50)\n",
    "print(f\"matrix_2:\\n{matrix_2}\\nmatrix_2.ndim : {matrix_2.ndim}\\nmatrix_2.shape: {matrix_2.shape}\\nmatrix_2.dtype: numpy.{matrix_2.dtype}\")\n",
    "print('-' * 50)\n",
    "print(f\"matrix_3:\\n{matrix_3}\\nmatrix_3.ndim : {matrix_3.ndim}\\nmatrix_3.shape: {matrix_3.shape}\\nmatrix_3.dtype: {matrix_3.dtype}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lst:\n",
      "[[[0, 1], [2, 3]], [[4, 5], [6, 7]]]\n",
      "ndim : 3\n",
      "dtype: <class 'int'>\n",
      "--------------------------------------------------\n",
      "arr:\n",
      "[[[0 1]\n",
      "  [2 3]]\n",
      "\n",
      " [[4 5]\n",
      "  [6 7]]]\n",
      "arr.ndim : 3\n",
      "arr.shape: (2, 2, 2)\n",
      "arr.dtype: numpy.int32\n",
      "--------------------------------------------------\n",
      "tsr:\n",
      "tensor([[[0, 1],\n",
      "         [2, 3]],\n",
      "\n",
      "        [[4, 5],\n",
      "         [6, 7]]])\n",
      "tsr.ndim : 3\n",
      "tsr.shape: torch.Size([2, 2, 2])\n",
      "tsr.dtype: torch.int64\n"
     ]
    }
   ],
   "source": [
    "# 3-dimensional list/array/tensor\n",
    "list_3d_1 = [[[0, 1], [2, 3]], [[4, 5], [6, 7]]]\n",
    "array_3d_1 = np.array(list_3d_1)\n",
    "tensor_3d_1 = torch.tensor(list_3d_1)\n",
    "\n",
    "# log\n",
    "print(f\"lst:\\n{list_3d_1}\\nndim : 3\\ndtype: {type(list_3d_1[0][0][0])}\")\n",
    "print('-' * 50)\n",
    "print(f\"arr:\\n{array_3d_1}\\narr.ndim : {array_3d_1.ndim}\\narr.shape: {array_3d_1.shape}\\narr.dtype: numpy.{array_3d_1.dtype}\")\n",
    "print('-' * 50)\n",
    "print(f\"tsr:\\n{tensor_3d_1}\\ntsr.ndim : {tensor_3d_1.ndim}\\ntsr.shape: {tensor_3d_1.shape}\\ntsr.dtype: {tensor_3d_1.dtype}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tensors\n",
    "   - [pytorch.org/docs/stable/torch.html#creation-ops](https://pytorch.org/docs/stable/torch.html#creation-ops)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ones_1:\n",
      "1.0\n",
      "ones_1.size() : torch.Size([])\n",
      "ones_1.ndim   : 0\n",
      "ones_1.dtype  : torch.float32\n",
      "type(ones_1)  : <class 'torch.Tensor'>\n",
      "--------------------------------------------------\n",
      "ones_2:\n",
      "tensor([[1., 1.],\n",
      "        [1., 1.]])\n",
      "ones_2.size() : torch.Size([2, 2])\n",
      "ones_2.ndim   : 2\n",
      "ones_2.dtype  : torch.float32\n",
      "type(ones_2)  : <class 'torch.Tensor'>\n",
      "--------------------------------------------------\n",
      "zeros_1:\n",
      "tensor([0., 0.])\n",
      "zeros_1.size() : torch.Size([2])\n",
      "zeros_1.ndim   : 1\n",
      "zeros_1.dtype  : torch.float32\n",
      "type(zeros_1)  : <class 'torch.Tensor'>\n",
      "--------------------------------------------------\n",
      "zeros_2:\n",
      "tensor([0, 0, 0], dtype=torch.int16)\n",
      "zeros_2.size() : torch.Size([3])\n",
      "zeros_2.ndim   : 1\n",
      "zeros_2.dtype  : torch.int16\n",
      "type(zeros_2)  : <class 'torch.Tensor'>\n",
      "--------------------------------------------------\n",
      "full_1:\n",
      "tensor([3, 3, 3], dtype=torch.int16)\n",
      "full_1.size() : torch.Size([3])\n",
      "full_1.ndim   : 1\n",
      "full_1.dtype  : torch.int16\n",
      "type(full_1)  : <class 'torch.Tensor'>\n",
      "--------------------------------------------------\n",
      "empty_1:\n",
      "tensor([[-1.7483e-18,  1.0258e-42,  0.0000e+00],\n",
      "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00]])\n",
      "empty_1.size() : torch.Size([2, 3])\n",
      "empty_1.ndim   : 2\n",
      "empty_1.dtype  : torch.float32\n",
      "type(empty_1)  : <class 'torch.Tensor'>\n",
      "--------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# ones\n",
    "ones_1 = torch.ones(size=())\n",
    "ones_2 = torch.ones(size=(2, 2))\n",
    "\n",
    "# zeros\n",
    "zeros_1 = torch.zeros(size=(2,))\n",
    "zeros_2 = torch.zeros(size=(3,), dtype=torch.int16)\n",
    "\n",
    "# full\n",
    "full_1 = torch.full(size=(3,), fill_value=3, dtype=torch.int16)\n",
    "\n",
    "# empty\n",
    "empty_1 = torch.empty(size=(2, 3))\n",
    "\n",
    "# log\n",
    "for variable in ['ones_1', 'ones_2', 'zeros_1', 'zeros_2', 'full_1', 'empty_1']:\n",
    "    print(f\"{variable}:\\n{eval(variable)}\")\n",
    "    print(f\"{variable}.size() : {eval(variable).size()}\")\n",
    "    print(f\"{variable}.ndim   : {eval(variable).ndim}\")\n",
    "    print(f\"{variable}.dtype  : {eval(variable).dtype}\")\n",
    "    print(f\"type({variable})  : {type(eval(variable))}\")\n",
    "    print('-' * 50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Index\n",
    "   - [pytorch.org/cppdocs/notes/tensor_indexing.html](https://pytorch.org/cppdocs/notes/tensor_indexing.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "index_1: tensor([1, 2, 3, 4])\n",
      "index_2: tensor([5, 6, 7, 8])\n",
      "index_3: tensor([ 9, 10, 11, 12])\n",
      "index_4: 1\n",
      "index_5: 11\n"
     ]
    }
   ],
   "source": [
    "tensor_2d_1 = torch.tensor([[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]])\n",
    "\n",
    "# index\n",
    "index_1 = tensor_2d_1[0]\n",
    "index_2 = tensor_2d_1[1]\n",
    "index_3 = tensor_2d_1[-1]\n",
    "index_4 = tensor_2d_1[0, 0]\n",
    "index_5 = tensor_2d_1[2, -2]\n",
    "\n",
    "# log\n",
    "print(f\"index_1: {index_1}\")\n",
    "print(f\"index_2: {index_2}\")\n",
    "print(f\"index_3: {index_3}\")\n",
    "print(f\"index_4: {index_4}\")\n",
    "print(f\"index_5: {index_5}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Slice\n",
    "   - [pytorch.org/cppdocs/notes/tensor_indexing.html](https://pytorch.org/cppdocs/notes/tensor_indexing.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "slice_1:\n",
      "tensor([0, 1, 2, 3])\n",
      "\n",
      "slice_2:\n",
      "tensor([1, 5, 9])\n",
      "\n",
      "slice_3:\n",
      "tensor([[2, 3],\n",
      "        [6, 7]])\n",
      "\n",
      "slice_4:\n",
      "tensor([8])\n"
     ]
    }
   ],
   "source": [
    "tensor_2d_2 = torch.arange(12).reshape((3, 4))\n",
    "\n",
    "# slice\n",
    "slice_1 = tensor_2d_2[0, :]    # same as tensor_2d_2[0]\n",
    "slice_2 = tensor_2d_2[:, 1]\n",
    "slice_3 = tensor_2d_2[:2, 2:]\n",
    "slice_4 = tensor_2d_2[-1:, 0]\n",
    "\n",
    "# log\n",
    "print(f\"slice_1:\\n{slice_1}\\n\")\n",
    "print(f\"slice_2:\\n{slice_2}\\n\")\n",
    "print(f\"slice_3:\\n{slice_3}\\n\")\n",
    "print(f\"slice_4:\\n{slice_4}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Arithmetic Element-wise Operations\n",
    "   - [pytorch.org/docs/stable/torch.html#pointwise-ops](https://pytorch.org/docs/stable/torch.html#pointwise-ops)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor_2d_3:\n",
      "tensor([[0, 1],\n",
      "        [2, 3]])\n",
      "\n",
      "tensor_2d_4:\n",
      "tensor([[2, 2],\n",
      "        [2, 2]])\n",
      "--------------------------------------------------\n",
      "arithmetic_1:\n",
      "tensor([[2, 3],\n",
      "        [4, 5]])\n",
      "\n",
      "arithmetic_2:\n",
      "tensor([[-2, -1],\n",
      "        [ 0,  1]])\n",
      "\n",
      "arithmetic_3:\n",
      "tensor([[0, 2],\n",
      "        [4, 6]])\n",
      "\n",
      "arithmetic_4:\n",
      "tensor([[0.0000, 0.5000],\n",
      "        [1.0000, 1.5000]])\n",
      "\n",
      "arithmetic_5:\n",
      "tensor([[0, 0],\n",
      "        [1, 1]])\n",
      "\n",
      "arithmetic_6:\n",
      "tensor([[0, 1],\n",
      "        [0, 1]])\n",
      "\n",
      "arithmetic_7:\n",
      "tensor([[0, 1],\n",
      "        [4, 9]])\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tensor_2d_3 = torch.arange(4).reshape(2, 2)\n",
    "tensor_2d_4 = torch.full(size=(2, 2), fill_value=2, dtype=torch.int64)\n",
    "\n",
    "# arithmetic operations\n",
    "arithmetic_1 = tensor_2d_3 + tensor_2d_4   # torch.add\n",
    "arithmetic_2 = tensor_2d_3 - tensor_2d_4   # torch.sub\n",
    "arithmetic_3 = tensor_2d_3 * tensor_2d_4   # torch.multiply\n",
    "arithmetic_4 = tensor_2d_3 / tensor_2d_4   # torch.divide\n",
    "arithmetic_5 = tensor_2d_3 // tensor_2d_4  # torch.floor_divide\n",
    "arithmetic_6 = tensor_2d_3 % tensor_2d_4   # torch.remainder\n",
    "arithmetic_7 = tensor_2d_3 ** tensor_2d_4  # torch.power\n",
    "\n",
    "# log\n",
    "print(f\"tensor_2d_3:\\n{tensor_2d_3}\\n\")\n",
    "print(f\"tensor_2d_4:\\n{tensor_2d_4}\")\n",
    "print('-' * 50)\n",
    "for i in range(7):\n",
    "    print(f\"arithmetic_{i+1}:\\n{eval(f'arithmetic_{i+1}')}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Broadcasting\n",
    "   - [pytorch.org/docs/stable/notes/broadcasting.html](https://pytorch.org/docs/stable/notes/broadcasting.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor_2d_5:\n",
      "tensor([[1, 2],\n",
      "        [3, 4]])\n",
      "\n",
      "tensor_2d_5:\n",
      "tensor([[1],\n",
      "        [2]])\n",
      "--------------------------------------------------\n",
      "broadcasting_1:\n",
      "tensor([[2, 3],\n",
      "        [4, 5]])\n",
      "\n",
      "broadcasting_2:\n",
      "tensor([[2, 3],\n",
      "        [5, 6]])\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tensor_2d_5 = torch.arange(4).reshape(2, 2) + 1\n",
    "tensor_2d_6 = torch.tensor([[1], [2]])\n",
    "\n",
    "# broadcasting\n",
    "broadcasting_1 = tensor_2d_5 + 1\n",
    "broadcasting_2 = tensor_2d_5 + tensor_2d_6\n",
    "\n",
    "# log\n",
    "print(f\"tensor_2d_5:\\n{tensor_2d_5}\\n\")\n",
    "print(f\"tensor_2d_5:\\n{tensor_2d_6}\")\n",
    "print('-' * 50)\n",
    "print(f\"broadcasting_1:\\n{broadcasting_1}\\n\")\n",
    "print(f\"broadcasting_2:\\n{broadcasting_2}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reshape & View\n",
    "   - torch.Tensor.**view**:\n",
    "      - requires the tensor to be contiguous\n",
    "      - less flexible due to the contiguity requirement\n",
    "      - generally faster since it doesn't involve copying data, just changes the metadata\n",
    "      - [pytorch.org/docs/stable/generated/torch.Tensor.reshape.html](https://pytorch.org/docs/stable/generated/torch.Tensor.reshape.html)\n",
    "   - torch.Tensor.**reshape**:\n",
    "      - it can handle non-contiguous tensors by copying data if necessary\n",
    "      - more flexible as it can work with both contiguous and non-contiguous tensors\n",
    "      - might be slower if it needs to copy the data to create a contiguous block\n",
    "      - [pytorch.org/docs/stable/generated/torch.Tensor.view.html](https://pytorch.org/docs/stable/generated/torch.Tensor.view.html)\n",
    "\n",
    "**Note**:\n",
    "   - it is advisable to use `reshape`, which returns a `view` if the shapes are compatible, and copies otherwise.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor_2d_7:\n",
      "tensor([[ 0,  1,  2,  3],\n",
      "        [ 4,  5,  6,  7],\n",
      "        [ 8,  9, 10, 11],\n",
      "        [12, 13, 14, 15]])\n",
      "\n",
      "--------------------------------------------------\n",
      "reshape_1:\n",
      "tensor([[ 0,  1,  2,  3,  4,  5,  6,  7],\n",
      "        [ 8,  9, 10, 11, 12, 13, 14, 15]])\n",
      "reshape_1.shape: torch.Size([2, 8])\n",
      "\n",
      "reshape_2:\n",
      "tensor([[[ 0,  1],\n",
      "         [ 2,  3],\n",
      "         [ 4,  5],\n",
      "         [ 6,  7]],\n",
      "\n",
      "        [[ 8,  9],\n",
      "         [10, 11],\n",
      "         [12, 13],\n",
      "         [14, 15]]])\n",
      "reshape_2.shape: torch.Size([2, 4, 2])\n"
     ]
    }
   ],
   "source": [
    "tensor_2d_7 = torch.arange(16).reshape(4, 4)\n",
    "\n",
    "# reshape\n",
    "reshape_1 = tensor_2d_7.reshape(2, 8)\n",
    "reshape_2 = tensor_2d_7.reshape(2, -1, 2)\n",
    "\n",
    "# log\n",
    "print(f\"tensor_2d_7:\\n{tensor_2d_7}\\n\")\n",
    "print('-' * 50)\n",
    "print(f\"reshape_1:\\n{reshape_1}\")\n",
    "print(f\"reshape_1.shape: {reshape_1.shape}\\n\")\n",
    "print(f\"reshape_2:\\n{reshape_2}\")\n",
    "print(f\"reshape_2.shape: {reshape_2.shape}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor_2d_7:\n",
      "tensor([[100,   1,   2,   3],\n",
      "        [  4,   5,   6,   7],\n",
      "        [  8,   9,  10,  11],\n",
      "        [ 12,  13,  14,  15]])\n",
      "\n",
      "reshape_1:\n",
      "tensor([[100,   1,   2,   3,   4,   5,   6,   7],\n",
      "        [  8,   9,  10,  11,  12,  13,  14,  15]])\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# assignment by index\n",
    "tensor_2d_7[0, 0] = 100\n",
    "\n",
    "# log\n",
    "print(f\"tensor_2d_7:\\n{tensor_2d_7}\\n\")\n",
    "print(f\"reshape_1:\\n{reshape_1}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Copy Tensors\n",
    "   - torch.**clone**: \n",
    "      - creates a hard/deep copy\n",
    "      - This function is differentiable, so gradients will flow back from the result of this operation to `input`\n",
    "   - [pytorch.org/docs/stable/generated/torch.clone.html](https://pytorch.org/docs/stable/generated/torch.clone.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "clone_1: tensor([0., 0., 0., 0., 0.])\n"
     ]
    }
   ],
   "source": [
    "tensor_1d_1 = torch.zeros(size=(5,))\n",
    "\n",
    "# clone\n",
    "clone_1 = tensor_1d_1.clone()\n",
    "\n",
    "# assignment by index\n",
    "tensor_1d_1[0] = 1\n",
    "\n",
    "# log\n",
    "print(f\"clone_1: {clone_1}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## torch.Tensor to numpy.ndarray\n",
    "   - [pytorch.org/docs/stable/generated/torch.Tensor.numpy.html](https://pytorch.org/docs/stable/generated/torch.Tensor.numpy.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor_1d_2        : tensor([0, 2, 6, 3])\n",
      "type(tensor_1d_2)  : <class 'torch.Tensor'>\n",
      "--------------------------------------------------\n",
      "tensor_to_numpy_1        : [0 2 6 3]\n",
      "type(tensor_to_numpy_1)  : <class 'numpy.ndarray'>\n",
      "\n",
      "tensor_to_numpy_2        : [1 2 6 3]\n",
      "type(tensor_to_numpy_2)  : <class 'numpy.ndarray'>\n"
     ]
    }
   ],
   "source": [
    "tensor_1d_2 = torch.tensor([1, 2, 6, 3])\n",
    "\n",
    "# convert + shared memory\n",
    "tensor_to_numpy_1 = tensor_1d_2.numpy()\n",
    "\n",
    "# convert + copy\n",
    "tensor_to_numpy_2 = np.array(tensor_1d_2)\n",
    "\n",
    "# assignment by index\n",
    "tensor_1d_2[0] = 0\n",
    "\n",
    "# log\n",
    "print(f\"tensor_1d_2        : {tensor_1d_2}\")\n",
    "print(f\"type(tensor_1d_2)  : {type(tensor_1d_2)}\")\n",
    "print('-' * 50)\n",
    "print(f\"tensor_to_numpy_1        : {tensor_to_numpy_1}\")\n",
    "print(f\"type(tensor_to_numpy_1)  : {type(tensor_to_numpy_1)}\\n\")\n",
    "print(f\"tensor_to_numpy_2        : {tensor_to_numpy_2}\")\n",
    "print(f\"type(tensor_to_numpy_2)  : {type(tensor_to_numpy_2)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## numpy.ndarray to torch.Tensor\n",
    "   - [pytorch.org/docs/stable/generated/torch.from_numpy.html](https://pytorch.org/docs/stable/generated/torch.from_numpy.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "array_1d_1       : [0 4 2 3]\n",
      "type(array_1d_1) : <class 'numpy.ndarray'>\n",
      "--------------------------------------------------\n",
      "numpy_to_tensor_1       : tensor([0, 4, 2, 3], dtype=torch.int32)\n",
      "type(numpy_to_tensor_1) : <class 'torch.Tensor'>\n",
      "\n",
      "numpy_to_tensor_2       : tensor([1, 4, 2, 3], dtype=torch.int32)\n",
      "type(numpy_to_tensor_2) : <class 'torch.Tensor'>\n"
     ]
    }
   ],
   "source": [
    "array_1d_1 = np.array([1, 4, 2, 3])\n",
    "\n",
    "# convert + shared memory\n",
    "numpy_to_tensor_1 = torch.from_numpy(array_1d_1)\n",
    "\n",
    "# convert + copy\n",
    "numpy_to_tensor_2 = torch.tensor(array_1d_1)\n",
    "\n",
    "# assignment by index\n",
    "array_1d_1[0] = 0\n",
    "\n",
    "# log\n",
    "print(f\"array_1d_1       : {array_1d_1}\")\n",
    "print(f\"type(array_1d_1) : {type(array_1d_1)}\")\n",
    "print('-' * 50)\n",
    "print(f\"numpy_to_tensor_1       : {numpy_to_tensor_1}\")\n",
    "print(f\"type(numpy_to_tensor_1) : {type(numpy_to_tensor_1)}\\n\")\n",
    "print(f\"numpy_to_tensor_2       : {numpy_to_tensor_2}\")\n",
    "print(f\"type(numpy_to_tensor_2) : {type(numpy_to_tensor_2)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## tensor on GPU\n",
    "   - PyTorch relies on the underlying [CUDA](https://developer.nvidia.com/cuda-gpus) and [ROCm](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/3rd-party-support-matrix.html#deep-learning) libraries for GPU support\n",
    "   - [pytorch.org/docs/stable/notes/cuda.html](https://pytorch.org/docs/stable/notes/cuda.html)\n",
    "\n",
    "**Note**:\n",
    "   - Tensors on the `GPU` cannot be directly converted to `np.ndarray` or other structures that do not support GPU operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "device: cuda\n"
     ]
    }
   ],
   "source": [
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')\n",
    "\n",
    "# log\n",
    "print(f\"device: {device}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "num_cuda_devices  : 1\n",
      "cuda 0:\n",
      "\tname                  : NVIDIA GeForce GTX 1650\n",
      "\ttotal_memory          : 4294639616 bytes\n",
      "\tmulti_processor_count : 14\n"
     ]
    }
   ],
   "source": [
    "if torch.cuda.is_available():\n",
    "\n",
    "    # number of cuda devices\n",
    "    num_cuda_devices = torch.cuda.device_count()\n",
    "    print(f\"num_cuda_devices  : {num_cuda_devices}\")\n",
    "\n",
    "    # cuda models\n",
    "    for i in range(num_cuda_devices):\n",
    "        print(f\"cuda {i}:\")\n",
    "        print(f\"\\tname                  : {torch.cuda.get_device_properties(i).name}\")\n",
    "        print(f\"\\ttotal_memory          : {torch.cuda.get_device_properties(i).total_memory} bytes\")\n",
    "        print(f\"\\tmulti_processor_count : {torch.cuda.get_device_properties(i).multi_processor_count}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor_1d_3        : tensor([1., 1., 1., 1., 1.])\n",
      "tensor_1d_3.device : cpu\n",
      "\n",
      "tensor_1d_4        : tensor([1., 1., 1., 1., 1.], device='cuda:0')\n",
      "tensor_1d_4.device : cuda:0\n",
      "\n",
      "tensor_1d_5        : tensor([1., 1., 1., 1., 1.], device='cuda:0')\n",
      "tensor_1d_5.device : cuda:0\n",
      "\n",
      "tensor_1d_6        : tensor([1., 1., 1., 1., 1.], device='cuda:0')\n",
      "tensor_1d_6.device : cuda:0\n"
     ]
    }
   ],
   "source": [
    "tensor_1d_3 = torch.ones(5)\n",
    "tensor_1d_4 = torch.ones(5, device=device)\n",
    "tensor_1d_5 = tensor_1d_3.to(device)\n",
    "tensor_1d_6 = tensor_1d_3.cuda()\n",
    "\n",
    "# log\n",
    "print(f\"tensor_1d_3        : {tensor_1d_3}\")\n",
    "print(f\"tensor_1d_3.device : {tensor_1d_3.device}\\n\")\n",
    "print(f\"tensor_1d_4        : {tensor_1d_4}\")\n",
    "print(f\"tensor_1d_4.device : {tensor_1d_4.device}\\n\")\n",
    "print(f\"tensor_1d_5        : {tensor_1d_5}\")\n",
    "print(f\"tensor_1d_5.device : {tensor_1d_5.device}\\n\")\n",
    "print(f\"tensor_1d_6        : {tensor_1d_6}\")\n",
    "print(f\"tensor_1d_6.device : {tensor_1d_6.device}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.\n"
     ]
    }
   ],
   "source": [
    "tensor_1d_7 = torch.ones(size=(5,), device=device)\n",
    "\n",
    "# torch.Tensor to numpy.ndarray\n",
    "try:\n",
    "    tensor_1d_7.numpy()\n",
    "except TypeError as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor_to_numpy_3       : [1. 1. 1. 1. 1.]\n",
      "type(tensor_to_numpy_3) : <class 'numpy.ndarray'>\n"
     ]
    }
   ],
   "source": [
    "tensor_1d_8 = torch.ones(size=(5,), device=device)\n",
    "\n",
    "# torch.Tensor to numpy.ndarray\n",
    "tensor_to_numpy_3 = tensor_1d_8.cpu().numpy()\n",
    "\n",
    "# log\n",
    "print(f\"tensor_to_numpy_3       : {tensor_to_numpy_3}\")\n",
    "print(f\"type(tensor_to_numpy_3) : {type(tensor_to_numpy_3)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reproducibility\n",
    "   - **seed** : an initial value used to initialize a pseudo-random number generator, ensuring reproducibility of random sequences\n",
    "   - Completely reproducible results are not guaranteed across PyTorch releases, individual commits, or different platforms\n",
    "   - Deterministic operations are often slower than nondeterministic operations\n",
    "   - Determinism may save time in development by facilitating experimentation, debugging, and regression testing\n",
    "   - [pytorch.org/docs/stable/notes/randomness.html](https://pytorch.org/docs/stable/notes/randomness.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.get_rng_state()      : tensor([42,  0,  0,  ...,  0,  0,  0], dtype=torch.uint8)\n",
      "torch.cuda.get_rng_state() : tensor([42,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0],\n",
      "       dtype=torch.uint8)\n"
     ]
    }
   ],
   "source": [
    "# set a seed for both CPU & GPU\n",
    "torch.manual_seed(42)\n",
    "\n",
    "# log\n",
    "print(f\"torch.get_rng_state()      : {torch.get_rng_state()}\")\n",
    "print(f\"torch.cuda.get_rng_state() : {torch.cuda.get_rng_state()}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.get_rng_state()      : tensor([42,  0,  0,  ...,  0,  0,  0], dtype=torch.uint8)\n",
      "torch.cuda.get_rng_state() : tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=torch.uint8)\n"
     ]
    }
   ],
   "source": [
    "# set a seed only for GPU\n",
    "torch.cuda.manual_seed(0)\n",
    "\n",
    "# log\n",
    "print(f\"torch.get_rng_state()      : {torch.get_rng_state()}\")\n",
    "print(f\"torch.cuda.get_rng_state() : {torch.cuda.get_rng_state()}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# forces CuDNN to use deterministic algorithms for convolution operations [default: False]\n",
    "torch.backends.cudnn.deterministic = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# disables CuDNN's benchmarking mode [default: False]\n",
    "torch.backends.cudnn.benchmark = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.are_deterministic_algorithms_enabled(): True\n"
     ]
    }
   ],
   "source": [
    "# enables deterministic algorithms for all PyTorch operations where possible\n",
    "# provides a broader scope of determinism compared to torch.backends.cudnn.deterministic = True\n",
    "# torch.manual_seed() affects random number generation rather than underlying algorithms used for computations\n",
    "# When enabled, operations will use deterministic algorithms when available, and if only nondeterministic algorithms are available they will throw a `RuntimeError` when called\n",
    "# https://pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html#torch.use_deterministic_algorithms\n",
    "torch.use_deterministic_algorithms(True)\n",
    "\n",
    "# check\n",
    "print(f\"torch.are_deterministic_algorithms_enabled(): {torch.are_deterministic_algorithms_enabled()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random Sampling\n",
    "   - [pytorch.org/docs/stable/torch.html#random-sampling](https://pytorch.org/docs/stable/torch.html#random-sampling)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rand_1       : tensor([0.8823, 0.9150, 0.3829, 0.9593, 0.3904])\n",
      "rand_1.dtype : torch.float32\n"
     ]
    }
   ],
   "source": [
    "# a tensor filled with random numbers from a uniform distribution on the interval [0,1)\n",
    "rand_1 = torch.rand(size=(5,))\n",
    "\n",
    "# log\n",
    "print(f\"rand_1       : {rand_1}\")\n",
    "print(f\"rand_1.dtype : {rand_1.dtype}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "normal_1        : tensor([ 0.0326, -0.0868,  0.1523,  0.0665, -0.1032])\n",
      "normal_1.mean() : 0.01226949505507946\n",
      "normal_1.std()  : 0.10736953467130661\n",
      "normal_1.dtype  : torch.float32\n"
     ]
    }
   ],
   "source": [
    "# a tensor of random numbers drawn from separate normal distributions\n",
    "normal_1 = torch.normal(mean=0, std=.1, size=(5,))\n",
    "\n",
    "# log\n",
    "print(f\"normal_1        : {normal_1}\")\n",
    "print(f\"normal_1.mean() : {normal_1.mean()}\")\n",
    "print(f\"normal_1.std()  : {normal_1.std()}\")\n",
    "print(f\"normal_1.dtype  : {normal_1.dtype}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## item\n",
    "   - What you see is not necessarily the actual value!\n",
    "   - [pytorch.org/docs/stable/generated/torch.Tensor.item.html](https://pytorch.org/docs/stable/generated/torch.Tensor.item.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor_1d_9: tensor([0.5739, 0.2666, 0.6274, 0.2696, 0.4414, 0.2969])\n",
      "--------------------------------------------------\n",
      "value_1       : 0.5739044547080994\n",
      "value_1.dtype : torch.float32\n",
      "\n",
      "value_2       : 0.5739044547080994\n",
      "type(value_2) : <class 'float'>\n"
     ]
    }
   ],
   "source": [
    "tensor_1d_9 = torch.rand(size=(6,))\n",
    "\n",
    "# item()\n",
    "value_1 = tensor_1d_9[0]\n",
    "value_2 = tensor_1d_9[0].item()\n",
    "\n",
    "# log\n",
    "print(f\"tensor_1d_9: {tensor_1d_9}\")\n",
    "print('-' * 50)\n",
    "print(f\"value_1       : {value_1}\")\n",
    "print(f\"value_1.dtype : {value_1.dtype}\\n\")\n",
    "print(f\"value_2       : {value_2}\")\n",
    "print(f\"type(value_2) : {type(value_2)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `torch.float32` is preferred rather than `torch.float64`\n",
    "   1. Performance and Speed:\n",
    "      - single-precision operations require less computational effort and memory bandwidth, leading to quicker processing times\n",
    "\n",
    "   1. Memory Usage:\n",
    "      - `torch.float32` requires 32 bits (4 bytes) per value\n",
    "      - `torch.float64` requires 64 bits (8 bytes) per value\n",
    "\n",
    "   1. Adequate Precision:\n",
    "      - for most deep learning tasks, the precision offered by torch.float32 is sufficient\n",
    "\n",
    "   1. Energy Efficiency:\n",
    "      - single-precision arithmetic is generally more energy-efficient than double-precision arithmetic\n",
    "\n",
    "   1. Industry Standards:\n",
    "      - the deep learning community and frameworks predominantly use single-precision floating-point arithmetic\n",
    "\n",
    "   1. Hardware Constraints:\n",
    "      - Many embedded systems and mobile devices have limited computational resources and memory\n",
    "\n",
    "**Notes**:\n",
    "   - `torch.float32` often referred to as `float` or `single-precision`\n",
    "   - `torch.float64` often referred to as `double` or `double-precision`"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}