6_Project1_hoon.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3gsI-o40lmaV"
      },
      "source": [
        "## 정보보호와 시스템보안\n",
        "### Team Project #1\n",
        "\n",
        "### 팀 정보\n",
        "- 팀 이름 : 보안?먹는건가요\n",
        "- 팀 번호 : 6\n",
        "- 팀원 목록 : 성창훈(20181628), 이영균(20181666), 이용준(20181669)\n",
        "\n",
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WIE5yEZzHDTo"
      },
      "source": [
        "### 프로젝트 TODO\n",
        "1. Data Preprocessing\n",
        "2. Data Scaling\n",
        "3. IP Voting\n",
        "\n",
        "### 프로젝트 데이터 셋\n",
        "- 국민대학교 네트워크 플로우 데이터셋(5시간)   \n",
        "\n",
        "### 참고자료\n",
        "- 네트워크 플로우([Network Flow](https://checkmk.com/guides/network-flow-monitoring)):   \n",
        "한번의 세션에서 출발지(Source IP)와 도착지(Destination IP) 사이에 오가는 패킷에 대한 통계적 데이터\n",
        "### 주의사항\n",
        "- 기존에 있는 파일과 폴더의 구조를 변경할 수 없음\n",
        "\n",
        "### 제출\n",
        "- `<팀 번호>_Project1.ipynb`파일을 팀장이 제출   \n",
        "팀 번호 : 구글 스프레드 시트 행 인덱스"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "f70kM6B5HE2G",
        "outputId": "44d76211-6826-4ebe-9b6d-126912150584"
      },
      "outputs": [],
      "source": [
        "# from google.colab import drive\n",
        "\n",
        "# drive_path = '/content/drive'\n",
        "# drive.mount(drive_path)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fAEQ51OHHcAD",
        "outputId": "309cb128-ae66-43a5-f995-145b3dbc214b"
      },
      "outputs": [],
      "source": [
        "# !7z x /content/drive/MyDrive/data/security/2022_security_project1.zip"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cPq6fU9dFsQq",
        "outputId": "25aac19a-6023-4ccc-a95a-dc952f08c9c7"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "/home/user/workspace/security/Security Project1\n"
          ]
        }
      ],
      "source": [
        "%cd \"./Security Project1\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "rY5Adez8HDTz"
      },
      "outputs": [],
      "source": [
        "from Utils import open_pickle\n",
        "import pandas as pd\n",
        "import numpy as np"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EyDy8yAtHDT2"
      },
      "source": [
        "### train, valid 데이터 셋 불러오기\n",
        "- 가상대학을 통해 다운받은 플로우 데이터를 불러온다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "KfaBhPp_HDT3"
      },
      "outputs": [],
      "source": [
        "train_path = r\"./train.csv\"\n",
        "valid_path = r\"./valid.csv\"\n",
        "\n",
        "train_df = pd.read_csv(train_path)\n",
        "valid_df = pd.read_csv(valid_path)\n",
        "test_df = pd.read_csv('./project2_test.csv')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NvNfhIhwHDT3",
        "outputId": "b6feb092-afa7-4ef4-98f9-8a72f77d9073"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "977270 325963\n"
          ]
        }
      ],
      "source": [
        "# 학습, 검증 플로우 데이터 개수 확인\n",
        "print(len(train_df), len(test_df))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "s_3M-0u1HDT4",
        "outputId": "901ba79f-483e-40ea-a384-36d3ea08b4be"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Index(['Src IP', 'Src Port', 'Dst IP', 'Dst Port', 'Protocol', 'Flags',\n",
              "       'Timestamp', 'Flow Duration', 'Total Fwd Packet', 'Total Bwd packets',\n",
              "       'Total Length of Fwd Packet', 'Total Length of Bwd Packet',\n",
              "       'Fwd Packet Length Max', 'Fwd Packet Length Min',\n",
              "       'Fwd Packet Length Std', 'Bwd Packet Length Max',\n",
              "       'Bwd Packet Length Min', 'Bwd Packet Length Std', 'Flow IAT Mean',\n",
              "       'Flow IAT Std', 'Flow IAT Max', 'Flow IAT Min', 'Fwd IAT Total',\n",
              "       'Fwd IAT Mean', 'Fwd IAT Std', 'Fwd IAT Max', 'Fwd IAT Min',\n",
              "       'Bwd IAT Total', 'Bwd IAT Mean', 'Bwd IAT Std', 'Bwd IAT Max',\n",
              "       'Bwd IAT Min', 'Fwd Header Length', 'Bwd Header Length',\n",
              "       'Packet Length Min', 'Packet Length Max', 'Packet Length Std',\n",
              "       'Packet Length Variance', 'Average Packet Size', 'Fwd Segment Size Avg',\n",
              "       'Bwd Segment Size Avg', 'FWD Init Win Bytes', 'Bwd Init Win Bytes',\n",
              "       'Fwd Act Data Pkts', 'Fwd Seg Size Min', 'Active Mean', 'Active Std',\n",
              "       'Active Max', 'Active Min', 'Idle Mean', 'Idle Std', 'Idle Max',\n",
              "       'Idle Min', 'Label'],\n",
              "      dtype='object')"
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# 주어진 플로우 컬럼 종류 확인\n",
        "train_df.keys()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Index(['Src IP', 'Src Port', 'Dst IP', 'Dst Port', 'Protocol', 'Flags',\n",
              "       'Timestamp', 'Flow Duration', 'Total Fwd Packet', 'Total Bwd packets',\n",
              "       'Total Length of Fwd Packet', 'Total Length of Bwd Packet',\n",
              "       'Fwd Packet Length Max', 'Fwd Packet Length Min',\n",
              "       'Fwd Packet Length Std', 'Bwd Packet Length Max',\n",
              "       'Bwd Packet Length Min', 'Bwd Packet Length Std', 'Flow IAT Mean',\n",
              "       'Flow IAT Std', 'Flow IAT Max', 'Flow IAT Min', 'Fwd IAT Total',\n",
              "       'Fwd IAT Mean', 'Fwd IAT Std', 'Fwd IAT Max', 'Fwd IAT Min',\n",
              "       'Bwd IAT Total', 'Bwd IAT Mean', 'Bwd IAT Std', 'Bwd IAT Max',\n",
              "       'Bwd IAT Min', 'Fwd Header Length', 'Bwd Header Length',\n",
              "       'Packet Length Min', 'Packet Length Max', 'Packet Length Std',\n",
              "       'Packet Length Variance', 'Average Packet Size', 'Fwd Segment Size Avg',\n",
              "       'Bwd Segment Size Avg', 'FWD Init Win Bytes', 'Bwd Init Win Bytes',\n",
              "       'Fwd Act Data Pkts', 'Fwd Seg Size Min', 'Active Mean', 'Active Std',\n",
              "       'Active Max', 'Active Min', 'Idle Mean', 'Idle Std', 'Idle Max',\n",
              "       'Idle Min'],\n",
              "      dtype='object')"
            ]
          },
          "execution_count": 33,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "test_df.keys()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CGgOGVVmHDT5",
        "outputId": "a27da546-f37e-40a4-86f4-e6d232e6c1fa"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "count        977270\n",
              "unique            2\n",
              "top       Benign_IP\n",
              "freq         571134\n",
              "Name: Label, dtype: object"
            ]
          },
          "execution_count": 6,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# 데이터 확인\n",
        "train_df['Label'].describe()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {},
      "outputs": [],
      "source": [
        "from collections import Counter\n",
        "\n",
        "train_src_port_counter = Counter(train_df['Src Port'])\n",
        "train_dst_port_counter = Counter(train_df['Dst Port'])\n",
        "train_src_ip_counter = Counter(train_df['Src IP'])\n",
        "train_dst_ip_counter = Counter(train_df['Dst IP'])\n",
        "\n",
        "valid_src_port_counter = Counter(valid_df['Src Port'])\n",
        "valid_dst_port_counter = Counter(valid_df['Dst Port'])\n",
        "valid_src_ip_counter = Counter(valid_df['Src IP'])\n",
        "valid_dst_ip_counter = Counter(valid_df['Dst IP'])\n",
        "\n",
        "test_src_port_counter = Counter(test_df['Src Port'])\n",
        "test_dst_port_counter = Counter(test_df['Dst Port'])\n",
        "test_src_ip_counter = Counter(test_df['Src IP'])\n",
        "test_dst_ip_counter = Counter(test_df['Dst IP'])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {},
      "outputs": [],
      "source": [
        "train_df['Src Port'] = train_df['Src Port'].map(train_src_port_counter)\n",
        "train_df['Dst Port'] = train_df['Dst Port'].map(train_dst_port_counter)\n",
        "train_df['Src IP'] = train_df['Src IP'].map(train_src_ip_counter)\n",
        "train_df['Dst IP'] = train_df['Dst IP'].map(train_dst_ip_counter)\n",
        "\n",
        "\n",
        "valid_df['Src Port'] = valid_df['Src Port'].map(valid_src_port_counter)\n",
        "valid_df['Dst Port'] = valid_df['Dst Port'].map(valid_dst_port_counter)\n",
        "valid_df['Src IP'] = valid_df['Src IP'].map(valid_src_ip_counter)\n",
        "valid_df['Dst IP'] = valid_df['Dst IP'].map(valid_dst_ip_counter)\n",
        "\n",
        "test_df['Src Port'] = test_df['Src Port'].map(test_src_port_counter)\n",
        "test_df['Dst Port'] = test_df['Dst Port'].map(test_dst_port_counter)\n",
        "test_df['Src IP'] = test_df['Src IP'].map(test_src_ip_counter)\n",
        "test_df['Dst IP'] = test_df['Dst IP'].map(test_dst_ip_counter)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 325963 entries, 0 to 325962\n",
            "Data columns (total 53 columns):\n",
            " #   Column                      Non-Null Count   Dtype  \n",
            "---  ------                      --------------   -----  \n",
            " 0   Src IP                      325963 non-null  int64  \n",
            " 1   Src Port                    325963 non-null  int64  \n",
            " 2   Dst IP                      325963 non-null  int64  \n",
            " 3   Dst Port                    325963 non-null  int64  \n",
            " 4   Protocol                    325963 non-null  object \n",
            " 5   Flags                       325963 non-null  object \n",
            " 6   Timestamp                   325963 non-null  object \n",
            " 7   Flow Duration               325963 non-null  int64  \n",
            " 8   Total Fwd Packet            325963 non-null  int64  \n",
            " 9   Total Bwd packets           325963 non-null  int64  \n",
            " 10  Total Length of Fwd Packet  325963 non-null  float64\n",
            " 11  Total Length of Bwd Packet  325963 non-null  float64\n",
            " 12  Fwd Packet Length Max       325963 non-null  float64\n",
            " 13  Fwd Packet Length Min       325963 non-null  float64\n",
            " 14  Fwd Packet Length Std       325963 non-null  float64\n",
            " 15  Bwd Packet Length Max       325963 non-null  float64\n",
            " 16  Bwd Packet Length Min       325963 non-null  float64\n",
            " 17  Bwd Packet Length Std       325963 non-null  float64\n",
            " 18  Flow IAT Mean               325963 non-null  float64\n",
            " 19  Flow IAT Std                325963 non-null  float64\n",
            " 20  Flow IAT Max                325963 non-null  float64\n",
            " 21  Flow IAT Min                325963 non-null  float64\n",
            " 22  Fwd IAT Total               325963 non-null  float64\n",
            " 23  Fwd IAT Mean                325963 non-null  float64\n",
            " 24  Fwd IAT Std                 325963 non-null  float64\n",
            " 25  Fwd IAT Max                 325963 non-null  float64\n",
            " 26  Fwd IAT Min                 325963 non-null  float64\n",
            " 27  Bwd IAT Total               325963 non-null  float64\n",
            " 28  Bwd IAT Mean                325963 non-null  float64\n",
            " 29  Bwd IAT Std                 325963 non-null  float64\n",
            " 30  Bwd IAT Max                 325963 non-null  float64\n",
            " 31  Bwd IAT Min                 325963 non-null  float64\n",
            " 32  Fwd Header Length           325963 non-null  int64  \n",
            " 33  Bwd Header Length           325963 non-null  int64  \n",
            " 34  Packet Length Min           325963 non-null  float64\n",
            " 35  Packet Length Max           325963 non-null  float64\n",
            " 36  Packet Length Std           325963 non-null  float64\n",
            " 37  Packet Length Variance      325963 non-null  float64\n",
            " 38  Average Packet Size         325963 non-null  float64\n",
            " 39  Fwd Segment Size Avg        325963 non-null  float64\n",
            " 40  Bwd Segment Size Avg        325963 non-null  float64\n",
            " 41  FWD Init Win Bytes          325963 non-null  int64  \n",
            " 42  Bwd Init Win Bytes          325963 non-null  int64  \n",
            " 43  Fwd Act Data Pkts           325963 non-null  int64  \n",
            " 44  Fwd Seg Size Min            325963 non-null  int64  \n",
            " 45  Active Mean                 325963 non-null  float64\n",
            " 46  Active Std                  325963 non-null  float64\n",
            " 47  Active Max                  325963 non-null  float64\n",
            " 48  Active Min                  325963 non-null  float64\n",
            " 49  Idle Mean                   325963 non-null  float64\n",
            " 50  Idle Std                    325963 non-null  float64\n",
            " 51  Idle Max                    325963 non-null  float64\n",
            " 52  Idle Min                    325963 non-null  float64\n",
            "dtypes: float64(37), int64(13), object(3)\n",
            "memory usage: 131.8+ MB\n"
          ]
        }
      ],
      "source": [
        "test_df.info()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/html": [
              "<style type=\"text/css\">\n",
              "#T_48d10_row18_col1, #T_48d10_row29_col1, #T_48d10_row44_col1 {\n",
              "  background-color: lightgreen;\n",
              "}\n",
              "</style>\n",
              "<table id=\"T_48d10\">\n",
              "  <thead>\n",
              "    <tr>\n",
              "      <th class=\"blank level0\" >&nbsp;</th>\n",
              "      <th id=\"T_48d10_level0_col0\" class=\"col_heading level0 col0\" >Description</th>\n",
              "      <th id=\"T_48d10_level0_col1\" class=\"col_heading level0 col1\" >Value</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
              "      <td id=\"T_48d10_row0_col0\" class=\"data row0 col0\" >session_id</td>\n",
              "      <td id=\"T_48d10_row0_col1\" class=\"data row0 col1\" >3144</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
              "      <td id=\"T_48d10_row1_col0\" class=\"data row1 col0\" >Target</td>\n",
              "      <td id=\"T_48d10_row1_col1\" class=\"data row1 col1\" >Label</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
              "      <td id=\"T_48d10_row2_col0\" class=\"data row2 col0\" >Target Type</td>\n",
              "      <td id=\"T_48d10_row2_col1\" class=\"data row2 col1\" >Binary</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
              "      <td id=\"T_48d10_row3_col0\" class=\"data row3 col0\" >Label Encoded</td>\n",
              "      <td id=\"T_48d10_row3_col1\" class=\"data row3 col1\" >Attack_IP: 0, Benign_IP: 1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
              "      <td id=\"T_48d10_row4_col0\" class=\"data row4 col0\" >Original Data</td>\n",
              "      <td id=\"T_48d10_row4_col1\" class=\"data row4 col1\" >(977270, 54)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
              "      <td id=\"T_48d10_row5_col0\" class=\"data row5 col0\" >Missing Values</td>\n",
              "      <td id=\"T_48d10_row5_col1\" class=\"data row5 col1\" >False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
              "      <td id=\"T_48d10_row6_col0\" class=\"data row6 col0\" >Numeric Features</td>\n",
              "      <td id=\"T_48d10_row6_col1\" class=\"data row6 col1\" >50</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
              "      <td id=\"T_48d10_row7_col0\" class=\"data row7 col0\" >Categorical Features</td>\n",
              "      <td id=\"T_48d10_row7_col1\" class=\"data row7 col1\" >2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
              "      <td id=\"T_48d10_row8_col0\" class=\"data row8 col0\" >Ordinal Features</td>\n",
              "      <td id=\"T_48d10_row8_col1\" class=\"data row8 col1\" >False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
              "      <td id=\"T_48d10_row9_col0\" class=\"data row9 col0\" >High Cardinality Features</td>\n",
              "      <td id=\"T_48d10_row9_col1\" class=\"data row9 col1\" >False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
              "      <td id=\"T_48d10_row10_col0\" class=\"data row10 col0\" >High Cardinality Method</td>\n",
              "      <td id=\"T_48d10_row10_col1\" class=\"data row10 col1\" >None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
              "      <td id=\"T_48d10_row11_col0\" class=\"data row11 col0\" >Transformed Train Set</td>\n",
              "      <td id=\"T_48d10_row11_col1\" class=\"data row11 col1\" >(977270, 78)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
              "      <td id=\"T_48d10_row12_col0\" class=\"data row12 col0\" >Transformed Test Set</td>\n",
              "      <td id=\"T_48d10_row12_col1\" class=\"data row12 col1\" >(325560, 78)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row13\" class=\"row_heading level0 row13\" >13</th>\n",
              "      <td id=\"T_48d10_row13_col0\" class=\"data row13 col0\" >Shuffle Train-Test</td>\n",
              "      <td id=\"T_48d10_row13_col1\" class=\"data row13 col1\" >True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row14\" class=\"row_heading level0 row14\" >14</th>\n",
              "      <td id=\"T_48d10_row14_col0\" class=\"data row14 col0\" >Stratify Train-Test</td>\n",
              "      <td id=\"T_48d10_row14_col1\" class=\"data row14 col1\" >False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row15\" class=\"row_heading level0 row15\" >15</th>\n",
              "      <td id=\"T_48d10_row15_col0\" class=\"data row15 col0\" >Fold Generator</td>\n",
              "      <td id=\"T_48d10_row15_col1\" class=\"data row15 col1\" >StratifiedKFold</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row16\" class=\"row_heading level0 row16\" >16</th>\n",
              "      <td id=\"T_48d10_row16_col0\" class=\"data row16 col0\" >Fold Number</td>\n",
              "      <td id=\"T_48d10_row16_col1\" class=\"data row16 col1\" >10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row17\" class=\"row_heading level0 row17\" >17</th>\n",
              "      <td id=\"T_48d10_row17_col0\" class=\"data row17 col0\" >CPU Jobs</td>\n",
              "      <td id=\"T_48d10_row17_col1\" class=\"data row17 col1\" >-1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row18\" class=\"row_heading level0 row18\" >18</th>\n",
              "      <td id=\"T_48d10_row18_col0\" class=\"data row18 col0\" >Use GPU</td>\n",
              "      <td id=\"T_48d10_row18_col1\" class=\"data row18 col1\" >True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row19\" class=\"row_heading level0 row19\" >19</th>\n",
              "      <td id=\"T_48d10_row19_col0\" class=\"data row19 col0\" >Log Experiment</td>\n",
              "      <td id=\"T_48d10_row19_col1\" class=\"data row19 col1\" >False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row20\" class=\"row_heading level0 row20\" >20</th>\n",
              "      <td id=\"T_48d10_row20_col0\" class=\"data row20 col0\" >Experiment Name</td>\n",
              "      <td id=\"T_48d10_row20_col1\" class=\"data row20 col1\" >clf-default-name</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row21\" class=\"row_heading level0 row21\" >21</th>\n",
              "      <td id=\"T_48d10_row21_col0\" class=\"data row21 col0\" >USI</td>\n",
              "      <td id=\"T_48d10_row21_col1\" class=\"data row21 col1\" >6937</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row22\" class=\"row_heading level0 row22\" >22</th>\n",
              "      <td id=\"T_48d10_row22_col0\" class=\"data row22 col0\" >Imputation Type</td>\n",
              "      <td id=\"T_48d10_row22_col1\" class=\"data row22 col1\" >simple</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row23\" class=\"row_heading level0 row23\" >23</th>\n",
              "      <td id=\"T_48d10_row23_col0\" class=\"data row23 col0\" >Iterative Imputation Iteration</td>\n",
              "      <td id=\"T_48d10_row23_col1\" class=\"data row23 col1\" >None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row24\" class=\"row_heading level0 row24\" >24</th>\n",
              "      <td id=\"T_48d10_row24_col0\" class=\"data row24 col0\" >Numeric Imputer</td>\n",
              "      <td id=\"T_48d10_row24_col1\" class=\"data row24 col1\" >mean</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row25\" class=\"row_heading level0 row25\" >25</th>\n",
              "      <td id=\"T_48d10_row25_col0\" class=\"data row25 col0\" >Iterative Imputation Numeric Model</td>\n",
              "      <td id=\"T_48d10_row25_col1\" class=\"data row25 col1\" >None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row26\" class=\"row_heading level0 row26\" >26</th>\n",
              "      <td id=\"T_48d10_row26_col0\" class=\"data row26 col0\" >Categorical Imputer</td>\n",
              "      <td id=\"T_48d10_row26_col1\" class=\"data row26 col1\" >constant</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row27\" class=\"row_heading level0 row27\" >27</th>\n",
              "      <td id=\"T_48d10_row27_col0\" class=\"data row27 col0\" >Iterative Imputation Categorical Model</td>\n",
              "      <td id=\"T_48d10_row27_col1\" class=\"data row27 col1\" >None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row28\" class=\"row_heading level0 row28\" >28</th>\n",
              "      <td id=\"T_48d10_row28_col0\" class=\"data row28 col0\" >Unknown Categoricals Handling</td>\n",
              "      <td id=\"T_48d10_row28_col1\" class=\"data row28 col1\" >least_frequent</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row29\" class=\"row_heading level0 row29\" >29</th>\n",
              "      <td id=\"T_48d10_row29_col0\" class=\"data row29 col0\" >Normalize</td>\n",
              "      <td id=\"T_48d10_row29_col1\" class=\"data row29 col1\" >True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row30\" class=\"row_heading level0 row30\" >30</th>\n",
              "      <td id=\"T_48d10_row30_col0\" class=\"data row30 col0\" >Normalize Method</td>\n",
              "      <td id=\"T_48d10_row30_col1\" class=\"data row30 col1\" >minmax</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row31\" class=\"row_heading level0 row31\" >31</th>\n",
              "      <td id=\"T_48d10_row31_col0\" class=\"data row31 col0\" >Transformation</td>\n",
              "      <td id=\"T_48d10_row31_col1\" class=\"data row31 col1\" >False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row32\" class=\"row_heading level0 row32\" >32</th>\n",
              "      <td id=\"T_48d10_row32_col0\" class=\"data row32 col0\" >Transformation Method</td>\n",
              "      <td id=\"T_48d10_row32_col1\" class=\"data row32 col1\" >None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row33\" class=\"row_heading level0 row33\" >33</th>\n",
              "      <td id=\"T_48d10_row33_col0\" class=\"data row33 col0\" >PCA</td>\n",
              "      <td id=\"T_48d10_row33_col1\" class=\"data row33 col1\" >False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row34\" class=\"row_heading level0 row34\" >34</th>\n",
              "      <td id=\"T_48d10_row34_col0\" class=\"data row34 col0\" >PCA Method</td>\n",
              "      <td id=\"T_48d10_row34_col1\" class=\"data row34 col1\" >None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row35\" class=\"row_heading level0 row35\" >35</th>\n",
              "      <td id=\"T_48d10_row35_col0\" class=\"data row35 col0\" >PCA Components</td>\n",
              "      <td id=\"T_48d10_row35_col1\" class=\"data row35 col1\" >None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row36\" class=\"row_heading level0 row36\" >36</th>\n",
              "      <td id=\"T_48d10_row36_col0\" class=\"data row36 col0\" >Ignore Low Variance</td>\n",
              "      <td id=\"T_48d10_row36_col1\" class=\"data row36 col1\" >False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row37\" class=\"row_heading level0 row37\" >37</th>\n",
              "      <td id=\"T_48d10_row37_col0\" class=\"data row37 col0\" >Combine Rare Levels</td>\n",
              "      <td id=\"T_48d10_row37_col1\" class=\"data row37 col1\" >False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row38\" class=\"row_heading level0 row38\" >38</th>\n",
              "      <td id=\"T_48d10_row38_col0\" class=\"data row38 col0\" >Rare Level Threshold</td>\n",
              "      <td id=\"T_48d10_row38_col1\" class=\"data row38 col1\" >None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row39\" class=\"row_heading level0 row39\" >39</th>\n",
              "      <td id=\"T_48d10_row39_col0\" class=\"data row39 col0\" >Numeric Binning</td>\n",
              "      <td id=\"T_48d10_row39_col1\" class=\"data row39 col1\" >False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row40\" class=\"row_heading level0 row40\" >40</th>\n",
              "      <td id=\"T_48d10_row40_col0\" class=\"data row40 col0\" >Remove Outliers</td>\n",
              "      <td id=\"T_48d10_row40_col1\" class=\"data row40 col1\" >False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row41\" class=\"row_heading level0 row41\" >41</th>\n",
              "      <td id=\"T_48d10_row41_col0\" class=\"data row41 col0\" >Outliers Threshold</td>\n",
              "      <td id=\"T_48d10_row41_col1\" class=\"data row41 col1\" >None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row42\" class=\"row_heading level0 row42\" >42</th>\n",
              "      <td id=\"T_48d10_row42_col0\" class=\"data row42 col0\" >Remove Multicollinearity</td>\n",
              "      <td id=\"T_48d10_row42_col1\" class=\"data row42 col1\" >False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row43\" class=\"row_heading level0 row43\" >43</th>\n",
              "      <td id=\"T_48d10_row43_col0\" class=\"data row43 col0\" >Multicollinearity Threshold</td>\n",
              "      <td id=\"T_48d10_row43_col1\" class=\"data row43 col1\" >None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row44\" class=\"row_heading level0 row44\" >44</th>\n",
              "      <td id=\"T_48d10_row44_col0\" class=\"data row44 col0\" >Remove Perfect Collinearity</td>\n",
              "      <td id=\"T_48d10_row44_col1\" class=\"data row44 col1\" >True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row45\" class=\"row_heading level0 row45\" >45</th>\n",
              "      <td id=\"T_48d10_row45_col0\" class=\"data row45 col0\" >Clustering</td>\n",
              "      <td id=\"T_48d10_row45_col1\" class=\"data row45 col1\" >False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row46\" class=\"row_heading level0 row46\" >46</th>\n",
              "      <td id=\"T_48d10_row46_col0\" class=\"data row46 col0\" >Clustering Iteration</td>\n",
              "      <td id=\"T_48d10_row46_col1\" class=\"data row46 col1\" >None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row47\" class=\"row_heading level0 row47\" >47</th>\n",
              "      <td id=\"T_48d10_row47_col0\" class=\"data row47 col0\" >Polynomial Features</td>\n",
              "      <td id=\"T_48d10_row47_col1\" class=\"data row47 col1\" >False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row48\" class=\"row_heading level0 row48\" >48</th>\n",
              "      <td id=\"T_48d10_row48_col0\" class=\"data row48 col0\" >Polynomial Degree</td>\n",
              "      <td id=\"T_48d10_row48_col1\" class=\"data row48 col1\" >None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row49\" class=\"row_heading level0 row49\" >49</th>\n",
              "      <td id=\"T_48d10_row49_col0\" class=\"data row49 col0\" >Trignometry Features</td>\n",
              "      <td id=\"T_48d10_row49_col1\" class=\"data row49 col1\" >False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row50\" class=\"row_heading level0 row50\" >50</th>\n",
              "      <td id=\"T_48d10_row50_col0\" class=\"data row50 col0\" >Polynomial Threshold</td>\n",
              "      <td id=\"T_48d10_row50_col1\" class=\"data row50 col1\" >None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row51\" class=\"row_heading level0 row51\" >51</th>\n",
              "      <td id=\"T_48d10_row51_col0\" class=\"data row51 col0\" >Group Features</td>\n",
              "      <td id=\"T_48d10_row51_col1\" class=\"data row51 col1\" >False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row52\" class=\"row_heading level0 row52\" >52</th>\n",
              "      <td id=\"T_48d10_row52_col0\" class=\"data row52 col0\" >Feature Selection</td>\n",
              "      <td id=\"T_48d10_row52_col1\" class=\"data row52 col1\" >False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row53\" class=\"row_heading level0 row53\" >53</th>\n",
              "      <td id=\"T_48d10_row53_col0\" class=\"data row53 col0\" >Feature Selection Method</td>\n",
              "      <td id=\"T_48d10_row53_col1\" class=\"data row53 col1\" >classic</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row54\" class=\"row_heading level0 row54\" >54</th>\n",
              "      <td id=\"T_48d10_row54_col0\" class=\"data row54 col0\" >Features Selection Threshold</td>\n",
              "      <td id=\"T_48d10_row54_col1\" class=\"data row54 col1\" >None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row55\" class=\"row_heading level0 row55\" >55</th>\n",
              "      <td id=\"T_48d10_row55_col0\" class=\"data row55 col0\" >Feature Interaction</td>\n",
              "      <td id=\"T_48d10_row55_col1\" class=\"data row55 col1\" >False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row56\" class=\"row_heading level0 row56\" >56</th>\n",
              "      <td id=\"T_48d10_row56_col0\" class=\"data row56 col0\" >Feature Ratio</td>\n",
              "      <td id=\"T_48d10_row56_col1\" class=\"data row56 col1\" >False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row57\" class=\"row_heading level0 row57\" >57</th>\n",
              "      <td id=\"T_48d10_row57_col0\" class=\"data row57 col0\" >Interaction Threshold</td>\n",
              "      <td id=\"T_48d10_row57_col1\" class=\"data row57 col1\" >None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row58\" class=\"row_heading level0 row58\" >58</th>\n",
              "      <td id=\"T_48d10_row58_col0\" class=\"data row58 col0\" >Fix Imbalance</td>\n",
              "      <td id=\"T_48d10_row58_col1\" class=\"data row58 col1\" >False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_48d10_level0_row59\" class=\"row_heading level0 row59\" >59</th>\n",
              "      <td id=\"T_48d10_row59_col0\" class=\"data row59 col0\" >Fix Imbalance Method</td>\n",
              "      <td id=\"T_48d10_row59_col1\" class=\"data row59 col1\" >SMOTE</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n"
            ],
            "text/plain": [
              "<pandas.io.formats.style.Styler at 0x7f560585bf10>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "from pycaret.classification import *\n",
        "clf = setup(\n",
        "    data=train_df, test_data=valid_df,\n",
        "    target='Label',\n",
        "    use_gpu=True,\n",
        "    categorical_features=['Flags'],\n",
        "    # high_cardinality_features=['Src Port', 'Dst Port'],\n",
        "    numeric_features=['Fwd Seg Size Min'],\n",
        "    ignore_features=['Timestamp'],\n",
        "    normalize=True,\n",
        "    normalize_method='minmax',\n",
        "    silent=True\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {},
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "[LightGBM] [Fatal] GPU Tree Learner was not enabled in this build.\n",
            "Please recompile with CMake option -DUSE_GPU=1\n",
            "[LightGBM] [Fatal] CUDA Tree Learner was not enabled in this build.\n",
            "Please recompile with CMake option -DUSE_CUDA=1\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Name</th>\n",
              "      <th>GPU Enabled</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>ID</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>lr</th>\n",
              "      <td>Logistic Regression</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>knn</th>\n",
              "      <td>K Neighbors Classifier</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>nb</th>\n",
              "      <td>Naive Bayes</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>dt</th>\n",
              "      <td>Decision Tree Classifier</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>svm</th>\n",
              "      <td>SVM - Linear Kernel</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>rbfsvm</th>\n",
              "      <td>SVM - Radial Kernel</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>gpc</th>\n",
              "      <td>Gaussian Process Classifier</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mlp</th>\n",
              "      <td>MLP Classifier</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>ridge</th>\n",
              "      <td>Ridge Classifier</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>rf</th>\n",
              "      <td>Random Forest Classifier</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>qda</th>\n",
              "      <td>Quadratic Discriminant Analysis</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>ada</th>\n",
              "      <td>Ada Boost Classifier</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>gbc</th>\n",
              "      <td>Gradient Boosting Classifier</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>lda</th>\n",
              "      <td>Linear Discriminant Analysis</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>et</th>\n",
              "      <td>Extra Trees Classifier</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>xgboost</th>\n",
              "      <td>Extreme Gradient Boosting</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>lightgbm</th>\n",
              "      <td>Light Gradient Boosting Machine</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>catboost</th>\n",
              "      <td>CatBoost Classifier</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>dummy</th>\n",
              "      <td>Dummy Classifier</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Bagging</th>\n",
              "      <td>Bagging Classifier</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Stacking</th>\n",
              "      <td>Stacking Classifier</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Voting</th>\n",
              "      <td>Voting Classifier</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>CalibratedCV</th>\n",
              "      <td>Calibrated Classifier CV</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                         Name  GPU Enabled\n",
              "ID                                                        \n",
              "lr                        Logistic Regression         True\n",
              "knn                    K Neighbors Classifier         True\n",
              "nb                                Naive Bayes        False\n",
              "dt                   Decision Tree Classifier        False\n",
              "svm                       SVM - Linear Kernel         True\n",
              "rbfsvm                    SVM - Radial Kernel        False\n",
              "gpc               Gaussian Process Classifier        False\n",
              "mlp                            MLP Classifier        False\n",
              "ridge                        Ridge Classifier         True\n",
              "rf                   Random Forest Classifier         True\n",
              "qda           Quadratic Discriminant Analysis        False\n",
              "ada                      Ada Boost Classifier        False\n",
              "gbc              Gradient Boosting Classifier        False\n",
              "lda              Linear Discriminant Analysis        False\n",
              "et                     Extra Trees Classifier        False\n",
              "xgboost             Extreme Gradient Boosting         True\n",
              "lightgbm      Light Gradient Boosting Machine        False\n",
              "catboost                  CatBoost Classifier         True\n",
              "dummy                        Dummy Classifier        False\n",
              "Bagging                    Bagging Classifier        False\n",
              "Stacking                  Stacking Classifier        False\n",
              "Voting                      Voting Classifier        False\n",
              "CalibratedCV         Calibrated Classifier CV        False"
            ]
          },
          "execution_count": 11,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "models(internal=True)[['Name', 'GPU Enabled']]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/html": [
              "<style type=\"text/css\">\n",
              "#T_85383_row5_col0, #T_85383_row5_col1, #T_85383_row5_col2, #T_85383_row5_col3, #T_85383_row5_col4, #T_85383_row5_col5, #T_85383_row5_col6 {\n",
              "  background: yellow;\n",
              "}\n",
              "</style>\n",
              "<table id=\"T_85383\">\n",
              "  <thead>\n",
              "    <tr>\n",
              "      <th class=\"blank level0\" >&nbsp;</th>\n",
              "      <th id=\"T_85383_level0_col0\" class=\"col_heading level0 col0\" >Accuracy</th>\n",
              "      <th id=\"T_85383_level0_col1\" class=\"col_heading level0 col1\" >AUC</th>\n",
              "      <th id=\"T_85383_level0_col2\" class=\"col_heading level0 col2\" >Recall</th>\n",
              "      <th id=\"T_85383_level0_col3\" class=\"col_heading level0 col3\" >Prec.</th>\n",
              "      <th id=\"T_85383_level0_col4\" class=\"col_heading level0 col4\" >F1</th>\n",
              "      <th id=\"T_85383_level0_col5\" class=\"col_heading level0 col5\" >Kappa</th>\n",
              "      <th id=\"T_85383_level0_col6\" class=\"col_heading level0 col6\" >MCC</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th id=\"T_85383_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
              "      <td id=\"T_85383_row0_col0\" class=\"data row0 col0\" >0.9470</td>\n",
              "      <td id=\"T_85383_row0_col1\" class=\"data row0 col1\" >0.9725</td>\n",
              "      <td id=\"T_85383_row0_col2\" class=\"data row0 col2\" >0.9229</td>\n",
              "      <td id=\"T_85383_row0_col3\" class=\"data row0 col3\" >0.9855</td>\n",
              "      <td id=\"T_85383_row0_col4\" class=\"data row0 col4\" >0.9532</td>\n",
              "      <td id=\"T_85383_row0_col5\" class=\"data row0 col5\" >0.8923</td>\n",
              "      <td id=\"T_85383_row0_col6\" class=\"data row0 col6\" >0.8949</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_85383_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
              "      <td id=\"T_85383_row1_col0\" class=\"data row1 col0\" >0.7381</td>\n",
              "      <td id=\"T_85383_row1_col1\" class=\"data row1 col1\" >0.9314</td>\n",
              "      <td id=\"T_85383_row1_col2\" class=\"data row1 col2\" >0.9076</td>\n",
              "      <td id=\"T_85383_row1_col3\" class=\"data row1 col3\" >0.7184</td>\n",
              "      <td id=\"T_85383_row1_col4\" class=\"data row1 col4\" >0.8020</td>\n",
              "      <td id=\"T_85383_row1_col5\" class=\"data row1 col5\" >0.4304</td>\n",
              "      <td id=\"T_85383_row1_col6\" class=\"data row1 col6\" >0.4568</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_85383_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
              "      <td id=\"T_85383_row2_col0\" class=\"data row2 col0\" >0.7965</td>\n",
              "      <td id=\"T_85383_row2_col1\" class=\"data row2 col1\" >0.8636</td>\n",
              "      <td id=\"T_85383_row2_col2\" class=\"data row2 col2\" >0.6812</td>\n",
              "      <td id=\"T_85383_row2_col3\" class=\"data row2 col3\" >0.9585</td>\n",
              "      <td id=\"T_85383_row2_col4\" class=\"data row2 col4\" >0.7964</td>\n",
              "      <td id=\"T_85383_row2_col5\" class=\"data row2 col5\" >0.6042</td>\n",
              "      <td id=\"T_85383_row2_col6\" class=\"data row2 col6\" >0.6398</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_85383_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
              "      <td id=\"T_85383_row3_col0\" class=\"data row3 col0\" >0.7534</td>\n",
              "      <td id=\"T_85383_row3_col1\" class=\"data row3 col1\" >0.8223</td>\n",
              "      <td id=\"T_85383_row3_col2\" class=\"data row3 col2\" >0.5871</td>\n",
              "      <td id=\"T_85383_row3_col3\" class=\"data row3 col3\" >0.9849</td>\n",
              "      <td id=\"T_85383_row3_col4\" class=\"data row3 col4\" >0.7356</td>\n",
              "      <td id=\"T_85383_row3_col5\" class=\"data row3 col5\" >0.5309</td>\n",
              "      <td id=\"T_85383_row3_col6\" class=\"data row3 col6\" >0.5942</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_85383_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
              "      <td id=\"T_85383_row4_col0\" class=\"data row4 col0\" >0.9578</td>\n",
              "      <td id=\"T_85383_row4_col1\" class=\"data row4 col1\" >0.9900</td>\n",
              "      <td id=\"T_85383_row4_col2\" class=\"data row4 col2\" >0.9428</td>\n",
              "      <td id=\"T_85383_row4_col3\" class=\"data row4 col3\" >0.9844</td>\n",
              "      <td id=\"T_85383_row4_col4\" class=\"data row4 col4\" >0.9631</td>\n",
              "      <td id=\"T_85383_row4_col5\" class=\"data row4 col5\" >0.9139</td>\n",
              "      <td id=\"T_85383_row4_col6\" class=\"data row4 col6\" >0.9150</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_85383_level0_row5\" class=\"row_heading level0 row5\" >Mean</th>\n",
              "      <td id=\"T_85383_row5_col0\" class=\"data row5 col0\" >0.8386</td>\n",
              "      <td id=\"T_85383_row5_col1\" class=\"data row5 col1\" >0.9160</td>\n",
              "      <td id=\"T_85383_row5_col2\" class=\"data row5 col2\" >0.8083</td>\n",
              "      <td id=\"T_85383_row5_col3\" class=\"data row5 col3\" >0.9264</td>\n",
              "      <td id=\"T_85383_row5_col4\" class=\"data row5 col4\" >0.8501</td>\n",
              "      <td id=\"T_85383_row5_col5\" class=\"data row5 col5\" >0.6743</td>\n",
              "      <td id=\"T_85383_row5_col6\" class=\"data row5 col6\" >0.7001</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_85383_level0_row6\" class=\"row_heading level0 row6\" >SD</th>\n",
              "      <td id=\"T_85383_row6_col0\" class=\"data row6 col0\" >0.0950</td>\n",
              "      <td id=\"T_85383_row6_col1\" class=\"data row6 col1\" >0.0639</td>\n",
              "      <td id=\"T_85383_row6_col2\" class=\"data row6 col2\" >0.1457</td>\n",
              "      <td id=\"T_85383_row6_col3\" class=\"data row6 col3\" >0.1045</td>\n",
              "      <td id=\"T_85383_row6_col4\" class=\"data row6 col4\" >0.0913</td>\n",
              "      <td id=\"T_85383_row6_col5\" class=\"data row6 col5\" >0.1949</td>\n",
              "      <td id=\"T_85383_row6_col6\" class=\"data row6 col6\" >0.1779</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n"
            ],
            "text/plain": [
              "<pandas.io.formats.style.Styler at 0x7f55fd57bd60>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "rf = create_model('rf', fold=5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/html": [
              "<style type=\"text/css\">\n",
              "#T_976b3_row10_col0, #T_976b3_row10_col1, #T_976b3_row10_col2, #T_976b3_row10_col3, #T_976b3_row10_col4, #T_976b3_row10_col5, #T_976b3_row10_col6 {\n",
              "  background: yellow;\n",
              "}\n",
              "</style>\n",
              "<table id=\"T_976b3\">\n",
              "  <thead>\n",
              "    <tr>\n",
              "      <th class=\"blank level0\" >&nbsp;</th>\n",
              "      <th id=\"T_976b3_level0_col0\" class=\"col_heading level0 col0\" >Accuracy</th>\n",
              "      <th id=\"T_976b3_level0_col1\" class=\"col_heading level0 col1\" >AUC</th>\n",
              "      <th id=\"T_976b3_level0_col2\" class=\"col_heading level0 col2\" >Recall</th>\n",
              "      <th id=\"T_976b3_level0_col3\" class=\"col_heading level0 col3\" >Prec.</th>\n",
              "      <th id=\"T_976b3_level0_col4\" class=\"col_heading level0 col4\" >F1</th>\n",
              "      <th id=\"T_976b3_level0_col5\" class=\"col_heading level0 col5\" >Kappa</th>\n",
              "      <th id=\"T_976b3_level0_col6\" class=\"col_heading level0 col6\" >MCC</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th id=\"T_976b3_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
              "      <td id=\"T_976b3_row0_col0\" class=\"data row0 col0\" >0.9756</td>\n",
              "      <td id=\"T_976b3_row0_col1\" class=\"data row0 col1\" >0.9989</td>\n",
              "      <td id=\"T_976b3_row0_col2\" class=\"data row0 col2\" >0.9587</td>\n",
              "      <td id=\"T_976b3_row0_col3\" class=\"data row0 col3\" >0.9995</td>\n",
              "      <td id=\"T_976b3_row0_col4\" class=\"data row0 col4\" >0.9787</td>\n",
              "      <td id=\"T_976b3_row0_col5\" class=\"data row0 col5\" >0.9501</td>\n",
              "      <td id=\"T_976b3_row0_col6\" class=\"data row0 col6\" >0.9513</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_976b3_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
              "      <td id=\"T_976b3_row1_col0\" class=\"data row1 col0\" >0.9881</td>\n",
              "      <td id=\"T_976b3_row1_col1\" class=\"data row1 col1\" >0.9999</td>\n",
              "      <td id=\"T_976b3_row1_col2\" class=\"data row1 col2\" >0.9798</td>\n",
              "      <td id=\"T_976b3_row1_col3\" class=\"data row1 col3\" >0.9999</td>\n",
              "      <td id=\"T_976b3_row1_col4\" class=\"data row1 col4\" >0.9897</td>\n",
              "      <td id=\"T_976b3_row1_col5\" class=\"data row1 col5\" >0.9756</td>\n",
              "      <td id=\"T_976b3_row1_col6\" class=\"data row1 col6\" >0.9759</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_976b3_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
              "      <td id=\"T_976b3_row2_col0\" class=\"data row2 col0\" >0.9957</td>\n",
              "      <td id=\"T_976b3_row2_col1\" class=\"data row2 col1\" >1.0000</td>\n",
              "      <td id=\"T_976b3_row2_col2\" class=\"data row2 col2\" >1.0000</td>\n",
              "      <td id=\"T_976b3_row2_col3\" class=\"data row2 col3\" >0.9927</td>\n",
              "      <td id=\"T_976b3_row2_col4\" class=\"data row2 col4\" >0.9964</td>\n",
              "      <td id=\"T_976b3_row2_col5\" class=\"data row2 col5\" >0.9912</td>\n",
              "      <td id=\"T_976b3_row2_col6\" class=\"data row2 col6\" >0.9912</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_976b3_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
              "      <td id=\"T_976b3_row3_col0\" class=\"data row3 col0\" >0.8904</td>\n",
              "      <td id=\"T_976b3_row3_col1\" class=\"data row3 col1\" >0.9755</td>\n",
              "      <td id=\"T_976b3_row3_col2\" class=\"data row3 col2\" >0.9352</td>\n",
              "      <td id=\"T_976b3_row3_col3\" class=\"data row3 col3\" >0.8841</td>\n",
              "      <td id=\"T_976b3_row3_col4\" class=\"data row3 col4\" >0.9089</td>\n",
              "      <td id=\"T_976b3_row3_col5\" class=\"data row3 col5\" >0.7718</td>\n",
              "      <td id=\"T_976b3_row3_col6\" class=\"data row3 col6\" >0.7737</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_976b3_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
              "      <td id=\"T_976b3_row4_col0\" class=\"data row4 col0\" >0.7532</td>\n",
              "      <td id=\"T_976b3_row4_col1\" class=\"data row4 col1\" >0.8170</td>\n",
              "      <td id=\"T_976b3_row4_col2\" class=\"data row4 col2\" >0.5960</td>\n",
              "      <td id=\"T_976b3_row4_col3\" class=\"data row4 col3\" >0.9702</td>\n",
              "      <td id=\"T_976b3_row4_col4\" class=\"data row4 col4\" >0.7384</td>\n",
              "      <td id=\"T_976b3_row4_col5\" class=\"data row4 col5\" >0.5289</td>\n",
              "      <td id=\"T_976b3_row4_col6\" class=\"data row4 col6\" >0.5859</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_976b3_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
              "      <td id=\"T_976b3_row5_col0\" class=\"data row5 col0\" >0.8677</td>\n",
              "      <td id=\"T_976b3_row5_col1\" class=\"data row5 col1\" >0.9755</td>\n",
              "      <td id=\"T_976b3_row5_col2\" class=\"data row5 col2\" >0.9838</td>\n",
              "      <td id=\"T_976b3_row5_col3\" class=\"data row5 col3\" >0.8240</td>\n",
              "      <td id=\"T_976b3_row5_col4\" class=\"data row5 col4\" >0.8968</td>\n",
              "      <td id=\"T_976b3_row5_col5\" class=\"data row5 col5\" >0.7165</td>\n",
              "      <td id=\"T_976b3_row5_col6\" class=\"data row5 col6\" >0.7386</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_976b3_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
              "      <td id=\"T_976b3_row6_col0\" class=\"data row6 col0\" >0.9910</td>\n",
              "      <td id=\"T_976b3_row6_col1\" class=\"data row6 col1\" >0.9994</td>\n",
              "      <td id=\"T_976b3_row6_col2\" class=\"data row6 col2\" >0.9869</td>\n",
              "      <td id=\"T_976b3_row6_col3\" class=\"data row6 col3\" >0.9977</td>\n",
              "      <td id=\"T_976b3_row6_col4\" class=\"data row6 col4\" >0.9922</td>\n",
              "      <td id=\"T_976b3_row6_col5\" class=\"data row6 col5\" >0.9815</td>\n",
              "      <td id=\"T_976b3_row6_col6\" class=\"data row6 col6\" >0.9815</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_976b3_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
              "      <td id=\"T_976b3_row7_col0\" class=\"data row7 col0\" >0.9638</td>\n",
              "      <td id=\"T_976b3_row7_col1\" class=\"data row7 col1\" >0.9959</td>\n",
              "      <td id=\"T_976b3_row7_col2\" class=\"data row7 col2\" >0.9728</td>\n",
              "      <td id=\"T_976b3_row7_col3\" class=\"data row7 col3\" >0.9655</td>\n",
              "      <td id=\"T_976b3_row7_col4\" class=\"data row7 col4\" >0.9691</td>\n",
              "      <td id=\"T_976b3_row7_col5\" class=\"data row7 col5\" >0.9253</td>\n",
              "      <td id=\"T_976b3_row7_col6\" class=\"data row7 col6\" >0.9254</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_976b3_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
              "      <td id=\"T_976b3_row8_col0\" class=\"data row8 col0\" >0.9994</td>\n",
              "      <td id=\"T_976b3_row8_col1\" class=\"data row8 col1\" >1.0000</td>\n",
              "      <td id=\"T_976b3_row8_col2\" class=\"data row8 col2\" >1.0000</td>\n",
              "      <td id=\"T_976b3_row8_col3\" class=\"data row8 col3\" >0.9990</td>\n",
              "      <td id=\"T_976b3_row8_col4\" class=\"data row8 col4\" >0.9995</td>\n",
              "      <td id=\"T_976b3_row8_col5\" class=\"data row8 col5\" >0.9988</td>\n",
              "      <td id=\"T_976b3_row8_col6\" class=\"data row8 col6\" >0.9988</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_976b3_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
              "      <td id=\"T_976b3_row9_col0\" class=\"data row9 col0\" >0.9485</td>\n",
              "      <td id=\"T_976b3_row9_col1\" class=\"data row9 col1\" >0.9905</td>\n",
              "      <td id=\"T_976b3_row9_col2\" class=\"data row9 col2\" >0.9518</td>\n",
              "      <td id=\"T_976b3_row9_col3\" class=\"data row9 col3\" >0.9597</td>\n",
              "      <td id=\"T_976b3_row9_col4\" class=\"data row9 col4\" >0.9557</td>\n",
              "      <td id=\"T_976b3_row9_col5\" class=\"data row9 col5\" >0.8941</td>\n",
              "      <td id=\"T_976b3_row9_col6\" class=\"data row9 col6\" >0.8942</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_976b3_level0_row10\" class=\"row_heading level0 row10\" >Mean</th>\n",
              "      <td id=\"T_976b3_row10_col0\" class=\"data row10 col0\" >0.9373</td>\n",
              "      <td id=\"T_976b3_row10_col1\" class=\"data row10 col1\" >0.9753</td>\n",
              "      <td id=\"T_976b3_row10_col2\" class=\"data row10 col2\" >0.9365</td>\n",
              "      <td id=\"T_976b3_row10_col3\" class=\"data row10 col3\" >0.9592</td>\n",
              "      <td id=\"T_976b3_row10_col4\" class=\"data row10 col4\" >0.9425</td>\n",
              "      <td id=\"T_976b3_row10_col5\" class=\"data row10 col5\" >0.8734</td>\n",
              "      <td id=\"T_976b3_row10_col6\" class=\"data row10 col6\" >0.8816</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_976b3_level0_row11\" class=\"row_heading level0 row11\" >SD</th>\n",
              "      <td id=\"T_976b3_row11_col0\" class=\"data row11 col0\" >0.0748</td>\n",
              "      <td id=\"T_976b3_row11_col1\" class=\"data row11 col1\" >0.0535</td>\n",
              "      <td id=\"T_976b3_row11_col2\" class=\"data row11 col2\" >0.1152</td>\n",
              "      <td id=\"T_976b3_row11_col3\" class=\"data row11 col3\" >0.0562</td>\n",
              "      <td id=\"T_976b3_row11_col4\" class=\"data row11 col4\" >0.0761</td>\n",
              "      <td id=\"T_976b3_row11_col5\" class=\"data row11 col5\" >0.1464</td>\n",
              "      <td id=\"T_976b3_row11_col6\" class=\"data row11 col6\" >0.1308</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n"
            ],
            "text/plain": [
              "<pandas.io.formats.style.Styler at 0x7f55fd663220>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "xgboost = create_model('xgboost')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/html": [
              "<style type=\"text/css\">\n",
              "#T_bcb89_row10_col0, #T_bcb89_row10_col1, #T_bcb89_row10_col2, #T_bcb89_row10_col3, #T_bcb89_row10_col4, #T_bcb89_row10_col5, #T_bcb89_row10_col6 {\n",
              "  background: yellow;\n",
              "}\n",
              "</style>\n",
              "<table id=\"T_bcb89\">\n",
              "  <thead>\n",
              "    <tr>\n",
              "      <th class=\"blank level0\" >&nbsp;</th>\n",
              "      <th id=\"T_bcb89_level0_col0\" class=\"col_heading level0 col0\" >Accuracy</th>\n",
              "      <th id=\"T_bcb89_level0_col1\" class=\"col_heading level0 col1\" >AUC</th>\n",
              "      <th id=\"T_bcb89_level0_col2\" class=\"col_heading level0 col2\" >Recall</th>\n",
              "      <th id=\"T_bcb89_level0_col3\" class=\"col_heading level0 col3\" >Prec.</th>\n",
              "      <th id=\"T_bcb89_level0_col4\" class=\"col_heading level0 col4\" >F1</th>\n",
              "      <th id=\"T_bcb89_level0_col5\" class=\"col_heading level0 col5\" >Kappa</th>\n",
              "      <th id=\"T_bcb89_level0_col6\" class=\"col_heading level0 col6\" >MCC</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th id=\"T_bcb89_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
              "      <td id=\"T_bcb89_row0_col0\" class=\"data row0 col0\" >0.9635</td>\n",
              "      <td id=\"T_bcb89_row0_col1\" class=\"data row0 col1\" >0.9979</td>\n",
              "      <td id=\"T_bcb89_row0_col2\" class=\"data row0 col2\" >0.9391</td>\n",
              "      <td id=\"T_bcb89_row0_col3\" class=\"data row0 col3\" >0.9983</td>\n",
              "      <td id=\"T_bcb89_row0_col4\" class=\"data row0 col4\" >0.9678</td>\n",
              "      <td id=\"T_bcb89_row0_col5\" class=\"data row0 col5\" >0.9257</td>\n",
              "      <td id=\"T_bcb89_row0_col6\" class=\"data row0 col6\" >0.9280</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_bcb89_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
              "      <td id=\"T_bcb89_row1_col0\" class=\"data row1 col0\" >0.9873</td>\n",
              "      <td id=\"T_bcb89_row1_col1\" class=\"data row1 col1\" >0.9997</td>\n",
              "      <td id=\"T_bcb89_row1_col2\" class=\"data row1 col2\" >0.9795</td>\n",
              "      <td id=\"T_bcb89_row1_col3\" class=\"data row1 col3\" >0.9987</td>\n",
              "      <td id=\"T_bcb89_row1_col4\" class=\"data row1 col4\" >0.9890</td>\n",
              "      <td id=\"T_bcb89_row1_col5\" class=\"data row1 col5\" >0.9739</td>\n",
              "      <td id=\"T_bcb89_row1_col6\" class=\"data row1 col6\" >0.9742</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_bcb89_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
              "      <td id=\"T_bcb89_row2_col0\" class=\"data row2 col0\" >0.9958</td>\n",
              "      <td id=\"T_bcb89_row2_col1\" class=\"data row2 col1\" >1.0000</td>\n",
              "      <td id=\"T_bcb89_row2_col2\" class=\"data row2 col2\" >0.9998</td>\n",
              "      <td id=\"T_bcb89_row2_col3\" class=\"data row2 col3\" >0.9930</td>\n",
              "      <td id=\"T_bcb89_row2_col4\" class=\"data row2 col4\" >0.9964</td>\n",
              "      <td id=\"T_bcb89_row2_col5\" class=\"data row2 col5\" >0.9914</td>\n",
              "      <td id=\"T_bcb89_row2_col6\" class=\"data row2 col6\" >0.9914</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_bcb89_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
              "      <td id=\"T_bcb89_row3_col0\" class=\"data row3 col0\" >0.8923</td>\n",
              "      <td id=\"T_bcb89_row3_col1\" class=\"data row3 col1\" >0.8787</td>\n",
              "      <td id=\"T_bcb89_row3_col2\" class=\"data row3 col2\" >0.9416</td>\n",
              "      <td id=\"T_bcb89_row3_col3\" class=\"data row3 col3\" >0.8821</td>\n",
              "      <td id=\"T_bcb89_row3_col4\" class=\"data row3 col4\" >0.9109</td>\n",
              "      <td id=\"T_bcb89_row3_col5\" class=\"data row3 col5\" >0.7752</td>\n",
              "      <td id=\"T_bcb89_row3_col6\" class=\"data row3 col6\" >0.7778</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_bcb89_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
              "      <td id=\"T_bcb89_row4_col0\" class=\"data row4 col0\" >0.7221</td>\n",
              "      <td id=\"T_bcb89_row4_col1\" class=\"data row4 col1\" >0.7508</td>\n",
              "      <td id=\"T_bcb89_row4_col2\" class=\"data row4 col2\" >0.5508</td>\n",
              "      <td id=\"T_bcb89_row4_col3\" class=\"data row4 col3\" >0.9545</td>\n",
              "      <td id=\"T_bcb89_row4_col4\" class=\"data row4 col4\" >0.6985</td>\n",
              "      <td id=\"T_bcb89_row4_col5\" class=\"data row4 col5\" >0.4732</td>\n",
              "      <td id=\"T_bcb89_row4_col6\" class=\"data row4 col6\" >0.5356</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_bcb89_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
              "      <td id=\"T_bcb89_row5_col0\" class=\"data row5 col0\" >0.9546</td>\n",
              "      <td id=\"T_bcb89_row5_col1\" class=\"data row5 col1\" >0.9920</td>\n",
              "      <td id=\"T_bcb89_row5_col2\" class=\"data row5 col2\" >0.9941</td>\n",
              "      <td id=\"T_bcb89_row5_col3\" class=\"data row5 col3\" >0.9328</td>\n",
              "      <td id=\"T_bcb89_row5_col4\" class=\"data row5 col4\" >0.9624</td>\n",
              "      <td id=\"T_bcb89_row5_col5\" class=\"data row5 col5\" >0.9054</td>\n",
              "      <td id=\"T_bcb89_row5_col6\" class=\"data row5 col6\" >0.9083</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_bcb89_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
              "      <td id=\"T_bcb89_row6_col0\" class=\"data row6 col0\" >0.9931</td>\n",
              "      <td id=\"T_bcb89_row6_col1\" class=\"data row6 col1\" >0.9999</td>\n",
              "      <td id=\"T_bcb89_row6_col2\" class=\"data row6 col2\" >0.9901</td>\n",
              "      <td id=\"T_bcb89_row6_col3\" class=\"data row6 col3\" >0.9981</td>\n",
              "      <td id=\"T_bcb89_row6_col4\" class=\"data row6 col4\" >0.9941</td>\n",
              "      <td id=\"T_bcb89_row6_col5\" class=\"data row6 col5\" >0.9858</td>\n",
              "      <td id=\"T_bcb89_row6_col6\" class=\"data row6 col6\" >0.9859</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_bcb89_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
              "      <td id=\"T_bcb89_row7_col0\" class=\"data row7 col0\" >0.9065</td>\n",
              "      <td id=\"T_bcb89_row7_col1\" class=\"data row7 col1\" >0.9758</td>\n",
              "      <td id=\"T_bcb89_row7_col2\" class=\"data row7 col2\" >0.9658</td>\n",
              "      <td id=\"T_bcb89_row7_col3\" class=\"data row7 col3\" >0.8847</td>\n",
              "      <td id=\"T_bcb89_row7_col4\" class=\"data row7 col4\" >0.9235</td>\n",
              "      <td id=\"T_bcb89_row7_col5\" class=\"data row7 col5\" >0.8038</td>\n",
              "      <td id=\"T_bcb89_row7_col6\" class=\"data row7 col6\" >0.8089</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_bcb89_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
              "      <td id=\"T_bcb89_row8_col0\" class=\"data row8 col0\" >0.9991</td>\n",
              "      <td id=\"T_bcb89_row8_col1\" class=\"data row8 col1\" >1.0000</td>\n",
              "      <td id=\"T_bcb89_row8_col2\" class=\"data row8 col2\" >0.9999</td>\n",
              "      <td id=\"T_bcb89_row8_col3\" class=\"data row8 col3\" >0.9986</td>\n",
              "      <td id=\"T_bcb89_row8_col4\" class=\"data row8 col4\" >0.9992</td>\n",
              "      <td id=\"T_bcb89_row8_col5\" class=\"data row8 col5\" >0.9982</td>\n",
              "      <td id=\"T_bcb89_row8_col6\" class=\"data row8 col6\" >0.9982</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_bcb89_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
              "      <td id=\"T_bcb89_row9_col0\" class=\"data row9 col0\" >0.9583</td>\n",
              "      <td id=\"T_bcb89_row9_col1\" class=\"data row9 col1\" >0.9936</td>\n",
              "      <td id=\"T_bcb89_row9_col2\" class=\"data row9 col2\" >0.9528</td>\n",
              "      <td id=\"T_bcb89_row9_col3\" class=\"data row9 col3\" >0.9752</td>\n",
              "      <td id=\"T_bcb89_row9_col4\" class=\"data row9 col4\" >0.9639</td>\n",
              "      <td id=\"T_bcb89_row9_col5\" class=\"data row9 col5\" >0.9145</td>\n",
              "      <td id=\"T_bcb89_row9_col6\" class=\"data row9 col6\" >0.9148</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_bcb89_level0_row10\" class=\"row_heading level0 row10\" >Mean</th>\n",
              "      <td id=\"T_bcb89_row10_col0\" class=\"data row10 col0\" >0.9373</td>\n",
              "      <td id=\"T_bcb89_row10_col1\" class=\"data row10 col1\" >0.9588</td>\n",
              "      <td id=\"T_bcb89_row10_col2\" class=\"data row10 col2\" >0.9313</td>\n",
              "      <td id=\"T_bcb89_row10_col3\" class=\"data row10 col3\" >0.9616</td>\n",
              "      <td id=\"T_bcb89_row10_col4\" class=\"data row10 col4\" >0.9406</td>\n",
              "      <td id=\"T_bcb89_row10_col5\" class=\"data row10 col5\" >0.8747</td>\n",
              "      <td id=\"T_bcb89_row10_col6\" class=\"data row10 col6\" >0.8823</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_bcb89_level0_row11\" class=\"row_heading level0 row11\" >SD</th>\n",
              "      <td id=\"T_bcb89_row11_col0\" class=\"data row11 col0\" >0.0797</td>\n",
              "      <td id=\"T_bcb89_row11_col1\" class=\"data row11 col1\" >0.0778</td>\n",
              "      <td id=\"T_bcb89_row11_col2\" class=\"data row11 col2\" >0.1287</td>\n",
              "      <td id=\"T_bcb89_row11_col3\" class=\"data row11 col3\" >0.0444</td>\n",
              "      <td id=\"T_bcb89_row11_col4\" class=\"data row11 col4\" >0.0856</td>\n",
              "      <td id=\"T_bcb89_row11_col5\" class=\"data row11 col5\" >0.1524</td>\n",
              "      <td id=\"T_bcb89_row11_col6\" class=\"data row11 col6\" >0.1359</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n"
            ],
            "text/plain": [
              "<pandas.io.formats.style.Styler at 0x7f55fd159370>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "catboost = create_model('catboost')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/html": [
              "<style type=\"text/css\">\n",
              "#T_e47e6_row5_col0, #T_e47e6_row5_col1, #T_e47e6_row5_col2, #T_e47e6_row5_col3, #T_e47e6_row5_col4, #T_e47e6_row5_col5, #T_e47e6_row5_col6 {\n",
              "  background: yellow;\n",
              "}\n",
              "</style>\n",
              "<table id=\"T_e47e6\">\n",
              "  <thead>\n",
              "    <tr>\n",
              "      <th class=\"blank level0\" >&nbsp;</th>\n",
              "      <th id=\"T_e47e6_level0_col0\" class=\"col_heading level0 col0\" >Accuracy</th>\n",
              "      <th id=\"T_e47e6_level0_col1\" class=\"col_heading level0 col1\" >AUC</th>\n",
              "      <th id=\"T_e47e6_level0_col2\" class=\"col_heading level0 col2\" >Recall</th>\n",
              "      <th id=\"T_e47e6_level0_col3\" class=\"col_heading level0 col3\" >Prec.</th>\n",
              "      <th id=\"T_e47e6_level0_col4\" class=\"col_heading level0 col4\" >F1</th>\n",
              "      <th id=\"T_e47e6_level0_col5\" class=\"col_heading level0 col5\" >Kappa</th>\n",
              "      <th id=\"T_e47e6_level0_col6\" class=\"col_heading level0 col6\" >MCC</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th id=\"T_e47e6_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
              "      <td id=\"T_e47e6_row0_col0\" class=\"data row0 col0\" >0.9699</td>\n",
              "      <td id=\"T_e47e6_row0_col1\" class=\"data row0 col1\" >0.9781</td>\n",
              "      <td id=\"T_e47e6_row0_col2\" class=\"data row0 col2\" >0.9612</td>\n",
              "      <td id=\"T_e47e6_row0_col3\" class=\"data row0 col3\" >0.9869</td>\n",
              "      <td id=\"T_e47e6_row0_col4\" class=\"data row0 col4\" >0.9739</td>\n",
              "      <td id=\"T_e47e6_row0_col5\" class=\"data row0 col5\" >0.9384</td>\n",
              "      <td id=\"T_e47e6_row0_col6\" class=\"data row0 col6\" >0.9388</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_e47e6_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
              "      <td id=\"T_e47e6_row1_col0\" class=\"data row1 col0\" >0.6931</td>\n",
              "      <td id=\"T_e47e6_row1_col1\" class=\"data row1 col1\" >0.9445</td>\n",
              "      <td id=\"T_e47e6_row1_col2\" class=\"data row1 col2\" >0.9768</td>\n",
              "      <td id=\"T_e47e6_row1_col3\" class=\"data row1 col3\" >0.6605</td>\n",
              "      <td id=\"T_e47e6_row1_col4\" class=\"data row1 col4\" >0.7881</td>\n",
              "      <td id=\"T_e47e6_row1_col5\" class=\"data row1 col5\" >0.3001</td>\n",
              "      <td id=\"T_e47e6_row1_col6\" class=\"data row1 col6\" >0.3897</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_e47e6_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
              "      <td id=\"T_e47e6_row2_col0\" class=\"data row2 col0\" >0.8669</td>\n",
              "      <td id=\"T_e47e6_row2_col1\" class=\"data row2 col1\" >0.8837</td>\n",
              "      <td id=\"T_e47e6_row2_col2\" class=\"data row2 col2\" >0.8127</td>\n",
              "      <td id=\"T_e47e6_row2_col3\" class=\"data row2 col3\" >0.9527</td>\n",
              "      <td id=\"T_e47e6_row2_col4\" class=\"data row2 col4\" >0.8771</td>\n",
              "      <td id=\"T_e47e6_row2_col5\" class=\"data row2 col5\" >0.7340</td>\n",
              "      <td id=\"T_e47e6_row2_col6\" class=\"data row2 col6\" >0.7451</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_e47e6_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
              "      <td id=\"T_e47e6_row3_col0\" class=\"data row3 col0\" >0.7534</td>\n",
              "      <td id=\"T_e47e6_row3_col1\" class=\"data row3 col1\" >0.9601</td>\n",
              "      <td id=\"T_e47e6_row3_col2\" class=\"data row3 col2\" >0.5868</td>\n",
              "      <td id=\"T_e47e6_row3_col3\" class=\"data row3 col3\" >0.9853</td>\n",
              "      <td id=\"T_e47e6_row3_col4\" class=\"data row3 col4\" >0.7355</td>\n",
              "      <td id=\"T_e47e6_row3_col5\" class=\"data row3 col5\" >0.5308</td>\n",
              "      <td id=\"T_e47e6_row3_col6\" class=\"data row3 col6\" >0.5943</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_e47e6_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
              "      <td id=\"T_e47e6_row4_col0\" class=\"data row4 col0\" >0.9587</td>\n",
              "      <td id=\"T_e47e6_row4_col1\" class=\"data row4 col1\" >0.9966</td>\n",
              "      <td id=\"T_e47e6_row4_col2\" class=\"data row4 col2\" >0.9465</td>\n",
              "      <td id=\"T_e47e6_row4_col3\" class=\"data row4 col3\" >0.9822</td>\n",
              "      <td id=\"T_e47e6_row4_col4\" class=\"data row4 col4\" >0.9640</td>\n",
              "      <td id=\"T_e47e6_row4_col5\" class=\"data row4 col5\" >0.9156</td>\n",
              "      <td id=\"T_e47e6_row4_col6\" class=\"data row4 col6\" >0.9165</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_e47e6_level0_row5\" class=\"row_heading level0 row5\" >Mean</th>\n",
              "      <td id=\"T_e47e6_row5_col0\" class=\"data row5 col0\" >0.8484</td>\n",
              "      <td id=\"T_e47e6_row5_col1\" class=\"data row5 col1\" >0.9526</td>\n",
              "      <td id=\"T_e47e6_row5_col2\" class=\"data row5 col2\" >0.8568</td>\n",
              "      <td id=\"T_e47e6_row5_col3\" class=\"data row5 col3\" >0.9135</td>\n",
              "      <td id=\"T_e47e6_row5_col4\" class=\"data row5 col4\" >0.8677</td>\n",
              "      <td id=\"T_e47e6_row5_col5\" class=\"data row5 col5\" >0.6838</td>\n",
              "      <td id=\"T_e47e6_row5_col6\" class=\"data row5 col6\" >0.7169</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_e47e6_level0_row6\" class=\"row_heading level0 row6\" >SD</th>\n",
              "      <td id=\"T_e47e6_row6_col0\" class=\"data row6 col0\" >0.1099</td>\n",
              "      <td id=\"T_e47e6_row6_col1\" class=\"data row6 col1\" >0.0386</td>\n",
              "      <td id=\"T_e47e6_row6_col2\" class=\"data row6 col2\" >0.1471</td>\n",
              "      <td id=\"T_e47e6_row6_col3\" class=\"data row6 col3\" >0.1271</td>\n",
              "      <td id=\"T_e47e6_row6_col4\" class=\"data row6 col4\" >0.0943</td>\n",
              "      <td id=\"T_e47e6_row6_col5\" class=\"data row6 col5\" >0.2415</td>\n",
              "      <td id=\"T_e47e6_row6_col6\" class=\"data row6 col6\" >0.2059</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n"
            ],
            "text/plain": [
              "<pandas.io.formats.style.Styler at 0x7f5605839a60>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "blend = blend_models([rf, xgboost, catboost], choose_better=True, optimize='f1', fold=5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "XGBClassifier(base_score=0.5, booster='gbtree', callbacks=None,\n",
              "              colsample_bylevel=1, colsample_bynode=1, colsample_bytree=1,\n",
              "              early_stopping_rounds=None, enable_categorical=False,\n",
              "              eval_metric=None, feature_types=None, gamma=0, gpu_id=0,\n",
              "              grow_policy='depthwise', importance_type=None,\n",
              "              interaction_constraints='', learning_rate=0.300000012,\n",
              "              max_bin=256, max_cat_threshold=64, max_cat_to_onehot=4,\n",
              "              max_delta_step=0, max_depth=6, max_leaves=0, min_child_weight=1,\n",
              "              missing=nan, monotone_constraints='()', n_estimators=100,\n",
              "              n_jobs=-1, num_parallel_tree=1, objective='binary:logistic',\n",
              "              predictor='auto', ...)"
            ]
          },
          "execution_count": 16,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "blend"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/html": [
              "<style type=\"text/css\">\n",
              "</style>\n",
              "<table id=\"T_1aa85\">\n",
              "  <thead>\n",
              "    <tr>\n",
              "      <th class=\"blank level0\" >&nbsp;</th>\n",
              "      <th id=\"T_1aa85_level0_col0\" class=\"col_heading level0 col0\" >Model</th>\n",
              "      <th id=\"T_1aa85_level0_col1\" class=\"col_heading level0 col1\" >Accuracy</th>\n",
              "      <th id=\"T_1aa85_level0_col2\" class=\"col_heading level0 col2\" >AUC</th>\n",
              "      <th id=\"T_1aa85_level0_col3\" class=\"col_heading level0 col3\" >Recall</th>\n",
              "      <th id=\"T_1aa85_level0_col4\" class=\"col_heading level0 col4\" >Prec.</th>\n",
              "      <th id=\"T_1aa85_level0_col5\" class=\"col_heading level0 col5\" >F1</th>\n",
              "      <th id=\"T_1aa85_level0_col6\" class=\"col_heading level0 col6\" >Kappa</th>\n",
              "      <th id=\"T_1aa85_level0_col7\" class=\"col_heading level0 col7\" >MCC</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th id=\"T_1aa85_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
              "      <td id=\"T_1aa85_row0_col0\" class=\"data row0 col0\" >Extreme Gradient Boosting</td>\n",
              "      <td id=\"T_1aa85_row0_col1\" class=\"data row0 col1\" >0.8658</td>\n",
              "      <td id=\"T_1aa85_row0_col2\" class=\"data row0 col2\" >0.8966</td>\n",
              "      <td id=\"T_1aa85_row0_col3\" class=\"data row0 col3\" >0.7948</td>\n",
              "      <td id=\"T_1aa85_row0_col4\" class=\"data row0 col4\" >0.9702</td>\n",
              "      <td id=\"T_1aa85_row0_col5\" class=\"data row0 col5\" >0.8738</td>\n",
              "      <td id=\"T_1aa85_row0_col6\" class=\"data row0 col6\" >0.7335</td>\n",
              "      <td id=\"T_1aa85_row0_col7\" class=\"data row0 col7\" >0.7502</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n"
            ],
            "text/plain": [
              "<pandas.io.formats.style.Styler at 0x7f5605830d30>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "pred_df = predict_model(blend, encoded_labels=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {},
      "outputs": [],
      "source": [
        "predict = list(pred_df['Label'].iloc[:, 1])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {},
      "outputs": [],
      "source": [
        "# 정상 외부 IP set\n",
        "outer_benignIP_set = open_pickle(\"./outer_benign_IP.pkl\")\n",
        "# 악성 외부 IP set\n",
        "outer_malIP_set = open_pickle(\"./outer_mal_IP.pkl\")\n",
        "\n",
        "valid_raw_df = pd.read_csv('./valid.csv')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Problem #3을 위해 dictionary형태로 변경\n",
        "labelIP_dict = dict()\n",
        "\n",
        "for ip in outer_benignIP_set:\n",
        "    labelIP_dict[ip] = 0\n",
        "for ip in outer_malIP_set:\n",
        "    labelIP_dict[ip] = 1"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {},
      "outputs": [],
      "source": [
        "predictIP_dict = dict()\n",
        "\n",
        "for ip in labelIP_dict.keys():\n",
        "  predictIP_dict[ip] = valid_raw_df.loc[(valid_raw_df['Src IP'] == ip) | (valid_raw_df['Dst IP'] == ip)].index\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {},
      "outputs": [],
      "source": [
        "attack_percentage_per_IP = []\n",
        "\n",
        "for ip in predictIP_dict.keys():\n",
        "  sum = 0\n",
        "  for idx in predictIP_dict[ip]:\n",
        "    sum += abs(1 - predict[idx])\n",
        "  percentage = sum / len(predictIP_dict[ip])\n",
        "  attack_percentage_per_IP.append(percentage)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {},
      "outputs": [],
      "source": [
        "pred_IP = []\n",
        "label_IP = []\n",
        "\n",
        "for k in predictIP_dict.keys():\n",
        "    pred_IP.append(predictIP_dict[k])\n",
        "    label_IP.append(labelIP_dict[k])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {},
      "outputs": [],
      "source": [
        "from sklearn.metrics import roc_curve, confusion_matrix, accuracy_score, recall_score, f1_score, precision_score"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {},
      "outputs": [],
      "source": [
        "fpr, tpr, th = roc_curve(label_IP, attack_percentage_per_IP)\n",
        "J = tpr-fpr\n",
        "ix = np.argmax(J)\n",
        "best_threshold = th[ix]\n",
        "\n",
        "predict_by_roc = (attack_percentage_per_IP>= best_threshold).astype('int') # 새로운 임계점 적용한 것"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "0.13043478260869565"
            ]
          },
          "execution_count": 26,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "best_threshold"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "confusion_matrix\n",
            " [[532  58]\n",
            " [ 13 444]]\n",
            "accuracy\t:  0.93219\n",
            "precision\t:  0.88446\n",
            "recall\t\t:  0.97155\n",
            "F-1\t\t:  0.92596\n"
          ]
        }
      ],
      "source": [
        "print(\"confusion_matrix\\n\", confusion_matrix(label_IP, predict_by_roc))\n",
        "print(f\"accuracy\\t: {accuracy_score(label_IP, predict_by_roc): .5f}\")\n",
        "print(f\"precision\\t: {precision_score(label_IP, predict_by_roc): .5f}\")\n",
        "print(f\"recall\\t\\t: {recall_score(label_IP, predict_by_roc): .5f}\")\n",
        "print(f\"F-1\\t\\t: {f1_score(label_IP, predict_by_roc): .5f}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 42,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Src IP</th>\n",
              "      <th>Src Port</th>\n",
              "      <th>Dst IP</th>\n",
              "      <th>Dst Port</th>\n",
              "      <th>Flow Duration</th>\n",
              "      <th>Total Fwd Packet</th>\n",
              "      <th>Total Bwd packets</th>\n",
              "      <th>Total Length of Fwd Packet</th>\n",
              "      <th>Total Length of Bwd Packet</th>\n",
              "      <th>Fwd Packet Length Max</th>\n",
              "      <th>...</th>\n",
              "      <th>Flags_..U...S.</th>\n",
              "      <th>Flags_.E.A..S.</th>\n",
              "      <th>Flags_CE....S.</th>\n",
              "      <th>Flags_CE.A..S.</th>\n",
              "      <th>Flags_CE.A..SF</th>\n",
              "      <th>Flags_CE.A.RS.</th>\n",
              "      <th>Flags_CE.A.RSF</th>\n",
              "      <th>Flags_CE.AP.S.</th>\n",
              "      <th>Flags_CE.AP.SF</th>\n",
              "      <th>Flags_CE.APRS.</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.005930</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000252</td>\n",
              "      <td>0.118292</td>\n",
              "      <td>0.024892</td>\n",
              "      <td>0.000017</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.005930</td>\n",
              "      <td>0.000011</td>\n",
              "      <td>0.000252</td>\n",
              "      <td>0.118292</td>\n",
              "      <td>0.024816</td>\n",
              "      <td>0.000017</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.005930</td>\n",
              "      <td>0.000057</td>\n",
              "      <td>0.000252</td>\n",
              "      <td>0.118292</td>\n",
              "      <td>0.024961</td>\n",
              "      <td>0.000017</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0.005930</td>\n",
              "      <td>0.000011</td>\n",
              "      <td>0.000252</td>\n",
              "      <td>0.118292</td>\n",
              "      <td>0.025030</td>\n",
              "      <td>0.000017</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0.003975</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000252</td>\n",
              "      <td>0.118292</td>\n",
              "      <td>0.025053</td>\n",
              "      <td>0.000017</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>325958</th>\n",
              "      <td>0.000075</td>\n",
              "      <td>0.001115</td>\n",
              "      <td>0.000873</td>\n",
              "      <td>0.001799</td>\n",
              "      <td>0.814971</td>\n",
              "      <td>0.065257</td>\n",
              "      <td>0.197632</td>\n",
              "      <td>0.013032</td>\n",
              "      <td>0.021930</td>\n",
              "      <td>0.031507</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>325959</th>\n",
              "      <td>0.000075</td>\n",
              "      <td>0.001115</td>\n",
              "      <td>0.000719</td>\n",
              "      <td>0.001799</td>\n",
              "      <td>0.829781</td>\n",
              "      <td>0.098200</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.019610</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.031507</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>325960</th>\n",
              "      <td>0.000075</td>\n",
              "      <td>0.001115</td>\n",
              "      <td>0.000699</td>\n",
              "      <td>0.001799</td>\n",
              "      <td>0.831896</td>\n",
              "      <td>0.098736</td>\n",
              "      <td>0.301192</td>\n",
              "      <td>0.019717</td>\n",
              "      <td>0.042704</td>\n",
              "      <td>0.031507</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>325961</th>\n",
              "      <td>0.000021</td>\n",
              "      <td>0.001035</td>\n",
              "      <td>0.000005</td>\n",
              "      <td>0.001799</td>\n",
              "      <td>0.802176</td>\n",
              "      <td>0.033656</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.047199</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.221233</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>325962</th>\n",
              "      <td>0.000075</td>\n",
              "      <td>0.001115</td>\n",
              "      <td>0.000303</td>\n",
              "      <td>0.001799</td>\n",
              "      <td>0.827627</td>\n",
              "      <td>0.097051</td>\n",
              "      <td>0.296102</td>\n",
              "      <td>0.019380</td>\n",
              "      <td>0.029003</td>\n",
              "      <td>0.031507</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>325963 rows × 78 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "          Src IP  Src Port    Dst IP  Dst Port  Flow Duration  \\\n",
              "0       0.005930  0.000000  0.000252  0.118292       0.024892   \n",
              "1       0.005930  0.000011  0.000252  0.118292       0.024816   \n",
              "2       0.005930  0.000057  0.000252  0.118292       0.024961   \n",
              "3       0.005930  0.000011  0.000252  0.118292       0.025030   \n",
              "4       0.003975  0.000000  0.000252  0.118292       0.025053   \n",
              "...          ...       ...       ...       ...            ...   \n",
              "325958  0.000075  0.001115  0.000873  0.001799       0.814971   \n",
              "325959  0.000075  0.001115  0.000719  0.001799       0.829781   \n",
              "325960  0.000075  0.001115  0.000699  0.001799       0.831896   \n",
              "325961  0.000021  0.001035  0.000005  0.001799       0.802176   \n",
              "325962  0.000075  0.001115  0.000303  0.001799       0.827627   \n",
              "\n",
              "        Total Fwd Packet  Total Bwd packets  Total Length of Fwd Packet  \\\n",
              "0               0.000017           0.000000                    0.000000   \n",
              "1               0.000017           0.000000                    0.000000   \n",
              "2               0.000017           0.000000                    0.000000   \n",
              "3               0.000017           0.000000                    0.000000   \n",
              "4               0.000017           0.000000                    0.000000   \n",
              "...                  ...                ...                         ...   \n",
              "325958          0.065257           0.197632                    0.013032   \n",
              "325959          0.098200           0.000000                    0.019610   \n",
              "325960          0.098736           0.301192                    0.019717   \n",
              "325961          0.033656           0.000000                    0.047199   \n",
              "325962          0.097051           0.296102                    0.019380   \n",
              "\n",
              "        Total Length of Bwd Packet  Fwd Packet Length Max  ...  \\\n",
              "0                         0.000000               0.000000  ...   \n",
              "1                         0.000000               0.000000  ...   \n",
              "2                         0.000000               0.000000  ...   \n",
              "3                         0.000000               0.000000  ...   \n",
              "4                         0.000000               0.000000  ...   \n",
              "...                            ...                    ...  ...   \n",
              "325958                    0.021930               0.031507  ...   \n",
              "325959                    0.000000               0.031507  ...   \n",
              "325960                    0.042704               0.031507  ...   \n",
              "325961                    0.000000               0.221233  ...   \n",
              "325962                    0.029003               0.031507  ...   \n",
              "\n",
              "        Flags_..U...S.  Flags_.E.A..S.  Flags_CE....S.  Flags_CE.A..S.  \\\n",
              "0                  0.0             0.0             0.0             0.0   \n",
              "1                  0.0             0.0             0.0             0.0   \n",
              "2                  0.0             0.0             0.0             0.0   \n",
              "3                  0.0             0.0             0.0             0.0   \n",
              "4                  0.0             0.0             0.0             0.0   \n",
              "...                ...             ...             ...             ...   \n",
              "325958             0.0             0.0             0.0             0.0   \n",
              "325959             0.0             0.0             0.0             0.0   \n",
              "325960             0.0             0.0             0.0             0.0   \n",
              "325961             0.0             0.0             0.0             0.0   \n",
              "325962             0.0             0.0             0.0             0.0   \n",
              "\n",
              "        Flags_CE.A..SF  Flags_CE.A.RS.  Flags_CE.A.RSF  Flags_CE.AP.S.  \\\n",
              "0                  0.0             0.0             0.0             0.0   \n",
              "1                  0.0             0.0             0.0             0.0   \n",
              "2                  0.0             0.0             0.0             0.0   \n",
              "3                  0.0             0.0             0.0             0.0   \n",
              "4                  0.0             0.0             0.0             0.0   \n",
              "...                ...             ...             ...             ...   \n",
              "325958             0.0             0.0             0.0             0.0   \n",
              "325959             0.0             0.0             0.0             0.0   \n",
              "325960             0.0             0.0             0.0             0.0   \n",
              "325961             0.0             0.0             0.0             0.0   \n",
              "325962             0.0             0.0             0.0             0.0   \n",
              "\n",
              "        Flags_CE.AP.SF  Flags_CE.APRS.  \n",
              "0                  0.0             0.0  \n",
              "1                  0.0             0.0  \n",
              "2                  0.0             0.0  \n",
              "3                  0.0             0.0  \n",
              "4                  0.0             0.0  \n",
              "...                ...             ...  \n",
              "325958             0.0             0.0  \n",
              "325959             0.0             0.0  \n",
              "325960             0.0             0.0  \n",
              "325961             0.0             0.0  \n",
              "325962             0.0             0.0  \n",
              "\n",
              "[325963 rows x 78 columns]"
            ]
          },
          "execution_count": 42,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "prep_pipe = get_config('prep_pipe')\n",
        "test_data = prep_pipe.transform(test_df)\n",
        "test_data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 54,
      "metadata": {},
      "outputs": [],
      "source": [
        "test_pred_df = predict_model(blend, data=test_df, encoded_labels=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 55,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Src IP</th>\n",
              "      <th>Src Port</th>\n",
              "      <th>Dst IP</th>\n",
              "      <th>Dst Port</th>\n",
              "      <th>Protocol</th>\n",
              "      <th>Flags</th>\n",
              "      <th>Timestamp</th>\n",
              "      <th>Flow Duration</th>\n",
              "      <th>Total Fwd Packet</th>\n",
              "      <th>Total Bwd packets</th>\n",
              "      <th>...</th>\n",
              "      <th>Active Mean</th>\n",
              "      <th>Active Std</th>\n",
              "      <th>Active Max</th>\n",
              "      <th>Active Min</th>\n",
              "      <th>Idle Mean</th>\n",
              "      <th>Idle Std</th>\n",
              "      <th>Idle Max</th>\n",
              "      <th>Idle Min</th>\n",
              "      <th>Label</th>\n",
              "      <th>Score</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>556</td>\n",
              "      <td>1</td>\n",
              "      <td>50</td>\n",
              "      <td>23150</td>\n",
              "      <td>TCP</td>\n",
              "      <td>......S.</td>\n",
              "      <td>03:33:43</td>\n",
              "      <td>2987074</td>\n",
              "      <td>3</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1</td>\n",
              "      <td>0.5636</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>556</td>\n",
              "      <td>2</td>\n",
              "      <td>50</td>\n",
              "      <td>23150</td>\n",
              "      <td>TCP</td>\n",
              "      <td>......S.</td>\n",
              "      <td>03:32:49</td>\n",
              "      <td>2977944</td>\n",
              "      <td>3</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1</td>\n",
              "      <td>0.5636</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>556</td>\n",
              "      <td>6</td>\n",
              "      <td>50</td>\n",
              "      <td>23150</td>\n",
              "      <td>TCP</td>\n",
              "      <td>......S.</td>\n",
              "      <td>04:23:46</td>\n",
              "      <td>2995375</td>\n",
              "      <td>3</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1</td>\n",
              "      <td>0.5558</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>556</td>\n",
              "      <td>2</td>\n",
              "      <td>50</td>\n",
              "      <td>23150</td>\n",
              "      <td>TCP</td>\n",
              "      <td>......S.</td>\n",
              "      <td>04:28:08</td>\n",
              "      <td>3003635</td>\n",
              "      <td>3</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1</td>\n",
              "      <td>0.6073</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>373</td>\n",
              "      <td>1</td>\n",
              "      <td>50</td>\n",
              "      <td>23150</td>\n",
              "      <td>TCP</td>\n",
              "      <td>......S.</td>\n",
              "      <td>03:46:39</td>\n",
              "      <td>3006388</td>\n",
              "      <td>3</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1</td>\n",
              "      <td>0.7305</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>325958</th>\n",
              "      <td>8</td>\n",
              "      <td>99</td>\n",
              "      <td>171</td>\n",
              "      <td>353</td>\n",
              "      <td>UDP</td>\n",
              "      <td>........</td>\n",
              "      <td>02:27:21</td>\n",
              "      <td>97796517</td>\n",
              "      <td>7784</td>\n",
              "      <td>7727</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1</td>\n",
              "      <td>0.9993</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>325959</th>\n",
              "      <td>8</td>\n",
              "      <td>99</td>\n",
              "      <td>141</td>\n",
              "      <td>353</td>\n",
              "      <td>UDP</td>\n",
              "      <td>........</td>\n",
              "      <td>02:27:19</td>\n",
              "      <td>99573675</td>\n",
              "      <td>11713</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1</td>\n",
              "      <td>0.9476</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>325960</th>\n",
              "      <td>8</td>\n",
              "      <td>99</td>\n",
              "      <td>137</td>\n",
              "      <td>353</td>\n",
              "      <td>UDP</td>\n",
              "      <td>........</td>\n",
              "      <td>02:27:19</td>\n",
              "      <td>99827550</td>\n",
              "      <td>11777</td>\n",
              "      <td>11776</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1</td>\n",
              "      <td>0.9998</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>325961</th>\n",
              "      <td>3</td>\n",
              "      <td>92</td>\n",
              "      <td>2</td>\n",
              "      <td>353</td>\n",
              "      <td>UDP</td>\n",
              "      <td>........</td>\n",
              "      <td>02:27:22</td>\n",
              "      <td>96261105</td>\n",
              "      <td>4015</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1</td>\n",
              "      <td>0.9862</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>325962</th>\n",
              "      <td>8</td>\n",
              "      <td>99</td>\n",
              "      <td>60</td>\n",
              "      <td>353</td>\n",
              "      <td>UDP</td>\n",
              "      <td>........</td>\n",
              "      <td>02:27:19</td>\n",
              "      <td>99315298</td>\n",
              "      <td>11576</td>\n",
              "      <td>11577</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1</td>\n",
              "      <td>0.9997</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>325963 rows × 55 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "        Src IP  Src Port  Dst IP  Dst Port Protocol     Flags Timestamp  \\\n",
              "0          556         1      50     23150      TCP  ......S.  03:33:43   \n",
              "1          556         2      50     23150      TCP  ......S.  03:32:49   \n",
              "2          556         6      50     23150      TCP  ......S.  04:23:46   \n",
              "3          556         2      50     23150      TCP  ......S.  04:28:08   \n",
              "4          373         1      50     23150      TCP  ......S.  03:46:39   \n",
              "...        ...       ...     ...       ...      ...       ...       ...   \n",
              "325958       8        99     171       353      UDP  ........  02:27:21   \n",
              "325959       8        99     141       353      UDP  ........  02:27:19   \n",
              "325960       8        99     137       353      UDP  ........  02:27:19   \n",
              "325961       3        92       2       353      UDP  ........  02:27:22   \n",
              "325962       8        99      60       353      UDP  ........  02:27:19   \n",
              "\n",
              "        Flow Duration  Total Fwd Packet  Total Bwd packets  ...  Active Mean  \\\n",
              "0             2987074                 3                  0  ...          0.0   \n",
              "1             2977944                 3                  0  ...          0.0   \n",
              "2             2995375                 3                  0  ...          0.0   \n",
              "3             3003635                 3                  0  ...          0.0   \n",
              "4             3006388                 3                  0  ...          0.0   \n",
              "...               ...               ...                ...  ...          ...   \n",
              "325958       97796517              7784               7727  ...          0.0   \n",
              "325959       99573675             11713                  0  ...          0.0   \n",
              "325960       99827550             11777              11776  ...          0.0   \n",
              "325961       96261105              4015                  0  ...          0.0   \n",
              "325962       99315298             11576              11577  ...          0.0   \n",
              "\n",
              "        Active Std  Active Max  Active Min  Idle Mean  Idle Std  Idle Max  \\\n",
              "0              0.0         0.0         0.0        0.0       0.0       0.0   \n",
              "1              0.0         0.0         0.0        0.0       0.0       0.0   \n",
              "2              0.0         0.0         0.0        0.0       0.0       0.0   \n",
              "3              0.0         0.0         0.0        0.0       0.0       0.0   \n",
              "4              0.0         0.0         0.0        0.0       0.0       0.0   \n",
              "...            ...         ...         ...        ...       ...       ...   \n",
              "325958         0.0         0.0         0.0        0.0       0.0       0.0   \n",
              "325959         0.0         0.0         0.0        0.0       0.0       0.0   \n",
              "325960         0.0         0.0         0.0        0.0       0.0       0.0   \n",
              "325961         0.0         0.0         0.0        0.0       0.0       0.0   \n",
              "325962         0.0         0.0         0.0        0.0       0.0       0.0   \n",
              "\n",
              "        Idle Min  Label   Score  \n",
              "0            0.0      1  0.5636  \n",
              "1            0.0      1  0.5636  \n",
              "2            0.0      1  0.5558  \n",
              "3            0.0      1  0.6073  \n",
              "4            0.0      1  0.7305  \n",
              "...          ...    ...     ...  \n",
              "325958       0.0      1  0.9993  \n",
              "325959       0.0      1  0.9476  \n",
              "325960       0.0      1  0.9998  \n",
              "325961       0.0      1  0.9862  \n",
              "325962       0.0      1  0.9997  \n",
              "\n",
              "[325963 rows x 55 columns]"
            ]
          },
          "execution_count": 55,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "test_pred_df"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 56,
      "metadata": {},
      "outputs": [],
      "source": [
        "predict = test_pred_df.loc[:, 'Label']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 49,
      "metadata": {},
      "outputs": [],
      "source": [
        "# test\n",
        "outer_testIP_set = open_pickle('./outer_ip_set.pkl')\n",
        "\n",
        "test_raw_df = pd.read_csv('./project2_test.csv')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 50,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Problem #3을 위해 dictionary형태로 변경\n",
        "labelIP_dict = dict()\n",
        "\n",
        "for ip in outer_testIP_set:\n",
        "    labelIP_dict[ip] = 0"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 51,
      "metadata": {},
      "outputs": [],
      "source": [
        "predictIP_dict = dict()\n",
        "\n",
        "for ip in labelIP_dict.keys():\n",
        "  predictIP_dict[ip] = test_raw_df.loc[(test_raw_df['Src IP'] == ip) | (test_raw_df['Dst IP'] == ip)].index\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 57,
      "metadata": {},
      "outputs": [],
      "source": [
        "attack_percentage_per_IP = []\n",
        "\n",
        "for ip in predictIP_dict.keys():\n",
        "  sum = 0\n",
        "  for idx in predictIP_dict[ip]:\n",
        "    sum += abs(1 - predict[idx])\n",
        "  percentage = sum / len(predictIP_dict[ip])\n",
        "  attack_percentage_per_IP.append(percentage)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 58,
      "metadata": {},
      "outputs": [],
      "source": [
        "pred_IP = []\n",
        "label_IP = []\n",
        "\n",
        "for k in predictIP_dict.keys():\n",
        "    pred_IP.append(predictIP_dict[k])\n",
        "    label_IP.append(labelIP_dict[k])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 59,
      "metadata": {},
      "outputs": [],
      "source": [
        "predict_by_roc = (attack_percentage_per_IP>= best_threshold).astype('int') # 새로운 임계점 적용한 것"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 67,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Counter({0: 538, 1: 509})"
            ]
          },
          "execution_count": 67,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "Counter(predict_by_roc)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 66,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'250.105.113.35': 0, '185.145.242.16': 1, '253.63.11.232': 1, '61.21.103.171': 1, '37.96.19.82': 1, '175.129.41.214': 1, '221.197.125.136': 1, '167.134.124.80': 1, '122.183.66.254': 0, '4.38.84.22': 0, '28.177.89.136': 0, '4.24.99.14': 0, '194.121.137.206': 1, '67.46.34.141': 0, '187.151.238.219': 1, '235.210.67.2': 0, '107.142.27.240': 1, '34.137.182.160': 1, '74.181.208.165': 1, '120.97.76.233': 0, '97.25.252.99': 0, '49.48.211.70': 0, '49.147.157.3': 0, '208.14.229.93': 0, '209.104.160.113': 1, '23.126.214.12': 1, '227.156.248.250': 1, '198.169.57.184': 1, '46.96.194.149': 1, '67.240.126.204': 0, '219.3.6.15': 1, '109.208.147.88': 1, '81.63.179.168': 1, '116.147.240.244': 0, '2.167.245.52': 0, '49.246.44.135': 0, '181.89.48.135': 1, '239.18.165.34': 1, '2.229.151.32': 0, '28.218.48.221': 0, '67.54.41.24': 0, '200.226.159.1': 1, '243.182.20.0': 1, '46.193.185.55': 1, '94.162.184.12': 1, '104.234.9.112': 0, '200.10.60.157': 1, '40.223.205.78': 0, '17.227.35.137': 0, '84.134.254.199': 1, '161.40.204.110': 0, '206.154.104.84': 1, '196.50.57.113': 1, '80.94.191.105': 1, '2.144.231.67': 0, '36.93.154.230': 0, '44.210.248.174': 1, '250.124.235.184': 0, '19.153.90.174': 0, '162.107.209.113': 0, '248.29.27.16': 1, '205.220.36.112': 0, '99.139.63.21': 1, '213.30.216.150': 1, '119.157.109.49': 1, '9.172.150.128': 0, '46.108.71.12': 1, '1.128.101.201': 0, '160.213.44.100': 1, '30.4.140.207': 1, '200.206.21.111': 1, '127.120.226.200': 1, '81.207.9.70': 1, '17.48.79.105': 0, '93.241.9.63': 1, '160.227.13.29': 1, '47.46.124.158': 0, '84.30.92.244': 0, '23.18.135.19': 0, '185.37.255.112': 1, '172.25.48.126': 0, '36.29.117.91': 1, '59.152.215.24': 0, '194.95.232.171': 0, '113.208.47.92': 0, '37.2.29.164': 0, '37.118.176.250': 1, '50.76.173.211': 0, '62.121.159.13': 0, '114.27.200.191': 1, '28.220.235.190': 0, '28.219.55.143': 0, '70.47.102.43': 1, '30.226.93.225': 1, '35.95.225.89': 1, '21.101.248.44': 0, '50.210.254.47': 0, '171.89.6.128': 0, '117.154.162.49': 1, '166.142.63.121': 0, '59.10.179.206': 0, '172.177.181.243': 0, '102.32.110.112': 0, '27.178.241.144': 1, '106.241.198.65': 1, '37.147.232.230': 1, '204.34.51.215': 0, '221.251.96.154': 0, '100.144.85.150': 0, '67.13.232.151': 1, '76.195.62.115': 1, '47.43.203.193': 0, '171.100.207.123': 0, '234.199.115.83': 1, '50.250.229.168': 0, '11.127.245.99': 0, '65.218.3.220': 0, '46.163.83.131': 1, '170.102.147.120': 0, '252.3.43.150': 1, '236.210.228.146': 1, '237.206.70.82': 1, '221.66.217.203': 1, '252.27.45.226': 1, '117.13.28.13': 1, '36.27.28.156': 0, '91.94.184.37': 0, '4.225.147.111': 1, '36.68.2.30': 1, '61.142.114.114': 0, '106.207.75.223': 0, '104.61.216.146': 0, '184.90.59.58': 1, '206.77.207.228': 0, '37.22.36.65': 0, '122.143.7.2': 1, '52.126.159.199': 1, '50.196.175.159': 0, '84.64.112.62': 1, '180.25.228.89': 0, '221.146.127.220': 1, '115.249.158.49': 1, '199.97.0.174': 1, '218.234.70.52': 1, '179.238.247.176': 0, '172.165.111.247': 0, '209.105.121.92': 1, '215.179.168.93': 0, '2.220.92.106': 0, '200.80.185.121': 0, '77.109.74.242': 0, '90.238.214.113': 0, '49.185.126.56': 0, '49.113.198.121': 1, '106.57.7.188': 0, '165.41.23.99': 1, '66.162.75.214': 0, '44.222.1.141': 1, '194.201.254.247': 0, '17.224.127.209': 0, '50.234.253.36': 0, '47.38.58.121': 0, '200.183.78.37': 1, '248.167.253.94': 1, '68.50.85.147': 1, '52.19.47.175': 1, '37.25.235.65': 0, '49.226.203.201': 0, '50.64.233.35': 0, '206.54.47.5': 1, '0.85.204.112': 0, '76.103.42.107': 1, '44.250.18.55': 0, '52.2.86.168': 1, '223.122.45.221': 0, '114.39.207.75': 1, '185.32.101.136': 0, '45.93.188.156': 1, '61.186.71.219': 0, '2.212.124.217': 0, '198.92.71.192': 1, '107.189.6.17': 1, '200.83.193.109': 1, '19.252.155.174': 0, '252.243.63.215': 1, '49.138.219.0': 0, '173.103.196.247': 0, '194.153.84.134': 0, '2.209.48.250': 0, '49.132.63.19': 0, '116.107.2.7': 0, '185.57.41.213': 1, '82.101.43.184': 1, '196.178.215.251': 1, '235.59.44.200': 1, '172.125.170.109': 1, '50.142.0.0': 0, '45.119.113.21': 1, '28.221.101.233': 0, '44.229.1.219': 0, '28.156.20.104': 0, '4.39.69.244': 1, '43.129.40.48': 0, '46.116.144.29': 1, '216.20.73.40': 0, '55.162.85.180': 0, '50.229.36.206': 0, '165.147.181.54': 0, '106.139.5.161': 1, '199.54.73.8': 1, '65.210.225.244': 0, '200.33.233.94': 1, '46.209.71.217': 1, '36.181.240.56': 0, '113.87.229.171': 1, '34.65.121.94': 1, '46.191.243.101': 1, '4.240.153.205': 0, '50.174.156.100': 0, '254.13.119.154': 1, '104.147.116.140': 0, '28.179.223.228': 0, '60.215.36.43': 0, '59.81.241.144': 1, '61.63.217.83': 1, '35.32.19.87': 1, '36.22.35.233': 0, '28.130.230.202': 0, '97.27.32.24': 0, '17.209.32.115': 0, '96.53.246.82': 0, '52.13.236.4': 1, '99.229.161.49': 0, '219.3.182.60': 1, '190.212.136.119': 1, '113.90.50.118': 1, '37.98.147.95': 1, '116.23.230.129': 0, '37.46.169.208': 0, '118.52.93.212': 0, '2.183.255.204': 0, '114.108.81.15': 0, '106.6.32.44': 0, '36.83.173.235': 0, '90.235.165.216': 1, '112.18.202.123': 0, '190.106.61.13': 1, '36.73.203.217': 0, '113.95.119.41': 0, '181.90.194.237': 1, '50.223.140.226': 0, '2.185.35.221': 0, '37.54.45.42': 0, '200.73.102.151': 1, '184.204.203.200': 0, '102.215.171.4': 0, '49.133.99.96': 0, '53.248.52.10': 1, '107.177.217.183': 1, '219.172.40.115': 1, '49.154.168.160': 0, '61.23.53.37': 1, '28.143.9.148': 0, '17.41.103.218': 0, '50.199.98.120': 0, '74.129.172.109': 1, '112.24.43.107': 0, '55.50.172.93': 0, '207.44.126.111': 1, '7.229.242.44': 1, '74.22.226.130': 0, '161.227.0.129': 1, '224.108.123.67': 1, '175.109.133.45': 1, '203.90.0.247': 0, '37.62.169.175': 0, '50.89.63.255': 0, '2.181.197.14': 0, '86.242.35.121': 0, '87.163.172.135': 0, '30.20.121.214': 1, '28.98.204.228': 0, '63.10.27.249': 0, '180.1.211.240': 0, '46.113.239.115': 1, '49.187.210.107': 0, '37.3.217.113': 0, '99.152.165.140': 1, '167.58.60.183': 1, '111.90.101.124': 1, '104.17.46.230': 0, '109.198.246.133': 0, '109.226.87.126': 1, '30.39.125.179': 1, '209.78.32.56': 1, '223.134.24.200': 1, '200.74.187.239': 1, '193.78.6.233': 1, '43.130.234.189': 1, '247.209.158.144': 0, '28.169.46.36': 0, '62.115.246.237': 1, '211.44.153.137': 0, '184.82.74.179': 1, '50.183.1.163': 0, '194.194.207.237': 1, '37.50.225.10': 0, '191.248.101.215': 1, '4.44.254.241': 1, '17.57.201.247': 0, '241.220.97.213': 1, '184.32.215.12': 1, '49.148.85.99': 0, '201.248.254.194': 0, '42.54.120.167': 1, '173.181.165.193': 0, '28.128.123.18': 0, '202.19.30.135': 1, '50.230.211.173': 0, '160.174.220.62': 1, '190.15.12.52': 1, '120.72.108.173': 0, '99.41.82.93': 1, '216.92.64.111': 1, '116.31.184.176': 0, '99.160.247.197': 1, '202.193.217.16': 1, '166.192.63.168': 1, '36.66.214.119': 1, '46.180.226.146': 1, '127.224.36.254': 1, '206.2.6.188': 0, '30.17.223.51': 0, '17.40.106.102': 0, '17.225.27.209': 0, '42.99.112.117': 1, '19.169.90.166': 0, '178.165.63.131': 1, '219.5.157.83': 1, '2.216.148.136': 0, '190.243.70.174': 1, '30.21.247.93': 1, '50.198.241.51': 0, '164.6.148.125': 0, '28.38.75.80': 0, '212.164.114.145': 1, '247.193.124.32': 1, '27.253.245.179': 0, '30.30.95.120': 0, '36.180.48.132': 0, '182.89.139.3': 1, '52.2.160.9': 1, '254.25.28.25': 1, '62.166.27.166': 0, '50.133.97.45': 0, '17.212.213.160': 0, '49.245.36.247': 0, '94.178.191.135': 1, '161.197.55.150': 0, '77.127.119.52': 1, '50.169.251.145': 0, '218.20.36.140': 1, '80.203.157.31': 1, '96.220.45.199': 1, '35.105.176.139': 0, '88.234.65.205': 1, '28.175.253.0': 0, '112.22.125.5': 1, '23.18.100.173': 1, '221.143.104.136': 1, '114.51.220.74': 1, '77.211.44.65': 0, '199.25.109.66': 1, '196.79.124.57': 1, '254.8.15.70': 1, '44.250.7.2': 1, '120.91.64.177': 0, '61.120.61.210': 0, '104.230.111.8': 1, '221.79.32.195': 0, '238.250.5.13': 0, '37.41.117.195': 0, '200.18.74.200': 0, '70.190.165.210': 1, '216.171.242.3': 1, '84.122.11.247': 1, '37.143.204.121': 1, '32.227.237.39': 1, '126.23.167.214': 1, '30.24.242.54': 1, '36.34.47.43': 1, '70.131.6.233': 0, '36.160.176.150': 0, '181.116.112.226': 1, '4.39.136.30': 0, '184.39.99.2': 1, '121.152.40.24': 1, '49.163.4.52': 0, '99.111.137.72': 0, '178.129.250.55': 1, '50.131.58.39': 0, '192.165.64.207': 0, '106.245.69.85': 1, '108.118.205.50': 0, '0.90.71.87': 0, '82.144.123.25': 1, '59.33.17.134': 1, '175.24.51.233': 1, '171.39.146.164': 1, '70.38.65.242': 0, '196.154.17.34': 0, '50.235.229.41': 0, '86.81.69.121': 1, '63.198.171.117': 1, '58.8.161.189': 0, '186.160.246.247': 1, '55.3.255.126': 1, '50.72.234.252': 0, '11.99.74.200': 1, '107.27.109.55': 0, '2.218.102.134': 0, '33.36.106.253': 1, '183.194.171.122': 1, '194.242.190.86': 1, '231.147.19.43': 1, '187.154.209.208': 1, '28.47.87.126': 0, '167.26.66.82': 1, '87.168.225.20': 1, '43.200.161.47': 0, '235.29.250.166': 1, '187.142.238.42': 1, '165.40.122.69': 0, '50.218.76.241': 0, '17.50.9.42': 0, '107.175.87.45': 1, '91.208.23.195': 1, '220.172.220.38': 1, '194.140.117.97': 1, '49.201.123.57': 0, '28.33.182.157': 0, '121.154.85.12': 1, '113.202.52.41': 0, '172.243.127.25': 0, '99.171.201.1': 1, '30.60.204.167': 0, '36.74.59.92': 0, '21.223.114.26': 0, '123.30.187.239': 1, '49.227.160.155': 0, '117.13.21.216': 1, '206.153.205.85': 1, '50.195.22.183': 0, '119.165.3.190': 0, '184.168.9.110': 0, '215.190.43.74': 0, '50.245.50.80': 0, '71.54.239.99': 1, '61.42.14.203': 1, '42.10.61.186': 0, '99.82.194.148': 1, '2.129.151.104': 0, '17.209.9.43': 1, '46.122.235.4': 1, '223.35.74.222': 1, '167.109.225.180': 0, '99.106.28.215': 1, '118.191.82.238': 0, '50.79.129.165': 0, '111.50.194.12': 1, '112.242.90.229': 0, '1.192.185.61': 0, '58.26.131.187': 0, '108.119.194.143': 0, '186.206.156.218': 1, '29.185.122.68': 1, '91.240.63.81': 1, '80.65.228.206': 0, '99.249.80.239': 0, '50.163.116.115': 0, '171.51.238.133': 1, '33.44.12.122': 0, '204.43.230.162': 0, '195.66.11.169': 1, '70.44.151.209': 1, '65.211.152.148': 0, '84.31.96.226': 1, '219.60.166.242': 0, '37.86.136.128': 1, '99.172.127.9': 1, '46.124.103.176': 1, '185.99.165.59': 1, '163.87.185.224': 0, '185.221.180.137': 1, '44.208.13.124': 1, '215.192.93.163': 1, '113.95.71.249': 1, '99.148.222.222': 1, '30.58.8.249': 1, '204.237.145.212': 1, '76.96.109.140': 0, '112.232.235.79': 0, '35.118.104.252': 0, '200.116.215.214': 1, '119.230.15.7': 0, '52.252.222.235': 0, '80.241.135.38': 0, '119.10.111.146': 1, '4.182.101.16': 0, '44.1.251.109': 1, '4.225.187.229': 1, '213.68.122.137': 1, '46.204.160.220': 1, '235.6.191.131': 0, '207.230.54.110': 1, '238.153.60.10': 0, '50.75.199.44': 0, '171.10.10.107': 1, '203.248.58.99': 1, '89.19.42.182': 0, '30.87.199.127': 1, '202.107.122.84': 0, '127.159.207.172': 1, '198.141.86.223': 1, '50.165.244.57': 0, '19.210.136.207': 0, '91.233.255.229': 1, '46.188.7.69': 1, '216.143.174.244': 1, '36.94.9.211': 0, '206.233.132.43': 1, '173.114.243.123': 0, '7.215.119.140': 0, '197.21.4.226': 0, '28.165.250.208': 0, '114.112.17.171': 1, '74.138.15.237': 1, '172.56.30.3': 0, '52.161.157.97': 1, '62.174.9.168': 0, '82.147.101.52': 0, '126.164.144.253': 1, '235.176.66.233': 1, '89.56.31.51': 0, '221.247.232.165': 1, '185.163.125.236': 1, '50.74.237.186': 0, '53.74.241.34': 1, '180.54.245.67': 0, '36.4.190.190': 1, '187.206.241.246': 1, '17.56.245.62': 0, '87.197.181.55': 0, '17.93.214.10': 1, '171.191.244.169': 0, '4.172.210.240': 0, '46.20.125.179': 1, '17.126.102.10': 0, '242.151.12.47': 0, '104.211.252.81': 1, '185.101.60.98': 1, '181.243.23.5': 0, '162.110.192.71': 0, '161.229.26.232': 1, '104.236.9.242': 0, '28.129.197.138': 0, '109.231.119.56': 0, '99.235.59.121': 1, '94.191.185.115': 1, '99.194.82.146': 1, '184.125.208.225': 0, '55.238.36.145': 0, '36.102.61.18': 0, '204.161.174.10': 1, '21.109.11.216': 0, '99.252.30.116': 1, '4.187.143.247': 0, '161.213.123.10': 0, '248.89.69.146': 1, '119.78.178.21': 0, '2.200.81.56': 0, '117.83.47.182': 1, '161.198.239.33': 0, '63.190.54.70': 1, '119.241.223.236': 0, '109.225.246.4': 1, '107.12.22.26': 0, '2.179.29.156': 0, '250.213.212.119': 1, '172.9.17.28': 0, '235.202.4.119': 1, '50.247.131.138': 0, '17.53.177.1': 0, '62.244.228.254': 0, '116.22.210.158': 0, '200.128.138.26': 0, '192.239.173.222': 1, '200.124.134.228': 1, '164.107.178.167': 1, '2.134.234.141': 0, '37.49.130.86': 0, '200.64.60.55': 0, '117.17.58.51': 1, '104.130.4.129': 1, '163.201.251.146': 0, '200.38.139.44': 1, '250.7.23.33': 1, '121.116.175.191': 1, '239.246.117.190': 1, '87.195.10.126': 0, '120.179.245.71': 0, '30.61.203.27': 0, '28.34.56.153': 0, '2.72.173.137': 0, '99.227.84.140': 0, '28.162.164.47': 0, '99.236.213.54': 1, '214.222.74.121': 0, '92.127.206.225': 1, '173.102.107.111': 0, '225.243.252.92': 1, '121.211.142.234': 1, '209.14.236.244': 1, '123.132.152.170': 1, '215.169.31.182': 0, '221.244.82.6': 1, '214.199.190.21': 0, '96.160.211.67': 0, '43.154.35.185': 0, '200.75.33.86': 0, '36.2.185.52': 1, '2.233.242.87': 0, '50.67.54.190': 0, '30.239.23.122': 1, '119.47.25.83': 0, '200.31.152.132': 0, '218.24.199.225': 1, '223.110.171.202': 1, '23.247.246.116': 0, '28.222.127.93': 0, '184.150.148.11': 0, '62.162.124.40': 1, '79.83.238.132': 0, '47.48.210.120': 0, '181.128.241.16': 1, '50.39.17.232': 1, '19.253.55.37': 0, '170.98.138.231': 1, '216.152.41.194': 1, '90.209.70.116': 1, '46.5.143.196': 0, '185.25.254.117': 0, '219.61.131.222': 0, '123.12.187.71': 1, '254.186.109.180': 0, '50.219.161.61': 0, '2.162.98.253': 0, '166.194.142.192': 0, '211.234.244.110': 0, '70.23.123.149': 0, '237.238.145.67': 1, '242.65.164.246': 1, '185.168.2.235': 1, '49.168.203.154': 0, '235.201.111.224': 1, '113.130.58.89': 1, '119.79.3.247': 0, '172.224.193.122': 0, '50.171.25.178': 0, '185.113.203.130': 1, '36.176.44.218': 0, '209.76.30.225': 0, '36.167.242.8': 0, '120.41.93.227': 1, '118.53.24.2': 0, '106.242.84.53': 1, '50.73.212.182': 0, '235.173.137.124': 1, '113.165.88.185': 0, '200.57.19.88': 1, '208.60.119.110': 1, '193.223.160.12': 0, '163.233.155.26': 0, '122.91.121.188': 0, '106.206.134.35': 1, '180.42.183.117': 0, '163.236.22.45': 1, '172.68.247.110': 0, '47.37.75.89': 0, '233.185.176.246': 1, '113.81.142.32': 0, '2.157.126.160': 0, '4.223.87.227': 0, '117.251.102.173': 1, '21.98.212.112': 0, '99.103.87.135': 0, '11.140.215.84': 1, '77.160.207.84': 1, '81.222.120.111': 1, '37.44.94.103': 0, '28.133.101.101': 0, '50.65.28.96': 0, '2.217.253.12': 0, '35.64.15.72': 0, '35.48.119.12': 1, '204.191.17.77': 0, '119.227.113.131': 0, '52.127.104.143': 1, '177.15.2.23': 1, '171.19.239.132': 0, '174.112.121.136': 1, '122.103.92.66': 0, '36.54.160.103': 1, '172.58.81.184': 0, '107.178.152.0': 1, '36.79.123.173': 0, '234.200.183.109': 0, '18.217.2.46': 0, '37.28.164.201': 0, '82.209.195.42': 1, '173.112.145.190': 0, '37.138.140.244': 1, '192.30.64.159': 0, '106.171.187.18': 1, '230.209.163.48': 1, '252.230.74.159': 1, '29.184.58.124': 1, '197.19.12.244': 0, '28.172.107.140': 0, '49.202.89.230': 0, '171.90.100.55': 0, '99.67.159.75': 1, '50.78.193.152': 0, '184.90.170.31': 0, '58.187.130.122': 0, '253.236.15.244': 1, '45.153.145.5': 1, '59.63.164.4': 1, '28.157.157.52': 0, '53.86.95.16': 1, '50.93.8.76': 0, '248.222.250.5': 1, '95.164.222.57': 0, '104.30.172.154': 0, '173.246.184.92': 1, '84.9.74.199': 1, '17.43.14.60': 0, '58.9.207.35': 1, '100.131.35.240': 0, '187.229.99.216': 1, '2.221.127.45': 0, '200.177.255.255': 0, '43.129.45.105': 1, '221.146.127.128': 1, '2.219.251.254': 0, '45.118.145.247': 0, '28.0.227.60': 1, '252.101.156.40': 0, '50.130.240.71': 0, '221.246.120.177': 1, '181.225.45.31': 0, '44.217.6.178': 0, '47.35.27.220': 0, '127.209.247.113': 1, '177.10.109.112': 1, '30.16.113.216': 0, '50.206.171.162': 0, '67.31.75.239': 0, '182.49.96.195': 1, '113.167.167.227': 1, '49.43.145.216': 0, '108.72.145.34': 0, '50.70.4.76': 0, '50.179.212.117': 0, '49.46.115.177': 0, '28.180.147.191': 0, '18.192.38.92': 0, '88.4.159.131': 1, '61.38.137.4': 1, '37.160.56.156': 1, '26.131.42.206': 1, '165.80.227.86': 1, '69.55.144.55': 1, '183.158.164.110': 0, '28.168.95.122': 0, '33.100.124.223': 1, '127.104.223.153': 1, '179.185.118.219': 1, '200.147.242.3': 0, '46.84.70.203': 1, '164.111.155.65': 1, '120.112.145.177': 0, '82.38.47.202': 1, '93.123.221.228': 0, '112.19.131.44': 1, '114.32.92.214': 1, '160.204.35.191': 1, '81.255.78.79': 1, '46.126.115.209': 1, '165.151.3.66': 0, '181.186.10.119': 0, '1.70.247.22': 0, '120.173.25.183': 1, '53.12.63.134': 0, '187.95.102.130': 1, '200.20.24.15': 0, '37.144.13.137': 1, '219.5.241.7': 1, '121.192.245.188': 1, '225.216.16.137': 0, '201.88.124.246': 1, '171.227.9.169': 0, '49.186.123.1': 0, '166.145.44.203': 0, '211.247.6.119': 0, '200.110.32.248': 0, '172.38.214.177': 0, '92.16.164.54': 1, '211.102.100.247': 1, '185.27.53.7': 1, '126.109.190.60': 0, '55.143.32.241': 1, '90.18.87.53': 1, '77.210.94.65': 0, '17.114.192.239': 0, '234.55.134.188': 0, '201.217.148.232': 1, '36.95.122.181': 0, '117.18.196.152': 1, '176.101.165.113': 1, '55.232.193.41': 1, '37.189.127.19': 1, '113.92.180.104': 1, '162.88.54.132': 0, '50.188.218.140': 1, '35.187.92.229': 1, '106.53.79.243': 0, '90.223.235.3': 0, '200.50.108.117': 0, '240.216.147.237': 1, '165.105.140.194': 1, '111.164.125.217': 0, '113.57.45.14': 1, '40.243.56.189': 0, '86.99.82.75': 0, '219.144.190.69': 1, '50.170.214.94': 0, '114.45.7.144': 1, '166.74.232.13': 1, '74.0.57.82': 1, '104.210.119.150': 1, '166.6.234.184': 0, '77.249.9.92': 1, '15.233.122.252': 0, '161.69.204.139': 1, '19.175.44.224': 0, '119.112.68.154': 0, '225.114.87.178': 1, '30.225.198.198': 1, '50.177.79.128': 0, '19.174.187.106': 0, '233.225.25.24': 1, '200.72.227.66': 1, '116.175.103.164': 1, '45.87.4.209': 1, '255.59.172.177': 1, '99.162.1.245': 1, '81.78.39.88': 1, '1.71.143.132': 1, '49.218.51.134': 0, '213.165.180.110': 1, '32.229.137.207': 1, '198.62.157.189': 1, '185.215.8.227': 1, '234.8.65.103': 1, '185.123.203.120': 0, '123.56.37.76': 1, '96.198.51.5': 1, '190.242.200.42': 1, '174.100.154.116': 1, '30.44.115.157': 1, '110.216.26.21': 1, '49.177.133.143': 0, '119.224.111.139': 0, '49.180.119.36': 0, '107.174.115.152': 1, '200.78.128.254': 0, '228.142.124.202': 1, '36.75.180.137': 0, '118.254.2.72': 0, '252.243.158.87': 0, '186.26.252.100': 1, '28.173.216.121': 0, '68.50.74.226': 1, '120.249.248.75': 0, '181.143.215.219': 0, '184.20.244.80': 1, '62.126.171.139': 0, '220.82.251.143': 1, '4.230.215.82': 0, '28.35.168.156': 0, '180.56.207.115': 0, '226.13.42.93': 1, '50.192.112.193': 0, '189.155.229.157': 1, '193.184.228.117': 1, '36.16.61.167': 1, '216.7.127.159': 1, '17.213.234.116': 0, '106.240.215.201': 1, '114.51.40.99': 0, '210.154.51.203': 1, '228.237.23.52': 0, '242.226.153.44': 0, '99.168.110.143': 0, '184.206.100.145': 1, '37.33.108.106': 0, '202.33.47.119': 1, '213.181.133.126': 1, '43.131.110.22': 1, '84.9.74.114': 1, '37.146.136.173': 1, '50.217.1.3': 0, '120.82.238.82': 0, '234.105.200.242': 1, '106.52.121.70': 0, '49.131.129.49': 0, '69.237.135.42': 1, '41.237.97.197': 1, '50.238.96.196': 0, '27.188.157.72': 1, '50.134.92.139': 0, '104.205.143.7': 1, '44.15.69.44': 1, '92.193.235.151': 0, '79.39.21.2': 0, '64.243.191.103': 1, '9.116.22.184': 1, '17.208.78.118': 0, '166.118.197.133': 0, '181.252.251.33': 0, '50.175.75.134': 0, '29.191.145.48': 0, '28.160.196.83': 0, '171.161.193.102': 1, '17.115.1.175': 0, '239.209.151.231': 1, '229.205.75.111': 1, '172.162.226.20': 0, '49.237.138.40': 1, '238.33.213.241': 1, '11.127.208.136': 1, '197.35.209.151': 0, '163.83.91.22': 1, '220.251.239.26': 1, '209.101.240.63': 1, '50.168.18.3': 0, '28.159.11.54': 0, '94.189.131.252': 1, '44.12.39.162': 0, '171.38.236.249': 1, '37.52.88.56': 0, '116.253.121.61': 0, '107.173.251.105': 1, '230.5.134.167': 1, '28.176.26.82': 0, '49.188.203.197': 0, '170.218.92.176': 1, '27.68.107.218': 0, '15.255.231.87': 1, '43.241.14.224': 0, '194.223.93.203': 1, '28.232.156.73': 0, '127.56.92.55': 0, '43.196.97.116': 0, '200.109.158.45': 1, '240.125.189.138': 1, '23.23.132.13': 0, '30.85.252.178': 1, '187.26.86.138': 0, '44.251.167.91': 0, '87.248.84.101': 1, '252.247.222.160': 1, '185.168.204.116': 0, '212.223.67.94': 1, '200.134.193.7': 0, '61.23.53.50': 1, '46.19.227.29': 0, '93.45.35.26': 1, '36.89.247.112': 0, '84.16.81.169': 0, '121.17.69.36': 0, '212.33.172.159': 1, '224.5.238.110': 1, '162.250.188.138': 1, '50.140.158.7': 0, '104.199.140.161': 1, '17.51.75.95': 0, '2.163.127.46': 0, '58.131.178.9': 0, '79.204.136.103': 0, '2.166.72.80': 0, '42.247.226.128': 1, '35.126.102.67': 0, '55.209.40.218': 0, '34.198.218.175': 1, '216.158.117.144': 1, '61.183.6.74': 0, '112.25.138.121': 0, '18.246.142.224': 0, '245.225.92.134': 0, '179.185.123.133': 1, '46.163.96.75': 1, '55.12.42.150': 0, '49.244.233.122': 0, '17.48.197.76': 1, '35.57.148.255': 1, '64.76.97.204': 1, '44.10.31.65': 0, '18.67.148.89': 0, '19.168.230.4': 0, '243.235.195.19': 1, '179.242.138.240': 1, '40.247.121.35': 0, '177.189.153.54': 0, '17.42.127.96': 0, '36.162.173.10': 0, '76.110.39.22': 0, '99.54.72.212': 1, '231.125.107.206': 1, '87.189.160.120': 0, '64.252.231.232': 0, '171.77.131.74': 1, '49.159.41.215': 0, '107.170.244.190': 0, '50.166.200.63': 0, '37.60.105.9': 0, '162.100.95.96': 0, '46.216.57.91': 1, '126.17.174.153': 0, '196.105.22.212': 1, '50.135.121.39': 0, '225.126.120.40': 1, '46.189.16.213': 1, '2.207.66.174': 0, '199.88.171.200': 0, '170.206.37.21': 1}\n"
          ]
        }
      ],
      "source": [
        "result = dict()\n",
        "for ip, label in zip(predictIP_dict.keys(), predict_by_roc):\n",
        "    result[ip] = label\n",
        "\n",
        "print(result)"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3.9.15 ('rapidsmm-22.10')",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.15"
    },
    "vscode": {
      "interpreter": {
        "hash": "7ffb406c183ba063dacefccde3ed94bbb3c7bbe2af981d5d49f0c02d6e46e228"
      }
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "0c56f9b4b76549b087baadd6117316ca": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1d6850d9a77a4d6bac92525a5052300b": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "3ef6af2d6f4b4a1f9b5d94a8950f6e6c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_eced2cd5c3d84b5cbe878c4b794e7cba",
            "max": 32,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_f71264c80887469d910f985ca2344a67",
            "value": 32
          }
        },
        "4236487094c541edb3e1d47d1e83677b": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "82fb019c2bc6409ba16395fb9a264e4a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_9d84404bc36146f59cd544851e7bb477",
            "placeholder": "​",
            "style": "IPY_MODEL_90c81c96bbad4e53af72dbcb7f1abe74",
            "value": " 32/32 [08:12&lt;00:00, 15.54s/it, epoch=32, loss=9.807953284428692e-05]"
          }
        },
        "90c81c96bbad4e53af72dbcb7f1abe74": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "9d84404bc36146f59cd544851e7bb477": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c77f618494594a1191c3691921ba6187": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_cbaf6de49d324ae3b9765a71fa994b50",
              "IPY_MODEL_3ef6af2d6f4b4a1f9b5d94a8950f6e6c",
              "IPY_MODEL_82fb019c2bc6409ba16395fb9a264e4a"
            ],
            "layout": "IPY_MODEL_4236487094c541edb3e1d47d1e83677b"
          }
        },
        "cbaf6de49d324ae3b9765a71fa994b50": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0c56f9b4b76549b087baadd6117316ca",
            "placeholder": "​",
            "style": "IPY_MODEL_1d6850d9a77a4d6bac92525a5052300b",
            "value": "100%"
          }
        },
        "eced2cd5c3d84b5cbe878c4b794e7cba": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f71264c80887469d910f985ca2344a67": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}