COMP6130 Dimensionality Reduction.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"COMP6130 Dimensionality Reduction.ipynb","provenance":[{"file_id":"https://gist.github.com/amjadraza/461bad6be2b1de881565290ae9e12e8a#file-scala-for-data-science-and-machine-learning-ipynb","timestamp":1618962427022}],"collapsed_sections":[],"toc_visible":true},"kernelspec":{"name":"scala","display_name":"Scala"},"language_info":{"name":"scala"}},"cells":[{"cell_type":"markdown","metadata":{"id":"jjfXAQTw1eko"},"source":["# COMP6130 - Big Data Analytics (Semester II, 2021)\n","Group Members:\n","- Shane Miller (620042891)\n","- Vincent Irving\n","- Jermaine Sammuels\n"]},{"cell_type":"markdown","metadata":{"id":"7bols0uZ23E4"},"source":["# Dimensionality Reduction"]},{"cell_type":"markdown","metadata":{"id":"6Z6hfgEq4u86"},"source":["## Problem and Data Description"]},{"cell_type":"markdown","metadata":{"id":"aBw0J0U_5AY1"},"source":["### [Kaggle Competition - Home Credit Default Risk](https://www.kaggle.com/c/home-credit-default-risk/overview)\n","Many people struggle to get loans due to insufficient or non-existent credit histories. And, unfortunately, this population is often taken advantage of by untrustworthy lenders.\n","\n","**Home Credit Group**\n","\n","Home Credit strives to broaden financial inclusion for the unbanked population by providing a positive and safe borrowing experience. In order to make sure this underserved population has a positive loan experience, Home Credit makes use of a variety of alternative data--including telco and transactional information--to predict their clients' repayment abilities.\n","\n","While Home Credit is currently using various statistical and machine learning methods to make these predictions, they're challenging Kagglers to help them unlock the full potential of their data. Doing so will ensure that clients capable of repayment are not rejected and that loans are given with a principal, maturity, and repayment calendar that will empower their clients to be successful."]},{"cell_type":"markdown","metadata":{"id":"94hMwJV1GVNj"},"source":["### Data Dictionary"]},{"cell_type":"markdown","metadata":{"id":"mdqUTMbWGhW6"},"source":["| Row                             | Description                                                                                                                                                                                                                                                                             |\n","| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n","| SK\\_ID\\_CURR                    | ID of loan in our sample                                                                                                                                                                                                                                                                |\n","| TARGET                          | Target variable (1 - client with payment difficulties: he/she had late payment more than X days on at least one of the first Y installments of the loan in our sample, 0 - all other cases)                                                                                             |\n","| NAME\\_CONTRACT\\_TYPE            | Identification if loan is cash or revolving                                                                                                                                                                                                                                             |\n","| CODE\\_GENDER                    | Gender of the client                                                                                                                                                                                                                                                                    |\n","| FLAG\\_OWN\\_CAR                  | Flag if the client owns a car                                                                                                                                                                                                                                                           |\n","| FLAG\\_OWN\\_REALTY               | Flag if client owns a house or flat                                                                                                                                                                                                                                                     |\n","| CNT\\_CHILDREN                   | Number of children the client has                                                                                                                                                                                                                                                       |\n","| AMT\\_INCOME\\_TOTAL              | Income of the client                                                                                                                                                                                                                                                                    |\n","| AMT\\_CREDIT                     | Credit amount of the loan                                                                                                                                                                                                                                                               |\n","| AMT\\_ANNUITY                    | Loan annuity                                                                                                                                                                                                                                                                            |\n","| AMT\\_GOODS\\_PRICE               | For consumer loans it is the price of the goods for which the loan is given                                                                                                                                                                                                             |\n","| NAME\\_TYPE\\_SUITE               | Who was accompanying client when he was applying for the loan                                                                                                                                                                                                                           |\n","| NAME\\_INCOME\\_TYPE              | Clients income type (businessman, working, maternity leave,…)                                                                                                                                                                                                                           |\n","| NAME\\_EDUCATION\\_TYPE           | Level of highest education the client achieved                                                                                                                                                                                                                                          |\n","| NAME\\_FAMILY\\_STATUS            | Family status of the client                                                                                                                                                                                                                                                             |\n","| NAME\\_HOUSING\\_TYPE             | What is the housing situation of the client (renting, living with parents, ...)                                                                                                                                                                                                         |\n","| REGION\\_POPULATION\\_RELATIVE    | Normalized population of region where client lives (higher number means the client lives in more populated region)                                                                                                                                                                      |\n","| DAYS\\_BIRTH                     | Client's age in days at the time of application                                                                                                                                                                                                                                         |\n","| DAYS\\_EMPLOYED                  | How many days before the application the person started current employment                                                                                                                                                                                                              |\n","| DAYS\\_REGISTRATION              | How many days before the application did client change his registration                                                                                                                                                                                                                 |\n","| DAYS\\_ID\\_PUBLISH               | How many days before the application did client change the identity document with which he applied for the loan                                                                                                                                                                         |\n","| OWN\\_CAR\\_AGE                   | Age of client's car                                                                                                                                                                                                                                                                     |\n","| FLAG\\_MOBIL                     | Did client provide mobile phone (1=YES, 0=NO)                                                                                                                                                                                                                                           |\n","| FLAG\\_EMP\\_PHONE                | Did client provide work phone (1=YES, 0=NO)                                                                                                                                                                                                                                             |\n","| FLAG\\_WORK\\_PHONE               | Did client provide home phone (1=YES, 0=NO)                                                                                                                                                                                                                                             |\n","| FLAG\\_CONT\\_MOBILE              | Was mobile phone reachable (1=YES, 0=NO)                                                                                                                                                                                                                                                |\n","| FLAG\\_PHONE                     | Did client provide home phone (1=YES, 0=NO)                                                                                                                                                                                                                                             |\n","| FLAG\\_EMAIL                     | Did client provide email (1=YES, 0=NO)                                                                                                                                                                                                                                                  |\n","| OCCUPATION\\_TYPE                | What kind of occupation does the client have                                                                                                                                                                                                                                            |\n","| CNT\\_FAM\\_MEMBERS               | How many family members does client have                                                                                                                                                                                                                                                |\n","| REGION\\_RATING\\_CLIENT          | Our rating of the region where client lives (1,2,3)                                                                                                                                                                                                                                     |\n","| REGION\\_RATING\\_CLIENT\\_W\\_CITY | Our rating of the region where client lives with taking city into account (1,2,3)                                                                                                                                                                                                       |\n","| WEEKDAY\\_APPR\\_PROCESS\\_START   | On which day of the week did the client apply for the loan                                                                                                                                                                                                                              |\n","| HOUR\\_APPR\\_PROCESS\\_START      | Approximately at what hour did the client apply for the loan                                                                                                                                                                                                                            |\n","| REG\\_REGION\\_NOT\\_LIVE\\_REGION  | Flag if client's permanent address does not match contact address (1=different, 0=same, at region level)                                                                                                                                                                                |\n","| REG\\_REGION\\_NOT\\_WORK\\_REGION  | Flag if client's permanent address does not match work address (1=different, 0=same, at region level)                                                                                                                                                                                   |\n","| LIVE\\_REGION\\_NOT\\_WORK\\_REGION | Flag if client's contact address does not match work address (1=different, 0=same, at region level)                                                                                                                                                                                     |\n","| REG\\_CITY\\_NOT\\_LIVE\\_CITY      | Flag if client's permanent address does not match contact address (1=different, 0=same, at city level)                                                                                                                                                                                  |\n","| REG\\_CITY\\_NOT\\_WORK\\_CITY      | Flag if client's permanent address does not match work address (1=different, 0=same, at city level)                                                                                                                                                                                     |\n","| LIVE\\_CITY\\_NOT\\_WORK\\_CITY     | Flag if client's contact address does not match work address (1=different, 0=same, at city level)                                                                                                                                                                                       |\n","| ORGANIZATION\\_TYPE              | Type of organization where client works                                                                                                                                                                                                                                                 |\n","| EXT\\_SOURCE\\_1                  | Normalized score from external data source                                                                                                                                                                                                                                              |\n","| EXT\\_SOURCE\\_2                  | Normalized score from external data source                                                                                                                                                                                                                                              |\n","| EXT\\_SOURCE\\_3                  | Normalized score from external data source                                                                                                                                                                                                                                              |\n","| APARTMENTS\\_AVG                 | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| BASEMENTAREA\\_AVG               | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| YEARS\\_BEGINEXPLUATATION\\_AVG   | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| YEARS\\_BUILD\\_AVG               | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| COMMONAREA\\_AVG                 | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| ELEVATORS\\_AVG                  | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| ENTRANCES\\_AVG                  | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| FLOORSMAX\\_AVG                  | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| FLOORSMIN\\_AVG                  | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| LANDAREA\\_AVG                   | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| LIVINGAPARTMENTS\\_AVG           | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| LIVINGAREA\\_AVG                 | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| NONLIVINGAPARTMENTS\\_AVG        | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| NONLIVINGAREA\\_AVG              | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| APARTMENTS\\_MODE                | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| BASEMENTAREA\\_MODE              | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| YEARS\\_BEGINEXPLUATATION\\_MODE  | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| YEARS\\_BUILD\\_MODE              | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| COMMONAREA\\_MODE                | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| ELEVATORS\\_MODE                 | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| ENTRANCES\\_MODE                 | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| FLOORSMAX\\_MODE                 | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| FLOORSMIN\\_MODE                 | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| LANDAREA\\_MODE                  | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| LIVINGAPARTMENTS\\_MODE          | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| LIVINGAREA\\_MODE                | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| NONLIVINGAPARTMENTS\\_MODE       | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| NONLIVINGAREA\\_MODE             | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| APARTMENTS\\_MEDI                | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| BASEMENTAREA\\_MEDI              | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| YEARS\\_BEGINEXPLUATATION\\_MEDI  | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| YEARS\\_BUILD\\_MEDI              | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| COMMONAREA\\_MEDI                | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| ELEVATORS\\_MEDI                 | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| ENTRANCES\\_MEDI                 | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| FLOORSMAX\\_MEDI                 | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| FLOORSMIN\\_MEDI                 | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| LANDAREA\\_MEDI                  | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| LIVINGAPARTMENTS\\_MEDI          | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| LIVINGAREA\\_MEDI                | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| NONLIVINGAPARTMENTS\\_MEDI       | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| NONLIVINGAREA\\_MEDI             | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| FONDKAPREMONT\\_MODE             | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| HOUSETYPE\\_MODE                 | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| TOTALAREA\\_MODE                 | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| WALLSMATERIAL\\_MODE             | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| EMERGENCYSTATE\\_MODE            | Normalized information about building where the client lives, What is average (\\_AVG suffix), modus (\\_MODE suffix), median (\\_MEDI suffix) apartment size, common area, living area, age of building, number of elevators, number of entrances, state of the building, number of floor |\n","| OBS\\_30\\_CNT\\_SOCIAL\\_CIRCLE    | How many observation of client's social surroundings with observable 30 DPD (days past due) default                                                                                                                                                                                     |\n","| DEF\\_30\\_CNT\\_SOCIAL\\_CIRCLE    | How many observation of client's social surroundings defaulted on 30 DPD (days past due)                                                                                                                                                                                                |\n","| OBS\\_60\\_CNT\\_SOCIAL\\_CIRCLE    | How many observation of client's social surroundings with observable 60 DPD (days past due) default                                                                                                                                                                                     |\n","| DEF\\_60\\_CNT\\_SOCIAL\\_CIRCLE    | How many observation of client's social surroundings defaulted on 60 (days past due) DPD                                                                                                                                                                                                |\n","| DAYS\\_LAST\\_PHONE\\_CHANGE       | How many days before application did client change phone                                                                                                                                                                                                                                |\n","| FLAG\\_DOCUMENT\\_2               | Did client provide document 2                                                                                                                                                                                                                                                           |\n","| FLAG\\_DOCUMENT\\_3               | Did client provide document 3                                                                                                                                                                                                                                                           |\n","| FLAG\\_DOCUMENT\\_4               | Did client provide document 4                                                                                                                                                                                                                                                           |\n","| FLAG\\_DOCUMENT\\_5               | Did client provide document 5                                                                                                                                                                                                                                                           |\n","| FLAG\\_DOCUMENT\\_6               | Did client provide document 6                                                                                                                                                                                                                                                           |\n","| FLAG\\_DOCUMENT\\_7               | Did client provide document 7                                                                                                                                                                                                                                                           |\n","| FLAG\\_DOCUMENT\\_8               | Did client provide document 8                                                                                                                                                                                                                                                           |\n","| FLAG\\_DOCUMENT\\_9               | Did client provide document 9                                                                                                                                                                                                                                                           |\n","| FLAG\\_DOCUMENT\\_10              | Did client provide document 10                                                                                                                                                                                                                                                          |\n","| FLAG\\_DOCUMENT\\_11              | Did client provide document 11                                                                                                                                                                                                                                                          |\n","| FLAG\\_DOCUMENT\\_12              | Did client provide document 12                                                                                                                                                                                                                                                          |\n","| FLAG\\_DOCUMENT\\_13              | Did client provide document 13                                                                                                                                                                                                                                                          |\n","| FLAG\\_DOCUMENT\\_14              | Did client provide document 14                                                                                                                                                                                                                                                          |\n","| FLAG\\_DOCUMENT\\_15              | Did client provide document 15                                                                                                                                                                                                                                                          |\n","| FLAG\\_DOCUMENT\\_16              | Did client provide document 16                                                                                                                                                                                                                                                          |\n","| FLAG\\_DOCUMENT\\_17              | Did client provide document 17                                                                                                                                                                                                                                                          |\n","| FLAG\\_DOCUMENT\\_18              | Did client provide document 18                                                                                                                                                                                                                                                          |\n","| FLAG\\_DOCUMENT\\_19              | Did client provide document 19                                                                                                                                                                                                                                                          |\n","| FLAG\\_DOCUMENT\\_20              | Did client provide document 20                                                                                                                                                                                                                                                          |\n","| FLAG\\_DOCUMENT\\_21              | Did client provide document 21                                                                                                                                                                                                                                                          |\n","| AMT\\_REQ\\_CREDIT\\_BUREAU\\_HOUR  | Number of enquiries to Credit Bureau about the client one hour before application                                                                                                                                                                                                       |\n","| AMT\\_REQ\\_CREDIT\\_BUREAU\\_DAY   | Number of enquiries to Credit Bureau about the client one day before application (excluding one hour before application)                                                                                                                                                                |\n","| AMT\\_REQ\\_CREDIT\\_BUREAU\\_WEEK  | Number of enquiries to Credit Bureau about the client one week before application (excluding one day before application)                                                                                                                                                                |\n","| AMT\\_REQ\\_CREDIT\\_BUREAU\\_MON   | Number of enquiries to Credit Bureau about the client one month before application (excluding one week before application)                                                                                                                                                              |\n","| AMT\\_REQ\\_CREDIT\\_BUREAU\\_QRT   | Number of enquiries to Credit Bureau about the client 3 month before application (excluding one month before application)                                                                                                                                                               |\n","| AMT\\_REQ\\_CREDIT\\_BUREAU\\_YEAR  | Number of enquiries to Credit Bureau about the client one day year (excluding last 3 months before application)                                                                                                                                                                         |"]},{"cell_type":"markdown","metadata":{"id":"S-kyiD5XJurs"},"source":["## Preliminary Activities"]},{"cell_type":"markdown","metadata":{"id":"sL1Vq6r6J6dA"},"source":["### Install the Scala Kernel on Google Colab\n","\n","The tutorial [Machine Learning with Scala in Google Colaboratory](https://medium.com/@shadaj/machine-learning-with-scala-in-google-colaboratory-e6f1661f1c88) by [Shadad Laddad](https://medium.com/@shadaj) outlines how the Scala kernel can be run in the google colab. The required code is also in the cells below.\n","\n","If you get a \"scala\" kernel not recognized warning when loading up the notebook for the first time, start by running the two cells below. Once you are done **reload the page** to load the notebook in the installed Scala kernel (also select scala kernel and not python)."]},{"cell_type":"code","metadata":{"id":"RoCcdvTCAqeR","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1619177666836,"user_tz":300,"elapsed":8153,"user":{"displayName":"Shane Miller","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GghBlg9vmzVSt3TEaPCSnTq_vxK8ZVRde3JtEL_Mw=s64","userId":"09995344781883736850"}},"outputId":"6780cfd6-adbe-433a-fdad-57dc251a0ead"},"source":["%%shell\n","SCALA_VERSION=2.12.8 ALMOND_VERSION=0.3.0+16-548dc10f-SNAPSHOT\n","curl -Lo coursier https://git.io/coursier-cli\n","chmod +x coursier\n","./coursier bootstrap \\\n","    -r jitpack -r sonatype:snapshots \\\n","    -i user -I user:sh.almond:scala-kernel-api_$SCALA_VERSION:$ALMOND_VERSION \\\n","    sh.almond:scala-kernel_$SCALA_VERSION:$ALMOND_VERSION \\\n","    --sources --default=true \\\n","    -o almond-snapshot --embed-files=false\n","rm coursier\n","./almond-snapshot --install --global --force\n","rm almond-snapshot"],"execution_count":null,"outputs":[{"output_type":"stream","text":["  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n","                                 Dload  Upload   Total   Spent    Left  Speed\n","  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\n","100   135  100   135    0     0    269      0 --:--:-- --:--:-- --:--:--   269\n","100 42577  100 42577    0     0  75894      0 --:--:-- --:--:-- --:--:-- 75894\n","Wrote /content/almond-snapshot\n","Installed scala kernel under /usr/local/share/jupyter/kernels/scala\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/plain":[""]},"metadata":{"tags":[]},"execution_count":5}]},{"cell_type":"code","metadata":{"id":"BZchIo9sArF4","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1619177745815,"user_tz":300,"elapsed":317,"user":{"displayName":"Shane Miller","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GghBlg9vmzVSt3TEaPCSnTq_vxK8ZVRde3JtEL_Mw=s64","userId":"09995344781883736850"}},"outputId":"22c1ff50-dc64-4418-bbe4-dc056d6a5923"},"source":["%%shell\n","echo \"{\n","  \\\"language\\\" : \\\"scala\\\",\n","  \\\"display_name\\\" : \\\"Scala\\\",\n","  \\\"argv\\\" : [\n","    \\\"bash\\\",\n","    \\\"-c\\\",\n","    \\\"env LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libpython3.6m.so:\\$LD_PRELOAD java -jar /usr/local/share/jupyter/kernels/scala/launcher.jar --connection-file {connection_file}\\\"\n","  ]\n","}\" > /usr/local/share/jupyter/kernels/scala/kernel.json"],"execution_count":12,"outputs":[{"output_type":"execute_result","data":{"text/plain":[""]},"metadata":{"tags":[]},"execution_count":12}]},{"cell_type":"code","metadata":{"id":"pdjsD-JBhBEt"},"source":[""],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"kQe78pN0OSuq"},"source":["### Scala Instalation Check\n","\n","Lets check if the scala is working or not. if it does not work refresh the page."]},{"cell_type":"code","metadata":{"id":"a9z0PxRg_iOp","colab":{"base_uri":"https://localhost:8080/","height":168},"executionInfo":{"status":"error","timestamp":1619177743643,"user_tz":300,"elapsed":551,"user":{"displayName":"Shane Miller","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GghBlg9vmzVSt3TEaPCSnTq_vxK8ZVRde3JtEL_Mw=s64","userId":"09995344781883736850"}},"outputId":"39656214-5e47-43e9-9cb3-e106ef446a57"},"source":["println(\"Hello, world!\")"],"execution_count":11,"outputs":[{"output_type":"error","ename":"NameError","evalue":"ignored","traceback":["\u001b[0;31m---------------------------------------------------------------------------\u001b[0m","\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)","\u001b[0;32m<ipython-input-11-55dcecfe56a8>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mprintln\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Hello, world!\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m","\u001b[0;31mNameError\u001b[0m: name 'println' is not defined"]}]},{"cell_type":"markdown","metadata":{"id":"R8Y0sp6fa604"},"source":[""]},{"cell_type":"markdown","metadata":{"id":"PeRzUzt0hIVc"},"source":["### Mount Google Drive "]},{"cell_type":"code","metadata":{"id":"UusQybIEu7Ox"},"source":["from google.colab import drive\n","\n","currentWorkingDir = !pwd\n","defaultWorkingDir = \"/content\"\n","\n","if ( currentWorkingDir[0] == defaultWorkingDir ): \n","\n","  drive.mount('/content/drive')\n","      \n","  %cd \"/content/drive/My Drive/Colab Notebooks/stock_portfolio\"\n","else:\n","  print(\"Currenting running app from: \")\n","  !pwd"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"JSje3CTkaaoz"},"source":["### Import datafile"]},{"cell_type":"markdown","metadata":{"id":"APzCTOnVGG81"},"source":["## Topic Introduction/Overview"]},{"cell_type":"markdown","metadata":{"id":"2ZIQZDhD3Fbo"},"source":["### What is considered to be High Dimensional Data?"]},{"cell_type":"markdown","metadata":{"id":"qcN6wZKo4Ths"},"source":["### What is Reduction of Dimensionality?"]},{"cell_type":"markdown","metadata":{"id":"G_1UHCGB4e1S"},"source":["### What is referred to as the Curse of Dimensionality?"]},{"cell_type":"markdown","metadata":{"id":"7CEJjHinHz_u"},"source":["## Strategies/approaches to Dimensionality Reduction"]},{"cell_type":"markdown","metadata":{"id":"jSmXndaCIZSi"},"source":["1. ### Strategy one (1)"]},{"cell_type":"markdown","metadata":{"id":"ZX-uy_8EIg3B"},"source":["2. ### Strategy two (2)"]},{"cell_type":"markdown","metadata":{"id":"r2xGz0gGIpvG"},"source":["3. ### Strategy three (3)"]},{"cell_type":"markdown","metadata":{"id":"YPG6tqA3I08M"},"source":["4. ### Principal Componet Analysis (PCA)"]},{"cell_type":"markdown","metadata":{"id":"XccwxfBYI7KT"},"source":["5. ### Singular Value Decomposition (SVD)"]}]}