From 266ddf55498f68625c513bb9291e4098feb3fbdc Mon Sep 17 00:00:00 2001 From: Samantha Feinstein Date: Mon, 26 Sep 2022 16:55:00 -0400 Subject: [PATCH 1/2] Updating authentication I was receiving an error when running the code below. Based on https://stackoverflow.com/a/71711274 I updated the notebook and found it worked for me. --- ... Satisfaction and GDP_Extended Version with Evaluation.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Week_01 - Introduction to Machine Learning/People Satisfaction and GDP_Extended Version with Evaluation.ipynb b/Week_01 - Introduction to Machine Learning/People Satisfaction and GDP_Extended Version with Evaluation.ipynb index 032aca3..0475355 100644 --- a/Week_01 - Introduction to Machine Learning/People Satisfaction and GDP_Extended Version with Evaluation.ipynb +++ b/Week_01 - Introduction to Machine Learning/People Satisfaction and GDP_Extended Version with Evaluation.ipynb @@ -1 +1 @@ -{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"People Satisfaction and GDP.ipynb","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"}},"cells":[{"cell_type":"markdown","metadata":{"id":"gseS97WXHRAJ"},"source":["[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1CkvVzG1NtJABPkPRofGLT-MXUCj7veLo#scrollTo=gseS97WXHRAJ)\n"]},{"cell_type":"markdown","metadata":{"id":"-OMhwXX0Euz1"},"source":["# Predict People Satisfaction Across the Globe\n","\n","**Objective:**\n","We would like to build a model that predicts satisfaction score for people of different countries given their country GDP."]},{"cell_type":"markdown","metadata":{"id":"-sSMTG_Buxcm"},"source":["#Note 1: How to enable code completion:\n","\n","Tools menu ==> click on settings ==> Editor ==> Enable \"Automatically trigger code completions\"\n","\n","\n"]},{"cell_type":"markdown","metadata":{"id":"x9L7_OwZix-k"},"source":["#Note 2: Instructions to create a copy of this notebook for youtself\n","\n","You do not have write access to this notebook.\n","\n","* From Menu bar, Go to File, \n","* Select \"Save a copy in my Drive\"\n","* Navigate to Google Drive\n","* Find a folder named \"Collab Notebook\" and open it to find your notebook. \n","* Rename it and start making changes.\n","\n","**Note:** If there is any file you should read in your code, make sure you copy the file from instructor folder to your own Gdrive by following below steps:\n","\n","* Right clicking on the file name \n","* Select \"Make a copy\"\n","* Click on the new file\n","* Move it to desired folder, preferrably where you have your notebook"]},{"cell_type":"markdown","metadata":{"id":"WaAyuEL-EfXy"},"source":["# Download Dataset\n","\n","Download the Better Life Index data (latest edition, currently it is 2017) from the [OECD’s website](http://homl.info/4) as well as stats about GDP per capita from the [IMF’s website](http://homl.info/5). Then you join the tables and sort by GDP per capita. "]},{"cell_type":"markdown","metadata":{"id":"ZWasUlHjGGHc"},"source":["# Import Dataset to Google Colab\n","\n","1. Download CSV and XLS files to your computer\n","2. Upload them to your Google Drive\n","3. Open the CSV files using Google Sheets so Google will create the dataset in format of Google Sheets\n","4. You can remove CSV and XLS files from your drive\n","5. Use the step by step guide from [here](https://colab.research.google.com/notebooks/io.ipynb#scrollTo=vz-jH8T_Uk2c) and scroll down to **\" Google Sheets\" ** cell to import data into dataframe\n","\n","NOTE: After creating Google Sheet into your Drive, make sure you are converting Column 2015 to 0.00 format before importing it into Colab otherwise Google will import it as a string and you will have hard time to clean the data\n","\n","\n"]},{"cell_type":"code","metadata":{"id":"RhyqGUE9GFTF","executionInfo":{"status":"ok","timestamp":1630462416669,"user_tz":240,"elapsed":115,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"}}},"source":["# Run below line of code for the first time to install gspread. Once installed comment it for future use\n","#!pip install --upgrade -q gspread\n","from google.colab import auth\n","auth.authenticate_user()\n","\n","import gspread\n","from oauth2client.client import GoogleCredentials\n","\n","gc = gspread.authorize(GoogleCredentials.get_application_default())"],"execution_count":6,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"1QrOorPEHbDg"},"source":["# Use gc to open Google Sheet Datasets"]},{"cell_type":"code","metadata":{"id":"4seJiwQyEYVG","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1630462418665,"user_tz":240,"elapsed":1879,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"}},"outputId":"e88a402a-3ac5-49bc-de2b-bc4f89ff08f2"},"source":["#Open given sheet\n","worksheet = gc.open('BLI_30012019054825599').sheet1\n","\n","# Read contents of CSV file\n","bli_rows = worksheet.get_all_values()\n","\n","# Convert to a DataFrame and render.\n","import pandas as pd\n","bli = pd.DataFrame.from_records(bli_rows, columns = bli_rows[0])\n","\n","# Remove rows where inequality has values other than TOT\n","bli = bli[bli[\"INEQUALITY\"]==\"TOT\"]\n","\n","# Reformat data based on \"indicator column\"\n","bli = bli.pivot(index=\"Country\", columns=\"Indicator\", values=\"Value\")\n","#bli.head()\n","bli[\"Life satisfaction\"].head()"],"execution_count":7,"outputs":[{"output_type":"execute_result","data":{"text/plain":["Country\n","Australia 7.3\n","Austria 7\n","Belgium 6.9\n","Brazil 6.6\n","Canada 7.3\n","Name: Life satisfaction, dtype: object"]},"metadata":{},"execution_count":7}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":360},"id":"PJ42_kqLt7Zq","executionInfo":{"status":"ok","timestamp":1630462418791,"user_tz":240,"elapsed":133,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"}},"outputId":"c48e3c42-99a5-4e29-e524-074c1d0500a7"},"source":["bli[0:5]"],"execution_count":8,"outputs":[{"output_type":"execute_result","data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
IndicatorAir pollutionDwellings without basic facilitiesEducational attainmentEmployees working very long hoursEmployment rateFeeling safe walking alone at nightHomicide rateHousehold net adjusted disposable incomeHousehold net financial wealthHousing expenditureLabour market insecurityLife expectancyLife satisfactionLong-term unemployment ratePersonal earningsQuality of support networkRooms per personSelf-reported healthStakeholder engagement for developing regulationsStudent skillsTime devoted to leisure and personal careVoter turnoutWater qualityYears in education
Country
Australia51.18013.27263.613341757462204.382.57.31.3652063942.3852.750214.35919221.2
Austria161856.787280.70.43254459574212.781.371.9448295921.6701.349214.55759317.1
Belgium152.3754.316270.7129968104084214.881.16.93.9849587922.2752.250315.77898418.2
Brazil106.7497.156437.327.6122277102204.974.76.63.3714024900.8702.239514.45797215.9
Canada70.2913.737380.91.42985085758223.981.57.30.8148403932.588352314.41689116.7
\n","
"],"text/plain":["Indicator Air pollution ... Years in education\n","Country ... \n","Australia 5 ... 21.2\n","Austria 16 ... 17.1\n","Belgium 15 ... 18.2\n","Brazil 10 ... 15.9\n","Canada 7 ... 16.7\n","\n","[5 rows x 24 columns]"]},"metadata":{},"execution_count":8}]},{"cell_type":"code","metadata":{"id":"01edCO9kHJ2p","colab":{"base_uri":"https://localhost:8080/","height":236},"executionInfo":{"status":"ok","timestamp":1630462419986,"user_tz":240,"elapsed":1198,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"}},"outputId":"13ea178f-cce1-4001-e5fe-a5a04840acaa"},"source":["#Open given sheet\n","worksheet = gc.open('WEO_Data').sheet1\n","\n","# Read contents of CSV file\n","WEO_rows = worksheet.get_all_values()\n","\n","# Convert to a DataFrame and render.\n","import pandas as pd\n","weo = pd.DataFrame.from_records(WEO_rows, columns = WEO_rows[0])\n","\n","# Drop the header row from data\n","weo = weo.reindex(weo.index.drop(0))\n","\n","# 1- Select only Country name and 2015 \n","# 2- then rename it to GDP Per capita\n","weo = weo[['Country','2015']].rename(columns={'2015':'GDP per capita'})\n","\n","# Set Country as index column\n","# Inplace command, will replace the results of command into the same DF\n","weo.set_index(\"Country\", inplace=True)\n","\n","#weo.drop_duplicates(inplace=True)\n","#Print top 5 rows\n","weo.head()"],"execution_count":9,"outputs":[{"output_type":"execute_result","data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
GDP per capita
Country
Afghanistan599.99
Albania3995.38
Algeria4318.14
Angola4100.32
Antigua and Barbuda14414.30
\n","
"],"text/plain":[" GDP per capita\n","Country \n","Afghanistan 599.99\n","Albania 3995.38\n","Algeria 4318.14\n","Angola 4100.32\n","Antigua and Barbuda 14414.30"]},"metadata":{},"execution_count":9}]},{"cell_type":"markdown","metadata":{"id":"XX_8yJTKNPkt"},"source":["# Merge/Join dataset"]},{"cell_type":"code","metadata":{"id":"ZibYb4W-HuEy","colab":{"base_uri":"https://localhost:8080/","height":360},"executionInfo":{"status":"ok","timestamp":1630462420472,"user_tz":240,"elapsed":127,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"}},"outputId":"d943679b-bf91-4a18-cb4f-30a6ac28e098"},"source":["# Now merge BLI and WEO datasets\n","df = pd.merge(left = weo, right = bli, left_index=True, right_index=True)\n","\n","#sort the dataframe by GPD per capita\n","df.sort_values(by=\"GDP per capita\", inplace=True)\n","df.head()\n"],"execution_count":10,"outputs":[{"output_type":"execute_result","data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
GDP per capitaAir pollutionDwellings without basic facilitiesEducational attainmentEmployees working very long hoursEmployment rateFeeling safe walking alone at nightHomicide rateHousehold net adjusted disposable incomeHousehold net financial wealthHousing expenditureLabour market insecurityLife expectancyLife satisfactionLong-term unemployment ratePersonal earningsQuality of support networkRooms per personSelf-reported healthStakeholder engagement for developing regulationsStudent skillsTime devoted to leisure and personal careVoter turnoutWater qualityYears in education
Country
Luxembourg101994.09120793.7666720.64131774141203.282.46.91.962636922701.548315.15918515.1
Hungary12239.89194.3833.056750.71.21682123289184.875.75.32.4221711841.2561.247415.06627616.6
Poland12495.33222.7916.686566.30.81890614997234.377.662.1425921891.1582.650414.42558017.7
Chile13340.91169.46510.066251.14.51658821409188.179.16.72.0228434841.9571.544314.9496917.3
Latvia13618.571112.9892.096960.76.61526917105236.874.65.93.7322389861.2462.448713.83597717.9
\n","
"],"text/plain":[" GDP per capita Air pollution ... Water quality Years in education\n","Country ... \n","Luxembourg 101994.09 12 ... 85 15.1\n","Hungary 12239.89 19 ... 76 16.6\n","Poland 12495.33 22 ... 80 17.7\n","Chile 13340.91 16 ... 69 17.3\n","Latvia 13618.57 11 ... 77 17.9\n","\n","[5 rows x 25 columns]"]},"metadata":{},"execution_count":10}]},{"cell_type":"code","metadata":{"id":"eoJn-lxSPfqP","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1630462420888,"user_tz":240,"elapsed":4,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"}},"outputId":"543cdb12-e9a2-40cd-a9e5-48b5f0501525"},"source":["df.iloc[3]"],"execution_count":11,"outputs":[{"output_type":"execute_result","data":{"text/plain":["GDP per capita 13340.91\n","Air pollution 16\n","Dwellings without basic facilities 9.4\n","Educational attainment 65\n","Employees working very long hours 10.06\n","Employment rate 62\n","Feeling safe walking alone at night 51.1\n","Homicide rate 4.5\n","Household net adjusted disposable income 16588\n","Household net financial wealth 21409\n","Housing expenditure 18\n","Labour market insecurity 8.1\n","Life expectancy 79.1\n","Life satisfaction 6.7\n","Long-term unemployment rate 2.02\n","Personal earnings 28434\n","Quality of support network 84\n","Rooms per person 1.9\n","Self-reported health 57\n","Stakeholder engagement for developing regulations 1.5\n","Student skills 443\n","Time devoted to leisure and personal care 14.9\n","Voter turnout 49\n","Water quality 69\n","Years in education 17.3\n","Name: Chile, dtype: object"]},"metadata":{},"execution_count":11}]},{"cell_type":"markdown","metadata":{"id":"MkSTXnclPGLd"},"source":["## Split dataset into Train & Test"]},{"cell_type":"code","metadata":{"id":"6HOaSt93Nqkz","executionInfo":{"status":"ok","timestamp":1630462423352,"user_tz":240,"elapsed":2,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"}}},"source":["# Below snippet is the most basic way of splitting data in Python; It is for illustration only and in \n","# later sections we will use proper library from Sklearn to split the data\n","test_indices = [0, 1, 6, 8, 33, 34, 35]\n","\n","#define the indices of the training set by substracting the test indices from all indices\n","train_indices = list(set(range(36)) - set(test_indices))\n","\n","#use the above indices to select parts of the dataframe as the training set and the test set\n","train = df[[\"GDP per capita\", 'Life satisfaction']].iloc[train_indices]\n","test = df[[\"GDP per capita\", 'Life satisfaction']].iloc[test_indices]"],"execution_count":12,"outputs":[]},{"cell_type":"code","metadata":{"id":"9CRVFFEX_voT","colab":{"base_uri":"https://localhost:8080/","height":236},"executionInfo":{"status":"ok","timestamp":1630462431988,"user_tz":240,"elapsed":116,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"}},"outputId":"fd22091b-f2d0-42ef-f5c2-ad591827f287"},"source":["train.head()"],"execution_count":13,"outputs":[{"output_type":"execute_result","data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
GDP per capitaLife satisfaction
Country
Poland12495.336
Chile13340.916.7
Latvia13618.575.9
Slovak Republic15991.746.1
Estonia17288.085.6
\n","
"],"text/plain":[" GDP per capita Life satisfaction\n","Country \n","Poland 12495.33 6\n","Chile 13340.91 6.7\n","Latvia 13618.57 5.9\n","Slovak Republic 15991.74 6.1\n","Estonia 17288.08 5.6"]},"metadata":{},"execution_count":13}]},{"cell_type":"code","metadata":{"id":"NqZnpDGyR_3f","executionInfo":{"status":"ok","timestamp":1630462433448,"user_tz":240,"elapsed":1190,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"}}},"source":["# Code example\n","import matplotlib.pyplot as plt\n","import numpy as np\n","import pandas as pd\n","import sklearn.linear_model\n","\n","# Prepare the data\n","X = np.c_[train[\"GDP per capita\"]]\n","y = np.c_[train[\"Life satisfaction\"]]"],"execution_count":14,"outputs":[]},{"cell_type":"code","metadata":{"id":"UzRnfo8yAEbI","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1630462433450,"user_tz":240,"elapsed":8,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"}},"outputId":"cd44d876-24ab-41b3-93da-1e346a10bfd3"},"source":["type(y)"],"execution_count":15,"outputs":[{"output_type":"execute_result","data":{"text/plain":["numpy.ndarray"]},"metadata":{},"execution_count":15}]},{"cell_type":"code","metadata":{"id":"BzUJ0hnrwYgd","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1630462433450,"user_tz":240,"elapsed":4,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"}},"outputId":"138676bc-658c-404f-de0f-3c0814a7cd26"},"source":["#Lets look at what is inside X and y. Print first 5 records\n","X[:5], y[:5]"],"execution_count":16,"outputs":[{"output_type":"execute_result","data":{"text/plain":["(array([['12495.33'],\n"," ['13340.91'],\n"," ['13618.57'],\n"," ['15991.74'],\n"," ['17288.08']], dtype=object), array([['6'],\n"," ['6.7'],\n"," ['5.9'],\n"," ['6.1'],\n"," ['5.6']], dtype=object))"]},"metadata":{},"execution_count":16}]},{"cell_type":"markdown","metadata":{"id":"WJFLcCBgPPXj"},"source":["# ML Starts Now\n","## Define a model with default values"]},{"cell_type":"code","metadata":{"id":"_mimmXfQNTD1","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1630462527023,"user_tz":240,"elapsed":88,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"}},"outputId":"30897cd6-848d-41eb-bcde-d077e4a38ec9"},"source":["# Select a basic linear model without setting any parameter (nothing inside paranthesis below)\n","model = sklearn.linear_model.LinearRegression()\n","\n","# See the model for yourself\n","model"],"execution_count":18,"outputs":[{"output_type":"execute_result","data":{"text/plain":["LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False)"]},"metadata":{},"execution_count":18}]},{"cell_type":"markdown","metadata":{"id":"0pf1NGpkPSnD"},"source":["## Start training the model using X and y"]},{"cell_type":"code","metadata":{"id":"NySTaTDMp_Ve","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1630462530244,"user_tz":240,"elapsed":98,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"}},"outputId":"eb0114ae-4a8f-493e-f1e6-d185b48f24b9"},"source":["# Train the model\n","model.fit(X, y)"],"execution_count":19,"outputs":[{"output_type":"execute_result","data":{"text/plain":["LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False)"]},"metadata":{},"execution_count":19}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"YzVQ4lQors1b","executionInfo":{"status":"ok","timestamp":1630462536558,"user_tz":240,"elapsed":89,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"}},"outputId":"7dc7cedc-4547-4f56-e759-41083f5b204b"},"source":["model.coef_"],"execution_count":20,"outputs":[{"output_type":"execute_result","data":{"text/plain":["array([[3.83407608e-05]])"]},"metadata":{},"execution_count":20}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"KF0UFvZxsgPA","executionInfo":{"status":"ok","timestamp":1630462542604,"user_tz":240,"elapsed":92,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"}},"outputId":"148f5a1e-91b7-4c11-cdd8-eeaed8d5583a"},"source":["model.intercept_"],"execution_count":21,"outputs":[{"output_type":"execute_result","data":{"text/plain":["array([5.25939616])"]},"metadata":{},"execution_count":21}]},{"cell_type":"markdown","metadata":{"id":"v519lr8DPW1v"},"source":["## Do prediction on test data"]},{"cell_type":"code","metadata":{"id":"cMGgmbBJlcvb","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1620953094422,"user_tz":240,"elapsed":200,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"}},"outputId":"fbc2a261-6f82-42df-a7a4-915cc50f4b8a"},"source":["# Make a prediction for Cyprus\n","X_new = [[17770]] # Cyprus' GDP per capita\n","print(model.predict(X_new)) # outputs [[5.95199478]]"],"execution_count":null,"outputs":[{"output_type":"stream","text":["[[5.94071148]]\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"46aihcuVpT0E"},"source":["# Make a prediction for our test data\n","pred = model.predict(test['GDP per capita'].values.reshape(-1,1))"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"D9QinOHvItbt","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1620942331580,"user_tz":240,"elapsed":55410,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"}},"outputId":"8edf9e2d-efb0-4b3f-86ed-e237db854b0b"},"source":["pred"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["array([[9.16992716],\n"," [5.72868285],\n"," [5.9210396 ],\n"," [5.95199478],\n"," [8.35254892],\n"," [5.59181055],\n"," [5.60481881]])"]},"metadata":{"tags":[]},"execution_count":17}]},{"cell_type":"markdown","metadata":{"id":"cUa_ypmpaMLd"},"source":["#Now, lets make it better!\n","\n","Use test dataset and predict the life expectancy using test dataset. "]},{"cell_type":"markdown","metadata":{"id":"m_aeltVpZcp3"},"source":["# Evaluate Model"]},{"cell_type":"code","metadata":{"id":"fTHMYUFvZuD9"},"source":["# Lets create train and test dataset, so we can use train dataset for training the model\n","# and use test dataset to evaluate model performance \n","X_train = np.c_[train[\"GDP per capita\"]]\n","y_train = np.c_[train[\"Life satisfaction\"]]\n","\n","X_test = np.c_[test[\"GDP per capita\"]]\n","y_test = np.c_[test[\"Life satisfaction\"]]\n","\n","model = model.fit(X_train,y_train)\n","\n","#Now apply the prediction on test dataset\n","y_pred_test = model.predict(X_test)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"8DdKjw72Z-YN","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1620942331581,"user_tz":240,"elapsed":55401,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"}},"outputId":"78b5a918-beed-43a6-9326-6e1ff4d9a41f"},"source":["# See predictions for yourself\n","y_pred_test"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["array([[9.16992716],\n"," [5.72868285],\n"," [5.9210396 ],\n"," [5.95199478],\n"," [8.35254892],\n"," [5.59181055],\n"," [5.60481881]])"]},"metadata":{"tags":[]},"execution_count":19}]},{"cell_type":"code","metadata":{"id":"XgO5TjY8O_QT","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1620942331582,"user_tz":240,"elapsed":55394,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"}},"outputId":"32af2221-5dd4-4f48-b468-0e8f35fdbf0a"},"source":["from sklearn.metrics import mean_squared_error\n","\n","#MSE: Mean Squared Error as a metric to evaluate a regression model\n","mean_squared_error(y_test, y_pred_test)\n"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["1.299498988620812"]},"metadata":{"tags":[]},"execution_count":20}]},{"cell_type":"code","metadata":{"id":"JRCvYZHAZmKZ","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1620942331582,"user_tz":240,"elapsed":55386,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"}},"outputId":"f86ab28d-8b53-4322-9e77-c7ea516d8115"},"source":["#RMSE\n","from math import sqrt\n","sqrt(mean_squared_error(y_test, y_pred_test))\n"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["1.1399556959026136"]},"metadata":{"tags":[]},"execution_count":21}]},{"cell_type":"markdown","metadata":{"id":"4g05y180fTDX"},"source":["# Question:\n","\n","### What would you expect if we normalize data and train the model again?"]},{"cell_type":"markdown","metadata":{"id":"h8hPONDlaxZM"},"source":["# Now, normalize data before prediction"]},{"cell_type":"code","metadata":{"id":"YW04HGsLapBM"},"source":["from sklearn.preprocessing import MinMaxScaler\n","# Define Scaling technique\n","scaler = MinMaxScaler()"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"hFxrvpPpa5D6","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1620942331746,"user_tz":240,"elapsed":55541,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"}},"outputId":"c588a578-77fe-4aef-f6ca-2f42f3c01270"},"source":["# Train escaling object \n","X_train_escaler = scaler.fit(X_train)\n","\n","# Apply scaling model to the data\n","X_train_escaled = X_train_escaler.transform(X_train)\n","X_train_escaled[:5]"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["array([[0.09837989],\n"," [0.11061207],\n"," [0.1146287 ],\n"," [0.14895901],\n"," [0.16771188]])"]},"metadata":{"tags":[]},"execution_count":23}]},{"cell_type":"markdown","metadata":{"id":"xcoCKpR1c7LC"},"source":["#Normalize Test Dataset"]},{"cell_type":"code","metadata":{"id":"7ey1DwyWbClL","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1620942331746,"user_tz":240,"elapsed":55533,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"}},"outputId":"b4b3e430-37ea-4209-893c-491c762fe654"},"source":["# Apply scaling model to the data\n","X_test_escaled = X_train_escaler.transform(X_test)\n","X_test_escaled"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["array([[1.39307026],\n"," [0.09468469],\n"," [0.16726112],\n"," [0.17894055],\n"," [1.08467248],\n"," [0.04304261],\n"," [0.04795064]])"]},"metadata":{"tags":[]},"execution_count":24}]},{"cell_type":"markdown","metadata":{"id":"-tHTOhKldbOC"},"source":["# Train Models using Scaled Data"]},{"cell_type":"markdown","metadata":{"id":"SGtl94_EddFR"},"source":["## Start training the model using X and y"]},{"cell_type":"code","metadata":{"id":"A9olddl0ddFR"},"source":["model.fit(X_train_escaled, y_train)\n","\n","y_pred_escaled = model.predict(X_train_escaled)\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"oN8ztq0gddFV"},"source":["## Do prediction on test data"]},{"cell_type":"code","metadata":{"id":"NYyxFRRWfH0e"},"source":["y_pred_escaled = model.predict(X_test_escaled)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"kg2h-_PkfNDz","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1620942331748,"user_tz":240,"elapsed":55523,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"}},"outputId":"a875c7e1-05b9-4b05-9638-8dcf195c8100"},"source":["#RMSE\n","from math import sqrt\n","sqrt(mean_squared_error(y_test, y_pred_escaled))\n"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["1.1399556959026143"]},"metadata":{"tags":[]},"execution_count":27}]},{"cell_type":"markdown","metadata":{"id":"Q4HfXFDKi_Y4"},"source":["# Can you conclude by comparing RMSE from normalized and not normalized dataset?"]},{"cell_type":"code","metadata":{"id":"6lW3c6z3fQTU"},"source":[""],"execution_count":null,"outputs":[]}]} \ No newline at end of file +{"cells":[{"cell_type":"markdown","metadata":{"id":"gseS97WXHRAJ"},"source":["[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1CkvVzG1NtJABPkPRofGLT-MXUCj7veLo#scrollTo=gseS97WXHRAJ)\n"]},{"cell_type":"markdown","metadata":{"id":"-OMhwXX0Euz1"},"source":["# Predict People Satisfaction Across the Globe\n","\n","**Objective:**\n","We would like to build a model that predicts satisfaction score for people of different countries given their country GDP."]},{"cell_type":"markdown","metadata":{"id":"-sSMTG_Buxcm"},"source":["#Note 1: How to enable code completion:\n","\n","Tools menu ==> click on settings ==> Editor ==> Enable \"Automatically trigger code completions\"\n","\n","\n"]},{"cell_type":"markdown","metadata":{"id":"x9L7_OwZix-k"},"source":["#Note 2: Instructions to create a copy of this notebook for youtself\n","\n","You do not have write access to this notebook.\n","\n","* From Menu bar, Go to File, \n","* Select \"Save a copy in my Drive\"\n","* Navigate to Google Drive\n","* Find a folder named \"Collab Notebook\" and open it to find your notebook. \n","* Rename it and start making changes.\n","\n","**Note:** If there is any file you should read in your code, make sure you copy the file from instructor folder to your own Gdrive by following below steps:\n","\n","* Right clicking on the file name \n","* Select \"Make a copy\"\n","* Click on the new file\n","* Move it to desired folder, preferrably where you have your notebook"]},{"cell_type":"markdown","metadata":{"id":"WaAyuEL-EfXy"},"source":["# Download Dataset\n","\n","Download the Better Life Index data (latest edition, currently it is 2017) from the [OECD’s website](http://homl.info/4) as well as stats about GDP per capita from the [IMF’s website](http://homl.info/5). Then you join the tables and sort by GDP per capita. "]},{"cell_type":"markdown","metadata":{"id":"ZWasUlHjGGHc"},"source":["# Import Dataset to Google Colab\n","\n","1. Download CSV and XLS files to your computer\n","2. Upload them to your Google Drive\n","3. Open the CSV files using Google Sheets so Google will create the dataset in format of Google Sheets\n","4. You can remove CSV and XLS files from your drive\n","5. Use the step by step guide from [here](https://colab.research.google.com/notebooks/io.ipynb#scrollTo=vz-jH8T_Uk2c) and scroll down to **\" Google Sheets\" ** cell to import data into dataframe\n","\n","NOTE: After creating Google Sheet into your Drive, make sure you are converting Column 2015 to 0.00 format before importing it into Colab otherwise Google will import it as a string and you will have hard time to clean the data\n","\n","\n"]},{"cell_type":"code","execution_count":6,"metadata":{"executionInfo":{"elapsed":115,"status":"ok","timestamp":1630462416669,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"RhyqGUE9GFTF","vscode":{"languageId":"python"}},"outputs":[],"source":["# Run below line of code for the first time to install gspread. Once installed comment it for future use\n","#!pip install --upgrade -q gspread\n","from google.colab import auth\n","auth.authenticate_user()\n","\n","import gspread\n","from google.auth import default\n","creds, _ = default()\n","\n","gc = gspread.authorize(creds)"]},{"cell_type":"markdown","metadata":{"id":"1QrOorPEHbDg"},"source":["# Use gc to open Google Sheet Datasets"]},{"cell_type":"code","execution_count":7,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":1879,"status":"ok","timestamp":1630462418665,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"4seJiwQyEYVG","outputId":"e88a402a-3ac5-49bc-de2b-bc4f89ff08f2","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["Country\n","Australia 7.3\n","Austria 7\n","Belgium 6.9\n","Brazil 6.6\n","Canada 7.3\n","Name: Life satisfaction, dtype: object"]},"execution_count":7,"metadata":{},"output_type":"execute_result"}],"source":["#Open given sheet\n","worksheet = gc.open('BLI_26092022223032438.xlsx').sheet1\n","\n","# Read contents of CSV file\n","bli_rows = worksheet.get_all_values()\n","\n","# Convert to a DataFrame and render.\n","import pandas as pd\n","bli = pd.DataFrame.from_records(bli_rows, columns = bli_rows[0])\n","\n","# Remove rows where inequality has values other than TOT\n","bli = bli[bli[\"INEQUALITY\"]==\"TOT\"]\n","\n","# Reformat data based on \"indicator column\"\n","bli = bli.pivot(index=\"Country\", columns=\"Indicator\", values=\"Value\")\n","#bli.head()\n","bli[\"Life satisfaction\"].head()"]},{"cell_type":"code","execution_count":8,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":360},"executionInfo":{"elapsed":133,"status":"ok","timestamp":1630462418791,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"PJ42_kqLt7Zq","outputId":"c48e3c42-99a5-4e29-e524-074c1d0500a7","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
IndicatorAir pollutionDwellings without basic facilitiesEducational attainmentEmployees working very long hoursEmployment rateFeeling safe walking alone at nightHomicide rateHousehold net adjusted disposable incomeHousehold net financial wealthHousing expenditureLabour market insecurityLife expectancyLife satisfactionLong-term unemployment ratePersonal earningsQuality of support networkRooms per personSelf-reported healthStakeholder engagement for developing regulationsStudent skillsTime devoted to leisure and personal careVoter turnoutWater qualityYears in education
Country
Australia51.18013.27263.613341757462204.382.57.31.3652063942.3852.750214.35919221.2
Austria161856.787280.70.43254459574212.781.371.9448295921.6701.349214.55759317.1
Belgium152.3754.316270.7129968104084214.881.16.93.9849587922.2752.250315.77898418.2
Brazil106.7497.156437.327.6122277102204.974.76.63.3714024900.8702.239514.45797215.9
Canada70.2913.737380.91.42985085758223.981.57.30.8148403932.588352314.41689116.7
\n","
"],"text/plain":["Indicator Air pollution ... Years in education\n","Country ... \n","Australia 5 ... 21.2\n","Austria 16 ... 17.1\n","Belgium 15 ... 18.2\n","Brazil 10 ... 15.9\n","Canada 7 ... 16.7\n","\n","[5 rows x 24 columns]"]},"execution_count":8,"metadata":{},"output_type":"execute_result"}],"source":["bli[0:5]"]},{"cell_type":"code","execution_count":9,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":236},"executionInfo":{"elapsed":1198,"status":"ok","timestamp":1630462419986,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"01edCO9kHJ2p","outputId":"13ea178f-cce1-4001-e5fe-a5a04840acaa","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
GDP per capita
Country
Afghanistan599.99
Albania3995.38
Algeria4318.14
Angola4100.32
Antigua and Barbuda14414.30
\n","
"],"text/plain":[" GDP per capita\n","Country \n","Afghanistan 599.99\n","Albania 3995.38\n","Algeria 4318.14\n","Angola 4100.32\n","Antigua and Barbuda 14414.30"]},"execution_count":9,"metadata":{},"output_type":"execute_result"}],"source":["#Open given sheet\n","worksheet = gc.open('WEO_Data').sheet1\n","\n","# Read contents of CSV file\n","WEO_rows = worksheet.get_all_values()\n","\n","# Convert to a DataFrame and render.\n","import pandas as pd\n","weo = pd.DataFrame.from_records(WEO_rows, columns = WEO_rows[0])\n","\n","# Drop the header row from data\n","weo = weo.reindex(weo.index.drop(0))\n","\n","# 1- Select only Country name and 2015 \n","# 2- then rename it to GDP Per capita\n","weo = weo[['Country','2015']].rename(columns={'2015':'GDP per capita'})\n","\n","# Set Country as index column\n","# Inplace command, will replace the results of command into the same DF\n","weo.set_index(\"Country\", inplace=True)\n","\n","#weo.drop_duplicates(inplace=True)\n","#Print top 5 rows\n","weo.head()"]},{"cell_type":"markdown","metadata":{"id":"XX_8yJTKNPkt"},"source":["# Merge/Join dataset"]},{"cell_type":"code","execution_count":10,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":360},"executionInfo":{"elapsed":127,"status":"ok","timestamp":1630462420472,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"ZibYb4W-HuEy","outputId":"d943679b-bf91-4a18-cb4f-30a6ac28e098","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
GDP per capitaAir pollutionDwellings without basic facilitiesEducational attainmentEmployees working very long hoursEmployment rateFeeling safe walking alone at nightHomicide rateHousehold net adjusted disposable incomeHousehold net financial wealthHousing expenditureLabour market insecurityLife expectancyLife satisfactionLong-term unemployment ratePersonal earningsQuality of support networkRooms per personSelf-reported healthStakeholder engagement for developing regulationsStudent skillsTime devoted to leisure and personal careVoter turnoutWater qualityYears in education
Country
Luxembourg101994.09120793.7666720.64131774141203.282.46.91.962636922701.548315.15918515.1
Hungary12239.89194.3833.056750.71.21682123289184.875.75.32.4221711841.2561.247415.06627616.6
Poland12495.33222.7916.686566.30.81890614997234.377.662.1425921891.1582.650414.42558017.7
Chile13340.91169.46510.066251.14.51658821409188.179.16.72.0228434841.9571.544314.9496917.3
Latvia13618.571112.9892.096960.76.61526917105236.874.65.93.7322389861.2462.448713.83597717.9
\n","
"],"text/plain":[" GDP per capita Air pollution ... Water quality Years in education\n","Country ... \n","Luxembourg 101994.09 12 ... 85 15.1\n","Hungary 12239.89 19 ... 76 16.6\n","Poland 12495.33 22 ... 80 17.7\n","Chile 13340.91 16 ... 69 17.3\n","Latvia 13618.57 11 ... 77 17.9\n","\n","[5 rows x 25 columns]"]},"execution_count":10,"metadata":{},"output_type":"execute_result"}],"source":["# Now merge BLI and WEO datasets\n","df = pd.merge(left = weo, right = bli, left_index=True, right_index=True)\n","\n","#sort the dataframe by GPD per capita\n","df.sort_values(by=\"GDP per capita\", inplace=True)\n","df.head()\n"]},{"cell_type":"code","execution_count":11,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":4,"status":"ok","timestamp":1630462420888,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"eoJn-lxSPfqP","outputId":"543cdb12-e9a2-40cd-a9e5-48b5f0501525","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["GDP per capita 13340.91\n","Air pollution 16\n","Dwellings without basic facilities 9.4\n","Educational attainment 65\n","Employees working very long hours 10.06\n","Employment rate 62\n","Feeling safe walking alone at night 51.1\n","Homicide rate 4.5\n","Household net adjusted disposable income 16588\n","Household net financial wealth 21409\n","Housing expenditure 18\n","Labour market insecurity 8.1\n","Life expectancy 79.1\n","Life satisfaction 6.7\n","Long-term unemployment rate 2.02\n","Personal earnings 28434\n","Quality of support network 84\n","Rooms per person 1.9\n","Self-reported health 57\n","Stakeholder engagement for developing regulations 1.5\n","Student skills 443\n","Time devoted to leisure and personal care 14.9\n","Voter turnout 49\n","Water quality 69\n","Years in education 17.3\n","Name: Chile, dtype: object"]},"execution_count":11,"metadata":{},"output_type":"execute_result"}],"source":["df.iloc[3]"]},{"cell_type":"markdown","metadata":{"id":"MkSTXnclPGLd"},"source":["## Split dataset into Train & Test"]},{"cell_type":"code","execution_count":12,"metadata":{"executionInfo":{"elapsed":2,"status":"ok","timestamp":1630462423352,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"6HOaSt93Nqkz","vscode":{"languageId":"python"}},"outputs":[],"source":["# Below snippet is the most basic way of splitting data in Python; It is for illustration only and in \n","# later sections we will use proper library from Sklearn to split the data\n","test_indices = [0, 1, 6, 8, 33, 34, 35]\n","\n","#define the indices of the training set by substracting the test indices from all indices\n","train_indices = list(set(range(36)) - set(test_indices))\n","\n","#use the above indices to select parts of the dataframe as the training set and the test set\n","train = df[[\"GDP per capita\", 'Life satisfaction']].iloc[train_indices]\n","test = df[[\"GDP per capita\", 'Life satisfaction']].iloc[test_indices]"]},{"cell_type":"code","execution_count":13,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":236},"executionInfo":{"elapsed":116,"status":"ok","timestamp":1630462431988,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"9CRVFFEX_voT","outputId":"fd22091b-f2d0-42ef-f5c2-ad591827f287","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
GDP per capitaLife satisfaction
Country
Poland12495.336
Chile13340.916.7
Latvia13618.575.9
Slovak Republic15991.746.1
Estonia17288.085.6
\n","
"],"text/plain":[" GDP per capita Life satisfaction\n","Country \n","Poland 12495.33 6\n","Chile 13340.91 6.7\n","Latvia 13618.57 5.9\n","Slovak Republic 15991.74 6.1\n","Estonia 17288.08 5.6"]},"execution_count":13,"metadata":{},"output_type":"execute_result"}],"source":["train.head()"]},{"cell_type":"code","execution_count":14,"metadata":{"executionInfo":{"elapsed":1190,"status":"ok","timestamp":1630462433448,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"NqZnpDGyR_3f","vscode":{"languageId":"python"}},"outputs":[],"source":["# Code example\n","import matplotlib.pyplot as plt\n","import numpy as np\n","import pandas as pd\n","import sklearn.linear_model\n","\n","# Prepare the data\n","X = np.c_[train[\"GDP per capita\"]]\n","y = np.c_[train[\"Life satisfaction\"]]"]},{"cell_type":"code","execution_count":15,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":8,"status":"ok","timestamp":1630462433450,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"UzRnfo8yAEbI","outputId":"cd44d876-24ab-41b3-93da-1e346a10bfd3","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["numpy.ndarray"]},"execution_count":15,"metadata":{},"output_type":"execute_result"}],"source":["type(y)"]},{"cell_type":"code","execution_count":16,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":4,"status":"ok","timestamp":1630462433450,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"BzUJ0hnrwYgd","outputId":"138676bc-658c-404f-de0f-3c0814a7cd26","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["(array([['12495.33'],\n"," ['13340.91'],\n"," ['13618.57'],\n"," ['15991.74'],\n"," ['17288.08']], dtype=object), array([['6'],\n"," ['6.7'],\n"," ['5.9'],\n"," ['6.1'],\n"," ['5.6']], dtype=object))"]},"execution_count":16,"metadata":{},"output_type":"execute_result"}],"source":["#Lets look at what is inside X and y. Print first 5 records\n","X[:5], y[:5]"]},{"cell_type":"markdown","metadata":{"id":"WJFLcCBgPPXj"},"source":["# ML Starts Now\n","## Define a model with default values"]},{"cell_type":"code","execution_count":18,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":88,"status":"ok","timestamp":1630462527023,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"_mimmXfQNTD1","outputId":"30897cd6-848d-41eb-bcde-d077e4a38ec9","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False)"]},"execution_count":18,"metadata":{},"output_type":"execute_result"}],"source":["# Select a basic linear model without setting any parameter (nothing inside paranthesis below)\n","model = sklearn.linear_model.LinearRegression()\n","\n","# See the model for yourself\n","model"]},{"cell_type":"markdown","metadata":{"id":"0pf1NGpkPSnD"},"source":["## Start training the model using X and y"]},{"cell_type":"code","execution_count":19,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":98,"status":"ok","timestamp":1630462530244,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"NySTaTDMp_Ve","outputId":"eb0114ae-4a8f-493e-f1e6-d185b48f24b9","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False)"]},"execution_count":19,"metadata":{},"output_type":"execute_result"}],"source":["# Train the model\n","model.fit(X, y)"]},{"cell_type":"code","execution_count":20,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":89,"status":"ok","timestamp":1630462536558,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"YzVQ4lQors1b","outputId":"7dc7cedc-4547-4f56-e759-41083f5b204b","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["array([[3.83407608e-05]])"]},"execution_count":20,"metadata":{},"output_type":"execute_result"}],"source":["model.coef_"]},{"cell_type":"code","execution_count":21,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":92,"status":"ok","timestamp":1630462542604,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"KF0UFvZxsgPA","outputId":"148f5a1e-91b7-4c11-cdd8-eeaed8d5583a","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["array([5.25939616])"]},"execution_count":21,"metadata":{},"output_type":"execute_result"}],"source":["model.intercept_"]},{"cell_type":"markdown","metadata":{"id":"v519lr8DPW1v"},"source":["## Do prediction on test data"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":200,"status":"ok","timestamp":1620953094422,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"cMGgmbBJlcvb","outputId":"fbc2a261-6f82-42df-a7a4-915cc50f4b8a","vscode":{"languageId":"python"}},"outputs":[{"name":"stdout","output_type":"stream","text":["[[5.94071148]]\n"]}],"source":["# Make a prediction for Cyprus\n","X_new = [[17770]] # Cyprus' GDP per capita\n","print(model.predict(X_new)) # outputs [[5.95199478]]"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"46aihcuVpT0E","vscode":{"languageId":"python"}},"outputs":[],"source":["# Make a prediction for our test data\n","pred = model.predict(test['GDP per capita'].values.reshape(-1,1))"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":55410,"status":"ok","timestamp":1620942331580,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"D9QinOHvItbt","outputId":"8edf9e2d-efb0-4b3f-86ed-e237db854b0b","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["array([[9.16992716],\n"," [5.72868285],\n"," [5.9210396 ],\n"," [5.95199478],\n"," [8.35254892],\n"," [5.59181055],\n"," [5.60481881]])"]},"execution_count":17,"metadata":{"tags":[]},"output_type":"execute_result"}],"source":["pred"]},{"cell_type":"markdown","metadata":{"id":"cUa_ypmpaMLd"},"source":["#Now, lets make it better!\n","\n","Use test dataset and predict the life expectancy using test dataset. "]},{"cell_type":"markdown","metadata":{"id":"m_aeltVpZcp3"},"source":["# Evaluate Model"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"fTHMYUFvZuD9","vscode":{"languageId":"python"}},"outputs":[],"source":["# Lets create train and test dataset, so we can use train dataset for training the model\n","# and use test dataset to evaluate model performance \n","X_train = np.c_[train[\"GDP per capita\"]]\n","y_train = np.c_[train[\"Life satisfaction\"]]\n","\n","X_test = np.c_[test[\"GDP per capita\"]]\n","y_test = np.c_[test[\"Life satisfaction\"]]\n","\n","model = model.fit(X_train,y_train)\n","\n","#Now apply the prediction on test dataset\n","y_pred_test = model.predict(X_test)"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":55401,"status":"ok","timestamp":1620942331581,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"8DdKjw72Z-YN","outputId":"78b5a918-beed-43a6-9326-6e1ff4d9a41f","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["array([[9.16992716],\n"," [5.72868285],\n"," [5.9210396 ],\n"," [5.95199478],\n"," [8.35254892],\n"," [5.59181055],\n"," [5.60481881]])"]},"execution_count":19,"metadata":{"tags":[]},"output_type":"execute_result"}],"source":["# See predictions for yourself\n","y_pred_test"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":55394,"status":"ok","timestamp":1620942331582,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"XgO5TjY8O_QT","outputId":"32af2221-5dd4-4f48-b468-0e8f35fdbf0a","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["1.299498988620812"]},"execution_count":20,"metadata":{"tags":[]},"output_type":"execute_result"}],"source":["from sklearn.metrics import mean_squared_error\n","\n","#MSE: Mean Squared Error as a metric to evaluate a regression model\n","mean_squared_error(y_test, y_pred_test)\n"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":55386,"status":"ok","timestamp":1620942331582,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"JRCvYZHAZmKZ","outputId":"f86ab28d-8b53-4322-9e77-c7ea516d8115","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["1.1399556959026136"]},"execution_count":21,"metadata":{"tags":[]},"output_type":"execute_result"}],"source":["#RMSE\n","from math import sqrt\n","sqrt(mean_squared_error(y_test, y_pred_test))\n"]},{"cell_type":"markdown","metadata":{"id":"4g05y180fTDX"},"source":["# Question:\n","\n","### What would you expect if we normalize data and train the model again?"]},{"cell_type":"markdown","metadata":{"id":"h8hPONDlaxZM"},"source":["# Now, normalize data before prediction"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"YW04HGsLapBM","vscode":{"languageId":"python"}},"outputs":[],"source":["from sklearn.preprocessing import MinMaxScaler\n","# Define Scaling technique\n","scaler = MinMaxScaler()"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":55541,"status":"ok","timestamp":1620942331746,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"hFxrvpPpa5D6","outputId":"c588a578-77fe-4aef-f6ca-2f42f3c01270","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["array([[0.09837989],\n"," [0.11061207],\n"," [0.1146287 ],\n"," [0.14895901],\n"," [0.16771188]])"]},"execution_count":23,"metadata":{"tags":[]},"output_type":"execute_result"}],"source":["# Train escaling object \n","X_train_escaler = scaler.fit(X_train)\n","\n","# Apply scaling model to the data\n","X_train_escaled = X_train_escaler.transform(X_train)\n","X_train_escaled[:5]"]},{"cell_type":"markdown","metadata":{"id":"xcoCKpR1c7LC"},"source":["#Normalize Test Dataset"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":55533,"status":"ok","timestamp":1620942331746,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"7ey1DwyWbClL","outputId":"b4b3e430-37ea-4209-893c-491c762fe654","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["array([[1.39307026],\n"," [0.09468469],\n"," [0.16726112],\n"," [0.17894055],\n"," [1.08467248],\n"," [0.04304261],\n"," [0.04795064]])"]},"execution_count":24,"metadata":{"tags":[]},"output_type":"execute_result"}],"source":["# Apply scaling model to the data\n","X_test_escaled = X_train_escaler.transform(X_test)\n","X_test_escaled"]},{"cell_type":"markdown","metadata":{"id":"-tHTOhKldbOC"},"source":["# Train Models using Scaled Data"]},{"cell_type":"markdown","metadata":{"id":"SGtl94_EddFR"},"source":["## Start training the model using X and y"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"A9olddl0ddFR","vscode":{"languageId":"python"}},"outputs":[],"source":["model.fit(X_train_escaled, y_train)\n","\n","y_pred_escaled = model.predict(X_train_escaled)\n"]},{"cell_type":"markdown","metadata":{"id":"oN8ztq0gddFV"},"source":["## Do prediction on test data"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"NYyxFRRWfH0e","vscode":{"languageId":"python"}},"outputs":[],"source":["y_pred_escaled = model.predict(X_test_escaled)"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":55523,"status":"ok","timestamp":1620942331748,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"kg2h-_PkfNDz","outputId":"a875c7e1-05b9-4b05-9638-8dcf195c8100","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["1.1399556959026143"]},"execution_count":27,"metadata":{"tags":[]},"output_type":"execute_result"}],"source":["#RMSE\n","from math import sqrt\n","sqrt(mean_squared_error(y_test, y_pred_escaled))\n"]},{"cell_type":"markdown","metadata":{"id":"Q4HfXFDKi_Y4"},"source":["# Can you conclude by comparing RMSE from normalized and not normalized dataset?"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"6lW3c6z3fQTU","vscode":{"languageId":"python"}},"outputs":[],"source":[]}],"metadata":{"colab":{"collapsed_sections":[],"name":"People Satisfaction and GDP.ipynb","provenance":[]},"kernelspec":{"display_name":"Python 3","name":"python3"}},"nbformat":4,"nbformat_minor":0} From b6a44e26ccd77d1de5c64522da0d3be4a8192605 Mon Sep 17 00:00:00 2001 From: Samantha Feinstein Date: Mon, 26 Sep 2022 17:25:07 -0400 Subject: [PATCH 2/2] Date and formatting updates I found that if I updated column F to 0.00 format as instructed, it also updated the header. Obviously it should be 2015, not 2,015.00 but if we're changing the name of the column anyway we might as well apply the format to the entirety of column F and avoid this little "gotcha". --- ... Satisfaction and GDP_Extended Version with Evaluation.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Week_01 - Introduction to Machine Learning/People Satisfaction and GDP_Extended Version with Evaluation.ipynb b/Week_01 - Introduction to Machine Learning/People Satisfaction and GDP_Extended Version with Evaluation.ipynb index 0475355..9f48953 100644 --- a/Week_01 - Introduction to Machine Learning/People Satisfaction and GDP_Extended Version with Evaluation.ipynb +++ b/Week_01 - Introduction to Machine Learning/People Satisfaction and GDP_Extended Version with Evaluation.ipynb @@ -1 +1 @@ -{"cells":[{"cell_type":"markdown","metadata":{"id":"gseS97WXHRAJ"},"source":["[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1CkvVzG1NtJABPkPRofGLT-MXUCj7veLo#scrollTo=gseS97WXHRAJ)\n"]},{"cell_type":"markdown","metadata":{"id":"-OMhwXX0Euz1"},"source":["# Predict People Satisfaction Across the Globe\n","\n","**Objective:**\n","We would like to build a model that predicts satisfaction score for people of different countries given their country GDP."]},{"cell_type":"markdown","metadata":{"id":"-sSMTG_Buxcm"},"source":["#Note 1: How to enable code completion:\n","\n","Tools menu ==> click on settings ==> Editor ==> Enable \"Automatically trigger code completions\"\n","\n","\n"]},{"cell_type":"markdown","metadata":{"id":"x9L7_OwZix-k"},"source":["#Note 2: Instructions to create a copy of this notebook for youtself\n","\n","You do not have write access to this notebook.\n","\n","* From Menu bar, Go to File, \n","* Select \"Save a copy in my Drive\"\n","* Navigate to Google Drive\n","* Find a folder named \"Collab Notebook\" and open it to find your notebook. \n","* Rename it and start making changes.\n","\n","**Note:** If there is any file you should read in your code, make sure you copy the file from instructor folder to your own Gdrive by following below steps:\n","\n","* Right clicking on the file name \n","* Select \"Make a copy\"\n","* Click on the new file\n","* Move it to desired folder, preferrably where you have your notebook"]},{"cell_type":"markdown","metadata":{"id":"WaAyuEL-EfXy"},"source":["# Download Dataset\n","\n","Download the Better Life Index data (latest edition, currently it is 2017) from the [OECD’s website](http://homl.info/4) as well as stats about GDP per capita from the [IMF’s website](http://homl.info/5). Then you join the tables and sort by GDP per capita. "]},{"cell_type":"markdown","metadata":{"id":"ZWasUlHjGGHc"},"source":["# Import Dataset to Google Colab\n","\n","1. Download CSV and XLS files to your computer\n","2. Upload them to your Google Drive\n","3. Open the CSV files using Google Sheets so Google will create the dataset in format of Google Sheets\n","4. You can remove CSV and XLS files from your drive\n","5. Use the step by step guide from [here](https://colab.research.google.com/notebooks/io.ipynb#scrollTo=vz-jH8T_Uk2c) and scroll down to **\" Google Sheets\" ** cell to import data into dataframe\n","\n","NOTE: After creating Google Sheet into your Drive, make sure you are converting Column 2015 to 0.00 format before importing it into Colab otherwise Google will import it as a string and you will have hard time to clean the data\n","\n","\n"]},{"cell_type":"code","execution_count":6,"metadata":{"executionInfo":{"elapsed":115,"status":"ok","timestamp":1630462416669,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"RhyqGUE9GFTF","vscode":{"languageId":"python"}},"outputs":[],"source":["# Run below line of code for the first time to install gspread. Once installed comment it for future use\n","#!pip install --upgrade -q gspread\n","from google.colab import auth\n","auth.authenticate_user()\n","\n","import gspread\n","from google.auth import default\n","creds, _ = default()\n","\n","gc = gspread.authorize(creds)"]},{"cell_type":"markdown","metadata":{"id":"1QrOorPEHbDg"},"source":["# Use gc to open Google Sheet Datasets"]},{"cell_type":"code","execution_count":7,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":1879,"status":"ok","timestamp":1630462418665,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"4seJiwQyEYVG","outputId":"e88a402a-3ac5-49bc-de2b-bc4f89ff08f2","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["Country\n","Australia 7.3\n","Austria 7\n","Belgium 6.9\n","Brazil 6.6\n","Canada 7.3\n","Name: Life satisfaction, dtype: object"]},"execution_count":7,"metadata":{},"output_type":"execute_result"}],"source":["#Open given sheet\n","worksheet = gc.open('BLI_26092022223032438.xlsx').sheet1\n","\n","# Read contents of CSV file\n","bli_rows = worksheet.get_all_values()\n","\n","# Convert to a DataFrame and render.\n","import pandas as pd\n","bli = pd.DataFrame.from_records(bli_rows, columns = bli_rows[0])\n","\n","# Remove rows where inequality has values other than TOT\n","bli = bli[bli[\"INEQUALITY\"]==\"TOT\"]\n","\n","# Reformat data based on \"indicator column\"\n","bli = bli.pivot(index=\"Country\", columns=\"Indicator\", values=\"Value\")\n","#bli.head()\n","bli[\"Life satisfaction\"].head()"]},{"cell_type":"code","execution_count":8,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":360},"executionInfo":{"elapsed":133,"status":"ok","timestamp":1630462418791,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"PJ42_kqLt7Zq","outputId":"c48e3c42-99a5-4e29-e524-074c1d0500a7","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
IndicatorAir pollutionDwellings without basic facilitiesEducational attainmentEmployees working very long hoursEmployment rateFeeling safe walking alone at nightHomicide rateHousehold net adjusted disposable incomeHousehold net financial wealthHousing expenditureLabour market insecurityLife expectancyLife satisfactionLong-term unemployment ratePersonal earningsQuality of support networkRooms per personSelf-reported healthStakeholder engagement for developing regulationsStudent skillsTime devoted to leisure and personal careVoter turnoutWater qualityYears in education
Country
Australia51.18013.27263.613341757462204.382.57.31.3652063942.3852.750214.35919221.2
Austria161856.787280.70.43254459574212.781.371.9448295921.6701.349214.55759317.1
Belgium152.3754.316270.7129968104084214.881.16.93.9849587922.2752.250315.77898418.2
Brazil106.7497.156437.327.6122277102204.974.76.63.3714024900.8702.239514.45797215.9
Canada70.2913.737380.91.42985085758223.981.57.30.8148403932.588352314.41689116.7
\n","
"],"text/plain":["Indicator Air pollution ... Years in education\n","Country ... \n","Australia 5 ... 21.2\n","Austria 16 ... 17.1\n","Belgium 15 ... 18.2\n","Brazil 10 ... 15.9\n","Canada 7 ... 16.7\n","\n","[5 rows x 24 columns]"]},"execution_count":8,"metadata":{},"output_type":"execute_result"}],"source":["bli[0:5]"]},{"cell_type":"code","execution_count":9,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":236},"executionInfo":{"elapsed":1198,"status":"ok","timestamp":1630462419986,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"01edCO9kHJ2p","outputId":"13ea178f-cce1-4001-e5fe-a5a04840acaa","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
GDP per capita
Country
Afghanistan599.99
Albania3995.38
Algeria4318.14
Angola4100.32
Antigua and Barbuda14414.30
\n","
"],"text/plain":[" GDP per capita\n","Country \n","Afghanistan 599.99\n","Albania 3995.38\n","Algeria 4318.14\n","Angola 4100.32\n","Antigua and Barbuda 14414.30"]},"execution_count":9,"metadata":{},"output_type":"execute_result"}],"source":["#Open given sheet\n","worksheet = gc.open('WEO_Data').sheet1\n","\n","# Read contents of CSV file\n","WEO_rows = worksheet.get_all_values()\n","\n","# Convert to a DataFrame and render.\n","import pandas as pd\n","weo = pd.DataFrame.from_records(WEO_rows, columns = WEO_rows[0])\n","\n","# Drop the header row from data\n","weo = weo.reindex(weo.index.drop(0))\n","\n","# 1- Select only Country name and 2015 \n","# 2- then rename it to GDP Per capita\n","weo = weo[['Country','2015']].rename(columns={'2015':'GDP per capita'})\n","\n","# Set Country as index column\n","# Inplace command, will replace the results of command into the same DF\n","weo.set_index(\"Country\", inplace=True)\n","\n","#weo.drop_duplicates(inplace=True)\n","#Print top 5 rows\n","weo.head()"]},{"cell_type":"markdown","metadata":{"id":"XX_8yJTKNPkt"},"source":["# Merge/Join dataset"]},{"cell_type":"code","execution_count":10,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":360},"executionInfo":{"elapsed":127,"status":"ok","timestamp":1630462420472,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"ZibYb4W-HuEy","outputId":"d943679b-bf91-4a18-cb4f-30a6ac28e098","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
GDP per capitaAir pollutionDwellings without basic facilitiesEducational attainmentEmployees working very long hoursEmployment rateFeeling safe walking alone at nightHomicide rateHousehold net adjusted disposable incomeHousehold net financial wealthHousing expenditureLabour market insecurityLife expectancyLife satisfactionLong-term unemployment ratePersonal earningsQuality of support networkRooms per personSelf-reported healthStakeholder engagement for developing regulationsStudent skillsTime devoted to leisure and personal careVoter turnoutWater qualityYears in education
Country
Luxembourg101994.09120793.7666720.64131774141203.282.46.91.962636922701.548315.15918515.1
Hungary12239.89194.3833.056750.71.21682123289184.875.75.32.4221711841.2561.247415.06627616.6
Poland12495.33222.7916.686566.30.81890614997234.377.662.1425921891.1582.650414.42558017.7
Chile13340.91169.46510.066251.14.51658821409188.179.16.72.0228434841.9571.544314.9496917.3
Latvia13618.571112.9892.096960.76.61526917105236.874.65.93.7322389861.2462.448713.83597717.9
\n","
"],"text/plain":[" GDP per capita Air pollution ... Water quality Years in education\n","Country ... \n","Luxembourg 101994.09 12 ... 85 15.1\n","Hungary 12239.89 19 ... 76 16.6\n","Poland 12495.33 22 ... 80 17.7\n","Chile 13340.91 16 ... 69 17.3\n","Latvia 13618.57 11 ... 77 17.9\n","\n","[5 rows x 25 columns]"]},"execution_count":10,"metadata":{},"output_type":"execute_result"}],"source":["# Now merge BLI and WEO datasets\n","df = pd.merge(left = weo, right = bli, left_index=True, right_index=True)\n","\n","#sort the dataframe by GPD per capita\n","df.sort_values(by=\"GDP per capita\", inplace=True)\n","df.head()\n"]},{"cell_type":"code","execution_count":11,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":4,"status":"ok","timestamp":1630462420888,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"eoJn-lxSPfqP","outputId":"543cdb12-e9a2-40cd-a9e5-48b5f0501525","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["GDP per capita 13340.91\n","Air pollution 16\n","Dwellings without basic facilities 9.4\n","Educational attainment 65\n","Employees working very long hours 10.06\n","Employment rate 62\n","Feeling safe walking alone at night 51.1\n","Homicide rate 4.5\n","Household net adjusted disposable income 16588\n","Household net financial wealth 21409\n","Housing expenditure 18\n","Labour market insecurity 8.1\n","Life expectancy 79.1\n","Life satisfaction 6.7\n","Long-term unemployment rate 2.02\n","Personal earnings 28434\n","Quality of support network 84\n","Rooms per person 1.9\n","Self-reported health 57\n","Stakeholder engagement for developing regulations 1.5\n","Student skills 443\n","Time devoted to leisure and personal care 14.9\n","Voter turnout 49\n","Water quality 69\n","Years in education 17.3\n","Name: Chile, dtype: object"]},"execution_count":11,"metadata":{},"output_type":"execute_result"}],"source":["df.iloc[3]"]},{"cell_type":"markdown","metadata":{"id":"MkSTXnclPGLd"},"source":["## Split dataset into Train & Test"]},{"cell_type":"code","execution_count":12,"metadata":{"executionInfo":{"elapsed":2,"status":"ok","timestamp":1630462423352,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"6HOaSt93Nqkz","vscode":{"languageId":"python"}},"outputs":[],"source":["# Below snippet is the most basic way of splitting data in Python; It is for illustration only and in \n","# later sections we will use proper library from Sklearn to split the data\n","test_indices = [0, 1, 6, 8, 33, 34, 35]\n","\n","#define the indices of the training set by substracting the test indices from all indices\n","train_indices = list(set(range(36)) - set(test_indices))\n","\n","#use the above indices to select parts of the dataframe as the training set and the test set\n","train = df[[\"GDP per capita\", 'Life satisfaction']].iloc[train_indices]\n","test = df[[\"GDP per capita\", 'Life satisfaction']].iloc[test_indices]"]},{"cell_type":"code","execution_count":13,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":236},"executionInfo":{"elapsed":116,"status":"ok","timestamp":1630462431988,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"9CRVFFEX_voT","outputId":"fd22091b-f2d0-42ef-f5c2-ad591827f287","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
GDP per capitaLife satisfaction
Country
Poland12495.336
Chile13340.916.7
Latvia13618.575.9
Slovak Republic15991.746.1
Estonia17288.085.6
\n","
"],"text/plain":[" GDP per capita Life satisfaction\n","Country \n","Poland 12495.33 6\n","Chile 13340.91 6.7\n","Latvia 13618.57 5.9\n","Slovak Republic 15991.74 6.1\n","Estonia 17288.08 5.6"]},"execution_count":13,"metadata":{},"output_type":"execute_result"}],"source":["train.head()"]},{"cell_type":"code","execution_count":14,"metadata":{"executionInfo":{"elapsed":1190,"status":"ok","timestamp":1630462433448,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"NqZnpDGyR_3f","vscode":{"languageId":"python"}},"outputs":[],"source":["# Code example\n","import matplotlib.pyplot as plt\n","import numpy as np\n","import pandas as pd\n","import sklearn.linear_model\n","\n","# Prepare the data\n","X = np.c_[train[\"GDP per capita\"]]\n","y = np.c_[train[\"Life satisfaction\"]]"]},{"cell_type":"code","execution_count":15,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":8,"status":"ok","timestamp":1630462433450,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"UzRnfo8yAEbI","outputId":"cd44d876-24ab-41b3-93da-1e346a10bfd3","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["numpy.ndarray"]},"execution_count":15,"metadata":{},"output_type":"execute_result"}],"source":["type(y)"]},{"cell_type":"code","execution_count":16,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":4,"status":"ok","timestamp":1630462433450,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"BzUJ0hnrwYgd","outputId":"138676bc-658c-404f-de0f-3c0814a7cd26","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["(array([['12495.33'],\n"," ['13340.91'],\n"," ['13618.57'],\n"," ['15991.74'],\n"," ['17288.08']], dtype=object), array([['6'],\n"," ['6.7'],\n"," ['5.9'],\n"," ['6.1'],\n"," ['5.6']], dtype=object))"]},"execution_count":16,"metadata":{},"output_type":"execute_result"}],"source":["#Lets look at what is inside X and y. Print first 5 records\n","X[:5], y[:5]"]},{"cell_type":"markdown","metadata":{"id":"WJFLcCBgPPXj"},"source":["# ML Starts Now\n","## Define a model with default values"]},{"cell_type":"code","execution_count":18,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":88,"status":"ok","timestamp":1630462527023,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"_mimmXfQNTD1","outputId":"30897cd6-848d-41eb-bcde-d077e4a38ec9","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False)"]},"execution_count":18,"metadata":{},"output_type":"execute_result"}],"source":["# Select a basic linear model without setting any parameter (nothing inside paranthesis below)\n","model = sklearn.linear_model.LinearRegression()\n","\n","# See the model for yourself\n","model"]},{"cell_type":"markdown","metadata":{"id":"0pf1NGpkPSnD"},"source":["## Start training the model using X and y"]},{"cell_type":"code","execution_count":19,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":98,"status":"ok","timestamp":1630462530244,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"NySTaTDMp_Ve","outputId":"eb0114ae-4a8f-493e-f1e6-d185b48f24b9","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False)"]},"execution_count":19,"metadata":{},"output_type":"execute_result"}],"source":["# Train the model\n","model.fit(X, y)"]},{"cell_type":"code","execution_count":20,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":89,"status":"ok","timestamp":1630462536558,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"YzVQ4lQors1b","outputId":"7dc7cedc-4547-4f56-e759-41083f5b204b","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["array([[3.83407608e-05]])"]},"execution_count":20,"metadata":{},"output_type":"execute_result"}],"source":["model.coef_"]},{"cell_type":"code","execution_count":21,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":92,"status":"ok","timestamp":1630462542604,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"KF0UFvZxsgPA","outputId":"148f5a1e-91b7-4c11-cdd8-eeaed8d5583a","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["array([5.25939616])"]},"execution_count":21,"metadata":{},"output_type":"execute_result"}],"source":["model.intercept_"]},{"cell_type":"markdown","metadata":{"id":"v519lr8DPW1v"},"source":["## Do prediction on test data"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":200,"status":"ok","timestamp":1620953094422,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"cMGgmbBJlcvb","outputId":"fbc2a261-6f82-42df-a7a4-915cc50f4b8a","vscode":{"languageId":"python"}},"outputs":[{"name":"stdout","output_type":"stream","text":["[[5.94071148]]\n"]}],"source":["# Make a prediction for Cyprus\n","X_new = [[17770]] # Cyprus' GDP per capita\n","print(model.predict(X_new)) # outputs [[5.95199478]]"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"46aihcuVpT0E","vscode":{"languageId":"python"}},"outputs":[],"source":["# Make a prediction for our test data\n","pred = model.predict(test['GDP per capita'].values.reshape(-1,1))"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":55410,"status":"ok","timestamp":1620942331580,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"D9QinOHvItbt","outputId":"8edf9e2d-efb0-4b3f-86ed-e237db854b0b","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["array([[9.16992716],\n"," [5.72868285],\n"," [5.9210396 ],\n"," [5.95199478],\n"," [8.35254892],\n"," [5.59181055],\n"," [5.60481881]])"]},"execution_count":17,"metadata":{"tags":[]},"output_type":"execute_result"}],"source":["pred"]},{"cell_type":"markdown","metadata":{"id":"cUa_ypmpaMLd"},"source":["#Now, lets make it better!\n","\n","Use test dataset and predict the life expectancy using test dataset. "]},{"cell_type":"markdown","metadata":{"id":"m_aeltVpZcp3"},"source":["# Evaluate Model"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"fTHMYUFvZuD9","vscode":{"languageId":"python"}},"outputs":[],"source":["# Lets create train and test dataset, so we can use train dataset for training the model\n","# and use test dataset to evaluate model performance \n","X_train = np.c_[train[\"GDP per capita\"]]\n","y_train = np.c_[train[\"Life satisfaction\"]]\n","\n","X_test = np.c_[test[\"GDP per capita\"]]\n","y_test = np.c_[test[\"Life satisfaction\"]]\n","\n","model = model.fit(X_train,y_train)\n","\n","#Now apply the prediction on test dataset\n","y_pred_test = model.predict(X_test)"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":55401,"status":"ok","timestamp":1620942331581,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"8DdKjw72Z-YN","outputId":"78b5a918-beed-43a6-9326-6e1ff4d9a41f","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["array([[9.16992716],\n"," [5.72868285],\n"," [5.9210396 ],\n"," [5.95199478],\n"," [8.35254892],\n"," [5.59181055],\n"," [5.60481881]])"]},"execution_count":19,"metadata":{"tags":[]},"output_type":"execute_result"}],"source":["# See predictions for yourself\n","y_pred_test"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":55394,"status":"ok","timestamp":1620942331582,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"XgO5TjY8O_QT","outputId":"32af2221-5dd4-4f48-b468-0e8f35fdbf0a","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["1.299498988620812"]},"execution_count":20,"metadata":{"tags":[]},"output_type":"execute_result"}],"source":["from sklearn.metrics import mean_squared_error\n","\n","#MSE: Mean Squared Error as a metric to evaluate a regression model\n","mean_squared_error(y_test, y_pred_test)\n"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":55386,"status":"ok","timestamp":1620942331582,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"JRCvYZHAZmKZ","outputId":"f86ab28d-8b53-4322-9e77-c7ea516d8115","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["1.1399556959026136"]},"execution_count":21,"metadata":{"tags":[]},"output_type":"execute_result"}],"source":["#RMSE\n","from math import sqrt\n","sqrt(mean_squared_error(y_test, y_pred_test))\n"]},{"cell_type":"markdown","metadata":{"id":"4g05y180fTDX"},"source":["# Question:\n","\n","### What would you expect if we normalize data and train the model again?"]},{"cell_type":"markdown","metadata":{"id":"h8hPONDlaxZM"},"source":["# Now, normalize data before prediction"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"YW04HGsLapBM","vscode":{"languageId":"python"}},"outputs":[],"source":["from sklearn.preprocessing import MinMaxScaler\n","# Define Scaling technique\n","scaler = MinMaxScaler()"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":55541,"status":"ok","timestamp":1620942331746,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"hFxrvpPpa5D6","outputId":"c588a578-77fe-4aef-f6ca-2f42f3c01270","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["array([[0.09837989],\n"," [0.11061207],\n"," [0.1146287 ],\n"," [0.14895901],\n"," [0.16771188]])"]},"execution_count":23,"metadata":{"tags":[]},"output_type":"execute_result"}],"source":["# Train escaling object \n","X_train_escaler = scaler.fit(X_train)\n","\n","# Apply scaling model to the data\n","X_train_escaled = X_train_escaler.transform(X_train)\n","X_train_escaled[:5]"]},{"cell_type":"markdown","metadata":{"id":"xcoCKpR1c7LC"},"source":["#Normalize Test Dataset"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":55533,"status":"ok","timestamp":1620942331746,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"7ey1DwyWbClL","outputId":"b4b3e430-37ea-4209-893c-491c762fe654","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["array([[1.39307026],\n"," [0.09468469],\n"," [0.16726112],\n"," [0.17894055],\n"," [1.08467248],\n"," [0.04304261],\n"," [0.04795064]])"]},"execution_count":24,"metadata":{"tags":[]},"output_type":"execute_result"}],"source":["# Apply scaling model to the data\n","X_test_escaled = X_train_escaler.transform(X_test)\n","X_test_escaled"]},{"cell_type":"markdown","metadata":{"id":"-tHTOhKldbOC"},"source":["# Train Models using Scaled Data"]},{"cell_type":"markdown","metadata":{"id":"SGtl94_EddFR"},"source":["## Start training the model using X and y"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"A9olddl0ddFR","vscode":{"languageId":"python"}},"outputs":[],"source":["model.fit(X_train_escaled, y_train)\n","\n","y_pred_escaled = model.predict(X_train_escaled)\n"]},{"cell_type":"markdown","metadata":{"id":"oN8ztq0gddFV"},"source":["## Do prediction on test data"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"NYyxFRRWfH0e","vscode":{"languageId":"python"}},"outputs":[],"source":["y_pred_escaled = model.predict(X_test_escaled)"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":55523,"status":"ok","timestamp":1620942331748,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"kg2h-_PkfNDz","outputId":"a875c7e1-05b9-4b05-9638-8dcf195c8100","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["1.1399556959026143"]},"execution_count":27,"metadata":{"tags":[]},"output_type":"execute_result"}],"source":["#RMSE\n","from math import sqrt\n","sqrt(mean_squared_error(y_test, y_pred_escaled))\n"]},{"cell_type":"markdown","metadata":{"id":"Q4HfXFDKi_Y4"},"source":["# Can you conclude by comparing RMSE from normalized and not normalized dataset?"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"6lW3c6z3fQTU","vscode":{"languageId":"python"}},"outputs":[],"source":[]}],"metadata":{"colab":{"collapsed_sections":[],"name":"People Satisfaction and GDP.ipynb","provenance":[]},"kernelspec":{"display_name":"Python 3","name":"python3"}},"nbformat":4,"nbformat_minor":0} +{"cells":[{"cell_type":"markdown","metadata":{"id":"gseS97WXHRAJ"},"source":["[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1CkvVzG1NtJABPkPRofGLT-MXUCj7veLo#scrollTo=gseS97WXHRAJ)\n"]},{"cell_type":"markdown","metadata":{"id":"-OMhwXX0Euz1"},"source":["# Predict People Satisfaction Across the Globe\n","\n","**Objective:**\n","We would like to build a model that predicts satisfaction score for people of different countries given their country GDP."]},{"cell_type":"markdown","metadata":{"id":"-sSMTG_Buxcm"},"source":["#Note 1: How to enable code completion:\n","\n","Tools menu ==> click on settings ==> Editor ==> Enable \"Automatically trigger code completions\"\n","\n","\n"]},{"cell_type":"markdown","metadata":{"id":"x9L7_OwZix-k"},"source":["#Note 2: Instructions to create a copy of this notebook for youtself\n","\n","You do not have write access to this notebook.\n","\n","* From Menu bar, Go to File, \n","* Select \"Save a copy in my Drive\"\n","* Navigate to Google Drive\n","* Find a folder named \"Collab Notebook\" and open it to find your notebook. \n","* Rename it and start making changes.\n","\n","**Note:** If there is any file you should read in your code, make sure you copy the file from instructor folder to your own Gdrive by following below steps:\n","\n","* Right clicking on the file name \n","* Select \"Make a copy\"\n","* Click on the new file\n","* Move it to desired folder, preferrably where you have your notebook"]},{"cell_type":"markdown","metadata":{"id":"WaAyuEL-EfXy"},"source":["# Download Dataset\n","\n","Download the Better Life Index data (latest edition, currently it is 2022) from the [OECD’s website](http://homl.info/4) as well as stats about GDP per capita from the [IMF’s website](http://homl.info/5). Then you join the tables and sort by GDP per capita. "]},{"cell_type":"markdown","metadata":{"id":"ZWasUlHjGGHc"},"source":["# Import Dataset to Google Colab\n","\n","1. Download CSV and XLS files to your computer\n","2. Upload them to your Google Drive\n","3. Open the CSV files using Google Sheets so Google will create the dataset in format of Google Sheets\n","4. You can remove CSV and XLS files from your drive\n","5. Use the step by step guide from [here](https://colab.research.google.com/notebooks/io.ipynb#scrollTo=vz-jH8T_Uk2c) and scroll down to **\" Google Sheets\" ** cell to import data into dataframe\n","\n","NOTE: After creating Google Sheet into your Drive, make sure you are converting Column 2015 to 0.00 format before importing it into Colab otherwise Google will import it as a string and you will have hard time to clean the data\n","\n","\n"]},{"cell_type":"code","execution_count":6,"metadata":{"executionInfo":{"elapsed":115,"status":"ok","timestamp":1630462416669,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"RhyqGUE9GFTF","vscode":{"languageId":"python"}},"outputs":[],"source":["# Run below line of code for the first time to install gspread. Once installed comment it for future use\n","#!pip install --upgrade -q gspread\n","from google.colab import auth\n","auth.authenticate_user()\n","\n","import gspread\n","from google.auth import default\n","creds, _ = default()\n","\n","gc = gspread.authorize(creds)"]},{"cell_type":"markdown","metadata":{"id":"1QrOorPEHbDg"},"source":["# Use gc to open Google Sheet Datasets"]},{"cell_type":"code","execution_count":7,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":1879,"status":"ok","timestamp":1630462418665,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"4seJiwQyEYVG","outputId":"e88a402a-3ac5-49bc-de2b-bc4f89ff08f2","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["Country\n","Australia 7.3\n","Austria 7\n","Belgium 6.9\n","Brazil 6.6\n","Canada 7.3\n","Name: Life satisfaction, dtype: object"]},"execution_count":7,"metadata":{},"output_type":"execute_result"}],"source":["#Open given sheet\n","worksheet = gc.open('BLI_26092022223032438.xlsx').sheet1\n","\n","# Read contents of CSV file\n","bli_rows = worksheet.get_all_values()\n","\n","# Convert to a DataFrame and render.\n","import pandas as pd\n","bli = pd.DataFrame.from_records(bli_rows, columns = bli_rows[0])\n","\n","# Remove rows where inequality has values other than TOT\n","bli = bli[bli[\"INEQUALITY\"]==\"TOT\"]\n","\n","# Reformat data based on \"indicator column\"\n","bli = bli.pivot(index=\"Country\", columns=\"Indicator\", values=\"Value\")\n","#bli.head()\n","bli[\"Life satisfaction\"].head()"]},{"cell_type":"code","execution_count":8,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":360},"executionInfo":{"elapsed":133,"status":"ok","timestamp":1630462418791,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"PJ42_kqLt7Zq","outputId":"c48e3c42-99a5-4e29-e524-074c1d0500a7","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
IndicatorAir pollutionDwellings without basic facilitiesEducational attainmentEmployees working very long hoursEmployment rateFeeling safe walking alone at nightHomicide rateHousehold net adjusted disposable incomeHousehold net financial wealthHousing expenditureLabour market insecurityLife expectancyLife satisfactionLong-term unemployment ratePersonal earningsQuality of support networkRooms per personSelf-reported healthStakeholder engagement for developing regulationsStudent skillsTime devoted to leisure and personal careVoter turnoutWater qualityYears in education
Country
Australia51.18013.27263.613341757462204.382.57.31.3652063942.3852.750214.35919221.2
Austria161856.787280.70.43254459574212.781.371.9448295921.6701.349214.55759317.1
Belgium152.3754.316270.7129968104084214.881.16.93.9849587922.2752.250315.77898418.2
Brazil106.7497.156437.327.6122277102204.974.76.63.3714024900.8702.239514.45797215.9
Canada70.2913.737380.91.42985085758223.981.57.30.8148403932.588352314.41689116.7
\n","
"],"text/plain":["Indicator Air pollution ... Years in education\n","Country ... \n","Australia 5 ... 21.2\n","Austria 16 ... 17.1\n","Belgium 15 ... 18.2\n","Brazil 10 ... 15.9\n","Canada 7 ... 16.7\n","\n","[5 rows x 24 columns]"]},"execution_count":8,"metadata":{},"output_type":"execute_result"}],"source":["bli[0:5]"]},{"cell_type":"code","execution_count":9,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":236},"executionInfo":{"elapsed":1198,"status":"ok","timestamp":1630462419986,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"01edCO9kHJ2p","outputId":"13ea178f-cce1-4001-e5fe-a5a04840acaa","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
GDP per capita
Country
Afghanistan599.99
Albania3995.38
Algeria4318.14
Angola4100.32
Antigua and Barbuda14414.30
\n","
"],"text/plain":[" GDP per capita\n","Country \n","Afghanistan 599.99\n","Albania 3995.38\n","Algeria 4318.14\n","Angola 4100.32\n","Antigua and Barbuda 14414.30"]},"execution_count":9,"metadata":{},"output_type":"execute_result"}],"source":["#Open given sheet\n","worksheet = gc.open('WEO_Data').sheet1\n","\n","# Read contents of CSV file\n","WEO_rows = worksheet.get_all_values()\n","\n","# Convert to a DataFrame and render.\n","import pandas as pd\n","weo = pd.DataFrame.from_records(WEO_rows, columns = WEO_rows[0])\n","\n","# Drop the header row from data\n","weo = weo.reindex(weo.index.drop(0))\n","\n","# 1- Select only Country name and 2015 \n","# 2- then rename it to GDP Per capita\n","weo = weo[['Country','2,015.00']].rename(columns={'2,015.00':'GDP per capita'})\n","\n","# Set Country as index column\n","# Inplace command, will replace the results of command into the same DF\n","weo.set_index(\"Country\", inplace=True)\n","\n","#weo.drop_duplicates(inplace=True)\n","#Print top 5 rows\n","weo.head()"]},{"cell_type":"markdown","metadata":{"id":"XX_8yJTKNPkt"},"source":["# Merge/Join dataset"]},{"cell_type":"code","execution_count":10,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":360},"executionInfo":{"elapsed":127,"status":"ok","timestamp":1630462420472,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"ZibYb4W-HuEy","outputId":"d943679b-bf91-4a18-cb4f-30a6ac28e098","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
GDP per capitaAir pollutionDwellings without basic facilitiesEducational attainmentEmployees working very long hoursEmployment rateFeeling safe walking alone at nightHomicide rateHousehold net adjusted disposable incomeHousehold net financial wealthHousing expenditureLabour market insecurityLife expectancyLife satisfactionLong-term unemployment ratePersonal earningsQuality of support networkRooms per personSelf-reported healthStakeholder engagement for developing regulationsStudent skillsTime devoted to leisure and personal careVoter turnoutWater qualityYears in education
Country
Luxembourg101994.09120793.7666720.64131774141203.282.46.91.962636922701.548315.15918515.1
Hungary12239.89194.3833.056750.71.21682123289184.875.75.32.4221711841.2561.247415.06627616.6
Poland12495.33222.7916.686566.30.81890614997234.377.662.1425921891.1582.650414.42558017.7
Chile13340.91169.46510.066251.14.51658821409188.179.16.72.0228434841.9571.544314.9496917.3
Latvia13618.571112.9892.096960.76.61526917105236.874.65.93.7322389861.2462.448713.83597717.9
\n","
"],"text/plain":[" GDP per capita Air pollution ... Water quality Years in education\n","Country ... \n","Luxembourg 101994.09 12 ... 85 15.1\n","Hungary 12239.89 19 ... 76 16.6\n","Poland 12495.33 22 ... 80 17.7\n","Chile 13340.91 16 ... 69 17.3\n","Latvia 13618.57 11 ... 77 17.9\n","\n","[5 rows x 25 columns]"]},"execution_count":10,"metadata":{},"output_type":"execute_result"}],"source":["# Now merge BLI and WEO datasets\n","df = pd.merge(left = weo, right = bli, left_index=True, right_index=True)\n","\n","#sort the dataframe by GPD per capita\n","df.sort_values(by=\"GDP per capita\", inplace=True)\n","df.head()\n"]},{"cell_type":"code","execution_count":11,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":4,"status":"ok","timestamp":1630462420888,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"eoJn-lxSPfqP","outputId":"543cdb12-e9a2-40cd-a9e5-48b5f0501525","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["GDP per capita 13340.91\n","Air pollution 16\n","Dwellings without basic facilities 9.4\n","Educational attainment 65\n","Employees working very long hours 10.06\n","Employment rate 62\n","Feeling safe walking alone at night 51.1\n","Homicide rate 4.5\n","Household net adjusted disposable income 16588\n","Household net financial wealth 21409\n","Housing expenditure 18\n","Labour market insecurity 8.1\n","Life expectancy 79.1\n","Life satisfaction 6.7\n","Long-term unemployment rate 2.02\n","Personal earnings 28434\n","Quality of support network 84\n","Rooms per person 1.9\n","Self-reported health 57\n","Stakeholder engagement for developing regulations 1.5\n","Student skills 443\n","Time devoted to leisure and personal care 14.9\n","Voter turnout 49\n","Water quality 69\n","Years in education 17.3\n","Name: Chile, dtype: object"]},"execution_count":11,"metadata":{},"output_type":"execute_result"}],"source":["df.iloc[3]"]},{"cell_type":"markdown","metadata":{"id":"MkSTXnclPGLd"},"source":["## Split dataset into Train & Test"]},{"cell_type":"code","execution_count":12,"metadata":{"executionInfo":{"elapsed":2,"status":"ok","timestamp":1630462423352,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"6HOaSt93Nqkz","vscode":{"languageId":"python"}},"outputs":[],"source":["# Below snippet is the most basic way of splitting data in Python; It is for illustration only and in \n","# later sections we will use proper library from Sklearn to split the data\n","test_indices = [0, 1, 6, 8, 33, 34, 35]\n","\n","#define the indices of the training set by substracting the test indices from all indices\n","train_indices = list(set(range(36)) - set(test_indices))\n","\n","#use the above indices to select parts of the dataframe as the training set and the test set\n","train = df[[\"GDP per capita\", 'Life satisfaction']].iloc[train_indices]\n","test = df[[\"GDP per capita\", 'Life satisfaction']].iloc[test_indices]"]},{"cell_type":"code","execution_count":13,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":236},"executionInfo":{"elapsed":116,"status":"ok","timestamp":1630462431988,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"9CRVFFEX_voT","outputId":"fd22091b-f2d0-42ef-f5c2-ad591827f287","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
GDP per capitaLife satisfaction
Country
Poland12495.336
Chile13340.916.7
Latvia13618.575.9
Slovak Republic15991.746.1
Estonia17288.085.6
\n","
"],"text/plain":[" GDP per capita Life satisfaction\n","Country \n","Poland 12495.33 6\n","Chile 13340.91 6.7\n","Latvia 13618.57 5.9\n","Slovak Republic 15991.74 6.1\n","Estonia 17288.08 5.6"]},"execution_count":13,"metadata":{},"output_type":"execute_result"}],"source":["train.head()"]},{"cell_type":"code","execution_count":14,"metadata":{"executionInfo":{"elapsed":1190,"status":"ok","timestamp":1630462433448,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"NqZnpDGyR_3f","vscode":{"languageId":"python"}},"outputs":[],"source":["# Code example\n","import matplotlib.pyplot as plt\n","import numpy as np\n","import pandas as pd\n","import sklearn.linear_model\n","\n","# Prepare the data\n","X = np.c_[train[\"GDP per capita\"]]\n","y = np.c_[train[\"Life satisfaction\"]]"]},{"cell_type":"code","execution_count":15,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":8,"status":"ok","timestamp":1630462433450,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"UzRnfo8yAEbI","outputId":"cd44d876-24ab-41b3-93da-1e346a10bfd3","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["numpy.ndarray"]},"execution_count":15,"metadata":{},"output_type":"execute_result"}],"source":["type(y)"]},{"cell_type":"code","execution_count":16,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":4,"status":"ok","timestamp":1630462433450,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"BzUJ0hnrwYgd","outputId":"138676bc-658c-404f-de0f-3c0814a7cd26","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["(array([['12495.33'],\n"," ['13340.91'],\n"," ['13618.57'],\n"," ['15991.74'],\n"," ['17288.08']], dtype=object), array([['6'],\n"," ['6.7'],\n"," ['5.9'],\n"," ['6.1'],\n"," ['5.6']], dtype=object))"]},"execution_count":16,"metadata":{},"output_type":"execute_result"}],"source":["#Lets look at what is inside X and y. Print first 5 records\n","X[:5], y[:5]"]},{"cell_type":"markdown","metadata":{"id":"WJFLcCBgPPXj"},"source":["# ML Starts Now\n","## Define a model with default values"]},{"cell_type":"code","execution_count":18,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":88,"status":"ok","timestamp":1630462527023,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"_mimmXfQNTD1","outputId":"30897cd6-848d-41eb-bcde-d077e4a38ec9","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False)"]},"execution_count":18,"metadata":{},"output_type":"execute_result"}],"source":["# Select a basic linear model without setting any parameter (nothing inside paranthesis below)\n","model = sklearn.linear_model.LinearRegression()\n","\n","# See the model for yourself\n","model"]},{"cell_type":"markdown","metadata":{"id":"0pf1NGpkPSnD"},"source":["## Start training the model using X and y"]},{"cell_type":"code","execution_count":19,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":98,"status":"ok","timestamp":1630462530244,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"NySTaTDMp_Ve","outputId":"eb0114ae-4a8f-493e-f1e6-d185b48f24b9","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False)"]},"execution_count":19,"metadata":{},"output_type":"execute_result"}],"source":["# Train the model\n","model.fit(X, y)"]},{"cell_type":"code","execution_count":20,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":89,"status":"ok","timestamp":1630462536558,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"YzVQ4lQors1b","outputId":"7dc7cedc-4547-4f56-e759-41083f5b204b","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["array([[3.83407608e-05]])"]},"execution_count":20,"metadata":{},"output_type":"execute_result"}],"source":["model.coef_"]},{"cell_type":"code","execution_count":21,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":92,"status":"ok","timestamp":1630462542604,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"KF0UFvZxsgPA","outputId":"148f5a1e-91b7-4c11-cdd8-eeaed8d5583a","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["array([5.25939616])"]},"execution_count":21,"metadata":{},"output_type":"execute_result"}],"source":["model.intercept_"]},{"cell_type":"markdown","metadata":{"id":"v519lr8DPW1v"},"source":["## Do prediction on test data"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":200,"status":"ok","timestamp":1620953094422,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"cMGgmbBJlcvb","outputId":"fbc2a261-6f82-42df-a7a4-915cc50f4b8a","vscode":{"languageId":"python"}},"outputs":[{"name":"stdout","output_type":"stream","text":["[[5.94071148]]\n"]}],"source":["# Make a prediction for Cyprus\n","X_new = [[17770]] # Cyprus' GDP per capita\n","print(model.predict(X_new)) # outputs [[5.95199478]]"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"46aihcuVpT0E","vscode":{"languageId":"python"}},"outputs":[],"source":["# Make a prediction for our test data\n","pred = model.predict(test['GDP per capita'].values.reshape(-1,1))"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":55410,"status":"ok","timestamp":1620942331580,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"D9QinOHvItbt","outputId":"8edf9e2d-efb0-4b3f-86ed-e237db854b0b","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["array([[9.16992716],\n"," [5.72868285],\n"," [5.9210396 ],\n"," [5.95199478],\n"," [8.35254892],\n"," [5.59181055],\n"," [5.60481881]])"]},"execution_count":17,"metadata":{"tags":[]},"output_type":"execute_result"}],"source":["pred"]},{"cell_type":"markdown","metadata":{"id":"cUa_ypmpaMLd"},"source":["#Now, lets make it better!\n","\n","Use test dataset and predict the life expectancy using test dataset. "]},{"cell_type":"markdown","metadata":{"id":"m_aeltVpZcp3"},"source":["# Evaluate Model"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"fTHMYUFvZuD9","vscode":{"languageId":"python"}},"outputs":[],"source":["# Lets create train and test dataset, so we can use train dataset for training the model\n","# and use test dataset to evaluate model performance \n","X_train = np.c_[train[\"GDP per capita\"]]\n","y_train = np.c_[train[\"Life satisfaction\"]]\n","\n","X_test = np.c_[test[\"GDP per capita\"]]\n","y_test = np.c_[test[\"Life satisfaction\"]]\n","\n","model = model.fit(X_train,y_train)\n","\n","#Now apply the prediction on test dataset\n","y_pred_test = model.predict(X_test)"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":55401,"status":"ok","timestamp":1620942331581,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"8DdKjw72Z-YN","outputId":"78b5a918-beed-43a6-9326-6e1ff4d9a41f","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["array([[9.16992716],\n"," [5.72868285],\n"," [5.9210396 ],\n"," [5.95199478],\n"," [8.35254892],\n"," [5.59181055],\n"," [5.60481881]])"]},"execution_count":19,"metadata":{"tags":[]},"output_type":"execute_result"}],"source":["# See predictions for yourself\n","y_pred_test"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":55394,"status":"ok","timestamp":1620942331582,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"XgO5TjY8O_QT","outputId":"32af2221-5dd4-4f48-b468-0e8f35fdbf0a","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["1.299498988620812"]},"execution_count":20,"metadata":{"tags":[]},"output_type":"execute_result"}],"source":["from sklearn.metrics import mean_squared_error\n","\n","#MSE: Mean Squared Error as a metric to evaluate a regression model\n","mean_squared_error(y_test, y_pred_test)\n"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":55386,"status":"ok","timestamp":1620942331582,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"JRCvYZHAZmKZ","outputId":"f86ab28d-8b53-4322-9e77-c7ea516d8115","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["1.1399556959026136"]},"execution_count":21,"metadata":{"tags":[]},"output_type":"execute_result"}],"source":["#RMSE\n","from math import sqrt\n","sqrt(mean_squared_error(y_test, y_pred_test))\n"]},{"cell_type":"markdown","metadata":{"id":"4g05y180fTDX"},"source":["# Question:\n","\n","### What would you expect if we normalize data and train the model again?"]},{"cell_type":"markdown","metadata":{"id":"h8hPONDlaxZM"},"source":["# Now, normalize data before prediction"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"YW04HGsLapBM","vscode":{"languageId":"python"}},"outputs":[],"source":["from sklearn.preprocessing import MinMaxScaler\n","# Define Scaling technique\n","scaler = MinMaxScaler()"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":55541,"status":"ok","timestamp":1620942331746,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"hFxrvpPpa5D6","outputId":"c588a578-77fe-4aef-f6ca-2f42f3c01270","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["array([[0.09837989],\n"," [0.11061207],\n"," [0.1146287 ],\n"," [0.14895901],\n"," [0.16771188]])"]},"execution_count":23,"metadata":{"tags":[]},"output_type":"execute_result"}],"source":["# Train escaling object \n","X_train_escaler = scaler.fit(X_train)\n","\n","# Apply scaling model to the data\n","X_train_escaled = X_train_escaler.transform(X_train)\n","X_train_escaled[:5]"]},{"cell_type":"markdown","metadata":{"id":"xcoCKpR1c7LC"},"source":["#Normalize Test Dataset"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":55533,"status":"ok","timestamp":1620942331746,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"7ey1DwyWbClL","outputId":"b4b3e430-37ea-4209-893c-491c762fe654","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["array([[1.39307026],\n"," [0.09468469],\n"," [0.16726112],\n"," [0.17894055],\n"," [1.08467248],\n"," [0.04304261],\n"," [0.04795064]])"]},"execution_count":24,"metadata":{"tags":[]},"output_type":"execute_result"}],"source":["# Apply scaling model to the data\n","X_test_escaled = X_train_escaler.transform(X_test)\n","X_test_escaled"]},{"cell_type":"markdown","metadata":{"id":"-tHTOhKldbOC"},"source":["# Train Models using Scaled Data"]},{"cell_type":"markdown","metadata":{"id":"SGtl94_EddFR"},"source":["## Start training the model using X and y"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"A9olddl0ddFR","vscode":{"languageId":"python"}},"outputs":[],"source":["model.fit(X_train_escaled, y_train)\n","\n","y_pred_escaled = model.predict(X_train_escaled)\n"]},{"cell_type":"markdown","metadata":{"id":"oN8ztq0gddFV"},"source":["## Do prediction on test data"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"NYyxFRRWfH0e","vscode":{"languageId":"python"}},"outputs":[],"source":["y_pred_escaled = model.predict(X_test_escaled)"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":55523,"status":"ok","timestamp":1620942331748,"user":{"displayName":"Saeid Abolfazli","photoUrl":"","userId":"03288941312928329554"},"user_tz":240},"id":"kg2h-_PkfNDz","outputId":"a875c7e1-05b9-4b05-9638-8dcf195c8100","vscode":{"languageId":"python"}},"outputs":[{"data":{"text/plain":["1.1399556959026143"]},"execution_count":27,"metadata":{"tags":[]},"output_type":"execute_result"}],"source":["#RMSE\n","from math import sqrt\n","sqrt(mean_squared_error(y_test, y_pred_escaled))\n"]},{"cell_type":"markdown","metadata":{"id":"Q4HfXFDKi_Y4"},"source":["# Can you conclude by comparing RMSE from normalized and not normalized dataset?"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"6lW3c6z3fQTU","vscode":{"languageId":"python"}},"outputs":[],"source":[]}],"metadata":{"colab":{"collapsed_sections":[],"name":"People Satisfaction and GDP.ipynb","provenance":[]},"kernelspec":{"display_name":"Python 3","name":"python3"}},"nbformat":4,"nbformat_minor":0}