{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "![MLU Logo](../../data/MLU_Logo.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Responsible AI - Final Project Solution\n", "\n", "Build a fair [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) that predicts the __credit_risk__ field (whether some is a credit risk or not) of the [German Credit Dataset](https://archive.ics.uci.edu/ml/datasets/South+German+Credit+%28UPDATE%29).\n", "\n", "### Final Project Problem: Loan Approval\n", "\n", "__Problem Definition:__\n", "Given a set of features about an individual (e.g. age, past credit history, immigration status, ...) predict whether a loan is repaid or not (is customer a credit risk). We impose the additional constraint that the model should be fair with respect to different age groups ($\\geq$ 25 yrs and $<$ 25 yrs).\n", "\n", "In the banking industry, there are certain regulations regarding the use of sensitive features (e.g., age, ethnicity, marital status, ...). According to those regulations, it would not be okay if age played a significant role in the model (loans should be approved/denied regardless of an individuals' age).\n", "\n", "\n", "``` \n", "F. Kamiran and T. Calders, \"Data Preprocessing Techniques for Classification without Discrimination,\" Knowledge and Information Systems, 2012\n", "```\n", "\n", "1. Read the datasets (Given) \n", "2. Data Processing (Implement)\n", " * Exploratory Data Analysis\n", " * Select features to build the model (Suggested)\n", " * Train - Validation - Test Datasets\n", " * Feature transformation\n", "3. Train a Classifier on the Training Dataset (Implement)\n", "4. Make Predictions on the Test Dataset (Implement)\n", "5. Evaluate Results (Given)\n", "\n", "\n", "__Datasets and Files:__\n", "\n", "\n", "- ```german_credit_training.csv```: Training data with loan applicants features, credit history, dependents, savings, account status, age group (and more). The label is __credit_risk__.\n", "\n", "- ```german_credit_test.csv```: Test data with same features as above apart from label. This will be the data to make predictions for to emulate a production environment." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This notebook assumes an installation of the SageMaker kernel `conda_pytorch_p39`. In addition, libraries from a requirements.txt need to be installed:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "tags": [] }, "outputs": [], "source": [ "!pip install --no-deps -U -q -r ../../requirements.txt" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "tags": [] }, "outputs": [], "source": [ "%%capture\n", "\n", "# Reshaping/basic libraries\n", "import pandas as pd\n", "import numpy as np\n", "\n", "# Plotting libraries\n", "import matplotlib.pyplot as plt\n", "\n", "%matplotlib inline\n", "import seaborn as sns\n", "\n", "sns.set_style(\"darkgrid\", {\"axes.facecolor\": \".9\"})\n", "\n", "# ML libraries\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.metrics import confusion_matrix, accuracy_score, f1_score\n", "from sklearn.impute import SimpleImputer\n", "from sklearn.preprocessing import OneHotEncoder, MinMaxScaler\n", "from sklearn.pipeline import Pipeline\n", "from sklearn.compose import ColumnTransformer\n", "from sklearn.linear_model import LogisticRegression\n", "\n", "# Operational libraries\n", "import sys\n", "\n", "sys.path.append(\"..\")\n", "sys.path.insert(1, \"..\")\n", "\n", "# Fairness libraries\n", "from folktables.acs import *\n", "from folktables.folktables import *\n", "from folktables.load_acs import *\n", "from fairlearn.reductions import EqualizedOdds\n", "from fairlearn.postprocessing import ThresholdOptimizer\n", "from fairlearn.metrics import MetricFrame, selection_rate\n", "\n", "# Jupyter(lab) libraries\n", "import warnings\n", "\n", "warnings.filterwarnings(\"ignore\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Read the datasets (Given)\n", "(Go to top)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then, we read the __training__ and __test__ datasets into dataframes, using [Pandas](https://pandas.pydata.org/pandas-docs/stable/user_guide/10min.html). This library allows us to read and manipulate our data." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The shape of the training dataset is: (800, 24)\n", "The shape of the test dataset is: (200, 23)\n" ] } ], "source": [ "training_data = pd.read_csv(\"../../data/final_project/german_credit_training.csv\")\n", "test_data = pd.read_csv(\"../../data/final_project/german_credit_test.csv\")\n", "\n", "print(\"The shape of the training dataset is:\", training_data.shape)\n", "print(\"The shape of the test dataset is:\", test_data.shape)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Data Processing (Implement)\n", "(Go to top) " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.1 Exploratory Data Analysis\n", "(Go to Data Processing)\n", "\n", "We look at number of rows, columns, and some simple statistics of the datasets." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
IDsexpersonal_statuschecking_account_statuscredit_duration_monthscredit_historycredit_purposecredit_amountsavingsemployed_since_years...age_yearsother_installment_planshousingnum_existing_creditsjob_statusnum_people_liable_fortelephoneforeign_workercredit_riskage_groups
0564malemarried/widowed... < 0 DM24existing credits paid back duly till nowretraining4712.. >= 1000 DM1 <= ... < 4 years...37bankrent2-3management/self-employed/highly qualified empl...0 to 2TrueFalse01
1484malemarried/widowed... >= 200 DM12all credits at this bank paid back dulyothers1163100 <= ... < 500 DM1 <= ... < 4 years...44nonerent1skilled employee/official0 to 2TrueFalse01
226femalesingle... >= 200 DM6delay in paying off in the pastfurniture/equipment426unknown/no savings account.. >= 7 years...39nonerent1unskilled - resident0 to 2FalseFalse01
3443malemarried/widowed... >= 200 DM12no credits taken/all credits paid back dulyrepairs719unknown/no savings account.. >= 7 years...41bankrent1unskilled - resident3 or moreFalseFalse11
4211female/malefemale (non-single) or male (single)... >= 200 DM36no credits taken/all credits paid back dulyfurniture/equipment3835.. >= 1000 DM.. >= 7 years...45nonerent1unskilled - resident0 to 2TrueFalse01
\n", "

5 rows × 24 columns

\n", "
" ], "text/plain": [ " ID sex personal_status \\\n", "0 564 male married/widowed \n", "1 484 male married/widowed \n", "2 26 female single \n", "3 443 male married/widowed \n", "4 211 female/male female (non-single) or male (single) \n", "\n", " checking_account_status credit_duration_months \\\n", "0 ... < 0 DM 24 \n", "1 ... >= 200 DM 12 \n", "2 ... >= 200 DM 6 \n", "3 ... >= 200 DM 12 \n", "4 ... >= 200 DM 36 \n", "\n", " credit_history credit_purpose \\\n", "0 existing credits paid back duly till now retraining \n", "1 all credits at this bank paid back duly others \n", "2 delay in paying off in the past furniture/equipment \n", "3 no credits taken/all credits paid back duly repairs \n", "4 no credits taken/all credits paid back duly furniture/equipment \n", "\n", " credit_amount savings employed_since_years ... \\\n", "0 4712 .. >= 1000 DM 1 <= ... < 4 years ... \n", "1 1163 100 <= ... < 500 DM 1 <= ... < 4 years ... \n", "2 426 unknown/no savings account .. >= 7 years ... \n", "3 719 unknown/no savings account .. >= 7 years ... \n", "4 3835 .. >= 1000 DM .. >= 7 years ... \n", "\n", " age_years other_installment_plans housing num_existing_credits \\\n", "0 37 bank rent 2-3 \n", "1 44 none rent 1 \n", "2 39 none rent 1 \n", "3 41 bank rent 1 \n", "4 45 none rent 1 \n", "\n", " job_status num_people_liable_for \\\n", "0 management/self-employed/highly qualified empl... 0 to 2 \n", "1 skilled employee/official 0 to 2 \n", "2 unskilled - resident 0 to 2 \n", "3 unskilled - resident 3 or more \n", "4 unskilled - resident 0 to 2 \n", "\n", " telephone foreign_worker credit_risk age_groups \n", "0 True False 0 1 \n", "1 True False 0 1 \n", "2 False False 0 1 \n", "3 False False 1 1 \n", "4 True False 0 1 \n", "\n", "[5 rows x 24 columns]" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "training_data.head()" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
IDsexpersonal_statuschecking_account_statuscredit_duration_monthscredit_historycredit_purposecredit_amountsavingsemployed_since_years...propertyage_yearsother_installment_planshousingnum_existing_creditsjob_statusnum_people_liable_fortelephoneforeign_workerage_groups
0963malemarried/widowed... >= 200 DM24no credits taken/all credits paid back dulyfurniture/equipment2397100 <= ... < 500 DM.. >= 7 years...building society savings agreement/life insurance35bankrent2-3skilled employee/official0 to 2TrueFalse1
1611female/malefemale (non-single) or male (single)0 <= ... < 200 DM10no credits taken/all credits paid back dulyothers1240... < 100 DM.. >= 7 years...real estate48noneown1unskilled - resident3 or moreFalseFalse1
2106malemarried/widowed... >= 200 DM18critical account/other credits existing (not a...others6458unknown/no savings account.. >= 7 years...real estate39bankrent2-3management/self-employed/highly qualified empl...3 or moreTrueFalse1
3891malemarried/widowed... >= 200 DM15all credits at this bank paid back dulyfurniture/equipment1829unknown/no savings account.. >= 7 years...building society savings agreement/life insurance46nonerent2-3skilled employee/official0 to 2TrueFalse1
4342femalesingle... < 0 DM18no credits taken/all credits paid back dulyfurniture/equipment3213100 <= ... < 500 DM... < 1 year...unknown/no property25nonefor free1skilled employee/official0 to 2FalseFalse1
\n", "

5 rows × 23 columns

\n", "
" ], "text/plain": [ " ID sex personal_status \\\n", "0 963 male married/widowed \n", "1 611 female/male female (non-single) or male (single) \n", "2 106 male married/widowed \n", "3 891 male married/widowed \n", "4 342 female single \n", "\n", " checking_account_status credit_duration_months \\\n", "0 ... >= 200 DM 24 \n", "1 0 <= ... < 200 DM 10 \n", "2 ... >= 200 DM 18 \n", "3 ... >= 200 DM 15 \n", "4 ... < 0 DM 18 \n", "\n", " credit_history credit_purpose \\\n", "0 no credits taken/all credits paid back duly furniture/equipment \n", "1 no credits taken/all credits paid back duly others \n", "2 critical account/other credits existing (not a... others \n", "3 all credits at this bank paid back duly furniture/equipment \n", "4 no credits taken/all credits paid back duly furniture/equipment \n", "\n", " credit_amount savings employed_since_years ... \\\n", "0 2397 100 <= ... < 500 DM .. >= 7 years ... \n", "1 1240 ... < 100 DM .. >= 7 years ... \n", "2 6458 unknown/no savings account .. >= 7 years ... \n", "3 1829 unknown/no savings account .. >= 7 years ... \n", "4 3213 100 <= ... < 500 DM ... < 1 year ... \n", "\n", " property age_years \\\n", "0 building society savings agreement/life insurance 35 \n", "1 real estate 48 \n", "2 real estate 39 \n", "3 building society savings agreement/life insurance 46 \n", "4 unknown/no property 25 \n", "\n", " other_installment_plans housing num_existing_credits \\\n", "0 bank rent 2-3 \n", "1 none own 1 \n", "2 bank rent 2-3 \n", "3 none rent 2-3 \n", "4 none for free 1 \n", "\n", " job_status num_people_liable_for \\\n", "0 skilled employee/official 0 to 2 \n", "1 unskilled - resident 3 or more \n", "2 management/self-employed/highly qualified empl... 3 or more \n", "3 skilled employee/official 0 to 2 \n", "4 skilled employee/official 0 to 2 \n", "\n", " telephone foreign_worker age_groups \n", "0 True False 1 \n", "1 False False 1 \n", "2 True False 1 \n", "3 True False 1 \n", "4 False False 1 \n", "\n", "[5 rows x 23 columns]" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "test_data.head()" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "tags": [] }, "outputs": [], "source": [ "# Implement more EDA here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.2 Select features to build the model \n", "(Go to Data Processing)\n", "\n", "Let's use all the features. Below you see a snippet of code that separates categorical and numerical columns based on their data type. This should only be used if we are sure that the data types are correctly assigned (check during EDA). Mindful with some of the feature names - they suggest numerical values but upon inspection it should become clear that they are actually categoricals (e.g. `employed_since_years` has been binned into groups).\n" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Categorical columns: ['sex', 'personal_status', 'checking_account_status', 'credit_history', 'credit_purpose', 'savings', 'employed_since_years', 'installment_rate', 'other_debtors_guarantors', 'present_residence_since', 'property', 'other_installment_plans', 'housing', 'num_existing_credits', 'job_status', 'num_people_liable_for']\n", "\n", "Numerical columns: ['ID', 'credit_duration_months', 'credit_amount', 'age_years', 'age_groups']\n" ] } ], "source": [ "# Grab model features/inputs and target/output\n", "categorical_features = (\n", " training_data.drop(\"credit_risk\", axis=1)\n", " .select_dtypes(include=\"object\")\n", " .columns.tolist()\n", ")\n", "print(\"Categorical columns:\", categorical_features)\n", "\n", "print(\"\")\n", "\n", "numerical_features = (\n", " training_data.drop(\"credit_risk\", axis=1)\n", " .select_dtypes(include=np.number)\n", " .columns.tolist()\n", ")\n", "print(\"Numerical columns:\", numerical_features)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We notice that `ID` is identified as numerical column. ID's should never be used as features for training as they are unique by row. Let's drop the ID from the model features after we have separated target and features. Also make sure to remove the sensitive feature so it does not end up as input for training." ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model features: ['sex', 'personal_status', 'checking_account_status', 'credit_history', 'credit_purpose', 'savings', 'employed_since_years', 'installment_rate', 'other_debtors_guarantors', 'present_residence_since', 'property', 'other_installment_plans', 'housing', 'num_existing_credits', 'job_status', 'num_people_liable_for', 'ID', 'credit_duration_months', 'credit_amount', 'age_years']\n", "\n", "\n", "Model target: credit_risk\n" ] } ], "source": [ "sensitive_feature = \"age_groups\"\n", "\n", "try:\n", " numerical_features.remove(sensitive_feature)\n", "except:\n", " pass\n", "\n", "try:\n", " categorical_features.remove(sensitive_feature)\n", "except:\n", " pass\n", "\n", "model_target = \"credit_risk\"\n", "model_features = categorical_features + numerical_features\n", "\n", "print(\"Model features: \", model_features)\n", "print(\"\\n\")\n", "print(\"Model target: \", model_target)" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "tags": [] }, "outputs": [], "source": [ "to_remove = \"ID\"\n", "\n", "# Drop 'ID' feature from the respective list(s)\n", "if to_remove in model_features:\n", " model_features.remove(to_remove)\n", "if to_remove in categorical_features:\n", " categorical_features.remove(to_remove)\n", "if to_remove in numerical_features:\n", " numerical_features.remove(to_remove)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's also remove `age_years` as this is an obvious proxy for the age groups." ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "tags": [] }, "outputs": [], "source": [ "to_remove = \"age_years\"\n", "\n", "# Drop 'ID' feature from the respective list(s)\n", "if to_remove in model_features:\n", " model_features.remove(to_remove)\n", "if to_remove in categorical_features:\n", " categorical_features.remove(to_remove)\n", "if to_remove in numerical_features:\n", " numerical_features.remove(to_remove)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.3 Feature transformation\n", "(Go to Data Processing)\n", "\n", "Here, you have different options. You can use Reweighing, Disparate Impact Remover or Suppression. However, in this notebook you should try to implement Equalized Odds postprocessing. Therefore, no transformation is required at this point." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.4 Train - Validation Datasets\n", "(Go to Data Processing)\n", "\n", "We already have training and test datasets, but no validation dataset (which you need to create). Furthermore, the test dataset is missing the labels - the goal of the project is to predict these labels. \n", "\n", "To produce a validation set to evaluate model performance, split the training dataset into train and validation subsets using sklearn's [train_test_split()](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) function. " ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "tags": [] }, "outputs": [], "source": [ "# Implement here" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Train - Test - Validation datasets shapes: (720, 24) (200, 23) (80, 24)\n" ] } ], "source": [ "# We only need to split between train and val (test is already separate)\n", "train_data, val_data = train_test_split(\n", " training_data, test_size=0.1, shuffle=True, random_state=23\n", ")\n", "\n", "# Print the shapes of the Train - Test Datasets\n", "print(\n", " \"Train - Test - Validation datasets shapes: \",\n", " train_data.shape,\n", " test_data.shape,\n", " val_data.shape,\n", ")\n", "\n", "train_data.reset_index(inplace=True, drop=True)\n", "val_data.reset_index(inplace=True, drop=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.5 Data processing with Pipeline\n", "(Go to Data Processing)\n", "\n", "Build a [pipeline](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html) to impute the missing values and scale the numerical features, and finally train a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) on the imputed and scaled dataset. \n" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "tags": [] }, "outputs": [], "source": [ "# Implement here" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "
Pipeline(steps=[('data_processing',\n",
       "                 ColumnTransformer(transformers=[('numerical_processing',\n",
       "                                                  Pipeline(steps=[('num_imputer',\n",
       "                                                                   SimpleImputer()),\n",
       "                                                                  ('num_scaler',\n",
       "                                                                   MinMaxScaler())]),\n",
       "                                                  ['credit_duration_months',\n",
       "                                                   'credit_amount']),\n",
       "                                                 ('categorical_processing',\n",
       "                                                  Pipeline(steps=[('cat_imputer',\n",
       "                                                                   SimpleImputer(fill_value='missing',\n",
       "                                                                                 strategy='constant')),\n",
       "                                                                  ('cat_encoder',\n",
       "                                                                   OneHot...))]),\n",
       "                                                  ['sex', 'personal_status',\n",
       "                                                   'checking_account_status',\n",
       "                                                   'credit_history',\n",
       "                                                   'credit_purpose', 'savings',\n",
       "                                                   'employed_since_years',\n",
       "                                                   'installment_rate',\n",
       "                                                   'other_debtors_guarantors',\n",
       "                                                   'present_residence_since',\n",
       "                                                   'property',\n",
       "                                                   'other_installment_plans',\n",
       "                                                   'housing',\n",
       "                                                   'num_existing_credits',\n",
       "                                                   'job_status',\n",
       "                                                   'num_people_liable_for'])])),\n",
       "                ('lg', LogisticRegression(random_state=0))])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
" ], "text/plain": [ "Pipeline(steps=[('data_processing',\n", " ColumnTransformer(transformers=[('numerical_processing',\n", " Pipeline(steps=[('num_imputer',\n", " SimpleImputer()),\n", " ('num_scaler',\n", " MinMaxScaler())]),\n", " ['credit_duration_months',\n", " 'credit_amount']),\n", " ('categorical_processing',\n", " Pipeline(steps=[('cat_imputer',\n", " SimpleImputer(fill_value='missing',\n", " strategy='constant')),\n", " ('cat_encoder',\n", " OneHot...))]),\n", " ['sex', 'personal_status',\n", " 'checking_account_status',\n", " 'credit_history',\n", " 'credit_purpose', 'savings',\n", " 'employed_since_years',\n", " 'installment_rate',\n", " 'other_debtors_guarantors',\n", " 'present_residence_since',\n", " 'property',\n", " 'other_installment_plans',\n", " 'housing',\n", " 'num_existing_credits',\n", " 'job_status',\n", " 'num_people_liable_for'])])),\n", " ('lg', LogisticRegression(random_state=0))])" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "### STEP 1 ###\n", "##############\n", "\n", "# Preprocess the numerical features\n", "numerical_processor = Pipeline(\n", " [(\"num_imputer\", SimpleImputer(strategy=\"mean\")), (\"num_scaler\", MinMaxScaler())]\n", ")\n", "# Preprocess the categorical features\n", "categorical_processor = Pipeline(\n", " [\n", " (\"cat_imputer\", SimpleImputer(strategy=\"constant\", fill_value=\"missing\")),\n", " (\"cat_encoder\", OneHotEncoder(handle_unknown=\"ignore\")),\n", " ]\n", ")\n", "\n", "### STEP 2 ###\n", "##############\n", "\n", "# Combine all data preprocessors from above\n", "data_processor = ColumnTransformer(\n", " [\n", " (\"numerical_processing\", numerical_processor, numerical_features),\n", " (\"categorical_processing\", categorical_processor, categorical_features),\n", " ]\n", ")\n", "\n", "\n", "### STEP 3 ###\n", "##############\n", "\n", "# Pipeline desired all data transformers, along with an estimator at the end\n", "# Later you can set/reach the parameters using the names issued - for hyperparameter tuning, for example\n", "pipeline = Pipeline(\n", " [\n", " (\"data_processing\", data_processor),\n", " (\"lg\", LogisticRegression(random_state=0)),\n", " ]\n", ")\n", "\n", "# Visualize the pipeline\n", "# This will come in handy especially when building more complex pipelines, stringing together multiple preprocessing steps\n", "from sklearn import set_config\n", "\n", "set_config(display=\"diagram\")\n", "pipeline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Train (and Tune) a Classifier (Implement)\n", "(Go to top)\n", "\n", "Train (and tune) the [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) pipeline. For tuning, you can try different imputation strategies, different scaling methods." ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "tags": [] }, "outputs": [], "source": [ "# Implement here" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "
ThresholdOptimizer(constraints='equalized_odds',\n",
       "                   estimator=LogisticRegression(random_state=0))
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
" ], "text/plain": [ "ThresholdOptimizer(constraints='equalized_odds',\n", " estimator=LogisticRegression(random_state=0))" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Get train data to train the classifier\n", "X_train = train_data[model_features]\n", "y_train = train_data[model_target]\n", "\n", "# Learn the transformation & extract feature names\n", "data_processor.fit(X_train)\n", "\n", "# To extract feature names we first need to fit the data processor as this will generate the one hot encoding\n", "ft_names = numerical_features + list(\n", " data_processor.transformers_[1][1]\n", " .named_steps[\"cat_encoder\"]\n", " .get_feature_names(categorical_features)\n", ")\n", "\n", "# Add column names and convert to data frame\n", "X_train_prep = pd.DataFrame(\n", " data_processor.transform(X_train).todense(), columns=ft_names\n", ")\n", "\n", "# Set up ThresholdOptimizer\n", "eo_model = ThresholdOptimizer(\n", " estimator=pipeline[-1],\n", " constraints=\"equalized_odds\",\n", " objective=\"accuracy_score\",\n", " grid_size=1000,\n", " flip=False,\n", " prefit=False,\n", " predict_method=\"deprecated\",\n", ")\n", "\n", "# Adjust the results that the classifier would produce by letting ThresholdOptimizer know what the sensitive features are\n", "eo_model.fit(X_train_prep, y_train, sensitive_features=train_data[\"age_groups\"].values)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. Make Predictions on the Test Dataset (Implement)\n", "(Go to top)\n", "\n", "Use the trained classifier to predict the labels on the test set. Below you will find a code snippet that evaluates for DI." ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "tags": [] }, "outputs": [], "source": [ "# Implement here\n", "\n", "# Get test data to test the classifier\n", "# ! test data should come from german_credit_test.csv !\n", "# ...\n", "\n", "# Use the trained model to make predictions on the test dataset\n", "# test_predictions = ..." ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "tags": [] }, "outputs": [], "source": [ "# You can now use the fitted equalized odds post-processor and use it to create adjusted outcomes (predictions)\n", "test_predictions = eo_model.predict(\n", " data_processor.transform(test_data[model_features]).todense(),\n", " sensitive_features=test_data[\"age_groups\"].values,\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. Evaluate Results (Given)\n", "(Go to top)" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "tags": [] }, "outputs": [], "source": [ "result_df = pd.DataFrame(columns=[\"ID\", \"credit_risk_pred\"])\n", "result_df[\"ID\"] = test_data[\"ID\"].tolist()\n", "result_df[\"credit_risk_pred\"] = test_predictions\n", "\n", "result_df.to_csv(\"../../data/final_project/project_day3_result.csv\", index=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Final Evaluation on Test Data - Disparate Impact\n", "To evaluate the fairness of the model predictions, we will calculate the disparate impact (DI) metric. For more details about DI you can have a look [here](https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-post-training-bias-metric-di.html)." ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "tags": [] }, "outputs": [], "source": [ "def calculate_di(test_data, pred_df, pred_col=\"credit_risk_pred\"):\n", " \"\"\"\n", " Function to calculate Disparate Impact metric using the results from this notebook.\n", " \"\"\"\n", " try:\n", " # Merge predictions with original test data to model per group\n", " di_df = pred_df.merge(test_data, on=\"ID\")\n", " # Count for group with members less than 25y old\n", " pos_outcomes_less25 = di_df[di_df[\"age_groups\"] == 0][pred_col].value_counts()[\n", " 0\n", " ] # value_counts()[0] takes the count of the '0 credit risk' == 'not credit risk'\n", " total_less25 = len(di_df[di_df[\"age_groups\"] == 0])\n", " # Count for group with members greater equal 25y old\n", " pos_outcomes_geq25 = di_df[di_df[\"age_groups\"] == 1][pred_col].value_counts()[\n", " 0\n", " ] # value_counts()[0] takes the count of the '0 credit risk' == 'not credit risk'\n", " total_geq25 = len(di_df[di_df[\"age_groups\"] == 1])\n", " # Check if correct number of gorups\n", " if total_geq25 == 0:\n", " print(\"There is only one group present in the data.\")\n", " elif total_less25 == 0:\n", " print(\"There is only one group present in the data.\")\n", " else:\n", " disparate_impact = (pos_outcomes_less25 / total_less25) / (\n", " pos_outcomes_geq25 / total_geq25\n", " )\n", " return disparate_impact\n", " except:\n", " print(\"Wrong inputs provided.\")" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "0.6343283582089553" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "calculate_di(test_data, result_df, \"credit_risk_pred\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Final Evaluation on Test Data - Accuracy & F1 Score\n", "In addition to fairness evaluation, we also need to check the general model performance. During the EDA stage we learned that the target distribution is skewed so we will use F1 score in addition to accuracy." ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "0.745" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "accuracy_score(\n", " pd.read_csv(\"../../data/final_project/german_credit_test_labels.csv\")[\n", " \"credit_risk\"\n", " ],\n", " result_df[\"credit_risk_pred\"],\n", ")" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "0.5405405405405405" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "f1_score(\n", " pd.read_csv(\"../../data/final_project/german_credit_test_labels.csv\")[\n", " \"credit_risk\"\n", " ],\n", " result_df[\"credit_risk_pred\"],\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is the end of the notebook." ] } ], "metadata": { "kernelspec": { "display_name": "conda_pytorch_p39", "language": "python", "name": "conda_pytorch_p39" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.15" } }, "nbformat": 4, "nbformat_minor": 4 }