{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "![MLU Logo](../../data/MLU_Logo.png)" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "# Responsible AI - Final Project\n", "\n", "Build a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) that predicts the __credit_risk__ field (whether some is a credit risk or not) of the [German Credit Dataset](https://archive.ics.uci.edu/ml/datasets/South+German+Credit+%28UPDATE%29).\n", "\n", "### Final Project Problem: Loan Approval\n", "\n", "__Problem Definition:__\n", "Given a set of features about an individual (e.g. age, past credit history, immigration status, ...) predict whether a loan is repaid or not (is customer a credit risk). We impose the additional constraint that the model should be fair with respect to different age groups ($\\geq$ 25 yrs and $<$ 25 yrs).\n", "\n", "In the banking industry, there are certain regulations regarding the use of sensitive features (e.g., age, ethnicity, marital status, ...). According to those regulations, it would not be okay if age played a significant role in the model (loans should be approved/denied regardless of an individuals' age).\n", "\n", "For example, certain laws declare it unlawful for creditors to discriminate against any applicant on the basis of age (or other sensitive attributes). For more details, have a look at this paper:\n", "\n", "``` \n", "F. Kamiran and T. Calders, \"Data Preprocessing Techniques for Classification without Discrimination,\" Knowledge and Information Systems, 2012\n", "```\n", "\n", "__Table of contents__\n", "\n", "1. Read the datasets (Given) \n", "2. Data Processing (Implement)\n", " * Exploratory Data Analysis\n", " * Select features to build the model (Suggested)\n", " * Train - Validation - Test Datasets\n", " * Data Processing with Pipeline\n", "3. Train (and Tune) a Classifier on the Training Dataset (Implement)\n", "4. Make Predictions on the Test Dataset (Implement)\n", "5. Evaluate Results (Given)\n", "\n", "\n", "__Datasets and Files__\n", "\n", "\n", "- ```german_credit_training.csv```: Training data with loan applicants features, credit history, dependents, savings, account status, age group (and more). The label is __credit_risk__.\n", "\n", "- ```german_credit_test.csv```: Test data with same features as above apart from label. This will be the data to make predictions for to emulate a production environment.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This notebook assumes an installation of the SageMaker kernel `conda_pytorch_p39`. In addition, libraries from a requirements.txt need to be installed:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip install --no-deps -U -q -r ../../requirements.txt" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Reshaping/basic libraries\n", "import pandas as pd\n", "import numpy as np\n", "\n", "# Plotting libraries\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "\n", "sns.set_style(\"darkgrid\", {\"axes.facecolor\": \".9\"})\n", "\n", "# ML libraries\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.metrics import confusion_matrix, accuracy_score, f1_score\n", "from sklearn.impute import SimpleImputer\n", "from sklearn.preprocessing import OneHotEncoder, MinMaxScaler\n", "from sklearn.pipeline import Pipeline\n", "from sklearn.compose import ColumnTransformer\n", "from sklearn.linear_model import LogisticRegression\n", "\n", "# Operational libraries\n", "import sys\n", "\n", "sys.path.append(\"..\")\n", "\n", "# Jupyter(lab) libraries\n", "import warnings\n", "\n", "warnings.filterwarnings(\"ignore\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Read the datasets (Given)\n", "(Go to top)\n", "\n", "Then, we read the __training__ and __test__ datasets into dataframes, using [Pandas](https://pandas.pydata.org/pandas-docs/stable/user_guide/10min.html). This library allows us to read and manipulate our data." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "training_data = pd.read_csv(\"../../data/final_project/german_credit_training.csv\")\n", "test_data = pd.read_csv(\"../../data/final_project/german_credit_test.csv\")\n", "\n", "print(\"The shape of the training dataset is:\", training_data.shape)\n", "print(\"The shape of the test dataset is:\", test_data.shape)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Data Processing (Implement)\n", "(Go to top) " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.1 Exploratory Data Analysis\n", "(Go to Data Processing)\n", "\n", "We look at number of rows, columns, and some simple statistics of the dataset." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "training_data.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "test_data.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Implement more EDA here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.2 Select features to build the model \n", "(Go to Data Processing)\n", "\n", "For a quick start, we recommend using only a few of the numerical and categorical features. However, feel free to explore other fields. In this case, we do not need to cast our features to numerical/objects. Mindful with some of the feature names - they suggest numerical values but upon inspection it should become clear that they are actually categoricals (e.g. `employed_since_years` has been binned into groups).\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Grab model features/inputs and target/output\n", "categorical_features = [\"job_status\", \"employed_since_years\", \"savings\", \"age_groups\"]\n", "\n", "numerical_features = [\"credit_amount\", \"credit_duration_months\"]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Separate features and the model target." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "model_target = \"credit_risk\"\n", "model_features = categorical_features + numerical_features\n", "\n", "print(\"Model features: \", model_features)\n", "print(\"Model target: \", model_target)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.3 Train - Validation Datasets\n", "(Go to Data Processing)\n", "\n", "We already have training and test datasets, but no validation dataset (which you need to create). Furthermore, the test dataset is missing the labels - the goal of the project is to predict these labels. \n", "\n", "To produce a validation set to evaluate model performance, split the training dataset into train and validation subsets using sklearn's [train_test_split()](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) function. Validation data you get here will be used later in section 3 to tune your classifier." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Implement here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.4 Data processing with Pipeline\n", "(Go to Data Processing)\n", "\n", "Build a [pipeline](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html) to impute the missing values and scale the numerical features, and finally train a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) on the imputed and scaled dataset. \n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Implement here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Train (and Tune) a Classifier (Implement)\n", "(Go to top)\n", "\n", "Train (and tune) the [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) pipeline. For tuning, you can try different imputation strategies, different scaling methods." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Implement here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. Make Predictions on the Test Dataset (Implement)\n", "(Go to top)\n", "\n", "Use the trained classifier to predict the labels on the test set. Below you will find a code snippet that evaluates for DI." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Implement here\n", "\n", "# Get test data to test the classifier\n", "# ! test data should come from german_credit_test.csv !\n", "# ...\n", "\n", "# Use the trained model to make predictions on the test dataset\n", "# test_predictions = ..." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. Evaluate Results (Given)\n", "(Go to top)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "result_df = pd.DataFrame(columns=[\"ID\", \"credit_risk_pred\"])\n", "result_df[\"ID\"] = test_data[\"ID\"].tolist()\n", "result_df[\"credit_risk_pred\"] = test_predictions\n", "\n", "result_df.to_csv(\"../../data/final_project/project_day1_result.csv\", index=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Final Evaluation on Test Data - Disparate Impact\n", "To evaluate the fairness of the model predictions, we will calculate the disparate impact (DI) metric. For more details about DI you can have a look [here](https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-post-training-bias-metric-di.html)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def calculate_di(test_data, pred_df, pred_col=\"credit_risk_pred\"):\n", " \"\"\"\n", " Function to calculate Disparate Impact metric using the results from this notebook.\n", " \"\"\"\n", " try:\n", " # Merge predictions with original test data to model per group\n", " di_df = pred_df.merge(test_data, on=\"ID\")\n", " # Count for group with members less than 25y old\n", " pos_outcomes_less25 = di_df[di_df[\"age_groups\"] == 0][pred_col].value_counts()[\n", " 0\n", " ] # value_counts()[0] takes the count of the '0 credit risk' == 'not credit risk'\n", " total_less25 = len(di_df[di_df[\"age_groups\"] == 0])\n", " # Count for group with members greater equal 25y old\n", " pos_outcomes_geq25 = di_df[di_df[\"age_groups\"] == 1][pred_col].value_counts()[\n", " 0\n", " ] # value_counts()[0] takes the count of the '0 credit risk' == 'not credit risk'\n", " total_geq25 = len(di_df[di_df[\"age_groups\"] == 1])\n", " # Check if correct number of gorups\n", " if total_geq25 == 0:\n", " print(\"There is only one group present in the data.\")\n", " elif total_less25 == 0:\n", " print(\"There is only one group present in the data.\")\n", " else:\n", " disparate_impact = (pos_outcomes_less25 / total_less25) / (\n", " pos_outcomes_geq25 / total_geq25\n", " )\n", " return disparate_impact\n", " except:\n", " print(\"Wrong inputs provided.\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "calculate_di(test_data, result_df, \"credit_risk_pred\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Final Evaluation on Test Data - Accuracy & F1 Score\n", "In addition to fairness evaluation, we also need to check the general model performance. During the EDA stage we learned that the target distribution is skewed so we will use F1 score in addition to accuracy." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "accuracy_score(\n", " pd.read_csv(\"../../data/final_project/german_credit_test_labels.csv\")[\n", " \"credit_risk\"\n", " ],\n", " result_df[\"credit_risk_pred\"],\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "f1_score(\n", " pd.read_csv(\"../../data/final_project/german_credit_test_labels.csv\")[\n", " \"credit_risk\"\n", " ],\n", " result_df[\"credit_risk_pred\"],\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is the end of the notebook." ] } ], "metadata": { "kernelspec": { "display_name": "conda_pytorch_p39", "language": "python", "name": "conda_pytorch_p39" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.15" } }, "nbformat": 4, "nbformat_minor": 4 }