{ "cells": [ { "cell_type": "markdown", "id": "083ded7d", "metadata": { "papermill": { "duration": 0.020102, "end_time": "2022-04-18T15:42:07.746324", "exception": false, "start_time": "2022-04-18T15:42:07.726222", "status": "completed" }, "tags": [] }, "source": [ "# Fairness and Explainability with SageMaker Clarify" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n", "\n", "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-west-2/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_outputs.ipynb)\n", "\n", "---" ] }, { "cell_type": "markdown", "id": "869afe6f", "metadata": { "papermill": { "duration": 0.0202, "end_time": "2022-04-18T15:42:07.786675", "exception": false, "start_time": "2022-04-18T15:42:07.766475", "status": "completed" }, "tags": [] }, "source": [ "## Runtime\n", "\n", "This notebook takes approximately 30 minutes to run.\n", "\n", "## Contents\n", "\n", "1. [Overview](#Overview)\n", "1. [Prerequisites and Data](#Prerequisites-and-Data)\n", " 1. [Initialize SageMaker](#Initialize-SageMaker)\n", " 1. [Download data](#Download-data)\n", " 1. [Loading the data: Adult Dataset](#Loading-the-data:-Adult-Dataset) \n", " 1. [Data inspection](#Data-inspection) \n", " 1. [Data encoding and upload to S3](#Encode-and-Upload-the-Data) \n", "1. [Train and Deploy XGBoost Model](#Train-XGBoost-Model)\n", " 1. [Train Model](#Train-Model)\n", " 1. [Create Model](#Create-Model)\n", "1. [Amazon SageMaker Clarify](#Amazon-SageMaker-Clarify)\n", " 1. [Detecting Bias](#Detecting-Bias)\n", " 1. [Writing BiasConfig](#Writing-BiasConfig)\n", " 1. [Pre-training Bias](#Pre-training-Bias)\n", " 1. [Post-training Bias](#Post-training-Bias)\n", " 1. [Viewing the Bias Report](#Viewing-the-Bias-Report)\n", " 1. [Explaining Predictions](#Explaining-Predictions)\n", " 1. [Viewing the Explainability Report](#Viewing-the-Explainability-Report)\n", "1. [Clean Up](#Clean-Up)\n", "\n", "## Overview\n", "Amazon SageMaker Clarify helps improve your machine learning models by detecting potential bias and helping explain how these models make predictions. The fairness and explainability functionality provided by SageMaker Clarify takes a step towards enabling AWS customers to build trustworthy and understandable machine learning models. The product comes with the tools to help you with the following tasks.\n", "\n", "* Measure biases that can occur during each stage of the ML lifecycle (data collection, model training and tuning, and monitoring of ML models deployed for inference).\n", "* Generate model governance reports targeting risk and compliance teams and external regulators.\n", "* Provide explanations of the data, models, and monitoring used to assess predictions.\n", "\n", "This sample notebook walks you through: \n", "1. Key terms and concepts needed to understand SageMaker Clarify\n", "1. Measuring the pre-training bias of a dataset and post-training bias of a model\n", "1. Explaining the importance of the various input features on the model's decision\n", "1. Accessing the reports through SageMaker Studio if you have an instance set up.\n", "\n", "In doing so, the notebook first trains a [SageMaker XGBoost](https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost.html) model using training dataset, then use SageMaker Clarify to analyze a testing dataset in CSV format. SageMaker Clarify also supports analyzing dataset in [SageMaker JSON Lines dense format](https://docs.aws.amazon.com/sagemaker/latest/dg/cdf-inference.html#common-in-formats), which is illustrated in [another notebook](https://github.com/aws/amazon-sagemaker-examples/blob/master/sagemaker-clarify/fairness_and_explainability/fairness_and_explainability_jsonlines_format.ipynb)." ] }, { "cell_type": "markdown", "id": "d009cb6d", "metadata": { "papermill": { "duration": 0.020058, "end_time": "2022-04-18T15:42:07.826883", "exception": false, "start_time": "2022-04-18T15:42:07.806825", "status": "completed" }, "tags": [] }, "source": [ "## Prerequisites and Data\n", "### Initialize SageMaker" ] }, { "cell_type": "code", "execution_count": 2, "id": "d05d4578", "metadata": { "execution": { "iopub.execute_input": "2022-04-18T15:42:07.873260Z", "iopub.status.busy": "2022-04-18T15:42:07.872366Z", "iopub.status.idle": "2022-04-18T15:42:09.367100Z", "shell.execute_reply": "2022-04-18T15:42:09.366670Z" }, "papermill": { "duration": 1.520269, "end_time": "2022-04-18T15:42:09.367251", "exception": false, "start_time": "2022-04-18T15:42:07.846982", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "from sagemaker import Session\n", "\n", "session = Session()\n", "bucket = session.default_bucket()\n", "prefix = \"sagemaker/DEMO-sagemaker-clarify\"\n", "region = session.boto_region_name\n", "# Define IAM role\n", "from sagemaker import get_execution_role\n", "import pandas as pd\n", "import numpy as np\n", "import os\n", "import boto3\n", "from datetime import datetime\n", "\n", "role = get_execution_role()\n", "s3_client = boto3.client(\"s3\")" ] }, { "cell_type": "markdown", "id": "0372a3cf", "metadata": { "papermill": { "duration": 0.021199, "end_time": "2022-04-18T15:42:09.408780", "exception": false, "start_time": "2022-04-18T15:42:09.387581", "status": "completed" }, "tags": [] }, "source": [ "### Download data\n", "Data Source: [https://archive.ics.uci.edu/ml/machine-learning-databases/adult/](https://archive.ics.uci.edu/ml/machine-learning-databases/adult/)\n", "\n", "Let's __download__ the data and save it in the local folder with the name adult.data and adult.test from UCI repository$^{[2]}$.\n", "\n", "$^{[2]}$Dua Dheeru, and Efi Karra Taniskidou. \"[UCI Machine Learning Repository](http://archive.ics.uci.edu/ml)\". Irvine, CA: University of California, School of Information and Computer Science (2017)." ] }, { "cell_type": "code", "execution_count": 3, "id": "fc24db78", "metadata": { "execution": { "iopub.execute_input": "2022-04-18T15:42:09.459240Z", "iopub.status.busy": "2022-04-18T15:42:09.458522Z", "iopub.status.idle": "2022-04-18T15:42:10.838357Z", "shell.execute_reply": "2022-04-18T15:42:10.837796Z" }, "papermill": { "duration": 1.409563, "end_time": "2022-04-18T15:42:10.838505", "exception": false, "start_time": "2022-04-18T15:42:09.428942", "status": "completed" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "adult.data saved!\n", "adult.test saved!\n" ] } ], "source": [ "adult_columns = [\n", " \"Age\",\n", " \"Workclass\",\n", " \"fnlwgt\",\n", " \"Education\",\n", " \"Education-Num\",\n", " \"Marital Status\",\n", " \"Occupation\",\n", " \"Relationship\",\n", " \"Ethnic group\",\n", " \"Sex\",\n", " \"Capital Gain\",\n", " \"Capital Loss\",\n", " \"Hours per week\",\n", " \"Country\",\n", " \"Target\",\n", "]\n", "if not os.path.isfile(\"adult.data\"):\n", " s3_client.download_file(\n", " \"sagemaker-sample-files\", \"datasets/tabular/uci_adult/adult.data\", \"adult.data\"\n", " )\n", " print(\"adult.data saved!\")\n", "else:\n", " print(\"adult.data already on disk.\")\n", "\n", "if not os.path.isfile(\"adult.test\"):\n", " s3_client.download_file(\n", " \"sagemaker-sample-files\", \"datasets/tabular/uci_adult/adult.test\", \"adult.test\"\n", " )\n", " print(\"adult.test saved!\")\n", "else:\n", " print(\"adult.test already on disk.\")" ] }, { "cell_type": "markdown", "id": "b00899c5", "metadata": { "papermill": { "duration": 0.02317, "end_time": "2022-04-18T15:42:10.885485", "exception": false, "start_time": "2022-04-18T15:42:10.862315", "status": "completed" }, "tags": [] }, "source": [ "### Loading the data: Adult Dataset\n", "From the UCI repository of machine learning datasets, this database contains 14 features concerning demographic characteristics of 45,222 rows (32,561 for training and 12,661 for testing). The task is to predict whether a person has a yearly income that is more or less than $50,000.\n", "\n", "Here are the features and their possible values:\n", "1. **Age**: continuous.\n", "1. **Workclass**: Private, Self-emp-not-inc, Self-emp-inc, Federal-gov, Local-gov, State-gov, Without-pay, Never-worked.\n", "1. **Fnlwgt**: continuous (the number of people the census takers believe that observation represents).\n", "1. **Education**: Bachelors, Some-college, 11th, HS-grad, Prof-school, Assoc-acdm, Assoc-voc, 9th, 7th-8th, 12th, Masters, 1st-4th, 10th, Doctorate, 5th-6th, Preschool.\n", "1. **Education-num**: continuous.\n", "1. **Marital-status**: Married-civ-spouse, Divorced, Never-married, Separated, Widowed, Married-spouse-absent, Married-AF-spouse.\n", "1. **Occupation**: Tech-support, Craft-repair, Other-service, Sales, Exec-managerial, Prof-specialty, Handlers-cleaners, Machine-op-inspct, Adm-clerical, Farming-fishing, Transport-moving, Priv-house-serv, Protective-serv, Armed-Forces.\n", "1. **Relationship**: Wife, Own-child, Husband, Not-in-family, Other-relative, Unmarried.\n", "1. **Ethnic group**: White, Asian-Pac-Islander, Amer-Indian-Eskimo, Other, Black.\n", "1. **Sex**: Female, Male.\n", " * **Note**: this data is extracted from the 1994 Census and enforces a binary option on Sex\n", "1. **Capital-gain**: continuous.\n", "1. **Capital-loss**: continuous.\n", "1. **Hours-per-week**: continuous.\n", "1. **Native-country**: United-States, Cambodia, England, Puerto-Rico, Canada, Germany, Outlying-US(Guam-USVI-etc), India, Japan, Greece, South, China, Cuba, Iran, Honduras, Philippines, Italy, Poland, Jamaica, Vietnam, Mexico, Portugal, Ireland, France, Dominican-Republic, Laos, Ecuador, Taiwan, Haiti, Columbia, Hungary, Guatemala, Nicaragua, Scotland, Thailand, Yugoslavia, El-Salvador, Trinadad&Tobago, Peru, Hong, Holand-Netherlands.\n", "\n", "Next, we specify our binary prediction task: \n", "15. **Target**: <=50,000, >$50,000." ] }, { "cell_type": "code", "execution_count": 4, "id": "be1e2a1f", "metadata": { "execution": { "iopub.execute_input": "2022-04-18T15:42:10.934980Z", "iopub.status.busy": "2022-04-18T15:42:10.934029Z", "iopub.status.idle": "2022-04-18T15:42:11.981113Z", "shell.execute_reply": "2022-04-18T15:42:11.982318Z" }, "papermill": { "duration": 1.076158, "end_time": "2022-04-18T15:42:11.982501", "exception": false, "start_time": "2022-04-18T15:42:10.906343", "status": "completed" }, "scrolled": true, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
AgeWorkclassfnlwgtEducationEducation-NumMarital StatusOccupationRelationshipEthnic groupSexCapital GainCapital LossHours per weekCountryTarget
039State-gov77516Bachelors13Never-marriedAdm-clericalNot-in-familyWhiteMale2174040United-States<=50K
150Self-emp-not-inc83311Bachelors13Married-civ-spouseExec-managerialHusbandWhiteMale0013United-States<=50K
238Private215646HS-grad9DivorcedHandlers-cleanersNot-in-familyWhiteMale0040United-States<=50K
353Private23472111th7Married-civ-spouseHandlers-cleanersHusbandBlackMale0040United-States<=50K
428Private338409Bachelors13Married-civ-spouseProf-specialtyWifeBlackFemale0040Cuba<=50K
\n", "
" ], "text/plain": [ " Age Workclass fnlwgt Education Education-Num \\\n", "0 39 State-gov 77516 Bachelors 13 \n", "1 50 Self-emp-not-inc 83311 Bachelors 13 \n", "2 38 Private 215646 HS-grad 9 \n", "3 53 Private 234721 11th 7 \n", "4 28 Private 338409 Bachelors 13 \n", "\n", " Marital Status Occupation Relationship Ethnic group Sex \\\n", "0 Never-married Adm-clerical Not-in-family White Male \n", "1 Married-civ-spouse Exec-managerial Husband White Male \n", "2 Divorced Handlers-cleaners Not-in-family White Male \n", "3 Married-civ-spouse Handlers-cleaners Husband Black Male \n", "4 Married-civ-spouse Prof-specialty Wife Black Female \n", "\n", " Capital Gain Capital Loss Hours per week Country Target \n", "0 2174 0 40 United-States <=50K \n", "1 0 0 13 United-States <=50K \n", "2 0 0 40 United-States <=50K \n", "3 0 0 40 United-States <=50K \n", "4 0 0 40 Cuba <=50K " ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "training_data = pd.read_csv(\n", " \"adult.data\", names=adult_columns, sep=r\"\\s*,\\s*\", engine=\"python\", na_values=\"?\"\n", ").dropna()\n", "\n", "testing_data = pd.read_csv(\n", " \"adult.test\", names=adult_columns, sep=r\"\\s*,\\s*\", engine=\"python\", na_values=\"?\", skiprows=1\n", ").dropna()\n", "\n", "training_data.head()" ] }, { "cell_type": "markdown", "id": "3b11ce50", "metadata": { "papermill": { "duration": 0.029795, "end_time": "2022-04-18T15:42:12.041291", "exception": false, "start_time": "2022-04-18T15:42:12.011496", "status": "completed" }, "tags": [] }, "source": [ "### Data inspection\n", "Plotting histograms for the distribution of the different features is a good way to visualize the data. Let's plot a few of the features that can be considered _sensitive_. \n", "Let's take a look specifically at the Sex feature of a census respondent. In the first plot we see that there are fewer Female respondents as a whole but especially in the positive outcomes, where they form ~$\\frac{1}{7}$th of respondents." ] }, { "cell_type": "code", "execution_count": 5, "id": "9050f337", "metadata": { "execution": { "iopub.execute_input": "2022-04-18T15:42:12.107265Z", "iopub.status.busy": "2022-04-18T15:42:12.106139Z", "iopub.status.idle": "2022-04-18T15:42:14.162289Z", "shell.execute_reply": "2022-04-18T15:42:14.162672Z" }, "papermill": { "duration": 2.09361, "end_time": "2022-04-18T15:42:14.162800", "exception": false, "start_time": "2022-04-18T15:42:12.069190", "status": "completed" }, "scrolled": true, "tags": [] }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAEICAYAAAC9E5gJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAYBUlEQVR4nO3df5RfdX3n8eerBBG1SIAxGxM0qNEusDWUFPFXD10UAlrBXcWwrQTLMbrCth7tduOPFguyiz/Qlj0UG9eswR8gu4jkaGyMWZV6FMkgkd+UgHBIGshAEEQ8KPjeP76fqZc4M5nMTGby4/k455653/fnc+/9fGEyr7mfe79zU1VIkvZsvzXVA5AkTT3DQJJkGEiSDANJEoaBJAnDQJKEYSDtFJLMSHJ1kp8muWCqx6M9j2GgXVKS/5SkP8mjSTYl+XqSV03CcSvJi3bArhcDDwD7VdV7hzju7CRXJHkgycNJbkpy+g4Yh/ZQ06Z6ANL2SvIeYAnwTmAV8AtgAXAS8N0pHNp4PB+4pYb/FOjngB+1fo8D/w74N5M0Nu0JqsrFZZdZgGcDjwJvHqHPPsDfAv/Slr8F9mltpwPf3ap/AS9q658FLgK+BvwU+AHwwtZ2dev7szaGtwAHAV8FfgJsAf4J+K1hxvUKYC3wcPv6is4xf0kv1B4FXjPEto8C80Z4z0cD32vj+BFwTOeYDwAHt9cvBR4Cfmeq/1+67FyL00Ta1bwceDpw5Qh9PkDvh+M8ej/8jgI+uB3HWAj8DTAdWA+cB1BVf9DaX1pVz6qqLwHvBTYAfcAM4P30AuMpkhxAL2AuBA4EPgF8LcmBVXU68AXgo22/3xxiTNcAFyVZmOR5W+17Vtv3h4EDgL8ArkjSV1XfA/4BWJ5kX+DzwF9V1W3b8d9DewDDQLuaA4EHquqJEfr8MXBOVW2uqgF6P9jfuh3HuLKqrm3H+AK9UBnOL4GZwPOr6pdV9U9VNdRUz+uAO6rqc1X1RFVdCtwG/NEox/RmemcdfwX8OMm6JL/f2v4EWFlVK6vqV1W1GugHTmztH6J3RnUtsJHemY/0FIaBdjUPAgclGel613OBezqv72m10bqvs/4Y8KwR+n6M3tnDN5LclWTJKMc0OK5ZoxlQVT1UVUuq6jB6ZyDrgK8kCb3rCG9O8pPBBXgVvZCiqn5JbyrqcOCCYcJKezjDQLua79O7gHryCH3+hd4PyEHPazXozfc/Y7AhybguwlbVT6vqvVX1AuANwHuSHDuKMQ2Oa+MYjvkA8HF6AXMAcC/wuarav7M8s6rOh3+dRjob+N/ABUn22d5javdnGGiXUlUPA39Nb/785CTPSLJ3khOSfLR1uxT4YJK+JAe1/p9vbT8CDksyL8nT6U2hbI/7gRcMvkjy+iQvar+hPww8CfxqiO1WAi9ut8ROS/IW4FB6F5+3KclHkhzetv1t4D8D66vqwfbe/ijJ8Un2SvL0JMe021FD76zgM8AZwCbg3O18z9oDGAba5VTVBcB76F0UHqD3m/FZwFdalw/TmzO/AbgR+GGrUVX/DJwDfBO4g+2/FfVD9C7G/iTJKcDctq9H6Z21/H1VfWuIMT8IvJ7eBecHgb8EXt9+yx+NZ9C7aP4T4C56ZxlvaPu+l95tte/n1/89/iu9f99/BjyH3kXjAt4GvC3Jq7fzfWs3F6cPJUmeGUiSDANJkmEgScIwkCSxC/+huoMOOqjmzJkz1cOQpF3Kdddd90BV9W1d32XDYM6cOfT390/1MCRpl5Jk60/CA04TSZIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSGMUnkJMcDFxC77mrBSytqr9LcgDwJWAOcDdwSlU91J6s9Hf0Hsb9GHB6Vf2w7WsRvQeSAHy4qpa3+pH0nsa0L70nQv25z2mVdqw5S7421UPYrdx9/uumegjjMpozgyeA91bVocDRwJlJDgWWAGuqai6wpr0GOIHe05/mAouBiwFaeJwNvAw4Cjg7yfS2zcXA2zvbLRj/W5MkjdY2w6CqNg3+Zl9VPwVuBWbRe8ze8tZtOb9+QPlJwCXVcw2wf5KZwPHA6qraUlUPAauBBa1tv6q6pp0NXMLIDzuXJE2w7bpmkGQOcATwA2BGVW1qTffRm0aCXlDc29lsQ6uNVN8wRF2SNElGHQZJngVcAby7qh7ptrXf6Hf4HH+SxUn6k/QPDAzs6MNJ0h5jVGGQZG96QfCFqvpyK9/fpnhoXze3+kbg4M7ms1ttpPrsIeq/oaqWVtX8qprf1/cbf45bkjRG2wyDdnfQZ4Bbq+oTnaYVwKK2vgi4qlM/LT1HAw+36aRVwHFJprcLx8cBq1rbI0mObsc6rbMvSdIkGM3DbV4JvBW4Mcm6Vns/cD5weZIzgHuAU1rbSnq3la6nd2vp2wCqakuSc4G1rd85VbWlrb+LX99a+vW2SJImyTbDoKq+C2SY5mOH6F/AmcPsaxmwbIh6P3D4tsYiSdox/ASyJMkwkCQZBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSGN0zkJcl2Zzkpk7tS0nWteXuwcdhJpmT5Oedtk91tjkyyY1J1ie5sD3vmCQHJFmd5I72dfoOeJ+SpBGM5szgs8CCbqGq3lJV86pqHnAF8OVO852DbVX1zk79YuDtwNy2DO5zCbCmquYCa9prSdIk2mYYVNXVwJah2tpv96cAl460jyQzgf2q6pr2jORLgJNb80nA8ra+vFOXJE2S8V4zeDVwf1Xd0akdkuT6JN9J8upWmwVs6PTZ0GoAM6pqU1u/D5gx3MGSLE7Sn6R/YGBgnEOXJA0abxicylPPCjYBz6uqI4D3AF9Mst9od9bOGmqE9qVVNb+q5vf19Y11zJKkrUwb64ZJpgH/AThysFZVjwOPt/XrktwJvBjYCMzubD671QDuTzKzqja16aTNYx2TJGlsxnNm8Brgtqr61+mfJH1J9mrrL6B3ofiuNg30SJKj23WG04Cr2mYrgEVtfVGnLkmaJKO5tfRS4PvAS5JsSHJGa1rIb144/gPghnar6f8F3llVgxef3wX8L2A9cCfw9VY/H3htkjvoBcz5Y387kqSx2OY0UVWdOkz99CFqV9C71XSo/v3A4UPUHwSO3dY4JEk7jp9AliQZBpIkw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkRveks2VJNie5qVP7UJKNSda15cRO2/uSrE9ye5LjO/UFrbY+yZJO/ZAkP2j1LyV52kS+QUnSto3mzOCzwIIh6p+sqnltWQmQ5FB6j8M8rG3z90n2as9Fvgg4ATgUOLX1BfhI29eLgIeAM7Y+kCRpx9pmGFTV1cCWbfVrTgIuq6rHq+rH9J53fFRb1lfVXVX1C+Ay4KQkAf49veclAywHTt6+tyBJGq/xXDM4K8kNbRppeqvNAu7t9NnQasPVDwR+UlVPbFUfUpLFSfqT9A8MDIxj6JKkrrGGwcXAC4F5wCbggoka0EiqamlVza+q+X19fZNxSEnaI0wby0ZVdf/gepJPA19tLzcCB3e6zm41hqk/COyfZFo7O+j2lyRNkjGdGSSZ2Xn5RmDwTqMVwMIk+yQ5BJgLXAusBea2O4eeRu8i84qqKuBbwJva9ouAq8YyJknS2G3zzCDJpcAxwEFJNgBnA8ckmQcUcDfwDoCqujnJ5cAtwBPAmVX1ZNvPWcAqYC9gWVXd3A7x34DLknwYuB74zES9OUnS6GwzDKrq1CHKw/7ArqrzgPOGqK8EVg5Rv4ve3UaSpCniJ5AlSYaBJMkwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJLEKMIgybIkm5Pc1Kl9LMltSW5IcmWS/Vt9TpKfJ1nXlk91tjkyyY1J1ie5MEla/YAkq5Pc0b5O3wHvU5I0gtGcGXwWWLBVbTVweFX9LvDPwPs6bXdW1by2vLNTvxh4OzC3LYP7XAKsqaq5wJr2WpI0ibYZBlV1NbBlq9o3quqJ9vIaYPZI+0gyE9ivqq6pqgIuAU5uzScBy9v68k5dkjRJJuKawZ8CX++8PiTJ9Um+k+TVrTYL2NDps6HVAGZU1aa2fh8wY7gDJVmcpD9J/8DAwAQMXZIE4wyDJB8AngC+0EqbgOdV1RHAe4AvJtlvtPtrZw01QvvSqppfVfP7+vrGMXJJUte0sW6Y5HTg9cCx7Yc4VfU48Hhbvy7JncCLgY08dSppdqsB3J9kZlVtatNJm8c6JknS2IzpzCDJAuAvgTdU1WOdel+Svdr6C+hdKL6rTQM9kuTodhfRacBVbbMVwKK2vqhTlyRNkm2eGSS5FDgGOCjJBuBsencP7QOsbneIXtPuHPoD4JwkvwR+BbyzqgYvPr+L3p1J+9K7xjB4neF84PIkZwD3AKdMyDuTJI3aNsOgqk4dovyZYfpeAVwxTFs/cPgQ9QeBY7c1DknSjuMnkCVJhoEkyTCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkiVGGQZJlSTYnualTOyDJ6iR3tK/TWz1JLkyyPskNSX6vs82i1v+OJIs69SOT3Ni2ubA9GlOSNElGe2bwWWDBVrUlwJqqmgusaa8BTqD37OO5wGLgYuiFB71HZr4MOAo4ezBAWp+3d7bb+liSpB1oVGFQVVcDW7YqnwQsb+vLgZM79Uuq5xpg/yQzgeOB1VW1paoeAlYDC1rbflV1TVUVcElnX5KkSTCeawYzqmpTW78PmNHWZwH3dvptaLWR6huGqP+GJIuT9CfpHxgYGMfQJUldE3IBuf1GXxOxr20cZ2lVza+q+X19fTv6cJK0x5g2jm3vTzKzqja1qZ7Nrb4ROLjTb3arbQSO2ar+7VafPUT/3cKcJV+b6iHsNu4+/3VTPQRptzWeM4MVwOAdQYuAqzr109pdRUcDD7fppFXAcUmmtwvHxwGrWtsjSY5udxGd1tmXJGkSjOrMIMml9H6rPyjJBnp3BZ0PXJ7kDOAe4JTWfSVwIrAeeAx4G0BVbUlyLrC29TunqgYvSr+L3h1L+wJfb4skaZKMKgyq6tRhmo4dom8BZw6zn2XAsiHq/cDhoxmLJGni+QlkSZJhIEkyDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEliHGGQ5CVJ1nWWR5K8O8mHkmzs1E/sbPO+JOuT3J7k+E59QautT7JkvG9KkrR9RvWks6FU1e3APIAke9F7iP2V9B5z+cmq+ni3f5JDgYXAYcBzgW8meXFrvgh4LbABWJtkRVXdMtaxSZK2z5jDYCvHAndW1T29Z9oP6STgsqp6HPhxkvXAUa1tfVXdBZDkstbXMJCkSTJR1wwWApd2Xp+V5IYky5JMb7VZwL2dPhtabbi6JGmSjDsMkjwNeAPwf1rpYuCF9KaQNgEXjPcYnWMtTtKfpH9gYGCiditJe7yJODM4AfhhVd0PUFX3V9WTVfUr4NP8eipoI3BwZ7vZrTZc/TdU1dKqml9V8/v6+iZg6JIkmJgwOJXOFFGSmZ22NwI3tfUVwMIk+yQ5BJgLXAusBeYmOaSdZSxsfSVJk2RcF5CTPJPeXUDv6JQ/mmQeUMDdg21VdXOSy+ldGH4COLOqnmz7OQtYBewFLKuqm8czLknS9hlXGFTVz4ADt6q9dYT+5wHnDVFfCawcz1gkSWPnJ5AlSYaBJMkwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJLEBIRBkruT3JhkXZL+Vjsgyeokd7Sv01s9SS5Msj7JDUl+r7OfRa3/HUkWjXdckqTRm6gzgz+sqnlVNb+9XgKsqaq5wJr2GuAEYG5bFgMXQy88gLOBlwFHAWcPBogkacfbUdNEJwHL2/py4ORO/ZLquQbYP8lM4HhgdVVtqaqHgNXAgh00NknSViYiDAr4RpLrkixutRlVtamt3wfMaOuzgHs7225oteHqT5FkcZL+JP0DAwMTMHRJEsC0CdjHq6pqY5LnAKuT3NZtrKpKUhNwHKpqKbAUYP78+ROyT0nSBJwZVNXG9nUzcCW9Of/72/QP7evm1n0jcHBn89mtNlxdkjQJxhUGSZ6Z5LcH14HjgJuAFcDgHUGLgKva+grgtHZX0dHAw206aRVwXJLp7cLxca0mSZoE450mmgFcmWRwX1+sqn9Msha4PMkZwD3AKa3/SuBEYD3wGPA2gKrakuRcYG3rd05VbRnn2CRJozSuMKiqu4CXDlF/EDh2iHoBZw6zr2XAsvGMR5I0Nn4CWZJkGEiSDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSGEcYJDk4ybeS3JLk5iR/3uofSrIxybq2nNjZ5n1J1ie5PcnxnfqCVlufZMn43pIkaXuN50lnTwDvraoftucgX5dkdWv7ZFV9vNs5yaHAQuAw4LnAN5O8uDVfBLwW2ACsTbKiqm4Zx9gkSdthzGHQHmS/qa3/NMmtwKwRNjkJuKyqHgd+nGQ9cFRrW98eoUmSy1pfw0CSJsmEXDNIMgc4AvhBK52V5IYky5JMb7VZwL2dzTa02nD1oY6zOEl/kv6BgYGJGLokiQkIgyTPAq4A3l1VjwAXAy8E5tE7c7hgvMcYVFVLq2p+Vc3v6+ubqN1K0h5vPNcMSLI3vSD4QlV9GaCq7u+0fxr4anu5ETi4s/nsVmOEuiRpEoznbqIAnwFurapPdOozO93eCNzU1lcAC5Psk+QQYC5wLbAWmJvkkCRPo3eRecVYxyVJ2n7jOTN4JfBW4MYk61rt/cCpSeYBBdwNvAOgqm5Ocjm9C8NPAGdW1ZMASc4CVgF7Acuq6uZxjEuStJ3GczfRd4EM0bRyhG3OA84bor5ypO0kSTuWn0CWJBkGkiTDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kSO1EYJFmQ5PYk65MsmerxSNKeZKcIgyR7ARcBJwCH0nuO8qFTOypJ2nPsFGEAHAWsr6q7quoXwGXASVM8JknaY0yb6gE0s4B7O683AC/bulOSxcDi9vLRJLdPwtj2FAcBD0z1IEaSj0z1CDRFdvrvTdilvj+fP1RxZwmDUamqpcDSqR7H7ihJf1XNn+pxSFvze3Ny7CzTRBuBgzuvZ7eaJGkS7CxhsBaYm+SQJE8DFgIrpnhMkrTH2CmmiarqiSRnAauAvYBlVXXzFA9rT+P0m3ZWfm9OglTVVI9BkjTFdpZpIknSFDIMJEmGwa4uyZNJ1nWWOTvwWHcnOWhH7V97jiSV5POd19OSDCT56ja2O2ZbfTQ2O8UFZI3Lz6tq3lQPQtpOPwMOT7JvVf0ceC3eTj6lPDPYDSU5Msl3klyXZFWSma3+7SSfTNKf5NYkv5/ky0nuSPLhzvZfadve3D71PdQx/iTJte1s5B/a35eStsdK4HVt/VTg0sGGJEcl+X6S65N8L8lLtt44yTOTLGvfh9cn8U/YjINhsOvbtzNFdGWSvYH/Cbypqo4ElgHndfr/on2a81PAVcCZwOHA6UkObH3+tG07H/izTh2AJP8WeAvwynZW8iTwxzvuLWo3dRmwMMnTgd8FftBpuw14dVUdAfw18N+H2P4DwP+rqqOAPwQ+luSZO3jMuy2niXZ9T5kmSnI4vR/uq5NA73Mbmzr9Bz/MdyNwc1VtatvdRe9T4A/SC4A3tn4HA3NbfdCxwJHA2naMfYHNE/qutNurqhvaNa5T6Z0ldD0bWJ5kLlDA3kPs4jjgDUn+or1+OvA84NYdM+Ldm2Gw+wm9H/IvH6b98fb1V531wdfTkhwDvAZ4eVU9luTb9P6RbX2M5VX1vokatPZYK4CPA8cA3TPQc4FvVdUbW2B8e4htA/zHqvIPVk4Ap4l2P7cDfUleDpBk7ySHbcf2zwYeakHwO8DRQ/RZA7wpyXPaMQ5IMuRfQpS2YRnwN1V141b1Z/PrC8qnD7PtKuC/pJ2eJjlih4xwD2EY7Gba8yDeBHwkyY+AdcArtmMX/0jvDOFW4HzgmiGOcQvwQeAbSW4AVgMzxzl07YGqakNVXThE00eB/5HkeoafwTiX3vTRDUlubq81Rv45CkmSZwaSJMNAkoRhIEnCMJAkYRhIkjAMJEkYBpIk4P8DMTgzDymu4/sAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "%matplotlib inline\n", "training_data[\"Sex\"].value_counts().sort_values().plot(kind=\"bar\", title=\"Counts of Sex\", rot=0)" ] }, { "cell_type": "code", "execution_count": 6, "id": "e7151483", "metadata": { "execution": { "iopub.execute_input": "2022-04-18T15:42:14.279169Z", "iopub.status.busy": "2022-04-18T15:42:14.277102Z", "iopub.status.idle": "2022-04-18T15:42:14.584591Z", "shell.execute_reply": "2022-04-18T15:42:14.584989Z" }, "papermill": { "duration": 0.399927, "end_time": "2022-04-18T15:42:14.585118", "exception": false, "start_time": "2022-04-18T15:42:14.185191", "status": "completed" }, "scrolled": true, "tags": [] }, "outputs": [ { "data": { "text/plain": [ "$50K'}>" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEICAYAAACzliQjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAYoklEQVR4nO3df5xddX3n8dfbhF+iJoGMMSbBoGSLQCXoGII/tlQ0hB811AUEKQSaNo91sXYrbjd0rUF+tNDqIu4D6bIlEtDy46FiUkRxBKJllR+DQWgIlFkkTWIgQxJSIAokfvaP8xm8Tu+duZPcuUPm+34+Hvcx53y/3/M933Nn7vuc+73nJooIzMysDK8Z6QGYmVn7OPTNzAri0DczK4hD38ysIA59M7OCOPTNzAri0LfdiqRJkn4o6TlJXxjp8bSDpO9Imj/S4wCQdO1Ij8F2jUN/lJP0MUndkp6XtCED5H1t2G9IOmgYul4IPAO8ISLOq7PfqZK+IekZSVsl/bOks4dhHG0TEcdFxNKRHkcjki6Q9HL+jfU93lpTP1PSA5K25c+ZNXXXSrq4Zv3Q/Dv9dJsPoxgO/VFM0qeALwJ/BUwCDgC+DMwbwWHtqrcAj0TjbxVeD6zNdvsDZwJPt2lsQyZp7Ajuey9J45pse76kNcDHJK2TtLhfk5si4nU1jydyuz2BZcBXgQnAUmBZlvffxxHAXcDFEfH5XTg0G0hE+DEKH8A44HnglAHa7EV1Uvh5Pr4I7JV1ZwN392sfwEG5fC1wJfBt4DngXuBtWffDbPtCjuGjwETgVuBZYDPwT8BrGozrPcD9wNb8+Z6afb4MvJT9frDOts8DMwc45tnAj3IcPwWOrtnnM8C0XD8c2AIc3KCfg4GuPJbHgFNr6k4AVgL/RnUCuqCmbno+NwuAf83n6mzgbuDzuc+fAcfVbLMC+KPa38sAbQ/MPp8Dvp+/o682OIYpOcavAR8c4PdxVB7nwfk76Oh73rL+ggH2MQdYD6im7F+BuTW/04uBWfn8/9FIv3ZG+2PEB+DHMP1iYS6wHRg7QJsLgXuAN+YL+UfARVl3NoOH/qZ8sY7N4LixXttc/2vg74A98vH+2iCoabdfhtmZ2e/pub5/zX4vHuCYvg/8X+A04IB+dVNyzMdTvcv9UK53ZP0lwJ3APsDDwCca7GNfqjA/J8d4RAbWIVl/NPDbuY93UL3TOCnrpudzc132s08+1y8DfwyMAT5OdRJWbrOC3wz9gdr+mOqEsCfwPqpQrxvI2f5NwKfzeNfk38Rb+7U5CfgXQMC1dfq4gOoEvRlYBXy8pu7PgO/0a38rcF7N7/N7ue2ZI/26KeHh6Z3Ra3/gmYjYPkCbM4ALI2JjRPQCn6MK22bdEhH35T6+BswcoO3LwGTgLRHxckT8U+Srvp8TgMcj4vqI2B4RNwCPAr/X5JhOoXoX8ZfAzyQ9KOndWfcHwG0RcVtE/CoiuoBuqpMAVOE1DriP6ur0ygb7OBF4MiK+kmNcCXwj901ErIiIh3MfDwE3AL/Tr48LIuKFiPhFrq+JiP8TETuopkAmU03J1VO3raQDgHcDn42IlyLibmD5QE9WRDwVEZ+PiN8GPgKMB+6RtELS4dmsi+qdw0+AIySdIGlMTTc3A2+nunD4Y+Czkk7PutdRnRBqbQVeX7M+O8u+M9BYrTUc+qPXJmDiIHPGb6a6uuuzJsua9VTN8jaqF3gjfwv0AN+T9ISkRU2OqW9cU5oZUERsiYhFEXEoVWg+CHxLkqjm+U+R9Gzfg+pqeHJu+zLVledhwBcanJTIfo7s188ZVFfNSDpS0l2SeiVtBf4z1fRWrbX91l95LiNiWy42ej4btX0zsLmmrN5+BvI41ZRXD9VUzvjcxwtU7+jOo7ravwxY0fe3FRGPRMTPI2JHRPwIuAI4Oft8HnhDv/28geok0udKqpNvl6QJQxiv7QSH/uj1Y+BFqrfmjfycKsD6HJBlUM3Hv7avQtKbdmUwEfFcRJwXEW8FPgx8StIxTYypb1zrd2Kfz1BNdbyZatpoLXB9RIyveewbEZcCSJoCLAa+AnxB0l4Nul4L/KBfP6+LiI9n/T9QXWFPi4hxVNNa6j+8oR5PEzYA+0l6bU3ZtIE2kDRG0nGSbqCaaz+BaipuakT84JXBVoF+J9XVfifVtNXMBt0Gvz7eVcA78qTb5x1Z3mcH8LHc/+2S+p8krIUc+qNURGwFPgtcKekkSa+VtEe+wP8mm90AfEZSh6SJ2f6rWfdT4NC83W5vqqmPoXgaqL1t70RJB+WLfyvVC/1Xdba7DfgPeavpWEkfBQ6hmgcelKTLJB2W276eas67JyI25bH9nqRjM+z2lnR03uYpqqv8a6g+ZN0AXNRgN7fmGM/M53QPSe+W9Pasfz3VFfcvJc2iCrRhFxFrqK6YL5C0p6SjGGBaTNIbgXVUd3fdQ/UZzEci4h9rpwUldUo6smbTt1HdBNCb9fMkTVBlFvBJqjt2oPo8Ygfwybxb6BNZfme/sb9MNT32DHCbpH137lmwwTj0R7GI+ALwKeAzVC/QtcAngG9lk4upQuIhqg/yfpJlRMS/UH2o932qt/13D3H3FwBLc/rjVGBG9vU81buQL0fEXXXGvIlqzvw8qimqPwdOzKv2ZrwWuIXq7pwnqN41fDj7Xkt1u+pf8Ovn479RvQ4+SfWB9l/mtM45wDmS3l9njM9R3ZVyGtU7k6eopjz63hn8F+BCSc9RnUhvbnLsrXAG1d02m6h+lzdRveOrZxvVXTRHRMQVAzzHO4BLJa2lmrb5DvBneZKB6nnooZqyuQ64LPJ7BRHxEtW7zbOofid/SPWh9kv9d5JlHwF+CfyjpH2GcNzWpL5P/M1sFJJ0E/BoRPS/r35n+7s2Is5uRV82MnylbzaK5DTT2yS9RtJcqnc23xrhYdmryIh9G9DMhsWbgG9S3bK7juqe+ZWt6txX+bs/T++YmRXE0ztmZgV5VU/vTJw4MaZPnz7SwzAz26088MADz0RER726V3XoT58+ne7u7pEehpnZbiX/RdS6PL1jZlYQh76ZWUEc+mZmBXHom5kVxKFvZlYQh76ZWUEc+mZmBXHom5kVxKFvZlaQV/U3cs1s101f9O2RHsKo8eSlJ4z0EHaZr/TNzAri0DczK4hD38ysIA59M7OCOPTNzAri0DczK4hD38ysIA59M7OCNBX6ksZL+rqkRyWtlnSUpP0kdUl6PH9OyLaS9CVJPZIekvTOmn7mZ/vHJc0froMyM7P6mr3SvwL4bkQcDBwOrAYWAXdExAzgjlwHOA6YkY+FwFUAkvYDFgNHArOAxX0nCjMza49BQ1/SOOA/AtcARMRLEfEsMA9Yms2WAifl8jzguqjcA4yXNBk4FuiKiM0RsQXoAua28FjMzGwQzVzpHwj0Al+RtFLS30vaF5gUERuyzVPApFyeAqyt2X5dljUqNzOzNmkm9McC7wSuiogjgBf49VQOABERQLRiQJIWSuqW1N3b29uKLs3MLDUT+uuAdRFxb65/neok8HRO25A/N2b9emBazfZTs6xR+W+IiKsjojMiOjs6OoZyLGZmNohBQz8ingLWSvqtLDoGeARYDvTdgTMfWJbLy4Gz8i6e2cDWnAa6HZgjaUJ+gDsny8zMrE2a/ff0/wT4mqQ9gSeAc6hOGDdLWgCsAU7NtrcBxwM9wLZsS0RslnQRcH+2uzAiNrfkKMzMrClNhX5EPAh01qk6pk7bAM5t0M8SYMkQxmdmZi3kb+SamRXEoW9mVhCHvplZQRz6ZmYFceibmRXEoW9mVhCHvplZQRz6ZmYFceibmRXEoW9mVhCHvplZQRz6ZmYFceibmRXEoW9mVhCHvplZQRz6ZmYFceibmRXEoW9mVhCHvplZQRz6ZmYFceibmRXEoW9mVhCHvplZQRz6ZmYFaSr0JT0p6WFJD0rqzrL9JHVJejx/TshySfqSpB5JD0l6Z00/87P945LmD88hmZlZI0O50v/diJgZEZ25vgi4IyJmAHfkOsBxwIx8LASuguokASwGjgRmAYv7ThRmZtYeuzK9Mw9YmstLgZNqyq+Lyj3AeEmTgWOBrojYHBFbgC5g7i7s38zMhqjZ0A/ge5IekLQwyyZFxIZcfgqYlMtTgLU1267Lskblv0HSQkndkrp7e3ubHJ6ZmTVjbJPt3hcR6yW9EeiS9GhtZUSEpGjFgCLiauBqgM7Ozpb0aWZmlaau9CNiff7cCNxCNSf/dE7bkD83ZvP1wLSazadmWaNyMzNrk0FDX9K+kl7ftwzMAf4ZWA703YEzH1iWy8uBs/IuntnA1pwGuh2YI2lCfoA7J8vMzKxNmpnemQTcIqmv/T9ExHcl3Q/cLGkBsAY4NdvfBhwP9ADbgHMAImKzpIuA+7PdhRGxuWVHYmZmgxo09CPiCeDwOuWbgGPqlAdwboO+lgBLhj5MMzNrBX8j18ysIA59M7OCOPTNzAri0DczK4hD38ysIA59M7OCOPTNzAri0DczK4hD38ysIA59M7OCOPTNzAri0DczK4hD38ysIA59M7OCOPTNzAri0DczK4hD38ysIA59M7OCOPTNzAri0DczK4hD38ysIA59M7OCOPTNzArSdOhLGiNppaRbc/1ASfdK6pF0k6Q9s3yvXO/J+uk1fZyf5Y9JOrblR2NmZgMaypX+nwKra9YvAy6PiIOALcCCLF8AbMnyy7Mdkg4BTgMOBeYCX5Y0ZteGb2ZmQ9FU6EuaCpwA/H2uC/gA8PVsshQ4KZfn5TpZf0y2nwfcGBEvRsTPgB5gVguOwczMmtTslf4XgT8HfpXr+wPPRsT2XF8HTMnlKcBagKzfmu1fKa+zzSskLZTULam7t7e3+SMxM7NBDRr6kk4ENkbEA20YDxFxdUR0RkRnR0dHO3ZpZlaMsU20eS/wYUnHA3sDbwCuAMZLGptX81OB9dl+PTANWCdpLDAO2FRT3qd2GzMza4NBr/Qj4vyImBoR06k+iL0zIs4A7gJOzmbzgWW5vDzXyfo7IyKy/LS8u+dAYAZwX8uOxMzMBtXMlX4j/x24UdLFwErgmiy/BrheUg+wmepEQUSsknQz8AiwHTg3Inbswv7NzGyIhhT6EbECWJHLT1Dn7puI+CVwSoPtLwEuGeogzcysNfyNXDOzgjj0zcwK4tA3MyuIQ9/MrCAOfTOzgjj0zcwK4tA3MyuIQ9/MrCAOfTOzgjj0zcwK4tA3MyuIQ9/MrCAOfTOzgjj0zcwK4tA3MyuIQ9/MrCAOfTOzgjj0zcwK4tA3MyuIQ9/MrCAOfTOzgjj0zcwK4tA3MyuIQ9/MrCCDhr6kvSXdJ+mnklZJ+lyWHyjpXkk9km6StGeW75XrPVk/vaav87P8MUnHDttRmZlZXc1c6b8IfCAiDgdmAnMlzQYuAy6PiIOALcCCbL8A2JLll2c7JB0CnAYcCswFvixpTAuPxczMBjFo6Efl+VzdIx8BfAD4epYvBU7K5Xm5TtYfI0lZfmNEvBgRPwN6gFmtOAgzM2tOU3P6ksZIehDYCHQB/w94NiK2Z5N1wJRcngKsBcj6rcD+teV1tqnd10JJ3ZK6e3t7h3xAZmbWWFOhHxE7ImImMJXq6vzg4RpQRFwdEZ0R0dnR0TFcuzEzK9KQ7t6JiGeBu4CjgPGSxmbVVGB9Lq8HpgFk/ThgU215nW3MzKwNmrl7p0PS+FzeB/gQsJoq/E/OZvOBZbm8PNfJ+jsjIrL8tLy750BgBnBfi47DzMyaMHbwJkwGluadNq8Bbo6IWyU9Atwo6WJgJXBNtr8GuF5SD7CZ6o4dImKVpJuBR4DtwLkRsaO1h2NmZgMZNPQj4iHgiDrlT1Dn7puI+CVwSoO+LgEuGfowzcysFfyNXDOzgjj0zcwK4tA3MyuIQ9/MrCAOfTOzgjj0zcwK4tA3MyuIQ9/MrCAOfTOzgjj0zcwK4tA3MyuIQ9/MrCAOfTOzgjj0zcwK4tA3MyuIQ9/MrCAOfTOzgjj0zcwK4tA3MyuIQ9/MrCAOfTOzgjj0zcwK4tA3MyvIoKEvaZqkuyQ9ImmVpD/N8v0kdUl6PH9OyHJJ+pKkHkkPSXpnTV/zs/3jkuYP32GZmVk9zVzpbwfOi4hDgNnAuZIOARYBd0TEDOCOXAc4DpiRj4XAVVCdJIDFwJHALGBx34nCzMzaY9DQj4gNEfGTXH4OWA1MAeYBS7PZUuCkXJ4HXBeVe4DxkiYDxwJdEbE5IrYAXcDcVh6MmZkNbEhz+pKmA0cA9wKTImJDVj0FTMrlKcDams3WZVmj8v77WCipW1J3b2/vUIZnZmaDaDr0Jb0O+AbwXyPi32rrIiKAaMWAIuLqiOiMiM6Ojo5WdGlmZqmp0Je0B1Xgfy0ivpnFT+e0DflzY5avB6bVbD41yxqVm5lZmzRz946Aa4DVEfE/a6qWA3134MwHltWUn5V38cwGtuY00O3AHEkT8gPcOVlmZmZtMraJNu8FzgQelvRglv0FcClws6QFwBrg1Ky7DTge6AG2AecARMRmSRcB92e7CyNicysOwszMmjNo6EfE3YAaVB9Tp30A5zboawmwZCgDNDOz1vE3cs3MCuLQNzMriEPfzKwgDn0zs4I49M3MCuLQNzMriEPfzKwgDn0zs4I49M3MCuLQNzMriEPfzKwgDn0zs4I49M3MCuLQNzMriEPfzKwgDn0zs4I49M3MCuLQNzMriEPfzKwgDn0zs4I49M3MCuLQNzMriEPfzKwgYwdrIGkJcCKwMSIOy7L9gJuA6cCTwKkRsUWSgCuA44FtwNkR8ZPcZj7wmez24ohY2tpDGTnTF317pIcwqjx56QkjPQSzUauZK/1rgbn9yhYBd0TEDOCOXAc4DpiRj4XAVfDKSWIxcCQwC1gsacKuDt7MzIZm0NCPiB8Cm/sVzwP6rtSXAifVlF8XlXuA8ZImA8cCXRGxOSK2AF38+xOJmZkNs52d058UERty+SlgUi5PAdbWtFuXZY3K/x1JCyV1S+ru7e3dyeGZmVk9u/xBbkQEEC0YS19/V0dEZ0R0dnR0tKpbMzNj50P/6Zy2IX9uzPL1wLSadlOzrFG5mZm10c6G/nJgfi7PB5bVlJ+lymxga04D3Q7MkTQhP8Cdk2VmZtZGzdyyeQNwNDBR0jqqu3AuBW6WtABYA5yazW+jul2zh+qWzXMAImKzpIuA+7PdhRHR/8NhMzMbZoOGfkSc3qDqmDptAzi3QT9LgCVDGp2ZmbWUv5FrZlYQh76ZWUEc+mZmBXHom5kVxKFvZlYQh76ZWUEc+mZmBXHom5kVxKFvZlYQh76ZWUEc+mZmBXHom5kVxKFvZlYQh76ZWUEc+mZmBXHom5kVxKFvZlYQh76ZWUEc+mZmBXHom5kVxKFvZlYQh76ZWUEc+mZmBXHom5kVpO2hL2mupMck9Uha1O79m5mVrK2hL2kMcCVwHHAIcLqkQ9o5BjOzkrX7Sn8W0BMRT0TES8CNwLw2j8HMrFhj27y/KcDamvV1wJG1DSQtBBbm6vOSHmvT2EowEXhmpAcxGF020iOwEeC/zdZ6S6OKdof+oCLiauDqkR7HaCSpOyI6R3ocZv35b7N92j29sx6YVrM+NcvMzKwN2h369wMzJB0oaU/gNGB5m8dgZlastk7vRMR2SZ8AbgfGAEsiYlU7x1A4T5vZq5X/NttEETHSYzAzszbxN3LNzAri0DczK4hDfzchaYekB2se04dxX09Kmjhc/Vs5JIWkr9asj5XUK+nWQbY7erA2tnNedffpW0O/iIiZIz0IsyF6AThM0j4R8QvgQ/g27RHlK/3dmKR3SfqBpAck3S5pcpavkHS5pG5JqyW9W9I3JT0u6eKa7b+V267Kb0LX28cfSLov31387/z3k8yG4jbghFw+Hbihr0LSLEk/lrRS0o8k/Vb/jSXtK2lJ/h2ulOR/umUXOPR3H/vUTO3cImkP4H8BJ0fEu4AlwCU17V/Kbzj+HbAMOBc4DDhb0v7Z5g9z207gkzXlAEh6O/BR4L35LmMHcMbwHaKNUjcCp0naG3gHcG9N3aPA+yPiCOCzwF/V2f5/AHdGxCzgd4G/lbTvMI951PL0zu7jN6Z3JB1GFeJdkqD63sOGmvZ9X3p7GFgVERtyuyeovhW9iSrofz/bTQNmZHmfY4B3AffnPvYBNrb0qGzUi4iH8jOo06mu+muNA5ZKmgEEsEedLuYAH5b06VzfGzgAWD08Ix7dHPq7L1GF+VEN6l/Mn7+qWe5bHyvpaOCDwFERsU3SCqoXU/99LI2I81s1aCvWcuDzwNFA7TvKi4C7IuL388Swos62Av5TRPgfX2wBT+/svh4DOiQdBSBpD0mHDmH7ccCWDPyDgdl12twBnCzpjbmP/SQ1/Nf7zAawBPhcRDzcr3wcv/5g9+wG294O/Iny7aakI4ZlhIVw6O+m8v8jOBm4TNJPgQeB9wyhi+9SXfGvBi4F7qmzj0eAzwDfk/QQ0AVM3sWhW4EiYl1EfKlO1d8Afy1pJY1nHi6imvZ5SNKqXLed5H+GwcysIL7SNzMriEPfzKwgDn0zs4I49M3MCuLQNzMriEPfzKwgDn0zs4L8f+dRfx5JxqxbAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "training_data[\"Sex\"].where(training_data[\"Target\"] == \">50K\").value_counts().sort_values().plot(\n", " kind=\"bar\", title=\"Counts of Sex earning >$50K\", rot=0\n", ")" ] }, { "cell_type": "markdown", "id": "c4a4afec", "metadata": { "papermill": { "duration": 0.086259, "end_time": "2022-04-18T15:42:14.767130", "exception": false, "start_time": "2022-04-18T15:42:14.680871", "status": "completed" }, "tags": [] }, "source": [ "### Encode and Upload the Dataset\n", "Here we encode the training and test data. Encoding input data is not necessary for SageMaker Clarify, but is necessary for the model." ] }, { "cell_type": "code", "execution_count": 7, "id": "c54fd61c", "metadata": { "execution": { "iopub.execute_input": "2022-04-18T15:42:14.886149Z", "iopub.status.busy": "2022-04-18T15:42:14.885365Z", "iopub.status.idle": "2022-04-18T15:42:16.872703Z", "shell.execute_reply": "2022-04-18T15:42:16.872261Z" }, "papermill": { "duration": 2.082627, "end_time": "2022-04-18T15:42:16.872819", "exception": false, "start_time": "2022-04-18T15:42:14.790192", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "from sklearn import preprocessing\n", "\n", "\n", "def number_encode_features(df):\n", " result = df.copy()\n", " encoders = {}\n", " for column in result.columns:\n", " if result.dtypes[column] == np.object:\n", " encoders[column] = preprocessing.LabelEncoder()\n", " result[column] = encoders[column].fit_transform(result[column].fillna(\"None\"))\n", " return result, encoders\n", "\n", "\n", "training_data = pd.concat([training_data[\"Target\"], training_data.drop([\"Target\"], axis=1)], axis=1)\n", "training_data, _ = number_encode_features(training_data)\n", "training_data.to_csv(\"train_data.csv\", index=False, header=False)\n", "\n", "testing_data, _ = number_encode_features(testing_data)\n", "test_features = testing_data.drop([\"Target\"], axis=1)\n", "test_target = testing_data[\"Target\"]\n", "test_features.to_csv(\"test_features.csv\", index=False, header=False)" ] }, { "cell_type": "markdown", "id": "0dff253a", "metadata": { "papermill": { "duration": 0.085611, "end_time": "2022-04-18T15:42:16.982827", "exception": false, "start_time": "2022-04-18T15:42:16.897216", "status": "completed" }, "tags": [] }, "source": [ "A quick note about our encoding: the \"Female\" Sex value has been encoded as 0 and \"Male\" as 1." ] }, { "cell_type": "code", "execution_count": 8, "id": "53706464", "metadata": { "execution": { "iopub.execute_input": "2022-04-18T15:42:17.176444Z", "iopub.status.busy": "2022-04-18T15:42:17.175481Z", "iopub.status.idle": "2022-04-18T15:42:17.179230Z", "shell.execute_reply": "2022-04-18T15:42:17.179614Z" }, "papermill": { "duration": 0.111035, "end_time": "2022-04-18T15:42:17.179749", "exception": false, "start_time": "2022-04-18T15:42:17.068714", "status": "completed" }, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
TargetAgeWorkclassfnlwgtEducationEducation-NumMarital StatusOccupationRelationshipEthnic groupSexCapital GainCapital LossHours per weekCountry
003957751691340141217404038
105048331191323041001338
2038221564611905141004038
305322347211725021004038
402823384099132952000404
\n", "
" ], "text/plain": [ " Target Age Workclass fnlwgt Education Education-Num Marital Status \\\n", "0 0 39 5 77516 9 13 4 \n", "1 0 50 4 83311 9 13 2 \n", "2 0 38 2 215646 11 9 0 \n", "3 0 53 2 234721 1 7 2 \n", "4 0 28 2 338409 9 13 2 \n", "\n", " Occupation Relationship Ethnic group Sex Capital Gain Capital Loss \\\n", "0 0 1 4 1 2174 0 \n", "1 3 0 4 1 0 0 \n", "2 5 1 4 1 0 0 \n", "3 5 0 2 1 0 0 \n", "4 9 5 2 0 0 0 \n", "\n", " Hours per week Country \n", "0 40 38 \n", "1 13 38 \n", "2 40 38 \n", "3 40 38 \n", "4 40 4 " ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "training_data.head()" ] }, { "cell_type": "markdown", "id": "fa10543a", "metadata": { "papermill": { "duration": 0.024025, "end_time": "2022-04-18T15:42:17.290367", "exception": false, "start_time": "2022-04-18T15:42:17.266342", "status": "completed" }, "tags": [] }, "source": [ "Lastly, let's upload the data to S3." ] }, { "cell_type": "code", "execution_count": 9, "id": "c1e828e1", "metadata": { "execution": { "iopub.execute_input": "2022-04-18T15:42:17.776575Z", "iopub.status.busy": "2022-04-18T15:42:17.472573Z", "iopub.status.idle": "2022-04-18T15:42:18.190540Z", "shell.execute_reply": "2022-04-18T15:42:18.190923Z" }, "papermill": { "duration": 0.813762, "end_time": "2022-04-18T15:42:18.191062", "exception": false, "start_time": "2022-04-18T15:42:17.377300", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "from sagemaker.s3 import S3Uploader\n", "from sagemaker.inputs import TrainingInput\n", "\n", "train_uri = S3Uploader.upload(\"train_data.csv\", \"s3://{}/{}\".format(bucket, prefix))\n", "train_input = TrainingInput(train_uri, content_type=\"csv\")\n", "test_uri = S3Uploader.upload(\"test_features.csv\", \"s3://{}/{}\".format(bucket, prefix))" ] }, { "cell_type": "markdown", "id": "d9e9b77c", "metadata": { "papermill": { "duration": 0.02369, "end_time": "2022-04-18T15:42:18.238778", "exception": false, "start_time": "2022-04-18T15:42:18.215088", "status": "completed" }, "tags": [] }, "source": [ "### Train XGBoost Model\n", "#### Train Model\n", "Since our focus is on understanding how to use SageMaker Clarify, we keep it simple by using a standard XGBoost model." ] }, { "cell_type": "code", "execution_count": 10, "id": "fcab8273", "metadata": { "execution": { "iopub.execute_input": "2022-04-18T15:42:18.371152Z", "iopub.status.busy": "2022-04-18T15:42:18.370192Z", "iopub.status.idle": "2022-04-18T15:45:44.661620Z", "shell.execute_reply": "2022-04-18T15:45:44.662200Z" }, "papermill": { "duration": 206.396586, "end_time": "2022-04-18T15:45:44.662348", "exception": false, "start_time": "2022-04-18T15:42:18.265762", "status": "completed" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "2022-04-18 15:42:18 Starting - Starting the training job...\n", "2022-04-18 15:42:34 Starting - Preparing the instances for training............\n", "2022-04-18 15:43:42 Downloading - Downloading input data...\n", "2022-04-18 15:44:02 Training - Downloading the training image...........\n", "2022-04-18 15:45:03 Training - Training image download completed. Training in progress......\n", "2022-04-18 15:45:34 Uploading - Uploading generated training model.\n", "2022-04-18 15:45:40 Completed - Training job completed\n" ] } ], "source": [ "from sagemaker.image_uris import retrieve\n", "from sagemaker.estimator import Estimator\n", "\n", "container = retrieve(\"xgboost\", region, version=\"1.2-1\")\n", "xgb = Estimator(\n", " container,\n", " role,\n", " instance_count=1,\n", " instance_type=\"ml.m5.xlarge\",\n", " disable_profiler=True,\n", " sagemaker_session=session,\n", ")\n", "\n", "xgb.set_hyperparameters(\n", " max_depth=5,\n", " eta=0.2,\n", " gamma=4,\n", " min_child_weight=6,\n", " subsample=0.8,\n", " objective=\"binary:logistic\",\n", " num_round=800,\n", ")\n", "\n", "xgb.fit({\"train\": train_input}, logs=False)" ] }, { "cell_type": "markdown", "id": "5d67780c", "metadata": { "papermill": { "duration": 0.044056, "end_time": "2022-04-18T15:45:44.748690", "exception": false, "start_time": "2022-04-18T15:45:44.704634", "status": "completed" }, "tags": [] }, "source": [ "#### Create Model\n", "Here we create the SageMaker model." ] }, { "cell_type": "code", "execution_count": 11, "id": "f39807a0", "metadata": { "execution": { "iopub.execute_input": "2022-04-18T15:45:44.844076Z", "iopub.status.busy": "2022-04-18T15:45:44.843305Z", "iopub.status.idle": "2022-04-18T15:45:45.241770Z", "shell.execute_reply": "2022-04-18T15:45:45.242483Z" }, "papermill": { "duration": 0.445712, "end_time": "2022-04-18T15:45:45.242628", "exception": false, "start_time": "2022-04-18T15:45:44.796916", "status": "completed" }, "tags": [] }, "outputs": [ { "data": { "text/plain": [ "'DEMO-clarify-model-18-04-2022-15-45-44'" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model_name = \"DEMO-clarify-model-{}\".format(datetime.now().strftime(\"%d-%m-%Y-%H-%M-%S\"))\n", "model = xgb.create_model(name=model_name)\n", "container_def = model.prepare_container_def()\n", "session.create_model(model_name, role, container_def)" ] }, { "cell_type": "markdown", "id": "efc6e484", "metadata": { "papermill": { "duration": 0.035834, "end_time": "2022-04-18T15:45:45.317431", "exception": false, "start_time": "2022-04-18T15:45:45.281597", "status": "completed" }, "tags": [] }, "source": [ "## Amazon SageMaker Clarify\n", "Now that you have your model set up, let's say hello to SageMaker Clarify!" ] }, { "cell_type": "code", "execution_count": 12, "id": "7d9f5fab", "metadata": { "execution": { "iopub.execute_input": "2022-04-18T15:45:45.412724Z", "iopub.status.busy": "2022-04-18T15:45:45.394186Z", "iopub.status.idle": "2022-04-18T15:45:45.416063Z", "shell.execute_reply": "2022-04-18T15:45:45.415616Z" }, "papermill": { "duration": 0.062687, "end_time": "2022-04-18T15:45:45.416177", "exception": false, "start_time": "2022-04-18T15:45:45.353490", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "from sagemaker import clarify\n", "\n", "clarify_processor = clarify.SageMakerClarifyProcessor(\n", " role=role, instance_count=1, instance_type=\"ml.m5.xlarge\", sagemaker_session=session\n", ")" ] }, { "cell_type": "markdown", "id": "dce26f7f", "metadata": { "papermill": { "duration": 0.035006, "end_time": "2022-04-18T15:45:45.488228", "exception": false, "start_time": "2022-04-18T15:45:45.453222", "status": "completed" }, "tags": [] }, "source": [ "### Detecting Bias\n", "SageMaker Clarify helps you detect possible pre- and post-training biases using a variety of metrics.\n", "#### Writing DataConfig and ModelConfig\n", "A `DataConfig` object communicates some basic information about data I/O to SageMaker Clarify. We specify where to find the input dataset, where to store the output, the target column (`label`), the header names, and the dataset type." ] }, { "cell_type": "code", "execution_count": 13, "id": "47bf995f", "metadata": { "execution": { "iopub.execute_input": "2022-04-18T15:45:45.570752Z", "iopub.status.busy": "2022-04-18T15:45:45.569644Z", "iopub.status.idle": "2022-04-18T15:45:45.572869Z", "shell.execute_reply": "2022-04-18T15:45:45.572368Z" }, "papermill": { "duration": 0.043789, "end_time": "2022-04-18T15:45:45.572994", "exception": false, "start_time": "2022-04-18T15:45:45.529205", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "bias_report_output_path = \"s3://{}/{}/clarify-bias\".format(bucket, prefix)\n", "bias_data_config = clarify.DataConfig(\n", " s3_data_input_path=train_uri,\n", " s3_output_path=bias_report_output_path,\n", " label=\"Target\",\n", " headers=training_data.columns.to_list(),\n", " dataset_type=\"text/csv\",\n", ")" ] }, { "cell_type": "markdown", "id": "533631dd", "metadata": { "papermill": { "duration": 0.040195, "end_time": "2022-04-18T15:45:45.654040", "exception": false, "start_time": "2022-04-18T15:45:45.613845", "status": "completed" }, "tags": [] }, "source": [ "A `ModelConfig` object communicates information about your trained model. To avoid additional traffic to your production models, SageMaker Clarify sets up and tears down a dedicated endpoint when processing.\n", "* `instance_type` and `instance_count` specify your preferred instance type and instance count used to run your model on during SageMaker Clarify's processing. The testing dataset is small so a single standard instance is good enough to run this example. If your have a large complex dataset, you may want to use a better instance type to speed up, or add more instances to enable Spark parallelization.\n", "* `accept_type` denotes the endpoint response payload format, and `content_type` denotes the payload format of request to the endpoint." ] }, { "cell_type": "code", "execution_count": 14, "id": "a8971b78", "metadata": { "execution": { "iopub.execute_input": "2022-04-18T15:45:45.730684Z", "iopub.status.busy": "2022-04-18T15:45:45.729722Z", "iopub.status.idle": "2022-04-18T15:45:45.732326Z", "shell.execute_reply": "2022-04-18T15:45:45.732966Z" }, "papermill": { "duration": 0.043473, "end_time": "2022-04-18T15:45:45.733104", "exception": false, "start_time": "2022-04-18T15:45:45.689631", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "model_config = clarify.ModelConfig(\n", " model_name=model_name,\n", " instance_type=\"ml.m5.xlarge\",\n", " instance_count=1,\n", " accept_type=\"text/csv\",\n", " content_type=\"text/csv\",\n", ")" ] }, { "cell_type": "markdown", "id": "29559ca9", "metadata": { "papermill": { "duration": 0.040697, "end_time": "2022-04-18T15:45:45.809694", "exception": false, "start_time": "2022-04-18T15:45:45.768997", "status": "completed" }, "tags": [] }, "source": [ "A `ModelPredictedLabelConfig` provides information on the format of your predictions. XGBoost model outputs probabilities of samples, so SageMaker Clarify invokes the endpoint then uses `probability_threshold` to convert the probability to binary labels for bias analysis. Prediction above the threshold is interpreted as label value `1` and below or equal as label value `0`." ] }, { "cell_type": "code", "execution_count": 15, "id": "1b8c33af", "metadata": { "execution": { "iopub.execute_input": "2022-04-18T15:45:45.887354Z", "iopub.status.busy": "2022-04-18T15:45:45.886505Z", "iopub.status.idle": "2022-04-18T15:45:45.888916Z", "shell.execute_reply": "2022-04-18T15:45:45.888503Z" }, "papermill": { "duration": 0.040649, "end_time": "2022-04-18T15:45:45.889059", "exception": false, "start_time": "2022-04-18T15:45:45.848410", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "predictions_config = clarify.ModelPredictedLabelConfig(probability_threshold=0.8)" ] }, { "cell_type": "markdown", "id": "23672b46", "metadata": { "papermill": { "duration": 0.034177, "end_time": "2022-04-18T15:45:45.957127", "exception": false, "start_time": "2022-04-18T15:45:45.922950", "status": "completed" }, "tags": [] }, "source": [ "#### Writing BiasConfig\n", "SageMaker Clarify also needs information on what the sensitive columns (`facets`) are, what the sensitive features (`facet_values_or_threshold`) may be, and what the desirable outcomes are (`label_values_or_threshold`).\n", "SageMaker Clarify can handle both categorical and continuous data for `facet_values_or_threshold` and for `label_values_or_threshold`. In this case we are using categorical data.\n", "\n", "We specify this information in the `BiasConfig` API. Here that the positive outcome is earning >$50,000, Sex is a sensitive category, and Female respondents are the sensitive group. `group_name` is used to form subgroups for the measurement of Conditional Demographic Disparity in Labels (CDDL) and Conditional Demographic Disparity in Predicted Labels (CDDPL) with regards to Simpson\u2019s paradox." ] }, { "cell_type": "code", "execution_count": 16, "id": "95207e36", "metadata": { "execution": { "iopub.execute_input": "2022-04-18T15:45:46.030157Z", "iopub.status.busy": "2022-04-18T15:45:46.029301Z", "iopub.status.idle": "2022-04-18T15:45:46.032167Z", "shell.execute_reply": "2022-04-18T15:45:46.031691Z" }, "papermill": { "duration": 0.04116, "end_time": "2022-04-18T15:45:46.032286", "exception": false, "start_time": "2022-04-18T15:45:45.991126", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "bias_config = clarify.BiasConfig(\n", " label_values_or_threshold=[1], facet_name=\"Sex\", facet_values_or_threshold=[0], group_name=\"Age\"\n", ")" ] }, { "cell_type": "markdown", "id": "c5e5b7be", "metadata": { "papermill": { "duration": 0.034256, "end_time": "2022-04-18T15:45:46.100764", "exception": false, "start_time": "2022-04-18T15:45:46.066508", "status": "completed" }, "tags": [] }, "source": [ "#### Pre-training Bias\n", "Bias can be present in your data before any model training occurs. Inspecting your data for bias before training begins can help detect any data collection gaps, inform your feature engineering, and help you understand what societal biases the data may reflect.\n", "\n", "Computing pre-training bias metrics does not require a trained model.\n", "\n", "#### Post-training Bias\n", "Computing post-training bias metrics does require a trained model.\n", "\n", "Unbiased training data (as determined by concepts of fairness measured by bias metric) may still result in biased model predictions after training. Whether this occurs depends on several factors including hyperparameter choices.\n", "\n", "\n", "You can run these options separately with `run_pre_training_bias()` and `run_post_training_bias()` or at the same time with `run_bias()` as shown below." ] }, { "cell_type": "code", "execution_count": 17, "id": "f7001f07", "metadata": { "execution": { "iopub.execute_input": "2022-04-18T15:45:46.174561Z", "iopub.status.busy": "2022-04-18T15:45:46.173716Z", "iopub.status.idle": "2022-04-18T15:54:02.714315Z", "shell.execute_reply": "2022-04-18T15:54:02.714703Z" }, "papermill": { "duration": 496.580275, "end_time": "2022-04-18T15:54:02.714839", "exception": false, "start_time": "2022-04-18T15:45:46.134564", "status": "completed" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Job Name: Clarify-Bias-2022-04-18-15-45-46-171\n", "Inputs: [{'InputName': 'dataset', 'AppManaged': False, 'S3Input': {'S3Uri': 's3://sagemaker-us-west-2-000000000000/sagemaker/DEMO-sagemaker-clarify/train_data.csv', 'LocalPath': '/opt/ml/processing/input/data', 'S3DataType': 'S3Prefix', 'S3InputMode': 'File', 'S3DataDistributionType': 'FullyReplicated', 'S3CompressionType': 'None'}}, {'InputName': 'analysis_config', 'AppManaged': False, 'S3Input': {'S3Uri': 's3://sagemaker-us-west-2-000000000000/sagemaker/DEMO-sagemaker-clarify/clarify-bias/analysis_config.json', 'LocalPath': '/opt/ml/processing/input/config', 'S3DataType': 'S3Prefix', 'S3InputMode': 'File', 'S3DataDistributionType': 'FullyReplicated', 'S3CompressionType': 'None'}}]\n", "Outputs: [{'OutputName': 'analysis_result', 'AppManaged': False, 'S3Output': {'S3Uri': 's3://sagemaker-us-west-2-000000000000/sagemaker/DEMO-sagemaker-clarify/clarify-bias', 'LocalPath': '/opt/ml/processing/output', 'S3UploadMode': 'EndOfJob'}}]\n", ".............................\u001b[34m2022-04-18 15:50:24,047 logging.conf not found when configuring logging, using default logging configuration.\u001b[0m\n", "\u001b[34m2022-04-18 15:50:24,047 Starting SageMaker Clarify Processing job\u001b[0m\n", "\u001b[34m2022-04-18 15:50:24,048 Analysis config path: /opt/ml/processing/input/config/analysis_config.json\u001b[0m\n", "\u001b[34m2022-04-18 15:50:24,048 Analysis result path: /opt/ml/processing/output\u001b[0m\n", "\u001b[34m2022-04-18 15:50:24,048 This host is algo-1.\u001b[0m\n", "\u001b[34m2022-04-18 15:50:24,048 This host is the leader.\u001b[0m\n", "\u001b[34m2022-04-18 15:50:24,048 Number of hosts in the cluster is 1.\u001b[0m\n", "\u001b[34m2022-04-18 15:50:24,194 Running Python / Pandas based analyzer.\u001b[0m\n", "\u001b[34m2022-04-18 15:50:24,195 Dataset type: text/csv uri: /opt/ml/processing/input/data\u001b[0m\n", "\u001b[34m2022-04-18 15:50:24,205 Loading dataset...\u001b[0m\n", "\u001b[34m2022-04-18 15:50:24,239 Loaded dataset. Dataset info:\u001b[0m\n", "\u001b[34m\u001b[0m\n", "\u001b[34mRangeIndex: 30162 entries, 0 to 30161\u001b[0m\n", "\u001b[34mData columns (total 14 columns):\n", " # Column Non-Null Count Dtype\u001b[0m\n", "\u001b[34m--- ------ -------------- -----\n", " 0 Age 30162 non-null int64\n", " 1 Workclass 30162 non-null int64\n", " 2 fnlwgt 30162 non-null int64\n", " 3 Education 30162 non-null int64\n", " 4 Education-Num 30162 non-null int64\n", " 5 Marital Status 30162 non-null int64\n", " 6 Occupation 30162 non-null int64\n", " 7 Relationship 30162 non-null int64\n", " 8 Ethnic group 30162 non-null int64\n", " 9 Sex 30162 non-null int64\n", " 10 Capital Gain 30162 non-null int64\n", " 11 Capital Loss 30162 non-null int64\n", " 12 Hours per week 30162 non-null int64\n", " 13 Country 30162 non-null int64\u001b[0m\n", "\u001b[34mdtypes: int64(14)\u001b[0m\n", "\u001b[34mmemory usage: 3.2 MB\u001b[0m\n", "\u001b[34m2022-04-18 15:50:24,377 Spinning up shadow endpoint\u001b[0m\n", "\u001b[34m2022-04-18 15:50:24,377 Creating endpoint-config with name sm-clarify-config-1650297024-8acd\u001b[0m\n", "\u001b[34m2022-04-18 15:50:24,486 Creating endpoint: 'sm-clarify-DEMO-clarify-model-18-04-2022-15-45--1650297024-12c3'\u001b[0m\n", "\u001b[34m2022-04-18 15:50:24,766 Using endpoint name: sm-clarify-DEMO-clarify-model-18-04-2022-15-45--1650297024-12c3\u001b[0m\n", "\u001b[34m2022-04-18 15:50:24,767 Waiting for endpoint ...\u001b[0m\n", "\u001b[34m2022-04-18 15:50:24,767 Checking endpoint status:\u001b[0m\n", "\u001b[34mLegend:\u001b[0m\n", "\u001b[34m(OutOfService: x, Creating: -, Updating: -, InService: !, RollingBack: <, Deleting: o, Failed: *)\u001b[0m\n", "\u001b[34m2022-04-18 15:53:25,164 Endpoint is in service after 180 seconds\u001b[0m\n", "\u001b[34m2022-04-18 15:53:25,164 Endpoint ready.\u001b[0m\n", "\u001b[34m2022-04-18 15:53:25,164 ======================================\u001b[0m\n", "\u001b[34m2022-04-18 15:53:25,164 Calculating post-training bias metrics\u001b[0m\n", "\u001b[34m2022-04-18 15:53:25,164 ======================================\u001b[0m\n", "\u001b[34m2022-04-18 15:53:25,164 Getting predictions from the endpoint\u001b[0m\n", "\u001b[34m2022-04-18 15:53:27,354 We assume a prediction above 0.800 indicates 1 and below or equal indicates 0.\u001b[0m\n", "\u001b[34m2022-04-18 15:53:27,355 Column Target with data uniqueness fraction 6.630860022544923e-05 is classifed as a CATEGORICAL column\u001b[0m\n", "\u001b[34m2022-04-18 15:53:27,357 Column Sex with data uniqueness fraction 6.630860022544923e-05 is classifed as a CATEGORICAL column\u001b[0m\n", "\u001b[34m2022-04-18 15:53:27,360 Column Target with data uniqueness fraction 6.630860022544923e-05 is classifed as a CATEGORICAL column\u001b[0m\n", "\u001b[34m2022-04-18 15:53:27,362 Column None with data uniqueness fraction 6.630860022544923e-05 is classifed as a CATEGORICAL column\u001b[0m\n", "\u001b[34m2022-04-18 15:53:28,676 Stop using endpoint: sm-clarify-DEMO-clarify-model-18-04-2022-15-45--1650297024-12c3\u001b[0m\n", "\u001b[34m2022-04-18 15:53:28,676 Deleting endpoint configuration with name: sm-clarify-config-1650297024-8acd\u001b[0m\n", "\u001b[34m2022-04-18 15:53:28,745 Deleting endpoint with name: sm-clarify-DEMO-clarify-model-18-04-2022-15-45--1650297024-12c3\u001b[0m\n", "\u001b[34m2022-04-18 15:53:28,834 Model endpoint delivered 0.56949 requests per second and a total of 2 requests over 4 seconds\u001b[0m\n", "\u001b[34m2022-04-18 15:53:28,835 Stop using endpoint: None\u001b[0m\n", "\u001b[34m2022-04-18 15:53:28,835 =====================================\u001b[0m\n", "\u001b[34m2022-04-18 15:53:28,835 Calculating pre-training bias metrics\u001b[0m\n", "\u001b[34m2022-04-18 15:53:28,835 =====================================\u001b[0m\n", "\u001b[34m2022-04-18 15:53:28,835 Column Target with data uniqueness fraction 6.630860022544923e-05 is classifed as a CATEGORICAL column\u001b[0m\n", "\u001b[34m2022-04-18 15:53:28,838 Column Sex with data uniqueness fraction 6.630860022544923e-05 is classifed as a CATEGORICAL column\u001b[0m\n", "\u001b[34m2022-04-18 15:53:28,840 Column Target with data uniqueness fraction 6.630860022544923e-05 is classifed as a CATEGORICAL column\u001b[0m\n", "\u001b[34m2022-04-18 15:53:29,145 ======================================\u001b[0m\n", "\u001b[34m2022-04-18 15:53:29,145 Calculating bias statistics for report\u001b[0m\n", "\u001b[34m2022-04-18 15:53:29,145 ======================================\u001b[0m\n", "\u001b[34m2022-04-18 15:53:29,146 Column Target with data uniqueness fraction 6.630860022544923e-05 is classifed as a CATEGORICAL column\u001b[0m\n", "\u001b[34m2022-04-18 15:53:29,149 Column Sex with data uniqueness fraction 6.630860022544923e-05 is classifed as a CATEGORICAL column\u001b[0m\n", "\u001b[34m2022-04-18 15:53:29,151 Column Target with data uniqueness fraction 6.630860022544923e-05 is classifed as a CATEGORICAL column\u001b[0m\n", "\u001b[34m2022-04-18 15:53:29,153 Column None with data uniqueness fraction 6.630860022544923e-05 is classifed as a CATEGORICAL column\u001b[0m\n", "\u001b[34m2022-04-18 15:53:29,161 Stop using endpoint: None\u001b[0m\n", "\u001b[34m2022-04-18 15:53:29,430 jupyter nbconvert --to html --output /opt/ml/processing/output/report.html /opt/ml/processing/output/report.ipynb --template sagemaker-xai\u001b[0m\n", "\u001b[34m[NbConvertApp] Converting notebook /opt/ml/processing/output/report.ipynb to html\u001b[0m\n", "\u001b[34m[NbConvertApp] Writing 344696 bytes to /opt/ml/processing/output/report.html\u001b[0m\n", "\u001b[34m2022-04-18 15:53:30,320 HTML report '/opt/ml/processing/output/report.html' generated successfully.\u001b[0m\n", "\u001b[34m2022-04-18 15:53:30,320 wkhtmltopdf -q /opt/ml/processing/output/report.html /opt/ml/processing/output/report.pdf\u001b[0m\n", "\u001b[34m2022-04-18 15:53:30,813 PDF report '/opt/ml/processing/output/report.pdf' generated successfully.\u001b[0m\n", "\u001b[34m2022-04-18 15:53:30,813 Collected analyses: \u001b[0m\n", "\u001b[34m{\n", " \"version\": \"1.0\",\n", " \"post_training_bias_metrics\": {\n", " \"label\": \"Target\",\n", " \"facets\": {\n", " \"Sex\": [\n", " {\n", " \"value_or_threshold\": \"0\",\n", " \"metrics\": [\n", " {\n", " \"name\": \"AD\",\n", " \"description\": \"Accuracy Difference (AD)\",\n", " \"value\": -0.1141572442143538\n", " },\n", " {\n", " \"name\": \"CDDPL\",\n", " \"description\": \"Conditional Demographic Disparity in Predicted Labels (CDDPL)\",\n", " \"value\": 0.19716203919079375\n", " },\n", " {\n", " \"name\": \"DAR\",\n", " \"description\": \"Difference in Acceptance Rates (DAR)\",\n", " \"value\": -0.007405223292617502\n", " },\n", " {\n", " \"name\": \"DCA\",\n", " \"description\": \"Difference in Conditional Acceptance (DCA)\",\n", " \"value\": -0.22750276729134145\n", " },\n", " {\n", " \"name\": \"DCR\",\n", " \"description\": \"Difference in Conditional Rejection (DCR)\",\n", " \"value\": 0.13282504190308553\n", " },\n", " {\n", " \"name\": \"DI\",\n", " \"description\": \"Disparate Impact (DI)\",\n", " \"value\": 0.32939129409419415\n", " },\n", " {\n", " \"name\": \"DPPL\",\n", " \"description\": \"Difference in Positive Proportions in Predicted Labels (DPPL)\",\n", " \"value\": 0.0922004707530946\n", " },\n", " {\n", " \"name\": \"DRR\",\n", " \"description\": \"Difference in Rejection Rates (DRR)\",\n", " \"value\": 0.13653296409568605\n", " },\n", " {\n", " \"name\": \"FT\",\n", " \"description\": \"Flip Test (FT)\",\n", " \"value\": -0.004600286240032713\n", " },\n", " {\n", " \"name\": \"RD\",\n", " \"description\": \"Recall Difference (RD)\",\n", " \"value\": 0.03556460647616988\n", " },\n", " {\n", " \"name\": \"TE\",\n", " \"description\": \"Treatment Equality (TE)\",\n", " \"value\": 24.11428571428572\n", " }\n", " ]\n", " }\n", " ]\n", " },\n", " \"label_value_or_threshold\": \"1\"\n", " },\n", " \"pre_training_bias_metrics\": {\n", " \"label\": \"Target\",\n", " \"facets\": {\n", " \"Sex\": [\n", " {\n", " \"value_or_threshold\": \"0\",\n", " \"metrics\": [\n", " {\n", " \"name\": \"CDDL\",\n", " \"description\": \"Conditional Demographic Disparity in Labels (CDDL)\",\n", " \"value\": 0.214915908649356\n", " },\n", " {\n", " \"name\": \"CI\",\n", " \"description\": \"Class Imbalance (CI)\",\n", " \"value\": 0.3513692725946555\n", " },\n", " {\n", " \"name\": \"DPL\",\n", " \"description\": \"Difference in Positive Proportions in Labels (DPL)\",\n", " \"value\": 0.20015891077100018\n", " },\n", " {\n", " \"name\": \"JS\",\n", " \"description\": \"Jensen-Shannon Divergence (JS)\",\n", " \"value\": 0.030756144659773006\n", " },\n", " {\n", " \"name\": \"KL\",\n", " \"description\": \"Kullback-Liebler Divergence (KL)\",\n", " \"value\": 0.14306865156306434\n", " },\n", " {\n", " \"name\": \"KS\",\n", " \"description\": \"Kolmogorov-Smirnov Distance (KS)\",\n", " \"value\": 0.20015891077100018\n", " },\n", " {\n", " \"name\": \"LP\",\n", " \"description\": \"L-p Norm (LP)\",\n", " \"value\": 0.2830674462421746\n", " },\n", " {\n", " \"name\": \"TVD\",\n", " \"description\": \"Total Variation Distance (TVD)\",\n", " \"value\": 0.20015891077100015\n", " }\n", " ]\n", " }\n", " ]\n", " },\n", " \"label_value_or_threshold\": \"1\"\n", " }\u001b[0m\n", "\u001b[34m}\u001b[0m\n", "\u001b[34m2022-04-18 15:53:30,814 exit_message: Completed: SageMaker XAI Analyzer ran successfully\u001b[0m\n", "\u001b[34m---!\u001b[0m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] } ], "source": [ "clarify_processor.run_bias(\n", " data_config=bias_data_config,\n", " bias_config=bias_config,\n", " model_config=model_config,\n", " model_predicted_label_config=predictions_config,\n", " pre_training_methods=\"all\",\n", " post_training_methods=\"all\",\n", ")" ] }, { "cell_type": "markdown", "id": "f7cda707", "metadata": { "papermill": { "duration": 0.048381, "end_time": "2022-04-18T15:54:02.817876", "exception": false, "start_time": "2022-04-18T15:54:02.769495", "status": "completed" }, "tags": [] }, "source": [ "#### Viewing the Bias Report\n", "In Studio, you can view the results under the experiments tab.\n", "\n", "\n", "\n", "Each bias metric has detailed explanations with examples that you can explore.\n", "\n", "\n", "\n", "You could also summarize the results in a handy table!\n", "\n", "\n" ] }, { "cell_type": "markdown", "id": "3caf52ba", "metadata": { "papermill": { "duration": 0.048103, "end_time": "2022-04-18T15:54:02.915109", "exception": false, "start_time": "2022-04-18T15:54:02.867006", "status": "completed" }, "tags": [] }, "source": [ "If you're not a Studio user yet, you can access the bias report in pdf, html and ipynb formats in the following S3 bucket:" ] }, { "cell_type": "code", "execution_count": 18, "id": "9847314b", "metadata": { "execution": { "iopub.execute_input": "2022-04-18T15:54:03.117134Z", "iopub.status.busy": "2022-04-18T15:54:03.110947Z", "iopub.status.idle": "2022-04-18T15:54:03.124304Z", "shell.execute_reply": "2022-04-18T15:54:03.121910Z" }, "papermill": { "duration": 0.128454, "end_time": "2022-04-18T15:54:03.124461", "exception": false, "start_time": "2022-04-18T15:54:02.996007", "status": "completed" }, "tags": [] }, "outputs": [ { "data": { "text/plain": [ "'s3://sagemaker-us-west-2-000000000000/sagemaker/DEMO-sagemaker-clarify/clarify-bias'" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "bias_report_output_path" ] }, { "cell_type": "markdown", "id": "cd6fa1c1", "metadata": { "papermill": { "duration": 0.0441, "end_time": "2022-04-18T15:54:03.260887", "exception": false, "start_time": "2022-04-18T15:54:03.216787", "status": "completed" }, "tags": [] }, "source": [ "### Explaining Predictions\n", "There are expanding business needs and legislative regulations that require explanations of _why_ a model made the decision it did. SageMaker Clarify uses SHAP to explain the contribution that each input feature makes to the final decision." ] }, { "cell_type": "markdown", "id": "3596d952", "metadata": { "papermill": { "duration": 0.042869, "end_time": "2022-04-18T15:54:03.346879", "exception": false, "start_time": "2022-04-18T15:54:03.304010", "status": "completed" }, "tags": [] }, "source": [ "Kernel SHAP algorithm requires a baseline (also known as background dataset). If not provided, a baseline is calculated automatically by SageMaker Clarify using K-means or K-prototypes in the input dataset. Baseline dataset type shall be the same as `dataset_type` of `DataConfig`, and baseline samples shall only include features. By definition, `baseline` should either be a S3 URI to the baseline dataset file, or an in-place list of samples. In this case we chose the latter, and put the first sample of the test dataset to the list." ] }, { "cell_type": "code", "execution_count": 19, "id": "8efafe15", "metadata": { "execution": { "iopub.execute_input": "2022-04-18T15:54:03.444618Z", "iopub.status.busy": "2022-04-18T15:54:03.443803Z", "iopub.status.idle": "2022-04-18T15:54:03.446753Z", "shell.execute_reply": "2022-04-18T15:54:03.446309Z" }, "papermill": { "duration": 0.052082, "end_time": "2022-04-18T15:54:03.446918", "exception": false, "start_time": "2022-04-18T15:54:03.394836", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "shap_config = clarify.SHAPConfig(\n", " baseline=[test_features.iloc[0].values.tolist()],\n", " num_samples=15,\n", " agg_method=\"mean_abs\",\n", " save_local_shap_values=True,\n", ")\n", "\n", "explainability_output_path = \"s3://{}/{}/clarify-explainability\".format(bucket, prefix)\n", "explainability_data_config = clarify.DataConfig(\n", " s3_data_input_path=train_uri,\n", " s3_output_path=explainability_output_path,\n", " label=\"Target\",\n", " headers=training_data.columns.to_list(),\n", " dataset_type=\"text/csv\",\n", ")" ] }, { "cell_type": "code", "execution_count": 20, "id": "2a001d69", "metadata": { "execution": { "iopub.execute_input": "2022-04-18T15:54:03.545718Z", "iopub.status.busy": "2022-04-18T15:54:03.537884Z", "iopub.status.idle": "2022-04-18T16:07:30.752399Z", "shell.execute_reply": "2022-04-18T16:07:30.750754Z" }, "papermill": { "duration": 807.262471, "end_time": "2022-04-18T16:07:30.752546", "exception": false, "start_time": "2022-04-18T15:54:03.490075", "status": "completed" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Job Name: Clarify-Explainability-2022-04-18-15-54-03-536\n", "Inputs: [{'InputName': 'dataset', 'AppManaged': False, 'S3Input': {'S3Uri': 's3://sagemaker-us-west-2-000000000000/sagemaker/DEMO-sagemaker-clarify/train_data.csv', 'LocalPath': '/opt/ml/processing/input/data', 'S3DataType': 'S3Prefix', 'S3InputMode': 'File', 'S3DataDistributionType': 'FullyReplicated', 'S3CompressionType': 'None'}}, {'InputName': 'analysis_config', 'AppManaged': False, 'S3Input': {'S3Uri': 's3://sagemaker-us-west-2-000000000000/sagemaker/DEMO-sagemaker-clarify/clarify-explainability/analysis_config.json', 'LocalPath': '/opt/ml/processing/input/config', 'S3DataType': 'S3Prefix', 'S3InputMode': 'File', 'S3DataDistributionType': 'FullyReplicated', 'S3CompressionType': 'None'}}]\n", "Outputs: [{'OutputName': 'analysis_result', 'AppManaged': False, 'S3Output': {'S3Uri': 's3://sagemaker-us-west-2-000000000000/sagemaker/DEMO-sagemaker-clarify/clarify-explainability', 'LocalPath': '/opt/ml/processing/output', 'S3UploadMode': 'EndOfJob'}}]\n", ".............................\u001b[34m2022-04-18 15:58:37,472 logging.conf not found when configuring logging, using default logging configuration.\u001b[0m\n", "\u001b[34m2022-04-18 15:58:37,473 Starting SageMaker Clarify Processing job\u001b[0m\n", "\u001b[34m2022-04-18 15:58:37,473 Analysis config path: /opt/ml/processing/input/config/analysis_config.json\u001b[0m\n", "\u001b[34m2022-04-18 15:58:37,473 Analysis result path: /opt/ml/processing/output\u001b[0m\n", "\u001b[34m2022-04-18 15:58:37,473 This host is algo-1.\u001b[0m\n", "\u001b[34m2022-04-18 15:58:37,473 This host is the leader.\u001b[0m\n", "\u001b[34m2022-04-18 15:58:37,473 Number of hosts in the cluster is 1.\u001b[0m\n", "\u001b[34m2022-04-18 15:58:37,640 Running Python / Pandas based analyzer.\u001b[0m\n", "\u001b[34m2022-04-18 15:58:37,640 Dataset type: text/csv uri: /opt/ml/processing/input/data\u001b[0m\n", "\u001b[34m2022-04-18 15:58:37,651 Loading dataset...\u001b[0m\n", "\u001b[34m2022-04-18 15:58:37,687 Loaded dataset. Dataset info:\u001b[0m\n", "\u001b[34m\u001b[0m\n", "\u001b[34mRangeIndex: 30162 entries, 0 to 30161\u001b[0m\n", "\u001b[34mData columns (total 14 columns):\n", " # Column Non-Null Count Dtype\u001b[0m\n", "\u001b[34m--- ------ -------------- -----\n", " 0 Age 30162 non-null int64\n", " 1 Workclass 30162 non-null int64\n", " 2 fnlwgt 30162 non-null int64\n", " 3 Education 30162 non-null int64\n", " 4 Education-Num 30162 non-null int64\n", " 5 Marital Status 30162 non-null int64\n", " 6 Occupation 30162 non-null int64\n", " 7 Relationship 30162 non-null int64\n", " 8 Ethnic group 30162 non-null int64\n", " 9 Sex 30162 non-null int64\n", " 10 Capital Gain 30162 non-null int64\n", " 11 Capital Loss 30162 non-null int64\n", " 12 Hours per week 30162 non-null int64\n", " 13 Country 30162 non-null int64\u001b[0m\n", "\u001b[34mdtypes: int64(14)\u001b[0m\n", "\u001b[34mmemory usage: 3.2 MB\u001b[0m\n", "\u001b[34m2022-04-18 15:58:37,835 Spinning up shadow endpoint\u001b[0m\n", "\u001b[34m2022-04-18 15:58:37,835 Creating endpoint-config with name sm-clarify-config-1650297517-31f2\u001b[0m\n", "\u001b[34m2022-04-18 15:58:37,915 Creating endpoint: 'sm-clarify-DEMO-clarify-model-18-04-2022-15-45--1650297517-f947'\u001b[0m\n", "\u001b[34m2022-04-18 15:58:38,339 Using endpoint name: sm-clarify-DEMO-clarify-model-18-04-2022-15-45--1650297517-f947\u001b[0m\n", "\u001b[34m2022-04-18 15:58:38,339 Waiting for endpoint ...\u001b[0m\n", "\u001b[34m2022-04-18 15:58:38,339 Checking endpoint status:\u001b[0m\n", "\u001b[34mLegend:\u001b[0m\n", "\u001b[34m(OutOfService: x, Creating: -, Updating: -, InService: !, RollingBack: <, Deleting: o, Failed: *)\u001b[0m\n", "\u001b[34m2022-04-18 16:01:38,784 Endpoint is in service after 180 seconds\u001b[0m\n", "\u001b[34m2022-04-18 16:01:38,785 Endpoint ready.\u001b[0m\n", "\u001b[34m2022-04-18 16:01:38,786 SHAP n_samples 15\u001b[0m\n", "\u001b[34m2022-04-18 16:01:38,904 =====================================================\u001b[0m\n", "\u001b[34m2022-04-18 16:01:38,904 Shap analyzer: explaining 30162 rows, 14 columns...\u001b[0m\n", "\u001b[34m2022-04-18 16:01:38,904 =====================================================\u001b[0m\n", "\u001b[34m 0% (0 of 30162) | | Elapsed Time: 0:00:00 ETA: --:--:--\u001b[0m\n", "\u001b[34m 9% (2798 of 30162) |# | Elapsed Time: 0:00:30 ETA: 0:04:53\u001b[0m\n", "\u001b[34m 18% (5681 of 30162) |### | Elapsed Time: 0:01:00 ETA: 0:04:14\u001b[0m\n", "\u001b[34m 28% (8703 of 30162) |##### | Elapsed Time: 0:01:30 ETA: 0:03:33\u001b[0m\n", "\u001b[34m 39% (11989 of 30162) |####### | Elapsed Time: 0:02:00 ETA: 0:02:45\u001b[0m\n", "\u001b[34m 51% (15453 of 30162) |######### | Elapsed Time: 0:02:30 ETA: 0:02:07\u001b[0m\n", "\u001b[34m 62% (18999 of 30162) |########### | Elapsed Time: 0:03:00 ETA: 0:01:34\u001b[0m\n", "\u001b[34m 74% (22510 of 30162) |############# | Elapsed Time: 0:03:30 ETA: 0:01:05\u001b[0m\n", "\u001b[34m 86% (25999 of 30162) |############### | Elapsed Time: 0:04:00 ETA: 0:00:35\u001b[0m\n", "\u001b[34m 97% (29442 of 30162) |################# | Elapsed Time: 0:04:30 ETA: 0:00:06\u001b[0m\n", "\u001b[34m100% (30162 of 30162) |##################| Elapsed Time: 0:04:36 Time: 0:04:36\u001b[0m\n", "\u001b[34m2022-04-18 16:06:15,695 getting explanations took 276.79 seconds.\u001b[0m\n", "\u001b[34m2022-04-18 16:06:15,695 ===================================================\u001b[0m\n", "\u001b[34m2022-04-18 16:06:18,084 converting explanations to tabular took 2.39 seconds.\u001b[0m\n", "\u001b[34m2022-04-18 16:06:18,084 ===================================================\u001b[0m\n", "\u001b[34m2022-04-18 16:06:18,088 Wrote baseline used to compute explanations to: /opt/ml/processing/output/explanations_shap/baseline.csv\u001b[0m\n", "\u001b[34m2022-04-18 16:06:18,731 Wrote 30162 local explanations to: /opt/ml/processing/output/explanations_shap/out.csv\u001b[0m\n", "\u001b[34m2022-04-18 16:06:18,731 writing local explanations took 0.65 seconds.\u001b[0m\n", "\u001b[34m2022-04-18 16:06:18,731 ===================================================\u001b[0m\n", "\u001b[34m2022-04-18 16:06:18,735 aggregating local explanations took 0.00 seconds.\u001b[0m\n", "\u001b[34m2022-04-18 16:06:18,735 ===================================================\u001b[0m\n", "\u001b[34m2022-04-18 16:06:18,736 Shap analysis finished.\u001b[0m\n", "\u001b[34m2022-04-18 16:06:18,736 Stop using endpoint: sm-clarify-DEMO-clarify-model-18-04-2022-15-45--1650297517-f947\u001b[0m\n", "\u001b[34m2022-04-18 16:06:18,736 Deleting endpoint configuration with name: sm-clarify-config-1650297517-31f2\u001b[0m\n", "\u001b[34m2022-04-18 16:06:18,853 Deleting endpoint with name: sm-clarify-DEMO-clarify-model-18-04-2022-15-45--1650297517-f947\u001b[0m\n", "\u001b[34m2022-04-18 16:06:18,997 Model endpoint delivered 107.74733 requests per second and a total of 30164 requests over 280 seconds\u001b[0m\n", "\u001b[34m2022-04-18 16:06:26,813 Stop using endpoint: None\u001b[0m\n", "\u001b[34m2022-04-18 16:06:50,264 jupyter nbconvert --to html --output /opt/ml/processing/output/report.html /opt/ml/processing/output/report.ipynb --template sagemaker-xai\u001b[0m\n", "\u001b[34m[NbConvertApp] Converting notebook /opt/ml/processing/output/report.ipynb to html\u001b[0m\n", "\u001b[34m[NbConvertApp] Writing 570437 bytes to /opt/ml/processing/output/report.html\u001b[0m\n", "\u001b[34m2022-04-18 16:06:51,317 HTML report '/opt/ml/processing/output/report.html' generated successfully.\u001b[0m\n", "\u001b[34m2022-04-18 16:06:51,317 wkhtmltopdf -q /opt/ml/processing/output/report.html /opt/ml/processing/output/report.pdf\u001b[0m\n", "\u001b[34m2022-04-18 16:06:51,874 PDF report '/opt/ml/processing/output/report.pdf' generated successfully.\u001b[0m\n", "\u001b[34m2022-04-18 16:06:51,875 Collected analyses: \u001b[0m\n", "\u001b[34m{\n", " \"version\": \"1.0\",\n", " \"explanations\": {\n", " \"kernel_shap\": {\n", " \"label0\": {\n", " \"global_shap_values\": {\n", " \"Age\": 0.03655626472022009,\n", " \"Workclass\": 0.017905832546722414,\n", " \"fnlwgt\": 0.021385894167534045,\n", " \"Education\": 0.018535316056790388,\n", " \"Education-Num\": 0.03609330944536093,\n", " \"Marital Status\": 0.02898470399620499,\n", " \"Occupation\": 0.026482349644306062,\n", " \"Relationship\": 0.03615980532944972,\n", " \"Ethnic group\": 0.020033663785676746,\n", " \"Sex\": 0.017880631469685705,\n", " \"Capital Gain\": 0.033581907850084025,\n", " \"Capital Loss\": 0.019556674983842386,\n", " \"Hours per week\": 0.021283579738450336,\n", " \"Country\": 0.04712774225543154\n", " },\n", " \"expected_value\": 0.0006380207487381995\n", " }\n", " }\n", " }\u001b[0m\n", "\u001b[34m}\u001b[0m\n", "\u001b[34m2022-04-18 16:06:51,875 exit_message: Completed: SageMaker XAI Analyzer ran successfully\u001b[0m\n", "\u001b[34m---!\u001b[0m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] } ], "source": [ "clarify_processor.run_explainability(\n", " data_config=explainability_data_config,\n", " model_config=model_config,\n", " explainability_config=shap_config,\n", ")" ] }, { "cell_type": "markdown", "id": "33d4727d", "metadata": { "papermill": { "duration": 0.066023, "end_time": "2022-04-18T16:07:30.893004", "exception": false, "start_time": "2022-04-18T16:07:30.826981", "status": "completed" }, "tags": [] }, "source": [ "#### Viewing the Explainability Report\n", "As with the bias report, you can view the explainability report in Studio under the experiments tab.\n", "\n", "\n", "\n", "\n", "The Model Insights tab contains direct links to the report and model insights.\n", "\n", "If you're not a Studio user yet, as with the Bias Report, you can access this report at the following S3 bucket." ] }, { "cell_type": "code", "execution_count": 21, "id": "5e571398", "metadata": { "execution": { "iopub.execute_input": "2022-04-18T16:07:31.028316Z", "iopub.status.busy": "2022-04-18T16:07:31.027352Z", "iopub.status.idle": "2022-04-18T16:07:31.030696Z", "shell.execute_reply": "2022-04-18T16:07:31.031231Z" }, "papermill": { "duration": 0.077024, "end_time": "2022-04-18T16:07:31.031371", "exception": false, "start_time": "2022-04-18T16:07:30.954347", "status": "completed" }, "tags": [] }, "outputs": [ { "data": { "text/plain": [ "'s3://sagemaker-us-west-2-000000000000/sagemaker/DEMO-sagemaker-clarify/clarify-explainability'" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "explainability_output_path" ] }, { "cell_type": "markdown", "id": "57a8f107", "metadata": { "papermill": { "duration": 0.061215, "end_time": "2022-04-18T16:07:31.154052", "exception": false, "start_time": "2022-04-18T16:07:31.092837", "status": "completed" }, "tags": [] }, "source": [ "#### Analysis of local explanations\n", "It is possible to visualize the the local explanations for single examples in your dataset. You can use the obtained results from running Kernel SHAP algorithm for global explanations.\n", "\n", "You can simply load the local explanations stored in your output path, and visualize the explanation (i.e., the impact that the single features have on the prediction of your model) for any single example." ] }, { "cell_type": "code", "execution_count": 22, "id": "90a09a11", "metadata": { "execution": { "iopub.execute_input": "2022-04-18T16:07:31.293968Z", "iopub.status.busy": "2022-04-18T16:07:31.292566Z", "iopub.status.idle": "2022-04-18T16:07:31.821690Z", "shell.execute_reply": "2022-04-18T16:07:31.821217Z" }, "papermill": { "duration": 0.598354, "end_time": "2022-04-18T16:07:31.821806", "exception": false, "start_time": "2022-04-18T16:07:31.223452", "status": "completed" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Example number: 111 \n", "with model prediction: False\n", "\n", "Feature values -- Label Target 0\n", "Age 21\n", "Workclass 2\n", "fnlwgt 199915\n", "Education 15\n", "Education-Num 10\n", "Marital Status 4\n", "Occupation 7\n", "Relationship 3\n", "Ethnic group 4\n", "Sex 0\n", "Capital Gain 0\n", "Capital Loss 0\n", "Hours per week 40\n", "Country 38\n", "Name: 120, dtype: int64\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY8AAAFMCAYAAAAz5jD9AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA8O0lEQVR4nO3dd7wcZb3H8c83CR1Dv4i0IEUMCgqRJvcCghCFK14FpYpK8YoFxQbXKyCIguViRUVAaYIgKlHpHRGEBNBQRCKggJTQkSaB3/3jeZbMWXbPOXPm2VOS7/v1Oq/szO78ZnazO7+ZpyoiMDMzq2PcSB+AmZmNPU4eZmZWm5OHmZnV5uRhZma1OXmYmVltTh5mZlabk8cYJmkLSfcM8z4nSQpJE4Zzv3nfu0m6oEexPyzpAUn/lLRMD+KP2Oc2EsbK+5V0qKRTRvo4xiInj8Ik3SVp65E+jrGu08knIk6NiG16sK8FgP8DtomIxSPi4QIx/T0wJG0p6VJJj0u6q8Pzh0uaKWmOpEPbnltB0jRJ/8i/hUnDdNiD4uRhBssDCwM3191QiX9HRpe7rKeAE4DPdNlsFvBZ4LcdnnsROA94d5EDLMxf+mEiaSFJ38xXEf/IjxeqPL+DpBslPSHpr5Km5vUfkHSrpCcl3SHpQzX2ubakCyU9Iuk2Se/J61fP69bPy6+SNFvSFnn5MklfkXRtPp6zJS3dZR9dj69VrCbpU5IelHSfpA9Unt9O0g15H3e3XXldkf99LBclbSLp/ZJ+V9l+U0nX5au66yRtWnnusnxVd1U+tgskLdvh+NcCbqvs65JBxj5C0lXA08Cr22KeDKwC/Dof+2crT+8m6e+SHpL0+co24yQdmP/vH5Z0RrfPPL9++/x9eUzS7yWtm9e/V9Kdkibm5bdJul/Scnn5W/mzfkLSDEn/Xol5qKQzJZ2SP7OZktaSdFD+/7tb0jaV19f5niwh6fj8HbhX0pckje/y2kPz+z8pH8fNkqZUng9Ja1SWfyLpS/lx6zv32cp37p2S3i7pL/l7/z9tu1xY0s/yvq6XtF4l9qsknaX0+7hT0sfbjvPn+fN6Anh/+3uJiGsj4mTgjk7vNSJOjIhzgSc7PPdARBwDXNdp2xEXEf4r+AfcBWzdYf1hwDXAvwHLAb8HDs/PbQg8DryVlNBXBNbOz20HrA4I2Jx0slo/P7cFcE+X41gMuBv4ADABeCPwEDA5P78PcAuwKHA+8PXKtpcB9wKvy3HOAk7Jz00CApgwyOObk9/7AsDb8/NLVZ5/fX7P6wIPAO/stJ+87v3A7/LjpYFHgT3y+9slLy9TeQ9/BdYCFsnLR3b5rNrf02Bi/x1YJz+/wEDfg8o+fpSPZz3gOeC1+fn9Sd+PlYCFgB8Cp3U53jcCDwIbAeOBPfP+FsrPnwr8BFgG+AewfWXb3fP6CcCngPuBhfNzhwLPAtvm508C7gQ+n///9gHuHOL35Jf5PS1G+g1cC3yoy/trHcfb8/v7CnBN5fkA1qgs/wT4Utt37uDKMc8Gfgq8Iv+fPQOsVtnX88CO+fWfzu95AdL3ckaOtSDpIuEOYNu2bd+ZX7tIP+eFrYG7+nn+FODQLs9NyO950kif3/oc10gfwLz2R/fk8Vfg7ZXlbVtfpvyjOnqQ8X8F7J8fb0H35PFe4Mq2dT8EDqksTwNmAn8in3jy+suonGiBycC/8g+5z0lhEMf3DH0TwIPAxl22/Wbrc+i0H/omjz2Aa9u2vxp4f+U9/G/luf2A87rst8++Bhn7sDrfg8o+VqqsuxbYOT++Fdiq8twKpBPTyz5n4PvkC4/KutuAzfPjJUnJbSbwwwGO81Fgvfz4UODCynP/CfwTGJ+XX5Hfw5J1viekYsHnqJxcSQn50i7HdChwUVvcZyrLAyWPZzoc80aV189g7kXKofRNTOOA+4B/JyXnv7cd20HAjyvbXjHI3+08lzxGdUuIecyrgL9Vlv+W1wGsDJzTaSNJbwMOIV1BjyPdKcwcxP5WBTaS9Fhl3QTg5Mryj0gJZN+IeK5t+7vbjnUBoFOxz0DH93BEzKksPw0snrfdCDiSdOW6IOmK+8xBvDd4+efZOs4VK8v3d9pvodh3MzTdjmlV4JeSXqw8/wLpxHtvW4xVgT0lfayybsF83ETEY5LOBA6grbxc0qeBvfJrA5hI3//XByqPnwEeiogXKsvkY34sPx7M92TVvP4+Sa114+j/M2z/nBaWNKHtu9TNwx2Ouf19Vb8LLx1HRLyo1IKx9fm8qu03NB64stO28xvXeQyff5B+RC2r5HWQvoCrt2+gVCdyFvB1YPmIWJKUZNT+2g7uBi6PiCUrf4tHxIdz7MVJV/rHA4d2KKteue1YnycVe5U6PkhFCdOAlSNiCeAHlW1jgG3bP8/WcbafaIdiMLEHOr6Bnm93N/C2tv+vhSOi0/u5Gzii7bWLRsRpAJLeAHwQOA34dmujXL/xWeA9pKLDJUnFpYP9/+pkwO9JPt7ngGUrxzsxItYZ4j6fJl2ktLxyiHFaXnoPSo0fViJ9B+4mFdNVP+dXRMTbK9vW/X+eZzh59MYCkhau/E0g/ZD/V9JyueL2YNKtKqQT+AckbZUrTleUtDZzr8ZnA3PyVf5gm6r+BlhL0h6SFsh/b5L02vz8t4DpEbE3qaXHD9q2313SZEmLkuosfl65mmtpcnyQihQeiYhnJW0I7Fp5bjaptcmrO26ZktRaknaVNEHSe0nFG7+psf9uSsR+gO7H3skPgCMkrQqQvyc7dHntj4D/lrSRksWUGh+8QtLCpO/V/5Dqu1aUtF/e7hWk+oDZwARJB5PuPJoY8HsSEfcBFwDfkDQxf8dXl7T5EPd5I7CrpPFKDUuGGqdlA0nvyr/TT5AS3TWkYsUnJX1O0iJ5f6+T9KbBBs7vdWHSnZfy+WDByvML5OfHkf5PFq42JMjPtRrWLJSXRwUnj944h3Rr3Po7FPgSMJ1UvzATuD6vIyKuJf3QjyZdCV4OrBoRTwIfB84glU3vSrpSH1DedhtgZ9JV1P3AUaQv4A7AVODD+eUHAOtL2q0S4mRSWfL9pGasH6dNk+PL9gMOk/QkKZmeUYn9NHAEcJVSi6KN2/b9MLA9qdL3YdIV9fYR0X7VW1uh2F8hXSw8louKBvIt0md3Qf48riGVuXc6vumkiuDvkj73Wcxt6fMV4O6I+H4uitwd+JKkNUkNI84D/kIqYnqW5sUuA35PsveRLjZuycf8c1K9zlDsT6qPeQzYjVTP1sTZpDrCViOJd0XE8zkJbg+8gVSJ/hBwHLBEjdj/QToHnEO6M3uGlEhbfpTX7UJqmPBMPoaWZ0j1TgB/Zm4x3IhTrpAxe4mky0itZo4b6WOx0cvfk/mb7zzMzKw2Jw8zM6vNxVZmZlab7zzMzKy2+aKT4LLLLhuTJk0a6cMwMxtTZsyY8VBELNfpufkieUyaNInp06eP9GGYmY0pktpHWniJi63MzKw2Jw8zM6vNycPMzGpz8jAzs9qcPMzMrDYnDzMzq61I8pA0VWmO7FmSDuzw/EJ5juBZkv4gaVLluYPy+tskbTtQzDwE9RFK8xHfWp1T2MzMhkfjfh557Pnvkebfvge4TtK0iLil8rK9gEcjYg1JO5OGBn+vpMmkIcPXIc3cdZGktfI23WK+nzR5y9p51q9/a/oezMysnhKdBDcEZkXEHQCSTgd2II3b37IDaU4LSOP4f1dpPsodgNPzvAN3SpqV49FPzA8Du0bEiwAR8WCB92BmNs+YdOBva73+riO3q72PEsVWK9J3Qpl76DvXc5/X5DmIHweW6Wfb/mKuTrprmS7p3DzJzctI2je/Zvrs2bOH9MbMzKyzsVhhvhDwbERMIc3CdUKnF0XEsRExJSKmLLdcx6FZzMxsiEokj3upTCBPmjz+3m6vyfMEL0Ga3rPbtv3FvAf4RX78S2Ddxu/AzMxqKZE8rgPWlLRanth9Z14+j/U0YM/8eEfgkkgTiUwDds6tsVYD1iRNOt9fzF8BW+bHm5PmYzYzs2HUuMI8IuZI+ihwPjAeOCEibpZ0GDA9IqYBxwMn5wrxR0jJgPy6M0gV4XOAj+RJ5+kUM+/ySOBUSZ8kTQy/d9P3YGZm9RQZkj0izgHOaVt3cOXxs8BOXbY9AjhiMDHz+seA+k0DzMysmLFYYW5mZiPMycPMzGpz8jAzs9qcPMzMrDYnDzMzq83Jw8zManPyMDOz2pw8zMysNicPMzOrzcnDzMxqc/IwM7PanDzMzKw2Jw8zM6vNycPMzGpz8jAzs9qcPMzMrDYnDzMzq83Jw8zManPyMDOz2pw8zMysNicPMzOrzcnDzMxqc/IwM7PanDzMzKw2Jw8zM6utSPKQNFXSbZJmSTqww/MLSfpZfv4PkiZVnjsor79N0rY1Yn5b0j9LHL+ZmdXTOHlIGg98D3gbMBnYRdLktpftBTwaEWsARwNH5W0nAzsD6wBTgWMkjR8opqQpwFJNj93MzIamxJ3HhsCsiLgjIv4FnA7s0PaaHYAT8+OfA1tJUl5/ekQ8FxF3ArNyvK4xc2L5GvDZAsduZmZDUCJ5rAjcXVm+J6/r+JqImAM8DizTz7b9xfwoMC0i7uvvoCTtK2m6pOmzZ8+u9YbMzKx/Y6rCXNKrgJ2A7wz02og4NiKmRMSU5ZZbrvcHZ2Y2HymRPO4FVq4sr5TXdXyNpAnAEsDD/Wzbbf0bgTWAWZLuAhaVNKvAezAzsxpKJI/rgDUlrSZpQVIF+LS210wD9syPdwQuiYjI63fOrbFWA9YEru0WMyJ+GxGvjIhJETEJeDpXwpuZ2TCa0DRARMyR9FHgfGA8cEJE3CzpMGB6REwDjgdOzncJj5CSAfl1ZwC3AHOAj0TECwCdYjY9VjMzK6Nx8gCIiHOAc9rWHVx5/CyprqLTtkcARwwmZofXLD6U4zUzs2bGVIW5mZmNDk4eZmZWm5OHmZnV5uRhZma1OXmYmVltTh5mZlabk4eZmdXm5GFmZrU5eZiZWW1OHmZmVpuTh5mZ1ebkYWZmtTl5mJlZbU4eZmZWm5OHmZnV5uRhZma1OXmYmVltTh5mZlabk4eZmdVWZA5zMxtbJh3421qvv+vI7Xp0JDZW+c7DzMxqc/IwM7PanDzMzKw2Jw8zM6utSPKQNFXSbZJmSTqww/MLSfpZfv4PkiZVnjsor79N0rYDxZR0al5/k6QTJC1Q4j2YmdngNU4eksYD3wPeBkwGdpE0ue1lewGPRsQawNHAUXnbycDOwDrAVOAYSeMHiHkqsDbwemARYO+m78HMzOopceexITArIu6IiH8BpwM7tL1mB+DE/PjnwFaSlNefHhHPRcSdwKwcr2vMiDgnMuBaYKUC78HMzGookTxWBO6uLN+T13V8TUTMAR4Hluln2wFj5uKqPYDzOh2UpH0lTZc0ffbs2TXfkpmZ9WcsdxI8BrgiIq7s9GREHAscCzBlypQYzgMbLr3u6DXW45tZ75RIHvcCK1eWV8rrOr3mHkkTgCWAhwfYtmtMSYcAywEfKnD8ZmZWU4liq+uANSWtJmlBUgX4tLbXTAP2zI93BC7JdRbTgJ1za6zVgDVJ9RhdY0raG9gW2CUiXixw/GZmVlPjO4+ImCPpo8D5wHjghIi4WdJhwPSImAYcD5wsaRbwCCkZkF93BnALMAf4SES8ANApZt7lD4C/AVenOnd+ERGHNX0fZmY2eEXqPCLiHOCctnUHVx4/C+zUZdsjgCMGEzOvH8v1NGZm8wT3MDczs9qcPMzMrDYnDzMzq83Jw8zManPyMDOz2pw8zMysNicPMzOrzcnDzMxqc/IwM7PanDzMzKw2Jw8zM6vNycPMzGpz8jAzs9qcPMzMrDYPb27zLE9zO+/y/+3I852HmZnV5uRhZma1OXmYmVltTh5mZlabk4eZmdXm5GFmZrU5eZiZWW1OHmZmVpuTh5mZ1ebkYWZmtRVJHpKmSrpN0ixJB3Z4fiFJP8vP/0HSpMpzB+X1t0nadqCYklbLMWblmAuWeA9mZjZ4jZOHpPHA94C3AZOBXSRNbnvZXsCjEbEGcDRwVN52MrAzsA4wFThG0vgBYh4FHJ1jPZpjm5nZMCpx57EhMCsi7oiIfwGnAzu0vWYH4MT8+OfAVpKU158eEc9FxJ3ArByvY8y8zVtyDHLMdxZ4D2ZmVkOJUXVXBO6uLN8DbNTtNRExR9LjwDJ5/TVt266YH3eKuQzwWETM6fD6PiTtC+wLsMoqq3Q88F6PzNnr+L0eKdTx+zeWvz9j/bMZ68ffa8NxPPNshXlEHBsRUyJiynLLLTfSh2NmNk8pkTzuBVauLK+U13V8jaQJwBLAw/1s2239w8CSOUa3fZmZWY+VSB7XAWvmVlALkirAp7W9ZhqwZ368I3BJRERev3NujbUasCZwbbeYeZtLcwxyzLMLvAczM6uhcZ1HrsP4KHA+MB44ISJulnQYMD0ipgHHAydLmgU8QkoG5NedAdwCzAE+EhEvAHSKmXf5OeB0SV8CbsixzcxsGBWZhjYizgHOaVt3cOXxs8BOXbY9AjhiMDHz+jtIrbHMzGyEzLMV5mZm1jtOHmZmVpuTh5mZ1ebkYWZmtRWpMDez8kZbr2WzKt95mJlZbU4eZmZWm5OHmZnV5uRhZma1OXmYmVltTh5mZlabk4eZmdXm5GFmZrW5k6DZELkTn83PfOdhZma1OXmYmVltTh5mZlabk4eZmdXm5GFmZrU5eZiZWW1OHmZmVpuTh5mZ1ebkYWZmtTl5mJlZbY2Sh6SlJV0o6fb871JdXrdnfs3tkvasrN9A0kxJsyR9W5L6iytpN0l/ytv8XtJ6TY7fzMyGpumdx4HAxRGxJnBxXu5D0tLAIcBGwIbAIZUk831gH2DN/Dd1gLh3AptHxOuBw4FjGx6/mZkNQdPksQNwYn58IvDODq/ZFrgwIh6JiEeBC4GpklYAJkbENRERwEmV7TvGjYjf5xgA1wArNTx+MzMbgqbJY/mIuC8/vh9YvsNrVgTurizfk9etmB+3rx9s3L2Ac7sdmKR9JU2XNH327NkDvhEzMxu8AYdkl3QR8MoOT32+uhARISlKHVh/cSVtSUoem/Wz3bHkYq0pU6YUPy4zs/nZgMkjIrbu9pykByStEBH35WKoBzu87F5gi8rySsBlef1KbevvzY+7xpW0LnAc8LaIeHig4zczs/KaFltNA1qtp/YEzu7wmvOBbSQtlSvKtwHOz8VST0jaOLeyel9l+45xJa0C/ALYIyL+0vDYzcxsiJomjyOBt0q6Hdg6LyNpiqTjACLiEVLLqOvy32F5HcB+pLuIWcBfmVuH0TEucDCwDHCMpBslTW94/GZmNgSNpqHNxUZbdVg/Hdi7snwCcEKX172uRty9q3HNzGxkeA5zMyvO87vP+zw8iZmZ1ebkYWZmtTl5mJlZbU4eZmZWm5OHmZnV5uRhZma1ualuD7m5opnNq3znYWZmtTl5mJlZbU4eZmZWm5OHmZnV5uRhZma1OXmYmVltTh5mZlabk4eZmdXm5GFmZrU5eZiZWW1OHmZmVpuTh5mZ1ebkYWZmtTl5mJlZbU4eZmZWm5OHmZnV5uRhZma1NUoekpaWdKGk2/O/S3V53Z75NbdL2rOyfgNJMyXNkvRtSRpMXElvkjRH0o5Njt/MzIam6Z3HgcDFEbEmcHFe7kPS0sAhwEbAhsAhlWTwfWAfYM38N3WguJLGA0cBFzQ8djMzG6KmyWMH4MT8+ETgnR1esy1wYUQ8EhGPAhcCUyWtAEyMiGsiIoCTKtv3F/djwFnAgw2P3czMhqhp8lg+Iu7Lj+8Hlu/wmhWBuyvL9+R1K+bH7eu7xpW0IvBfpDuWfknaV9J0SdNnz549yLdjZmaDMWGgF0i6CHhlh6c+X12IiJAUpQ6sS9xvAp+LiBdz9Uh/2x0LHAswZcqU4sdlZjY/GzB5RMTW3Z6T9ICkFSLivlwM1ako6V5gi8rySsBlef1KbevvzY+7xZ0CnJ4Tx7LA2yXNiYhfDfQ+zMysnKbFVtOAVuupPYGzO7zmfGAbSUvlivJtgPNzsdQTkjbOrazeV9m+Y9yIWC0iJkXEJODnwH5OHGZmw69p8jgSeKuk24Gt8zKSpkg6DiAiHgEOB67Lf4fldQD7AccBs4C/Auf2F9fMzEYHpYZO87YpU6bE9OnTX7Z+0oG/rRXnriO3K3VIZmajnqQZETGl03PuYW5mZrU5eZiZWW1OHmZmVpuTh5mZ1ebkYWZmtTl5mJlZbU4eZmZWm5OHmZnV5uRhZma1OXmYmVltTh5mZlabk4eZmdXm5GFmZrU5eZiZWW1OHmZmVpuTh5mZ1ebkYWZmtTl5mJlZbU4eZmZWm5OHmZnV5uRhZma1OXmYmVltTh5mZlbbhJE+gJF015HbjfQhmJmNSY3uPCQtLelCSbfnf5fq8ro982tul7RnZf0GkmZKmiXp25I0UFxJW0i6UdLNki5vcvxmZjY0TYutDgQujog1gYvzch+SlgYOATYCNgQOqSSD7wP7AGvmv6n9xZW0JHAM8I6IWAfYqeHxm5nZEDRNHjsAJ+bHJwLv7PCabYELI+KRiHgUuBCYKmkFYGJEXBMRAZxU2b5b3F2BX0TE3wEi4sGGx29mZkPQNHksHxH35cf3A8t3eM2KwN2V5XvyuhXz4/b1/cVdC1hK0mWSZkh6X7cDk7SvpOmSps+ePbvWmzIzs/4NWGEu6SLglR2e+nx1ISJCUpQ6sC5xJwAbAFsBiwBXS7omIv7SYbtjgWMBpkyZUvy4zMzmZwMmj4jYuttzkh6QtEJE3JeLoToVI90LbFFZXgm4LK9fqW39vflxt7j3AA9HxFPAU5KuANYDXpY8zMysd5oWW00DWq2n9gTO7vCa84FtJC2VK8q3Ac7PxVJPSNo4t7J6X2X7bnHPBjaTNEHSoqRK+FsbvgczM6upafI4EnirpNuBrfMykqZIOg4gIh4BDgeuy3+H5XUA+wHHAbOAvwLn9hc3Im4FzgP+BFwLHBcRNzV8D2ZmVpNSQ6d525QpU2L69OkjfRhmZmOKpBkRMaXjc/ND8pA0G/hbjU2WBR7q0eE4vuOP5vhj+dgdv3z8VSNiuU5PzBfJoy5J07tlW8d3/Hk5/lg+dscf3vgeGNHMzGpz8jAzs9qcPDo71vEdfz6NP5aP3fGHMb7rPMzMrDbfeZiZWW1OHmZmVpuTh5mZ1ebkMQ+QtNBg1s2vJH1V0kRJC0i6WNJsSbuP9HGNBpIWlnSApF9IOkvSJyUtPNLHVYekTSXtKul9rb+CsVdv/ZbyLKYfz5PSlYp/WNvyeEmnloqfYy5TMl6LkwcgaXlJx0s6Ny9PlrRXwfhvlrRYfry7pP+TtGqp+MDVg1w3ZPk9XCjpL5LukHSnpDtK7iPvZ2KehnjpPAtlCdtExBPA9sBdwBrAZwrFRtJakn4k6QJJl7T+SsXP+1hG0nckXZ/nsvlWoZPCScA6wHeA7wKTgZMLxH1J+28pnyAPKRT7ZODrwGbAm/JfyU52ZwEvSFqD1FJpZeCnBeOvLOkgeOmC7xfA7QXjA1wj6UxJb8+D0BYx4JDs84mfAD9m7hwlfwF+BhxfKP73gfUkrQd8ijQY5EnA5k2CSnolaQKtRSS9EWh9MSYCizaJ3cHxwCeBGcALhWMj6UPAF4FngVYTwABeXSB863u+HXBmRDxe8DcEcCbwA+BH9OCzyU4HrgDenZd3I31Hu06ZMEivi4jJleVLJd3SMGa7rSS9G9gLWJr0e7u8UOwpwOToXbPRFyNijqT/Ar4TEd+RdEPB+B8ETs0JZEvgnIj4ZsH4kCbR2zrv69uSzgB+0mkepFoiYr7/A67L/95QWXdjwfjX538PBvaqrmsYd0/gUuDJ/G/rbxrwrsKf0R96/H9wO7Bsj2IfCfwZuAFYAFiu5PsBZvTys8n7uKnDupkF4p4CbFxZ3gg4qQfH/17SmEp/A95cMO6ZwAo9/Nz/AOwC3ASs1u3/Yghx16/8bQTcCHyvta6H72dL0rxJj5ES+CZDjeV+HoCky0hXdBdGxPqSNgaOiohGdwaV+JeThpL/APAfpMmt/hgRry8U/90RcVaJWP3s40hgPOm2+rnW+oi4vlD880gJ7+kS8TrEXxp4PCJeyHPBTIyI+wvFPpT0f/pL+n42j3TbZgj7+D/SNARn5FU7AhtGxKcbxr0VeA3w97xqFeA2YA5pIs91m8TP+1gTOBGYCbwWuAU4oMT/taRLgTeQPpvqZ/+OprFz/MnAfwNXR8RpklYD3hMRRzWMe2k/T0dEvKVJ/LZ9LQPsDuwBPEAqRZhG+tzOjIjVhhTXyQMkrU8q830d6QpjOWDHiPhTofivBHYl3eFcKWkVYIuIOKlQ/AM6rH6cdEV8Y6F9dPqyF/uS52K3H5Ou9KongY8XiN2xArXg539n5/BRosittY8ngcVIxWIi1Vc+VdnXxCHG7bfuLSLqjEbdbR9/Bj4aERflMvcDgA9GxDoFYne8wIuIUsVi1X0tBaxc6rwwXCT9hVSP9eOIuKftuc8NNRE6eWSSJpCuwATcFhHPF4y9GPBsvupdC1gbOLfUPiT9lFT2++u8anvShFmTSFcWXy2xn16SdC3wO9LV6Yut9RFxYoHY36ksLgxsRSo23LFp7LEuX8i8TET8vdP6Ie5jYqQGC9V1a0XTMvdhkEsl3kGqN5tBusO8KiI6XbANJf7ywJeBV0XE2/KdziYRUaS+VdJ44KsR8akS8frEdvIASe/qsPpxUplyp3nZ68afAfw7sBRwFWlGxX9FxG5NY+f4VwBvj4h/5uXFgd8CU0l3H5P7236Q+1gCOIRU7AapvPSwiHi8aewc/4aIeGOJWIPY15LA6RExtVC8BYAPM/ezuQz4YYmLA0lrR8Sf893xyzQtNpQ0k9QwQaTEuhrp4qnxXUFlH60T5IoRMbXECVLS7yJis3xHVj2JiQZ3Yh32c0NEvFHS3qS7jkMk/alEcV6Ofy65sU5ErJcvYm8oVaSd93F1RGxSKl6LW1slewGbkCqbAbYgXWWsJumwiGjadFER8XRusnhMRHxV0h8bxqz6NypFPcDzwPIR8Yyk57psU9cJpCK99+TlPUhf+k6JdyjOlbQv6e6pJ/UGFU+RTpKlfJ9UEX9MXt4jr9u7QOwDgH2Bb3R4LoBGxYbtJ6mcpPZrErODn1C4NWNEbJb/fUXTgxvABEkrkL73nx/oxUOwbESc0WquG6llV+kWezdKmkZqXNAq6iQiftEkqJNHMgF4bUQ8AC9dKZ1EagVxBc3bvUvSJqTmla027yX72JwK/EHS2Xn5P4Gf5uKyUs0uV4+Id1eWvyjpxkKxIbVoATiosq5IU11Jv2bu1ek4Ul+GM5vGrXhTRKxXWb6k1MVBROyb/92yRLxB7O96SRsVDtvzE6SkfyPdOZH3UarY7TDgfFJR1XWSXk3ZfhhP5QrtAMiNdYrczVcsDDxM3wuNIDV+GTInj2TlVuLIHszrHpFUol5if9JJ8ZcRcXP+AvbX2qKWiDg83/6+Oa/674hoTdpepGgMeEbSZhHxO0idBoFnCsVmqC0+BunrlcdzgL+1Vxw29IKk1SPirwD5/7cXfWE2JdVjvfS7bVrp39bYYhypqeg/msTsoGcnSEnvIN2VvYr0u10VuJXU8bGxiDiTyoVGRNzB3L42JRxAavm0uqSryI11CsYHOC4irqquyL/fRlznAUg6htREsfUleTdwD6kX8m+G66pvqCQdTrpD+n1EPDXQ64e4jzeQmlsuQSpXfgR4f0QUucLuZYsoSUdFxOcGWtcg/lakYpk7SJ/NqsAHIqLYBYJST+rVSf0BWokpmrZGU9+e3nNIPfDPiohnm8Rt20fPWjPmO7y3ABfluoktgd0josgIEZJWIh1762R7JbB/yYuPXjbWyfGvj4j1B1pXO66TRypTIpXdb5ZXPUqqM/hIofjLAZ8lXQ1Vb61LNXP9AKlCfhNSh8ErgSsi4ux+NxzaviYCtLeeKRC3Zy2iuvx4ilV65ngLkU4AkE4ApeqaWvFvpYc9qXMjC1qNLgrFfBNwd0Tcn0+QHyJdmN0CHFyiPkt5Tu6cRN4YES9K+mNbMWKT+BeShiNpFV3vDuwWEW8tFH9R0t3HqhGxj1KfmNdExG8KxN4E2BT4BHB05amJwH81/YxcbEW6fFMap2ljYCfgTtKYNqWcSqog3J7U4WhPYHap4BHxY+DHSv1J3gN8mlTJ2rgyUdLuEXFKW/EGysN7RMT/Nd1HjvOxtvhLkobkGDJJHyZV/r5aUvUq9xWkVm+NSHpLRFzSobXeGpIaV0i2uQl4JXBfwZhIeh3pxLh0Xn4I2DMibioQ/ofMHT5lU1KF88dIndOOpUzxzGM58V1BGubjQSqVwgUsl39fLT+R9ImC8X9MapzTag11L6kEpHHyABYEFied56vngico8NnP18kj97nYJf89RDrBqwfFVMtExPGS9o/UeelySdeVCi7pOFIl8AOku44dgSI9v0kd06BzIurlbWuJFlE/Bc4FvgIcWFn/ZKFWXJsDl5AaKLRrXCEJfSr7XwHcotQfpmRP6mNJvb0vzfvbIq/btGFcgPGVz/m9wLGRRkI4q2Bjix1IdW+fJNXvLUGq5C7lYaURmE/Ly7uQKp9LWT0i3itpF4DcKrPIwGuVc81PokBnz3bzdfIgjXd0JbB9RMwCkPTJHuynVYZ5n6TtSBWSpUaMBViGNHTIY6S6iIciYk6JwBHxw/zwol5UulVidWoRdUb3LQaW+6A8Tm7JVWmRs7ikxZu2yImIVn3BYRHRp5e50jAWJXx94Jc0sli1biYiLsut9EoYL2lC/i5uRbobbily7qnU8b1IqpMr7YOkOo+jSd/P3wPvLxj/X5IWYW5jgtXp2+y+hIUkHcvLG1s0KzaPHg3ANRb+gHeSikbuJo2IuhVwZw/2sz3piuh1pFZWM4B39GA/ryWVb/4NuKdw7JcN5NhpXYP4m1f+3gysVDD2f5KaVz5FKpJ8Ebi5x59N0cESSXeA4/LjtUi9nhcoEPeXwBfyiWUS8L+kVoEljvnzpOLBs0mDUrbqWNcgNX1tEnsv4DOV5XtJxTFPklobFvvsO+z76wVjbUPqcDubVLx9F2noopLH+0dSJ9YNgQ1af03jusKcl4YP2YF0hfoWUh+PX0bEBSN6YIMkaXtShfl/AEsC1wBXRsQJBWL3tNJtOPSqRY6ktUmNIL5K3/lBJpJObCV7afdklAKl8Zq+SGosEqQ78S9GxKPNjvil+BsDKwAXRL5LyMXFi0eD3vG52HdqRDycl2/I/7cLA+dHoUFNu+z77xHRcViXIcZbhlTfKuCaiHioVOwcf0ZEbFAyJrjYCnjp1venpI51S5EqzT8HNEoeuQVR1+wcBQb9y6aSfvTfiojSbfR7WummNKhgt88oImL1pvsAno+IhyWNkzQuIi6V9M0CcV9Duqtckr71Hk8C+xSIX1V8lAKlcY9+ET1sih4R13RYV2JMK7USR3Zmjv1sLgbqpWKTwUg6hXTncWVE/LlU3Da/lrQfhUd99p1HD0nas7/no8Cgf8NF0qrRg0o3vXw2vHHMbTF2ffTt1T7UfVxEKqL8CrAsqTPZmyKiRKUwkjaJiKIzN3bYxw2klmNHk+aEuVnSzGg4BpKki0lD4Zfu1dxTkmZFxBod1o8DZkXDEY3VfRZLkaZTWKlJ/Mp+tiTdUf47qR/PDaRm9t8qET/voyejPjt5jGF6+aBwLz1FwcHh8r563VdlHGlMqM+QOsJ9OSKKDK2SiyWfISWmVoucU5peeVXiL0wqg2//bD5YIn7ex3+QEupVEXGUUi/2TzS9e1Ua0uaNwIX0Hfeo1F1xTyh17H0kIv63bf2XSMOh/HfD+K074k53GY1PvG37Gk+aPndLUlP+ZyJi7VLxe8XJYxjkjkY7RcRjeXkp0qiu247ogdUg6QJSU+ZPU+mrEg17aSuNSPtBUlPL3wFHRm75Vop638P8TFLLvV1JzUR3A26NiP1LxO+lbnfHo/2uOF8QHEc66baK79YDpgN7R8HOjr2U7/wWA64mFT3/LgqM5N22j56M3uDkMQwk3RgRb2hbd0MUHII8X70sT9+meCXnZJgRERtUe2ZLui4i3tQw7j2kYTG+ydzZ7F4SBTraqcc9zCuVtX+KiHVzQrwyIjYuET/vYy1S4p5EyeaWY1y+A2s1TLgl8vhiY4Wko0mtn54jNYS4gjRrYbFx49Sj0RtcYT48XpC0SutkrjR7W7GsLeljpLk2HmDuREoBFBt+g971VbmIdKzr5b+qRh3tuvQwF6kBQOMe5hWtz+ax3GP7ftIw+SWdCfyAdLVdbNBFzZ3Po+px0hX8l9oqpUedSAMV3jHSxzFUEfFJAEmvIPUf+TFpJIGFCu6j+OgN4DuPYSFpKqnX7uWkk9e/Ax+KiPMKxZ8FbNTLH3puDnwlsDKp09REUpPOab3aZ1NKE1gtRe96mLf2szdpOJt1ST/+xYEvxNwOliX20ZPmlpK+SkpGP82rdgYWJSXAzSKiU+95K0TSR0nngw1IfTyuJN21XtLDfS4A3BQRrxnwxf3FcfIYHpKWJbXlhsJtuZXmF39rFOpVPtIk/SYiti8QZ1FSM93n8/JrgLeThmQvOe5Uz0k6lNRKrGxzy35GXC3Rmmus6qe1FVBukjJJnyYljBm9+v22jd4wntSZ+IyIOLD7VoOI6+TRe0qzER5cWR4HnNy0g1cl3vGkPge/pe+JpcighXkfJ5KGon4sLy8FfKNki6LKvorUBylNz7tXRNwuaQ3gWlIv3snAdU1/PJX9LAMcSuoZ3+pod3jJO8GeNbdMfUX2iYhr8/KbSPM/rFe6Xq6kXp/ch7O1Va9JqnaYLDafjes8hsfKkg6KiK8oDd19Bqk9dyOSTo6IPUjDXB9N6tC3YNO4XazbShwAEfGopF6dWBp/NtlSEdGa9W1P4LSI+JikBUlDxBRJHqTy4yuYO0nQbqSWaVt33aKm6N1kWXsDJygPyU7q4LhXbs30lR7ts4QZ9HNyp+EMlD38vIddRFyuNDtqq3FLkZkQfecxDCSJdMU7k9SW+9yIOLr/rQYV9xbSCeo80rzrfRQu1/8jacydR/Py0sDlo7lYo61l2FXA1yLiV3m55JwPN0XE69rWFS3yyeXUHyYNQQNwGfDDKDRxUK4fYqx1FhwO+S57Tfr24bli5I6oHknvAb5G+s606lw/ExE/bxLXdx49pDSDWsu3SPMbXEUaJnn9aDC2T/YD4GLS0OXTK+tFofm/K74BXJ37NIg0NMkRpYIrjdB7KGkWvgnM7ejY5D38SdLXSYPmrUEebia3NinpAkk7M3cU4B1J816X9H1gAeCYvLxHXrd3ieBjOWn08uSeG0PsD6xE6ry6MalPRuMm0rl5/UXR+5lKP08aUeHBvN/lSK0cGyUP33n0UK7I7iYK9s7+fkR8uESsAfYzmbk/mktK9QDPsf9M6ig4g0pT1Cb1BkpjHO1PGpjvhMhT5irNBb56RJzc3/Y19vMkqaNXq5n0OOb21o4o0NO/051Sybunsarbyb3gb2smqbjnmoh4g9JgmF+OiPYJwIYav+fDw7TfBec61z82vTP2nUcPRcSW+T9qp4j4WQ/3MxyJYxXgn8C06rqCHREfj4hzC8UCIHe0OrLD+t+T5mUotZ/GMzYOwguSVm91gsud44r19xjD9mfuyX3L1sm9YPxnIw22iKSFIuLPudVeKf8EZiqNQtGr4WHOk3Q+cye0ei9wTtOgTh49FmlO5c+QKlDHst8yt7nfIqSistuY27u3qUslfY3UKbDaYqzUjIg9ozTu1MsULhf/DOkzuoNUpLcq8IGmQSV9BDi1rRXdLhFxTL8bjh69Prnfk4s5fwVcKOlR0nw5pfyCAjNOdpJbGC4fEZ9Rmip5s/zU1aQ62GbxXWzVe5KOZO40t9Wri2IV2sMt1+fsFxFFyty7FPEVK9rrpdyOvmVh0qQ7M0ofe26p1zox3hYRjWec0zAMndNLkn5JSqKfIBWpPkqaJOvtPdjX5qRBNc8t1VAhx10EWCUibisVM8f9DXBQRMxsW/96UtFbow6gTh7DoFdt9Efa/NyJrD+SVga+GWWGk39LRFySrxxfpmlnx1ymv27kE0GuxP1TFJzIarj04uReaQ7f77oG8f+TNNXwghGxmqQ3kKY1bjo3fb9jz5X47brYahjMC23GJR1QWRwHrE8a36pU/CVI43O1ioAuJ/2IhlyR2Naz9mVK/EC7uIfUi7eEzYFL6DvZVEujsb+y84CfSWoNpfKhvG5MqJ7II+Ly1jpSa7QS+iTRnFxLDhNzKOlO9TKAiLgx12eVsGQ/zzWeMMvJYxj0uo3+MKlWCs8h1YGcVTD+CcBNpImgIP34fww0adXy9aYHNRjqO2PkOOANQJG6mog4JD88LCL63MFKKnFR8jlSwmg1uriQNPjiWNGTk7ukg4D/ARaR9ERrNfAv0jh1pTwfEY+nrmAvebHbi2uaLmmfiPhRdWVuoTajaXAXWw0DSceR2ui35kjYA3ihVH3BvKBL2fvL1o1G6jsnxhzgrogoOWpvtzGoejJY4lhQPbkDT7dWk0/uEXFQof18pVSsLvGPJ/XVOpA0QsHHSXU2jSazyrGXJ42F9i/mJosppFEo/isi7m8U38mj98ZyG/3hKvqRdDWp1+vv8vKbga9HxCYFYq9JGmpjMn07khUpHlAayuPZiHghL48HFoqIp/vfclCx1yZdXX+V1OKqZSLp8xpS3YSkMyLiPeo8JDtRaK6TXuvVyV3S2rnl1vqdni/VClBp8M7PA9uQkt/5pHHRni0RP+9jS6A1AsLNUWjEXiePYSDpelJfj2ob/Z+3X0mORpo7qNq7SPMMnJKXdwEeiDwfQYH9vIF0Z7YE6Uf0CPD+Vse+hrF/R6pPOZpUd/ABYFxUBqtsGP8aYOvIs9cpjRN1QRSYI13SDqT5199BpY8NaQyq03OflaHEXSEi7lOaW+Zlogfz1ZfU65O7pGMjYt/hagUoaWKO+2TJuL3k5NFDkj5B6oy2JPAjoFVmPQn4YKkrgOEgaXpETBloXYH9TASIiCcGem2NmK1ZEF9qYVKyyGc4itwkbRIRV5eKN9YN98m9V5RGMT6BuXWKj5PODY3rJHrNFea9tRJpetXXkkayfAS4FDgrIoq1VBomi0l6daSZ21qVtYs1DSpp94g4pa01F60KxCgzrPxzSj39b1eafOde0oRNpTylylhlkjYAik0jmt2QO/StQ9+it0ZD4ucmwEeRZj4Uc8cUazykSi9FxL75356OCyVpYdJslJsxd7j9HxQsVjqe1F/qyry/zUgNRUZ9saGTRw9FxKcBlIYAnwJsShr99iBJj0XE5BE8vLo+CVzW1sP5QwXithJQpyE+St0W70+aHe/jwOHAlsD7CsWG1EHtTEn/IH02ryQNAVHSycCfgW2Bw0jDvt9aIO5Xgf+MiBKxht0wnNxPIhURtuYB35X0f7FTofgvtBIHQET8TtKYmNTNxVbDIPdh2IQ0WdAmpGKsmRHReHiJ4ZR7OK+dF/9coodzJfab21sodVo3xNg7RcSZA61ruI8F6Nv7u2gz7Favb+Vh5vP+royIjQfcuP+4V0XEmwsd5rCTdAbp5N6qi9sVWDIiipzcJd3SfpHXaV2D+N8ktRg7jZT83gs8S34/pSrme8HJo4ckHUsqZngS+ANwDWkAt0dH9MBqkPTZiPhqftznhCvpyxHxP4X203U61NEcO8fq+fhQkq6NiA2VZkfcjzTH+LVNW4xJ+hbpTulX9B1TbExM0zsMJ/dTgO9GxDV5eSPgIxFR5M61S51Ny6iuu3GxVW+tAixEqu+4l9Tz+LGRPKAh2JlUtAFwEFC9Wp9Kams/ZJI2IRXnLddW7zGRNN9yk9hvI81ZvqKkb7fFLlk0sE9EfK+1EGmWxX2YO/dGCcfmpPQFUqurxYESrcUmkvpJbFNZV6Ln+nC5XtLGbSf36QNsU8cGwO8ltUaPXgW4rdXEuWmT5l7X2fSSk0cPRcRUpZrfdUgnyE8Br5P0CGnOgUP6DTA6qMvjTstDsSDpRDiBvvUeT5AmVWriH6QTyTvo26P2SVIdTinjJSmiz/hQRacDjohWr+/LKTjJ11grOu2gpyd30gWSdeBiq2EiaSVSncemwPbAMhGx5Ige1CBUi3fai3oKF/2s2qu+BZImRETPKiGVZitchTRTJMB/A3+PiE8ViH1Af883bY2mNKvcPqTm4y9dTDZtxTVcuvVTaSn1nZL0b/Rt5VZqHpsxy3cePSTp46RksSnwPKnPx+9J7bpn9rPpaLKe0tg+4uXj/CzcfbPanlaaz6O9KeqQy3xbvahJzVx72Yv6C6QT8H55+XxSE8wSej3R1NmkFkoXMQYnl2olh16d3CW9gzQF86uAB0mtDG+l3Dw2Y5aTR29NItURfDIi7hvhYxmSiGhU71DDqaT5TrYnXbnvCcxuGHP//O/2DeN0JGkCada6DwB359WrAHeQBkhsfDKOiC82jTGARSPicz3eR88Mw8n9cNLUthfl1m5bArsXio2knYDzIuJJSf9LGq36S6O5lVXLuJE+gHlZRBwQEWeN1cQxzJaJiONJo4xenotNGrU0ycNvjAd+EhF/a/8rcMxfA5YGXh0R6+civNVIQ6wUHdFX0lqSLpZ0U15eN59smvqNpOITJw2j1sn9L5GmPtiK1KqxlOcj4mFgnKRxEXEpqc9WKV/IiWMzYGvSHev3C8bvGScPGy1a/SLuk7SdpDeSTsyNRBqs8MXc16a07UktrV4ajyg//jCplVdJPyK1dns+7+dPpJZwQyLpyVwEuT8pgTwj6YnK+rGi1yf3x5TGKrsCODU3bX5qgG3qaN2dbkcaDfi3FG5s0SsutrLR4kv5BP8pUm/eiZRrEfVPYKakC+k7DfDHG8aNVgurtpUvdKpjaWjRiLhWfed9GHIjgIjodV3KcGk/uT9I2ZP7DqShZj5J6tW/BKmHfyn3Kk3E9VbgqNwRd0xc1Dt52KgQEb/JDx8nDR9S0i/oTb+FWyS9LyJOqq6UtDtpKJGSHpK0OnnIFkk7Ao2LQyVdHBFbDbRuFOvJyV3SGsDylREOXgROzMVLSwIPN91H9h5Sc+CvR8Rjklag79D7o5ab6tqoIOlEYP+2XtrfGM1NRiWtSEpKz9B3sp1FSJPt3FtwX68mzWC3KfAoaYTm3YZad5PHhFqMNMXtFsztszORVIG7dpdNR4UOJ/fW+s2A+yJPf9Ag/m+AgyJiZtv61wNfjohO0wLX3cd40vwao/qz7sZ3HjZarNtKHPBSL+03lgisHk0GlZPDRpLewtzWPedExMVN4nbZ1x3A1koTT40j9QrfGRhqxf+HSAM6voq+U+Y+AXx36Ec6bL5JqgNq93h+runJffn2xAEQETMlTWoYuxXrBUm3SVplLPYbcfKw0WKcpKVa435JWppy388fM3cyqC3Jk0EVik2keVl6MjeL0vwmHwFWJPXJuCgvfwr4E6mJc20R8S3gW5I+FhHfGXCD0afXJ/cl+3lukQLxW5YCbpZ0LX3r44rM0NlLTh42WnwDuFrSmaQilB2BIwrFXiQiLs5DiPwNOFTSDMqMDdVrJ5OKqa4mdUT8POnz+a+IuHGoQVsDXkbEd3o54GUPLdnPcyVO7tMl7RMRP6qulLQ3fYe6aeoLBWMNK9d52KghaTJz+3ZcEhG3FIr7e9J8Dz8n3SHcCxwZEa/pd8NRQH1nPxxPqiRfJRrOVzFcw870iqTTSN+RTif3t0ZEo/lUJC0P/BL4F33rsxYkJe77m8SfFzh52KggaZVO60uUBStN9Xkr6Wr1cFKLnK+2RmIdzXp1YleeH6T9cafl0Wi4Tu65R/nr8uLNUXjqaElPMnfSswWBBYCnYpTP5AhOHjZKtEZBzYuLkHpq3xYR8/UYQpJeYG5ZuEifzdM0nC52rN95tPT65D6clDrx7ABsHBEHjvTxDMTJw0YlSeuT5nbeu0GMaf09PxYqJXulkpSqCYm8vHBELDBSxza/Gwt3fuAKcxulIuJ6pYl9mtiENGDhaaSZHEvMPzJPGMYBL60fkt5VWRxHKnorNf96Tzl52KjQNm/FONLoov9oGPaVpGEfdiHNbf1b4LSIuLlhXLNSqv1R5gB3kYquRj0XW9moIKk6q2LrR3RW01ZFlfgLkZLI14AvRsRY6AhnNmo5edg8LSeN7UiJYxJp/u8TSg4dYjZUSjOMfoc0yyikibn2j4h7Ru6oBsfJw0aUpF8zt5XVyzSp1JZ0EqklzjnA6RFx01BjmfVCHun5p6TOoJAmmtotIt46ckc1OE4eNqIkbZ4fvotUR3FKXt4FeCAihjwsu6QXmdvMtfpFb9TM1awUSTdGxBsGWjcaucLcRlREXA4g6RsRUZ3E59eSpjeMPSbmRbD52sN5CP/T8vIulBvuvaf847LRYrE87DgAklYjDRluNi/7IGlOj/tJQ8/sSBq4c9RzsZWNCpKmkuaruINUrLQq8KGIOH9ED8zMOnLysFEjt4xqTYzz54h4biSPx6xXJH2H/huKNJ0iuedcbGUjStJnK4vviIg/5r/nJH15xA7MrLemkwZ0nAG8o/K49Tfq+c7DRtS8MkCf2VCNlbGs2vnOw0aaujzutGw2LxqTV/BOHjbSosvjTstmNkq42MpGlIcGt/lR2yRQi9L3ez8mOrA6eZiZWW0utjIzs9qcPMzMrDYnDzMzq83Jw8zManPyMDOz2v4fVX7iADJW0JQAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "local_explanations_out = pd.read_csv(explainability_output_path + \"/explanations_shap/out.csv\")\n", "feature_names = [str.replace(c, \"_label0\", \"\") for c in local_explanations_out.columns.to_series()]\n", "local_explanations_out.columns = feature_names\n", "\n", "selected_example = 111\n", "print(\n", " \"Example number:\",\n", " selected_example,\n", " \"\\nwith model prediction:\",\n", " sum(local_explanations_out.iloc[selected_example]) > 0,\n", ")\n", "print(\"\\nFeature values -- Label\", training_data.iloc[selected_example])\n", "local_explanations_out.iloc[selected_example].plot(\n", " kind=\"bar\", title=\"Local explanation for the example number \" + str(selected_example), rot=90\n", ")" ] }, { "cell_type": "markdown", "id": "5d169fff", "metadata": { "papermill": { "duration": 0.054899, "end_time": "2022-04-18T16:07:31.931845", "exception": false, "start_time": "2022-04-18T16:07:31.876946", "status": "completed" }, "tags": [] }, "source": [ "### Clean Up\n", "Finally, don't forget to clean up the resources we set up and used for this demo!" ] }, { "cell_type": "code", "execution_count": 23, "id": "4a66aede", "metadata": { "execution": { "iopub.execute_input": "2022-04-18T16:07:32.047815Z", "iopub.status.busy": "2022-04-18T16:07:32.047054Z", "iopub.status.idle": "2022-04-18T16:07:32.151107Z", "shell.execute_reply": "2022-04-18T16:07:32.150588Z" }, "papermill": { "duration": 0.164477, "end_time": "2022-04-18T16:07:32.151256", "exception": false, "start_time": "2022-04-18T16:07:31.986779", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "session.delete_model(model_name)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Notebook CI Test Results\n", "\n", "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n", "\n", "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-east-1/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_outputs.ipynb)\n", "\n", "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-east-2/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_outputs.ipynb)\n", "\n", "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-west-1/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_outputs.ipynb)\n", "\n", "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ca-central-1/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_outputs.ipynb)\n", "\n", "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/sa-east-1/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_outputs.ipynb)\n", "\n", "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-west-1/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_outputs.ipynb)\n", "\n", "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-west-2/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_outputs.ipynb)\n", "\n", "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-west-3/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_outputs.ipynb)\n", "\n", "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-central-1/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_outputs.ipynb)\n", "\n", "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-north-1/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_outputs.ipynb)\n", "\n", "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-southeast-1/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_outputs.ipynb)\n", "\n", "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-southeast-2/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_outputs.ipynb)\n", "\n", "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-northeast-1/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_outputs.ipynb)\n", "\n", "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-northeast-2/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_outputs.ipynb)\n", "\n", "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-south-1/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_outputs.ipynb)\n" ] } ], "metadata": { "instance_type": "ml.t3.medium", "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.9" }, "papermill": { "default_parameters": {}, "duration": 1526.483007, "end_time": "2022-04-18T16:07:32.926299", "environment_variables": {}, "exception": null, "input_path": "fairness_and_explainability.ipynb", "output_path": "/opt/ml/processing/output/fairness_and_explainability-2022-04-18-15-28-21.ipynb", "parameters": { "kms_key": "arn:aws:kms:us-west-2:000000000000:1234abcd-12ab-34cd-56ef-1234567890ab" }, "start_time": "2022-04-18T15:42:06.443292", "version": "2.3.4" } }, "nbformat": 4, "nbformat_minor": 5 }