{
"cells": [
{
"cell_type": "markdown",
"id": "6a46bd98",
"metadata": {
"tags": []
},
"source": [
"# Tabular Online Explainability with SageMaker Clarify"
]
},
{
"cell_type": "markdown",
"id": "8a4b915e",
"metadata": {},
"source": [
"---\n",
"\n",
"This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
"\n",
"\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"id": "e2789005",
"metadata": {},
"source": [
"1. [Introduction](#Introduction)\n",
"1. [General Setup](#General-Setup)\n",
" 1. [Install dependencies](#Install-dependencies)\n",
" 1. [Import libraries](#Import-libraries)\n",
" 1. [Set configurations](#Set-configurations)\n",
" 1. [Create serializer and deserializer](#Create-serializer-and-deserializer)\n",
" 1. [For visualization](#For-visualization)\n",
"1. [Prepare data](#Prepare-data)\n",
" 1. [Download data](#Download-data)\n",
" 1. [Loading the data: Adult Dataset](#Loading-the-data:-Adult-Dataset)\n",
" 1. [Encode and Upload the Dataset](#Encode-and-Upload-the-Dataset)\n",
"1. [Train XGBoost Model](#Train-XGBoost-Model)\n",
"1. [Create endpoint](#Create-endpoint)\n",
" 1. [Create model](#Create-model)\n",
" 1. [Create endpoint config](#Create-endpoint-config)\n",
" 1. [Create endpoint](#Create-endpoint)\n",
"1. [Invoke endpoint](#Invoke-endpoint)\n",
" 1. [Single record request](#Single-record-request)\n",
" 1. [Single record request, no explanation](#Single-record-request,-no-explanation)\n",
" 1. [Batch request, explain both](#Batch-request,-explain-both)\n",
" 1. [Batch request with more records, explain some of the records](#Batch-request-with-more-records,-explain-some-of-the-records)\n",
"1. [Cleanup](#Cleanup)"
]
},
{
"cell_type": "markdown",
"id": "2116c025",
"metadata": {},
"source": [
"## Introduction\n",
"\n",
"Amazon SageMaker Clarify helps improve your machine learning models by detecting potential bias and helping explain how these models make predictions. The fairness and explainability functionality provided by SageMaker Clarify takes a step towards enabling AWS customers to build trustworthy and understandable machine learning models. \n",
"\n",
"SageMaker Clarify currently supports explainability for SageMaker models as an offline processing job. This example notebook showcases a new feature for explainability on a [SageMaker real-time inference](https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints.html) endpoint, a.k.a. [Online Explainability](https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-online-explainability.html).\n",
"\n",
"This example notebook walks you through: \n",
"1. Key terms and concepts needed to understand SageMaker Clarify\n",
"1. Training the model on a training dataset.\n",
"1. Create a model from trained model artifacts, create an endpoint configuration with the new SageMaker Clarify explainer configuration, and create an endpoint using the same explainer configuration.\n",
"1. Invoke the endpoint with single and batch request with different `EnableExplanations` query.\n",
"1. Explaining the importance of the various input features on the model's decision.\n",
"\n",
"\n",
"In doing so, the notebook will first train a [SageMaker XGBoost](https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost.html) model using training dataset, then use SageMaker Clarify to analyze a testing dataset in CSV format."
]
},
{
"cell_type": "markdown",
"id": "bf64fc0a",
"metadata": {},
"source": [
"## General Setup\n",
"\n",
"We recommend you use `Python 3 (Data Science)` kernel on SageMaker Studio or `conda_python3` kernel on SageMaker Notebook Instance."
]
},
{
"cell_type": "markdown",
"id": "48047964",
"metadata": {},
"source": [
"### Install dependencies\n",
"\n",
"Install required dependencies. We use `shap` and `matplotlib` to visualize the feature attributions."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "ec349708",
"metadata": {},
"outputs": [],
"source": [
"! pip install --upgrade -r requirements.txt --quiet"
]
},
{
"cell_type": "markdown",
"id": "7ec24d29",
"metadata": {},
"source": [
"### Import libraries"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "85c9245b",
"metadata": {},
"outputs": [],
"source": [
"import boto3\n",
"import io\n",
"import os\n",
"import pprint\n",
"import pandas as pd\n",
"import numpy as np\n",
"from sagemaker import get_execution_role, Session\n",
"from sagemaker.serializers import CSVSerializer\n",
"from sagemaker.deserializers import JSONDeserializer\n",
"from sagemaker.utils import unique_name_from_base"
]
},
{
"cell_type": "markdown",
"id": "b662efac",
"metadata": {},
"source": [
"### Set configurations"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "0787380b",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Region: us-west-2\n",
"Role: arn:aws:iam::000000000000:role/service-role/SMClarifySageMaker-ExecutionRole\n",
"Demo S3 key: s3://sagemaker-us-west-2-000000000000/sagemaker/DEMO-Tabular-Adult-1686294366-1e35\n",
"Demo model name: DEMO-Tabular-Adult-1686294366-1e35-model\n",
"Demo endpoint config name: DEMO-Tabular-Adult-1686294366-1e35-endpoint-config\n",
"Demo endpoint name: DEMO-Tabular-Adult-1686294366-1e35-endpoint\n"
]
}
],
"source": [
"boto3_session = boto3.session.Session()\n",
"sagemaker_client = boto3.client(\"sagemaker\")\n",
"sagemaker_runtime_client = boto3.client(\"sagemaker-runtime\")\n",
"\n",
"# Initialize sagemaker session\n",
"sagemaker_session = Session(\n",
" boto_session=boto3_session,\n",
" sagemaker_client=sagemaker_client,\n",
" sagemaker_runtime_client=sagemaker_runtime_client,\n",
")\n",
"\n",
"region = sagemaker_session.boto_region_name\n",
"print(f\"Region: {region}\")\n",
"\n",
"role = get_execution_role()\n",
"print(f\"Role: {role}\")\n",
"\n",
"s3_client = boto3.client(\"s3\")\n",
"\n",
"prefix = unique_name_from_base(\"DEMO-Tabular-Adult\")\n",
"\n",
"s3_bucket = sagemaker_session.default_bucket()\n",
"s3_prefix = f\"sagemaker/{prefix}\"\n",
"s3_key = f\"s3://{s3_bucket}/{s3_prefix}\"\n",
"print(f\"Demo S3 key: {s3_key}\")\n",
"\n",
"model_name = f\"{prefix}-model\"\n",
"print(f\"Demo model name: {model_name}\")\n",
"endpoint_config_name = f\"{prefix}-endpoint-config\"\n",
"print(f\"Demo endpoint config name: {endpoint_config_name}\")\n",
"endpoint_name = f\"{prefix}-endpoint\"\n",
"print(f\"Demo endpoint name: {endpoint_name}\")\n",
"\n",
"# Instance type for training and hosting\n",
"instance_type = \"ml.m5.xlarge\""
]
},
{
"cell_type": "markdown",
"id": "7c44c759",
"metadata": {},
"source": [
"### Create serializer and deserializer"
]
},
{
"cell_type": "markdown",
"id": "9f0a7aba",
"metadata": {},
"source": [
"CSV serializer to serialize test data to string"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "79569f81",
"metadata": {},
"outputs": [],
"source": [
"csv_serializer = CSVSerializer()"
]
},
{
"cell_type": "markdown",
"id": "6e64de94",
"metadata": {},
"source": [
"JSON deserializer to deserialize invoke endpoint response"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "201230ef",
"metadata": {},
"outputs": [],
"source": [
"json_deserializer = JSONDeserializer()"
]
},
{
"cell_type": "markdown",
"id": "57480844",
"metadata": {},
"source": [
"### For visualization\n",
"\n",
"SHAP plots are useful visualization tools to interpret the explanations. For example, [SHAP additive force layout](https://shap.readthedocs.io/en/latest/generated/shap.plots.force.html) shows how each feature contributes to pushing the base value (also called the expected value which is the mean predictions of the training dataset) to the corresponding prediction. Features that push the prediction higher are in red color, while those push the prediction lower are in blue. We have some methods implemented for visualization in `visualization_utils.py` file."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "6f780c60",
"metadata": {},
"outputs": [],
"source": [
"%run visualization_utils.py"
]
},
{
"cell_type": "markdown",
"id": "aefb875e",
"metadata": {},
"source": [
"## Prepare data\n",
"\n",
"### Download data\n",
"Data Source: [https://archive.ics.uci.edu/ml/machine-learning-databases/adult/](https://archive.ics.uci.edu/ml/machine-learning-databases/adult/)\n",
"\n",
"Let's __download__ the data and save it in the local folder with the name adult.data and adult.test from UCI repository$^{[2]}$.\n",
"\n",
"$^{[2]}$Dua Dheeru, and Efi Karra Taniskidou. \"[UCI Machine Learning Repository](http://archive.ics.uci.edu/ml)\". Irvine, CA: University of California, School of Information and Computer Science (2017)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b94940af",
"metadata": {},
"outputs": [],
"source": [
"adult_columns = [\n",
" \"Age\",\n",
" \"Workclass\",\n",
" \"fnlwgt\",\n",
" \"Education\",\n",
" \"Education-Num\",\n",
" \"Marital Status\",\n",
" \"Occupation\",\n",
" \"Relationship\",\n",
" \"Ethnic group\",\n",
" \"Sex\",\n",
" \"Capital Gain\",\n",
" \"Capital Loss\",\n",
" \"Hours per week\",\n",
" \"Country\",\n",
" \"Target\",\n",
"]\n",
"if not os.path.isfile(\"adult.data\"):\n",
" s3_client.download_file(\n",
" f\"sagemaker-example-files-prod-{region}\",\n",
" \"datasets/tabular/uci_adult/adult.data\",\n",
" \"adult.data\",\n",
" )\n",
" print(f\"adult.data saved!\")\n",
"else:\n",
" print(f\"adult.data already on disk.\")\n",
"\n",
"if not os.path.isfile(\"adult.test\"):\n",
" s3_client.download_file(\n",
" f\"sagemaker-example-files-prod-{region}\",\n",
" \"datasets/tabular/uci_adult/adult.test\",\n",
" \"adult.test\",\n",
" )\n",
" print(f\"adult.test saved!\")\n",
"else:\n",
" print(f\"adult.test already on disk.\")"
]
},
{
"cell_type": "markdown",
"id": "9b39109e",
"metadata": {},
"source": [
"### Loading the data: Adult Dataset\n",
"From the UCI repository of machine learning datasets, this database contains 14 features concerning demographic characteristics of 45,222 rows (32,561 for training and 12,661 for testing). The task is to predict whether a person has a yearly income that is more or less than $50,000.\n",
"\n",
"Here are the features and their possible values:\n",
"1. **Age**: continuous.\n",
"1. **Workclass**: Private, Self-emp-not-inc, Self-emp-inc, Federal-gov, Local-gov, State-gov, Without-pay, Never-worked.\n",
"1. **Fnlwgt**: continuous (the number of people the census takers believe that observation represents).\n",
"1. **Education**: Bachelors, Some-college, 11th, HS-grad, Prof-school, Assoc-acdm, Assoc-voc, 9th, 7th-8th, 12th, Masters, 1st-4th, 10th, Doctorate, 5th-6th, Preschool.\n",
"1. **Education-num**: continuous.\n",
"1. **Marital-status**: Married-civ-spouse, Divorced, Never-married, Separated, Widowed, Married-spouse-absent, Married-AF-spouse.\n",
"1. **Occupation**: Tech-support, Craft-repair, Other-service, Sales, Exec-managerial, Prof-specialty, Handlers-cleaners, Machine-op-inspct, Adm-clerical, Farming-fishing, Transport-moving, Priv-house-serv, Protective-serv, Armed-Forces.\n",
"1. **Relationship**: Wife, Own-child, Husband, Not-in-family, Other-relative, Unmarried.\n",
"1. **Ethnic group**: White, Asian-Pac-Islander, Amer-Indian-Eskimo, Other, Black.\n",
"1. **Sex**: Female, Male.\n",
" * **Note**: this data is extracted from the 1994 Census and enforces a binary option on Sex\n",
"1. **Capital-gain**: continuous.\n",
"1. **Capital-loss**: continuous.\n",
"1. **Hours-per-week**: continuous.\n",
"1. **Native-country**: United-States, Cambodia, England, Puerto-Rico, Canada, Germany, Outlying-US(Guam-USVI-etc), India, Japan, Greece, South, China, Cuba, Iran, Honduras, Philippines, Italy, Poland, Jamaica, Vietnam, Mexico, Portugal, Ireland, France, Dominican-Republic, Laos, Ecuador, Taiwan, Haiti, Columbia, Hungary, Guatemala, Nicaragua, Scotland, Thailand, Yugoslavia, El-Salvador, Trinadad&Tobago, Peru, Hong, Holand-Netherlands.\n",
"\n",
"Next, we specify our binary prediction task: \n",
"15. **Target**: <=50,000, >$50,000."
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "3f6d7062",
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" Age | \n",
" Workclass | \n",
" fnlwgt | \n",
" Education | \n",
" Education-Num | \n",
" Marital Status | \n",
" Occupation | \n",
" Relationship | \n",
" Ethnic group | \n",
" Sex | \n",
" Capital Gain | \n",
" Capital Loss | \n",
" Hours per week | \n",
" Country | \n",
" Target | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" 39 | \n",
" State-gov | \n",
" 77516 | \n",
" Bachelors | \n",
" 13 | \n",
" Never-married | \n",
" Adm-clerical | \n",
" Not-in-family | \n",
" White | \n",
" Male | \n",
" 2174 | \n",
" 0 | \n",
" 40 | \n",
" United-States | \n",
" <=50K | \n",
"
\n",
" \n",
" | 1 | \n",
" 50 | \n",
" Self-emp-not-inc | \n",
" 83311 | \n",
" Bachelors | \n",
" 13 | \n",
" Married-civ-spouse | \n",
" Exec-managerial | \n",
" Husband | \n",
" White | \n",
" Male | \n",
" 0 | \n",
" 0 | \n",
" 13 | \n",
" United-States | \n",
" <=50K | \n",
"
\n",
" \n",
" | 2 | \n",
" 38 | \n",
" Private | \n",
" 215646 | \n",
" HS-grad | \n",
" 9 | \n",
" Divorced | \n",
" Handlers-cleaners | \n",
" Not-in-family | \n",
" White | \n",
" Male | \n",
" 0 | \n",
" 0 | \n",
" 40 | \n",
" United-States | \n",
" <=50K | \n",
"
\n",
" \n",
" | 3 | \n",
" 53 | \n",
" Private | \n",
" 234721 | \n",
" 11th | \n",
" 7 | \n",
" Married-civ-spouse | \n",
" Handlers-cleaners | \n",
" Husband | \n",
" Black | \n",
" Male | \n",
" 0 | \n",
" 0 | \n",
" 40 | \n",
" United-States | \n",
" <=50K | \n",
"
\n",
" \n",
" | 4 | \n",
" 28 | \n",
" Private | \n",
" 338409 | \n",
" Bachelors | \n",
" 13 | \n",
" Married-civ-spouse | \n",
" Prof-specialty | \n",
" Wife | \n",
" Black | \n",
" Female | \n",
" 0 | \n",
" 0 | \n",
" 40 | \n",
" Cuba | \n",
" <=50K | \n",
"
\n",
" \n",
"
\n",
"
\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" Target | \n",
" Age | \n",
" Workclass | \n",
" fnlwgt | \n",
" Education | \n",
" Education-Num | \n",
" Marital Status | \n",
" Occupation | \n",
" Relationship | \n",
" Ethnic group | \n",
" Sex | \n",
" Capital Gain | \n",
" Capital Loss | \n",
" Hours per week | \n",
" Country | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" 0 | \n",
" 39 | \n",
" 5 | \n",
" 77516 | \n",
" 9 | \n",
" 13 | \n",
" 4 | \n",
" 0 | \n",
" 1 | \n",
" 4 | \n",
" 1 | \n",
" 2174 | \n",
" 0 | \n",
" 40 | \n",
" 38 | \n",
"
\n",
" \n",
" | 1 | \n",
" 0 | \n",
" 50 | \n",
" 4 | \n",
" 83311 | \n",
" 9 | \n",
" 13 | \n",
" 2 | \n",
" 3 | \n",
" 0 | \n",
" 4 | \n",
" 1 | \n",
" 0 | \n",
" 0 | \n",
" 13 | \n",
" 38 | \n",
"
\n",
" \n",
" | 2 | \n",
" 0 | \n",
" 38 | \n",
" 2 | \n",
" 215646 | \n",
" 11 | \n",
" 9 | \n",
" 0 | \n",
" 5 | \n",
" 1 | \n",
" 4 | \n",
" 1 | \n",
" 0 | \n",
" 0 | \n",
" 40 | \n",
" 38 | \n",
"
\n",
" \n",
" | 3 | \n",
" 0 | \n",
" 53 | \n",
" 2 | \n",
" 234721 | \n",
" 1 | \n",
" 7 | \n",
" 2 | \n",
" 5 | \n",
" 0 | \n",
" 2 | \n",
" 1 | \n",
" 0 | \n",
" 0 | \n",
" 40 | \n",
" 38 | \n",
"
\n",
" \n",
" | 4 | \n",
" 0 | \n",
" 28 | \n",
" 2 | \n",
" 338409 | \n",
" 9 | \n",
" 13 | \n",
" 2 | \n",
" 9 | \n",
" 5 | \n",
" 2 | \n",
" 0 | \n",
" 0 | \n",
" 0 | \n",
" 40 | \n",
" 4 | \n",
"
\n",
" \n",
"
\n",
"