{ "cells": [ { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "# Fairness and Explainability with SageMaker Clarify - JSON Lines Format" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n", "\n", "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-west-2/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_jsonlines_format.ipynb)\n", "\n", "---" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Runtime\n", "\n", "This notebook takes approximately 30 minutes to run.\n", "\n", "## Contents\n", "\n", "\n", "1. [Overview](#Overview)\n", "1. [Prerequisites and Data](#Prerequisites-and-Data)\n", " 1. [Import Libraries](#Import-Libraries)\n", " 1. [Set Configurations](#Set-Configurations)\n", " 1. [Download data](#Download-data)\n", " 1. [Loading the data: Adult Dataset](#Loading-the-data:-Adult-Dataset) \n", " 1. [Data inspection](#Data-inspection) \n", " 1. [Encode and Upload the Dataset](#Encode-and-Upload-the-Dataset) \n", "1. [Train and Deploy XGBoost Model](#Train-XGBoost-Model)\n", " 1. [Train Model](#Train-Model)\n", " 1. [Create Model](#Create-Model)\n", "1. [Amazon SageMaker Clarify](#Amazon-SageMaker-Clarify)\n", " 1. [Detecting Bias](#Detecting-Bias)\n", " 1. [Writing DataConfig](#Writing-DataConfig)\n", " 1. [Writing ModelConfig](#Writing-ModelConfig)\n", " 1. [Writing ModelPredictedLabelConfig](#Writing-ModelPredictedLabelConfig)\n", " 1. [Writing BiasConfig](#Writing-BiasConfig)\n", " 1. [Pre-training Bias](#Pre-training-Bias)\n", " 1. [Post-training Bias](#Post-training-Bias)\n", " 1. [Viewing the Bias Report](#Viewing-the-Bias-Report)\n", " 1. [Explaining Predictions](#Explaining-Predictions)\n", " 1. [Viewing the Explainability Report](#Viewing-the-Explainability-Report)\n", "1. [Clean Up](#Clean-Up)\n", "\n", "## Overview\n", "Amazon SageMaker Clarify helps improve your machine learning models by detecting potential bias and helping explain how these models make predictions. The fairness and explainability functionality provided by SageMaker Clarify takes a step towards enabling AWS customers to build trustworthy and understandable machine learning models. The product comes with the tools to help you with the following tasks.\n", "\n", "* Measure biases that can occur during each stage of the ML lifecycle (data collection, model training and tuning, and monitoring of ML models deployed for inference).\n", "* Generate model governance reports targeting risk and compliance teams and external regulators.\n", "* Provide explanations of the data, models, and monitoring used to assess predictions.\n", "\n", "This sample notebook walks you through: \n", "1. Key terms and concepts needed to understand SageMaker Clarify\n", "1. Measuring the pre-training bias of a dataset and post-training bias of a model\n", "1. Explaining the importance of the various input features on the model's decision\n", "1. Accessing the reports through SageMaker Studio if you have an instance set up.\n", "\n", "In doing so, the notebook will first train a [SageMaker Linear Learner](https://docs.aws.amazon.com/sagemaker/latest/dg/linear-learner.html) model using training dataset, then use [Amazon SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable/) to launch SageMaker Clarify jobs to analyze an example dataset in [SageMaker JSON Lines dense format](https://docs.aws.amazon.com/sagemaker/latest/dg/cdf-inference.html#common-in-formats). SageMaker Clarify also supports analyzing CSV dataset, which is illustrated in [another notebook](https://github.com/aws/amazon-sagemaker-examples/blob/master/sagemaker_processing/fairness_and_explainability/fairness_and_explainability.ipynb)." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Prerequisites and Data\n", "### Import Libraries" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "tags": [] }, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np\n", "import os\n", "import boto3\n", "from datetime import datetime\n", "from sagemaker import get_execution_role, session" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### Set Configurations" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Region: us-west-2\n", "Role: arn:aws:iam::000000000000:role/service-role/AmazonSageMaker-ExecutionRole-20220304T121686\n" ] } ], "source": [ "# Initialize sagemaker session\n", "sagemaker_session = session.Session()\n", "\n", "region = sagemaker_session.boto_region_name\n", "print(f\"Region: {region}\")\n", "\n", "role = get_execution_role()\n", "print(f\"Role: {role}\")\n", "\n", "bucket = sagemaker_session.default_bucket()\n", "\n", "prefix = \"sagemaker/DEMO-sagemaker-clarify-jsonlines\"" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### Download data\n", "Data Source: [https://archive.ics.uci.edu/ml/machine-learning-databases/adult/](https://archive.ics.uci.edu/ml/machine-learning-databases/adult/)\n", "\n", "Let's __download__ the data and save it in the local folder with the name adult.data and adult.test from UCI repository$^{[2]}$.\n", "\n", "$^{[2]}$Dua Dheeru, and Efi Karra Taniskidou. \"[UCI Machine Learning Repository](http://archive.ics.uci.edu/ml)\". Irvine, CA: University of California, School of Information and Computer Science (2017)." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "adult.data saved!\n", "adult.test saved!\n" ] } ], "source": [ "from sagemaker.s3 import S3Downloader\n", "\n", "adult_columns = [\n", " \"Age\",\n", " \"Workclass\",\n", " \"fnlwgt\",\n", " \"Education\",\n", " \"Education-Num\",\n", " \"Marital Status\",\n", " \"Occupation\",\n", " \"Relationship\",\n", " \"Ethnic group\",\n", " \"Sex\",\n", " \"Capital Gain\",\n", " \"Capital Loss\",\n", " \"Hours per week\",\n", " \"Country\",\n", " \"Target\",\n", "]\n", "if not os.path.isfile(\"adult.data\"):\n", " S3Downloader.download(\n", " s3_uri=\"s3://{}/{}\".format(\n", " f\"sagemaker-example-files-prod-{region}\", \"datasets/tabular/uci_adult/adult.data\"\n", " ),\n", " local_path=\"./\",\n", " sagemaker_session=sagemaker_session,\n", " )\n", " print(\"adult.data saved!\")\n", "else:\n", " print(\"adult.data already on disk.\")\n", "\n", "if not os.path.isfile(\"adult.test\"):\n", " S3Downloader.download(\n", " s3_uri=\"s3://{}/{}\".format(\n", " f\"sagemaker-example-files-prod-{region}\", \"datasets/tabular/uci_adult/adult.test\"\n", " ),\n", " local_path=\"./\",\n", " sagemaker_session=sagemaker_session,\n", " )\n", " print(\"adult.test saved!\")\n", "else:\n", " print(\"adult.test already on disk.\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### Loading the data: Adult Dataset\n", "From the UCI repository of machine learning datasets, this database contains 14 features concerning demographic characteristics of 45,222 rows (32,561 for training and 12,661 for testing). The task is to predict whether a person has a yearly income that is more or less than $50,000.\n", "\n", "Here are the features and their possible values:\n", "\n", "1. **Age**: continuous.\n", "1. **Workclass**: Private, Self-emp-not-inc, Self-emp-inc, Federal-gov, Local-gov, State-gov, Without-pay, Never-worked.\n", "1. **Fnlwgt**: continuous (the number of people the census takers believe that observation represents).\n", "1. **Education**: Bachelors, Some-college, 11th, HS-grad, Prof-school, Assoc-acdm, Assoc-voc, 9th, 7th-8th, 12th, Masters, 1st-4th, 10th, Doctorate, 5th-6th, Preschool.\n", "1. **Education-num**: continuous.\n", "1. **Marital-status**: Married-civ-spouse, Divorced, Never-married, Separated, Widowed, Married-spouse-absent, Married-AF-spouse.\n", "1. **Occupation**: Tech-support, Craft-repair, Other-service, Sales, Exec-managerial, Prof-specialty, Handlers-cleaners, Machine-op-inspct, Adm-clerical, Farming-fishing, Transport-moving, Priv-house-serv, Protective-serv, Armed-Forces.\n", "1. **Relationship**: Wife, Own-child, Husband, Not-in-family, Other-relative, Unmarried.\n", "1. **Ethnic group**: White, Asian-Pac-Islander, Amer-Indian-Eskimo, Other, Black.\n", "1. **Sex**: Female, Male.\n", " * **Note**: this data is extracted from the 1994 Census and enforces a binary option on Sex\n", "1. **Capital-gain**: continuous.\n", "1. **Capital-loss**: continuous.\n", "1. **Hours-per-week**: continuous.\n", "1. **Native-country**: United-States, Cambodia, England, Puerto-Rico, Canada, Germany, Outlying-US(Guam-USVI-etc), India, Japan, Greece, South, China, Cuba, Iran, Honduras, Philippines, Italy, Poland, Jamaica, Vietnam, Mexico, Portugal, Ireland, France, Dominican-Republic, Laos, Ecuador, Taiwan, Haiti, Columbia, Hungary, Guatemala, Nicaragua, Scotland, Thailand, Yugoslavia, El-Salvador, Trinadad&Tobago, Peru, Hong, Holand-Netherlands.\n", "\n", "Next, we specify our binary prediction task: \n", "\n", "15. **Target**: <=50,000, >$50,000." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "scrolled": true, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
AgeWorkclassfnlwgtEducationEducation-NumMarital StatusOccupationRelationshipEthnic groupSexCapital GainCapital LossHours per weekCountryTarget
039State-gov77516Bachelors13Never-marriedAdm-clericalNot-in-familyWhiteMale2174040United-States<=50K
150Self-emp-not-inc83311Bachelors13Married-civ-spouseExec-managerialHusbandWhiteMale0013United-States<=50K
238Private215646HS-grad9DivorcedHandlers-cleanersNot-in-familyWhiteMale0040United-States<=50K
353Private23472111th7Married-civ-spouseHandlers-cleanersHusbandBlackMale0040United-States<=50K
428Private338409Bachelors13Married-civ-spouseProf-specialtyWifeBlackFemale0040Cuba<=50K
\n", "
" ], "text/plain": [ " Age Workclass fnlwgt Education Education-Num \\\n", "0 39 State-gov 77516 Bachelors 13 \n", "1 50 Self-emp-not-inc 83311 Bachelors 13 \n", "2 38 Private 215646 HS-grad 9 \n", "3 53 Private 234721 11th 7 \n", "4 28 Private 338409 Bachelors 13 \n", "\n", " Marital Status Occupation Relationship Ethnic group Sex \\\n", "0 Never-married Adm-clerical Not-in-family White Male \n", "1 Married-civ-spouse Exec-managerial Husband White Male \n", "2 Divorced Handlers-cleaners Not-in-family White Male \n", "3 Married-civ-spouse Handlers-cleaners Husband Black Male \n", "4 Married-civ-spouse Prof-specialty Wife Black Female \n", "\n", " Capital Gain Capital Loss Hours per week Country Target \n", "0 2174 0 40 United-States <=50K \n", "1 0 0 13 United-States <=50K \n", "2 0 0 40 United-States <=50K \n", "3 0 0 40 United-States <=50K \n", "4 0 0 40 Cuba <=50K " ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "training_data = pd.read_csv(\n", " \"adult.data\", names=adult_columns, sep=r\"\\s*,\\s*\", engine=\"python\", na_values=\"?\"\n", ").dropna()\n", "\n", "testing_data = pd.read_csv(\n", " \"adult.test\", names=adult_columns, sep=r\"\\s*,\\s*\", engine=\"python\", na_values=\"?\", skiprows=1\n", ").dropna()\n", "\n", "training_data.head()" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### Data inspection\n", "Plotting histograms for the distribution of the different features is a good way to visualize the data. Let's plot a few of the features that can be considered _sensitive_. \n", "Let's take a look specifically at the Sex feature of a census respondent. In the first plot we see that there are fewer Female respondents as a whole but especially in the positive outcomes, where they form ~$\\frac{1}{7}$th of respondents." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "scrolled": true, "tags": [] }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjkAAAGxCAYAAACA4KdFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de1RVdf7/8dcR5IgEJxXhcBwUnJkcDTWFUnQctBK8TzOamkVSfWn8opmgq6JypvxOkmaO31HTckybNLUZL9MMxoCXMEfwlpSoY+VA6E8QLwjqKKDu3x8t9rcjeMHBkM3zsdZei70/7/05n89ZHXn12XsfbIZhGAIAALCYJvU9AAAAgFuBkAMAACyJkAMAACyJkAMAACyJkAMAACyJkAMAACyJkAMAACyJkAMAACyJkAMAACyJkAM0cl988YWeeOIJhYaGqlmzZrrjjjvUvXt3zZw5U6dOnarv4UmSPvjgA82ZM6deXvvUqVMaPXq0AgICZLPZ9NBDD121trKyUm+//bbuvfdetWzZUs2bN1e7du3085//XGvXrv0eRw1Akmz8WQeg8Vq0aJESEhLUoUMHJSQkqFOnTqqsrNSuXbu0aNEide3a9bb45TxkyBDl5uYqPz//e3/txMREvfXWW3r33Xf1wx/+UC1bttRdd91VY+3o0aO1Zs0aTZo0SX379pXdbte//vUvpaWlqXXr1lq4cOH3PHqgcSPkAI1UVlaW+vTpo/79+2vdunWy2+1u7RUVFUpLS9OwYcPqaYT/pz5DTv/+/fX//t//0/79+69Zl5eXp/bt2+vXv/61Xn311Wrtly9fVpMmLJ4D3yc+cUAjNX36dNlsNr3zzjvVAo4keXl5uQWcy5cva+bMmfrJT34iu92ugIAAPf744zpy5IjbeSEhIYqLi6vWX9++fdW3b19z/5NPPpHNZtOKFSv00ksvyeVyyc/PTw8++KAOHjzodl5qaqq++eYb2Ww2c6uyYMECde3aVXfccYd8fX31k5/8RC+++OJ153/q1CklJCSoTZs28vLyUvv27fXSSy+pvLxckpSfny+bzaYNGzbowIED5ut+8sknNfZ38uRJSVJQUFCN7VcGnLKyMk2ZMkWhoaHy8vJSmzZtNGnSJJ07d86sWblypWw2m+bNm+d27m9+8xt5eHgoIyPjuvMEGjUDQKNz8eJFo3nz5kaPHj1u+Jynn37akGRMmDDBSEtLMxYuXGi0bt3aCA4ONo4fP27WtWvXzhg7dmy186OiooyoqChzf/PmzYYkIyQkxHj00UeN1NRUY8WKFUbbtm2NH//4x8bFixcNwzCMffv2Gb179zacTqeRlZVlboZhGCtWrDAkGc8884yRnp5ubNiwwVi4cKExceLEa87l/PnzRpcuXQwfHx9j1qxZRnp6ujF16lTD09PTGDRokGEYhnHhwgUjKyvL6Natm9G+fXvzdUtLS2vs8+zZs8add95pOJ1O4+233zby8vKu+vrnzp0z7rnnHsPf39+YPXu2sWHDBuN///d/DYfDYdx///3G5cuXzdpx48YZXl5exs6dOw3DMIyNGzcaTZo0MV5++eVrzhGAYRBygEaoqKjIkGSMHj36huoPHDhgSDISEhLcjm/fvt2QZLz44ovmsdqGnKpQUeXDDz80JJlBxjAMY/DgwUa7du2q9TlhwgTjzjvvvKE5fNfChQsNScaHH37odnzGjBmGJCM9Pd1t3HffffcN9Zuammr4+/sbkgxJRqtWrYyHH37Y+Oijj9zqUlJSjCZNmpjBpcqf//xnQ5Kxfv1689iFCxeMbt26GaGhocb+/fuNwMBAIyoqygyBAK6Oy1UArmvz5s2SVO0y1H333aeOHTtq48aNN933lff8dOnSRZL0zTffXPfc++67T6dPn9Yjjzyiv/zlLzpx4sQNveamTZvk4+OjESNGuB2vmt/NzmfQoEEqKCjQ2rVrNWXKFN19991at26dhg0bpgkTJph1f/vb3xQWFqZ77rlHFy9eNLeYmJhql8Tsdrs+/PBDnTx5Ut27d5dhGFqxYoU8PDxuaoxAY0LIARohf39/NW/eXHl5eTdUf637TVwul9l+M1q1auW2X3V/0Pnz5697bmxsrN5991198803Gj58uAICAtSjR4/r3qty8uRJOZ1Ot3t7JCkgIECenp7/0Xy8vb310EMP6Y033lBmZqa+/vprderUSfPnz9e+ffskSceOHdMXX3yhpk2bum2+vr4yDKNaWPvRj36kPn366MKFC3r00Uevet8PAHeEHKAR8vDw0AMPPKDdu3dXu3G4JlVBpLCwsFrb0aNH5e/vb+43a9bMvHn3u250laW2nnjiCW3btk2lpaVKTU2VYRgaMmTINVeCWrVqpWPHjsm44uHS4uJiXbx40W0+/6m2bdvq6aefliQz5Pj7+6tz587auXNnjdvUqVPd+vjDH/6g1NRU3XfffZo3b562b99eZ+MDrIyQAzRSycnJMgxD8fHxqqioqNZeWVmpv/71r5Kk+++/X5K0bNkyt5qdO3fqwIEDeuCBB8xjISEh+uKLL9zqvvzyS7cnpmrLbrdfd2XHx8dHAwcO1EsvvaSKigozUNTkgQce0NmzZ7Vu3Tq343/84x/N9to6c+aMzp49W2PbgQMHJH276iV9+0j8oUOH1KpVK0VERFTbQkJCzHP37t2riRMn6vHHH9enn36qLl26aNSoUSopKan1GIHGxrO+BwCgfkRGRmrBggVKSEhQeHi4/vu//1t33323KisrtWfPHr3zzjsKCwvT0KFD1aFDBz399NOaO3eumjRpooEDByo/P19Tp05VcHCwEhMTzX5jY2P12GOPKSEhQcOHD9c333yjmTNnqnXr1jc91s6dO2vNmjVasGCBwsPD1aRJE0VERCg+Pl7e3t7q3bu3goKCVFRUpJSUFDkcDt17771X7e/xxx/X/PnzNXbsWOXn56tz587aunWrpk+frkGDBunBBx+s9RgPHjyomJgYjR49WlFRUQoKClJJSYlSU1P1zjvvqG/fvurVq5ckadKkSVq9erV+9rOfKTExUV26dNHly5dVUFCg9PR0TZ48WT169NC5c+c0cuRIhYaG6q233pKXl5c+/PBDde/eXU888US1kAbgCvV62zOAepeTk2OMHTvWaNu2reHl5WX4+PgY3bp1M379618bxcXFZt2lS5eMGTNmGHfddZfRtGlTw9/f33jssceMw4cPu/V3+fJlY+bMmUb79u2NZs2aGREREcamTZuu+nTVn/70J7fz8/LyDEnGkiVLzGOnTp0yRowYYdx5552GzWYzqv7peu+994x+/foZgYGBhpeXl+FyuYyRI0caX3zxxXXnffLkSWPcuHFGUFCQ4enpabRr185ITk42Lly44FZ3o09XlZSUGL/97W+N+++/32jTpo35Xt5zzz3Gb3/7W+Pf//63W/3Zs2eNl19+2ejQoYPh5eVlOBwOo3PnzkZiYqJRVFRkGIZhPPbYY0bz5s2Nffv2uZ37pz/9yZBk/O53v7vuuIDGjG88BgAAlsQ9OQAAwJIIOQAAwJIIOQAAwJIIOQAAwJIIOQAAwJIIOQAAwJIa9ZcBXr58WUePHpWvr2+1v2EDAABuT4Zh6MyZM3K5XGrS5OrrNY065Bw9elTBwcH1PQwAAHATDh8+rB/84AdXbW/UIcfX11fSt2+Sn59fPY8GAADciLKyMgUHB5u/x6+mUYecqktUfn5+hBwAABqY691qwo3HAADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkjzrewAAgLoV8kJqfQ8B36P81wfX9xBuW6zkAAAASyLkAAAAS6pVyElJSdG9994rX19fBQQE6KGHHtLBgwfdasrLy/XMM8/I399fPj4+GjZsmI4cOeJWU1BQoKFDh8rHx0f+/v6aOHGiKioq3GoyMzMVHh6uZs2aqX379lq4cGG18bz11lsKDQ1Vs2bNFB4erk8//bQ20wEAABZWq5CTmZmp8ePHKzs7WxkZGbp48aKio6N17tw5s2bSpElau3atVq5cqa1bt+rs2bMaMmSILl26JEm6dOmSBg8erHPnzmnr1q1auXKlVq9ercmTJ5t95OXladCgQerTp4/27NmjF198URMnTtTq1avNmlWrVmnSpEl66aWXtGfPHvXp00cDBw5UQUHBf/qeAAAAC7AZhmHc7MnHjx9XQECAMjMz9bOf/UylpaVq3bq13n//fY0aNUqSdPToUQUHB2v9+vWKiYnRxx9/rCFDhujw4cNyuVySpJUrVyouLk7FxcXy8/PT888/r48++kgHDhwwX2vcuHH6/PPPlZWVJUnq0aOHunfvrgULFpg1HTt21EMPPaSUlJQbGn9ZWZkcDodKS0vl5+d3s28DANxWuPG4cWmMNx7f6O/v/+ienNLSUklSy5YtJUm7d+9WZWWloqOjzRqXy6WwsDBt27ZNkpSVlaWwsDAz4EhSTEyMysvLtXv3brPmu31U1ezatUuVlZWqqKjQ7t27q9VER0ebr1OT8vJylZWVuW0AAMCabjrkGIahpKQk/fSnP1VYWJgkqaioSF5eXmrRooVbbWBgoIqKisyawMBAt/YWLVrIy8vrmjWBgYG6ePGiTpw4oRMnTujSpUs11lT1UZOUlBQ5HA5zCw4OvrnJAwCA295Nh5wJEyboiy++0IoVK65baxiGbDabuf/dn2+0puqq2vVqauq7SnJyskpLS83t8OHD1x07AABomG4q5DzzzDP66KOPtHnzZv3gBz8wjzudTlVUVKikpMStvri42Fx1cTqd1VZbSkpKVFlZec2a4uJieXp6qlWrVvL395eHh0eNNVeu7nyX3W6Xn5+f2wYAAKypViHHMAxNmDBBa9as0aZNmxQaGurWHh4erqZNmyojI8M8VlhYqNzcXPXq1UuSFBkZqdzcXBUWFpo16enpstvtCg8PN2u+20dVTUREhJo2bSovLy+Fh4dXq8nIyDBfBwAANG61+rMO48eP1wcffKC//OUv8vX1NVdSHA6HvL295XA49NRTT2ny5Mlq1aqVWrZsqSlTpqhz58568MEHJX17c3CnTp0UGxurN954Q6dOndKUKVMUHx9vrqyMGzdO8+bNU1JSkuLj45WVlaXFixe7XRpLSkpSbGysIiIiFBkZqXfeeUcFBQUaN25cXb03AACgAatVyKl6XLtv375ux5csWaK4uDhJ0u9+9zt5enpq5MiROn/+vB544AEtXbpUHh4ekiQPDw+lpqYqISFBvXv3lre3t8aMGaNZs2aZ/YWGhmr9+vVKTEzU/Pnz5XK59Pvf/17Dhw83a0aNGqWTJ09q2rRpKiwsVFhYmNavX6927drdzPsAAAAs5j/6npyGju/JAWBFfE9O48L35Nyi78kBAAC4XRFyAACAJRFyAACAJRFyAACAJRFyAACAJRFyAACAJRFyAACAJRFyAACAJRFyAACAJRFyAACAJRFyAACAJRFyAACAJRFyAACAJRFyAACAJRFyAACAJRFyAACAJRFyAACAJRFyAACAJRFyAACAJRFyAACAJRFyAACAJRFyAACAJRFyAACAJRFyAACAJRFyAACAJRFyAACAJRFyAACAJRFyAACAJRFyAACAJRFyAACAJRFyAACAJRFyAACAJdU65GzZskVDhw6Vy+WSzWbTunXr3NptNluN2xtvvGHWhISEVGt/4YUX3PopKCjQ0KFD5ePjI39/f02cOFEVFRVuNZmZmQoPD1ezZs3Uvn17LVy4sLbTAQAAFuVZ2xPOnTunrl276oknntDw4cOrtRcWFrrtf/zxx3rqqaeq1U6bNk3x8fHm/h133GH+fOnSJQ0ePFitW7fW1q1bdfLkSY0dO1aGYWju3LmSpLy8PA0aNEjx8fFatmyZ/vGPfyghIUGtW7eucVwAAKBxqXXIGThwoAYOHHjVdqfT6bb/l7/8Rf369VP79u3djvv6+larrZKenq79+/fr8OHDcrlckqQ333xTcXFxeu211+Tn56eFCxeqbdu2mjNnjiSpY8eO2rVrl2bNmkXIAQAAt/aenGPHjik1NVVPPfVUtbYZM2aoVatWuueee/Taa6+5XYrKyspSWFiYGXAkKSYmRuXl5dq9e7dZEx0d7dZnTEyMdu3apcrKyhrHU15errKyMrcNAABYU61Xcmrjvffek6+vr375y1+6HX/22WfVvXt3tWjRQjt27FBycrLy8vL0hz/8QZJUVFSkwMBAt3NatGghLy8vFRUVXbUmMDBQFy9e1IkTJxQUFFRtPCkpKXr11VfrcooAAOA2dUtDzrvvvqtHH31UzZo1czuemJho/tylSxe1aNFCI0aMMFd3pG9vYL6SYRhux6+sMQzjqudKUnJyspKSksz9srIyBQcH13JWAACgIbhlIefTTz/VwYMHtWrVquvW9uzZU5L09ddfq1WrVnI6ndq+fbtbTUlJiSorK83VG6fTaa7qVCkuLpanp6cZlK5kt9tlt9tvZjoAAKCBuWX35CxevFjh4eHq2rXrdWv37NkjSeYlpsjISOXm5ro9qZWeni673a7w8HCzJiMjw62f9PR0RUREqGnTpnU1DQAA0EDVeiXn7Nmz+vrrr839vLw85eTkqGXLlmrbtq2kby8D/elPf9Kbb75Z7fysrCxlZ2erX79+cjgc2rlzpxITEzVs2DDz/OjoaHXq1EmxsbF64403dOrUKU2ZMkXx8fHy8/OTJI0bN07z5s1TUlKS4uPjlZWVpcWLF2vFihU39UYAAABrqXXI2bVrl/r162fuV93jMnbsWC1dulSStHLlShmGoUceeaTa+Xa7XatWrdKrr76q8vJytWvXTvHx8XruuefMGg8PD6WmpiohIUG9e/eWt7e3xowZo1mzZpk1oaGhWr9+vRITEzV//ny5XC79/ve/5/FxAAAgSbIZVXfrNkJlZWVyOBwqLS01V4gAoKELeSG1voeA71H+64Prewjfuxv9/c3frgIAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZU65CzZcsWDR06VC6XSzabTevWrXNrj4uLk81mc9t69uzpVlNeXq5nnnlG/v7+8vHx0bBhw3TkyBG3moKCAg0dOlQ+Pj7y9/fXxIkTVVFR4VaTmZmp8PBwNWvWTO3bt9fChQtrOx0AAGBRtQ45586dU9euXTVv3ryr1gwYMECFhYXmtn79erf2SZMmae3atVq5cqW2bt2qs2fPasiQIbp06ZIk6dKlSxo8eLDOnTunrVu3auXKlVq9erUmT55s9pGXl6dBgwapT58+2rNnj1588UVNnDhRq1evru2UAACABXnW9oSBAwdq4MCB16yx2+1yOp01tpWWlmrx4sV6//339eCDD0qSli1bpuDgYG3YsEExMTFKT0/X/v37dfjwYblcLknSm2++qbi4OL322mvy8/PTwoUL1bZtW82ZM0eS1LFjR+3atUuzZs3S8OHDazstAABgMbfknpxPPvlEAQEBuuuuuxQfH6/i4mKzbffu3aqsrFR0dLR5zOVyKSwsTNu2bZMkZWVlKSwszAw4khQTE6Py8nLt3r3brPluH1U1u3btUmVlZY3jKi8vV1lZmdsGAACsqc5DzsCBA7V8+XJt2rRJb775pnbu3Kn7779f5eXlkqSioiJ5eXmpRYsWbucFBgaqqKjIrAkMDHRrb9Gihby8vK5ZExgYqIsXL+rEiRM1ji0lJUUOh8PcgoOD62TOAADg9lPry1XXM2rUKPPnsLAwRUREqF27dkpNTdUvf/nLq55nGIZsNpu5/92fb7TGMIyrnitJycnJSkpKMvfLysoIOgAAWNQtf4Q8KChI7dq101dffSVJcjqdqqioUElJiVtdcXGxuTLjdDrNFZsqJSUlqqysvGZNcXGxPD091apVqxrHYrfb5efn57YBAABruuUh5+TJkzp8+LCCgoIkSeHh4WratKkyMjLMmsLCQuXm5qpXr16SpMjISOXm5qqwsNCsSU9Pl91uV3h4uFnz3T6qaiIiItS0adNbPS0AAHCbq3XIOXv2rHJycpSTkyPp20e5c3JyVFBQoLNnz2rKlCnKyspSfn6+PvnkEw0dOlT+/v76xS9+IUlyOBx66qmnNHnyZG3cuFF79uzRY489ps6dO5tPW0VHR6tTp06KjY3Vnj17tHHjRk2ZMkXx8fHm6su4ceP0zTffKCkpSQcOHNC7776rxYsXa8qUKXX13gAAgAas1vfk7Nq1S/369TP3q+5xGTt2rBYsWKC9e/fqj3/8o06fPq2goCD169dPq1atkq+vr3nO7373O3l6emrkyJE6f/68HnjgAS1dulQeHh6SJA8PD6WmpiohIUG9e/eWt7e3xowZo1mzZpl9hIaGav369UpMTNT8+fPlcrn0+9//nsfHAQCAJMlmVN2t2wiVlZXJ4XCotLSU+3MAWEbIC6n1PQR8j/JfH1zfQ/je3ejvb/52FQAAsCRCDgAAsCRCDgAAsCRCDgAAsCRCDgAAsCRCDgAAsCRCDgAAsCRCDgAAsCRCDgAAsCRCDgAAsCRCDgAAsCRCDgAAsCRCDgAAsCRCDgAAsCRCDgAAsCRCDgAAsCRCDgAAsCRCDgAAsCRCDgAAsCRCDgAAsCRCDgAAsCRCDgAAsCRCDgAAsCRCDgAAsCRCDgAAsCRCDgAAsCRCDgAAsCRCDgAAsCRCDgAAsCRCDgAAsCRCDgAAsCRCDgAAsCRCDgAAsKRah5wtW7Zo6NChcrlcstlsWrdundlWWVmp559/Xp07d5aPj49cLpcef/xxHT161K2PkJAQ2Ww2t+2FF15wqykoKNDQoUPl4+Mjf39/TZw4URUVFW41mZmZCg8PV7NmzdS+fXstXLiwttMBAAAWVeuQc+7cOXXt2lXz5s2r1vbvf/9bn332maZOnarPPvtMa9as0Zdffqlhw4ZVq502bZoKCwvN7eWXXzbbLl26pMGDB+vcuXPaunWrVq5cqdWrV2vy5MlmTV5engYNGqQ+ffpoz549evHFFzVx4kStXr26tlMCAAAW5FnbEwYOHKiBAwfW2OZwOJSRkeF2bO7cubrvvvtUUFCgtm3bmsd9fX3ldDpr7Cc9PV379+/X4cOH5XK5JElvvvmm4uLi9Nprr8nPz08LFy5U27ZtNWfOHElSx44dtWvXLs2aNUvDhw+vsd/y8nKVl5eb+2VlZTc+cQAA0KDc8ntySktLZbPZdOedd7odnzFjhlq1aqV77rlHr732mtulqKysLIWFhZkBR5JiYmJUXl6u3bt3mzXR0dFufcbExGjXrl2qrKyscSwpKSlyOBzmFhwcXFfTBAAAt5lbGnIuXLigF154QWPGjJGfn595/Nlnn9XKlSu1efNmTZgwQXPmzFFCQoLZXlRUpMDAQLe+WrRoIS8vLxUVFV21JjAwUBcvXtSJEydqHE9ycrJKS0vN7fDhw3U1VQAAcJup9eWqG1VZWanRo0fr8uXLeuutt9zaEhMTzZ+7dOmiFi1aaMSIEebqjiTZbLZqfRqG4Xb8yhrDMK56riTZ7XbZ7fabmxAAAGhQbslKTmVlpUaOHKm8vDxlZGS4reLUpGfPnpKkr7/+WpLkdDrNFZsqJSUlqqysNFdvaqopLi6Wp6enGZQAAEDjVechpyrgfPXVV9qwYcMNBY49e/ZIkoKCgiRJkZGRys3NVWFhoVmTnp4uu92u8PBws+bKm5zT09MVERGhpk2b1tV0AABAA1Xry1Vnz541V1ykbx/lzsnJUcuWLeVyuTRixAh99tln+tvf/qZLly6Zqy0tW7aUl5eXsrKylJ2drX79+snhcGjnzp1KTEzUsGHDzKevoqOj1alTJ8XGxuqNN97QqVOnNGXKFMXHx5urQuPGjdO8efOUlJSk+Ph4ZWVlafHixVqxYkVdvC8AAKCBsxlVN7LcoE8++UT9+vWrdnzs2LF65ZVXFBoaWuN5mzdvVt++ffXZZ58pISFB//znP1VeXq527dpp9OjReu6559S8eXOzvqCgQAkJCdq0aZO8vb01ZswYzZo1y+2emszMTCUmJmrfvn1yuVx6/vnnNW7cuBueS1lZmRwOh0pLS697SQ0AGoqQF1Lrewj4HuW/Pri+h/C9u9Hf37UOOVZCyAFgRYScxoWQc/Xf3/ztKgAAYEmEHAAAYEmEHAAAYEmEHAAAYEmEHAAAYEmEHAAAYEmEHAAAYEmEHAAAYEmEHAAAYEmEHAAAYEmEHAAAYEmEHAAAYEmEHAAAYEmEHAAAYEmEHAAAYEmEHAAAYEmEHAAAYEmEHAAAYEmEHAAAYEmEHAAAYEmEHAAAYEmEHAAAYEmEHAAAYEmEHAAAYEmEHAAAYEmEHAAAYEmEHAAAYEmEHAAAYEmEHAAAYEmEHAAAYEmEHAAAYEmEHAAAYEm1DjlbtmzR0KFD5XK5ZLPZtG7dOrd2wzD0yiuvyOVyydvbW3379tW+ffvcakpKShQbGyuHwyGHw6HY2FidPn3arWbv3r2KioqSt7e32rRpo2nTpskwDLea1atXq1OnTrLb7erUqZPWrl1b2+kAAACLqnXIOXfunLp27ap58+bV2D5z5kzNnj1b8+bN086dO+V0OtW/f3+dOXPGrBkzZoxycnKUlpamtLQ05eTkKDY21mwvKytT//795XK5tHPnTs2dO1ezZs3S7NmzzZqsrCyNGjVKsbGx+vzzzxUbG6uRI0dq+/bttZ0SAACwIJtx5fJIbU622bR27Vo99NBDkr5dxXG5XJo0aZKef/55SVJ5ebkCAwM1Y8YM/epXv9KBAwfUqVMnZWdnq0ePHpKk7OxsRUZG6p///Kc6dOigBQsWKDk5WceOHc6UywoAABf/SURBVJPdbpckvf7665o7d66OHDkim82mUaNGqaysTB9//LE5ngEDBqhFixZasWLFDY2/rKxMDodDpaWl8vPzu9m3AQBuKyEvpNb3EPA9yn99cH0P4Xt3o7+/6/SenLy8PBUVFSk6Oto8ZrfbFRUVpW3btkn6dgXG4XCYAUeSevbsKYfD4VYTFRVlBhxJiomJ0dGjR5Wfn2/WfPd1qmqq+qhJeXm5ysrK3DYAAGBNdRpyioqKJEmBgYFuxwMDA822oqIiBQQEVDs3ICDAraamPr77GlerqWqvSUpKinkfkMPhUHBwcG2mBwAAGpBb8nSVzWZz2zcMw+3Yle03UlN1Ve16NTX1XSU5OVmlpaXmdvjw4RuYDQAAaIg867Izp9Mp6dtVlqCgIPN4cXGxueridDp17NixauceP37crebKFZni4mJJum7Nlas732W3290ugQEAAOuq05ATGhoqp9OpjIwMdevWTZJUUVGhzMxMzZgxQ5IUGRmp0tJS7dixQ/fdd58kafv27SotLVWvXr3MmhdffFEVFRXy8vKSJKWnp8vlcikkJMSsycjIUGJiovn66enpZh+4Nm5MbFwa442JAFDry1Vnz55VTk6OcnJyJH17s3FOTo4KCgpks9k0adIkTZ8+XWvXrlVubq7i4uLUvHlzjRkzRpLUsWNHDRgwQPHx8crOzlZ2drbi4+M1ZMgQdejQQdK3j5jb7XbFxcUpNzdXa9eu1fTp05WUlGRejnr22WeVnp6uGTNm6J///KdmzJihDRs2aNKkSXX13gAAgAas1is5u3btUr9+/cz9pKQkSdLYsWO1dOlSPffcczp//rwSEhJUUlKiHj16KD09Xb6+vuY5y5cv18SJE82no4YNG+b2vTsOh0MZGRkaP368IiIi1KJFCyUlJZmvJUm9evXSypUr9fLLL2vq1Kn64Q9/qFWrVrk9tQUAABqv/+h7chq6xvw9OVyualy4XNW48PluXBrj57tevicHAADgdkHIAQAAlkTIAQAAlkTIAQAAlkTIAQAAlkTIAQAAlkTIAQAAlkTIAQAAlkTIAQAAlkTIAQAAlkTIAQAAlkTIAQAAlkTIAQAAlkTIAQAAlkTIAQAAlkTIAQAAlkTIAQAAlkTIAQAAlkTIAQAAlkTIAQAAlkTIAQAAlkTIAQAAlkTIAQAAlkTIAQAAlkTIAQAAlkTIAQAAlkTIAQAAlkTIAQAAlkTIAQAAlkTIAQAAlkTIAQAAlkTIAQAAllTnISckJEQ2m63aNn78eElS3759q7WNHj3arY+SkhLFxsbK4XDI4XAoNjZWp0+fdqvZu3evoqKi5O3trTZt2mjatGkyDKOupwMAABooz7rucOfOnbp06ZK5n5ubq/79++vhhx82j8XHx2vatGnmvre3t1sfY8aM0ZEjR5SWliZJevrppxUbG6u//vWvkqSysjL1799f/fr1086dO/Xll18qLi5OPj4+mjx5cl1PCQAANEB1HnJat27ttv/666/rhz/8oaKiosxjzZs3l9PprPH8AwcOKC0tTdnZ2erRo4ckadGiRYqMjNTBgwfVoUMHLV++XBcuXNDSpUtlt9sVFhamL7/8UrNnz1ZSUpJsNltdTwsAADQwt/SenIqKCi1btkxPPvmkW/BYvny5/P39dffdd2vKlCk6c+aM2ZaVlSWHw2EGHEnq2bOnHA6Htm3bZtZERUXJbrebNTExMTp69Kjy8/OvOp7y8nKVlZW5bQAAwJrqfCXnu9atW6fTp08rLi7OPPboo48qNDRUTqdTubm5Sk5O1ueff66MjAxJUlFRkQICAqr1FRAQoKKiIrMmJCTErT0wMNBsCw0NrXE8KSkpevXVV+tgZgAA4HZ3S0PO4sWLNXDgQLlcLvNYfHy8+XNYWJh+/OMfKyIiQp999pm6d+8uSTVebjIMw+34lTVVNx1f61JVcnKykpKSzP2ysjIFBwfXclYAAKAhuGUh55tvvtGGDRu0Zs2aa9Z1795dTZs21VdffaXu3bvL6XTq2LFj1eqOHz9urtY4nU5zVadKcXGxpP9b0amJ3W53u8QFAACs65bdk7NkyRIFBARo8ODB16zbt2+fKisrFRQUJEmKjIxUaWmpduzYYdZs375dpaWl6tWrl1mzZcsWVVRUmDXp6elyuVzVLmMBAIDG6ZaEnMuXL2vJkiUaO3asPD3/b7Ho0KFDmjZtmnbt2qX8/HytX79eDz/8sLp166bevXtLkjp27KgBAwYoPj5e2dnZys7OVnx8vIYMGaIOHTpI+vYRc7vdrri4OOXm5mrt2rWaPn06T1YBAADTLQk5GzZsUEFBgZ588km3415eXtq4caNiYmLUoUMHTZw4UdHR0dqwYYM8PDzMuuXLl6tz586Kjo5WdHS0unTpovfff99sdzgcysjI0JEjRxQREaGEhAQlJSW53W8DAAAat1tyT050dHSN3z4cHByszMzM657fsmVLLVu27Jo1nTt31pYtW256jAAAwNr421UAAMCSCDkAAMCSCDkAAMCSCDkAAMCSCDkAAMCSCDkAAMCSCDkAAMCSCDkAAMCSCDkAAMCSCDkAAMCSCDkAAMCSCDkAAMCSCDkAAMCSCDkAAMCSCDkAAMCSCDkAAMCSCDkAAMCSCDkAAMCSCDkAAMCSCDkAAMCSCDkAAMCSCDkAAMCSCDkAAMCSCDkAAMCSCDkAAMCSCDkAAMCSCDkAAMCSCDkAAMCSCDkAAMCSCDkAAMCSCDkAAMCSCDkAAMCS6jzkvPLKK7LZbG6b0+k02w3D0CuvvCKXyyVvb2/17dtX+/btc+ujpKREsbGxcjgccjgcio2N1enTp91q9u7dq6ioKHl7e6tNmzaaNm2aDMOo6+kAAIAG6pas5Nx9990qLCw0t71795ptM2fO1OzZszVv3jzt3LlTTqdT/fv315kzZ8yaMWPGKCcnR2lpaUpLS1NOTo5iY2PN9rKyMvXv318ul0s7d+7U3LlzNWvWLM2ePftWTAcAADRAnrekU09Pt9WbKoZhaM6cOXrppZf0y1/+UpL03nvvKTAwUB988IF+9atf6cCBA0pLS1N2drZ69OghSVq0aJEiIyN18OBBdejQQcuXL9eFCxe0dOlS2e12hYWF6csvv9Ts2bOVlJQkm812K6YFAAAakFuykvPVV1/J5XIpNDRUo0eP1r/+9S9JUl5enoqKihQdHW3W2u12RUVFadu2bZKkrKwsORwOM+BIUs+ePeVwONxqoqKiZLfbzZqYmBgdPXpU+fn5Vx1XeXm5ysrK3DYAAGBNdR5yevTooT/+8Y/6+9//rkWLFqmoqEi9evXSyZMnVVRUJEkKDAx0OycwMNBsKyoqUkBAQLV+AwIC3Gpq6qOq7WpSUlLM+3wcDoeCg4NvfqIAAOC2VuchZ+DAgRo+fLg6d+6sBx98UKmpqZK+vSxV5crLSYZhuB2r6XLT9Wqqbjq+1qWq5ORklZaWmtvhw4drMTMAANCQ3PJHyH18fNS5c2d99dVX5n06V662FBcXmysxTqdTx44dq9bP8ePH3Wpq6kOqvkr0XXa7XX5+fm4bAACwplsecsrLy3XgwAEFBQUpNDRUTqdTGRkZZntFRYUyMzPVq1cvSVJkZKRKS0u1Y8cOs2b79u0qLS11q9myZYsqKirMmvT0dLlcLoWEhNzqKQEAgAagzkPOlClTlJmZqby8PG3fvl0jRoxQWVmZxo4dK5vNpkmTJmn69Olau3atcnNzFRcXp+bNm2vMmDGSpI4dO2rAgAGKj49Xdna2srOzFR8fryFDhqhDhw6Svn3E3G63Ky4uTrm5uVq7dq2mT5/Ok1UAAMBU54+QHzlyRI888ohOnDih1q1bq2fPnsrOzla7du0kSc8995zOnz+vhIQElZSUqEePHkpPT5evr6/Zx/LlyzVx4kTzKaxhw4Zp3rx5ZrvD4VBGRobGjx+viIgItWjRQklJSUpKSqrr6QAAgAbKZjTirwkuKyuTw+FQaWlpo7s/J+SF1PoeAr5H+a8Pru8h4HvE57txaYyf7xv9/c3frgIAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZEyAEAAJZU5yEnJSVF9957r3x9fRUQEKCHHnpIBw8edKvp27evbDab2zZ69Gi3mpKSEsXGxsrhcMjhcCg2NlanT592q9m7d6+ioqLk7e2tNm3aaNq0aTIMo66nBAAAGqA6DzmZmZkaP368srOzlZGRoYsXLyo6Olrnzp1zq4uPj1dhYaG5vf32227tY8aMUU5OjtLS0pSWlqacnBzFxsaa7WVlZerfv79cLpd27typuXPnatasWZo9e3ZdTwkAADRAnnXdYVpamtv+kiVLFBAQoN27d+tnP/uZebx58+ZyOp019nHgwAGlpaUpOztbPXr0kCQtWrRIkZGROnjwoDp06KDly5frwoULWrp0qex2u8LCwvTll19q9uzZSkpKks1mq+upAQCABuSW35NTWloqSWrZsqXb8eXLl8vf31933323pkyZojNnzphtWVlZcjgcZsCRpJ49e8rhcGjbtm1mTVRUlOx2u1kTExOjo0ePKj8/v8axlJeXq6yszG0DAADWVOcrOd9lGIaSkpL005/+VGFhYebxRx99VKGhoXI6ncrNzVVycrI+//xzZWRkSJKKiooUEBBQrb+AgAAVFRWZNSEhIW7tgYGBZltoaGi181NSUvTqq6/W1fQAAMBt7JaGnAkTJuiLL77Q1q1b3Y7Hx8ebP4eFhenHP/6xIiIi9Nlnn6l79+6SVOPlJsMw3I5fWVN10/HVLlUlJycrKSnJ3C8rK1NwcHAtZwUAABqCWxZynnnmGX300UfasmWLfvCDH1yztnv37mratKm++uorde/eXU6nU8eOHatWd/z4cXO1xul0mqs6VYqLiyX934rOlex2u9vlLQAAYF11fk+OYRiaMGGC1qxZo02bNtV42ehK+/btU2VlpYKCgiRJkZGRKi0t1Y4dO8ya7du3q7S0VL169TJrtmzZooqKCrMmPT1dLper2mUsAADQ+NR5yBk/fryWLVumDz74QL6+vioqKlJRUZHOnz8vSTp06JCmTZumXbt2KT8/X+vXr9fDDz+sbt26qXfv3pKkjh07asCAAYqPj1d2drays7MVHx+vIUOGqEOHDpK+fcTcbrcrLi5Oubm5Wrt2raZPn86TVQAAQNItCDkLFixQaWmp+vbtq6CgIHNbtWqVJMnLy0sbN25UTEyMOnTooIkTJyo6OlobNmyQh4eH2c/y5cvVuXNnRUdHKzo6Wl26dNH7779vtjscDmVkZOjIkSOKiIhQQkKCkpKS3O65AQAAjVed35NzvW8cDg4OVmZm5nX7admypZYtW3bNms6dO2vLli21Gh8AAGgc+NtVAADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkgg5AADAkhp8yHnrrbcUGhqqZs2aKTw8XJ9++ml9DwkAANwGGnTIWbVqlSZNmqSXXnpJe/bsUZ8+fTRw4EAVFBTU99AAAEA9a9AhZ/bs2Xrqqaf0X//1X+rYsaPmzJmj4OBgLViwoL6HBgAA6plnfQ/gZlVUVGj37t164YUX3I5HR0dr27ZtNZ5TXl6u8vJyc7+0tFSSVFZWdusGepu6XP7v+h4CvkeN8b/xxozPd+PSGD/fVXM2DOOadQ025Jw4cUKXLl1SYGCg2/HAwEAVFRXVeE5KSopeffXVaseDg4NvyRiB24VjTn2PAMCt0pg/32fOnJHD4bhqe4MNOVVsNpvbvmEY1Y5VSU5OVlJSkrl/+fJlnTp1Sq1atbrqObCOsrIyBQcH6/Dhw/Lz86vv4QCoQ3y+GxfDMHTmzBm5XK5r1jXYkOPv7y8PD49qqzbFxcXVVneq2O122e12t2N33nnnLRsjbk9+fn78IwhYFJ/vxuNaKzhVGuyNx15eXgoPD1dGRobb8YyMDPXq1aueRgUAAG4XDXYlR5KSkpIUGxuriIgIRUZG6p133lFBQYHGjRtX30MDAAD1rEGHnFGjRunkyZOaNm2aCgsLFRYWpvXr16tdu3b1PTTchux2u37zm99Uu2QJoOHj842a2IzrPX8FAADQADXYe3IAAACuhZADAAAsiZADAAAsiZADAAAsiZADXEd+fr5sNptycnLqeygAaoHPLgg5uK3ExcXJZrNV277++uv6HhqA70HVvwE1fd9ZQkKCbDab4uLivv+BoUEi5OC2M2DAABUWFrptoaGh9T0sAN+T4OBgrVy5UufPnzePXbhwQStWrFDbtm3rcWRoaAg5uO3Y7XY5nU63zcPDQ4ZhaObMmWrfvr28vb3VtWtX/fnPfzbP++STT2Sz2fT3v/9d3bp1k7e3t+6//34VFxfr448/VseOHeXn56dHHnlE//73v83z0tLS9NOf/lR33nmnWrVqpSFDhujQoUPXHOP+/fs1aNAg3XHHHQoMDFRsbKxOnDhxy94ToDHp3r272rZtqzVr1pjH1qxZo+DgYHXr1s08xmcX10PIQYPx8ssva8mSJVqwYIH27dunxMREPfbYY8rMzHSre+WVVzRv3jxt27ZNhw8f1siRIzVnzhx98MEHSk1NVUZGhubOnWvWnzt3TklJSdq5c6c2btyoJk2a6Be/+IUuX75c4zgKCwsVFRWle+65R7t27VJaWpqOHTumkSNH3tL5A43JE088oSVLlpj77777rp588km3Gj67uC4DuI2MHTvW8PDwMHx8fMxtxIgRxtmzZ41mzZoZ27Ztc6t/6qmnjEceecQwDMPYvHmzIcnYsGGD2Z6SkmJIMg4dOmQe+9WvfmXExMRcdQzFxcWGJGPv3r2GYRhGXl6eIcnYs2ePYRiGMXXqVCM6OtrtnMOHDxuSjIMHD/5nbwDQyI0dO9b4+c9/bhw/ftyw2+1GXl6ekZ+fbzRr1sw4fvy48fOf/9wYO3Zsjefy2cWVGvTfroI19evXTwsWLDD3fXx8tH//fl24cEH9+/d3q62oqHBbvpakLl26mD8HBgaqefPmat++vduxHTt2mPuHDh3S1KlTlZ2drRMnTpj/F1hQUKCwsLBq49u9e7c2b96sO+64o1rboUOHdNddd9VyxgCu5O/vr8GDB+u9996TYRgaPHiw/P393Wr47OJ6CDm47fj4+OhHP/qR27GCggJJUmpqqtq0aePWduUf5GvatKn5s81mc9uvOvbd5eyhQ4cqODhYixYtksvl0uXLlxUWFqaKiooax3f58mUNHTpUM2bMqNYWFBR0AzMEcCOefPJJTZgwQZI0f/78au18dnE9hBw0CJ06dZLdbldBQYGioqLqrN+TJ0/qwIEDevvtt9WnTx9J0tatW695Tvfu3bV69WqFhITI05OPEHCrDBgwwAwsMTExbm18dnEjuPEYDYKvr6+mTJmixMREvffeezp06JD27Nmj+fPn67333rvpflu0aKFWrVrpnXfe0ddff61NmzYpKSnpmueMHz9ep06d0iOPPKIdO3boX//6l9LT0/Xkk0/q0qVLNz0WAO48PDx04MABHThwQB4eHm5tfHZxIwg5aDD+53/+R7/+9a+VkpKijh07KiYmRn/961//o+/QadKkiVauXKndu3crLCxMiYmJeuONN655jsvl0j/+8Q9dunRJMTExCgsL07PPPiuHw6EmTfhIAXXJz89Pfn5+1Y7z2cWNsBmGYdT3IAAAAOoa0RUAAFgSIQcAAFgSIQcAAFgSIQcAAFgSIQcAAFgSIQcAAFgSIQcAAFgSIQcAAFgSIQcAAFgSIQcAAFgSIQcAAFjS/wf/qnzQRo8KOQAAAABJRU5ErkJggg==", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "%matplotlib inline\n", "training_data[\"Sex\"].value_counts().sort_values().plot(kind=\"bar\", title=\"Counts of Sex\", rot=0)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "scrolled": true, "tags": [] }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjAAAAGxCAYAAAB89YyPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de1xVVf7/8fcR5IAoRwQ9iKHiNGMaWoZl6DRoXtC8dLFMLdJsKNNyFP01mVNZU1JmZmWjo1PpTCk1GdVEIXSzHCEvI5MaY+UlcBTxQqCOAur6/dGD/fUEXlATF76ej8d+PDhrf/baa8E58nbfcBljjAAAACxSr7YHAAAAUFMEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQY1FlfffWV7rzzTkVHRyswMFANGzbUFVdcoenTp2vv3r21PTxJ0qJFizRr1qxa2ffevXs1dOhQNWvWTC6XSzfccMNxaysqKvTnP/9ZV155pZo0aaIGDRqoVatWuv7665WWlnYOR31+a926tUaOHFnbwzht3bt319atW6tdt3XrVrlcrmqX1NTUKvX/+te/1KtXLzVs2FCNGzfWTTfdpM2bN1fb54wZM3zajxw5olGjRsnlcunJJ588a/ND3eJf2wMAfg7z58/XmDFj1LZtW/2///f/1L59e1VUVGj16tWaO3eusrOzz4tfvIsWLdL69es1fvz4c77vP/7xj0pLS9Mrr7yiX/ziF2rSpMlxaxMTE/X2229r/Pjxeuyxx+R2u7V582ZlZGRo6dKluvHGG8/hyM9faWlpCgkJqe1h/Kzuv/9+DR8+3Kftl7/8pc/r//znP+revbsuv/xyvfnmmzp06JAeeeQRXXPNNcrNzVXTpk2P2395ebmGDRumd955R3/605907733/izzQB1ggDpmxYoVxs/Pz/Tt29ccOnSoyvqysjLz7rvv1sLIqurfv79p1apVrey7V69epl27diet27x5s5FkHnnkkWrXHzly5GwP7bzwv//9zxw9erS2h3HKvvnmm9Mab35+vhkyZIgJDw83kkz9+vVNVFSUGT58uE/dli1bjCTzzDPPnLTPW265xYSHh5uSkhKnbevWraZ+/frmgQceOG6f+/fvN7169TL169c3ixcvrvFccGHhFBLqnGnTpsnlcmnevHlyu91V1gcEBGjQoEHO66NHj2r69Om65JJL5Ha71axZM91xxx3atm2bz3bHOz3QvXt3de/e3Xn92WefyeVyafHixZoyZYoiIyMVEhKiXr16aePGjT7bpaen6/vvv/c5HF9pzpw5uuyyy9SwYUM1atRIl1xyiR566KGTzn/v3r0aM2aMWrRooYCAALVp00ZTpkxRWVmZpP87bP/RRx8pLy/P2e9nn31WbX979uyRJDVv3rza9fXq+f4zUlpaqkmTJik6OloBAQFq0aKFxo8frwMHDjg1qampcrlcmj17ts+2jz76qPz8/JSVlXXSeb7xxhuKi4tTcHCwGjZsqISEBK1du9anZvXq1Ro6dKhat26toKAgtW7dWsOGDdP333/vU7dgwQK5XC5lZmZq1KhRatq0qRo0aKCysjJNnTpVLpdLGzZs0LBhw+TxeOT1ejVq1CiVlJT49PPT98ipvhckyRijadOmqVWrVgoMDFTnzp2VlZVV5f11PElJSWrVqpUeeOCBKt+HE7npppv0+eef69lnn1VsbKwWLlyoRx99VIcOHTrlPo51+PBhvf/++xo8eLDP0ahWrVqpR48exz3yWVxcrF69eumf//yn3nnnHQ0dOvS09o8LSG0nKOBsOnz4sGnQoIHp0qXLKW9z9913G0nmvvvuMxkZGWbu3LmmadOmJioqyuzatcupa9WqlRkxYkSV7ePj4018fLzz+tNPPzWSTOvWrc1tt91m0tPTzeLFi03Lli3NL3/5S3P48GFjjDEbNmww3bp1MxERESY7O9tZjDFm8eLFRpK5//77TWZmpvnoo4/M3Llzzbhx4044l4MHD5qOHTua4OBgM2PGDJOZmWkefvhh4+/vb6677jpjjDGHDh0y2dnZplOnTqZNmzbOfo/93/Kx9u/fbxo3bmwiIiLMn//8Z7Nly5bj7v/AgQPm8ssvN+Hh4WbmzJnmo48+Ms8//7zxeDzm2muv9TlCMHr0aBMQEGBWrVpljDHm448/NvXq1TN/+MMfTjhHY4x58sknjcvlMqNGjTLvv/++efvtt01cXJwJDg42GzZscOr+/ve/m0ceecSkpaWZZcuWmdTUVBMfH2+aNm3q87N99dVXjSTTokULc/fdd5sPP/zQvPXWW+bw4cPm0UcfNZJM27ZtzSOPPGKysrLMzJkzjdvtNnfeeafPuH76HjnV94IxxkyePNlIMnfffbfJyMgw8+fPNy1btjTNmzf3eX8dz4YNG8zkyZNNmzZtjCRzySWXmMcee8x88803x91m7969RpJ57rnnjDE/vpeP9/OtPFoSFhZm6tevb4KCgky3bt2qHM38z3/+YySZl156qUofkyZNMi6Xyxw8eNCnz+TkZBMTE2M8Ho/54osvTjpXwBhjCDCoUwoLC40kM3To0FOqz8vLM5LMmDFjfNq//PJLI8k89NBDTltNA0xlYKj05ptvGklOSDHm+KeQ7rvvPtO4ceNTmsOx5s6daySZN99806f96aefNpJMZmamz7gvvfTSU+o3PT3dOcVQ+UvslltuMe+9955PXUpKiqlXr54TSiq99dZbRpL54IMPnLZDhw6ZTp06mejoaPP1118br9dr4uPjfX6pVyc/P9/4+/ub+++/36d93759JiIiwgwZMuS42x4+fNjs37/fBAcHm+eff95prwwwd9xxR5VtKgPM9OnTfdrHjBljAgMDfULZ8QLMyd4Le/fuNW6329x6660+ddnZ2UbSKQWYY61cudIkJyebiy66yEgysbGx5tlnnzXbtm2r8v1o2LChufHGG82hQ4dOGGC2b99ukpKSzJtvvmm++OIL8/rrr5urr77aSDLz58936v75z38aSdWeApo2bZqRZLZv326M+b8AU7kc+/4EToZTSLigffrpp5JU5dTQVVddpXbt2unjjz8+7b6PPU0lSR07dpSkKqcvqnPVVVfphx9+0LBhw/Tuu+9q9+7dp7TPTz75RMHBwbr55pt92ivnd7rzue6665Sfn6+0tDRNmjRJl156qd555x0NGjRI9913n1P3/vvvKyYmRpdffrkOHz7sLAkJCVVOU7ndbr355pvas2ePrrjiChljtHjxYvn5+Z1wLEuXLtXhw4d1xx13+OwjMDBQ8fHxPvvYv3+/fv/73+viiy+Wv7+//P391bBhQx04cEB5eXlV+h48ePBx91vdz/PQoUMqKio6yXfv5O+FnJwclZWVaciQIT51V199tVq3bn3S/n/qyiuv1LPPPqv8/Hx9/vnnuvrqq/X000+rZcuWmjhxolPn5+en+fPn6+OPP5bX69W//vUvPfXUU3r33Xd15MgRnz6bN2+uefPm6ZZbbtGvf/1rDR8+XJ9//rk6deqkBx98UIcPH/apP/Z06E/9dF1CQoLcbreSk5O1a9euGs8XFyYCDOqU8PBwNWjQQFu2bDml+hNd3xEZGemsPx1hYWE+ryuvxzl48OBJt01MTNQrr7yi77//XoMHD1azZs3UpUuXk14bsmfPHkVERFT5BdGsWTP5+/uf0XyCgoJ0ww036JlnntGyZcv03XffqX379nrppZe0YcMGSdLOnTv11VdfqX79+j5Lo0aNZIypEsQuvvhiXXPNNTp06JBuu+22415nc6ydO3dK+vGX9E/388Ybb/jsY/jw4Zo9e7Z++9vfaunSpVq5cqVWrVqlpk2bVvtzONH+z+TnebJtK38uXq+3yrbVtZ2qiooKlZSU6IcfftDBgwcVEBCgRo0a+dQMHTpUW7du1fz58xUWFqY1a9bo5ptvVkxMjLZv337C/uvXr69bb71Ve/bs0bfffivp/+Za3Xtt7969crlcaty4sU97r169lJaWpm+//VY9evQ4pVAIEGBQp/j5+alnz55as2ZNlYtwq1P5j+2OHTuqrNu+fbvCw8Od14GBgc6FsMc61aMjNXXnnXdqxYoVKikpUXp6uowxGjBgwAmP4ISFhWnnzp0yxvi0FxUV6fDhwz7zOVMtW7bU3XffLUlOgAkPD1eHDh20atWqapeHH37Yp4+//OUvSk9P11VXXaXZs2fryy+/POl+K+fw1ltvVbuPyj5KSkr0/vvv64EHHtCDDz6onj176sorr1SHDh2O+xygEx01+DlVvg8rw9mxCgsLa9TX4cOHtXTpUo0aNUper1c33HCDioqK9Pzzz2vnzp2aOnVqlW1CQ0N1yy23qFWrVvr73/+uNWvWaPPmzXr88cdPur/K91rlxdy/+MUvFBQUpHXr1lWpXbdunS6++GIFBgZWWdevXz+9++672rRpk3r06FHt9wI4FgEGdc7kyZNljFFSUpLKy8urrK+oqNA//vEPSdK1114rSXrttdd8alatWqW8vDz17NnTaWvdurW++uorn7pvvvmmyt0kNeF2u0/6P/jg4GD169dPU6ZMUXl5uRMWqtOzZ0/t379f77zzjk/7X//6V2d9Te3bt0/79++vdl3laZjIyEhJ0oABA7Rp0yaFhYWpc+fOVZZjT4esW7dO48aN0x133KEvvvhCHTt21K233qri4uITjichIUH+/v7atGlTtfvo3LmzpB/DiDGmyp1of/nLX6qcHqltXbp0kdvt1htvvOHTnpOTc0qnHCUpOztbo0ePVvPmzdW3b1/l5eXpscce03//+19lZmbqzjvvrPKMmp8G3UodO3ZUeHj4SY+EVFRU6I033lB4eLguvvhiSZK/v78GDhyot99+W/v27XNq8/Pz9emnn+qmm246bn8JCQl69913tXnzZvXo0aPG4Q0XFh5khzonLi5Oc+bM0ZgxYxQbG6t7771Xl156qSoqKrR27VrNmzdPMTExGjhwoNq2bau7775bL774ourVq6d+/fpp69atevjhhxUVFaUJEyY4/SYmJur222/XmDFjNHjwYH3//feaPn36CR/KdTIdOnTQ22+/rTlz5ig2Nlb16tVT586dlZSUpKCgIHXr1k3NmzdXYWGhUlJS5PF4dOWVVx63vzvuuEMvvfSSRowYoa1bt6pDhw5avny5pk2bpuuuu069evWq8Rg3btyohIQEDR06VPHx8WrevLmKi4uVnp6uefPmqXv37urataskafz48VqyZIl+85vfaMKECerYsaOOHj2q/Px8ZWZmauLEierSpYsOHDigIUOGKDo6Wn/6058UEBCgN998U1dccYXuvPPOKgHsWK1bt9bjjz+uKVOmaPPmzerbt69CQ0O1c+dOrVy5UsHBwXrssccUEhKi3/zmN3rmmWcUHh6u1q1ba9myZXr55ZernMKobU2aNFFycrJSUlIUGhqqG2+8Udu2bdNjjz2m5s2bV7lVvTqTJ0/W7t27NX78eA0fPlzR0dEn3eb777/X0KFDde+996pjx44qKyvTunXrlJKSou3bt+v66693apOTk1VRUaFu3bopIiJCBQUFevHFF5Wbm6tXX33V59qlxx57TFdeeaUGDBigBx980HmQXXh4uM81ONXp06eP3nvvPV1//fXq0aOHPvnkk1M6tYgLUO1dPwz8vHJzc82IESNMy5YtTUBAgAkODjadOnUyjzzyiCkqKnLqjhw5Yp5++mnzq1/9ytSvX9+Eh4eb22+/3RQUFPj0d/ToUTN9+nTTpk0bExgYaDp37mw++eST496F9Pe//91n+8o7Ll599VWnbe/evebmm282jRs3Ni6Xy1R+JBcuXGh69OhhvF6vCQgIMJGRkWbIkCHmq6++Oum89+zZY0aPHm2aN29u/P39TatWrczkyZOrPNTvVO9CKi4uNk888YS59tprTYsWLZzv5eWXX26eeOIJ87///c+nfv/+/eYPf/iDadu2rQkICDAej8d06NDBTJgwwRQWFhpjjLn99ttNgwYNfG55NubH2551zG29J/LOO++YHj16mJCQEON2u02rVq3MzTffbD766COnZtu2bWbw4MEmNDTUNGrUyPTt29esX7++yt1ClXch/fTuKWP+7y6kY2+7PnabY+/aOd5dSKfyXjh69Kh54oknzEUXXWQCAgJMx44dzfvvv28uu+wyc+ONN570+1F5Z09NHDhwwEydOtVcddVVpkmTJkaSCQ4ONh07djRz5871qX355ZedOn9/fxMaGmoSEhLM0qVLq+179erVpmfPnqZBgwYmJCTE3HDDDea7776r9vtQ3cPxPvroIxMUFGTatm1r/vvf/9Z4bqj7XMYc5xgiAKBWbdmyRZdccokeffTRU3qI4Znq3r27FixYcFp3PgHnGqeQAOA88O9//1uLFy9W165dFRISoo0bN2r69OkKCQnRXXfdVdvDA847BBgAOA8EBwdr9erVevnll/XDDz/I4/Goe/fuevLJJ8/oVuqaGDly5Hl3fRBwPJxCAgAA1uE2agAAYB0CDAAAsA4BBgAAWKfOXsR79OhRbd++XY0aNaq1x4MDAICaMcZo3759ioyMPOFDHOtsgNm+fbuioqJqexgAAOA0FBQU6KKLLjru+jobYCr/4mpBQUGVv/8BAADOT6WlpYqKiqryl9N/qs4GmMrTRiEhIQQYAAAsc7LLP7iIFwAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6/rU9AADAqWv9YHptDwHn0Nan+tf2EM5bHIEBAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1qlxgPnvf/+r22+/XWFhYWrQoIEuv/xyrVmzxllvjNHUqVMVGRmpoKAgde/eXRs2bPDpo7i4WImJifJ4PPJ4PEpMTNQPP/zgU7Nu3TrFx8crKChILVq00OOPPy5jzGlOEwAA1CU1CjDFxcXq1q2b6tevrw8//FBff/21nn32WTVu3NipmT59umbOnKnZs2dr1apVioiIUO/evbVv3z6nZvjw4crNzVVGRoYyMjKUm5urxMREZ31paal69+6tyMhIrVq1Si+++KJmzJihmTNnnoUpAwAA2/nXpPjpp59WVFSUXn31VaetdevWztfGGM2aNUtTpkzRTTfdJElauHChvF6vFi1apHvuuUd5eXnKyMhQTk6OunTpIkmaP3++4uLitHHjRrVt21avv/66Dh06pAULFsjtdismJkbffPONZs6cqeTkZLlcrrMwdQAAYKsaHYF577331LlzZ91yyy1q1qyZOnXqpPnz5zvrt2zZosLCQvXp08dpc7vdio+P14oVKyRJ2dnZ8ng8TniRpKuvvloej8enJj4+Xm6326lJSEjQ9u3btXXr1mrHVlZWptLSUp8FAADUTTUKMJs3b9acOXP0y1/+UkuXLtXo0aM1btw4/fWvf5UkFRYWSpK8Xq/Pdl6v11lXWFioZs2aVem7WbNmPjXV9XHsPn4qJSXFuabG4/EoKiqqJlMDAAAWqVGAOXr0qK644gpNmzZNnTp10j333KOkpCTNmTPHp+6np3iMMT5t1Z0COllN5QW8xzt9NHnyZJWUlDhLQUFBTaYGAAAsUqMA07x5c7Vv396nrV27dsrPz5ckRURESKp6lKSoqMg5ghIREaGdO3dW6XvXrl0+NdX1IVU9ulPJ7XYrJCTEZwEAAHVTjQJMt27dtHHjRp+2b775Rq1atZIkRUdHKyIiQllZWc768vJyLVu2TF27dpUkxcXFqaSkRCtXrnRqvvzyS5WUlPjUfP755yovL3dqMjMzFRkZ6XPRMAAAuDDVKMBMmDBBOTk5mjZtmr777jstWrRI8+bN09ixYyX9eHpn/PjxmjZtmtLS0rR+/XqNHDlSDRo00PDhwyX9eMSmb9++SkpKUk5OjnJycpSUlKQBAwaobdu2kn68zdrtdmvkyJFav3690tLSNG3aNO5AAgAAkmp4G/WVV16ptLQ0TZ48WY8//riio6M1a9Ys3XbbbU7NAw88oIMHD2rMmDEqLi5Wly5dlJmZqUaNGjk1r7/+usaNG+fcrTRo0CDNnj3bWe/xeJSVlaWxY8eqc+fOCg0NVXJyspKTk890vgAAoA5wmTr6eNvS0lJ5PB6VlJRwPQyAOqP1g+m1PQScQ1uf6l/bQzjnTvX3N38LCQAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOvUKMBMnTpVLpfLZ4mIiHDWG2M0depURUZGKigoSN27d9eGDRt8+iguLlZiYqI8Ho88Ho8SExP1ww8/+NSsW7dO8fHxCgoKUosWLfT444/LGHMG0wQAAHVJjY/AXHrppdqxY4ezrFu3zlk3ffp0zZw5U7Nnz9aqVasUERGh3r17a9++fU7N8OHDlZubq4yMDGVkZCg3N1eJiYnO+tLSUvXu3VuRkZFatWqVXnzxRc2YMUMzZ848w6kCAIC6wr/GG/j7+xx1qWSM0axZszRlyhTddNNNkqSFCxfK6/Vq0aJFuueee5SXl6eMjAzl5OSoS5cukqT58+crLi5OGzduVNu2bfX666/r0KFDWrBggdxut2JiYvTNN99o5syZSk5OlsvlOsMpAwAA29X4CMy3336ryMhIRUdHa+jQodq8ebMkacuWLSosLFSfPn2cWrfbrfj4eK1YsUKSlJ2dLY/H44QXSbr66qvl8Xh8auLj4+V2u52ahIQEbd++XVu3bj3uuMrKylRaWuqzAACAuqlGAaZLly7661//qqVLl2r+/PkqLCxU165dtWfPHhUWFkqSvF6vzzZer9dZV1hYqGbNmlXpt1mzZj411fVRue54UlJSnOtqPB6PoqKiajI1AABgkRoFmH79+mnw4MHq0KGDevXqpfT0dEk/niqq9NNTPMYYn7bqTgGdrKbyAt4TnT6aPHmySkpKnKWgoKAGMwMAADY5o9uog4OD1aFDB3377bfOdTE/PUpSVFTkHEGJiIjQzp07q/Sza9cun5rq+pCqHt05ltvtVkhIiM8CAADqpjMKMGVlZcrLy1Pz5s0VHR2tiIgIZWVlOevLy8u1bNkyde3aVZIUFxenkpISrVy50qn58ssvVVJS4lPz+eefq7y83KnJzMxUZGSkWrdufSbDBQAAdUSNAsykSZO0bNkybdmyRV9++aVuvvlmlZaWasSIEXK5XBo/frymTZumtLQ0rV+/XiNHjlSDBg00fPhwSVK7du3Ut29fJSUlKScnRzk5OUpKStKAAQPUtm1bST/eZu12uzVy5EitX79eaWlpmjZtGncgAQAAR41uo962bZuGDRum3bt3q2nTprr66quVk5OjVq1aSZIeeOABHTx4UGPGjFFxcbG6dOmizMxMNWrUyOnj9ddf17hx45y7lQYNGqTZs2c76z0ej7KysjR27Fh17txZoaGhSk5OVnJy8tmYLwAAqANcpo4+4ra0tFQej0clJSVcDwOgzmj9YHptDwHn0Nan+tf2EM65U/39zd9CAgAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDpnFGBSUlLkcrk0fvx4p62srEz333+/wsPDFRwcrEGDBmnbtm0+2+Xn52vgwIEKDg5WeHi4xo0bp/Lycp+aZcuWKTY2VoGBgWrTpo3mzp17JkMFAAB1yGkHmFWrVmnevHnq2LGjT/v48eOVlpam1NRULV++XPv379eAAQN05MgRSdKRI0fUv39/HThwQMuXL1dqaqqWLFmiiRMnOn1s2bJF1113na655hqtXbtWDz30kMaNG6clS5ac7nABAEAd4n86G+3fv1+33Xab5s+fryeeeMJpLykp0csvv6y//e1v6tWrlyTptddeU1RUlD766CMlJCQoMzNTX3/9tQoKChQZGSlJevbZZzVy5Eg9+eSTCgkJ0dy5c9WyZUvNmjVLktSuXTutXr1aM2bM0ODBg890zgAAwHKndQRm7Nix6t+/vxNSKq1Zs0YVFRXq06eP0xYZGamYmBitWLFCkpSdna2YmBgnvEhSQkKCysrKtGbNGqfm2D4qa1avXq2Kiopqx1RWVqbS0lKfBQAA1E01DjCpqan617/+pZSUlCrrCgsLFRAQoNDQUJ92r9erwsJCp8br9fqsDw0NVUBAwAlrvF6vDh8+rN27d1c7rpSUFHk8HmeJioqq6dQAAIAlahRgCgoK9Lvf/U6vvfaaAgMDT3k7Y4xcLpfz+tivT7XGGHPcbSVp8uTJKikpcZaCgoJTHh8AALBLjQLMmjVrVFRUpNjYWPn7+8vf31/Lli3TCy+8IH9/f3m9XpWXl6u4uNhnu6KiIueISkREhHOkpek6YrkAABfKSURBVFJxcbEqKipOWFNUVCR/f3+FhYVVOza3262QkBCfBQAA1E01CjA9e/bUunXrlJub6yydO3fWbbfd5nxdv359ZWVlOdvs2LFD69evV9euXSVJcXFxWr9+vXbs2OHUZGZmyu12KzY21qk5to/Kmsr+AQDAha1GdyE1atRIMTExPm3BwcEKCwtz2u+66y5NnDhRYWFhatKkiSZNmqQOHTo4F/z26dNH7du3V2Jiop555hnt3btXkyZNUlJSknPUZPTo0Zo9e7aSk5OVlJSk7Oxsvfzyy1q8ePHZmDMAALDcad1GfSLPPfec/P39NWTIEB08eFA9e/bUggUL5OfnJ0ny8/NTenq6xowZo27duikoKEjDhw/XjBkznD6io6P1wQcfaMKECXrppZcUGRmpF154gVuoAQCAJMllKq+OrWNKS0vl8XhUUlLC9TAA6ozWD6bX9hBwDm19qn9tD+GcO9Xf3/wtJAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxTowAzZ84cdezYUSEhIQoJCVFcXJw+/PBDZ31ZWZnuv/9+hYeHKzg4WIMGDdK2bdt8+sjPz9fAgQMVHBys8PBwjRs3TuXl5T41y5YtU2xsrAIDA9WmTRvNnTv3DKYIAADqmhoFmIsuukhPPfWUVq9erdWrV+vaa6/V9ddfrw0bNkiSxo8fr7S0NKWmpmr58uXav3+/BgwYoCNHjkiSjhw5ov79++vAgQNavny5UlNTtWTJEk2cONHZx5YtW3Tdddfpmmuu0dq1a/XQQw9p3LhxWrJkyVmcNgAAsJnLGGPOpIMmTZromWee0c0336ymTZvqb3/7m2699VZJ0vbt2xUVFaUPPvhACQkJ+vDDDzVgwAAVFBQoMjJSkpSamqqRI0eqqKhIISEh+v3vf6/33ntPeXl5zj5Gjx6tf//738rOzj7lcZWWlsrj8aikpEQhISFnMkUAOG+0fjC9toeAc2jrU/1rewjn3Kn+/j7ta2COHDmi1NRUHThwQHFxcVqzZo0qKirUp08fpyYyMlIxMTFasWKFJCk7O1sxMTFOeJGkhIQElZWVac2aNU7NsX1U1qxevVoVFRXHHU9ZWZlKS0t9FgAAUDfVOMCsW7dODRs2lNvt1ujRo5WWlqb27dursLBQAQEBCg0N9an3er0qLCyUJBUWFsrr9fqsDw0NVUBAwAlrvF6vDh8+rN27dx93XCkpKfJ4PM4SFRVV06kBAABL1DjAtG3bVrm5ucrJydG9996rESNG6Ouvvz5uvTFGLpfLeX3s16daU3mWq7ptK02ePFklJSXOUlBQcMpzAgAAdvGv6QYBAQG6+OKLJUmdO3fWqlWr9Pzzz+vWW29VeXm5iouLfY7CFBUVqWvXrpKkiIgIffnllz79FRcXq6KiwjnqEhER4RyNObYPf39/hYWFHXdcbrdbbre7ptMBAAAWOuPnwBhjVFZWptjYWNWvX19ZWVnOuh07dmj9+vVOgImLi9P69eu1Y8cOpyYzM1Nut1uxsbFOzbF9VNZ07txZ9evXP9PhAgCAOqBGR2Aeeugh9evXT1FRUdq3b59SU1P12WefKSMjQx6PR3fddZcmTpyosLAwNWnSRJMmTVKHDh3Uq1cvSVKfPn3Uvn17JSYm6plnntHevXs1adIkJSUlOVcajx49WrNnz1ZycrKSkpKUnZ2tl19+WYsXLz77swcAAFaqUYDZuXOnEhMTtWPHDnk8HnXs2FEZGRnq3bu3JOm5556Tv7+/hgwZooMHD6pnz55asGCB/Pz8JEl+fn5KT0/XmDFj1K1bNwUFBWn48OGaMWOGs4/o6Gh98MEHmjBhgl566SVFRkbqhRde0ODBg8/itAEAgM3O+Dkw5yueAwOgLuI5MBcWngPzMzwHBgAAoLYQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGCdGgWYlJQUXXnllWrUqJGaNWumG264QRs3bvSpKSsr0/3336/w8HAFBwdr0KBB2rZtm09Nfn6+Bg4cqODgYIWHh2vcuHEqLy/3qVm2bJliY2MVGBioNm3aaO7cuac5RQAAUNfUKMAsW7ZMY8eOVU5OjrKysnT48GH16dNHBw4ccGrGjx+vtLQ0paamavny5dq/f78GDBigI0eOSJKOHDmi/v3768CBA1q+fLlSU1O1ZMkSTZw40eljy5Ytuu6663TNNddo7dq1euihhzRu3DgtWbLkLE0bAADYzGWMMae78a5du9SsWTMtW7ZMv/nNb1RSUqKmTZvqb3/7m2699VZJ0vbt2xUVFaUPPvhACQkJ+vDDDzVgwAAVFBQoMjJSkpSamqqRI0eqqKhIISEh+v3vf6/33ntPeXl5zr5Gjx6tf//738rOzj6lsZWWlsrj8aikpEQhISGnO0UAOK+0fjC9toeAc2jrU/1rewjn3Kn+/j6ja2BKSkokSU2aNJEkrVmzRhUVFerTp49TExkZqZiYGK1YsUKSlJ2drZiYGCe8SFJCQoLKysq0Zs0ap+bYPiprVq9erYqKimrHUlZWptLSUp8FAADUTacdYIwxSk5O1q9//WvFxMRIkgoLCxUQEKDQ0FCfWq/Xq8LCQqfG6/X6rA8NDVVAQMAJa7xerw4fPqzdu3dXO56UlBR5PB5niYqKOt2pAQCA89xpB5j77rtPX331lRYvXnzSWmOMXC6X8/rYr0+1pvJMV3XbStLkyZNVUlLiLAUFBac0DwAAYJ/TCjD333+/3nvvPX366ae66KKLnPaIiAiVl5eruLjYp76oqMg5ohIREeEcaalUXFysioqKE9YUFRXJ399fYWFh1Y7J7XYrJCTEZwEAAHVTjQKMMUb33Xef3n77bX3yySeKjo72WR8bG6v69esrKyvLaduxY4fWr1+vrl27SpLi4uK0fv167dixw6nJzMyU2+1WbGysU3NsH5U1nTt3Vv369Ws2QwAAUOfUKMCMHTtWr732mhYtWqRGjRqpsLBQhYWFOnjwoCTJ4/Horrvu0sSJE/Xxxx9r7dq1uv3229WhQwf16tVLktSnTx+1b99eiYmJWrt2rT7++GNNmjRJSUlJzlGT0aNH6/vvv1dycrLy8vL0yiuv6OWXX9akSZPO8vQBAICNahRg5syZo5KSEnXv3l3Nmzd3ljfeeMOpee6553TDDTdoyJAh6tatmxo0aKB//OMf8vPzkyT5+fkpPT1dgYGB6tatm4YMGaIbbrhBM2bMcPqIjo7WBx98oM8++0yXX365/vjHP+qFF17Q4MGDz9K0AQCAzc7oOTDnM54DA6Au4jkwFxaeA/MzPQcGAACgNhBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHX8a3sAOPtaP5he20PAObT1qf61PQQAOOc4AgMAAKxT4wDz+eefa+DAgYqMjJTL5dI777zjs94Yo6lTpyoyMlJBQUHq3r27NmzY4FNTXFysxMREeTweeTweJSYm6ocffvCpWbduneLj4xUUFKQWLVro8ccflzHmNKYIAADqmhoHmAMHDuiyyy7T7Nmzq10/ffp0zZw5U7Nnz9aqVasUERGh3r17a9++fU7N8OHDlZubq4yMDGVkZCg3N1eJiYnO+tLSUvXu3VuRkZFatWqVXnzxRc2YMUMzZ848jSkCAIC6psbXwPTr10/9+vWrdp0xRrNmzdKUKVN00003SZIWLlwor9erRYsW6Z577lFeXp4yMjKUk5OjLl26SJLmz5+vuLg4bdy4UW3bttXrr7+uQ4cOacGCBXK73YqJidE333yjmTNnKjk5WS6X6wymDAAAbHdWr4HZsmWLCgsL1adPH6fN7XYrPj5eK1askCRlZ2fL4/E44UWSrr76ank8Hp+a+Ph4ud1upyYhIUHbt2/X1q1bq913WVmZSktLfRYAAFA3ndUAU1hYKEnyer0+7V6v11lXWFioZs2aVdm2WbNmPjXV9XHsPn4qJSXFuabG4/EoKirqzCYDAADOWz/LXUg/PcVjjPFpq+4U0MlqKi/gPd7po8mTJ6ukpMRZCgoKTnv8AADg/HZWnwMTEREh6cejJM2bN3fai4qKnCMoERER2rlzZ5Vtd+3a5VPz0yMtRUVFkqoe3ankdrt9TjkBAIC666wegYmOjlZERISysrKctvLyci1btkxdu3aVJMXFxamkpEQrV650ar788kuVlJT41Hz++ecqLy93ajIzMxUZGanWrVufzSEDAAAL1TjA7N+/X7m5ucrNzZX044W7ubm5ys/Pl8vl0vjx4zVt2jSlpaVp/fr1GjlypBo0aKDhw4dLktq1a6e+ffsqKSlJOTk5ysnJUVJSkgYMGKC2bdtK+vE2a7fbrZEjR2r9+vVKS0vTtGnTuAMJAABIOo1TSKtXr1aPHj2c18nJyZKkESNGaMGCBXrggQd08OBBjRkzRsXFxerSpYsyMzPVqFEjZ5vXX39d48aNc+5WGjRokM9zZTwej7KysjR27Fh17txZoaGhSk5OdvYFAAAubC5TRx9vW1paKo/Ho5KSEoWEhNT2cM4p/hbShYW/hXRh4fN9YbkQP9+n+vubv4UEAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdc7rAPOnP/1J0dHRCgwMVGxsrL744ovaHhIAADgPnLcB5o033tD48eM1ZcoUrV27Vtdcc4369eun/Pz82h4aAACoZedtgJk5c6buuusu/fa3v1W7du00a9YsRUVFac6cObU9NAAAUMv8a3sA1SkvL9eaNWv04IMP+rT36dNHK1asqHabsrIylZWVOa9LSkokSaWlpT/fQM9TR8v+V9tDwDl0Ib7HL2R8vi8sF+Lnu3LOxpgT1p2XAWb37t06cuSIvF6vT7vX61VhYWG126SkpOixxx6r0h4VFfWzjBE4X3hm1fYIAPxcLuTP9759++TxeI67/rwMMJVcLpfPa2NMlbZKkydPVnJysvP66NGj2rt3r8LCwo67DeqO0tJSRUVFqaCgQCEhIbU9HABnEZ/vC4sxRvv27VNkZOQJ687LABMeHi4/P78qR1uKioqqHJWp5Ha75Xa7fdoaN278s40R56eQkBD+gQPqKD7fF44THXmpdF5exBsQEKDY2FhlZWX5tGdlZalr1661NCoAAHC+OC+PwEhScnKyEhMT1blzZ8XFxWnevHnKz8/X6NGja3toAACglp23AebWW2/Vnj179Pjjj2vHjh2KiYnRBx98oFatWtX20HAecrvdevTRR6ucRgRgPz7fqI7LnOw+JQAAgPPMeXkNDAAAwIkQYAAAgHUIMAAAwDoEGAAAYB0CDC5oW7dulcvlUm5ubm0PBUAN8NkFAQbnzMiRI+Vyuaos3333XW0PDcA5UPlvQHXP8xozZoxcLpdGjhx57gcGKxFgcE717dtXO3bs8Fmio6Nre1gAzpGoqCilpqbq4MGDTtuhQ4e0ePFitWzZshZHBtsQYHBOud1uRURE+Cx+fn4yxmj69Olq06aNgoKCdNlll+mtt95ytvvss8/kcrm0dOlSderUSUFBQbr22mtVVFSkDz/8UO3atVNISIiGDRum//3vf852GRkZ+vWvf63GjRsrLCxMAwYM0KZNm044xq+//lrXXXedGjZsKK/Xq8TERO3evftn+54AF5IrrrhCLVu21Ntvv+20vf3224qKilKnTp2cNj67OBkCDM4Lf/jDH/Tqq69qzpw52rBhgyZMmKDbb79dy5Yt86mbOnWqZs+erRUrVqigoEBDhgzRrFmztGjRIqWnpysrK0svvviiU3/gwAElJydr1apV+vjjj1WvXj3deOONOnr0aLXj2LFjh+Lj43X55Zdr9erVysjI0M6dOzVkyJCfdf7AheTOO+/Uq6++6rx+5ZVXNGrUKJ8aPrs4KQOcIyNGjDB+fn4mODjYWW6++Wazf/9+ExgYaFasWOFTf9ddd5lhw4YZY4z59NNPjSTz0UcfOetTUlKMJLNp0yan7Z577jEJCQnHHUNRUZGRZNatW2eMMWbLli1Gklm7dq0xxpiHH37Y9OnTx2ebgoICI8ls3LjxzL4BwAVuxIgR5vrrrze7du0ybrfbbNmyxWzdutUEBgaaXbt2meuvv96MGDGi2m357OKnztu/hYS6qUePHpozZ47zOjg4WF9//bUOHTqk3r17+9SWl5f7HFKWpI4dOzpfe71eNWjQQG3atPFpW7lypfN606ZNevjhh5WTk6Pdu3c7/3vLz89XTExMlfGtWbNGn376qRo2bFhl3aZNm/SrX/2qhjMG8FPh4eHq37+/Fi5cKGOM+vfvr/DwcJ8aPrs4GQIMzqng4GBdfPHFPm35+fmSpPT0dLVo0cJn3U//eFv9+vWdr10ul8/ryrZjDzEPHDhQUVFRmj9/viIjI3X06FHFxMSovLy82vEdPXpUAwcO1NNPP11lXfPmzU9hhgBOxahRo3TfffdJkl566aUq6/ns4mQIMKh17du3l9vtVn5+vuLj489av3v27FFeXp7+/Oc/65prrpEkLV++/ITbXHHFFVqyZIlat24tf38+HsDPpW/fvk4YSUhI8FnHZxengot4UesaNWqkSZMmacKECVq4cKE2bdqktWvX6qWXXtLChQtPu9/Q0FCFhYVp3rx5+u677/TJJ58oOTn5hNuMHTtWe/fu1bBhw7Ry5Upt3rxZmZmZGjVqlI4cOXLaYwHgy8/PT3l5ecrLy5Ofn5/POj67OBUEGJwX/vjHP+qRRx5RSkqK2rVrp4SEBP3jH/84o2fE1KtXT6mpqVqzZo1iYmI0YcIEPfPMMyfcJjIyUv/85z915MgRJSQkKCYmRr/73e/k8XhUrx4fF+BsCgkJUUhISJV2Prs4FS5jjKntQQAAANQEsRQAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1vn/C9LGU0CgA6kAAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "training_data[\"Sex\"].where(training_data[\"Target\"] == \">50K\").value_counts().sort_values().plot(\n", " kind=\"bar\", title=\"Counts of Sex earning >$50K\", rot=0\n", ")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### Encode and Upload the Dataset\n", "Here we encode the training and test data. Encoding input data is not necessary for SageMaker Clarify, but is necessary for the model." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "from sklearn import preprocessing\n", "\n", "\n", "def number_encode_features(df):\n", " result = df.copy()\n", " encoders = {}\n", " for column in result.columns:\n", " if result.dtypes[column] == np.object:\n", " encoders[column] = preprocessing.LabelEncoder()\n", " # print('Column:', column, result[column])\n", " result[column] = encoders[column].fit_transform(result[column].fillna(\"None\"))\n", " return result, encoders\n", "\n", "\n", "training_data, _ = number_encode_features(training_data)\n", "testing_data, _ = number_encode_features(testing_data)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "Then save the testing dataset to a JSON Lines file. The file conforms to [SageMaker JSON Lines dense format](https://docs.aws.amazon.com/sagemaker/latest/dg/cdf-inference.html#common-in-formats), with an additional field to hold the ground truth label." ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "tags": [] }, "outputs": [], "source": [ "import json\n", "\n", "\n", "def dump_to_jsonlines_file(df, filename):\n", " with open(filename, \"w\") as f:\n", " for _, row in df.iterrows():\n", " sample = {\"features\": row[0:-1].tolist(), \"label\": int(row[-1])}\n", " print(json.dumps(sample), file=f)\n", "\n", "\n", "dump_to_jsonlines_file(testing_data, \"test_data.jsonl\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "A quick note about our encoding: the \"Female\" Sex value has been encoded as 0 and \"Male\" as 1." ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\"features\": [25, 2, 226802, 1, 7, 4, 6, 3, 2, 1, 0, 0, 40, 37], \"label\": 0}\n", "{\"features\": [38, 2, 89814, 11, 9, 2, 4, 0, 4, 1, 0, 0, 50, 37], \"label\": 0}\n", "{\"features\": [28, 1, 336951, 7, 12, 2, 10, 0, 4, 1, 0, 0, 40, 37], \"label\": 1}\n", "{\"features\": [44, 2, 160323, 15, 10, 2, 6, 0, 2, 1, 7688, 0, 40, 37], \"label\": 1}\n", "{\"features\": [34, 2, 198693, 0, 6, 4, 7, 1, 4, 1, 0, 0, 30, 37], \"label\": 0}\n" ] } ], "source": [ "!head -n 5 test_data.jsonl" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
AgeWorkclassfnlwgtEducationEducation-NumMarital StatusOccupationRelationshipEthnic groupSexCapital GainCapital LossHours per weekCountryTarget
025222680217463210040370
138289814119240410050370
22813369517122100410040371
34421603231510260217688040371
534219869306471410030370
\n", "
" ], "text/plain": [ " Age Workclass fnlwgt Education Education-Num Marital Status \\\n", "0 25 2 226802 1 7 4 \n", "1 38 2 89814 11 9 2 \n", "2 28 1 336951 7 12 2 \n", "3 44 2 160323 15 10 2 \n", "5 34 2 198693 0 6 4 \n", "\n", " Occupation Relationship Ethnic group Sex Capital Gain Capital Loss \\\n", "0 6 3 2 1 0 0 \n", "1 4 0 4 1 0 0 \n", "2 10 0 4 1 0 0 \n", "3 6 0 2 1 7688 0 \n", "5 7 1 4 1 0 0 \n", "\n", " Hours per week Country Target \n", "0 40 37 0 \n", "1 50 37 0 \n", "2 40 37 1 \n", "3 40 37 1 \n", "5 30 37 0 " ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "testing_data.head()" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "Lastly, let's upload the data to S3" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "tags": [] }, "outputs": [], "source": [ "from sagemaker.s3 import S3Uploader\n", "\n", "test_data_uri = S3Uploader.upload(\"test_data.jsonl\", \"s3://{}/{}\".format(bucket, prefix))" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### Train Linear Learner Model\n", "#### Train Model\n", "Since our focus is on understanding how to use SageMaker Clarify, we keep it simple by using a standard Linear Learner model.\n", "\n", "It takes about 5 minutes for the model to be trained." ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "INFO:sagemaker.image_uris:Same images used for training and inference. Defaulting to image scope: inference.\n", "INFO:sagemaker.image_uris:Ignoring unnecessary instance type: None.\n", "INFO:sagemaker:Creating training-job with name: linear-learner-2023-02-07-03-39-06-446\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "2023-02-07 03:39:06 Starting - Starting the training job..\n", "2023-02-07 03:39:21 Starting - Preparing the instances for training..........\n", "2023-02-07 03:40:13 Downloading - Downloading input data....\n", "2023-02-07 03:40:38 Training - Downloading the training image..........\n", "2023-02-07 03:41:34 Training - Training image download completed. Training in progress....\n", "2023-02-07 03:41:54 Uploading - Uploading generated training model.\n", "2023-02-07 03:42:06 Completed - Training job completed\n" ] } ], "source": [ "from sagemaker.image_uris import retrieve\n", "from sagemaker.amazon.linear_learner import LinearLearner\n", "\n", "ll = LinearLearner(\n", " role,\n", " instance_count=1,\n", " instance_type=\"ml.m5.xlarge\",\n", " predictor_type=\"binary_classifier\",\n", " sagemaker_session=sagemaker_session,\n", ")\n", "training_target = training_data[\"Target\"].to_numpy().astype(np.float32)\n", "training_features = training_data.drop([\"Target\"], axis=1).to_numpy().astype(np.float32)\n", "ll.fit(ll.record_set(training_features, training_target), logs=False)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "#### Create Model\n", "Here we create the SageMaker model." ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "INFO:sagemaker.image_uris:Same images used for training and inference. Defaulting to image scope: inference.\n", "INFO:sagemaker.image_uris:Ignoring unnecessary instance type: None.\n", "INFO:sagemaker:Creating model with name: DEMO-clarify-ll-model-07-02-2023-03-42-08\n" ] }, { "data": { "text/plain": [ "'DEMO-clarify-ll-model-07-02-2023-03-42-08'" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model_name = \"DEMO-clarify-ll-model-{}\".format(datetime.now().strftime(\"%d-%m-%Y-%H-%M-%S\"))\n", "model = ll.create_model(name=model_name)\n", "container_def = model.prepare_container_def()\n", "sagemaker_session.create_model(model_name, role, container_def)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Amazon SageMaker Clarify\n", "With your model set up, it's time to explore SageMaker Clarify. For a general overview of how SageMaker Clarify processing jobs work, refer [the provided link](https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-processing-job-configure-how-it-works.html)." ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "INFO:sagemaker.image_uris:Defaulting to the only supported framework/algorithm version: 1.0.\n", "INFO:sagemaker.image_uris:Ignoring unnecessary instance type: None.\n" ] } ], "source": [ "from sagemaker import clarify\n", "\n", "# Initialize a SageMakerClarifyProcessor to compute bias metrics and model explanations.\n", "clarify_processor = clarify.SageMakerClarifyProcessor(\n", " role=role, instance_count=1, instance_type=\"ml.m5.xlarge\", sagemaker_session=sagemaker_session\n", ")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### Detecting Bias\n", "SageMaker Clarify helps you detect possible [pre-training](https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-detect-data-bias.html) and [post-training](https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-detect-post-training-bias.html) biases using a variety of metrics.\n", "\n", "#### Writing DataConfig\n", "\n", "A [DataConfig](https://sagemaker.readthedocs.io/en/stable/api/training/processing.html#sagemaker.clarify.DataConfig) object communicates some basic information about data I/O to SageMaker Clarify. For our example here we provide the below information:\n", "\n", "* `s3_data_input_path`: S3 URI of the train dataset we uploaded above\n", "* `s3_output_path`: S3 URI at which our output report will be uploaded\n", "* `headers`: The list of column names in the dataset. SageMaker Clarify will load the JSON Lines dataset into tabular representation for further analysis, and argument `headers` is the list of column names. The label header should be the last one in the headers list, and the order of feature headers should be the same as the order of features in a sample.\n", "* `dataset_type`: specifies the format of your dataset, for this example as we are using JSON Lines dataset this will be `application/jsonlines`\n", "* `label`: Specifies the ground truth label, which is also known as observed label or target attribute.\n", "* `features`: JMESPath expression to locate the feature columns for bias metrics if the dataset format is JSON Lines.\n", "\n", "**Note:** Argument `features` or `label` above are **NOT** header string. Instead, it is a [JMESPath string](https://jmespath.org/specification.html) to locate the features list or label in the dataset. For example, for a sample like below, `features` should be `data.features.values`, and `label` should be `data.label`. \n", "\n", "```\n", "{\"data\": {\"features\": {\"values\": [25, 2, 226802, 1, 7, 4, 6, 3, 2, 1, 0, 0, 40, 37]}, \"label\": 0}}\n", "```" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "tags": [] }, "outputs": [], "source": [ "bias_report_output_path = \"s3://{}/{}/clarify-bias\".format(bucket, prefix)\n", "bias_data_config = clarify.DataConfig(\n", " s3_data_input_path=test_data_uri,\n", " s3_output_path=bias_report_output_path,\n", " features=\"features\",\n", " label=\"label\",\n", " headers=testing_data.columns.to_list(),\n", " dataset_type=\"application/jsonlines\",\n", ")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "#### Writing ModelConfig\n", "\n", "A [ModelConfig](https://sagemaker.readthedocs.io/en/stable/api/training/processing.html#sagemaker.clarify.ModelConfig) object communicates information about your trained model. To avoid additional traffic to the production models, SageMaker Clarify sets up and tears down a dedicated endpoint when processing. For our example here we provide the below information:\n", "\n", "* `model_name`: name of the concerned model, using name of the linear learner model trained earlier\n", "* `instance_type` and `initial_instance_count` specify your preferred instance type and instance count used to run your model on during SageMaker Clarify's processing. The example dataset is small, so a single standard instance is good enough to run this example.\n", "* `accept_type` denotes the endpoint response payload format, and `content_type` denotes the payload format of request to the endpoint. As per the example model we created above both of these will be `application/jsonlines`.\n", "* `content_template` is used by SageMaker Clarify to compose the request payload if the content type is JSON Lines. To be more specific, the placeholder `$features` will be replaced by the features list from samples. The request payload of a sample from the testing dataset happens to be similar to the sample itself, like `'{\"features\": [25, 2, 226802, 1, 7, 4, 6, 3, 2, 1, 0, 0, 40, 37]}'`, because both the dataset and the model input conform to [SageMaker JSON Lines dense format](https://docs.aws.amazon.com/sagemaker/latest/dg/cdf-inference.html#common-in-formats)." ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "tags": [] }, "outputs": [], "source": [ "model_config = clarify.ModelConfig(\n", " model_name=model_name,\n", " instance_type=\"ml.m5.xlarge\",\n", " instance_count=1,\n", " accept_type=\"application/jsonlines\",\n", " content_type=\"application/jsonlines\",\n", " content_template='{\"features\":$features}',\n", ")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "#### Writing ModelPredictedLabelConfig\n", "\n", "A `ModelPredictedLabelConfig` provides information on the format of your predictions. The argument `label` is a JMESPath string to locate the predicted label in endpoint response. In this case, the response payload for a single sample request looks like `'{\"predicted_label\": 0, \"score\": 0.013525663875043}'`, so SageMaker Clarify can find predicted label `0` by JMESPath `'predicted_label'`. There is also probability score in the response, so it is possible to use another combination of arguments to decide the predicted label by a custom threshold, for example `probability='score'` and `probability_threshold=0.8`." ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "tags": [] }, "outputs": [], "source": [ "predictions_config = clarify.ModelPredictedLabelConfig(label=\"predicted_label\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "If you are building your own model, then you may choose a different JSON Lines format, as long as it has the key elements like label and features list, and request payload built using `content_template` is supported by the model (you can customize the template but the placeholder of features list must be `$features`). Also, `dataset_type`, `accept_type` and `content_type` don't have to be the same, for example, a use case may use CSV dataset and content type, but JSON Lines accept type." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "#### Writing BiasConfig\n", "[BiasConfig](https://sagemaker.readthedocs.io/en/stable/api/training/processing.html#sagemaker.clarify.BiasConfig) contains configuration values for detecting bias using a Clarify container." ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "tags": [] }, "outputs": [], "source": [ "bias_config = clarify.BiasConfig(\n", " label_values_or_threshold=[1], facet_name=\"Sex\", facet_values_or_threshold=[0], group_name=\"Age\"\n", ")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "For our demo we provide the following information in BiasConfig API:\n", "\n", "* `label_values_or_threshold`: List of label value(s) or threshold to indicate positive outcome used for bias metrics. Here positive outcome is earning >$50,000.\n", "* `facet_name`: Sensitive columns of the dataset, \"Sex\" is the category\n", "* `facet_values_or_threshold`: values of the sensitive group, \"Female\" respondents are the sensitive group.\n", "* `group_name`: This example has selected the \"Age\" column which is used to form subgroups for the measurement of bias metric [Conditional Demographic Disparity (CDD)](https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-data-bias-metric-cddl.html) or [Conditional Demographic Disparity in Predicted Labels (CDDPL)](https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-post-training-bias-metric-cddpl.html).\n", "\n", "SageMaker Clarify can handle both categorical and continuous data for `facet: values_or_threshold` and for `label_values_or_threshold`. In this case we are using categorical data. The results will show if the model has a preference for records of one sex over the other." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "#### Pre-training Bias\n", "Bias can be present in your data before any model training occurs. Inspecting your data for bias before training begins can help detect any data collection gaps, inform your feature engineering, and help you understand what societal biases the data may reflect.\n", "\n", "Computing pre-training bias metrics does not require a trained model.\n", "\n", "#### Post-training Bias\n", "Computing post-training bias metrics does require a trained model.\n", "\n", "Unbiased training data (as determined by concepts of fairness measured by bias metric) may still result in biased model predictions after training. Whether this occurs depends on several factors including hyperparameter choices.\n", "\n", "\n", "You can run these options separately with `run_pre_training_bias()` and `run_post_training_bias()` or at the same time with `run_bias()` as shown below. We use following additional parameters for the api call:\n", "* `pre_training_methods`: Pre-training bias metrics to be computed. The detailed description of the metrics can be found on [Measure Pre-training Bias](https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-measure-data-bias.html). This example sets methods to \"all\" to compute all the pre-training bias metrics.\n", "* `post_training_methods`: Post-training bias metrics to be computed. The detailed description of the metrics can be found on [Measure Post-training Bias](https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-detect-post-training-bias.html). This example sets methods to \"all\" to compute all the post-training bias metrics." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "# The job takes about 10 minutes to run\n", "clarify_processor.run_bias(\n", " data_config=bias_data_config,\n", " bias_config=bias_config,\n", " model_config=model_config,\n", " model_predicted_label_config=predictions_config,\n", " pre_training_methods=\"all\",\n", " post_training_methods=\"all\",\n", ")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "#### Viewing the Bias Report\n", "In Studio, you can view the results under the experiments tab.\n", "\n", "\n", "\n", "Each bias metric has detailed explanations with examples that you can explore.\n", "\n", "\n", "\n", "You could also summarize the results in a handy table!\n", "\n", "\n" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "If you're not a Studio user yet, you can access the bias report in PDF, HTML and ipynb formats in the following S3 bucket:" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "'s3://sagemaker-us-west-2-000000000000/sagemaker/DEMO-sagemaker-clarify-jsonlines/clarify-bias'" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "bias_report_output_path" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### Explaining Predictions\n", "There are expanding business needs and legislative regulations that require explanations of _why_ a model made the decision it did. SageMaker Clarify uses Kernel SHAP to explain the contribution that each input feature makes to the final decision.\n", "\n", "For run_explainability API call we need similar `DataConfig` and `ModelConfig` objects we defined above. [SHAPConfig](https://sagemaker.readthedocs.io/en/stable/api/training/processing.html#sagemaker.clarify.SHAPConfig) here is the config class for Kernel SHAP algorithm.\n", "\n", "For our demo we pass the following information in `SHAPConfig`:\n", "\n", "* `baseline`: Kernel SHAP algorithm requires a baseline (also known as background dataset). If not provided, a baseline is calculated automatically by SageMaker Clarify using K-means or K-prototypes in the input dataset. Baseline dataset type shall be the same as dataset_type, and baseline samples shall only include features. By definition, baseline should either be a S3 URI to the baseline dataset file, or an in-place list of samples. In this case we chose the latter, and use the mean of our dataset as baseline. For more details on baseline selection please [refer this documentation](https://docs.aws.amazon.com/en_us/sagemaker/latest/dg/clarify-feature-attribute-shap-baselines.html).\n", "* `num_samples`: Number of samples to be used in the Kernel SHAP algorithm. This number determines the size of the generated synthetic dataset to compute the SHAP values. \n", "* `agg_method`: Aggregation method for global SHAP values. For our example here we are using `mean_abs` i.e. mean of absolute SHAP values for all instances\n", "* `save_local_shap_values`: Indicates whether to save the local SHAP values in the output location." ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['Age', 'Workclass', 'fnlwgt', 'Education', 'Education-Num', 'Marital Status', 'Occupation', 'Relationship', 'Ethnic group', 'Sex', 'Capital Gain', 'Capital Loss', 'Hours per week', 'Country']\n" ] } ], "source": [ "# Similarly, excluding label header from headers list\n", "headers = testing_data.columns.to_list()\n", "headers.remove(\"Target\")\n", "print(headers)\n", "\n", "explainability_output_path = \"s3://{}/{}/clarify-explainability\".format(bucket, prefix)\n", "explainability_data_config = clarify.DataConfig(\n", " s3_data_input_path=test_data_uri,\n", " s3_output_path=explainability_output_path,\n", " features=\"features\",\n", " headers=headers,\n", " dataset_type=\"application/jsonlines\",\n", ")" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "tags": [] }, "outputs": [], "source": [ "baseline_record = testing_data.mean().iloc[:-1].round().values.tolist()\n", "baseline = {\"features\": baseline_record}\n", "\n", "shap_config = clarify.SHAPConfig(\n", " baseline=[baseline], num_samples=15, agg_method=\"mean_abs\", save_local_shap_values=False\n", ")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "Run the explainability job, note that Kernel SHAP algorithm requires probability prediction, so JMESPath `\"score\"` is used to extract the probability." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "# The job takes about 10 minutes to run\n", "clarify_processor.run_explainability(\n", " data_config=explainability_data_config,\n", " model_config=model_config,\n", " explainability_config=shap_config,\n", " model_scores=\"score\",\n", ")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "#### Viewing the Explainability Report\n", "As with the bias report, you can view the explainability report in Studio under the experiments tab\n", "\n", "\n", "\n", "\n", "The Model Insights tab contains direct links to the report and model insights.\n", "\n", "If you're not a Studio user yet, as with the Bias Report, you can access this report at the following S3 bucket." ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "'s3://sagemaker-us-west-2-000000000000/sagemaker/DEMO-sagemaker-clarify-jsonlines/clarify-explainability'" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "explainability_output_path" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "**Note:** You can run both bias and explainability jobs at the same time with `run_bias_and_explainability()`, refer [API Documentation](https://sagemaker.readthedocs.io/en/stable/api/training/processing.html#sagemaker.clarify.SageMakerClarifyProcessor.run_bias_and_explainability) for more details." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### Clean Up\n", "Finally, don't forget to clean up the resources we set up and used for this demo!" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "INFO:sagemaker:Deleting model with name: DEMO-clarify-ll-model-07-02-2023-03-42-08\n" ] } ], "source": [ "sagemaker_session.delete_model(model_name)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Notebook CI Test Results\n", "\n", "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n", "\n", "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-east-1/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_jsonlines_format.ipynb)\n", "\n", "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-east-2/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_jsonlines_format.ipynb)\n", "\n", "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-west-1/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_jsonlines_format.ipynb)\n", "\n", "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ca-central-1/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_jsonlines_format.ipynb)\n", "\n", "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/sa-east-1/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_jsonlines_format.ipynb)\n", "\n", "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-west-1/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_jsonlines_format.ipynb)\n", "\n", "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-west-2/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_jsonlines_format.ipynb)\n", "\n", "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-west-3/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_jsonlines_format.ipynb)\n", "\n", "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-central-1/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_jsonlines_format.ipynb)\n", "\n", "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-north-1/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_jsonlines_format.ipynb)\n", "\n", "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-southeast-1/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_jsonlines_format.ipynb)\n", "\n", "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-southeast-2/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_jsonlines_format.ipynb)\n", "\n", "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-northeast-1/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_jsonlines_format.ipynb)\n", "\n", "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-northeast-2/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_jsonlines_format.ipynb)\n", "\n", "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-south-1/sagemaker-clarify|fairness_and_explainability|fairness_and_explainability_jsonlines_format.ipynb)\n" ] } ], "metadata": { "instance_type": "ml.t3.medium", "kernelspec": { "display_name": "Python 3 (Data Science 3.0)", "language": "python", "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-data-science-310-v1" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.6" } }, "nbformat": 4, "nbformat_minor": 4 }