{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"collapsed": true,
"id": "3112b95d-4e53-40ca-9934-531bf6ebcb56"
},
"source": [
"# Notebook for generating training data distribution and configuring Fairness\n",
"\n",
"This notebook analyzes training data and outputs a JSON which contains information related to data distribution and fairness configuration. In order to use this notebook you need to do the following:\n",
"\n",
"1. Read the training data into a pandas dataframe called \"data_df\". \n",
"2. Edit the below cells and provide the training data and fairness configuration information. \n",
"3. Run the notebook. It will generate a JSON and a download link for the JSON will be present at the very end of the notebook.\n",
"4. Download the JSON by clicking on the link and upload it in the IBM AI OpenScale GUI.\n",
"\n",
"If you have multiple models (deployments), you will have to repeat the above steps for each model (deployment).\n",
"\n",
"**Note:** Please restart the kernel after executing below cell"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"id": "02149ed4-01c9-4d65-bc4e-ea435b71e101"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: pandas in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (1.3.4)\n",
"Requirement already satisfied: python-dateutil>=2.7.3 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from pandas) (2.8.2)\n",
"Requirement already satisfied: pytz>=2017.3 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from pandas) (2021.3)\n",
"Requirement already satisfied: numpy>=1.17.3 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from pandas) (1.20.3)\n",
"Requirement already satisfied: six>=1.5 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from python-dateutil>=2.7.3->pandas) (1.15.0)\n",
"Requirement already satisfied: ibm-cos-sdk in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (2.11.0)\n",
"Requirement already satisfied: ibm-cos-sdk-core==2.11.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from ibm-cos-sdk) (2.11.0)\n",
"Requirement already satisfied: ibm-cos-sdk-s3transfer==2.11.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from ibm-cos-sdk) (2.11.0)\n",
"Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from ibm-cos-sdk) (0.10.0)\n",
"Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from ibm-cos-sdk-core==2.11.0->ibm-cos-sdk) (2.8.2)\n",
"Requirement already satisfied: requests<3.0,>=2.26 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from ibm-cos-sdk-core==2.11.0->ibm-cos-sdk) (2.26.0)\n",
"Requirement already satisfied: urllib3<1.27,>=1.26.7 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from ibm-cos-sdk-core==2.11.0->ibm-cos-sdk) (1.26.7)\n",
"Requirement already satisfied: six>=1.5 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->ibm-cos-sdk-core==2.11.0->ibm-cos-sdk) (1.15.0)\n",
"Requirement already satisfied: charset-normalizer~=2.0.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from requests<3.0,>=2.26->ibm-cos-sdk-core==2.11.0->ibm-cos-sdk) (2.0.4)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from requests<3.0,>=2.26->ibm-cos-sdk-core==2.11.0->ibm-cos-sdk) (2021.10.8)\n",
"Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from requests<3.0,>=2.26->ibm-cos-sdk-core==2.11.0->ibm-cos-sdk) (3.3)\n",
"Requirement already satisfied: numpy in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (1.20.3)\n",
"Collecting scikit-learn==0.24.1\n",
" Downloading scikit_learn-0.24.1-cp39-cp39-manylinux2010_x86_64.whl (23.8 MB)\n",
"\u001b[K |████████████████████████████████| 23.8 MB 4.2 MB/s eta 0:00:01\n",
"\u001b[?25hRequirement already satisfied: joblib>=0.11 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from scikit-learn==0.24.1) (0.17.0)\n",
"Requirement already satisfied: scipy>=0.19.1 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from scikit-learn==0.24.1) (1.7.3)\n",
"Requirement already satisfied: numpy>=1.13.3 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from scikit-learn==0.24.1) (1.20.3)\n",
"Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from scikit-learn==0.24.1) (2.2.0)\n",
"Installing collected packages: scikit-learn\n",
" Attempting uninstall: scikit-learn\n",
" Found existing installation: scikit-learn 1.0.2\n",
" Uninstalling scikit-learn-1.0.2:\n",
" Successfully uninstalled scikit-learn-1.0.2\n",
"Successfully installed scikit-learn-0.24.1\n",
"Collecting pyspark\n",
" Downloading pyspark-3.3.0.tar.gz (281.3 MB)\n",
"\u001b[K |████████████████████████████████| 281.3 MB 137 kB/s eta 0:00:01\n",
"\u001b[?25hCollecting py4j==0.10.9.5\n",
" Downloading py4j-0.10.9.5-py2.py3-none-any.whl (199 kB)\n",
"\u001b[K |████████████████████████████████| 199 kB 99.7 MB/s eta 0:00:01\n",
"\u001b[?25hBuilding wheels for collected packages: pyspark\n",
" Building wheel for pyspark (setup.py) ... \u001b[?25ldone\n",
"\u001b[?25h Created wheel for pyspark: filename=pyspark-3.3.0-py2.py3-none-any.whl size=281764027 sha256=71e7ad3f13443a688e8771947c68f2416776071382982d1c0b6cc764b33c2812\n",
" Stored in directory: /tmp/1000660000/.cache/pip/wheels/05/75/73/81f84d174299abca38dd6a06a5b98b08ae25fce50ab8986fa1\n",
"Successfully built pyspark\n",
"Installing collected packages: py4j, pyspark\n",
"Successfully installed py4j-0.10.9.5 pyspark-3.3.0\n",
"Collecting lime\n",
" Downloading lime-0.2.0.1.tar.gz (275 kB)\n",
"\u001b[K |████████████████████████████████| 275 kB 4.5 MB/s eta 0:00:01\n",
"\u001b[?25hRequirement already satisfied: matplotlib in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from lime) (3.5.0)\n",
"Requirement already satisfied: numpy in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from lime) (1.20.3)\n",
"Requirement already satisfied: scipy in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from lime) (1.7.3)\n",
"Requirement already satisfied: tqdm in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from lime) (4.62.3)\n",
"Requirement already satisfied: scikit-learn>=0.18 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from lime) (0.24.1)\n",
"Requirement already satisfied: scikit-image>=0.12 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from lime) (0.18.3)\n",
"Requirement already satisfied: networkx>=2.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from scikit-image>=0.12->lime) (2.6.3)\n",
"Requirement already satisfied: pillow!=7.1.0,!=7.1.1,>=4.3.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from scikit-image>=0.12->lime) (9.0.1)\n",
"Requirement already satisfied: imageio>=2.3.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from scikit-image>=0.12->lime) (2.9.0)\n",
"Requirement already satisfied: tifffile>=2019.7.26 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from scikit-image>=0.12->lime) (2021.7.2)\n",
"Requirement already satisfied: PyWavelets>=1.1.1 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from scikit-image>=0.12->lime) (1.1.1)\n",
"Requirement already satisfied: python-dateutil>=2.7 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from matplotlib->lime) (2.8.2)\n",
"Requirement already satisfied: cycler>=0.10 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from matplotlib->lime) (0.11.0)\n",
"Requirement already satisfied: pyparsing>=2.2.1 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from matplotlib->lime) (3.0.4)\n",
"Requirement already satisfied: fonttools>=4.22.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from matplotlib->lime) (4.25.0)\n",
"Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from matplotlib->lime) (1.3.1)\n",
"Requirement already satisfied: packaging>=20.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from matplotlib->lime) (21.3)\n",
"Requirement already satisfied: six>=1.5 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from python-dateutil>=2.7->matplotlib->lime) (1.15.0)\n",
"Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from scikit-learn>=0.18->lime) (2.2.0)\n",
"Requirement already satisfied: joblib>=0.11 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from scikit-learn>=0.18->lime) (0.17.0)\n",
"Building wheels for collected packages: lime\n",
" Building wheel for lime (setup.py) ... \u001b[?25ldone\n",
"\u001b[?25h Created wheel for lime: filename=lime-0.2.0.1-py3-none-any.whl size=283857 sha256=9d50cdc60211137f741277b6b9c09e689090a33cc08e120949ce75239a5e7292\n",
" Stored in directory: /tmp/1000660000/.cache/pip/wheels/ed/d7/c9/5a0130d06d6310bc6cbe55220e6e72dcb8c4eff9a478717066\n",
"Successfully built lime\n",
"Installing collected packages: lime\n",
"Successfully installed lime-0.2.0.1\n",
"Requirement already satisfied: ibm-watson-openscale in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (3.0.17)\n",
"Collecting ibm-watson-openscale\n",
" Downloading ibm_watson_openscale-3.0.24-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.2 MB)\n",
"\u001b[K |████████████████████████████████| 1.2 MB 4.7 MB/s eta 0:00:01\n",
"\u001b[?25hRequirement already satisfied: python-dateutil>=2.5.3 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from ibm-watson-openscale) (2.8.2)\n",
"Requirement already satisfied: requests<3.0,>=2.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from ibm-watson-openscale) (2.26.0)\n",
"Requirement already satisfied: pandas>=0.25.3 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from ibm-watson-openscale) (1.3.4)\n",
"Requirement already satisfied: ibm-cloud-sdk-core==3.10.1 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from ibm-watson-openscale) (3.10.1)\n",
"Requirement already satisfied: PyJWT<3.0.0,>=2.0.1 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from ibm-cloud-sdk-core==3.10.1->ibm-watson-openscale) (2.1.0)\n",
"Requirement already satisfied: pytz>=2017.3 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from pandas>=0.25.3->ibm-watson-openscale) (2021.3)\n",
"Requirement already satisfied: numpy>=1.17.3 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from pandas>=0.25.3->ibm-watson-openscale) (1.20.3)\n",
"Requirement already satisfied: six>=1.5 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from python-dateutil>=2.5.3->ibm-watson-openscale) (1.15.0)\n",
"Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from requests<3.0,>=2.0->ibm-watson-openscale) (1.26.7)\n",
"Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from requests<3.0,>=2.0->ibm-watson-openscale) (3.3)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from requests<3.0,>=2.0->ibm-watson-openscale) (2021.10.8)\n",
"Requirement already satisfied: charset-normalizer~=2.0.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from requests<3.0,>=2.0->ibm-watson-openscale) (2.0.4)\n",
"Installing collected packages: ibm-watson-openscale\n",
" Attempting uninstall: ibm-watson-openscale\n",
" Found existing installation: ibm-watson-openscale 3.0.17\n",
" Uninstalling ibm-watson-openscale-3.0.17:\n",
" Successfully uninstalled ibm-watson-openscale-3.0.17\n",
"Successfully installed ibm-watson-openscale-3.0.24\n",
"Collecting ibm-wos-utils==4.1.1\n",
" Downloading ibm_wos_utils-4.1.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (9.1 MB)\n",
"\u001b[K |████████████████████████████████| 9.1 MB 4.5 MB/s eta 0:00:01\n",
"\u001b[?25hCollecting jenkspy\n",
" Downloading jenkspy-0.3.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (523 kB)\n",
"\u001b[K |████████████████████████████████| 523 kB 80.2 MB/s eta 0:00:01\n",
"\u001b[?25hRequirement already satisfied: pandas in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from ibm-wos-utils==4.1.1) (1.3.4)\n",
"Collecting nltk\n",
" Downloading nltk-3.7-py3-none-any.whl (1.5 MB)\n",
"\u001b[K |████████████████████████████████| 1.5 MB 39.3 MB/s eta 0:00:01\n",
"\u001b[?25hCollecting more-itertools>=8.6.0\n",
" Downloading more_itertools-8.14.0-py3-none-any.whl (52 kB)\n",
"\u001b[K |████████████████████████████████| 52 kB 2.6 MB/s eta 0:00:01\n",
"\u001b[?25hRequirement already satisfied: requests in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from ibm-wos-utils==4.1.1) (2.26.0)\n",
"Requirement already satisfied: scikit-image in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from ibm-wos-utils==4.1.1) (0.18.3)\n",
"Collecting scikit-learn==1.0.2\n",
" Downloading scikit_learn-1.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (26.4 MB)\n",
"\u001b[K |████████████████████████████████| 26.4 MB 95.3 MB/s eta 0:00:01\n",
"\u001b[?25hRequirement already satisfied: numpy in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from ibm-wos-utils==4.1.1) (1.20.3)\n",
"Requirement already satisfied: tqdm in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from ibm-wos-utils==4.1.1) (4.62.3)\n",
"Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from scikit-learn==1.0.2->ibm-wos-utils==4.1.1) (2.2.0)\n",
"Requirement already satisfied: scipy>=1.1.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from scikit-learn==1.0.2->ibm-wos-utils==4.1.1) (1.7.3)\n",
"Requirement already satisfied: joblib>=0.11 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from scikit-learn==1.0.2->ibm-wos-utils==4.1.1) (0.17.0)\n",
"Requirement already satisfied: click in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from nltk->ibm-wos-utils==4.1.1) (8.0.3)\n",
"Requirement already satisfied: regex>=2021.8.3 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from nltk->ibm-wos-utils==4.1.1) (2021.11.2)\n",
"Requirement already satisfied: python-dateutil>=2.7.3 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from pandas->ibm-wos-utils==4.1.1) (2.8.2)\n",
"Requirement already satisfied: pytz>=2017.3 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from pandas->ibm-wos-utils==4.1.1) (2021.3)\n",
"Requirement already satisfied: six>=1.5 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from python-dateutil>=2.7.3->pandas->ibm-wos-utils==4.1.1) (1.15.0)\n",
"Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from requests->ibm-wos-utils==4.1.1) (3.3)\n",
"Requirement already satisfied: charset-normalizer~=2.0.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from requests->ibm-wos-utils==4.1.1) (2.0.4)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from requests->ibm-wos-utils==4.1.1) (2021.10.8)\n",
"Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from requests->ibm-wos-utils==4.1.1) (1.26.7)\n",
"Requirement already satisfied: matplotlib!=3.0.0,>=2.0.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from scikit-image->ibm-wos-utils==4.1.1) (3.5.0)\n",
"Requirement already satisfied: networkx>=2.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from scikit-image->ibm-wos-utils==4.1.1) (2.6.3)\n",
"Requirement already satisfied: pillow!=7.1.0,!=7.1.1,>=4.3.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from scikit-image->ibm-wos-utils==4.1.1) (9.0.1)\n",
"Requirement already satisfied: imageio>=2.3.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from scikit-image->ibm-wos-utils==4.1.1) (2.9.0)\n",
"Requirement already satisfied: tifffile>=2019.7.26 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from scikit-image->ibm-wos-utils==4.1.1) (2021.7.2)\n",
"Requirement already satisfied: PyWavelets>=1.1.1 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from scikit-image->ibm-wos-utils==4.1.1) (1.1.1)\n",
"Requirement already satisfied: pyparsing>=2.2.1 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image->ibm-wos-utils==4.1.1) (3.0.4)\n",
"Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image->ibm-wos-utils==4.1.1) (1.3.1)\n",
"Requirement already satisfied: cycler>=0.10 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image->ibm-wos-utils==4.1.1) (0.11.0)\n",
"Requirement already satisfied: packaging>=20.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image->ibm-wos-utils==4.1.1) (21.3)\n",
"Requirement already satisfied: fonttools>=4.22.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image->ibm-wos-utils==4.1.1) (4.25.0)\n",
"Installing collected packages: scikit-learn, nltk, more-itertools, jenkspy, ibm-wos-utils\n",
" Attempting uninstall: scikit-learn\n",
" Found existing installation: scikit-learn 0.24.1\n",
" Uninstalling scikit-learn-0.24.1:\n",
" Successfully uninstalled scikit-learn-0.24.1\n",
"Successfully installed ibm-wos-utils-4.1.1 jenkspy-0.3.1 more-itertools-8.14.0 nltk-3.7 scikit-learn-1.0.2\n"
]
}
],
"source": [
"!pip install pandas\n",
"!pip install ibm-cos-sdk\n",
"!pip install numpy\n",
"!pip install scikit-learn==0.24.1 \n",
"!pip install pyspark\n",
"!pip install lime\n",
"!pip install --upgrade ibm-watson-openscale\n",
"!pip install \"ibm-wos-utils==4.1.1\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f90c1d27-0d3a-4ebe-9fc5-86a877756c85"
},
"source": [
"# Read training data into a pandas data frame"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"id": "3f5ac70d587d411d8dd7a75e1a0ae951"
},
"outputs": [],
"source": [
"VERSION = \"5.0.1\""
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"id": "57698194-fcf6-43ae-9c3e-6a9f86f8e32b"
},
"outputs": [],
"source": [
"import pandas as pd\n",
"data_df=pd.read_csv('https://ibm-aws-immersion-day.s3.us-east-2.amazonaws.com/publicdata/Data-region-RI-SM-encoded.csv')"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"id": "87317635df74441f85f18ebd1cf24fe1"
},
"outputs": [
{
"data": {
"text/plain": [
"REGION int64\n",
"Total_cases int64\n",
"Risk_Index int64\n",
"dtype: object"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data_df.dtypes"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dd6d8871-689b-4783-af32-ba9e9aae1860"
},
"source": [
"# Select the services for which configuration information needs to be generated\n",
"\n",
"This notebook has support to generaton configuration information related to fairness , explainability and drift service. The below can be used by the user to control service specific configuration information.\n",
"\n",
"Details of the service speicifc flags available:\n",
"\n",
"- enable_fairness : Flag to allow generation of fairness specific data distribution needed for configuration\n",
"- enable_explainability : Flag to allow generation of explainability specific information\n",
"- enable_drift: Flag to allow generation of drift detection model needed by drift service\n",
"\n",
"\n",
"service_configuration_support = {
\n",
" \"enable_fairness\": True, \n",
" \"enable_explainability\": True, \n",
" \"enable_drift\": False \n",
" } \n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"id": "51e4bdd950b742be89da60aebdb41fa2"
},
"outputs": [],
"source": [
"data_df['REGION'] = pd.to_numeric(data_df['REGION'])\n",
"data_df['Total_cases'] = pd.to_numeric(data_df['Total_cases'])\n",
"data_df['Risk_Index'] = pd.to_numeric(data_df['Risk_Index'])"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"id": "04ce6ae3ea67447e81b202a06a405de9"
},
"outputs": [
{
"data": {
"text/plain": [
"REGION int64\n",
"Total_cases int64\n",
"Risk_Index int64\n",
"dtype: object"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data_df.dtypes"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"id": "c3101683-cf10-47c0-bb9f-d07e7a6aaaa9"
},
"outputs": [],
"source": [
"service_configuration_support = {\n",
" \"enable_fairness\": True,\n",
" \"enable_explainability\": True,\n",
" \"enable_drift\": True\n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c083a482-77d4-4a81-8514-077df848aa21"
},
"source": [
"# Training Data and Fairness Configuration Information\n",
"\n",
"Please provide information about the training data which is used to train the model. "
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"id": "969fe4a4-1060-4809-ab5e-135e46eab24f"
},
"outputs": [],
"source": [
"training_data_info = {\n",
" \"class_label\": \"Risk_Index\",\n",
" \"feature_columns\": ['REGION','Total_cases'],\n",
" \"categorical_columns\": ['REGION']\n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "10f682bf-4bf8-439c-b247-99d840a89aa7"
},
"source": [
"# Specify the Model Type\n",
"\n",
"In the next cell, specify the type of your model. If your model is a binary classification model, then set the type to \"binary\". If it is a multi-class classifier then set the type to \"multiclass\". If it is a regression model (e.g., Linear Regression), then set it to \"regression\"."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"id": "5bcccecd-1ae4-4e0a-a17b-d54b1c2d69cd"
},
"outputs": [],
"source": [
"#Set model_type. Acceptable values are:[\"binary\",\"multiclass\",\"regression\"]\n",
"model_type = \"multiclass\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fd3bc3cf-e0f1-4b1f-8e6f-a0f58c0372c7"
},
"source": [
"# Specify the Fairness Configuration\n",
"\n",
"You need to provide the following information for the fairness configuration: \n",
"\n",
"- fairness_attributes: These are the attributes on which you wish to monitor fairness. \n",
"- With Indirect Bias support, you can also monitor protected attributes for fairness. The protected attributes are those attributes which are present in the training data but are not used to train the model. To check if there exists indirect bias with respect to some protected attribute due to possible correlation with some feature column, it can be specified in fairness configuration.\n",
"- type: The data type of the fairness attribute (e.g., float or int or double)\n",
"- minority: The minority group for which we want to ensure that the model is not biased. "
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"id": "95b3bbd0-fb54-46dd-8c52-fb0dab72a93b"
},
"outputs": [],
"source": [
"fairness_attributes = [{\n",
" \"type\" : \"int\", #data type of the column eg: float or int or double\n",
" \"feature\": \"REGION\", \n",
" \"majority\": [\n",
" [1,1] # range of values for column eg: [31, 45] for int or [31.4, 45.1] for float\n",
" ],\n",
" \"minority\": [\n",
" [0,0], # range of values for column eg: [80, 100] for int or [80.0, 99.9] for float \n",
" ],\n",
" \"threshold\": 0.9 \n",
" }]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4c5fae13-458f-40b7-b327-60fd3950d1f3"
},
"source": [
"# Specify the Favorable and Unfavorable class values\n",
"\n",
"The second part of fairness configuration is about the favourable and unfavourable class values. In other words in order to measure fairness, we need to know the target field values which can be considered as being favourable and those values which can be considered as unfavourable. "
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"id": "b1530a2a-dcc3-4a8e-9138-934fbf7c0b69"
},
"outputs": [],
"source": [
"# For classification models use the below.\n",
"parameters = {\n",
" \"favourable_class\" : [0],\n",
" \"unfavourable_class\": [2]\n",
" }"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2b0a28bf-900c-40e0-b280-f9aa245ec9af"
},
"source": [
"# Specify the number of records which should be processed for Fairness\n",
"\n",
"The final piece of information that needs to be provided is the number of records (min_records) that should be used for computing the fairness."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"id": "0c71844b-deeb-4429-a662-f3590a7fc5d1"
},
"outputs": [],
"source": [
"# min_records = \n",
"min_records = 10"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "eb2c48ef-7a26-4090-b4d4-ef5bfe06ae70"
},
"source": [
"# End of Input \n",
"\n",
"You need not edit anything beyond this point. Run the notebook and go to the very last cell. There will be a link to download the JSON file (called: \"Download training data distribution JSON file\"). Download the file and upload it using the IBM AI OpenScale GUI.\n",
"\n",
"*Note: drop_na parameter of TrainingStats object should be set to 'False' if NA values are taken care while reading the training data in the above cells*"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"id": "b89d8f5d-7ec9-439a-a57d-51c40ab72014"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: ibm-wos-utils in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (4.1.1)\n",
"Requirement already satisfied: nltk in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from ibm-wos-utils) (3.7)\n",
"Requirement already satisfied: tqdm in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from ibm-wos-utils) (4.62.3)\n",
"Requirement already satisfied: numpy in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from ibm-wos-utils) (1.20.3)\n",
"Requirement already satisfied: jenkspy in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from ibm-wos-utils) (0.3.1)\n",
"Requirement already satisfied: scikit-learn==1.0.2 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from ibm-wos-utils) (1.0.2)\n",
"Requirement already satisfied: more-itertools>=8.6.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from ibm-wos-utils) (8.14.0)\n",
"Requirement already satisfied: requests in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from ibm-wos-utils) (2.26.0)\n",
"Requirement already satisfied: scikit-image in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from ibm-wos-utils) (0.18.3)\n",
"Requirement already satisfied: pandas in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from ibm-wos-utils) (1.3.4)\n",
"Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from scikit-learn==1.0.2->ibm-wos-utils) (2.2.0)\n",
"Requirement already satisfied: scipy>=1.1.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from scikit-learn==1.0.2->ibm-wos-utils) (1.7.3)\n",
"Requirement already satisfied: joblib>=0.11 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from scikit-learn==1.0.2->ibm-wos-utils) (0.17.0)\n",
"Requirement already satisfied: regex>=2021.8.3 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from nltk->ibm-wos-utils) (2021.11.2)\n",
"Requirement already satisfied: click in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from nltk->ibm-wos-utils) (8.0.3)\n",
"Requirement already satisfied: python-dateutil>=2.7.3 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from pandas->ibm-wos-utils) (2.8.2)\n",
"Requirement already satisfied: pytz>=2017.3 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from pandas->ibm-wos-utils) (2021.3)\n",
"Requirement already satisfied: six>=1.5 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from python-dateutil>=2.7.3->pandas->ibm-wos-utils) (1.15.0)\n",
"Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from requests->ibm-wos-utils) (1.26.7)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from requests->ibm-wos-utils) (2021.10.8)\n",
"Requirement already satisfied: charset-normalizer~=2.0.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from requests->ibm-wos-utils) (2.0.4)\n",
"Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from requests->ibm-wos-utils) (3.3)\n",
"Requirement already satisfied: matplotlib!=3.0.0,>=2.0.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from scikit-image->ibm-wos-utils) (3.5.0)\n",
"Requirement already satisfied: networkx>=2.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from scikit-image->ibm-wos-utils) (2.6.3)\n",
"Requirement already satisfied: pillow!=7.1.0,!=7.1.1,>=4.3.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from scikit-image->ibm-wos-utils) (9.0.1)\n",
"Requirement already satisfied: imageio>=2.3.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from scikit-image->ibm-wos-utils) (2.9.0)\n",
"Requirement already satisfied: tifffile>=2019.7.26 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from scikit-image->ibm-wos-utils) (2021.7.2)\n",
"Requirement already satisfied: PyWavelets>=1.1.1 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from scikit-image->ibm-wos-utils) (1.1.1)\n",
"Requirement already satisfied: pyparsing>=2.2.1 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image->ibm-wos-utils) (3.0.4)\n",
"Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image->ibm-wos-utils) (1.3.1)\n",
"Requirement already satisfied: packaging>=20.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image->ibm-wos-utils) (21.3)\n",
"Requirement already satisfied: cycler>=0.10 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image->ibm-wos-utils) (0.11.0)\n",
"Requirement already satisfied: fonttools>=4.22.0 in /opt/conda/envs/Python-3.9/lib/python3.9/site-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image->ibm-wos-utils) (4.25.0)\n"
]
}
],
"source": [
"from ibm_watson_openscale.utils.training_stats import TrainingStats\n",
"\n",
"enable_explainability = service_configuration_support.get('enable_explainability')\n",
"enable_fairness = service_configuration_support.get('enable_fairness')\n",
"\n",
"if enable_explainability or enable_fairness:\n",
" fairness_inputs = None\n",
" if enable_fairness:\n",
" fairness_inputs = {\n",
" \"fairness_attributes\": fairness_attributes,\n",
" \"min_records\" : min_records,\n",
" \"favourable_class\" : parameters[\"favourable_class\"],\n",
" \"unfavourable_class\": parameters[\"unfavourable_class\"]\n",
" }\n",
" \n",
" input_parameters = {\n",
" \"label_column\": training_data_info[\"class_label\"],\n",
" \"feature_columns\": training_data_info[\"feature_columns\"],\n",
" \"categorical_columns\": training_data_info[\"categorical_columns\"],\n",
" \"fairness_inputs\": fairness_inputs, \n",
" \"problem_type\" : model_type \n",
" }\n",
"\n",
" training_stats = TrainingStats(data_df,input_parameters, explain=enable_explainability, fairness=enable_fairness, drop_na=True)\n",
" config_json = training_stats.get_training_statistics()\n",
" config_json[\"notebook_version\"] = VERSION\n",
"#print(config_json)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5677fd74-4bb7-470c-85bd-38e7a5b946eb"
},
"source": [
"### Indirect Bias\n",
"In case of Indirect bias i.e if protected attributes(the sensitive attributes like race, gender etc which are present in the training data but are not used to train the model) are being monitored for fairness:\n",
"- Bias service identifies correlations between the protected attribute and model features. Correlated attributes are also known as proxy features.\n",
"- Existence of correlations with model features can result in indirect bias w.r.t protected attribute even though it is not used to train the model.\n",
"- Highly correlated attributes based on their correlation strength are considered while computing bias for a given protected attribute.\n",
"\n",
"The following cell identifies if user has configured protected attribute for fairness by checking the feature, non-feature columns and the fairness configuration. If protected attribute/s are configured then it identifies correlations and stores it in the fairness configuration."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"id": "a2c90389-6611-4f5b-9d9a-47b2e38943af"
},
"outputs": [],
"source": [
"# Checking if protected attributes are configured for fairness monitoring. If yes, then computing correlation information for each meta-field and updating it in the fairness configuration\n",
"if enable_fairness:\n",
" fairness_configuration = config_json.get('fairness_configuration')\n",
" training_columns = data_df.columns.tolist()\n",
" label_column = training_data_info.get('class_label')\n",
" training_columns.remove(label_column)\n",
" feature_columns = training_data_info.get('feature_columns')\n",
" non_feature_columns = list(set(training_columns) - set(feature_columns))\n",
" if non_feature_columns is not None and len(non_feature_columns) > 0:\n",
" protected_attributes = []\n",
" fairness_attributes_list = [attribute.get('feature') for attribute in fairness_attributes]\n",
" for col in non_feature_columns:\n",
" if col in fairness_attributes_list:\n",
" protected_attributes.append(col)\n",
" if len(protected_attributes) > 0:\n",
" from ibm_watson_openscale.utils.indirect_bias_processor import IndirectBiasProcessor\n",
" fairness_configuration = IndirectBiasProcessor().get_correlated_attributes(data_df, fairness_configuration, feature_columns, protected_attributes, label_column) "
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"id": "c0653cd1-a5ab-45cf-b33b-d6a1be5051ce"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Finished generating training distribution data\n"
]
},
{
"data": {
"text/html": [
"Download training data distribution JSON file"
],
"text/plain": [
""
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import json\n",
"\n",
"print(\"Finished generating training distribution data\")\n",
"\n",
"# Create a file download link\n",
"import base64\n",
"from IPython.display import HTML\n",
"\n",
"def create_download_link( title = \"Download training data distribution JSON file\", filename = \"training_distribution.json\"): \n",
" if enable_explainability or enable_fairness:\n",
" output_json = json.dumps(config_json, indent=2)\n",
" b64 = base64.b64encode(output_json.encode())\n",
" payload = b64.decode()\n",
" html = '{title}'\n",
" html = html.format(payload=payload,title=title,filename=filename)\n",
" return HTML(html)\n",
" else:\n",
" print('No download link generated as fairness/explainability services are disabled.')\n",
"\n",
"create_download_link()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "95d14d0a-98ae-4458-a7b9-11dcb54fbe16"
},
"source": [
"# Drift detection model generation\n",
"\n",
"Please update the score function which will be used forgenerating drift detection model which will used for drift detection . This might take sometime to generate model and time taken depends on the training dataset size. The output of the score function should be a 2 arrays 1. Array of model prediction 2. Array of probabilities \n",
"\n",
"- User is expected to make sure that the data type of the \"class label\" column selected and the prediction column are same . For eg : If class label is numeric , the prediction array should also be numeric\n",
"\n",
"- Each entry of a probability array should have all the probabities of the unique class lable .\n",
" For eg: If the model_type=multiclass and unique class labels are A, B, C, D . Each entry in the probability array should be a array of size 4 . Eg : [ [0.50,0.30,0.10,0.10] ,[0.40,0.20,0.30,0.10]...]\n",
" \n",
"**Note:**\n",
"- *User is expected to add \"score\" method , which should output prediction column array and probability column array.*\n",
"- *The data type of the label column and prediction column should be same . User needs to make sure that label column and prediction column array should have the same unique class labels*\n",
"- **Please update the score function below with the help of templates documented [here](https://github.com/IBM-Watson/aios-data-distribution/blob/master/Score%20function%20templates%20for%20drift%20detection.md)**"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "df95ed8ce11f4290b1fd820b49e58435"
},
"source": [
"## Update SageMaker Credentials in the below cell"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"id": "89ed9c8b62964821851957ac7c822ca3"
},
"outputs": [],
"source": [
"SAGEMAKER_CREDENTIALS = {\n",
" \"access_key_id\": '',\n",
" \"secret_access_key\": '',\n",
" \"region\": 'us-east-2'\n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "a94c5ecb-18b7-489e-87cb-065409ca58ec"
},
"source": [
"### Update SageMaker endpoint name in the below cell"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"id": "6958c5864c5241a09b0da12b0f5a22a1"
},
"outputs": [],
"source": [
"def score(training_data_frame):\n",
" #User input needed\n",
" endpoint_name = 'UPDATE THE SAGEMAKER ENDPOINT NAME'\n",
"\n",
" access_id = SAGEMAKER_CREDENTIALS.get('access_key_id')\n",
" secret_key = SAGEMAKER_CREDENTIALS.get('secret_access_key')\n",
" region = SAGEMAKER_CREDENTIALS.get('region')\n",
" \n",
" #Covert the training data frames to bytes\n",
" import io\n",
" import numpy as np\n",
" train_df_bytes = io.BytesIO()\n",
" np.savetxt(train_df_bytes, training_data_frame.values, delimiter=',', fmt='%g')\n",
" payload_data = train_df_bytes.getvalue().decode().rstrip()\n",
"\n",
" #Score the training data\n",
" import requests\n",
" import time\n",
" import json\n",
" import boto3\n",
"\n",
" runtime = boto3.client('sagemaker-runtime', region_name=region, aws_access_key_id=access_id, aws_secret_access_key=secret_key)\n",
" start_time = time.time()\n",
"\n",
" response = runtime.invoke_endpoint(EndpointName=endpoint_name, ContentType='text/csv', Body=payload_data)\n",
" response_time = int((time.time() - start_time)*1000)\n",
" results_decoded = json.loads(response['Body'].read().decode())\n",
"\n",
" #Extract the details\n",
" results = results_decoded['predictions']\n",
"\n",
" predicted_vector_list = []\n",
" probability_array_list = []\n",
"\n",
" for value in results:\n",
" predicted_vector_list.append(value['predicted_label'])\n",
" probability_array_list.append(value['score'])\n",
"\n",
" #Conver to numpy arrays\n",
" probability_array = np.array(probability_array_list)\n",
" predicted_vector = np.array(predicted_vector_list)\n",
"\n",
" return probability_array, predicted_vector"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"id": "c4a4ad22ae5a424c8b0aa7f0293de341"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2220\n",
"\n"
]
}
],
"source": [
"print(data_df.shape[0])\n",
"print(score)"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"id": "03d34d7b-e163-4eb7-848f-550b4acca573"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Scoring training dataframe...: 100%|██████████| 1776/1776 [00:02<00:00, 627.03rows/s]\n",
"Optimising Drift Detection Model...: 100%|██████████| 40/40 [00:37<00:00, 1.06models/s]\n",
"Scoring training dataframe...: 100%|██████████| 444/444 [00:00<00:00, 775.00rows/s]\n",
"Computing feature stats...: 100%|██████████| 2/2 [00:00<00:00, 461.88features/s]\n",
"Learning single feature constraints...: 100%|██████████| 3/3 [00:00<00:00, 404.76constraints/s]\n",
"Learning two feature constraints...: 100%|██████████| 2/2 [00:00<00:00, 121.22constraints/s]\n"
]
}
],
"source": [
"#Generate drift detection model\n",
"from ibm_wos_utils.drift.drift_trainer import DriftTrainer\n",
"enable_drift = service_configuration_support.get('enable_drift')\n",
"if enable_drift:\n",
" drift_detection_input = {\n",
" \"feature_columns\":training_data_info.get('feature_columns'),\n",
" \"categorical_columns\":training_data_info.get('categorical_columns'),\n",
" \"label_column\": training_data_info.get('class_label'),\n",
" \"problem_type\": model_type\n",
" }\n",
" \n",
" drift_trainer = DriftTrainer(data_df,drift_detection_input)\n",
" if model_type != \"regression\":\n",
" #Note: batch_size can be customized by user as per the training data size\n",
" drift_trainer.generate_drift_detection_model(score,batch_size=32)\n",
" \n",
" #Note:\n",
" # - Two column constraints are not computed beyond two_column_learner_limit(default set to 200)\n",
" # - Categorical columns with large (determined by categorical_unique_threshold; default > 0.8) number of unique values relative to total rows in the column are discarded. \n",
" #User can adjust the value depending on the requirement\n",
" \n",
" drift_trainer.learn_constraints(two_column_learner_limit=2, categorical_unique_threshold=0.8)\n",
" drift_trainer.create_archive()"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"id": "1ebac730-61e9-47c4-8187-2f1977afacfe"
},
"outputs": [
{
"data": {
"text/html": [
"Download Drift detection model"
],
"text/plain": [
""
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#Generate a download link for drift detection model\n",
"from IPython.display import HTML\n",
"import base64\n",
"import io\n",
"\n",
"def create_download_link_for_ddm( title = \"Download Drift detection model\", filename = \"drift_detection_model.tar.gz\"): \n",
" \n",
" #Retains stats information \n",
" if enable_drift:\n",
" with open(filename,'rb') as file:\n",
" ddm = file.read()\n",
" b64 = base64.b64encode(ddm)\n",
" payload = b64.decode()\n",
" \n",
" html = '{title}'\n",
" html = html.format(payload=payload,title=title,filename=filename)\n",
" return HTML(html)\n",
" else:\n",
" print(\"Drift Detection is not enabled. Please enable and rerun the notebook\")\n",
"\n",
"create_download_link_for_ddm()\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "86dd5a03fb824f6f8d4f52c340dfe705"
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.6"
}
},
"nbformat": 4,
"nbformat_minor": 1
}