{ "cells": [ { "cell_type": "markdown", "id": "32f98d3f", "metadata": {}, "source": [ "# Demand Prediction for Dockless Vehicles using Amzon SageMaker and Amazon Athena\n", "\n", "We demonstrate building a machine learning model for short term demand predictions of dockless vehicles in different neighborhood of Loisville, KY. The project has been inspired by several Kaagle competitions and publications.\n", "\n", "The goal is to develop a demand model that predicts the number of vehicles needed within the next hour for each neighborhood.\n", "\n", "The data set is provided by the Office of Advanced Planning of Loisville, Kentucky.\n", "The data set contains trip data of individual vehicles with start and end locations. The start and end times are truncated to 15 minute intervals, though we aggregate over the full hour.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "a4b5b53d", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": 2, "id": "27509a1b", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Role: arn:aws:iam::969171869770:role/service-role/AmazonSageMaker-ExecutionRole-20210428T175911\n", "ml_train_test_data.csv\n" ] } ], "source": [ "import boto3\n", "import re\n", "import s3fs\n", "import pandas as pd\n", "\n", "import numpy as np\n", "import datetime\n", "\n", "\n", "import sagemaker\n", "ROLE = sagemaker.get_execution_role()\n", "print(f\"Role: {ROLE}\")\n", "\n", "REGION = 33.Session().region_name\n", "\n", "S3BUCKET = f'athena-blog-1'\n", "# S3PREFIX = 'taxi_data_raw/athena_data'\n", "S3PREFIX = 'dockless_vehicles/ml'\n", "\n", "DATAFILE = 'ml_train_test_data.csv'\n", "print(DATAFILE)\n", "\n", "def printShape(df):\n", " print(f\"Number of records: {df.shape[0]:,}, number of columns: {df.shape[1]}\")\n", " " ] }, { "cell_type": "code", "execution_count": 6, "id": "4fe7c771", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number of records: 895,413, number of columns: 17\n" ] } ], "source": [ "df = pd.read_csv(DATAFILE)\n", "df.dropna(inplace=True)\n", "printShape(df)" ] }, { "cell_type": "code", "execution_count": 7, "id": "8b273097", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
datenbidhourdown_startn_endn_start_1n_start_2n_start_3n_start_4n_end_1n_end_2n_end_3n_end_4y_start_1y_start_2y_start_3
42018-08-0935100.00.00.00.00.00.00.00.00.00.00.00.00.0
52018-08-09352300.00.00.00.00.00.00.00.00.00.00.00.00.0
62018-08-0935600.00.00.00.00.00.00.00.00.00.00.00.00.0
72018-08-0935200.00.00.00.00.00.00.00.00.00.00.00.00.0
82018-08-09351100.00.00.00.00.00.00.00.00.00.00.00.00.0
\n", "
" ], "text/plain": [ " date nbid hour dow n_start n_end n_start_1 n_start_2 \\\n", "4 2018-08-09 35 1 0 0.0 0.0 0.0 0.0 \n", "5 2018-08-09 35 23 0 0.0 0.0 0.0 0.0 \n", "6 2018-08-09 35 6 0 0.0 0.0 0.0 0.0 \n", "7 2018-08-09 35 2 0 0.0 0.0 0.0 0.0 \n", "8 2018-08-09 35 11 0 0.0 0.0 0.0 0.0 \n", "\n", " n_start_3 n_start_4 n_end_1 n_end_2 n_end_3 n_end_4 y_start_1 \\\n", "4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n", "5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n", "6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n", "7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n", "8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n", "\n", " y_start_2 y_start_3 \n", "4 0.0 0.0 \n", "5 0.0 0.0 \n", "6 0.0 0.0 \n", "7 0.0 0.0 \n", "8 0.0 0.0 " ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.head()" ] }, { "cell_type": "code", "execution_count": 8, "id": "742627a4", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Index(['date', 'nbid', 'hour', 'dow', 'n_start', 'n_end', 'n_start_1',\n", " 'n_start_2', 'n_start_3', 'n_start_4', 'n_end_1', 'n_end_2', 'n_end_3',\n", " 'n_end_4', 'y_start_1', 'y_start_2', 'y_start_3'],\n", " dtype='object')" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.columns" ] }, { "cell_type": "markdown", "id": "5dff857c", "metadata": {}, "source": [ "Extract all-zero rows and use a sub-sample to balanace data set" ] }, { "cell_type": "code", "execution_count": 15, "id": "e338feae", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number of records: 10,000, number of columns: 17\n", "Number of records: 304,619, number of columns: 17\n", "Number of records: 314,619, number of columns: 17\n" ] } ], "source": [ "num_cols = ['n_start', 'n_end','n_start_1', 'n_start_2', 'n_start_3', 'n_start_4', 'n_end_1', 'n_end_2', 'n_end_3', 'n_end_4', 'y_start_1', 'y_start_2', 'y_start_3']\n", "mask = df[num_cols].values.sum(axis=1)>0\n", "mask.shape, mask.sum()\n", "\n", "df_zero = df[~mask].sample(10000)\n", "printShape(df_zero)\n", "df_nonzero = df[mask]\n", "printShape(df_nonzero)\n", "df2 = pd.concat([df_zero, df_nonzero])\n", "printShape(df2)" ] }, { "cell_type": "markdown", "id": "30a3a42c", "metadata": {}, "source": [ "# Processing Data" ] }, { "cell_type": "code", "execution_count": 13, "id": "24058744", "metadata": {}, "outputs": [], "source": [ "from sklearn.model_selection import train_test_split\n", "from sklearn.preprocessing import StandardScaler, OneHotEncoder, LabelBinarizer, KBinsDiscretizer\n", "from sklearn.preprocessing import PolynomialFeatures\n", "from sklearn.compose import make_column_transformer\n", "from sklearn.exceptions import DataConversionWarning" ] }, { "cell_type": "code", "execution_count": 16, "id": "83518370", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
y_start_1nbidhourdown_start_1n_start_2n_start_3n_start_4n_end_1n_end_2n_end_3n_end_4
8953670.091740.00.00.00.00.00.00.00.0
8953680.091101.00.00.00.01.00.00.00.0
8953690.09900.01.00.00.00.01.00.00.0
8953700.091800.00.01.00.00.00.01.00.0
8953710.09500.00.00.01.00.00.00.01.0
\n", "
" ], "text/plain": [ " y_start_1 nbid hour dow n_start_1 n_start_2 n_start_3 \\\n", "895367 0.0 9 17 4 0.0 0.0 0.0 \n", "895368 0.0 9 11 0 1.0 0.0 0.0 \n", "895369 0.0 9 9 0 0.0 1.0 0.0 \n", "895370 0.0 9 18 0 0.0 0.0 1.0 \n", "895371 0.0 9 5 0 0.0 0.0 0.0 \n", "\n", " n_start_4 n_end_1 n_end_2 n_end_3 n_end_4 \n", "895367 0.0 0.0 0.0 0.0 0.0 \n", "895368 0.0 1.0 0.0 0.0 0.0 \n", "895369 0.0 0.0 1.0 0.0 0.0 \n", "895370 0.0 0.0 0.0 1.0 0.0 \n", "895371 1.0 0.0 0.0 0.0 1.0 " ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Number of rows: 314,619\n" ] } ], "source": [ "features = ['y_start_1', 'nbid', 'hour', 'dow', 'n_start_1', 'n_start_2', 'n_start_3', 'n_start_4',\n", " 'n_end_1', 'n_end_2', 'n_end_3', 'n_end_4' ]\n", "display(df2[features].tail())\n", " \n", "### Save the data to S3 bucket\n", "input_data = df2[features].dropna().copy()\n", "print(f\"Number of rows: {input_data.shape[0]:,}\")\n", "\n", "dataInPath = f's3://{S3BUCKET}/{S3PREFIX}/data/data_in.csv'\n", "input_data.to_csv(dataInPath, sep = ',', index = False)" ] }, { "cell_type": "code", "execution_count": 17, "id": "c57629f2", "metadata": {}, "outputs": [], "source": [ "### Import required libraries\n", " \n", "import boto3\n", "import sagemaker\n", "from sagemaker import get_execution_role\n", "from sagemaker.sklearn.processing import SKLearnProcessor \n", " \n", "### Create an scikit-learn processor\n", " \n", "sklearn_processor = SKLearnProcessor(\n", " framework_version = '0.20.0', # specify the scikit-learn version that we want to use - the image will be this version\n", " role = ROLE, # insert the role\n", " instance_type = 'ml.m5.xlarge', # this is the \"normal\" instance type\n", " instance_count = 1, # we only use 1 instance\n", " base_job_name = f'dockless-processing', # need to follow ADA Datalab guideline \n", ")" ] }, { "cell_type": "markdown", "id": "f282fd7e", "metadata": {}, "source": [ "## Process script" ] }, { "cell_type": "code", "execution_count": 18, "id": "0b6731ec", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Overwriting preprocessing.py\n" ] } ], "source": [ "%%writefile preprocessing.py\n", " \n", "import argparse\n", "import os\n", "import warnings\n", "import pandas as pd\n", "import numpy as np\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.preprocessing import StandardScaler, OneHotEncoder, LabelBinarizer, KBinsDiscretizer\n", "from sklearn.preprocessing import PolynomialFeatures\n", "from sklearn.compose import make_column_transformer\n", "from sklearn.exceptions import DataConversionWarning\n", "warnings.filterwarnings(action='ignore', category=DataConversionWarning)\n", " \n", "def print_shape(df):\n", " pass\n", " \n", "if __name__=='__main__':\n", " \n", " ### Specifying input path - Note how set the input paths\n", " \n", " input_data_path = os.path.join('/opt/ml/processing/input', 'data_in.csv')\n", " \n", " ### Reading + basic data handling\n", " \n", " print('Reading input data from {}'.format(input_data_path))\n", " data_raw = pd.read_csv(input_data_path) #### , parse_dates=['t_hour'])\n", "# features = ['y_start_1', 'location_id', 'hr', 'dayofweek', 'n_start_1', 'n_start_2', 'n_start_3', 'n_start_4',\n", "# 'n_end_1', 'n_end_2', 'n_end_3', 'n_end_4' ]\n", " features = list(data_raw.columns)\n", " label_col = features[0]\n", " data = data_raw[features]\n", " # data = pd.DataFrame(data = data)\n", " \n", " ### Transform response variable\n", " \n", " \n", " \n", " ### data split in three sets, training, validation and batch inference\n", " \n", " rand_split = np.random.rand(len(data))\n", " train_list = rand_split < 0.8\n", " val_list = (rand_split >= 0.8) & (rand_split < 0.9)\n", " test_list = rand_split >= 0.9\n", " \n", " ### Split data to train, validation and test\n", " \n", " data_train = data[train_list]\n", " data_val = data[val_list]\n", " data_test = data[test_list]\n", " test_class = data_test[label_col]\n", " data_test = data_test.drop([label_col], axis = 1)\n", " \n", " ### Specifying output paths - Note how set the output paths\n", " \n", " train_features_output_path = os.path.join('/opt/ml/processing/train', 'train_data.csv')\n", " validation_features_output_path = os.path.join('/opt/ml/processing/validation', 'validation_data.csv')\n", " test_features_output_path = os.path.join('/opt/ml/processing/test', 'test_data.csv')\n", " test_classes_output_path = os.path.join('/opt/ml/processing/test', 'test_class.csv')\n", " \n", " ### Save files to output destinations\n", " \n", " pd.DataFrame(data_train).to_csv(train_features_output_path, header = False, index = False)\n", " pd.DataFrame(data_val).to_csv(validation_features_output_path, header = False, index = False)\n", " pd.DataFrame(data_test).to_csv(test_features_output_path, header = False, index = False)\n", " pd.DataFrame(test_class).to_csv(test_classes_output_path, header = False, index = False)" ] }, { "cell_type": "markdown", "id": "f968c1cd", "metadata": {}, "source": [ "## Upload and Run" ] }, { "cell_type": "code", "execution_count": 20, "id": "78af9c0b", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "s3://athena-blog-1/dockless_vehicles/ml/scripts/preprocessing.py\n", "\n", "Job Name: dockless-processing-2021-05-13-09-31-44-921\n", "Inputs: [{'InputName': 'input-1', 'AppManaged': False, 'S3Input': {'S3Uri': 's3://athena-blog-1/dockless_vehicles/ml/data/data_in.csv', 'LocalPath': '/opt/ml/processing/input', 'S3DataType': 'S3Prefix', 'S3InputMode': 'File', 'S3DataDistributionType': 'FullyReplicated', 'S3CompressionType': 'None'}}, {'InputName': 'code', 'AppManaged': False, 'S3Input': {'S3Uri': 's3://athena-blog-1/dockless_vehicles/ml/scripts/preprocessing.py', 'LocalPath': '/opt/ml/processing/input/code', 'S3DataType': 'S3Prefix', 'S3InputMode': 'File', 'S3DataDistributionType': 'FullyReplicated', 'S3CompressionType': 'None'}}]\n", "Outputs: [{'OutputName': 'train_data', 'AppManaged': False, 'S3Output': {'S3Uri': 's3://athena-blog-1/dockless_vehicles/ml/train/', 'LocalPath': '/opt/ml/processing/train', 'S3UploadMode': 'EndOfJob'}}, {'OutputName': 'validation_data', 'AppManaged': False, 'S3Output': {'S3Uri': 's3://athena-blog-1/dockless_vehicles/ml/validation/', 'LocalPath': '/opt/ml/processing/validation', 'S3UploadMode': 'EndOfJob'}}, {'OutputName': 'test_data', 'AppManaged': False, 'S3Output': {'S3Uri': 's3://athena-blog-1/dockless_vehicles/ml/test/', 'LocalPath': '/opt/ml/processing/test', 'S3UploadMode': 'EndOfJob'}}]\n", ".......................\u001b[34m/miniconda3/lib/python3.7/site-packages/sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:47: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses\n", " import imp\u001b[0m\n", "\u001b[34mReading input data from /opt/ml/processing/input/data_in.csv\u001b[0m\n", "\n" ] } ], "source": [ "### Copy the preprocessing code over to the s3 bucket\n", " \n", "sess = sagemaker.Session()\n", "codeprefix = S3PREFIX + '/scripts'\n", "codeupload = sess.upload_data('preprocessing.py', bucket = S3BUCKET, key_prefix = codeprefix)\n", "print(codeupload)\n", " \n", "### Import ProcessingInput and ProcessingOutput function\n", "from sagemaker.processing import ProcessingInput, ProcessingOutput\n", " \n", "### Set our data destination path in S3\n", "dataDestination = 's3://' + S3BUCKET + '/' + S3PREFIX\n", " \n", "### Run the processing job\n", "sklearn_processor.run(\n", " code = codeupload,\n", " inputs = [ProcessingInput(source = dataInPath, destination = '/opt/ml/processing/input')],\n", " outputs = [\n", " ProcessingOutput(\n", " output_name = 'train_data',\n", " source = '/opt/ml/processing/train',\n", " destination = dataDestination + '/train/'\n", " ),\n", " ProcessingOutput(\n", " output_name = 'validation_data', \n", " source = '/opt/ml/processing/validation',\n", " destination = dataDestination + '/validation/'\n", " ),\n", " ProcessingOutput(\n", " output_name = 'test_data',\n", " source = '/opt/ml/processing/test',\n", " destination = dataDestination + '/test/'\n", " )\n", " ]\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "b9a2a28d", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "72a6b6a7", "metadata": {}, "source": [ "# Training Data" ] }, { "cell_type": "code", "execution_count": 22, "id": "f45bc73a", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "The method get_image_uri has been renamed in sagemaker>=2.\n", "See: https://sagemaker.readthedocs.io/en/stable/v2.html for details.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Using ML image '257758044811.dkr.ecr.us-east-2.amazonaws.com/sagemaker-xgboost:1.2-1' in region 'us-east-2'\n", "training artifacts will be uploaded to: s3://athena-blog-1/dockless_vehicles/ml\n", "InProgress\n", "Training job ended with status: Completed\n" ] } ], "source": [ "### This is to get the image df_nonzerohe XGBoost\n", " \n", "from sagemaker.amazon.amazon_estimator import get_image_uri\n", "\n", "image = get_image_uri(REGION, 'xgboost', '1.2-1')\n", "#image = sagemaker.image_uris.retrieve(\"xgboost\", REGION, \"1.2-1\")\n", "print(f\"Using ML image '{image}' in region '{REGION}'\")\n", " \n", "### This is the output location of the model\n", " \n", "output_location = 's3://{}/{}'.format(S3BUCKET, S3PREFIX)\n", "print('training artifacts will be uploaded to: {}'.format(output_location))\n", " \n", "### Training Parameters\n", "import datetime\n", "ts = datetime.datetime.now().strftime(\"%Y%m%d-%H%M%S\")\n", "job_name = f\"dockless-training-{ts}\"\n", "\n", "create_training_params = {\n", " \"AlgorithmSpecification\": {\n", " \"TrainingImage\": image, ### This is the image of the XGBoost that we want to use as built-in algorithm directly\n", " \"TrainingInputMode\": \"File\"\n", " },\n", " \"RoleArn\": ROLE, \n", " \"OutputDataConfig\": {\n", " \"S3OutputPath\": output_location ### This is the output location (with some addition on prefix, etc) where model.tar.gz will be stored\n", " },\n", " \"ResourceConfig\": {\n", " \"InstanceCount\": 1, \n", " \"InstanceType\": \"ml.m5.4xlarge\",\n", " \"VolumeSizeInGB\": 50,\n", " },\n", " \"TrainingJobName\": job_name, ### This is the training job name that we have provided previously\n", " \"HyperParameters\": {\n", " ## \"objective\": \"binary:logistic\", ### For our case, we use binary:logistic as our objective function (0/1 case)\n", " \"objective\": \"reg:squarederror\", ### For our case, we use binary:logistic as our objective function (0/1 case)\n", " \"max_depth\": \"5\",\n", " \"eta\": \"0.2\",\n", " \"gamma\": \"4\",\n", " \"min_child_weight\": \"6\",\n", " \"subsample\": \"0.8\",\n", " # \"silent\": \"0\",\n", " \"num_round\": \"500\"\n", " },\n", " \"StoppingCondition\": {\n", " \"MaxRuntimeInSeconds\": 60 * 60\n", " },\n", " \"InputDataConfig\": [\n", " {\n", " \"ChannelName\": \"train\", ### This contains the channels that we want to use for training, here we are using train and validation channel\n", " \"DataSource\": {\n", " \"S3DataSource\": {\n", " \"S3DataType\": \"S3Prefix\",\n", " \"S3Uri\": 's3://{}/{}/train'.format(S3BUCKET, S3PREFIX),\n", " \"S3DataDistributionType\": \"FullyReplicated\"\n", " }\n", " },\n", " \"CompressionType\": \"None\",\n", " \"RecordWrapperType\": \"None\",\n", " \"ContentType\": \"text/csv\"\n", " },\n", " {\n", " \"ChannelName\": \"validation\",\n", " \"DataSource\": {\n", " \"S3DataSource\": {\n", " \"S3DataType\": \"S3Prefix\",\n", " \"S3Uri\": 's3://{}/{}/validation'.format(S3BUCKET, S3PREFIX),\n", " \"S3DataDistributionType\": \"FullyReplicated\"\n", " }\n", " },\n", " \"CompressionType\": \"None\",\n", " \"RecordWrapperType\": \"None\",\n", " \"ContentType\": \"text/csv\"\n", " }\n", " ],\n", "}\n", " \n", "sagemaker = boto3.client('sagemaker')\n", " \n", "### Create Training Job\n", " \n", "sagemaker.create_training_job(**create_training_params)\n", " \n", "### Describe the status of training job\n", " \n", "status = sagemaker.describe_training_job(TrainingJobName = job_name)['TrainingJobStatus']\n", "print(status)\n", " \n", "try:\n", " sagemaker.get_waiter('training_job_completed_or_stopped').wait(TrainingJobName = job_name)\n", "finally:\n", " status = sagemaker.describe_training_job(TrainingJobName = job_name)['TrainingJobStatus']\n", " print(\"Training job ended with status: \" + status)\n", " if status == 'Failed':\n", " message = sagemaker.describe_training_job(TrainingJobName = job_name)['FailureReason']\n", " print('Training failed with the following error: {}'.format(message))\n", " raise Exception('Training job failed')" ] }, { "cell_type": "code", "execution_count": null, "id": "bd89e269", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "416277d6", "metadata": {}, "source": [ "# Create Model" ] }, { "cell_type": "code", "execution_count": 26, "id": "4ff18a1f", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Model Info\n", "----------\n", "Location of model parameters: s3://athena-blog-1/dockless_vehicles/ml/dockless-training-20210513-093745/output/model.tar.gz\n", "Container image: 257758044811.dkr.ecr.us-east-2.amazonaws.com/sagemaker-xgboost:1.2-1\n", "Training job: dockless-training-20210513-093745\n", "\n", "\n", "arn:aws:sagemaker:us-east-2:969171869770:model/dockless-regression-model\n" ] } ], "source": [ "sagemaker = boto3.client('sagemaker')\n", "info = sagemaker.describe_training_job(TrainingJobName = job_name) # This is to see all the details of your training job\n", "model_data = info['ModelArtifacts']['S3ModelArtifacts'] # That detail includes where the model.tar.gz is saved in your S3 bucket\n", " \n", "print(f\"\"\"\n", "Model Info\n", "----------\n", "Location of model parameters: {model_data}\n", "Container image: {image}\n", "Training job: {job_name}\n", "\n", "\"\"\")\n", "\n", "primary_container = {\n", " 'Image': image, # This is the image for inference (not for training). But in our case, this is the same as training image.\n", " 'ModelDataUrl': model_data # This is the model artifacts, the one with model.tar.gz\n", "}\n", " \n", "create_model_response = sagemaker.create_model(\n", " ModelName = f'dockless-regression-model', # Specify the job name of the training job earlier on\n", " ExecutionRoleArn = ROLE, # Specify the role, which equals to get_execution_role()\n", " PrimaryContainer = primary_container, # Specify the primary container. \n", ")\n", " \n", "print(create_model_response['ModelArn'])" ] }, { "cell_type": "code", "execution_count": null, "id": "ed88d904", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "93020190", "metadata": {}, "source": [ "# Eval" ] }, { "cell_type": "code", "execution_count": 27, "id": "ec1ff60c", "metadata": {}, "outputs": [], "source": [ "import boto3\n", "sagemaker_runtime = boto3.client('sagemaker-runtime')" ] }, { "cell_type": "code", "execution_count": 29, "id": "3b025761", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
y_start_1nbidhourdown_start_1n_start_2n_start_3n_start_4n_end_1n_end_2n_end_3n_end_4
4700.0351620.00.00.00.00.00.00.00.0
4710.0351100.00.00.00.01.00.00.00.0
4720.035900.00.00.00.00.01.00.00.0
4730.035700.00.00.00.00.00.01.00.0
4740.0352000.00.00.00.00.00.00.01.0
\n", "
" ], "text/plain": [ " y_start_1 nbid hour dow n_start_1 n_start_2 n_start_3 n_start_4 \\\n", "470 0.0 35 16 2 0.0 0.0 0.0 0.0 \n", "471 0.0 35 11 0 0.0 0.0 0.0 0.0 \n", "472 0.0 35 9 0 0.0 0.0 0.0 0.0 \n", "473 0.0 35 7 0 0.0 0.0 0.0 0.0 \n", "474 0.0 35 20 0 0.0 0.0 0.0 0.0 \n", "\n", " n_end_1 n_end_2 n_end_3 n_end_4 \n", "470 0.0 0.0 0.0 0.0 \n", "471 1.0 0.0 0.0 0.0 \n", "472 0.0 1.0 0.0 0.0 \n", "473 0.0 0.0 1.0 0.0 \n", "474 0.0 0.0 0.0 1.0 " ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "features = ['y_start_1', 'nbid', 'hour', 'dow', 'n_start_1', 'n_start_2', 'n_start_3', 'n_start_4',\n", " 'n_end_1', 'n_end_2', 'n_end_3', 'n_end_4' ]\n", "df_nonzero[features].dropna().head()" ] }, { "cell_type": "code", "execution_count": 30, "id": "42ce4d34", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "114599\ty=0.0\ty_hat=0.03657257556915283\t23.0,14.0,0.0,0.0,1.0,0.0,0.0,1.0,0.0,0.0,0.0\n", "577313\ty=0.0\ty_hat=0.6612421274185181\t20.0,22.0,0.0,4.0,1.0,6.0,0.0,1.0,1.0,6.0,0.0\n", "763932\ty=0.0\ty_hat=-0.006854534149169922\t49.0,6.0,0.0,0.0,1.0,0.0,0.0,0.0,0.0,0.0,0.0\n", "395856\ty=10.0\ty_hat=0.4053572416305542\t47.0,6.0,1.0,5.0,1.0,5.0,0.0,3.0,0.0,9.0,0.0\n", "147445\ty=0.0\ty_hat=0.0036890506744384766\t3.0,15.0,3.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0\n", "196262\ty=0.0\ty_hat=0.022508323192596436\t39.0,4.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.0,0.0\n", "419153\ty=0.0\ty_hat=0.0021598339080810547\t19.0,19.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0\n", "876123\ty=0.0\ty_hat=0.05208203196525574\t72.0,0.0,0.0,0.0,1.0,1.0,0.0,0.0,0.0,0.0,0.0\n", "801412\ty=0.0\ty_hat=0.05565375089645386\t70.0,5.0,0.0,1.0,0.0,0.0,0.0,1.0,0.0,1.0,0.0\n", "327177\ty=0.0\ty_hat=0.024671852588653564\t63.0,16.0,2.0,0.0,0.0,0.0,0.0,0.0,0.0,1.0,0.0\n", "42257\ty=1.0\ty_hat=1.0007874965667725\t22.0,11.0,6.0,0.0,0.0,0.0,6.0,0.0,0.0,0.0,5.0\n", "802636\ty=1.0\ty_hat=0.6643890738487244\t70.0,15.0,5.0,1.0,7.0,1.0,0.0,1.0,2.0,2.0,0.0\n", "882518\ty=1.0\ty_hat=-0.0007305145263671875\t9.0,0.0,0.0,0.0,1.0,0.0,0.0,0.0,0.0,0.0,0.0\n", "413048\ty=0.0\ty_hat=0.0002645254135131836\t30.0,8.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0\n", "620443\ty=6.0\ty_hat=0.0058509111404418945\t60.0,22.0,0.0,2.0,0.0,0.0,0.0,6.0,0.0,0.0,0.0\n", "154856\ty=1.0\ty_hat=0.0002645254135131836\t26.0,6.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0\n", "706889\ty=0.0\ty_hat=0.011612534523010254\t58.0,4.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.0\n", "764850\ty=0.0\ty_hat=-0.006854534149169922\t49.0,3.0,0.0,0.0,1.0,0.0,0.0,0.0,0.0,0.0,0.0\n", "121845\ty=0.0\ty_hat=0.0021598339080810547\t26.0,17.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0\n", "21848\ty=0.0\ty_hat=0.1311495304107666\t1.0,12.0,2.0,0.0,1.0,0.0,0.0,2.0,1.0,0.0,0.0\n" ] } ], "source": [ "\n", "for i, r in df_nonzero[features].dropna().sample(20).iterrows():\n", " b = ','.join([ str(r[c]) for c in features[1:] ])\n", " y = r['y_start_1']\n", " res = sagemaker_runtime.invoke_endpoint(\n", " EndpointName='dockless-regression-ep',\n", " Body = b, ###. '79.0,2.0,3.0,727.0,721.0,1.0,3.0,602.0,421.0,1.0,1.0',\n", " ContentType = 'text/csv'\n", " )\n", " y_hat = float(res['Body'].read())\n", " print(f\"{i}\\ty={y}\\ty_hat={y_hat}\\t{b}\")" ] }, { "cell_type": "code", "execution_count": 113, "id": "30229d28", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(16928, 18)" ] }, "execution_count": 113, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_nonzero.shape" ] }, { "cell_type": "code", "execution_count": 33, "id": "bdbbb46e", "metadata": {}, "outputs": [], "source": [ "def predict_row(r):\n", " features = ['y_start_1', 'nbid', 'hour', 'dow', 'n_start_1', 'n_start_2', 'n_start_3', 'n_start_4',\n", " 'n_end_1', 'n_end_2', 'n_end_3', 'n_end_4' ] \n", " \n", " b = ','.join([ str(r[c]) for c in features[1:] ])\n", " y = r['y_start_1']\n", " res = sagemaker_runtime.invoke_endpoint(\n", " EndpointName='dockless-regression-ep',\n", " Body = b, ###. '79.0,2.0,3.0,727.0,721.0,1.0,3.0,602.0,421.0,1.0,1.0',\n", " ContentType = 'text/csv'\n", " )\n", " y_hat = float(res['Body'].read())\n", " return y_hat\n", " " ] }, { "cell_type": "code", "execution_count": 66, "id": "7bbe12db", "metadata": {}, "outputs": [], "source": [ "%%time\n", "df_hat = df_nonzero.sample(10000).copy()\n", "df_hat['y_hat'] = df_hat.apply(predict_row, axis=1)\n", "df_hat['y_predict'] = df_hat.y_hat.round()\n", "df_hat.rename({'y_start_1': 'y_actual'}, axis=1, inplace=True)\n", "printShape(df_hat)\n", "\n", "\n", "confusion_matrix = pd.pivot_table(df_hat, index='y_actual', columns='y_predict', values='date', aggfunc='count', fill_value=0)\n", "printShape(confusion_matrix)" ] }, { "cell_type": "code", "execution_count": 72, "id": "63a209c8", "metadata": {}, "outputs": [], "source": [ "def plot_confusion_matrix(subdf, label=''):\n", " confusion_matrix = pd.pivot_table(subdf, index='y_actual', columns='y_predict', values='date', aggfunc='count', fill_value=0)\n", " print(f\"Confusion Matrix: {label}\")\n", " printShape(confusion_matrix)\n", " mae = np.abs(df_hat.y_actual-df_hat.y_predict).mean()\n", " print(f\"MAE: {mae}\")\n", " plt.figure(figsize=(12,12))\n", " plt.imshow(np.log1p(confusion_matrix))\n", " plt.colorbar()\n", " plt.ylabel('Actual')\n", " plt.xlabel('Predict')\n", " plt.show()\n", " plt.close()\n", " return confusion_matrix" ] }, { "cell_type": "code", "execution_count": null, "id": "0c6b49b3", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": 74, "id": "7d5646f1", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Confusion Matrix: \n", "Number of records: 80, number of columns: 73\n", "MAE: 1.0903\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAApcAAAKgCAYAAADOE1aEAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAvO0lEQVR4nO3dfbBtd1kn+O/DuTevECAEMCY6wHSMDdgEO4Mo0zYQceLLAF3T9kCVXYzNdOwu20bHHsX+Yyy7xiqm7LKkHMfuDKCpEVFEGWhagRhFdMoGEoyGGFIoIoSEXELkJQmQ5J5n/jg7cAlnrXPuOWuftc8+n0/Vrv2y9lr7t39n39ST51m/Z1V3BwAApvCIuQcAAMD6EFwCADAZwSUAAJMRXAIAMBnBJQAAkzk29wAAAI6q/+555/an7j456xhu+PMvvqO7r5zqeIJLAICZfOruk3nvO75+1jFsXPihC6Y8nrI4AACTEVwCADAZZXEAgJl0ks1szj2MSclcAgAwGZlLAIDZdE62zCUAAGxLcAkAwGSUxQEAZrK1oKfnHsakZC4BAJiMzCUAwIy0IgIAgAGCSwAAJqMsDgAwk07nZFvQAwAA25K5BACYkVZEAAAwQHAJAMBklMUBAGbSSU4qiwMAwPZkLgEAZmRBDwAADBBcAgAwGWVxAICZdOIKPQAAMETmEgBgRptzD2BiMpcAAExGcAkAwGSUxQEAZtJpV+gBAIAhgksAACajLA4AMJdOTq5XVfxwBJdnbJzdZ2+ct6d9+4EHBrfVWWcObrvkG/529LifODm8712ff+TIeMaTxTXWj2Dkx1cPjh42NbZx7Lg79Ed4xP3DO9fmyLaT4wfujZF5Gv0y4+qLO0zUEoz9BndSx48v5bgAJJ/L397V3Y+fexzrZpbgsqquTPLqJBtJXtPdrxp7/9kb5+Xbvuale/qsB2/7+OC2jb9z6eC233nHb4we92fv/q8Ht73m5ucMbnvgzrNHj3vs3uGg6hEjscSZnxqPuGosgDw5Mp77xv936lEfu39w2/HPDm/buHd4W5KcPPeMwW2bZ+79Z3vGX5/Y8757NfYb3Mmxr7loKccFIPm9ftPfzD2Gjj6X+1ZVG0l+Mcl3JXlqkpdW1VMPehwAAExvjgU9z0ryl9394e6+P8mvJ3nRDOMAAGBic5TFL0rysVOe35bkWx7+pqq6KslVSXLWxqMOZmQAAAeqcnI/iwlW0ByZy+1m8KtO7Ovuq7v78u6+/IxHjJ+nCADAapgjc3lbkq875fnFSW4f26EfeGDPixeOXTyyIOLmWwe3/a+feObocd91+yXDx73jnMFtZ58Yj+eP3Tu8beOLw4trHnfz50ePO+b43fcNbnvg/OHvkowvkOlHD6+ar8/cMz6om4f/3vv6P6Kx38N+Ft6MHHc/LNoBWG+dZKS5yqE0R+byfUkuqaonV9UZSV6S5K0zjAMAgIkdeOayux+sqn+V5B3ZakX0uu6++aDHAQDA9Gbpc9ndv5Pkd+b4bACAVWJBDwAAR0ZV/WhV3VxVH6iqN1TVWWPvF1wCAMyks5W5nPM2pqouSvKvk1ze3U/P1imNLxnbR3AJAMCYY0nOrqpjSc7JDl1+BJcAAEfbBVV1/Sm3qx7a0N0fT/Lvk3w0yR1JPtPd7xw72CwLeg7SXvsE/ucPP210+xfuO2Nw2/F7h1PMj7h//HOP3zfc7GrjCyPjefzweJLkrE8Of/BYz8njo0fdYX5vG9608bRLR497LHvrR7lTv8mxfXca08mRvqj6UQKwV5s9+4Keu7r78u02VNVjs3WZ7icn+XSS36yq7+/uXx06mMwlAABDviPJX3f3J7v7gSS/neTbxnZY+8wlAMCqemhBzwr7aJJnV9U5ST6f5Iok14/tIHMJAMC2uvs9Sd6U5P1JbspW7Hj12D4ylwAADOrun0ryU7t9v+ASAGAmncrJNSskr9e3AQBgVjKXA75wx7njbxg593bjvuGNxz4/ftixVkUbDwy3KdrcGD/u8bvvG9zWj37k4LaxNkXJzq1/hnz+wuHPTJLj5460Vhpp+7NTS6Cx8T440mroKNlPOycATt8KtCKalMwlAACTEVwCADAZZXEAgJkcgj6Xp03mEgCAyQguAQCYjLI4AMBsKid7vXJ9gssB53x8vLfPFx8z3Bbo+L3D+23cP7xfkjxw7vB5F5sjf63H3TTyoUlOjrT22Txz+MBn7NCKaMxYy5rjO7WzedY37flzxyyrjc5oi6ND1rrnsI0XgNUiuAQAmEkn2VyzsxTX69sAADArwSUAAJNRFgcAmJE+lwAAMEDmEgBgJt3r14povb4NAACzkrkccManx/tRpofPj3jkx08Obts8Pn5eRZ05vO3cTzw4uO2+i88ZPe7xzw7ve9Ydw70sH/za80ePe+z2uwe3bf6DZw6P5+77Ro978r03jW4fHM9Iv8lkvIfjfvbVGxIAtgguAQBmtGlBDwAAbE/mEgBgJp3k5Jrl+tbr2wAAMCvBJQAAk1EWBwCYzfr1uRRcDnjczV8Y3f7JZ549uO3YF4bbGN2/QyuiM+7ZHNx2/J7hdkLn3nRi9Lh79YWnjLfnOSvDrYoe8Ud/OrhtuFnT/uzUEmis3ZB2QgCwf4JLAICZdJLNNTtLcb2+DQAAsxJcAgAwGWVxAIAZnRy5pPRhJHMJAMBkBJcAAExGWXzAFx5/xuj2c04MtwyqB4dbEZ194oHR495z0fDn3nvRmYPbTp75NaPHPeuOewa31WeGt531n947etyx1j4Z2daPfuToccfGNEY7IQAOk065/CMAAAyRuQQAmNHmml2hZ72+DQAAsxJcAgAwGWVxAICZdGJBDwAADJG5BACYSafW7go9gssB59x23+j2B84b7ke5eXw4IfyFC46PHvf4fcP9M499Ybh/5thnJsnJc0f6dp57/vBnjh51730lNx596Z6PO9Zbc+NpOxz35lvHB7ZHo/0+d3DYenOOfdfD9l0AmJ6yOAAAk5G5BACY0eaa5frW69sAADArmUsAgJl0JyddoQcAALYnuAQAYDLK4gOO3X736PaNex85uO2+rz9vcFudHG4nlCTH7xluRXT8ngeHt9093jpptBXRPoy1/hn7zJPvvWnPxx1tJ3Tb6GGX1kbnKLXgOUrfFWD5KptZrz6XMpcAAExG5hIAYCYdC3oAAGCQ4BIAgMkoiwMAzOjkmuX61uvbAAAwK5nLATu1W9l49HCrnDM/9YXBbefeNN7i6LPPunjkuMOtiMbaH+00po177x/c1o8ebrmUJPWZe4Y33rz3ljUnx9oN7cN+2ugsq40RAEdXp7LZWhEBAMC2lhZcVtXrqupEVX3glNfOr6prq+pDi/vHLuvzAQA4eMvMXP5Kkisf9tork1zX3ZckuW7xHADgyDqZR8x6m9rSgsvufneSh59g+KIk1yweX5Pkxcv6fAAADt5Bn3P5xO6+I0kW908YemNVXVVV11fV9Q/kiwc2QAAA9m5lV4t399VJrk6S8+r8nnk4AACT6ySbLv+4L3dW1YVJsrg/ccCfDwDAEh105vKtSV6W5FWL+7cc8OdPZqy/40aGe0M++LXnjx73vPfeNrjt89/4NYPbzvnoZ0ePO2asp+RYb8dk/PtsjPTIHO2PmfG+kXP1m9TLEoDpVU5Gn8tdqao3JPmTJJdW1W1V9fJsBZUvqKoPJXnB4jkAAGtiaZnL7n7pwKYrlvWZAADMa2UX9AAArDsLegAAYITMJQDAjFZ9QU9VXZrkN0556SlJ/rfu/vnt3i+4BABgUHffmuSyJKmqjSQfT/LmofcLLveoR9rsjDl2+8OviLn74579wU8Mbrv3m7529Lhn/u77BrdtPO3S4fGMHjXZuPf+wW1j7Yb209ZnFVsCzdUeacwqjgmAQ++KJH/V3X8z9AbBJQDATLrrsC3oeUmSN4y94VB9GwAAJndBVV1/yu2q7d5UVWckeWGS3xw7mMwlAMCMTs6fubyruy/fxfu+K8n7u/vOsTfN/m0AADgUXpodSuKJ4BIAgB1U1TnZunT3b+/0XmVxAICZdJLNFe9zmSTdfV+Sx+3mvTKXAABMRuZyj8Z6ON7/5CcM73j+OXv+zEfcfOvgtnN26LtZYz0PR4471isxGe+XOLbvTsfd62fux37GNIf9/G0AWBW1Cgt6JrVe3wYAgFkJLgEAmIyyOADATDrJZq/+gp7TIXMJAMBkZC4BAGZ0cs1yfev1bQAAmJXM5RKc8dcnBrft1B5m42mXDm47uecR7d1O491r+55VbCd02Fr3HLbxAnA0CC4BAGbSKQt6AABgiOASAIDJKIsDAMxoc81yfev1bQAAmJXMJQDATLqTk2u2oEdwuUf96Efuab9j2aFVzmfuGd53rM3OyH7J8trWjB13P22BxvYd+8xltU4CAHZHWRwAgMnIXAIAzEifSwAAGCBzCQAwk60r9KxXrm+9vg0AALMSXAIAMBllcQCAGZ3Mei3oEVzuUY30ldxP78ex/pl7/cydPnc/PTDHjjvaC/S28eMuq38mW/Yzh8vqmQrAehBcAgDMpKMVEQAADBJcAgAwGWVxAIDZ6HMJAACDZC4BAGa0qRURyd5b5ezYMih73PdZ3zR63Nx+9/j2PRprN3Ty5luX8pnr1gpnWa2VxuZpjjnc6Xuu298V4KhSFgcAYDIylwAAM+lOTupzCQAA2xNcAgAwGWVxAIAZ6XMJAAADZC6XYD8tVfa873tvGj/u3o66o7F2Q2OtZ8ZaGO103P3YV5uoPe67rBY8y2phtCxaDQF8tU5l04IeAADYnuASAIDJKIsDAMxo3S7/KHMJAMBkZC4BAGbSiQU9AAAwRHAJAMBklMWXYBV7Ke7Vvsb7tecPbtu49/7R42487dLBbXvtrZnM04N0Wf0d9Y0EWA+u0AMAAANkLgEA5tKu0AMAAIMElwAATEZZHABgJh1X6AEAgEEyl0swR4uYZX3mflr77OfHtYptdvbTYor14/cATMWCnl2qqq+rqj+oqluq6uaqesXi9fOr6tqq+tDi/rHLGgMAAAdrmWXxB5P8WHf/3STPTvJDVfXUJK9Mcl13X5LkusVzAADWwNLK4t19R5I7Fo8/V1W3JLkoyYuSPHfxtmuSvCvJTyxrHAAAq6qjLL4nVfWkJM9M8p4kT1wEng8FoE84iDEAALB8S1/QU1WPTPJbSX6kuz9btbvovKquSnJVkpyVc5Y3QACAGclcnoaqOp6twPL13f3bi5fvrKoLF9svTHJiu327++ruvry7Lz+eM5c5TAAAJrK0zGVtpShfm+SW7v65Uza9NcnLkrxqcf+WZY1h3eyn9ckcbVPWqSXTso/N4eP3ALC9ZZbFn5Pknya5qapuXLz2b7MVVL6xql6e5KNJvm+JYwAAWFmdWruy+DJXi/9xMng9oyuW9bkAAMzH5R8BAJiMyz8CAMxoc7DQezjJXAIAMBmZSwCAubQ+lwAAMEjmcsWM9c7beNqlwzvetvfjjvXAZHfm6CO6099NH0YAplBVj0nymiRPz9bl0P9Zd//J0PsFlwAAM+kcirL4q5O8vbv/cVWdkYxfl1twCQDAtqrqvCTfnuR/SpLuvj/J/WP7CC4BAGa04pnLpyT5ZJJfrqpnJLkhySu6+96hHSzoAQA42i6oqutPuV11yrZjSb45yS919zOT3JvklWMHk7kEADja7uruywe23Zbktu5+z+L5myK4BABYTZ1a6bJ4d3+iqj5WVZd2961JrkjyF2P7CC4PkZM337qU4x6lljX7aRk01grqwZG/zbJaBvm7bTlK8wAwkx9O8vrFSvEPJ/mBsTcLLgEAZtQrnLlMku6+MclQ2fyrWNADAMBkBJcAAExGWRwAYEabWe2y+OmSuQQAYDIylwAAM+le+Sv0nDbBJaMtduoz94zue9jawOxnvMtqBcXODtvvDOAoUxYHAGAyMpcAADNa9T6Xp0vmEgCAychcAgDMZrWvLb4XMpcAAExGcAkAwGSUxQEAZrRuC3oEl6xV/8ZjF180un2Ofolz9Wgcmwt9IwFYFmVxAAAmI3MJADCTzvpd/lHmEgCAychcAgDMpZPuuQcxLZlLAAAmI7gEAGAyyuKslaPU9mdZbZc2nnbp6PZ1al0FsAo2Y0EPAABsS+YSAGAmnfW7Qo/MJQAAkxFcAgAwGWVxAIDZlCv0AADAEJnLPdqpDcyQuVrlsFxz/F2X9Zmr2GpojlZPAAfFFXoAAGCA4BIAgMkoiwMAzEifSwAAGCBzCQAwk26ZSwAAGCS4BABgMsrie7RqvfV26ru5auOFh9tr71iAw84VegAAYIDgEgCAySiLAwDMyOUfAQBggMwlAMCM9LkEAIABMpdrYlmtho5Si6P9tMKZY/7Xae6T9fs+AEeV4BIAYCadUhYHAIAhMpcAADNas05EMpcAAExHcAkAwGSUxQEA5tLr1+dScLli9tp6Zlktg/bTHmYV2+is4pjGrOKYxhy2+V3F9lMAh53gEgBgTmu2omdp51xW1VlV9d6q+rOqurmqfnrx+vlVdW1VfWhx/9hljQEAgIO1zAU9X0zy/O5+RpLLklxZVc9O8sok13X3JUmuWzwHAGANLK0s3t2d5J7F0+OLWyd5UZLnLl6/Jsm7kvzEssYBALDK1m1Bz1JbEVXVRlXdmOREkmu7+z1JntjddyTJ4v4JA/teVVXXV9X1D+SLyxwmAAATWeqCnu4+meSyqnpMkjdX1dNPY9+rk1ydJOfV+Wt2qisAwJZesyjnQJqod/ens1X+vjLJnVV1YZIs7k8cxBgAAFi+pWUuq+rxSR7o7k9X1dlJviPJ/5HkrUleluRVi/u3LGsMh9Ec/Sj3Y+Nplw5ue/DmWw9wJLtzlHoT7rWH405ztE69LFdxvACH3TLL4hcmuaaqNrKVIX1jd7+tqv4kyRur6uVJPprk+5Y4BgCAldVZvwU9y1wt/udJnrnN659KcsWyPhcAgPm4Qg8AwFw6yZplLg9kQQ8AAEeD4BIAgMkoiwMAzGjd+lwKLgfs1MZFC5Mt9Zl7dn4TK2fs97tuv/3DNl6Aw05ZHACAychcAgDMSVkcAICjpKo+kuRzSU4mebC7Lx96r+ASAGA2dZiu0PO87r5rpzc55xIAgMkILgEAjrYLqur6U25XbfOeTvLOqrphYPuXKIsDAMxp/gU9d42dQ7nwnO6+vaqekOTaqvpgd797uzcKLgfojbc75mm5xnpO7jT3/jbLtZ+/DcBh0923L+5PVNWbkzwrieASAGCldFZ+QU9VnZvkEd39ucXj70zy74beL7gEAGDME5O8uaqSrdjx17r77UNvFlwCADCouz+c5Bm7fb/gEgBgTvMv6JmUVkQAAExG5hIAYFarvaDndAkuWStj7WESLWIAYNmUxQEAmIzMJQDAnCzoAQCA7clcAgDMSeYSAAC2J7gEAGAyyuKspLGWQmPthNat1dAc32dZnzlXm6i9/pZ2sm6/NWAmnaTXq8+lzCUAAJORuQQAmFFb0AMAANsTXAIAMBllcQCAOSmLAwDA9gSXAABMRlmcpfUB3I916iG48bRLR7efvPnWAxrJ7iyrH6Xf0v6t4r9VYAL6XAIAwPZkLgEAZlQW9AAAwPYElwAATEZZHABgLh19LgEAYIjM5SGyrDYkh62FyWFrx7JqrYZ2sopzOIdltWTaj1X82xy2f4+wekorIgAAGCK4BABgMsriAABzsqAHAAC2J3MJADAnmUsAANiezOUhoq3HlrnmQcuVo8XfdHfME/BwgksAgDkpiwMAwPYGM5dV9QsZiaW7+18vZUQAAEdFZ+2u0DNWFr/+wEYBAMBaGAwuu/uagxwIAACH344Leqrq8Ul+IslTk5z10Ovd/fwljgsA4EioI7ig5/VJbkny5CQ/neQjSd63xDEBAHBI7aYV0eO6+7VV9Yru/sMkf1hVf7jsgcGq0c9vNW087dLR7SdvvvWARgKwR2uWudxNcPnA4v6OqvqeJLcnuXh5QwIA4LDaTXD5v1fVo5P8WJJfSHJekh9d6qgAADiUdgwuu/tti4efSfK85Q4HAIDDbDerxX8525wN0N3/bCkjAgDg0NpNWfxtpzw+K8k/ytZ5lwAA8BV2Uxb/rVOfV9Ubkvze0kYEAHCErFufy91kLh/ukiRfP/VA4FTHLr5ocJuWQEfP6O9hh1ZDfksAB2s351x+Ll95zuUnsnXFnl2pqo1sXaf84939vVV1fpLfSPKkbDVk/yfd/benMWYAgPXRNfcIJrXjFXq6+1Hdfd4pt294eKl8B6/I1hV+HvLKJNd19yVJrls8BwBgDewYXFbVdbt5bWDfi5N8T5LXnPLyi5Jcs3h8TZIX7+ZYAACsvsGyeFWdleScJBdU1WOTPJSzPS/J1+7y+D+f5MeTPOqU157Y3XckSXffUVVPGPj8q5JclSRn5ZxdfhwAwCHSOVKXf/zBJD+SrUDyhnw5uPxskl/c6cBV9b1JTnT3DVX13NMdWHdfneTqJDmvzl+zaQcAWE+DwWV3vzrJq6vqh7v7F/Zw7OckeWFVfXe2+mOeV1W/muTOqrpwkbW8MMmJPY0cAGAdrFkKbTetiDar6jHd/ekkWZTIX9rd/9fYTt39k0l+crHPc5P8m+7+/qr62SQvS/Kqxf1b9jx6Vtp+WsBoEcOp9vN78FsCOFg7LuhJ8s8fCiyTZNE26J/v4zNfleQFVfWhJC9YPAcAYA3sJnP5iKqq7u7kS30rzzidD+nudyV51+Lxp5JccXrDBABYT0fxCj3vSPLGqvoP2Tor4F8k+d2ljgoAgENpN8HlT2SrJdC/zNaK8T9NcuEyBwUAcGSsWeZyN1fo2UzyX5J8OMnl2Spp3zK6EwAAR9JYE/VvSPKSJC9N8qlsXQ883f28gxkaAACHzVhZ/INJ/ijJf9/df5kkVfWjBzIqAICjYs3K4mPB5f+QrczlH1TV25P8er58lR7Ykf6CrLP99HEFWGeD51x295u7+39M8o3ZaiP0o0meWFW/VFXfeUDjAwBYW9Xz36a2mwU993b367v7e5NcnOTGJK+cfigAABx2u7lCz5d0993d/R+7+/nLGhAAAIfXbvpcAgCwLL1eS1pOK3MJAMDRU1UbVfWnVfW2nd4ruAQAYCevyC4voqMsjpYqrLdnfdPwtvfetOfDjv3b8G8KOC0r3ueyqi5O8j1JfibJ/7LT+2UuAQCOtguq6vpTblc9bPvPJ/nxJJu7OZjMJQDAjJbRa/I03dXdl2+3oaq+N8mJ7r6hqp67m4PJXAIAMOQ5SV5YVR/J1tUan19Vvzq2g+ASAIBtdfdPdvfF3f2kbF0W/Pe7+/vH9lEWBwCY0/xl8UkJLgEA2FF3vyvJu3Z6n+ASrVHW1Fg7nDHr9ns4dvvdg9seXNJnrtscAkvUK7GgZ1LOuQQAYDKCSwAAJqMsDgAwJ2VxAADYnswlAMCcZC4BAGB7gksAACajLA4TGOspuZ+eh/s5rl6LWw7bPCzrtwSsLn0uAQBggOASAIDJCC4BAJiM4BIAgMlY0AMAMCcLegAAYHsyl0eE9ibLtaw59LdZP2P/FhN/czhyWisiAAAYJLgEAGAyyuIAAHNSFgcAgO0JLgEAmIyyOADAnNasLC64XBPam8Dh4N/ivLRlg+UTXAIAzKSizyUAAAwSXAIAMBllcQCAOSmLAwDA9mQuAQDm0hb0AADAIJnLNaE/2/7pFQrrz79jWD7BJQDAnJTFAQBgezKXAABzkrkEAIDtCS4BAJiMsjgAwIzWrc+l4BIWtChhKmNtrXb6ne1nX4BVILgEAJjTmmUunXMJAMBkBJcAAExGWRwAYC4dZXEAABgicwkAMCOtiIBDT7ub5drPHK7a/I/9VpLVGy8wv6UGl1X1kSSfS3IyyYPdfXlVnZ/kN5I8KclHkvyT7v7bZY4DAICDcRDnXD6vuy/r7ssXz1+Z5LruviTJdYvnAABHU898m9gcC3pelOSaxeNrkrx4hjEAALAEyw4uO8k7q+qGqrpq8doTu/uOJFncP2HJYwAA4IAse0HPc7r79qp6QpJrq+qDu91xEYxelSRn5ZxljQ8AYFbrtlp8qZnL7r59cX8iyZuTPCvJnVV1YZIs7k8M7Ht1d1/e3Zcfz5nLHCYAABNZWnBZVedW1aMeepzkO5N8IMlbk7xs8baXJXnLssYAALDy1mxBzzLL4k9M8uaqeuhzfq27315V70vyxqp6eZKPJvm+JY4B2IbehJxK31NgSksLLrv7w0mesc3rn0pyxbI+FwCA+bhCDwDAXJZUmp7THH0uAQBYUzKXAAAzqcVtnchcAgAwGcElAACTURYHWANj7YSS8ZZC2g3BzCzoAQCA7clcAgDMyLXFAQBggOASAIDJKIsDAMxJWRwAALYncwkAMKc1y1wKLuEIGuuJqOfh4bSKfze/Mzj8quqsJO9Ocma24sY3dfdPje0juAQAYMgXkzy/u++pquNJ/riqfre7/8vQDoJLAIC59Gr3uezuTnLP4unxxW10xBb0AAAcbRdU1fWn3K46dWNVbVTVjUlOJLm2u98zdjCZSwCAo+2u7r58aGN3n0xyWVU9Jsmbq+rp3f2BoffLXAIAzKlnvu12mN2fTvKuJFeOvU9wCQDAtqrq8YuMZarq7CTfkeSDY/soiwOwFNoNwe6s8oKeJBcmuaaqNrKVlHxjd79tbAfBJQAA2+ruP0/yzNPZR1kcAIDJyFwCAMxptcvip03mEgCAychcAgDMaMUX9Jw2mUsAACYjcwlHkBYxcPqOXXzR4Db/puDLBJcAAHM5zavkHAbK4gAATEbmEgBgTjKXAACwPcElAACTURYHAJhJRZ9LAAAYJHMJsAbGejDuRI/G3TFPLI3MJQAAbE9wCQDAZJTFAQBmVL1edXGZSwAAJiNzCQAwF9cWBwCAYTKXAGvgsLXJ2al10mH7PsCXCS4BAGbkCj0AADBAcAkAwGSUxQEA5qQsDgAA25O5BACY0bot6BFcrpix9hxacwDrwn/PYH0piwMAMBmZSwCAOa1ZWVzmEgCAychcAgDMpddvQY/MJQAAkxFcAgAwGWVxAIA5rVlZXHC5YvR+Axjv+TvGf0NhfoJLAICZVCzoAQCAQYJLAAAmoywOADCnXq+6uMwlAACTkbkEAJjRui3oEVwCsHLmaCm0U/sjbY5gd5ZaFq+qx1TVm6rqg1V1S1V9a1WdX1XXVtWHFvePXeYYAAA4OMs+5/LVSd7e3d+Y5BlJbknyyiTXdfclSa5bPAcAOHp6BW4TW1pwWVXnJfn2JK9Nku6+v7s/neRFSa5ZvO2aJC9e1hgAADhYyzzn8ilJPpnkl6vqGUluSPKKJE/s7juSpLvvqKonbLdzVV2V5KokOSvnLHGYAADzqc25RzCtZZbFjyX55iS/1N3PTHJvTqME3t1Xd/fl3X358Zy5rDECADChZQaXtyW5rbvfs3j+pmwFm3dW1YVJsrg/scQxAABwgJZWFu/uT1TVx6rq0u6+NckVSf5icXtZklct7t+yrDEAwG4tq9WQFkfsSJ/L0/LDSV5fVWck+XCSH8hWtvSNVfXyJB9N8n1LHgMAAAdkqcFld9+Y5PJtNl2xzM8FAGAertADADCjdbv847KbqAMAcITIXAIAzKWT9HqlLmUuAQCYjOASAIDJKIsDHBJj/RL1Slxd/jbsxIIeAAAYIHMJADAnmUsAANie4BIAgMkoiwMAzKRiQQ8AAAySuQQ4JLS02dlYu6bEHLKCul2hBwAAhgguAQCYjOASAGBG1fPeRsdW9XVV9QdVdUtV3VxVr9jp+zjnEgCAIQ8m+bHufn9VPSrJDVV1bXf/xdAOgksAgDmt8Hqe7r4jyR2Lx5+rqluSXJREcAkAwLYuqKrrT3l+dXdf/fA3VdWTkjwzyXvGDia4BE7LWKsXbV6Ym98g7Mld3X352Buq6pFJfivJj3T3Z8feK7gEAJjRql+hp6qOZyuwfH13//ZO77daHACAbVVVJXltklu6++d2s4/gEgCAIc9J8k+TPL+qblzcvntsB2VxAIC5dJLN1a2Ld/cfJ6nT2UfmEgCAychcAgDMaXUTl3sicwkAwGRkLoHToo8gfLWx/q878W+KdSO4BACY0ar3uTxdyuIAAExG5hIAYE69XqlLmUsAACYjuAQAYDLK4gAAM1q3BT2CSwDYp2W1ExprcaSFEatKcAkAMJeOK/QAAMAQwSUAAJNRFgcAmEklKX0uAQBgezKXAABz2px7ANOSuQQAYDKCSwAAJqMsDgAwIwt6AABggMwlAMBcXKEHAACGCS4BAJiMsjgAwGw6WbMFPYJLAJjRsYsvGtz24G0fP8CRwDSUxQEAmIzMJQDAjGq9quIylwAATEfmEgBgTmu2oEfmEgCAyQguAQCYjLI4wMS0luF0+E0ccZ3U5tyDmJbMJQAAk5G5BACYkwU9AACwPcElAACTURYHAJjTelXFZS4BAJiOzCXAxLSW4SCMtbxK/A4Pk7KgZ3eq6tKquvGU22er6keq6vyquraqPrS4f+yyxgAAwMFaWnDZ3bd292XdfVmSv5/kviRvTvLKJNd19yVJrls8BwBgDRzUOZdXJPmr7v6bJC9Kcs3i9WuSvPiAxgAAsHq6571N7KCCy5ckecPi8RO7+44kWdw/Ybsdquqqqrq+qq5/IF88oGECALAfS1/QU1VnJHlhkp88nf26++okVyfJeXX+ep3pCgCQbLUhcm3x0/ZdSd7f3Xcunt9ZVRcmyeL+xAGMAQCAA3AQweVL8+WSeJK8NcnLFo9fluQtBzAGAAAOwFLL4lV1TpIXJPnBU15+VZI3VtXLk3w0yfctcwwAsI6W1cdS/8yDVem163O51OCyu+9L8riHvfapbK0eBwBgzbhCDwDAnNYsc+na4gAATEZwCQDAZJTFAQDmpCwOAADbk7kEgCNmrN2QVkPsl+ASAGAuLv8IAADDZC4BAGa0blfokbkEAGAygksAACajLA4AMCdlcQAA2J7MJQCsoTl6WeqfuRctcwkAAEMElwAATEZZHABgLh1lcQAAGCK4BACY0+bMtx1U1euq6kRVfWA3X0dwCQDAmF9JcuVu3+ycSwBYQ3O0/hn7zLE2RTvty7y6+91V9aTdvl9wCQAwo5p/Qc8FVXX9Kc+v7u6r93owwSUAwNF2V3dfPtXBBJcAAHOaP3M5KQt6AACYjOASAIBBVfWGJH+S5NKquq2qXj72fmVxAIC5dJLN1S6Ld/dLT+f9gksAYNfGWgqNtRPSaujoEFwCAMymLegBAIAhgksAACajLA4AMCdlcQAA2J7gEgCAySiLAwDMac3K4oJLAGDXltWvcq/9M1k9gksAgLkcgiv0nC7nXAIAMBnBJQAAk1EWBwCYTSe9OfcgJiVzCQDAZGQuAQDmpBURAMC09tpuaKyF0X6Oy94piwMAMBmZSwCAuehzCQAAw2QuAQDmtGYLemQuAQCYjOASAIDJKIsDACttrN3QWrQaUhYHAIDtyVwCAMymZS4BAGCI4BIAgMkoiwMAzKWTbG7OPYpJyVwCADAZwSUAAJNRFgcAVtp+elmO9cjMx/Z82GlZLQ4AANuTuQQAmJPMJQAAbE9wCQDAZJTFAQBm08mmsjgAAGzrUGQuP5e/vev3+k1/c8pLFyS5a67xHBLmaGfmaHfM087M0e6Yp52Zo52d3hyNtxv6r/Y5lv3rpHu9rtBzKILL7n78qc+r6vruvnyu8RwG5mhn5mh3zNPOzNHumKedmaOdmaPVpywOAMBkDkXmEgBgbVnQsxKunnsAh4A52pk52h3ztDNztDvmaWfmaGfmaMVVr1lXeACAw+LRxx7f3/qoF806hnd8+rU3THke62HNXAIAsIIElwAATOZQBZdVdWVV3VpVf1lVr5x7PKuiql5XVSeq6gOnvHZ+VV1bVR9a3D92zjHOraq+rqr+oKpuqaqbq+oVi9fN00JVnVVV762qP1vM0U8vXjdHD1NVG1X1p1X1tsVzc/QwVfWRqrqpqm6squsXr5mnU1TVY6rqTVX1wcV/m77VHH2lqrp08Rt66PbZqvqRtZqn7mRzc97bxA5NcFlVG0l+Mcl3JXlqkpdW1VPnHdXK+JUkVz7stVcmua67L0ly3eL5UfZgkh/r7r+b5NlJfmjx+zFPX/bFJM/v7mckuSzJlVX17Jij7bwiyS2nPDdH23ted192yrlc5ukrvTrJ27v7G5M8I1u/KXN0iu6+dfEbuizJ309yX5I3xzyttEMTXCZ5VpK/7O4Pd/f9SX49ybxnwK6I7n53krsf9vKLklyzeHxNkhcf5JhWTXff0d3vXzz+XLb+I35RzNOX9JZ7Fk+PL24dc/QVquriJN+T5DWnvGyOdsc8LVTVeUm+Pclrk6S77+/uT8ccjbkiyV91999k3eape97bxA5TcHlRvvIiTrctXmN7T+zuO5KtwCrJE2Yez8qoqicleWaS98Q8fYVFuffGJCeSXNvd5uir/XySH09yai3JHH21TvLOqrqhqq5avGaevuwpST6Z5JcXp1i8pqrOjTka85Ikb1g8Nk8r7DAFl7XNa/oocVqq6pFJfivJj3T3Z+cez6rp7pOL8tPFSZ5VVU+feUgrpaq+N8mJ7r5h7rEcAs/p7m/O1qlMP1RV3z73gFbMsSTfnOSXuvuZSe6N0u6gqjojyQuT/ObcY2Fnhym4vC3J153y/OIkt880lsPgzqq6MEkW9ydmHs/squp4tgLL13f3by9eNk/bWJTn3pWtc3nN0Zc9J8kLq+oj2To15/lV9asxR1+lu29f3J/I1jlyz4p5OtVtSW5bVAeS5E3ZCjbN0fa+K8n7u/vOxfO1mqfe3Jz1NrXDFFy+L8klVfXkxf/BvCTJW2ce0yp7a5KXLR6/LMlbZhzL7KqqsnVu0y3d/XOnbDJPC1X1+Kp6zOLx2Um+I8kHY46+pLt/srsv7u4nZeu/Qb/f3d8fc/QVqurcqnrUQ4+TfGeSD8Q8fUl3fyLJx6rq0sVLVyT5i5ijIS/Nl0viiXlaaYfqCj1V9d3ZOt9pI8nruvtn5h3RaqiqNyR5bpILktyZ5KeS/L9J3pjk65N8NMn3dffDF/0cGVX13yb5oyQ35cvnyv3bbJ13aZ6SVNXfy9aJ8RvZ+h/PN3b3v6uqx8UcfZWqem6Sf9Pd32uOvlJVPSVb2cpkq/z7a939M+bpK1XVZdlaGHZGkg8n+YEs/u3FHH1JVZ2TrTUXT+nuzyxeW5vf0qM3HtfPPut7Zh3DO+/7fya9Qs+hCi4BANbJOgaXh6ksDgDAijs29wAAAI6sTrK5XlVkmUsAACYjuAQAYDLK4gAAc+rpe03OSeYSWClVdbKqbqyqD1TVby7akOz1WL9SVf948fg1VfXUkfc+t6q+ba+fBcAWwSWwaj7f3Zd199OT3J/kX5y6sao29nLQ7v6fu/svRt7y3CSCS+BAdZLe7FlvUxNcAqvsj5L8nUVW8Q+q6teS3FRVG1X1s1X1vqr686r6wWTrSkxV9X9W1V9U1X9O8oSHDlRV76qqyxePr6yq91fVn1XVdVX1pGwFsT+6yJr+g4P/qgDrwTmXwEqqqmPZup7w2xcvPSvJ07v7r6vqqiSf6e7/pqrOTPL/VdU7kzwzyaVJvinJE7N1Ob3XPey4j0/yfyf59sWxzu/uu6vqPyS5p7v//YF8QYA1JbgEVs3ZVXXj4vEfZeua8N+W5L3d/deL178zyd976HzKJI9OckmSb0/yhu4+meT2qvr9bY7/7CTvfuhYh/WSccCa6F67BT2CS2DVfL67Lzv1hapKkntPfSnJD3f3Ox72vu/O1ilMY2oX7wFgj5xzCRxG70jyL6vqeJJU1TdU1blJ3p3kJYtzMi9M8rxt9v2TJP+wqp682Pf8xeufS/Ko5Q8d4CtZ0AMwv9dk63zK91fVB5L8x2xVYt6c5ENJbkryS0n+8OE7dvcnk1yV5Ler6s+S/MZi039K8o8s6AHYn+pWHQIAmMN5dX5/yyNeMOsYfm/zjTd09+VD26vqyiSvTrKR5DXd/aqx4znnEgBgTiu8oGfRW/gXk7wgyW1J3ldVbx3rG6wsDgDAkGcl+cvu/nB335/k15O8aGwHmUsAgJl8Ln/7jt/rN10w8zDOqqrrT3l+dXdfvXh8UZKPnbLttiTfMnYwwSUAwEy6+8q5x7CD2ua10QU7yuIAAAy5LcnXnfL84iS3j+0guAQAYMj7klxSVU+uqjOSvCTJW8d2UBYHAGBb3f1gVf2rbF28YiPJ67r75rF99LkEAGAyyuIAAExGcAkAwGQElwAATEZwCQDAZASXAABMRnAJAMBkBJcAAEzm/wdksHT/AqVEJAAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" }, { "data": { "text/plain": [ "''" ] }, "execution_count": 74, "metadata": {}, "output_type": "execute_result" } ], "source": [ "plot_confusion_matrix(df_hat)\n", ";\n" ] }, { "cell_type": "code", "execution_count": null, "id": "5041c7d2", "metadata": {}, "outputs": [], "source": [ "\n" ] }, { "cell_type": "code", "execution_count": null, "id": "7f535f91", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "id": "a5181e5e", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": 79, "id": "1487c96a", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Confusion Matrix: \n", "Number of records: 71, number of columns: 70\n", "MAE: 1.0903\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqAAAAKgCAYAAABEPM/FAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAr8klEQVR4nO3dfbBtdXkn+O8jQQm+ASJXBBPRIUk7zvgyxKBUMihtBl8ido3WYKYTxnGajp34kkqqQ/wjmUxNpazqrlQk2tq31UiqjWnbaKTVlhBaW9u3cCX4BlqgRr0BIWLEFxyEe5754+ybHK/37H3uPXettc/Zn0/Vrr3XWnut9dvrbC5PPc/+Pau6OwAAMJb7TD0AAABWiwAUAIBRCUABABiVABQAgFEJQAEAGNUPTT0AAIBV9b889f59x9cPTDqGj3/y7qu6+8IxzykABQCYyB1fP5C/vOpHJh3DcaffdOrY51SCBwBgVAJQAABGpQQPADCRTrKWtamHMToZUAAARiUDCgAwmc6BlgEFAIBBCUABABiVEjwAwETWJyH11MMYnQwoAACjkgEFAJiQNkwAADAwASgAAKNSggcAmEinc6BNQgIAgEHJgAIATEgbJgAAGJgAFACAUSnBAwBMpJMcUIIHAIBhyYACAEzIJCQAABiYABQAgFEpwQMATKQTd0ICAIChyYACAExobeoBTEAGFACAUQlAAQAYlRI8AMBEOu1OSAAAMDQBKAAAo1KCBwCYSicHVq8CLwMKAMC4ZEABACbS0QcUAAAGJwAFAGBUSvAAAJOpHEhNPYjRyYACADAqGVAAgIl0kjVtmAAAYFgCUAAARqUEDwAwIZOQAABgYDKgAAAT6ciAAgDA4ASgAACMSgkeAGBCa60EDwAAg5IBBQCYiElIAAAwAgEoAACjUoIHAJhIp3JgBfOBq/eJAQCYlAwoAMCEVrEN044IQI+/7/37hBNPPqp968675m7vB584d/uB+23+pVg7fv65zzz5jrnbT7rP2vwDzPGle+4/d/sJ97ln0233q3vn7nvL//fg+Sf/5uZfm0XX5Phv96bb7nnA/P8A53ykheeed95k+9+T7Rx7O+fdzrEBVt238ndf6+6HTj2OVTRJAFpVFyZ5VZLjkry+u1857/0nnHhynvDTLz2qc93v3dfO3X73T//k3O3fePTmUc1dD5sf1Pzr518xd/tz7n/0wcO/+Jtz527/sRO/uum2R9339rn7/t83/tzc7Wt/8ZBNty26Jg//0ObB7y3nzf86nvjV+QHqvHPPO2+y/e/Jdo69nfNu59gAq+4v+m1fmnoMq2r0ALSqjkvymiRPT7I/ybVVdWV33zD2WAAApqQP6HielOTm7v5Cd38vyZ8kuWiCcQAAMIEpAtAzknxlw/L+2brvU1WXVtW+qtp3z/e+M9rgAAAY1hS/AT1cnvkHfrzX3XuT7E2SB5505vwfFgIA7EiVA716XTGn+MT7kzxiw/KZSW6ZYBwAAExgigzotUnOrqqzkvxNkouT/Py8HerOu456tu/dz9reLOI92zj2b331/5i7/dfnzNq+55QDc/d98A3z/3QfnrPtpM/P72f07QWz0R8+d//5fZjmXe+z3j13121Z9LdaZN64t3vsoz0vADtfJ1lbwfsCjR6Adve9VfUrSa7KehumN3b3Z8YeBwAA05ikD2h3vyfJe6Y4NwAA09oRd0ICANit9AEFAICByYACAEykWxsmAAAY3K7PgE7ZxuY75941d/v9P3ripttO+tD83vu3nDd/+8M/dO+m2xZdk4fn6FtXzWtblcxvWbTdv9V2jn3bS58yd/ueyzdvbKVVEgAcmV0fgAIALLM1k5AAAGBYMqAAABPpJAdWMB+4ep8YAIBJCUABABiVEjwAwGT0AQUAgMHJgG7Dov6PZyzop/mln9u8V+dJnz+qIf29eWOb1y9z0b5b2X+ev/nf79l0W5335Ln7nvWKj8zdvp3PPK/P56ra7vcEgMU6ydoK5gNX7xMDADApASgAAKNSggcAmNCBdickAAAYlAAUAIBRKcEDAEykU27FCQAAQ5MB3YbbXvqUudtP+vzmPS8XmdcvM0ke9fPzezDOG9t2xrXIot6Qj3r35tsWXc/tGLJn5W7tl7lTxw2w06y5ExIAAAxLAAoAwKiU4AEAJtKJSUgAADA0GVAAgIl0yp2QAABgaDKgS+pRP3/93O2LWv/c9bDedNtJn59/7kXtkPZc/uGjHtfRHneZaVcEAEdGAAoAMKG1FSxIr94nBgBgUjKgAAAT6U4OuBMSAAAMSwAKAMColOABACZTWYs+oAAAMCgZ0G3Ybt/KH3v35tsW9dNc1HvyrDnHXmTPgu3zxrasPTG3ez0BYAgdk5AAAGBwAlAAAEalBA8AMKEDK5gPXL1PDADApASgAAAT6VTWetrHPFX1iKp6X1XdWFWfqaqXHeY951fVnVV1/ezxW4s+txI8AACbuTfJr3X3dVX1wCQfr6qru/uGQ973we5+9lYPKgMKAMBhdfet3X3d7PW3ktyY5IztHlcGdBu221ty3v63nDf/T3Pio58yd/s8J33+nrnbh+yJedtLNx/3or6qenkCsBstwSSkU6tq34blvd2999A3VdUjkzwhyccOc4wnV9UnktyS5Ne7+zPzTigABQBYbV/r7nPmvaGqHpDkT5O8vLu/ecjm65L8aHd/u6qemeTPkpw973iTh9wAACyvqjo+68Hnm7v77Ydu7+5vdve3Z6/fk+T4qjp13jFlQAEAJtJJ1pb4VpxVVUnekOTG7v69Td7zsCS3dXdX1ZOynuC8Y95xBaAAAGzmvCS/kORTVXX9bN0rkvxIknT365I8L8mLq+reJN9NcnF397yDCkABACZTOZD5vTin1N3/LZk/wO5+dZJXH8lxlzfnCwDAriQDug1Dtv056xUfmbv9i7/75G0c/fi5W/cs2HuqdkfbaWs15JgXtYdaZKe2j5rqegOw8wlAAQAmsuyTkIayep8YAIBJyYACAExomSchDUUGFACAUQlAAQAYlRI8AMBEusskJAAAGJoM6IAW9Yf8xqM378e5qBfnwz9071GMaGsWjXtej8dF/UnPesWHj2pMWzFvXNv5TNs57262qp8b4Fg7IAMKAADDEoACADAqJXgAgIl0kjV9QAEAYFgyoAAAkymTkAAAYGgCUAAARqUEP6BFfRLvmtMzc1HfykVuOW/zP+1Zr/jI3H1ve+lT5m4/ac7YFh17KtvtWTnv76EfJgBHq5OstUlIx0xVvbGqbq+qT29Yd0pVXV1VN82eTx7q/AAALKchS/BvSnLhIesuS3JNd5+d5JrZMgDAyjqQ+0z6mMJgZ+3uDyT5+iGrL0pyxez1FUmeO9T5AQBYTmOHvXu6+9YkmT2fttkbq+rSqtpXVfvuyd2jDRAAgGEt7SSk7t6bZG+SPKhO6YmHAwBwzHXKJKQR3FZVpyfJ7Pn2kc8PAMDExs6AXpnkkiSvnD2/c+Tzj2pRK6XttCwa8tgnff6eudvntXh6eI6+fdSidkaLPvOQ7ZC0WgKAY2ewALSq3pLk/CSnVtX+JL+d9cDzrVX1oiRfTvL8oc4PALATrK3gfYEGC0C7+wWbbLpgqHMCALD8lnYSEgDAbtedHDAJCQAAhiUABQBgVErwAAAT0gcUAAAGJgM6oO30tVy075Q9L89692CnnmtZe3FO2Z90ke18xwAY3vqdkFYvH7h6nxgAgEkJQAEAGJUSPADAhA7EJCQAABiUDCgAwEQ62jABAMDgBKAAAIxKCZ5jalFPzO0Ysm/lkOMe0jL3IAVgK/QBBQCAwcmAAgBMaE0bJgAAGJYAFACAUSnBAwBMpDs5oA8oAAAMSwZ0QttpkbNT2+8Y17G1U8cNwGoTgAIATEgfUAAAGJgMKADARDqVNZOQAABgWAJQAABGpQQPADAht+IEAICByYBuw5S9OJe1/+OyjgsAllEnJiEBAMDQBKAAAIxKCR4AYELuhAQAAAOTAQUAmEq7ExIAAAxOAAoAwKiU4LdhUc/L7fQJnbLH6CLzxvaNRx8/d989l394kPMmO/d6TmXRNVlkFa8ZwLHWcSckAAAYnAwoAMCETEICAICBCUABABiVEjwAwEQ6SvAAADA4GdABbadNzaJ9v/i7T567/axXfOSoz70d22mztMiQ13NK222HNM+8z72srbyS5f57ARxrMqAAADAwASgAAKNSggcAmEinlOABAGBoAlAAAEalBA8AMKG1KMEDAMCgZEB3qKn6fCY7s0fjdvtOztt/O/tuZf/tHHtZ7cTvEMAgWh9QAAAYnAAUAIBRKcEDAEykowQPAACDkwEFAJiQDCgAAAxMAAoAwKiU4Hep7fSH3E5fy288+vijPu+eyz981Psm2+vVuch29h+y56V+mgA7W6eU4AEAYGgyoAAAE2oZUAAAGJYAFACAUSnBAwBMaC1K8AAAMCgZ0Akta9ug7Rx7z4J9t9MeapEhP/OQfyt2D98T4Eh1uxMSAAAMbrAAtKoeUVXvq6obq+ozVfWy2fpTqurqqrpp9nzyUGMAAGD5DJkBvTfJr3X3P0pybpJfrqrHJLksyTXdfXaSa2bLAAArqbsmfUxhsAC0u2/t7utmr7+V5MYkZyS5KMkVs7ddkeS5Q40BAIDlM8okpKp6ZJInJPlYkj3dfWuyHqRW1Wmb7HNpkkuT5IScOMYwAQBG5l7wg6iqByT50yQv7+5vbnW/7t7b3ed09znH537DDRAAgFENGoBW1fFZDz7f3N1vn62+rapOn20/PcntQ44BAICjs9mk8kPeU1V1eVXdXFWfrKonLjruYCX4qqokb0hyY3f/3oZNVya5JMkrZ8/vHGoMy25eX8BF/TIX9RRc1n6EO7UX4k4dN+PyPQGOxlQTgbbo4KTy66rqgUk+XlVXd/cNG97zjCRnzx4/leS1s+dNDZkBPS/JLyR5WlVdP3s8M+uB59Or6qYkT58tAwCwZOZMKt/ooiR/1Os+muSkg9XuzQyWAe3u/5ZsenPTC4Y6LwAAR+TUqtq3YXlvd+899E2HTCrf6IwkX9mwvH+27tbNTuhWnAAAE+ksxa04v9bd58x7w4JJ5Yf7AD3veG7FCQDApjaZVL7R/iSP2LB8ZpJb5h1TAAoAMJVOeuLHPHMmlW90ZZJfnM2GPzfJnQd7vm9GCR4AgM0cnFT+qaq6frbuFUl+JEm6+3VJ3pPkmUluTnJXkhcuOqgAFACAw1owqfzgezrJLx/JcQWgu9R2eoxu59iraLs9W3fquQE4Ntbmx3e7kt+AAgAwKhlQAICJdJb+TkiDkAEFAGBUAlAAAEalBA8AMJlahjshjU4GFACAUcmALqkh2+dozXNsLbqeQ7ZKWsW/pdZTwG6z6G5Eu5EMKAAAoxKAAgAwKiV4AIAJ6QMKAAADkwEFAJhItwwoAAAMTgAKAMColODZNeb1h5yyN6S+lMeW6wnsNu6EBAAAAxOAAgAwKiV4AIAJuRUnAAAMTAYUAGBC+oACAMDABKAAAIxKCZ5dY6r+kPP6jyY7t2/lbS99ytztey7/8EgjAdi9OqUEDwAAQ5MBBQCY0Ap2YZIBBQBgXAJQAABGpQQPADCV1gcUAAAGJwMK27RT2ywtsqxtlnZr2ytgha3gLCQZUAAARiUABQBgVErwAAATMgkJAAAGJgMKADChNgkJAACGJQAFAGBUSvA7lF6I7GaLvt8Au0XHJCQAABicDCgAwFQ6iQwoAAAMSwAKAMColOABACakDygAAAxMAAoAwKiU4HcofT6Pre30nRzyb7Gq/V536+cCOCwleAAAGJYMKADAZMqdkAAAYGgCUAAARqUEDwAwJZOQAABgWDKgE5rXYkcbmmNrp7YzWtZxLbJTr/d22nEly/u5gCXWMQkJAACGJgAFAGBUSvAAAFMyCQkAAIYlAwoAMCmTkAAAYFACUAAARqUEPyE9A8fjWv+g7fbq3I19bHfquIEdziSkY6eqTqiqv6yqT1TVZ6rqd2brT6mqq6vqptnzyUONAQCA5TNkCf7uJE/r7scleXySC6vq3CSXJbmmu89Ocs1sGQBgNfXEjwkMFoD2um/PFo+fPTrJRUmumK2/IslzhxoDAADLZ9BJSFV1XFVdn+T2JFd398eS7OnuW5Nk9nzaJvteWlX7qmrfPbl7yGECADCiQSchdfeBJI+vqpOSvKOqHnsE++5NsjdJHlSnrODPcwGAXa+TtD6gg+jubyR5f5ILk9xWVacnyez59jHGAADAchhyFvxDZ5nPVNUPJ/nHST6b5Mokl8zedkmSdw41BgCAZdc97WMKQ5bgT09yRVUdl/VA963d/a6q+kiSt1bVi5J8OcnzBxwD7Gjb7dW5nX2HPPdutRt7owIMYbAAtLs/meQJh1l/R5ILhjovAADLzZ2QAACmtIJTrd0LHgCAUQlAAQAYlRI8AMCU9AEFAIBhyYCya2iBc2y5ZgDjKJOQAABgWAJQAABGpQQPADCVjj6gAAAwNBlQAIDJlDZMAAAwNAEoAACjUoJn19iNfSt342dK5vdsTYb93EP2i92tfy9gYCYhAQDAsGRAAQCmJAMKAADDEoACADAqJXgAgCkpwQMAwLA2zYBW1R9kTkze3S8dZESwy0zZcmhZTfmZd+P19h2DHayzkndCmleC3zfaKAAAWBmbBqDdfcWYAwEAYDUsnIRUVQ9N8htJHpPkhIPru/tpA44LAGAllElIh/XmJDcmOSvJ7yT56yR+UAQAwFHZSgD6kO5+Q5J7uvu/dvf/meTcgccFALAaeuLHBLbSB/Se2fOtVfWsJLckOXO4IQEAsJttJQD9f6vqwUl+LckfJHlQkl8ddFQAAOxaCwPQ7n7X7OWdSZ467HA4SF+/H7RTr8myjosjt6zfwSm/Y8t6TYDltpVZ8H+Yw/xCYPZbUAAAdrGqemOSZye5vbsfe5jt5yd5Z5Ivzla9vbv/n3nH3EoJ/l0bXp+Q5J9k/XegAADsfm9K8uokfzTnPR/s7mdv9YBbKcH/6cblqnpLkr/Y6gkAANjcsvcB7e4PVNUjj+Uxt9KG6VBnJ/mRYzkIAAAmc2pV7dvwuPQojvHkqvpEVf3nqvrvF715K78B/Va+/zegX836nZEAANiurqlH8LXuPmcb+1+X5Ee7+9tV9cwkf5b1hOWmtlKCf+A2BgQAwC7W3d/c8Po9VfVvqurU7v7aZvssLMFX1TVbWQcAwOqpqodVVc1ePynr8eUd8/bZNANaVSckOTHrvws4OcnB/PCDkjz8mIyYTemd94OWtdehv9Xq8Lf+Qa4JbNOEt8PcqtkE9POzHhPuT/LbSY5Pku5+XZLnJXlxVd2b5LtJLu7uuZ9qXgn+nyd5edaDzY/nHwLQbyZ5zVF/CgAAdozufsGC7a/OepumLds0AO3uVyV5VVW9pLv/4EgOCgDAFi15BnQIW2nDtFZVJx1cqKqTq+pfDDckAAB2s60EoP+su79xcKG7/y7JPxtsRAAA7GpbuRXnfaqqDv6YtKqOS3LfYYcFALAalv1OSEPYSgB6VZK3VtXrsv4rhV9K8p8HHRUAALvWVgLQ30hyaZIXZ30m/F8lOX3IQcGy0Wpm57jtpU+Zu33P5R8eaSQAW7SCGdCFvwHt7rUkH03yhSTnJLkgyY0DjwsAgF1qXiP6H0tycZIXZL2b/X9Iku5+6jhDAwBgN5pXgv9skg8m+bnuvjlJqupXRxkVAMCqUIL/Pv9rkq8meV9V/buquiD/cDckAAA4KpsGoN39ju7+35L8RJL3J/nVJHuq6rVV9bMjjQ8AYNeqnv4xha1MQvpOd7+5u5+d5Mwk1ye5bOiBAQCwO23lTkh/r7u/3t3/trufNtSAAADY3bbSBxRgx1jU5/PuZ/3kptv0ewUm0as3xeaIMqAAALBdAlAAAEalBA8AMCV9QAEAYFgyoAAAE5qqF+eUZEABABiVABQAgFEpwXNMzeuxmOizyPR8B4GlowQPAADDkgEFAJhKm4QEAACDE4ACADAqJXgAgCkpwQMAwLBkQFfQkK2StLhh2X3xd5+86bazXvGRwc6rRRmwKRlQAAAY1uABaFUdV1V/VVXvmi2fUlVXV9VNs+eThx4DAADLY4wM6MuS3Lhh+bIk13T32UmumS0DAKyk6mkfUxg0AK2qM5M8K8nrN6y+KMkVs9dXJHnukGMAAGC5DJ0B/f0k/zLJ2oZ1e7r71iSZPZ92uB2r6tKq2ldV++7J3QMPEwCAsQwWgFbVs5Pc3t0fP5r9u3tvd5/T3eccn/sd49EBADCVIdswnZfkOVX1zCQnJHlQVf37JLdV1endfWtVnZ7k9gHHAADAkhksAO3u30zym0lSVecn+fXu/qdV9a+SXJLklbPndw41Bg5Pv0GS1e1L+fAP3TvJeXfr9QSOAX1AR/HKJE+vqpuSPH22DADAihjlTkjd/f4k75+9viPJBWOcFwBgqU3YCmlK7oQEAMCoBKAAAIxqlBI8AACbUIIHAIBhCUABABiVEjwssSF7da5qX8pV/dzAElOCBwCAYcmAAgBMpKIPKAAADE4ACgDAqJTgAQCmpAQPAADDkgHdpea179GGZufwt1odQ7bcApZYm4QEAACDE4ACADAqJXgAgCkpwQMAwLBkQAEApiQDCgAAwxKAAgAwKiX4XUrPQNhZ/Dc7Lr2SWSb6gAIAwMBkQAEApiQDCgAAwxKAAgAwKiV4AICpdJTgAQBgaDKgAAATWsU2TAJQyPyegIm+gLDb+G8apqUEDwDAqGRAAQCmtIIleBlQAABGJQAFAGBUSvAAABNaxVnwMqAAAIxKBhSiJQvD226rr3n7+/7CDicDCgAAwxKAAgAwKiV4AICpdJTgAQBgaDKgAAATqdlj1ciAAgAwKgEoAACjUoIHGMF2e3VO1etzu/1LgS0wCQkAAIYlAwoAMCH3ggcAgIEJQAEAGJUSPADAlJTgAQBgWDKgAABTWsEMqAAUYMXN6/WpzycwBCV4AABGJQMKADCV1gcUAAAGJwAFAGBUSvAAAFNSggcAgGHJgMKKmtd6J9F+Z5UM+bfW4gkWMwkJAAAGJgAFAGBUAlAAgCn1xI8FquqNVXV7VX16k+1VVZdX1c1V9cmqeuKiYwpAAQCY501JLpyz/RlJzp49Lk3y2kUHNAkJAGBCyz4Jqbs/UFWPnPOWi5L8UXd3ko9W1UlVdXp337rZDjKgAACr7dSq2rfhcekR7n9Gkq9sWN4/W7cpGVAAgNX2te4+Zxv712HWzc3rCkBhRenByBh24/dMD12OqS1OBFpy+5M8YsPymUlumbfDoCX4qvrrqvpUVV1fVftm606pqqur6qbZ88lDjgEAgEFdmeQXZ7Phz01y57zffybjZECf2t1f27B8WZJruvuVVXXZbPk3RhgHAMDyWfIMaFW9Jcn5Wf+t6P4kv53k+CTp7tcleU+SZya5OcldSV646JhTlOAvyvqHSJIrkrw/AlAAgKXU3S9YsL2T/PKRHHPoWfCd5M+r6uMbZlTtOZiWnT2fdrgdq+rSg7Ox7sndAw8TAICxDJ0BPa+7b6mq05JcXVWf3eqO3b03yd4keVCdsuTJaQCAI1dZ/j6gQxg0A9rdt8yeb0/yjiRPSnJbVZ2eJLPn24ccAwAAy2WwALSq7l9VDzz4OsnPJvl01mdKXTJ72yVJ3jnUGAAAlt6S3wt+CEOW4PckeUdVHTzPH3f3e6vq2iRvraoXJflykucPOAaAladv5bHlesH2DRaAdvcXkjzuMOvvSHLBUOcFAGC5uRMSAMCEqldvFtLQbZgAAOD7yIACAExld9wL/ojJgAIAMCoBKAAAo1KCB9jllrVtkPZQsM6dkAAAYGACUAAARqUEDwAwJSV4AAAYlgwoAMCETEICAICBCUABABiVEvyE5vXA0/8O2O38OwczSvAAADAsGVAAgKm0SUgAADA4ASgAAKNSggcAmJISPAAADEsGFABgIpXVnIQkAJ2QHngAm5vXK3kR/77CclOCBwBgVDKgAABT6tWrwcuAAgAwKhlQAIAJreIkJBlQAABGJQAFAGBUSvAALKWpWiktav+kxRPHVMedkAAAYGgyoAAAE6q1qUcwPhlQAABGJQAFAGBUSvAAAFMyCQkAAIYlAAUAYFRK8ACwwZB9PvUY5XDcihMAAAYmAwoAMJVO0quXApUBBQBgVAJQAABGpQQPADAhk5AAAGBgMqAAAFNawQyoABSAXWdZ+23q8wnrlOABABiVDCgAwEQqJiEBAMDgZEABAKbS7U5IAAAwNAEoAACjUoIHYNeZst3RohZQ82jTtJpMQgIAgIHJgAIATEkGFAAAhiUABQBgVErwAAATMgkJAAAGJgAFAGBUSvAAcAxtp5fnoh6i+oTuQp1kbfVq8DKgAACMSgYUAGBKq5cAlQEFAGBcAlAAAEalBA8AMCF9QAEAYGAyoAAAU+rVS4EKQAFGoL8jW+F7wKoYtARfVSdV1duq6rNVdWNVPbmqTqmqq6vqptnzyUOOAQCA5TL0b0BfleS93f0TSR6X5MYklyW5prvPTnLNbBkAYCVVT/uYwmABaFU9KMnPJHlDknT397r7G0kuSnLF7G1XJHnuUGMAAGD5DJkBfVSSv03yh1X1V1X1+qq6f5I93X1rksyeTzvczlV1aVXtq6p99+TuAYcJADCRXoLHBIYMQH8oyROTvLa7n5DkOzmCcnt37+3uc7r7nONzv6HGCADAyIYMQPcn2d/dH5stvy3rAeltVXV6ksyebx9wDAAALJnB2jB191er6itV9ePd/bkkFyS5Yfa4JMkrZ8/vHGoMAMtCex2OhXntvHzHdqZKUvqAHnMvSfLmqrpvki8keWHWs65vraoXJflykucPPAYAAJbIoAFod1+f5JzDbLpgyPMCAOwYa1MPYHzuBQ8AwKgEoAAAjMq94AEAJrSKk5BkQAEAGJUMKADAVCa8G9GUBKAAsEMM2etTj1HGpAQPAMCoZEABACbTiUlIAAAwLAEoAACjUoIHAJhQrV4FXgYUAIBxyYACAExpBSchCUABYAXM6/OZ6PXJuJTgAQAYlQwoAMBUOqm1qQcxPhlQAABGJQAFAJhS97SPBarqwqr6XFXdXFWXHWb7+VV1Z1VdP3v81qJjKsEDAHBYVXVcktckeXqS/Umuraoru/uGQ976we5+9laPKwMKAMBmnpTk5u7+Qnd/L8mfJLlouweVAQWAXWDKNktaPG3T9G1AT62qfRuW93b33tnrM5J8ZcO2/Ul+6jDHeHJVfSLJLUl+vbs/M++EAlAAgNX2te4+Z5NtdZh1h4bM1yX50e7+dlU9M8mfJTl73gmV4AEAJlTdkz4W2J/kERuWz8x6lvPvdfc3u/vbs9fvSXJ8VZ0676ACUAAANnNtkrOr6qyqum+Si5NcufENVfWwqqrZ6ydlPb68Y95BleABADis7r63qn4lyVVJjkvyxu7+TFX90mz765I8L8mLq+reJN9NcnH3/NSqABQAYEpb6MU5pVlZ/T2HrHvdhtevTvLqIzmmEjwAAKOSAQUAmEonWcF7wQtAAWAXmLLX5qJzz+sTqkfoalKCBwBgVDKgAAATqWypF+euIwMKAMCoZEABAKYkAwoAAMMSgAIAMColeACAKa1gCV4ACgDMNa+PZ7K4l6denxxKCR4AgFHJgAIATGVFb8UpAwoAwKhkQAEAJuROSAAAMDABKAAAo1KCBwDmGrKN0nZbPO0KSvAAADAsGVAAgMm0DCgAAAxNAAoAwKiU4AEAptJRggcAgKHJgAIATGkF7wUvAAUAJrPdPp/z+oiuRA/RHUoJHgCAUcmAAgBMqExCAgCAYcmAAgBMSQYUAACGJQAFAGBUSvAAAFPpJGurV4IXgAIAS2ten89Er8+dSgAKADCZNgkJAACGJgAFAGBUSvAAAFNSggcAgGEJQAEAGJUSPACwtLbTZmlRC6e8621HfexjSgn+2KmqH6+q6zc8vllVL6+qU6rq6qq6afZ88lBjAABg+QwWgHb357r78d39+CT/U5K7krwjyWVJrunus5NcM1sGAFg9B++ENOVjAmP9BvSCJJ/v7i8luSjJFbP1VyR57khjAABgCYwVgF6c5C2z13u6+9YkmT2fdrgdqurSqtpXVfvuyd0jDRMAgKENPgmpqu6b5DlJfvNI9uvuvUn2JsmD6pTV+3UuALACOum1qQcxujEyoM9Icl133zZbvq2qTk+S2fPtI4wBAIAlMUYA+oL8Q/k9Sa5Mcsns9SVJ3jnCGAAAllP3tI8JVA944qo6MclXkjyqu++crXtIkrcm+ZEkX07y/O7++oLj/G2SL21YdWqSrw0y6N3J9TpyrtmRcb2OnGt2ZFyvI+eaLfaj3f3QKQfw4Pvt6aec/vNTDiHv/dLvf7y7zxnznIP+BrS770rykEPW3ZH1WfFHcpzv+3JU1b6xL9RO5nodOdfsyLheR841OzKu15FzzVhm7oQEADCVg31AV4x7wQMAMKqdmgHdO/UAdhjX68i5ZkfG9TpyrtmRcb2OnGu2U6zgveAHnYQEAMDmHnzfPf2UPRdPOob37r989ElISvAAAIxqp5bgAQB2hxWsRu+oDGhVXVhVn6uqm6vqsqnHs4yq6o1VdXtVfXrDulOq6uqqumn2fPKUY1wmVfWIqnpfVd1YVZ+pqpfN1rtmm6iqE6rqL6vqE7Nr9juz9a7ZHFV1XFX9VVW9a7bses1RVX9dVZ+qquurat9snWu2iao6qareVlWfnf179mTXi2W2YwLQqjouyWuyfmvPxyR5QVU9ZtpRLaU3JbnwkHWXJbmmu89Ocs1smXX3Jvm17v5HSc5N8suz75Vrtrm7kzytux+X5PFJLqyqc+OaLfKyJDduWHa9Fntqdz9+w2/TXLPNvSrJe7v7J5I8LuvfNddrR5j4LkgTZV93TACa5ElJbu7uL3T395L8SZKLJh7T0unuDyQ59M5SFyW5Yvb6iiTPHXNMy6y7b+3u62avv5X1f7TPiGu2qV737dni8bNHxzXbVFWdmeRZSV6/YbXrdeRcs8Ooqgcl+Zkkb0iS7v5ed38jrhdLbCcFoGdk/baeB+2frWOxPd19a7IecCU5beLxLKWqemSSJyT5WFyzuWbl5OuT3J7k6u52zeb7/ST/MsnahnWu13yd5M+r6uNVdelsnWt2eI9K8rdJ/nD2M4/XV9X943qxxHbSJKQ6zLrV+9Uug6iqByT50yQv7+5vVh3u68ZB3X0gyeOr6qQk76iqx048pKVVVc9Ocnt3f7yqzp94ODvJed19S1WdluTqqvrs1ANaYj+U5IlJXtLdH6uqV0W5fefoJGtrC9+22+ykDOj+JI/YsHxmklsmGstOc1tVnZ4ks+fbJx7PUqmq47MefL65u98+W+2abcGszPf+rP/u2DU7vPOSPKeq/jrrPx16WlX9+7hec3X3LbPn25O8I+s/w3LNDm9/kv2zSkSSvC3rAanrxdLaSQHotUnOrqqzquq+SS5OcuXEY9oprkxyyez1JUneOeFYlkqtpzrfkOTG7v69DZtcs01U1UNnmc9U1Q8n+cdJPhvX7LC6+ze7+8zufmTW/936L939T+N6baqq7l9VDzz4OsnPJvl0XLPD6u6vJvlKVf34bNUFSW6I68US2zEl+O6+t6p+JclVSY5L8sbu/szEw1o6VfWWJOcnObWq9if57SSvTPLWqnpRki8nef50I1w65yX5hSSfmv2mMUleEddsntOTXDHrTHGfJG/t7ndV1Ufimh0J37HN7cn6TzuS9f9P/XF3v7eqro1rtpmXJHnzLEHzhSQvzOy/T9drB1jBPqBuxQkAMJEHH39aP+Uhz5t0DO+97bWj34pzx2RAAQB2pRVMBu6k34ACALALCEABABiVEjwAwGQ6WVOCBwCAQQlAgaVSVQeq6vqq+nRV/ceqOnEbx3pTVT1v9vr1VfWYOe89v6qecrTnAjgqnXSvTfqYggAUWDbf7e7Hd/djk3wvyS9t3DjrP3rEuvv/6u4b5rzl/CQCUIARCECBZfbBJP/dLDv5vqr646zfNOC4qvpXVXVtVX2yqv55sn5nq6p6dVXdUFXvTnLawQNV1fur6pzZ6wur6rqq+kRVXVNVj8x6oPurs+zrT4//UQFWh0lIwFKqqh9K8owk752telKSx3b3F6vq0iR3dvdPVtX9knyoqv48yROS/HiS/yHrd9O5IckbDznuQ5P8uyQ/MzvWKd399ap6XZJvd/e/HuUDAhy0gpOQBKDAsvnhDbdF/WCSN2S9NP6X3f3F2fqfTfI/Hvx9Z5IHJzk7yc8keUt3H0hyS1X9l8Mc/9wkHzh4rO7++jAfA4DNCECBZfPd7n78xhWze4J/Z+OqJC/p7qsOed8zkyxKJdQW3gMwHndCAtgRrkry4qo6Pkmq6seq6v5JPpDk4tlvRE9P8tTD7PuRJP9zVZ012/eU2fpvJXng8EMHQAAK7ESvz/rvO6+rqk8n+bdZr+i8I8lNST6V5LVJ/uuhO3b33ya5NMnbq+oTSf7DbNN/SvJPTEICGF71CqZ9AQCWwYOPO7Wf/IDnTDqGq775hx/v7nPGPKcMKAAAozIJCQBgSitYjZYBBQBgVAJQAABGpQQPADChXlubegijkwEFAGBUMqAAAJNpk5AAAGBoAlAAAEalBA8AMJVOsqYEDwAAgxKAAgAwKiV4AIAptT6gAAAwKBlQAICJdJI2CQkAAIYlAAUAYFRK8AAAU+k2CQkAAIYmAwoAMCGTkAAAYGACUAAANlVVF1bV56rq5qq67DDbq6oun23/ZFU9cdExleABAKa0xJOQquq4JK9J8vQk+5NcW1VXdvcNG972jCRnzx4/leS1s+dNyYACALCZJyW5ubu/0N3fS/InSS465D0XJfmjXvfRJCdV1enzDioDCgAwkW/l7676i37bqRMP44Sq2rdheW937529PiPJVzZs258fzG4e7j1nJLl1sxMKQAEAJtLdF049hgXqMOsOnba/lfd8HyV4AAA2sz/JIzYsn5nklqN4z/cRgAIAsJlrk5xdVWdV1X2TXJzkykPec2WSX5zNhj83yZ3dvWn5PVGCBwBgE919b1X9SpKrkhyX5I3d/Zmq+qXZ9tcleU+SZya5OcldSV646LjVvXrd9wEAmI4SPAAAoxKAAgAwKgEoAACjEoACADAqASgAAKMSgAIAMCoBKAAAo/r/AWZ9kqoJy1V+AAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" }, { "data": { "text/plain": [ "''" ] }, "execution_count": 79, "metadata": {}, "output_type": "execute_result" } ], "source": [ "plot_confusion_matrix(df_hat[df_hat.nbid==17])\n", ";\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "id": "ac433967", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "b580ca44", "metadata": {}, "source": [ "former value: 0.19214895122203912" ] }, { "cell_type": "code", "execution_count": null, "id": "5449b33c", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "id": "95aa2d46", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "id": "15bbfe6c", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "conda_python3", "language": "python", "name": "conda_python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.13" }, "toc-autonumbering": true }, "nbformat": 4, "nbformat_minor": 5 }