{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## Implementing a Review classification model with Keras, MXNet and SageMaker\n", " The solution demotrates a Distributed Learning system implementing custom generator to train on sprase libsvm data." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Background " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Data set: Amazon Customer Reviews Dataset\n", " \n", " Amazon Customer Reviews (a.k.a. Product Reviews) is one of Amazon’s iconic products. In a period of over two decades since the first review in 1995, millions of Amazon customers have contributed over a hundred million reviews to express opinions and describe their experiences regarding products on the Amazon.com website. Over 130+ million customer reviews are available to researchers as part of this dataset.\n", " \n", "#### Approach\n", " 1. The Review classficiation is NLP machine learning model to predict whether a review posted by the customer is positive or negative. For the sake of simplification, we have converted the ratings provided by the customer into a binary target variable with value equals to 1 when ratings were either 4 or 5. \n", "\n", " 2. In present work, after spilitting the data into train and test we created a feature engineerng pipeline on training data using natural language processing techiniques. The dataset was futher split into 10 chunks to demonstrate distributed learning, where during training not more than 1204 [batch_size] records are loaded onto memory at a time.\n", " \n", " \n", "#### Setup\n", "This notebook was created and tested on an ml.p2.xlarge notebook instance.\n", "\n", "Let's start by specifying:\n", "\n", "1. The S3 bucket and prefix that you want to use for training and model data. This should be within the same region as the Notebook Instance, training, and hosting.\n", "2. The IAM role arn used to give training and hosting access to your data. See the documentation for how to create these. Note, if more than one role is required for notebook instances, training, and/or hosting, please replace the get_execution_role() call with the appropriate full IAM role arn string(s)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sagemaker import get_execution_role\n", "from sagemaker.session import Session\n", "import boto3\n", "\n", "bucket = ''\n", "custom_code_upload_location = 's3://{}/sagemaker/DEMO-Distributed-learning-Keras/customcode'.format(bucket)\n", "model_artifacts_location = 's3://{}/sagemaker/DEMO-Distributed-learning-Keras/artifacts'.format(bucket)\n", "\n", "role = get_execution_role()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Data\n", "##### Explore\n", "Let's start by bringing in our dataset from an S3 public bucket. As mentioned above, this contains 1 to 5 star ratings from over 2M Amazon customers on over 160K digital videos. More details on this dataset can be found at its AWS Public Datasets page.\n", "\n", "Note, because this dataset is over a half gigabyte, the load from S3 may take ~10 minutes. Also, since Amazon SageMaker Notebooks start with a 5GB persistent volume by default, and we don't need to keep this data on our instance for long, we'll bring it to the temporary volume (which has up to 20GB of storage).\n", "\n", "\n", "Let's read the data into a Pandas DataFrame so that we can begin to understand it.\n", "\n", "Note, we'll set error_bad_lines=False when reading the file in as there appear to be a very small number of records which would create a problem otherwise." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "b'Skipping line 92523: expected 15 fields, saw 22\\n'\n", "b'Skipping line 343254: expected 15 fields, saw 22\\n'\n", "b'Skipping line 524626: expected 15 fields, saw 22\\n'\n", "b'Skipping line 623024: expected 15 fields, saw 22\\n'\n", "b'Skipping line 977412: expected 15 fields, saw 22\\n'\n", "b'Skipping line 1496867: expected 15 fields, saw 22\\n'\n", "b'Skipping line 1711638: expected 15 fields, saw 22\\n'\n", "b'Skipping line 1787213: expected 15 fields, saw 22\\n'\n", "b'Skipping line 2395306: expected 15 fields, saw 22\\n'\n", "b'Skipping line 2527690: expected 15 fields, saw 22\\n'\n" ] } ], "source": [ "import pandas as pd\n", "import csv\n", "fname = 's3://amazon-reviews-pds/tsv/amazon_reviews_us_Digital_Video_Download_v1_00.tsv.gz'\n", "df = pd.read_csv(fname,sep='\\t',delimiter='\\t',error_bad_lines=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "We can see this dataset includes information like:\n", "\n", " 1. marketplace: 2-letter country code (in this case all \"US\").\n", " 2. customer_id: Random identifier that can be used to aggregate reviews written by a single author.\n", " 3. review_id: A unique ID for the review.\n", " 4. product_id: The Amazon Standard Identification Number (ASIN). http://www.amazon.com/dp/ links to the product's detail page.\n", " 5. product_parent: The parent of that ASIN. Multiple ASINs (color or format variations of the same product) can roll up into a single parent parent.\n", " 6. product_title: Title description of the product.\n", " 7. product_category: Broad product category that can be used to group reviews (in this case digital videos).\n", " 8. star_rating: The review's rating (1 to 5 stars).\n", " 9. helpful_votes: Number of helpful votes for the review.\n", " 10. total_votes: Number of total votes the review received.\n", " 11. vine: Was the review written as part of the Vine program?\n", " 12. verified_purchase: Was the review from a verified purchase?\n", " 13. review_headline: The title of the review itself.\n", " 14. review_body: The text of the review.\n", " 15. review_date: The date the review was written.\n", " \n", " \n", "For this example, let's limit ourselves 'verified_purchase','review_headline','review_body','product_title','helpful_votes and star_rating. \n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Pre-processing\n", "1. Replace any NaN values with empty string\n", "2. Create Target column based the star rating" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "def pre_process(df):\n", " df.fillna(\n", " value={'review_body': '', 'review_headline': ''}, inplace=True)\n", " df.fillna(\n", " value={'verified_purchase': 'Unk'}, inplace=True)\n", "\n", " df.fillna(0, inplace=True)\n", " return df\n", "\n", "df = pre_process(df)\n", "\n", "df.review_date = pd.to_datetime(df.review_date)\n", "df['target'] = np.where(df['star_rating']>=4,1,0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Split the data into Train, validation and Test set\n", "Note: We have used stratefied sampling as the criteria for splittng the data; stratefied sampling on the target variable insures same distribution of the varible across all the splits." ] }, { "cell_type": "code", "execution_count": 75, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.7920667165021078\n", "0.7920667275251041\n", "0.7920667165021078\n" ] } ], "source": [ "from sklearn.model_selection import StratifiedShuffleSplit\n", "sss = StratifiedShuffleSplit(n_splits=2, test_size=0.10, random_state=0)\n", "\n", "sss.get_n_splits(df, df['target'])\n", "for train_index, test_index in sss.split(df, df['target']):\n", " X_train_vallid , X_test = df.iloc[train_index], df.iloc[test_index]\n", " \n", " \n", "sss.get_n_splits(X_train_vallid, X_train_vallid['target']) \n", "for train_index, test_index in sss.split(X_train_vallid, X_train_vallid['target']):\n", " X_train , X_valid = X_train_vallid.iloc[train_index], X_train_vallid.iloc[test_index]\n", " \n", " \n", "print(X_train.target.value_counts()[1]/(X_train.target.value_counts().sum()))\n", "print(X_test.target.value_counts()[1]/(X_test.target.value_counts().sum()))\n", "print(X_valid.target.value_counts()[1]/(X_valid.target.value_counts().sum()))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Feature Engineering\n", " 1. One-hot encoder for 'verified_purchase'\n", " 2. CountVectorizer followed by absolute scaling for 'review_headline'\n", " 3. CountVectorizer followed by absolute scaling for 'review_body'\n", " 4. CountVectorizer followed by absolute scaling for 'product_title'\n", " 5. SimpleImputer followed by MinMaxScaler for 'helpful_votes'\n", " 6. Concatenate all the previous vector in one Pipeline" ] }, { "cell_type": "code", "execution_count": 68, "metadata": {}, "outputs": [], "source": [ "from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer\n", "import sklearn.pipeline\n", "from sklearn.preprocessing import OneHotEncoder,MaxAbsScaler,MinMaxScaler\n", "from sklearn.impute import SimpleImputer\n", "from sklearn.compose import ColumnTransformer\n", "\n", "cat_transformer = sklearn.pipeline.Pipeline(steps=[('cat_Vectorizer', \n", " OneHotEncoder(handle_unknown='ignore'))])\n", "\n", "headline_vectorizer = CountVectorizer(stop_words='english',\n", " binary=False,\n", " max_df=0.7,\n", " min_df=100,\n", " ngram_range=(3, 3),\n", " max_features=500,\n", " analyzer='char_wb')\n", "\n", "headline_transformer = sklearn.pipeline.Pipeline(steps=[\n", " ('Vectorizer', headline_vectorizer),\n", " ('feature_vec_sclr', MaxAbsScaler())])\n", "\n", "\n", "body_vectorizer = CountVectorizer(stop_words='english',\n", " binary=False,\n", " max_df=0.7,\n", " min_df=100,\n", " ngram_range=(3, 3),\n", " max_features=5000,\n", " analyzer='char_wb')\n", "\n", "body_transformer = sklearn.pipeline.Pipeline(steps=[\n", " ('Vectorizer', body_vectorizer),\n", " ('feature_vec_sclr', MaxAbsScaler())])\n", "\n", "\n", "product_vectorizer = CountVectorizer(stop_words='english',\n", " binary=False,\n", " max_df=0.7,\n", " min_df=100,\n", " ngram_range=(3, 3),\n", " max_features=500,\n", " analyzer='char_wb')\n", "\n", "product_transformer = sklearn.pipeline.Pipeline(steps=[\n", " ('Vectorizer', product_vectorizer),\n", " ('feature_vec_sclr', MaxAbsScaler())])\n", "\n", "steps = [('feature_Imputer', SimpleImputer(strategy='most_frequent'))\n", " , ('feature_scaler', MinMaxScaler())]\n", "numeric_transformer = sklearn.pipeline.Pipeline(steps)\n", " \n", "\n", "union = ColumnTransformer(\n", " transformers=[ ('cat', cat_transformer, [0]),\n", " ('headline', headline_transformer, 1),\n", " ('body', body_transformer, 2),\n", " ('product', product_transformer, 3),\n", " ('num', numeric_transformer, [4])\n", " ],\n", " remainder=\"drop\")\n", "\n", "steps = [('features', union)]\n", "pipeline = sklearn.pipeline.Pipeline(steps)" ] }, { "cell_type": "code", "execution_count": 76, "metadata": {}, "outputs": [], "source": [ "cols = ['verified_purchase','review_headline','review_body','product_title','helpful_votes']\n", "import time\n", "start = time.time()\n", "train_vec = pipeline.fit_transform(X_train[cols])\n", "end = time.time()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It took approximatly 10 mins to fit the pipeline on 3M records" ] }, { "cell_type": "code", "execution_count": 112, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "mkdir: cannot create directory ‘./data’: File exists\r\n" ] } ], "source": [ "! mkdir ./data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Splitting the data into 10 different chunks to demonstrate distributed learning" ] }, { "cell_type": "code", "execution_count": 200, "metadata": {}, "outputs": [], "source": [ "from sklearn.datasets import dump_svmlight_file\n", "train_df = np.array_split(X_train, 10)\n", "i=0\n", "for df_split in train_df: \n", " trans = pipeline.transform(df_split[cols])\n", " dump_svmlight_file(trans,df_split['target'].values,'data/train/file_'+str(i))\n", " i=i+1\n", " \n", "train_df = np.array_split(X_valid, 10)\n", "i=0\n", "for df_split in train_df: \n", " trans = pipeline.transform(df_split[cols]) \n", " dump_svmlight_file(trans,df_split['target'].values,'data/valid/file_'+str(i))\n", " i=i+1 \n", " \n", "train_df = np.array_split(X_test, 10)\n", "i=0\n", "for df_split in train_df: \n", " trans = pipeline.transform(df_split[cols]) \n", " dump_svmlight_file(trans,df_split['target'].values,'data/holdout/file_'+str(i))\n", " i=i+1 " ] }, { "cell_type": "code", "execution_count": 297, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/bin/sh: your: No such file or directory\r\n" ] } ], "source": [ "### repalce with your bucket name\n", "! aws s3 cp ./data/ s3:///sagemaker/DEMO-Distributed-learning-Keras/data/ --recursive --acl bucket-owner-full-control" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Train with SageMaker\n", "\n", "\n", "Wrap Code\n", "To use SageMaker's pre-built MXNet container, we'll need to wrap our code from above into a Python script. There's a great deal of flexibility in using SageMaker's pre-built containers, and detailed documentation can be found here, but for our example, it consisted of:\n", "\n", " 1.f1 custom function to optmize around\n", " 2.readfile function to read individual files\n", " 3.Custom generator to loop through files and records to return batchs in sequence\n", " 4.parse_args to parser to read all arguements\n", " 5.train function to train the Neural network model\n", " 6.threadsafe_iter class and threadsafe_generator to make the custom generator threadsafe\n", " 7.input_fn function to convert inference payload to numpy vector\n" ] }, { "cell_type": "code", "execution_count": 299, "metadata": { "collapsed": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "# #Standard Libraries\r\n", "import sys\r\n", "import os\r\n", "from os import walk\r\n", "\r\n", "import json\r\n", "import threading\r\n", "\r\n", "import argparse\r\n", "\r\n", "\r\n", "import numpy as np\r\n", "\r\n", "\r\n", "import mxnet as mx\r\n", "\r\n", "##Keras Imports\r\n", "import keras\r\n", "from keras import backend as K\r\n", "from keras.layers import Dense, Dropout, Input, BatchNormalization, Activation\r\n", "\r\n", "from keras.models import Sequential\r\n", "from keras.models import save_mxnet_model\r\n", "from keras.models import load_model\r\n", "from keras.optimizers import RMSprop, Adam, SGD\r\n", "\r\n", "from keras.utils import multi_gpu_model\r\n", "\r\n", "\r\n", "input_shape = (6003,)\r\n", "\r\n", "\r\n", "def f1(y_true, y_pred):\r\n", " def recall(y_true, y_pred):\r\n", " \"\"\"Recall metric.\r\n", "\r\n", " Only computes a batch-wise average of recall.\r\n", "\r\n", " Computes the recall, a metric for multi-label classification of\r\n", " how many relevant items are selected.\r\n", " \"\"\"\r\n", " true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\r\n", " possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))\r\n", " recall = true_positives / (possible_positives + K.epsilon())\r\n", " return recall\r\n", "\r\n", " def precision(y_true, y_pred):\r\n", " \"\"\"Precision metric.\r\n", "\r\n", " Only computes a batch-wise average of precision.\r\n", "\r\n", " Computes the precision, a metric for multi-label classification of\r\n", " how many selected items are relevant.\r\n", " \"\"\"\r\n", " true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\r\n", " predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))\r\n", " precision = true_positives / (predicted_positives + K.epsilon())\r\n", " return precision\r\n", "\r\n", " precision = precision(y_true, y_pred)\r\n", " recall = recall(y_true, y_pred)\r\n", " return 2 * ((precision * recall) / (precision + recall + K.epsilon()))\r\n", "\r\n", "\r\n", "def input_fn(request_body, request_content_type):\r\n", " if request_content_type == 'text/csv':\r\n", " print('I am here:')\r\n", " Matrix = []\r\n", " print(type(request_body))\r\n", " for line in request_body.splitlines():\r\n", " data = line.split()\r\n", " target = float(data[0])\r\n", " row = np.zeros(input_shape[0], float)\r\n", " for i, (idx, value) in enumerate([item.split(':') for item in data[1:]]):\r\n", " row[int(idx)] = value\r\n", " Matrix.append(np.array(row))\r\n", " Matrix = np.array(Matrix)\r\n", " print('Matrix Shape', Matrix.shape)\r\n", " print('Matrix Type', type(Matrix))\r\n", " return mx.io.NDArrayIter(Matrix)\r\n", " else:\r\n", " # Handle other content-types here or raise an Exception\r\n", " # if the content type is not supported.\r\n", " pass\r\n", "\r\n", "\r\n", "\r\n", "class threadsafe_iter:\r\n", " \"\"\"Takes an iterator/generator and makes it thread-safe by\r\n", " serializing call to the `next` method of given iterator/generator.\r\n", " \"\"\"\r\n", "\r\n", " def __init__(self, it):\r\n", " self.it = it\r\n", " self.lock = threading.Lock()\r\n", "\r\n", " def __iter__(self):\r\n", " return self\r\n", "\r\n", " def next(self):\r\n", " with self.lock:\r\n", " return self.it.next()\r\n", "\r\n", "\r\n", "def threadsafe_generator(f):\r\n", " \"\"\"A decorator that takes a generator function and makes it thread-safe.\r\n", " \"\"\"\r\n", "\r\n", " def g(*a, **kw):\r\n", " return threadsafe_iter(f(*a, **kw))\r\n", "\r\n", " return g\r\n", "\r\n", "\r\n", "def readfile(filepath):\r\n", " with open(filepath) as fp:\r\n", " lines = fp.readlines()\r\n", " Matrix = []\r\n", " labels = []\r\n", " for line in lines:\r\n", " data = line.split()\r\n", " target = labels.append(float(data[0]))\r\n", " row = np.zeros(input_shape[0], float)\r\n", " for i, (idx, value) in enumerate([item.split(':') for item in data[1:]]):\r\n", " row[int(idx)] = value\r\n", " Matrix.append(np.array(row))\r\n", " Matrix = np.array(Matrix)\r\n", " return labels, Matrix\r\n", "\r\n", "\r\n", "def generator(files, batch_size):\r\n", " # print('start generator')\r\n", " while 1:\r\n", " # print('loop generator')\r\n", " for file in files:\r\n", " try:\r\n", " # data = load_svmlight_file(file)\r\n", " Y, X = readfile(file)\r\n", " recs = X.shape[0]\r\n", " batches = int(np.ceil(recs / batch_size))\r\n", " for i in range(0, batches):\r\n", " x_batch = X[i * batch_size:min(len(X), i * batch_size + batch_size), ]\r\n", " y_batch = Y[i * batch_size:min(len(X), i * batch_size + batch_size)]\r\n", " yield x_batch, y_batch\r\n", " except EOFError:\r\n", " print(\"error\" + file)\r\n", "\r\n", "\r\n", "def parse_args():\r\n", " parser = argparse.ArgumentParser()\r\n", "\r\n", " # retrieve the hyperparameters we set in notebook (with some defaults)\r\n", " parser.add_argument('--epochs', type=int, default=5)\r\n", " parser.add_argument('--learning-rate', type=float, default=0.01)\r\n", "\r\n", " parser.add_argument('--model-dir', type=str, default=os.environ['SM_MODEL_DIR'])\r\n", " parser.add_argument('--training_channel', type=str, default=os.environ['SM_CHANNEL_TRAIN'])\r\n", " parser.add_argument('--test_channel', type=str, default=os.environ['SM_CHANNEL_TEST'])\r\n", "\r\n", " parser.add_argument('--current-host', type=str, default=os.environ['SM_CURRENT_HOST'])\r\n", " parser.add_argument('--hosts', type=list, default=json.loads(os.environ['SM_HOSTS']))\r\n", "\r\n", " parser.add_argument('--batch_size', type=int, default=1024)\r\n", "\r\n", " return parser.parse_args()\r\n", "\r\n", "\r\n", "def train(current_host, hosts, num_cpus, num_gpus, training_dir, val_dir, model_dir, batch_size, epochs, learning_rate):\r\n", " print('Parameters: ', num_cpus, num_gpus, training_dir, val_dir, model_dir, batch_size, epochs, learning_rate)\r\n", "\r\n", " train_files = [os.path.join(training_dir, file) for file in os.listdir(training_dir)]\r\n", " train_files = [x for x in train_files if x.split('/')[-1] != '.ipynb_checkpoints']\r\n", " train_gen = generator(train_files, batch_size)\r\n", " print('Number of training files: ', len(train_files))\r\n", "\r\n", " test_files = [os.path.join(val_dir, file) for file in os.listdir(val_dir)]\r\n", " test_files = [x for x in test_files if x.split('/')[-1] != '.ipynb_checkpoints']\r\n", " test_gen = generator(test_files, batch_size)\r\n", " print('Number of test files: ', len(test_files))\r\n", "\r\n", " # shape = load_svmlight_file(train_files[0])[0].shape[1:]\r\n", " # shape = (scipy.sparse.load_npz(train_files[0]).shape[1] -1 ,)## Substracting 1 for label columns\r\n", " print('shape:', input_shape)\r\n", "\r\n", " # steps = np.array([int(np.ceil(scipy.sparse.load_npz(x).shape[0]/batch_size)) for x in train_files]).sum()\r\n", " # steps_val = np.array([int(np.ceil(scipy.sparse.load_npz(x).shape[0]/batch_size)) for x in test_files]).sum()\r\n", "\r\n", " steps = np.array([int(np.ceil(readfile(x)[1].shape[0] / batch_size)) for x in train_files]).sum()\r\n", " steps_val = np.array([int(np.ceil(readfile(x)[1].shape[0] / batch_size)) for x in test_files]).sum()\r\n", " print('Steps', steps, steps_val)\r\n", " print('Number of gpus', num_gpus)\r\n", "\r\n", " model = Sequential()\r\n", " model.add(Dense(1024, input_shape=input_shape))\r\n", " model.add(Activation('relu'))\r\n", " model.add(Dropout(0.25))\r\n", "\r\n", " model.add(Dense(128))\r\n", " model.add(Activation('relu'))\r\n", " model.add(Dropout(0.25))\r\n", "\r\n", " model.add(Dense(32))\r\n", " model.add(Activation('relu'))\r\n", " model.add(Dropout(0.25))\r\n", "\r\n", " model.add(Dense(1))\r\n", " model.add(Activation('sigmoid'))\r\n", "\r\n", " print(model.summary())\r\n", "\r\n", " if num_gpus > 1:\r\n", " model = multi_gpu_model(model, gpus=num_gpus)\r\n", "\r\n", " filepath = 'best_model.h5'\r\n", " checkpoint = keras.callbacks.ModelCheckpoint(filepath, monitor='val_loss',\r\n", " verbose=1, save_best_only=True, mode='min')\r\n", " callbacks_list = [checkpoint]\r\n", " opt = Adam(lr=learning_rate, rescale_grad=1. / batch_size)\r\n", "\r\n", " model.compile(optimizer=opt, loss='binary_crossentropy', metrics=[f1], callbacks=callbacks_list)\r\n", " # model.fit_generator(train, epochs=epochs, shuffle=True, class_weight=[0.6,0.4],\r\n", " # validation_data=test, callbacks=[es, mc, lrp], max_queue_size=30, verbose=verbose)\r\n", "\r\n", " history = model.fit_generator(train_gen, steps_per_epoch=steps, epochs=epochs,\r\n", " shuffle=True,\r\n", " validation_data=test_gen,\r\n", " validation_steps=steps_val,\r\n", " callbacks=callbacks_list,\r\n", " verbose=2)\r\n", "\r\n", " print(np.sqrt(history.history['loss']))\r\n", " print(np.sqrt(history.history['val_loss']))\r\n", " print('model trained: ')\r\n", " return model\r\n", "\r\n", "\r\n", "if __name__ == '__main__':\r\n", " args = parse_args()\r\n", " num_cpus = int(os.environ['SM_NUM_CPUS'])\r\n", " num_gpus = int(os.environ['SM_NUM_GPUS'])\r\n", " print('number of cpus', num_cpus)\r\n", " print('number of gpus', num_gpus)\r\n", "\r\n", " os.environ['MXNET_ENABLE_GPU_P2P'] = '0'\r\n", " os.environ['MXNET_CUDNN_AUTOTUNE_DEFAULT'] = '0'\r\n", "\r\n", " model = train(args.current_host, args.hosts, num_cpus, num_gpus, args.training_channel, args.test_channel,\r\n", " args.model_dir,\r\n", " (args.batch_size), args.epochs, args.learning_rate)\r\n", "\r\n", " if args.current_host == args.hosts[0]:\r\n", " model.save(os.path.join(args.model_dir, 'model'))\r\n", " print('model saved')\r\n", " # Get our best model\r\n", " # best_model = load_model(os.path.join(args.model_dir, 'model'))\r\n", " dependencies = {'f1': f1}\r\n", " best_model = load_model('best_model.h5', custom_objects=dependencies)\r\n", " print('model loaded:-')\r\n", " print( os.path.join(args.model_dir, 'model'))\r\n", " #\r\n", " # Make one random prediction to initialize for the save\r\n", " # sample = np.random.randint(args.vocab_size, size=(args.maxlen,))\r\n", " # best_model.predict(np.array([sample]))\r\n", "\r\n", " test_files = [os.path.join(args.test_channel, file) for file in os.listdir(args.test_channel)]\r\n", " # sample = load_svmlight_file(test_files[0])[0].todense()\r\n", " Y, X = readfile(test_files[0])\r\n", " print(test_files[0])\r\n", " print(X.shape)\r\n", " best_model.predict(X)\r\n", "\r\n", " print( os.path.join(args.model_dir, 'model'))\r\n", " data_names, data_shapes = save_mxnet_model(best_model, os.path.join(args.model_dir, 'model'))\r\n", "\r\n", " signature = [{'name': data_names[0], 'shape': [dim for dim in data_desc.shape]} for data_desc in data_shapes]\r\n", " with open(os.path.join(args.model_dir, 'model-shapes.json'), 'w') as f:\r\n", " json.dump(signature, f)\r\n" ] } ], "source": [ "!cat Mailerclass_mxnet_dnn_II.py" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "### Submitting a sagemaker training job\n", "\n", "from sagemaker.mxnet import MXNet\n", "mnist_dnn_estimator = MXNet(entry_point='Mailerclass_mxnet_dnn_II.py',\n", " role=role,\n", " output_path=model_artifacts_location,\n", " code_location=custom_code_upload_location,\n", " train_instance_count=1,\n", " train_instance_type='ml.p3.16xlarge',\n", " framework_version='1.4.1',\n", " py_version='py3',\n", " train_volume_size = 100,\n", " distributions={'parameter_server': {'enabled': True}},\n", " hyperparameters={'learning-rate': 0.01})\n", "\n", "region = boto3.Session().region_name\n", "train_data_location = 's3://'+bucket+'/sagemaker/DEMO-Distributed-learning-Keras/data/train/'\n", "test_data_location = 's3://'+bucket+'/sagemaker/DEMO-Distributed-learning-Keras/data/valid/'\n", "\n", "mnist_dnn_estimator.fit({'train': train_data_location, 'test': test_data_location})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Batch inference\n", "Run the batch inference on the test set" ] }, { "cell_type": "code", "execution_count": 272, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Parameter image will be renamed to image_uri in SageMaker Python SDK v2.\n", "'create_image_uri' will be deprecated in favor of 'ImageURIProvider' class in SageMaker Python SDK v2.\n" ] } ], "source": [ "output_path = 's3://'+bucket+'/sagemaker/DEMO-Distributed-learning-Keras/data/holdout-predictions/'\n", "input_path = 's3://'+bucket+'/sagemaker/DEMO-Distributed-learning-Keras/data/holdout/'\n", "\n", "transformer = mnist_dnn_estimator.transformer(instance_count=2, \n", " instance_type='ml.p3.16xlarge', \n", " assemble_with='Line', \n", " accept='text/csv',\n", " max_payload=3,\n", " output_path=output_path,\n", " env = {'SAGEMAKER_MODEL_SERVER_TIMEOUT' : '3600' })\n", "transformer.transform(input_path, \n", " content_type='text/csv',\n", " #input_filter=\"$[1:]\" ,\n", " #join_source='Input',\n", " split_type='Line')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Evaluation\n", "1. read_predictions: Read results from the batch transform\n", "2. Analyze and plot the PR-AUC and ROC-AUC curve " ] }, { "cell_type": "code", "execution_count": 284, "metadata": {}, "outputs": [], "source": [ "### Reading results and Analyzing\n", "from sklearn import metrics\n", "import matplotlib\n", "import pandas as pd\n", "matplotlib.use('agg', warn=False, force=True)\n", "from matplotlib import pyplot as plt\n", "\n", "%matplotlib inline \n", "\n", "def analyze_results(labels, predictions):\n", " precision, recall, thresholds = metrics.precision_recall_curve(labels, predictions)\n", " auc = metrics.auc(recall, precision)\n", " \n", " fpr, tpr, _ = metrics.roc_curve(labels, predictions)\n", " roc_auc_score = metrics.roc_auc_score(labels, predictions)\n", " \n", " print('Neural-Nets: ROC auc=%.3f' % ( roc_auc_score))\n", " \n", " plt.plot(fpr, tpr, label=\"data 1, auc=\" + str(roc_auc_score))\n", " plt.xlabel('1-Specificity')\n", " plt.ylabel('Sensitivity')\n", " plt.legend(loc=4)\n", " plt.show()\n", " \n", " \n", " lr_precision, lr_recall, _ = metrics.precision_recall_curve(labels, predictions)\n", " lr_auc = metrics.auc(lr_recall, lr_precision)\n", " # summarize scores\n", " print('Neural-Nets: PR auc=%.3f' % ( lr_auc))\n", " # plot the precision-recall curves\n", " no_skill = len(labels[labels==1.0]) / len(labels)\n", " plt.plot([0, 1], [no_skill, no_skill], linestyle='--', label='No Skill')\n", " \n", " plt.plot(lr_recall, lr_precision, marker='.', label='Neural-Nets')\n", " # axis labels\n", " plt.xlabel('Recall')\n", " plt.ylabel('Precision')\n", " # show the legend\n", " plt.legend()\n", " # show the plot\n", " plt.show()\n", " \n", " \n", " return auc\n", "\n", "def read_predictions(prefix):\n", " s3 = boto3.resource('s3')\n", " my_bucket = s3.Bucket(bucket)\n", " files = 0 \n", " preds=[]\n", " for object_summary in my_bucket.objects.filter(Prefix=prefix):\n", " print(object_summary.key)\n", " obj = s3.Object(bucket, object_summary.key)\n", " filename = 's3://'+ bucket+ '/' + object_summary.key\n", " res = pd.read_csv(filename, header=None , sep='\\t')\n", " print(\"length:\",len(res.iloc[:,0].values.tolist()))\n", " preds = preds + res.iloc[:,0].values.tolist()\n", " files = files +1\n", " return preds\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "prefix ='sagemaker/DEMO-Distributed-learning-Keras/data/holdout-predictions/'\n", "y_dnn_pred = read_predictions(prefix)" ] }, { "cell_type": "code", "execution_count": 286, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Neural-Nets: ROC auc=0.975\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deZQU1fn/8fczCwz7jkG2QdlBHGXcvypRo2gixiUR0LhEMX4TYzTxm/iLiVviiVGjiceVqDESAyomBpVEo2JiDCgQFgERCYIMIPs6CzM9/fz+6J62Z+iZ6YGp6Rnq8zqnT1ffulX9VMHU0/dW1S1zd0REJLyyMh2AiIhklhKBiEjIKRGIiIScEoGISMgpEYiIhFxOpgNoqO7du3t+fn6mwxARaVHmz5+/xd17pJrX4hJBfn4+8+bNy3QYIiItipmtqW2euoZEREJOiUBEJOSUCEREQk6JQEQk5JQIRERCLrBEYGZPmdkmM1tSy3wzswfNbKWZLTazo4OKRUREahdki+BpYGwd888GBsVf1wCPBhiLiIjUIrD7CNz9n2aWX0eV84BnPDYO9hwz62xmvdx9Q1AxiTSUuxN1qIw6lVEnEo0mTXtiOurJ7+xTFnVi79Gk6arypDqVUQdi0+7gienYcPFR91h5fB0OkDRdvTy27NY9e+nSrlWKbUuxvXXsh/T3War1pl6+YTGkt96GjKxf23YFEVeD1ltLXKcPO4Qj+3auJZL9l8kbynoDa5M+F8XL9kkEZnYNsVYD/fr1a5LgJPMilVFKKyopraikrPzz6dLySsoq4q9IJWUVUcoqKtkb+fx9b0WUvZFKyiNR9kaiVFTGXp9PO+Xx6fJIlPLKKJFKj31Omo5E9bwOyQyzfct6dsw76BJBis2sLTn6ZGAyQGFhof4ym7nKqLOjpJxtxbHXztIKdpRWsKu0gp2lFewui7C7LEJpRYQ9eyspK48d4EvKIxTvraS4PEJZRSUVlfv3T52bbeTlZNMqJ4vWOVm0ir9ys+PvWVm0yc2mU5tccrMtVp4dm59T9Tkni5wsIyfLyM6KlWdnGdkWf8+yWJkZWUnln09DVo2yLDOyDKyqvMZ0VvwvP8uMrCwwquoAfF7f4nWqDhRZWbEyi68jXj0xbYn17ruvLNWfYaq/TFIfmGqpmvjO9OqmGVctdRtSL9V6a6+bar21xNWQGNLdiCaUyURQBPRN+twHWJ+hWKQOkcoo26oO7HvK2Ro/wMfe98amk8p3lJRT2w9pM2jfOoeOebm0aZVNu1bZtGmVTY8OrWmT25Z2rbNp2yqHNq2yaZMbf7XKJi83m7zcLNompmPzqsrzcmLTrXKyyE51xBORWmUyEcwArjOzacBxwE6dH2h60aizYVcZa7eVsGFnKRt2lrFhRxkbdpaxaXcZRdtL2VZcnnJZM+jcJpeu7VrRrV1rBvZoT9cBrejWrhVd469u7VrTuW0undrk0rFNLh1a55ClA7VIsxJYIjCzqcAYoLuZFQG3AbkA7v4YMBM4B1gJlABXBhVL2JVHoqzeWsyqzXtYvbWET7eVsGZrMWu3lfLZzjLKK6PV6ndqk8sXOuZxSKc8RhzaiUM6to4f3FvHDu7tYwf5zm1yycnWrSgiLV2QVw1NqGe+A98J6vvDaG+kkrXbSvh44x4+3rSH5Z/tYvmG3azeWlytq6Zz21zyu7XjyL6dOfuIPPp1bUu/rm3p1akNh3bOo22rFjcorYgcAP3Ft1DlkShL1+9k0dodLF63k2Xrd/Hxpj3xyw9j+nVty7BeHTjniF4c3rMdA3t0oH/3tnTMy81g5CLS3CgRtBBlFZUsLtrJ+59sZfaqrcxfs52yiliXTo8OrRlxaEdOG9qTQYe05/Ae7RnYs71+2YtIWnSkaMY27irj78s2Mmv5Jt5ZuYXySOzAP/QLHRh/TD+OG9CVI/t2plenvGZ5SZqItAxKBM2Iu7Ny0x7+tuQz3vhwI4uKdgLQt2sbJh7bj5MGdmd0/y50TXGXqIjI/lIiaAY+3VrCXxau46WF6/jv5mIAjuzTiR+OHcJpQ3sy5JAO+sUvIoFRIsiQkvIILy9az58XrGPOqm0AHJvflStOzOdLw7/AFzrlZThCEQkLJYImtnXPXp7+92r+MGcN20sqGNC9Hd//0mAuHN2H3p3bZDo8EQkhJYImsnZbCQ/PWslLC9dRVhHljGGHcM0ph3FMfhd1+4hIRikRBGztthIeemslf1pQRJYZ5x/Vm6tPPoyBPdtnOjQREUCJIDDbi8t5/J+reOpfnwAw/ph+fPuLh9Ork7p/RKR5USJoZJHKKNPmruXXb3zMlj17+WrBofzo7KFKACLSbCkRNKKVm/bwgxcWsWjtDkb378LTVx7DyN6dMh2WiEidlAgagbvz7HufcvuMpbTJzebBCUdx7qheOgksIi2CEsEBKquo5Md/+oA/LVjHmCE9uO9rR9K9fetMhyUikjYlggOwfkcpV/1+Hss/28UNZwziu6cN0tOxRKTFUSLYT2u2FjNh8hx2llbw5OWFnDb0kEyHJCKyX5QI9sPqLcVc8sR7lFRU8ty3TtAJYRFp0ZQIGmjp+p1c8bu5VFRG+cNVxykJiEiLpwfONsAnW2LdQdlmPHeNWgIicnBQiyBNO0rKuer3c8nJzuKFa0+gb9e2mQ5JRKRRqEWQhkhllOv+uIBPt5bw6CVHKwmIyEFFLYI0/PJvy/nXyi388sIjOO6wbpkOR0SkUalFUI83P9zIb9/5hInH9ePiY/plOhwRkUanRFCHzbv38qMXFzP0Cx249SvDMx2OiEgg1DVUhzteXsrO0gr+OOl48nKzMx2OiEgg1CKoxb8+3sIrizfw3dMGMfiQDpkOR0QkMEoEKUSjzt1/+5BenfL41qmHZTocEZFAKRGk8PqyjSxZt4vvf2kwrXPUJSQiBzclghrKKiq5a+YyBvZsz1eP6p3pcEREAqdEUMML89aydlspt35lOLnZ2j0icvDTkS5JRWWU377zCSN7d+TkQd0zHY6ISJMINBGY2Vgz+8jMVprZzSnm9zOzWWa2wMwWm9k5QcZTn78u+YxPt5XwvdMH6zGTIhIagSUCM8sGHgbOBoYDE8ys5l1ZPwGed/ejgPHAI0HFk44X5q2lV6c8Th/aM5NhiIg0qSBbBMcCK919lbuXA9OA82rUcaBjfLoTsD7AeOpUtL2Ed1du4cKj+5Clx02KSIgEmQh6A2uTPhfFy5LdDlxqZkXATOC7qVZkZteY2Twzm7d58+YgYmX6/CKiDhOO03hCIhIuQSaCVD+rvcbnCcDT7t4HOAeYYmb7xOTuk9290N0Le/To0eiBujsvzCvifwZ2p3fnNo2+fhGR5izIRFAE9E363Id9u36uAp4HcPfZQB7Q5JfrLCraybodpbpvQERCKchEMBcYZGYDzKwVsZPBM2rU+RQ4HcDMhhFLBMH0/dThb0s+IyfLOGOYThKLSPgElgjcPQJcB7wGfEjs6qClZnanmY2LV/sBMMnMFgFTgSvcvWb3UeDeWr6RYwd0pXPbVk391SIiGRfoMNTuPpPYSeDksluTppcBJwUZQ3027S5jxcY9nH9Un0yGISKSMaG/s3jW8k0AnDq48U9Ci4i0BKFPBP/+71a6t2/NsF565oCIhFPoE8G81ds57rCuGlJCREIr1IlgW3E563aUckTvTpkORUQkY0KdCBau3Q5AQd/OGY5ERCRzQp0IFhftxAy1CEQk1EKdCBau3cHgnh1o1zrQq2hFRJq1UCeCDzfsYvihHeuvKCJyEAttItiws5SNu/aqW0hEQi+0ieA/a3YAcFQ/nSgWkXALbSL4aONuzFDXkIiEXmgTwbL1Ozmsezta52RnOhQRkYwKbSL4aONuhh+q8wMiIqFMBKXllRRtL2Vgj/aZDkVEJONCmQjW7SjBHfp102MpRURCmQj+u7kYgMO6q0UgIhLKRLBueykAfbqoRSAiEspEsGZrMe1aZdO1nR5NKSISykTw6bYS8ru30zMIREQIaSLYsLOMXp3ULSQiAiFNBJt376Vnx9aZDkNEpFkIXSIoq6hka3E5X+iYl+lQRESahdAlgrXbSgDo17VthiMREWkeQpcINu7aC0CvTmoRiIhACBPB1uJYItCloyIiMaFLBFVdQ326qGtIRARCmAg27tpLx7wc2rTS8NMiIhDCRLBw7Q56dNCloyIiVUKXCNq1zqai0jMdhohIsxG6RLBs/S4O79Eu02GIiDQbaSUCM3vRzL5sZg1KHGY21sw+MrOVZnZzLXW+bmbLzGypmf2xIevfHx3ycomqQSAikpDugf1RYCLwsZndbWZD61vAzLKBh4GzgeHABDMbXqPOIOD/ASe5+wjghoYEvz+2l5QzqKeeQyAiUiWtRODub7j7JcDRwGrg72b2bzO70sxya1nsWGClu69y93JgGnBejTqTgIfdfXv8ezbtz0akqzLqlJRX0j4vJ8ivERFpUdLu6jGzbsAVwNXAAuA3xBLD32tZpDewNulzUbws2WBgsJm9a2ZzzGxsLd99jZnNM7N5mzdvTjfkfRSXRwBo10qJQESkSlpHRDP7EzAUmAKc6+4b4rOeM7N5tS2Woqxm73wOMAgYA/QB3jGzke6+o9pC7pOByQCFhYX73cO/uyyWCDq2USIQEamS7hHxCXefmVxgZq3dfa+7F9ayTBHQN+lzH2B9ijpz3L0C+MTMPiKWGOamGVeDbN0TG17CUuYoEZFwSrdr6OcpymbXs8xcYJCZDTCzVsB4YEaNOi8BXwQws+7EuopWpRlTg5VVRAHo1VkDzomIVKmzRWBmXyDWr9/GzI7i8+6ejkCdg/W4e8TMrgNeA7KBp9x9qZndCcxz9xnxeWea2TKgEvg/d996QFtUhz17KwBo31pdQyIiVeo7Ip5F7ARxH+D+pPLdwI/rW3m8O2lmjbJbk6Yd+H78FbiS8koA2ikRiIgk1HlEdPffA783swvd/cUmiikwVYmgTa4GnBMRqVJf19Cl7v4HIN/M9vnV7u73p1is2SrZG7tqqK1GHhURSaivj6RqUJ6D4lbc4niLQDeUiYh8rr6uocfjk4+4+/7fydVM7NkbITfbaJ2jFoGISJV0Lx/9t5m9bmZXmVmXQCMKUPHeCG11V7GISDXpjjU0CPgJMAKYb2avmNmlgUYWgFWbi3WiWESkhrTHGnL39939+8QGk9sG/D6wqALSsU0OO0rLMx2GiEizku7zCDqa2eVm9lfg38AGYgmhRYlUOvnd9FAaEZFk6XaYLyI2HMSd7l7f0BLNVkVllNzs0D2UTUSkTukmgsPidwG3aOWVUVrlKBGIiCSr74ayX7v7DcAMM9snEbj7uMAiC8Deiiit1CIQEammvhbBlPj7fUEH0hT2RqJ00M1kIiLV1HdD2fz4ZIG7/yZ5npl9D/hHUIEFoayikjxdPioiUk26/SSXpyi7ohHjaBJ7I1Fa6xyBiEg19Z0jmABMBAaYWfJDZToAgT03ICh7I5UaXkJEpIb6Osyr7hnoDvwqqXw3sDiooIJSHtFVQyIiNdV3jmANsAY4oWnCCVZFpes+AhGRGurrGvqXu/+Pme0Gki8fNWIPGOsYaHSNTC0CEZF91dci+J/4e4emCSc40ajrhjIRkRTSHWvocDNrHZ8eY2bXm1nnYENrXBXRKICuGhIRqSHdo+KLQKWZDQSeBAYAfwwsqgBEKmM9WzlZluFIRESal3QTQdTdI8D5wK/d/UagV3BhNb5EItDJYhGRatI9KlbE7ym4HHglXpYbTEjBKK+MdQ21ylaLQEQkWbqJ4Epil5De5e6fmNkA4A/BhdX4IvFzBGoRiIhUl9YIbO6+DLg+6fMnwN1BBRWEqq6hbJ0jEBGpJq1EYGYnAbcD/ePLVN1HcFhwoTWuaPxxCtmmRCAikizdMZmfBG4E5gOVwYUTnMqoWgQiIqmkmwh2uvtfA40kYFUtgiwlAhGRatJNBLPM7F7gT8DeqkJ3/08gUQUgftGQuoZERGpINxEcF38vTCpz4LTGDSc4n3cNZTgQEZFmJt2rhr4YdCBBS3QNqUUgIlJNumMNHWJmT5rZX+Ofh5vZVWksN9bMPjKzlWZ2cx31LjIzN7PC2uocKJ0sFhFJLd2OkqeB14BD459XADfUtYCZZQMPA2cDw4EJZjY8Rb0OxO5ReC/NWPZLpU4Wi4iklG4i6O7uzwNRgPi4Q/VdRnossNLdV7l7OTANOC9FvZ8B9wBlacayX6JR3UcgIpJKuomg2My6EX84jZkdD+ysZ5newNqkz0XxsgQzOwro6+6vUAczu8bM5pnZvM2bN6cZcnVVT9VRHhARqS7dq4a+D8wADjezd4EewEX1LJPqkJt4ypmZZQEPAFfU9+XuPhmYDFBYWOj1VK9lHVVBKROIiCSrs0VgZseY2Rfi9wucCvyY2H0ErxP7hV+XIqBv0uc+wPqkzx2AkcDbZrYaOB6YEeQJY1CLQESkpvq6hh4HyuPTJwK3EDsBvJ34L/Q6zAUGmdkAM2sFjCfWqgDA3Xe6e3d3z3f3fGAOMM7d5zV8M+rnvl8NCRGRg159XUPZ7r4tPn0xMNndXwReNLOFdS3o7hEzu47Y1UbZwFPuvtTM7gTmufuMupZvbIlzBE35pSIiLUC9icDMcuJXCZ0OXNOAZXH3mcDMGmW31lJ3TH3rOxCuTCAiklJ9B/OpwD/MbAtQCrwDEH92cX1XDTVLOlksIlJdnYnA3e8yszeJPZ/4df+8oz0L+G7QwTUmR+cIRERSSad7Z06KshXBhBOgqstH1SAQEakmNGNx6hSBiEhqoUkEVUxNAhGRakKTCHQbgYhIauFJBPHOITUIRESqC00iqKI8ICJSXWgSgeuqIRGRlMKTCDIdgIhIMxWeRKAxJkREUgpNIqiiriERkepCkwjUHhARSS00iUAnCUREUgtNIvj8PgK1CUREkoUmEVRRGhARqS40iUD3EYiIpBa6RCAiItWFJxHE3/WEMhGR6kKTCKqoa0hEpLrQJAJX35CISErhSQSZDkBEpJkKTSKooq4hEZHqQpMIEpeP6mSxiEg1oUkE6AllIiIphSYR6FyxiEhqoUkEVdQiEBGpLjSJQDeUiYikFp5EoLGGRERSCk8i0J0EIiIpBZoIzGysmX1kZivN7OYU879vZsvMbLGZvWlm/YOMBzQMtYhITYElAjPLBh4GzgaGAxPMbHiNaguAQncfBUwH7gkqHnUNiYikFmSL4FhgpbuvcvdyYBpwXnIFd5/l7iXxj3OAPkEFo44hEZHUgkwEvYG1SZ+L4mW1uQr4a6oZZnaNmc0zs3mbN28+wLDUJBARSRZkIkh1xE35w9zMLgUKgXtTzXf3ye5e6O6FPXr02K9gqkYfVdeQiEh1OQGuuwjom/S5D7C+ZiUzOwO4BTjV3fcGGE/s+4L+AhGRFibIFsFcYJCZDTCzVsB4YEZyBTM7CngcGOfumwKMRUNMiIjUIrBE4O4R4DrgNeBD4Hl3X2pmd5rZuHi1e4H2wAtmttDMZtSyukZj6hsSEakmyK4h3H0mMLNG2a1J02cE+f3Vvrdq9NGm+kIRkRYiPHcW6z4CEZGUQpcIRESkutAkgioafVREpLrQJILEMNTKAyIi1YQnEahvSEQkpfAkgkwHICLSTIUmEVRR15CISHXhSQSJy0eVCUREkoUmEeiGMhGR1EKTCEREJLXQJALdWSwiklp4EkH8XTeUiYhUF55EoBaBiEhKoUkEIiKSWmgSga4aEhFJLTyJ4POTBCIikiQ8iSD+rpPFIiLVhSYRiIhIauFJBPG+IV01JCJSXWgSgU4RiIikFp5EoEHnRERSysl0ACIAFRUVFBUVUVZWlulQRFq0vLw8+vTpQ25ubtrLhCYRVD2hTO2B5qmoqIgOHTqQn5+vVpvIfnJ3tm7dSlFREQMGDEh7ufB0DcXfdYxpnsrKyujWrZuSgMgBMDO6devW4JZ1aBJBFd1H0HwpCYgcuP35OwpNItCz60VEUgtPIqia0I9OScPtt9/OfffdV2edl156iWXLljVovcuXL+eEE06gdevW9a6/qbk7119/PQMHDmTUqFH85z//SVnvueeeY9SoUYwYMYIf/vCHifIbb7yRgoICCgoKGDx4MJ07d6623K5du+jduzfXXXddoqy8vJxrrrmGwYMHM3ToUF588cXEvOeff57hw4czYsQIJk6cCMCsWbMS31FQUEBeXh4vvfQSAJdccglDhgxh5MiRfPOb36SiogKAt99+m06dOiWWufPOO6vFVVlZyVFHHcVXvvKVRFlt66paX0FBASNGjODUU09NlOfn53PEEUdQUFBAYWFhovynP/0po0aNoqCggDPPPJP169entb9T7a8xY8YwZMiQxLZs2rQp5b9Rg7l7i3qNHj3a98dv//lf7/+jV3xnafl+LS/BWrZsWaZDqOa2227ze++9t846l19+ub/wwgsNWu/GjRv9/fff9x//+Mf1rr+pvfrqqz527FiPRqM+e/ZsP/bYY/eps2XLFu/bt69v2rTJ3d0vu+wyf+ONN/ap9+CDD/qVV15Zrez666/3CRMm+He+851E2a233uq33HKLu7tXVlb65s2b3d19xYoVXlBQ4Nu2bXP32H6raevWrd6lSxcvLi5OxB+NRj0ajfr48eP9kUcecXf3WbNm+Ze//OVat/tXv/qVT5gwoVqd2ta1fft2HzZsmK9Zs2afuPr375+IP9nOnTsT07/5zW/8W9/6VuI76trfqfbXqaee6nPnzq11W6qk+nsC5nktx9XQXDVURQ2C5u+Ol5eybP2uRl3n8EM7ctu5I+qsc9ddd/HMM8/Qt29fevTowejRowH47W9/y+TJkykvL2fgwIFMmTKFhQsXMmPGDP7xj3/w85//nBdffJG33nprn3pt27at9h09e/akZ8+evPrqq2nHfuedd/Lyyy9TWlrKiSeeyOOPP46ZMWbMGO677z4KCwvZsmULhYWFrF69msrKSn70ox/x2muvYWZMmjSJ7373u/V+z1/+8hcuu+wyzIzjjz+eHTt2sGHDBnr16pWos2rVKgYPHkyPHj0AOOOMM3jxxRc5/fTTq61r6tSp3HHHHYnP8+fPZ+PGjYwdO5Z58+Ylyp966imWL18OQFZWFt27d0/s8+985zt06dIlsd9qmj59OmeffXZiH59zzjmJecceeyxFRUX1bnNRURGvvvoqt9xyC/fff3+ivLZ1/fGPf+SCCy6gX79+tcZVU8eOHRPTxcXFiT78uvZ3bfsrKKHpGhKpy/z585k2bRoLFizgT3/6E3Pnzk3Mu+CCC5g7dy6LFi1i2LBhPPnkk5x44omMGzeOe++9l4ULF3L44YenrNcYrrvuOubOncuSJUsoLS3llVdeqbP+5MmT+eSTT1iwYAGLFy/mkksuAap33SS/7r77bgDWrVtH3759E+vp06cP69atq7bugQMHsnz5clavXk0kEuGll15i7dq11eqsWbOGTz75hNNOOw2AaDTKD37wA+69995q9Xbs2AHEuk6OPvpovva1r7Fx40YAVqxYwYoVKzjppJM4/vjj+dvf/rbPdk6bNo0JEybsU15RUcGUKVMYO3Zsomz27NkceeSRnH322SxdujRRfsMNN3DPPfeQlZX6UFhzXStWrGD79u2MGTOG0aNH88wzzyTqmhlnnnkmo0ePZvLkydXWc8stt9C3b1+effbZRNdUbfu7tv1V5corr6SgoICf/exnicviD1RoWgS6s7jlqO+XexDeeecdzj///MSvy3HjxiXmLVmyhJ/85Cfs2LGDPXv2cNZZZ6VcR7r1GmrWrFncc889lJSUsG3bNkaMGMG5555ba/033niDa6+9lpyc2J93165dAXjggQfq/J5UB5Wafy9dunTh0Ucf5eKLLyYrK4sTTzyRVatWVaszbdo0LrroIrKzswF45JFHOOecc6od9AAikQhFRUWcdNJJ3H///dx///3cdNNNTJkyhUgkwscff8zbb79NUVERJ598MkuWLEmcd9iwYQMffPBByn387W9/m1NOOYWTTz4ZgKOPPpo1a9bQvn17Zs6cyVe/+lU+/vhjXnnlFXr27Mno0aN5++23U+6TmuuKRCLMnz+fN998k9LSUk444QSOP/54Bg8ezLvvvsuhhx7Kpk2b+NKXvsTQoUM55ZRTgFhr86677uIXv/gFDz30EHfccUet+7u2/QXw7LPP0rt3b3bv3s2FF17IlClTuOyyy1LG3hCBtgjMbKyZfWRmK83s5hTzW5vZc/H575lZflCx6ME0Up/afiRcccUVPPTQQ3zwwQfcdttttV6jnW69higrK+Pb3/4206dP54MPPmDSpEmJ9ebk5BCNRhP1qrh7ym2pr0XQp0+far/ui4qKOPTQQ/dZz7nnnst7773H7NmzGTJkCIMGDao2v+Yv9dmzZ/PQQw+Rn5/PTTfdxDPPPMPNN99Mt27daNu2Leeffz4AX/va1xInTPv06cN5551Hbm4uAwYMYMiQIXz88ceJdT7//POcf/75+9w9e8cdd7B58+Zq3TwdO3akffv2QKzLp6Kigi1btvDuu+8yY8YM8vPzGT9+PG+99RaXXnppnevq06cPY8eOpV27dnTv3p1TTjmFRYsWAST2Vc+ePTn//PN5//3399l3EydOTJwQr21/17a/AHr37g1Ahw4dmDhxYsrv2B+BJQIzywYeBs4GhgMTzGx4jWpXAdvdfSDwAPDLoOL5PK6gv0FaolNOOYU///nPlJaWsnv3bl5++eXEvN27d9OrVy8qKip49tlnE+UdOnRg9+7d9dZL1+mnn75PV0zVAb579+7s2bOH6dOnJ+bl5+czf/58gGrlZ555Jo899hiRSASAbdu2AbEWwcKFC/d5VR1kxo0bxzPPPIO7M2fOHDp16lTt/ECVqitVtm/fziOPPMLVV1+dmPfRRx+xfft2TjjhhETZs88+y6effsrq1au57777uOyyy7j77rsxM84999zEr/E333yT4cNjh4ivfvWrzJo1C4AtW7awYsUKDjvssMQ6p06duk+30BNPPMFrr73G1KlTq3X1fPbZZ4lf3++//z7RaJRu3brxi1/8gqKiIlavXs20adM47bTT+HHOyBQAAAoBSURBVMMf/lDnus477zzeeecdIpEIJSUlvPfeewwbNozi4uLE/4Xi4mJef/11Ro4cCVAtgc2YMYOhQ4fWub9r21+RSIQtW7YAsS6rV155JfEdByrIrqFjgZXuvgrAzKYB5wHJ19udB9wen54OPGRm5o3V8ZVE9xFIXY4++mguvvhiCgoK6N+/f6IrAOBnP/sZxx13HP379+eII45I/MGPHz+eSZMm8eCDDzJ9+vRa6yX77LPPKCwsZNeuXWRlZfHrX/+aZcuW0b59e1auXJnoxqnSuXNnJk2axBFHHEF+fj7HHHNMYt5NN93E17/+daZMmZLojwe4+uqrWbFiBaNGjSI3N5dJkyZVuwSxNueccw4zZ85k4MCBtG3blt/97neJeQUFBSxcuBCA733ve4lfwbfeeiuDBw9O1Js6dSrjx49Puwv2l7/8Jd/4xje44YYb6NGjR+I7zzrrLF5//XWGDx9OdnY29957L926dQNg9erVrF27ttqlmwDXXnst/fv3TyShCy64gFtvvZXp06fz6KOPkpOTQ5s2bZg2bVq98dW2rmHDhjF27FhGjRpFVlYWV199NSNHjmTVqlWJlk0kEmHixImJ8wo333wzH330EVlZWfTv35/HHnus3v2dyt69eznrrLOoqKigsrKSM844g0mTJqW1n+tjARxzYys2uwgY6+5Xxz9/AzjO3a9LqrMkXqco/vm/8TpbaqzrGuAagH79+o1es2ZNg+P5+7KNvLRgHfdffCStc7L3d7MkIB9++CHDhg3LdBgZs2TJEp566qlq3RAi+yvV35OZzXf3wlT1g2wRpEq5NbNOOnVw98nAZIDCwsL9ylxfGn4IXxp+yP4sKhK4kSNHKglIxgR5srgISD7t3QdYX1sdM8sBOgHbAoxJRERqCDIRzAUGmdkAM2sFjAdm1KgzA7g8Pn0R8FYQ5wekZdA/vciB25+/o8ASgbtHgOuA14APgefdfamZ3WlmVRdpPwl0M7OVwPeBfS4xlXDIy8tj69atSgYiB8DjzyPIy8tr0HKBnSwOSmFhoTfFLdfStPSEMpHGUdsTyjJ1slgkbVU3DolI09NYQyIiIadEICISckoEIiIh1+JOFpvZZqDhtxbHdAe21Fvr4KJtDgdtczgcyDb3d/ceqWa0uERwIMxsXm1nzQ9W2uZw0DaHQ1DbrK4hEZGQUyIQEQm5sCWCyfVXOehom8NB2xwOgWxzqM4RiIjIvsLWIhARkRqUCEREQu6gTARmNtbMPjKzlWa2z4imZtbazJ6Lz3/PzPKbPsrGlcY2f9/MlpnZYjN708z6ZyLOxlTfNifVu8jM3Mxa/KWG6WyzmX09/m+91Mz+2NQxNrY0/m/3M7NZZrYg/v/7nEzE2VjM7Ckz2xR/gmOq+WZmD8b3x2IzO/qAv9TdD6oXkA38FzgMaAUsAobXqPNt4LH49HjguUzH3QTb/EWgbXz6f8OwzfF6HYB/AnOAwkzH3QT/zoOABUCX+OeemY67CbZ5MvC/8enhwOpMx32A23wKcDSwpJb55wB/JfaEx+OB9w70Ow/GFsGxwEp3X+Xu5cA04Lwadc4Dfh+fng6cbuk+bbt5qneb3X2Wu5fEP84h9sS4liydf2eAnwH3AAfD+NbpbPMk4GF33w7g7puaOMbGls42O9AxPt2JfZ+E2KK4+z+p+0mN5wHPeMwcoLOZ9TqQ7zwYE0FvYG3S56J4Wco6HnuAzk6gW5NEF4x0tjnZVcR+UbRk9W6zmR0F9HX3V5oysACl8+88GBhsZu+a2RwzG9tk0QUjnW2+HbjUzIqAmcB3mya0jGno33u9DsbnEaT6ZV/zGtl06rQkaW+PmV0KFAKnBhpR8OrcZjPLAh4ArmiqgJpAOv/OOcS6h8YQa/W9Y2Yj3X1HwLEFJZ1tngA87e6/MrMTgCnxbY4GH15GNPrx62BsERQBfZM+92HfpmKijpnlEGtO1tUUa+7S2WbM7AzgFmCcu+9totiCUt82dwBGAm+b2WpifakzWvgJ43T/b//F3Svc/RPgI2KJoaVKZ5uvAp4HcPfZQB6xwdkOVmn9vTfEwZgI5gKDzGyAmbUidjJ4Ro06M4DL49MXAW95/CxMC1XvNse7SR4nlgRaer8x1LPN7r7T3bu7e7675xM7LzLO3Vvyc07T+b/9ErELAzCz7sS6ilY1aZSNK51t/hQ4HcDMhhFLBJubNMqmNQO4LH710PHATnffcCArPOi6htw9YmbXAa8Ru+LgKXdfamZ3AvPcfQbwJLHm40piLYHxmYv4wKW5zfcC7YEX4ufFP3X3cRkL+gCluc0HlTS3+TXgTDNbBlQC/+fuWzMX9YFJc5t/APzWzG4k1kVyRUv+YWdmU4l17XWPn/e4DcgFcPfHiJ0HOQdYCZQAVx7wd7bg/SUiIo3gYOwaEhGRBlAiEBEJOSUCEZGQUyIQEQk5JQIRkZBTIpCDUn0jOMbr3BIfoXOxmS00s+MaOYaZZtY5Pn29mX1oZs+a2bi6RkuN1/93/D3fzCY2ZlwiNenyUTkomdkpwB5ig3ONTDH/BOB+YIy7743ffNXK3QMZsMzMlgNnx+/2bchyY4Cb3P0rQcQlAmoRyEEqjREcewFbqobacPctVUnAzFab2S/N7P34a2C8vIeZvWhmc+Ovk+Ll7c3sd2b2Qbx1cWHSerqb2WPEhlGeYWY3mtkVZvZQvM4hZvZnM1sUf50YL98Tj/Nu4OR4i+VGM3vHzAqqNiI+uNyoRtx1EkJKBBJWrwN9zWyFmT1iZjUH4dvl7scCDwG/jpf9BnjA3Y8BLgSeiJf/lNht/ke4+yjgreQVufu1xMaC+aK7P1Djex4E/uHuRxIbg35pjfk3A++4e0F82SeID6RnZoOB1u6+eD+2XyRBiUBCyd33AKOBa4iNS/OcmV2RVGVq0vsJ8ekzgIfMbCGx8V46mlmHePnDSeve3oBQTgMejS9X6e4766n/AvAVM8sFvgk83YDvEknpoBtrSCQVM+sLvBz/+Ji7P+bulcDbxEYo/YDYQIRPx+sknzyrms4CTnD30hrrNppoGHN3LzGzvxN7OMnXiQ0pLnJA1CKQUHD3tfHulQJ3f8zMhphZ8vDMBcCapM8XJ73Pjk+/DlxXVSGpr75meZcGhPYmsUeHYmbZZtaxxvzdxIbUTvYEsS6lue7ekodPl2ZCiUAOSvERHGcDQ8ysyMyuqlGlPfB7iz3kfTGxZ93enjS/tZm9B3wPuDFedj1QGD8hvAy4Nl7+c6CLmS0xs0XEh4FO0/eAL8ZbJPOBETXmLwYi8RPJNwK4+3xgF/C7BnyPSK10+ahIDRZ7kE2hu2/JdCypmNmhxLq0hh7ET+GSJqQWgUgLYmaXAe8BtygJSGNRi0BEJOTUIhARCTklAhGRkFMiEBEJOSUCEZGQUyIQEQm5/w9vBOL06X/Z/AAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Neural-Nets: PR auc=0.993\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/IPython/core/pylabtools.py:132: UserWarning: Creating legend with loc=\"best\" can be slow with large amounts of data.\n", " fig.canvas.print_figure(bytes_io, **kw)\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAeb0lEQVR4nO3dfZhVZb3/8fdHHhwzRRHyoMNThQbYADKSpobnkIBEmtpRMxU0I000PXrKwqNoceqcH/00ozRKfKAHNMqCsrwUJTVRGQRUUE6IKBMcG0HxCeTpe/7Yi3GYWczsYWbNnj3zeV3XXLPWutfa+3sPw/7Mvdba91ZEYGZmVttehS7AzMxaJweEmZmlckCYmVkqB4SZmaVyQJiZWaqOhS6guXTr1i369OlT6DLMzIrKokWLXouI7mltbSYg+vTpQ0VFRaHLMDMrKpJe3l2bTzGZmVkqB4SZmaVyQJiZWSoHhJmZpXJAmJlZqswCQtIMSf+Q9Nxu2iXpZkkrJT0j6cgabeMk/S35GpdVjWZmtntZ3uZ6BzANuGs37ScB/ZKvTwC3AJ+Q1BW4DigHAlgkaU5EvJ5ZpWuegtWPQp/joeew3bfBrvvtbNvnINi0Pv37/y7NdeOfBueW3/5Heg2bXod3XoOOnWHbltz3zW/Ctvdy7dvfg+1bAcGOrRA73j82dsCOHbnnoZXPzjt5Y6ErMLM8ZRYQEfGIpD717HIKcFfk5ht/QtIBknoAJwAPRMQGAEkPAKOBX2VS6EuPwszPwY7tsFdH+MRX4EP9oWMJvLEG5v8nbN8Ge3UAIvdC3KEjlF8AFTOSF+1W/qLcmkzuUugKio9D1QqkkG+UOxRYU2O9Mtm2u+11SJoATADo1avXnlWx6mHYsS23vGMrLJiWvt+OGn+xb98CT966Z89n1lgtEaoOIUtRyIBQyraoZ3vdjRHTgekA5eXle/Zn/GGjYcGPcy/6HTrBGXflRhBbN0PlQvjDFbngUIfkSbfDXp3g6IvgiVuTEcSOpOxI+W5WBPY0hBwsbVohA6IS6FljvRRYm2w/odb2+ZlV0XMYjJuTfg2i+2HQrd/ur0F8bKyvQVj71phgcZgUHWX5kaPJNYg/RMQRKW2fASYCY8hdpL45IoYlF6kXATvvanoaGLrzmsTulJeXh+diaqV83cHy4QApCEmLIqI8rS2zEYSkX5EbCXSTVEnuzqROABFxK3AfuXBYCbwLnJ+0bZD0bWBh8lA3NBQO1sr5P37jtNdAbajf/j1qcZmOIFqSRxBmjdRWgsjB0SQFGUGYWSu3py+srS1Y0upxaDQLB4SZNU5jXnwLFSYOjWbhgDCz7DT0otySAVL7uRwYDXJAmFnhFDJAaj62wyKVA8LMWq/dvXA3d3A4LFI5IMys+KS9iH/7Q7k3lDb5sR0WOzkgzKxt+I+UWQqaOtLYeXw7DQoHhJm1XbVf2Pd4zqnkuC694Ipnm1ZTEXFAmFn7UTMw9iQsNr7SrkYVDggza5+aGhaTuwB7weTsPsus0PyZ1GZmkze+/9UoO1rfO8ubkQPCzKymPQmKyV3aZFD4FJOZWZo9OQXVxq5PeARhZtaQxo4q2showgFhZpavxgRFGzjt5IAwM2usdjKacECYme2Jxo4mipADwsysKfINiiIMCQeEmVlzaIMh4YAwM2su+YwmiigkHBBmZs2tjYSEA8LMLAttICQcEGZmWSnykHBAmJllqYhDwgFhZpa1Ig0JB4SZWUsowpBwQJiZtZQim+XVAWFm1pLqC4lWNopwQJiZtbQiCQkHhJmZpXJAmJkVQhGMIhwQZmaF0spDwgFhZmapHBBmZoXUikcRDggzs0Jrpe+PcECYmbVmBRxFOCDMzFqDVjiKcECYmbV2BRpFOCDMzFqLVjaKyDQgJI2WtELSSklXp7T3ljRP0jOS5ksqrdG2XdKS5GtOlnWamVldmQWEpA7Aj4CTgAHAFyQNqLXbVOCuiCgDbgC+W6NtU0QMTr5OzqpOM7NWZXejiAKcZspyBDEMWBkRqyJiCzALOKXWPgOAecnywyntZmZWIFkGxKHAmhrrlcm2mpYCpyfLpwL7STooWS+RVCHpCUmfS3sCSROSfSqqqqqas3Yzs3Yvy4BQyraotX4VMFzSYmA48HdgW9LWKyLKgbOBmyR9pM6DRUyPiPKIKO/evXszlm5mVkCt5DRTxwwfuxLoWWO9FFhbc4eIWAucBiDpg8DpEbGxRhsRsUrSfGAI8GKG9ZqZWQ1ZjiAWAv0k9ZXUGTgL2OVuJEndJO2s4ZvAjGT7gZL23rkPcCywPMNazcyslswCIiK2AROB+4HngXsiYpmkGyTtvCvpBGCFpP8BDgamJNv7AxWSlpK7eP29iHBAmFn70QpOM2V5iomIuA+4r9a2a2sszwZmpxz3OPDxLGszM7P6+Z3UZmaWygFhZtZaFXjqDQeEmVmxaaHrEA4IMzNL5YAwM7NUDggzs9asgNchHBBmZsWoBa5DOCDMzCyVA8LMzFI5IMzMWrsCXYdwQJiZWSoHhJlZscr4QrUDwszMUjkgzMwslQPCzKwYFOBCtQPCzMxSOSDMzIrZ9V0ze2gHhJlZMYvtmT20A8LMzFI5IMzMikULX6h2QJiZWSoHhJmZpXJAmJlZKgeEmZmlckCYmVmqjvnuKOlQoHfNYyLikSyKMjOzRpjcJZM7nPIKCEn/BZwJLAd2visjAAeEmVkble8I4nPA4RHxXpbFmJlZ65HvNYhVQKcsCzEzszy04Jvl8h1BvAsskTQPqB5FRMRlmVTVTLZu3UplZSWbN28udCntTklJCaWlpXTq5L8rzIpVvgExJ/kqKpWVley333706dMHSYUup92ICNavX09lZSV9+/YtdDlmtofyCoiIuFNSZ+CwZNOKiNiaXVnNY/PmzQ6HApDEQQcdRFVVVaFLMbMmyPcuphOAO4HVgICeksYVw22uDofC8M/drPjle5H6+8DIiBgeEZ8CRgE3ZldW2yGJK6+8snp96tSpTJ48Oe/jX331VcaOHcugQYMYMGAAY8aMAWD+/PmMHTu2zv5z5szhe9/7HgCTJ09m6tSpAIwfP57Zs2c3oSdm1t7kGxCdImLFzpWI+B98V1Ne9t57b37729/y2muv7dHx1157LSeeeCJLly5l+fLl1S/+u3PyySdz9dVX79FzmZnVlG9AVEi6TdIJyddPgUVZFtZWdOzYkQkTJnDjjXUHXC+//DIjRoygrKyMESNG8Morr9TZZ926dZSWllavl5WV1dln4cKFDBkyhFWrVnHHHXcwceLE5u2EmbVL+d7FdDFwCXAZuWsQjwA/zqqorJz5kwV1to0t68G5x/Rh05btjL/9qTrtnx9ayr+W92TDO1u4+Oe7ZuLdXzkmr+e95JJLKCsr4+tf//ou2ydOnMh5553HuHHjmDFjBpdddhm/+93v6hx75plnMm3aND796U9z/vnnc8ghh1S3P/7441x66aX8/ve/p1evXjzySKu/LGRmWchguo28RhAR8V5E/P+IOC0iTo2IG/2u6vztv//+nHfeedx88827bF+wYAFnn302AOeeey6PPfZYnWNHjRrFqlWr+PKXv8wLL7zAkCFDqu8Oev7555kwYQJz586lV69e2XfEzNqVekcQku6JiDMkPUtu7qVdRETd8x27Hj8a+AHQAfhZRHyvVntvYAbQHdgAnBMRlUnbOOCaZNfvRMSd+XVp9+r7i3+fzh3qbe+6b+e8RwxpLr/8co488kjOP//83e6zuzt/unbtytlnn83ZZ5/N2LFjeeSRRzjooIPo0aMHmzdvZvHixbuMKszMmkNDI4ivJd/HAp9N+dotSR2AHwEnAQOAL0gaUGu3qcBdSdDcAHw3ObYrcB3wCWAYcJ2kA/PsU6vUtWtXzjjjDG677bbqbZ/85CeZNWsWAL/4xS847rjj6hz30EMP8e677wLw1ltv8eKLL1aPFg444AD++Mc/8q1vfYv58+dn3wkzax1aaLqNegMiItYli68BayLiZWBvYBCwtoHHHgasjIhVEbEFmAWcUmufAcC8ZPnhGu2jgAciYkNEvA48AIzOoz+t2pVXXrnL3Uw333wzt99+O2VlZcycOZMf/OAHdY5ZtGgR5eXllJWVccwxx3DhhRdy1FFHVbcffPDBzJ07l0suuYQnn3yyRfphZu2DIuqcOaq7k7QIOB44EHgCqADejYgv1nPM54HREXFhsn4u8ImImFhjn18CT0bEDySdBvwG6AacD5RExHeS/f4D2BQRU2s9xwRgAkCvXr2Gvvzyy7vU8Pzzz9O/f/8G+2fZ8M/fLEOTu6Rsa/zIQtKiiChPa8v3NldFxLvAacAPI+JUcn/913tMyrbaaXQVMFzSYmA48HdgW57HEhHTI6I8Isq7d+/eUB/MzKwR8g4ISccAXwT+mGxr6BbZSqBnjfVSap2Wioi1yZ1RQ4BJybaN+RxrZmbZyjcgLge+CdwbEcskfZjcNYP6LAT6SeqbTPR3FrVmhJXUTdLOGr5J7o4mgPuBkZIOTC5Oj0y2mZlZC8l3Nte/AH+psb6K3Jvm6jtmm6SJ5F7YOwAzknC5AaiIiDnACcB3Je38+NJLkmM3SPo2uZABuCEiNjSqZ2Zm1iQNvQ/ipoi4XNJc0q8BnFzf8RFxH3BfrW3X1lieDaTOIBcRM3h/RGFmZi2soRHEzOT71Hr3MjOzNqeh90HsnHyoAng0Iv6SnG56jPdP/1g9mjrd95464YQTqKioqLN9/vz5SGLu3LnV28aOHdvgG+3uuOMO1q71fQJm7Um+F6nnAR+osb4P8GDzl9P2NHW6792JCHbs2LFHx5aWljJlypRGHeOAMGt/8g2Ikoh4e+dKsvyBevYvXmuegke/n/veDOqb7ruqqorTTz+do446iqOOOoq//vWvwK4f9ANwxBFHsHr1alavXk3//v356le/ypFHHsmaNWu4+OKLKS8vZ+DAgVx33XV51TRo0CC6dOnCAw88UKdt0aJFDB8+nKFDhzJq1CjWrVvH7Nmzqaio4Itf/CKDBw9m06ZNXH311QwYMICysjKuuuqqPfzpmFlrlu903+9IOjIingaQNBTYlF1ZGfjT1fC/z9a/z3tvwqvPQewA7QUHHwF777/7/f/p43BS/R/gA7uf7vtrX/saV1xxBccddxyvvPIKo0aN4vnnn6/3sVasWMHtt9/Oj3+cm219ypQpdO3ale3btzNixAieeeaZ1M+MqO2aa67hmmuu4cQTT6zetnXr1uqpw7t3787dd9/NpEmTmDFjBtOmTWPq1KmUl5ezYcMG7r33Xl544QUk8cYbbzT4fGZWfPINiMuBX0vaeY6hB3BmNiUV0OaNuXCA3PfNG+sPiDzVnO57n332qd7+4IMPsnz58ur1N998k7feeqvex+rduzdHH3109fo999zD9OnT2bZtG+vWrWP58uV5BcTxxx8PwKOPPlq9bcWKFTz33HPVobF9+3Z69OiR2p+SkhIuvPBCPvOZz6R+9KmZFb983wexUNLHgMPJTYPxQkRszbSy5pbHX/qseQruPBm2b4EOneH0n0HPYc3y9GnTfe/YsYMFCxbsEhqQOy1V8/rC5s2bq5f33Xff6uWXXnqJqVOnsnDhQg488EDGjx+/y74A9957L9dffz0AP/vZz3ZpmzRpElOmTKFjx9yvQUQwcOBAFiyo+8FKtet76qmnmDdvHrNmzWLatGk89NBD+fwYzKyI5HUNQtIHgG8AX4uIZ4E+ktren409h8G4OfAvk3LfmykcIH2675EjRzJt2rTq9SVLlgDQp08fnn76aQCefvppXnrppdTHfPPNN9l3333p0qULr776Kn/605/q7HPqqaeyZMkSlixZQnn5rvNxjRw5ktdff52lS5cCcPjhh1NVVVUdEFu3bmXZsmUA7LffftWjm7fffpuNGzcyZswYbrrppuq6zaxtyfci9e3AFmDnJ+ZUAt/JpKJC6zkMjr+yWcNhp7TpvisqKigrK2PAgAHceuutAJx++uls2LCBwYMHc8stt3DYYYelPt6gQYMYMmQIAwcO5IILLuDYY49tdE2TJk2isrISgM6dOzN79my+8Y1vMGjQIAYPHszjjz8OwPjx47nooosYPHgwb731FmPHjqWsrIzhw4enXoA3s+KX73TfFRFRLmlxMrEekpZGxKDMK8xTeXl51L7v39NNF5Z//mYZSpvuGxo95XdzTPe9RdI+JNNtSPoI4M+kNjNrw/K9i+k64M9AT0m/AI4FxmdVlJmZFV6DIwhJAl4g92FB44FfAeURMT/TyszMbPda4HOpGxxBRERI+l1EDOX9DwsqGhFBLuOsJeVzbcvMWrd8r0E8IemoTCvJQElJCevXr/eLVQuLCNavX09JSUmhSzGzJsj3GsQ/AxdJWg28Q+7NchERDb9lt4BKS0uprKykqqqq0KW0OyUlJZSWlha6DDNrgnwD4qRMq8hIp06d6Nu3b6HLMDMrSg19olwJcBHwUeBZ4LaI2NYShZmZWWE1dA3iTqCcXDicBHw/84rMzKxVaOgU04CI+DiApNuA5vmQBDMza/UaGkFUz9jqU0tmZu1LQyOIQZLeTJYF7JOs77yLqekflmBmZq1SvQERER1aqhAzM2td8n2jnJmZtTMOCDMzS+WAMDOzVA4IMzNL5YAwM7NUDggzM0vlgDAzs1QOCDMzS+WAMDOzVA4IMzNL5YAwM7NUDggzM0vlgDAzs1QOCDMzS+WAMDOzVA4IMzNLlWlASBotaYWklZKuTmnvJelhSYslPSNpTLK9j6RNkpYkX7dmWaeZmdXV0EeO7jFJHYAfAScClcBCSXMiYnmN3a4B7omIWyQNAO4D+iRtL0bE4KzqMzOz+mU5ghgGrIyIVRGxBZgFnFJrnwB2fq51F2BthvWYmVkjZBkQhwJraqxXJttqmgycI6mS3Ojh0hptfZNTT3+RdHzaE0iaIKlCUkVVVVUzlm5mZlkGhFK2Ra31LwB3REQpMAaYKWkvYB3QKyKGAP8G/FLS/rWOJSKmR0R5RJR37969mcs3M2vfsgyISqBnjfVS6p5C+hJwD0BELABKgG4R8V5ErE+2LwJeBA7LsFYzM6sly4BYCPST1FdSZ+AsYE6tfV4BRgBI6k8uIKokdU8uciPpw0A/YFWGtZqZWS2Z3cUUEdskTQTuBzoAMyJimaQbgIqImANcCfxU0hXkTj+Nj4iQ9CngBknbgO3ARRGxIatazcysrswCAiAi7iN38bnmtmtrLC8Hjk057jfAb7KszczM6ud3UpuZWSoHhJmZpXJAmJlZKgeEmZmlckCYmVkqB4SZmaVyQJiZWSoHhJmZpXJAmJlZKgeEmZmlckCYmVkqB4SZmaVyQJiZWSoHhJmZpXJAmJlZKgeEmZmlckCYmVkqB4SZmaVyQJiZWSoHhJmZpXJAmJlZKgeEmZmlckCYmVkqB4SZmaVyQJiZWSoHhJmZpXJAmJlZKgeEmZmlckCYmVkqB4SZmaVyQJiZWSoHhJmZpXJAmJlZKgeEmZmlckCYmVkqB4SZmaVyQJiZWSoHhJmZpco0ICSNlrRC0kpJV6e095L0sKTFkp6RNKZG2zeT41ZIGpVlnWZmVlfHrB5YUgfgR8CJQCWwUNKciFheY7drgHsi4hZJA4D7gD7J8lnAQOAQ4EFJh0XE9qzqNTOzXWU5ghgGrIyIVRGxBZgFnFJrnwD2T5a7AGuT5VOAWRHxXkS8BKxMHs/MzFpIlgFxKLCmxnplsq2mycA5kirJjR4ubcSxSJogqUJSRVVVVXPVbWZmZBsQStkWtda/ANwREaXAGGCmpL3yPJaImB4R5RFR3r179yYXbGZm78vsGgS5v/p71lgv5f1TSDt9CRgNEBELJJUA3fI81szMMpTlCGIh0E9SX0mdyV10nlNrn1eAEQCS+gMlQFWy31mS9pbUF+gHPJVhrWZmVktmI4iI2CZpInA/0AGYERHLJN0AVETEHOBK4KeSriB3Cml8RASwTNI9wHJgG3CJ72AyM2tZWZ5iIiLuI3fxuea2a2ssLweO3c2xU4ApWdZnZma753dSm5lZqkxHEMXkzJ8sqLNtbFkPzj2mD5u2bGf87XUvgXx+aCn/Wt6TDe9s4eKfL6rTfs7RvfnsoENY+8Ymrrh7SZ32Lx//YT494GBerHqbb/322Trtl/5LP47r141lazdyw9zlddq/PvpwhvbuyqKXN/Dff15Rp/3azw5g4CFdeOxvr/HDh/5Wp/0/T/s4H+n+QR5c/io/fXRVnfYbzxzMIQfsw9yla/n5Ey/Xab/lnKF03bczv65Yw+xFlXXa7zh/GPt07sDMBav5wzPr6rTf/ZVjAJj+yIvMe/4fu7SVdOrAnRfk3vpy87y/8deVr+3SfuAHOnPruUMB+K8/v8DTL7++S3uPLiXcdNYQAK6fu4zla9/cpf3D3fflu6eVAfDN3z7Dqqp3dmkfcMj+XPfZgQBcPmsx6zZu3qX9yN4H8o3RHwPgopmLeP3dLbu0H/vRblw2oh8A42Y8xeatu54hHdH/Q0z41EcA/+75d2/Pf/d2kLvlM+22z+bgEYSZmaVS7ppw8SsvL4+KiopCl2Fm1nImd0nZtrFRDyFpUUSUp7V5BGFmVqxqh0Ejw6EhvgZhZlbMmjkUavIIwszMUjkgzMwslQPCzMxSOSDMzCyVA8LMzFI5IMzMLFWbeaOcpCqg7nvy89cNeK3BvdqW9tbn9tZfcJ/bi6b0uXdEpH7iWpsJiKaSVLG7dxO2Ve2tz+2tv+A+txdZ9dmnmMzMLJUDwszMUjkg3je90AUUQHvrc3vrL7jP7UUmffY1CDMzS+URhJmZpXJAmJlZqnYVEJJGS1ohaaWkq1Pa95Z0d9L+pKQ+LV9l88qjz/8mabmkZyTNk9S7EHU2p4b6XGO/z0sKSUV/S2Q+fZZ0RvJvvUzSL1u6xuaWx+92L0kPS1qc/H6PKUSdzUXSDEn/kPTcbtol6ebk5/GMpCOb/KQR0S6+gA7Ai8CHgc7AUmBArX2+CtyaLJ8F3F3oulugz/8MfCBZvrg99DnZbz/gEeAJoLzQdbfAv3M/YDFwYLL+oULX3QJ9ng5cnCwPAFYXuu4m9vlTwJHAc7tpHwP8idxHVB8NPNnU52xPI4hhwMqIWBURW4BZwCm19jkFuDNZng2MkJTV54G3hAb7HBEPR8S7yeoTQGkL19jc8vl3Bvg28N/A5pS2YpNPn78M/CgiXgeIiH+0cI3NLZ8+B7B/stwFWNuC9TW7iHgE2FDPLqcAd0XOE8ABkno05TnbU0AcCqypsV6ZbEvdJyK2ARuBg1qkumzk0+eavkTuL5Bi1mCfJQ0BekbEH1qysAzl8+98GHCYpL9KekLS6BarLhv59HkycI6kSuA+4NKWKa1gGvv/vUHt6SNH00YCte/xzWefYpJ3fySdA5QDwzOtKHv19lnSXsCNwPiWKqgF5PPv3JHcaaYTyI0SH5V0RES8kXFtWcmnz18A7oiI70s6BpiZ9HlH9uUVRLO/frWnEUQl0LPGeil1h5zV+0jqSG5YWt+QrrXLp89I+jQwCTg5It5rodqy0lCf9wOOAOZLWk3uXO2cIr9Qne/v9u8jYmtEvASsIBcYxSqfPn8JuAcgIhYAJeQmtWur8vr/3hjtKSAWAv0k9ZXUmdxF6Dm19pkDjEuWPw88FMnVnyLVYJ+T0y0/IRcOxX5eGhroc0RsjIhuEdEnIvqQu+5yckRUFKbcZpHP7/bvyN2QgKRu5E45rWrRKptXPn1+BRgBIKk/uYCoatEqW9Yc4LzkbqajgY0Rsa4pD9huTjFFxDZJE4H7yd0BMSMilkm6AaiIiDnAbeSGoSvJjRzOKlzFTZdnn/8f8EHg18n1+Fci4uSCFd1Eefa5Tcmzz/cDIyUtB7YD/x4R6wtXddPk2ecrgZ9KuoLcqZbxxfwHn6RfkTtF2C25rnId0AkgIm4ld51lDLASeBc4v8nPWcQ/LzMzy1B7OsVkZmaN4IAwM7NUDggzM0vlgDAzs1QOCDMzS+WAMGsESdslLZH0nKS5kg5o5scfL2lasjxZ0lXN+fhmjeGAMGucTRExOCKOIPdemUsKXZBZVhwQZntuATUmQ5P075IWJnPxX19j+3nJtqWSZibbPpt85shiSQ9KOrgA9ZvVq928k9qsOUnqQG4ah9uS9ZHk5jYaRm7StDmSPgWsJzfP1bER8ZqkrslDPAYcHREh6ULg6+Te+WvWajggzBpnH0lLgD7AIuCBZPvI5Gtxsv5BcoExCJgdEa8BRMTOyR9LgbuT+fo7Ay+1SPVmjeBTTGaNsykiBgO9yb2w77wGIeC7yfWJwRHx0Yi4LdmeNp/ND4FpEfFx4CvkJpIza1UcEGZ7ICI2ApcBV0nqRG7SuAskfRBA0qGSPgTMA86QdFCyfecppi7A35PlcZi1Qj7FZLaHImKxpKXAWRExM5lSekEyK+7bwDnJDKNTgL9I2k7uFNR4cp929mtJfyc35XjfQvTBrD6ezdXMzFL5FJOZmaVyQJiZWSoHhJmZpXJAmJlZKgeEmZmlckCYmVkqB4SZmaX6PzR/aHtByfpFAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" }, { "data": { "text/plain": [ "0.992858168571398" ] }, "execution_count": 286, "metadata": {}, "output_type": "execute_result" } ], "source": [ "analyze_results(X_test['target'].values,y_dnn_pred)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Results\n", "1. Model achieved an ROC-AUC of 0.97 on the holdout set below are some reviews classified as good and bad by the model\n", "\n", "#### Reviews classified as Good by the model\n", " 1. 'Awesome show, wish they would have continued it!',\n", " 2. 'Great movie.',\n", " 3. 'Absolutely beautiful!!!', \n", " 4. 'I love this show <3',\n", " 5. 'One of my favorite kind of shows. Love I!'\n", " \n", "#### Reviews classified as Bad by the model \n", " 1. If I could give this less than 1 star I would. The worst movie I've seen in ages. The acting was less than high school level. The Mockingjay character was mind numbingly boring and droll. She was terrible. The characters were beyond pathetic. 1 hour and 40 minutes of pure torture and inane plot, script combined with terrible CGI and effects I found myself wanting to consume mass quantities of alcohol. The whole Hunger Game sage is a total waste of time. How anyone can say this is anything but pure boredom is beyond me.\n", " 2. slow and boring, boring, boring, just like the book\n", " 3. \"Awful. A waste of money. A waste of time. Boring. Predictable. I gave up for the sake of my sanity about 20 minutes before it was over. Sandler and Barrymore must have been bored to tears with nothing to do, under contract to do something, and just didn't care........\",\n", " 4.'Horrible movie. Mediocre acting, terrible plot, confusing and boring.',\n", " 5.'The movie is a real waste of time and money, poor actors, poor story, poor locations... really awful movie. I waste my money with this piece of crap.',\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "conda_mxnet_p36", "language": "python", "name": "conda_mxnet_p36" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.10" } }, "nbformat": 4, "nbformat_minor": 4 }