{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build a Customer Churn Model for Music Streaming App Users: Model Selection and Model Explainability\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-west-2/use-cases|customer_churn|2_cust_churn_train_deploy_infer.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Background\n",
    "\n",
    "This notebook is one of a sequence of notebooks that show you how to use various SageMaker functionalities to build, train, and deploy the model from end to end, including data pre-processing steps like ingestion, cleaning and processing, feature engineering, training and hyperparameter tuning, model explainability, and eventually deploy the model. There are two parts of the demo: \n",
    "\n",
    "1. Build a Customer Churn Model for Music Streaming App Users: Overview and Data Preparation - you will process the data with the help of Data Wrangler, then create features from the cleaned data. By the end of part 1, you will have a complete feature data set that contains all attributes built for each user, and it is ready for modeling.\n",
    "1. Build a Customer Churn Model for Music Streaming App Users: Model Selection and Model Explainability (current notebook) - you will use the data set built from part 1 to find an optimal model for the use case, then test the model predictability with the test data. \n",
    "\n",
    "For how to set up the SageMaker Studio Notebook environment, please check the [onboarding video]( https://www.youtube.com/watch?v=wiDHCWVrjCU&feature=youtu.be). And for a list of services covered in the use case demo, please check the documentation linked in each section.\n",
    "\n",
    "\n",
    "## Content\n",
    "* [Model Selection](#Model-Selection)\n",
    "* [Training with SageMaker Estimator and Experiment](#Training-with-SageMaker-Estimator-and-Experiment)\n",
    "* [Hyperparameter Tuning with SageMaker Hyperparameter Tuning Job](#Hyperparameter-Tuning-with-SageMaker-Hyperparameter-Tuning-Job)\n",
    "* [Deploy the model with SageMaker Batch-transform](#Deploy-the-model-with-SageMaker-Batch-transform)\n",
    "* [Model Explainability with SageMaker Clarify](#Model-Explainability-with-SageMaker-Clarify)\n",
    "* [Optional: Automate your training and model selection with SageMaker Autopilot (Console)](#Optional:-Automate-your-training-and-model-selection-with-SageMaker-Autopilot-(Console))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "### What is Customer Churn and why is it important for businesses?\n",
    "\n",
    "Customer churn, or customer retention/attrition, means a customer has the tendency to leave and stop paying for a business. It is one of the primary metrics companies want to track to get a sense of their customer satisfaction, especially for a subscription-based business model. The company can track churn rate (defined as the percentage of customers churned during a period) as a health indicator for the business, but we would love to identify the at-risk customers before they churn and offer appropriate treatment to keep them with the business, and this is where machine learning comes into play.\n",
    "### Use Cases for Customer Churn\n",
    "\n",
    "Any subscription-based business would track customer churn as one of the most critical Key Performance Indicators (KPIs). Such companies and industries include Telecom companies (cable, cell phone, internet, etc.), digital subscriptions of media (news, forums, blogposts platforms, etc.), music and video streaming services, and other Software as a Service (SaaS) providers (e-commerce, CRM, Mar-Tech, cloud computing, video conference provider, and visualization and data science tools, etc.)\n",
    "\n",
    "### Define Business problem\n",
    "\n",
    "To start with, here are some common business problems to consider depending on your specific use cases and your focus:\n",
    "\n",
    " * Will this customer churn (cancel the plan, cancel the subscription)?\n",
    " * Will this customer downgrade a pricing plan?\n",
    " * For a subscription business model, will a customer renew his/her subscription?\n",
    "\n",
    "### Machine learning problem formulation\n",
    "\n",
    "#### Classification: will this customer churn?\n",
    "\n",
    "To goal of classification is to identify the at-risk customers and sometimes their unusual behavior, such as: will this customer churn or downgrade their plan? Is there any unusual behavior for a customer? The latter question can be formulated as an anomaly detection problem.\n",
    "\n",
    "#### Time Series: will this customer churn in the next X months? When will this customer churn?\n",
    "\n",
    "You can further explore your users by formulating the problem as a time series one and detect when will the customer churn.\n",
    "\n",
    "### Data Requirements\n",
    "\n",
    "#### Data collection Sources\n",
    "\n",
    "Some most common data sources used to construct a data set for churn analysis are:\n",
    "* Customer Relationship Management platform (CRM), \n",
    "* engagement and usage data (analytics services), \n",
    "* passive feedback (ratings based on your request), and active feedback (customer support request, feedback on social media and review platforms).\n",
    "\n",
    "#### Construct a Data Set for Churn Analysis\n",
    "\n",
    "Most raw data collected from the sources mentioned above are huge and often needs a lot of cleaning and pre-processing. For example, usage data is usually event-based log data and can be more than a few gigabytes every day; you can aggregate the data to user-level daily for further analysis. Feedback and review data are mostly text data, so you would need to clean and pre-process the natural language data to be normalized, machine-readable data. If you are joining multiple data sources (especially from different platforms) together, you would want to make sure all data points are consistent, and the user identity can be matched across different platforms.\n",
    "           \n",
    "#### Challenges with Customer Churn\n",
    "\n",
    "* Business related\n",
    "    * Importance of domain knowledge: this is critical when you start building features for the machine learning model. It is important to understand the business enough to decide which features would trigger retention.\n",
    "* Data issues\n",
    "    * fewer churn data available (imbalanced classes): data for churn analysis is often very imbalanced as most of the customers of a business are happy customers (usually).\n",
    "    * User identity mapping problem: if you are joining data from different platforms (CRM, email, feedback, mobile app, and website usage data), you would want to make sure user A is recognized as the same user across multiple platforms. There are third-party solutions that help you tackle this problem.\n",
    "    * Not collecting the right data for the use case or Lacking enough data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Selection\n",
    "\n",
    "You can experiment with all your model choices and see which one gives better results. A few things to note when you choose algorithms:\n",
    "* **Start with simple ones**: Usually for tabular data classification that does not contain complex unstructured data (text, audio, image, etc.), you can start with logistic regression to see how your data performs, as sometimes the simplest model gives great results if your data have a strong linear pattern.\n",
    "\n",
    "* **Think about your data structure**: For imbalanced class data like churn analysis, you can experiment with tree-based models like the random forest, gradient boosting, or XGboost since they are less sensitive to class imbalance.\n",
    "\n",
    "* **Interpretability**: logistic regression model generally has better interpretability because of its linearity. You can also use feature importance from tree-based models or Support Vector Machines as an overall observation, but not to your predicting instance level. Instead, you can utilize tools like SHAP or the SageMaker new feature SageMaker Clarify to better visualize which feature contributing more to your prediction results.\n",
    "\n",
    "In this use case, a tree-based model XGBoost is chosen due to consideration of imbalanced class, and in the family of tree based models, XGBoost usually gives best results as its built for model performance and computational speed. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training with SageMaker Estimator and Experiment\n",
    "\n",
    "Once you decide on a range of models you want to experiment with, you can start training and comparing model results to choose the best one. A few things left for you to make a decision:\n",
    "* SageMaker estimator configuration\n",
    "  * to initialize your training job, you would need to config your SageMaker estimator and SageMaker training image by specifying the model choice, instance size, and type.\n",
    "* Choose evaluation methods\n",
    "    * You can check the [model parameter documentation page](https://github.com/dmlc/xgboost/blob/master/doc/parameter.rst#learning-task-parameters) for all the evaluation metrics you can choose for a model. For a imbalanced classification problem, you can choose F1 as your evaluation especially for comparing different models;  area under curve (auc) is also a good choice when your output is probability.\n",
    "* Hyper-parameters\n",
    "    * You can look at the [documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost_hyperparameters.html) for a complete list of hyper-parameters tunable for the model (The XGBoost model here was given as an example). For best performances, you can experiment with a range of combinations for the hyper-parameters and compare the validation results.\n",
    "   \n",
    "### How to create a training job as a trial in SageMaker Experiment    \n",
    "\n",
    "#### Get ECR image URIs for pre-built SageMaker Docker images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install sagemaker-experiments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "import json\n",
    "import pandas as pd\n",
    "import glob\n",
    "import s3fs\n",
    "import boto3\n",
    "from datetime import datetime\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sagemaker_session = sagemaker.Session()\n",
    "s3 = sagemaker_session.boto_session.resource(\"s3\")\n",
    "\n",
    "region = boto3.Session().region_name\n",
    "role = sagemaker.get_execution_role()\n",
    "smclient = boto3.Session().client(\"sagemaker\")\n",
    "bucket = sagemaker_session.default_bucket()\n",
    "prefix = \"music-streaming\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='4'></a>\n",
    "\n",
    "### Download Data and Upload to S3\n",
    "\n",
    "We ingest the simulated data from the public SageMaker S3 training database. If you want to see how the train, test, and validation datasets are created in detail, look at [Build a Customer Churn Model for Music Streaming App Users: Overview and Data Preparation](0_cust_churn_overview_dw.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "##### Alternative: copy data from a public S3 bucket to your own bucket\n",
    "##### data file should include full_data.csv and sample.json\n",
    "#### cell 5 - 7 is not needed; the processing job before data wrangler screenshots is not needed\n",
    "!mkdir -p data/raw\n",
    "s3 = boto3.client(\"s3\")\n",
    "s3.download_file(\n",
    "    f\"sagemaker-example-files-prod-{region}\",\n",
    "    \"datasets/tabular/customer-churn/customer-churn-data-v2.zip\",\n",
    "    \"data/raw/customer-churn-data.zip\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!unzip -o ./data/raw/customer-churn-data.zip -d ./data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# unzip the partitioned data files into the same folder\n",
    "!unzip -o data/simu-1.zip -d data/raw\n",
    "!unzip -o data/simu-2.zip -d data/raw\n",
    "!unzip -o data/simu-3.zip -d data/raw\n",
    "!unzip -o data/simu-4.zip -d data/raw"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm ./data/raw/*.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!unzip -o data/sample.zip -d data/raw"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!aws s3 cp  ./data/raw s3://$bucket/$prefix/data/json/  --recursive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_input_train = (\n",
    "    boto3.Session()\n",
    "    .resource(\"s3\")\n",
    "    .Bucket(bucket)\n",
    "    .Object(os.path.join(prefix, \"train/train.csv\"))\n",
    "    .upload_file(\"data/train_updated.csv\")\n",
    ")\n",
    "s3_input_validation = (\n",
    "    boto3.Session()\n",
    "    .resource(\"s3\")\n",
    "    .Bucket(bucket)\n",
    "    .Object(os.path.join(prefix, \"validation/validation.csv\"))\n",
    "    .upload_file(\"data/validation_updated.csv\")\n",
    ")\n",
    "s3_input_validation = (\n",
    "    boto3.Session()\n",
    "    .resource(\"s3\")\n",
    "    .Bucket(bucket)\n",
    "    .Object(os.path.join(prefix, \"test/test_labeled.csv\"))\n",
    "    .upload_file(\"data/test_updated.csv\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Initialize Model Hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hyperparameters = {\n",
    "    \"max_depth\": \"12\",\n",
    "    \"eta\": \"0.08\",\n",
    "    \"gamma\": \"4\",\n",
    "    \"min_child_weight\": \"7\",\n",
    "    \"subsample\": \"0.7\",\n",
    "    \"eval_metric\": \"auc\",\n",
    "    \"objective\": \"binary:logistic\",\n",
    "    \"num_round\": \"800\",\n",
    "    \"early_stopping_rounds\": \"50\",\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define SageMaker estimator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "from time import gmtime, strftime\n",
    "\n",
    "container = sagemaker.image_uris.retrieve(\n",
    "    \"xgboost\", region, version=\"1.0-1\", instance_type=\"ml.m4.xlarge\"\n",
    ")\n",
    "\n",
    "\n",
    "xgb = sagemaker.estimator.Estimator(\n",
    "    container,\n",
    "    role,\n",
    "    instance_count=1,\n",
    "    instance_type=\"ml.m4.xlarge\",\n",
    "    output_path=\"s3://{}/{}/output\".format(bucket, prefix),\n",
    "    sagemaker_session=sagemaker_session,\n",
    ")\n",
    "\n",
    "xgb.set_hyperparameters(**hyperparameters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.inputs import TrainingInput\n",
    "\n",
    "content_type = \"csv\"\n",
    "train_input = TrainingInput(\n",
    "    \"s3://{}/{}/{}/\".format(bucket, prefix, \"train\"), content_type=content_type\n",
    ")\n",
    "validation_input = TrainingInput(\n",
    "    \"s3://{}/{}/{}/\".format(bucket, prefix, \"validation\"), content_type=content_type\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "xgb.fit(inputs={\"train\": train_input, \"validation\": validation_input}, wait=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define SageMaker Experiment and Trial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# custom trial name\n",
    "experiment_name = \"music-streaming-churn-exp-{}\".format(datetime.now().strftime(\"%Y%m%d-%H%M%S\"))\n",
    "trial_name_xgb = \"xgboost-{}\".format(datetime.now().strftime(\"%Y%m%d-%H%M%S\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "from smexperiments import experiment, trial\n",
    "from sagemaker import analytics\n",
    "\n",
    "# create experiment if it doesn't exist\n",
    "try:\n",
    "    my_experiment = experiment.Experiment.load(experiment_name=experiment_name)\n",
    "    print(f\"Experiment loaded {experiment_name}: SUCCESS\")\n",
    "except Exception as ex:\n",
    "    if \"ResourceNotFound\" in str(ex):\n",
    "        my_experiment = experiment.Experiment.create(experiment_name=experiment_name)\n",
    "        print(f\"Experiment creation {experiment_name}: SUCCESS\")\n",
    "\n",
    "# create the trial if it doesn't exist\n",
    "try:\n",
    "    my_trial = trial.Trial.load(trial_name=trial_name_xgb)\n",
    "    print(f\"Trial loaded {trial_name_xgb}: SUCCESS\")\n",
    "except Exception as ex:\n",
    "    if \"ResourceNotFound\" in str(ex):\n",
    "        my_trial = trial.Trial.create(experiment_name=experiment_name, trial_name=trial_name_xgb)\n",
    "        print(f\"Create trial {my_trial.trial_name}: SUCCESSFUL\")\n",
    "\n",
    "\n",
    "xgb.fit(\n",
    "    inputs={\"train\": train_input, \"validation\": validation_input},\n",
    "    wait=True,\n",
    "    experiment_config={\n",
    "        \"ExperimentName\": my_experiment.experiment_name,\n",
    "        \"TrialName\": my_trial.trial_name,\n",
    "        \"TrialComponentDisplayName\": \"churn-xgboost\",\n",
    "    },\n",
    "    logs=True,\n",
    ")\n",
    "\n",
    "trial_component_analytics = analytics.ExperimentAnalytics(\n",
    "    experiment_name=my_experiment.experiment_name\n",
    ")\n",
    "\n",
    "analytic_table = trial_component_analytics.dataframe()\n",
    "analytic_table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyperparameter Tuning with SageMaker Hyperparameter Tuning Job\n",
    "\n",
    "Now that you understand how training one model works and how to create a SageMaker experiment, and selected the XGBoost model as the final model, you will need to fine-tune the hyperparameters for the best model performances. For a xgboost model, you can start with defining ranges for the eta, alpha, min_child_weight, and max_depth. You can check the [documentation when considering what haperparameter to tune](https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning-considerations.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Specify the Hyperparameter Tuning Job Settings\n",
    "\n",
    "To specify settings for the hyperparameter tuning job, you define a JSON object. You pass the object as the value of the HyperParameterTuningJobConfig parameter to CreateHyperParameterTuningJob when you create the tuning job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tuning_job_config = {\n",
    "    \"ParameterRanges\": {\n",
    "        \"CategoricalParameterRanges\": [],\n",
    "        \"ContinuousParameterRanges\": [\n",
    "            {\"MaxValue\": \"1\", \"MinValue\": \"0\", \"Name\": \"eta\"},\n",
    "            {\"MaxValue\": \"2\", \"MinValue\": \"0\", \"Name\": \"alpha\"},\n",
    "            {\"MaxValue\": \"10\", \"MinValue\": \"1\", \"Name\": \"min_child_weight\"},\n",
    "        ],\n",
    "        \"IntegerParameterRanges\": [{\"MaxValue\": \"10\", \"MinValue\": \"1\", \"Name\": \"max_depth\"}],\n",
    "    },\n",
    "    \"ResourceLimits\": {\"MaxNumberOfTrainingJobs\": 20, \"MaxParallelTrainingJobs\": 3},\n",
    "    \"Strategy\": \"Bayesian\",\n",
    "    \"TrainingJobEarlyStoppingType\": \"Auto\",\n",
    "    \"HyperParameterTuningJobObjective\": {\"MetricName\": \"validation:auc\", \"Type\": \"Maximize\"},\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure the Training Jobs\n",
    "\n",
    "To configure the training jobs that the tuning job launches, define a JSON object that you pass as the value of the TrainingJobDefinition parameter of the CreateHyperParameterTuningJob call."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_input_train = \"s3://{}/{}/train\".format(bucket, prefix)\n",
    "s3_input_validation = \"s3://{}/{}/validation\".format(bucket, prefix)\n",
    "\n",
    "training_job_definition = {\n",
    "    \"AlgorithmSpecification\": {\"TrainingImage\": container, \"TrainingInputMode\": \"File\"},\n",
    "    \"InputDataConfig\": [\n",
    "        {\n",
    "            \"ChannelName\": \"train\",\n",
    "            \"CompressionType\": \"None\",\n",
    "            \"ContentType\": \"csv\",\n",
    "            \"DataSource\": {\n",
    "                \"S3DataSource\": {\n",
    "                    \"S3DataDistributionType\": \"FullyReplicated\",\n",
    "                    \"S3DataType\": \"S3Prefix\",\n",
    "                    \"S3Uri\": s3_input_train,\n",
    "                }\n",
    "            },\n",
    "        },\n",
    "        {\n",
    "            \"ChannelName\": \"validation\",\n",
    "            \"CompressionType\": \"None\",\n",
    "            \"ContentType\": \"csv\",\n",
    "            \"DataSource\": {\n",
    "                \"S3DataSource\": {\n",
    "                    \"S3DataDistributionType\": \"FullyReplicated\",\n",
    "                    \"S3DataType\": \"S3Prefix\",\n",
    "                    \"S3Uri\": s3_input_validation,\n",
    "                }\n",
    "            },\n",
    "        },\n",
    "    ],\n",
    "    \"OutputDataConfig\": {\"S3OutputPath\": \"s3://{}/{}/output\".format(bucket, prefix)},\n",
    "    \"ResourceConfig\": {\"InstanceCount\": 2, \"InstanceType\": \"ml.c4.2xlarge\", \"VolumeSizeInGB\": 10},\n",
    "    \"RoleArn\": role,\n",
    "    \"StaticHyperParameters\": {\n",
    "        \"eval_metric\": \"auc\",\n",
    "        \"num_round\": \"100\",\n",
    "        \"objective\": \"binary:logistic\",\n",
    "        \"rate_drop\": \"0.3\",\n",
    "        \"tweedie_variance_power\": \"1.4\",\n",
    "    },\n",
    "    \"StoppingCondition\": {\"MaxRuntimeInSeconds\": 43200},\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Name and Launch the Hyperparameter Tuning Job\n",
    "\n",
    "Now you can provide a name for the hyperparameter tuning job and then launch it by calling the CreateHyperParameterTuningJob API. Pass tuning_job_config, and training_job_definition that you created in previous steps as the values of the parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# custom a tuner job name\n",
    "tuning_job_name = \"ChurnPredictTune-{}\".format(datetime.now().strftime(\"%Y%m%d-%H%M%S\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check if tuner job has been created\n",
    "list_tuning_job = smclient.list_hyper_parameter_tuning_jobs(NameContains=tuning_job_name)\n",
    "job_results = [[i for i in list_tuning_job[x]] for x in list_tuning_job.keys()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "smclient.list_hyper_parameter_tuning_jobs(NameContains=tuning_job_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.tuner import HyperparameterTuner\n",
    "\n",
    "# create the tuning job if it doesn't exist\n",
    "try:\n",
    "    if tuning_job_name == job_results[0][0][\"HyperParameterTuningJobName\"]:\n",
    "        print(f\"Tuning job exists\")\n",
    "except Exception as ex:\n",
    "    # create hyperparameter tuning job\n",
    "    smclient.create_hyper_parameter_tuning_job(\n",
    "        HyperParameterTuningJobName=tuning_job_name,\n",
    "        HyperParameterTuningJobConfig=tuning_job_config,\n",
    "        TrainingJobDefinition=training_job_definition,\n",
    "    )\n",
    "    print(f\"Create tuning job {tuning_job_name}: SUCCESSFUL\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Monitor the Progress of a Hyperparameter Tuning Job\n",
    "\n",
    "To monitor the progress of a hyperparameter tuning job and the training jobs that it launches, you can use the Amazon SageMaker console.\n",
    "\n",
    "<div>\n",
    "<img src=\"image/hp1.PNG\" width=\"600\"/>\n",
    "   </div>\n",
    "   \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy the model with SageMaker Batch-transform\n",
    "\n",
    "You can directly deploy the best model from your hyperparameter tuning job by getting the best training job from your tuner."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "# check status\n",
    "import time\n",
    "\n",
    "status = boto3.client(\"sagemaker\").describe_hyper_parameter_tuning_job(\n",
    "    HyperParameterTuningJobName=tuning_job_name\n",
    ")[\"HyperParameterTuningJobStatus\"]\n",
    "\n",
    "while status == \"InProgress\":\n",
    "    print(status)\n",
    "    time.sleep(60)\n",
    "    status = boto3.client(\"sagemaker\").describe_hyper_parameter_tuning_job(\n",
    "        HyperParameterTuningJobName=tuning_job_name\n",
    "    )[\"HyperParameterTuningJobStatus\"]\n",
    "\n",
    "print(status)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "# Attach to an existing hyperparameter tuning job.\n",
    "tuning_job_details = smclient.describe_hyper_parameter_tuning_job(\n",
    "    HyperParameterTuningJobName=tuning_job_name\n",
    ")\n",
    "xgb_tuner = HyperparameterTuner.attach(\n",
    "    tuning_job_name,\n",
    "    job_details=tuning_job_details,\n",
    "    sagemaker_session=sagemaker_session,\n",
    "    estimator_cls=None,\n",
    ")\n",
    "\n",
    "# Get the best XGBoost training job name from the HPO job\n",
    "xgb_best_training_job = xgb_tuner.best_training_job()\n",
    "print(xgb_best_training_job)\n",
    "# Attach estimator to the best training job name\n",
    "best_estimator = sagemaker.estimator.Estimator.attach(xgb_best_training_job)\n",
    "\n",
    "# Create model to be passed to the inference pipeline\n",
    "best_model = sagemaker.model.Model(\n",
    "    model_data=best_estimator.model_data,\n",
    "    role=sagemaker.get_execution_role(),\n",
    "    image_uri=best_estimator.image_uri,\n",
    ")\n",
    "\n",
    "predictor = best_model.deploy(initial_instance_count=1, instance_type=\"ml.m5.large\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also use the following code to find and deploy the best model. Replace with your best model output path. Go to SageMaker Console $\\rightarrow$ Hyperparameter Tuning Job $\\rightarrow$ [Your hyperparameter Tuning Job] $\\rightarrow$ Best model $\\rightarrow$ Output. You can also choose to **create a model** from the console under **best training job**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# replace with your best model output path\n",
    "# model_artifacts = 's3://{}/{}/output/ChurnPrediction-TuningJob-020-20eee831/output/model.tar.gz'.format(bucket, prefix)\n",
    "# best_model = sagemaker.model.Model(\n",
    "#    model_data= model_artifacts,\n",
    "#    image_uri =container,\n",
    "#    role=role)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# The location of the test dataset\n",
    "test_data = pd.read_csv(\"data/test_w_header.csv\")\n",
    "test_set = test_data.drop(columns=[\"user_churned\"])\n",
    "test_set.to_csv(\"data/test.csv\", index=False, header=False)\n",
    "s3_input_validation = (\n",
    "    boto3.Session()\n",
    "    .resource(\"s3\")\n",
    "    .Bucket(bucket)\n",
    "    .Object(os.path.join(prefix, \"test/testdata/test.csv\"))\n",
    "    .upload_file(\"data/test.csv\")\n",
    ")\n",
    "\n",
    "batch_input = \"s3://{}/{}/test/testdata\".format(bucket, prefix)\n",
    "\n",
    "batch_output = \"s3://{}/{}/batch-inference\".format(bucket, prefix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "transformer = best_model.transformer(\n",
    "    instance_count=1, instance_type=\"ml.m4.xlarge\", output_path=batch_output\n",
    ")\n",
    "transformer.transform(\n",
    "    data=batch_input, data_type=\"S3Prefix\", content_type=\"text/csv\", split_type=\"Line\"\n",
    ")\n",
    "transformer.wait()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wait for the previous step to be completed. Once done, you can download the prediction results to your instance for further analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3.download_file(bucket, prefix + \"/batch-inference/test.csv.out\", \"batch_results\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"batch_results\") as f:\n",
    "    results = f.readlines()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compare model results to actual\n",
    "\n",
    "You can evaluate your model results with the test data you left out earlier and check the precision and recall. In customer churn problem, precision and recall means:\n",
    "\n",
    "* Precision – Of all the users that the algorithm predicts will churn, how many of them do actually churn?\n",
    "* Recall – What percentage of users that end up churning does the algorithm successfully find?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_data[\"predicted_results\"] = pd.to_numeric(results)\n",
    "# define a threshold to convert probability to class, you can set as 0.5 by default\n",
    "test_data[\"predicted_binary\"] = [1 if x >= 0.5 else 0 for x in test_data[\"predicted_results\"]]\n",
    "test_data[[\"user_churned\", \"predicted_results\", \"predicted_binary\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You did a good job and your model can detect 85% of the users who are going to churn. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import metrics\n",
    "\n",
    "test_labels = test_data[\"user_churned\"]\n",
    "test_pred = test_data[\"predicted_binary\"]\n",
    "\n",
    "test_f1 = metrics.f1_score(test_labels, test_pred, average=None)\n",
    "# fbeta_test= metrics.f1_score(mtest_labels, mpreds_test_xgb, average=None)\n",
    "prec, rec, fbeta_test, support = metrics.precision_recall_fscore_support(\n",
    "    test_labels, test_pred, average=None\n",
    ")\n",
    "metrics.precision_recall_fscore_support(test_labels, test_pred, average=None)\n",
    "print(\"Test Evaluation: \")\n",
    "print(\"Average F1 Score: \", (test_f1[0] + test_f1[1]) / 2)\n",
    "print(\"Precision Score: \", (prec[1]))\n",
    "print(\"Recall Score: \", (rec[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Explainability with SageMaker Clarify\n",
    "\n",
    "You can visualize which feature contributes most to your prediction results by using the new SageMaker feature SageMaker Clarify. It  will provide SHAP values which measures the importance of a feature by replacing it with a dummy and seeing how it affects the prediciton. (In reality, SHAP is smart about the choice of dummy and also takes into account feature interactions.)  For a more general overview of model interpretability, see [this post](https://towardsdatascience.com/guide-to-interpretable-machine-learning-d40e8a64b6cf). For other capabilities of SageMaker Clarify, please see the [documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-fairness-and-explainability.html) and the [example notebook](https://github.com/aws/amazon-sagemaker-examples/blob/master/sagemaker-clarify/fairness_and_explainability/fairness_and_explainability.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker import clarify\n",
    "\n",
    "clarify_processor = clarify.SageMakerClarifyProcessor(\n",
    "    role=role, instance_count=1, instance_type=\"ml.c4.xlarge\", sagemaker_session=sagemaker_session\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.s3 import S3Uploader\n",
    "from sagemaker.inputs import TrainingInput\n",
    "\n",
    "train_uri = \"s3://{}/{}/train/train.csv\".format(bucket, prefix).format(bucket, prefix)\n",
    "train_input = TrainingInput(train_uri, content_type=\"csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# variables used for parameterizing the notebook run\n",
    "xgb_endpoint_name = smclient.list_endpoints()[\"Endpoints\"][0][\"EndpointName\"]\n",
    "xgb_model_name = smclient.list_models()[\"Models\"][0][\"ModelName\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "set(test_data.columns) - set(test_set.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shap_config = clarify.SHAPConfig(\n",
    "    baseline=[test_set.iloc[0].values.tolist()], num_samples=100, agg_method=\"mean_abs\"\n",
    ")\n",
    "\n",
    "explainability_output_path = \"s3://{}/{}/clarify-explainability\".format(bucket, prefix)\n",
    "\n",
    "explainability_data_config = clarify.DataConfig(\n",
    "    s3_data_input_path=train_uri,\n",
    "    s3_output_path=explainability_output_path,\n",
    "    label=\"user_churned\",\n",
    "    headers=test_data.drop([\"predicted_results\", \"predicted_binary\"], axis=1).columns.to_list(),\n",
    "    dataset_type=\"text/csv\",\n",
    ")\n",
    "\n",
    "model_config = clarify.ModelConfig(\n",
    "    model_name=xgb_model_name,\n",
    "    instance_type=\"ml.m4.xlarge\",\n",
    "    instance_count=1,\n",
    "    accept_type=\"text/csv\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "clarify_processor.run_explainability(\n",
    "    data_config=explainability_data_config,\n",
    "    model_config=model_config,\n",
    "    explainability_config=shap_config,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optional: Automate your training and model selection with SageMaker Autopilot (Console)\n",
    "\n",
    "With [SageMaker Autopilot](https://aws.amazon.com/blogs/aws/amazon-sagemaker-autopilot-fully-managed-automatic-machine-learning/), you can skip all the steps above and let it automatically tracks the inputs, parameters, configurations, and results of your iterations as trials. Go to SageMaker Experiments List on the left navigation pane, then choose **Create Experiment**. You will be directed to the experiment creating page. All you need to do is do give the Experiment job a name, specify your input and output data location, specify your target variable, and choose your ML problem type (classification or regression), or leave it as auto.\n",
    "\n",
    "<div>\n",
    "<img src=\"image/sm1.PNG\" width=\"400\"/>\n",
    "   </div>\n",
    "<div>\n",
    "<img src=\"image/sm2.PNG\" width=\"400\"/>\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data = pd.read_csv(\"data/train_w_header.csv\")\n",
    "validation_data = pd.read_csv(\"data/validation_w_header.csv\")\n",
    "\n",
    "data_for_experiment = pd.concat([train_data, validation_data])\n",
    "data_for_experiment.to_csv(\"full_feature_data.csv\", index=False)\n",
    "s3_input_full_set = (\n",
    "    boto3.Session()\n",
    "    .resource(\"s3\")\n",
    "    .Bucket(bucket)\n",
    "    .Object(os.path.join(prefix, \"full/fullset.csv\"))\n",
    "    .upload_file(\"full_feature_data.csv\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The experiment job will take some time to run (for the default 250 trials). It will go through data exploration, feature engineering, model selection and hyperparameter tuning. It will create a **data exploration notebook** that describes the data (missing values, numerical feature distributions, etc.), and a **candidate generation notebook** that describes the AutoML job. At the end of the Experiment job, the best model chosen is highlighted, and you can directly deploy the model for real-time prediction from the SageMaker Experiment Console. \n",
    "<div>\n",
    "<img src=\"image/sm3.PNG\" width=\"800\"/>\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optional: deploy model created from SageMaker Autopilot\n",
    "\n",
    "#### From Console\n",
    "\n",
    "You can navigate to the SageMaker Hyperparameter Tuning Job from the console to find the best model by go to **SageMaker  $\\rightarrow$ Hyperparameter tuning jobs  $\\rightarrow$ \\<your most recent job name marked as complete\\> $\\rightarrow$ Best training job** then choose **Create model**.\n",
    "<div>\n",
    "<img src=\"image/sm4.PNG\" width=\"800\"/>\n",
    "\n",
    "</div>\n",
    "\n",
    "#### With Code\n",
    "\n",
    "Alternatively, you can take look at the candidate generation notebook that describes the AutoML job. As part of the job, the input dataset has been randomly split into two pieces, one for training and one for validation. The notebook helps you inspect and modify the data transformation approaches proposed by Amazon SageMaker Autopilot. You can interactively train the data transformation models and use them to transform the data. Finally, you can execute a multiple algorithm hyperparameter optimization (multi-algo HPO) job that helps you find the best model for your dataset by jointly optimizing the data transformations and machine learning algorithms. Note that with the data transformation pipeline created by the AutoML job, the final model may contain more features than what you already created in this notebook, hence it would be better to test the models created by the AutoML job in the **Candidate generation notebook** to make sure your test data is also processed by the data transformation pipeline and has all the feature needed.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Disclaimer\n",
    "\n",
    "The data used in this notebook is synthetic and does not contain real user data. The results (all the names, emails, IP addresses, and browser information) of this simulation are fake."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Citation\n",
    "\n",
    "The data used in this notebook is simulated using the [EventSim](https://github.com/Interana/eventsim)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-east-1/use-cases|customer_churn|2_cust_churn_train_deploy_infer.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-east-2/use-cases|customer_churn|2_cust_churn_train_deploy_infer.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-west-1/use-cases|customer_churn|2_cust_churn_train_deploy_infer.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ca-central-1/use-cases|customer_churn|2_cust_churn_train_deploy_infer.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/sa-east-1/use-cases|customer_churn|2_cust_churn_train_deploy_infer.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-west-1/use-cases|customer_churn|2_cust_churn_train_deploy_infer.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-west-2/use-cases|customer_churn|2_cust_churn_train_deploy_infer.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-west-3/use-cases|customer_churn|2_cust_churn_train_deploy_infer.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-central-1/use-cases|customer_churn|2_cust_churn_train_deploy_infer.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-north-1/use-cases|customer_churn|2_cust_churn_train_deploy_infer.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-southeast-1/use-cases|customer_churn|2_cust_churn_train_deploy_infer.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-southeast-2/use-cases|customer_churn|2_cust_churn_train_deploy_infer.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-northeast-1/use-cases|customer_churn|2_cust_churn_train_deploy_infer.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-northeast-2/use-cases|customer_churn|2_cust_churn_train_deploy_infer.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-south-1/use-cases|customer_churn|2_cust_churn_train_deploy_infer.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   }
  ],
  "kernelspec": {
   "display_name": "Python 3 (Data Science 3.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/sagemaker-data-science-310-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}