{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Amazon SageMaker Autopilot Candidate Definition Notebook\n", "\n", "This notebook was automatically generated by the AutoML job **mangrove-2013**.\n", "This notebook allows you to customize the candidate definitions and execute the SageMaker Autopilot workflow.\n", "\n", "The dataset has **8** columns and the column named **label** is used as\n", "the target column. This is being treated as a **BinaryClassification** problem. The dataset also has **2** classes.\n", "This notebook will build a **[BinaryClassification](https://en.wikipedia.org/wiki/Binary_classification)** model that\n", "**maximizes** the \"**AUC**\" quality metric of the trained models.\n", "The \"**AUC**\" metric applies to binary classification with a positive and negative class. It calculates the area under the ROC curve which is created by plotting the true positive rate against the false positive rate..\n", "\n", "As part of the AutoML job, the input dataset has been randomly split into two pieces, one for **training** and one for\n", "**validation**. This notebook helps you inspect and modify the data transformation approaches proposed by Amazon SageMaker Autopilot. You can interactively\n", "train the data transformation models and use them to transform the data. Finally, you can execute a multiple algorithm hyperparameter optimization (multi-algo HPO)\n", "job that helps you find the best model for your dataset by jointly optimizing the data transformations and machine learning algorithms.\n", "\n", "
💡 Available Knobs\n", "Look for sections like this for recommended settings that you can change.\n", "
\n", "\n", "\n", "---\n", "\n", "## Contents\n", "\n", "1. [Sagemaker Setup](#Sagemaker-Setup)\n", " 1. [Downloading Generated Candidates](#Downloading-Generated-Modules)\n", " 1. [SageMaker Autopilot Job and Amazon Simple Storage Service (Amazon S3) Configuration](#SageMaker-Autopilot-Job-and-Amazon-Simple-Storage-Service-(Amazon-S3)-Configuration)\n", "1. [Candidate Pipelines](#Candidate-Pipelines)\n", " 1. [Generated Candidates](#Generated-Candidates)\n", " 1. [Selected Candidates](#Selected-Candidates)\n", "1. [Executing the Candidate Pipelines](#Executing-the-Candidate-Pipelines)\n", " 1. [Run Data Transformation Steps](#Run-Data-Transformation-Steps)\n", " 1. [Multi Algorithm Hyperparameter Tuning](#Multi-Algorithm-Hyperparameter-Tuning)\n", "1. [Model Selection and Deployment](#Model-Selection-and-Deployment)\n", " 1. [Tuning Job Result Overview](#Tuning-Job-Result-Overview)\n", " 1. [Model Deployment](#Model-Deployment)\n", "\n", "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Sagemaker Setup\n", "\n", "Before you launch the SageMaker Autopilot jobs, we'll setup the environment for Amazon SageMaker\n", "- Check environment & dependencies.\n", "- Create a few helper objects/function to organize input/output data and SageMaker sessions." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Minimal Environment Requirements**\n", "\n", "- Jupyter: Tested on `JupyterLab 1.0.6`, `jupyter_core 4.5.0` and `IPython 6.4.0`\n", "- Kernel: `conda_python3`\n", "- Dependencies required\n", " - `sagemaker-python-sdk>=2.40.0`\n", " - Use `!pip install sagemaker==2.40.0` to download this dependency.\n", " - Kernel may need to be restarted after download.\n", "- Expected Execution Role/permission\n", " - S3 access to the bucket that stores the notebook." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Downloading Generated Modules\n", "Download the generated data transformation modules and an SageMaker Autopilot helper module used by this notebook.\n", "Those artifacts will be downloaded to **mangrove-2013-artifacts** folder." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "!mkdir -p mangrove-2013-artifacts\n", "!aws s3 sync s3://sagemaker-gis/LANDSAT_LC08_C01_T1_SR/Year2013/mangrove-2013/sagemaker-automl-candidates/mangrove-2013-pr-1-3fbd4f34dd314d0dbd0bf3ad17f37ce4f4b731e72ed2/generated_module mangrove-2013-artifacts/generated_module --only-show-errors\n", "!aws s3 sync s3://sagemaker-gis/LANDSAT_LC08_C01_T1_SR/Year2013/mangrove-2013/sagemaker-automl-candidates/mangrove-2013-pr-1-3fbd4f34dd314d0dbd0bf3ad17f37ce4f4b731e72ed2/notebooks/sagemaker_automl mangrove-2013-artifacts/sagemaker_automl --only-show-errors\n", "\n", "import sys\n", "sys.path.append(\"mangrove-2013-artifacts\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### SageMaker Autopilot Job and Amazon Simple Storage Service (Amazon S3) Configuration\n", "\n", "The following configuration has been derived from the SageMaker Autopilot job. These items configure where this notebook will\n", "look for generated candidates, and where input and output data is stored on Amazon S3." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "This notebook is initialized to use the following configuration: \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
NameValue
GeneralRolearn:aws:iam::766126314930:role/sm-service-access
Base AutoML JobJob Namemangrove-2013
Base Output S3 Paths3://sagemaker-gis/LANDSAT_LC08_C01_T1_SR/Year2013/mangrove-2013
Interactive JobJob Namemangrove-2-notebook-run-21-16-54-33
Base Output S3 Paths3://sagemaker-gis/LANDSAT_LC08_C01_T1_SR/Year2013/mangrove-2013/mangrove-2-notebook-run-21-16-54-33
Data Processing Trained Model Directorys3://sagemaker-gis/LANDSAT_LC08_C01_T1_SR/Year2013/mangrove-2013/mangrove-2-notebook-run-21-16-54-33/data-processor-models
Data Processing Transformed Outputs3://sagemaker-gis/LANDSAT_LC08_C01_T1_SR/Year2013/mangrove-2013/mangrove-2-notebook-run-21-16-54-33/transformed-data
Algo Tuning Model Output Directorys3://sagemaker-gis/LANDSAT_LC08_C01_T1_SR/Year2013/mangrove-2013/mangrove-2-notebook-run-21-16-54-33/multi-algo-tuning
\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from sagemaker_automl import uid, AutoMLLocalRunConfig\n", "\n", "# Where the preprocessed data from the existing AutoML job is stored\n", "BASE_AUTOML_JOB_NAME = 'mangrove-2013'\n", "BASE_AUTOML_JOB_CONFIG = {\n", " 'automl_job_name': BASE_AUTOML_JOB_NAME,\n", " 'automl_output_s3_base_path': 's3://sagemaker-gis/LANDSAT_LC08_C01_T1_SR/Year2013/mangrove-2013',\n", " 'data_transformer_image_repo_version': '2.4-1-cpu-py3',\n", " 'algo_image_repo_versions': {'xgboost': '1.2-2-cpu-py3', 'linear-learner': 'latest', 'mlp': 'training-cpu'},\n", " 'algo_inference_image_repo_versions': {'xgboost': '1.2-2-cpu-py3', 'linear-learner': 'latest', 'mlp': 'inference-cpu'}\n", "}\n", "\n", "# Path conventions of the output data storage path from the local AutoML job run of this notebook\n", "LOCAL_AUTOML_JOB_NAME = 'mangrove-2-notebook-run-{}'.format(uid())\n", "LOCAL_AUTOML_JOB_CONFIG = {\n", " 'local_automl_job_name': LOCAL_AUTOML_JOB_NAME,\n", " 'local_automl_job_output_s3_base_path': 's3://sagemaker-gis/LANDSAT_LC08_C01_T1_SR/Year2013/mangrove-2013/{}'.format(LOCAL_AUTOML_JOB_NAME),\n", " 'data_processing_model_dir': 'data-processor-models',\n", " 'data_processing_transformed_output_dir': 'transformed-data',\n", " 'multi_algo_tuning_output_dir': 'multi-algo-tuning'\n", "}\n", "\n", "AUTOML_LOCAL_RUN_CONFIG = AutoMLLocalRunConfig(\n", " role='arn:aws:iam::766126314930:role/sm-service-access',\n", " base_automl_job_config=BASE_AUTOML_JOB_CONFIG,\n", " local_automl_job_config=LOCAL_AUTOML_JOB_CONFIG,\n", " security_config={'EnableInterContainerTrafficEncryption': False, 'VpcConfig': {}})\n", "\n", "AUTOML_LOCAL_RUN_CONFIG.display()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Candidate Pipelines\n", "\n", "The `AutoMLLocalRunner` keeps track of selected candidates and automates many of the steps needed to execute feature engineering and tuning steps." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "from sagemaker_automl import AutoMLInteractiveRunner, AutoMLLocalCandidate\n", "\n", "automl_interactive_runner = AutoMLInteractiveRunner(AUTOML_LOCAL_RUN_CONFIG)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Generated Candidates\n", "\n", "The SageMaker Autopilot Job has analyzed the dataset and has generated **7** machine learning\n", "pipeline(s) that use **3** algorithm(s). Each pipeline contains a set of feature transformers and an\n", "algorithm.\n", "\n", "
💡 Available Knobs\n", "\n", "1. The resource configuration: instance type & count\n", "1. Select candidate pipeline definitions by cells\n", "1. The linked data transformation script can be reviewed and updated. Please refer to the [README.md](./mangrove-2013-artifacts/generated_module/README.md) for detailed customization instructions.\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**[dpp0-xgboost](mangrove-2013-artifacts/generated_module/candidate_data_processors/dpp0.py)**: This data transformation strategy first transforms 'numeric' features using [RobustImputer (converts missing values to nan)](https://github.com/aws/sagemaker-scikit-learn-extension/blob/master/src/sagemaker_sklearn_extension/impute/base.py). It merges all the generated features and applies [RobustStandardScaler](https://github.com/aws/sagemaker-scikit-learn-extension/blob/master/src/sagemaker_sklearn_extension/preprocessing/data.py). The\n", "transformed data will be used to tune a *xgboost* model. Here is the definition:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2021-10-21 16:54:46,582 INFO sagemaker.image_uris: Same images used for training and inference. Defaulting to image scope: inference.\n", "2021-10-21 16:54:46,604 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n", "2021-10-21 16:54:46,606 INFO sagemaker.image_uris: Same images used for training and inference. Defaulting to image scope: inference.\n", "2021-10-21 16:54:46,674 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n", "2021-10-21 16:54:46,675 INFO sagemaker.image_uris: Same images used for training and inference. Defaulting to image scope: inference.\n", "2021-10-21 16:54:46,693 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n" ] } ], "source": [ "automl_interactive_runner.select_candidate({\n", " \"data_transformer\": {\n", " \"name\": \"dpp0\",\n", " \"training_resource_config\": {\n", " \"instance_type\": \"ml.m5.4xlarge\",\n", " \"instance_count\": 1,\n", " \"volume_size_in_gb\": 50\n", " },\n", " \"transform_resource_config\": {\n", " \"instance_type\": \"ml.m5.4xlarge\",\n", " \"instance_count\": 1,\n", " },\n", " \"transforms_label\": True,\n", " \"transformed_data_format\": \"text/csv\",\n", " \"sparse_encoding\": False\n", " },\n", " \"algorithm\": {\n", " \"name\": \"xgboost\",\n", " \"training_resource_config\": {\n", " \"instance_type\": \"ml.m5.4xlarge\",\n", " \"instance_count\": 1,\n", " },\n", " }\n", "})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**[dpp1-xgboost](mangrove-2013-artifacts/generated_module/candidate_data_processors/dpp1.py)**: This data transformation strategy first transforms 'numeric' features using [RobustImputer](https://github.com/aws/sagemaker-scikit-learn-extension/blob/master/src/sagemaker_sklearn_extension/impute/base.py). It merges all the generated features and applies [RobustPCA](https://github.com/aws/sagemaker-scikit-learn-extension/blob/master/src/sagemaker_sklearn_extension/decomposition/robust_pca.py) followed by [RobustStandardScaler](https://github.com/aws/sagemaker-scikit-learn-extension/blob/master/src/sagemaker_sklearn_extension/preprocessing/data.py). The\n", "transformed data will be used to tune a *xgboost* model. Here is the definition:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2021-10-21 16:54:48,014 INFO sagemaker.image_uris: Same images used for training and inference. Defaulting to image scope: inference.\n", "2021-10-21 16:54:48,044 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n", "2021-10-21 16:54:48,046 INFO sagemaker.image_uris: Same images used for training and inference. Defaulting to image scope: inference.\n", "2021-10-21 16:54:48,061 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n", "2021-10-21 16:54:48,063 INFO sagemaker.image_uris: Same images used for training and inference. Defaulting to image scope: inference.\n", "2021-10-21 16:54:48,082 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n" ] } ], "source": [ "automl_interactive_runner.select_candidate({\n", " \"data_transformer\": {\n", " \"name\": \"dpp1\",\n", " \"training_resource_config\": {\n", " \"instance_type\": \"ml.m5.4xlarge\",\n", " \"instance_count\": 1,\n", " \"volume_size_in_gb\": 50\n", " },\n", " \"transform_resource_config\": {\n", " \"instance_type\": \"ml.m5.4xlarge\",\n", " \"instance_count\": 1,\n", " },\n", " \"transforms_label\": True,\n", " \"transformed_data_format\": \"text/csv\",\n", " \"sparse_encoding\": False\n", " },\n", " \"algorithm\": {\n", " \"name\": \"xgboost\",\n", " \"training_resource_config\": {\n", " \"instance_type\": \"ml.m5.4xlarge\",\n", " \"instance_count\": 1,\n", " },\n", " }\n", "})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**[dpp2-linear-learner](mangrove-2013-artifacts/generated_module/candidate_data_processors/dpp2.py)**: This data transformation strategy first transforms 'numeric' features using [combined RobustImputer and RobustMissingIndicator](https://github.com/aws/sagemaker-scikit-learn-extension/blob/master/src/sagemaker_sklearn_extension/impute/base.py) followed by [QuantileExtremeValuesTransformer](https://github.com/aws/sagemaker-scikit-learn-extension/blob/master/src/sagemaker_sklearn_extension/preprocessing/base.py), 'categorical' features using [ThresholdOneHotEncoder](https://github.com/aws/sagemaker-scikit-learn-extension/blob/master/src/sagemaker_sklearn_extension/preprocessing/encoders.py). It merges all the generated features and applies [RobustPCA](https://github.com/aws/sagemaker-scikit-learn-extension/blob/master/src/sagemaker_sklearn_extension/decomposition/robust_pca.py) followed by [RobustStandardScaler](https://github.com/aws/sagemaker-scikit-learn-extension/blob/master/src/sagemaker_sklearn_extension/preprocessing/data.py). The\n", "transformed data will be used to tune a *linear-learner* model. Here is the definition:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2021-10-21 16:54:49,463 INFO sagemaker.image_uris: Same images used for training and inference. Defaulting to image scope: inference.\n", "2021-10-21 16:54:49,479 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n", "2021-10-21 16:54:49,480 INFO sagemaker.image_uris: Same images used for training and inference. Defaulting to image scope: inference.\n", "2021-10-21 16:54:49,482 WARNING sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: 1. Ignoring framework/algorithm version: latest.\n", "2021-10-21 16:54:49,496 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n", "2021-10-21 16:54:49,497 INFO sagemaker.image_uris: Same images used for training and inference. Defaulting to image scope: inference.\n", "2021-10-21 16:54:49,498 WARNING sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: 1. Ignoring framework/algorithm version: latest.\n", "2021-10-21 16:54:49,515 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n" ] } ], "source": [ "automl_interactive_runner.select_candidate({\n", " \"data_transformer\": {\n", " \"name\": \"dpp2\",\n", " \"training_resource_config\": {\n", " \"instance_type\": \"ml.m5.4xlarge\",\n", " \"instance_count\": 1,\n", " \"volume_size_in_gb\": 50\n", " },\n", " \"transform_resource_config\": {\n", " \"instance_type\": \"ml.m5.4xlarge\",\n", " \"instance_count\": 1,\n", " },\n", " \"transforms_label\": True,\n", " \"transformed_data_format\": \"application/x-recordio-protobuf\",\n", " \"sparse_encoding\": False\n", " },\n", " \"algorithm\": {\n", " \"name\": \"linear-learner\",\n", " \"training_resource_config\": {\n", " \"instance_type\": \"ml.m5.4xlarge\",\n", " \"instance_count\": 1,\n", " },\n", " }\n", "})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**[dpp3-xgboost](mangrove-2013-artifacts/generated_module/candidate_data_processors/dpp3.py)**: This data transformation strategy first transforms 'numeric' features using [RobustImputer (converts missing values to nan)](https://github.com/aws/sagemaker-scikit-learn-extension/blob/master/src/sagemaker_sklearn_extension/impute/base.py), 'categorical' features using [ThresholdOneHotEncoder](https://github.com/aws/sagemaker-scikit-learn-extension/blob/master/src/sagemaker_sklearn_extension/preprocessing/encoders.py). It merges all the generated features and applies [RobustStandardScaler](https://github.com/aws/sagemaker-scikit-learn-extension/blob/master/src/sagemaker_sklearn_extension/preprocessing/data.py). The\n", "transformed data will be used to tune a *xgboost* model. Here is the definition:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2021-10-21 16:54:50,991 INFO sagemaker.image_uris: Same images used for training and inference. Defaulting to image scope: inference.\n", "2021-10-21 16:54:51,011 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n", "2021-10-21 16:54:51,012 INFO sagemaker.image_uris: Same images used for training and inference. Defaulting to image scope: inference.\n", "2021-10-21 16:54:51,023 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n", "2021-10-21 16:54:51,024 INFO sagemaker.image_uris: Same images used for training and inference. Defaulting to image scope: inference.\n", "2021-10-21 16:54:51,035 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n" ] } ], "source": [ "automl_interactive_runner.select_candidate({\n", " \"data_transformer\": {\n", " \"name\": \"dpp3\",\n", " \"training_resource_config\": {\n", " \"instance_type\": \"ml.m5.4xlarge\",\n", " \"instance_count\": 1,\n", " \"volume_size_in_gb\": 50\n", " },\n", " \"transform_resource_config\": {\n", " \"instance_type\": \"ml.m5.4xlarge\",\n", " \"instance_count\": 1,\n", " },\n", " \"transforms_label\": True,\n", " \"transformed_data_format\": \"application/x-recordio-protobuf\",\n", " \"sparse_encoding\": True\n", " },\n", " \"algorithm\": {\n", " \"name\": \"xgboost\",\n", " \"training_resource_config\": {\n", " \"instance_type\": \"ml.m5.4xlarge\",\n", " \"instance_count\": 1,\n", " },\n", " }\n", "})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**[dpp4-xgboost](mangrove-2013-artifacts/generated_module/candidate_data_processors/dpp4.py)**: This data transformation strategy first transforms 'numeric' features using [RobustImputer (converts missing values to nan)](https://github.com/aws/sagemaker-scikit-learn-extension/blob/master/src/sagemaker_sklearn_extension/impute/base.py), 'categorical' features using [ThresholdOneHotEncoder](https://github.com/aws/sagemaker-scikit-learn-extension/blob/master/src/sagemaker_sklearn_extension/preprocessing/encoders.py). It merges all the generated features and applies [RobustStandardScaler](https://github.com/aws/sagemaker-scikit-learn-extension/blob/master/src/sagemaker_sklearn_extension/preprocessing/data.py). The\n", "transformed data will be used to tune a *xgboost* model. Here is the definition:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2021-10-21 16:54:52,984 INFO sagemaker.image_uris: Same images used for training and inference. Defaulting to image scope: inference.\n", "2021-10-21 16:54:53,000 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n", "2021-10-21 16:54:53,003 INFO sagemaker.image_uris: Same images used for training and inference. Defaulting to image scope: inference.\n", "2021-10-21 16:54:53,018 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n", "2021-10-21 16:54:53,020 INFO sagemaker.image_uris: Same images used for training and inference. Defaulting to image scope: inference.\n", "2021-10-21 16:54:53,035 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n" ] } ], "source": [ "automl_interactive_runner.select_candidate({\n", " \"data_transformer\": {\n", " \"name\": \"dpp4\",\n", " \"training_resource_config\": {\n", " \"instance_type\": \"ml.m5.4xlarge\",\n", " \"instance_count\": 1,\n", " \"volume_size_in_gb\": 50\n", " },\n", " \"transform_resource_config\": {\n", " \"instance_type\": \"ml.m5.4xlarge\",\n", " \"instance_count\": 1,\n", " },\n", " \"transforms_label\": True,\n", " \"transformed_data_format\": \"application/x-recordio-protobuf\",\n", " \"sparse_encoding\": True\n", " },\n", " \"algorithm\": {\n", " \"name\": \"xgboost\",\n", " \"training_resource_config\": {\n", " \"instance_type\": \"ml.m5.4xlarge\",\n", " \"instance_count\": 1,\n", " },\n", " }\n", "})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**[dpp5-xgboost](mangrove-2013-artifacts/generated_module/candidate_data_processors/dpp5.py)**: This data transformation strategy first transforms 'numeric' features using [RobustImputer (converts missing values to nan)](https://github.com/aws/sagemaker-scikit-learn-extension/blob/master/src/sagemaker_sklearn_extension/impute/base.py), 'categorical' features using [ThresholdOneHotEncoder](https://github.com/aws/sagemaker-scikit-learn-extension/blob/master/src/sagemaker_sklearn_extension/preprocessing/encoders.py). It merges all the generated features and applies [RobustStandardScaler](https://github.com/aws/sagemaker-scikit-learn-extension/blob/master/src/sagemaker_sklearn_extension/preprocessing/data.py). The\n", "transformed data will be used to tune a *xgboost* model. Here is the definition:" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2021-10-21 16:54:54,315 INFO sagemaker.image_uris: Same images used for training and inference. Defaulting to image scope: inference.\n", "2021-10-21 16:54:54,330 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n", "2021-10-21 16:54:54,331 INFO sagemaker.image_uris: Same images used for training and inference. Defaulting to image scope: inference.\n", "2021-10-21 16:54:54,343 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n", "2021-10-21 16:54:54,344 INFO sagemaker.image_uris: Same images used for training and inference. Defaulting to image scope: inference.\n", "2021-10-21 16:54:54,425 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n" ] } ], "source": [ "automl_interactive_runner.select_candidate({\n", " \"data_transformer\": {\n", " \"name\": \"dpp5\",\n", " \"training_resource_config\": {\n", " \"instance_type\": \"ml.m5.4xlarge\",\n", " \"instance_count\": 1,\n", " \"volume_size_in_gb\": 50\n", " },\n", " \"transform_resource_config\": {\n", " \"instance_type\": \"ml.m5.4xlarge\",\n", " \"instance_count\": 1,\n", " },\n", " \"transforms_label\": True,\n", " \"transformed_data_format\": \"application/x-recordio-protobuf\",\n", " \"sparse_encoding\": True\n", " },\n", " \"algorithm\": {\n", " \"name\": \"xgboost\",\n", " \"training_resource_config\": {\n", " \"instance_type\": \"ml.m5.4xlarge\",\n", " \"instance_count\": 1,\n", " },\n", " }\n", "})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**[dpp6-mlp](mangrove-2013-artifacts/generated_module/candidate_data_processors/dpp6.py)**: This data transformation strategy transforms 'numeric' features using [RobustImputer](https://github.com/aws/sagemaker-scikit-learn-extension/blob/master/src/sagemaker_sklearn_extension/impute/base.py) followed by [RobustStandardScaler](https://github.com/aws/sagemaker-scikit-learn-extension/blob/master/src/sagemaker_sklearn_extension/preprocessing/data.py). The\n", "transformed data will be used to tune a *mlp* model. Here is the definition:" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2021-10-21 16:54:55,831 INFO sagemaker.image_uris: Same images used for training and inference. Defaulting to image scope: inference.\n", "2021-10-21 16:54:55,849 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n", "2021-10-21 16:54:55,850 INFO sagemaker.image_uris: Same images used for training and inference. Defaulting to image scope: inference.\n", "2021-10-21 16:54:55,852 WARNING sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: 1. Ignoring framework/algorithm version: training-cpu.\n", "2021-10-21 16:54:55,862 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n", "2021-10-21 16:54:55,864 INFO sagemaker.image_uris: Same images used for training and inference. Defaulting to image scope: inference.\n", "2021-10-21 16:54:55,865 WARNING sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: 1. Ignoring framework/algorithm version: inference-cpu.\n", "2021-10-21 16:54:55,877 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n" ] } ], "source": [ "automl_interactive_runner.select_candidate({\n", " \"data_transformer\": {\n", " \"name\": \"dpp6\",\n", " \"training_resource_config\": {\n", " \"instance_type\": \"ml.m5.4xlarge\",\n", " \"instance_count\": 1,\n", " \"volume_size_in_gb\": 50\n", " },\n", " \"transform_resource_config\": {\n", " \"instance_type\": \"ml.m5.4xlarge\",\n", " \"instance_count\": 1,\n", " },\n", " \"transforms_label\": True,\n", " \"transformed_data_format\": \"text/csv\",\n", " \"sparse_encoding\": False\n", " },\n", " \"algorithm\": {\n", " \"name\": \"mlp\",\n", " \"training_resource_config\": {\n", " \"instance_type\": \"ml.m5.4xlarge\",\n", " \"instance_count\": 1,\n", " },\n", " \"candidate_specific_static_hyperparameters\": {\n", " \"num_categorical_features\": '0',\n", " }\n", " }\n", "})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Selected Candidates\n", "\n", "You have selected the following candidates (please run the cell below and click on the feature transformer links for details):" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "
Candidate NameAlgorithmFeature Transformer
dpp0-xgboostxgboostdpp0.py
dpp1-xgboostxgboostdpp1.py
dpp2-linear-learnerlinear-learnerdpp2.py
dpp3-xgboostxgboostdpp3.py
dpp4-xgboostxgboostdpp4.py
dpp5-xgboostxgboostdpp5.py
dpp6-mlpmlpdpp6.py
\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "automl_interactive_runner.display_candidates()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The feature engineering pipeline consists of two SageMaker jobs:\n", "\n", "1. Generated trainable data transformer Python modules like [dpp0.py](mangrove-2013-artifacts/generated_module/candidate_data_processors/dpp0.py), which has been downloaded to the local file system\n", "2. A **training** job to train the data transformers\n", "3. A **batch transform** job to apply the trained transformation to the dataset to generate the algorithm compatible data\n", "\n", "The transformers and its training pipeline are built using open sourced **[sagemaker-scikit-learn-container][]** and **[sagemaker-scikit-learn-extension][]**.\n", "\n", "[sagemaker-scikit-learn-container]: https://github.com/aws/sagemaker-scikit-learn-container\n", "[sagemaker-scikit-learn-extension]: https://github.com/aws/sagemaker-scikit-learn-extension" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Executing the Candidate Pipelines\n", "\n", "Each candidate pipeline consists of two steps, feature transformation and algorithm training.\n", "For efficiency first execute the feature transformation step which will generate a featurized dataset on S3\n", "for each pipeline.\n", "\n", "After each featurized dataset is prepared, execute a multi-algorithm tuning job that will run tuning jobs\n", "in parallel for each pipeline. This tuning job will execute training jobs to find the best set of\n", "hyper-parameters for each pipeline, as well as finding the overall best performing pipeline.\n", "\n", "### Run Data Transformation Steps\n", "\n", "Now you are ready to start execution all data transformation steps. The cell below may take some time to finish,\n", "feel free to go grab a cup of coffee. To expedite the process you can set the number of `parallel_jobs` to be up to 10.\n", "Please check the account limits to increase the limits before increasing the number of jobs to run in parallel." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2021-10-21 17:28:33,715 INFO root: [Worker_3:dpp3-xgboost]Executing step: train_data_transformer\n", "2021-10-21 17:28:33,717 INFO sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: latest.\n", "2021-10-21 17:28:33,722 INFO root: [Worker_4:dpp4-xgboost]Executing step: train_data_transformer\n", "2021-10-21 17:28:33,727 INFO sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: latest.\n", "2021-10-21 17:28:33,870 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n", "2021-10-21 17:28:33,871 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n", "2021-10-21 17:28:34,005 INFO sagemaker: Creating training-job with name: mangrove-2-notebook-run-21-16-54-33-dpp4-train-21-17-28-31\n", "2021-10-21 17:28:34,009 INFO sagemaker: Creating training-job with name: mangrove-2-notebook-run-21-16-54-33-dpp3-train-21-17-28-31\n", "\n", "2021-10-21 17:28:34 Starting - Starting the training job\n", "2021-10-21 17:28:34 Starting - Starting the training job2021-10-21 17:28:34,716 INFO root: [Worker_1:dpp1-xgboost]Executing step: train_data_transformer\n", "2021-10-21 17:28:34,717 INFO sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: latest.\n", "2021-10-21 17:28:34,729 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n", "2021-10-21 17:28:34,821 INFO sagemaker: Creating training-job with name: mangrove-2-notebook-run-21-16-54-33-dpp1-train-21-17-28-31\n", "2021-10-21 17:28:35,723 INFO root: [Worker_5:dpp5-xgboost]Executing step: train_data_transformer\n", "2021-10-21 17:28:35,724 INFO sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: latest.\n", "2021-10-21 17:28:35,736 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n", "2021-10-21 17:28:35,837 INFO sagemaker: Creating training-job with name: mangrove-2-notebook-run-21-16-54-33-dpp5-train-21-17-28-31\n", "2021-10-21 17:28:36,368 INFO root: We are getting throttled, retrying in 1s\n", "2021-10-21 17:28:37,369 INFO sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: latest.\n", "2021-10-21 17:28:37,382 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n", "2021-10-21 17:28:37,487 INFO sagemaker: Creating training-job with name: mangrove-2-notebook-run-21-16-54-33-dpp1-train-21-17-28-31\n", "2021-10-21 17:28:37,726 INFO root: [Worker_6:dpp6-mlp]Executing step: train_data_transformer\n", "2021-10-21 17:28:37,727 INFO sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: latest.\n", "2021-10-21 17:28:37,741 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n", "2021-10-21 17:28:37,863 INFO sagemaker: Creating training-job with name: mangrove-2-notebook-run-21-16-54-33-dpp6-train-21-17-28-31\n", "\n", "2021-10-21 17:28:38 Starting - Starting the training job\n", "2021-10-21 17:28:36 Starting - Launching requested ML instances\n", "2021-10-21 17:28:36 Starting - Launching requested ML instances\n", "2021-10-21 17:28:39 Starting - Starting the training job2021-10-21 17:28:40,715 INFO root: [Worker_2:dpp2-linear-learner]Executing step: train_data_transformer\n", "2021-10-21 17:28:40,716 INFO sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: latest.\n", "2021-10-21 17:28:40,730 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n", "2021-10-21 17:28:40,836 INFO sagemaker: Creating training-job with name: mangrove-2-notebook-run-21-16-54-33-dpp2-train-21-17-28-31\n", "\n", "2021-10-21 17:28:40 Starting - Starting the training job2021-10-21 17:28:41,715 INFO root: [Worker_0:dpp0-xgboost]Executing step: train_data_transformer\n", "2021-10-21 17:28:41,717 INFO sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: latest.\n", "2021-10-21 17:28:41,730 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n", "2021-10-21 17:28:41,853 INFO sagemaker: Creating training-job with name: mangrove-2-notebook-run-21-16-54-33-dpp0-train-21-17-28-31\n", "\n", "2021-10-21 17:28:42 Starting - Starting the training job\n", "2021-10-21 17:28:40 Starting - Launching requested ML instances\n", "2021-10-21 17:28:43 Starting - Starting the training job..\n", "2021-10-21 17:28:41 Starting - Launching requested ML instances\n", "2021-10-21 17:28:42 Starting - Launching requested ML instances\n", "2021-10-21 17:28:44 Starting - Launching requested ML instances.\n", "2021-10-21 17:28:45 Starting - Launching requested ML instances.........................................................\n", "2021-10-21 17:29:29 Starting - Preparing the instances for training...\n", "2021-10-21 17:29:33 Starting - Preparing the instances for training...\n", "2021-10-21 17:29:35 Starting - Preparing the instances for training\n", "2021-10-21 17:29:35 Starting - Preparing the instances for training.......\n", "2021-10-21 17:29:38 Starting - Preparing the instances for training.........\n", "2021-10-21 17:29:45 Starting - Preparing the instances for training............\n", "2021-10-21 17:29:55 Starting - Preparing the instances for training........................................................\n", "2021-10-21 17:30:39 Downloading - Downloading input data...............\n", "2021-10-21 17:30:49 Downloading - Downloading input data....\n", "2021-10-21 17:30:50 Training - Downloading the training image\n", "2021-10-21 17:30:51 Downloading - Downloading input data\n", "2021-10-21 17:30:53 Training - Downloading the training image\n", "2021-10-21 17:30:55 Downloading - Downloading input data\n", "2021-10-21 17:30:56 Downloading - Downloading input data\n", "2021-10-21 17:30:57 Downloading - Downloading input data.......................\n", "2021-10-21 17:31:11 Training - Downloading the training image...\n", "2021-10-21 17:31:18 Training - Downloading the training image....\n", "2021-10-21 17:31:17 Training - Downloading the training image\n", "2021-10-21 17:31:20 Training - Downloading the training image.\n", "2021-10-21 17:31:21 Downloading - Downloading input data...............\n", "2021-10-21 17:31:30 Training - Training image download completed. Training in progress..\n", "2021-10-21 17:31:33 Training - Training image download completed. Training in progress.\n", "2021-10-21 17:31:35 Training - Training image download completed. Training in progress.\n", "2021-10-21 17:31:37 Training - Training image download completed. Training in progress..\n", "2021-10-21 17:31:39 Uploading - Uploading generated training model\n", "2021-10-21 17:31:38 Uploading - Uploading generated training model..\n", "2021-10-21 17:31:41 Uploading - Uploading generated training model\n", "2021-10-21 17:31:43 Uploading - Uploading generated training model..\n", "2021-10-21 17:31:43 Completed - Training job completed\n", "\n", "2021-10-21 17:31:44 Uploading - Uploading generated training model\n", "2021-10-21 17:31:43 Uploading - Uploading generated training model2021-10-21 17:31:47,403 INFO root: [Worker_3:dpp3-xgboost]Executing step: create_transformer_model\n", "2021-10-21 17:31:47,434 INFO sagemaker: Creating model with name: sagemaker-sklearn-automl-2021-10-21-17-31-47-404\n", "..2021-10-21 17:31:49,756 INFO root: [Worker_3:dpp3-xgboost]Executing step: perform_data_transform\n", "2021-10-21 17:31:49,761 INFO sagemaker: Creating transform job with name: mangrove-2-notebook-run-21-16-54-33-dpp3-transform-21-17-28-31\n", "\n", "2021-10-21 17:31:47 Training - Downloading the training image.\n", "2021-10-21 17:31:46 Completed - Training job completed\n", ".\n", "2021-10-21 17:31:51 Completed - Training job completed\n", "2021-10-21 17:31:52,361 INFO root: [Worker_4:dpp4-xgboost]Executing step: create_transformer_model\n", "2021-10-21 17:31:52,383 INFO sagemaker: Creating model with name: sagemaker-sklearn-automl-2021-10-21-17-31-52-361\n", "\n", "2021-10-21 17:31:49 Completed - Training job completed\n", "\n", "2021-10-21 17:31:50 Completed - Training job completed\n", "2021-10-21 17:31:54,713 INFO root: [Worker_4:dpp4-xgboost]Executing step: perform_data_transform\n", "2021-10-21 17:31:54,714 INFO sagemaker: Creating transform job with name: mangrove-2-notebook-run-21-16-54-33-dpp4-transform-21-17-28-31\n", "...\n", "2021-10-21 17:31:52 Completed - Training job completed\n", "2021-10-21 17:31:58,921 INFO root: [Worker_1:dpp1-xgboost]Executing step: create_transformer_model\n", "2021-10-21 17:31:58,947 INFO sagemaker: Creating model with name: sagemaker-sklearn-automl-2021-10-21-17-31-58-922\n", "...2021-10-21 17:32:00,606 INFO root: [Worker_6:dpp6-mlp]Executing step: create_transformer_model\n", "2021-10-21 17:32:00,629 INFO sagemaker: Creating model with name: sagemaker-sklearn-automl-2021-10-21-17-32-00-607\n", "2021-10-21 17:32:01,113 INFO root: [Worker_2:dpp2-linear-learner]Executing step: create_transformer_model\n", "2021-10-21 17:32:01,136 INFO sagemaker: Creating model with name: sagemaker-sklearn-automl-2021-10-21-17-32-01-114\n", "2021-10-21 17:32:02,291 INFO root: [Worker_1:dpp1-xgboost]Executing step: perform_data_transform\n", "2021-10-21 17:32:02,292 INFO sagemaker: Creating transform job with name: mangrove-2-notebook-run-21-16-54-33-dpp1-transform-21-17-28-31\n", ".2021-10-21 17:32:03,109 INFO root: [Worker_0:dpp0-xgboost]Executing step: create_transformer_model\n", "2021-10-21 17:32:03,135 INFO sagemaker: Creating model with name: sagemaker-sklearn-automl-2021-10-21-17-32-03-110\n", ".\n", "2021-10-21 17:32:03 Training - Training image download completed. Training in progress..2021-10-21 17:32:06,967 INFO root: [Worker_6:dpp6-mlp]Executing step: perform_data_transform\n", "2021-10-21 17:32:06,968 INFO sagemaker: Creating transform job with name: mangrove-2-notebook-run-21-16-54-33-dpp6-transform-21-17-28-31\n", ".....2021-10-21 17:32:11,315 INFO root: [Worker_2:dpp2-linear-learner]Executing step: perform_data_transform\n", "2021-10-21 17:32:11,316 INFO sagemaker: Creating transform job with name: mangrove-2-notebook-run-21-16-54-33-dpp2-transform-21-17-28-31\n", "...2021-10-21 17:32:14,092 INFO root: [Worker_0:dpp0-xgboost]Executing step: perform_data_transform\n", "2021-10-21 17:32:14,093 INFO sagemaker: Creating transform job with name: mangrove-2-notebook-run-21-16-54-33-dpp0-transform-21-17-28-31\n", "..\n", "2021-10-21 17:32:14 Uploading - Uploading generated training model.............\n", "2021-10-21 17:32:22 Completed - Training job completed\n", "....2021-10-21 17:32:29,138 INFO root: [Worker_5:dpp5-xgboost]Executing step: create_transformer_model\n", "2021-10-21 17:32:29,162 INFO sagemaker: Creating model with name: sagemaker-sklearn-automl-2021-10-21-17-32-29-139\n", "......2021-10-21 17:32:33,452 INFO root: [Worker_5:dpp5-xgboost]Executing step: perform_data_transform\n", "2021-10-21 17:32:33,453 INFO sagemaker: Creating transform job with name: mangrove-2-notebook-run-21-16-54-33-dpp5-transform-21-17-28-31\n", "..............................................................................................................................................................................................................................................................................................................................!\n", "2021-10-21 17:36:21,197 INFO root: Successfully fit data transformer for dpp3-xgboost\n", ".............................!\n", "2021-10-21 17:36:46,277 INFO root: Successfully fit data transformer for dpp4-xgboost\n", "!\n", "2021-10-21 17:36:47,852 INFO root: Successfully fit data transformer for dpp2-linear-learner\n", ".........!\n", "2021-10-21 17:36:58,867 INFO root: Successfully fit data transformer for dpp1-xgboost\n", "..!\n", "2021-10-21 17:37:03,598 INFO root: Successfully fit data transformer for dpp6-mlp\n", ".!\n", "2021-10-21 17:37:05,918 INFO root: Successfully fit data transformer for dpp0-xgboost\n", "!\n", "2021-10-21 17:37:09,963 INFO root: Successfully fit data transformer for dpp5-xgboost\n", "2021-10-21 17:37:09,965 INFO root: Successfully fit 7 data transformers\n", "CPU times: user 3.46 s, sys: 397 ms, total: 3.86 s\n", "Wall time: 8min 38s\n" ] } ], "source": [ "%%time\n", "automl_interactive_runner.fit_data_transformers(parallel_jobs=7)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Multi Algorithm Hyperparameter Tuning\n", "\n", "Now that the algorithm compatible transformed datasets are ready, you can start the multi-algorithm model tuning job\n", "to find the best predictive model. The following algorithm training job configuration for each\n", "algorithm is auto-generated by the AutoML Job as part of the recommendation.\n", "\n", "
💡 Available Knobs\n", "\n", "1. Hyperparameter ranges\n", "2. Objective metrics\n", "3. Recommended static algorithm hyperparameters.\n", "\n", "Please refers to [Xgboost tuning](https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost-tuning.html) and [Linear learner tuning](https://docs.aws.amazon.com/sagemaker/latest/dg/linear-learner-tuning.html) for detailed explanations of the parameters.\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The AutoML recommendation job has recommended the following hyperparameters, objectives and accuracy metrics for\n", "the algorithm and problem type:" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [], "source": [ "ALGORITHM_OBJECTIVE_METRICS = {\n", " 'xgboost': 'validation:auc',\n", " 'linear-learner': 'validation:roc_auc_score',\n", " 'mlp': 'validation:roc_auc',\n", "}\n", "\n", "STATIC_HYPERPARAMETERS = {\n", " 'xgboost': {\n", " 'objective': 'binary:logistic',\n", " 'eval_metric': 'accuracy,f1_binary,auc',\n", " '_kfold': 5,\n", " '_num_cv_round': 3,\n", " },\n", " 'linear-learner': {\n", " 'predictor_type': 'binary_classifier',\n", " 'loss': 'logistic',\n", " 'mini_batch_size': 800,\n", " 'binary_classifier_model_selection_criteria': 'loss_function',\n", " 'num_models': 1,\n", " },\n", " 'mlp': {\n", " 'problem_type': 'binary_classification',\n", " 'positive_example_weight_mult': 'auto',\n", " 'ml_application': 'mlp',\n", " 'use_batchnorm': 'true',\n", " 'activation': 'relu',\n", " 'warmup_epochs': 10,\n", " 'reporting_metrics': 'accuracy,binary_f_1,roc_auc',\n", " 'eval_metric': 'roc_auc',\n", " 'kfold': 5,\n", " 'num_cv_rounds': 3,\n", " },\n", "}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The following tunable hyperparameters search ranges are recommended for the Multi-Algo tuning job:" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "from sagemaker.parameter import CategoricalParameter, ContinuousParameter, IntegerParameter\n", "\n", "ALGORITHM_TUNABLE_HYPERPARAMETER_RANGES = {\n", " 'xgboost': {\n", " 'num_round': IntegerParameter(64, 1024, scaling_type='Logarithmic'),\n", " 'max_depth': IntegerParameter(2, 8, scaling_type='Logarithmic'),\n", " 'eta': ContinuousParameter(1e-3, 1.0, scaling_type='Logarithmic'),\n", " 'gamma': ContinuousParameter(1e-6, 64.0, scaling_type='Logarithmic'),\n", " 'min_child_weight': ContinuousParameter(1e-6, 32.0, scaling_type='Logarithmic'),\n", " 'subsample': ContinuousParameter(0.5, 1.0, scaling_type='Linear'),\n", " 'colsample_bytree': ContinuousParameter(0.3, 1.0, scaling_type='Linear'),\n", " 'lambda': ContinuousParameter(1e-6, 2.0, scaling_type='Logarithmic'),\n", " 'alpha': ContinuousParameter(1e-6, 2.0, scaling_type='Logarithmic'),\n", " },\n", " 'linear-learner': {\n", " 'wd': ContinuousParameter(1e-7, 1.0, scaling_type='Logarithmic'),\n", " 'l1': ContinuousParameter(1e-7, 1.0, scaling_type='Logarithmic'),\n", " 'learning_rate': ContinuousParameter(1e-5, 1.0, scaling_type='Logarithmic'),\n", " },\n", " 'mlp': {\n", " 'mini_batch_size': IntegerParameter(128, 512, scaling_type='Linear'),\n", " 'learning_rate': ContinuousParameter(1e-6, 1e-2, scaling_type='Logarithmic'),\n", " 'weight_decay': ContinuousParameter(1e-12, 1e-2, scaling_type='Logarithmic'),\n", " 'dropout_prob': ContinuousParameter(0.25, 0.5, scaling_type='Linear'),\n", " 'embedding_size_factor': ContinuousParameter(0.65, 0.95, scaling_type='Linear'),\n", " 'network_type': CategoricalParameter(['feedforward', 'widedeep']),\n", " 'layers': CategoricalParameter(['256', '50, 25', '100, 50', '200, 100', '256, 128', '300, 150', '200, 100, 50']),\n", " },\n", "}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Prepare Multi-Algorithm Tuner Input\n", "\n", "To use the multi-algorithm HPO tuner, prepare some inputs and parameters. Prepare a dictionary whose key is the name of the trained pipeline candidates and the values are respectively:\n", "\n", "1. Estimators for the recommended algorithm\n", "2. Hyperparameters search ranges\n", "3. Objective metrics" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "multi_algo_tuning_parameters = automl_interactive_runner.prepare_multi_algo_parameters(\n", " objective_metrics=ALGORITHM_OBJECTIVE_METRICS,\n", " static_hyperparameters=STATIC_HYPERPARAMETERS,\n", " hyperparameters_search_ranges=ALGORITHM_TUNABLE_HYPERPARAMETER_RANGES)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Below you prepare the inputs data to the multi-algo tuner:" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "multi_algo_tuning_inputs = automl_interactive_runner.prepare_multi_algo_inputs()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Create Multi-Algorithm Tuner\n", "\n", "With the recommended Hyperparameter ranges and the transformed dataset, create a multi-algorithm model tuning job\n", "that coordinates hyper parameter optimizations across the different possible algorithms and feature processing strategies.\n", "\n", "
💡 Available Knobs\n", "\n", "1. Tuner strategy: [Bayesian](https://en.wikipedia.org/wiki/Hyperparameter_optimization#Bayesian_optimization), [Random Search](https://en.wikipedia.org/wiki/Hyperparameter_optimization#Random_search)\n", "2. Objective type: `Minimize`, `Maximize`, see [optimization](https://en.wikipedia.org/wiki/Mathematical_optimization)\n", "3. Max Job size: the max number of training jobs HPO would be launching to run experiments. Note the default value is **250**\n", " which is the default of the managed flow.\n", "4. Parallelism. Number of jobs that will be executed in parallel. Higher value will expedite the tuning process.\n", " Please check the account limits to increase the limits before increasing the number of jobs to run in parallel\n", "5. Please use a different tuning job name if you re-run this cell after applied customizations.\n", "
" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [], "source": [ "from sagemaker.tuner import HyperparameterTuner\n", "\n", "base_tuning_job_name = \"{}-tuning\".format(AUTOML_LOCAL_RUN_CONFIG.local_automl_job_name)\n", "\n", "tuner = HyperparameterTuner.create(\n", " base_tuning_job_name=base_tuning_job_name,\n", " strategy='Bayesian',\n", " objective_type='Maximize',\n", " max_parallel_jobs=10,\n", " max_jobs=50,\n", " **multi_algo_tuning_parameters,\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Run Multi-Algorithm Tuning\n", "\n", "Now you are ready to start running the **Multi-Algo Tuning** job. After the job is finished, store the tuning job name which you use to select models in the next section.\n", "The tuning process will take some time, please track the progress in the Amazon SageMaker Hyperparameter tuning jobs console." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2021-10-21 20:16:09,756 INFO sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: latest.\n", "2021-10-21 20:16:09,803 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n", "2021-10-21 20:16:09,804 INFO sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: latest.\n", "2021-10-21 20:16:09,819 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n", "2021-10-21 20:16:09,820 INFO sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: latest.\n", "2021-10-21 20:16:09,834 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n", "2021-10-21 20:16:09,836 INFO sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: latest.\n", "2021-10-21 20:16:09,846 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n", "2021-10-21 20:16:09,847 INFO sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: latest.\n", "2021-10-21 20:16:09,864 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n", "2021-10-21 20:16:09,867 INFO sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: latest.\n", "2021-10-21 20:16:09,878 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n", "2021-10-21 20:16:09,881 INFO sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: latest.\n", "2021-10-21 20:16:09,892 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n", "2021-10-21 20:16:09,897 INFO sagemaker: Creating hyperparameter tuning job with name: mangrove-2-notebook--211021-2016\n", "...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................!\n" ] }, { "data": { "text/markdown": [ "Tuning Job mangrove-2-notebook--211021-2016 started, please track the progress from [here](https://us-east-2.console.aws.amazon.com/sagemaker/home?region=us-east-2#/hyper-tuning-jobs/mangrove-2-notebook--211021-2016)" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "!\n", "CPU times: user 6 s, sys: 425 ms, total: 6.43 s\n", "Wall time: 1h 17min 39s\n" ] } ], "source": [ "%%time\n", "from IPython.display import display, Markdown\n", "\n", "# Run tuning\n", "tuner.fit(inputs=multi_algo_tuning_inputs, include_cls_metadata=None)\n", "tuning_job_name = tuner.latest_tuning_job.name\n", "\n", "display(\n", " Markdown(f\"Tuning Job {tuning_job_name} started, please track the progress from [here](https://{AUTOML_LOCAL_RUN_CONFIG.region}.console.aws.amazon.com/sagemaker/home?region={AUTOML_LOCAL_RUN_CONFIG.region}#/hyper-tuning-jobs/{tuning_job_name})\"))\n", "\n", "# Wait for tuning job to finish\n", "tuner.wait()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Model Selection and Deployment\n", "\n", "This section guides you through the model selection process. Afterward, you construct an inference pipeline\n", "on Amazon SageMaker to host the best candidate.\n", "\n", "Because you executed the feature transformation and algorithm training in two separate steps, you now need to manually\n", "link each trained model with the feature transformer that it is associated with. When running a regular Amazon\n", "SageMaker Autopilot job, this will automatically be done for you." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Tuning Job Result Overview\n", "\n", "The performance of each candidate pipeline can be viewed as a Pandas dataframe. For more interactive usage please\n", "refers to [model tuning monitor](https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning-monitor.html)." ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
AUCrun_time
candidate
dpp6-mlp0.960082711.0
dpp4-xgboost0.95236385.0
dpp3-xgboost0.95095202.0
dpp4-xgboost0.95069458.0
dpp3-xgboost0.95015361.0
\n", "
" ], "text/plain": [ " AUC run_time\n", "candidate \n", "dpp6-mlp 0.96008 2711.0\n", "dpp4-xgboost 0.95236 385.0\n", "dpp3-xgboost 0.95095 202.0\n", "dpp4-xgboost 0.95069 458.0\n", "dpp3-xgboost 0.95015 361.0" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sagemaker.analytics import HyperparameterTuningJobAnalytics\n", "\n", "SAGEMAKER_SESSION = AUTOML_LOCAL_RUN_CONFIG.sagemaker_session\n", "SAGEMAKER_ROLE = AUTOML_LOCAL_RUN_CONFIG.role\n", "\n", "tuner_analytics = HyperparameterTuningJobAnalytics(\n", " tuner.latest_tuning_job.name, sagemaker_session=SAGEMAKER_SESSION)\n", "\n", "df_tuning_job_analytics = tuner_analytics.dataframe()\n", "\n", "# Sort the tuning job analytics by the final metrics value\n", "df_tuning_job_analytics.sort_values(\n", " by=['FinalObjectiveValue'],\n", " inplace=True,\n", " ascending=False if tuner.objective_type == \"Maximize\" else True)\n", "\n", "# select the columns to display and rename\n", "select_columns = [\"TrainingJobDefinitionName\", \"FinalObjectiveValue\", \"TrainingElapsedTimeSeconds\"]\n", "rename_columns = {\n", " \"TrainingJobDefinitionName\": \"candidate\",\n", " \"FinalObjectiveValue\": \"AUC\", \n", " \"TrainingElapsedTimeSeconds\": \"run_time\" \n", "}\n", "\n", "# Show detailed analytics for the top 5 models\n", "df_tuning_job_analytics.rename(columns=rename_columns)[rename_columns.values()].set_index(\"candidate\").head(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The best training job can be selected as below:\n", "\n", "
💡 Tips: \n", "You could select alternative job by using the value from `TrainingJobName` column above and assign to `best_training_job` below\n", "
" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Best Multi Algorithm HPO training job name is mangrove-2-notebook--211021-2016-012-500271c8\n" ] } ], "source": [ "attached_tuner = HyperparameterTuner.attach(tuner.latest_tuning_job.name, sagemaker_session=SAGEMAKER_SESSION)\n", "best_training_job = attached_tuner.best_training_job()\n", "\n", "print(\"Best Multi Algorithm HPO training job name is {}\".format(best_training_job))" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "dpp4-xgboost 14\n", "dpp5-xgboost 8\n", "dpp3-xgboost 6\n", "dpp0-xgboost 6\n", "dpp1-xgboost 6\n", "dpp2-linear-learner 5\n", "dpp6-mlp 5\n", "Name: TrainingJobDefinitionName, dtype: int64" ] }, "execution_count": 39, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Number of models built per algorithm\n", "df_tuning_job_analytics.TrainingJobDefinitionName.value_counts()" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
38
TrainingJobNamemangrove-2-notebook--211021-2016-012-500271c8
TrainingJobStatusCompleted
FinalObjectiveValue0.96008
TrainingStartTime2021-10-21 20:22:55+00:00
TrainingEndTime2021-10-21 21:08:06+00:00
TrainingElapsedTimeSeconds2711
TrainingJobDefinitionNamedpp6-mlp
dropout_prob0.415778
embedding_size_factor0.849226
layers256
learning_rate0.00013862
mini_batch_size317
network_typefeedforward
weight_decay1.29323e-12
\n", "
" ], "text/plain": [ " 38\n", "TrainingJobName mangrove-2-notebook--211021-2016-012-500271c8\n", "TrainingJobStatus Completed\n", "FinalObjectiveValue 0.96008\n", "TrainingStartTime 2021-10-21 20:22:55+00:00\n", "TrainingEndTime 2021-10-21 21:08:06+00:00\n", "TrainingElapsedTimeSeconds 2711\n", "TrainingJobDefinitionName dpp6-mlp\n", "dropout_prob 0.415778\n", "embedding_size_factor 0.849226\n", "layers 256\n", "learning_rate 0.00013862\n", "mini_batch_size 317\n", "network_type feedforward\n", "weight_decay 1.29323e-12" ] }, "execution_count": 40, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_tuning_job_analytics.loc[df_tuning_job_analytics.TrainingJobName==best_training_job].T.dropna()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Linking Best Training Job with Feature Pipelines\n", "\n", "Finally, deploy the best training job to Amazon SageMaker along with its companion feature engineering models.\n", "At the end of the section, you get an endpoint that's ready to serve online inference or start batch transform jobs!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Deploy a [PipelineModel](https://sagemaker.readthedocs.io/en/stable/pipeline.html) that has multiple containers of the following:\n", "\n", "1. Data Transformation Container: a container built from the model we selected and trained during the data transformer sections\n", "2. Algorithm Container: a container built from the trained model we selected above from the best HPO training job.\n", "3. Inverse Label Transformer Container: a container that converts numerical intermediate prediction value back to non-numerical label value.\n", "\n", "Get both best data transformation model and algorithm model from best training job and create an pipeline model:" ] }, { "cell_type": "code", "execution_count": 45, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2021-10-22 01:57:14,565 INFO root: Chosen Data Processing pipeline candidate name is dpp6-mlp\n", "\n", "2021-10-21 17:32:49 Starting - Preparing the instances for training\n", "2021-10-21 17:32:49 Downloading - Downloading input data\n", "2021-10-21 17:32:49 Training - Training image download completed. Training in progress.\n", "2021-10-21 17:32:49 Uploading - Uploading generated training model\n", "2021-10-21 17:32:49 Completed - Training job completed\n", "\n", "2021-10-21 21:08:06 Starting - Preparing the instances for training\n", "2021-10-21 21:08:06 Downloading - Downloading input data\n", "2021-10-21 21:08:06 Training - Training image download completed. Training in progress.\n", "2021-10-21 21:08:06 Uploading - Uploading generated training model\n", "2021-10-21 21:08:06 Completed - Training job completed\n", "\n", "2021-10-21 17:32:49 Starting - Preparing the instances for training\n", "2021-10-21 17:32:49 Downloading - Downloading input data\n", "2021-10-21 17:32:49 Training - Training image download completed. Training in progress.\n", "2021-10-21 17:32:49 Uploading - Uploading generated training model\n", "2021-10-21 17:32:49 Completed - Training job completed\n" ] } ], "source": [ "from sagemaker.estimator import Estimator\n", "from sagemaker import PipelineModel\n", "from sagemaker_automl import select_inference_output\n", "\n", "# Get a data transformation model from chosen candidate\n", "best_candidate = automl_interactive_runner.choose_candidate(df_tuning_job_analytics, best_training_job)\n", "best_data_transformer_model = best_candidate.get_data_transformer_model(role=SAGEMAKER_ROLE, sagemaker_session=SAGEMAKER_SESSION)\n", "\n", "# Our first data transformation container will always return recordio-protobuf format\n", "best_data_transformer_model.env[\"SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT\"] = 'application/x-recordio-protobuf'\n", "# Add environment variable for sparse encoding\n", "if best_candidate.data_transformer_step.sparse_encoding:\n", " best_data_transformer_model.env[\"AUTOML_SPARSE_ENCODE_RECORDIO_PROTOBUF\"] = '1'\n", "\n", "# Get a algo model from chosen training job of the candidate\n", "algo_estimator = Estimator.attach(best_training_job)\n", "best_algo_model = algo_estimator.create_model(**best_candidate.algo_step.get_inference_container_config())\n", "\n", "# Final pipeline model is composed of data transformation models and algo model and an\n", "# inverse label transform model if we need to transform the intermediates back to non-numerical value\n", "model_containers = [best_data_transformer_model, best_algo_model]\n", "if best_candidate.transforms_label:\n", " model_containers.append(best_candidate.get_data_transformer_model(\n", " transform_mode=\"inverse-label-transform\",\n", " role=SAGEMAKER_ROLE,\n", " sagemaker_session=SAGEMAKER_SESSION))\n", "\n", "# This model can emit response ['predicted_label', 'probability', 'labels', 'probabilities']. To enable the model to emit one or more\n", "# of the response content, pass the keys to `output_key` keyword argument in the select_inference_output method.\n", "\n", "model_containers = select_inference_output(\"BinaryClassification\", model_containers, output_keys=['predicted_label'])\n", "\n", "\n", "pipeline_model = PipelineModel(\n", " name=f\"mangrove-automl-2013\",\n", " role=SAGEMAKER_ROLE,\n", " models=model_containers,\n", " vpc_config=AUTOML_LOCAL_RUN_CONFIG.vpc_config)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Deploying Best Pipeline\n", "\n", "
💡 Available Knobs\n", "\n", "1. You can customize the initial instance count and instance type used to deploy this model.\n", "2. Endpoint name can be changed to avoid conflict with existing endpoints.\n", "\n", "
\n", "\n", "Finally, deploy the model to SageMaker to make it functional." ] }, { "cell_type": "code", "execution_count": 46, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2021-10-22 01:57:37,570 INFO sagemaker: Creating model with name: mangrove-automl-2013\n", "2021-10-22 01:57:38,310 INFO sagemaker: Creating endpoint with name mangrove-automl-2013\n", "---------------!CPU times: user 434 ms, sys: 47.1 ms, total: 481 ms\n", "Wall time: 7min 32s\n" ] } ], "source": [ "%%time\n", "pipeline_model.deploy(initial_instance_count=1,\n", " instance_type='ml.m5.2xlarge',\n", " endpoint_name=pipeline_model.name,\n", " wait=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Congratulations! Now you could visit the sagemaker\n", "[endpoint console page](https://us-east-2.console.aws.amazon.com/sagemaker/home?region=us-east-2#/endpoints) to find the deployed endpoint (it'll take a few minutes to be in service).\n", "\n", "
\n", " To rerun this notebook, delete or change the name of your endpoint!
\n", "If you rerun this notebook, you'll run into an error on the last step because the endpoint already exists. You can either delete the endpoint from the endpoint console page or you can change the endpoint_name in the previous code block.\n", "
" ] } ], "metadata": { "instance_type": "ml.t3.medium", "kernelspec": { "display_name": "Python 3 (Data Science)", "language": "python", "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-2:429704687514:image/datascience-1.0" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.10" } }, "nbformat": 4, "nbformat_minor": 4 }