"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"automl_interactive_runner.display_candidates()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The feature engineering pipeline consists of two SageMaker jobs:\n",
"\n",
"1. Generated trainable data transformer Python modules like [dpp0.py](mangrove-2013-artifacts/generated_module/candidate_data_processors/dpp0.py), which has been downloaded to the local file system\n",
"2. A **training** job to train the data transformers\n",
"3. A **batch transform** job to apply the trained transformation to the dataset to generate the algorithm compatible data\n",
"\n",
"The transformers and its training pipeline are built using open sourced **[sagemaker-scikit-learn-container][]** and **[sagemaker-scikit-learn-extension][]**.\n",
"\n",
"[sagemaker-scikit-learn-container]: https://github.com/aws/sagemaker-scikit-learn-container\n",
"[sagemaker-scikit-learn-extension]: https://github.com/aws/sagemaker-scikit-learn-extension"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Executing the Candidate Pipelines\n",
"\n",
"Each candidate pipeline consists of two steps, feature transformation and algorithm training.\n",
"For efficiency first execute the feature transformation step which will generate a featurized dataset on S3\n",
"for each pipeline.\n",
"\n",
"After each featurized dataset is prepared, execute a multi-algorithm tuning job that will run tuning jobs\n",
"in parallel for each pipeline. This tuning job will execute training jobs to find the best set of\n",
"hyper-parameters for each pipeline, as well as finding the overall best performing pipeline.\n",
"\n",
"### Run Data Transformation Steps\n",
"\n",
"Now you are ready to start execution all data transformation steps. The cell below may take some time to finish,\n",
"feel free to go grab a cup of coffee. To expedite the process you can set the number of `parallel_jobs` to be up to 10.\n",
"Please check the account limits to increase the limits before increasing the number of jobs to run in parallel."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-10-21 17:28:33,715 INFO root: [Worker_3:dpp3-xgboost]Executing step: train_data_transformer\n",
"2021-10-21 17:28:33,717 INFO sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: latest.\n",
"2021-10-21 17:28:33,722 INFO root: [Worker_4:dpp4-xgboost]Executing step: train_data_transformer\n",
"2021-10-21 17:28:33,727 INFO sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: latest.\n",
"2021-10-21 17:28:33,870 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n",
"2021-10-21 17:28:33,871 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n",
"2021-10-21 17:28:34,005 INFO sagemaker: Creating training-job with name: mangrove-2-notebook-run-21-16-54-33-dpp4-train-21-17-28-31\n",
"2021-10-21 17:28:34,009 INFO sagemaker: Creating training-job with name: mangrove-2-notebook-run-21-16-54-33-dpp3-train-21-17-28-31\n",
"\n",
"2021-10-21 17:28:34 Starting - Starting the training job\n",
"2021-10-21 17:28:34 Starting - Starting the training job2021-10-21 17:28:34,716 INFO root: [Worker_1:dpp1-xgboost]Executing step: train_data_transformer\n",
"2021-10-21 17:28:34,717 INFO sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: latest.\n",
"2021-10-21 17:28:34,729 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n",
"2021-10-21 17:28:34,821 INFO sagemaker: Creating training-job with name: mangrove-2-notebook-run-21-16-54-33-dpp1-train-21-17-28-31\n",
"2021-10-21 17:28:35,723 INFO root: [Worker_5:dpp5-xgboost]Executing step: train_data_transformer\n",
"2021-10-21 17:28:35,724 INFO sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: latest.\n",
"2021-10-21 17:28:35,736 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n",
"2021-10-21 17:28:35,837 INFO sagemaker: Creating training-job with name: mangrove-2-notebook-run-21-16-54-33-dpp5-train-21-17-28-31\n",
"2021-10-21 17:28:36,368 INFO root: We are getting throttled, retrying in 1s\n",
"2021-10-21 17:28:37,369 INFO sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: latest.\n",
"2021-10-21 17:28:37,382 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n",
"2021-10-21 17:28:37,487 INFO sagemaker: Creating training-job with name: mangrove-2-notebook-run-21-16-54-33-dpp1-train-21-17-28-31\n",
"2021-10-21 17:28:37,726 INFO root: [Worker_6:dpp6-mlp]Executing step: train_data_transformer\n",
"2021-10-21 17:28:37,727 INFO sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: latest.\n",
"2021-10-21 17:28:37,741 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n",
"2021-10-21 17:28:37,863 INFO sagemaker: Creating training-job with name: mangrove-2-notebook-run-21-16-54-33-dpp6-train-21-17-28-31\n",
"\n",
"2021-10-21 17:28:38 Starting - Starting the training job\n",
"2021-10-21 17:28:36 Starting - Launching requested ML instances\n",
"2021-10-21 17:28:36 Starting - Launching requested ML instances\n",
"2021-10-21 17:28:39 Starting - Starting the training job2021-10-21 17:28:40,715 INFO root: [Worker_2:dpp2-linear-learner]Executing step: train_data_transformer\n",
"2021-10-21 17:28:40,716 INFO sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: latest.\n",
"2021-10-21 17:28:40,730 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n",
"2021-10-21 17:28:40,836 INFO sagemaker: Creating training-job with name: mangrove-2-notebook-run-21-16-54-33-dpp2-train-21-17-28-31\n",
"\n",
"2021-10-21 17:28:40 Starting - Starting the training job2021-10-21 17:28:41,715 INFO root: [Worker_0:dpp0-xgboost]Executing step: train_data_transformer\n",
"2021-10-21 17:28:41,717 INFO sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: latest.\n",
"2021-10-21 17:28:41,730 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n",
"2021-10-21 17:28:41,853 INFO sagemaker: Creating training-job with name: mangrove-2-notebook-run-21-16-54-33-dpp0-train-21-17-28-31\n",
"\n",
"2021-10-21 17:28:42 Starting - Starting the training job\n",
"2021-10-21 17:28:40 Starting - Launching requested ML instances\n",
"2021-10-21 17:28:43 Starting - Starting the training job..\n",
"2021-10-21 17:28:41 Starting - Launching requested ML instances\n",
"2021-10-21 17:28:42 Starting - Launching requested ML instances\n",
"2021-10-21 17:28:44 Starting - Launching requested ML instances.\n",
"2021-10-21 17:28:45 Starting - Launching requested ML instances.........................................................\n",
"2021-10-21 17:29:29 Starting - Preparing the instances for training...\n",
"2021-10-21 17:29:33 Starting - Preparing the instances for training...\n",
"2021-10-21 17:29:35 Starting - Preparing the instances for training\n",
"2021-10-21 17:29:35 Starting - Preparing the instances for training.......\n",
"2021-10-21 17:29:38 Starting - Preparing the instances for training.........\n",
"2021-10-21 17:29:45 Starting - Preparing the instances for training............\n",
"2021-10-21 17:29:55 Starting - Preparing the instances for training........................................................\n",
"2021-10-21 17:30:39 Downloading - Downloading input data...............\n",
"2021-10-21 17:30:49 Downloading - Downloading input data....\n",
"2021-10-21 17:30:50 Training - Downloading the training image\n",
"2021-10-21 17:30:51 Downloading - Downloading input data\n",
"2021-10-21 17:30:53 Training - Downloading the training image\n",
"2021-10-21 17:30:55 Downloading - Downloading input data\n",
"2021-10-21 17:30:56 Downloading - Downloading input data\n",
"2021-10-21 17:30:57 Downloading - Downloading input data.......................\n",
"2021-10-21 17:31:11 Training - Downloading the training image...\n",
"2021-10-21 17:31:18 Training - Downloading the training image....\n",
"2021-10-21 17:31:17 Training - Downloading the training image\n",
"2021-10-21 17:31:20 Training - Downloading the training image.\n",
"2021-10-21 17:31:21 Downloading - Downloading input data...............\n",
"2021-10-21 17:31:30 Training - Training image download completed. Training in progress..\n",
"2021-10-21 17:31:33 Training - Training image download completed. Training in progress.\n",
"2021-10-21 17:31:35 Training - Training image download completed. Training in progress.\n",
"2021-10-21 17:31:37 Training - Training image download completed. Training in progress..\n",
"2021-10-21 17:31:39 Uploading - Uploading generated training model\n",
"2021-10-21 17:31:38 Uploading - Uploading generated training model..\n",
"2021-10-21 17:31:41 Uploading - Uploading generated training model\n",
"2021-10-21 17:31:43 Uploading - Uploading generated training model..\n",
"2021-10-21 17:31:43 Completed - Training job completed\n",
"\n",
"2021-10-21 17:31:44 Uploading - Uploading generated training model\n",
"2021-10-21 17:31:43 Uploading - Uploading generated training model2021-10-21 17:31:47,403 INFO root: [Worker_3:dpp3-xgboost]Executing step: create_transformer_model\n",
"2021-10-21 17:31:47,434 INFO sagemaker: Creating model with name: sagemaker-sklearn-automl-2021-10-21-17-31-47-404\n",
"..2021-10-21 17:31:49,756 INFO root: [Worker_3:dpp3-xgboost]Executing step: perform_data_transform\n",
"2021-10-21 17:31:49,761 INFO sagemaker: Creating transform job with name: mangrove-2-notebook-run-21-16-54-33-dpp3-transform-21-17-28-31\n",
"\n",
"2021-10-21 17:31:47 Training - Downloading the training image.\n",
"2021-10-21 17:31:46 Completed - Training job completed\n",
".\n",
"2021-10-21 17:31:51 Completed - Training job completed\n",
"2021-10-21 17:31:52,361 INFO root: [Worker_4:dpp4-xgboost]Executing step: create_transformer_model\n",
"2021-10-21 17:31:52,383 INFO sagemaker: Creating model with name: sagemaker-sklearn-automl-2021-10-21-17-31-52-361\n",
"\n",
"2021-10-21 17:31:49 Completed - Training job completed\n",
"\n",
"2021-10-21 17:31:50 Completed - Training job completed\n",
"2021-10-21 17:31:54,713 INFO root: [Worker_4:dpp4-xgboost]Executing step: perform_data_transform\n",
"2021-10-21 17:31:54,714 INFO sagemaker: Creating transform job with name: mangrove-2-notebook-run-21-16-54-33-dpp4-transform-21-17-28-31\n",
"...\n",
"2021-10-21 17:31:52 Completed - Training job completed\n",
"2021-10-21 17:31:58,921 INFO root: [Worker_1:dpp1-xgboost]Executing step: create_transformer_model\n",
"2021-10-21 17:31:58,947 INFO sagemaker: Creating model with name: sagemaker-sklearn-automl-2021-10-21-17-31-58-922\n",
"...2021-10-21 17:32:00,606 INFO root: [Worker_6:dpp6-mlp]Executing step: create_transformer_model\n",
"2021-10-21 17:32:00,629 INFO sagemaker: Creating model with name: sagemaker-sklearn-automl-2021-10-21-17-32-00-607\n",
"2021-10-21 17:32:01,113 INFO root: [Worker_2:dpp2-linear-learner]Executing step: create_transformer_model\n",
"2021-10-21 17:32:01,136 INFO sagemaker: Creating model with name: sagemaker-sklearn-automl-2021-10-21-17-32-01-114\n",
"2021-10-21 17:32:02,291 INFO root: [Worker_1:dpp1-xgboost]Executing step: perform_data_transform\n",
"2021-10-21 17:32:02,292 INFO sagemaker: Creating transform job with name: mangrove-2-notebook-run-21-16-54-33-dpp1-transform-21-17-28-31\n",
".2021-10-21 17:32:03,109 INFO root: [Worker_0:dpp0-xgboost]Executing step: create_transformer_model\n",
"2021-10-21 17:32:03,135 INFO sagemaker: Creating model with name: sagemaker-sklearn-automl-2021-10-21-17-32-03-110\n",
".\n",
"2021-10-21 17:32:03 Training - Training image download completed. Training in progress..2021-10-21 17:32:06,967 INFO root: [Worker_6:dpp6-mlp]Executing step: perform_data_transform\n",
"2021-10-21 17:32:06,968 INFO sagemaker: Creating transform job with name: mangrove-2-notebook-run-21-16-54-33-dpp6-transform-21-17-28-31\n",
".....2021-10-21 17:32:11,315 INFO root: [Worker_2:dpp2-linear-learner]Executing step: perform_data_transform\n",
"2021-10-21 17:32:11,316 INFO sagemaker: Creating transform job with name: mangrove-2-notebook-run-21-16-54-33-dpp2-transform-21-17-28-31\n",
"...2021-10-21 17:32:14,092 INFO root: [Worker_0:dpp0-xgboost]Executing step: perform_data_transform\n",
"2021-10-21 17:32:14,093 INFO sagemaker: Creating transform job with name: mangrove-2-notebook-run-21-16-54-33-dpp0-transform-21-17-28-31\n",
"..\n",
"2021-10-21 17:32:14 Uploading - Uploading generated training model.............\n",
"2021-10-21 17:32:22 Completed - Training job completed\n",
"....2021-10-21 17:32:29,138 INFO root: [Worker_5:dpp5-xgboost]Executing step: create_transformer_model\n",
"2021-10-21 17:32:29,162 INFO sagemaker: Creating model with name: sagemaker-sklearn-automl-2021-10-21-17-32-29-139\n",
"......2021-10-21 17:32:33,452 INFO root: [Worker_5:dpp5-xgboost]Executing step: perform_data_transform\n",
"2021-10-21 17:32:33,453 INFO sagemaker: Creating transform job with name: mangrove-2-notebook-run-21-16-54-33-dpp5-transform-21-17-28-31\n",
"..............................................................................................................................................................................................................................................................................................................................!\n",
"2021-10-21 17:36:21,197 INFO root: Successfully fit data transformer for dpp3-xgboost\n",
".............................!\n",
"2021-10-21 17:36:46,277 INFO root: Successfully fit data transformer for dpp4-xgboost\n",
"!\n",
"2021-10-21 17:36:47,852 INFO root: Successfully fit data transformer for dpp2-linear-learner\n",
".........!\n",
"2021-10-21 17:36:58,867 INFO root: Successfully fit data transformer for dpp1-xgboost\n",
"..!\n",
"2021-10-21 17:37:03,598 INFO root: Successfully fit data transformer for dpp6-mlp\n",
".!\n",
"2021-10-21 17:37:05,918 INFO root: Successfully fit data transformer for dpp0-xgboost\n",
"!\n",
"2021-10-21 17:37:09,963 INFO root: Successfully fit data transformer for dpp5-xgboost\n",
"2021-10-21 17:37:09,965 INFO root: Successfully fit 7 data transformers\n",
"CPU times: user 3.46 s, sys: 397 ms, total: 3.86 s\n",
"Wall time: 8min 38s\n"
]
}
],
"source": [
"%%time\n",
"automl_interactive_runner.fit_data_transformers(parallel_jobs=7)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Multi Algorithm Hyperparameter Tuning\n",
"\n",
"Now that the algorithm compatible transformed datasets are ready, you can start the multi-algorithm model tuning job\n",
"to find the best predictive model. The following algorithm training job configuration for each\n",
"algorithm is auto-generated by the AutoML Job as part of the recommendation.\n",
"\n",
" 💡 Available Knobs\n",
"\n",
"1. Hyperparameter ranges\n",
"2. Objective metrics\n",
"3. Recommended static algorithm hyperparameters.\n",
"\n",
"Please refers to [Xgboost tuning](https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost-tuning.html) and [Linear learner tuning](https://docs.aws.amazon.com/sagemaker/latest/dg/linear-learner-tuning.html) for detailed explanations of the parameters.\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The AutoML recommendation job has recommended the following hyperparameters, objectives and accuracy metrics for\n",
"the algorithm and problem type:"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [],
"source": [
"ALGORITHM_OBJECTIVE_METRICS = {\n",
" 'xgboost': 'validation:auc',\n",
" 'linear-learner': 'validation:roc_auc_score',\n",
" 'mlp': 'validation:roc_auc',\n",
"}\n",
"\n",
"STATIC_HYPERPARAMETERS = {\n",
" 'xgboost': {\n",
" 'objective': 'binary:logistic',\n",
" 'eval_metric': 'accuracy,f1_binary,auc',\n",
" '_kfold': 5,\n",
" '_num_cv_round': 3,\n",
" },\n",
" 'linear-learner': {\n",
" 'predictor_type': 'binary_classifier',\n",
" 'loss': 'logistic',\n",
" 'mini_batch_size': 800,\n",
" 'binary_classifier_model_selection_criteria': 'loss_function',\n",
" 'num_models': 1,\n",
" },\n",
" 'mlp': {\n",
" 'problem_type': 'binary_classification',\n",
" 'positive_example_weight_mult': 'auto',\n",
" 'ml_application': 'mlp',\n",
" 'use_batchnorm': 'true',\n",
" 'activation': 'relu',\n",
" 'warmup_epochs': 10,\n",
" 'reporting_metrics': 'accuracy,binary_f_1,roc_auc',\n",
" 'eval_metric': 'roc_auc',\n",
" 'kfold': 5,\n",
" 'num_cv_rounds': 3,\n",
" },\n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The following tunable hyperparameters search ranges are recommended for the Multi-Algo tuning job:"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [],
"source": [
"from sagemaker.parameter import CategoricalParameter, ContinuousParameter, IntegerParameter\n",
"\n",
"ALGORITHM_TUNABLE_HYPERPARAMETER_RANGES = {\n",
" 'xgboost': {\n",
" 'num_round': IntegerParameter(64, 1024, scaling_type='Logarithmic'),\n",
" 'max_depth': IntegerParameter(2, 8, scaling_type='Logarithmic'),\n",
" 'eta': ContinuousParameter(1e-3, 1.0, scaling_type='Logarithmic'),\n",
" 'gamma': ContinuousParameter(1e-6, 64.0, scaling_type='Logarithmic'),\n",
" 'min_child_weight': ContinuousParameter(1e-6, 32.0, scaling_type='Logarithmic'),\n",
" 'subsample': ContinuousParameter(0.5, 1.0, scaling_type='Linear'),\n",
" 'colsample_bytree': ContinuousParameter(0.3, 1.0, scaling_type='Linear'),\n",
" 'lambda': ContinuousParameter(1e-6, 2.0, scaling_type='Logarithmic'),\n",
" 'alpha': ContinuousParameter(1e-6, 2.0, scaling_type='Logarithmic'),\n",
" },\n",
" 'linear-learner': {\n",
" 'wd': ContinuousParameter(1e-7, 1.0, scaling_type='Logarithmic'),\n",
" 'l1': ContinuousParameter(1e-7, 1.0, scaling_type='Logarithmic'),\n",
" 'learning_rate': ContinuousParameter(1e-5, 1.0, scaling_type='Logarithmic'),\n",
" },\n",
" 'mlp': {\n",
" 'mini_batch_size': IntegerParameter(128, 512, scaling_type='Linear'),\n",
" 'learning_rate': ContinuousParameter(1e-6, 1e-2, scaling_type='Logarithmic'),\n",
" 'weight_decay': ContinuousParameter(1e-12, 1e-2, scaling_type='Logarithmic'),\n",
" 'dropout_prob': ContinuousParameter(0.25, 0.5, scaling_type='Linear'),\n",
" 'embedding_size_factor': ContinuousParameter(0.65, 0.95, scaling_type='Linear'),\n",
" 'network_type': CategoricalParameter(['feedforward', 'widedeep']),\n",
" 'layers': CategoricalParameter(['256', '50, 25', '100, 50', '200, 100', '256, 128', '300, 150', '200, 100, 50']),\n",
" },\n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Prepare Multi-Algorithm Tuner Input\n",
"\n",
"To use the multi-algorithm HPO tuner, prepare some inputs and parameters. Prepare a dictionary whose key is the name of the trained pipeline candidates and the values are respectively:\n",
"\n",
"1. Estimators for the recommended algorithm\n",
"2. Hyperparameters search ranges\n",
"3. Objective metrics"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [],
"source": [
"multi_algo_tuning_parameters = automl_interactive_runner.prepare_multi_algo_parameters(\n",
" objective_metrics=ALGORITHM_OBJECTIVE_METRICS,\n",
" static_hyperparameters=STATIC_HYPERPARAMETERS,\n",
" hyperparameters_search_ranges=ALGORITHM_TUNABLE_HYPERPARAMETER_RANGES)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Below you prepare the inputs data to the multi-algo tuner:"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [],
"source": [
"multi_algo_tuning_inputs = automl_interactive_runner.prepare_multi_algo_inputs()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Create Multi-Algorithm Tuner\n",
"\n",
"With the recommended Hyperparameter ranges and the transformed dataset, create a multi-algorithm model tuning job\n",
"that coordinates hyper parameter optimizations across the different possible algorithms and feature processing strategies.\n",
"\n",
" 💡 Available Knobs\n",
"\n",
"1. Tuner strategy: [Bayesian](https://en.wikipedia.org/wiki/Hyperparameter_optimization#Bayesian_optimization), [Random Search](https://en.wikipedia.org/wiki/Hyperparameter_optimization#Random_search)\n",
"2. Objective type: `Minimize`, `Maximize`, see [optimization](https://en.wikipedia.org/wiki/Mathematical_optimization)\n",
"3. Max Job size: the max number of training jobs HPO would be launching to run experiments. Note the default value is **250**\n",
" which is the default of the managed flow.\n",
"4. Parallelism. Number of jobs that will be executed in parallel. Higher value will expedite the tuning process.\n",
" Please check the account limits to increase the limits before increasing the number of jobs to run in parallel\n",
"5. Please use a different tuning job name if you re-run this cell after applied customizations.\n",
"
"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [],
"source": [
"from sagemaker.tuner import HyperparameterTuner\n",
"\n",
"base_tuning_job_name = \"{}-tuning\".format(AUTOML_LOCAL_RUN_CONFIG.local_automl_job_name)\n",
"\n",
"tuner = HyperparameterTuner.create(\n",
" base_tuning_job_name=base_tuning_job_name,\n",
" strategy='Bayesian',\n",
" objective_type='Maximize',\n",
" max_parallel_jobs=10,\n",
" max_jobs=50,\n",
" **multi_algo_tuning_parameters,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Run Multi-Algorithm Tuning\n",
"\n",
"Now you are ready to start running the **Multi-Algo Tuning** job. After the job is finished, store the tuning job name which you use to select models in the next section.\n",
"The tuning process will take some time, please track the progress in the Amazon SageMaker Hyperparameter tuning jobs console."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-10-21 20:16:09,756 INFO sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: latest.\n",
"2021-10-21 20:16:09,803 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n",
"2021-10-21 20:16:09,804 INFO sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: latest.\n",
"2021-10-21 20:16:09,819 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n",
"2021-10-21 20:16:09,820 INFO sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: latest.\n",
"2021-10-21 20:16:09,834 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n",
"2021-10-21 20:16:09,836 INFO sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: latest.\n",
"2021-10-21 20:16:09,846 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n",
"2021-10-21 20:16:09,847 INFO sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: latest.\n",
"2021-10-21 20:16:09,864 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n",
"2021-10-21 20:16:09,867 INFO sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: latest.\n",
"2021-10-21 20:16:09,878 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n",
"2021-10-21 20:16:09,881 INFO sagemaker.image_uris: Defaulting to the only supported framework/algorithm version: latest.\n",
"2021-10-21 20:16:09,892 INFO sagemaker.image_uris: Ignoring unnecessary instance type: None.\n",
"2021-10-21 20:16:09,897 INFO sagemaker: Creating hyperparameter tuning job with name: mangrove-2-notebook--211021-2016\n",
"...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................!\n"
]
},
{
"data": {
"text/markdown": [
"Tuning Job mangrove-2-notebook--211021-2016 started, please track the progress from [here](https://us-east-2.console.aws.amazon.com/sagemaker/home?region=us-east-2#/hyper-tuning-jobs/mangrove-2-notebook--211021-2016)"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"!\n",
"CPU times: user 6 s, sys: 425 ms, total: 6.43 s\n",
"Wall time: 1h 17min 39s\n"
]
}
],
"source": [
"%%time\n",
"from IPython.display import display, Markdown\n",
"\n",
"# Run tuning\n",
"tuner.fit(inputs=multi_algo_tuning_inputs, include_cls_metadata=None)\n",
"tuning_job_name = tuner.latest_tuning_job.name\n",
"\n",
"display(\n",
" Markdown(f\"Tuning Job {tuning_job_name} started, please track the progress from [here](https://{AUTOML_LOCAL_RUN_CONFIG.region}.console.aws.amazon.com/sagemaker/home?region={AUTOML_LOCAL_RUN_CONFIG.region}#/hyper-tuning-jobs/{tuning_job_name})\"))\n",
"\n",
"# Wait for tuning job to finish\n",
"tuner.wait()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Model Selection and Deployment\n",
"\n",
"This section guides you through the model selection process. Afterward, you construct an inference pipeline\n",
"on Amazon SageMaker to host the best candidate.\n",
"\n",
"Because you executed the feature transformation and algorithm training in two separate steps, you now need to manually\n",
"link each trained model with the feature transformer that it is associated with. When running a regular Amazon\n",
"SageMaker Autopilot job, this will automatically be done for you."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Tuning Job Result Overview\n",
"\n",
"The performance of each candidate pipeline can be viewed as a Pandas dataframe. For more interactive usage please\n",
"refers to [model tuning monitor](https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning-monitor.html)."
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" AUC | \n",
" run_time | \n",
"
\n",
" \n",
" candidate | \n",
" | \n",
" | \n",
"
\n",
" \n",
" \n",
" \n",
" dpp6-mlp | \n",
" 0.96008 | \n",
" 2711.0 | \n",
"
\n",
" \n",
" dpp4-xgboost | \n",
" 0.95236 | \n",
" 385.0 | \n",
"
\n",
" \n",
" dpp3-xgboost | \n",
" 0.95095 | \n",
" 202.0 | \n",
"
\n",
" \n",
" dpp4-xgboost | \n",
" 0.95069 | \n",
" 458.0 | \n",
"
\n",
" \n",
" dpp3-xgboost | \n",
" 0.95015 | \n",
" 361.0 | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" AUC run_time\n",
"candidate \n",
"dpp6-mlp 0.96008 2711.0\n",
"dpp4-xgboost 0.95236 385.0\n",
"dpp3-xgboost 0.95095 202.0\n",
"dpp4-xgboost 0.95069 458.0\n",
"dpp3-xgboost 0.95015 361.0"
]
},
"execution_count": 36,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from sagemaker.analytics import HyperparameterTuningJobAnalytics\n",
"\n",
"SAGEMAKER_SESSION = AUTOML_LOCAL_RUN_CONFIG.sagemaker_session\n",
"SAGEMAKER_ROLE = AUTOML_LOCAL_RUN_CONFIG.role\n",
"\n",
"tuner_analytics = HyperparameterTuningJobAnalytics(\n",
" tuner.latest_tuning_job.name, sagemaker_session=SAGEMAKER_SESSION)\n",
"\n",
"df_tuning_job_analytics = tuner_analytics.dataframe()\n",
"\n",
"# Sort the tuning job analytics by the final metrics value\n",
"df_tuning_job_analytics.sort_values(\n",
" by=['FinalObjectiveValue'],\n",
" inplace=True,\n",
" ascending=False if tuner.objective_type == \"Maximize\" else True)\n",
"\n",
"# select the columns to display and rename\n",
"select_columns = [\"TrainingJobDefinitionName\", \"FinalObjectiveValue\", \"TrainingElapsedTimeSeconds\"]\n",
"rename_columns = {\n",
" \"TrainingJobDefinitionName\": \"candidate\",\n",
" \"FinalObjectiveValue\": \"AUC\", \n",
" \"TrainingElapsedTimeSeconds\": \"run_time\" \n",
"}\n",
"\n",
"# Show detailed analytics for the top 5 models\n",
"df_tuning_job_analytics.rename(columns=rename_columns)[rename_columns.values()].set_index(\"candidate\").head(5)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The best training job can be selected as below:\n",
"\n",
" 💡 Tips: \n",
"You could select alternative job by using the value from `TrainingJobName` column above and assign to `best_training_job` below\n",
"
"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Best Multi Algorithm HPO training job name is mangrove-2-notebook--211021-2016-012-500271c8\n"
]
}
],
"source": [
"attached_tuner = HyperparameterTuner.attach(tuner.latest_tuning_job.name, sagemaker_session=SAGEMAKER_SESSION)\n",
"best_training_job = attached_tuner.best_training_job()\n",
"\n",
"print(\"Best Multi Algorithm HPO training job name is {}\".format(best_training_job))"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"dpp4-xgboost 14\n",
"dpp5-xgboost 8\n",
"dpp3-xgboost 6\n",
"dpp0-xgboost 6\n",
"dpp1-xgboost 6\n",
"dpp2-linear-learner 5\n",
"dpp6-mlp 5\n",
"Name: TrainingJobDefinitionName, dtype: int64"
]
},
"execution_count": 39,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Number of models built per algorithm\n",
"df_tuning_job_analytics.TrainingJobDefinitionName.value_counts()"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" 38 | \n",
"
\n",
" \n",
" \n",
" \n",
" TrainingJobName | \n",
" mangrove-2-notebook--211021-2016-012-500271c8 | \n",
"
\n",
" \n",
" TrainingJobStatus | \n",
" Completed | \n",
"
\n",
" \n",
" FinalObjectiveValue | \n",
" 0.96008 | \n",
"
\n",
" \n",
" TrainingStartTime | \n",
" 2021-10-21 20:22:55+00:00 | \n",
"
\n",
" \n",
" TrainingEndTime | \n",
" 2021-10-21 21:08:06+00:00 | \n",
"
\n",
" \n",
" TrainingElapsedTimeSeconds | \n",
" 2711 | \n",
"
\n",
" \n",
" TrainingJobDefinitionName | \n",
" dpp6-mlp | \n",
"
\n",
" \n",
" dropout_prob | \n",
" 0.415778 | \n",
"
\n",
" \n",
" embedding_size_factor | \n",
" 0.849226 | \n",
"
\n",
" \n",
" layers | \n",
" 256 | \n",
"
\n",
" \n",
" learning_rate | \n",
" 0.00013862 | \n",
"
\n",
" \n",
" mini_batch_size | \n",
" 317 | \n",
"
\n",
" \n",
" network_type | \n",
" feedforward | \n",
"
\n",
" \n",
" weight_decay | \n",
" 1.29323e-12 | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" 38\n",
"TrainingJobName mangrove-2-notebook--211021-2016-012-500271c8\n",
"TrainingJobStatus Completed\n",
"FinalObjectiveValue 0.96008\n",
"TrainingStartTime 2021-10-21 20:22:55+00:00\n",
"TrainingEndTime 2021-10-21 21:08:06+00:00\n",
"TrainingElapsedTimeSeconds 2711\n",
"TrainingJobDefinitionName dpp6-mlp\n",
"dropout_prob 0.415778\n",
"embedding_size_factor 0.849226\n",
"layers 256\n",
"learning_rate 0.00013862\n",
"mini_batch_size 317\n",
"network_type feedforward\n",
"weight_decay 1.29323e-12"
]
},
"execution_count": 40,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df_tuning_job_analytics.loc[df_tuning_job_analytics.TrainingJobName==best_training_job].T.dropna()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Linking Best Training Job with Feature Pipelines\n",
"\n",
"Finally, deploy the best training job to Amazon SageMaker along with its companion feature engineering models.\n",
"At the end of the section, you get an endpoint that's ready to serve online inference or start batch transform jobs!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Deploy a [PipelineModel](https://sagemaker.readthedocs.io/en/stable/pipeline.html) that has multiple containers of the following:\n",
"\n",
"1. Data Transformation Container: a container built from the model we selected and trained during the data transformer sections\n",
"2. Algorithm Container: a container built from the trained model we selected above from the best HPO training job.\n",
"3. Inverse Label Transformer Container: a container that converts numerical intermediate prediction value back to non-numerical label value.\n",
"\n",
"Get both best data transformation model and algorithm model from best training job and create an pipeline model:"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-10-22 01:57:14,565 INFO root: Chosen Data Processing pipeline candidate name is dpp6-mlp\n",
"\n",
"2021-10-21 17:32:49 Starting - Preparing the instances for training\n",
"2021-10-21 17:32:49 Downloading - Downloading input data\n",
"2021-10-21 17:32:49 Training - Training image download completed. Training in progress.\n",
"2021-10-21 17:32:49 Uploading - Uploading generated training model\n",
"2021-10-21 17:32:49 Completed - Training job completed\n",
"\n",
"2021-10-21 21:08:06 Starting - Preparing the instances for training\n",
"2021-10-21 21:08:06 Downloading - Downloading input data\n",
"2021-10-21 21:08:06 Training - Training image download completed. Training in progress.\n",
"2021-10-21 21:08:06 Uploading - Uploading generated training model\n",
"2021-10-21 21:08:06 Completed - Training job completed\n",
"\n",
"2021-10-21 17:32:49 Starting - Preparing the instances for training\n",
"2021-10-21 17:32:49 Downloading - Downloading input data\n",
"2021-10-21 17:32:49 Training - Training image download completed. Training in progress.\n",
"2021-10-21 17:32:49 Uploading - Uploading generated training model\n",
"2021-10-21 17:32:49 Completed - Training job completed\n"
]
}
],
"source": [
"from sagemaker.estimator import Estimator\n",
"from sagemaker import PipelineModel\n",
"from sagemaker_automl import select_inference_output\n",
"\n",
"# Get a data transformation model from chosen candidate\n",
"best_candidate = automl_interactive_runner.choose_candidate(df_tuning_job_analytics, best_training_job)\n",
"best_data_transformer_model = best_candidate.get_data_transformer_model(role=SAGEMAKER_ROLE, sagemaker_session=SAGEMAKER_SESSION)\n",
"\n",
"# Our first data transformation container will always return recordio-protobuf format\n",
"best_data_transformer_model.env[\"SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT\"] = 'application/x-recordio-protobuf'\n",
"# Add environment variable for sparse encoding\n",
"if best_candidate.data_transformer_step.sparse_encoding:\n",
" best_data_transformer_model.env[\"AUTOML_SPARSE_ENCODE_RECORDIO_PROTOBUF\"] = '1'\n",
"\n",
"# Get a algo model from chosen training job of the candidate\n",
"algo_estimator = Estimator.attach(best_training_job)\n",
"best_algo_model = algo_estimator.create_model(**best_candidate.algo_step.get_inference_container_config())\n",
"\n",
"# Final pipeline model is composed of data transformation models and algo model and an\n",
"# inverse label transform model if we need to transform the intermediates back to non-numerical value\n",
"model_containers = [best_data_transformer_model, best_algo_model]\n",
"if best_candidate.transforms_label:\n",
" model_containers.append(best_candidate.get_data_transformer_model(\n",
" transform_mode=\"inverse-label-transform\",\n",
" role=SAGEMAKER_ROLE,\n",
" sagemaker_session=SAGEMAKER_SESSION))\n",
"\n",
"# This model can emit response ['predicted_label', 'probability', 'labels', 'probabilities']. To enable the model to emit one or more\n",
"# of the response content, pass the keys to `output_key` keyword argument in the select_inference_output method.\n",
"\n",
"model_containers = select_inference_output(\"BinaryClassification\", model_containers, output_keys=['predicted_label'])\n",
"\n",
"\n",
"pipeline_model = PipelineModel(\n",
" name=f\"mangrove-automl-2013\",\n",
" role=SAGEMAKER_ROLE,\n",
" models=model_containers,\n",
" vpc_config=AUTOML_LOCAL_RUN_CONFIG.vpc_config)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Deploying Best Pipeline\n",
"\n",
" 💡 Available Knobs\n",
"\n",
"1. You can customize the initial instance count and instance type used to deploy this model.\n",
"2. Endpoint name can be changed to avoid conflict with existing endpoints.\n",
"\n",
"
\n",
"\n",
"Finally, deploy the model to SageMaker to make it functional."
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-10-22 01:57:37,570 INFO sagemaker: Creating model with name: mangrove-automl-2013\n",
"2021-10-22 01:57:38,310 INFO sagemaker: Creating endpoint with name mangrove-automl-2013\n",
"---------------!CPU times: user 434 ms, sys: 47.1 ms, total: 481 ms\n",
"Wall time: 7min 32s\n"
]
}
],
"source": [
"%%time\n",
"pipeline_model.deploy(initial_instance_count=1,\n",
" instance_type='ml.m5.2xlarge',\n",
" endpoint_name=pipeline_model.name,\n",
" wait=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Congratulations! Now you could visit the sagemaker\n",
"[endpoint console page](https://us-east-2.console.aws.amazon.com/sagemaker/home?region=us-east-2#/endpoints) to find the deployed endpoint (it'll take a few minutes to be in service).\n",
"\n",
"\n",
" To rerun this notebook, delete or change the name of your endpoint!
\n",
"If you rerun this notebook, you'll run into an error on the last step because the endpoint already exists. You can either delete the endpoint from the endpoint console page or you can change the endpoint_name
in the previous code block.\n",
"
"
]
}
],
"metadata": {
"instance_type": "ml.t3.medium",
"kernelspec": {
"display_name": "Python 3 (Data Science)",
"language": "python",
"name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-2:429704687514:image/datascience-1.0"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.10"
}
},
"nbformat": 4,
"nbformat_minor": 4
}