{ "cells": [ { "cell_type": "markdown", "id": "65e8d9da-ab2f-4e52-ac05-a0c0f46de9b4", "metadata": {}, "source": [ "---\n", "\n", "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.\n", "\n", "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-west-2/sagemaker-pipelines|selective-step-execution|Selective_Execution_Abalone.ipynb)\n", "\n", "---" ] }, { "cell_type": "markdown", "id": "bf679922-4032-40ef-b370-6a9b34293ea7", "metadata": {}, "source": [ "# SageMaker Pipelines: Selective Execution Demo" ] }, { "cell_type": "markdown", "id": "bf30edaa-ad14-4ae0-a490-afe1a0626661", "metadata": {}, "source": [ " In this demo notebook, we will walk a user through,\n", "1/ Setting up a new SageMaker pipeline which consists of Pre-Processing, Training, Evaluation and Model Registration - aka end-2-end ML Pipeline\n", "2/ How to leverage the previously defined end-2-end pipeline for Selective Execution - to selectively run subsets of your pipeline\n", "\n", "The flow of this notebook is as follows,\n", "- [Download Dataset](#Download-Dataset)\n", "- [Define a New Pipeline](#Define-a-New-Pipeline)\n", " - [Setup](#Setup)\n", " - [PreProcessing](#PreProcessing)\n", " - [Train](#Train)\n", " - [Evaluation](#Evaluation)\n", " - [Create Model](#Create-Model)\n", " - [Register Model](#Register-Model)\n", " - [Conditional Check](#Conditional-Check)\n", "- [Execute Pipeline](#Execute-Pipeline)\n", "- [Selective Execution](#Selective-Execution)\n", " - [Setup](#Setup)\n", " - [Selective Execution Scenarios](#Selective-Execution-Scenarios)\n", " - [Execute Single Pipeline Step with Changes to Parameters](#Execute-Single-Pipeline-Step-with-Changes-to-Parameters)\n", " - [Execute Multiple Contiguous Pipeline Steps or the Entire Pipeline Manually](#Execute-Multiple-Contiguous-Pipeline-Steps-or-the-Entire-Pipeline-Manually)" ] }, { "cell_type": "code", "execution_count": 2, "id": "1a69b6b5-dc28-4987-a945-0a7bc4e76613", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: sagemaker==2.162.0 in /opt/conda/lib/python3.10/site-packages (2.162.0)\n", "Requirement already satisfied: attrs<24,>=23.1.0 in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (23.1.0)\n", "Requirement already satisfied: boto3<2.0,>=1.26.131 in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (1.26.153)\n", "Requirement already satisfied: cloudpickle==2.2.1 in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (2.2.1)\n", "Requirement already satisfied: google-pasta in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (0.2.0)\n", "Requirement already satisfied: numpy<2.0,>=1.9.0 in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (1.24.3)\n", "Requirement already satisfied: protobuf<4.0,>=3.1 in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (3.20.3)\n", "Requirement already satisfied: protobuf3-to-dict<1.0,>=0.1.5 in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (0.1.5)\n", "Requirement already satisfied: smdebug-rulesconfig==1.0.1 in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (1.0.1)\n", "Requirement already satisfied: importlib-metadata<5.0,>=1.4.0 in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (4.11.3)\n", "Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (21.3)\n", "Requirement already satisfied: pandas in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (1.4.4)\n", "Requirement already satisfied: pathos in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (0.3.0)\n", "Requirement already satisfied: schema in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (0.7.5)\n", "Requirement already satisfied: PyYAML==6.0 in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (6.0)\n", "Requirement already satisfied: jsonschema in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (3.2.0)\n", "Requirement already satisfied: platformdirs in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (2.5.2)\n", "Requirement already satisfied: tblib==1.7.0 in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (1.7.0)\n", "Requirement already satisfied: botocore<1.30.0,>=1.29.153 in /opt/conda/lib/python3.10/site-packages (from boto3<2.0,>=1.26.131->sagemaker==2.162.0) (1.29.153)\n", "Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /opt/conda/lib/python3.10/site-packages (from boto3<2.0,>=1.26.131->sagemaker==2.162.0) (0.10.0)\n", "Requirement already satisfied: s3transfer<0.7.0,>=0.6.0 in /opt/conda/lib/python3.10/site-packages (from boto3<2.0,>=1.26.131->sagemaker==2.162.0) (0.6.0)\n", "Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.10/site-packages (from importlib-metadata<5.0,>=1.4.0->sagemaker==2.162.0) (3.8.0)\n", "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/conda/lib/python3.10/site-packages (from packaging>=20.0->sagemaker==2.162.0) (3.0.9)\n", "Requirement already satisfied: six in /opt/conda/lib/python3.10/site-packages (from protobuf3-to-dict<1.0,>=0.1.5->sagemaker==2.162.0) (1.16.0)\n", "Requirement already satisfied: pyrsistent>=0.14.0 in /opt/conda/lib/python3.10/site-packages (from jsonschema->sagemaker==2.162.0) (0.18.0)\n", "Requirement already satisfied: setuptools in /opt/conda/lib/python3.10/site-packages (from jsonschema->sagemaker==2.162.0) (67.8.0)\n", "Requirement already satisfied: python-dateutil>=2.8.1 in /opt/conda/lib/python3.10/site-packages (from pandas->sagemaker==2.162.0) (2.8.2)\n", "Requirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.10/site-packages (from pandas->sagemaker==2.162.0) (2022.1)\n", "Requirement already satisfied: ppft>=1.7.6.6 in /opt/conda/lib/python3.10/site-packages (from pathos->sagemaker==2.162.0) (1.7.6.6)\n", "Requirement already satisfied: dill>=0.3.6 in /opt/conda/lib/python3.10/site-packages (from pathos->sagemaker==2.162.0) (0.3.6)\n", "Requirement already satisfied: pox>=0.3.2 in /opt/conda/lib/python3.10/site-packages (from pathos->sagemaker==2.162.0) (0.3.2)\n", "Requirement already satisfied: multiprocess>=0.70.14 in /opt/conda/lib/python3.10/site-packages (from pathos->sagemaker==2.162.0) (0.70.14)\n", "Requirement already satisfied: contextlib2>=0.5.5 in /opt/conda/lib/python3.10/site-packages (from schema->sagemaker==2.162.0) (21.6.0)\n", "Requirement already satisfied: urllib3<1.27,>=1.25.4 in /opt/conda/lib/python3.10/site-packages (from botocore<1.30.0,>=1.29.153->boto3<2.0,>=1.26.131->sagemaker==2.162.0) (1.26.16)\n", "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n", "\u001b[0m\n", "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.1.2\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.2.1\u001b[0m\n", "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n" ] } ], "source": [ "!python3 -m pip install -U sagemaker==2.162.0" ] }, { "cell_type": "code", "execution_count": 3, "id": "9dfe4f3c-5d4f-4a70-80db-3a59a89b5bea", "metadata": { "tags": [] }, "outputs": [], "source": [ "import boto3\n", "import pprint\n", "import sagemaker\n", "from sagemaker.workflow.pipeline_context import PipelineSession" ] }, { "cell_type": "code", "execution_count": 4, "id": "4112dc0c-102c-4094-b15f-bd36911187fd", "metadata": { "tags": [] }, "outputs": [], "source": [ "sagemaker_session = sagemaker.session.Session()\n", "region = sagemaker_session.boto_region_name\n", "role = sagemaker.get_execution_role()\n", "pipeline_session = PipelineSession()\n", "default_bucket = sagemaker_session.default_bucket()\n", "sm_client = boto3.client(\"sagemaker\")\n", "s3 = boto3.resource(\"s3\")\n", "model_package_group_name = \"Abalone-Models\"" ] }, { "cell_type": "markdown", "id": "d081b072-3023-49b4-8d03-e6f65bcd482d", "metadata": {}, "source": [ "# Download Dataset " ] }, { "cell_type": "code", "execution_count": 5, "id": "e58905df-0b7f-4f39-b4af-ecfcfefc0874", "metadata": { "tags": [] }, "outputs": [], "source": [ "!mkdir -p raw-data/" ] }, { "cell_type": "code", "execution_count": 6, "id": "3117a396-d41f-42a9-9b91-2c9fde3fa2b8", "metadata": { "tags": [] }, "outputs": [], "source": [ "s3.Bucket(f\"sagemaker-example-files-prod-{region}\").download_file(\n", " \"datasets/tabular/uci_abalone/abalone.csv\", \"raw-data/abalone.csv\"\n", ")" ] }, { "cell_type": "code", "execution_count": 7, "id": "264e1cac-cb5f-4858-b24c-5532f7434f39", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "s3://sagemaker-us-east-1-681580474417/sample-dataset/abalone/abalone.csv\n" ] } ], "source": [ "input_data_uri = sagemaker.s3.S3Uploader.upload(\n", " local_path=\"raw-data/abalone.csv\",\n", " desired_s3_uri=f\"s3://{default_bucket}/sample-dataset/abalone\",\n", ")\n", "print(input_data_uri)" ] }, { "cell_type": "markdown", "id": "4ca7c782-a4df-4d59-95e1-80929389b152", "metadata": { "tags": [] }, "source": [ "# Define a New Pipeline" ] }, { "cell_type": "markdown", "id": "e2c85b0c-08e8-4245-a798-04bf4998e91f", "metadata": { "tags": [] }, "source": [ "## Setup" ] }, { "cell_type": "code", "execution_count": 8, "id": "ee6395a7-5573-427d-a51f-16fdb4730235", "metadata": { "tags": [] }, "outputs": [], "source": [ "from sagemaker.workflow.parameters import (\n", " ParameterInteger,\n", " ParameterString,\n", " ParameterFloat,\n", ")\n", "from sagemaker.workflow.steps import CacheConfig" ] }, { "cell_type": "markdown", "id": "51d36af3-9869-49a0-8759-2032ffd1347f", "metadata": {}, "source": [ "Lets define parameters which would be leveraged throught our pipeline, we can change these parameters during execution and control various aspects of our pipeline" ] }, { "cell_type": "code", "execution_count": 9, "id": "f6230bc2-07c1-4797-af29-215685742044", "metadata": { "tags": [] }, "outputs": [], "source": [ "step_cache_config = CacheConfig(\n", " enable_caching=True,\n", " expire_after=\"T12H\",\n", ")\n", "# processing params\n", "processing_instance_count = ParameterInteger(name=\"ProcessingInstanceCount\", default_value=1)\n", "processing_instance_type = ParameterString(\n", " name=\"ProcessingInstanceType\", default_value=\"ml.c5.xlarge\"\n", ")\n", "# training params\n", "train_instance_count = ParameterInteger(name=\"TrainingInstanceCount\", default_value=1)\n", "train_instance_type = ParameterString(name=\"TrainingInstanceType\", default_value=\"ml.c5.xlarge\")\n", "\n", "xgb_num_round = ParameterInteger(name=\"XGBNumRounds\", default_value=50)\n", "xgb_max_depth = ParameterInteger(name=\"XGBMaxDepth\", default_value=5)\n", "xgb_eta = ParameterFloat(name=\"XGBETA\", default_value=0.2)\n", "xgb_gamma = ParameterInteger(name=\"XGBGamma\", default_value=4)\n", "xgb_min_child_weight = ParameterInteger(name=\"XGBMinChildWeight\", default_value=6)\n", "xgb_subsample = ParameterFloat(name=\"XGBSubSample\", default_value=0.7)\n", "\n", "# model params\n", "model_path = ParameterString(\n", " name=\"ModelPath\", default_value=f\"s3://{default_bucket}/Abalone/models/\"\n", ")\n", "\n", "default_model_status = ParameterString(name=\"ApprovalStatus\", default_value=\"PendingManualApproval\")\n", "\n", "input_data = ParameterString(\n", " name=\"InputData\",\n", " default_value=input_data_uri,\n", ")\n", "\n", "threshold = ParameterFloat(name=\"MseThreshold\", default_value=6.0)" ] }, { "cell_type": "markdown", "id": "f9d62033-efd6-4cc9-a6f1-b74d83ea02e3", "metadata": {}, "source": [ "## PreProcessing " ] }, { "cell_type": "code", "execution_count": 10, "id": "9bd273d5-526d-48e2-819a-592239337e1d", "metadata": { "tags": [] }, "outputs": [], "source": [ "from sagemaker.sklearn.processing import SKLearnProcessor\n", "from sagemaker.processing import ProcessingInput, ProcessingOutput\n", "from sagemaker.workflow.steps import ProcessingStep" ] }, { "cell_type": "code", "execution_count": 11, "id": "8501a89a-c780-4f3a-b6ab-5582b319878d", "metadata": { "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "The input argument instance_type of function (sagemaker.image_uris.retrieve) is a pipeline variable (), which is not allowed. The default_value of this Parameter object will be used to override it. Please make sure the default_value is valid.\n" ] } ], "source": [ "sklearn_processor = SKLearnProcessor(\n", " framework_version=\"0.23-1\",\n", " instance_type=processing_instance_type,\n", " instance_count=processing_instance_count,\n", " base_job_name=\"abalone-preprocess\",\n", " role=role,\n", " sagemaker_session=pipeline_session,\n", ")" ] }, { "cell_type": "code", "execution_count": 12, "id": "78305be6-e205-4f59-96e8-468682e31175", "metadata": { "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/opt/conda/lib/python3.10/site-packages/sagemaker/workflow/pipeline_context.py:286: UserWarning: Running within a PipelineSession, there will be No Wait, No Logs, and No Job being started.\n", " warnings.warn(\n" ] } ], "source": [ "processor_args = sklearn_processor.run(\n", " inputs=[\n", " ProcessingInput(source=input_data, destination=\"/opt/ml/processing/input\"),\n", " ],\n", " outputs=[\n", " ProcessingOutput(output_name=\"train\", source=\"/opt/ml/processing/output/train\"),\n", " ProcessingOutput(output_name=\"validation\", source=\"/opt/ml/processing/output/validation\"),\n", " ProcessingOutput(output_name=\"test\", source=\"/opt/ml/processing/output/test\"),\n", " ],\n", " code=\"code/preprocessing.py\",\n", " arguments=[\n", " \"--train_size\",\n", " str(0.7),\n", " \"--val_size\",\n", " str(0.2),\n", " \"--test_size\",\n", " str(0.1),\n", " \"--random_state\",\n", " str(10),\n", " \"--cat_feature_cols\",\n", " \"sex\",\n", " \"--num_feature_cols\",\n", " \"length,diameter,height,whole_weight,shucked_weight,viscera_weight,shell_weight\",\n", " \"--target_col\",\n", " \"rings\",\n", " ],\n", ")\n", "\n", "step_preprocess = ProcessingStep(\n", " name=\"Abalone-Preprocess\", step_args=processor_args, cache_config=step_cache_config\n", ")" ] }, { "cell_type": "markdown", "id": "de432bf9-ba4d-4cf2-9286-62943bd7c1e6", "metadata": { "tags": [] }, "source": [ "## Train" ] }, { "cell_type": "code", "execution_count": 13, "id": "72bbb9cf-6b68-42ca-9443-4452b23126d4", "metadata": { "tags": [] }, "outputs": [], "source": [ "from sagemaker.estimator import Estimator\n", "from sagemaker.inputs import TrainingInput\n", "from sagemaker.inputs import TrainingInput\n", "from sagemaker.workflow.steps import TrainingStep" ] }, { "cell_type": "code", "execution_count": 14, "id": "9f7490db-845c-432c-abce-b54ae8e42cba", "metadata": { "tags": [] }, "outputs": [], "source": [ "image_uri = sagemaker.image_uris.retrieve(\n", " framework=\"xgboost\",\n", " region=region,\n", " version=\"1.0-1\",\n", " py_version=\"py3\",\n", " instance_type=train_instance_type.default_value,\n", ")\n", "xgb_train = Estimator(\n", " image_uri=image_uri,\n", " instance_count=train_instance_count,\n", " output_path=model_path,\n", " base_job_name=\"abalone-train\",\n", " instance_type=train_instance_type.default_value,\n", " role=role,\n", " sagemaker_session=pipeline_session,\n", ")\n", "xgb_train.set_hyperparameters(\n", " objective=\"reg:linear\",\n", " num_round=xgb_num_round,\n", " max_depth=xgb_max_depth,\n", " eta=xgb_eta,\n", " gamma=xgb_gamma,\n", " min_child_weight=xgb_gamma,\n", " subsample=xgb_subsample,\n", ")" ] }, { "cell_type": "code", "execution_count": 15, "id": "30997087-53ec-45ab-af11-d513d2505d4c", "metadata": { "tags": [] }, "outputs": [], "source": [ "train_args = xgb_train.fit(\n", " inputs={\n", " \"train\": TrainingInput(\n", " s3_data=step_preprocess.properties.ProcessingOutputConfig.Outputs[\n", " \"train\"\n", " ].S3Output.S3Uri,\n", " content_type=\"text/csv\",\n", " ),\n", " \"validation\": TrainingInput(\n", " s3_data=step_preprocess.properties.ProcessingOutputConfig.Outputs[\n", " \"validation\"\n", " ].S3Output.S3Uri,\n", " content_type=\"text/csv\",\n", " ),\n", " }\n", ")\n", "\n", "step_train = TrainingStep(\n", " name=\"Abalone-Train\", step_args=train_args, cache_config=step_cache_config\n", ")" ] }, { "cell_type": "markdown", "id": "4e173cc4-4562-420f-afb0-b98a043105e4", "metadata": {}, "source": [ "## Evaluation" ] }, { "cell_type": "code", "execution_count": 16, "id": "52154379-ee37-4cc2-be56-448830ffa651", "metadata": { "tags": [] }, "outputs": [], "source": [ "from sagemaker.processing import ScriptProcessor\n", "from sagemaker.workflow.properties import PropertyFile" ] }, { "cell_type": "code", "execution_count": 17, "id": "ac8bba1a-31bc-46a8-b153-444010f28aba", "metadata": { "tags": [] }, "outputs": [], "source": [ "script_eval = ScriptProcessor(\n", " image_uri=image_uri,\n", " command=[\"python3\"],\n", " instance_type=processing_instance_type,\n", " volume_size_in_gb=10,\n", " instance_count=1,\n", " base_job_name=\"abalone-postproc\",\n", " role=role,\n", " sagemaker_session=pipeline_session,\n", ")" ] }, { "cell_type": "code", "execution_count": 18, "id": "c492ecba-1ae4-4414-b863-a090268fb1be", "metadata": { "tags": [] }, "outputs": [], "source": [ "eval_args = script_eval.run(\n", " inputs=[\n", " ProcessingInput(\n", " source=step_train.properties.ModelArtifacts.S3ModelArtifacts,\n", " destination=\"/opt/ml/processing/model\",\n", " ),\n", " ProcessingInput(\n", " source=step_preprocess.properties.ProcessingOutputConfig.Outputs[\"test\"].S3Output.S3Uri,\n", " destination=\"/opt/ml/processing/output/test\",\n", " ),\n", " ],\n", " outputs=[\n", " ProcessingOutput(output_name=\"evaluation\", source=\"/opt/ml/processing/evaluation\"),\n", " ],\n", " code=\"code/evaluation.py\",\n", ")\n", "\n", "evaluation_report = PropertyFile(\n", " name=\"EvaluationReport\", output_name=\"evaluation\", path=\"evaluation.json\"\n", ")\n", "\n", "step_eval = ProcessingStep(\n", " name=\"Abalone-Evaluate\",\n", " step_args=eval_args,\n", " cache_config=step_cache_config,\n", " property_files=[evaluation_report],\n", ")" ] }, { "cell_type": "markdown", "id": "924756cb-1613-4644-b804-574be2f1071a", "metadata": {}, "source": [ "## Create Model" ] }, { "cell_type": "code", "execution_count": 19, "id": "30baedc4-0af3-4abc-a8e4-a679cef394de", "metadata": { "tags": [] }, "outputs": [], "source": [ "from sagemaker.model import Model\n", "from sagemaker.inputs import CreateModelInput\n", "from sagemaker.workflow.model_step import ModelStep" ] }, { "cell_type": "code", "execution_count": 20, "id": "f876f774-748e-4259-ac6f-77a667c29309", "metadata": { "tags": [] }, "outputs": [], "source": [ "model = Model(\n", " image_uri=image_uri,\n", " model_data=step_train.properties.ModelArtifacts.S3ModelArtifacts,\n", " sagemaker_session=pipeline_session,\n", " role=role,\n", ")" ] }, { "cell_type": "code", "execution_count": 21, "id": "8619e368-54eb-4d2f-9252-27bb085c05f7", "metadata": { "tags": [] }, "outputs": [], "source": [ "step_create_model = ModelStep(\n", " name=\"Abalone-CreateModel\",\n", " step_args=model.create(instance_type=\"ml.m5.large\", accelerator_type=\"ml.eia1.medium\"),\n", ")" ] }, { "cell_type": "markdown", "id": "4adc007a-4536-4f88-9c65-959a8c3dbc0a", "metadata": { "tags": [] }, "source": [ "## Register Model" ] }, { "cell_type": "code", "execution_count": 22, "id": "29a25590-9d5d-4590-8568-704060c98c75", "metadata": { "tags": [] }, "outputs": [], "source": [ "from sagemaker.model_metrics import MetricsSource, ModelMetrics" ] }, { "cell_type": "code", "execution_count": 23, "id": "313fcdcb-35e7-477a-bcb1-8b777c15aabc", "metadata": { "tags": [] }, "outputs": [], "source": [ "model_metrics = ModelMetrics(\n", " model_statistics=MetricsSource(\n", " s3_uri=\"{}/evaluation.json\".format(\n", " step_eval.arguments[\"ProcessingOutputConfig\"][\"Outputs\"][0][\"S3Output\"][\"S3Uri\"]\n", " ),\n", " content_type=\"application/json\",\n", " )\n", ")" ] }, { "cell_type": "code", "execution_count": 24, "id": "658a5f92-4154-47c8-ab6f-f31379e26dd6", "metadata": { "tags": [] }, "outputs": [], "source": [ "register_args = model.register(\n", " content_types=[\"text/csv\"],\n", " response_types=[\"text/csv\"],\n", " inference_instances=[\"ml.t2.medium\", \"ml.m5.xlarge\"],\n", " transform_instances=[\"ml.m5.xlarge\"],\n", " model_package_group_name=model_package_group_name,\n", " approval_status=default_model_status,\n", " model_metrics=model_metrics,\n", ")\n", "\n", "step_register = ModelStep(name=\"Abalone-ModelRegister\", step_args=register_args)" ] }, { "cell_type": "markdown", "id": "550ca0c0-916c-4a42-bcf9-5c123d61df3b", "metadata": {}, "source": [ "## Conditional Check" ] }, { "cell_type": "code", "execution_count": 25, "id": "693c4936-ecff-4b70-8f7c-512af15b2231", "metadata": { "tags": [] }, "outputs": [], "source": [ "from sagemaker.workflow.fail_step import FailStep\n", "from sagemaker.workflow.functions import Join\n", "from sagemaker.workflow.conditions import ConditionLessThanOrEqualTo\n", "from sagemaker.workflow.condition_step import ConditionStep\n", "from sagemaker.workflow.functions import JsonGet" ] }, { "cell_type": "code", "execution_count": 26, "id": "2c2a3351-73aa-4e18-9a55-ac2dba566f1f", "metadata": { "tags": [] }, "outputs": [], "source": [ "step_fail = FailStep(\n", " name=\"Abalone-FailNotify\",\n", " error_message=Join(on=\" \", values=[\"Execution failed due to MSE >\", threshold]),\n", ")\n", "\n", "cond_lte = ConditionLessThanOrEqualTo(\n", " left=JsonGet(\n", " step_name=step_eval.name,\n", " property_file=evaluation_report,\n", " json_path=\"regression_metrics.mse.value\",\n", " ),\n", " right=threshold,\n", ")" ] }, { "cell_type": "code", "execution_count": 27, "id": "490dcd45-b727-4d95-998e-ce2184abddf2", "metadata": { "tags": [] }, "outputs": [], "source": [ "step_cond = ConditionStep(\n", " name=\"Abalone-MSECheck\",\n", " conditions=[cond_lte],\n", " if_steps=[step_register, step_create_model],\n", " else_steps=[step_fail],\n", ")" ] }, { "cell_type": "markdown", "id": "a976de6e-c5a5-4061-9fa8-cdb088c50dec", "metadata": {}, "source": [ "# Execute Pipeline" ] }, { "cell_type": "code", "execution_count": 28, "id": "4359ccfc-cb81-41a1-8ceb-0a1e00076faf", "metadata": { "tags": [] }, "outputs": [], "source": [ "from sagemaker.workflow.pipeline import Pipeline\n", "from sagemaker.workflow.execution_variables import ExecutionVariables\n", "from sagemaker.workflow.pipeline import PipelineExperimentConfig" ] }, { "cell_type": "code", "execution_count": 29, "id": "d83036c2-06d2-4f33-92c4-d96ad4941bd9", "metadata": { "tags": [] }, "outputs": [], "source": [ "experiment_name = \"AbaloneExperimentsPipeline\"\n", "\n", "# Pipeline experiment config\n", "abalone_experiments_config = PipelineExperimentConfig(\n", " experiment_name, ExecutionVariables.PIPELINE_EXECUTION_ID\n", ")" ] }, { "cell_type": "code", "execution_count": 30, "id": "9fc5c324-cb5f-400f-b56a-711e542d9e96", "metadata": { "tags": [] }, "outputs": [], "source": [ "pipeline_name = \"AbalonePipeline-SelectExec\"\n", "\n", "pipeline = Pipeline(\n", " name=pipeline_name,\n", " pipeline_experiment_config=abalone_experiments_config,\n", " parameters=[\n", " processing_instance_count,\n", " processing_instance_type,\n", " train_instance_count,\n", " train_instance_type,\n", " xgb_num_round,\n", " xgb_max_depth,\n", " xgb_eta,\n", " xgb_gamma,\n", " xgb_min_child_weight,\n", " xgb_subsample,\n", " model_path,\n", " default_model_status,\n", " input_data,\n", " threshold,\n", " ],\n", " steps=[step_preprocess, step_train, step_eval, step_cond],\n", " sagemaker_session=pipeline_session,\n", ")" ] }, { "cell_type": "code", "execution_count": 31, "id": "05290a86-8bd9-4488-aad7-f334fcbda78f", "metadata": { "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "WARNING:sagemaker.workflow._utils:Popping out 'CertifyForMarketplace' from the pipeline definition since it will be overridden in pipeline execution time.\n" ] }, { "data": { "text/plain": [ "{'Version': '2020-12-01',\n", " 'Metadata': {},\n", " 'Parameters': [{'Name': 'ProcessingInstanceCount',\n", " 'Type': 'Integer',\n", " 'DefaultValue': 1},\n", " {'Name': 'ProcessingInstanceType',\n", " 'Type': 'String',\n", " 'DefaultValue': 'ml.c5.xlarge'},\n", " {'Name': 'TrainingInstanceCount', 'Type': 'Integer', 'DefaultValue': 1},\n", " {'Name': 'TrainingInstanceType',\n", " 'Type': 'String',\n", " 'DefaultValue': 'ml.c5.xlarge'},\n", " {'Name': 'XGBNumRounds', 'Type': 'Integer', 'DefaultValue': 50},\n", " {'Name': 'XGBMaxDepth', 'Type': 'Integer', 'DefaultValue': 5},\n", " {'Name': 'XGBETA', 'Type': 'Float', 'DefaultValue': 0.2},\n", " {'Name': 'XGBGamma', 'Type': 'Integer', 'DefaultValue': 4},\n", " {'Name': 'XGBMinChildWeight', 'Type': 'Integer', 'DefaultValue': 6},\n", " {'Name': 'XGBSubSample', 'Type': 'Float', 'DefaultValue': 0.7},\n", " {'Name': 'ModelPath',\n", " 'Type': 'String',\n", " 'DefaultValue': 's3://sagemaker-us-east-1-681580474417/Abalone/models/'},\n", " {'Name': 'ApprovalStatus',\n", " 'Type': 'String',\n", " 'DefaultValue': 'PendingManualApproval'},\n", " {'Name': 'InputData',\n", " 'Type': 'String',\n", " 'DefaultValue': 's3://sagemaker-us-east-1-681580474417/sample-dataset/abalone/abalone.csv'},\n", " {'Name': 'MseThreshold', 'Type': 'Float', 'DefaultValue': 6.0}],\n", " 'PipelineExperimentConfig': {'ExperimentName': 'AbaloneExperimentsPipeline',\n", " 'TrialName': {'Get': 'Execution.PipelineExecutionId'}},\n", " 'Steps': [{'Name': 'Abalone-Preprocess',\n", " 'Type': 'Processing',\n", " 'Arguments': {'ProcessingResources': {'ClusterConfig': {'InstanceType': {'Get': 'Parameters.ProcessingInstanceType'},\n", " 'InstanceCount': {'Get': 'Parameters.ProcessingInstanceCount'},\n", " 'VolumeSizeInGB': 30}},\n", " 'AppSpecification': {'ImageUri': '683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-scikit-learn:0.23-1-cpu-py3',\n", " 'ContainerArguments': ['--train_size',\n", " '0.7',\n", " '--val_size',\n", " '0.2',\n", " '--test_size',\n", " '0.1',\n", " '--random_state',\n", " '10',\n", " '--cat_feature_cols',\n", " 'sex',\n", " '--num_feature_cols',\n", " 'length,diameter,height,whole_weight,shucked_weight,viscera_weight,shell_weight',\n", " '--target_col',\n", " 'rings'],\n", " 'ContainerEntrypoint': ['python3',\n", " '/opt/ml/processing/input/code/preprocessing.py']},\n", " 'RoleArn': 'arn:aws:iam::681580474417:role/SageMakerCFDomainCreation-SageMakerExecutionRole-12TMZ4EI9FOSA',\n", " 'ProcessingInputs': [{'InputName': 'input-1',\n", " 'AppManaged': False,\n", " 'S3Input': {'S3Uri': {'Get': 'Parameters.InputData'},\n", " 'LocalPath': '/opt/ml/processing/input',\n", " 'S3DataType': 'S3Prefix',\n", " 'S3InputMode': 'File',\n", " 'S3DataDistributionType': 'FullyReplicated',\n", " 'S3CompressionType': 'None'}},\n", " {'InputName': 'code',\n", " 'AppManaged': False,\n", " 'S3Input': {'S3Uri': 's3://sagemaker-us-east-1-681580474417/AbalonePipeline-SelectExec/code/5d9f8cf978eb635fdf8359a0de190535/preprocessing.py',\n", " 'LocalPath': '/opt/ml/processing/input/code',\n", " 'S3DataType': 'S3Prefix',\n", " 'S3InputMode': 'File',\n", " 'S3DataDistributionType': 'FullyReplicated',\n", " 'S3CompressionType': 'None'}}],\n", " 'ProcessingOutputConfig': {'Outputs': [{'OutputName': 'train',\n", " 'AppManaged': False,\n", " 'S3Output': {'S3Uri': {'Std:Join': {'On': '/',\n", " 'Values': ['s3:/',\n", " 'sagemaker-us-east-1-681580474417',\n", " 'AbalonePipeline-SelectExec',\n", " {'Get': 'Execution.PipelineExecutionId'},\n", " 'Abalone-Preprocess',\n", " 'output',\n", " 'train']}},\n", " 'LocalPath': '/opt/ml/processing/output/train',\n", " 'S3UploadMode': 'EndOfJob'}},\n", " {'OutputName': 'validation',\n", " 'AppManaged': False,\n", " 'S3Output': {'S3Uri': {'Std:Join': {'On': '/',\n", " 'Values': ['s3:/',\n", " 'sagemaker-us-east-1-681580474417',\n", " 'AbalonePipeline-SelectExec',\n", " {'Get': 'Execution.PipelineExecutionId'},\n", " 'Abalone-Preprocess',\n", " 'output',\n", " 'validation']}},\n", " 'LocalPath': '/opt/ml/processing/output/validation',\n", " 'S3UploadMode': 'EndOfJob'}},\n", " {'OutputName': 'test',\n", " 'AppManaged': False,\n", " 'S3Output': {'S3Uri': {'Std:Join': {'On': '/',\n", " 'Values': ['s3:/',\n", " 'sagemaker-us-east-1-681580474417',\n", " 'AbalonePipeline-SelectExec',\n", " {'Get': 'Execution.PipelineExecutionId'},\n", " 'Abalone-Preprocess',\n", " 'output',\n", " 'test']}},\n", " 'LocalPath': '/opt/ml/processing/output/test',\n", " 'S3UploadMode': 'EndOfJob'}}]}},\n", " 'CacheConfig': {'Enabled': True, 'ExpireAfter': 'T12H'}},\n", " {'Name': 'Abalone-Train',\n", " 'Type': 'Training',\n", " 'Arguments': {'AlgorithmSpecification': {'TrainingInputMode': 'File',\n", " 'TrainingImage': '683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-xgboost:1.0-1-cpu-py3'},\n", " 'OutputDataConfig': {'S3OutputPath': {'Get': 'Parameters.ModelPath'}},\n", " 'StoppingCondition': {'MaxRuntimeInSeconds': 86400},\n", " 'ResourceConfig': {'VolumeSizeInGB': 30,\n", " 'InstanceCount': {'Get': 'Parameters.TrainingInstanceCount'},\n", " 'InstanceType': 'ml.c5.xlarge'},\n", " 'RoleArn': 'arn:aws:iam::681580474417:role/SageMakerCFDomainCreation-SageMakerExecutionRole-12TMZ4EI9FOSA',\n", " 'InputDataConfig': [{'DataSource': {'S3DataSource': {'S3DataType': 'S3Prefix',\n", " 'S3Uri': {'Get': \"Steps.Abalone-Preprocess.ProcessingOutputConfig.Outputs['train'].S3Output.S3Uri\"},\n", " 'S3DataDistributionType': 'FullyReplicated'}},\n", " 'ContentType': 'text/csv',\n", " 'ChannelName': 'train'},\n", " {'DataSource': {'S3DataSource': {'S3DataType': 'S3Prefix',\n", " 'S3Uri': {'Get': \"Steps.Abalone-Preprocess.ProcessingOutputConfig.Outputs['validation'].S3Output.S3Uri\"},\n", " 'S3DataDistributionType': 'FullyReplicated'}},\n", " 'ContentType': 'text/csv',\n", " 'ChannelName': 'validation'}],\n", " 'HyperParameters': {'objective': 'reg:linear',\n", " 'num_round': {'Std:Join': {'On': '',\n", " 'Values': [{'Get': 'Parameters.XGBNumRounds'}]}},\n", " 'max_depth': {'Std:Join': {'On': '',\n", " 'Values': [{'Get': 'Parameters.XGBMaxDepth'}]}},\n", " 'eta': {'Std:Join': {'On': '', 'Values': [{'Get': 'Parameters.XGBETA'}]}},\n", " 'gamma': {'Std:Join': {'On': '',\n", " 'Values': [{'Get': 'Parameters.XGBGamma'}]}},\n", " 'min_child_weight': {'Std:Join': {'On': '',\n", " 'Values': [{'Get': 'Parameters.XGBGamma'}]}},\n", " 'subsample': {'Std:Join': {'On': '',\n", " 'Values': [{'Get': 'Parameters.XGBSubSample'}]}}},\n", " 'DebugHookConfig': {'S3OutputPath': {'Get': 'Parameters.ModelPath'},\n", " 'CollectionConfigurations': []},\n", " 'ProfilerConfig': {'S3OutputPath': {'Get': 'Parameters.ModelPath'},\n", " 'DisableProfiler': False}},\n", " 'CacheConfig': {'Enabled': True, 'ExpireAfter': 'T12H'}},\n", " {'Name': 'Abalone-Evaluate',\n", " 'Type': 'Processing',\n", " 'Arguments': {'ProcessingResources': {'ClusterConfig': {'InstanceType': {'Get': 'Parameters.ProcessingInstanceType'},\n", " 'InstanceCount': 1,\n", " 'VolumeSizeInGB': 10}},\n", " 'AppSpecification': {'ImageUri': '683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-xgboost:1.0-1-cpu-py3',\n", " 'ContainerEntrypoint': ['python3',\n", " '/opt/ml/processing/input/code/evaluation.py']},\n", " 'RoleArn': 'arn:aws:iam::681580474417:role/SageMakerCFDomainCreation-SageMakerExecutionRole-12TMZ4EI9FOSA',\n", " 'ProcessingInputs': [{'InputName': 'input-1',\n", " 'AppManaged': False,\n", " 'S3Input': {'S3Uri': {'Get': 'Steps.Abalone-Train.ModelArtifacts.S3ModelArtifacts'},\n", " 'LocalPath': '/opt/ml/processing/model',\n", " 'S3DataType': 'S3Prefix',\n", " 'S3InputMode': 'File',\n", " 'S3DataDistributionType': 'FullyReplicated',\n", " 'S3CompressionType': 'None'}},\n", " {'InputName': 'input-2',\n", " 'AppManaged': False,\n", " 'S3Input': {'S3Uri': {'Get': \"Steps.Abalone-Preprocess.ProcessingOutputConfig.Outputs['test'].S3Output.S3Uri\"},\n", " 'LocalPath': '/opt/ml/processing/output/test',\n", " 'S3DataType': 'S3Prefix',\n", " 'S3InputMode': 'File',\n", " 'S3DataDistributionType': 'FullyReplicated',\n", " 'S3CompressionType': 'None'}},\n", " {'InputName': 'code',\n", " 'AppManaged': False,\n", " 'S3Input': {'S3Uri': 's3://sagemaker-us-east-1-681580474417/AbalonePipeline-SelectExec/code/af3d0ac9a2a46ba2e53409799b1c4dbe/evaluation.py',\n", " 'LocalPath': '/opt/ml/processing/input/code',\n", " 'S3DataType': 'S3Prefix',\n", " 'S3InputMode': 'File',\n", " 'S3DataDistributionType': 'FullyReplicated',\n", " 'S3CompressionType': 'None'}}],\n", " 'ProcessingOutputConfig': {'Outputs': [{'OutputName': 'evaluation',\n", " 'AppManaged': False,\n", " 'S3Output': {'S3Uri': 's3://sagemaker-us-east-1-681580474417/abalone-postproc-2023-07-25-20-12-29-933/output/evaluation',\n", " 'LocalPath': '/opt/ml/processing/evaluation',\n", " 'S3UploadMode': 'EndOfJob'}}]}},\n", " 'CacheConfig': {'Enabled': True, 'ExpireAfter': 'T12H'},\n", " 'PropertyFiles': [{'PropertyFileName': 'EvaluationReport',\n", " 'OutputName': 'evaluation',\n", " 'FilePath': 'evaluation.json'}]},\n", " {'Name': 'Abalone-MSECheck',\n", " 'Type': 'Condition',\n", " 'Arguments': {'Conditions': [{'Type': 'LessThanOrEqualTo',\n", " 'LeftValue': {'Std:JsonGet': {'PropertyFile': {'Get': 'Steps.Abalone-Evaluate.PropertyFiles.EvaluationReport'},\n", " 'Path': 'regression_metrics.mse.value'}},\n", " 'RightValue': {'Get': 'Parameters.MseThreshold'}}],\n", " 'IfSteps': [{'Name': 'Abalone-ModelRegister-RegisterModel',\n", " 'Type': 'RegisterModel',\n", " 'Arguments': {'ModelPackageGroupName': 'Abalone-Models',\n", " 'ModelMetrics': {'ModelQuality': {'Statistics': {'ContentType': 'application/json',\n", " 'S3Uri': 's3://sagemaker-us-east-1-681580474417/abalone-postproc-2023-07-25-20-12-29-933/output/evaluation/evaluation.json'}},\n", " 'Bias': {},\n", " 'Explainability': {}},\n", " 'InferenceSpecification': {'Containers': [{'Image': '683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-xgboost:1.0-1-cpu-py3',\n", " 'Environment': {},\n", " 'ModelDataUrl': {'Get': 'Steps.Abalone-Train.ModelArtifacts.S3ModelArtifacts'}}],\n", " 'SupportedContentTypes': ['text/csv'],\n", " 'SupportedResponseMIMETypes': ['text/csv'],\n", " 'SupportedRealtimeInferenceInstanceTypes': ['ml.t2.medium',\n", " 'ml.m5.xlarge'],\n", " 'SupportedTransformInstanceTypes': ['ml.m5.xlarge']},\n", " 'ModelApprovalStatus': {'Get': 'Parameters.ApprovalStatus'}}},\n", " {'Name': 'Abalone-CreateModel-CreateModel',\n", " 'Type': 'Model',\n", " 'Arguments': {'ExecutionRoleArn': 'arn:aws:iam::681580474417:role/SageMakerCFDomainCreation-SageMakerExecutionRole-12TMZ4EI9FOSA',\n", " 'PrimaryContainer': {'Image': '683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-xgboost:1.0-1-cpu-py3',\n", " 'Environment': {},\n", " 'ModelDataUrl': {'Get': 'Steps.Abalone-Train.ModelArtifacts.S3ModelArtifacts'}}}}],\n", " 'ElseSteps': [{'Name': 'Abalone-FailNotify',\n", " 'Type': 'Fail',\n", " 'Arguments': {'ErrorMessage': {'Std:Join': {'On': ' ',\n", " 'Values': ['Execution failed due to MSE >',\n", " {'Get': 'Parameters.MseThreshold'}]}}}}]}}]}" ] }, "execution_count": 31, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import json\n", "\n", "definition = json.loads(pipeline.definition())\n", "definition" ] }, { "cell_type": "code", "execution_count": 32, "id": "0c2aca96-1cd7-4410-aa26-dc22f9689f1e", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "{'PipelineArn': 'arn:aws:sagemaker:us-east-1:681580474417:pipeline/AbalonePipeline-SelectExec',\n", " 'ResponseMetadata': {'RequestId': '18bfd020-370d-4e95-ba59-b674c77832b8',\n", " 'HTTPStatusCode': 200,\n", " 'HTTPHeaders': {'x-amzn-requestid': '18bfd020-370d-4e95-ba59-b674c77832b8',\n", " 'content-type': 'application/x-amz-json-1.1',\n", " 'content-length': '94',\n", " 'date': 'Tue, 25 Jul 2023 20:12:33 GMT'},\n", " 'RetryAttempts': 0}}" ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pipeline.upsert(role_arn=role)" ] }, { "cell_type": "code", "execution_count": 33, "id": "d5267f20-8875-457a-b900-5e4d069623c0", "metadata": { "tags": [] }, "outputs": [], "source": [ "execution = pipeline.start()" ] }, { "cell_type": "code", "execution_count": 34, "id": "289f8ff5-1122-4203-a7f5-20dfaa70f264", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "{'PipelineArn': 'arn:aws:sagemaker:us-east-1:681580474417:pipeline/AbalonePipeline-SelectExec',\n", " 'PipelineExecutionArn': 'arn:aws:sagemaker:us-east-1:681580474417:pipeline/AbalonePipeline-SelectExec/execution/is6ph1fp4go8',\n", " 'PipelineExecutionDisplayName': 'execution-1690315954896',\n", " 'PipelineExecutionStatus': 'Executing',\n", " 'PipelineExperimentConfig': {'ExperimentName': 'AbaloneExperimentsPipeline',\n", " 'TrialName': 'is6ph1fp4go8'},\n", " 'CreationTime': datetime.datetime(2023, 7, 25, 20, 12, 34, 781000, tzinfo=tzlocal()),\n", " 'LastModifiedTime': datetime.datetime(2023, 7, 25, 20, 12, 34, 781000, tzinfo=tzlocal()),\n", " 'CreatedBy': {'UserProfileArn': 'arn:aws:sagemaker:us-east-1:681580474417:user-profile/d-biwxq9wovumj/ds-user',\n", " 'UserProfileName': 'ds-user',\n", " 'DomainId': 'd-biwxq9wovumj'},\n", " 'LastModifiedBy': {'UserProfileArn': 'arn:aws:sagemaker:us-east-1:681580474417:user-profile/d-biwxq9wovumj/ds-user',\n", " 'UserProfileName': 'ds-user',\n", " 'DomainId': 'd-biwxq9wovumj'},\n", " 'ResponseMetadata': {'RequestId': '8e12d252-e17c-456e-ba25-88c42feb477f',\n", " 'HTTPStatusCode': 200,\n", " 'HTTPHeaders': {'x-amzn-requestid': '8e12d252-e17c-456e-ba25-88c42feb477f',\n", " 'content-type': 'application/x-amz-json-1.1',\n", " 'content-length': '821',\n", " 'date': 'Tue, 25 Jul 2023 20:12:35 GMT'},\n", " 'RetryAttempts': 0}}" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "execution.describe()" ] }, { "cell_type": "code", "execution_count": 35, "id": "3a666e2f-71d9-430b-bf8f-2fa2a345c6b1", "metadata": { "tags": [] }, "outputs": [], "source": [ "execution.wait()" ] }, { "cell_type": "markdown", "id": "baab933d-ecc4-488a-b2a4-92a77a4040a5", "metadata": { "tags": [] }, "source": [ "# Selective Execution" ] }, { "cell_type": "code", "execution_count": 36, "id": "c1428589-ddd5-496c-a2b4-9f3c19e652a7", "metadata": { "tags": [] }, "outputs": [], "source": [ "from sagemaker.workflow.selective_execution_config import SelectiveExecutionConfig" ] }, { "cell_type": "markdown", "id": "e8712e01-e68c-4ade-bb07-698e84f0ec6e", "metadata": { "tags": [] }, "source": [ "## Setup" ] }, { "cell_type": "code", "execution_count": 37, "id": "562a85e3-090f-4d68-8240-2e340ce288ff", "metadata": { "tags": [] }, "outputs": [], "source": [ "pipeline_name = \"AbalonePipeline-SelectExec\"\n", "\n", "pipeline_arn = Pipeline(name=pipeline_name)\n", "\n", "# filter for succesfull executions only\n", "pipeline_executions = [\n", " _exec\n", " for _exec in pipeline_arn.list_executions()[\"PipelineExecutionSummaries\"]\n", " if _exec[\"PipelineExecutionStatus\"] == \"Succeeded\"\n", "]" ] }, { "cell_type": "code", "execution_count": 38, "id": "36b2f3b9-c360-4823-8654-b0fccaa68450", "metadata": { "tags": [] }, "outputs": [], "source": [ "# get the last successful execution\n", "latest_pipeline_arn = pipeline_executions[0][\"PipelineExecutionArn\"]" ] }, { "cell_type": "code", "execution_count": 39, "id": "b2df32c6-6fd1-4f66-8707-6393df1b8786", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "'arn:aws:sagemaker:us-east-1:681580474417:pipeline/AbalonePipeline-SelectExec/execution/is6ph1fp4go8'" ] }, "execution_count": 39, "metadata": {}, "output_type": "execute_result" } ], "source": [ "latest_pipeline_arn" ] }, { "cell_type": "code", "execution_count": 40, "id": "f5342572-9651-4365-b07a-cbfe082d92d3", "metadata": { "tags": [] }, "outputs": [], "source": [ "# list pipeline steps\n", "execution_steps = sm_client.list_pipeline_execution_steps(PipelineExecutionArn=latest_pipeline_arn)[\n", " \"PipelineExecutionSteps\"\n", "]" ] }, { "cell_type": "code", "execution_count": 41, "id": "ad2c45f8-a6d0-4dd7-a3e5-e7f70fa9a9c4", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "[{'StepName': 'Abalone-Preprocess',\n", " 'StartTime': datetime.datetime(2023, 7, 25, 20, 12, 35, 695000, tzinfo=tzlocal()),\n", " 'EndTime': datetime.datetime(2023, 7, 25, 20, 17, 1, 128000, tzinfo=tzlocal()),\n", " 'StepStatus': 'Succeeded',\n", " 'AttemptCount': 0,\n", " 'Metadata': {'ProcessingJob': {'Arn': 'arn:aws:sagemaker:us-east-1:681580474417:processing-job/pipelines-is6ph1fp4go8-Abalone-Preprocess-vRZZNaYdbc'}}},\n", " {'StepName': 'Abalone-Train',\n", " 'StartTime': datetime.datetime(2023, 7, 25, 20, 17, 2, 192000, tzinfo=tzlocal()),\n", " 'EndTime': datetime.datetime(2023, 7, 25, 20, 19, 33, 989000, tzinfo=tzlocal()),\n", " 'StepStatus': 'Succeeded',\n", " 'AttemptCount': 0,\n", " 'Metadata': {'TrainingJob': {'Arn': 'arn:aws:sagemaker:us-east-1:681580474417:training-job/pipelines-is6ph1fp4go8-Abalone-Train-5F3a1Eb2Zd'}}},\n", " {'StepName': 'Abalone-Evaluate',\n", " 'StartTime': datetime.datetime(2023, 7, 25, 20, 19, 34, 645000, tzinfo=tzlocal()),\n", " 'EndTime': datetime.datetime(2023, 7, 25, 20, 24, 4, 114000, tzinfo=tzlocal()),\n", " 'StepStatus': 'Succeeded',\n", " 'AttemptCount': 0,\n", " 'Metadata': {'ProcessingJob': {'Arn': 'arn:aws:sagemaker:us-east-1:681580474417:processing-job/pipelines-is6ph1fp4go8-Abalone-Evaluate-3Ntw7qvw1A'}}},\n", " {'StepName': 'Abalone-MSECheck',\n", " 'StartTime': datetime.datetime(2023, 7, 25, 20, 24, 4, 650000, tzinfo=tzlocal()),\n", " 'EndTime': datetime.datetime(2023, 7, 25, 20, 24, 4, 863000, tzinfo=tzlocal()),\n", " 'StepStatus': 'Succeeded',\n", " 'AttemptCount': 0,\n", " 'Metadata': {'Condition': {'Outcome': 'True'}}},\n", " {'StepName': 'Abalone-ModelRegister-RegisterModel',\n", " 'StartTime': datetime.datetime(2023, 7, 25, 20, 24, 5, 676000, tzinfo=tzlocal()),\n", " 'EndTime': datetime.datetime(2023, 7, 25, 20, 24, 6, 529000, tzinfo=tzlocal()),\n", " 'StepStatus': 'Succeeded',\n", " 'AttemptCount': 0,\n", " 'Metadata': {'RegisterModel': {'Arn': 'arn:aws:sagemaker:us-east-1:681580474417:model-package/abalone-models/14'}}},\n", " {'StepName': 'Abalone-CreateModel-CreateModel',\n", " 'StartTime': datetime.datetime(2023, 7, 25, 20, 24, 5, 676000, tzinfo=tzlocal()),\n", " 'EndTime': datetime.datetime(2023, 7, 25, 20, 24, 7, 178000, tzinfo=tzlocal()),\n", " 'StepStatus': 'Succeeded',\n", " 'AttemptCount': 0,\n", " 'Metadata': {'Model': {'Arn': 'arn:aws:sagemaker:us-east-1:681580474417:model/pipelines-is6ph1fp4go8-abalone-createmodel--m2fs0kxign'}}}]" ] }, "execution_count": 41, "metadata": {}, "output_type": "execute_result" } ], "source": [ "execution_steps[::-1]" ] }, { "cell_type": "code", "execution_count": 42, "id": "47557f03-ffdb-4580-b020-956fa29e2374", "metadata": { "tags": [] }, "outputs": [], "source": [ "# list all configureable pipeline parameters - params can be altered during selective execution\n", "parameters = sm_client.list_pipeline_parameters_for_execution(\n", " PipelineExecutionArn=latest_pipeline_arn\n", ")[\"PipelineParameters\"]" ] }, { "cell_type": "code", "execution_count": 43, "id": "572a1abd-01bb-4fac-9443-7e5249073d22", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "[{'Name': 'XGBNumRounds', 'Value': '50'},\n", " {'Name': 'XGBSubSample', 'Value': '0.7'},\n", " {'Name': 'XGBGamma', 'Value': '4'},\n", " {'Name': 'TrainingInstanceCount', 'Value': '1'},\n", " {'Name': 'XGBMinChildWeight', 'Value': '6'},\n", " {'Name': 'XGBETA', 'Value': '0.2'},\n", " {'Name': 'ApprovalStatus', 'Value': 'PendingManualApproval'},\n", " {'Name': 'ProcessingInstanceCount', 'Value': '1'},\n", " {'Name': 'ProcessingInstanceType', 'Value': 'ml.c5.xlarge'},\n", " {'Name': 'MseThreshold', 'Value': '6.0'},\n", " {'Name': 'ModelPath',\n", " 'Value': 's3://sagemaker-us-east-1-681580474417/Abalone/models/'},\n", " {'Name': 'XGBMaxDepth', 'Value': '5'},\n", " {'Name': 'TrainingInstanceType', 'Value': 'ml.c5.xlarge'},\n", " {'Name': 'InputData',\n", " 'Value': 's3://sagemaker-us-east-1-681580474417/sample-dataset/abalone/abalone.csv'}]" ] }, "execution_count": 43, "metadata": {}, "output_type": "execute_result" } ], "source": [ "parameters" ] }, { "cell_type": "markdown", "id": "56d17938-66c5-4457-8aee-852068e2ec41", "metadata": {}, "source": [ "## Selective Execution Scenarios" ] }, { "cell_type": "markdown", "id": "81f1abe2-1d0e-4605-89dc-1e44f0f652fe", "metadata": {}, "source": [ "### Execute Single Pipeline Step with Changes to Parameters" ] }, { "cell_type": "markdown", "id": "f63440eb-96b1-4eee-af2c-8550c4febff9", "metadata": {}, "source": [ "I would like to retrain the model in hopes to get a better performant model with different hyper parameters without retraining the whole pipeline flow, this is useful in scenarios where a DS would prefer to just test custom hyper parameters before executing the full pipeline flow" ] }, { "cell_type": "code", "execution_count": 44, "id": "daae6090-470d-46d9-9a1d-b7ae8a6bfa18", "metadata": { "tags": [] }, "outputs": [], "source": [ "selective_execution_config = SelectiveExecutionConfig(\n", " source_pipeline_execution_arn=pipeline_executions[0][\"PipelineExecutionArn\"],\n", " selected_steps=[\"Abalone-Train\"],\n", ")" ] }, { "cell_type": "code", "execution_count": 45, "id": "ec964a47-fa5a-401d-80b5-0f5156bd2dcb", "metadata": { "tags": [] }, "outputs": [], "source": [ "select_execution = pipeline_arn.start(\n", " selective_execution_config=selective_execution_config,\n", " parameters={\n", " \"XGBNumRounds\": 120,\n", " \"XGBSubSample\": 0.9,\n", " \"XGBGamma\": 2,\n", " \"XGBMinChildWeight\": 4,\n", " \"XGBETA\": 0.25,\n", " \"XGBMaxDepth\": 12,\n", " },\n", ")" ] }, { "cell_type": "markdown", "id": "f9516d60-1be4-47c5-b71a-083a96870349", "metadata": {}, "source": [ "Before:\n", "\n", "![Before Single Step](./media/BeforeSingleStep.jpg)\n", "\n", "After\n", "\n", "![After Single Step](./media/AfterSingleStep.jpg)" ] }, { "cell_type": "markdown", "id": "f2f577fd-afcc-4b9e-b357-2907e99365c4", "metadata": { "tags": [] }, "source": [ "### Execute Multiple Contiguous Pipeline Steps or the Entire Pipeline Manually" ] }, { "cell_type": "code", "execution_count": 46, "id": "92543458-21df-4539-ab63-4df1e4f7e8ea", "metadata": { "tags": [] }, "outputs": [], "source": [ "selective_execution_config = SelectiveExecutionConfig(\n", " source_pipeline_execution_arn=latest_pipeline_arn,\n", " selected_steps=[\"Abalone-Train\", \"Abalone-Evaluate\", \"Abalone-MSECheck\"],\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "73c09375-0782-4156-a7ba-f138a8bb74a5", "metadata": { "tags": [] }, "outputs": [], "source": [ "select_execution = pipeline_arn.start(\n", " selective_execution_config=selective_execution_config,\n", " parameters={\n", " \"XGBNumRounds\": 120,\n", " \"XGBSubSample\": 0.9,\n", " \"XGBGamma\": 2,\n", " \"XGBMinChildWeight\": 4,\n", " \"XGBETA\": 0.25,\n", " \"XGBMaxDepth\": 12,\n", " \"MseThreshold\": 6,\n", " },\n", ")" ] }, { "cell_type": "markdown", "id": "596da7bd-61ba-474a-a129-c4e970b02989", "metadata": {}, "source": [ "Execute Contiguous Blocks of the Pipeline\n", "\n", "\n", "![Multiple Steps](./media/MultipleSteps.jpg)\n", "\n", "\n", "OR Execute the full pipeline \n", "\n", "![Full Steps](./media/FullPipeline.jpg)" ] }, { "cell_type": "markdown", "id": "98a9d988-11a2-432c-b3df-67395eb8e3e1", "metadata": {}, "source": [ "## Notebook CI Test Results\n", "\n", "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n", "\n", "\n", "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-east-1/sagemaker-pipelines|selective-step-execution|Selective_Execution_Abalone.ipynb)\n", "\n", "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-east-2/sagemaker-pipelines|selective-step-execution|Selective_Execution_Abalone.ipynb)\n", "\n", "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-west-1/sagemaker-pipelines|selective-step-execution|Selective_Execution_Abalone.ipynb)\n", "\n", "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ca-central-1/sagemaker-pipelines|selective-step-execution|Selective_Execution_Abalone.ipynb)\n", "\n", "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/sa-east-1/sagemaker-pipelines|selective-step-execution|Selective_Execution_Abalone.ipynb)\n", "\n", "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-west-1/sagemaker-pipelines|selective-step-execution|Selective_Execution_Abalone.ipynb)\n", "\n", "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-west-2/sagemaker-pipelines|selective-step-execution|Selective_Execution_Abalone.ipynb)\n", "\n", "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-west-3/sagemaker-pipelines|selective-step-execution|Selective_Execution_Abalone.ipynb)\n", "\n", "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-central-1/sagemaker-pipelines|selective-step-execution|Selective_Execution_Abalone.ipynb)\n", "\n", "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-north-1/sagemaker-pipelines|selective-step-execution|Selective_Execution_Abalone.ipynb)\n", "\n", "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-southeast-1/sagemaker-pipelines|selective-step-execution|Selective_Execution_Abalone.ipynb)\n", "\n", "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-southeast-2/sagemaker-pipelines|selective-step-execution|Selective_Execution_Abalone.ipynb)\n", "\n", "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-northeast-1/sagemaker-pipelines|selective-step-execution|Selective_Execution_Abalone.ipynb)\n", "\n", "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-northeast-2/sagemaker-pipelines|selective-step-execution|Selective_Execution_Abalone.ipynb)\n", "\n", "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-south-1/sagemaker-pipelines|selective-step-execution|Selective_Execution_Abalone.ipynb)\n" ] }, { "cell_type": "code", "execution_count": null, "id": "ace16482-c9cf-4134-8cf9-290b3221d8f6", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "availableInstances": [ { "_defaultOrder": 0, "_isFastLaunch": true, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 4, "name": "ml.t3.medium", "vcpuNum": 2 }, { "_defaultOrder": 1, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.t3.large", "vcpuNum": 2 }, { "_defaultOrder": 2, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.t3.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 3, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.t3.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 4, "_isFastLaunch": true, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.m5.large", "vcpuNum": 2 }, { "_defaultOrder": 5, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.m5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 6, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.m5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 7, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.m5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 8, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.m5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 9, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.m5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 10, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.m5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 11, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.m5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 12, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.m5d.large", "vcpuNum": 2 }, { "_defaultOrder": 13, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.m5d.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 14, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.m5d.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 15, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.m5d.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 16, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.m5d.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 17, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.m5d.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 18, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.m5d.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 19, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.m5d.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 20, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": true, "memoryGiB": 0, "name": "ml.geospatial.interactive", "supportedImageNames": [ "sagemaker-geospatial-v1-0" ], "vcpuNum": 0 }, { "_defaultOrder": 21, "_isFastLaunch": true, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 4, "name": "ml.c5.large", "vcpuNum": 2 }, { "_defaultOrder": 22, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.c5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 23, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.c5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 24, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.c5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 25, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 72, "name": "ml.c5.9xlarge", "vcpuNum": 36 }, { "_defaultOrder": 26, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 96, "name": "ml.c5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 27, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 144, "name": "ml.c5.18xlarge", "vcpuNum": 72 }, { "_defaultOrder": 28, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.c5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 29, "_isFastLaunch": true, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.g4dn.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 30, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.g4dn.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 31, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.g4dn.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 32, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.g4dn.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 33, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.g4dn.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 34, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.g4dn.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 35, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 61, "name": "ml.p3.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 36, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 244, "name": "ml.p3.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 37, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 488, "name": "ml.p3.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 38, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 768, "name": "ml.p3dn.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 39, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.r5.large", "vcpuNum": 2 }, { "_defaultOrder": 40, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.r5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 41, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.r5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 42, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.r5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 43, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.r5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 44, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.r5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 45, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 512, "name": "ml.r5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 46, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 768, "name": "ml.r5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 47, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.g5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 48, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.g5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 49, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.g5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 50, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.g5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 51, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.g5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 52, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.g5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 53, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.g5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 54, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 768, "name": "ml.g5.48xlarge", "vcpuNum": 192 }, { "_defaultOrder": 55, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 1152, "name": "ml.p4d.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 56, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 1152, "name": "ml.p4de.24xlarge", "vcpuNum": 96 } ], "instance_type": "ml.t3.medium", "kernelspec": { "display_name": "Python 3 (Data Science 3.0)", "language": "python", "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-data-science-310-v1" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.6" }, "toc-autonumbering": false, "toc-showcode": false, "toc-showmarkdowntxt": false }, "nbformat": 4, "nbformat_minor": 5 }