{ "cells": [ { "cell_type": "markdown", "id": "fd591cb1-84c5-42d7-a7b8-44c58df18952", "metadata": { "tags": [] }, "source": [ "## Module 4: Build End-to-end Machine Learning Pipeline on SageMaker\n", "\n", "MLOps (Machine Learning Operations) is important because it enables the seamless deployment, monitoring, and management of machine learning models in production. It involves a combination of DevOps, DataOps, and ML engineering practices to build and deploy machine learning models at scale.\n", "\n", "One important aspect of MLOps is the use of an orchestration pipeline, which helps to automate the process of building, testing, and deploying machine learning models. An orchestration pipeline provides a framework for managing the entire machine learning workflow, from data preparation to model deployment and monitoring.\n", "\n", "SageMaker pipeline provides a fully-managed continuous integration and continuous delivery (CI/CD) service for building, training, and deploying machine learning models. It helps address some of the operational challenges in MLOps by providing a visual interface to create and manage machine learning workflows, automating common tasks like data preprocessing and model training, and enabling easy model deployment and monitoring.\n", "\n", "In this module, we are going to convert the manually executed notebook in module 3 into a fully automated ML pipeline using SageMaker Pipelines.\n", "\n", "![Pipeline Image](statics/module_04_pipeline01.png)\n", "\n", "**If you DID NOT run the previous modules, please run [0_setup.ipynb notebook](0_setup.ipynb) first before running this notebook**\n", "\n", "**This Demo is optimized for SageMaker Studio using Studio notebook in Data Science Kernel**" ] }, { "cell_type": "code", "execution_count": null, "id": "7980e9bf-c90f-4ce5-b44f-6d23f6fb0dd1", "metadata": { "tags": [] }, "outputs": [], "source": [ "!pip install -Uq pip --quiet\n", "\n", "!pip install -Uq awswrangler sagemaker boto3 --quiet" ] }, { "cell_type": "markdown", "id": "30f58d90-b59b-49b9-ae5a-8885f27784ca", "metadata": {}, "source": [ "Import Libraries we need" ] }, { "cell_type": "code", "execution_count": null, "id": "83d23a42-b298-4ea7-a87b-0d2e6577b2e8", "metadata": { "tags": [] }, "outputs": [], "source": [ "import json\n", "import boto3\n", "import pandas as pd\n", "import string\n", "\n", "from sagemaker.processing import (\n", " ProcessingInput,\n", " ProcessingOutput,\n", ")\n", "from sagemaker.sklearn.processing import SKLearnProcessor\n", "from sagemaker.xgboost import XGBoostProcessor\n", "\n", "from sagemaker.workflow.steps import (\n", " ProcessingStep, \n", " TrainingStep, \n", " CacheConfig, \n", " CreateModelStep\n", ")\n", "\n", "import sagemaker\n", "\n", "from sagemaker.workflow.conditions import (\n", " ConditionGreaterThanOrEqualTo,\n", ")\n", "from sagemaker.workflow.condition_step import ConditionStep, JsonGet\n", "# from sagemaker.workflow.functions import JsonGet\n", "\n", "from sagemaker.xgboost.estimator import XGBoost\n", "from sagemaker.workflow.pipeline import Pipeline\n", "from sagemaker.workflow.properties import PropertyFile\n", "from sagemaker.model_metrics import MetricsSource, ModelMetrics\n", "from sagemaker.workflow.step_collections import RegisterModel\n", "from sagemaker.workflow.pipeline_context import PipelineSession" ] }, { "cell_type": "markdown", "id": "7bf1bd33-15f9-4a11-8dd3-5d95c17ac594", "metadata": {}, "source": [ "Define global parameters" ] }, { "cell_type": "code", "execution_count": null, "id": "06b9acf4-396a-4318-a2b4-79a8149cbe5e", "metadata": { "tags": [] }, "outputs": [], "source": [ "sagemaker_session = sagemaker.Session()\n", "\n", "region = sagemaker_session.boto_region_name\n", "sagemaker_role = sagemaker.get_execution_role()\n", "\n", "bucket = sagemaker_session.default_bucket()\n", "\n", "s3_client = boto3.client(\"s3\", region_name=region)\n", "sagemaker_client = boto3.client(\"sagemaker\")\n", "\n", "%store -r fg_name\n", "\n", "%store -r prefix" ] }, { "cell_type": "code", "execution_count": null, "id": "86a1be93-90f1-46a0-ba34-824b2c5428aa", "metadata": { "tags": [] }, "outputs": [], "source": [ "# ======> variables used for parameterizing the notebook run\n", "flow_instance_count = 1\n", "flow_instance_type = \"ml.m5.4xlarge\"\n", "\n", "dataprep_instance_count = 1\n", "dataprep_instance_type = \"ml.m5.xlarge\"\n", "\n", "train_instance_count = 1\n", "train_instance_type = \"ml.m5.xlarge\"\n", "\n", "eval_instance_count = 1\n", "eval_instance_type = \"ml.m5.xlarge\"\n", "\n", "deploy_instance_count = 1\n", "deploy_instance_type = \"ml.m5.xlarge\"\n", "\n", "cache_config = CacheConfig(enable_caching=True, expire_after=\"PT1H\")" ] }, { "cell_type": "markdown", "id": "b1b148dc-671d-4a9a-ad1a-5bfe1c6c0529", "metadata": {}, "source": [ "## Architecture: Create a SageMaker Pipeline to Automate All the Steps from Data Prep to Model Deployment\n", "----\n", "\n", "## Breaking down the process into steps\n", "\n", "- [Step 1: Data Wrangler Preprocessing Step](#Step-1:-Data-Wrangler-Preprocessing-Step)\n", "- [Step 2: Create Dataset and Train/Test Split](#Step-2:-Create-Dataset-and-Train/Test-Split)\n", "- [Step 3: Train XGBoost Model](#Step-3:-Train-XGBoost-Model)\n", "- [Step 4: Model Evaluation](#Step-4:-Model-Evaluation)\n", "- [Step 5: Register Model](#Step-5:-Register-Model)\n", "- [Step 6: Deploy Model](#Step-6:-Deploy-Model)\n", "- [Step 7: Combine and Run the Pipeline Steps](#Step-7:-Combine-and-Run-the-Pipeline-Steps)" ] }, { "cell_type": "markdown", "id": "b4384c42-4b64-4d15-8226-c0a6c9d4495e", "metadata": {}, "source": [ "### Define Pipeline Parameters\n", "Pipeline parameters are variables that can be defined and used within a SageMaker pipeline to enable dynamic configuration of pipeline components and their execution." ] }, { "cell_type": "code", "execution_count": null, "id": "b3d81afe-822b-45a2-bf40-6901695294a2", "metadata": { "tags": [] }, "outputs": [], "source": [ "from sagemaker.workflow.parameters import ParameterInteger, ParameterFloat, ParameterString\n", "\n", "train_instance_param = ParameterString(\n", " name=\"TrainingInstance\",\n", " default_value=\"ml.m5.xlarge\",\n", ")\n", "\n", "model_approval_status = ParameterString(\n", " name=\"ModelApprovalStatus\", default_value=\"PendingManualApproval\"\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "6c95eed7-edc6-48d5-91ca-eb439d12b87b", "metadata": { "tags": [] }, "outputs": [], "source": [ "cell5g_s3_key = f\"{prefix}/data/raw/5gcell.csv\"\n", "\n", "s3_client.upload_file(\n", " Filename=\"data/5gcell.csv\", Bucket=bucket, Key=cell5g_s3_key\n", ")" ] }, { "cell_type": "markdown", "id": "ce1ab916-4af1-4e0e-8155-39d079b18f03", "metadata": {}, "source": [ "## Step 1: Data Wrangler Preprocessing Step" ] }, { "cell_type": "markdown", "id": "1ab60773-9b4c-4612-b1bf-2c375a979e9b", "metadata": {}, "source": [ "### Update attributes within the `.flow` file \n", "Data Wrangler will generate a .flow file. It contains a reference to an S3 bucket used during the Wrangling. This may be different from the one you have as a default in this notebook eg if the Wrangling was done by someone else, you will probably not have access to their bucket and you now need to point to your own S3 bucket so you can actually load the .flow file into Data Wrangler or access the data.\n", "\n", "After running the cell below you can open the `5gcell.flow` file and export the data to S3" ] }, { "cell_type": "code", "execution_count": null, "id": "59cbce1c-a5f7-4bd0-9b7a-3e79b7c11ba0", "metadata": { "tags": [] }, "outputs": [], "source": [ "template_file = \"pipeline/5gcell_flow_template_2\"\n", "\n", "with open(template_file, \"r\") as f:\n", " template = f.read()\n", " template = template.replace('${bucket}', bucket)\n", " template = template.replace('${prefix}', prefix)\n", " flow = json.loads(template)\n", "\n", "flow_file = \"pipeline/5gcell.flow\"\n", "with open(flow_file, \"w\") as f:\n", " json.dump(flow, f)" ] }, { "cell_type": "markdown", "id": "650a51df-7a87-4325-a6a5-f8a7de85b100", "metadata": {}, "source": [ "#### Upload flow to S3\n", "This will become an input to the first step and, as such, needs to be in S3." ] }, { "cell_type": "code", "execution_count": null, "id": "ca55066e-febb-4219-91f6-91df725e5b89", "metadata": { "tags": [] }, "outputs": [], "source": [ "flow_key = f\"{prefix}/dataprep-notebooks/{flow_file.split('/')[-1]}\"\n", "\n", "s3_client.upload_file(\n", " Filename=flow_file, Bucket=bucket, Key=flow_key\n", ")\n", "flow_uri = f\"s3://{bucket}/{flow_key}\"\n", "print(f\"Flow file uploaded: {flow_uri}\")" ] }, { "cell_type": "markdown", "id": "c0473f60-98e4-4c19-914d-6ec184f65787", "metadata": {}, "source": [ "#### Define the first Data Wrangler step's inputs" ] }, { "cell_type": "code", "execution_count": null, "id": "60fab2c1-344e-4072-bef7-37e7ba470572", "metadata": { "tags": [] }, "outputs": [], "source": [ "with open(flow_file, \"r\") as f:\n", " flow = json.load(f)\n", "\n", "flow_step_inputs = []\n", "\n", "#input data location in the process container\n", "processing_dir='/opt/ml/processing'\n", "\n", "# flow file contains the code for each transformation\n", "flow_file_input = sagemaker.processing.ProcessingInput(\n", " source=flow_uri, destination=f\"{processing_dir}/flow\", input_name=\"flow\"\n", ")\n", "\n", "flow_step_inputs.append(flow_file_input)\n", "\n", "# parse the flow file for S3 inputs to Data Wranger job\n", "for node in flow[\"nodes\"]:\n", " if \"dataset_definition\" in node[\"parameters\"]:\n", " data_def = node[\"parameters\"][\"dataset_definition\"]\n", " name = data_def[\"name\"]\n", " s3_input = sagemaker.processing.ProcessingInput(\n", " source=data_def[\"s3ExecutionContext\"][\"s3Uri\"],\n", " destination=f\"{processing_dir}/{name}\",\n", " input_name=name,\n", " )\n", " flow_step_inputs.append(s3_input)" ] }, { "cell_type": "markdown", "id": "9556c612-960b-48c0-b3f7-3f9daf53fe65", "metadata": {}, "source": [ "#### Define outputs for first Data Wranger step" ] }, { "cell_type": "code", "execution_count": null, "id": "a87f8d10-55fa-4606-b91f-96e38ffc89bc", "metadata": { "tags": [] }, "outputs": [], "source": [ "output_name = (\n", " f\"{flow['nodes'][-1]['node_id']}.{flow['nodes'][-1]['outputs'][0]['name']}\"\n", ")\n", "\n", "flow_step_outputs = []\n", "\n", "flow_output = sagemaker.processing.ProcessingOutput(\n", " output_name=output_name,\n", " feature_store_output=sagemaker.processing.FeatureStoreOutput(feature_group_name=fg_name),\n", " app_managed=True,\n", ")\n", "\n", "flow_step_outputs.append(flow_output)" ] }, { "cell_type": "markdown", "id": "4cef78a6-d332-4535-88b5-5b7b21c225b0", "metadata": {}, "source": [ "#### Define processor and processing step" ] }, { "cell_type": "code", "execution_count": null, "id": "353581a9-6ea4-45fc-9f41-a566cc7ec48f", "metadata": { "tags": [] }, "outputs": [], "source": [ "# Pulls the latest data-wrangler container tag, i.e. \"1.x\"\n", "# The latest tested container version was \"1.11.0\"\n", "image_uri = sagemaker.image_uris.retrieve(framework=\"data-wrangler\", region=region)\n", "\n", "print(\"image_uri: {}\".format(image_uri))\n", "\n", "flow_processor = sagemaker.processing.Processor(\n", " role=sagemaker_role,\n", " image_uri=image_uri,\n", " instance_count=flow_instance_count,\n", " instance_type=flow_instance_type,\n", " max_runtime_in_seconds=86400,\n", ")\n", "\n", "# Output configuration used as processing job container arguments\n", "output_config = {output_name: {\"content_type\": \"CSV\"}}\n", "\n", "flow_step = ProcessingStep(\n", " name=\"DataWranglerProcessingStep\",\n", " processor=flow_processor,\n", " inputs=flow_step_inputs,\n", " outputs=flow_step_outputs,\n", " job_arguments=[f\"--output-config '{json.dumps(output_config)}'\"],\n", " cache_config=cache_config\n", ")" ] }, { "cell_type": "markdown", "id": "350a3dc7-56be-4daa-bfe2-cc8f2f9cef2f", "metadata": {}, "source": [ "### Step 2: Create Dataset and Train/Test Split" ] }, { "cell_type": "code", "execution_count": null, "id": "f89d401d-03c1-4fe2-9bc0-b71d6a9261c6", "metadata": { "tags": [] }, "outputs": [], "source": [ "create_dataset_script = \"pipeline/create_dataset.py\"\n", "create_dataset_key = f\"{prefix}/code/{create_dataset_script.split('/')[-1]}\"\n", "\n", "s3_client.upload_file(\n", " Filename=create_dataset_script, Bucket=bucket, Key=create_dataset_key\n", ")\n", "\n", "create_dataset_script_uri = f\"s3://{bucket}/{create_dataset_key}\"\n", "\n", "create_dataset_processor = SKLearnProcessor(\n", " framework_version=\"0.23-1\",\n", " role=sagemaker_role,\n", " instance_type=dataprep_instance_type,\n", " instance_count=dataprep_instance_count,\n", " base_job_name=f\"{prefix}-dataprep\",\n", " sagemaker_session=sagemaker_session,\n", ")\n", "\n", "create_dataset_step = ProcessingStep(\n", " name=\"CreateDataset\",\n", " processor=create_dataset_processor,\n", " outputs=[\n", " sagemaker.processing.ProcessingOutput(\n", " output_name=\"train_data\", source=\"/opt/ml/processing/output/train\"\n", " ),\n", " sagemaker.processing.ProcessingOutput(\n", " output_name=\"test_data\", source=\"/opt/ml/processing/output/test\"\n", " ),\n", " ],\n", " job_arguments=[\n", " \"--feature-group-name\",\n", " fg_name,\n", " \"--bucket-name\",\n", " bucket,\n", " \"--bucket-prefix\",\n", " prefix,\n", " \"--region\",\n", " region,\n", " ],\n", " code=create_dataset_script_uri,\n", " depends_on=[flow_step.name],\n", " cache_config=cache_config\n", ")" ] }, { "cell_type": "markdown", "id": "efc5152e-4319-4fbe-9cfc-41f01e64731d", "metadata": {}, "source": [ "### Step 3: Train XGBoost Model\n", "In this step we use the ParameterString `train_instance_param` defined at the beginning of the pipeline." ] }, { "cell_type": "code", "execution_count": null, "id": "9c2626b6-ece1-409b-8357-6bb30a0076e6", "metadata": { "tags": [] }, "outputs": [], "source": [ "hyperparameters = {\n", " \"max_depth\": \"3\",\n", " \"eta\": \"0.2\",\n", " \"objective\": \"binary:logistic\",\n", " \"num_round\": \"100\",\n", " \"region\":region\n", "}\n", "\n", "training_job_output_path = f\"s3://{bucket}/{prefix}/training_jobs\"\n", "\n", "xgb_estimator = XGBoost(\n", " entry_point=\"xgboost_starter_script.py\",\n", " source_dir=\"pipeline/code\",\n", " output_path=training_job_output_path,\n", " code_location=training_job_output_path,\n", " hyperparameters=hyperparameters,\n", " role=sagemaker_role,\n", " instance_count=train_instance_count,\n", " instance_type=train_instance_type, \n", " framework_version=\"1.5-1\",\n", ")\n", "\n", "train_step = TrainingStep(\n", " name=\"XgboostTrain\",\n", " estimator=xgb_estimator,\n", " inputs={\n", " \"train\": sagemaker.inputs.TrainingInput(\n", " s3_data=create_dataset_step.properties.ProcessingOutputConfig.Outputs[\n", " \"train_data\"\n", " ].S3Output.S3Uri\n", " )\n", " },\n", " cache_config=cache_config\n", ")" ] }, { "cell_type": "markdown", "id": "38646373-0fb6-4e6e-bb27-d77569dc8da9", "metadata": {}, "source": [ "#### Create SageMaker Model" ] }, { "cell_type": "code", "execution_count": null, "id": "f9d27e69-e358-49b5-9ddc-30a0d3aef316", "metadata": { "tags": [] }, "outputs": [], "source": [ "model = sagemaker.model.Model(\n", " name=\"XgboostAnomalyModel\",\n", " image_uri=train_step.properties.AlgorithmSpecification.TrainingImage,\n", " model_data=train_step.properties.ModelArtifacts.S3ModelArtifacts,\n", " sagemaker_session=sagemaker_session,\n", " role=sagemaker_role,\n", ")\n", "\n", "inputs = sagemaker.inputs.CreateModelInput(instance_type=deploy_instance_type)\n", "\n", "create_model_step = CreateModelStep(name=\"XgboostModel\", model=model, inputs=inputs)" ] }, { "cell_type": "markdown", "id": "5145ccfd-6dce-4ea7-852e-6245d2fd2737", "metadata": {}, "source": [ "### Step 4: Model Evaluation" ] }, { "cell_type": "code", "execution_count": null, "id": "aef36268-8dba-4dda-8152-e2e339a7be3d", "metadata": { "tags": [] }, "outputs": [], "source": [ "model_eval_script = \"pipeline/model_eval.py\"\n", "model_eval_key = f\"{prefix}/code/{model_eval_script.split('/')[-1]}\"\n", "\n", "s3_client.upload_file(\n", " Filename=model_eval_script, Bucket=bucket, Key=model_eval_key\n", ")\n", "\n", "model_eval_script_uri = f\"s3://{bucket}/{model_eval_key}\"\n", "\n", "eval_processor = XGBoostProcessor(\n", " framework_version='1.5-1',\n", " role=sagemaker_role,\n", " instance_type=eval_instance_type,\n", " instance_count=eval_instance_count,\n", " base_job_name=f'{prefix}-eval',\n", " sagemaker_session=PipelineSession(),\n", ")\n", "\n", "step_args = eval_processor.run(\n", " code=\"pipeline/model_eval.py\",\n", " inputs=[ProcessingInput(source=train_step.properties.ModelArtifacts.S3ModelArtifacts, \n", " destination=\"/opt/ml/processing/model\"),\n", " ProcessingInput(source=create_dataset_step.properties.ProcessingOutputConfig.Outputs[\"test_data\"].S3Output.S3Uri, \n", " destination=\"/opt/ml/processing/input/test\")\n", " ],\n", " outputs=[\n", " ProcessingOutput(output_name=\"evaluation\", source=\"/opt/ml/processing/evaluation\"),\n", " ],\n", " arguments=[\"--model-file\", \"model.tar.gz\", \"--cutoff-threshold\", \"0.4\"],\n", ")\n", "\n", "\n", "evaluation_report = PropertyFile(\n", " name=\"EvaluationReport\",\n", " output_name=\"evaluation\",\n", " path=\"evaluation.json\",\n", ")\n", "\n", "eval_step = ProcessingStep(\n", " name=f\"ModelEval\",\n", " step_args=step_args,\n", " property_files=[evaluation_report],\n", " cache_config=cache_config\n", ")" ] }, { "cell_type": "markdown", "id": "313441f6-5b46-4a60-86ac-aa688bbb6dea", "metadata": {}, "source": [ "### Step 5: Register Model\n", "In this step you will use the ParameterString `model_approval_status` defined at the outset of the pipeline code." ] }, { "cell_type": "code", "execution_count": null, "id": "2fecdaa2-0d96-408f-a5e5-d296d93afccc", "metadata": { "tags": [] }, "outputs": [], "source": [ "model_metrics = ModelMetrics(\n", " model_statistics=MetricsSource(\n", " s3_uri=\"{}/evaluation.json\".format(\n", " eval_step.arguments[\"ProcessingOutputConfig\"][\"Outputs\"][0][\"S3Output\"][\n", " \"S3Uri\"\n", " ]\n", " ),\n", " content_type=\"application/json\",\n", " )\n", ")\n", "\n", "register_step = RegisterModel(\n", " name=\"Xgboost\",\n", " estimator=xgb_estimator,\n", " model_data=train_step.properties.ModelArtifacts.S3ModelArtifacts,\n", " content_types=[\"text/csv\"],\n", " response_types=[\"text/csv\"],\n", " inference_instances=[\"ml.m5.xlarge\"],\n", " transform_instances=[\"ml.m5.xlarge\"],\n", " model_package_group_name=f\"{prefix}-model\",\n", " approval_status=model_approval_status,\n", " model_metrics=model_metrics,\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "5013f6f5-0ef4-4a5d-bd74-06bd7b88a082", "metadata": { "tags": [] }, "outputs": [], "source": [ "# Condition step for evaluating model quality and branching execution\n", "cond_lte = ConditionGreaterThanOrEqualTo( # You can change the condition here\n", " left=JsonGet(\n", " step=eval_step,\n", " property_file=evaluation_report,\n", " json_path=\"multiclass_classification_metrics.accuracy.value\", # This should follow the structure of your report_dict defined in the evaluate.py file.\n", " ),\n", " right=0.8, # You can change the threshold here\n", ")\n", "cond_step = ConditionStep(\n", " name=\"AccuracyCond\",\n", " conditions=[cond_lte],\n", " if_steps=[register_step],\n", " else_steps=[],\n", ")" ] }, { "cell_type": "markdown", "id": "1aa2d160-9681-4804-85eb-ac59b44f5e27", "metadata": {}, "source": [ "### Step 6: Deploy Model" ] }, { "cell_type": "code", "execution_count": null, "id": "56e9c8d2-5c5c-4ac2-b1f8-a754596c4317", "metadata": { "tags": [] }, "outputs": [], "source": [ "model_deploy_script = \"pipeline/deploy_model.py\"\n", "model_deploy_key = f\"{prefix}/code/{model_deploy_script.split('/')[-1]}\"\n", "\n", "s3_client.upload_file(\n", " Filename=model_deploy_script, Bucket=bucket, Key=model_deploy_key\n", ")\n", "\n", "model_deploy_script_uri = f\"s3://{bucket}/{model_deploy_key}\"\n", "\n", "deploy_model_processor = SKLearnProcessor(\n", " framework_version=\"0.23-1\",\n", " role=sagemaker_role,\n", " instance_type=deploy_instance_type,\n", " instance_count=deploy_instance_count,\n", " base_job_name=f\"{prefix}-deploy\",\n", " sagemaker_session=sagemaker_session,\n", ")\n", "\n", "deploy_step = ProcessingStep(\n", " name=\"DeployModel\",\n", " processor=deploy_model_processor,\n", " job_arguments=[\n", " \"--model-name\",\n", " create_model_step.properties.ModelName,\n", " \"--region\",\n", " region,\n", " \"--endpoint-instance-type\",\n", " deploy_instance_type,\n", " \"--endpoint-name\",\n", " f\"{prefix}-endpoint\",\n", " ],\n", " code=model_deploy_script_uri,\n", ")" ] }, { "cell_type": "markdown", "id": "c8adaf57-0b38-4f7b-acbb-97eb2694ce05", "metadata": {}, "source": [ "### Step 7: Combine and Run the Pipeline Steps" ] }, { "cell_type": "code", "execution_count": null, "id": "14f87753-4456-4457-9d14-e4679904e12f", "metadata": { "tags": [] }, "outputs": [], "source": [ "pipeline = Pipeline(\n", " name=prefix,\n", " parameters=[model_approval_status],\n", " steps=[\n", " flow_step,\n", " create_dataset_step,\n", " train_step,\n", " create_model_step,\n", " eval_step,\n", " cond_step,\n", " deploy_step\n", " ],\n", ")" ] }, { "cell_type": "markdown", "id": "3fa536e8-8879-45e1-9cee-186c96ab9539", "metadata": {}, "source": [ "### Submit the pipeline definition to SageMaker\n", "Note: If an existing pipeline has the same name it will be overwritten." ] }, { "cell_type": "code", "execution_count": null, "id": "147ede88-8133-49bd-abc3-7099f099eaa1", "metadata": { "tags": [] }, "outputs": [], "source": [ "pipeline.upsert(role_arn=sagemaker_role)" ] }, { "cell_type": "code", "execution_count": null, "id": "cab32fb6-0191-4648-b955-b5f6c1765cee", "metadata": { "tags": [] }, "outputs": [], "source": [ "start_response = pipeline.start()\n", "# start_response.wait(delay=60, max_attempts=500)\n", "start_response.describe()" ] }, { "cell_type": "markdown", "id": "6dd3333a-e024-4afe-bf35-c4600cba4c79", "metadata": {}, "source": [ "## Clean Up\n", "----\n", "After running the demo, you should remove the resources which were created. You can also delete all the objects in the project's S3 directory by passing the keyword argument `delete_s3_objects=True`." ] }, { "cell_type": "code", "execution_count": null, "id": "2281ac65-b5fa-4853-b870-4d320a6dc1ca", "metadata": { "tags": [] }, "outputs": [], "source": [ "from delete_pipeline import delete_pipeline_resources" ] }, { "cell_type": "code", "execution_count": null, "id": "218e5fa8-a81f-4adc-b82f-644a18af515f", "metadata": { "tags": [] }, "outputs": [], "source": [ "delete_pipeline_resources(\n", " sagemaker_boto_client=sagemaker_client,\n", " pipeline_name=prefix,\n", " mpg_name=f\"{prefix}-model\",\n", " prefix=prefix,\n", " delete_s3_objects=True,\n", " bucket_name=bucket,\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "a3ed4053-a720-4cca-87a1-be3d4781ac8d", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "availableInstances": [ { "_defaultOrder": 0, "_isFastLaunch": true, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 4, "name": "ml.t3.medium", "vcpuNum": 2 }, { "_defaultOrder": 1, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.t3.large", "vcpuNum": 2 }, { "_defaultOrder": 2, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.t3.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 3, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.t3.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 4, "_isFastLaunch": true, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.m5.large", "vcpuNum": 2 }, { "_defaultOrder": 5, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.m5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 6, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.m5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 7, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.m5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 8, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.m5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 9, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.m5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 10, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.m5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 11, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.m5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 12, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.m5d.large", "vcpuNum": 2 }, { "_defaultOrder": 13, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.m5d.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 14, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.m5d.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 15, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.m5d.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 16, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.m5d.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 17, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.m5d.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 18, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.m5d.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 19, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.m5d.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 20, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": true, "memoryGiB": 0, "name": "ml.geospatial.interactive", "supportedImageNames": [ "sagemaker-geospatial-v1-0" ], "vcpuNum": 0 }, { "_defaultOrder": 21, "_isFastLaunch": true, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 4, "name": "ml.c5.large", "vcpuNum": 2 }, { "_defaultOrder": 22, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.c5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 23, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.c5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 24, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.c5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 25, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 72, "name": "ml.c5.9xlarge", "vcpuNum": 36 }, { "_defaultOrder": 26, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 96, "name": "ml.c5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 27, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 144, "name": "ml.c5.18xlarge", "vcpuNum": 72 }, { "_defaultOrder": 28, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.c5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 29, "_isFastLaunch": true, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.g4dn.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 30, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.g4dn.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 31, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.g4dn.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 32, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.g4dn.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 33, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.g4dn.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 34, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.g4dn.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 35, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 61, "name": "ml.p3.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 36, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 244, "name": "ml.p3.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 37, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 488, "name": "ml.p3.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 38, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 768, "name": "ml.p3dn.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 39, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.r5.large", "vcpuNum": 2 }, { "_defaultOrder": 40, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.r5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 41, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.r5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 42, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.r5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 43, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.r5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 44, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.r5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 45, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 512, "name": "ml.r5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 46, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 768, "name": "ml.r5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 47, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.g5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 48, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.g5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 49, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.g5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 50, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.g5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 51, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.g5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 52, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.g5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 53, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.g5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 54, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 768, "name": "ml.g5.48xlarge", "vcpuNum": 192 } ], "instance_type": "ml.t3.medium", "kernelspec": { "display_name": "Python 3 (Data Science)", "language": "python", "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/datascience-1.0" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.10" } }, "nbformat": 4, "nbformat_minor": 5 }