{ "cells": [ { "cell_type": "markdown", "id": "ed086066-7a2c-4be4-9e95-5192e7b0d5b3", "metadata": {}, "source": [ "## Module 3: Training Custom Models Using Studio Notebook\n", "\n", "In many situations, you may choose to build a custom model when you need to tackle a unique problem or when there isn't a pre-built model that meets your needs. In such cases, building a custom model might involve selecting an appropriate algorithm, fine-tuning its parameters, and optimizing its performance through iterative experimentation. In this module we will going through following steps to build, track, deploy and monitor a custom model using Amazon SageMaker Studio Notebook.\n", "\n", "- [Step 1: Pull Data from Offline Feature Store](#Pull-data-from-offline-feature-store)\n", "- [Step 2: Train, Track, and Deploy a Xgboost Model](#Train-XGBoost-Model)\n", "- [Step 3: Train, Track, and Deploy an Isolation Forest Model](#Train-Isolation-Forest-Model)\n", "- [Step 4: Model Monitoring](#Model-Monitoring)\n", "- [Step 5: Clean Up](#Clean-up)\n", "\n", "**If you DID NOT run the previous modules, please run [0_setup.ipynb notebook](0_setup.ipynb) first before running this notebook**\n", "\n", "**This Demo is optimized for SageMaker Studio using Studio notebook in Data Science Kernel**\n", "\n" ] }, { "cell_type": "markdown", "id": "12e7f468-dbfb-4b5b-bc97-11cf9043fca3", "metadata": { "tags": [] }, "source": [ "### Setup\n", "\n", "Install required and/or update libraries" ] }, { "cell_type": "code", "execution_count": null, "id": "df25350d-9eb7-45dd-b594-b0020d93ebfd", "metadata": { "tags": [] }, "outputs": [], "source": [ "!pip install -Uq pip --quiet\n", "\n", "!pip install -Uq awswrangler sagemaker boto3 --quiet" ] }, { "cell_type": "markdown", "id": "da76413c-8691-428a-ae8c-5b7097b26e9e", "metadata": {}, "source": [ "### Import & Global Parameters" ] }, { "cell_type": "code", "execution_count": null, "id": "3372368e-1e64-4f10-867d-e76d403ed02c", "metadata": { "tags": [] }, "outputs": [], "source": [ "import boto3\n", "import sagemaker\n", "import pandas as pd\n", "\n", "sagemaker_session = sagemaker.Session()\n", "\n", "region = sagemaker_session.boto_region_name\n", "sagemaker_role = sagemaker.get_execution_role()\n", "\n", "bucket = sagemaker_session.default_bucket()\n", "\n", "s3_client = boto3.client(\"s3\", region_name=region)\n", "sagemaker_client = boto3.client(\"sagemaker\")\n", "\n", "prefix = \"telco-5g-observability\"\n", "\n", "%store region\n", "%store bucket\n", "%store sagemaker_role\n", "%store prefix" ] }, { "cell_type": "markdown", "id": "1a39b797-117b-45db-8f48-d181c2f51874", "metadata": { "tags": [] }, "source": [ "### Pull data from offline feature store\n", "----\n", "In Module 1 of this workshop, we prepared the raw data and upload the final data into an Offline Feature Store. This dataset is now cataloged in a central location for management and discovery. Now we want to extract that data and build our observability models. SageMaker feature store uses athena query to pull the data and can cast the data directly into a pandas dataframe for further processing." ] }, { "cell_type": "code", "execution_count": null, "id": "c2eb1aed-2bcb-4251-a773-688951a59de9", "metadata": { "tags": [] }, "outputs": [], "source": [ "from sagemaker.feature_store.feature_group import FeatureGroup\n", "\n", "%store -r fg_name\n", "\n", "anomaly_features = FeatureGroup(name=fg_name, sagemaker_session=sagemaker_session)\n", "\n", "query = anomaly_features.athena_query()\n", "\n", "table_name = query.table_name\n", " \n", "query_string = f\"\"\"\n", "SELECT * FROM \"{table_name}\"\n", "\"\"\"\n", "\n", "query.run(query_string=query_string, output_location=f\"s3://{bucket}/{prefix}/data/query_results\")\n", "query.wait()\n", "\n", "dataset = query.as_dataframe()\n", "\n", "dataset" ] }, { "cell_type": "markdown", "id": "12085adb-bf4e-41b9-aca2-c361af728fd8", "metadata": { "tags": [] }, "source": [ "### Train XGBoost Model\n", "----\n", "\n", "In real world, data scientist goes through hundreds of iterations to experiment with different algorithm to come up with the best model for the ML use case. Here you are going to start with a supervised learning approach and use XGboost model for our problem.\n", "\n", "To get Your features ready for XGBoost, we need to move the target varibale to the first column for our xgboost model. You will also split the data into train & test dataset to keep a holdout set to validate model performance." ] }, { "cell_type": "code", "execution_count": null, "id": "efdf60e5-b1da-4b2e-bd19-330ee48d7bb2", "metadata": { "tags": [] }, "outputs": [], "source": [ "col_order = [\"anomaly\"] + list(dataset.drop([\"location_id\", \"anomaly\", \"eventtime\", \"write_time\",\"api_invocation_time\",'is_deleted'], axis=1).columns)\n", "\n", "train = dataset.sample(frac=0.80, random_state=0)[col_order]\n", "test = dataset.drop(train.index)[col_order]" ] }, { "cell_type": "markdown", "id": "98e213e0-8814-4a3d-bebe-42bce9e5ef1e", "metadata": {}, "source": [ "Upload the training data to S3" ] }, { "cell_type": "code", "execution_count": null, "id": "01ae425b-8b66-497c-b7db-3875476f5698", "metadata": { "tags": [] }, "outputs": [], "source": [ "train.to_csv(\"data/train.csv\", index=False)\n", "key = f\"{prefix}/data/xgboost/train.csv\"\n", "\n", "s3_client.upload_file(\n", " Filename=\"data/train.csv\",\n", " Bucket=bucket,\n", " Key=key,\n", ")\n", "\n", "train_s3_path = f\"s3://{bucket}/{key}\"\n", "print(f\"training data is uploaded to {train_s3_path}\")" ] }, { "cell_type": "markdown", "id": "7b4591aa-c85a-45e9-9815-3447503e543b", "metadata": { "tags": [] }, "source": [ "#### Set the hyperparameters\n", "These are the parameters which will be sent to our training script in order to train the model. Although they are all defined as \"hyperparameters\" here, they can encompass XGBoost's [Learning Task Parameters](https://xgboost.readthedocs.io/en/latest/parameter.html#learning-task-parameters), [Tree Booster Parameters](https://xgboost.readthedocs.io/en/latest/parameter.html#parameters-for-tree-booster), or any other parameters you'd like to configure for XGBoost.\n", "\n", "#### Setup Experiment Run Context\n", "Amazon SageMaker Experiment allows data you to organize, track, compare, and evaluate experiments during the model building and training process. Experiment tracking is extremely important because it enables you to keep track of model performance and changes over time, making it easier to debug and optimize the model. It also helps in reproducing and sharing the results with others, leading to better collaboration and faster iteration. For more details reference [SageMaker documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/experiments-create.html).\n", "\n", "### Train a custom model on SageMaker\n", "When it comes to training a model on SageMaker, you start by specifying the type of instance, the framework container, and any hyperparameters you want to use. When you call `estimator.fit()`, you supply the location of your training data. SageMaker will then spin up the specified instance and download your training data onto it. In the example below, we are also supplying a custom training script. This way, SageMaker will copy the script into the container and run. This makes it easy for you to iterate your code." ] }, { "cell_type": "code", "execution_count": null, "id": "048f2863-a3d3-4949-844e-bae48e824473", "metadata": { "scrolled": true, "tags": [] }, "outputs": [], "source": [ "from sagemaker.xgboost.estimator import XGBoost\n", "from sagemaker.experiments.run import Run, load_run\n", "from sagemaker.utils import unique_name_from_base\n", "\n", "train_instance_count=1\n", "train_instance_type=\"ml.m5.xlarge\" \n", "\n", "experiment_name = unique_name_from_base(prefix)\n", "\n", "run_name = unique_name_from_base(\"xgboost-experiment\")\n", "\n", "with Run(experiment_name=experiment_name, run_name=run_name, \n", " sagemaker_session=sagemaker_session) as run:\n", " \n", " run.log_file(\"data/train.csv\", is_output=False)\n", " \n", " hyperparameters = {\n", " \"max_depth\": \"3\",\n", " \"eta\": \"0.2\",\n", " \"objective\": \"binary:logistic\",\n", " \"num_round\": \"100\",\n", " \"region\":region\n", " }\n", "\n", " xgb_estimator = XGBoost(\n", " entry_point=\"xgboost_starter_script.py\",\n", " source_dir=\"code\",\n", " hyperparameters=hyperparameters,\n", " role=sagemaker_role,\n", " instance_count=train_instance_count,\n", " instance_type=train_instance_type, \n", " framework_version=\"1.5-1\",\n", " )\n", " \n", " xgb_estimator.fit(inputs={\"train\": train_s3_path})" ] }, { "cell_type": "markdown", "id": "3b917f34-85d8-4666-b7ff-70921b86321a", "metadata": {}, "source": [ "#### Deploy model to an endpoint\n", "We are going to enable data capturing for model monitoring" ] }, { "cell_type": "code", "execution_count": null, "id": "f65dff6e-e964-4396-a338-2955e2c125fe", "metadata": { "tags": [] }, "outputs": [], "source": [ "from sagemaker.serializers import CSVSerializer\n", "from sagemaker.model_monitor import DataCaptureConfig\n", "\n", "data_capture_config = DataCaptureConfig(\n", " enable_capture=True,\n", " sampling_percentage=100,\n", " destination_s3_uri=f\"s3://{bucket}/{prefix}/monitoring/datacapture\"\n", ")\n", "\n", "\n", "predictor = xgb_estimator.deploy(\n", " initial_instance_count=1, instance_type=\"ml.m5.xlarge\", serializer=CSVSerializer(), data_capture_config=data_capture_config\n", ")" ] }, { "cell_type": "markdown", "id": "40e42e0c-95c0-4fc6-bf78-0f6f875496a9", "metadata": {}, "source": [ "#### Test inference on endpoint\n", "Function below calls the sagemaker endpoint and capture the predictions to generate the confussion matrix." ] }, { "cell_type": "code", "execution_count": null, "id": "9c221ae2-5c59-4156-8283-e42d53479aec", "metadata": { "tags": [] }, "outputs": [], "source": [ "import numpy as np\n", "def predict(data, rows=500):\n", " split_array = np.array_split(data, int(data.shape[0] / float(rows) + 1))\n", " predictions =[]\n", " for array in split_array:\n", " predictions = predictions + sum(predictor.predict(array), [])\n", "\n", " return [float(i) for i in predictions]\n", "\n", "def calibrate(probabilities, cutoff=.2):\n", " predictions = []\n", " for p in probabilities:\n", " if p <= cutoff:\n", " predictions.append(0)\n", " else:\n", " predictions.append(1)\n", " return predictions" ] }, { "cell_type": "markdown", "id": "23336fcb-b392-447c-9b3d-9311a91ec034", "metadata": {}, "source": [ "You can load SageMaker Experiment at any time and track more information about the run. Here you will invoke the SageMaker endpoint for batch prediction and we will put the results in a simple [confusion matrix](https://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/). We will also capture the chart and track it inside our experiment run.\n", "\n", "After you complete the cell below, you can go to SageMaker Experiment to see all the information SageMaker has captured for you in this experiment run.\n", "\n", "![EX01](statics/module_03_ex01.png)" ] }, { "cell_type": "code", "execution_count": null, "id": "fee8c75f-d203-4467-bb90-a7f0cdbb0dfe", "metadata": { "tags": [] }, "outputs": [], "source": [ "from sklearn.metrics import f1_score\n", "\n", "with load_run(experiment_name=experiment_name, run_name=run_name) as run:\n", "\n", " # run batch prediction\n", " probabilities = predict(test.to_numpy()[:, 1:])\n", " # run calibration and visualize the results\n", " predictions = np.asarray(calibrate(probabilities, 0.4))\n", " run.log_confusion_matrix(test[\"anomaly\"], predictions, unique_name_from_base(\"Confusion-Matrix\"))\n", "\n", "print(f\"Experiment Name: {experiment_name}\\n\")\n", "\n", "print(f\"Run Name: {run_name}\\n\")\n", "\n", "f1 = f1_score(test[\"anomaly\"], predictions, average='micro')\n", "\n", "print(f\"F1 Score: {f1}\")\n", "pd.crosstab(\n", " index=test.iloc[:, 0],\n", " columns=predictions,\n", " rownames=[\"actual\"],\n", " colnames=[\"predictions\"],\n", ")" ] }, { "cell_type": "markdown", "id": "f8fe950a-d9f0-4dc1-b345-5a0b8ea41560", "metadata": {}, "source": [ "### Train Isolation Forest Model\n", "----\n", "Now let's experiment with a different approach. This time you are going to use a common unsupervised learning algorithm called isolation forest. In the cells below, you will use the full dataset to build an isolation forest model trying to isolate anomalies base on how different they are from each other. More on [isolation forest algorithm](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.IsolationForest.html#sklearn.ensemble.IsolationForest)." ] }, { "cell_type": "markdown", "id": "453f91a9-6aa0-4b59-a418-fdccc556e64b", "metadata": {}, "source": [ "Upload isolation forest dataset to S3." ] }, { "cell_type": "code", "execution_count": null, "id": "aa7cdf22-b8fb-4a3a-82b3-4575cb37f9c5", "metadata": { "tags": [] }, "outputs": [], "source": [ "iso_input = dataset.drop([\"location_id\", \"anomaly\", \"eventtime\", \n", " \"write_time\",\"api_invocation_time\",'is_deleted'], axis=1)\n", "iso_input.to_csv(\"data/iso_input.csv\", index=False)\n", "key = f\"{prefix}/data/isoforest/iso_input.csv\"\n", "\n", "s3_client.upload_file(\n", " Filename=\"data/iso_input.csv\",\n", " Bucket=bucket,\n", " Key=key,\n", ")\n", "\n", "input_s3_path = f\"s3://{bucket}/{key}\"\n", "print(f\"training data is uploaded to {input_s3_path}\")" ] }, { "cell_type": "code", "execution_count": null, "id": "5d84fa7d-8122-422e-bd08-ef23cbab4f77", "metadata": { "scrolled": true, "tags": [] }, "outputs": [], "source": [ "from sagemaker.sklearn.estimator import SKLearn\n", "\n", "run_name = unique_name_from_base(\"isoforest-experiment\")\n", "\n", "with Run(experiment_name=experiment_name, run_name=run_name, \n", " sagemaker_session=sagemaker_session) as run:\n", " \n", " run.log_file(\"data/iso_input.csv\", is_output=False)\n", " FRAMEWORK_VERSION = \"1.0-1\"\n", "\n", " sklearn = SKLearn(\n", " entry_point=\"isolation_forest_script.py\",\n", " source_dir=\"code\",\n", " framework_version=\"1.0-1\",\n", " instance_count=train_instance_count,\n", " instance_type=train_instance_type,\n", " role=sagemaker_role,\n", " sagemaker_session=sagemaker_session,\n", " hyperparameters={\"max_samples\": 512,\n", " \"random_state\": 42,\n", " \"region\":region},\n", " )\n", " sklearn.fit({\"train\": input_s3_path})" ] }, { "cell_type": "markdown", "id": "f435e5bd-2cea-423b-824e-3b5c1e613f8a", "metadata": {}, "source": [ "#### Deploy IsoForest Model to an endpoint" ] }, { "cell_type": "code", "execution_count": null, "id": "bf6581c0-2fb7-481e-a26f-53f818bf17b2", "metadata": { "tags": [] }, "outputs": [], "source": [ "from sagemaker.deserializers import JSONDeserializer\n", "\n", "isoforest_predictor = sklearn.deploy(\n", " initial_instance_count=1, instance_type=\"ml.m5.xlarge\", serializer=CSVSerializer(), deserializer = JSONDeserializer()\n", ")" ] }, { "cell_type": "markdown", "id": "913e7ad6-d600-46cc-b39d-6f3cd8cb2428", "metadata": {}, "source": [ "#### Test inference on endpoint\n", "Capture the confussion matrix results in the experiemnt for historic reference. It turns out, Isolation forest does not perform well in this situation, so we will go with the xgboost model fore the rest of the module and lab." ] }, { "cell_type": "code", "execution_count": null, "id": "7dcd43d9-5abe-4c70-95f3-ccaa6aebdb2d", "metadata": { "tags": [] }, "outputs": [], "source": [ "with load_run(experiment_name=experiment_name, run_name=run_name) as run:\n", "\n", " results = isoforest_predictor.predict(test.to_numpy()[:, 1:])\n", " \n", " # run fix -1 value to 0\n", " predictions = []\n", " for x in results:\n", " if x <= 0:\n", " predictions.append(0)\n", " else:\n", " predictions.append(x)\n", " \n", " predictions = np.asarray(predictions)\n", " run.log_confusion_matrix(test[\"anomaly\"], predictions, unique_name_from_base(\"IsoForest-Confusion-Matrix\"))\n", "\n", "print(f\"Experiment Name: {experiment_name}\\n\")\n", "\n", "print(f\"Run Name: {run_name}\\n\")\n", "\n", "f1 = f1_score(test[\"anomaly\"], predictions, average='micro')\n", "\n", "print(f\"F1 Score: {f1}\")\n", "\n", "pd.crosstab(\n", " index=test.iloc[:, 0],\n", " columns=predictions,\n", " rownames=[\"actual\"],\n", " colnames=[\"predictions\"],\n", ")" ] }, { "cell_type": "markdown", "id": "fd5db1da-a43d-4679-ac49-c103b851e8c8", "metadata": {}, "source": [ "## Model Monitoring\n", "\n", "![MON01](statics/module_03_monitor01.png)\n", "\n", "Model monitoring is crucial to ensure that machine learning models continue to perform as expected after deployment. It involves tracking various metrics such as accuracy, precision, recall, and F1 score to detect and diagnose performance degradation, identify data drift, and other issues that may arise over time.\n", "\n", "Amazon SageMaker model monitoring allows you to automatically monitor your deployed models using predefined rules, and alerts you when the model's performance deviates from the expected behavior. In this example you are going to manually setup data drift detection for our xgboost endpoint.\n", "\n", "#### 1. Create a baselining job with training dataset\n", "Now that you have the training data ready in Amazon S3, start a job to suggest constraints. DefaultModelMonitor.suggest_baseline(..) starts a ProcessingJob using an Amazon SageMaker provided Model Monitor container to generate the constraints." ] }, { "cell_type": "code", "execution_count": null, "id": "45fb2e50-6eea-4c0a-b293-99050dfa49ce", "metadata": { "scrolled": true, "tags": [] }, "outputs": [], "source": [ "from sagemaker.model_monitor import DefaultModelMonitor\n", "from sagemaker.model_monitor.dataset_format import DatasetFormat\n", "\n", "# this is our training dataset\n", "baseline_data_uri = train_s3_path\n", "baseline_results_prefix = f\"{prefix}/monitoring/baselining/results\"\n", "baseline_results_uri = f\"s3://{bucket}/{baseline_results_prefix}\"\n", "\n", "\n", "my_default_monitor = DefaultModelMonitor(\n", " role=sagemaker_role,\n", " instance_count=1,\n", " instance_type=\"ml.m5.xlarge\",\n", " volume_size_in_gb=20,\n", " max_runtime_in_seconds=3600,\n", ")\n", "\n", "my_default_monitor.suggest_baseline(\n", " baseline_dataset=baseline_data_uri,\n", " dataset_format=DatasetFormat.csv(header=True),\n", " output_s3_uri=baseline_results_uri,\n", " wait=True,\n", ")" ] }, { "cell_type": "markdown", "id": "a0d2d41b-fadb-4fdb-a696-59069781dbdc", "metadata": {}, "source": [ "Baseline process generates a contraints and statistics configuration files, we are going to preview the generated json files. Keep in mind, you can also supply your own." ] }, { "cell_type": "code", "execution_count": null, "id": "23d23c2d-e122-4f5a-a2c1-ab655d619b2c", "metadata": { "tags": [] }, "outputs": [], "source": [ "result = s3_client.list_objects(Bucket=bucket, Prefix=baseline_results_prefix)\n", "report_files = [report_file.get(\"Key\") for report_file in result.get(\"Contents\")]\n", "print(\"Found Files:\")\n", "print(\"\\n \".join(report_files))" ] }, { "cell_type": "markdown", "id": "d3d3bd4a-541b-48f1-9c96-16b55553de47", "metadata": {}, "source": [ "**Statistics** refer to the expected statistical properties of the input data, such as mean and standard deviation" ] }, { "cell_type": "code", "execution_count": null, "id": "2bdd6d7d-6375-4ae7-9ff3-5436ed22dad4", "metadata": { "tags": [] }, "outputs": [], "source": [ "import pandas as pd\n", "\n", "baseline_job = my_default_monitor.latest_baselining_job\n", "schema_df = pd.io.json.json_normalize(baseline_job.baseline_statistics().body_dict[\"features\"])\n", "schema_df.head(10)" ] }, { "cell_type": "markdown", "id": "d70ce650-eda1-4fc5-b11c-b80287138bda", "metadata": {}, "source": [ "**Constraints** are rules that are used to ensure that the model's performance does not degrade beyond a certain threshold" ] }, { "cell_type": "code", "execution_count": null, "id": "0ff450b7-5ae3-494a-9e25-c2999ca23bee", "metadata": { "tags": [] }, "outputs": [], "source": [ "constraints_df = pd.io.json.json_normalize(\n", " baseline_job.suggested_constraints().body_dict[\"features\"]\n", ")\n", "constraints_df.head(10)" ] }, { "cell_type": "markdown", "id": "78b94347-8a5d-427a-a750-d365284c1c00", "metadata": {}, "source": [ "#### 2. Create a schedule to analyze collected data for data quality issues" ] }, { "cell_type": "code", "execution_count": null, "id": "092a6a67-d0c3-4826-9c80-9d51e7c75e61", "metadata": { "tags": [] }, "outputs": [], "source": [ "from time import gmtime, strftime\n", "from sagemaker.model_monitor import CronExpressionGenerator\n", "\n", "mon_schedule_name = unique_name_from_base(f\"{prefix}-monitoring-job\")\n", "\n", "s3_report_path = f\"s3://{bucket}/{prefix}/montoring/report\"\n", "\n", "my_default_monitor.create_monitoring_schedule(\n", " monitor_schedule_name=mon_schedule_name,\n", " endpoint_input=predictor.endpoint, #predictor endpoint name\n", " output_s3_uri=s3_report_path,\n", " statistics=my_default_monitor.baseline_statistics(),\n", " constraints=my_default_monitor.suggested_constraints(),\n", " schedule_cron_expression=CronExpressionGenerator.hourly(), #Hourly\n", " enable_cloudwatch_metrics=True,\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "1bbf953b-1edc-4196-83b4-23a8e702c036", "metadata": { "tags": [] }, "outputs": [], "source": [ "desc_schedule_result = my_default_monitor.describe_schedule()\n", "print(\"Schedule status: {}\".format(desc_schedule_result[\"MonitoringScheduleStatus\"]))" ] }, { "cell_type": "markdown", "id": "2732643a-21c5-4320-8624-eb53c09bd21e", "metadata": {}, "source": [ "### Generate some artificial traffic" ] }, { "cell_type": "code", "execution_count": null, "id": "7568fa71-2f23-425c-8d4a-d39b6f8c2be9", "metadata": { "scrolled": true, "tags": [] }, "outputs": [], "source": [ "count = 0\n", "for i in range(100):\n", "\n", " predict(test.to_numpy()[:, 1:])\n", " count+=1\n", " if count%10 == 0:\n", " print(f\"predicting artificial traffic batch {count} ...\")" ] }, { "cell_type": "markdown", "id": "5d4cafc7-293f-4b80-81b2-5f6cb2a1fef1", "metadata": {}, "source": [ "Depend on the cron job you defined, you may need to wait a bit for your monitor job to execute. Once it does, the execution will be listed below." ] }, { "cell_type": "code", "execution_count": null, "id": "875cb453-6662-40b5-8d92-eb09e4ee5690", "metadata": { "tags": [] }, "outputs": [], "source": [ "mon_executions = my_default_monitor.list_executions()\n", "mon_executions" ] }, { "cell_type": "markdown", "id": "d7bbce37-004f-4dcd-80ee-edfc37992e37", "metadata": { "tags": [] }, "source": [ "## Clean up" ] }, { "cell_type": "markdown", "id": "7df94c6d-26ef-430d-a5ab-29c3c583ee1d", "metadata": {}, "source": [ "Delete feature group" ] }, { "cell_type": "code", "execution_count": null, "id": "6330e71e-f238-4127-a0cf-4e82d9dea7d0", "metadata": { "tags": [] }, "outputs": [], "source": [ "feature_group_name = fg_name\n", "sagemaker_client.delete_feature_group(\n", " FeatureGroupName= feature_group_name\n", ")" ] }, { "cell_type": "markdown", "id": "0665e257-bc91-4ae2-bb5f-491fabc05c7a", "metadata": {}, "source": [ "Remove experiments" ] }, { "cell_type": "code", "execution_count": null, "id": "18e60355-6025-4d6f-881f-8082a855561a", "metadata": { "tags": [] }, "outputs": [], "source": [ "import time\n", "def remove_experiment(experiment_name):\n", " trials = sagemaker_client.list_trials(ExperimentName=experiment_name)['TrialSummaries']\n", " print('TrialNames:')\n", " for trial in trials:\n", " trial_name = trial['TrialName']\n", " print(f\"\\n{trial_name}\")\n", "\n", " components_in_trial = sagemaker_client.list_trial_components(TrialName=trial_name)\n", " print('\\tTrialComponentNames:')\n", " for component in components_in_trial['TrialComponentSummaries']:\n", " component_name = component['TrialComponentName']\n", " print(f\"\\t{component_name}\")\n", " sagemaker_client.disassociate_trial_component(TrialComponentName=component_name, TrialName=trial_name)\n", " try:\n", " # comment out to keep trial components\n", " sagemaker_client.delete_trial_component(TrialComponentName=component_name)\n", " except:\n", " # component is associated with another trial\n", " continue\n", " # to prevent throttling\n", " time.sleep(.5)\n", " sagemaker_client.delete_trial(TrialName=trial_name)\n", " sagemaker_client.delete_experiment(ExperimentName=experiment_name)\n", " print(f\"\\nExperiment {experiment_name} deleted\")\n", "\n", "remove_experiment(experiment_name)" ] }, { "cell_type": "markdown", "id": "a9b7c166-7189-4407-acdf-f4501747bf6a", "metadata": {}, "source": [ "Remove Model Monitor" ] }, { "cell_type": "code", "execution_count": null, "id": "a0febe27-c8fa-4433-a870-8080870aaa99", "metadata": { "tags": [] }, "outputs": [], "source": [ "my_default_monitor.delete_monitoring_schedule()" ] }, { "cell_type": "markdown", "id": "5e2f897b-4811-4ebb-aa60-da0e44b5281f", "metadata": {}, "source": [ "Remove endpoints" ] }, { "cell_type": "code", "execution_count": null, "id": "c069a782-6683-4213-81dd-76b0005c0c29", "metadata": { "tags": [] }, "outputs": [], "source": [ "def remove_endpoint(endpoint_name):\n", " monitor_schedules = sagemaker_client.list_monitoring_schedules(EndpointName=endpoint_name)['MonitoringScheduleSummaries']\n", " print('Monitoring Schedule:')\n", " for ms in monitor_schedules:\n", " ms_name = ms['MonitoringScheduleName']\n", " print(f\"\\n{ms_name}\")\n", "\n", " sagemaker_client.delete_monitoring_schedule(MonitoringScheduleName=ms_name)\n", " \n", " sagemaker_client.delete_endpoint(EndpointName=endpoint_name)\n", " print(f\"Endpoint {endpoint_name} deleted\")\n", "\n", "#xgboost\n", "remove_endpoint(predictor.endpoint_name)\n", "# #isolation forest\n", "remove_endpoint(isoforest_predictor.endpoint_name)" ] }, { "cell_type": "code", "execution_count": null, "id": "667ffc53-9d22-49bf-bbca-39591c1c4e80", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "availableInstances": [ { "_defaultOrder": 0, "_isFastLaunch": true, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 4, "name": "ml.t3.medium", "vcpuNum": 2 }, { "_defaultOrder": 1, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.t3.large", "vcpuNum": 2 }, { "_defaultOrder": 2, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.t3.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 3, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.t3.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 4, "_isFastLaunch": true, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.m5.large", "vcpuNum": 2 }, { "_defaultOrder": 5, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.m5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 6, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.m5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 7, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.m5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 8, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.m5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 9, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.m5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 10, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.m5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 11, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.m5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 12, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.m5d.large", "vcpuNum": 2 }, { "_defaultOrder": 13, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.m5d.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 14, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.m5d.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 15, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.m5d.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 16, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.m5d.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 17, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.m5d.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 18, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.m5d.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 19, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.m5d.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 20, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": true, "memoryGiB": 0, "name": "ml.geospatial.interactive", "supportedImageNames": [ "sagemaker-geospatial-v1-0" ], "vcpuNum": 0 }, { "_defaultOrder": 21, "_isFastLaunch": true, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 4, "name": "ml.c5.large", "vcpuNum": 2 }, { "_defaultOrder": 22, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.c5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 23, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.c5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 24, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.c5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 25, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 72, "name": "ml.c5.9xlarge", "vcpuNum": 36 }, { "_defaultOrder": 26, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 96, "name": "ml.c5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 27, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 144, "name": "ml.c5.18xlarge", "vcpuNum": 72 }, { "_defaultOrder": 28, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.c5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 29, "_isFastLaunch": true, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.g4dn.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 30, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.g4dn.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 31, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.g4dn.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 32, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.g4dn.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 33, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.g4dn.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 34, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.g4dn.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 35, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 61, "name": "ml.p3.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 36, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 244, "name": "ml.p3.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 37, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 488, "name": "ml.p3.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 38, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 768, "name": "ml.p3dn.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 39, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.r5.large", "vcpuNum": 2 }, { "_defaultOrder": 40, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.r5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 41, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.r5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 42, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.r5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 43, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.r5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 44, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.r5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 45, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 512, "name": "ml.r5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 46, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 768, "name": "ml.r5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 47, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.g5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 48, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.g5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 49, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.g5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 50, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.g5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 51, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.g5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 52, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.g5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 53, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.g5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 54, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 768, "name": "ml.g5.48xlarge", "vcpuNum": 192 } ], "instance_type": "ml.t3.medium", "kernelspec": { "display_name": "Python 3 (Data Science)", "language": "python", "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/datascience-1.0" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.10" } }, "nbformat": 4, "nbformat_minor": 5 }