{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Clean up\n",
"This notebook provides code for end-to-end clean up of your SageMaker environment and preparation for stack deletion.\n",
"\n",
" ❗ This is a destructive action. All SageMaker data, notebooks, projects, artifacts and correspoinding S3 buckets will be deleted! All data on the EFS will be deleted (SageMaker home directories). You may want to backup the EFS before deletion❗ \n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load environment variables\n",
"If you haven't done so, you have to run [00-setup notebook](00-setup.ipynb) first!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import boto3\n",
"import json\n",
"import sagemaker\n",
"\n",
"sm = boto3.client(\"sagemaker\")\n",
"cf = boto3.client(\"cloudformation\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%store -r domain_id\n",
"%store -r security_group_ids\n",
"%store -r subnets\n",
"%store -r processing_role\n",
"%store -r execution_role\n",
"%store -r model_execution_role\n",
"%store -r pipeline_role\n",
"%store -r data_bucket\n",
"%store -r model_bucket\n",
"%store -r ebs_key_id\n",
"%store -r s3_key_id\n",
"%store -r network_config\n",
"%store -r env_name\n",
"%store -r env_type\n",
"%store -r env_type_staging_name\n",
"%store -r env_type_prod_name\n",
"\n",
"try:\n",
" domain_id\n",
"except NameError:\n",
" print(\"++++++++++++++++++++++++++++++++++++++++++++++\")\n",
" print(\"[ERROR] YOU HAVE TO RUN 00-SETUP notebook \")\n",
" print(\"++++++++++++++++++++++++++++++++++++++++++++++\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step1: Clean up MLOps projects"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Enumerate projects and select projects to be deleted"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Get all projects created by the current domain\n",
"projects = [\n",
" {\"ProjectName\":p[\"ProjectName\"], \"ProjectId\":p[\"ProjectId\"]} for p in sm.list_projects(MaxResults=100, SortBy=\"CreationTime\")[\"ProjectSummaryList\"] \n",
" if p[\"ProjectStatus\"] in [\"CreateCompleted\"] and sm.describe_project(ProjectName=p[\"ProjectName\"])[\"CreatedBy\"][\"DomainId\"] == domain_id\n",
"]\n",
"\n",
"print(f\"These projects have been created by domain {domain_id}: {json.dumps(projects, indent=2)}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Select projects to be deleted\n",
"projects_to_delete = []\n",
"\n",
"for p in projects:\n",
" print(f\"Are you sure you want to delete this project: {p['ProjectName']}? (y/n)\")\n",
" choice = input()\n",
" if choice == 'y':\n",
" projects_to_delete.append(p)\n",
" \n",
"print(f\"The following projects will be deleted: {json.dumps(projects_to_delete, indent=2)}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Remove CloudFormation stack sets\n",
"This code removes stack sets which were used for model deployment"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import time\n",
"\n",
"for p in projects_to_delete:\n",
" for ss in [\n",
" f\"sagemaker-{p['ProjectName']}-{p['ProjectId']}-deploy-{env_type_staging_name}\",\n",
" f\"sagemaker-{p['ProjectName']}-{p['ProjectId']}-deploy-{env_type_prod_name}\"\n",
" ]:\n",
" try:\n",
" accounts = [a[\"Account\"] for a in cf.list_stack_instances(StackSetName=ss)[\"Summaries\"]]\n",
" print(f\"delete stack set instances for {ss} stack set for the accounts {accounts}\")\n",
" r = cf.delete_stack_instances(\n",
" StackSetName=ss,\n",
" Accounts=accounts,\n",
" Regions=[boto3.session.Session().region_name],\n",
" RetainStacks=False,\n",
" )\n",
" print(r)\n",
" time.sleep(180)\n",
" except Exception:\n",
" print(f\"skip {ss} because there is no such stackset\")\n",
" pass\n",
" else:\n",
" print(f\"delete stack set {ss}\")\n",
" r = cf.delete_stack_set(\n",
" StackSetName=ss\n",
" )\n",
" print(r)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Delete projects"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"for p in projects_to_delete:\n",
" try:\n",
" print(f\"Deleting project {p['ProjectName']}:{sm.delete_project(ProjectName=p['ProjectName'])}\")\n",
" except Exception:\n",
" pass"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Empty project S3 buckets"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"for p in projects_to_delete:\n",
" !aws s3 rb s3://sm-mlops-cp-{p['ProjectName']}-{p['ProjectId']} --force "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 2: Empty data and model S3 buckets\n",
"\n",
" \n",
"❗ This is a destructive action. All SageMaker data, logs, and model artifacts in data and models S3 buckets will be deleted❗\n",
"
"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!aws s3 rm s3://{data_bucket} --recursive --quiet\n",
"!aws s3 rm s3://{model_bucket} --recursive --quiet"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 3: Generate clean up CLI commands \n",
"This section generates the clean-up CLI commands based on your environment. You must run this commands in your CLI terminal in the solution working directory (git clone directory) after you close Studio."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def print_aws_cli(stackname):\n",
" if type(stackname) == list:\n",
" stackname = stackname[0]['StackName'] if len(stackname) else 'None'\n",
"\n",
" if stackname and stackname != \"None\":\n",
" ## condition for Service Catalog deployment-type\n",
" if f\"SC-{sagemaker.get_execution_role().split(':')[4]}\" in stackname: \n",
" print(f\"pipenv run python3 functions/pipeline/terminate-sc-product-cli.py {('-').join(stackname.split('-')[-2:])}\")\n",
" else:\n",
" print(f\"aws cloudformation delete-stack --stack-name {stackname}\")\n",
" print(f\"aws cloudformation wait stack-delete-complete --stack-name {stackname}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# find data science environment stack based on the domain id\n",
"env_stackname = [\n",
" s for s in cf.list_stacks(StackStatusFilter=[\"CREATE_COMPLETE\"])[\"StackSummaries\"] \n",
" if not s.get(\"ParentId\") and cf.describe_stacks(StackName=s[\"StackName\"])[\"Stacks\"][0].get(\"Outputs\") \n",
" and len([o for o in cf.describe_stacks(StackName=s[\"StackName\"])[\"Stacks\"][0][\"Outputs\"] \n",
" if o[\"OutputKey\"] == \"SageMakerDomainId\" and o[\"OutputValue\"] == domain_id]) != 0\n",
"]\n",
"\n",
"# find core infrastructure stack name based on the output name \n",
"core_stackname = [\n",
" s for s in cf.list_stacks(StackStatusFilter=[\"CREATE_COMPLETE\"])[\"StackSummaries\"] \n",
" if not s.get(\"ParentId\") and cf.describe_stacks(StackName=s[\"StackName\"])[\"Stacks\"][0].get(\"Outputs\") \n",
" and len([o for o in cf.describe_stacks(StackName=s[\"StackName\"])[\"Stacks\"][0][\"Outputs\"] \n",
" if o[\"OutputKey\"] == \"SCLaunchRoleArn\"]) != 0\n",
"]\n",
"\n",
"# find (optional) CloudFormation package stack name\n",
"package_cfn_stackname = [\n",
" s for s in cf.list_stacks(StackStatusFilter=[\"CREATE_COMPLETE\"])[\"StackSummaries\"] \n",
" if cf.describe_stacks(StackName=s[\"StackName\"])[\"Stacks\"][0].get(\"Outputs\") \n",
" and len([o for o in cf.describe_stacks(StackName=s[\"StackName\"])[\"Stacks\"][0][\"Outputs\"] \n",
" if o[\"OutputKey\"] == \"StartBuildCLICommand\"]) != 0\n",
"]\n",
"\n",
"print(f\"Data science environment stack name: {env_stackname}\")\n",
"print(f\"Core infrastructure stack name: {core_stackname}\")\n",
"print(f\"CloudFormation template package stack name: {package_cfn_stackname}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Copy the output of the following code cell and run it in your CLI terminal in the solution directory **after** you close Studio.\n",
"\n",
" ❗ The follwing code contains AWS CLI command to delete SageMaker EFS. All data in user home directories will be deleted when you run this code in your CLI terminal❗ \n",
"
"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(\"*********************************************************\")\n",
"print(\"* COPY THE LINES BELOW AND PASTE INTO YOUR CLI TERMINAL *\")\n",
"print(\"* !run in the solution directory! *\")\n",
"print(\"↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓\")\n",
"print_aws_cli(env_stackname)\n",
"print_aws_cli(core_stackname)\n",
"print_aws_cli(package_cfn_stackname)\n",
"print(f\"pipenv run python3 functions/pipeline/clean-up-efs-cli.py {domain_id}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" \n",
"Now you can close Studio and run the generated CLI commands in your terminal.\n",
"
"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"instance_type": "ml.t3.medium",
"kernelspec": {
"display_name": "Python 3.8.2 64-bit",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.2"
},
"vscode": {
"interpreter": {
"hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
}
}
},
"nbformat": 4,
"nbformat_minor": 4
}