{ "cells": [ { "cell_type": "markdown", "id": "f2ed4510", "metadata": {}, "source": [ "# Clean Up Resources\n", "This notebook demonstrates how to clean up all the resources created in a previous notebook by loading saved variable names. \n", "\n", "(!) If you have created resources using multiple notebooks and have multiple dataset groups, run the cell that begins `# store for cleanup` at the end of the notebook and then immediatly after that run this notebook for each of them (if you have created two dataset groups, you wil have to run this notebook twice). This code only deletes the resources specified in `# store for cleanup` and if you used multiple notebooks, the values saved will correspond to the last notebook you ran).\n", "\n", "This notebook uses the functions defined below, to iterate throught the resources inside a dataset group. \n", "\n", "You can use this notebook to delete Amazon Personalize resources The resource ARNs are defined in the previous notebook or you can enter them manually bellow." ] }, { "cell_type": "code", "execution_count": null, "id": "a771e164", "metadata": {}, "outputs": [], "source": [ "%store -r" ] }, { "cell_type": "code", "execution_count": null, "id": "24882c55", "metadata": {}, "outputs": [], "source": [ "# If you cannot/do not want to use \"%store -r\" to load the resources to delete, \n", "# you can uncomment the code bellow and enter them manually\n", "\n", "# dataset_group_arn='XXXXX'\n", "# role_name='XXXXX'\n", "# region='XXXXX'" ] }, { "cell_type": "markdown", "id": "d317efa0", "metadata": {}, "source": [ "Print the resources to be deleted. Please check that these correspond to the resources you want to delete. \n", "This operation cannot be undone." ] }, { "cell_type": "code", "execution_count": null, "id": "a856088b", "metadata": {}, "outputs": [], "source": [ "print ('dataset_group_arn:', dataset_group_arn)\n", "print ('role_name:', role_name)\n", "print ('region:', region)" ] }, { "cell_type": "code", "execution_count": null, "id": "2d2adadd", "metadata": {}, "outputs": [], "source": [ "schema_arns = []" ] }, { "cell_type": "code", "execution_count": null, "id": "4013ba1b", "metadata": {}, "outputs": [], "source": [ "import sys\n", "import getopt\n", "import logging\n", "import botocore\n", "import boto3\n", "import time\n", "from packaging import version\n", "from time import sleep\n", "from botocore.exceptions import ClientError\n", "\n", "logger = logging.getLogger()\n", "handler = logging.StreamHandler(sys.stdout)\n", "handler.setLevel(logging.INFO)\n", "logger.setLevel(logging.INFO)\n", "logger.addHandler(handler)\n", "\n", "personalize = None" ] }, { "cell_type": "code", "execution_count": null, "id": "f358d77f", "metadata": {}, "outputs": [], "source": [ "def _delete_event_trackers(dataset_group_arn):\n", " event_tracker_arns = []\n", "\n", " event_trackers_paginator = personalize.get_paginator('list_event_trackers')\n", " for event_tracker_page in event_trackers_paginator.paginate(datasetGroupArn = dataset_group_arn):\n", " for event_tracker in event_tracker_page['eventTrackers']:\n", " if event_tracker['status'] in [ 'ACTIVE', 'CREATE FAILED' ]:\n", " logger.info('Deleting event tracker {}'.format(event_tracker['eventTrackerArn']))\n", " personalize.delete_event_tracker(eventTrackerArn = event_tracker['eventTrackerArn'])\n", " elif event_tracker['status'].startswith('DELETE'):\n", " logger.warning('Event tracker {} is already being deleted so will wait for delete to complete'.format(event_tracker['eventTrackerArn']))\n", " else:\n", " raise Exception('Solution {} has a status of {} so cannot be deleted'.format(event_tracker['eventTrackerArn'], event_tracker['status']))\n", "\n", " event_tracker_arns.append(event_tracker['eventTrackerArn'])\n", "\n", " max_time = time.time() + 30*60 # 30 mins\n", " while time.time() < max_time:\n", " for event_tracker_arn in event_tracker_arns:\n", " try:\n", " describe_response = personalize.describe_event_tracker(eventTrackerArn = event_tracker_arn)\n", " logger.debug('Event tracker {} status is {}'.format(event_tracker_arn, describe_response['eventTracker']['status']))\n", " except ClientError as e:\n", " error_code = e.response['Error']['Code']\n", " if error_code == 'ResourceNotFoundException':\n", " event_tracker_arns.remove(event_tracker_arn)\n", "\n", " if len(event_tracker_arns) == 0:\n", " logger.info('All event trackers have been deleted or none exist for dataset group')\n", " break\n", " else:\n", " logger.info('Waiting for {} event tracker(s) to be deleted'.format(len(event_tracker_arns)))\n", " time.sleep(20)\n", "\n", " if len(event_tracker_arns) > 0:\n", " raise Exception('Timed out waiting for all event trackers to be deleted')\n", "\n", "def _delete_filters(dataset_group_arn):\n", " filter_arns = []\n", "\n", " filters_response = personalize.list_filters(datasetGroupArn = dataset_group_arn, maxResults = 100)\n", " for filter in filters_response['Filters']:\n", " logger.info('Deleting filter ' + filter['filterArn'])\n", " personalize.delete_filter(filterArn = filter['filterArn'])\n", " filter_arns.append(filter['filterArn'])\n", "\n", " max_time = time.time() + 30*60 # 30 mins\n", " while time.time() < max_time:\n", " for filter_arn in filter_arns:\n", " try:\n", " describe_response = personalize.describe_filter(filterArn = filter_arn)\n", " logger.debug('Filter {} status is {}'.format(filter_arn, describe_response['filter']['status']))\n", " except ClientError as e:\n", " error_code = e.response['Error']['Code']\n", " if error_code == 'ResourceNotFoundException':\n", " filter_arns.remove(filter_arn)\n", "\n", " if len(filter_arns) == 0:\n", " logger.info('All filters have been deleted or none exist for dataset group')\n", " break\n", " else:\n", " logger.info('Waiting for {} filter(s) to be deleted'.format(len(filter_arns)))\n", " time.sleep(20)\n", "\n", " if len(filter_arns) > 0:\n", " raise Exception('Timed out waiting for all filter(s) to be deleted')\n", " \n", "def _delete_recommenders(dataset_group_arn):\n", " recommender_arns = []\n", " recommenders_response = personalize.list_recommenders(datasetGroupArn = dataset_group_arn, maxResults = 100)\n", " for recommender in recommenders_response['recommenders']:\n", " logger.info('Deleting recommender ' + recommender['recommenderArn'])\n", " recommender_status = personalize.describe_recommender(recommenderArn = recommender['recommenderArn'])['recommender']['status']\n", " if not (recommender_status == 'DELETE IN_PROGRESS'):\n", " personalize.delete_recommender(recommenderArn = recommender['recommenderArn'])\n", " recommender_arns.append(recommender['recommenderArn'])\n", " max_time = time.time() + 30*60 # 30 mins\n", " while time.time() < max_time:\n", " for recommender_arn in recommender_arns:\n", " try:\n", " describe_response = personalize.describe_recommender(recommenderArn = recommender_arn)\n", " logger.debug('Recommender {} status is {}'.format(recommender_arn, describe_response['recommender']['status']))\n", " except ClientError as e:\n", " error_code = e.response['Error']['Code']\n", " if error_code == 'ResourceNotFoundException':\n", " recommender_arns.remove(recommender_arn)\n", "\n", " if len(recommender_arns) == 0:\n", " logger.info('All recommenders have been deleted or none exist for dataset group')\n", " break\n", " else:\n", " logger.info('Waiting for {} recommender(s) to be deleted'.format(len(recommender_arns)))\n", " time.sleep(20)\n", "\n", " if len(recommender_arns) > 0:\n", " raise Exception('Timed out waiting for all recommender(s) to be deleted')\n", " \n", "\n", "def _delete_datasets_and_schemas(dataset_group_arn, schema_arns):\n", " dataset_arns = []\n", " \n", " dataset_paginator = personalize.get_paginator('list_datasets')\n", " for dataset_page in dataset_paginator.paginate(datasetGroupArn = dataset_group_arn):\n", " for dataset in dataset_page['datasets']:\n", " describe_response = personalize.describe_dataset(datasetArn = dataset['datasetArn'])\n", " schema_arns.append(describe_response['dataset']['schemaArn'])\n", "\n", " if dataset['status'] in ['ACTIVE', 'CREATE FAILED']:\n", " logger.info('Deleting dataset ' + dataset['datasetArn'])\n", " personalize.delete_dataset(datasetArn = dataset['datasetArn'])\n", " elif dataset['status'].startswith('DELETE'):\n", " logger.warning('Dataset {} is already being deleted so will wait for delete to complete'.format(dataset['datasetArn']))\n", " else:\n", " raise Exception('Dataset {} has a status of {} so cannot be deleted'.format(dataset['datasetArn'], dataset['status']))\n", "\n", " dataset_arns.append(dataset['datasetArn'])\n", "\n", " max_time = time.time() + 30*60 # 30 mins\n", " while time.time() < max_time:\n", " for dataset_arn in dataset_arns:\n", " try:\n", " describe_response = personalize.describe_dataset(datasetArn = dataset_arn)\n", " logger.debug('Dataset {} status is {}'.format(dataset_arn, describe_response['dataset']['status']))\n", " except ClientError as e:\n", " error_code = e.response['Error']['Code']\n", " if error_code == 'ResourceNotFoundException':\n", " dataset_arns.remove(dataset_arn)\n", "\n", " if len(dataset_arns) == 0:\n", " logger.info('All datasets have been deleted or none exist for dataset group')\n", " break\n", " else:\n", " logger.info('Waiting for {} dataset(s) to be deleted'.format(len(dataset_arns)))\n", " time.sleep(20)\n", "\n", " if len(dataset_arns) > 0:\n", " raise Exception('Timed out waiting for all datasets to be deleted')\n", "\n", " for schema_arn in schema_arns:\n", " try:\n", " logger.info('Deleting schema ' + schema_arn)\n", " personalize.delete_schema(schemaArn = schema_arn)\n", " except ClientError as e:\n", " error_code = e.response['Error']['Code']\n", " if error_code == 'ResourceInUseException':\n", " logger.info('Schema {} is still in-use by another dataset (likely in another dataset group)'.format(schema_arn))\n", " else:\n", " raise e\n", "\n", " logger.info('All schemas used exclusively by datasets have been deleted or none exist for dataset group')\n", "\n", "def _delete_dataset_group(dataset_group_arn):\n", " logger.info('Deleting dataset group ' + dataset_group_arn)\n", " personalize.delete_dataset_group(datasetGroupArn = dataset_group_arn)\n", "\n", " max_time = time.time() + 30*60 # 30 mins\n", " while time.time() < max_time:\n", " try:\n", " describe_response = personalize.describe_dataset_group(datasetGroupArn = dataset_group_arn)\n", " logger.debug('Dataset group {} status is {}'.format(dataset_group_arn, describe_response['datasetGroup']['status']))\n", " break\n", " except ClientError as e:\n", " error_code = e.response['Error']['Code']\n", " if error_code == 'ResourceNotFoundException':\n", " logger.info('Dataset group {} has been fully deleted'.format(dataset_group_arn))\n", " else:\n", " raise e\n", "\n", " logger.info('Waiting for dataset group to be deleted')\n", " time.sleep(20)\n", "\n", "def _delete_metric_attributions(dataset_group_arn):\n", " # delete metric attributions in this dataset group \n", " metric_attribution_list = personalize.list_metric_attributions(datasetGroupArn = dataset_group_arn)['metricAttributions']\n", "\n", " for metric in metric_attribution_list:\n", " response = personalize.delete_metric_attribution(\n", " metricAttributionArn = metric['metricAttributionArn']\n", " ) \n", " \n", " logger.info('All metric attributions have been deleted or none exist for dataset group')\n", " \n", " #wait for metrics to delete\n", " time.sleep (120)\n", "\n", "def delete_dataset_groups(dataset_group_arns, schema_arns, region = None):\n", " global personalize\n", " personalize = boto3.client(service_name = 'personalize', region_name = region)\n", "\n", " for dataset_group_arn in dataset_group_arns:\n", " logger.info('Dataset Group ARN: ' + dataset_group_arn)\n", "\n", " # 1. Delete Recommenders\n", " _delete_recommenders(dataset_group_arn)\n", " \n", " # 2. Delete event trackers\n", " _delete_event_trackers(dataset_group_arn)\n", "\n", " # 3. Delete filters\n", " _delete_filters(dataset_group_arn)\n", " \n", " # 4. Delete metric attributions\n", " _delete_metric_attributions(dataset_group_arn)\n", "\n", " # 5. Delete datasets and their schemas\n", " _delete_datasets_and_schemas(dataset_group_arn, schema_arns)\n", "\n", " # 6. Delete dataset group\n", " _delete_dataset_group(dataset_group_arn)\n", "\n", " logger.info(f'Dataset group {dataset_group_arn} fully deleted')\n" ] }, { "cell_type": "code", "execution_count": null, "id": "3632a568", "metadata": {}, "outputs": [], "source": [ "delete_dataset_groups([dataset_group_arn], schema_arns, region)" ] }, { "cell_type": "markdown", "id": "84e94a02", "metadata": {}, "source": [ "## Clean up the IAM role\n", "Start by deleting the role." ] }, { "cell_type": "code", "execution_count": null, "id": "6372bd68", "metadata": {}, "outputs": [], "source": [ "iam = boto3.Session().client(\n", " service_name='iam', region_name=region)" ] }, { "cell_type": "markdown", "id": "31952392", "metadata": {}, "source": [ "Identify the name of the role you want to delete.\n", "\n", "You cannot delete an IAM role which still has policies attached to it. So after you have identified the relevant role, let's list the attached policies of that role." ] }, { "cell_type": "code", "execution_count": null, "id": "b646683f", "metadata": {}, "outputs": [], "source": [ "attached_policies = iam.list_attached_role_policies( RoleName = role_name)\n", "print (attached_policies)\n", "\n", "# detaching the role policies\n", "for attached_policy in attached_policies['AttachedPolicies']:\n", " iam.detach_role_policy(\n", " RoleName = role_name,\n", " PolicyArn = attached_policy['PolicyArn']\n", " )" ] }, { "cell_type": "code", "execution_count": null, "id": "f3e1c048", "metadata": {}, "outputs": [], "source": [ "# removing the inline policies\n", "in_line_policies = iam.list_role_policies(\n", " RoleName = role_name\n", ")\n", "print (in_line_policies)\n", "\n", "for in_line_policy in in_line_policies['PolicyNames']:\n", " iam.delete_role_policy(\n", " RoleName = role_name,\n", " PolicyName = in_line_policy\n", ")\n" ] }, { "cell_type": "markdown", "id": "88ac5ea3", "metadata": {}, "source": [ "Finally, you should be able to delete the IAM role." ] }, { "cell_type": "code", "execution_count": null, "id": "306e1bb4", "metadata": {}, "outputs": [], "source": [ "iam.delete_role(\n", " RoleName = role_name\n", ")" ] }, { "cell_type": "markdown", "id": "4fa0cf14", "metadata": {}, "source": [ "## Deleting the Amazon S3 bucket\n", "To delete an Amazon S3 bucket, it first needs to be empty. The easiest way to delete an Amazon S3 bucket, is just to navigate to Amazon S3 in the AWS console, delete the objects in the bucket, and then delete the Amazon S3 bucket itself." ] }, { "cell_type": "code", "execution_count": null, "id": "ff15d740", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "conda_python3", "language": "python", "name": "conda_python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.12" } }, "nbformat": 4, "nbformat_minor": 5 }