{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "### Setup\n", "\n", "This notebook performs the following setup actions for this example use of Amazon SageMaker Feature Store:\n", "\n", "1. Create online-only feature groups\n", "2. Create source and destination Amazon Managed Streaming for Kafka (MSK) topics.\n", "3. Create an Amazon Kinesis Data Applications (KDA) application that aggregates data from source topic and loads to the target topic.\n", "\n", "**Recommended settings to run this notebook in SageMaker Studio:**\n", "\n", "- Image: Data Science\n", "- Kernel: Python3\n", "- Instance type: ml.m5.large (2 vCPU + 8 GiB)\n", "\n", "**Important Note:**\n", "\n", "DO NOT \"Run All Cells\" on this notebook, manual steps are needed for successful execution." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Get ARN's of Lambda functions from CloudFormation stack outputs\n", "1. InvokeFraudEndpointLambdaARN\n", "2. StreamingAggLambdaARN" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Be sure to set the stack name in cell below to match the name of the CloudFormation stack used to create this SageMaker domain. If you used the default stack name, you should not need to make any updates\n", "\n", "![SegmentLocal](images/get-cloudformation-stack-name.gif \"get_cf_name\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "STACK_NAME = 'sagemaker-featurestore-msk-kda-stack' # if you're not using the default stack name, replace this\n", "%store STACK_NAME" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "import sys\n", "import boto3\n", "\n", "cf_client = boto3.client('cloudformation')\n", "\n", "try:\n", " outputs = cf_client.describe_stacks(StackName=STACK_NAME)['Stacks'][0]['Outputs']\n", " for o in outputs:\n", " if o['OutputKey'] == 'IngestLambdaFunctionARN':\n", " lambda_to_fs_arn = o['OutputValue']\n", " if o['OutputKey'] == 'PredictLambdaFunctionARN':\n", " lambda_to_model_arn = o['OutputValue']\n", " if o['OutputKey'] == 'PredictLambdaFunctionName':\n", " predict_lambda_name = o['OutputValue']\n", " if o['OutputKey'] == 'AggregateFeatureStoreNameOutput':\n", " aggregate_feature_group_name = o['OutputValue'] \n", " if o['OutputKey'] == 'AggregateBatchFeatureStoreNameOutput':\n", " aggregate_batch_feature_group_name = o['OutputValue']\n", "except:\n", " msg = f'CloudFormation stack {STACK_NAME} was not found. Please set the STACK_NAME properly and re-run this cell'\n", " sys.exit(ValueError(msg))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "print(f'lambda_to_model_arn: {lambda_to_model_arn}')\n", "print(f'lambda_to_fs_arn: {lambda_to_fs_arn}')\n", "print(f'predict_lambda_name: {predict_lambda_name}')\n", "print(f'aggregate_feature_group_name: {aggregate_feature_group_name}')\n", "print(f'aggregate_batch_feature_group_name: {aggregate_batch_feature_group_name}')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "try:\n", " response = cf_client.describe_stack_resource(\n", " StackName=STACK_NAME,\n", " LogicalResourceId='MSKCluster'\n", " )\n", " MSKClusterArn = response[\"StackResourceDetail\"][\"PhysicalResourceId\"]\n", "except:\n", " msg = f'CloudFormation stack {STACK_NAME} was not found. Please set the STACK_NAME properly and re-run this cell'\n", " sys.exit(ValueError(msg))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "print(f'msk_cluster_arn: {MSKClusterArn}')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "%store lambda_to_model_arn" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "%store predict_lambda_name" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "%store aggregate_feature_group_name" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "%store aggregate_batch_feature_group_name" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "%store MSKClusterArn" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "from IPython.display import display_html\n", "def restartkernel() :\n", " display_html(\"\",raw=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Imports and other setup" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "from sagemaker import get_execution_role\n", "import sagemaker\n", "import boto3\n", "import json\n", "\n", "role = get_execution_role()\n", "sm = boto3.Session().client(service_name='sagemaker')\n", "smfs_runtime = boto3.Session().client(service_name='sagemaker-featurestore-runtime')" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Create online-only feature groups\n", "When using Amazon SageMaker Feature Store, a core design decision is the definition of feature groups. For our credit card fraud detection use case, we have decided to use two of them:\n", "\n", "1. aggregate_feature_group_name - holds aggregate features that will be updated in near real-time (streaming ingestion)\n", "2. aggregate_batch_feature_group_name - holds aggregate features that will be updated in batch\n", "\n", "Establishing a feature group is a one-time step and is done using the `CreateFeatureGroup` API. \n", "\n", "Feature groups can be created as **online-only**, **offline-only**, or both **online and offline**, which replicates updates from an online store to an offline store in Amazon S3. Since our focus in this example is on demonstrating the use of the feature store for online inference and streaming aggregation of features, we make each of our feature groups online-only.\n", "\n", "In addition to a feature group name, we provide metadata about each feature in the group. We are using a json file to define the schema, but this is not a requirement. We use a schema file to demonstrate how you might capture the feature group definitions, enabling you to recreate them consistently as you move from a development environment to a test or production environment. In our schema file, we also highlight the record identifier and the event timestamp. All feature groups must have these two features, but you get to decide how to name them.\n", "\n", "Here is a visual summary of the feature groups we will create below.\n", "\n", "![SegmentLocal](images/feature_groups.png \"connection\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### cc-agg-fg schema" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "!pygmentize schema/cc-agg-fg-schema.json" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### cc-agg-batch-fg schema" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "!pygmentize schema/cc-agg-batch-fg-schema.json" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Utility functions to simplify creation of feature groups\n", "`schema_to_defs` takes our schema file and returns feature definitions, and the names of the record identifier and event timestamp feature." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "def schema_to_defs(filename):\n", " schema = json.loads(open(filename).read())\n", " \n", " feature_definitions = []\n", " \n", " for col in schema['Features']:\n", " feature = {'FeatureName': col['name']}\n", " if col['type'] == 'double':\n", " feature['FeatureType'] = 'Fractional'\n", " elif col['type'] == 'bigint':\n", " feature['FeatureType'] = 'Integral'\n", " else:\n", " feature['FeatureType'] = 'String'\n", " feature_definitions.append(feature)\n", "\n", " return feature_definitions, schema['record_identifier_feature_name'], schema['event_time_feature_name']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`schema_to_fg` creates a feature group from a schema file. If no s3 URI is passed, an online-only feature group is created." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "def create_feature_group_from_schema(filename, fg_name, role_arn=None, s3_uri=None):\n", " schema = json.loads(open(filename).read())\n", " \n", " feature_defs = []\n", " \n", " for col in schema['features']:\n", " feature = {'FeatureName': col['name']}\n", " if col['type'] == 'double':\n", " feature['FeatureType'] = 'Fractional'\n", " elif col['type'] == 'bigint':\n", " feature['FeatureType'] = 'Integral'\n", " else:\n", " feature['FeatureType'] = 'String'\n", " feature_defs.append(feature)\n", "\n", " record_identifier_name = schema['record_identifier_feature_name']\n", " event_time_name = schema['event_time_feature_name']\n", "\n", " if role_arn is None:\n", " role_arn = get_execution_role()\n", "\n", " if s3_uri is None:\n", " offline_config = {}\n", " else:\n", " offline_config = {'OfflineStoreConfig': {'S3StorageConfig': {'S3Uri': s3_uri}}}\n", " \n", " sm.create_feature_group(\n", " FeatureGroupName = fg_name,\n", " RecordIdentifierFeatureName = record_identifier_name,\n", " EventTimeFeatureName = event_time_name,\n", " FeatureDefinitions = feature_defs,\n", " Description = schema['description'],\n", " Tags = schema['tags'],\n", " OnlineStoreConfig = {'EnableOnlineStore': True},\n", " RoleArn = role_arn,\n", " **offline_config)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Create the two feature groups" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "create_feature_group_from_schema('schema/cc-agg-fg-schema.json', aggregate_feature_group_name)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "create_feature_group_from_schema('schema/cc-agg-batch-fg-schema.json', aggregate_batch_feature_group_name)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Show that the feature store is aware of the new feature groups" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "sm.list_feature_groups()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Describe each feature group\n", "Note that each feature group gets its own ARN, allowing you to manage IAM policies that control access to individual feature groups. The feature names and types are displayed, and the record identifier and event time features are called out specifically. Notice that there is only an `OnlineStoreConfig` and no `OfflineStoreConfig`, as we have decided not to replicate features offline for these groups." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "sm.describe_feature_group(FeatureGroupName=aggregate_feature_group_name)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "sm.describe_feature_group(FeatureGroupName=aggregate_batch_feature_group_name)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Create Source and Destination Kafka topics" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Why install Apache Kafka Libraries?\n", "\n", "Amazon Managed Streaming for Apache Kafka (MSK) lets you use Apache Kafka data-plane operations to create topics and to produce and consume data. In MSK, You can use the AWS Management Console, the AWS Command Line Interface (AWS CLI), or the APIs in the SDK to perform control-plane operations. But since Kafka topic creation is a data-plane activity, we install Apache Kafka client libraries from a separate SageMaker Studio terminal window and execute it. For more details on how MSK works, refer to https://docs.aws.amazon.com/msk/latest/developerguide/what-is-msk.html. The required network and VPC configuration is already setup via the CloudFormation template." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Execute below commands in a separate terminal window in this SageMaker domain. Detailed instructions on how to create topics are available at https://docs.aws.amazon.com/msk/latest/developerguide/create-topic.html." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Retrieve MSK cluster connection string\n", "\n", "The MSK cluster required would be created via the CloudFormation template. Retrieve the connection string for the pre-created MSK cluster using steps shown in this animation. Save off the connection string in a temporary location.\n", "\n", "![SegmentLocal](images/get-msk-cluster-connection-string.gif \"connection\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Apache Kafka client installation from SageMaker Studio terminal window\n", "\n", "\n", "1. Navigate to SageMaker Studio environment.\n", "2. In the top menu, hit \"File\" and choose \"New\" -> \"Terminal\".\n", "3. Once a terminal window is fully available, execute commands in below cells. \n", "4. Replace the *cluster-connection-string* in commands below with Kafka connection string previously saved off in a temporary location." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Prepare the terminal environment\n", "\n", "```\n", "sudo yum -y update\n", "sudo yum -y install java-1.8.0\n", "sudo yum -y install wget\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Create directory for Apache Kafka client download\n", "\n", "```\n", "mkdir kafka\n", "chmod 777 kafka\n", "cd kafka\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Download and install Apache Kafka client\n", "\n", "```\n", "sudo wget https://archive.apache.org/dist/kafka/2.6.2/kafka_2.12-2.6.2.tgz\n", "tar -xzf kafka_2.12-2.6.2.tgz\n", "cd kafka_2.12-2.6.2\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Create source topic (replace cluster-connection-string with MSK cluster connection string stored in temporary location)\n", "\n", "```\n", "bin/kafka-topics.sh --create --bootstrap-server --replication-factor 2 --partitions 1 --topic cctopic\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Create destination topic (replace cluster-connection-string with connection string stored in temporary location)\n", "\n", "```\n", "bin/kafka-topics.sh --create --bootstrap-server --replication-factor 2 --partitions 1 --topic ccdesttopic\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Verify created topics (replace cluster-connection-string with connection string stored in temporary location)\n", "\n", "```\n", "bin/kafka-topics.sh --bootstrap-server --list\n", "```" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "##### To make cleanup easier remove the kafka client. Re-install using steps above if it had to be referred again.\n", "\n", "```\n", "cd ..\n", "rm -rf kafka_2.12-2.6.2\n", "rm -rf kafka_2.12-2.6.2.tgz\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Exit out of the terminal window\n", "\n", "```\n", "exit\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Map the Kafka source topic as an event source for Lambda fraud detection" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "lambda_client = boto3.client('lambda')\n", "\n", "source_topic_name = 'cctopic'\n", "\n", "lambda_client.create_event_source_mapping(EventSourceArn=MSKClusterArn,\n", " FunctionName=lambda_to_model_arn,\n", " StartingPosition='LATEST',\n", " Enabled=True,\n", " Topics=[source_topic_name]\n", " ) #DestinationConfig would handle discarded record" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Map the Kafka destination topic as an event source for Lambda streaming ingest features" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "lambda_client = boto3.client('lambda')\n", "\n", "dest_topic_name = 'ccdesttopic'\n", "\n", "lambda_client.create_event_source_mapping(EventSourceArn=MSKClusterArn,\n", " FunctionName=lambda_to_fs_arn,\n", " StartingPosition='LATEST',\n", " Enabled=True,\n", " Topics=[dest_topic_name]\n", " ) #DestinationConfig would handle discarded records" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Create Amazon Kinesis Data Analytics for Apache Flink application\n", "\n", "Now that we have source/destination MSK topics and have them hooked up to the respective Lambda functions, let us focus on creating a Amazon Kinesis Data Analytics for Apache Flink application using a pre-created Apache Zeppelin studio environment.\n", "\n", "The MSK cluster connection string saved off in a temporary location would be needed again for steps below." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Open the pre-created KDA Studio Environment\n", "\n", "For new projects, we recommend that you use the new Kinesis Data Analytics Studio. Kinesis Data Analytics Studio combines ease of use with advanced analytical capabilities, which makes it possible to build sophisticated stream processing applications in minutes. Let’s see how that works in practice. Studio notebooks for Kinesis Data Analytics allows you to interactively query data streams in real time, and easily build and run stream processing applications using standard SQL, Python, and Scala. With a few clicks in the AWS Management console, you can launch a serverless notebook to query data streams and get results in seconds. In steps below, we navigate to the Kinesis console, run the pre-created KDA Studio environment, upload sample notebook, key in MSK connection strings, build and deploy an executable KDA streaming application.\n", "\n", "![SegmentLocal](images/start-open-kda-studio-environment.gif \"studio\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Download pre-created Zeppelin Notebook\n", "\n", "Download the pre-built Zeppelin notebook (from GitHub) into your local environment. This notebook defines the schema for source and destination Kafka topics. In addition, it also defines the logic for aggregating data in source topic and feeds into the destination topic using Apache Flink Streaming SQL. To learn more on how this works, refer to https://docs.aws.amazon.com/kinesisanalytics/latest/java/how-zeppelin-interactive.html.\n", "\n", "![SegmentLocal](images/download-zeppelin-notebook.gif \"zpln_download\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Upload pre-created Zeppelin Notebook\n", "\n", "Upload the pre-built Zeppelin notebook (from GitHub) into this Studio environment. \n", "\n", "Do not execute invdividual cells in the imported notebook. The notebook would be built and deployed directly from Zeppelin environment. \n", "\n", "![SegmentLocal](images/upload-kda-msk-flink-note-zeppelin.gif \"zpln_upload\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Add MSK cluster connection string to cells\n", "\n", "Using Apache Kafka client libraries, the source and target topics were created in the MSK cluster previously from SageMaker Studio terminal. In this step, the MSK cluster's connection string (saved off in a temporary location) is keyed into the Zeppelin notebook cells. In these cells, the notebook creates the metadata for the topics which is registered via AWS Glue Schema Registry.\n", "\n", "![SegmentLocal](images/key-in-msk-cluster-connection-string.gif \"zpln_connection_string\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Build KDA app from Zeppelin Notebook\n", "\n", "Building a Kinesis Data Analytics Application by building the Zeppelin notebook. This step approximately takes about 5 minutes.\n", "\n", "![SegmentLocal](images/build-kda-app.gif \"zpln_build_app\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "##### Deploy KDA app and run\n", "\n", "Now the application code is built and uploaded to S3. Next steps are to create a KDA application from it and start it. Starting a Kinesis Data Analytics application typically takes several minutes. Ensure that the CloudFormation created role that has text \"KDAStreamingApplicationExecutionRole\" completely or partly in its name.\n", "\n", "![SegmentLocal](images/deploy-run-kda-app.gif \"zpln_deploy_app\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "availableInstances": [ { "_defaultOrder": 0, "_isFastLaunch": true, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 4, "name": "ml.t3.medium", "vcpuNum": 2 }, { "_defaultOrder": 1, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.t3.large", "vcpuNum": 2 }, { "_defaultOrder": 2, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.t3.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 3, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.t3.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 4, "_isFastLaunch": true, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.m5.large", "vcpuNum": 2 }, { "_defaultOrder": 5, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.m5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 6, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.m5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 7, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.m5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 8, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.m5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 9, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.m5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 10, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.m5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 11, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.m5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 12, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.m5d.large", "vcpuNum": 2 }, { "_defaultOrder": 13, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.m5d.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 14, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.m5d.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 15, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.m5d.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 16, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.m5d.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 17, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.m5d.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 18, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.m5d.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 19, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.m5d.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 20, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": true, "memoryGiB": 0, "name": "ml.geospatial.interactive", "supportedImageNames": [ "sagemaker-geospatial-v1-0" ], "vcpuNum": 0 }, { "_defaultOrder": 21, "_isFastLaunch": true, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 4, "name": "ml.c5.large", "vcpuNum": 2 }, { "_defaultOrder": 22, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.c5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 23, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.c5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 24, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.c5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 25, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 72, "name": "ml.c5.9xlarge", "vcpuNum": 36 }, { "_defaultOrder": 26, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 96, "name": "ml.c5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 27, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 144, "name": "ml.c5.18xlarge", "vcpuNum": 72 }, { "_defaultOrder": 28, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.c5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 29, "_isFastLaunch": true, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.g4dn.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 30, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.g4dn.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 31, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.g4dn.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 32, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.g4dn.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 33, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.g4dn.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 34, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.g4dn.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 35, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 61, "name": "ml.p3.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 36, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 244, "name": "ml.p3.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 37, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 488, "name": "ml.p3.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 38, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 768, "name": "ml.p3dn.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 39, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.r5.large", "vcpuNum": 2 }, { "_defaultOrder": 40, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.r5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 41, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.r5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 42, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.r5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 43, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.r5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 44, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.r5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 45, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 512, "name": "ml.r5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 46, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 768, "name": "ml.r5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 47, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.g5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 48, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.g5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 49, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.g5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 50, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.g5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 51, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.g5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 52, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.g5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 53, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.g5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 54, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 768, "name": "ml.g5.48xlarge", "vcpuNum": 192 } ], "instance_type": "ml.g4dn.xlarge", "kernelspec": { "display_name": "Python 3 (Data Science)", "language": "python", "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/datascience-1.0" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.10" }, "vscode": { "interpreter": { "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6" } } }, "nbformat": 4, "nbformat_minor": 4 }