{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Amazon SageMaker Feature Store: How to securely store an image dataset in your Feature Store with a KMS key?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
"\n",
"\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This notebook demonstrates how to securely store a dataset of images into your Feature Store using a KMS key. This is demonstrated using the [MNIST dataset](http://yann.lecun.com/exdb/mnist/). \n",
"\n",
"The example in this notebook starts by retrieving the dataset from an Amazon S3 bucket (you can substitute your own S3 bucket storing your image dataset), and then prepare your dataset for ingestion to an online or offline feature store. We use a [Key Management Service (KMS)](https://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html) key for server-side encryption to ensure that your data is securely stored in your feature store. Finally, we query the ingested dataset from your feature store and then demonstrate how to retrieve your image dataset.\n",
"\n",
"This notebook uses KMS key for server side encryption for your Feature Store. For more information on server-side encryption, see [Feature Store: Encrypt Data in your Online or Offline Feature Store using KMS key](https://sagemaker-examples.readthedocs.io/en/latest/sagemaker-featurestore/feature_store_kms_key_encryption.html). \n",
"\n",
"\n",
"If you would like to encrypt your data on the client side prior to ingestion, see [Amazon SageMaker Feature Store: Client-side Encryption using AWS Encryption SDK](https://sagemaker-examples.readthedocs.io/en/latest/sagemaker-featurestore/feature_store_client_side_encryption.html) for a demonstration. \n",
"\n",
"## Overview\n",
"1. Set up\n",
"2. Load in your image data set\n",
"3. Create Feature Groups and ingest your encrypted data into them\n",
"4. Query your data in your feature store using Amazon Athena\n",
"5. Plot your image data set\n",
"\n",
"## Prerequisites\n",
"This notebook uses the Python SDK library for Feature Store, and the `Python 3 (Data Science)` kernel. To encrypt your data with KMS key for server side encryption, you will need to have an active KMS key. If you do not have a KMS key, then you can create one by following the [KMS Policy Template](https://sagemaker-examples.readthedocs.io/en/latest/sagemaker-featurestore/feature_store_kms_key_encryption.html#KMS-Policy-Template) steps, or you can visit the [KMS section in the console](https://console.aws.amazon.com/kms/home) and follow the button prompts for creating a KMS key. This notebook is compatible with SageMaker Studio, Jupyter, and JupyterLab. \n",
"\n",
"## Library Dependencies:\n",
"* `sagemaker>=2.0.0`\n",
"* `numpy`\n",
"* `pandas`\n",
"* `boto3`\n",
"\n",
"## Data\n",
"This notebook uses the [MNIST dataset](http://yann.lecun.com/exdb/mnist/)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"from time import gmtime, strftime\n",
"from sagemaker.feature_store.feature_group import FeatureGroup\n",
"\n",
"import sagemaker\n",
"import boto3\n",
"import pandas as pd\n",
"import numpy as np\n",
"import pickle\n",
"import gzip\n",
"import time\n",
"import ast\n",
"import matplotlib.pyplot as plt\n",
"import os.path"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Set up"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"sagemaker_session = sagemaker.Session()\n",
"s3_bucket_name = sagemaker_session.default_bucket() # This is the bucket for your offline store.\n",
"public_s3_bucket_name = f\"sagemaker-example-files-prod-{sagemaker_session.boto_region_name}\" # This is the name of the public S3 bucket.\n",
"prefix = \"sagemaker-featurestore-demo\"\n",
"role = sagemaker.get_execution_role()\n",
"region = sagemaker_session.boto_region_name"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Download MNIST\n",
"We are using the MNIST data set. It is stored on a publically available S3 bucket. Below is a method to download a file to your current working directory. We use it to download the MNIST data set from our public S3 bucket that already has the data. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"def download_file_from_s3(bucket, path, filename):\n",
" \"\"\"\n",
" Download filename to your current directory.\n",
" Parameters:\n",
" bucket: S3 bucket name\n",
" path: path to file\n",
" filename: the name of the file you are downloading\n",
" Returns:\n",
" None\n",
" \"\"\"\n",
" if not os.path.exists(filename):\n",
" s3 = boto3.client(\"s3\", region_name=\"us-east-1\")\n",
" s3.download_file(Bucket=bucket, Key=path, Filename=filename)\n",
"\n",
"\n",
"download_file_from_s3(\n",
" public_s3_bucket_name, path=\"datasets/image/MNIST/mnist.pkl.gz\", filename=\"mnist.pkl.gz\"\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Additional - Helper Method\n",
"Below is a method that you can use to get images from your S3 bucket into a `numpy` array. Specifically, if you have `jpg` or `jpeg` images in a S3 bucket that you want to load directly into a `numpy` array, then you can provide the bucket name, `s3_bucket_name`, and prefix path, `prefix_path` to `load_images_into_array` which does just this. Note: This is an additional method that you can use, but we do not use it in this notebook. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"def load_images_into_array(s3_bucket_name, prefix_path):\n",
" \"\"\"\n",
" Return a numpy array of images.\n",
" Parameters:\n",
" s3_bucket_name: S3 bucket name\n",
" prefix_path: path to images in your S3 bucket\n",
" Returns:\n",
" Numpy array.\n",
" \"\"\"\n",
" s3 = boto3.resource(\"s3\")\n",
" bucket = s3.Bucket(s3_bucket_name)\n",
"\n",
" def s3_get_image_paths(bucket, prefix_path, img_exts=[\"jpg\", \"jpeg\"]):\n",
" \"\"\"\n",
" Return a list of paths of images.\n",
" Parameters:\n",
" bucket: S3 bucket name\n",
" prefix_path: path to images in your S3 bucket\n",
" img_exts: image extentions\n",
" Returns:\n",
" A list of paths to images.\n",
" \"\"\"\n",
" img_path_lst = []\n",
" for _ in bucket.objects.filter(Prefix=prefix_path):\n",
" if _.key.endswith(tuple(img_exts)):\n",
" img_path_lst.append(_.key)\n",
" return img_path_lst\n",
"\n",
" img_path_lst = s3_get_image_paths(bucket, prefix_path)\n",
"\n",
" lst = []\n",
" for _ in img_path_lst:\n",
" object = bucket.Object(_)\n",
" response = object.get()\n",
" file_stream = response[\"Body\"]\n",
" lst.append(np.array(Image.open(file_stream)))\n",
" return np.array(lst)\n",
"\n",
"\n",
"# Below demonstrates how to use this method.\n",
"# img_lst = load_images_into_array(s3_bucket_name, prefix_path=image_path)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Unzip and load in dataset"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"with gzip.open(\"mnist.pkl.gz\", \"rb\") as f:\n",
" train_set, validation_set, test_set = pickle.load(f, encoding=\"latin1\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"train_x, train_y = train_set\n",
"# Reshape the image so it can be plotted\n",
"train_x = train_x.reshape(train_x.shape[0], 28, 28)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the following example, we plot a single image."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"plt.imshow(train_x[0])\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Create a data frame of our images.\n",
"We represent the image as a flattened array and also store the original shape of the image in our data frame. Both will be in our data frame that will be ingested into your feature store. \n",
"\n",
"**Important:** At this time, Feature store only supports flattened images with maximum length 350k."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"def create_data_frame(img_lst, col_names=[\"img\", \"shape\"]):\n",
" \"\"\"\n",
" Return a Pandas data frame where each row corresponds to an\n",
" image represented as an array, the original shape of that image and an id.\n",
" Parameters:\n",
" img_lst: a list of images.\n",
" col_names: names of the columns in your data frame\n",
" Returns:\n",
" Pandas data frame.\n",
"\n",
" \"\"\"\n",
" img_col = []\n",
" img_shape_col = []\n",
" ids = []\n",
"\n",
" for index, img in enumerate(img_lst):\n",
" img_flat = img.reshape(-1)\n",
" img_as_str = str(\n",
" np.array2string(img_flat, precision=2, separator=\",\", suppress_small=True)\n",
" ).encode(\"utf-8\")\n",
" img_shape = list(img.shape)\n",
" img_col.append(img_as_str)\n",
" img_shape_col.append(img_shape)\n",
" ids.append(index)\n",
"\n",
" return pd.DataFrame({\"id\": ids, col_names[0]: img_col, col_names[1]: img_shape_col})\n",
"\n",
"\n",
"df = create_data_frame(train_x[:5])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"df.head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"df.dtypes"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"def cast_object_to_string(data_frame):\n",
" \"\"\"\n",
" Cast all columns of data_frame of type object to type string and return it.\n",
" Parameters:\n",
" data_frame: A pandas Dataframe\n",
" Returns:\n",
" Data frame\n",
" \"\"\"\n",
" for label in data_frame.columns:\n",
" if data_frame.dtypes[label] == object:\n",
" data_frame[label] = data_frame[label].astype(\"str\").astype(\"string\")\n",
" return data_frame"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"# Cast columns of df of type object to string.\n",
"df = cast_object_to_string(df)\n",
"\n",
"# Get rid of newlines so it can be ingested into the feature group later\n",
"df.img = df.img.str.replace(\"\\\\n \", \"\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Create your Feature Group and Ingest your data into it\n",
"\n",
"Below we start by appending the `EventTime` feature to your data to timestamp entries, then we load the feature definition, and instantiate the Feature Group object. Then lastly we ingest the data into your feature store."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"feature_group_name = \"mnist-feature-group-\" + strftime(\"%d-%H-%M-%S\", gmtime())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Instantiate a `FeatureGroup` object for your data. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"feature_group = FeatureGroup(name=feature_group_name, sagemaker_session=sagemaker_session)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"record_identifier_feature_name = \"id\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Append the `EventTime` feature to your data frame. This parameter is required, and time stamps each data point."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"current_time_sec = int(round(time.time()))\n",
"event_time_feature_name = \"EventTime\"\n",
"# append EventTime feature\n",
"df[event_time_feature_name] = pd.Series([current_time_sec] * len(df), dtype=\"float64\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Load Feature Definition's of your data into your feature group."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"feature_group.load_feature_definitions(data_frame=df)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Create your feature group.\n",
"\n",
"**Important**: You will need to substitute your KMS Key ARN for `kms_key` for server side encryption (SSE). The cell below demonstrates how to enable SSE for an offline store. If you choose to use an online store, you will need to assign `enable_online_store` to `True`. To enable SSE for an online store you will need to assign `online_store_kms_key_id` to your KMS key. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"feature_group.create(\n",
" s3_uri=f\"s3://{s3_bucket_name}/{prefix}\",\n",
" record_identifier_name=record_identifier_feature_name,\n",
" event_time_feature_name=\"EventTime\",\n",
" role_arn=role,\n",
" enable_online_store=False,\n",
" offline_store_kms_key_id=kms_key, # Substitute kms_key with your kms key.\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"feature_group.describe()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Continually check your offline store until your data is available in it."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"def check_feature_group_status(feature_group):\n",
" \"\"\"\n",
" Print when the feature group has been successfully created\n",
" Parameters:\n",
" feature_group: FeatureGroup\n",
" Returns:\n",
" None\n",
" \"\"\"\n",
" status = feature_group.describe().get(\"FeatureGroupStatus\")\n",
" while status == \"Creating\":\n",
" print(\"Waiting for Feature Group to be Created\")\n",
" time.sleep(5)\n",
" status = feature_group.describe().get(\"FeatureGroupStatus\")\n",
" print(f\"FeatureGroup {feature_group.name} successfully created.\")\n",
"\n",
"\n",
"check_feature_group_status(feature_group)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Ingest your data into your feature group."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"feature_group.ingest(data_frame=df, max_workers=5, wait=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"time.sleep(30)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"s3_client = sagemaker_session.boto_session.client(\"s3\", region_name=region)\n",
"\n",
"feature_group_s3_uri = (\n",
" feature_group.describe()\n",
" .get(\"OfflineStoreConfig\")\n",
" .get(\"S3StorageConfig\")\n",
" .get(\"ResolvedOutputS3Uri\")\n",
")\n",
"\n",
"feature_group_s3_prefix = feature_group_s3_uri.replace(f\"s3://{s3_bucket_name}/\", \"\")\n",
"offline_store_contents = None\n",
"while offline_store_contents is None:\n",
" objects_in_bucket = s3_client.list_objects(\n",
" Bucket=s3_bucket_name, Prefix=feature_group_s3_prefix\n",
" )\n",
" if \"Contents\" in objects_in_bucket and len(objects_in_bucket[\"Contents\"]) > 1:\n",
" offline_store_contents = objects_in_bucket[\"Contents\"]\n",
" else:\n",
" print(\"Waiting for data in offline store...\\n\")\n",
" time.sleep(60)\n",
"\n",
"print(\"Data available.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Use Amazon Athena to Query your Encrypted Data in your Feature Store\n",
"Using Amazon Athena, we query the image data set that we stored in our feature store to demonstrate how to extract your data set of images."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"query = feature_group.athena_query()\n",
"table = query.table_name\n",
"query_table = 'SELECT * FROM \"' + table + '\"'\n",
"print(\"Running \" + query_table)\n",
"# Run the Athena query\n",
"query.run(\n",
" query_string=query_table,\n",
" output_location=\"s3://\" + s3_bucket_name + \"/\" + prefix + \"/query_results/\",\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"time.sleep(60)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"dataset = query.as_dataframe()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"print(dataset.dtypes)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Below is the data queried from your feature store."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"dataset"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"def parse_show_image(df):\n",
" \"\"\"\n",
" Return a numpy array of your images that have been reshaped into it's corresponding shape.\n",
" Parameters:\n",
" df: dataframe of your data\n",
" Returns:\n",
" Numpy array\n",
" \"\"\"\n",
" import ast\n",
"\n",
" images = []\n",
" for index, entry in enumerate(np.array(df[\"img\"])):\n",
" entry = entry.strip(\"b\").strip(\"'\").replace(\"\\\\n\", \"\")\n",
" entry = np.array(ast.literal_eval(entry))\n",
" shape = ast.literal_eval(df[\"shape\"][index])\n",
" entry = entry.reshape(shape[0], shape[1])\n",
" images.append(entry)\n",
" return np.array(images)\n",
"\n",
"\n",
"images = parse_show_image(dataset)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"# Below shows the shape of your image data set.\n",
"images.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Plot the images to demonstrate that you can view the images stored in your feature store. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"for img in images:\n",
" plt.imshow(img)\n",
" plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Clean up resources\n",
"Remove the Feature Group that was created. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"feature_group.delete()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Next steps\n",
"\n",
"In this notebook we covered how to securely store data sets of images in a feature store using KMS key.\n",
"\n",
"If you are interested in understanding more on how server-side encryption is done with Feature Store, see [Feature Store: Encrypt Data in your Online or Offline Feature Store using KMS key](https://sagemaker-examples.readthedocs.io/en/latest/sagemaker-featurestore/feature_store_kms_key_encryption.html).\n",
"\n",
"If you are interested in understanding how to do client-side encryption to encrypt your image data set prior to storing it in your feature store, see [Amazon SageMaker Feature Store: Client-side Encryption using AWS Encryption SDK](https://sagemaker-examples.readthedocs.io/en/latest/sagemaker-featurestore/feature_store_client_side_encryption.html). For more information on the AWS Encryption library, see [AWS Encryption SDK library](https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/introduction.html).\n",
"\n",
"For detailed information about Feature Store, see the [Developer Guide](https://docs.aws.amazon.com/sagemaker/latest/dg/feature-store.html)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Notebook CI Test Results\n",
"\n",
"This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n"
]
}
],
"metadata": {
"availableInstances": [
{
"_defaultOrder": 0,
"_isFastLaunch": true,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 4,
"name": "ml.t3.medium",
"vcpuNum": 2
},
{
"_defaultOrder": 1,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 8,
"name": "ml.t3.large",
"vcpuNum": 2
},
{
"_defaultOrder": 2,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 16,
"name": "ml.t3.xlarge",
"vcpuNum": 4
},
{
"_defaultOrder": 3,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 32,
"name": "ml.t3.2xlarge",
"vcpuNum": 8
},
{
"_defaultOrder": 4,
"_isFastLaunch": true,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 8,
"name": "ml.m5.large",
"vcpuNum": 2
},
{
"_defaultOrder": 5,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 16,
"name": "ml.m5.xlarge",
"vcpuNum": 4
},
{
"_defaultOrder": 6,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 32,
"name": "ml.m5.2xlarge",
"vcpuNum": 8
},
{
"_defaultOrder": 7,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 64,
"name": "ml.m5.4xlarge",
"vcpuNum": 16
},
{
"_defaultOrder": 8,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 128,
"name": "ml.m5.8xlarge",
"vcpuNum": 32
},
{
"_defaultOrder": 9,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 192,
"name": "ml.m5.12xlarge",
"vcpuNum": 48
},
{
"_defaultOrder": 10,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 256,
"name": "ml.m5.16xlarge",
"vcpuNum": 64
},
{
"_defaultOrder": 11,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 384,
"name": "ml.m5.24xlarge",
"vcpuNum": 96
},
{
"_defaultOrder": 12,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 8,
"name": "ml.m5d.large",
"vcpuNum": 2
},
{
"_defaultOrder": 13,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 16,
"name": "ml.m5d.xlarge",
"vcpuNum": 4
},
{
"_defaultOrder": 14,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 32,
"name": "ml.m5d.2xlarge",
"vcpuNum": 8
},
{
"_defaultOrder": 15,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 64,
"name": "ml.m5d.4xlarge",
"vcpuNum": 16
},
{
"_defaultOrder": 16,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 128,
"name": "ml.m5d.8xlarge",
"vcpuNum": 32
},
{
"_defaultOrder": 17,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 192,
"name": "ml.m5d.12xlarge",
"vcpuNum": 48
},
{
"_defaultOrder": 18,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 256,
"name": "ml.m5d.16xlarge",
"vcpuNum": 64
},
{
"_defaultOrder": 19,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 384,
"name": "ml.m5d.24xlarge",
"vcpuNum": 96
},
{
"_defaultOrder": 20,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": true,
"memoryGiB": 0,
"name": "ml.geospatial.interactive",
"supportedImageNames": [
"sagemaker-geospatial-v1-0"
],
"vcpuNum": 0
},
{
"_defaultOrder": 21,
"_isFastLaunch": true,
"category": "Compute optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 4,
"name": "ml.c5.large",
"vcpuNum": 2
},
{
"_defaultOrder": 22,
"_isFastLaunch": false,
"category": "Compute optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 8,
"name": "ml.c5.xlarge",
"vcpuNum": 4
},
{
"_defaultOrder": 23,
"_isFastLaunch": false,
"category": "Compute optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 16,
"name": "ml.c5.2xlarge",
"vcpuNum": 8
},
{
"_defaultOrder": 24,
"_isFastLaunch": false,
"category": "Compute optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 32,
"name": "ml.c5.4xlarge",
"vcpuNum": 16
},
{
"_defaultOrder": 25,
"_isFastLaunch": false,
"category": "Compute optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 72,
"name": "ml.c5.9xlarge",
"vcpuNum": 36
},
{
"_defaultOrder": 26,
"_isFastLaunch": false,
"category": "Compute optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 96,
"name": "ml.c5.12xlarge",
"vcpuNum": 48
},
{
"_defaultOrder": 27,
"_isFastLaunch": false,
"category": "Compute optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 144,
"name": "ml.c5.18xlarge",
"vcpuNum": 72
},
{
"_defaultOrder": 28,
"_isFastLaunch": false,
"category": "Compute optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 192,
"name": "ml.c5.24xlarge",
"vcpuNum": 96
},
{
"_defaultOrder": 29,
"_isFastLaunch": true,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 16,
"name": "ml.g4dn.xlarge",
"vcpuNum": 4
},
{
"_defaultOrder": 30,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 32,
"name": "ml.g4dn.2xlarge",
"vcpuNum": 8
},
{
"_defaultOrder": 31,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 64,
"name": "ml.g4dn.4xlarge",
"vcpuNum": 16
},
{
"_defaultOrder": 32,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 128,
"name": "ml.g4dn.8xlarge",
"vcpuNum": 32
},
{
"_defaultOrder": 33,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 4,
"hideHardwareSpecs": false,
"memoryGiB": 192,
"name": "ml.g4dn.12xlarge",
"vcpuNum": 48
},
{
"_defaultOrder": 34,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 256,
"name": "ml.g4dn.16xlarge",
"vcpuNum": 64
},
{
"_defaultOrder": 35,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 61,
"name": "ml.p3.2xlarge",
"vcpuNum": 8
},
{
"_defaultOrder": 36,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 4,
"hideHardwareSpecs": false,
"memoryGiB": 244,
"name": "ml.p3.8xlarge",
"vcpuNum": 32
},
{
"_defaultOrder": 37,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 8,
"hideHardwareSpecs": false,
"memoryGiB": 488,
"name": "ml.p3.16xlarge",
"vcpuNum": 64
},
{
"_defaultOrder": 38,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 8,
"hideHardwareSpecs": false,
"memoryGiB": 768,
"name": "ml.p3dn.24xlarge",
"vcpuNum": 96
},
{
"_defaultOrder": 39,
"_isFastLaunch": false,
"category": "Memory Optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 16,
"name": "ml.r5.large",
"vcpuNum": 2
},
{
"_defaultOrder": 40,
"_isFastLaunch": false,
"category": "Memory Optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 32,
"name": "ml.r5.xlarge",
"vcpuNum": 4
},
{
"_defaultOrder": 41,
"_isFastLaunch": false,
"category": "Memory Optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 64,
"name": "ml.r5.2xlarge",
"vcpuNum": 8
},
{
"_defaultOrder": 42,
"_isFastLaunch": false,
"category": "Memory Optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 128,
"name": "ml.r5.4xlarge",
"vcpuNum": 16
},
{
"_defaultOrder": 43,
"_isFastLaunch": false,
"category": "Memory Optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 256,
"name": "ml.r5.8xlarge",
"vcpuNum": 32
},
{
"_defaultOrder": 44,
"_isFastLaunch": false,
"category": "Memory Optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 384,
"name": "ml.r5.12xlarge",
"vcpuNum": 48
},
{
"_defaultOrder": 45,
"_isFastLaunch": false,
"category": "Memory Optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 512,
"name": "ml.r5.16xlarge",
"vcpuNum": 64
},
{
"_defaultOrder": 46,
"_isFastLaunch": false,
"category": "Memory Optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 768,
"name": "ml.r5.24xlarge",
"vcpuNum": 96
},
{
"_defaultOrder": 47,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 16,
"name": "ml.g5.xlarge",
"vcpuNum": 4
},
{
"_defaultOrder": 48,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 32,
"name": "ml.g5.2xlarge",
"vcpuNum": 8
},
{
"_defaultOrder": 49,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 64,
"name": "ml.g5.4xlarge",
"vcpuNum": 16
},
{
"_defaultOrder": 50,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 128,
"name": "ml.g5.8xlarge",
"vcpuNum": 32
},
{
"_defaultOrder": 51,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 256,
"name": "ml.g5.16xlarge",
"vcpuNum": 64
},
{
"_defaultOrder": 52,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 4,
"hideHardwareSpecs": false,
"memoryGiB": 192,
"name": "ml.g5.12xlarge",
"vcpuNum": 48
},
{
"_defaultOrder": 53,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 4,
"hideHardwareSpecs": false,
"memoryGiB": 384,
"name": "ml.g5.24xlarge",
"vcpuNum": 96
},
{
"_defaultOrder": 54,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 8,
"hideHardwareSpecs": false,
"memoryGiB": 768,
"name": "ml.g5.48xlarge",
"vcpuNum": 192
},
{
"_defaultOrder": 55,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 8,
"hideHardwareSpecs": false,
"memoryGiB": 1152,
"name": "ml.p4d.24xlarge",
"vcpuNum": 96
},
{
"_defaultOrder": 56,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 8,
"hideHardwareSpecs": false,
"memoryGiB": 1152,
"name": "ml.p4de.24xlarge",
"vcpuNum": 96
}
],
"kernelspec": {
"display_name": "Python 3 (Data Science 3.0)",
"language": "python",
"name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/sagemaker-data-science-310-v1"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.6"
}
},
"nbformat": 4,
"nbformat_minor": 4
}