{ "cells": [ { "cell_type": "markdown", "id": "65bd2951", "metadata": {}, "source": [ "# Module 2: Speed ML Development with Apache Iceberg offline store compaction\n", "\n", "Feature Store supports Apache Iceberg as a table format for storing features. This accelerates model development by enabling faster query performance when extracting ML training datasets, taking advantage of Iceberg table compaction. Depending on the design of your feature groups and their scale, you can experience training query performance improvements of 10x to 100x by using this new capability.\n", "\n", "**Note:** Please set kernel to `Python 3 (Data Science)` and select instance to `ml.m5.4xlarge`\n", "\n", "**This notebook uses the feature groups created in `module-2` and only applies to feature groups created with table format `Iceberg`**" ] }, { "cell_type": "markdown", "id": "93001c3d", "metadata": {}, "source": [ "---\n", "\n", "## Contents\n", "\n", "1. [Background](#Background)\n", "1. [Setup](#Setup)\n", "1. [Compaction](#Compaction)\n", " 1. [Compaction using Athena](#Compaction-using-Athena)\n", " 1. [Compaction using Spark](#Compaction-using-Spark)\n", "1. [Scheduled Compaction](#Scheduled-Compaction)" ] }, { "cell_type": "markdown", "id": "5fb98418", "metadata": {}, "source": [ "# Background" ] }, { "cell_type": "markdown", "id": "358b68fc", "metadata": {}, "source": [ "In this notebook, you will learn how to schedule table maintenance operations (compaction, storage reclaim) using Amazon Athena and AWS Glue\n", "\n" ] }, { "cell_type": "markdown", "id": "7210e3bf", "metadata": {}, "source": [ "[Apache Iceberg](https://iceberg.apache.org/) is an open table format for very large analytic datasets. It manages large collections of files as tables, and it supports modern analytical data lake operations such as record-level insert, update, delete, and time travel queries. \n", "\n", "With Feature Store, you can create feature groups with Iceberg table format as an alternative to the default standard Glue format. With that, customers can leverage the table format to use Iceberg’s file compaction and data pruning features as per their use case and optimization requirements. Iceberg also enables customers to perform deletion, time-travel queries, high concurrent transactions, and higher performance of selective queries.\n", "\n", "By combining Iceberg as a table format and table maintenance operations such as compaction, customer will benefit from faster query performance when working with offline feature groups at scale and as a result help customer build training dataset faster. \n", "\n", "The following diagram shows the structure of the offline store using Iceberg as a table format.\n", "\n", "![Offline File Compaction](../images/smfs_apache_iceberg_compaction.png \"Offline File Compaction\")\n", "\n", "In this notebook, you will learn how execute Iceberg’s table management procedures using AWS Athena and use AWS services to schedule these tasks to run on-demand or on a schedule." ] }, { "cell_type": "markdown", "id": "8f23e6d1", "metadata": { "tags": [] }, "source": [ "# Setup" ] }, { "cell_type": "markdown", "id": "614e4a31", "metadata": {}, "source": [ "#### Imports" ] }, { "cell_type": "code", "execution_count": null, "id": "ee622efb", "metadata": {}, "outputs": [], "source": [ "from sagemaker.feature_store.feature_group import FeatureGroup\n", "from sagemaker import get_execution_role\n", "import sagemaker\n", "import logging\n", "import boto3\n", "import pandas as pd\n", "import time\n", "import re\n", "import os\n", "import sys\n", "import subprocess\n", "import importlib\n", "sys.path.append('..')\n", "from utilities import Utils\n", "from utilities import feature_store_helper" ] }, { "cell_type": "code", "execution_count": null, "id": "feb1aa77-0960-446f-aa24-2a417fb45429", "metadata": { "tags": [] }, "outputs": [], "source": [ "sm_version = sagemaker.__version__\n", "major, minor, patch = sm_version.split('.')\n", "if int(major) < 2 or int(minor) < 125:\n", " subprocess.check_call([sys.executable, '-m', 'pip', 'install', 'sagemaker==2.125.0'])\n", " importlib.reload(sagemaker)" ] }, { "cell_type": "code", "execution_count": null, "id": "c25ec110", "metadata": {}, "outputs": [], "source": [ "logger = logging.getLogger('__name__')\n", "logger.setLevel(logging.DEBUG)\n", "logger.addHandler(logging.StreamHandler())" ] }, { "cell_type": "code", "execution_count": null, "id": "85c2978c", "metadata": {}, "outputs": [], "source": [ "logger.info(f'Using SageMaker version: {sagemaker.__version__}')\n", "logger.info(f'Using Pandas version: {pd.__version__}')" ] }, { "cell_type": "markdown", "id": "16eea690", "metadata": {}, "source": [ "#### Essentials" ] }, { "cell_type": "code", "execution_count": null, "id": "2104e08e", "metadata": {}, "outputs": [], "source": [ "sagemaker_session = sagemaker.Session()\n", "role = sagemaker.get_execution_role()\n", "default_bucket = sagemaker_session.default_bucket()\n", "region = sagemaker_session.boto_region_name\n", "logger.info(f'Default S3 bucket = {default_bucket}')\n", "prefix = 'sagemaker-feature-store'\n", "\n", "boto_session = boto3.Session(region_name=region)\n", "sagemaker_client = boto_session.client(service_name='sagemaker', region_name=region)\n", "featurestore_runtime = boto_session.client(service_name='sagemaker-featurestore-runtime', region_name=region)\n", "\n", "feature_store_session = sagemaker.Session(boto_session=boto_session, \n", " sagemaker_client=sagemaker_client, \n", " sagemaker_featurestore_runtime_client=featurestore_runtime)" ] }, { "cell_type": "markdown", "id": "1caeeb98", "metadata": {}, "source": [ "#### Retrieve Feature Group" ] }, { "cell_type": "code", "execution_count": null, "id": "e76d425f-8a3c-4cc9-8be7-b3ea5f0fa1ab", "metadata": { "tags": [] }, "outputs": [], "source": [ "# Retrieve FG name\n", "%store -r orders_feature_group_name\n", "orders_fg = FeatureGroup(name=orders_feature_group_name, sagemaker_session=feature_store_session)" ] }, { "cell_type": "markdown", "id": "efa27c05", "metadata": {}, "source": [ "#### Query Data" ] }, { "cell_type": "code", "execution_count": null, "id": "3a4b2242", "metadata": {}, "outputs": [], "source": [ "orders_query = orders_fg.athena_query()\n", "orders_table = orders_query.table_name\n", "orders_database = orders_query.database" ] }, { "cell_type": "code", "execution_count": null, "id": "fc86e2c3", "metadata": {}, "outputs": [], "source": [ "query_string = f'SELECT * ' \\\n", " f'FROM' \\\n", " f' (SELECT *,' \\\n", " f' row_number()' \\\n", " f' OVER (PARTITION BY \"order_id\"' \\\n", " f' ORDER BY event_time desc, Api_Invocation_Time DESC, write_time DESC) AS row_num' \\\n", " f' FROM \"{orders_database}\".\"{orders_table}\")' \\\n", " f'WHERE row_num = 1 and NOT is_deleted limit 10000;'\n", "\n", "%store query_string\n", "query_string" ] }, { "cell_type": "code", "execution_count": null, "id": "2f57db5e", "metadata": {}, "outputs": [], "source": [ "query_results= 'sagemaker-featurestore/athena-results'\n", "output_location = f's3://{default_bucket}/{query_results}/query_results/'\n", "print(f'Athena query output location: \\n{output_location}')" ] }, { "cell_type": "code", "execution_count": null, "id": "ea662c6e", "metadata": {}, "outputs": [], "source": [ "%%timeit\n", "orders_query.run(query_string=query_string, output_location=output_location)\n", "orders_query.wait()" ] }, { "cell_type": "code", "execution_count": null, "id": "32c65223-f517-437c-83b6-9320f323ee75", "metadata": {}, "outputs": [], "source": [ "joined_df = orders_query.as_dataframe()\n", "joined_df.head()" ] }, { "cell_type": "markdown", "id": "de783a36-6e39-424b-8e73-7063f3bfebd7", "metadata": {}, "source": [ "# Compaction" ] }, { "cell_type": "markdown", "id": "ee893aaf-b01b-42de-81e1-fffe3fa75791", "metadata": {}, "source": [ "## Compaction using Athena" ] }, { "cell_type": "markdown", "id": "a479d47f-be68-4cf9-820d-49a4d975c301", "metadata": {}, "source": [ "[Amazon Athena](https://aws.amazon.com/athena/) is a serverless SQL query engine that natively supports Iceberg management procedures. In this section, you will use Athena to manually compact the offline feature group you created. \n", "\n", "Note you will need to use [Athena engine version 3](https://docs.aws.amazon.com/athena/latest/ug/engine-versions-reference-0003.html). For this, you can create a new workgroup, or configure an existing workgroup, and select the recommended Athena engine version 3. For more information and instructions for changing your Athena engine version, refer to Changing Athena engine versions.\n", "\n", "![Offline File Compaction](../images/smfs_athena_workgroup.png \"Offline File Compaction\")" ] }, { "cell_type": "markdown", "id": "38a75237-63fd-402f-85cf-6b4a4ed2e9a3", "metadata": { "tags": [] }, "source": [ "#### Query pre compaction" ] }, { "cell_type": "markdown", "id": "4e94e37b-1328-4de4-83ba-730b1be36c6e", "metadata": {}, "source": [ "The $partitions table provides a detailed overview of the partitions of the Iceberg table. You can retrieve the information about the partitions of the Iceberg table. Let's have a look at the number of files and total bytes size (pre-compaction)." ] }, { "cell_type": "code", "execution_count": null, "id": "b27ab824-a3b7-4f23-acd5-d7f744f4a705", "metadata": { "tags": [] }, "outputs": [], "source": [ "stats_query_string = f'SELECT sum(total_size), sum(file_count) ' \\\n", " f'FROM \"{orders_table}$partitions\"' \\\n", "\n", "%store stats_query_string\n", "stats_query_string" ] }, { "cell_type": "code", "execution_count": null, "id": "2aa616f2-ff81-4211-b775-2d0ad68f6ab8", "metadata": { "tags": [] }, "outputs": [], "source": [ "%%timeit\n", "orders_query.run(query_string=stats_query_string, output_location=output_location)\n", "orders_query.wait()" ] }, { "cell_type": "code", "execution_count": null, "id": "a0f37039-a86a-401e-966b-26ed15ee4712", "metadata": { "tags": [] }, "outputs": [], "source": [ "stats_joined_df = orders_query.as_dataframe()\n", "stats_joined_df.head()" ] }, { "cell_type": "markdown", "id": "36ee91c8-b568-4796-8876-99d0c2511245", "metadata": {}, "source": [ "#### Iceberg Compaction Procedures" ] }, { "cell_type": "markdown", "id": "80b6bccb-ba61-443c-b6f9-a48abfdb8bae", "metadata": {}, "source": [ "As data accumulates into an Iceberg table, queries gradually become less efficient because of the increased processing time required to open additional files. Compactions optimizes the structural layout of the table without altering table content. \n", "\n", "For this, you will use the `OPTIMIZE table REWRITE DATA` compaction table maintenance command in Athena. The following syntax shows how to optimize data layout of a Feature Group stored using the Iceberg table format. See [documentation](https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.html) for more detail.\n", "\n", "```sql\n", "OPTIMIZE \"sagemaker_featurestore\".\"\" REWRITE DATA USING BIN_PACK\n", "```" ] }, { "cell_type": "markdown", "id": "9b6c4a61-4037-4de2-b672-b4b26d1d21f4", "metadata": {}, "source": [ "
💡 Iceberg Optimize more Runs needed: \n", "In the event the table could not be compacted within the quey execution timeout for Athena, you will receive an error ICEBERG_OPTIMIZE_MORE_RUNS_NEEDED. This happens if the entire table could not be compacted within the execution time and you should rerun the command, to resume compaction for remaining partitions.\n", "
\n", "\n", "![Offline File Compaction](../images/smfs_optimize_query_more_runs_needed.png \"Offline File Compaction\")" ] }, { "cell_type": "markdown", "id": "d79a8845-03dd-4e5d-9fdd-6b30fd50a2f3", "metadata": {}, "source": [ "After running the optimize command, you will use the `VACUUM` procedure which performs snapshot expiration and removal of orphan files. These actions reduce metadata size and remove files not in the current table state that are also older than the retention period specified for the table. See [documentation](https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.html) for more detail.\n", "\n", "```sql\n", "VACUUM \"sagemaker_featurestore\".\"\"\n", "```" ] }, { "cell_type": "markdown", "id": "738cbec8-a383-4eda-825e-6a5686f2c91f", "metadata": {}, "source": [ "We created a script to run the procedures in Python. Let's explore it.\n", "\n", "* The script executes the OPTIMIZE query using boto3.\n", "* It handles the error message error ICEBERG_OPTIMIZE_MORE_RUNS_NEEDED which requires you to resume compaction for remaining partitions.\n", "* It executes VACUUM query if the OPTIMIZE query is successful." ] }, { "cell_type": "code", "execution_count": null, "id": "1bd3dc32-e48e-4361-be18-fca47f33971b", "metadata": {}, "outputs": [], "source": [ "!pip3 install pyspark\n", "!pip3 install git+https://github.com/awslabs/aws-glue-libs.git" ] }, { "cell_type": "code", "execution_count": null, "id": "f44b0883-e451-4466-90fd-f88a8342d13b", "metadata": {}, "outputs": [], "source": [ "!pygmentize ./smfs_offline_compaction.py" ] }, { "cell_type": "markdown", "id": "c0cc7147-ac6a-4932-bc90-55635c3c6bcc", "metadata": {}, "source": [ "#### Execute the compaction procedures" ] }, { "cell_type": "code", "execution_count": null, "id": "e548e8f5-faae-458f-9cfd-53524e00d7cd", "metadata": {}, "outputs": [], "source": [ "!python smfs_offline_compaction.py --region {region} --database {orders_database} --table {orders_table} --workgroup primary --outputlocation {output_location}" ] }, { "cell_type": "markdown", "id": "dbf92b3d-a803-4841-9293-0156f94783d2", "metadata": {}, "source": [ "#### Query post compaction" ] }, { "cell_type": "code", "execution_count": null, "id": "5838a56b-86d3-43d8-af14-8458df0f1714", "metadata": {}, "outputs": [], "source": [ "%%timeit\n", "orders_query.run(query_string=query_string, output_location=output_location)\n", "orders_query.wait()" ] }, { "cell_type": "code", "execution_count": null, "id": "183e8e05-0ef8-480f-b9c1-da9a73b4b855", "metadata": {}, "outputs": [], "source": [ "joined_df = orders_query.as_dataframe()\n", "joined_df.head()" ] }, { "cell_type": "markdown", "id": "df2e0f9b-2dc0-49f6-8822-445112be5852", "metadata": {}, "source": [ "Depending on the design of your feature groups and their scale, you can experience training query performance improvements of 10x to 100x by using this new capability. Since the dataset used in this example is small, you might not see a noticeable improvement in query performance. Let's have a look at the number of files and total bytes size post compaction." ] }, { "cell_type": "code", "execution_count": null, "id": "f1575301-8bde-48c9-993a-74f5eb65eafc", "metadata": {}, "outputs": [], "source": [ "%%timeit\n", "orders_query.run(query_string=stats_query_string, output_location=output_location)\n", "orders_query.wait()" ] }, { "cell_type": "code", "execution_count": null, "id": "750d486d-e5b4-48bf-aaf9-55412e3e4bad", "metadata": {}, "outputs": [], "source": [ "stats_joined_df = orders_query.as_dataframe()\n", "stats_joined_df.head()" ] }, { "cell_type": "markdown", "id": "caf84c93-6a7b-4cd4-86ae-2ba7057f8445", "metadata": {}, "source": [ "## Compaction using Spark" ] }, { "cell_type": "markdown", "id": "a51b23dc-2d7e-4216-bd55-163256fa11d7", "metadata": {}, "source": [ "Customers can also use Spark to manage the compaction jobs and maintenance methods. For more detail on the Spark procedures, see the [documentation](https://iceberg.apache.org/docs/latest/spark-procedures/).\n", "\n", "You first need to configure some of the common properties.\n", "\n", "``` sql\n", "%%configure -f\n", "{\n", " \"conf\": {\n", " \"spark.sql.catalog.smfs\": \"org.apache.iceberg.spark.SparkCatalog\",\n", " \"spark.sql.catalog.smfs.catalog-impl\": \"org.apache.iceberg.aws.glue.GlueCatalog\",\n", " \"spark.sql.catalog.smfs.warehouse\": \"\",\n", " \"spark.sql.extensions\":\"org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions\",\n", " \"spark.sql.catalog.smfs.glue.skip-name-validation\": \"true\"\n", " }\n", "}\n", "```\n", "\n", "The following code can be used to optimize the feature groups via Spark.\n", "\n", "``` sql\n", "spark.sql(f\"\"\"CALL smfs.system.rewrite_data_files(table => '{DATABASE}.`{ICEBERG_TABLE}`')\"\"\")\n", "```\n", "\n", "You can then execute the next two table maintenance procedures to remove older snapshots and orphan files which are no longer needed.\n", "\n", "``` sql\n", "spark.sql(f\"\"\"CALL smfs.system.expire_snapshots(table => '{DATABASE}.`{ICEBERG_TABLE}`', older_than => TIMESTAMP '{one_day_ago}', retain_last => 1)\"\"\")\n", "spark.sql(f\"\"\"CALL smfs.system.remove_orphan_files(table => '{DATABASE}.`{ICEBERG_TABLE}`')\"\"\")\n", "```\n", "\n", "You can then incorporate the above Spark commands into your Spark environment. For example, you can create a job that performs the optimization above on a desired schedule or in a pipeline after ingestion. \n" ] }, { "cell_type": "markdown", "id": "7a6362b8-5714-4cbb-8af1-cad0cbb6be26", "metadata": {}, "source": [ "# Scheduled Compaction" ] }, { "cell_type": "markdown", "id": "79fb6799-f796-4d14-b5c9-e2123f3d2d42", "metadata": {}, "source": [ "In this section, you will learn how to automate the table management procedures to compact your offline feature store. The diagram below illustrates the architecture for creating Feature Groups in Iceberg table format and a fully automated table management solution which includes file compaction and cleanup operations. \n", "\n", "![Offline File Compaction](../images/smfs_scheduled_compaction.png \"Offline File Compaction\")" ] }, { "cell_type": "markdown", "id": "8246db6b-6c61-4fbc-b86a-c24cb30ecf4a", "metadata": {}, "source": [ "* At a high level, you created a Feature Group using the Iceberg table format and ingested records into the online feature store in the previous section. \n", "* Feature values were automatically replicated from the online store to the historical offline store. \n", "* [Athena](https://aws.amazon.com/athena/) will be used to run the Iceberg management procedures.\n", "* To schedule the procedures, you will setup a [AWS Glue](https://aws.amazon.com/glue/) job using a Python shell script and create an AWS Glue job schedule. " ] }, { "cell_type": "markdown", "id": "d7c8a568-90db-49c6-a7e0-f90560769189", "metadata": {}, "source": [ "### AWS Glue Job Setup" ] }, { "cell_type": "markdown", "id": "75aedf0e-cc37-46f0-b801-db7aecaaefdf", "metadata": {}, "source": [ "#### Create IAM Role" ] }, { "cell_type": "markdown", "id": "26c8cc84-414b-45da-924d-62c8d5c0cd9c", "metadata": {}, "source": [ "First you need to create a role for AWS Glue to have permissions to access Amazon Athena, Amazon S3, and CloudWatch.\n", "* Navigate to the IAM console.\n", "* Create an AWS Identity and Access Management (IAM) service role for AWS Glue. \n", "* Attach a policy that allows access to Athena, Amazon Simple Storage Service (Amazon S3), and Amazon CloudWatch Logs. For example, you can add [AmazonAthenaFullAccess](https://docs.aws.amazon.com/athena/latest/ug/managed-policies.html#amazonathenafullaccess-managed-policy) and [CloudWatchLogsFullAccess](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/iam-identity-based-access-control-cwl.html#managed-policies-cwl) to the role. AmazonAthenaFullAccess allows full access to Athena and includes basic permissions for Amazon S3. CloudWatchLogsFullAccess allows full access to CloudWatch Logs.\n" ] }, { "cell_type": "markdown", "id": "67fbfc42-3462-4c2e-b39e-78b69945461d", "metadata": {}, "source": [ "![Offline File Compaction](../images/smfs_aws_glue_role.png \"Offline File Compaction\")" ] }, { "cell_type": "markdown", "id": "9a006ea4-b676-4cbf-acca-3fa9829a8cca", "metadata": {}, "source": [ "#### Create AWS Glue Job" ] }, { "cell_type": "markdown", "id": "c1465da5-2627-46c1-bb0f-c6b92b660078", "metadata": {}, "source": [ "Next, you need to create the actual AWS Glue Job.\n", "\n", "* Navigate to the AWS Glue console.\n", "* Choose the **Jobs** tab under AWS Glue Studio.\n", "* Select **Python Shell script editor**. \n", "* Choose to *Upload and edit an existing script*. Use the script located in this module [smfs_offine_compaction.py](./smfs_offine_compaction.py)\n", "* Click **Create**.\n", "* Edit the name of the job (as per screenshot below).\n", "\n", "![Offline File Compaction](../images/smfs_glue_script_upload.png \"Offline File Compaction\")\n", "\n" ] }, { "cell_type": "markdown", "id": "b2bbb4ff-e168-45c7-b6de-f1382623af14", "metadata": {}, "source": [ "The **Job details** lets you configure the AWS Glue job.\n", "\n", "* Select *IAM role* you created earlier on.\n", "* Select *Python 3.9* or the latest available Python version.\n", "* In the same tab, you can also define a number of other configuration options such as **Number of retries** or **Job timeout**. Leave the value as default.\n", "\n", "![Offline File Compaction](../images/smfs_glue_job_details.png \"Offline File Compaction\")\n" ] }, { "cell_type": "markdown", "id": "0708262b-75d7-452b-ae9b-d52adb998a6a", "metadata": {}, "source": [ "* In the **Advanced properties** section, you can add **Job parameters** to execute the script, as per the example screenshot below.\n", "* Click \"Save\".\n", "\n", "![Offline File Compaction](../images/smfs_glue_job_parameters.png \"Offline File Compaction\")" ] }, { "cell_type": "markdown", "id": "aefec6f1-983d-4b93-a4a9-c550e1f540a5", "metadata": {}, "source": [ "* In the **Schedules** tab, you can define the schedule to run the feature store maintenance procedures. For example, the screenshot below shows you how to run the job on a schedule of every 6 hours. For more information, check the [documentation](https://docs.aws.amazon.com/glue/latest/dg/monitor-data-warehouse-schedule.html).\n", "\n", "![Offline File Compaction](../images/smfs_glue_schedule.png \"Offline File Compaction\")" ] }, { "cell_type": "markdown", "id": "57c9cc1f-13c5-4963-b113-63e743665d46", "metadata": {}, "source": [ "* You can click \"Save\" and \"Run\" to start a compaction job.\n", "\n", "![Offline File Compaction](../images/smfs_glue_run.png \"Offline File Compaction\")" ] }, { "cell_type": "markdown", "id": "adab950b-3eda-47e5-8e52-7ce4f7a43eef", "metadata": {}, "source": [ "* You can also monitor job runs to understand runtime metrics such as completion status, duration, and start time. You can check the CloudWatch logs for the AWS Glue job to check that the procedures run successfully.\n", "\n", "![Offline File Compaction](../images/smfs_glue_logs.png \"Offline File Compaction\")" ] }, { "cell_type": "code", "execution_count": null, "id": "bf67e1f2-d6ba-4ab6-b232-3e1c1c1cd004", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "id": "ed728b08-07dd-48ff-b479-2b1fb80eca01", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "availableInstances": [ { "_defaultOrder": 0, "_isFastLaunch": true, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 4, "name": "ml.t3.medium", "vcpuNum": 2 }, { "_defaultOrder": 1, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.t3.large", "vcpuNum": 2 }, { "_defaultOrder": 2, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.t3.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 3, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.t3.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 4, "_isFastLaunch": true, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.m5.large", "vcpuNum": 2 }, { "_defaultOrder": 5, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.m5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 6, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.m5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 7, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.m5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 8, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.m5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 9, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.m5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 10, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.m5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 11, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.m5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 12, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.m5d.large", "vcpuNum": 2 }, { "_defaultOrder": 13, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.m5d.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 14, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.m5d.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 15, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.m5d.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 16, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.m5d.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 17, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.m5d.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 18, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.m5d.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 19, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.m5d.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 20, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": true, "memoryGiB": 0, "name": "ml.geospatial.interactive", "supportedImageNames": [ "sagemaker-geospatial-v1-0" ], "vcpuNum": 0 }, { "_defaultOrder": 21, "_isFastLaunch": true, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 4, "name": "ml.c5.large", "vcpuNum": 2 }, { "_defaultOrder": 22, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.c5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 23, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.c5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 24, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.c5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 25, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 72, "name": "ml.c5.9xlarge", "vcpuNum": 36 }, { "_defaultOrder": 26, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 96, "name": "ml.c5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 27, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 144, "name": "ml.c5.18xlarge", "vcpuNum": 72 }, { "_defaultOrder": 28, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.c5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 29, "_isFastLaunch": true, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.g4dn.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 30, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.g4dn.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 31, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.g4dn.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 32, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.g4dn.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 33, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.g4dn.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 34, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.g4dn.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 35, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 61, "name": "ml.p3.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 36, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 244, "name": "ml.p3.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 37, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 488, "name": "ml.p3.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 38, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 768, "name": "ml.p3dn.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 39, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.r5.large", "vcpuNum": 2 }, { "_defaultOrder": 40, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.r5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 41, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.r5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 42, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.r5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 43, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.r5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 44, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.r5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 45, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 512, "name": "ml.r5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 46, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 768, "name": "ml.r5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 47, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.g5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 48, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.g5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 49, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.g5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 50, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.g5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 51, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.g5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 52, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.g5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 53, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.g5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 54, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 768, "name": "ml.g5.48xlarge", "vcpuNum": 192 } ], "instance_type": "ml.t3.medium", "kernelspec": { "display_name": "Python 3 (Data Science)", "language": "python", "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/datascience-1.0" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.10" } }, "nbformat": 4, "nbformat_minor": 5 }