{ "cells": [ { "cell_type": "markdown", "id": "fcec45af-2406-4a3c-a212-3f6288c05924", "metadata": {}, "source": [ "# Model Training with MLlib and Hosting on SageMaker\n", "\n", "This notebook is tested using `Studio SparkMagic - PySpark Kernel` running on a `ml.t3.medium` instance and connected to an EMR clsuter with an `m5.xlarge` Master node and 2 `m5.xlarge` Core nodes. Please ensure that you see `PySpark (SparkMagic)` in the top right on your notebook.\n" ] }, { "cell_type": "code", "execution_count": null, "outputs": [], "source": [ "%%local\n", "!pip install -U -q sagemaker" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "id": "f8f4aad7-bdb2-42f9-b296-66ba54560c1c", "metadata": {}, "source": [ "### Connection to EMR Cluster\n", "\n", "In the cell below, the code block is autogenerated. You can generate this code by clicking on the \"Cluster\" link on the top of the notebook and select the EMR cluster. \n", "\n", "For our workshop we be passing our SageMaker execution role to the cluster, but this works equally well for Kerberos, LDAP and HTTP auth mechanisms\n", "![img](https://user-images.githubusercontent.com/18154355/216500654-a18ac11a-c405-4704-b9f6-c6cd4f4fb324.png)" ] }, { "cell_type": "code", "execution_count": null, "id": "d742680d-e7de-4fd6-877e-53770e2ae405", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "6ad483fb-270b-4656-b802-77875973ae15", "metadata": {}, "source": [ "## MLeap Dependency" ] }, { "cell_type": "code", "execution_count": null, "id": "9d0d695d-0acb-4baa-9d69-aa1eaa63de66", "metadata": { "tags": [] }, "outputs": [], "source": [ "%%configure -f\n", "{ \"conf\":{\n", " \"spark.jars.packages\": \"ml.combust.mleap:mleap-spark_2.12:0.20.0,ml.combust.mleap:mleap-spark-base_2.12:0.20.0\",\n", " \"spark.pyspark.python\": \"python3\",\n", " \"spark.pyspark.virtualenv.enabled\": \"true\",\n", " \"spark.pyspark.virtualenv.type\":\"native\",\n", " \"spark.pyspark.virtualenv.bin.path\":\"/usr/bin/virtualenv\"\n", " }\n", "}" ] }, { "cell_type": "markdown", "id": "fabc330f-a36c-4d8e-8c66-1c6ce88943eb", "metadata": {}, "source": [ "## Overview\n", "\n", "In this notebook, we'll use a remote EMR cluster to build a ML model using Spark MLLib and then we'll host that model on SageMaker\n" ] }, { "cell_type": "code", "execution_count": null, "id": "ecaac1c0-2813-4d07-bdb4-9c763709755b", "metadata": { "tags": [] }, "outputs": [], "source": [ "%%local\n", "import sagemaker\n", "\n", "sess = sagemaker.Session()\n", "bucket = sess.default_bucket()\n", "region = sess.boto_region_name" ] }, { "cell_type": "code", "execution_count": null, "id": "e2d642dd-dfdd-46f2-b56e-1b4662052071", "metadata": { "tags": [] }, "outputs": [], "source": [ "%%send_to_spark -i bucket -t str -n bucket\n" ] }, { "cell_type": "code", "execution_count": null, "id": "58ba2b2d-a05f-493f-a8d6-0202af507591", "metadata": { "tags": [] }, "outputs": [], "source": [ "%%send_to_spark -i region -t str -n region" ] }, { "cell_type": "code", "execution_count": null, "id": "16d208da-cc06-4dfe-ba2b-aa27afe0978d", "metadata": { "tags": [] }, "outputs": [], "source": [ "sc.install_pypi_package(\"sagemaker\")" ] }, { "cell_type": "code", "execution_count": null, "id": "878370fc-dd13-4277-b951-50f80fa531b9", "metadata": { "tags": [] }, "outputs": [], "source": [ "sc.install_pypi_package(\"mleap==0.20.0\")" ] }, { "cell_type": "code", "execution_count": null, "id": "8e150a31-7010-4f21-ac18-a4c44616abba", "metadata": { "tags": [] }, "outputs": [], "source": [ "from mleap.pyspark.spark_support import SimpleSparkSerializer\n", "from pyspark.ml.regression import RandomForestRegressor\n", "from pyspark.sql.types import StructField, StructType, StringType, DoubleType\n", "\n", "schema = StructType(\n", " [\n", " StructField(\"sex\", StringType(), True),\n", " StructField(\"length\", DoubleType(), True),\n", " StructField(\"diameter\", DoubleType(), True),\n", " StructField(\"height\", DoubleType(), True),\n", " StructField(\"whole_weight\", DoubleType(), True),\n", " StructField(\"shucked_weight\", DoubleType(), True),\n", " StructField(\"viscera_weight\", DoubleType(), True),\n", " StructField(\"shell_weight\", DoubleType(), True),\n", " StructField(\"rings\", DoubleType(), True),\n", " ]\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "8cd6baa1-86a1-480f-851b-41715d4da850", "metadata": { "tags": [] }, "outputs": [], "source": [ "total_df = spark.read.csv(\n", " \"s3://sagemaker-sample-files/datasets/tabular/uci_abalone/abalone.csv\", header=False, schema=schema\n", ")\n", "total_df.show(5)\n", "(train_df, validation_df) = total_df.randomSplit([0.8, 0.2])" ] }, { "cell_type": "code", "execution_count": null, "id": "6b42b116-1e09-4938-95a2-6d931f6a5cc4", "metadata": { "tags": [] }, "outputs": [], "source": [ "from pyspark.ml.feature import (\n", " StringIndexer,\n", " VectorIndexer,\n", " OneHotEncoder,\n", " VectorAssembler,\n", " IndexToString,\n", ")\n", "\n", "\n", "sex_indexer = StringIndexer(inputCol=\"sex\", outputCol=\"indexed_sex\")\n", "\n", "sex_encoder = OneHotEncoder(inputCols=[\"indexed_sex\"], outputCols=[\"sex_vec\"])\n", "\n", "assembler = VectorAssembler(\n", " inputCols=[\n", " \"sex_vec\",\n", " \"length\",\n", " \"diameter\",\n", " \"height\",\n", " \"whole_weight\",\n", " \"shucked_weight\",\n", " \"viscera_weight\",\n", " \"shell_weight\",\n", " ],\n", " outputCol=\"features\",\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "0e030f68-369d-4093-a3cc-b630b429a545", "metadata": { "tags": [] }, "outputs": [], "source": [ "from pyspark.ml import Pipeline\n", "from pyspark.ml.regression import RandomForestRegressor\n", "\n", "rf = RandomForestRegressor(labelCol=\"rings\", featuresCol=\"features\", maxDepth=6, numTrees=18)\n", "pipeline = Pipeline(stages=[sex_indexer, sex_encoder, assembler, rf])\n", "model = pipeline.fit(train_df)" ] }, { "cell_type": "code", "execution_count": null, "id": "a4a26a13-b69b-4e5f-b4f3-c0151a0fd17a", "metadata": { "tags": [] }, "outputs": [], "source": [ "transformed_train_df = model.transform(train_df)\n", "transformed_validation_df = model.transform(validation_df)\n", "transformed_validation_df.select(\"prediction\").show(5)" ] }, { "cell_type": "markdown", "id": "f1eb1d26-db67-4705-9702-886b2de8cced", "metadata": {}, "source": [ "------\n", "\n", "## Optionally record your experiment using SageMaker Experiments. \n", "\n", "Note: This requires your network to be configured correctly. If you're running this as an AWS provided workshop then you're all set" ] }, { "cell_type": "code", "execution_count": null, "id": "0207a8d7-f326-4da7-b80a-24538e33bd44", "metadata": { "tags": [] }, "outputs": [], "source": [ "from pyspark.ml.evaluation import RegressionEvaluator\n", "\n", "evaluator = RegressionEvaluator(labelCol=\"rings\", predictionCol=\"prediction\", metricName=\"rmse\")\n", "train_rmse = evaluator.evaluate(transformed_train_df)\n", "validation_rmse = evaluator.evaluate(transformed_validation_df)\n", "print(\"Train RMSE = %g\" % train_rmse)\n", "print(\"Validation RMSE = %g\" % validation_rmse)" ] }, { "cell_type": "code", "execution_count": null, "id": "a9b04cad-c1a0-43ab-9f4a-7df3d0dc697d", "metadata": { "tags": [] }, "outputs": [], "source": [ "import boto3\n", "from sagemaker.session import Session\n", "from sagemaker.experiments import Run\n", "\n", "boto_session = boto3.session.Session(region_name=region)\n", "sagemaker_session = Session(boto_session=boto_session)\n", "\n", "# The run name is an optional argument to `run.init()`\n", "with Run(experiment_name = 'mllib-experiment', sagemaker_session=sagemaker_session) as run:\n", " run.log_metric(name = \"Train RMSE\", value = train_rmse)\n", " run.log_metric(name = \"Val RMSE\", value = validation_rmse)\n", "\n", "print(\"Done!\")" ] }, { "cell_type": "markdown", "id": "4f4e5eec-63e8-4ee1-a0d4-ce1f0571d87a", "metadata": {}, "source": [ "------\n", "## Save Model to S3 then host on SageMaker" ] }, { "cell_type": "code", "execution_count": null, "id": "91ce77f3-3623-4fca-8488-deb77feabe9e", "metadata": { "tags": [] }, "outputs": [], "source": [ "model.serializeToBundle(\"jar:file:/tmp/model.zip\", transformed_validation_df)" ] }, { "cell_type": "code", "execution_count": null, "id": "60a013a3-dd74-4694-bec4-a02e0e081a72", "metadata": {}, "outputs": [], "source": [ "import zipfile\n", "\n", "with zipfile.ZipFile(\"/tmp/model.zip\") as zf:\n", " zf.extractall(\"/tmp/model\")\n", "\n", "import tarfile\n", "\n", "with tarfile.open(\"/tmp/model.tar.gz\", \"w:gz\") as tar:\n", " tar.add(\"/tmp/model/bundle.json\", arcname=\"bundle.json\")\n", " tar.add(\"/tmp/model/root\", arcname=\"root\")" ] }, { "cell_type": "code", "execution_count": null, "id": "53ede086-dd6e-481d-ade9-6b06a0cbf9b6", "metadata": { "tags": [] }, "outputs": [], "source": [ "# Please replace the bucket name with your bucket name where you want to upload the model\n", "import os\n", "s3 = boto3.resource(\"s3\")\n", "file_name = os.path.join(\"emr/abalone/mleap\", \"model.tar.gz\")\n", "s3.Bucket(bucket).upload_file(\"/tmp/model.tar.gz\", file_name)" ] }, { "cell_type": "code", "execution_count": null, "id": "b2419d70-fa0d-4c28-8891-2568b0c83047", "metadata": {}, "outputs": [], "source": [ "%%local\n", "import json\n", "\n", "schema = {\n", " \"input\": [\n", " {\"name\": \"sex\", \"type\": \"string\"},\n", " {\"name\": \"length\", \"type\": \"double\"},\n", " {\"name\": \"diameter\", \"type\": \"double\"},\n", " {\"name\": \"height\", \"type\": \"double\"},\n", " {\"name\": \"whole_weight\", \"type\": \"double\"},\n", " {\"name\": \"shucked_weight\", \"type\": \"double\"},\n", " {\"name\": \"viscera_weight\", \"type\": \"double\"},\n", " {\"name\": \"shell_weight\", \"type\": \"double\"},\n", " ],\n", " \"output\": {\"name\": \"prediction\", \"type\": \"double\"},\n", "}\n", "schema_json = json.dumps(schema, indent=2)\n" ] }, { "cell_type": "code", "execution_count": null, "id": "798ffb0f-1d4b-4c0d-8624-8bca3f028b90", "metadata": { "tags": [] }, "outputs": [], "source": [ "%%local\n", "from time import gmtime, strftime\n", "import time\n", "\n", "timestamp_prefix = strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n", "\n", "import boto3\n", "import sagemaker\n", "from sagemaker import get_execution_role\n", "from sagemaker.sparkml.model import SparkMLModel\n", "\n", "boto3_session = boto3.session.Session()\n", "sagemaker_client = boto3.client(\"sagemaker\")\n", "sagemaker_runtime_client = boto3.client(\"sagemaker-runtime\")\n", "\n", "# Initialize sagemaker session\n", "session = sagemaker.Session(\n", " boto_session=boto3_session,\n", " sagemaker_client=sagemaker_client,\n", " sagemaker_runtime_client=sagemaker_runtime_client,\n", ")\n", "\n", "role = get_execution_role()" ] }, { "cell_type": "code", "execution_count": null, "id": "47f614b9-e3e3-49df-8cf5-cb06181ebd19", "metadata": { "tags": [] }, "outputs": [], "source": [ "%%local\n", "# S3 location of where you uploaded your trained and serialized SparkML model\n", "sparkml_data = \"s3://{}/{}/{}\".format(\n", " bucket, \"emr/abalone/mleap\", \"model.tar.gz\"\n", ")\n", "model_name = \"sparkml-abalone-\" + timestamp_prefix\n", "sparkml_model = SparkMLModel(\n", " model_data=sparkml_data,\n", " role=role,\n", " spark_version='3.3',\n", " sagemaker_session=session,\n", " name=model_name,\n", " # passing the schema defined above by using an environment\n", " # variable that sagemaker-sparkml-serving understands\n", " env={\"SAGEMAKER_SPARKML_SCHEMA\": schema_json},\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "694b34ed-789c-4b2b-a34c-46d3c7de4f3c", "metadata": { "tags": [] }, "outputs": [], "source": [ "%%local\n", "endpoint_name = \"sparkml-abalone-ep-\" + timestamp_prefix\n", "sparkml_model.deploy(\n", " initial_instance_count=1, instance_type=\"ml.c4.xlarge\", endpoint_name=endpoint_name\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "e95f97d5-ea07-4869-95fb-743db59d9c7b", "metadata": { "tags": [] }, "outputs": [], "source": [ "%%local\n", "from sagemaker.predictor import Predictor\n", "from sagemaker.serializers import CSVSerializer, JSONSerializer\n", "from sagemaker.deserializers import JSONDeserializer\n", "\n", "\n", "payload = \"F,0.515,0.425,0.14,0.766,0.304,0.1725,0.255\"\n", "\n", "predictor = Predictor(\n", " endpoint_name=endpoint_name, sagemaker_session=session, serializer=CSVSerializer()\n", ")\n", "print(predictor.predict(payload))" ] }, { "cell_type": "code", "execution_count": null, "id": "5b850183-57e2-436a-9da1-cbe85a8b1b9e", "metadata": { "tags": [] }, "outputs": [], "source": [ "%%local\n", "payload = {\"data\": [\"F\", 0.515, 0.425, 0.14, 0.766, 0.304, 0.1725, 0.255]}\n", "\n", "predictor = Predictor(\n", " endpoint_name=endpoint_name, sagemaker_session=session, serializer=JSONSerializer()\n", ")\n", "print(predictor.predict(payload))" ] }, { "cell_type": "code", "execution_count": null, "id": "40c5483b-d9a8-4974-823b-2147c6874fb1", "metadata": { "tags": [] }, "outputs": [], "source": [ "%%local\n", "payload = {\n", " \"schema\": {\n", " \"input\": [\n", " {\"name\": \"length\", \"type\": \"double\"},\n", " {\"name\": \"sex\", \"type\": \"string\"},\n", " {\"name\": \"diameter\", \"type\": \"double\"},\n", " {\"name\": \"height\", \"type\": \"double\"},\n", " {\"name\": \"whole_weight\", \"type\": \"double\"},\n", " {\"name\": \"shucked_weight\", \"type\": \"double\"},\n", " {\"name\": \"viscera_weight\", \"type\": \"double\"},\n", " {\"name\": \"shell_weight\", \"type\": \"double\"},\n", " ],\n", " \"output\": {\"name\": \"prediction\", \"type\": \"double\"},\n", " },\n", " \"data\": [0.515, \"F\", 0.425, 0.14, 0.766, 0.304, 0.1725, 0.255],\n", "}\n", "\n", "predictor = Predictor(\n", " endpoint_name=endpoint_name, sagemaker_session=session, serializer=JSONSerializer()\n", ")\n", "print(predictor.predict(payload))" ] }, { "cell_type": "markdown", "id": "d3c23286-5a0a-4c7c-9923-0fadf36d793a", "metadata": {}, "source": [ "## Clean Up" ] }, { "cell_type": "code", "execution_count": null, "id": "e36c2070-6395-4620-9a45-54b131df7d8f", "metadata": { "tags": [] }, "outputs": [], "source": [ "%%cleanup -f" ] }, { "cell_type": "code", "execution_count": null, "id": "1b329764-7da2-4a45-825b-80f06bedabbb", "metadata": { "tags": [] }, "outputs": [], "source": [ "%%local\n", "predictor.delete_endpoint()" ] }, { "cell_type": "code", "execution_count": null, "id": "c1b03084-8301-46c6-95db-3b5442faa4a1", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "availableInstances": [ { "_defaultOrder": 0, "_isFastLaunch": true, "category": "General purpose", "gpuNum": 0, "memoryGiB": 4, "name": "ml.t3.medium", "vcpuNum": 2 }, { "_defaultOrder": 1, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "memoryGiB": 8, "name": "ml.t3.large", "vcpuNum": 2 }, { "_defaultOrder": 2, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "memoryGiB": 16, "name": "ml.t3.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 3, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "memoryGiB": 32, "name": "ml.t3.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 4, "_isFastLaunch": true, "category": "General purpose", "gpuNum": 0, "memoryGiB": 8, "name": "ml.m5.large", "vcpuNum": 2 }, { "_defaultOrder": 5, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "memoryGiB": 16, "name": "ml.m5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 6, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "memoryGiB": 32, "name": "ml.m5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 7, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "memoryGiB": 64, "name": "ml.m5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 8, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "memoryGiB": 128, "name": "ml.m5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 9, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "memoryGiB": 192, "name": "ml.m5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 10, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "memoryGiB": 256, "name": "ml.m5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 11, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "memoryGiB": 384, "name": "ml.m5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 12, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "memoryGiB": 8, "name": "ml.m5d.large", "vcpuNum": 2 }, { "_defaultOrder": 13, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "memoryGiB": 16, "name": "ml.m5d.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 14, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "memoryGiB": 32, "name": "ml.m5d.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 15, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "memoryGiB": 64, "name": "ml.m5d.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 16, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "memoryGiB": 128, "name": "ml.m5d.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 17, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "memoryGiB": 192, "name": "ml.m5d.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 18, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "memoryGiB": 256, "name": "ml.m5d.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 19, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "memoryGiB": 384, "name": "ml.m5d.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 20, "_isFastLaunch": true, "category": "Compute optimized", "gpuNum": 0, "memoryGiB": 4, "name": "ml.c5.large", "vcpuNum": 2 }, { "_defaultOrder": 21, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "memoryGiB": 8, "name": "ml.c5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 22, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "memoryGiB": 16, "name": "ml.c5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 23, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "memoryGiB": 32, "name": "ml.c5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 24, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "memoryGiB": 72, "name": "ml.c5.9xlarge", "vcpuNum": 36 }, { "_defaultOrder": 25, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "memoryGiB": 96, "name": "ml.c5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 26, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "memoryGiB": 144, "name": "ml.c5.18xlarge", "vcpuNum": 72 }, { "_defaultOrder": 27, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "memoryGiB": 192, "name": "ml.c5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 28, "_isFastLaunch": true, "category": "Accelerated computing", "gpuNum": 1, "memoryGiB": 16, "name": "ml.g4dn.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 29, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "memoryGiB": 32, "name": "ml.g4dn.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 30, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "memoryGiB": 64, "name": "ml.g4dn.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 31, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "memoryGiB": 128, "name": "ml.g4dn.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 32, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "memoryGiB": 192, "name": "ml.g4dn.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 33, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "memoryGiB": 256, "name": "ml.g4dn.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 34, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "memoryGiB": 61, "name": "ml.p3.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 35, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "memoryGiB": 244, "name": "ml.p3.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 36, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "memoryGiB": 488, "name": "ml.p3.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 37, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "memoryGiB": 768, "name": "ml.p3dn.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 38, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "memoryGiB": 16, "name": "ml.r5.large", "vcpuNum": 2 }, { "_defaultOrder": 39, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "memoryGiB": 32, "name": "ml.r5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 40, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "memoryGiB": 64, "name": "ml.r5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 41, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "memoryGiB": 128, "name": "ml.r5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 42, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "memoryGiB": 256, "name": "ml.r5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 43, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "memoryGiB": 384, "name": "ml.r5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 44, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "memoryGiB": 512, "name": "ml.r5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 45, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "memoryGiB": 768, "name": "ml.r5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 46, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "memoryGiB": 16, "name": "ml.g5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 47, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "memoryGiB": 32, "name": "ml.g5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 48, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "memoryGiB": 64, "name": "ml.g5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 49, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "memoryGiB": 128, "name": "ml.g5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 50, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "memoryGiB": 256, "name": "ml.g5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 51, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "memoryGiB": 192, "name": "ml.g5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 52, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "memoryGiB": 384, "name": "ml.g5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 53, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "memoryGiB": 768, "name": "ml.g5.48xlarge", "vcpuNum": 192 } ], "instance_type": "ml.t3.medium", "kernelspec": { "display_name": "PySpark (SparkMagic)", "language": "python", "name": "pysparkkernel__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-sparkmagic" }, "language_info": { "codemirror_mode": { "name": "python", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "pyspark", "pygments_lexer": "python3" } }, "nbformat": 4, "nbformat_minor": 5 }