{ "cells": [ { "cell_type": "markdown", "id": "e1c348b8", "metadata": {}, "source": [ "# mpt-7b-Instruct を SageMaker で Hosting\n", "## このノートブックについて\n", "このノートブックは、mpt-7b を Instruction tuning した mpt-7b-Instruct モデルを、SageMaker でリアルタイム推論エンドポイントを Hosting するノートブックです。 \n", "以下の環境で動作確認を行ってます。\n", "* SageMaker Studio Notebooks\n", " * `ml.g5.2xlarge(NVIDIA A10G Tensor Core GPU 搭載 VRAM 24GB, RAM 32GB, vCPU 8)` : `PyTorch 1.13 Python 3.9 GPU Optimized`\n", " * `ml.m5.2xlarge(RAM 32GB, vCPU 8) ` : `PyTorch 1.13 Python 3.9 CPU Optimized`\n", "* SageMaker Notebooks\n", " * `ml.g5.2xlarge(NVIDIA A10G Tensor Core GPU 搭載 VRAM 24GB, RAM 32GB, vCPU 8)` : `conda_pytorch_p39`\n", " * `ml.m5.2xlarge(RAM 32GB, vCPU 8) ` : `conda_pytorch_p39` \n", "[各インスタンスの料金についてはこちら](https://aws.amazon.com/jp/sagemaker/pricing/)をご確認ください。 \n", "\n", "## 使用するモデルについて\n", "モデルの詳細については [Hugging Face mpt-7b-instruct](https://huggingface.co/mosaicml/mpt-7b-instruct) や [MosaicML の blog](https://www.mosaicml.com/blog/mpt-7b) を参照してください。 \n", "モデルのライセンスは上記リンクにあるとおり `CC-By-SA-3.0` です。\n", "\n", "ノートブックは外部ファイルを参照していないので、どのディレクトリに配置してあっても動作します。 \n", "\n", "また、ノートブックを動かすにあたって、各セルを上から順番に実行すれば動きますが、SageMaker 上での推論の仕組みについては、[AI/ML DarkPark](https://www.youtube.com/playlist?list=PLAOq15s3RbuL32mYUphPDoeWKUiEUhcug) の特に [Amazon SageMaker 推論 Part2すぐにプロダクション利用できる！モデルをデプロイして推論する方法 【ML-Dark-04】【AWS Black Belt】](https://youtu.be/sngNd79GpmE) をご参照ください。" ] }, { "cell_type": "markdown", "id": "46a2fddb-de17-4ce9-b622-227d06290e17", "metadata": {}, "source": [ "## 準備\n", "### ノートブックを動かすに当たって必要なモジュールのインストール" ] }, { "cell_type": "code", "execution_count": null, "id": "f0436c68", "metadata": { "scrolled": true, "tags": [] }, "outputs": [], "source": [ "pip install transformers==4.26 einops sagemaker flash_attn -U" ] }, { "cell_type": "markdown", "id": "bdcddf57-19a5-455e-95e7-25703783a2a0", "metadata": {}, "source": [ "### 今回扱うモデルの動かし方について\n", "[How to use](https://huggingface.co/mosaicml/mpt-7b-instruct#how-to-use) に沿って実行すると動かせます。 \n", "例えば、以下のコードをこのノートブックで実行するとテキストを生成できます。 \n", "実行したい場合は別途セルを用意して実行してみてください。g5.2xlarge インスタンスで実行に 10 分程度かかります。(ほとんどはモデルのロード時間です) \n", "このノートブックでは以下のコードをベースに SageMaker で Hosting できるようにします。 \n", "```python\n", "import torch\n", "from transformers import AutoTokenizer, AutoModelForCausalLM\n", "\n", "# tokenizer を Hugging Face からロード\n", "tokenizer = AutoTokenizer.from_pretrained(\n", " \"mosaicml/mpt-7b-Instruct\",\n", " trust_remote_code=True\n", ")\n", "# model を Hugging Face からロード\n", "model = AutoModelForCausalLM.from_pretrained(\n", " \"mosaicml/mpt-7b-Instruct\", \n", " torch_dtype=torch.float16,\n", " trust_remote_code=True\n", ").to(\"cuda:0\")\n", "\n", "\n", "# prompt の書き方は後述\n", "prompt = '''Python で フィボナッチ数列の 10 番目を知りたいです。\n", "### Instruction:\n", "フィボナッチ数列を求める関数とその関数の実行コードを記載してください。\n", "### Response:'''\n", "\n", "# prompt のトークン化\n", "inputs = tokenizer(prompt, return_tensors='pt').to(\"cuda:0\")\n", "input_length = inputs.input_ids.shape[1]\n", "\n", "# テキストを生成\n", "with torch.no_grad():\n", " outputs = model.generate(\n", " **inputs, \n", " max_new_tokens=128, \n", " do_sample=True, \n", " temperature=0.01, \n", " top_p=0.7, \n", " top_k=50, \n", " return_dict_in_generate=True\n", " )\n", "\n", "# 生成結果を出力\n", "token = outputs.sequences[0, input_length:]\n", "output_str = tokenizer.decode(token)\n", "output_str = output_str[:output_str.find('<|endoftext|>')]\n", "\n", "print(output_str)\n", "```" ] }, { "cell_type": "markdown", "id": "7a31ac1d-e300-44a3-8d38-f709e7370ad8", "metadata": {}, "source": [ "### モジュール読み込み" ] }, { "cell_type": "code", "execution_count": null, "id": "636d9a5e", "metadata": { "tags": [] }, "outputs": [], "source": [ "import torch\n", "from transformers import AutoTokenizer, AutoModelForCausalLM\n", "import gc\n", "import sagemaker\n", "import boto3\n", "from sagemaker.huggingface import HuggingFaceModel\n", "from sagemaker.serializers import JSONSerializer\n", "from sagemaker.deserializers import JSONDeserializer\n", "import json\n", "region = boto3.session.Session().region_name\n", "role = sagemaker.get_execution_role()\n", "sm = boto3.client('sagemaker')\n", "smr = boto3.client('sagemaker-runtime')\n", "endpoint_inservice_waiter = sm.get_waiter('endpoint_in_service')" ] }, { "cell_type": "markdown", "id": "e45fe9a8", "metadata": {}, "source": [ "### モデルのダウンロード\n", "SageMaker で機械学習モデルをホスティングする際は、一般的にはモデルや推論コードなどを tar.gz の形に固めます。 \n", "tokenizer と model を `from_pretrained` メソッドを利用してモデルをインターネットからロードして、そのままファイルをディレクトリに出力します。 " ] }, { "cell_type": "code", "execution_count": null, "id": "05c6a8fb-28b7-414d-a00f-e70256021df5", "metadata": { "tags": [] }, "outputs": [], "source": [ "# 既存のディレクトリがある場合のときのため削除\n", "model_dir = './inference'\n", "!rm -rf {model_dir}\n", "!mkdir -p {model_dir}'/code'" ] }, { "cell_type": "markdown", "id": "09c0b8e0-33e5-4482-90bc-2a47eb4290ae", "metadata": {}, "source": [ "#### tokenizer の取得と保存" ] }, { "cell_type": "code", "execution_count": null, "id": "ab583c32", "metadata": { "tags": [] }, "outputs": [], "source": [ "%%time\n", "\n", "tokenizer = AutoTokenizer.from_pretrained(\n", " \"mosaicml/mpt-7b-Instruct\",\n", " trust_remote_code=True\n", ")\n", "tokenizer.save_pretrained(model_dir)" ] }, { "cell_type": "markdown", "id": "22afb4a9-2812-4c3a-969d-fe8550b6812e", "metadata": {}, "source": [ "#### モデルの取得と保存\n", "以下のセルは 10GB 以上のモデルを DL して保存するため 9 分ほど時間がかかります。" ] }, { "cell_type": "code", "execution_count": null, "id": "d9e4a1bf", "metadata": { "scrolled": true, "tags": [] }, "outputs": [], "source": [ "%%time\n", "model = AutoModelForCausalLM.from_pretrained(\n", " \"mosaicml/mpt-7b-Instruct\", \n", " torch_dtype=torch.float16,\n", " trust_remote_code=True\n", ")\n", "model.save_pretrained(model_dir)" ] }, { "cell_type": "markdown", "id": "488f84c0-2f23-4350-ab31-8844ebca0add", "metadata": {}, "source": [ "モデルは SageMaker で動かすのでメモリから開放します" ] }, { "cell_type": "code", "execution_count": null, "id": "ce465780", "metadata": { "tags": [] }, "outputs": [], "source": [ "del model\n", "del tokenizer\n", "gc.collect()" ] }, { "cell_type": "markdown", "id": "67ed6305", "metadata": {}, "source": [ "### 推論コードの作成\n", "先程実行したコードをもとに記述していきます。 \n", "まずは必要なモジュールを記述した requirements.txt を用意します。 \n", "今回は [deep-learning-containers](https://github.com/aws/deep-learning-containers)の HuggingFace のコンテナを使います。 \n", "einops だけ不足しているので requirements.txt に記載します。" ] }, { "cell_type": "code", "execution_count": null, "id": "26fdc6ac", "metadata": { "tags": [] }, "outputs": [], "source": [ "%%writefile inference/code/requirements.txt\n", "einops" ] }, { "cell_type": "markdown", "id": "1aae3d83-b4c9-4b2b-a1d5-9fba9a35c93b", "metadata": {}, "source": [ "先述のコードを SageMaker Inference 向けに改変します。\n", "1. `model_fn` でモデルを読み込みます。先程は huggingface のモデルを直接ロードしましたが、`model_dir` に展開されたモデルを読み込みます。\n", "2. `input_fn` で前処理を行います。\n", " * json 形式のみを受け付け他の形式は弾くようにします。\n", " * json 文字列を dict 形式に変換して返します。\n", "3. `predict_fn` で推論します。\n", " 1. リクエストされたテキストを token 化します。\n", " 2. パラメータを展開します。\n", " 3. 推論（生成）します。\n", " 4. 生成結果をテキストにして返します。\n", "4. `output_fn` で結果を json 形式にして返します。" ] }, { "cell_type": "code", "execution_count": null, "id": "1fbc4cf4", "metadata": { "tags": [] }, "outputs": [], "source": [ "%%writefile inference/code/inference.py\n", "import torch\n", "from transformers import AutoTokenizer, AutoModelForCausalLM\n", "import json\n", "\n", "DEVICE = 'cuda:0'\n", "\n", "def model_fn(model_dir):\n", " tokenizer = AutoTokenizer.from_pretrained(\n", " model_dir,\n", " trust_remote_code=True\n", " )\n", " model = AutoModelForCausalLM.from_pretrained(\n", " model_dir,\n", " trust_remote_code=True,\n", " torch_dtype=torch.float16,\n", " ).to(DEVICE)\n", " return {'tokenizer':tokenizer,'model':model}\n", "\n", "def input_fn(data, content_type):\n", " if content_type == 'application/json':\n", " data = json.loads(data)\n", " else:\n", " raise TypeError('content_type is only allowed application/json')\n", " return data\n", "\n", "def predict_fn(data, model):\n", " inputs = model['tokenizer'](data['prompt'], return_tensors='pt').to(DEVICE)\n", " input_length = inputs.input_ids.shape[1]\n", " max_new_tokens = data['max_new_tokens']\n", " do_sample = data['do_sample']\n", " temperature = data['temperature']\n", " top_p = data['top_p']\n", " top_k = data['top_k']\n", " return_dict_in_generate = data['return_dict_in_generate']\n", " with torch.no_grad():\n", " outputs = model['model'].generate(\n", " **inputs, \n", " max_new_tokens=max_new_tokens, \n", " do_sample=do_sample, \n", " temperature=temperature, \n", " top_p=top_p, \n", " top_k=top_k, \n", " return_dict_in_generate=return_dict_in_generate\n", " )\n", " \n", " token = outputs.sequences[0, input_length:]\n", " output_str = model['tokenizer'].decode(token)\n", " \n", " return output_str\n", "\n", "def output_fn(data, accept_type):\n", " if accept_type == 'application/json':\n", " data = json.dumps({'result' : data})\n", " else:\n", " raise TypeError('content_type is only allowed application/json')\n", " return data" ] }, { "cell_type": "markdown", "id": "54a1a2d0", "metadata": {}, "source": [ "### モデルアーティファクトの作成と S3 アップロード\n", "アーティファクト(推論コード + モデル)を tar.gz に固めます。時間がかかるので `pigz` で並列処理を行います。 \n", "ml.g5.2xlarge, ml.m5.2xlarge で 3-4 分ほどかかります。" ] }, { "cell_type": "markdown", "id": "1c89a3a6-804d-4512-81f5-64a0bbfee47c", "metadata": { "tags": [] }, "source": [ "※ SageMaker Studio のカーネルには pigz が入っていないので、下記 apt のセルを実行してください。SageMaker Notebooks の場合は不要です。" ] }, { "cell_type": "code", "execution_count": null, "id": "b1200c53-ff5d-4b12-b7e5-8303a22f0680", "metadata": { "scrolled": true, "tags": [] }, "outputs": [], "source": [ "!apt update -y\n", "!apt install pigz -y" ] }, { "cell_type": "code", "execution_count": null, "id": "b003b0fd", "metadata": { "scrolled": true, "tags": [] }, "outputs": [], "source": [ "%%time\n", "\n", "!rm model.tar.gz\n", "%cd {model_dir}\n", "!tar cv ./ | pigz -p 8 > ../model.tar.gz # 8 並列でアーカイブ\n", "%cd .." ] }, { "cell_type": "markdown", "id": "6a5f9124-a8d9-4460-a7f5-c350b8d05629", "metadata": {}, "source": [ "アーティファクトを S3 にアップロードします。60 秒程度で完了します。" ] }, { "cell_type": "code", "execution_count": null, "id": "87e9a3a0", "metadata": { "scrolled": true, "tags": [] }, "outputs": [], "source": [ "%%time\n", "\n", "model_s3_uri = sagemaker.session.Session().upload_data(\n", " 'model.tar.gz',\n", " key_prefix='mpt-7b-Instruct'\n", ")\n", "print(model_s3_uri)" ] }, { "cell_type": "markdown", "id": "67269333", "metadata": {}, "source": [ "## SageMaker で Hosting する\n", "g5.2xlarge インスタンス(NVIDIA A10G Tensor Core GPU 搭載 VRAM 24GB, RAM 32GB) の場合レスポンスに 6 秒程度で済むため、リアルタイム推論エンドポイントを立てます。 \n", "(再掲)g5.2xlarge の[料金はこちら](https://aws.amazon.com/sagemaker/pricing/?nc1=h_ls)で確認してください。 \n", "\n", "リアルタイム推論エンドポイントを立てて推論するにあたって、SageMaker Python SDK を用いる場合と Boto3 を用いる場合の 2 パターンを紹介します。" ] }, { "cell_type": "markdown", "id": "489d0ab5", "metadata": {}, "source": [ "### SageMaker Python SDKを用いる場合\n", "\n", "#### Hosting\n", "使用している API の詳細は以下を確認してください。 \n", "[Amazon SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable/index.html)" ] }, { "cell_type": "markdown", "id": "1ac8e337-19bc-414e-9828-44fe06cd6854", "metadata": {}, "source": [ "##### 定数の設定" ] }, { "cell_type": "code", "execution_count": null, "id": "cdfe392f", "metadata": { "tags": [] }, "outputs": [], "source": [ "model_name = 'mpt-7b-Instruct'\n", "endpoint_config_name = model_name + 'Config'\n", "endpoint_name = model_name + 'Endpoint'\n", "instance_type = 'ml.g5.2xlarge'" ] }, { "cell_type": "markdown", "id": "5f6e9517-82f5-4c53-8015-725397aad119", "metadata": {}, "source": [ "##### 使用するコンテナイメージの URI を取得" ] }, { "cell_type": "code", "execution_count": null, "id": "440e5c49", "metadata": { "tags": [] }, "outputs": [], "source": [ "image_uri = sagemaker.image_uris.retrieve(\n", " framework='huggingface',\n", " region=region,\n", " version='4.26',\n", " image_scope='inference',\n", " base_framework_version='pytorch1.13',\n", " instance_type = instance_type\n", ")" ] }, { "cell_type": "markdown", "id": "2102d46c-e265-411c-89e0-1cf7b8071b85", "metadata": {}, "source": [ "##### モデルの定義\n", "先程 S3 にアップロードしたアーティファクトの tar.gz の URI と、コンテナイメージの URI, ロールを設定します。" ] }, { "cell_type": "code", "execution_count": null, "id": "68d03094", "metadata": { "tags": [] }, "outputs": [], "source": [ "huggingface_model = HuggingFaceModel(\n", " model_data = model_s3_uri,\n", " role = role,\n", " image_uri = image_uri\n", ")" ] }, { "cell_type": "markdown", "id": "3960d528-946b-4efa-9b46-6197c5544c8e", "metadata": {}, "source": [ "デプロイ" ] }, { "cell_type": "code", "execution_count": null, "id": "5a98cf65", "metadata": { "scrolled": true, "tags": [] }, "outputs": [], "source": [ "predictor = huggingface_model.deploy(\n", " initial_instance_count=1,\n", " instance_type=instance_type,\n", " endpoint_name=endpoint_name,\n", " serializer=JSONSerializer(),\n", " deserializer=JSONDeserializer()\n", ")" ] }, { "cell_type": "markdown", "id": "d4fccf8b", "metadata": {}, "source": [ "#### 推論" ] }, { "cell_type": "markdown", "id": "fff1f0eb-b1a9-4512-b8ad-edcb5b22f655", "metadata": {}, "source": [ "##### promptについて\n", "[mosaicml/mpt-7b-instruct の Formatting](https://huggingface.co/mosaicml/mpt-7b-instruct#formatting) にある通り、dolly-15k の format で学習しているので、prompt は dolly-15k の format である、以下の通りにすると良い結果が得られやすいです。 \n", "```text\n", "{状況や前提}\n", "### Instruction:\n", "{命令文}\n", "### Response:\n", "```" ] }, { "cell_type": "code", "execution_count": null, "id": "2e3c1a93", "metadata": { "tags": [] }, "outputs": [], "source": [ "# prompt 例\n", "prompt = '''Python で フィボナッチ数列の 10 番目を知りたいです。\n", "### Instruction:\n", "フィボナッチ数列を求める関数とその関数の実行コードを記載してください。\n", "### Response:'''" ] }, { "cell_type": "markdown", "id": "4551782e-9ec4-49e2-a398-80ffb5836f64", "metadata": {}, "source": [ "##### 推論リクエスト\n", "model_fn の実行に時間がかかってしまい、エンドポイントが IN_SERVICE になっても、初回推論はしばらく動かないことがあります。 \n", "CloudWatch Logs に以下のような表示がある場合はしばらく待てば使えるようになります。 \n", "`[WARN] pool-3-thread-1 com.amazonaws.ml.mms.metrics.MetricCollector - worker pid is not available yet.` \n", "モデルがロードされるまで 6 分程度かかるため、リトライを入れています。\n", "実際の推論時間は 6 秒程度です。" ] }, { "cell_type": "code", "execution_count": null, "id": "3e7e3b3e", "metadata": { "scrolled": true, "tags": [] }, "outputs": [], "source": [ "from time import sleep\n", "request = {\n", " 'prompt' : prompt,\n", " 'max_new_tokens' : 128,\n", " 'do_sample' : True,\n", " 'temperature' : 0.01,\n", " 'top_p' : 0.7,\n", " 'top_k' : 50,\n", " 'return_dict_in_generate' : True\n", "}\n", "\n", "for i in range(10):\n", " try:\n", " output_str = predictor.predict(request)['result']\n", " break\n", " except:\n", " sleep(60)\n", "\n", "print(output_str[:output_str.find('<|endoftext|>')])" ] }, { "cell_type": "markdown", "id": "44ec071c-4b2f-4218-8705-9ea99606f516", "metadata": {}, "source": [ "##### エンドポイントの削除" ] }, { "cell_type": "code", "execution_count": null, "id": "f85c17da", "metadata": { "tags": [] }, "outputs": [], "source": [ "predictor.delete_model()\n", "predictor.delete_endpoint()" ] }, { "cell_type": "markdown", "id": "7896098a", "metadata": {}, "source": [ "### Boto3 を用いる場合\n", "標準だと SageMaker SDK が入っていない環境からデプロイや推論する場合(例:AWS Lambda など)は、boto3 でデプロイや推論することも多いです。 \n", "以下のセルは boto3 で実行する方法を記述しています。\n", "各 API の詳細は Document を確認してください。 \n", "[SageMaker](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html) \n", "[SageMakerRuntime](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker-runtime.html) \n", "\n", "#### Hosting" ] }, { "cell_type": "markdown", "id": "bc667c06-cf76-4076-808d-5105389a7740", "metadata": {}, "source": [ "##### モデルの作成" ] }, { "cell_type": "code", "execution_count": null, "id": "6754ae19", "metadata": { "tags": [] }, "outputs": [], "source": [ "response = sm.create_model(\n", " ModelName=model_name,\n", " PrimaryContainer={\n", " 'Image': image_uri,\n", " 'ModelDataUrl': model_s3_uri,\n", " 'Environment': {\n", " 'SAGEMAKER_CONTAINER_LOG_LEVEL': '20',\n", " 'SAGEMAKER_REGION': region,\n", " }\n", " },\n", " ExecutionRoleArn=role,\n", ")" ] }, { "cell_type": "markdown", "id": "819e4d68-56a0-440c-bce0-65c61f2f364d", "metadata": { "tags": [] }, "source": [ "##### エンドポイントコンフィグの作成" ] }, { "cell_type": "code", "execution_count": null, "id": "a118dc38", "metadata": { "tags": [] }, "outputs": [], "source": [ "response = sm.create_endpoint_config(\n", " EndpointConfigName=endpoint_config_name,\n", " ProductionVariants=[\n", " {\n", " 'VariantName': 'AllTrafic',\n", " 'ModelName': model_name,\n", " 'InitialInstanceCount': 1,\n", " 'InstanceType': 'ml.g5.2xlarge',\n", " },\n", " ]\n", ")" ] }, { "cell_type": "markdown", "id": "c4d159b7-b6ca-401e-82f5-0682a597d162", "metadata": {}, "source": [ "##### エンドポイントの作成" ] }, { "cell_type": "code", "execution_count": null, "id": "3e39686d", "metadata": { "tags": [] }, "outputs": [], "source": [ "response = sm.create_endpoint(\n", " EndpointName=endpoint_name,\n", " EndpointConfigName=endpoint_config_name,\n", ")\n", "endpoint_inservice_waiter.wait(\n", " EndpointName=endpoint_name,\n", " WaiterConfig={'Delay': 5,}\n", ")" ] }, { "cell_type": "markdown", "id": "2017ad04-23c8-4259-83a6-4812b3784291", "metadata": {}, "source": [ "#### 推論" ] }, { "cell_type": "code", "execution_count": null, "id": "6e6b059d-89d1-45c0-819a-e7f3b412e74a", "metadata": { "tags": [] }, "outputs": [], "source": [ "# prompt 確認\n", "print(request)" ] }, { "cell_type": "code", "execution_count": null, "id": "69159367", "metadata": { "tags": [] }, "outputs": [], "source": [ "%%time\n", "\n", "# 推論\n", "smr = boto3.client('sagemaker-runtime')\n", "\n", "for i in range(10):\n", " try:\n", " response = smr.invoke_endpoint(\n", " EndpointName=endpoint_name,\n", " ContentType='application/json',\n", " Accept='application/json',\n", " Body=json.dumps(request)\n", " )\n", " break\n", " except:\n", " sleep(60)\n", "output_str = json.loads(response['Body'].read().decode('utf-8'))['result']\n", "print(output_str[:output_str.find('<|endoftext|>')])" ] }, { "cell_type": "markdown", "id": "7cf1199f-f812-46f5-8b2d-423f0e9b6ece", "metadata": {}, "source": [ "#### お片付け" ] }, { "cell_type": "code", "execution_count": null, "id": "3b036b51", "metadata": { "tags": [] }, "outputs": [], "source": [ "sm.delete_endpoint(EndpointName=endpoint_name)\n", "sm.delete_endpoint_config(EndpointConfigName=endpoint_config_name)\n", "sm.delete_model(ModelName=model_name)" ] }, { "cell_type": "code", "execution_count": null, "id": "34d781e1-e2f1-41c8-83a6-59608b133c08", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "availableInstances": [ { "_defaultOrder": 0, "_isFastLaunch": true, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 4, "name": "ml.t3.medium", "vcpuNum": 2 }, { "_defaultOrder": 1, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.t3.large", "vcpuNum": 2 }, { "_defaultOrder": 2, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.t3.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 3, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.t3.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 4, "_isFastLaunch": true, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.m5.large", "vcpuNum": 2 }, { "_defaultOrder": 5, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.m5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 6, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.m5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 7, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.m5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 8, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.m5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 9, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.m5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 10, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.m5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 11, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.m5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 12, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.m5d.large", "vcpuNum": 2 }, { "_defaultOrder": 13, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.m5d.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 14, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.m5d.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 15, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.m5d.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 16, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.m5d.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 17, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.m5d.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 18, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.m5d.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 19, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.m5d.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 20, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": true, "memoryGiB": 0, "name": "ml.geospatial.interactive", "supportedImageNames": [ "sagemaker-geospatial-v1-0" ], "vcpuNum": 0 }, { "_defaultOrder": 21, "_isFastLaunch": true, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 4, "name": "ml.c5.large", "vcpuNum": 2 }, { "_defaultOrder": 22, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.c5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 23, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.c5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 24, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.c5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 25, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 72, "name": "ml.c5.9xlarge", "vcpuNum": 36 }, { "_defaultOrder": 26, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 96, "name": "ml.c5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 27, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 144, "name": "ml.c5.18xlarge", "vcpuNum": 72 }, { "_defaultOrder": 28, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.c5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 29, "_isFastLaunch": true, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.g4dn.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 30, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.g4dn.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 31, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.g4dn.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 32, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.g4dn.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 33, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.g4dn.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 34, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.g4dn.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 35, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 61, "name": "ml.p3.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 36, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 244, "name": "ml.p3.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 37, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 488, "name": "ml.p3.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 38, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 768, "name": "ml.p3dn.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 39, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.r5.large", "vcpuNum": 2 }, { "_defaultOrder": 40, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.r5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 41, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.r5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 42, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.r5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 43, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.r5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 44, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.r5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 45, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 512, "name": "ml.r5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 46, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 768, "name": "ml.r5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 47, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.g5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 48, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.g5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 49, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.g5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 50, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.g5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 51, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.g5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 52, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.g5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 53, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.g5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 54, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 768, "name": "ml.g5.48xlarge", "vcpuNum": 192 }, { "_defaultOrder": 55, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 1152, "name": "ml.p4d.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 56, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 1152, "name": "ml.p4de.24xlarge", "vcpuNum": 96 } ], "instance_type": "ml.m5.2xlarge", "kernelspec": { "display_name": "Python 3 (PyTorch 1.13 Python 3.9 CPU Optimized)", "language": "python", "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/pytorch-1.13-cpu-py39" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.16" } }, "nbformat": 4, "nbformat_minor": 5 }