{ "cells": [ { "cell_type": "markdown", "id": "bc5ab391", "metadata": { "tags": [] }, "source": [ "# Kor LLM 모델 서빙" ] }, { "cell_type": "markdown", "id": "694d9185", "metadata": {}, "source": [ "### 참조: \n", "- Model 정보\n", " - beomi/KoAlpaca-Polyglot-12.8B\n", " - This model is a fine-tuned version of EleutherAI/polyglot-ko-12.8b on a KoAlpaca Dataset v1.1b\n", " - https://huggingface.co/beomi/KoAlpaca-Polyglot-12.8B\n", " - EleutherAI/polyglot-ko-12.8b\n", " - Polyglot-Ko-12.8B was trained for 167 billion tokens over 301,000 steps on 256 A100 GPUs with the GPT-NeoX framework. It was trained as an autoregressive language model, using cross-entropy loss to maximize the likelihood of predicting the next token.\n", " - License: Apache 2.0\n", " - https://huggingface.co/EleutherAI/polyglot-ko-12.8b\n", " \n", "- 블로그\n", " - https://aws.amazon.com/ko/blogs/machine-learning/deploy-large-models-on-amazon-sagemaker-using-djlserving-and-deepspeed-model-parallel-inference/\n", "- 코드\n", " - Boto3\n", " - https://github.com/aws/amazon-sagemaker-examples/blob/main/advanced_functionality/pytorch_deploy_large_GPT_model/GPT-J-6B-model-parallel-inference-DJL.ipynb\n", " - Python SDK\n", " - https://github.com/aws/amazon-sagemaker-examples/blob/main/inference/generativeai/deepspeed/GPT-J-6B_DJLServing_with_PySDK.ipynb" ] }, { "cell_type": "markdown", "id": "7b1ae4f8-2cf7-40eb-adf8-6dca4260c92e", "metadata": { "tags": [] }, "source": [ "# 1. 기본 환경 설정" ] }, { "cell_type": "code", "execution_count": 2, "id": "cee2723b-2c1e-4924-9853-664a26cd7075", "metadata": { "tags": [] }, "outputs": [], "source": [ "%load_ext autoreload\n", "%autoreload 2\n", "\n", "# src 폴더 경로 설정\n", "import sys\n", "sys.path.append('../common_code')" ] }, { "cell_type": "markdown", "id": "81c2bdf4", "metadata": {}, "source": [ "# 2. SageMaker endpoint 의 추론 도커 이미지 인 DLC image URL 가져오기\n", "- We get DLC image URL for djl-deepspeed 0.21.0 and set SageMaker settings" ] }, { "cell_type": "code", "execution_count": 3, "id": "2876d11c", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "'763104351884.dkr.ecr.us-east-1.amazonaws.com/djl-inference:0.21.0-deepspeed0.8.3-cu117'" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import sagemaker, boto3\n", "from sagemaker import image_uris\n", "\n", "\n", "role = sagemaker.get_execution_role() # execution role for the endpoint\n", "session = sagemaker.session.Session() # sagemaker session for interacting with different AWS APIs\n", "region = session._region_name\n", "bucket = session.default_bucket() # bucket to house artifacts\n", "\n", "img_uri = image_uris.retrieve(framework=\"djl-deepspeed\", region=region, version=\"0.21.0\")\n", "img_uri" ] }, { "cell_type": "markdown", "id": "bdff2960-a103-4016-be3e-83947b4d36d6", "metadata": {}, "source": [ "# 3. Set configuration" ] }, { "cell_type": "markdown", "id": "e6bf555e-7128-4378-9f48-103674ebc859", "metadata": {}, "source": [ "## 테스트 모델 지정" ] }, { "cell_type": "code", "execution_count": 4, "id": "a2ca985f-a53f-459e-bd8c-6adfeb2977e4", "metadata": { "tags": [] }, "outputs": [], "source": [ "serve_model = 'KoAlpaca-12-8B'\n", "# serve_model = 'Polyglot-Kor-5-8B'\n", "# serve_model = 'Kullm-polyglot-12-8b-v2'" ] }, { "cell_type": "code", "execution_count": 5, "id": "665498b3-9904-4f92-9d24-9b684132b169", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "instance_type : ml.g5.12xlarge\n" ] } ], "source": [ "model_artifact_name = f'{serve_model}.tar.gz'\n", "\n", "instance_type = \"ml.g5.12xlarge\"\n", "# instance_type = \"ml.g5.48xlarge\"\n", "\n", "print(\"instance_type :\", instance_type) \n" ] }, { "cell_type": "code", "execution_count": 6, "id": "47ab31d1", "metadata": { "scrolled": true, "tags": [] }, "outputs": [ { "data": { "text/plain": [ "'s3://sagemaker-us-east-1-057716757052/KoAlpaca-12-8B/'" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s3_location = f\"s3://{bucket}/{serve_model}/\"\n", "s3_location" ] }, { "cell_type": "markdown", "id": "1ac32e96", "metadata": {}, "source": [ "# 4. 모델 추론 코드 및 모델 설정 파일을 패키징\n", "- `model.py` and `serving.properties`\n", "- The code below creates the SageMaker model file (`model.tar.gz`) and upload it to S3. " ] }, { "cell_type": "code", "execution_count": 7, "id": "4a724622-069c-453a-8c2d-3a43adbc0a91", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "KoAlpaca-12-8B\n", "KoAlpaca-12-8B.tar.gz\n", "KoAlpaca-12-8B/\n", "KoAlpaca-12-8B/model.py\n", "KoAlpaca-12-8B/serving.properties\n" ] } ], "source": [ "%%sh -s {serve_model} {model_artifact_name}\n", "serve_model=$1\n", "model_artifact_name=$2\n", "echo $serve_model\n", "echo $model_artifact_name\n", "\n", "rm -rf $serve_model/.ipynb_checkpoints\n", "\n", "tar -czvf $model_artifact_name $serve_model/\n", "\n" ] }, { "cell_type": "markdown", "id": "4b9bbee9-224b-4daf-915e-a1e40d72e47f", "metadata": {}, "source": [ "## mode.tar.gz 를 S3 업로드" ] }, { "cell_type": "code", "execution_count": 8, "id": "db47f969", "metadata": { "tags": [] }, "outputs": [], "source": [ "model_tar_url = sagemaker.s3.S3Uploader.upload(model_artifact_name, s3_location)" ] }, { "cell_type": "markdown", "id": "c507e3ef", "metadata": {}, "source": [ "# 5. SageMaker endpoint 생성" ] }, { "cell_type": "markdown", "id": "32589338", "metadata": {}, "source": [ "- Now we create our [SageMaker model](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_model). Make sure your execution role has access to your model artifacts and ECR image. Please check out our SageMaker Roles [documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.html) for more details. " ] }, { "cell_type": "code", "execution_count": 9, "id": "026d27d2", "metadata": { "tags": [] }, "outputs": [], "source": [ "from datetime import datetime\n", "\n", "sm_client = boto3.client(\"sagemaker\")\n", "\n", "time_stamp = datetime.now().strftime(\"%Y-%m-%d-%H-%M-%S\")\n", "model_name = f\"{serve_model}-\" + time_stamp\n", "\n", "create_model_response = sm_client.create_model(\n", " ModelName=model_name,\n", " ExecutionRoleArn=role,\n", " PrimaryContainer={\"Image\": img_uri, \"ModelDataUrl\": model_tar_url},\n", ")" ] }, { "cell_type": "markdown", "id": "22d2fc2b", "metadata": {}, "source": [ "Now we create an endpoint configuration that SageMaker hosting services uses to deploy models. Note that we configured `ModelDataDownloadTimeoutInSeconds` and `ContainerStartupHealthCheckTimeoutInSeconds` to accommodate the large size of our model. " ] }, { "cell_type": "code", "execution_count": 10, "id": "84e25dd4", "metadata": { "tags": [] }, "outputs": [], "source": [ "initial_instance_count = 1\n", "variant_name = \"AllTraffic\"\n", "endpoint_config_name = f\"{serve_model}-config-\" + time_stamp\n", "\n", "production_variants = [\n", " {\n", " \"VariantName\": variant_name,\n", " \"ModelName\": model_name,\n", " \"InitialInstanceCount\": initial_instance_count,\n", " \"InstanceType\": instance_type,\n", " \"ModelDataDownloadTimeoutInSeconds\": 300,\n", " \"ContainerStartupHealthCheckTimeoutInSeconds\": 300,\n", " }\n", "]\n", "\n", "endpoint_config = {\n", " \"EndpointConfigName\": endpoint_config_name,\n", " \"ProductionVariants\": production_variants,\n", "}\n", "\n", "ep_conf_res = sm_client.create_endpoint_config(**endpoint_config)" ] }, { "cell_type": "markdown", "id": "e4b3bc26", "metadata": {}, "source": [ "We are ready to create an endpoint using the model and the endpoint configuration created from above steps. " ] }, { "cell_type": "code", "execution_count": 11, "id": "962a1aef", "metadata": { "tags": [] }, "outputs": [], "source": [ "endpoint_name = f\"{serve_model}-\" + time_stamp\n", "ep_res = sm_client.create_endpoint(\n", " EndpointName=endpoint_name, EndpointConfigName=endpoint_config_name\n", ")" ] }, { "cell_type": "code", "execution_count": 12, "id": "d044bed4-9f47-4f15-893b-5af2c087e3a1", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "endpoint_name: KoAlpaca-12-8B-2023-06-04-11-40-34\n" ] } ], "source": [ "print(\"endpoint_name: \", endpoint_name)" ] }, { "cell_type": "code", "execution_count": 13, "id": "62d9c280-76b3-4c58-8fb9-2b54e5bc5623", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Endpoint is Creating\n", "Endpoint is Creating\n", "Endpoint is Creating\n", "Endpoint is Creating\n", "Endpoint is Creating\n", "Endpoint is Creating\n", "Endpoint is Creating\n", "Endpoint is InService\n", "CPU times: user 977 ms, sys: 55.6 ms, total: 1.03 s\n", "Wall time: 7min 2s\n" ] } ], "source": [ "%%time \n", "\n", "from inference_lib import descirbe_endpoint\n", "descirbe_endpoint(endpoint_name) " ] }, { "cell_type": "markdown", "id": "8f1221df-caf9-4050-9541-34bf558538c7", "metadata": {}, "source": [ "# 6. 엔드포인트 추론 " ] }, { "cell_type": "code", "execution_count": 20, "id": "5ed7a325", "metadata": { "tags": [] }, "outputs": [], "source": [ "from inference_lib import invoke_inference_DJ" ] }, { "cell_type": "markdown", "id": "63c5791c-5b67-4e3f-98a6-327baa81d7b2", "metadata": { "tags": [] }, "source": [ "### options for generation\n", "* **temperature**: Controls randomness in the model. Lower values will make the model more deterministic and higher values will make the model more random. Default value is 1.0.\n", "* **max_new_tokens**: The maximum number of tokens to generate. Default value is 20, max value is 512.\n", "* **repetition_penalty**: Controls the likelihood of repetition, defaults to null.\n", "* **seed**: The seed to use for random generation, default is null.\n", "* **stop**: A list of tokens to stop the generation. The generation will stop when one of the tokens is generated.\n", "* **top_k**: The number of highest probability vocabulary tokens to keep for top-k-filtering. Default value is null, which disables top-k-filtering.\n", "* **top_p**: The cumulative probability of parameter highest probability vocabulary tokens to keep for nucleus sampling, default to null\n", "* **do_sample**: Whether or not to use sampling ; use greedy decoding otherwise. Default value is false.\n", "* **best_of**: Generate best_of sequences and return the one if the highest token logprobs, default to null.\n", "* **details**: Whether or not to return details about the generation. Default value is false.\n", "* **return_full_text**: Whether or not to return the full text or only the generated part. Default value is false.\n", "* **truncate**: Whether or not to truncate the input to the maximum length of the model. Default value is true.\n", "* **typical_p**: The typical probability of a token. Default value is null.\n", "* **watermark**: The watermark to use for the generation. Default value is false." ] }, { "cell_type": "code", "execution_count": 21, "id": "3bd74737-4f00-4a18-8c2d-042a8cecf65c", "metadata": { "tags": [] }, "outputs": [], "source": [ "params = {\n", " \"do_sample\":False, \n", " \"max_new_tokens\":128,\n", " \"temperature\":1.0,\n", " \"top_k\":0,\n", " \"top_p\":0.9,\n", " \"return_full_text\":False,\n", " \"repetition_penalty\":1.1,\n", " \"presence_penalty\":None,\n", " \"eos_token_id\":2,\n", "}" ] }, { "cell_type": "markdown", "id": "aa002070-b85d-4f3d-a3da-57ed0d63ac16", "metadata": { "tags": [] }, "source": [ "## (1) 맥락 (Context) 없이 질문" ] }, { "cell_type": "code", "execution_count": 22, "id": "74e7eeea-2eef-464a-aca8-7b4d5116ba1e", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "prompt_wo_c: \n", " {'prompt': ['### 질문: 홈플러스 중계점은 몇시까지 장사해?\\n\\n### 답변:'], 'params': {'do_sample': False, 'max_new_tokens': 128, 'temperature': 1.0, 'top_k': 0, 'top_p': 0.9, 'return_full_text': False, 'repetition_penalty': 1.1, 'presence_penalty': None, 'eos_token_id': 2}}\n" ] } ], "source": [ "q = \"홈플러스 중계점은 몇시까지 장사해?\"\n", "c = \"\"#\"홈플러스 영업시간은 오전 10시 부터 오후 12시까지 입니다.\"\n", "prompt_wo_c = f\"### 질문: {q}\\n\\n### 맥락: {c}\\n\\n### 답변:\" if c else f\"### 질문: {q}\\n\\n### 답변:\" \n", "data = {\n", " \"prompt\": [prompt_wo_c,],\n", " \"params\": params\n", "}\n", "print(\"prompt_wo_c: \\n\", data)" ] }, { "cell_type": "code", "execution_count": 23, "id": "7b73fca4-9318-401c-97ba-3005c40adfde", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[\n", " [\n", " {\n", " \"generated_text\":\"홈플러스의 매장 운영 시간은 점포마다 다릅니다. 대부분의 매장은 10시에 문을 닫으며, 일부 매장은 11시까지 영업합니다. 예를 들어, 홈플러스 동대문점은 11시까지 영업하며, 홈플러스 영등포점과 강서점은 9시에 문을 닫습니다. 또한, 홈플러스 김해점과 밀양점은 8시에 문을 닫고 있습니다. \"\n", " }\n", " ]\n", "]\n", "CPU times: user 37.2 ms, sys: 366 µs, total: 37.6 ms\n", "Wall time: 4.89 s\n" ] }, { "data": { "text/plain": [ "'[\\n [\\n {\\n \"generated_text\":\"홈플러스의 매장 운영 시간은 점포마다 다릅니다. 대부분의 매장은 10시에 문을 닫으며, 일부 매장은 11시까지 영업합니다. 예를 들어, 홈플러스 동대문점은 11시까지 영업하며, 홈플러스 영등포점과 강서점은 9시에 문을 닫습니다. 또한, 홈플러스 김해점과 밀양점은 8시에 문을 닫고 있습니다. \"\\n }\\n ]\\n]'" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "%%time \n", "invoke_inference_DJ(endpoint_name, data)" ] }, { "cell_type": "markdown", "id": "c47feb14-8bc0-42f7-8319-fa37ca1ee340", "metadata": {}, "source": [ "## (2) 맥락 (Context) 가지고 질문" ] }, { "cell_type": "code", "execution_count": 24, "id": "6864581e-186d-412e-bd7f-c5cd049539c8", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "prompt_w_c:\n", " ### 질문: 홈플러스 중계점은 몇시까지 장사해?\n", "\n", "### 맥락: 홈플러스 영업시간은 오전 10시 부터 오후 10시까지 입니다. 홈플러스 매장 찾기(영업시간 확인)는 이 주소를 이용하세요: http://corporate.homeplus.co.kr/Store.aspx?isA=%C1%F6%BF%B4%C7%B0%BF%AE%C0%C7%C1%F2%B5%B5%B4%F6 \n", "\n", "### 답변:\n" ] } ], "source": [ "q = \"홈플러스 중계점은 몇시까지 장사해?\"\n", "c = \"홈플러스 영업시간은 오전 10시 부터 오후 10시까지 입니다. 홈플러스 매장 찾기(영업시간 확인)는 이 주소를 이용하세요: http://corporate.homeplus.co.kr/Store.aspx?isA=%C1%F6%BF%B4%C7%B0%BF%AE%C0%C7%C1%F2%B5%B5%B4%F6 \"\n", "prompt_w_c = f\"### 질문: {q}\\n\\n### 맥락: {c}\\n\\n### 답변:\" if c else f\"### 질문: {q}\\n\\n### 답변:\" \n", "data = {\n", " \"prompt\": [prompt_w_c,],\n", " \"params\": params\n", "}\n", "print(\"prompt_w_c:\\n\", prompt_w_c)" ] }, { "cell_type": "code", "execution_count": 25, "id": "a7b53302-ec17-4a8d-837e-e93d7c5b90cf", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[\n", " [\n", " {\n", " \"generated_text\":\"홈플러스의 영업 시간은 오전 10시부터 오후 10시까지입니다. 각 지점마다 약간씩 차이가 있을 수 있으므로, 방문 전에 영업 시간을 꼭 확인해보시기 바랍니다. 아래 링크에서 홈플러스 매장 찾기를 통해 영업 시간을 확인하실 수 있습니다: http://corporate.homeplus.co.kr/Store.aspx?isA=%C1%F6%BF%B4%C7%B0%BF%AE%C0%C7%C1%F2%\"\n", " }\n", " ]\n", "]\n", "CPU times: user 37.4 ms, sys: 0 ns, total: 37.4 ms\n", "Wall time: 7.58 s\n" ] }, { "data": { "text/plain": [ "'[\\n [\\n {\\n \"generated_text\":\"홈플러스의 영업 시간은 오전 10시부터 오후 10시까지입니다. 각 지점마다 약간씩 차이가 있을 수 있으므로, 방문 전에 영업 시간을 꼭 확인해보시기 바랍니다. 아래 링크에서 홈플러스 매장 찾기를 통해 영업 시간을 확인하실 수 있습니다: http://corporate.homeplus.co.kr/Store.aspx?isA=%C1%F6%BF%B4%C7%B0%BF%AE%C0%C7%C1%F2%\"\\n }\\n ]\\n]'" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "%%time \n", "invoke_inference_DJ(endpoint_name, data)" ] }, { "cell_type": "markdown", "id": "92e83c91", "metadata": {}, "source": [ "# 7. [중요] 클린업 엔트포인트 " ] }, { "cell_type": "code", "execution_count": 107, "id": "a15980a3", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "{'ResponseMetadata': {'RequestId': '995450b5-7fa3-4810-be70-967959b79ebd',\n", " 'HTTPStatusCode': 200,\n", " 'HTTPHeaders': {'x-amzn-requestid': '995450b5-7fa3-4810-be70-967959b79ebd',\n", " 'content-type': 'application/x-amz-json-1.1',\n", " 'content-length': '0',\n", " 'date': 'Thu, 01 Jun 2023 14:03:56 GMT'},\n", " 'RetryAttempts': 0}}" ] }, "execution_count": 107, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# sm_client.delete_endpoint(EndpointName=endpoint_name)\n", "# sm_client.delete_endpoint_config(EndpointConfigName=endpoint_config_name)\n", "# sm_client.delete_model(ModelName=model_name)" ] }, { "cell_type": "code", "execution_count": null, "id": "c7ca877d-400d-467f-8501-976b94e015cb", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "id": "c0d06b7b-8997-4025-bada-793b1e33c7fa", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "availableInstances": [ { "_defaultOrder": 0, "_isFastLaunch": true, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 4, "name": "ml.t3.medium", "vcpuNum": 2 }, { "_defaultOrder": 1, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.t3.large", "vcpuNum": 2 }, { "_defaultOrder": 2, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.t3.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 3, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.t3.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 4, "_isFastLaunch": true, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.m5.large", "vcpuNum": 2 }, { "_defaultOrder": 5, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.m5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 6, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.m5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 7, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.m5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 8, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.m5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 9, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.m5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 10, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.m5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 11, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.m5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 12, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.m5d.large", "vcpuNum": 2 }, { "_defaultOrder": 13, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.m5d.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 14, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.m5d.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 15, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.m5d.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 16, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.m5d.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 17, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.m5d.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 18, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.m5d.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 19, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.m5d.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 20, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": true, "memoryGiB": 0, "name": "ml.geospatial.interactive", "supportedImageNames": [ "sagemaker-geospatial-v1-0" ], "vcpuNum": 0 }, { "_defaultOrder": 21, "_isFastLaunch": true, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 4, "name": "ml.c5.large", "vcpuNum": 2 }, { "_defaultOrder": 22, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.c5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 23, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.c5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 24, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.c5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 25, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 72, "name": "ml.c5.9xlarge", "vcpuNum": 36 }, { "_defaultOrder": 26, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 96, "name": "ml.c5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 27, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 144, "name": "ml.c5.18xlarge", "vcpuNum": 72 }, { "_defaultOrder": 28, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.c5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 29, "_isFastLaunch": true, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.g4dn.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 30, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.g4dn.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 31, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.g4dn.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 32, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.g4dn.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 33, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.g4dn.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 34, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.g4dn.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 35, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 61, "name": "ml.p3.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 36, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 244, "name": "ml.p3.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 37, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 488, "name": "ml.p3.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 38, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 768, "name": "ml.p3dn.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 39, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.r5.large", "vcpuNum": 2 }, { "_defaultOrder": 40, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.r5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 41, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.r5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 42, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.r5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 43, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.r5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 44, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.r5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 45, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 512, "name": "ml.r5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 46, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 768, "name": "ml.r5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 47, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.g5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 48, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.g5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 49, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.g5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 50, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.g5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 51, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.g5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 52, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.g5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 53, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.g5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 54, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 768, "name": "ml.g5.48xlarge", "vcpuNum": 192 } ], "instance_type": "ml.m5.2xlarge", "kernelspec": { "display_name": "Python 3 (PyTorch 1.13 Python 3.9 CPU Optimized)", "language": "python", "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/pytorch-1.13-cpu-py39" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.16" }, "vscode": { "interpreter": { "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6" } } }, "nbformat": 4, "nbformat_minor": 5 }