{ "cells": [ { "cell_type": "markdown", "id": "03050374-58d4-4388-a334-d06a395bfd40", "metadata": {}, "source": [ "# Finetuning Foundation Models - HuggingFace Text2Text- FLAN" ] }, { "cell_type": "markdown", "id": "362b27d1-cd77-46d7-88f0-68748156703d", "metadata": {}, "source": [ "In this demo notebook, we use the SageMaker Python SDK to **fine-tune a Text2Text model**. Such a model takes prompting text as input and generates text as output. The prompt can include a task description in natural language. Accordingly, the model can be used for a variety of NLP tasks (e.g., text summarization, question answering, etc.).\n", "\n", "We will fine-tune a pre-trained **FLAN T5 model** from [Hugging Face](https://huggingface.co/docs/transformers/model_doc/flan-t5). While pre-trained FLAN T5 models can be used \"as is\" for many tasks, fine-tuning can improve model performance on a particular task or language domain. As an example, we will fine-tune the model for a task that was not used for pre-training. After fine-tuning we will deploy two inference endpoints, one with a pre-trained and one with a fine-tuned model. We will then run the same inference query against both endpoints and compare results." ] }, { "cell_type": "markdown", "id": "d1706e56-3d74-4f2c-b5cd-695acca57d5c", "metadata": {}, "source": [ "#### In this notebook:\n", "1. [Setting up](#1.-Setting-up)\n", "1. [Fine-tuning a model](#2.-Fine-tuning-a-model)\n", "1. [Deploying inference endpoints](#3.-Deploying-inference-endpoints)\n", "1. [Running inference queries](#4.-Running-inference-queries)\n", "1. [Cleaning up resources](#5.-Cleaning-up-resources)" ] }, { "cell_type": "markdown", "id": "1c30630a-685b-4be1-9c32-e50252a77b87", "metadata": {}, "source": [ "### 1. Setting up" ] }, { "cell_type": "markdown", "id": "5b97ab4c-f05c-4696-8040-f7b0a17ba21e", "metadata": {}, "source": [ "We begin by installing and upgrading necessary packages. Restart the kernel after executing the cell below." ] }, { "cell_type": "code", "execution_count": 3, "id": "ef9bf59d-f7fb-47e1-b2cf-7be6a819be96", "metadata": { "tags": [] }, "outputs": [], "source": [ "#!pip install nest-asyncio==1.5.5 --quiet\n", "#!pip install ipywidgets==8.0.4 --quiet\n", "#!pip install sagemaker==2.148.0 --quiet" ] }, { "cell_type": "markdown", "id": "714a91ac-431d-4adc-8285-4e1dd73721d1", "metadata": { "tags": [] }, "source": [ "We will use the following variables throughout the notebook. In particular, we select FLAN T5 model size and select training and inference instance types. We also obtain execution role associated with the current notebook instance." ] }, { "cell_type": "code", "execution_count": 4, "id": "c83140fe-f5d2-49ee-a433-70109cf23bd5", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[1maws_region:\u001b[0m us-east-1\n", "\u001b[1maws_role:\u001b[0m arn:aws:iam::509957658284:role/service-role/AmazonSageMaker-ExecutionRole-20211126T131684\n", "\u001b[1moutput_bucket:\u001b[0m sagemaker-us-east-1-509957658284\n" ] } ], "source": [ "import boto3\n", "import sagemaker\n", "\n", "# Get current region, role, and default bucket\n", "aws_region = boto3.Session().region_name\n", "aws_role = sagemaker.session.Session().get_caller_identity_arn()\n", "output_bucket = sagemaker.Session().default_bucket()\n", "\n", "# This will be useful for printing\n", "newline, bold, unbold = \"\\n\", \"\\033[1m\", \"\\033[0m\"\n", "\n", "print(f\"{bold}aws_region:{unbold} {aws_region}\")\n", "print(f\"{bold}aws_role:{unbold} {aws_role}\")\n", "print(f\"{bold}output_bucket:{unbold} {output_bucket}\")" ] }, { "cell_type": "markdown", "id": "c5f41438-e53c-4d04-bea7-a2a1f9408128", "metadata": { "tags": [] }, "source": [ "## Select Flan model" ] }, { "cell_type": "code", "execution_count": 5, "id": "23ec3986-9905-4c40-b81e-ca54f78311cf", "metadata": { "tags": [] }, "outputs": [], "source": [ "import IPython\n", "from ipywidgets import Dropdown\n", "from sagemaker.jumpstart.filters import And\n", "from sagemaker.jumpstart.notebook_utils import list_jumpstart_models\n", "\n", "# Default model choice\n", "model_id = \"huggingface-text2text-flan-t5-small\"\n", "model_version = \"*\"" ] }, { "cell_type": "code", "execution_count": 6, "id": "905ddba9-de01-4059-af82-3c178ecbed60", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[1mmodel_id:\u001b[0m huggingface-text2text-flan-t5-small\n", "\u001b[1mtraining_instance_type:\u001b[0m ml.p3.2xlarge\n", "\u001b[1minference_instance_type:\u001b[0m ml.g5.2xlarge\n" ] } ], "source": [ "from sagemaker.instance_types import retrieve_default\n", "\n", "# Instance types for training and inference\n", "training_instance_type = retrieve_default(\n", " model_id=model_id, model_version=model_version, scope=\"training\"\n", ")\n", "\n", "training_instance_type = \"ml.p3.2xlarge\" \n", "inference_instance_type = \"ml.g5.2xlarge\"\n", "print(f\"{bold}model_id:{unbold} {model_id}\")\n", "print(f\"{bold}training_instance_type:{unbold} {training_instance_type}\")\n", "print(f\"{bold}inference_instance_type:{unbold} {inference_instance_type}\")" ] }, { "cell_type": "markdown", "id": "237c7754-ffc3-40a0-9f85-bda3b25161d0", "metadata": {}, "source": [ "### 2. Fine-tuning a model" ] }, { "cell_type": "markdown", "id": "d898c437-0b3a-47b2-9b25-827d824f83ec", "metadata": {}, "source": [ "FLAN T5 models were pre-trained on a variety of tasks. In this demo, we fine-tune a model for a new task. In this task, given a piece of text, the model is asked to generate questions that are relevant to the text, but cannot be answered based on provided information. Examples are given in the inference section of this notebook." ] }, { "cell_type": "markdown", "id": "0c377423-c7d5-4087-a175-717227d47936", "metadata": { "tags": [] }, "source": [ "#### 2.1. Preparing training data\n", "We will use a subset of SQuAD2.0 for supervised fine-tuning. This dataset contains questions posed by human annotators on a set of Wikipedia articles. In addition to questions with answers, SQuAD2.0 contains about 50k unanswerable questions. Such questions are plausible, but cannot be directly answered from the articles' content. We only use unanswerable questions for our task.\n", "\n", "*Citation: @article{rajpurkar2018know, title={Know what you don't know: Unanswerable questions for SQuAD},\n", "author={Rajpurkar, Pranav and Jia, Robin and Liang, Percy}, journal={arXiv preprint arXiv:1806.03822}, year={2018} }*\n", "\n", "License: [Creative Commons Attribution-ShareAlike License (CC BY-SA 4.0)](https://creativecommons.org/licenses/by-sa/4.0/legalcode)\n", "#original_data_location = f\"s3://sagemaker-sample-files/datasets/text/squad2.0/{original_data_file}\"" ] }, { "cell_type": "code", "execution_count": 7, "id": "96bb8fb1-84e1-43d3-981b-769856e1c204", "metadata": { "tags": [] }, "outputs": [], "source": [ "from sagemaker.s3 import S3Downloader\n", "\n", "original_data_file = \"train-v2.0.json\"" ] }, { "cell_type": "markdown", "id": "7d9c3efd-63ae-4d01-9b13-cd9c48c1af92", "metadata": {}, "source": [ "The Text2Text generation model can be fine-tuned on any text data provided that the data is in the expected format. The data must include a training and an optional validation parts. The best model is selected according to the validation loss, calculated at the end of each epoch. If a validation set is not given, an (adjustable) percentage of the training data is automatically split and used for validation.\n", "\n", "The training data must be formatted in JSON lines (`.jsonl`) format, where each line is a dictionary representing a single data sample. All training data must be in a single folder, however it can be saved in multiple jsonl files. The `.jsonl` file extension is mandatory. The training folder can also contain a `template.json` file describing the input and output formats.\n", "\n", "If no template file is given, the following default template will be used:\n", "```json\n", "{\n", " \"prompt\": \"{prompt}\",\n", " \"completion\": \"{completion}\"\n", "}\n", "```\n", "In this case, the data in the JSON lines entries must include `prompt` and `completion` fields.\n", "\n", "In this demo, we are going to use a custom template (see below)." ] }, { "cell_type": "code", "execution_count": 8, "id": "3002bc68-2779-4fc0-a68b-cbdd0b3663bd", "metadata": { "tags": [] }, "outputs": [], "source": [ "import json\n", "\n", "local_data_file = \"task-data.jsonl\" # any name with .jsonl extension\n", "\n", "with open(original_data_file) as f:\n", " data = json.load(f)\n", "\n", "with open(local_data_file, \"w\") as f:\n", " for article in data[\"data\"]:\n", " for paragraph in article[\"paragraphs\"]:\n", " # iterate over questions for a given paragraph\n", " for qas in paragraph[\"qas\"]:\n", " if qas[\"is_impossible\"]:\n", " # the question is relevant, but cannot be answered\n", " example = {\"context\": paragraph[\"context\"], \"question\": qas[\"question\"]}\n", " json.dump(example, f)\n", " f.write(\"\\n\")\n", "\n", "template = {\n", " \"prompt\": \"Ask a question which is related to the following text, but cannot be answered based on the text. Text: {context}\",\n", " \"completion\": \"{question}\",\n", "}\n", "with open(\"template.json\", \"w\") as f:\n", " json.dump(template, f)" ] }, { "cell_type": "markdown", "id": "67f18513-50ab-49a0-9ed1-cf436a6b65dc", "metadata": {}, "source": [ "### Upload to S3" ] }, { "cell_type": "code", "execution_count": 9, "id": "0fc435de-a9c3-403e-a24a-f5ef032755cb", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[1mtraining data:\u001b[0m s3://sagemaker-us-east-1-509957658284/train_data\n" ] } ], "source": [ "from sagemaker.s3 import S3Uploader\n", "\n", "train_data_location = f\"s3://{output_bucket}/train_data\"\n", "S3Uploader.upload(local_data_file, train_data_location)\n", "S3Uploader.upload(\"template.json\", train_data_location)\n", "print(f\"{bold}training data:{unbold} {train_data_location}\")" ] }, { "cell_type": "markdown", "id": "36c448df-769f-41de-ba3b-e9ed0dccb041", "metadata": {}, "source": [ "#### 2.2. Start training\n", "\n", "We are now ready to launch a training job." ] }, { "cell_type": "code", "execution_count": 10, "id": "2e35da74-e166-41e8-b7d3-3804911b522e", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[1mimage uri:\u001b[0m 763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-training:1.10.2-transformers4.17.0-gpu-py38-cu113-ubuntu20.04\n", "\u001b[1mmodel uri:\u001b[0m s3://jumpstart-cache-prod-us-east-1/huggingface-training/train-huggingface-text2text-flan-t5-small.tar.gz\n", "\u001b[1mscript uri:\u001b[0m s3://jumpstart-cache-prod-us-east-1/source-directory-tarballs/huggingface/transfer_learning/text2text/prepack/v1.0.3/sourcedir.tar.gz\n", "\u001b[1moutput location:\u001b[0m s3://sagemaker-us-east-1-509957658284/demo-fine-tune-flan-t5/\n" ] } ], "source": [ "from sagemaker import image_uris, model_uris, script_uris\n", "\n", "# Training instance will use this image\n", "train_image_uri = image_uris.retrieve(\n", " region=aws_region,\n", " framework=None, # automatically inferred from model_id\n", " model_id=model_id,\n", " model_version=model_version,\n", " image_scope=\"training\",\n", " instance_type=training_instance_type,\n", ")\n", "\n", "# Pre-trained model\n", "train_model_uri = model_uris.retrieve(\n", " model_id=model_id, model_version=model_version, model_scope=\"training\"\n", ")\n", "\n", "# Script to execute on the training instance\n", "train_script_uri = script_uris.retrieve(\n", " model_id=model_id, model_version=model_version, script_scope=\"training\"\n", ")\n", "\n", "output_location = f\"s3://{output_bucket}/demo-fine-tune-flan-t5/\"\n", "\n", "print(f\"{bold}image uri:{unbold} {train_image_uri}\")\n", "print(f\"{bold}model uri:{unbold} {train_model_uri}\")\n", "print(f\"{bold}script uri:{unbold} {train_script_uri}\")\n", "print(f\"{bold}output location:{unbold} {output_location}\")" ] }, { "cell_type": "code", "execution_count": 11, "id": "1ed0df7c-0414-49d9-953d-1d48675236c4", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'epochs': '2', 'max_steps': '-1', 'seed': '42', 'batch_size': '64', 'learning_rate': '0.0001', 'lr_scheduler_type': 'constant_with_warmup', 'warmup_ratio': '0.0', 'warmup_steps': '0', 'validation_split_ratio': '0.05', 'train_data_split_seed': '0', 'max_train_samples': '-1', 'max_eval_samples': '-1', 'max_input_length': '-1', 'max_output_length': '128', 'pad_to_max_length': 'True', 'gradient_accumulation_steps': '1', 'weight_decay': '0.0', 'adam_beta1': '0.9', 'adam_beta2': '0.999', 'adam_epsilon': '1e-08', 'max_grad_norm': '1.0', 'load_best_model_at_end': 'True', 'early_stopping_patience': '3', 'early_stopping_threshold': '0.0', 'label_smoothing_factor': '0', 'logging_strategy': 'steps', 'logging_first_step': 'False', 'logging_steps': '500', 'logging_nan_inf_filter': 'True', 'save_strategy': 'epoch', 'save_steps': '500', 'save_total_limit': '2', 'dataloader_drop_last': 'False', 'dataloader_num_workers': '0', 'evalaution_strategy': 'epoch', 'eval_steps': '500', 'eval_accumulation_steps': 'None', 'gradient_checkpointing': 'True', 'auto_find_batch_size': 'False', 'preprocessing_num_workers': 'None'}\n" ] } ], "source": [ "from sagemaker import hyperparameters\n", "\n", "# Retrieve the default hyper-parameters for fine-tuning the model\n", "hyperparameters = hyperparameters.retrieve_default(model_id=model_id, model_version=model_version)\n", "\n", "# We will override some default hyperparameters with custom values\n", "hyperparameters[\"epochs\"] = \"2\"\n", "print(hyperparameters)" ] }, { "cell_type": "markdown", "id": "6fdf3377-4cd7-40df-876b-e0fa2f57dc8f", "metadata": {}, "source": [ "We are now ready to start the training job. This can take a while to complete, from 20 minutes to several hours, depending on the model size, amount of data, and so on (e.g., it can take a few hours for the xl model, 40k examples and 3 epochs)." ] }, { "cell_type": "code", "execution_count": 12, "id": "068d6c5a-00a1-48d0-a2cd-86bcdd15cb0b", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[1mjob name:\u001b[0m js-demo-flan-t5-small-2-2023-05-29-03-24-42-284\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:sagemaker:Creating training-job with name: js-demo-flan-t5-small-2-2023-05-29-03-24-42-284\n" ] } ], "source": [ "from sagemaker.estimator import Estimator\n", "from sagemaker.utils import name_from_base\n", "\n", "model_name = \"-\".join(model_id.split(\"-\")[2:]) # get the most informative part of ID\n", "training_job_name = name_from_base(f\"js-demo-{model_name}-{hyperparameters['epochs']}\")\n", "print(f\"{bold}job name:{unbold} {training_job_name}\")\n", "\n", "training_metric_definitions = [\n", " {\"Name\": \"val_loss\", \"Regex\": \"'eval_loss': ([0-9\\\\.]+)\"},\n", " {\"Name\": \"train_loss\", \"Regex\": \"'loss': ([0-9\\\\.]+)\"},\n", " {\"Name\": \"epoch\", \"Regex\": \"'epoch': ([0-9\\\\.]+)\"},\n", "]\n", "\n", "# Create SageMaker Estimator instance\n", "sm_estimator = Estimator(\n", " role=aws_role,\n", " image_uri=train_image_uri,\n", " model_uri=train_model_uri,\n", " source_dir=train_script_uri,\n", " entry_point=\"transfer_learning.py\",\n", " instance_count=1,\n", " instance_type=training_instance_type,\n", " volume_size=300,\n", " max_run=360000,\n", " hyperparameters=hyperparameters,\n", " output_path=output_location,\n", " metric_definitions=training_metric_definitions,\n", ")\n", "\n", "# Launch a SageMaker training job over data located in the given S3 path\n", "# Training jobs can take hours, it is recommended to set wait=False,\n", "# and monitor job status through SageMaker console\n", "sm_estimator.fit({\"training\": train_data_location}, job_name=training_job_name, wait=False)" ] }, { "cell_type": "markdown", "id": "4524dead-769d-4b5d-88b2-e9587dab474e", "metadata": {}, "source": [ "Performance metrics such as training and validation loss can be accessed through CloudWatch during training. We can also fetch the most recent snapshot of metrics as follows." ] }, { "cell_type": "code", "execution_count": 14, "id": "2683497e-dc98-4fe3-ac15-d09cd58817e6", "metadata": { "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "WARNING:sagemaker.analytics:Warning: No metrics called train_loss found\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
timestampmetric_namevalue
00.0val_loss2.475903
10.0epoch1.666667
\n", "
" ], "text/plain": [ " timestamp metric_name value\n", "0 0.0 val_loss 2.475903\n", "1 0.0 epoch 1.666667" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sagemaker import TrainingJobAnalytics\n", "\n", "# This can be called while the job is still running\n", "df = TrainingJobAnalytics(training_job_name=training_job_name).dataframe()\n", "df.head(10)" ] }, { "cell_type": "markdown", "id": "9e7ba317-699a-4b86-9cf4-254a3f5bdbe6", "metadata": { "tags": [] }, "source": [ "### 3. Deploying inference endpoints" ] }, { "cell_type": "markdown", "id": "629eaa2c-b415-4bd7-b8bc-500f118862e5", "metadata": {}, "source": [ "Remainder of the notebook should be executed once the training job is successfully completed. Recall that variable `training_job_name` contains job name and `output_location` points to an S3 location with a fine-tuned model artifact.\n", "\n", "We will create two inference endpoints, one for the original pre-trained model, and one for the fine-tuned model. We will then run the same request against the two endpoints and compare the results.\n", "\n", "Note that each endpoint deployment can take a few minutes." ] }, { "cell_type": "code", "execution_count": 15, "id": "c37e490b-2c5e-493d-a3f8-03345742f994", "metadata": { "tags": [] }, "outputs": [], "source": [ "from sagemaker import image_uris\n", "\n", "# Retrieve the inference docker image URI. This is the base HuggingFace container image\n", "deploy_image_uri = image_uris.retrieve(\n", " region=aws_region,\n", " framework=None, # automatically inferred from model_id\n", " model_id=model_id,\n", " model_version=model_version,\n", " image_scope=\"inference\",\n", " instance_type=inference_instance_type,\n", ")" ] }, { "cell_type": "code", "execution_count": 16, "id": "487a9d26-11fa-4788-a768-2535060b0aa9", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[1mimage URI:\u001b[0m\n", " 763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-inference:1.10.2-transformers4.17.0-gpu-py38-cu113-ubuntu20.04\n", "\u001b[1mmodel URI:\u001b[0m\n", " s3://jumpstart-cache-prod-us-east-1/huggingface-infer/prepack/v1.0.1/infer-prepack-huggingface-text2text-flan-t5-small.tar.gz\n", "Deploying an endpoint ...\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:sagemaker:Creating model with name: jumpstart-demo-pre-trained-huggingface--2023-05-29-03-37-21-524\n", "INFO:sagemaker:Creating endpoint-config with name jumpstart-demo-pre-trained-huggingface--2023-05-29-03-37-21-524\n", "INFO:sagemaker:Creating endpoint with name jumpstart-demo-pre-trained-huggingface--2023-05-29-03-37-21-524\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "-------!\n", "Deployed an endpoint jumpstart-demo-pre-trained-huggingface--2023-05-29-03-37-21-524\n" ] } ], "source": [ "from sagemaker import model_uris, script_uris\n", "from sagemaker.model import Model\n", "from sagemaker.predictor import Predictor\n", "from sagemaker.utils import name_from_base\n", "\n", "# Retrieve the URI of the pre-trained model\n", "pre_trained_model_uri = model_uris.retrieve(\n", " model_id=model_id, model_version=model_version, model_scope=\"inference\"\n", ")\n", "\n", "pre_trained_name = name_from_base(f\"jumpstart-demo-pre-trained-{model_id}\")\n", "\n", "# Create the SageMaker model instance of the pre-trained model\n", "if (\"small\" in model_id) or (\"base\" in model_id):\n", " deploy_source_uri = script_uris.retrieve(\n", " model_id=model_id, model_version=model_version, script_scope=\"inference\"\n", " )\n", " pre_trained_model = Model(\n", " image_uri=deploy_image_uri,\n", " source_dir=deploy_source_uri,\n", " entry_point=\"inference.py\",\n", " model_data=pre_trained_model_uri,\n", " role=aws_role,\n", " predictor_cls=Predictor,\n", " name=pre_trained_name,\n", " )\n", "else:\n", " # For those large models, we already repack the inference script and model\n", " # artifacts for you, so the `source_dir` argument to Model is not required.\n", " pre_trained_model = Model(\n", " image_uri=deploy_image_uri,\n", " model_data=pre_trained_model_uri,\n", " role=aws_role,\n", " predictor_cls=Predictor,\n", " name=pre_trained_name,\n", " )\n", "\n", "print(f\"{bold}image URI:{unbold}{newline} {deploy_image_uri}\")\n", "print(f\"{bold}model URI:{unbold}{newline} {pre_trained_model_uri}\")\n", "print(\"Deploying an endpoint ...\")\n", "\n", "# Deploy the pre-trained model. Note that we need to pass Predictor class when we deploy model\n", "# through Model class, for being able to run inference through the SageMaker API\n", "pre_trained_predictor = pre_trained_model.deploy(\n", " initial_instance_count=1,\n", " instance_type=inference_instance_type,\n", " predictor_cls=Predictor,\n", " endpoint_name=pre_trained_name,\n", ")\n", "print(f\"{newline}Deployed an endpoint {pre_trained_name}\")" ] }, { "cell_type": "code", "execution_count": 17, "id": "07cab2be-f433-41c3-bee5-e8e0d61a7a91", "metadata": { "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "INFO:sagemaker:Creating model with name: jumpstart-demo-fine-tuned-huggingface-t-2023-05-29-03-42-28-773\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\u001b[1mimage URI:\u001b[0m\n", " 763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-inference:1.10.2-transformers4.17.0-gpu-py38-cu113-ubuntu20.04\n", "\u001b[1mmodel URI:\u001b[0m\n", " s3://sagemaker-us-east-1-509957658284/demo-fine-tune-flan-t5/js-demo-flan-t5-small-2-2023-05-29-03-24-42-284/output/model.tar.gz\n", "Deploying an endpoint ...\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:sagemaker:Creating endpoint-config with name jumpstart-demo-fine-tuned-huggingface-t-2023-05-29-03-42-28-773\n", "INFO:sagemaker:Creating endpoint with name jumpstart-demo-fine-tuned-huggingface-t-2023-05-29-03-42-28-773\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "-------!\n", "Deployed an endpoint jumpstart-demo-fine-tuned-huggingface-t-2023-05-29-03-42-28-773\n" ] } ], "source": [ "from sagemaker.model import Model\n", "from sagemaker.predictor import Predictor\n", "from sagemaker.utils import name_from_base\n", "\n", "fine_tuned_name = name_from_base(f\"jumpstart-demo-fine-tuned-{model_id}\")\n", "fine_tuned_model_uri = f\"{output_location}{training_job_name}/output/model.tar.gz\"\n", "\n", "# Create the SageMaker model instance of the fine-tuned model\n", "fine_tuned_model = Model(\n", " image_uri=deploy_image_uri,\n", " model_data=fine_tuned_model_uri,\n", " role=aws_role,\n", " predictor_cls=Predictor,\n", " name=fine_tuned_name,\n", ")\n", "\n", "print(f\"{bold}image URI:{unbold}{newline} {deploy_image_uri}\")\n", "print(f\"{bold}model URI:{unbold}{newline} {fine_tuned_model_uri}\")\n", "print(\"Deploying an endpoint ...\")\n", "\n", "# Deploy the fine-tuned model.\n", "fine_tuned_predictor = fine_tuned_model.deploy(\n", " initial_instance_count=1,\n", " instance_type=inference_instance_type,\n", " predictor_cls=Predictor,\n", " endpoint_name=fine_tuned_name,\n", ")\n", "print(f\"{newline}Deployed an endpoint {fine_tuned_name}\")" ] }, { "cell_type": "markdown", "id": "58114831-c08e-4542-bc5e-852826785356", "metadata": {}, "source": [ "### 4. Running inference queries" ] }, { "cell_type": "markdown", "id": "d5ebf33c-dc6f-41a9-b137-1e2ffaa8de2e", "metadata": {}, "source": [ "As the name suggests, a Text2Text model such as FLAN T5 receives a piece of text as input, and generates text as output. The input text will contain the description of the task. In this demo, our task is to generate questions given a piece of text. The questions must be relevant to the text, but the text should contain no answer. Such a task could arise when automating gathering additional information, or identifying gaps in technical documentation." ] }, { "cell_type": "code", "execution_count": 18, "id": "2598072e-fc52-40c0-8343-1a74e35cfa3a", "metadata": { "tags": [] }, "outputs": [], "source": [ "prompt = \"Ask a question which is related to the following text, but cannot be answered based on the text. Text: {context}\"\n", "\n", "# Sources: Wikipedia, AWS Documentation\n", "test_paragraphs = [\n", " \"\"\"\n", "Adelaide is the capital city of South Australia, the state's largest city and the fifth-most populous city in Australia. \"Adelaide\" may refer to either Greater Adelaide (including the Adelaide Hills) or the Adelaide city centre. The demonym Adelaidean is used to denote the city and the residents of Adelaide. The Traditional Owners of the Adelaide region are the Kaurna people. The area of the city centre and surrounding parklands is called Tarndanya in the Kaurna language.\n", "Adelaide is situated on the Adelaide Plains north of the Fleurieu Peninsula, between the Gulf St Vincent in the west and the Mount Lofty Ranges in the east. Its metropolitan area extends 20 km (12 mi) from the coast to the foothills of the Mount Lofty Ranges, and stretches 96 km (60 mi) from Gawler in the north to Sellicks Beach in the south.\n", "\"\"\",\n", " \"\"\"\n", "Amazon Elastic Block Store (Amazon EBS) provides block level storage volumes for use with EC2 instances. EBS volumes behave like raw, unformatted block devices. You can mount these volumes as devices on your instances. EBS volumes that are attached to an instance are exposed as storage volumes that persist independently from the life of the instance. You can create a file system on top of these volumes, or use them in any way you would use a block device (such as a hard drive). You can dynamically change the configuration of a volume attached to an instance.\n", "We recommend Amazon EBS for data that must be quickly accessible and requires long-term persistence. EBS volumes are particularly well-suited for use as the primary storage for file systems, databases, or for any applications that require fine granular updates and access to raw, unformatted, block-level storage. Amazon EBS is well suited to both database-style applications that rely on random reads and writes, and to throughput-intensive applications that perform long, continuous reads and writes.\n", "\"\"\",\n", " \"\"\"\n", "Amazon Comprehend uses natural language processing (NLP) to extract insights about the content of documents. It develops insights by recognizing the entities, key phrases, language, sentiments, and other common elements in a document. Use Amazon Comprehend to create new products based on understanding the structure of documents. For example, using Amazon Comprehend you can search social networking feeds for mentions of products or scan an entire document repository for key phrases. \n", "You can access Amazon Comprehend document analysis capabilities using the Amazon Comprehend console or using the Amazon Comprehend APIs. You can run real-time analysis for small workloads or you can start asynchronous analysis jobs for large document sets. You can use the pre-trained models that Amazon Comprehend provides, or you can train your own custom models for classification and entity recognition. \n", "All of the Amazon Comprehend features accept UTF-8 text documents as the input. In addition, custom classification and custom entity recognition accept image files, PDF files, and Word files as input. \n", "Amazon Comprehend can examine and analyze documents in a variety of languages, depending on the specific feature. For more information, see Languages supported in Amazon Comprehend. Amazon Comprehend's Dominant language capability can examine documents and determine the dominant language for a far wider selection of languages.\n", "\"\"\",\n", "]" ] }, { "cell_type": "code", "execution_count": 19, "id": "bb2f1f73-4ba9-422a-adf1-d94a686c2b70", "metadata": { "tags": [] }, "outputs": [], "source": [ "import boto3\n", "import json\n", "\n", "# Parameters of (output) text generation. A great introduction to generation\n", "# parameters can be found at https://huggingface.co/blog/how-to-generate\n", "parameters = {\n", " \"max_length\": 40, # restrict the length of the generated text\n", " \"num_return_sequences\": 5, # we will inspect several model outputs\n", " \"num_beams\": 10, # use beam search\n", "}\n", "\n", "\n", "# Helper functions for running inference queries\n", "def query_endpoint_with_json_payload(payload, endpoint_name):\n", " encoded_json = json.dumps(payload).encode(\"utf-8\")\n", " client = boto3.client(\"runtime.sagemaker\")\n", " response = client.invoke_endpoint(\n", " EndpointName=endpoint_name, ContentType=\"application/json\", Body=encoded_json\n", " )\n", " return response\n", "\n", "\n", "def parse_response_multiple_texts(query_response):\n", " model_predictions = json.loads(query_response[\"Body\"].read())\n", " generated_text = model_predictions[\"generated_texts\"]\n", " return generated_text\n", "\n", "\n", "def generate_questions(endpoint_name, text):\n", " expanded_prompt = prompt.replace(\"{context}\", text)\n", " payload = {\"text_inputs\": expanded_prompt, **parameters}\n", " query_response = query_endpoint_with_json_payload(payload, endpoint_name=endpoint_name)\n", " generated_texts = parse_response_multiple_texts(query_response)\n", " for i, generated_text in enumerate(generated_texts):\n", " print(f\"Response {i}: {generated_text}{newline}\")" ] }, { "cell_type": "code", "execution_count": 20, "id": "42d04ea2-82a0-446a-b64d-ab68442081a9", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[1mPrompt:\u001b[0m 'Ask a question which is related to the following text, but cannot be answered based on the text. Text: {context}'\n", "--------------------------------------------------------------------------------\n", "\n", "Adelaide is the capital city of South Australia, the state's largest city and the fifth-most populous city in Australia. \"Adelaide\" may refer to either Greater Adelaide (including the Adelaide Hills) or the Adelaide city centre. The demonym Adelaidean is used to denote the city and the residents of Adelaide. The Traditional Owners of the Adelaide region are the Kaurna people. The area of the city centre and surrounding parklands is called Tarndanya in the Kaurna language.\n", "Adelaide is situated on the Adelaide Plains north of the Fleurieu Peninsula, between the Gulf St Vincent in the west and the Mount Lofty Ranges in the east. Its metropolitan area extends 20 km (12 mi) from the coast to the foothills of the Mount Lofty Ranges, and stretches 96 km (60 mi) from Gawler in the north to Sellicks Beach in the south.\n", "\n", "--------------------------------------------------------------------------------\n", "\u001b[1mpre-trained\u001b[0m\n", "Response 0: Where is Adelaide located?\n", "\n", "Response 1: What is the name of Adelaide's metropolitan area?\n", "\n", "Response 2: What is the name of the capital city of South Australia?\n", "\n", "Response 3: What is the name of the city Adelaide is located in?\n", "\n", "Response 4: What is the capital city of Adelaide?\n", "\n", "\u001b[1mfine-tuned\u001b[0m\n", "Response 0: Who is the capital city of South Australia, the state's largest city and the fifth-most populous city in Australia?\n", "\n", "Response 1: Adelaide is the capital city of South Australia, the state's largest city and the fifth-most populous city in Australia?\n", "\n", "Response 2: Adelaide is the capital city of South Australia, which is the state's largest city and the fifth-most populous city in Australia?\n", "\n", "Response 3: Adelaide is the capital city of South Australia, the state's largest city and what is the fifth-most populous city in Australia?\n", "\n", "Response 4: Where is Adelaide the capital city of South Australia, the state's largest city and the fifth-most populous city in Australia?\n", "\n", "--------------------------------------------------------------------------------\n", "\n", "Amazon Elastic Block Store (Amazon EBS) provides block level storage volumes for use with EC2 instances. EBS volumes behave like raw, unformatted block devices. You can mount these volumes as devices on your instances. EBS volumes that are attached to an instance are exposed as storage volumes that persist independently from the life of the instance. You can create a file system on top of these volumes, or use them in any way you would use a block device (such as a hard drive). You can dynamically change the configuration of a volume attached to an instance.\n", "We recommend Amazon EBS for data that must be quickly accessible and requires long-term persistence. EBS volumes are particularly well-suited for use as the primary storage for file systems, databases, or for any applications that require fine granular updates and access to raw, unformatted, block-level storage. Amazon EBS is well suited to both database-style applications that rely on random reads and writes, and to throughput-intensive applications that perform long, continuous reads and writes.\n", "\n", "--------------------------------------------------------------------------------\n", "\u001b[1mpre-trained\u001b[0m\n", "Response 0: How does Amazon Elastic Block Store provide block level storage volumes for use with EC2 instances?\n", "\n", "Response 1: What is the name of Amazon Elastic Block Store?\n", "\n", "Response 2: How does Amazon Elastic Block Store work?\n", "\n", "Response 3: How does Amazon Elastic Block Store provide block level storage volumes for EC2 instances?\n", "\n", "Response 4: How does Amazon Elastic Block Store provide block level storage volumes for use with EC2 instance?\n", "\n", "\u001b[1mfine-tuned\u001b[0m\n", "Response 0: Amazon Elastic Block Store (Amazon EBS) provides block level storage volumes for use with EC2 instances?\n", "\n", "Response 1: Amazon Elastic Block Store provides block level storage volumes for use with EC2 instances?\n", "\n", "Response 2: EBS volumes that are attached to an instance are exposed as storage volumes that persist independently from the life of the instance?\n", "\n", "Response 3: EBS volumes that are attached to an instance are exposed as storage volumes that persist independently from the life of what?\n", "\n", "Response 4: Amazon Elastic Block Store (Amazon EBS) provides block level storage volumes for what?\n", "\n", "--------------------------------------------------------------------------------\n", "\n", "Amazon Comprehend uses natural language processing (NLP) to extract insights about the content of documents. It develops insights by recognizing the entities, key phrases, language, sentiments, and other common elements in a document. Use Amazon Comprehend to create new products based on understanding the structure of documents. For example, using Amazon Comprehend you can search social networking feeds for mentions of products or scan an entire document repository for key phrases. \n", "You can access Amazon Comprehend document analysis capabilities using the Amazon Comprehend console or using the Amazon Comprehend APIs. You can run real-time analysis for small workloads or you can start asynchronous analysis jobs for large document sets. You can use the pre-trained models that Amazon Comprehend provides, or you can train your own custom models for classification and entity recognition. \n", "All of the Amazon Comprehend features accept UTF-8 text documents as the input. In addition, custom classification and custom entity recognition accept image files, PDF files, and Word files as input. \n", "Amazon Comprehend can examine and analyze documents in a variety of languages, depending on the specific feature. For more information, see Languages supported in Amazon Comprehend. Amazon Comprehend's Dominant language capability can examine documents and determine the dominant language for a far wider selection of languages.\n", "\n", "--------------------------------------------------------------------------------\n", "\u001b[1mpre-trained\u001b[0m\n", "Response 0: What is Amazon Comprehend's dominant language capability?\n", "\n", "Response 1: What is Amazon Comprehend's ability to analyze documents in a variety of languages?\n", "\n", "Response 2: What is Amazon Comprehend's ability to analyze documents?\n", "\n", "Response 3: How does Amazon Comprehend work?\n", "\n", "Response 4: What is Amazon Comprehend's Dominant language capability?\n", "\n", "\u001b[1mfine-tuned\u001b[0m\n", "Response 0: What uses Amazon Comprehend to extract insights about the content of documents?\n", "\n", "Response 1: Amazon Comprehend uses natural language processing (NLP) to extract insights about the content of what?\n", "\n", "Response 2: What does Amazon Comprehend use to extract insights about the content of documents?\n", "\n", "Response 3: What uses natural language processing to extract insights about the content of documents?\n", "\n", "Response 4: Amazon Comprehend uses what to extract insights about the content of documents?\n", "\n" ] } ], "source": [ "print(f\"{bold}Prompt:{unbold} {repr(prompt)}\")\n", "for paragraph in test_paragraphs:\n", " print(\"-\" * 80)\n", " print(paragraph)\n", " print(\"-\" * 80)\n", " print(f\"{bold}pre-trained{unbold}\")\n", " generate_questions(pre_trained_name, paragraph)\n", " print(f\"{bold}fine-tuned{unbold}\")\n", " generate_questions(fine_tuned_name, paragraph)" ] }, { "cell_type": "markdown", "id": "de13af34-3364-4ca0-ad89-929996642a80", "metadata": {}, "source": [ "The pre-trained model was not specifically trained to generate unanswerable questions. Despite the input prompt, it tends to generate questions that can be answered from the text. The fine-tuned model is generally better at this task, and the improvement is more prominent for larger models (e.g., xl rather than base)." ] }, { "cell_type": "markdown", "id": "8beafe5b-07d9-4095-9f4e-7f75b4104c95", "metadata": {}, "source": [ "### 5. Cleaning up resources" ] }, { "cell_type": "code", "execution_count": 21, "id": "99076e38-16db-4b77-9545-d21796af4ab9", "metadata": { "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "INFO:sagemaker:Deleting model with name: jumpstart-demo-pre-trained-huggingface--2023-05-29-03-37-21-524\n", "INFO:sagemaker:Deleting endpoint configuration with name: jumpstart-demo-pre-trained-huggingface--2023-05-29-03-37-21-524\n", "INFO:sagemaker:Deleting endpoint with name: jumpstart-demo-pre-trained-huggingface--2023-05-29-03-37-21-524\n", "INFO:sagemaker:Deleting model with name: jumpstart-demo-fine-tuned-huggingface-t-2023-05-29-03-42-28-773\n", "INFO:sagemaker:Deleting endpoint configuration with name: jumpstart-demo-fine-tuned-huggingface-t-2023-05-29-03-42-28-773\n", "INFO:sagemaker:Deleting endpoint with name: jumpstart-demo-fine-tuned-huggingface-t-2023-05-29-03-42-28-773\n" ] } ], "source": [ "# Delete resources\n", "pre_trained_predictor.delete_model()\n", "pre_trained_predictor.delete_endpoint()\n", "fine_tuned_predictor.delete_model()\n", "fine_tuned_predictor.delete_endpoint()" ] }, { "cell_type": "code", "execution_count": null, "id": "5d3ff128-2a21-4681-9dc3-3e34573c770a", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "availableInstances": [ { "_defaultOrder": 0, "_isFastLaunch": true, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 4, "name": "ml.t3.medium", "vcpuNum": 2 }, { "_defaultOrder": 1, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.t3.large", "vcpuNum": 2 }, { "_defaultOrder": 2, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.t3.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 3, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.t3.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 4, "_isFastLaunch": true, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.m5.large", "vcpuNum": 2 }, { "_defaultOrder": 5, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.m5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 6, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.m5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 7, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.m5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 8, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.m5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 9, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.m5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 10, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.m5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 11, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.m5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 12, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.m5d.large", "vcpuNum": 2 }, { "_defaultOrder": 13, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.m5d.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 14, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.m5d.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 15, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.m5d.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 16, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.m5d.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 17, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.m5d.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 18, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.m5d.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 19, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.m5d.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 20, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": true, "memoryGiB": 0, "name": "ml.geospatial.interactive", "supportedImageNames": [ "sagemaker-geospatial-v1-0" ], "vcpuNum": 0 }, { "_defaultOrder": 21, "_isFastLaunch": true, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 4, "name": "ml.c5.large", "vcpuNum": 2 }, { "_defaultOrder": 22, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.c5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 23, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.c5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 24, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.c5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 25, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 72, "name": "ml.c5.9xlarge", "vcpuNum": 36 }, { "_defaultOrder": 26, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 96, "name": "ml.c5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 27, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 144, "name": "ml.c5.18xlarge", "vcpuNum": 72 }, { "_defaultOrder": 28, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.c5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 29, "_isFastLaunch": true, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.g4dn.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 30, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.g4dn.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 31, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.g4dn.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 32, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.g4dn.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 33, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.g4dn.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 34, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.g4dn.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 35, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 61, "name": "ml.p3.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 36, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 244, "name": "ml.p3.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 37, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 488, "name": "ml.p3.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 38, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 768, "name": "ml.p3dn.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 39, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.r5.large", "vcpuNum": 2 }, { "_defaultOrder": 40, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.r5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 41, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.r5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 42, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.r5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 43, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.r5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 44, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.r5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 45, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 512, "name": "ml.r5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 46, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 768, "name": "ml.r5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 47, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.g5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 48, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.g5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 49, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.g5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 50, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.g5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 51, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.g5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 52, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.g5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 53, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.g5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 54, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 768, "name": "ml.g5.48xlarge", "vcpuNum": 192 }, { "_defaultOrder": 55, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 1152, "name": "ml.p4d.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 56, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 1152, "name": "ml.p4de.24xlarge", "vcpuNum": 96 } ], "instance_type": "ml.t3.medium", "kernelspec": { "display_name": "Python 3 (Data Science)", "language": "python", "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/datascience-1.0" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.10" } }, "nbformat": 4, "nbformat_minor": 5 }