{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"id": "1975e354-ccfa-485f-aa92-71c49570a904",
"metadata": {
"tags": []
},
"source": [
"# Lab 2\n",
"\n",
"## Introduction\n",
"\n",
"Language models have recently exploded in both size and popularity. In 2018, BERT-large entered the scene and, with its 340M parameters and novel transformer architecture, set the standard on NLP task accuracy. Within just a few years, state-of-the-art NLP model size has grown by more than 500x with models such as OpenAI’s 175 billion parameter GPT-3 and similarly sized open source Bloom 176B raising the bar on NLP accuracy. This increase in the number of parameters is driven by the simple and empirically-demonstrated positive relationship between model size and accuracy: more is better. With easy access from models zoos such as HuggingFace and improved accuracy in NLP tasks such as classification and text generation, practitioners are increasingly reaching for these large models. These models can be used in pretrained form as so called \"Foundation Models\". \n",
"\n",
"However, since they are trained on large datasets of generic data they are often not suited for use cases requiring domain-specific knowledge. Especially the models of large parameter size are usually able to generalize well and perform surprisingly good in various zero-shot/few-shot scenarios. Nevertheless, zero-shot/few-shot performance in complex tasks like question-answering or handling human-like conversation is decreasing rapidly the more specifik the tasks become. \n",
"\n",
"In these cases, fine-tuning a model on a smaller, domain-specific use case can help increase the performance to a satisfying level. However, training/finetuning these models can be a challenge because of their size.\n",
"\n",
"In this Lab, we'll explore how to finetune a large language model on Amazon SageMaker using Sagemaker Training, one of many ready-to-use AWS Deep Learning Containers (DLCs) and the built-in HuggingFace integration of the Sagemaker SDK. \n",
"\n",
"## Background and Details\n",
"Since training such models requires even more resources than hosting them, for this lab we'll be working with a rather small Large Language Model (LLM) to learn the basic concepts of finetuning LLMs . However, the proposed approach works similarily at scale for larger models. \n",
"\n",
"'distilGPT', a transformer-based large language model with around 82M parameters is the distilled version of GPT2 (predecessor of GPT-3/4), which was pre-trained on the WebText dataset. Since it is a decoder-only model it was trained using a causal language modeling (CLM) loss. We will use the exact approach for finetuning the data on the 'tiny_shakespeare' dataset to adjust the model output in terms of writing-style and content of the generated text. \n",
"\n",
"The 'tiny_shakespeare' is a dataset consisting of 40000 lines of Shakespeare from a variety of Shakespeare's plays avaliable in a train/test/validation split. It can be retrieved conveniently from the HuggingFace datasets hub.\n",
"\n",
"Finally, we will deploy both the original and the finetuned model to experience the impact of the performed training.\n",
"\n",
"\n",
"## Instructions\n",
"\n",
"### Prerequisites\n",
"\n",
"#### To run this workshop...\n",
"You need a computer with a web browser, preferably with the latest version of Chrome / FireFox.\n",
"Sequentially read and follow the instructions described in AWS Hosted Event and Work Environment Set Up\n",
"\n",
"#### Recommended background\n",
"It will be easier for you to run this workshop if you have:\n",
"\n",
"- Experience with Deep learning models\n",
"- Familiarity with Python or other similar programming languages\n",
"- Experience with Jupyter notebooks\n",
"- Begineers level knowledge and experience with SageMaker Hosting/Inference.\n",
"\n",
"#### Target audience\n",
"Data Scientists, ML Engineering, ML Infrastructure, MLOps Engineers, Technical Leaders.\n",
"Intended for customers working with large Generative AI models including Language, Computer vision and Multi-modal use-cases.\n",
"Customers using EKS/EC2/ECS/On-prem for hosting or experience with SageMaker.\n",
"\n",
"Level of expertise - 400\n",
"\n",
"#### Time to complete\n",
"Approximately 1 hour."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "61bc66e0-d96f-4bb9-8f54-430a3533882f",
"metadata": {},
"source": [
"# Import of required dependencies\n",
"\n",
"For this lab, we will use the following libraries:\n",
"\n",
" - boto3, the AWS SDK for python\n",
" - SageMaker SDK for interacting with Amazon SageMaker. We especially want to highlight the classes 'HuggingFaceModel' and 'HuggingFace', utilizing the built-in HuggingFace integration into SageMaker SDK. These classes are used to encapsulate functionality around the model and the deployed endpoint we will use. They inherit from the generic 'Model' and 'Estimator' classes of the native SageMaker SDK, however implementing some additional functionality specific to HuggingFace and the HuggingFace model hub.\n",
" - os, a python library implementing miscellaneous operating system interfaces \n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "54e705f7-8ff0-405f-8515-4d6486bf8279",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"import boto3\n",
"import sagemaker\n",
"import sagemaker.session\n",
"import os\n",
"\n",
"from sagemaker.huggingface import HuggingFace, HuggingFaceModel"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "ab767806-52a7-499d-90db-cf13ee85e9cb",
"metadata": {},
"source": [
"# Setup of notebook environment\n",
"\n",
"Before we begin with the actual work for finetuning and deploying the model to Amazon SageMaker, we need to setup the notebook environment respectively. This includes:\n",
"\n",
"- retrieval of the execution role our SageMaker Studio domain is associated with for later usage\n",
"- retrieval of our account_id for later usage\n",
"- retrieval of the chosen region for later usage"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e35d59f7-39c1-477a-b6c7-e4077226713a",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"# Retrieve SM execution role\n",
"role = sagemaker.get_execution_role()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "adfeccf0-0729-464f-a56a-f276119bb32f",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"# Create a new STS client\n",
"sts_client = boto3.client('sts')\n",
"\n",
"# Call the GetCallerIdentity operation to retrieve the account ID\n",
"response = sts_client.get_caller_identity()\n",
"account_id = response['Account']\n",
"account_id"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7465b465",
"metadata": {},
"outputs": [],
"source": [
"# Retrieve region\n",
"region = boto3.Session().region_name\n",
"region"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "1f320544-4a5a-4734-8386-63c419eb3ad4",
"metadata": {},
"source": [
"# Setup of S3 bucket for storage of training artifacts\n",
"When training a model with AWS SageMaker Training several artifacts can be written to an S3 bucket. This includes the trained model in form of a ‘model.tar.gz’ but also other artifacts like log files and the source code base. For this purpose, (if not already present) we create a dedicated S3 bucket."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "35114d1a-a340-4615-bd2f-39146e01efcd",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"# specifying bucket name for model artifact storage\n",
"model_bucket_name = f'immersion-day-bucket-{account_id}'\n",
"model_bucket_name"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2b7907ee-e233-4dda-9aae-4ed7cc5b6337",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"# Create S3 bucket\n",
"s3_client = boto3.client('s3', region_name=region)\n",
"location = {'LocationConstraint': region}\n",
"\n",
"bucket_name = model_bucket_name\n",
"\n",
"# Check if bucket already exists\n",
"bucket_exists = True\n",
"try:\n",
" s3_client.head_bucket(Bucket=bucket_name)\n",
"except:\n",
" bucket_exists = False\n",
"\n",
"# Create bucket if it does not exist\n",
"if not bucket_exists:\n",
" if region == 'us-east-1':\n",
" s3_client.create_bucket(Bucket=bucket_name)\n",
" else: \n",
" s3_client.create_bucket(Bucket=bucket_name,\n",
" CreateBucketConfiguration=location)\n",
" print(f\"Bucket '{bucket_name}' created successfully\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "006fdef7-e740-4910-9a24-1645914d0dd0",
"metadata": {
"tags": []
},
"source": [
"\n",
"# Diving deep into the training code\n",
"\n",
"The code artifacts required for finetuning are residing in the finetuning directory. This directory is composed as follows:\n",
"\n",
"`finetuning/`\n",
"- `finetuning.py`\n",
"- `requirements.txt`\n",
"\n",
"The \"finetuning\" directory contains your training script (finetuning.py) and your requirements.txt file (for installation of additional dependencies not preinstalled in the container image upon start of the training container). We will now take a closer look into the training code:\n",
"\n",
"## Import of required dependencies\n",
"\n",
"On top of several commodity Python libraries, for this training script we will use the following DL specific libraries:\n",
" \n",
"- torch: PyTorch is a Python package that provides two high-level features: 1/Tensor computation (like NumPy) with strong GPU acceleration and 2/Deep neural networks built on a tape-based autograd system\n",
"- transformers: HuggingFace transformers provides APIs and tools to easily download and train state-of-the-art pretrained models. Transformers support framework interoperability between PyTorch, TensorFlow, and JAX. \n",
"- evaluate: HuggingFace evaluate is a library for easily evaluating machine learning models and datasets.\n",
"- datasets: HuggingFace datasets is a library for easily accessing and sharing datasets for Audio, Computer Vision, and Natural Language Processing (NLP) tasks in the context of the HuggingFace dataset hub.\n",
"\n",
"## Script invocation and hyperparameter parsing\n",
"\n",
"After the ephemeral training cluster has been provisioned and the respective Docker image has been pulled onto the machines, SageMaker Training starts the container which invokes the training python script 'finetuning.py' as entrypoint. Thereby it passes the defined hyperparameters as command line arguments. We will dive deeper into our hyperparameter selection at a later point.\n",
"\n",
"The hyperparameters can be parsed by an 'argpars' ArgumentParser:\n",
"\n",
"```python\n",
"parser = argparse.ArgumentParser()\n",
" \n",
"# Training parameters\n",
"parser.add_argument(\"--model_name_or_path\", default=\"distilgpt2\")\n",
"\n",
"args = parser.parse_args()\n",
"\n",
"```\n",
"\n",
"## Logging\n",
"\n",
"For logging we use the 'logging' library. We first setup the basic config: \n",
"\n",
"```python\n",
" # Setup logging\n",
"logging.basicConfig(\n",
" format=\"%(asctime)s - %(levelname)s - %(name)s - %(message)s\",\n",
" datefmt=\"%m/%d/%Y %H:%M:%S\",\n",
" handlers=[logging.StreamHandler(sys.stdout)],\n",
")\n",
"\n",
"```\n",
"\n",
" Then we set the log level to 'INFO':\n",
" \n",
"```python\n",
"log_level = logging.INFO\n",
"logger.setLevel(log_level)\n",
"\n",
"```\n",
"\n",
"Finally we configure logging for the HuggingFace frameworks 'datasets' and 'transformers'.\n",
"\n",
"```python\n",
"datasets.utils.logging.set_verbosity(log_level)\n",
"transformers.utils.logging.set_verbosity(log_level)\n",
"transformers.utils.logging.enable_default_handler()\n",
"transformers.utils.logging.enable_explicit_format()\n",
"\n",
"```\n",
"\n",
"## Loading the dataset\n",
"\n",
"We then use the 'datasets' library to load our dataset from the HuggingFace dataset hub:\n",
"\n",
"```python\n",
"# Downloading and loading a dataset from the hub.\n",
"raw_datasets = load_dataset(args.dataset_name)\n",
"```\n",
"\n",
"In case the training script we are loading is not available with a train/test split, there is additional functionality implemented to achieve this.\n",
"\n",
"In a real world scenario training data could be ingested from various data sources like S3, databases, ... . \n",
"\n",
"\n",
"## Preprocessing\n",
"\n",
"Since we want to finetune the model on a CLM task we need to consider both NLP-related steps and CLM-related steps when it comes to data preprocessing:\n",
"\n",
"For training NLP models the full text string has to be tokenized to enable the model to \"digest\" it as an input. Beyond access to a huge amount of open-source NLP models, the HuggingFace model hub offers also compatible tokenizers. By utilizing the 'transformers' library we are downloading a tokenizer for the revision of 'distilGPT2' which will be finetuned later on:\n",
"\n",
"```python\n",
"tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path, use_fast = True, revision = args.model_revision)\n",
"\n",
"```\n",
"\n",
"In a similar fashion we are also loading the model artifacts from the HuggingFace model hub:\n",
"\n",
"```python\n",
"model = AutoModelForCausalLM.from_pretrained(args.model_name_or_path, revision=args.model_revision, torch_dtype=\"auto\")\n",
"\n",
"```\n",
"\n",
"The tokenizer is now wrapped into an object of the 'AutoTokenizer' class, while the model resides in an object of the 'AutoModelForCausalLM' class. \n",
"\n",
"### Tokenization\n",
"\n",
"For tokenization we define a function taking care of the actual tokenization task:\n",
"\n",
"```python\n",
"def tokenize_function(examples):\n",
" \n",
" ... \n",
" \n",
" output = tokenizer(examples[text_column_name])\n",
" \n",
" ...\n",
" \n",
" return output\n",
"\n",
"```\n",
"\n",
"\n",
" \n",
" \n",
"Then we utilize it as a higher order function in a map approach on the dataset:\n",
"\n",
"```python\n",
"tokenized_datasets = raw_datasets.map(\n",
" tokenize_function,\n",
" batched=True,\n",
" remove_columns=column_names,\n",
" desc=\"Running tokenizer on dataset\"\n",
")\n",
"\n",
"```\n",
"\n",
"### CLM-related tasks\n",
"\n",
"The utilized training task consumes token blocks of 'block_size' (number of token a model is consuming in one forward pass. This is model specific plus bound to the instance type used for training.) and trains the model using a CLM loss (for details read [this](https://huggingface.co/docs/transformers/tasks/language_modeling#causal-language-modeling)). Therefore we need to group our tokenized dataset into token blocks of 'block_size'. We again define a function that performs the acutual grouping task:\n",
"\n",
"# Main data processing function that will concatenate all texts from our dataset and generate chunks of block_size.\n",
"\n",
"```python\n",
"def group_texts(examples):\n",
" # Concatenate all texts.\n",
" concatenated_examples = {k: list(chain(*examples[k])) for k in examples.keys()}\n",
" total_length = len(concatenated_examples[list(examples.keys())[0]])\n",
" # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can\n",
" # customize this part to your needs.\n",
" if total_length >= block_size:\n",
" total_length = (total_length // block_size) * block_size\n",
" # Split by chunks of max_len.\n",
" result = {\n",
" k: [t[i : i + block_size] for i in range(0, total_length, block_size)]\n",
" for k, t in concatenated_examples.items()\n",
" }\n",
" result[\"labels\"] = result[\"input_ids\"].copy()\n",
" return result\n",
"\n",
"```\n",
" \n",
" \n",
"Then we utilize it as a higher order function in a map approach on the tokenized dataset:\n",
"\n",
"```python\n",
"lm_datasets = tokenized_datasets.map(\n",
" group_texts,\n",
" batched=True,\n",
" desc=f\"Grouping texts in chunks of {block_size}\",\n",
")\n",
"\n",
"```\n",
"\n",
"\n",
"## Training\n",
"\n",
"Since we specified that we want to run evaluations on the model to be finetuned (both stepwise during and after the training process), we need to define our evaluation metric first. Therefore we load one of various pre-implemented metrics available using HuggingFace's 'evaluate' library:\n",
"\n",
"```python\n",
"metric = evaluate.load(\"accuracy\")\n",
"\n",
"```\n",
"\n",
"\n",
"Then we define a function computing the actual metrics tied to our training job:\n",
"\n",
"```python\n",
"def compute_metrics(eval_preds):\n",
" preds, labels = eval_preds\n",
" # preds have the same shape as the labels, after the argmax(-1) has been calculated\n",
" # by preprocess_logits_for_metrics but we need to shift the labels\n",
" labels = labels[:, 1:].reshape(-1)\n",
" preds = preds[:, :-1].reshape(-1)\n",
" return metric.compute(predictions=preds, references=labels)\n",
" \n",
"```\n",
"\n",
"The next step is configuring the actual training job. Therefore we first initialize a 'TrainingArguments' object fed with our hyperparamters plus a seed.:\n",
"\n",
"\n",
"```python\n",
"# Specifying training_args. Going with default values for every parameter not explicitly specified. See documentation for more information: https://huggingface.co/docs/transformers/v4.27.2/en/main_classes/trainer#transformers.TrainingArguments \n",
"training_args = TrainingArguments(\n",
" per_device_train_batch_size = int(args.per_device_train_batch_size), \n",
" per_device_eval_batch_size=int(args.per_device_eval_batch_size), \n",
" output_dir=args.output_dir, \n",
" seed=42, \n",
" disable_tqdm=False\n",
")\n",
" \n",
"```\n",
" \n",
"Then we initialize the Trainer object, which will orchestrate the training and evaluation process holistically. Several artifacts defined in the flow we executed so far are passed as parameters (model, training_args, datasets, compute_metrics function):\n",
"\n",
"```python\n",
"# Initialize our Trainer\n",
"trainer = Trainer(\n",
" model=model,\n",
" args=training_args,\n",
" train_dataset=train_dataset if args.do_train else None,\n",
" eval_dataset=eval_dataset if args.do_eval else None,\n",
" tokenizer=tokenizer,\n",
" # Data collator will default to DataCollatorWithPadding, so we change it.\n",
" data_collator=default_data_collator,\n",
" compute_metrics=compute_metrics if args.do_eval and not is_torch_tpu_available() else None,\n",
" preprocess_logits_for_metrics=preprocess_logits_for_metrics\n",
" if args.do_eval and not is_torch_tpu_available()\n",
" else None,\n",
")\n",
"```\n",
"\n",
"Finally the Trainer's .train() function is invoked executing the actual training. After successful completion the model artifacts are persisted according to the 'output_path' configuration.\n",
" \n",
"```python\n",
"train_result = trainer.train()\n",
"trainer.save_model() # Saves the tokenizer too for easy upload\n",
"\n",
"```\n",
" \n",
"## Evaluation\n",
"\n",
"After successful completion of the training run, we perform a final evaluation: \n",
"\n",
"```python\n",
"metrics = trainer.evaluate()\n",
"\n",
"```"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "50ad9741-ee3f-4c4b-9f30-556241280e5e",
"metadata": {},
"source": [
"# Hyperparameters\n",
"\n",
"For the finetuning job to be conducted we specify the following hyperparameters explicitly:\n",
"- model_name_or_path: model id in HuggingFace ecosystem\n",
"- dataset_name: dataset id in HuggingFace ecosystem\n",
"- do_train: boolean variable indicating if training run should be executed. In our case 1.\n",
"- do_eval: boolean variable indicating if evaluation run should be executed. In our case 1.\n",
"- output_dir: directory path for storing the produced model artifacts locally within the container. We pick the default output directory of our SageMaker Training job (will be uploaded to S3 upon job success) '/opt/ml/model'.\n",
"- per_device_train_batch_size: batch size to be used when training. We choose 2.\n",
"- per_device_eval_batch_size: batch size to used when evaluating. We choose 2.\n",
"\n",
"The default values for the remaining configurable parameters can be found in the [Trainer](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Trainer) and [TrainingArguments](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Trainer) documentation."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "d910db15-418f-4b4c-9f5d-fc6af1593145",
"metadata": {},
"source": [
"# Configure the environment for model finetuning using the SageMaker HuggingFace Estimator and a AWS HuggingFace DLC\n",
"\n",
"For conveniently training a model with AWS SageMaker Training we can use the Estimator class of SageMaker. Thanks to the AWS x HuggingFace partnership we can use the HuggingFace Estimator natively integrated into the SageMaker SDK, implementing some additional functionality specific to HuggingFace and the HuggingFace model hub. This enables us to finetune the model by providing the training script and some configuration parameters only, while SageMaker is taking care of all the undifferentiated heavy lifting in the background for you. \n",
"In the constructor we specify the following parameters:\n",
"- source_dir: directory path to where the training script file is residing. In our case, this is the relative path to the 'finetuning' directory. Please note, that we've also created a 'requirements.txt' file for installing dependencies the training script requires on container-start time.\n",
"- entry_point: file in which the training script is implemented. Residing in the 'finetuning' directory, this is 'finetuning.py'. \n",
"- instance_type: EC2 instance type for executing the training job. We pick the 'ml.p3.2xlarge', an instance with 16 GB GPU acceleration (NVIDIA Tesla V100 GPU), 8 vCPUs and 61GB RAM.\n",
"- instance_count: size of the ephemeral training cluster. We pick a single node cluster.\n",
"- image_uri: The image uri of a Docker image used for training the model. We will be using on of the many ready-to-use Deep Learning Containers AWS is providing [here](https://aws.amazon.com/machine-learning/containers/). Deep Learning Containers are Docker images that are preinstalled and tested with the latest versions of popular deep learning frameworks. Deep Learning Containers let you train models in custom ML environments quickly without building and optimizing your environments from scratch. Since we will be training a model from the HuggingFace model hub by leveraging various HuggingFace frameworks, we will use one of the HuggingFace DLCs, coming with preinstalled python 3.8, pytorch 1.10.2, transformers 4.17.0 dependencies and optimized for training in GPU-accelerated environments. \n",
"- py_version: version of the python runtime installed in the container. This parameter is redundant, since we have explicitly specified a container image uri.\n",
"- hyperparameters: hyperparameters, passed as command line arguments to the training script. \n",
"- output_path: S3 path for storing the artifacts produced by the training job. Therefore, we use the S3 bucket we created in the beginning.\n",
"\n",
"Finally, we execute the SageMaker Training job by calling the .fit() function. This will take a couple of minutes. \n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "01a8ed23-805a-4c8d-8a6d-85536d00f76b",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"hyperparameters = {\n",
" \"model_name_or_path\": 'distilgpt2',\n",
" \"dataset_name\": 'tiny_shakespeare',\n",
" \"do_train\": 1, \n",
" \"do_eval\": 1, \n",
" \"output_dir\": '/opt/ml/model',\n",
" \"per_device_train_batch_size\": 2, \n",
" \"per_device_eval_batch_size\": 2,\n",
" }"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1c7949eb-4f3c-4c92-a419-32620752f5da",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"huggingface_estimator = HuggingFace(\n",
" source_dir='finetuning',\n",
" entry_point='finetuning.py',\n",
" instance_type='ml.p3.2xlarge',\n",
" instance_count=1,\n",
" role=role,\n",
" image_uri=f'763104351884.dkr.ecr.{region}.amazonaws.com/huggingface-pytorch-training:1.10.2-transformers4.17.0-gpu-py38-cu113-ubuntu20.04',\n",
" py_version=None,\n",
" hyperparameters = hyperparameters,\n",
" output_path = f's3://{model_bucket_name}'\n",
" )\n",
" "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0e93debc-900a-474b-8467-ac135182fa73",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"huggingface_estimator.fit()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "a60ffb91-b45e-4555-a872-8bd1e9b7367e",
"metadata": {},
"source": [
"# Model deployment\n",
"\n",
"Now we want to deploy both the original 'distilGPT2' model and our finetuned 'shakespeare-distilGPT2' model. Therefore we first retrieve the S3 path to the model artifact archive of our finetuned model:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "eb86c23d-8f09-4a7e-8c4c-8cc36478ce6a",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"latest_job_name = huggingface_estimator.latest_training_job.job_name\n",
"latest_job_name"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b38d0303-ce67-43d7-b791-fa46765b0b96",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"def get_s3_artifact_path(training_job_name):\n",
" # Get the ModelArtifacts object for the training job\n",
" sagemaker_session = sagemaker.Session()\n",
"\n",
" training_job = sagemaker_session.describe_training_job(training_job_name)\n",
"\n",
" model_artifacts = training_job['ModelArtifacts']\n",
"\n",
" # Retrieve the S3 path to the model artifact\n",
" s3_path = model_artifacts['S3ModelArtifacts']\n",
" return s3_path"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5f7448e5-b197-4200-96de-1087fb4f4621",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"s3_path = get_s3_artifact_path(latest_job_name)\n",
"s3_path"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "f6618c04-47b4-4622-9a1c-d86feeba1670",
"metadata": {},
"source": [
"Then we deploy the model to a 'ml.g4dn.xlarge' instance using the HuggingFaceModel class:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3fc0e9a9-5d85-4311-803d-e11bc1993336",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"# create Hugging Face Model Class\n",
"huggingface_model_finetuned = HuggingFaceModel(\n",
" image_uri=f'763104351884.dkr.ecr.{region}.amazonaws.com/huggingface-pytorch-inference:1.10.2-transformers4.17.0-gpu-py38-cu113-ubuntu20.04',\n",
" model_data=s3_path ,\n",
"\trole=role\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6de7915b-3dc7-4b15-8fbc-274dec2da47c",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"predictor_finetuned = huggingface_model_finetuned.deploy(\n",
" initial_instance_count=1, # number of instances\n",
" instance_type='ml.g4dn.xlarge', \n",
" endpoint_name='sm-endpoint-distilgpt2-shakespeare-immersion-day',\n",
")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "3a25abe7",
"metadata": {},
"source": [
"We also deploy the original model to a 'ml.g4dn.xlarge' instance. Therefore we use a cool feature built-in into the SageMaker SDK - we can define a model to be deployed directly from the HuggingFace model hub together with the model task to be performed directly as environment variables when creating a HuggingFaceModel, SageMaker Inference handles the rest:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "aa2b629b-f2dc-4e04-b8eb-b2a9980567b4",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"hub = {\n",
" 'HF_MODEL_ID':'distilgpt2', # model_id from hf.co/models\n",
" 'HF_TASK':'text-generation' # NLP task you want to use for predictions\n",
"}\n",
"\n",
"# create Hugging Face Model Class\n",
"huggingface_model_plain = HuggingFaceModel(\n",
" env=hub, # configuration for loading model from Hub\n",
" role=role, # IAM role with permissions to create an endpoint\n",
" image_uri=f'763104351884.dkr.ecr.{region}.amazonaws.com/huggingface-pytorch-inference:1.10.2-transformers4.17.0-gpu-py38-cu113-ubuntu20.04' # Python version used\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a9b9df69-2f8b-41ac-97d9-1658b15ef64a",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"predictor_plain = huggingface_model_plain.deploy(\n",
" initial_instance_count=1, # number of instances\n",
" instance_type='ml.g4dn.xlarge', \n",
" endpoint_name='sm-endpoint-distilgpt2-immersion-day',\n",
")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "57fb5f87-c532-41c9-ae17-ab713226a04f",
"metadata": {},
"source": [
"# Inference\n",
"\n",
"Having the two endpoints available, we can experiment and observe the impact the finetuning has in terms of performance of the text-generation task."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f0ef1354-fc99-4efb-bd27-207d393a8572",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"predictor_finetuned.predict({\"inputs\": \"The meaning of life\",\n",
"\"parameters\": {\n",
" \"min_length\": 50,\n",
" \"max_length\": 100\n",
"}})[0]['generated_text']"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0968db01-a88c-4057-b4ab-699ef5b6bfcc",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"predictor_plain.predict({\"inputs\": \"The meaning of life\",\n",
"\"parameters\": {\n",
" \"min_length\": 50,\n",
" \"max_length\": 100\n",
"}})[0]['generated_text']"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "6880a14c-73e8-4bf9-a5aa-df9b0ecc9f69",
"metadata": {},
"source": [
"# Cleanup\n",
"Finally, we clean up all resources not needed anymore since we pledge for the responsible use of compute resources. In this case this is the created endpoint together with the respective endpoint configuration. "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "223e5554-f70d-4238-b94f-dba3f84e85f6",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"predictor_finetuned.delete_endpoint(delete_endpoint_config=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9602cab9-2929-44fe-a3c0-a8c1a804a845",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"predictor_plain.delete_endpoint(delete_endpoint_config=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "266e2b3c-e4f3-4c74-b699-88b0f27913a4",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"availableInstances": [
{
"_defaultOrder": 0,
"_isFastLaunch": true,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 4,
"name": "ml.t3.medium",
"vcpuNum": 2
},
{
"_defaultOrder": 1,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 8,
"name": "ml.t3.large",
"vcpuNum": 2
},
{
"_defaultOrder": 2,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 16,
"name": "ml.t3.xlarge",
"vcpuNum": 4
},
{
"_defaultOrder": 3,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 32,
"name": "ml.t3.2xlarge",
"vcpuNum": 8
},
{
"_defaultOrder": 4,
"_isFastLaunch": true,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 8,
"name": "ml.m5.large",
"vcpuNum": 2
},
{
"_defaultOrder": 5,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 16,
"name": "ml.m5.xlarge",
"vcpuNum": 4
},
{
"_defaultOrder": 6,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 32,
"name": "ml.m5.2xlarge",
"vcpuNum": 8
},
{
"_defaultOrder": 7,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 64,
"name": "ml.m5.4xlarge",
"vcpuNum": 16
},
{
"_defaultOrder": 8,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 128,
"name": "ml.m5.8xlarge",
"vcpuNum": 32
},
{
"_defaultOrder": 9,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 192,
"name": "ml.m5.12xlarge",
"vcpuNum": 48
},
{
"_defaultOrder": 10,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 256,
"name": "ml.m5.16xlarge",
"vcpuNum": 64
},
{
"_defaultOrder": 11,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 384,
"name": "ml.m5.24xlarge",
"vcpuNum": 96
},
{
"_defaultOrder": 12,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 8,
"name": "ml.m5d.large",
"vcpuNum": 2
},
{
"_defaultOrder": 13,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 16,
"name": "ml.m5d.xlarge",
"vcpuNum": 4
},
{
"_defaultOrder": 14,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 32,
"name": "ml.m5d.2xlarge",
"vcpuNum": 8
},
{
"_defaultOrder": 15,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 64,
"name": "ml.m5d.4xlarge",
"vcpuNum": 16
},
{
"_defaultOrder": 16,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 128,
"name": "ml.m5d.8xlarge",
"vcpuNum": 32
},
{
"_defaultOrder": 17,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 192,
"name": "ml.m5d.12xlarge",
"vcpuNum": 48
},
{
"_defaultOrder": 18,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 256,
"name": "ml.m5d.16xlarge",
"vcpuNum": 64
},
{
"_defaultOrder": 19,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 384,
"name": "ml.m5d.24xlarge",
"vcpuNum": 96
},
{
"_defaultOrder": 20,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": true,
"memoryGiB": 0,
"name": "ml.geospatial.interactive",
"supportedImageNames": [
"sagemaker-geospatial-v1-0"
],
"vcpuNum": 0
},
{
"_defaultOrder": 21,
"_isFastLaunch": true,
"category": "Compute optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 4,
"name": "ml.c5.large",
"vcpuNum": 2
},
{
"_defaultOrder": 22,
"_isFastLaunch": false,
"category": "Compute optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 8,
"name": "ml.c5.xlarge",
"vcpuNum": 4
},
{
"_defaultOrder": 23,
"_isFastLaunch": false,
"category": "Compute optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 16,
"name": "ml.c5.2xlarge",
"vcpuNum": 8
},
{
"_defaultOrder": 24,
"_isFastLaunch": false,
"category": "Compute optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 32,
"name": "ml.c5.4xlarge",
"vcpuNum": 16
},
{
"_defaultOrder": 25,
"_isFastLaunch": false,
"category": "Compute optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 72,
"name": "ml.c5.9xlarge",
"vcpuNum": 36
},
{
"_defaultOrder": 26,
"_isFastLaunch": false,
"category": "Compute optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 96,
"name": "ml.c5.12xlarge",
"vcpuNum": 48
},
{
"_defaultOrder": 27,
"_isFastLaunch": false,
"category": "Compute optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 144,
"name": "ml.c5.18xlarge",
"vcpuNum": 72
},
{
"_defaultOrder": 28,
"_isFastLaunch": false,
"category": "Compute optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 192,
"name": "ml.c5.24xlarge",
"vcpuNum": 96
},
{
"_defaultOrder": 29,
"_isFastLaunch": true,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 16,
"name": "ml.g4dn.xlarge",
"vcpuNum": 4
},
{
"_defaultOrder": 30,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 32,
"name": "ml.g4dn.2xlarge",
"vcpuNum": 8
},
{
"_defaultOrder": 31,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 64,
"name": "ml.g4dn.4xlarge",
"vcpuNum": 16
},
{
"_defaultOrder": 32,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 128,
"name": "ml.g4dn.8xlarge",
"vcpuNum": 32
},
{
"_defaultOrder": 33,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 4,
"hideHardwareSpecs": false,
"memoryGiB": 192,
"name": "ml.g4dn.12xlarge",
"vcpuNum": 48
},
{
"_defaultOrder": 34,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 256,
"name": "ml.g4dn.16xlarge",
"vcpuNum": 64
},
{
"_defaultOrder": 35,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 61,
"name": "ml.p3.2xlarge",
"vcpuNum": 8
},
{
"_defaultOrder": 36,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 4,
"hideHardwareSpecs": false,
"memoryGiB": 244,
"name": "ml.p3.8xlarge",
"vcpuNum": 32
},
{
"_defaultOrder": 37,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 8,
"hideHardwareSpecs": false,
"memoryGiB": 488,
"name": "ml.p3.16xlarge",
"vcpuNum": 64
},
{
"_defaultOrder": 38,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 8,
"hideHardwareSpecs": false,
"memoryGiB": 768,
"name": "ml.p3dn.24xlarge",
"vcpuNum": 96
},
{
"_defaultOrder": 39,
"_isFastLaunch": false,
"category": "Memory Optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 16,
"name": "ml.r5.large",
"vcpuNum": 2
},
{
"_defaultOrder": 40,
"_isFastLaunch": false,
"category": "Memory Optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 32,
"name": "ml.r5.xlarge",
"vcpuNum": 4
},
{
"_defaultOrder": 41,
"_isFastLaunch": false,
"category": "Memory Optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 64,
"name": "ml.r5.2xlarge",
"vcpuNum": 8
},
{
"_defaultOrder": 42,
"_isFastLaunch": false,
"category": "Memory Optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 128,
"name": "ml.r5.4xlarge",
"vcpuNum": 16
},
{
"_defaultOrder": 43,
"_isFastLaunch": false,
"category": "Memory Optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 256,
"name": "ml.r5.8xlarge",
"vcpuNum": 32
},
{
"_defaultOrder": 44,
"_isFastLaunch": false,
"category": "Memory Optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 384,
"name": "ml.r5.12xlarge",
"vcpuNum": 48
},
{
"_defaultOrder": 45,
"_isFastLaunch": false,
"category": "Memory Optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 512,
"name": "ml.r5.16xlarge",
"vcpuNum": 64
},
{
"_defaultOrder": 46,
"_isFastLaunch": false,
"category": "Memory Optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 768,
"name": "ml.r5.24xlarge",
"vcpuNum": 96
},
{
"_defaultOrder": 47,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 16,
"name": "ml.g5.xlarge",
"vcpuNum": 4
},
{
"_defaultOrder": 48,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 32,
"name": "ml.g5.2xlarge",
"vcpuNum": 8
},
{
"_defaultOrder": 49,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 64,
"name": "ml.g5.4xlarge",
"vcpuNum": 16
},
{
"_defaultOrder": 50,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 128,
"name": "ml.g5.8xlarge",
"vcpuNum": 32
},
{
"_defaultOrder": 51,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 256,
"name": "ml.g5.16xlarge",
"vcpuNum": 64
},
{
"_defaultOrder": 52,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 4,
"hideHardwareSpecs": false,
"memoryGiB": 192,
"name": "ml.g5.12xlarge",
"vcpuNum": 48
},
{
"_defaultOrder": 53,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 4,
"hideHardwareSpecs": false,
"memoryGiB": 384,
"name": "ml.g5.24xlarge",
"vcpuNum": 96
},
{
"_defaultOrder": 54,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 8,
"hideHardwareSpecs": false,
"memoryGiB": 768,
"name": "ml.g5.48xlarge",
"vcpuNum": 192
},
{
"_defaultOrder": 55,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 8,
"hideHardwareSpecs": false,
"memoryGiB": 1152,
"name": "ml.p4d.24xlarge",
"vcpuNum": 96
},
{
"_defaultOrder": 56,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 8,
"hideHardwareSpecs": false,
"memoryGiB": 1152,
"name": "ml.p4de.24xlarge",
"vcpuNum": 96
}
],
"instance_type": "ml.t3.medium",
"kernelspec": {
"display_name": "Python 3 (Data Science)",
"language": "python",
"name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/datascience-1.0"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.10"
},
"vscode": {
"interpreter": {
"hash": "5c7b89af1651d0b8571dde13640ecdccf7d5a6204171d6ab33e7c296e100e08a"
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}