{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deploy Transformers model using SageMaker Hugging Face Deep Learning Container\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. [Introduction](#Introduction)  \n",
    "2. [Setup](#Setup)\n",
    "3. [Download Hugging Face Pretrained Model](#Download-Hugging-Face-Pretrained-Model)\n",
    "4. [Package the saved model to tar.gz format](#Package-the-saved-model-to-tar.gz-format)\n",
    "5. [Upload Pre-trained Model to S3](#Upload-Pre-trained-Model-to-S3)\n",
    "6. [Deploy the model using `model_data`](#Deploy-the-model-using-model_data)  \n",
    "7. [Prediction with Amazon SageMaker endpoint](#Prediction-with-Amazon-SageMaker-endpoint)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For inference, you can use your trained Hugging Face model or one of the pretrained Hugging Face models to deploy an inference job with SageMaker. \n",
    "\n",
    "### How to deploy an inference job using the Hugging Face Deep Learning Containers\n",
    "You have two options for running inference with SageMaker. You can run inference using a model that you trained, or deploy a pre-trained Hugging Face model.\n",
    "\n",
    "* Run inference with your trained model: You have two options for running inference with your own trained model. You can run inference with a model that you trained using an existing Hugging Face model with the SageMaker Hugging Face Deep Learning Containers, or you can bring your own existing Hugging Face model and deploy it using SageMaker. When you run inference with a model that you trained with the SageMaker Hugging Face Estimator, you can deploy the model immediately after training completes or you can upload the trained model to an Amazon S3 bucket and ingest it when running inference later. If you bring your own existing Hugging Face model, you must upload the trained model to an Amazon S3 bucket and ingest that bucket when running inference.\n",
    "\n",
    "* Run inference with a pre-trained HuggingFace model: You can use one of the thousands of pre-trained Hugging Face models to run your inference jobs with no additional training needed. We will see this in our lab today.\n",
    "\n",
    "In this notebook, we will use the `Hugging Face Inference DLCs and Amazon SageMaker Python SDK` to deploy a pre-trained transformer model for real-time inference. We will perform following steps:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!pip install \"sagemaker>=2.48.0\" --upgrade\n",
    "!pip install torch -q\n",
    "!pip install transformers -q\n",
    "!pip install ipywidgets -q"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Note: Restart the notebook after installing the above packages.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import display_html\n",
    "def restartkernel() :\n",
    "    display_html(\"<script>Jupyter.notebook.kernel.restart()</script>\",raw=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "restartkernel()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy a Hugging Face Transformer model from S3 to SageMaker for inference\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download Hugging Face Pretrained Model\n",
    "\n",
    "In this example we are downloading a pre-trained HuggingFace model - `facebook/bart-large-mnli` from the HuggingFace library. We will use this model for classifying the tweets as `POSTIVE` or `NEGATIVE`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pip install huggingface-hub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoModelForSequenceClassification, AutoTokenizer\n",
    "MODEL = 'facebook/bart-large-mnli'\n",
    "model = AutoModelForSequenceClassification.from_pretrained(MODEL)\n",
    "tokenizer = AutoTokenizer.from_pretrained(MODEL)\n",
    "model.save_pretrained('model_token')\n",
    "tokenizer.save_pretrained('model_token')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Package the saved model to tar.gz format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the model is downloaded, we need to package (tokenizer and model weights) it to `.tar.gz` format as expected by Amazon SageMaker. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cd model_token && tar zcvf model.tar.gz * \n",
    "!mv model_token/model.tar.gz ./model.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload Pre-trained Model to S3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to use the `sagemaker.s3.S3Uploader` api to upload our model to an S3 location. We will provide this s3 path to the `HuggingFaceModel` class during deployment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "from sagemaker.s3 import S3Uploader,s3_path_join\n",
    "\n",
    "# get the s3 bucket\n",
    "sess = sagemaker.Session()\n",
    "role = sagemaker.get_execution_role()\n",
    "sagemaker_session_bucket = sess.default_bucket()\n",
    "# uploads a given file to S3.\n",
    "upload_path = s3_path_join(\"s3://\",sagemaker_session_bucket,\"bart_model\")\n",
    "print(f\"Uploading Model to {upload_path}\")\n",
    "model_uri = S3Uploader.upload('model.tar.gz',upload_path)\n",
    "print(f\"Uploaded model to {model_uri}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store model_uri"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy the model using `model_data`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will deploy our pre-trained model which we packaged and uploaded to s3 in the previous steps, using the model_data argument to specify the s3 location of your tokenizer and model weights."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Parameters for `HuggingFaceModel` class\n",
    "We will use following parameters in this lab for deploying the model. \n",
    "* `model_data (str)` – The Amazon S3 location of a SageMaker model data .tar.gz file.\n",
    "\n",
    "* `role (str)` – An AWS IAM role specified with either the name or full ARN. The Amazon SageMaker training jobs and APIs that create Amazon SageMaker endpoints use this role to access training data and model artifacts. After the endpoint is created, the inference code might use the IAM role, if it needs to access an AWS resource.\n",
    "\n",
    "* `transformers_version (str)` – Transformers version you want to use for executing your model training code. Defaults to None. Required unless image_uri is provided.\n",
    "\n",
    "* `pytorch_version (str)` – PyTorch version you want to use for executing your inference code. Defaults to None. Required unless tensorflow_version is provided. List of supported versions: https://github.com/aws/sagemaker-python-sdk#huggingface-sagemaker-estimators.\n",
    "\n",
    "* `py_version (str)` – Python version you want to use for executing your model training code. Defaults to None. Required unless image_uri is provided.\n",
    "\n",
    "For details about other paramets, please click [here](#https://sagemaker.readthedocs.io/en/stable/frameworks/huggingface/sagemaker.huggingface.html?highlight=huggingfacemodel#sagemaker.huggingface.model.HuggingFaceModel)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.huggingface import HuggingFaceModel\n",
    "import sagemaker \n",
    "\n",
    "role = sagemaker.get_execution_role()\n",
    "\n",
    "# create Hugging Face Model Class\n",
    "huggingface_model = HuggingFaceModel(\n",
    "   model_data=model_uri,  # path to your trained sagemaker model\n",
    "   role=role, # iam role with permissions to create an Endpoint\n",
    "   transformers_version=\"4.17\", # transformers version used\n",
    "   pytorch_version=\"1.10\", # pytorch version used\n",
    "   py_version=\"py38\", # python version of the DLC\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# deploy model to SageMaker Inference\n",
    "import time\n",
    "ts = time.strftime(\"%Y-%m-%d-%H-%M-%S\", time.gmtime())\n",
    "endpoint_name = \"bart-large-blog-\" + ts\n",
    "\n",
    "predictor = huggingface_model.deploy(\n",
    "   initial_instance_count=1,\n",
    "   instance_type=\"ml.g4dn.xlarge\",\n",
    "   endpoint_name=endpoint_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prediction with Amazon SageMaker endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# example request, you always need to define \"inputs\"\n",
    "data = {\n",
    "   \"inputs\": \"The new Hugging Face SageMaker DLC makes it super easy to deploy models in production. I love it!\"\n",
    "}\n",
    "\n",
    "# request\n",
    "predictor.predict(data)"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.g4dn.xlarge",
  "interpreter": {
   "hash": "c281c456f1b8161c8906f4af2c08ed2c40c50136979eaae69688b01f70e9f4a9"
  },
  "kernelspec": {
   "display_name": "Python 3 (Data Science)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/datascience-1.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}