{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deploy Hugging Face BART transformer models with multi-model endpoints \n",
    "\n",
    "This notebook is a step-by-step tutorial on deploying multiple pre-trained PyTorch Hugging Face model [BART](https://huggingface.co/transformers/model_doc/bart.html) with multi-model endpoint on Amazon SageMaker. Bart uses a standard seq2seq/machine translation architecture with a bidirectional encoder (like BERT) and a left-to-right decoder (like GPT). Specifically, we use the BART Model with a language modeling head [BartForConditionalGeneration](https://huggingface.co/transformers/model_doc/bart.html#transformers.BartForConditionalGeneration) for summarization task. \n",
    "\n",
    "We will describe the steps for deploying a multi-model endpoint on Amazon SageMaker with TorchServe serving stack. An additional step compared to single model deployment is the requirement to create a manifest file for each model prior to deployment. For training Hugging Face models on SageMaker, refer the examples [here](https://github.com/huggingface/notebooks/tree/master/sagemaker)\n",
    "\n",
    "The outline of steps is as follows:\n",
    "\n",
    "1. Download 2 pre-trained Hugging Face model\n",
    "2. Use torch-archiver to create a manifest file for each model\n",
    "3. Save and upload model artifact to S3\n",
    "4. Create an inference entrypoint script\n",
    "5. Deploy multi-model endpoint\n",
    "6. Trigger endpoint invocation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "from sagemaker import get_execution_role\n",
    "from sagemaker.utils import name_from_base\n",
    "from sagemaker.pytorch import PyTorchModel\n",
    "import boto3\n",
    "import torch\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "us-east-1\n",
      "arn:aws:iam::208480242416:role/service-role/AmazonSageMaker-ExecutionRole-endtoendml\n",
      "sagemaker-us-east-1-208480242416\n"
     ]
    }
   ],
   "source": [
    "from sagemaker import get_execution_role\n",
    "\n",
    "role = get_execution_role()\n",
    "region = boto3.Session().region_name\n",
    "sagemaker_session = sagemaker.session.Session()\n",
    "bucket = sagemaker_session.default_bucket()\n",
    "prefix = 'hf-multimodel-deploy-pytorch'\n",
    "hf_cache_dir = 'hf_cache_dir/'\n",
    "\n",
    "print(region)\n",
    "print(role)\n",
    "print(bucket)\n",
    "\n",
    "model_data_path = 's3://{0}/{1}/models'.format(bucket,prefix)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download the Hugging Face pretrained model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install transformers==4.5.1 --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "transformers==4.5.1\r\n"
     ]
    }
   ],
   "source": [
    "!pip freeze | grep transformers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "pip install -U ipywidgets --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import BartForConditionalGeneration, BartTokenizer, BartConfig"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e4ca39bb5ffa4712bd8ff861e1424498",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/1.40k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ea2da84786ff4b4898a0389342090b64",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/1.63G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Download a pre-tuned bart transformer and move the model artifact to  S3 bucket\n",
    "PRE_TRAINED_MODEL_NAME1='facebook/bart-large-cnn'\n",
    "# Note that we use a specific HF cache dir, to avoid using the default cache dirs that might fill \n",
    "# root disk space.\n",
    "model1 = BartForConditionalGeneration.from_pretrained(PRE_TRAINED_MODEL_NAME1, cache_dir=hf_cache_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "92c008372f244ee6948b24fe768db960",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/1.65k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ed628ff2b5f94b969e9eaf09028c01bf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/1.22G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Create a fine-tuned bart transformer  and host on S3 bucket\n",
    "#This is generally a training process to fine tune as per your own data\n",
    "#In this example, we save the exact pretrained model itself\n",
    "PRE_TRAINED_MODEL_NAME2='sshleifer/distilbart-cnn-12-6'\n",
    "model2 = BartForConditionalGeneration.from_pretrained(PRE_TRAINED_MODEL_NAME2, cache_dir=hf_cache_dir)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3e75bf0510554d4e8bed58c10d730187",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/899k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "88c864cd0c77442a838c893128539d30",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/456k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "90e317bf4ff04cf4b36f2f6fe715fcb9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/1.36M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1b4df079ca42422e98fd623d62e8e047",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/899k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fda09734796949ab97b6725ba77fc13a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/456k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3f7ff8c3e9f34354a0ce2cdb22da38cc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/26.0 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tokenizer1 = BartTokenizer.from_pretrained(PRE_TRAINED_MODEL_NAME1)\n",
    "tokenizer2 = BartTokenizer.from_pretrained(PRE_TRAINED_MODEL_NAME2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create and upload model archive to S3 \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('./models/model1/bart_tokenizer/tokenizer_config.json',\n",
       " './models/model1/bart_tokenizer/special_tokens_map.json',\n",
       " './models/model1/bart_tokenizer/vocab.json',\n",
       " './models/model1/bart_tokenizer/merges.txt',\n",
       " './models/model1/bart_tokenizer/added_tokens.json')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model1.save_pretrained('./models/model1/bart_model/')\n",
    "tokenizer1.save_pretrained('./models/model1/bart_tokenizer/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('./models/model2/bart_tokenizer/tokenizer_config.json',\n",
       " './models/model2/bart_tokenizer/special_tokens_map.json',\n",
       " './models/model2/bart_tokenizer/vocab.json',\n",
       " './models/model2/bart_tokenizer/merges.txt',\n",
       " './models/model2/bart_tokenizer/added_tokens.json')"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model2.save_pretrained('./models/model2/bart_model/')\n",
    "tokenizer2.save_pretrained('./models/model2/bart_tokenizer/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q torch-model-archiver"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from model_archiver import model_packaging\n",
    "from model_archiver import model_packaging_utils\n",
    "from model_archiver import arg_parser\n",
    "\n",
    "#we add the handler from the sagemaker pytorch inference toolkit to the list of known handlers to model archiver\n",
    "model_packaging_utils.model_handlers['sagemaker_pytorch_serving_container.handler_service'] = 'toolkit_handler'\n",
    "\n",
    "arguments = '--model-name summarizer1 --version 1.0 \\\n",
    "--export-path models \\\n",
    "--extra-files models/model1/ \\\n",
    "--handler sagemaker_pytorch_serving_container.handler_service --archive-format no-archive'\n",
    "args = arg_parser.ArgParser.export_model_args_parser().parse_args(arguments.split(' '))\n",
    "manifest = model_packaging_utils.ModelExportUtils.generate_manifest_json(args)\n",
    "model_packaging.package_model(args, manifest=manifest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -rf models/model1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from model_archiver import model_packaging\n",
    "from model_archiver import model_packaging_utils\n",
    "from model_archiver import arg_parser\n",
    "\n",
    "#we add the handler from the sagemaker pytorch inference toolkit to the list of known handlers to model archiver\n",
    "model_packaging_utils.model_handlers['sagemaker_pytorch_serving_container.handler_service'] = 'toolkit_handler'\n",
    "\n",
    "arguments = '--model-name summarizer2 --version 1.0 \\\n",
    "--export-path models/ \\\n",
    "--extra-files models/model2/ \\\n",
    "--handler sagemaker_pytorch_serving_container.handler_service --archive-format no-archive'\n",
    "args = arg_parser.ArgParser.export_model_args_parser().parse_args(arguments.split(' '))\n",
    "manifest = model_packaging_utils.ModelExportUtils.generate_manifest_json(args)\n",
    "model_packaging.package_model(args, manifest=manifest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -rf models/model2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add inference code "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are manually adding the inference code  to the model folder, to avoid the SM Python SDK having to repack the model.tar.gz archive when executing deployment. Since there are large models, the repack operation can take some time (downlaod from S3, repack, re-upload).\n",
    "The custom inference code must be stored in the code/ folder in the model archive, and the name of the entrypoint module is inference.py by default. You can customize that by passing an environment variable named SAGEMAKER_PROGRAM when creating the Model object (see below)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir models/summarizer1/code\n",
    "!mkdir models/summarizer2/code\n",
    "\n",
    "! cp source_dir/model1/inference.py models/summarizer1/code/inference.py\n",
    "! cp source_dir/model2/inference.py models/summarizer2/code/inference.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create model archive and upload to S3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "./\n",
      "./code/\n",
      "./code/inference.py\n",
      "./bart_tokenizer/\n",
      "./bart_tokenizer/merges.txt\n",
      "./bart_tokenizer/tokenizer_config.json\n",
      "./bart_tokenizer/vocab.json\n",
      "./bart_tokenizer/special_tokens_map.json\n",
      "./bart_model/\n",
      "./bart_model/config.json\n",
      "./bart_model/pytorch_model.bin\n"
     ]
    }
   ],
   "source": [
    "!tar -czvf models/summarizer1.tar.gz -C models/summarizer1/ .\n",
    "!tar -czvf models/summarizer2.tar.gz -C models/summarizer2/ ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.s3 import S3Uploader\n",
    "S3Uploader.upload('models/summarizer1.tar.gz', model_data_path)\n",
    "S3Uploader.upload('models/summarizer2.tar.gz', model_data_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a MultiDataModel and deploy to a SageMaker endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.model import Model\n",
    "from sagemaker.multidatamodel import MultiDataModel\n",
    "from sagemaker.serializers import JSONSerializer\n",
    "from sagemaker.deserializers import JSONDeserializer\n",
    "\n",
    "from sagemaker.image_uris import retrieve\n",
    "\n",
    "deploy_instance_type = 'ml.m5.4xlarge'\n",
    "\n",
    "pytorch_inference_image_uri = retrieve('pytorch',\n",
    "                                       region,\n",
    "                                       version='1.8.1',\n",
    "                                       py_version='py3',\n",
    "                                       instance_type = deploy_instance_type,\n",
    "                                       accelerator_type=None,\n",
    "                                       image_scope='inference')\n",
    "print(pytorch_inference_image_uri)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "dummy_model = Model(name = 'dummy_model_pt',\n",
    "                    image_uri = pytorch_inference_image_uri,\n",
    "                    role = role,\n",
    "                    model_data = '')\n",
    "\n",
    "multi_model = MultiDataModel(name              = 'pytorch-multi-model-summarizer-' + time.strftime(\"%Y-%m-%d-%H-%M-%S\", time.gmtime()),\n",
    "                             model             = dummy_model,\n",
    "                             model_data_prefix = model_data_path)\n",
    "\n",
    "endpoint_name = 'torchserve-multimodel-summarizer-endpoint-' + time.strftime(\"%Y-%m-%d-%H-%M-%S\", time.gmtime())\n",
    "\n",
    "predictor = multi_model.deploy(instance_type=deploy_instance_type,\n",
    "                               initial_instance_count=1,\n",
    "                               endpoint_name = endpoint_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3://sagemaker-us-east-1-208480242416/hf-multimodel-deploy-pytorch/models\n"
     ]
    }
   ],
   "source": [
    "print(model_data_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.predictor import Predictor\n",
    "pred = Predictor(endpoint_name)\n",
    "pred.serializer = sagemaker.serializers.JSONSerializer()\n",
    "pred.deserializer = sagemaker.deserializers.JSONDeserializer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Invoke endpoint "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'text': 'The Amazon Technical Academy upskilling program targets Amazon employees aspiring to become software engineers. Its leader says education is key to long-term success. The benefits are vast for Amazon employees accepted into Amazon Technical Academy, one of six training programs included in Upskilling 2025, Amazon’s $700 million commitment to equip more than 100,000 Amazon employees with new professional skills by 2025.  Amazon Technical Academy trains employees in the essential skills needed to transition to entry-level software developer engineer roles at Amazon. The program requires no previous computer training from applicants, only a high school diploma or GED—and the fortitude to get through a rigorous nine-month, full-time program created by expert Amazon software engineers.  Hundreds of Amazon employees have enrolled in Amazon Technical Academy since its launch in 2017. Amazon Technical Academy has placed 98% of its graduates into software development engineer roles within Amazon, with their salary and compensation packages increasing an average of 93%. Applicants accepted into the tuition-free program receive a stipend to cover living costs and a subsidy to maintain their benefits plan.  As part of its commitment to provide career advancement opportunities for employees, the company invested more than $12 million into this program in 2020 alone.  Ashley Rajagopal, a longtime Amazon employee who has held various roles across Amazon’s Consumer business, leads the program. She joined Amazon Technical Academy early on, as a small team of engineers and product managers evaluating whether Amazon could upskill employees into software engineering careers regardless of their tech skills or backgrounds.  Ashely Rajagopal wears a black dress with a jean jacket and a long gold necklace. She smiles as she stands outside in a park. Ashley Rajagopal leads the Amazon Technical Academy. Photo by Mitch Pittman/Amazon “Key to our success has been our deliberate effort to demystify the skills it takes to become a software engineer,” Rajagopal said. “As we’ve defined those skills, we have intentionally evolved our curriculum and teaching approach to be accessible to participants who didn’t have the opportunity, either because of background or financial limitations, to pursue a college degree in software engineering.”  Graduates come from a vast array of professional backgrounds at Amazon, including fulfillment center associates, program managers, recruitment coordinators, executive assistants, and financial analysts. Their personal backgrounds are just as varied: single parents, immigrants, college graduates, GED recipients. The diversity is a reflection of Amazon Technical Academy’s intentional accessibility.  What graduates all have in common, Rajagopal said, is career ownership and a desire to pursue a new professional path.  “Our graduates all had a vision for their future and an unwavering commitment to advance into a technical role. Amazon Technical Academy was simply here to open the door to a role as a software engineer and offer the support they needed to get there. I love their passion to pursue their dreams,” Rajagopal said.  “Someone saw something in me when I came to work at Amazon 11 years ago,” said added. “I had managers invest in me. I feel like it’s really important for me to share that with other people and to pass that along. I believe education is the key to giving people a vision and path to reach their potential and taking control of their career progression.”  Software engineering for all Since its inception, Amazon Technical Academy has aimed to not only help individuals advance their careers to better support themselves and their families, but to provide Amazon hiring managers with high-performing software engineers who understand Amazon’s systems and culture.  “We had an idea—a really big idea—that we could reimagine how Amazon trains and recruits software engineers. In true Amazon fashion, we focused on building what would work best for our customers, our customers being both the participants and our hiring managers,” Rajagopal said.  Over the last four years, the team worked tirelessly to build the right curriculum. They conducted extensive focus groups with software development managers and software engineers from across the company to identify all of the skills that software engineers need to use day-to-day in their job and throughout their career.  Ashely Rajagopal sits on a bench outside with vibrant greenery behind her. She is focused while she works on her laptop. Photo by Mitch Pittman/Amazon “We broke up complex software engineering topics into small, discreet skills,” Rajagopal said.  With the catalog of skills, Amazon Technical Academy sought to reimagine how to teach these skills to make them more accessible to a broad audience. The learning environment is structured in a flipped classroom environment where students read and watch the lecture materials before coming to class. This gives them the opportunity to spend as much time as they need to learn the material before deep diving into the topics in a classroom with other students and an instructor.  The lecture materials and assignments teach the skills using broadly understood, real-world examples outside the stereotypical software engineering culture. “When we deep dive into a particular topic, we don’t teach in abstract, mathematical concepts that are regularly used in a traditional computer science university setting,” said Rajagopal. “Instead, we relate the concept to real life examples like cleaning your room, growing a flower, or opening a Russian nesting doll that many people are familiar with.”  Amazon Technical Academy takes off This unusual approach has allowed Amazon Technical Academy to attract and successfully train participants from a wide array of educational and professional backgrounds. While the program’s pilot cohort was limited to corporate employees in Seattle, the program subsequently opened applications to all employees in the U.S., and now “a third of participants don’t have a college degree and 40% of our participants were previously in the hourly workforce,” Rajagopal said.  “We pursue people across all backgrounds. Holistically, for Amazon, it’s important,” she added. “We’re looking to rethink and reimagine, and eliminate some of the barriers that exist in particular industries.  Amazon Technical Academy is now offered as a free, core training and job-placement program that equips Amazon employees with essential skills needed to transition to and thrive in technical careers at Amazon. The program focuses on combining instructor-led, project-based learning with real-world application.  The result is graduates who can expertly work with the most widely used software engineering tools, including Amazon Web Services (AWS) cloud computing technology. When the program ends, Amazon Technical Academy graduates transition into full-time, entry-level software engineering roles.  “Over the last two years, we’ve focused on building out this program,” Rajagopal said. “We started as a small proof-of-concept for corporate employees with live, in-person instruction from a tenured Amazon software engineer.”  Amazon Technical Academy expands beyond Amazon With a solid foundation of coursework and graduate success in place, Amazon Technical Academy is now working with two online training partners—Lambda School and Kenzie Academy—to bring its rigorous curriculum to students outside Amazon. Graduates will leave the programs with deep knowledge of software engineering skills and tools, including AWS cloud computing technology.  The new Amazon-backed engineering-focused programs will offer more people access to Amazon Technical Academy coursework, which is especially helpful for individuals who did not pursue a more traditional four-year computer science degree. Graduates will be armed with the industry-leading, bar-raising skills required of Amazon software engineers.  Lambda School and Kenzie Academy already have strong technical foundations: Lambda School focuses on training data scientists and web developers, and the Kenzie Academy offers programs in software engineering and UX design. Both schools are adopting Amazon Technical Academy’s curriculum and making adjustments to align with their program structure and student needs. Lambda School will begin accepting applications in August 2021 and Kenzie Academy will begin accepting applications immediately.  The schools aim to recruit a diverse student body (e.g., gender, racial, and financial diversity in their applicant pool). Lambda School’s Enterprise Backend Development Program will be a nine-month, full-time, fully remote course. Kenzie Academy’s Software Engineering Program will be a nine- to 12-month, full-time, fully remote course with no fixed class time.  Their programs will ready graduates for a market that, according to the Bureau of Labor Statistics, is expected to grow twice as fast for computer science professionals than for the rest of the labor market from 2014 to 2024. The bureau’s research also found that in 2019, the median annual salary for computer science occupations was about $48,000 more than the median wage for all occupations in the U.S.  “We are very proud to partner with Amazon on our forthcoming Enterprise Backend Development Program,” said Austen Allred, Lambda School CEO. “We’re thrilled to provide this opportunity to our future students. Backend is a skill set that is incredibly in demand with all of our largest hiring partners and this enables us to deliver a world class curriculum.”  “We are excited to work with Amazon to expand access to the Amazon Technical Academy software engineering curriculum to the general public,” said Chok Ooi, Executive Director, Kenzie Academy. “At Kenzie Academy, our focus is on leveling the playing field to enable more Americans to pursue tech careers. This is a great opportunity to combine Amazon’s exclusive software engineering curriculum with our expertise in propelling a diverse range of learners to professional success.”  To learn more about Amazon’s Upskilling 2025 commitment and Amazon Technical Academy, you can visit Amazon’s Upskilling website, or email: ata-contact-us@amazon.com.'}"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with open('article.txt') as f:\n",
    "    content = f.read()\n",
    "content = content.replace('\\n', ' ')\n",
    "\n",
    "json_request_data = {\"text\": \"{0}\"}\n",
    "json_request_data[\"text\"] = json_request_data[\"text\"].format(content)\n",
    "\n",
    "json_request_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 11.2 ms, sys: 0 ns, total: 11.2 ms\n",
      "Wall time: 3.45 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Amazon Technical Academy trains employees in the essential skills needed to transition to entry-level software developer engineer roles at Amazon. The program requires no previous computer training from applicants, only a high school diploma or GED. Hundreds of Amazon employees have enrolled in Amazon Technical Academy since its launch in 2017.'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "model_archive = '/summarizer1.tar.gz'\n",
    "pred.predict(json_request_data, target_model=model_archive, target_variant=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.99 ms, sys: 0 ns, total: 3.99 ms\n",
      "Wall time: 2.61 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "' Amazon upskilling program targets Amazon employees aspiring to become software engineers. The program requires no previous computer training from applicants, only a high school diploma or GED. Amazon Technical Academy has placed 98% of its graduates into software development engineer roles within Amazon. Graduates come from a vast array of professional backgrounds at Amazon.'"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "model_archive = '/summarizer2.tar.gz'\n",
    "pred.predict(json_request_data, target_model=model_archive, target_variant=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Predictions for HuggingFace PyTorch multi-model endpoints with SageMaker : \n",
      "\n",
      "\n",
      "P95: 3644.779336452484 ms\n",
      "\n",
      "P90: 3639.8558855056763 ms\n",
      "\n",
      "Average: 3600.4682779312134 ms\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import time,numpy as np\n",
    "results = []\n",
    "for i in (1,100):\n",
    "    start = time.time()\n",
    "    model_archive = '/summarizer1.tar.gz'\n",
    "    pred.predict(json_request_data, target_model=model_archive, target_variant=None)\n",
    "    results.append((time.time() - start) * 1000)\n",
    "print(\"\\nPredictions for HuggingFace PyTorch multi-model endpoints with SageMaker : \\n\")\n",
    "print('\\nP95: ' + str(np.percentile(results, 95)) + ' ms\\n')    \n",
    "print('P90: ' + str(np.percentile(results, 90)) + ' ms\\n')\n",
    "print('Average: ' + str(np.average(results)) + ' ms\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleanup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "predictor.delete_endpoint(delete_endpoint_config=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor.delete_model()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_pytorch_p36",
   "language": "python",
   "name": "conda_pytorch_p36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}