{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QoT3NSUxzrFb"
      },
      "source": [
        "# Deploy and perform inference on Model Package from AWS Marketplace \n",
        "\n",
        "This notebook provides you instructions on how to deploy and perform inference on model packages from AWS Marketplace Hugging Face Named Entity Recognition model.\n",
        "\n",
        "This notebook is compatible only with those Hugging Face Named Entity Recognition model packages which this notebook is linked to.\n",
        "\n",
        "#### Pre-requisites:\n",
        "1. **Note**: This notebook contains elements which render correctly in Jupyter interface. Open this notebook from an Amazon SageMaker Notebook Instance or Amazon SageMaker Studio.\n",
        "1. Ensure that IAM role used has **AmazonSageMakerFullAccess**\n",
        "1. To deploy this ML model successfully, ensure that:\n",
        "    1. Either your IAM role has these three permissions and you have authority to make AWS Marketplace subscriptions in the AWS account used: \n",
        "        1. **aws-marketplace:ViewSubscriptions**\n",
        "        1. **aws-marketplace:Unsubscribe**\n",
        "        1. **aws-marketplace:Subscribe**  \n",
        "    2. or your AWS account has a subscription to this Hugging Face Named Entity Recognition model. If so, skip step: [Subscribe to the model package](#1.-Subscribe-to-the-model-package)\n",
        "\n",
        "#### Contents:\n",
        "1. [Subscribe to the model package](#1.-Subscribe-to-the-model-package)\n",
        "2. [Create an endpoint and perform real-time inference](#2.-Create-an-endpoint-and-perform-real-time-inference)\n",
        "   1. [Create an endpoint](#A.-Create-an-endpoint)\n",
        "   2. [Create input payload](#B.-Create-input-payload)\n",
        "   3. [Perform real-time inference](#C.-Perform-real-time-inference)\n",
        "   4. [Delete the endpoint](#D.-Delete-the-endpoint)\n",
        "3. [Perform batch inference](#3.-Perform-batch-inference) \n",
        "4. [Clean-up](#4.-Clean-up)\n",
        "    1. [Delete the model](#A.-Delete-the-model)\n",
        "    2. [Unsubscribe to the listing (optional)](#B.-Unsubscribe-to-the-listing-(optional))\n",
        "    \n",
        "\n",
        "#### Usage instructions\n",
        "You can run this notebook one cell at a time (By using Shift+Enter for running a cell).\n",
        "\n",
        "**Note** - This notebook requires you to follow instructions and specify values for parameters, as instructed."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bWFzr4MNzrFe"
      },
      "source": [
        "### 1. Subscribe to the model package"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ip7SYwwuzrFe"
      },
      "source": [
        "To subscribe to the model package:\n",
        "1. Open the model package listing page you opened this notebook for.\n",
        "1. On the AWS Marketplace listing, click on the **Continue to subscribe** button.\n",
        "1. On the **Subscribe to this software** page, review and click on **\"Accept Offer\"** if you and your organization agrees with EULA, pricing, and support terms. \n",
        "1. Once you click on **Continue to configuration button** and then choose a **region**, you will see a **Product Arn** displayed. This is the model package ARN that you need to specify while creating a deployable model using Boto3. Copy the ARN corresponding to your region and specify the same in the following cell."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ceFu8-pvzrFf"
      },
      "outputs": [],
      "source": [
        "model_package_arn='<Customer to specify Model package ARN corresponding to their AWS region>' "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NMjlrf1rzrFg"
      },
      "outputs": [],
      "source": [
        "import json \n",
        "from sagemaker import ModelPackage\n",
        "import sagemaker as sage\n",
        "from sagemaker import get_execution_role"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xTLmsbRNzrFg"
      },
      "outputs": [],
      "source": [
        "role = get_execution_role()\n",
        "sagemaker_session = sage.Session()\n",
        "boto3 = sagemaker_session.boto_session\n",
        "bucket = sagemaker_session.default_bucket()\n",
        "region = sagemaker_session.boto_region_name\n",
        "\n",
        "s3 = boto3.client(\"s3\")\n",
        "runtime= boto3.client('runtime.sagemaker')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XeAgfxGazrFh"
      },
      "source": [
        "In next step, you would be deploying the model for real-time inference. For  information on how real-time inference with Amazon SageMaker works, see [Documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/how-it-works-hosting.html)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dvJzR1nrzrFh"
      },
      "source": [
        "### 2. Create an endpoint and perform real-time inference"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ojXcE60UzrFi"
      },
      "outputs": [],
      "source": [
        "model_name='named-entity-recognition-model'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_21CpTtNzrFi"
      },
      "outputs": [],
      "source": [
        "#The Hugging Facce Named Entity Recognition model packages this notebook notebook is compatible with, support application/x-text as the \n",
        "#content-type.\n",
        "content_type='application/x-text'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oh-7M2pPzrFj"
      },
      "source": [
        "Review and update the compatible instance type for the model package in the following cell."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "a5b61sJBzrFj"
      },
      "outputs": [],
      "source": [
        "real_time_inference_instance_type='ml.g4dn.xlarge'\n",
        "batch_transform_inference_instance_type='ml.p2.xlarge'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JZfOBRyszrFj"
      },
      "source": [
        "#### A. Create an endpoint"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gAWD_mtnzrFk"
      },
      "outputs": [],
      "source": [
        "#create a deployable model from the model package.\n",
        "model = ModelPackage(role=role,\n",
        "                    model_package_arn=model_package_arn,\n",
        "                    sagemaker_session=sagemaker_session)\n",
        "\n",
        "#Deploy the model\n",
        "predictor = model.deploy(1, real_time_inference_instance_type, endpoint_name=model_name)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wh6jA0WUzrFk"
      },
      "source": [
        "Once endpoint has been created, you would be able to perform real-time inference."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dvYKbpDHzrFk"
      },
      "source": [
        "#### B. Prepare input file for performing real-time inference\n",
        "#### Let's put in some example input text. You can put in any English text, the model predicts named entities in the text."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_O1j66zuzrFk"
      },
      "outputs": [],
      "source": [
        "input_text = \"My name is Wolfgang and I live in Berlin\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XR4LZ5LFzrFl"
      },
      "source": [
        "#### C. Query endpoint that you have created"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "O28mDdmhzrFl"
      },
      "outputs": [],
      "source": [
        "#perform_inference method performs inference on the endpoint and prints predictions.\n",
        "newline = '\\n'\n",
        "bold = '\\033[1m'\n",
        "unbold = '\\033[0m'\n",
        "def query_endpoint(encoded_text):\n",
        "    endpoint_name = model_name\n",
        "    response = runtime.invoke_endpoint(EndpointName=endpoint_name, ContentType=content_type, Body=encoded_text)\n",
        "    model_predictions = json.loads(response['Body'].read())\n",
        "    return model_predictions\n",
        "\n",
        "model_predictions = query_endpoint(json.dumps(input_text).encode('utf-8'))\n",
        "print (f\"Inference:{newline}\"\n",
        "        f\"input text: {input_text}{newline}\"\n",
        "        f\"model prediction: {bold}{model_predictions}{unbold}{newline}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vHZtid2rzrFl"
      },
      "source": [
        "#### D. Delete the endpoint"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a7HmqxjuzrFl"
      },
      "source": [
        "Now that you have successfully performed a real-time inference, you do not need the endpoint any more. You can terminate the endpoint to avoid being charged."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7Ll8GY8ZzrFm"
      },
      "outputs": [],
      "source": [
        "model.sagemaker_session.delete_endpoint(model_name)\n",
        "model.sagemaker_session.delete_endpoint_config(model_name)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GnERE-zzzrFm"
      },
      "source": [
        "### 3. Perform batch inference"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "maABVgLczrFm"
      },
      "source": [
        "In this section, you will perform batch inference using multiple input payloads together. If you are not familiar with batch transform, and want to learn more, see [How to run a batch transform job](https://docs.aws.amazon.com/sagemaker/latest/dg/how-it-works-batch.html)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VSGSqxWrzrFm"
      },
      "outputs": [],
      "source": [
        "#upload the batch-transform job input files to S3\n",
        "transform_input_key_prefix = 'named-entity-recognition-model-transform-input'\n",
        "f = open(\"transform-input-data.txt\", \"w\")\n",
        "f.write(\"My name is Wolfgang and I live in Berlin\")\n",
        "f.close()\n",
        "transform_input = sagemaker_session.upload_data(\"transform-input-data.txt\", key_prefix=transform_input_key_prefix) \n",
        "print(\"Transform input uploaded to \" + transform_input)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b48RwPq7zrFm"
      },
      "outputs": [],
      "source": [
        "#Run the batch-transform job\n",
        "transformer = model.transformer(1, batch_transform_inference_instance_type)\n",
        "transformer.transform(transform_input, content_type=content_type)\n",
        "transformer.wait()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8Ll-cI_2zrFn"
      },
      "outputs": [],
      "source": [
        "# output is available on following path\n",
        "transformer.output_path"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ea20tyqzzrFn"
      },
      "outputs": [],
      "source": [
        "output_bucket_name, output_path = transformer.output_path.replace(\"s3://\", \"\").split(\"/\", 1)\n",
        "obj = s3.get_object(Bucket=output_bucket_name, Key=output_path + '/transform-input-data.txt.out')\n",
        "batch_prediction = obj['Body'].read().decode('utf-8')\n",
        "\n",
        "# print out batch-transform job output\n",
        "print(batch_prediction)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G5OI2L61zrFn"
      },
      "source": [
        "### 4. Clean-up"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TBvxFImFzrFn"
      },
      "source": [
        "#### A. Delete the model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Q6Jee8PQzrFn"
      },
      "outputs": [],
      "source": [
        "model.delete_model()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q6J-gWOEzrFn"
      },
      "source": [
        "#### B. Unsubscribe to the listing (optional)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cn1FoQuazrFn"
      },
      "source": [
        "If you would like to unsubscribe to the model package, follow these steps. Before you cancel the subscription, ensure that you do not have any [deployable model](https://console.aws.amazon.com/sagemaker/home#/models) created from the model package or using the algorithm. Note - You can find this information by looking at the container name associated with the model. \n",
        "\n",
        "**Steps to unsubscribe to product from AWS Marketplace**:\n",
        "1. Navigate to __Machine Learning__ tab on [__Your Software subscriptions page__](https://aws.amazon.com/marketplace/ai/library?productType=ml&ref_=mlmp_gitdemo_indust)\n",
        "2. Locate the listing that you want to cancel the subscription for, and then choose __Cancel Subscription__  to cancel the subscription.\n",
        "\n"
      ]
    }
  ],
  "metadata": {
    "instance_type": "ml.t3.medium",
    "kernelspec": {
      "display_name": "conda_pytorch_latest_p36",
      "language": "python",
      "name": "conda_pytorch_latest_p36"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.13"
    },
    "colab": {
      "name": "note.ipynb",
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}