{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bc5ab391",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Kor LLM 모델 서빙 (SageMaker Python SDK)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "694d9185",
   "metadata": {},
   "source": [
    "### 참조: \n",
    "- Model 정보\n",
    "    - beomi/KoAlpaca-Polyglot-12.8B\n",
    "        - This model is a fine-tuned version of EleutherAI/polyglot-ko-12.8b on a KoAlpaca Dataset v1.1b\n",
    "        - https://huggingface.co/beomi/KoAlpaca-Polyglot-12.8B\n",
    "    - EleutherAI/polyglot-ko-12.8b\n",
    "        - Polyglot-Ko-12.8B was trained for 167 billion tokens over 301,000 steps on 256 A100 GPUs with the GPT-NeoX framework. It was trained as an autoregressive language model, using cross-entropy loss to maximize the likelihood of predicting the next token.\n",
    "        - License: Apache 2.0\n",
    "        - https://huggingface.co/EleutherAI/polyglot-ko-12.8b\n",
    "        \n",
    "- Doc\n",
    "    - Large model inference tutorials\n",
    "        - https://docs.aws.amazon.com/sagemaker/latest/dg/large-model-inference-tutorials.html\n",
    "    - Use DJL with the SageMaker Python SDK\n",
    "        - https://sagemaker.readthedocs.io/en/stable/frameworks/djl/using_djl.html\n",
    "        \n",
    "- 블로그\n",
    "    - https://aws.amazon.com/ko/blogs/machine-learning/deploy-large-models-on-amazon-sagemaker-using-djlserving-and-deepspeed-model-parallel-inference/\n",
    "    - 코드\n",
    "        - https://github.com/aws/amazon-sagemaker-examples/blob/main/inference/generativeai/deepspeed/GPT-J-6B_DJLServing_with_PySDK.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b1ae4f8-2cf7-40eb-adf8-6dca4260c92e",
   "metadata": {},
   "source": [
    "# 1. 기본 환경 설정"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "cee2723b-2c1e-4924-9853-664a26cd7075",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "# src 폴더 경로 설정\n",
    "import sys\n",
    "sys.path.append('../common_code')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81c2bdf4",
   "metadata": {},
   "source": [
    "# 2. SageMaker endpoint 의 추론 도커 이미지 인 DLC image URL 가져오기\n",
    "- We get DLC image URL for djl-deepspeed 0.21.0 and set SageMaker settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "2876d11c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'763104351884.dkr.ecr.us-east-1.amazonaws.com/djl-inference:0.21.0-deepspeed0.8.3-cu117'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sagemaker, boto3\n",
    "from sagemaker import image_uris\n",
    "\n",
    "\n",
    "role = sagemaker.get_execution_role()  # execution role for the endpoint\n",
    "session = sagemaker.session.Session()  # sagemaker session for interacting with different AWS APIs\n",
    "region = session._region_name\n",
    "bucket = session.default_bucket()  # bucket to house artifacts\n",
    "\n",
    "img_uri = image_uris.retrieve(framework=\"djl-deepspeed\", region=region, version=\"0.21.0\")\n",
    "img_uri"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdff2960-a103-4016-be3e-83947b4d36d6",
   "metadata": {},
   "source": [
    "# 3. Set configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6bf555e-7128-4378-9f48-103674ebc859",
   "metadata": {},
   "source": [
    "## 테스트 모델 지정"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a2ca985f-a53f-459e-bd8c-6adfeb2977e4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "serve_model:  KoAlpaca-12-8B\n",
      "model_artifact_name:  KoAlpaca-12-8B.tar.gz\n",
      "model packaging s3 location:  s3://sagemaker-us-east-1-057716757052/KoAlpaca-12-8B\n"
     ]
    }
   ],
   "source": [
    "serve_model = 'KoAlpaca-12-8B'\n",
    "# serve_model = 'Polyglot-Kor-5-8B'\n",
    "# serve_model = 'Kullm-polyglot-12-8b-v2'\n",
    "\n",
    "# 모델 패키징 할 파일 이름\n",
    "model_artifact_name = f'{serve_model}.tar.gz'\n",
    "\n",
    "# 모델 패키징 S3 위치\n",
    "s3_location = f\"s3://{bucket}/{serve_model}\"\n",
    "\n",
    "print(\"serve_model: \", serve_model)\n",
    "print(\"model_artifact_name: \", model_artifact_name)\n",
    "print(\"model packaging s3 location: \", s3_location)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0629748a-a8d3-4478-ade6-dee7ba9b4907",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 모델 다운로드"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "99d3f352-5179-4442-8cae-32580cf0ed9e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "##수정중\n",
    "# from huggingface_hub import snapshot_download\n",
    "# from pathlib import Path\n",
    "# import os\n",
    "\n",
    "# # - This will download the model into the current directory where ever the jupyter notebook is running\n",
    "# local_model_path = Path(\".\")\n",
    "# local_model_path.mkdir(exist_ok=True)\n",
    "# model_name = \"beomi/KoAlpaca-Polyglot-12.8B\"\n",
    "\n",
    "# # Only download pytorch checkpoint files\n",
    "# allow_patterns = [\"*.json\", \"*.pt\", \"*.bin\", \"*.txt\", \"*.model\"]\n",
    "\n",
    "# # - Leverage the snapshot library to donload the model since the model is stored in repository using LFS\n",
    "# model_download_path = snapshot_download(\n",
    "#     repo_id=model_name,\n",
    "#     cache_dir=local_model_path,\n",
    "#     allow_patterns=allow_patterns,\n",
    "# )\n",
    "# model_download_path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c896de9e-87ce-4e19-a9fa-a93cbf21a487",
   "metadata": {},
   "source": [
    "## 로컬 모드 혹은 클라우드 모드"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a2e4c5f7-b590-4470-ba3d-cb1cfff12231",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "instance_type  : ml.g5.12xlarge\n"
     ]
    }
   ],
   "source": [
    "use_local_mode = False\n",
    "# use_local_mode = True\n",
    "\n",
    "if use_local_mode:\n",
    "    instance_type = \"local_gpu\"\n",
    "    from sagemaker.local import LocalSession\n",
    "    sagemaker_session = LocalSession()\n",
    "    sagemaker_session.config = {'local': {'local_code': True}}\n",
    "else:\n",
    "    sagemaker_session = sagemaker.session.Session()\n",
    "    instance_type = \"ml.g5.12xlarge\"\n",
    "    \n",
    "print(\"instance_type  :\", instance_type)      "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ac32e96",
   "metadata": {},
   "source": [
    "# 4. 모델 추론 코드 및 모델 설정 파일을 패키징\n",
    "- `model.py` and `serving.properties`\n",
    "- The code below creates the SageMaker model file (`model.tar.gz`) and upload it to S3. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4a724622-069c-453a-8c2d-3a43adbc0a91",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "KoAlpaca-12-8B\n",
      "KoAlpaca-12-8B.tar.gz\n",
      "KoAlpaca-12-8B/\n",
      "KoAlpaca-12-8B/serving.properties\n",
      "KoAlpaca-12-8B/model.py\n"
     ]
    }
   ],
   "source": [
    "%%sh -s {serve_model} {model_artifact_name}\n",
    "serve_model=$1\n",
    "model_artifact_name=$2\n",
    "echo $serve_model\n",
    "echo $model_artifact_name\n",
    "\n",
    "rm -rf $serve_model/.ipynb_checkpoints\n",
    "\n",
    "tar -czvf $model_artifact_name $serve_model/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b9bbee9-224b-4daf-915e-a1e40d72e47f",
   "metadata": {},
   "source": [
    "## mode.tar.gz 및 pretrained model을 S3 업로드"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "db47f969",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model_tar_url:  s3://sagemaker-us-east-1-057716757052/KoAlpaca-12-8B/KoAlpaca-12-8B.tar.gz\n"
     ]
    }
   ],
   "source": [
    "model_tar_url = sagemaker.s3.S3Uploader.upload(model_artifact_name, s3_location)\n",
    "print(\"model_tar_url: \", model_tar_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "90325ad7-06bc-4f30-a02f-94f1c6e43e09",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "## 수정 중\n",
    "# define a variable to contain the s3url of the location that has the model\n",
    "#pretrained_model_location = f'{s3_location}/pretrained/' \n",
    "#print(f\"Pretrained model will be uploaded to ---- > {pretrained_model_location}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8f584935-bf66-4f33-84e8-52909450efb4",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "## 수정 중\n",
    "# model_artifact = session.upload_data(path=model_download_path, key_prefix=f'{serve_model}/pretrained')\n",
    "# print(f\"Model uploaded to --- > {model_artifact}\")\n",
    "# print(f\"We will set option.s3url={model_artifact}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "1f3646c9-70a7-4307-bfa3-e3d0f6824bbf",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "## 수정 중\n",
    "# we plug in the appropriate model location into our `serving.properties` file based on the region in which this notebook is running\n",
    "#jinja_env = jinja2.Environment()\n",
    "#template = jinja_env.from_string(Path(\"mymodel/serving.properties\").open().read())\n",
    "#Path(\"mymodel/serving.properties\").open(\"w\").write(template.render(s3url=pretrained_model_location))\n",
    "#!pygmentize mymodel/serving.properties | cat -n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b80a0e7e-b29f-40f7-b8ab-eb8a009e12da",
   "metadata": {},
   "source": [
    "# 5. SageMaker Model 생성"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "01ea57da-b463-4eee-92b2-49e4b5dbe9de",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.model import Model\n",
    "from sagemaker import image_uris, get_execution_role\n",
    "\n",
    "def create_model(model_name, role, sagemaker_session, inference_image_uri, model_s3_url):\n",
    "    model = Model(\n",
    "        image_uri=inference_image_uri,\n",
    "        model_data=model_s3_url,\n",
    "        role=role,\n",
    "        name=model_name,\n",
    "        sagemaker_session=sagemaker_session,\n",
    "    )\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "61e1b1d7-4674-4c30-97bc-70db65e37a43",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "time_stamp = datetime.now().strftime(\"%Y-%m-%d-%H-%M-%S\")\n",
    "model_name = f\"{serve_model}-\" + time_stamp\n",
    "\n",
    "sm_model = create_model(model_name, role, sagemaker_session, img_uri, model_tar_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d32fe7b7-926f-4fa7-afbb-459c30740f85",
   "metadata": {
    "tags": []
   },
   "source": [
    "# 6. SageMaker Endpoint 생성\n",
    "- 클라우드 배포시 약 8분 걸림. 10 이상 걸리면 무엇인가 문제 있음"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "ecb50049-daa1-46a3-959e-95bf78fba7c0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker import serializers, deserializers\n",
    "\n",
    "def deploy_model(model, sagemaker_session, instance_type, _endpoint_name):\n",
    "    model.deploy(\n",
    "        initial_instance_count=1,\n",
    "        instance_type=instance_type,\n",
    "        endpoint_name=_endpoint_name\n",
    "    )\n",
    "    predictor = sagemaker.Predictor(\n",
    "        endpoint_name=_endpoint_name,\n",
    "        sagemaker_session=sagemaker_session,\n",
    "        serializer=serializers.JSONSerializer(),\n",
    "        deserializer=deserializers.JSONDeserializer()\n",
    "    )\n",
    "    return predictor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "793b40cf-c8d8-4ca6-bfbe-f2b8ca8edd6f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------!CPU times: user 127 ms, sys: 41.2 ms, total: 169 ms\n",
      "Wall time: 12min 34s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "endpoint_name = f\"{serve_model}-\" + time_stamp\n",
    "predictor = deploy_model(sm_model, sagemaker_session, instance_type, endpoint_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f1221df-caf9-4050-9541-34bf558538c7",
   "metadata": {},
   "source": [
    "# 7. 엔드포인트 추론 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "5ed7a325",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from inference_lib import invoke_inference_DJ    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e210bb4f-4683-4fc1-9b9a-b487e0deaa7d",
   "metadata": {},
   "source": [
    "### options for generation\n",
    "* **temperature**: Controls randomness in the model. Lower values will make the model more deterministic and higher values will make the model more random. Default value is 1.0.\n",
    "* **max_new_tokens**: The maximum number of tokens to generate. Default value is 20, max value is 512.\n",
    "* **repetition_penalty**: Controls the likelihood of repetition, defaults to null.\n",
    "* **seed**: The seed to use for random generation, default is null.\n",
    "* **stop**: A list of tokens to stop the generation. The generation will stop when one of the tokens is generated.\n",
    "* **top_k**: The number of highest probability vocabulary tokens to keep for top-k-filtering. Default value is null, which disables top-k-filtering.\n",
    "* **top_p**: The cumulative probability of parameter highest probability vocabulary tokens to keep for nucleus sampling, default to null\n",
    "* **do_sample**: Whether or not to use sampling ; use greedy decoding otherwise. Default value is false.\n",
    "* **best_of**: Generate best_of sequences and return the one if the highest token logprobs, default to null.\n",
    "* **details**: Whether or not to return details about the generation. Default value is false.\n",
    "* **return_full_text**: Whether or not to return the full text or only the generated part. Default value is false.\n",
    "* **truncate**: Whether or not to truncate the input to the maximum length of the model. Default value is true.\n",
    "* **typical_p**: The typical probability of a token. Default value is null.\n",
    "* **watermark**: The watermark to use for the generation. Default value is false."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "af70195f-79f5-4dcb-85db-67862911b105",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "params = {\n",
    "    \"do_sample\":False, \n",
    "    \"max_new_tokens\":128,\n",
    "    \"temperature\":1.0,\n",
    "    \"top_k\":0,\n",
    "    \"top_p\":0.9,\n",
    "    \"return_full_text\":False,\n",
    "    \"repetition_penalty\":1.1,\n",
    "    \"presence_penalty\":None,\n",
    "    \"eos_token_id\":2,\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa002070-b85d-4f3d-a3da-57ed0d63ac16",
   "metadata": {
    "tags": []
   },
   "source": [
    "## (1) 맥락 (Context) 없이 질문"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "74e7eeea-2eef-464a-aca8-7b4d5116ba1e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "prompt_wo_c: \n",
      " {'prompt': ['### 질문: 홈플러스 중계점은 몇시까지 장사해?\\n\\n### 답변:'], 'params': {'do_sample': False, 'max_new_tokens': 128, 'temperature': 1.0, 'top_k': 0, 'top_p': 0.9, 'return_full_text': False, 'repetition_penalty': 1.1, 'presence_penalty': None, 'eos_token_id': 2}}\n"
     ]
    }
   ],
   "source": [
    "q = \"홈플러스 중계점은 몇시까지 장사해?\"\n",
    "c = \"\"#\"홈플러스 영업시간은 오전 10시 부터 오후 12시까지 입니다.\"\n",
    "prompt_wo_c = f\"### 질문: {q}\\n\\n### 맥락: {c}\\n\\n### 답변:\" if c else f\"### 질문: {q}\\n\\n### 답변:\" \n",
    "data = {\n",
    "    \"prompt\": [prompt_wo_c,],\n",
    "    \"params\": params\n",
    "}\n",
    "print(\"prompt_wo_c: \\n\", data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "7b73fca4-9318-401c-97ba-3005c40adfde",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "  [\n",
      "    {\n",
      "      \"generated_text\":\"홈플러스의 매장 운영 시간은 점포마다 다릅니다. 대부분의 매장은 10시에 문을 닫으며, 일부 매장은 11시까지 영업합니다. 예를 들어, 홈플러스 동대문점은 11시까지 영업하며, 홈플러스 영등포점과 강서점은 9시에 문을 닫습니다. 또한, 홈플러스 김해점과 밀양점은 8시에 문을 닫고 있습니다. \"\n",
      "    }\n",
      "  ]\n",
      "]\n",
      "CPU times: user 52.3 ms, sys: 10.6 ms, total: 63 ms\n",
      "Wall time: 5.79 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'[\\n  [\\n    {\\n      \"generated_text\":\"홈플러스의 매장 운영 시간은 점포마다 다릅니다. 대부분의 매장은 10시에 문을 닫으며, 일부 매장은 11시까지 영업합니다. 예를 들어, 홈플러스 동대문점은 11시까지 영업하며, 홈플러스 영등포점과 강서점은 9시에 문을 닫습니다. 또한, 홈플러스 김해점과 밀양점은 8시에 문을 닫고 있습니다. \"\\n    }\\n  ]\\n]'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time \n",
    "invoke_inference_DJ(endpoint_name, data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c47feb14-8bc0-42f7-8319-fa37ca1ee340",
   "metadata": {},
   "source": [
    "## (2) 맥락 (Context) 가지고 질문"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "6864581e-186d-412e-bd7f-c5cd049539c8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "prompt_w_c:\n",
      " ### 질문: 홈플러스 중계점은 몇시까지 장사해?\n",
      "\n",
      "### 맥락: 홈플러스 영업시간은 오전 10시 부터 오후 10시까지 입니다. 홈플러스 매장 찾기(영업시간 확인)는 이 주소를 이용하세요:  http://corporate.homeplus.co.kr/Store.aspx?isA=%C1%F6%BF%B4%C7%B0%BF%AE%C0%C7%C1%F2%B5%B5%B4%F6 \n",
      "\n",
      "### 답변:\n"
     ]
    }
   ],
   "source": [
    "q = \"홈플러스 중계점은 몇시까지 장사해?\"\n",
    "c = \"홈플러스 영업시간은 오전 10시 부터 오후 10시까지 입니다. 홈플러스 매장 찾기(영업시간 확인)는 이 주소를 이용하세요:  http://corporate.homeplus.co.kr/Store.aspx?isA=%C1%F6%BF%B4%C7%B0%BF%AE%C0%C7%C1%F2%B5%B5%B4%F6 \"\n",
    "prompt_w_c = f\"### 질문: {q}\\n\\n### 맥락: {c}\\n\\n### 답변:\" if c else f\"### 질문: {q}\\n\\n### 답변:\" \n",
    "data = {\n",
    "    \"prompt\": [prompt_w_c,],\n",
    "    \"params\": params\n",
    "}\n",
    "print(\"prompt_w_c:\\n\", prompt_w_c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "a7b53302-ec17-4a8d-837e-e93d7c5b90cf",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "  [\n",
      "    {\n",
      "      \"generated_text\":\"홈플러스의 영업 시간은 오전 10시부터 오후 10시까지입니다. 각 지점마다 약간씩 차이가 있을 수 있으므로, 방문 전에 영업 시간을 꼭 확인해보시기 바랍니다. 아래 링크에서 홈플러스 매장 찾기를 통해 영업 시간을 확인하실 수 있습니다: http://corporate.homeplus.co.kr/Store.aspx?isA=%C1%F6%BF%B4%C7%B0%BF%AE%C0%C7%C1%F2%\"\n",
      "    }\n",
      "  ]\n",
      "]\n",
      "CPU times: user 15.3 ms, sys: 1.32 ms, total: 16.6 ms\n",
      "Wall time: 8.3 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'[\\n  [\\n    {\\n      \"generated_text\":\"홈플러스의 영업 시간은 오전 10시부터 오후 10시까지입니다. 각 지점마다 약간씩 차이가 있을 수 있으므로, 방문 전에 영업 시간을 꼭 확인해보시기 바랍니다. 아래 링크에서 홈플러스 매장 찾기를 통해 영업 시간을 확인하실 수 있습니다: http://corporate.homeplus.co.kr/Store.aspx?isA=%C1%F6%BF%B4%C7%B0%BF%AE%C0%C7%C1%F2%\"\\n    }\\n  ]\\n]'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time \n",
    "invoke_inference_DJ(endpoint_name,  data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92e83c91",
   "metadata": {},
   "source": [
    "# 7. [중요] 클린업 엔트포인트 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "a15980a3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# delete endpoint\n",
    "predictor.delete_model()\n",
    "predictor.delete_endpoint()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a46913c-00c0-4ac9-b0f9-8c7c39c2119d",
   "metadata": {},
   "source": [
    "# Trouble Shooting"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f09ee2a-566d-460c-842d-42a68dcd622a",
   "metadata": {},
   "source": [
    "3vep2qi5ar-algo-1-zk8i8 | INFO  ModelServer Model server stopped.\n",
    "\n",
    "3vep2qi5ar-algo-1-zk8i8 | ERROR ModelServer Invalid configuration: Workflow KoAlpaca_12_8B is already registered."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ada9b60-c190-4b65-89c8-615cff143bc7",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   }
  ],
  "instance_type": "ml.m5.large",
  "kernelspec": {
   "display_name": "Python 3 (PyTorch 1.13 Python 3.9 CPU Optimized)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/pytorch-1.13-cpu-py39"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  },
  "vscode": {
   "interpreter": {
    "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}