{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"id": "f8acc98c-07fe-4d78-abe4-52510e2f7c4d",
"metadata": {},
"source": [
"# YoloV8 Model Inference in Amazon SageMaker\n",
"This notebook will demonstrate how to create an endpoint for real time inference with the trained YoloV8 (see [1] and [2]) model.\n",
"\n",
"References:\n",
"----\n",
"[1] https://docs.ultralytics.com/ </br>\n",
"[2] https://github.com/ultralytics/ultralytics"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "ee535e76-caf1-437c-b9cb-911718b13fd0",
"metadata": {},
"source": [
"## 1. SageMaker Initialization\n",
"First we upgrade SageMaker to the latest version. If your notebook is already using latest Sagemaker 2.x API, you may skip the next cell."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8a61d6dc-d23a-4ed8-88c6-548ff177d712",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"! pip install --upgrade pip\n",
"! python3 -m pip install --upgrade sagemaker"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6abe8f55-6bd4-442c-b571-7e2a45f220ce",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"import boto3\n",
"import sagemaker\n",
"from sagemaker import get_execution_role\n",
"\n",
"role = (\n",
" get_execution_role()\n",
") # provide a pre-existing role ARN as an alternative to creating a new role\n",
"print(f\"SageMaker Execution Role:{role}\")\n",
"\n",
"client = boto3.client('sts')\n",
"account = client.get_caller_identity()['Account']\n",
"print(f'AWS account:{account}')\n",
"\n",
"session = boto3.session.Session()\n",
"aws_region = session.region_name\n",
"print(f\"AWS region:{aws_region}\")\n",
"\n",
"container_name = \"inference-container\""
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "cf18f5ea-f785-4c34-a0c3-a8c26a621e7b",
"metadata": {},
"source": [
"## 2. Build, Test and Push Amazon SageMaker Serving Container Images\n",
"For this step, the [IAM Role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) attached to this Studio notebook needs full access to [Amazon ECR service](https://aws.amazon.com/ecr/) and access to [Amazon EC2 service](https://aws.amazon.com/ec2/). Check this [page](https://github.com/aws-samples/sagemaker-studio-docker-cli-extension#prerequsites) for prerequistes to use this notebook on SageMaker Studio. We use prebuild [YoloV8 docker image](https://hub.docker.com/r/ultralytics/ultralytics/tags) as a base to build on top and configure it to work with SageMaker."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "8f8b31f9-d308-4c11-a6d9-64220d754bbf",
"metadata": {
"tags": []
},
"source": [
"### 2.1. Use local Mode to develop and test our code"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "28b0d4ad-72cd-4411-ab19-1c6d6baccbbf",
"metadata": {},
"source": [
"#### 2.1.1. Docker Environment Preparation on SageMaker Studio Notebooks\n",
"By default, SageMaker Studio does not support docker operations. This [github repo](https://github.com/aws-samples/sagemaker-studio-docker-cli-extension) enables us to build, test and push images on Studio. You can skip this step if you running this notebook on your local machine or on Amazon SageMaker Notebook Instance.\n",
"</br></br>\n",
"Run the below cell to clone [SageMaker Studio Docker CLI extension](https://github.com/aws-samples/sagemaker-studio-docker-cli-extension) and install required dependencies missing from **Data Science** kernel"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "875f92a5-d391-44f4-94a4-b85ca0eb63e8",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"!cd ~ && git clone https://github.com/aws-samples/sagemaker-studio-docker-cli-extension\n",
"\n",
"# fix dependancies\n",
"!conda update --force -y conda\n",
"!conda install -y pyyaml==5.4.1\n",
"#!apt-get install -y procps\n",
"\n",
"# setup the extension\n",
"!cd ~/sagemaker-studio-docker-cli-extension && ./setup.sh"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "457c9f8a-608c-4f20-8f5b-1dbb2d80534e",
"metadata": {},
"source": [
"We will use **m5.xlarge** instance to build and test YoloV8 SageMaker docker image"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a00eec62-d8a4-4675-a8d0-2b1de6d10e20",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"!sdocker create-host --instance-type m5.xlarge"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "5c1885e9-e297-42b0-9765-8c5be5b39776",
"metadata": {
"tags": []
},
"source": [
"#### 2.1.2. Build docker image\n",
"We first build docker image to test it in local mode before deploying it to ECR"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cd53c111-4d14-4a1e-8d05-3c6b8a3e562c",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"!cd inference-container && docker build . -t yolov8-sagemaker-inference:latest"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "87edce2a-068f-4236-af2d-0de4a6c1ef79",
"metadata": {},
"source": [
"#### 2.2.1. Create Local Inference Endpoint"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "5e873a86-0b30-441a-83c6-5836cd71445e",
"metadata": {},
"source": [
"##### 2.2.1.1. Define Amazon SageMaker Model\n",
"We first download pretrained model, then we define SageMaker model."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "368662b7-3087-4500-8299-5e6972b9af3e",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"!mkdir -p inference-container/model && cd inference-container/model && wget https://github.com/ultralytics/assets/releases/download/v0.0.0/yolov8n-pose.pt"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "e850e866",
"metadata": {},
"source": [
"The container can have the following environment variables:\n",
"\n",
"| Env | Value |\n",
"|---------------|---------------------------|\n",
"| SM_MODEL_NAME | Name of YoloV8 model |\n",
"| SM_TRACKING | `Enabled` or `Disabled` |\n",
"| SM_HP_CLASSES | List of classes ex. `[0]` |\n",
"| SM_HP_CONF | Default `0.25` |\n",
"| SM_HP_IOU | Default `0.7` |\n",
"| SM_HP_HALF | Default `False` |\n",
"| SM_HP_TRACKER | `botsort` or `bytetrack` |"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e7e11a77-6843-4121-b2a8-b18f947e1f66",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"from sagemaker.local import LocalSession\n",
"\n",
"sagemaker_session = LocalSession()\n",
"sagemaker_session.config = {'local': {'local_code': True}}\n",
"\n",
"image_uri = \"yolov8-sagemaker-inference:latest\"\n",
"model_name = \"yolov8-model-1\" # set the name of the model\n",
"\n",
"model_uri = \"file://inference-container/model/yolov8n-pose.pt\" # define the local pretrained model URI manually.\n",
"\n",
"serving_container_def = {\n",
" 'Image': image_uri,\n",
" 'ModelDataUrl': model_uri,\n",
" 'Mode': 'SingleModel',\n",
" 'Environment': {\n",
" 'SM_MODEL_NAME' : 'yolov8n-pose.pt'\n",
" }\n",
"}\n",
"\n",
"create_model_response = sagemaker_session.create_model(name=model_name, \n",
" role=role, \n",
" container_defs=serving_container_def)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "1edbf312-38ff-4f3b-8bcb-1cc3b72efed9",
"metadata": {},
"source": [
"##### 2.2.1.2. Create Endpoint Configuration\n",
"Next, we set the name of the Amaozn SageMaker hosted service endpoint configuration.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c0172c8e-b1ad-4aab-884c-e58e8db01773",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"endpoint_config_name = f\"{model_name}-endpoint-config\"\n",
"print(endpoint_config_name)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "d007261f-0b9a-446c-896f-3bc198ccd707",
"metadata": {},
"source": [
"Then create the local Amazon SageMaker hosted service endpoint configuration that uses one a local endpoint container for testing purposes."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f25964e5-b54b-4b00-b001-31f03646490d",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"epc = sagemaker_session.create_endpoint_config(\n",
" name=endpoint_config_name,\n",
" model_name=model_name,\n",
" initial_instance_count=1,\n",
" instance_type=\"local\",\n",
")\n",
"print(epc)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "80e9240e-81aa-4f1d-aee3-56d7e98a3355",
"metadata": {},
"source": [
"Next we specify the Amazon SageMaker endpoint name for the endpoint used to serve the model."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a85172d9-a58f-438a-86d3-1b724389439f",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"endpoint_name = f\"{model_name}-endpoint\"\n",
"print(endpoint_name)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "3dd2da8f-7af9-411e-84dc-a7b132533f49",
"metadata": {},
"source": [
"##### 2.2.1.3. Create Endpoint\n",
"In this step, we create the Amazon SageMaker endpoint using the endpoint configuration we created above."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a8dca9bd-5e9f-4d7f-afc3-e259f32067c5",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"ep = sagemaker_session.create_endpoint(\n",
" endpoint_name=endpoint_name, config_name=endpoint_config_name, wait=True\n",
")\n",
"print(ep)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "939acbd0-3e45-4b3d-a9ea-95598f9b035d",
"metadata": {},
"source": [
"### 2.2.2. Test Local Endpoint"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "ff70e515-513a-42a8-9435-343f3f8fedf1",
"metadata": {},
"source": [
"#### 2.2.2.1. Visualization Helper Functions\n",
"Draw the bounding box, pose sekeleton and ID for each tracked object in the raw frames."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4466fe0d-0d23-4f87-85b9-f831f79000f7",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"!apt-get update && apt-get install ffmpeg libsm6 libxext6 -y\n",
"!pip install opencv-python"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e1dae137-4a5e-45e3-b0aa-2300909baeca",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"import cv2\n",
"import numpy as np\n",
"\n",
"pose_palette = np.array([[255, 128, 0], [255, 153, 51], [255, 178, 102], [230, 230, 0], [255, 153, 255],\n",
" [153, 204, 255], [255, 102, 255], [255, 51, 255], [102, 178, 255], [51, 153, 255],\n",
" [255, 153, 153], [255, 102, 102], [255, 51, 51], [153, 255, 153], [102, 255, 102],\n",
" [51, 255, 51], [0, 255, 0], [0, 0, 255], [255, 0, 0], [255, 255, 255]],\n",
" dtype=np.uint8)\n",
"\n",
"skeleton = [[16, 14], [14, 12], [17, 15], [15, 13], [12, 13], [6, 12], [7, 13], [6, 7], [6, 8], [7, 9],\n",
" [8, 10], [9, 11], [2, 3], [1, 2], [1, 3], [2, 4], [3, 5], [4, 6], [5, 7]]\n",
"\n",
"limb_color = pose_palette[[9, 9, 9, 9, 7, 7, 7, 0, 0, 0, 0, 0, 16, 16, 16, 16, 16, 16, 16]]\n",
"kpt_color = pose_palette[[16, 16, 16, 16, 16, 0, 0, 0, 0, 0, 0, 9, 9, 9, 9, 9, 9]]\n",
"\n",
"\n",
"def get_color(idx):\n",
" idx = idx * 3\n",
" color = (75, 95, 230)\n",
"\n",
" return color\n",
"\n",
"def draw_res(det_boxes, frame, frame_id, image_w):\n",
" i = 0\n",
" indexIDs = []\n",
" boxes = []\n",
" person_num = 0\n",
" conf = None\n",
" text_scale = max(1, image_w / 1600.)\n",
" text_thickness = 3\n",
" line_thickness = max(3, int(image_w/ 500.))\n",
" for det_box in det_boxes:\n",
" name, class_id, conf, box, track_id, keypoints = det_box.values()\n",
" indexIDs.append(track_id)\n",
" x1, y1, x2, y2 = box.values()\n",
" intbox = tuple(map(int, (x1, y1, x2, y2)))\n",
" textbox = tuple(map(int, (x1 - line_thickness, y1, x2 + line_thickness, y1 - 15)))\n",
" color = get_color(abs(int(track_id)))\n",
" cv2.rectangle(frame, textbox[0:2], textbox[2:4], color=color, thickness=-1)\n",
" cv2.rectangle(frame, intbox[0:2], intbox[2:4], color=color, thickness=line_thickness)\n",
" kx = keypoints[\"x\"]\n",
" ky = keypoints[\"y\"]\n",
" cv2.line(frame, (int(kx[skeleton[-2][0] - 1]), int(ky[skeleton[-2][0] - 1])), (int(kx[skeleton[-2][1] - 1]), int(ky[skeleton[-2][1] - 1])), tuple(limb_color[-2].tolist()), 2)\n",
" cv2.line(frame, (int(kx[skeleton[-1][0] - 1]), int(ky[skeleton[-1][0] - 1])), (int(kx[skeleton[-1][1] - 1]), int(ky[skeleton[-1][1] - 1])), tuple(limb_color[-1].tolist()), 2)\n",
" for i, (x, y) in enumerate(zip(kx, ky)):\n",
" cv2.line(frame, (int(kx[skeleton[i][0] - 1]), int(ky[skeleton[i][0] - 1])), (int(kx[skeleton[i][1] - 1]), int(ky[skeleton[i][1] - 1])), tuple(limb_color[i].tolist()), 2)\n",
" cv2.circle(frame, (int(x), int(y)), 3, tuple(kpt_color[i].tolist()), -1)\n",
" cv2.putText(frame, f\"ID: {str(track_id)} - {str(round(conf,4))}\", (intbox[0], intbox[1]), cv2.FONT_HERSHEY_PLAIN, text_scale, (255, 255, 255),thickness=2)\n",
" cv2.putText(frame, 'frame:{}'.format(frame_id), (int(25), int(25)),0, text_scale, (230,95,75),3)\n",
" i += 1\n",
" return frame"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "709a937c-6469-4483-8175-1dc395b5098f",
"metadata": {},
"source": [
"#### 2.2.2.2. Invoke endpoint\n",
"\n",
"Next, we download a [video](https://motchallenge.net/sequenceVideos/MOT17-09-FRCNN-raw.mp4) from MOT17 dataset to test our endpoint. We create a directory input for saving the processed result, and then download video to input directory with MP4 format."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a04c8591-88ef-425a-a719-f2d532253f6a",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"!mkdir -p input\n",
"!mkdir -p output\n",
"!cd input && wget \"https://motchallenge.net/sequenceVideos/MOT17-09-FRCNN-raw.mp4\" -O test.mp4"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "acf3803c-cb02-4878-9f21-4f311a0f7df6",
"metadata": {},
"source": [
"After preparing the test data, we invoke the endpoint to run the real time inferece on the test video."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ac2547ea-b4fb-4e83-88a5-123985955b1b",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"import os\n",
"import cv2\n",
"import json\n",
"import time\n",
"import base64\n",
"\n",
"sm_runtime = sagemaker_session.sagemaker_runtime_client\n",
"\n",
"data_path = \"input/test.mp4\" \n",
"cap = cv2.VideoCapture(data_path)\n",
"frame_w = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))\n",
"frame_h = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))\n",
"resize_factor = 1\n",
"\n",
"fourcc = cv2.VideoWriter_fourcc(*'MP4V')\n",
"file_path = f\"output/out-t-{time.localtime().tm_min}-{time.localtime().tm_sec}.mp4\" \n",
"out = cv2.VideoWriter(file_path, fourcc, 25, (int(frame_w / resize_factor), int(frame_h / resize_factor)))\n",
"\n",
"processing_time = 0\n",
"frame_id = 0\n",
"\n",
"\n",
"i = 0\n",
"while True:\n",
" ret, frame = cap.read()\n",
" if ret != True:\n",
" break\n",
" \n",
" if resize_factor == 1:\n",
" res = frame\n",
" else:\n",
" res = cv2.resize(frame, dsize=(int(frame_w / resize_factor), int(frame_h / resize_factor)), interpolation=cv2.INTER_CUBIC)\n",
" Body = {\"frame_id\": frame_id}\n",
" Body[\"frame_w\"] = int(frame_w / resize_factor)\n",
" Body[\"frame_h\"] = int(frame_h / resize_factor)\n",
" Body[\"frame_data\"] = base64.b64encode(res).decode(\"utf-8\")\n",
" \n",
" request_time=time.time()\n",
" body = json.dumps(Body).encode(\"utf-8\")\n",
" response = sm_runtime.invoke_endpoint(EndpointName=ep, Body=body, ContentType=\"application/json\")\n",
"\n",
" if frame_id > 0:\n",
" processing_time += (time.time() - request_time)\n",
" print(f'frame-{frame_id} Processing time: {(time.time() - request_time)}')\n",
" body = response[\"Body\"].read()\n",
" msg = body.decode(\"utf-8\")\n",
" data = json.loads(msg)\n",
" frame_res = draw_res(data[0], res, frame_id, int(frame_w / resize_factor))\n",
" out.write(frame_res)\n",
" frame_id += 1\n",
"\n",
"out.release()\n",
"cap.release()\n",
"print('average processing time: ', processing_time/frame_id)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "42a9b8d7-44e8-4a98-94b0-0732aa30719f",
"metadata": {},
"source": [
"#### 2.2.3. Cleanup resources"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0e82166d-bb4a-4377-9b60-0e6e99e5c9af",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"sagemaker_session.delete_model(model_name)\n",
"sagemaker_session.delete_endpoint_config(epc)\n",
"sagemaker_session.delete_endpoint(ep)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "b39e44ec-2d80-4607-80fe-0627c7ddd932",
"metadata": {},
"source": [
"### 2.3. Push tested image to ECR\n",
"Now that we built and tested our image, we can push it to ECR to be able to use it with SageMaker Endpoints"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b4559a54-bbf0-4f08-ba52-a58110614c2c",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"%%bash\n",
"\n",
"set -x\n",
"# This script shows how to build the Docker image and push it to ECR to be ready for use\n",
"# by SageMaker.\n",
"\n",
"image=\"yolov8-sagemaker-inference\"\n",
"tag=\"latest\"\n",
"region=$REGION_NAME\n",
"\n",
"# Get the account number associated with the current IAM credentials\n",
"account=$(aws sts get-caller-identity --query Account --output text)\n",
"\n",
"if [ $? -ne 0 ]\n",
"then\n",
" exit 255\n",
"fi\n",
"\n",
"\n",
"fullname=\"${account}.dkr.ecr.${region}.amazonaws.com/${image}:${tag}\"\n",
"\n",
"# If the repository doesn't exist in ECR, create it.\n",
"aws ecr describe-repositories --region ${region} --repository-names \"${image}\" > /dev/null 2>&1\n",
"if [ $? -ne 0 ]; then\n",
" aws ecr create-repository --region ${region} --repository-name \"${image}\" > /dev/null\n",
"fi\n",
"\n",
"\n",
"# Build the docker image locally with the image name and then push it to ECR\n",
"# with the full name.\n",
"\n",
"docker tag ${image}:${tag} ${fullname}\n",
"\n",
"# Get the login command from ECR and execute it directly\n",
"$(aws ecr get-login --region ${region} --no-include-email)\n",
"docker push ${fullname}\n",
"if [ $? -eq 0 ]; then\n",
"\techo \"Amazon ECR URI: ${fullname}\"\n",
"else\n",
"\techo \"Error: Image build and push failed\"\n",
"\texit 1\n",
"fi"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "ee22e159-68a3-4f12-9cf0-b02635a34d6f",
"metadata": {},
"source": [
"#### 2.3.1. Delete Docker Host\n",
"Make sure to delete docker host after finishing with local mode to avoid extra charges"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "27d0bb07-9075-419d-b9e6-8574086c6dae",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"!sdocker terminate-current-host"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "cff74aa3-c99d-41ca-8fd4-7b45e6103cd2",
"metadata": {},
"source": [
"### 2.4. Deploy on SageMaker Endpoint\n",
"Now that we pushed the inference image to ECR we can test it on SageMaker. First we package pretrained model into **.tar.gz** archive and upload to s3"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7c400171",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"sagemaker_session = sagemaker.session.Session(boto_session=session)\n",
"s3_bucket_name = sagemaker_session.default_bucket()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "41b3ae28-82e4-48af-966d-135d9f7c91aa",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"!cd inference-container/model && tar -czvf model.tar.gz yolov8n-pose.pt && rm yolov8n-pose.pt"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6f3d7c54-c3f6-4cd1-9bfc-fdac6fa628c4",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"!aws s3 cp inference-container/model/model.tar.gz s3://{s3_bucket_name}/yolov8/model/model.tar.gz"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "2430bfcd-4ec2-4000-a5c5-da2a245bac03",
"metadata": {},
"source": [
"#### 2.4.1. Define SageMaker Model"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "acf0c96e-31d3-418d-a745-018450b596aa",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"image = \"yolov8-sagemaker-inference\"\n",
"tag = \"latest\"\n",
"region = os.getenv(\"AWS_REGION\")\n",
"\n",
"sagemaker_session = sagemaker.session.Session(boto_session=session)\n",
"\n",
"s3_bucket_name = sagemaker_session.default_bucket()\n",
"\n",
"image_uri = f\"{account}.dkr.ecr.{region}.amazonaws.com/{image}:{tag}\"\n",
"model_name = \"yolov8-model-1\" # set the name of the model\n",
"\n",
"model_uri = f\"s3://{s3_bucket_name}/yolov8/model/model.tar.gz\" # define the local pretrained model URI manually.\n",
"\n",
"serving_container_def = {\n",
" 'Image': image_uri,\n",
" 'ModelDataUrl': model_uri,\n",
" 'Mode': 'SingleModel',\n",
" 'Environment': {\n",
" 'SM_MODEL_NAME' : 'yolov8n-pose.pt',\n",
" }\n",
"}\n",
"\n",
"create_model_response = sagemaker_session.create_model(name=model_name, \n",
" role=role, \n",
" container_defs=serving_container_def)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "5e7e0b13-f980-4a4f-896f-e829c366fca6",
"metadata": {},
"source": [
"#### 2.4.2. Define SageMaker Endpoint Configuration"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "03e51717-b8ad-4a68-9884-836421bd66f2",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"endpoint_config_name = f\"{model_name}-endpoint-config\"\n",
"epc = sagemaker_session.create_endpoint_config(\n",
" name=endpoint_config_name,\n",
" model_name=model_name,\n",
" initial_instance_count=1,\n",
" instance_type=\"ml.m5.2xlarge\",\n",
")\n",
"print(epc)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "38d44649-9489-4c57-b31a-9676ad271777",
"metadata": {},
"source": [
"#### 2.4.3. Create SageMaker Endpoint"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e4d95ce3-3c9a-44b4-89a0-cbba5d1c48b2",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"endpoint_name = f\"{model_name}-endpoint\"\n",
"ep = sagemaker_session.create_endpoint(\n",
" endpoint_name=endpoint_name, config_name=endpoint_config_name, wait=True\n",
")\n",
"print(ep)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "3ac2b013-16dc-405d-a110-3c459a5d9a50",
"metadata": {},
"source": [
"#### 2.4.3. Test Endpoint"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "49ec18cf-7755-42e1-b93b-75c774cd0bb4",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"import os\n",
"import cv2\n",
"import json\n",
"import time\n",
"import base64\n",
"\n",
"sm_runtime = sagemaker_session.sagemaker_runtime_client\n",
"\n",
"data_path = \"input/test.mp4\" \n",
"cap = cv2.VideoCapture(data_path)\n",
"frame_w = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))\n",
"frame_h = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))\n",
"resize_factor = 1\n",
"\n",
"fourcc = cv2.VideoWriter_fourcc(*'MP4V')\n",
"file_path = f\"output/out-t-{time.localtime().tm_min}-{time.localtime().tm_sec}.mp4\" \n",
"out = cv2.VideoWriter(file_path, fourcc, 25, (int(frame_w / resize_factor), int(frame_h / resize_factor)))\n",
"\n",
"processing_time = 0\n",
"frame_id = 0\n",
"\n",
"\n",
"i = 0\n",
"while True:\n",
" ret, frame = cap.read()\n",
" if ret != True:\n",
" break\n",
" \n",
" if resize_factor == 1:\n",
" res = frame\n",
" else:\n",
" res = cv2.resize(frame, dsize=(int(frame_w / resize_factor), int(frame_h / resize_factor)), interpolation=cv2.INTER_CUBIC)\n",
" Body = {\"frame_id\": frame_id}\n",
" Body[\"frame_w\"] = int(frame_w / resize_factor)\n",
" Body[\"frame_h\"] = int(frame_h / resize_factor)\n",
" Body[\"frame_data\"] = base64.b64encode(res).decode(\"utf-8\")\n",
" \n",
" request_time=time.time()\n",
" body = json.dumps(Body).encode(\"utf-8\")\n",
" response = sm_runtime.invoke_endpoint(EndpointName=ep, Body=body, ContentType=\"application/json\")\n",
"\n",
" if frame_id > 0:\n",
" processing_time += (time.time() - request_time)\n",
" print(f'frame-{frame_id} Processing time: {(time.time() - request_time)}')\n",
" body = response[\"Body\"].read()\n",
" msg = body.decode(\"utf-8\")\n",
" data = json.loads(msg)\n",
" frame_res = draw_res(data[0], res, frame_id, int(frame_w / resize_factor))\n",
" out.write(frame_res)\n",
" frame_id += 1\n",
"\n",
"out.release()\n",
"cap.release()\n",
"print('average processing time: ', processing_time/frame_id)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "814cd422-286b-4f3a-a402-16e827193574",
"metadata": {},
"source": [
"#### 2.4.4. Cleanup resources"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7f17a51e-e1c3-4804-ba0a-b3e17b2bcc04",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"sagemaker_session.delete_model(model_name)\n",
"sagemaker_session.delete_endpoint_config(epc)\n",
"sagemaker_session.delete_endpoint(ep)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "07972ee7-79a9-4402-aaa0-f6ef86be6c17",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"availableInstances": [
{
"_defaultOrder": 0,
"_isFastLaunch": true,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 4,
"name": "ml.t3.medium",
"vcpuNum": 2
},
{
"_defaultOrder": 1,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 8,
"name": "ml.t3.large",
"vcpuNum": 2
},
{
"_defaultOrder": 2,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 16,
"name": "ml.t3.xlarge",
"vcpuNum": 4
},
{
"_defaultOrder": 3,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 32,
"name": "ml.t3.2xlarge",
"vcpuNum": 8
},
{
"_defaultOrder": 4,
"_isFastLaunch": true,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 8,
"name": "ml.m5.large",
"vcpuNum": 2
},
{
"_defaultOrder": 5,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 16,
"name": "ml.m5.xlarge",
"vcpuNum": 4
},
{
"_defaultOrder": 6,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 32,
"name": "ml.m5.2xlarge",
"vcpuNum": 8
},
{
"_defaultOrder": 7,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 64,
"name": "ml.m5.4xlarge",
"vcpuNum": 16
},
{
"_defaultOrder": 8,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 128,
"name": "ml.m5.8xlarge",
"vcpuNum": 32
},
{
"_defaultOrder": 9,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 192,
"name": "ml.m5.12xlarge",
"vcpuNum": 48
},
{
"_defaultOrder": 10,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 256,
"name": "ml.m5.16xlarge",
"vcpuNum": 64
},
{
"_defaultOrder": 11,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 384,
"name": "ml.m5.24xlarge",
"vcpuNum": 96
},
{
"_defaultOrder": 12,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 8,
"name": "ml.m5d.large",
"vcpuNum": 2
},
{
"_defaultOrder": 13,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 16,
"name": "ml.m5d.xlarge",
"vcpuNum": 4
},
{
"_defaultOrder": 14,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 32,
"name": "ml.m5d.2xlarge",
"vcpuNum": 8
},
{
"_defaultOrder": 15,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 64,
"name": "ml.m5d.4xlarge",
"vcpuNum": 16
},
{
"_defaultOrder": 16,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 128,
"name": "ml.m5d.8xlarge",
"vcpuNum": 32
},
{
"_defaultOrder": 17,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 192,
"name": "ml.m5d.12xlarge",
"vcpuNum": 48
},
{
"_defaultOrder": 18,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 256,
"name": "ml.m5d.16xlarge",
"vcpuNum": 64
},
{
"_defaultOrder": 19,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 384,
"name": "ml.m5d.24xlarge",
"vcpuNum": 96
},
{
"_defaultOrder": 20,
"_isFastLaunch": false,
"category": "General purpose",
"gpuNum": 0,
"hideHardwareSpecs": true,
"memoryGiB": 0,
"name": "ml.geospatial.interactive",
"supportedImageNames": [
"sagemaker-geospatial-v1-0"
],
"vcpuNum": 0
},
{
"_defaultOrder": 21,
"_isFastLaunch": true,
"category": "Compute optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 4,
"name": "ml.c5.large",
"vcpuNum": 2
},
{
"_defaultOrder": 22,
"_isFastLaunch": false,
"category": "Compute optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 8,
"name": "ml.c5.xlarge",
"vcpuNum": 4
},
{
"_defaultOrder": 23,
"_isFastLaunch": false,
"category": "Compute optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 16,
"name": "ml.c5.2xlarge",
"vcpuNum": 8
},
{
"_defaultOrder": 24,
"_isFastLaunch": false,
"category": "Compute optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 32,
"name": "ml.c5.4xlarge",
"vcpuNum": 16
},
{
"_defaultOrder": 25,
"_isFastLaunch": false,
"category": "Compute optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 72,
"name": "ml.c5.9xlarge",
"vcpuNum": 36
},
{
"_defaultOrder": 26,
"_isFastLaunch": false,
"category": "Compute optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 96,
"name": "ml.c5.12xlarge",
"vcpuNum": 48
},
{
"_defaultOrder": 27,
"_isFastLaunch": false,
"category": "Compute optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 144,
"name": "ml.c5.18xlarge",
"vcpuNum": 72
},
{
"_defaultOrder": 28,
"_isFastLaunch": false,
"category": "Compute optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 192,
"name": "ml.c5.24xlarge",
"vcpuNum": 96
},
{
"_defaultOrder": 29,
"_isFastLaunch": true,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 16,
"name": "ml.g4dn.xlarge",
"vcpuNum": 4
},
{
"_defaultOrder": 30,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 32,
"name": "ml.g4dn.2xlarge",
"vcpuNum": 8
},
{
"_defaultOrder": 31,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 64,
"name": "ml.g4dn.4xlarge",
"vcpuNum": 16
},
{
"_defaultOrder": 32,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 128,
"name": "ml.g4dn.8xlarge",
"vcpuNum": 32
},
{
"_defaultOrder": 33,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 4,
"hideHardwareSpecs": false,
"memoryGiB": 192,
"name": "ml.g4dn.12xlarge",
"vcpuNum": 48
},
{
"_defaultOrder": 34,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 256,
"name": "ml.g4dn.16xlarge",
"vcpuNum": 64
},
{
"_defaultOrder": 35,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 61,
"name": "ml.p3.2xlarge",
"vcpuNum": 8
},
{
"_defaultOrder": 36,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 4,
"hideHardwareSpecs": false,
"memoryGiB": 244,
"name": "ml.p3.8xlarge",
"vcpuNum": 32
},
{
"_defaultOrder": 37,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 8,
"hideHardwareSpecs": false,
"memoryGiB": 488,
"name": "ml.p3.16xlarge",
"vcpuNum": 64
},
{
"_defaultOrder": 38,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 8,
"hideHardwareSpecs": false,
"memoryGiB": 768,
"name": "ml.p3dn.24xlarge",
"vcpuNum": 96
},
{
"_defaultOrder": 39,
"_isFastLaunch": false,
"category": "Memory Optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 16,
"name": "ml.r5.large",
"vcpuNum": 2
},
{
"_defaultOrder": 40,
"_isFastLaunch": false,
"category": "Memory Optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 32,
"name": "ml.r5.xlarge",
"vcpuNum": 4
},
{
"_defaultOrder": 41,
"_isFastLaunch": false,
"category": "Memory Optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 64,
"name": "ml.r5.2xlarge",
"vcpuNum": 8
},
{
"_defaultOrder": 42,
"_isFastLaunch": false,
"category": "Memory Optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 128,
"name": "ml.r5.4xlarge",
"vcpuNum": 16
},
{
"_defaultOrder": 43,
"_isFastLaunch": false,
"category": "Memory Optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 256,
"name": "ml.r5.8xlarge",
"vcpuNum": 32
},
{
"_defaultOrder": 44,
"_isFastLaunch": false,
"category": "Memory Optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 384,
"name": "ml.r5.12xlarge",
"vcpuNum": 48
},
{
"_defaultOrder": 45,
"_isFastLaunch": false,
"category": "Memory Optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 512,
"name": "ml.r5.16xlarge",
"vcpuNum": 64
},
{
"_defaultOrder": 46,
"_isFastLaunch": false,
"category": "Memory Optimized",
"gpuNum": 0,
"hideHardwareSpecs": false,
"memoryGiB": 768,
"name": "ml.r5.24xlarge",
"vcpuNum": 96
},
{
"_defaultOrder": 47,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 16,
"name": "ml.g5.xlarge",
"vcpuNum": 4
},
{
"_defaultOrder": 48,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 32,
"name": "ml.g5.2xlarge",
"vcpuNum": 8
},
{
"_defaultOrder": 49,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 64,
"name": "ml.g5.4xlarge",
"vcpuNum": 16
},
{
"_defaultOrder": 50,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 128,
"name": "ml.g5.8xlarge",
"vcpuNum": 32
},
{
"_defaultOrder": 51,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 1,
"hideHardwareSpecs": false,
"memoryGiB": 256,
"name": "ml.g5.16xlarge",
"vcpuNum": 64
},
{
"_defaultOrder": 52,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 4,
"hideHardwareSpecs": false,
"memoryGiB": 192,
"name": "ml.g5.12xlarge",
"vcpuNum": 48
},
{
"_defaultOrder": 53,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 4,
"hideHardwareSpecs": false,
"memoryGiB": 384,
"name": "ml.g5.24xlarge",
"vcpuNum": 96
},
{
"_defaultOrder": 54,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 8,
"hideHardwareSpecs": false,
"memoryGiB": 768,
"name": "ml.g5.48xlarge",
"vcpuNum": 192
},
{
"_defaultOrder": 55,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 8,
"hideHardwareSpecs": false,
"memoryGiB": 1152,
"name": "ml.p4d.24xlarge",
"vcpuNum": 96
},
{
"_defaultOrder": 56,
"_isFastLaunch": false,
"category": "Accelerated computing",
"gpuNum": 8,
"hideHardwareSpecs": false,
"memoryGiB": 1152,
"name": "ml.p4de.24xlarge",
"vcpuNum": 96
}
],
"instance_type": "ml.t3.medium",
"kernelspec": {
"display_name": "Python 3 (Data Science)",
"language": "python",
"name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:ap-southeast-2:452832661640:image/datascience-1.0"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.10"
},
"toc-autonumbering": false,
"toc-showtags": false
},
"nbformat": 4,
"nbformat_minor": 5
}