{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# [모듈 3.7] 추론 및 결과 분석\n", "\n", "이 노트북은 이전 노트북에서 생성한 앤드포인트를 대상으로 실제 추론을 하여, 평가 결과를 얻습니다. 테스트 데이터 세트는 1000개를 사용하여 결과를 확인 합니다.\n", "- 0. 기본 세이지 메이커 정보 및 기본 변수 로딩\n", "- 1. 테스트 데이터 세트 로딩\n", "- 2. 추론할 Payload 생성\n", "- 3. 추론 실행\n", "- 4. 예측의 확률값을 0, 1로 변화하여 리포팅\n", "- 5. 엔드포인트 삭제\n", "\n", "---\n", "### 노트북 커널\n", "- 이 워크샵은 노트북 커널이 `conda_python3` 를 사용합니다. 다른 커널일 경우 변경 해주세요.\n", "---\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 0. 기본 세이지 메이커 정보 및 기본 변수 로딩" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import boto3\n", "import sagemaker\n", "import pandas as pd\n", "\n", "region = boto3.Session().region_name\n", "sagemaker_session = sagemaker.session.Session()\n", "role = sagemaker.get_execution_role()\n", "\n", "%store -r \n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "아래는 사용자 정의의 함수를 로딩 할때마다, 캐시에 있는 것보다는 원본을 로딩함." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "%load_ext autoreload\n", "%autoreload 2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. 테스트 데이터 세트 로딩" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "

\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "

	fraud	vehicle_claim	total_claim_amount	customer_age	months_as_customer	num_claims_past_year	num_insurers_past_5_years	policy_deductable	policy_annual_premium	customer_zip	...	collision_type_missing	incident_severity_Major	incident_severity_Minor	incident_severity_Totaled	authorities_contacted_Ambulance	authorities_contacted_Fire	authorities_contacted_None	authorities_contacted_Police	police_report_available_No	police_report_available_Yes
0	0	23901.432507	36201.432507	56	118	0	1	750	3000	98039	...	0	0	0	1	0	0	0	1	0	1
1	0	29095.295736	73695.295736	36	71	0	1	750	2650	95307	...	0	0	0	1	0	0	0	1	0	1
2	0	11232.831826	25032.831826	24	11	1	4	750	3000	98391	...	0	0	1	0	0	0	1	0	1	0
3	0	11125.579356	25125.579356	23	65	0	1	750	3000	91302	...	0	0	1	0	0	0	1	0	1	0
4	0	4529.773913	22329.773913	37	142	0	1	750	3000	90744	...	0	0	1	0	0	0	0	1	1	0

\n", "

5 rows × 59 columns

\n", "

" ], "text/plain": [ " fraud vehicle_claim total_claim_amount customer_age months_as_customer \\\n", "0 0 23901.432507 36201.432507 56 118 \n", "1 0 29095.295736 73695.295736 36 71 \n", "2 0 11232.831826 25032.831826 24 11 \n", "3 0 11125.579356 25125.579356 23 65 \n", "4 0 4529.773913 22329.773913 37 142 \n", "\n", " num_claims_past_year num_insurers_past_5_years policy_deductable \\\n", "0 0 1 750 \n", "1 0 1 750 \n", "2 1 4 750 \n", "3 0 1 750 \n", "4 0 1 750 \n", "\n", " policy_annual_premium customer_zip ... collision_type_missing \\\n", "0 3000 98039 ... 0 \n", "1 2650 95307 ... 0 \n", "2 3000 98391 ... 0 \n", "3 3000 91302 ... 0 \n", "4 3000 90744 ... 0 \n", "\n", " incident_severity_Major incident_severity_Minor \\\n", "0 0 0 \n", "1 0 0 \n", "2 0 1 \n", "3 0 1 \n", "4 0 1 \n", "\n", " incident_severity_Totaled authorities_contacted_Ambulance \\\n", "0 1 0 \n", "1 1 0 \n", "2 0 0 \n", "3 0 0 \n", "4 0 0 \n", "\n", " authorities_contacted_Fire authorities_contacted_None \\\n", "0 0 0 \n", "1 0 0 \n", "2 0 1 \n", "3 0 1 \n", "4 0 0 \n", "\n", " authorities_contacted_Police police_report_available_No \\\n", "0 1 0 \n", "1 1 0 \n", "2 0 1 \n", "3 0 1 \n", "4 1 1 \n", "\n", " police_report_available_Yes \n", "0 1 \n", "1 1 \n", "2 0 \n", "3 0 \n", "4 0 \n", "\n", "[5 rows x 59 columns]" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import os\n", "test_preproc_dir_artifact_file= os.path.join(test_preproc_dir_artifact, 'test.csv')\n", "test_df = pd.read_csv(test_preproc_dir_artifact_file)\n", "test_df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. 추론할 Payload 생성\n", "\n", "- 추론에 입력될 CST String 형태의 데이터로 변경 (payload)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "payload_list: \n", " ['23901.43250714285,36201.43250714285,56,118,0,1,750,3000,98039,2012,3,3,1,12300,9,12,3,14,0,0,0,0,0,1,0,0,1,0,0,1,0,0,0,0,0,0,1,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,1']\n" ] } ], "source": [ "def get_payload_list(test_df, label_col):\n", " '''\n", " CSV String 행태로 변경\n", " '''\n", " s = test_df.drop(columns=[label_col])\n", " s = s.to_csv(header=None, index=None)\n", " payload_list = s.splitlines()\n", " return payload_list\n", "\n", "payload_list = get_payload_list(test_df, label_col='fraud')\n", "y_test = test_df['fraud'].tolist()\n", "\n", "print(\"payload_list: \\n\", payload_list[0:1]) # 샘플로 5개의 리스트 보여줌.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. 추론 실행\n", "- 앤드포인트가 InServie로 유효하면 추론을 하여, 확률값을 제공함.\n", "- 위에서 준비한 테스트 데이터 세트에서 실험적으로 5개의 샘플만을 추론을 함." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "pred_prob_list: \n", " [0.10092484205961227, 0.08251918852329254, 0.4293206036090851, 0.23539546132087708, 0.669212281703949]\n", "CPU times: user 2 s, sys: 53.8 ms, total: 2.06 s\n", "Wall time: 8.65 s\n" ] } ], "source": [ "%%time\n", "\n", "import numpy as np\n", "import boto3\n", "sagemaker_boto_client = boto3.client('sagemaker')\n", "\n", "from sagemaker.deserializers import CSVDeserializer\n", "csv_deserializer = CSVDeserializer(accept='text/csv') # 디폴트가 accept가 'text/csv' 이지만 직관적 이유로 기술함.\n", "\n", "from src.p_utils import is_available_endpoint, get_predictor, get_payload, predict, delete_endpoint\n", "\n", "pipeline_predictor = get_predictor(all_pipeline_endpoint_name, sagemaker_session, csv_deserializer)\n", "pred_prob_list = []\n", "payload_list = get_payload_list(test_df, label_col='fraud')\n", "for payload in payload_list:\n", " pred_prob = predict(pipeline_predictor, payload)\n", " pred_prob_list.append(float(pred_prob[0]))\n", "\n", "\n", "print(\"pred_prob_list: \\n\", pred_prob_list[0:5])\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. 예측의 확률값을 0, 1로 변화하여 리포팅\n", "- threshold 값이 보다 작으면 0, 크면 1로 할당한후에, Confustion Matrix로 보여줌" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "import itertools\n", "def plot_conf_mat(cm, classes, title, cmap = plt.cm.Greens):\n", " plt.imshow(cm, interpolation='nearest', cmap=cmap)\n", " plt.title(title)\n", " plt.colorbar()\n", " tick_marks = np.arange(len(classes))\n", " plt.xticks(tick_marks, classes, rotation=45)\n", " plt.yticks(tick_marks, classes)\n", "\n", " fmt = 'd'\n", " thresh = cm.max() / 2.\n", " for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n", " plt.text(j, i, format(cm[i, j], fmt),\n", " horizontalalignment=\"center\",\n", " color=\"black\" if cm[i, j] > thresh else \"black\")\n", "\n", " plt.tight_layout()\n", " plt.ylabel('True label')\n", " plt.xlabel('Predicted label')\n", " plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 결과 해석\n", "\n", "아래는 하나의 Confustion 행렬의 예로서, 해석의 방법은 아래와 같습니다.\n", "- 전체 1000 개의 테스트 데이터 세트 이고, 이중에 33개 (7+26) 가 fraud 임. \n", "- 33개 중에서 26개를 잘 예측 했음 (fraud:1 에 대한 recall 이 0.79, 26/33). (True Positive, TP)\n", "- 정상인 967 개 중에서 697개를 잘 예측 했음(fraud:0 에 대한 recall 이 0.72, 697/967). (True Negative, TN)\n", "- 실제 fraud 이지만, 예측을 정상으로 한것이 7개 (False Negative, FN)\n", "- 실제 정상이지만, fraud 로 예측한 것이 270개 (False Positive, FP)\n", " - 조금 더 상세한 의미는 인터넷에 많이 있습니다. 하나의 참조입니다. --> [분류 성능 평가](https://datascienceschool.net/03%20machine%20learning/09.04%20분류%20성능평가.html) \n", "\n", "```\n", " legit fraud\n", "legit 697 270\n", "fraud 7 26\n", "```" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUIAAAEYCAYAAAApuP8NAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAj8UlEQVR4nO3deZxU1Z3+8c/TDQIKIsgqSEQDGtCAiriNxiXuG5kExSVBx4zjT42JMYtm0UQl40ySSTRqItFJSExU1BhJNG5ER40ruEVxgcQNQRAEFUHZvr8/7mkp2+6qAutSXV3P21e9uurcc+893W1/Ods9RxGBmVk9a6h2AczMqs2B0MzqngOhmdU9B0Izq3sOhGZW9xwIzazuORAaAJK6SPqTpDclXfcRrnOspNsrWbZqkPQXSeOrXQ5bPxwIa4ykYyRNk7RE0tz0B/svFbj054C+wKYRMXZdLxIRv4uI/StQng+QtJekkPSHZukjUvrdZV7ne5KuKpUvIg6KiEnrWFyrMQ6ENUTSV4GfAj8gC1qDgMuAIypw+Y8Bz0fEygpcKy+vA7tJ2rQgbTzwfKVuoIz/LupNRPhVAy+gO7AEGFskTyeyQDknvX4KdErH9gJmA2cC84G5wAnp2PeB5cCKdI8Tge8BVxVcewsggA7p8/HAP4G3gReAYwvS7ys4bzfgEeDN9HW3gmN3A+cDf0vXuR3o1cr31lT+XwCnprTGlHYOcHdB3ouAV4C3gOnAHin9wGbf5xMF5ZiQyrEM+HhK+2I6/nPg+oLr/xcwFVC1/7/wqzIv/8tXO3YFOgM3FsnzbWAXYCQwAhgNfKfgeD+ygDqALNhdKqlHRJxLVsu8NiK6RsSVxQoiaSPgYuCgiOhGFuwebyFfT+DmlHdT4H+Am5vV6I4BTgD6ABsAXyt2b+A3wBfS+wOAp8mCfqFHyH4GPYHfA9dJ6hwRtzb7PkcUnPN54CSgG/BSs+udCXxS0vGS9iD72Y2PFBWt9jkQ1o5NgQVRvOl6LHBeRMyPiNfJanqfLzi+Ih1fERG3kNWKtl7H8qwGtpXUJSLmRsTTLeQ5BJgZEb+NiJURcTXwLHBYQZ5fRcTzEbEMmEwWwFoVEfcDPSVtTRYQf9NCnqsiYmG654/Jasqlvs9fR8TT6ZwVza63FDiOLJBfBXwpImaXuJ7VEAfC2rEQ6CWpQ5E8m/HB2sxLKe39azQLpEuBrmtbkIh4BzgKOBmYK+lmSduUUZ6mMg0o+PzaOpTnt8BpwN60UEOWdKakZ9II+GKyWnCvEtd8pdjBiHiYrCtAZAHb2hEHwtrxAPAuMKZInjlkgx5NBvHhZmO53gE2LPjcr/BgRNwWEfsB/clqeb8sozxNZXp1HcvU5LfAKcAtqbb2vtR0/SZwJNAjIjYh659UU9FbuWbRZq6kU8lqlnOAb6xzya1NciCsERHxJtmgwKWSxkjaUFJHSQdJ+u+U7WrgO5J6S+qV8pecKtKKx4E9JQ2S1B04u+mApL6SDk99he+RNbFXtXCNW4ChacpPB0lHAcOAP69jmQCIiBeAT5H1iTbXDVhJNsLcQdI5wMYFx+cBW6zNyLCkocAFZM3jzwPfkDRy3UpvbZEDYQ2JiP8Bvko2API6WXPuNOCPKcsFwDTgSeDvwKMpbV3udQdwbbrWdD4YvBrIBhDmAG+QBaVTWrjGQuDQlHchWU3q0IhYsC5lanbt+yKipdrubcBfyKbUvERWiy5s9jZNFl8o6dFS90ldEVcB/xURT0TETOBbwG8ldfoo34O1HfLAl5nVO9cIzazuORCaWd1zIDSzuudAaGZ1r9jk3JqnDRqCzu36W6xJWwzqX+0iWDMLXl3I24uWqHTO8qlX52D56tIZ315xW0QcWMl7r632HSU6d4Cd+1S7FNbMuZe0NP3Pqun7n51Q+YsuX13e39+dr5Z66id37TsQmln1CGisaCUzNw6EZpYfORCaWb2rjTjoQGhmeZFrhGZW59xHaGaGm8ZmVucENNRGJHQgNLP81EYcdCA0sxy5Rmhmdc1NYzMz3DQ2s3rneYRmVu88j9DMDNcIzczcR2hm9c2jxmZmOBCamdXKrkgOhGaWD3n6jJmZB0vMzDyP0Mzqm3DT2MzMTWMzsxqZPlMjg9tmVnOaJlSXepVzKWkTSddLelbSM5J2ldRT0h2SZqavPQryny1plqTnJB1Q6voOhGaWEyGVfpXpIuDWiNgGGAE8A5wFTI2IIcDU9BlJw4BxwHDgQOAySY3FLu5AaGa5qUQglLQxsCdwJUBELI+IxcARwKSUbRIwJr0/ArgmIt6LiBeAWcDoYvdwIDSz3DTNqS72AnpJmlbwOqnZZbYEXgd+JekxSVdI2gjoGxFzAdLXPin/AOCVgvNnp7RWebDEzHIhQWND6brWKlgQEaOKZOkA7AB8KSIeknQRqRnc2q1bSItiZXCN0MxyU6E+wtnA7Ih4KH2+niwwzpPUP92nPzC/IP/mBecPBOYUu4EDoZnlpDKDJRHxGvCKpK1T0r7ADGAKMD6ljQduSu+nAOMkdZI0GBgCPFzsHm4am1kuBDRUbh7hl4DfSdoA+CdwAllFbrKkE4GXgbEAEfG0pMlkwXIlcGpErCp2cQdCM8uHWJvpMUVFxONAS/2I+7aSfwIwodzrOxCaWW5UI8/YORCaWW4qVSPMmwOhmeVCiMYaedbYgdDMcuMaoZnVtwoOluTNgdDMclMjcdCB0Mzykc0jrI1nNhwIzSwna7XMVlU5EJpZPtxHaGttxWp4ZhEsWZl9HrYJNDbAs4tgZUCXDrBtD+jQAHOXwktL1py7ZAXs3Bu6bVCVordXC+e+wRXf/BVvLngLNYhPHbkH+39hXy47YyKvvTAPgKVvLWPDjbtw3h+/C8CfL/8L997wNxoaGjjm20ex3R7Dq/ktVF2NxEEHwjbj+cWwaWf45EawOmBVwGMLYEh36NEJXn0nC35bbQz9N8xekAXBJxY6COagsbGRo745li2GD2LZknf5/mcnMHy3T3DKT9Ysl3fNhdfRpVsXAF6dNYeHb5nGBX8+l8Xz3+SHJ/yEC289n4bG2ugnq7Ra6iOsjVK2dytXw6LlsFkKbg2Cjg3wzkrYJAW4TTvB/GUfPve1pdC3y/orax3ZpE93thg+CIAuXTvTf6v+LJ63+P3jEcHDt05n50N2AuCxqU8w+uBRdNygI70H9qLPoD7888kXqlH0NqNBKvlqCxwI24JlK2GDBpixGB6cDzMWwarV0LUjvP5ulmfeMni3hQU05i2Dfhuu1+LWowWzF/DyMy+z5YjB76c9P20m3TftRr8t+gKwaN5ievZ/f/8gevbrwaKCwFl3yliduo3EwfUbCCUtKZ2r1XPPk/Tp9P4rktrPX38Ab6+AgRvBLn2gUfDikqyfcPY78ND8rKnc/Lf15vKs9ti1YxUKXT/efeddLjn9co4++0i6dF1T+37o5kfY+ZDCrTA+vAhyrQwW5EGV3bwpVzVTI4yIcyLizvTxK0D7CYSdGrNX99QM7tMF3loOG3WEHXrBzn2y5m+XZl26ry11bTBnK1es4pLTL2fXw0Yzav8d3k9ftXIV0+94jNEHr1kZqkffHrwxd9H7n994bRGb9Om+Xsvb1jSooeSrLahaKSR9XdIjkp6U9P2C9O+mvUvvkHS1pK+l9F9L+pyk04HNgLsk3VWt8ldUp0bo3AjvrMg+v/FeVstbnprCEfDC2zBgozXnRGR9hu4fzE1E8Kvv/IbNturHASfs94FjMx54hv6D+9Gz35qm8Pb7jODhW6axYvkKXp+9gPkvzWfLTw5uftm6Uis1wqqMGkvan2z57NFkg0tTJO0JLAU+C2yfyvYoML3w3Ii4WNJXgb0jYkEL1z4JyIb1OhfdyrRt2bo7PLUoC3BdOsCwHtk0mdmpN6F3lzWDKZANrnRqhA098J+XmY/+g/tvepCBQwdwzpjzAfjsGWMY8anteOjmaex86E4fyD9gyGbsdNCOfPuQ79HY2Mhx5xxdtyPGTdpInCupWn9F+6fXY+lzV7LA2A24KSKWAUj609peOCImAhMBtPEGRXeualO6bZA1gQsN6pq9WtKzE4zu0/Ixq4ihO36cXz17eYvHvnjh8S2mH3bywRx28sE5lqp2yBOqSxLwnxHxgf/LJJ1RpfKYWcXJ8whLuA34N0ldASQNkNQHuA84TFLndOyQVs5/m6z2aGZtWK1Mn6lKjTAibpf0CeCBVHVeAhwXEY9ImgI8AbwETAPebOESE4G/SJobEXuvr3Kb2dqpVNNY0otkFaBVwMqIGCWpJ3AtsAXwInBkRCxK+c8GTkz5T4+I24pdf73WCCOia8H7iyJiu/TaNSL+kQ79KCK2BsYAW5MGSyLi+Ii4Pr3/WURs4yBo1nY19RFWcNR474gYGRFNc5bOAqZGxBBgavqMpGHAOGA4cCBwmaSiI6dtsQE/UdLjZCPGN0TEo1Uuj5mto4YGlXx9BEcAk9L7SWSVp6b0ayLivYh4AZhFNkOlVW1u7kVEHFPtMphZJVR0nmAAt0sK4PI0O6RvRMwFiIi5aZwBYADwYMG5s1Naq9pcIDSz9qPMQNhL0rSCzxNToCu0e0TMScHuDknPFrttC2lFp9I5EJpZLtZiHuGCgn6/FkXEnPR1vqQbyZq68yT1T7XB/sD8lH02sHnB6QOBOcWu3xb7CM2snahEH6GkjSR1a3pP9jDGU8AUYHzKNh64Kb2fAoyT1EnSYLKHNR4udg/XCM0sP5XpI+wL3Jhqlx2A30fErZIeASZLOhF4GRgLEBFPS5oMzABWAqdGRAtr2K3hQGhmOanMYElE/BMY0UL6QmDfVs6ZAEwo9x4OhGaWC4mPOj1mvXEgNLPceNEFM6t7DoRmVvdqJA46EJpZPqTaWYbLgdDMcuOmsZnVPQdCM6tvbWjh1VIcCM0sF6qhpfodCM0sN24am1ndq5E46EBoZjnxdp5mZiD3EZpZPRNQI2suOBCaWV4qumdJrhwIzSwfgoZaD4SSfkaRDU8i4vRcSmRm7YKAxloPhMC0IsfMzEqq+RphREwq/Cxpo4h4J/8imVl7IGpn+kzJsW1Ju0qaATyTPo+QdFnuJTOzGicaVPrVFpQzyeenwAHAQoCIeALYM8cymVk7IGV9hKVebUFZsx0j4pVmSUW3xjMzAypWI5TUKOkxSX9On3tKukPSzPS1R0HesyXNkvScpAPKKmcZeV6RtBsQkjaQ9DVSM9nMrBhJJV9l+jIfjDtnAVMjYggwNX1G0jBgHDAcOBC4TFJjqYuXEwhPBk4FBgCvAiPTZzOzVmVPlnz0GqGkgcAhwBUFyUcATQO6k4AxBenXRMR7EfECMAsYXeoeJSdUR8QC4NiSpTUz+4Cy+wB7SSqcrjcxIiYWfP4p8A2gW0Fa34iYCxARcyX1SekDgAcL8s1OaUWVDISStgQuAnYhm2D9AHBG2n3ezKxFKv/JkgURMarla+hQYH5ETJe0Vzm3bSGt1QdDmpTTNP49MBnoD2wGXAdcXcZ5ZlbnKtBHuDtwuKQXgWuAfSRdBcyT1D/doz8wP+WfDWxecP5AYE6pm5QTCBURv42Ilel1FWVEWDOzj9pHGBFnR8TAiNiCbBDkrxFxHDAFGJ+yjQduSu+nAOMkdZI0GBgCPFyqnMWeNe6Z3t4l6SyyaBzAUcDNpS5sZvUt52eNLwQmSzoReBkYCxART0uaDMwAVgKnRkTJ6X7F+ginkwW+pu/kPwqOBXD+2pfdzOpHZZ8ciYi7gbvT+4XAvq3kmwBMWJtrF3vWePDaXMjMrJDa21L9krYFhgGdm9Ii4jd5FcrM2oe28ghdKeVMnzkX2IssEN4CHATcBzgQmlmrmiZU14JyRo0/R9YWfy0iTgBGAJ1yLZWZtQu1svpMOU3jZRGxWtJKSRuTzdfZMudymVnNa197lkyTtAnwS7KR5CWUMS/HzOpb0zJctaCcZ41PSW9/IelWYOOIeDLfYplZe9BWmr6lFJtQvUOxYxHxaD5FMrP2oJYGS4rVCH9c5FgA+1S4LBW3w9Bt+dut91W7GNbMytUrql0Ea+biTj/P5bo130cYEXuvz4KYWXsjGlXWIvhV5w3ezSwXa7EMV9U5EJpZbtTi8oBtjwOhmeWm5vsImyj7To4FtoyI8yQNAvpFhOcSmlmrhGgsvW9Sm1BOT+ZlwK7A0enz28CluZXIzNqN9vSI3c4RsYOkxwAiYpGkDXIul5m1A+2maQysSPuCBoCk3sDqXEtlZjVP6b9aUE4gvBi4EegjaQLZajTfybVUZlb7BI0N7WQeYUT8TtJ0sqW4BIyJiGdKnGZmdU5AQ1nDENVXzqjxIGAp8KfCtIh4Oc+CmVmta1/LcN3Mmk2cOgODgeeA4TmWy8zagUoEQkmdgXvIFoTuAFwfEeemnTavBbYAXgSOjIhF6ZyzgROBVcDpEXFbsXuU0zTerlmhduCDO9qZmX1Itp1nRZrG7wH7RMQSSR2B+yT9BfhXYGpEXJi2HD4L+KakYWR7IA8HNgPulDS02Laea13KtPzWTuvwzZhZPVFl5hFGZkn62DG9AjgCmJTSJwFj0vsjgGsi4r2IeAGYBYwudo9y+gi/WvCxAdgBeL1k6c2szpU9faaXpGkFnydGxMQPXCmbwjcd+DhwaUQ8JKlvRMwFiIi5kvqk7AOABwtOn53SWlVOH2G3gvcryfoMbyjjPDOrY9nCrGU1OhdExKhiGVKzdmTaNuTGtMVwsVt/6BLFrl80EKYo3DUivl4sn5lZS8oMhGWLiMWS7gYOBOZJ6p9qg/3JNpaDrAa4ecFpA4E5RcvZ2gFJHVIUbnXJfjOz1pXuHyynj1BS71QTRFIX4NPAs8AUYHzKNh64Kb2fAoyT1EnSYGAIJTacK1YjfJgsCD4uaQpwHfBO08GI+EPJ78DM6pao2HqE/YFJqYXaAEyOiD9LegCYLOlE4GVgLEBEPC1pMjCDrDvv1GIjxlBeH2FPYCHZHiVN8wkDcCA0s6IqsbpM2jVz+xbSF5I98dbSOROACeXeo1gg7JNGjJ9iTQB8/z7l3sDM6pREQ42sR1gsEDYCXVmHERgzs/aynefciDhvvZXEzNqd9vCscW18B2bWJlXwEbvcFQuELXZCmpmVR6jWA2FEvLE+C2Jm7U9DjTQsvZ2nmeVCah99hGZmH4Eq/ohdXhwIzSw3bhqbWV0T1P5giZnZR9O+tvM0M1sn7iM0s7rnUWMzq2vZvsYOhGZWz9S+9jU2M1sn7WEZLjOzdVbBFapz50BoZjkpb0+StsCB0Mxy4xqhmdW1bIVq9xGaWV2rnVHj2pj2bWY1qQGVfJUiaXNJd0l6RtLTkr6c0ntKukPSzPS1R8E5Z0uaJek5SQeULqeZWR7SeoSlXmVYCZwZEZ8AdgFOlTQMOAuYGhFDgKnpM+nYOGA4cCBwWdoTuVUOhG3Y8889z8477vL+q0+PfvzsokuqXay6NPuV2Ry836HsuN1odhqxC5f97OfvH/vFpZez/fBR7DRiF75z1jlVLGXbkvURNpR8lRIRcyPi0fT+beAZYABwBDApZZsEjEnvjwCuiYj3IuIFYBYwutg93EfYhg3deigPTX8QgFWrVrHVoI9z+JjDq1yq+tShQwd+8N8XMHL7kbz99tvssfNe7LPv3syfP5+b/3QLDz76Nzp16sTr81+vdlHbEKHy6lq9JE0r+DwxIia2eEVpC7LN3h8C+kbEXMiCpaQ+KdsA4MGC02antFY5ENaIu6bexeAtt+RjHxtU7aLUpX79+9Gvfz8AunXrxtbbDGXOnLn8+spJfPXrZ9CpUycAevfpXc1itjllziNcEBGjSmWS1BW4AfhKRLxVpFm91nuxu2lcI66bfD1Hjhtb7WIY8NKLL/HkE39n1OgdmTVzFvffdz97774vB+57MNOnPVrt4rUZTU+WlPqvrGtJHcmC4O8i4g8peZ6k/ul4f2B+Sp8NbF5w+kBgTrHr5xYIJZ2eRnl+V+Hr7iXpz5W8Zlu3fPlybv7TLfzr5z5T7aLUvSVLlnDcUV/gwh/9gI033piVK1exePFi/nrfnVxw4fmMP+Z4IopWPuqIKtJHqKzqdyXwTET8T8GhKcD49H48cFNB+jhJnSQNBoYADxe7R55N41OAg1JnJQCSOkTEyhzv2S7dduvtjNx+BH379q12UeraihUrOO6oL3Dk0WM54jNZX+2AgZtx+JjDkMSonXakoaGBBQsW0rt3ryqXtm2o0DzC3YHPA3+X9HhK+xZwITBZ0onAy8BYgIh4WtJkYAbZiPOpEbGq2A1yCYSSfgFsCUyRNAi4FtgCWCDpW8BvgY1S9tMi4n5JewFfi4hD0zUuAaZFxK8lHQj8FFgA1F3bY/I117lZXGURwaknncbW2wzlS1857f30Qw8/hP+76x72+NQezHx+FsuXr6BXr02rWNK2I1uP8KM3OiPiPlru9wPYt5VzJgATyr1HLoEwIk5OwWtv4DTgMOBfImKZpA2B/SLiXUlDgKuBVjtKJXUGfgnsQzYMfm2xe0s6CTgJYPNBmxfLWhOWLl3KX+/8K5f8/OJqF6WuPXD/g1z9u2sZvu0wdhv1LwCce/45fP744zjl309j9Mhd2WCDjlx+5WU18zRF7ryv8YdMiYhl6X1H4BJJI4FVwNAS524DvBARMwEkXUUKdC1Jw+4TAXYctUPNd9ZsuOGGvDr/lWoXo+7ttvuuvL18cYvHrpjU4kwP877GH/JOwfszgHnACLLBmndT+ko+OHjTueB9zQc0s3pUK6vPVCNcdwfmRsRqsg7QpkdfXgKGpZGe7qxp+z8LDJa0Vfp89HotrZmtk0pOn8lbNSZUXwbcIGkscBepthgRr6SRnieBmcBjKf3d1O93s6QFwH3AtlUot5mtFXmD94jYIr39XrP0mcAnC5LOLjj2DeAbLVzrVrK+QjOrIR4sMbO611aavqU4EJpZbhwIzayuydNnzMxcIzSzeucnS8zMXCM0szrnPkIzM1wjNDNzH6GZmWuEZlbX3EdoZga0vrB02+JAaGb58DxCM7Pa6SOsjQa8mdWcpj7CCmzn+b+S5kt6qiCtp6Q7JM1MX3sUHDtb0ixJz0k6oJyyOhCaWW4qtEL1r4EDm6WdBUyNiCHA1PQZScOAccDwdM5lkhopwYHQzHJTiUAYEfcAbzRLPgKYlN5PAsYUpF8TEe+lPdVnAaNL3cN9hGaWmzIHS3pJmlbweWLajbKYvhExFyAi5krqk9IHAA8W5Jud0opyIDSzXKzFPMIFEdHq3uZrfdsPK7kLppvGZpabHHexmyepP0D6Oj+lzwY2L8g3EJhT6mIOhGaWI5XxWidTgPHp/XjgpoL0cWlb4MHAEODhUhdz09jMclOJWYSSrgb2IutLnA2cC1wITJZ0IvAyMBYgIp5O2wLPAFYCp0bEqlL3cCA0s9xUYl/jiDi6lUP7tpJ/AjBhbe7hQGhmuamN50ocCM0sNx+pD3C9ciA0s1zIiy6YmdXOogsOhGaWm1oJhJ5HaGZ1zzVCM8uN+wjNrM59pEfo1isHQjPLkQOhmdWx2plF6EBoZjmqxCN264MDoZnlxjVCM7MaCYUOhGaWE3n6jJnVt2ywxIHQzOqeA6GZ1bnaCIMOhGaWI/cRmlmd8yN2ZmbUSuPYgdDM8lFDK1TXxvMvZmY5UkRUuwy5kfQ68FK1y1EhvYAF1S6EfUB7+p18LCJ6V/KCkm4l+xmVsiAiDqzkvddWuw6E7YmkaRExqtrlsDX8O2k/3DQ2s7rnQGhmdc+BsHZMrHYB7EP8O2kn3EdoZnXPNUIzq3sOhGZW9xwIa5RqZcp+HVKtbNRh7/MvrIZI6lTwsXvVCmIfImmYpJ9L6hARq/0PVW1xIKwR6Q9rrKQjJe0NXC+pm//gqi/VAAV0An4kqTEiwr+b2uFR4xogqSewCmgEniH7ozs0Ih6WpPAvsWoKf/6SPgv8O/A48O2IWOXfT21wjbCNk9QZ+AqwaUS8AVwBvAU0PdrlWkcVFQTBrwLHkz3bPhK4ODWTw32GbZ9rhDVAUndgY+AI4I/Au8A9wFUR8QNJo4B3I+Kp6pWyfqXfz43A5yLiDUnbAWcA84HvRMTKqhbQSvK/VG1YQR/TMqAHsDdwDLAEOAT4vKRLyIJjv2qUsd5J6kr2D1MfYMeU/BzwFHA4cEGVimZrwYGwjWrqW5K0NfB1YCZwDrAd8GXgVbLA+DAwNiLurFph65SkvYCT08cfAGdI2i0ilgPzgKuBi6tTOlsbbhq3YZI+DZwK7ED2XOslwEDga8A/gCsi4rXqlbC+NB/4kDQG+AzwEHAv2e/pP4EpwAHA/hExswpFtbXkQNhGSdqerEZxJLAL8ElgDvBTYChZLfG7EfHPapWxXknaOSIeSu8PBg4jawpfAXwc2ASYExEvVK2Qtla8Z0kbU1Dr2BJ4KiKeBJ6UdABwPll3xo+BkyLinSoWtS5J6gOcLem5iPhmRNwiqSNwHlk/7pUR8XR1S2lry32EbUTBwMgG6esjQKOkQwEi4raUNhjYLyLe8bSM/DWfFB0R88n6AwdKOj+l3QQ8CQwgG9iyGuOmcRsiaX/gULJRxxlkfU59gLnAg8CFwP3A4Ig4ulrlrBfNJksfT9aCWh4Rv0lTls4kC3x3AccB/x4RL1ervLbuXKOoMkkd0te9gIuAycApwF7AH8gC4G7Ad8nmpt2eZVeX9V/a+lIQBL9M9sTILOBSSd+MiGlkgVBkU5nOdBCsXe4jrBJJg4E3IuLNtJjCbmQBcBnZvLQrI+JlSQsi4sYU+PYBJgDjI8JNsBykprAiYnX6PBDYDzgYOBF4APi6pO4R8S3gBEld/PuobQ6E1bMV8KikLSNikaSXyTrcNwQOj4hXJX0O6AtcGhHLJA0Fjo6IZ6pY7vZuo4hYAiDpC8ArZM3e3YHPRsTukg4Cbpb0VkRc6CBY+9w0rpI0AfpoYLqkTcjmoS0FrgReS9NnzgX+WXDOTxwE8yPpcLLpSUjaDxgH/D0iFpP9rTyUsm4I/BfZY3XWDniwpMrSPLSfkA2M7EE2J20k2WozP04jkpYzSZsC15J1T+wIfBF4JCLOSsf3A/4NWAnsBBwQES9VqbhWYQ6EbYCkQ4AfAqMjYknqP3wvIuZ4Gaf1Q1I34DqyhRJ2Iquh9wV+FBH3pjy7kM0V/EdEPF+tslrluY+wDYiImyWtBv4h6ROFTyQ4CK4fEfG2pL+SdUd8LyJ+KOkC4BBJRMS9EfFglYtpOXGNsA1JNcOlEXFXtctSjyR9DBhC9kz3D4FbgP8AepEteeZA2E45ELZBbg5Xl6QdyPoLLwDuBMaTLXAxv6oFs9w4EJq1QNII4K/Al4BrI2JVlYtkOXIgNGtFWml6WUTMqnZZLF8OhGZW9zyh2szqngOhmdU9B0Izq3sOhGZW9xwIzazuORC2c5JWSXpc0lOSrpO04Ue41q/T0mBIukLSsCJ595K02zrc40VJvcpNb5ZnyVre63uSvra2ZbT2x4Gw/VsWESMjYltgOWv24QVAUuO6XDQivhgRM4pk2YtssVmzNs+BsL7cC3w81dbukvR74O+SGiX9UNIjkp6U9B+QPeon6RJJMyTdTLZ/CunY3WnfDiQdKOlRSU9ImippC7KAe0aqje4hqbekG9I9HpG0ezp3U0m3S3pM0uVkS98XJemPkqZLelrSSc2O/TiVZaqk3iltK0m3pnPulbRNRX6a1m549Zk6kfZGOQi4NSWNBraNiBdSMHkzInZK2wb8TdLtwPbA1sB2ZEtSzQD+t9l1ewO/BPZM1+oZEW9I+gWwJCJ+lPL9HvhJRNwnaRBwG/AJstVe7ouI89KiEx8IbK34t3SPLsAjkm6IiIXARsCjEXGmpHPStU8DJgInR8RMSTsDl5Fte2AGOBDWgy6SHk/v7yVbAXs34OGC5b72Bz7Z1P8HdCdbhWVP4Or0nO2ctExVc7sA9zRdKyLeaKUcnwaGac3umBunNQD3BP41nXuzpEVlfE+nS/pMer95KutCYDXZYgkAVwF/kNQ1fb/XFdy7Uxn3sDriQNj+LYuIkYUJKSAUbg4v4Etp7+TCfAcDpZ7BVBl5IOuG2bX5/h6pLGU/56lst79Pp2stlXQ30LmV7JHuu7j5z8CskPsIDbJm6v+T1BFA0lBJGwH3AONSH2J/YO8Wzn0A+FRaVRtJPVP620C3gny3kzVTSflGprf3AMemtIPIVoAupjuwKAXBbchqpE0agKZa7TFkTe63gBckjU33UFpZxux9DoQGcAVZ/9+jkp4CLidrLdwIzAT+Dvwc+L/mJ0bE62T9en+Q9ARrmqZ/Aj7TNFgCnA6MSoMxM1gzev19YE9Jj5I10UvtDXwr0EHSk8D5ZPs+N3kHGC5pOlkf4Hkp/VjgxFS+p4EjyviZWB3x6jNmVvdcIzSzuudAaGZ1z4HQzOqeA6GZ1T0HQjOrew6EZlb3HAjNrO79f5n8bUBMzeoZAAAAAElFTkSuQmCC\n", "text/plain": [ "

" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "[[697 270]\n", " [ 7 26]]\n", " precision recall f1-score support\n", "\n", " 0 0.99 0.72 0.83 967\n", " 1 0.09 0.79 0.16 33\n", "\n", " accuracy 0.72 1000\n", " macro avg 0.54 0.75 0.50 1000\n", "weighted avg 0.96 0.72 0.81 1000\n", "\n" ] } ], "source": [ "from sklearn.metrics import classification_report\n", "from sklearn.metrics import confusion_matrix\n", "\n", "threshold = 0.5\n", "predictions = [1 if e >= threshold else 0 for e in pred_prob_list ] \n", " \n", "\n", "cm = confusion_matrix(y_true= y_test, y_pred= predictions) \n", "plot_conf_mat(cm, classes=['legit','fraud'], \n", " title='Confusion Matrix')\n", "print(cm)\n", "\n", "print(f\"{classification_report(y_true=y_test, y_pred = predictions)}\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. 엔드포인트 삭제\n", "- **[중요] 과금을 막기 위해서 아래 코드를 꼭 수행 바랍니다.**\n", "- is_del_model= True 를 통해서 생성한 모델 및 이전 파이프라인을 통한 엔드포인트 컨피그 와 엔드포인트를 삭제 합니다." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "--- Deleted model: pipelines-kxuameqozk3b-Fraud-Basic-CreateMo-QxDsU13ks1\n", "--- Deleted endpoint: all-pipeline-endpoint-983792\n", "--- Deleted endpoint_config: pipelines-kxuameqozk3b-Fraud-Basic-CreateMo-QxDsU13ks1-config\n" ] } ], "source": [ "import boto3\n", "boto3_client = boto3.client('sagemaker')\n", "delete_endpoint(boto3_client, all_pipeline_endpoint_name, is_del_model=True )\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "instance_type": "ml.t3.medium", "kernelspec": { "display_name": "conda_python3", "language": "python", "name": "conda_python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.13" } }, "nbformat": 4, "nbformat_minor": 4 }