{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "1099b04396475b6a0143fa303da9fa44ad87b660"
   },
   "source": [
    "# Amazon Comprehend의 데이터셋 준비와 Custom entity recognizer 실행하기\n",
    "\n",
    "### [(원본)](https://github.com/aws-samples/amazon-comprehend-custom-entity/blob/master/1-AWS-Comprehend-Custom-Entities.ipynb)\n",
    "\n",
    "이 노트북은 Amazon Comprehend의 custom entities 의 학습 데이터셋을 위한 준비하는 방법에 대해 살펴보도록 하겠습니다.\n",
    "\n",
    "Custom entity recongizer 모델을 생성하는 방법에 대한 추가적인 정보는 다음 링크를 참조하시기 바랍니다. \n",
    "https://docs.aws.amazon.com/comprehend/latest/dg/training-recognizers.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_kg_hide-input": true,
    "_kg_hide-output": true,
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5"
   },
   "outputs": [],
   "source": [
    "# library imports\n",
    "import re\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib\n",
    "import csv\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "54e810d8b9c1936c8569093badabc4d7b25ea881"
   },
   "source": [
    "\n",
    "이번 예제는 twitter dataset을 사용할 예정으로 다음 링크에서 데이터셋을 다운로드하여 압축을 푼 후, ./data 폴더에 업로드를 합니다. https://www.kaggle.com/thoughtvector/customer-support-on-twitter\n",
    "\n",
    "업로드 한 데이데셋을 읽어 확인합니다"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "_uuid": "9365c16e4481ec49f5c084f7c3b0cf50dd55047f",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2811774, 7)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tweet_id</th>\n",
       "      <th>author_id</th>\n",
       "      <th>inbound</th>\n",
       "      <th>created_at</th>\n",
       "      <th>text</th>\n",
       "      <th>response_tweet_id</th>\n",
       "      <th>in_response_to_tweet_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>sprintcare</td>\n",
       "      <td>False</td>\n",
       "      <td>Tue Oct 31 22:10:47 +0000 2017</td>\n",
       "      <td>@115712 I understand. I would like to assist y...</td>\n",
       "      <td>2</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>115712</td>\n",
       "      <td>True</td>\n",
       "      <td>Tue Oct 31 22:11:45 +0000 2017</td>\n",
       "      <td>@sprintcare and how do you propose we do that</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>115712</td>\n",
       "      <td>True</td>\n",
       "      <td>Tue Oct 31 22:08:27 +0000 2017</td>\n",
       "      <td>@sprintcare I have sent several private messag...</td>\n",
       "      <td>1</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>sprintcare</td>\n",
       "      <td>False</td>\n",
       "      <td>Tue Oct 31 21:54:49 +0000 2017</td>\n",
       "      <td>@115712 Please send us a Private Message so th...</td>\n",
       "      <td>3</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>115712</td>\n",
       "      <td>True</td>\n",
       "      <td>Tue Oct 31 21:49:35 +0000 2017</td>\n",
       "      <td>@sprintcare I did.</td>\n",
       "      <td>4</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   tweet_id   author_id  inbound                      created_at  \\\n",
       "0         1  sprintcare    False  Tue Oct 31 22:10:47 +0000 2017   \n",
       "1         2      115712     True  Tue Oct 31 22:11:45 +0000 2017   \n",
       "2         3      115712     True  Tue Oct 31 22:08:27 +0000 2017   \n",
       "3         4  sprintcare    False  Tue Oct 31 21:54:49 +0000 2017   \n",
       "4         5      115712     True  Tue Oct 31 21:49:35 +0000 2017   \n",
       "\n",
       "                                                text response_tweet_id  \\\n",
       "0  @115712 I understand. I would like to assist y...                 2   \n",
       "1      @sprintcare and how do you propose we do that               NaN   \n",
       "2  @sprintcare I have sent several private messag...                 1   \n",
       "3  @115712 Please send us a Private Message so th...                 3   \n",
       "4                                 @sprintcare I did.                 4   \n",
       "\n",
       "   in_response_to_tweet_id  \n",
       "0                      3.0  \n",
       "1                      1.0  \n",
       "2                      4.0  \n",
       "3                      5.0  \n",
       "4                      6.0  "
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets = pd.read_csv('./data/twcs.csv',encoding='utf-8')\n",
    "print(tweets.shape)\n",
    "tweets.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "845eba8749f15e1e2b10aa43414f40860259f4e0"
   },
   "source": [
    "<a id='data-wrangling'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "5e053048057a5566a30aab3f0278aa529449938a"
   },
   "source": [
    "## 데이터 전처리\n",
    "\n",
    "tweet 데이터 셋은 흥미롭게도 약 3백만 개의 트윗을 포함하고 tweet 작성자와 tweet이 질의인지 대답인지 (\"inbound\" 컬럼)를 포함하고 있습니다.\n",
    "만약 tweet이 질의이면 \"response_tweet_id\" 컬럼에는 support team의 답변이 제공됩니다. \n",
    "\n",
    "각 행에서 질의-응답 데이터의 결과값을 얻기위해서 [이 링크](https://www.kaggle.com/soaxelbrooke/first-inbound-and-response-tweets)에서 코드를 가져왔습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
    "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data shape: (794299, 14)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tweet_id_x</th>\n",
       "      <th>author_id_x</th>\n",
       "      <th>inbound_x</th>\n",
       "      <th>created_at_x</th>\n",
       "      <th>text_x</th>\n",
       "      <th>response_tweet_id_x</th>\n",
       "      <th>in_response_to_tweet_id_x</th>\n",
       "      <th>tweet_id_y</th>\n",
       "      <th>author_id_y</th>\n",
       "      <th>inbound_y</th>\n",
       "      <th>created_at_y</th>\n",
       "      <th>text_y</th>\n",
       "      <th>response_tweet_id_y</th>\n",
       "      <th>in_response_to_tweet_id_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8</td>\n",
       "      <td>115712</td>\n",
       "      <td>True</td>\n",
       "      <td>Tue Oct 31 21:45:10 +0000 2017</td>\n",
       "      <td>@sprintcare is the worst customer service</td>\n",
       "      <td>9,6,10</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6</td>\n",
       "      <td>sprintcare</td>\n",
       "      <td>False</td>\n",
       "      <td>Tue Oct 31 21:46:24 +0000 2017</td>\n",
       "      <td>@115712 Can you please send us a private messa...</td>\n",
       "      <td>5,7</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8</td>\n",
       "      <td>115712</td>\n",
       "      <td>True</td>\n",
       "      <td>Tue Oct 31 21:45:10 +0000 2017</td>\n",
       "      <td>@sprintcare is the worst customer service</td>\n",
       "      <td>9,6,10</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9</td>\n",
       "      <td>sprintcare</td>\n",
       "      <td>False</td>\n",
       "      <td>Tue Oct 31 21:46:14 +0000 2017</td>\n",
       "      <td>@115712 I would love the chance to review the ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>115712</td>\n",
       "      <td>True</td>\n",
       "      <td>Tue Oct 31 21:45:10 +0000 2017</td>\n",
       "      <td>@sprintcare is the worst customer service</td>\n",
       "      <td>9,6,10</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10</td>\n",
       "      <td>sprintcare</td>\n",
       "      <td>False</td>\n",
       "      <td>Tue Oct 31 21:45:59 +0000 2017</td>\n",
       "      <td>@115712 Hello! We never like our customers to ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>18</td>\n",
       "      <td>115713</td>\n",
       "      <td>True</td>\n",
       "      <td>Tue Oct 31 19:56:01 +0000 2017</td>\n",
       "      <td>@115714 y’all lie about your “great” connectio...</td>\n",
       "      <td>17</td>\n",
       "      <td>NaN</td>\n",
       "      <td>17</td>\n",
       "      <td>sprintcare</td>\n",
       "      <td>False</td>\n",
       "      <td>Tue Oct 31 19:59:13 +0000 2017</td>\n",
       "      <td>@115713 H there! We'd definitely like to work ...</td>\n",
       "      <td>16</td>\n",
       "      <td>18.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>20</td>\n",
       "      <td>115715</td>\n",
       "      <td>True</td>\n",
       "      <td>Tue Oct 31 22:03:34 +0000 2017</td>\n",
       "      <td>@115714 whenever I contact customer support, t...</td>\n",
       "      <td>19</td>\n",
       "      <td>NaN</td>\n",
       "      <td>19</td>\n",
       "      <td>sprintcare</td>\n",
       "      <td>False</td>\n",
       "      <td>Tue Oct 31 22:10:10 +0000 2017</td>\n",
       "      <td>@115715 Please send me a private message so th...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>20.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   tweet_id_x author_id_x  inbound_x                    created_at_x  \\\n",
       "0           8      115712       True  Tue Oct 31 21:45:10 +0000 2017   \n",
       "1           8      115712       True  Tue Oct 31 21:45:10 +0000 2017   \n",
       "2           8      115712       True  Tue Oct 31 21:45:10 +0000 2017   \n",
       "3          18      115713       True  Tue Oct 31 19:56:01 +0000 2017   \n",
       "4          20      115715       True  Tue Oct 31 22:03:34 +0000 2017   \n",
       "\n",
       "                                              text_x response_tweet_id_x  \\\n",
       "0          @sprintcare is the worst customer service              9,6,10   \n",
       "1          @sprintcare is the worst customer service              9,6,10   \n",
       "2          @sprintcare is the worst customer service              9,6,10   \n",
       "3  @115714 y’all lie about your “great” connectio...                  17   \n",
       "4  @115714 whenever I contact customer support, t...                  19   \n",
       "\n",
       "   in_response_to_tweet_id_x  tweet_id_y author_id_y  inbound_y  \\\n",
       "0                        NaN           6  sprintcare      False   \n",
       "1                        NaN           9  sprintcare      False   \n",
       "2                        NaN          10  sprintcare      False   \n",
       "3                        NaN          17  sprintcare      False   \n",
       "4                        NaN          19  sprintcare      False   \n",
       "\n",
       "                     created_at_y  \\\n",
       "0  Tue Oct 31 21:46:24 +0000 2017   \n",
       "1  Tue Oct 31 21:46:14 +0000 2017   \n",
       "2  Tue Oct 31 21:45:59 +0000 2017   \n",
       "3  Tue Oct 31 19:59:13 +0000 2017   \n",
       "4  Tue Oct 31 22:10:10 +0000 2017   \n",
       "\n",
       "                                              text_y response_tweet_id_y  \\\n",
       "0  @115712 Can you please send us a private messa...                 5,7   \n",
       "1  @115712 I would love the chance to review the ...                 NaN   \n",
       "2  @115712 Hello! We never like our customers to ...                 NaN   \n",
       "3  @115713 H there! We'd definitely like to work ...                  16   \n",
       "4  @115715 Please send me a private message so th...                 NaN   \n",
       "\n",
       "   in_response_to_tweet_id_y  \n",
       "0                        8.0  \n",
       "1                        8.0  \n",
       "2                        8.0  \n",
       "3                       18.0  \n",
       "4                       20.0  "
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "first_inbound = tweets[pd.isnull(tweets.in_response_to_tweet_id) & tweets.inbound]\n",
    "\n",
    "QnR = pd.merge(first_inbound, tweets, left_on='tweet_id', \n",
    "                                  right_on='in_response_to_tweet_id')\n",
    "\n",
    "# Filter to only outbound replies (from companies)\n",
    "QnR = QnR[QnR.inbound_y ^ True]\n",
    "print(f'Data shape: {QnR.shape}')\n",
    "QnR.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "_uuid": "0428e41c670dbe801090613580cf22e3b41723b5"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>author_id_x</th>\n",
       "      <th>created_at_x</th>\n",
       "      <th>text_x</th>\n",
       "      <th>author_id_y</th>\n",
       "      <th>created_at_y</th>\n",
       "      <th>text_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>115712</td>\n",
       "      <td>Tue Oct 31 21:45:10 +0000 2017</td>\n",
       "      <td>@sprintcare is the worst customer service</td>\n",
       "      <td>sprintcare</td>\n",
       "      <td>Tue Oct 31 21:46:24 +0000 2017</td>\n",
       "      <td>@115712 Can you please send us a private messa...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>115712</td>\n",
       "      <td>Tue Oct 31 21:45:10 +0000 2017</td>\n",
       "      <td>@sprintcare is the worst customer service</td>\n",
       "      <td>sprintcare</td>\n",
       "      <td>Tue Oct 31 21:46:14 +0000 2017</td>\n",
       "      <td>@115712 I would love the chance to review the ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>115712</td>\n",
       "      <td>Tue Oct 31 21:45:10 +0000 2017</td>\n",
       "      <td>@sprintcare is the worst customer service</td>\n",
       "      <td>sprintcare</td>\n",
       "      <td>Tue Oct 31 21:45:59 +0000 2017</td>\n",
       "      <td>@115712 Hello! We never like our customers to ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>115713</td>\n",
       "      <td>Tue Oct 31 19:56:01 +0000 2017</td>\n",
       "      <td>@115714 y’all lie about your “great” connectio...</td>\n",
       "      <td>sprintcare</td>\n",
       "      <td>Tue Oct 31 19:59:13 +0000 2017</td>\n",
       "      <td>@115713 H there! We'd definitely like to work ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>115715</td>\n",
       "      <td>Tue Oct 31 22:03:34 +0000 2017</td>\n",
       "      <td>@115714 whenever I contact customer support, t...</td>\n",
       "      <td>sprintcare</td>\n",
       "      <td>Tue Oct 31 22:10:10 +0000 2017</td>\n",
       "      <td>@115715 Please send me a private message so th...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  author_id_x                    created_at_x  \\\n",
       "0      115712  Tue Oct 31 21:45:10 +0000 2017   \n",
       "1      115712  Tue Oct 31 21:45:10 +0000 2017   \n",
       "2      115712  Tue Oct 31 21:45:10 +0000 2017   \n",
       "3      115713  Tue Oct 31 19:56:01 +0000 2017   \n",
       "4      115715  Tue Oct 31 22:03:34 +0000 2017   \n",
       "\n",
       "                                              text_x author_id_y  \\\n",
       "0          @sprintcare is the worst customer service  sprintcare   \n",
       "1          @sprintcare is the worst customer service  sprintcare   \n",
       "2          @sprintcare is the worst customer service  sprintcare   \n",
       "3  @115714 y’all lie about your “great” connectio...  sprintcare   \n",
       "4  @115714 whenever I contact customer support, t...  sprintcare   \n",
       "\n",
       "                     created_at_y  \\\n",
       "0  Tue Oct 31 21:46:24 +0000 2017   \n",
       "1  Tue Oct 31 21:46:14 +0000 2017   \n",
       "2  Tue Oct 31 21:45:59 +0000 2017   \n",
       "3  Tue Oct 31 19:59:13 +0000 2017   \n",
       "4  Tue Oct 31 22:10:10 +0000 2017   \n",
       "\n",
       "                                              text_y  \n",
       "0  @115712 Can you please send us a private messa...  \n",
       "1  @115712 I would love the chance to review the ...  \n",
       "2  @115712 Hello! We never like our customers to ...  \n",
       "3  @115713 H there! We'd definitely like to work ...  \n",
       "4  @115715 Please send me a private message so th...  "
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#데이터프레임에서 필요한 컬럼만 추출합니다\n",
    "\n",
    "QnR = QnR[[\"author_id_x\",\"created_at_x\",\"text_x\",\"author_id_y\",\"created_at_y\",\"text_y\"]]\n",
    "QnR.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 통신사 tweet만 필터링하기\n",
    "\n",
    "이 예제에서는 스마트폰 디바이스를 인식하기위한 custom entity를 생성합니다. \n",
    "우선 데이터프레임에서 T-Mobile과 Sprint의 tweet만 필터링하여 telco tweets 데이터를 만듭니다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweet_telco = QnR[QnR[\"author_id_y\"].isin([\"TMobileHelp\", \"sprintcare\"])]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "필터링된 데이터에서 질문과 응답의 text 데이터를 결합하여 하나의 text 컬럼으로 생성합니다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweet_telco['text'] = tweet_telco['text_x']+ ' | ' + tweet_telco['text_y']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "telco tweets 데이터를 csv 파일로 저장합니다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweet_telco['text'].to_csv('./data/tweet_telco.csv', encoding='utf-8', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "데이셋을 생성하기 위해서는 DEVICE라는 새 클래스의 Entity 목록을 제공해야 합니다. \n",
    "\n",
    "관련 Entity를 찾기 위해서는, 코퍼스를 word2vec 모델로 로드하고 유사한 키워드 목록을 생성할 수 있습니다. \n",
    "이 기술은 두 번째 예에서 사용될 것입니다.\n",
    "\n",
    "우리는 Device를 찾기 위해 스마프톤의 다른 철자 목록을 생성할 것입니다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "sphones = ['iPhone X', 'iPhoneX', 'iphoneX', 'Samsung Galaxy', 'Samsung Note', 'iphone', 'iPhone', 'android', 'Android']\n",
    "\n",
    "df_entity_list = pd.DataFrame(sphones, columns=['Text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>iPhone X</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>iPhoneX</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>iphoneX</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Samsung Galaxy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Samsung Note</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>iphone</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>iPhone</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>android</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Android</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Text\n",
       "0        iPhone X\n",
       "1         iPhoneX\n",
       "2         iphoneX\n",
       "3  Samsung Galaxy\n",
       "4    Samsung Note\n",
       "5          iphone\n",
       "6          iPhone\n",
       "7         android\n",
       "8         Android"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_entity_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DEVICE 클래스 레이블로 다른 컬럼을 추가합니다. 이것은 Amazon Comprehend의 학습 데이터셋의 요구 사양입니다. \n",
    "\n",
    "자세한 내용은 다음 링크를 확인해주시기 바랍니다. \n",
    "\n",
    "https://docs.aws.amazon.com/comprehend/latest/dg/cer-entity-list.html\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_entity_list['Type'] = 'DEVICE'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Text</th>\n",
       "      <th>Type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>iPhone X</td>\n",
       "      <td>DEVICE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>iPhoneX</td>\n",
       "      <td>DEVICE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>iphoneX</td>\n",
       "      <td>DEVICE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Samsung Galaxy</td>\n",
       "      <td>DEVICE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Samsung Note</td>\n",
       "      <td>DEVICE</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Text    Type\n",
       "0        iPhone X  DEVICE\n",
       "1         iPhoneX  DEVICE\n",
       "2         iphoneX  DEVICE\n",
       "3  Samsung Galaxy  DEVICE\n",
       "4    Samsung Note  DEVICE"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_entity_list.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "학습 데이터파일을 생성합니다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 파일이 크면 학습이 늦어지므로 일부만 학습하도록 합니다\n",
    "#tweet_telco['text'].tail(5000).to_csv('./data/raw_txt.csv', encoding='utf-8', index=False)\n",
    "\n",
    "tweet_telco['text'].to_csv('./data/raw_txt.csv', encoding='utf-8', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"I feel like #Sprint hates me. \n",
      "\n",
      "Maybe I deserve consistent slow LTE. Thanks Sprint, you only give me a low amount of first world problems. \n",
      "\n",
      "I miss #TMobile &lt;/3 they sucked too, but not as bad as Sprint. | @803494 This sounds concerning. How long have you been experiencing slow speeds on your device? Does this happen in specific locations? Do you notice this indoors or outdoors? - JF\"\n",
      ".@115990 you all are pathetic I have been with you all 10 years and have been with uber 3 with a discount now you are forcing me back to #Sprint | @176582 Hey there! We'd love to see you as part of our family. Please check our great offers in this link: https://t.co/dwtDgGbWzU . - EG\n",
      "\"Very #disappointed again by the services of @116297 @115817 and @115911 - We keep paying and yet no one delivers.... #complaint #technews #tmobile #asurion #ups - Thanks for nothing! | @221395 Hi Athan, DM us and tell us what's going on. We'll make sure you get what you're expecting. *JoanO\"\n",
      "\"@TMobileHelp @115911 I'm in Norwalk, CT and haven't had service in hours. Help! I saw on Down Detector that there is an outage, any update on when it'll be fixed? | @803724 Oh no! DM me and let's look into this ASAP! https://t.co/86MicZnF9C *WarrenCamp\"\n",
      "@TMobileHelp came through for me ✊🤘 | @792214 We will leave the #MagentaLights On if you need us again my friend :)  Take care and enjoy the rest of your day! *KimWilliams\n",
      "\".@115913 Note 4, Note 5, S7 Edge, S8+, and now Note 8. 100% of my Jumps have had an issue caused by T-Mobile to where I am unable to even leave the store with the phone. Why can't this be fixed by your staff? | @803725 @115913 Being able to get a new phone should always be an amazing feeling and should go as smoothly as possible. Send us a DM so I can look into this and we can find a solution for you right away. https://t.co/plE6XCdHhz *CharlesOpacki\"\n"
     ]
    }
   ],
   "source": [
    "!head ./data/raw_txt.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Entity 목록 파일을 생성합니다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_entity_list.to_csv('./data/entity_list.csv', encoding='utf-8', index=False)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text,Type\n",
      "iPhone X,DEVICE\n",
      "iPhoneX,DEVICE\n",
      "iphoneX,DEVICE\n",
      "Samsung Galaxy,DEVICE\n",
      "Samsung Note,DEVICE\n",
      "iphone,DEVICE\n",
      "iPhone,DEVICE\n",
      "android,DEVICE\n",
      "Android,DEVICE\n"
     ]
    }
   ],
   "source": [
    "!head ./data/entity_list.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "원본 telco tweet 데이터셋에서 테스트 파일을 만듭니다.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweet_telco['text'].tail(100).to_csv('./data/telco_test.csv', encoding='utf-8', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## S3에 파일 업로드\n",
    "\n",
    "S3에 생성된 entry_list.csv, raw_txt.csv, telco_test.csv 파일을 업로드합니다.  \n",
    "\n",
    "\n",
    "![title](./img/s3.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 모델 학습하기\n",
    "\n",
    "Amazon Comprehend 콘솔을 통해 Custom entity recognizer의 Job을 생성하겠습니다. \n",
    "custom entity 설정은 다음과 같습니다. \n",
    "\n",
    "![title](./img/ter.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Amazon Comprehend가 생성한 S3버킷에 접근하기 위해서는 다음과 같이 IAM role을 생성합니다. \n",
    "![title](./img/iam.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Custom entity 모델 테스트하기"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(1) 생성된 Custom entity 모델을 테스트하기 위해서 Amazon Comprehend 콘솔의 Analysis jobs를 통해 분석 job을 생성합니다. \n",
    "\n",
    "![title](./img/ajob.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(2) Comprehend CLI를 통해서 S3에 올려놓은 테스트 파일에 대한 분석 job을 실행할 수 있다. 다음은 그 예제입니다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!aws comprehend start-entities-detection-job \\\n",
    "     --entity-recognizer-arn \"arn:aws:comprehend:us-east-1:{account number}:entity-recognizer/twitter-device-cer\" \\\n",
    "     --job-name Test \\\n",
    "     --data-access-role-arn \"arn:aws:iam::{account number}:role/service-role/AmazonComprehendServiceRole-cer-test\" \\\n",
    "     --language-code en \\\n",
    "     --input-data-config \"S3Uri=s3://{bucket name}/custom_entities/telco_test.csv\" \\\n",
    "     --output-data-config \"S3Uri=s3://{bucket name}/custom_entities/output/\" \\\n",
    "     --region \"us-east-1\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "39751a13337cd09b32588e2d0fc5f7e7817cca8b"
   },
   "source": [
    "\n",
    "참고로 CLI로 모델을 테스트하기 위해서는 IAM role 설정이 추가적으로 필요합니다.\n",
    "data-access-role-arn 의 role은 해당 버킷에 PutObject 권한이 필요하고, 노트북 인스턴스의 role은 iam:PassRole 설정이 필요합니다.\n",
    "관련 내용은 다음 링크를 참고하시기 바랍니다. \n",
    "https://docs.aws.amazon.com/comprehend/latest/dg/access-control-managing-permissions.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "39751a13337cd09b32588e2d0fc5f7e7817cca8b"
   },
   "source": [
    "테스트가 완료되면 --output-data-config 경로에 압축된 output json 파일이 생성됩니다. \n",
    "![title](./img/output.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "39751a13337cd09b32588e2d0fc5f7e7817cca8b"
   },
   "source": [
    "## Output 결과 확인하기\n",
    " \n",
    "Ouptput 결과는 json 파일을 직접 열어 확인하거나 AWS Glue와 Athena를 통해 확인가능합니다. \n",
    "AWS Glue의 crawler를 통해 S3의 output 파일을 읽어 Database를 만들고 Athena를 통해 조회가 가능합니다. \n",
    "다음은 Glue의 Crawler 설정 예제입니다. \n",
    "![title](./img/glue.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "39751a13337cd09b32588e2d0fc5f7e7817cca8b"
   },
   "source": [
    "Athena에서는 다음과 같은 query를 통해 Custom entity 인식 결과를 Device별로 조회할 수 있습니다. \n",
    "\n",
    "\"SELECT col3, count(col3) FROM \"comprehend\".\"telco_device_test_json\" group by col3;\"\n",
    "\n",
    "![title](./img/test.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "위 결과 중, \"ipone\"은 데이터셋에 태깅된 Entity 목록에 포함되지 않으나 Comprehend는 어느 정도 정확한 신뢰도를 가지고 이를 선택할 수 있습니다. \n",
    "\n",
    "\n",
    "### 참고\n",
    "https://aws.amazon.com/blogs/machine-learning/getting-started-with-amazon-comprehend-custom-entities/\n",
    "https://aws.amazon.com/blogs/machine-learning/build-a-custom-entity-recognizer-using-amazon-comprehend/\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernel_info": {
   "name": "python3"
  },
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "nteract": {
   "version": "0.15.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}