"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Extract the text from all the sample documents in the list\n",
"\n",
"We will create yet another small utility function that receives the document bytearray, extracts text from the document with Textract and returns the extracted text."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def extract_text(doc):\n",
" response = call_textract(input_document=doc)\n",
" page_string = get_string(textract_json=response, output_type=[Textract_Pretty_Print.LINES])\n",
" return page_string"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"files_df['DocText'] = files_df.apply(lambda row : extract_text(row['Document']), axis = 1)\n",
"files_df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We have the extracted text in the dataframe for each of our document, the next step is to use the Amazon Comprehend real-time endpoint to classify them. We will create a small utility function that does the classification using the endpoint and returns the document type. Note that Comprehend will return all the classes of documents with a confidence score linked to each class in an array of key-value pairs (Name-Score), we will pick only the document class with the highest confidence score from the endpoint's response."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import time \n",
"from datetime import datetime\n",
"\n",
"def classify_doc(document):\n",
" now = datetime.now()\n",
" current_time = now.strftime(\"%H:%M:%S\")\n",
" print(f'{current_time} : Processing')\n",
" time.sleep(10) #to avoid Comprehend API throttling\n",
" try:\n",
" response = comprehend.classify_document(\n",
" Text= document,\n",
" EndpointArn=ENDPOINT_ARN\n",
" )\n",
" print(response)\n",
" response_df = pd.DataFrame(response['Classes'])\n",
" result = response_df.iloc[response_df['Score'].idxmax()] # choose the class with highest score \n",
" return result.Name # return the corresponding class name\n",
" except Exception as e:\n",
" print(e)\n",
" return 'error'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Lets now run the inference on our data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"# import time\n",
"files_df['DocType'] = files_df.apply(lambda row : classify_doc(row['DocText']), axis = 1)\n",
"files_df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We have now classified all these documents into their respective classes. Let's review to check if the classifier did a correct job of classifying."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"document = files_df.iloc[0]['Document']\n",
"document_type = files_df.iloc[0]['DocType']\n",
"\n",
"display(HTML(f'![\"IDP](\"./images/Insurance_doc_classify.png\")
\n",
"