{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Moderation - PII Detection/Redaction and Sentiment Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PII (Personal Identifiable Information) detection and Sentiment Analysis are popular use cases of text moderation. You can accurately analyze customer interactions, including social media posts, reviews, customer interaction transcripts to improve your products and services. You can determine if the sentiment is positive, negative, neutral, or mixed. For example, you can use sentiment analysis to determine the sentiments of comments on a blog posting to determine if your readers liked the post."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [Setup notebook](#step1)\n",
    "- [Detect and redact PII entities in real-time](#step2)\n",
    "- [Detect PII entities as a batch job with redaction](#step3)\n",
    "- [Analyze text sentiment in real-time](#step4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Set up Notebook <a id=\"step1\"></a>\n",
    "Run the below cell to install/update Python dependencies if you run the lab using a local IDE. It is optional if you use a SageMaker Studio Juypter Notebook, which already includes the dependencies in the kernel. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First, let's get the latest installations of our dependencies\n",
    "%pip install -qU pip\n",
    "%pip install boto3 -qU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "import sagemaker as sm\n",
    "import json\n",
    "from IPython.display import clear_output\n",
    "\n",
    "# variables\n",
    "data_bucket = sm.Session().default_bucket()\n",
    "region = boto3.session.Session().region_name\n",
    "role = sm.get_execution_role()\n",
    "\n",
    "s3=boto3.client('s3')\n",
    "comprehend=boto3.client('comprehend', region_name=region)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Detect/redact PII entities in real-time<a id=\"step2\"></a>\n",
    "\n",
    "You can use the Amazon Comprehend console or APIs to detect personally identifiable information (PII) in English text documents. PII is a textual reference to personal data that could be used to identify an individual. PII examples include addresses, bank account numbers, and phone numbers.\n",
    "\n",
    "In the below cell, we will call the Amazon Comprehend [DetectPiiEntity](https://docs.aws.amazon.com/comprehend/latest/dg/realtime-pii-api.html) API, which analyzes a single document in real-time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"Here are my contact details: Call me 751-01-1111 or email me at someone@domain.com\"\n",
    "\n",
    "print('Calling DetectPii')\n",
    "response = comprehend.detect_pii_entities(Text=text, LanguageCode='en')\n",
    "display(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The PII API responsee has \"Entities\" node, a list of PII enitites detected by the Comprehend API. Each detected entity contains:\n",
    "- Score: the confidence score of this entity.\n",
    "- Type: PII type. For a full list of the PII entity types support by Comprehend, refer to this [doc](https://docs.aws.amazon.com/comprehend/latest/dg/how-pii.html).\n",
    "- BeginOffset: A character offset in the input text that shows where the PII entity begins.\n",
    "- EndOffset: A character offset in the input text that shows where the PII entity ends. \n",
    "\n",
    "The real-time PII detection endpoint doesn't support masking out of the box. But you can apply a light logic easily using the metadata in the response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pii_redaction(text, entities, mask_character='*'):\n",
    "    for entity in entities:\n",
    "        mask = ''\n",
    "        for i in range(entity[\"BeginOffset\"], entity[\"EndOffset\"]):\n",
    "            mask += mask_character\n",
    "        text = f'{text[0:entity[\"BeginOffset\"]]}{mask}{text[entity[\"EndOffset\"]:len(text)]}'\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = pii_redaction(text, response[\"Entities\"])\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Detect PII entities as a batch job with redaction <a id=\"step3\"></a>\n",
    "Now, let's try to start an asynchronous  [PII entity detection job](https://docs.aws.amazon.com/comprehend/latest/dg/get-started-api-pii.html) to process a collection of documents. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will generate a CSV file with some test messages containing PII information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "msgs = [\n",
    "    \"Hello, My name is John Doe and my address is 1 main st, Pleasanton,CA,94000\",\n",
    "    \"Hello, I m John Doe and my phone number is 785-000-0000\",\n",
    "    \"My credit card number is 5370-0000-1234-0000,111\",\n",
    "    \"For verification purposes, my SSN is 172-00-0000\",\n",
    "    \"Please send me the receipt on my email address someone@domain.com\",\n",
    "    \"I do not wish to subscribe to your marketing messages, please unsubscribe me\",\n",
    "    \"Good morning, everybody. My name is Jan Doe, and today I feel like sharing a whole lot of personal information with you. Let's start with my Email address jandoe@domain.com. My address is 2 park blvd, Los Angeles,CA,92000. My phone number is 818-111-1111. My Social security number is 548-00-0001. My Bank account number is 94000000001 and routing number 195111111. My credit card number is 5534816011111110,my pin 123456. Well, I think that's it. You know a whole lot about me. And I hope that Amazon comprehend is doing a good job at identifying PII entities so you can redact my personal information away from this document. Let's check.\"\"\",\n",
    "]\n",
    "with open('pii-test-data.csv','w') as f:\n",
    "    for m in msgs:\n",
    "        f.write(f'\"{m}\"\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Upload the CSV file to S3 for Comprehend batch process to access"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_key = 'content-moderation-im/text-moderation/pii-test-data.csv'\n",
    "s3.upload_file('pii-test-data.csv', data_bucket, s3_key)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import uuid\n",
    "\n",
    "InputS3URI= f\"s3://{data_bucket}/{s3_key}\"\n",
    "OutputS3URI=f\"s3://{data_bucket}/content-moderation-im/text-moderation/pii-detection-redaction\"\n",
    "job_name = f\"pii-job-{uuid.uuid1()}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One parameter we passed to Comprehend start_pii_entities_detection_job API is called RedactionConfig. In the below example, we ask the API to detect the PII entities, then apply redaction logic to these entities by replacing them using a mask character *."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = comprehend.start_pii_entities_detection_job(\n",
    "    InputDataConfig={\n",
    "        'S3Uri': InputS3URI,\n",
    "        'InputFormat': 'ONE_DOC_PER_FILE'\n",
    "    },\n",
    "    OutputDataConfig={\n",
    "        'S3Uri': OutputS3URI\n",
    "       \n",
    "    },\n",
    "    Mode='ONLY_REDACTION',\n",
    "    RedactionConfig={\n",
    "        'PiiEntityTypes': [\n",
    "           'ALL',\n",
    "        ],\n",
    "        'MaskMode': 'MASK',\n",
    "        'MaskCharacter': '*'\n",
    "    },\n",
    "    DataAccessRoleArn = role,\n",
    "    JobName=job_name,\n",
    "    LanguageCode='en',\n",
    "    \n",
    ")\n",
    "events_job_id = response['JobId']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The job will take roughly ~10 minutes. The below code is to check the status of the job. The cell execution would be completed after the job is completed "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "# Loop through and wait for the training to complete . Takes up to 10 mins \n",
    "import time\n",
    "from datetime import datetime\n",
    "\n",
    "max_time = time.time() + 3*60*60 # 3 hours\n",
    "while time.time() < max_time:\n",
    "    now = datetime.now()\n",
    "    current_time = now.strftime(\"%H:%M:%S\")\n",
    "    \n",
    "    response = comprehend.describe_pii_entities_detection_job(JobId=events_job_id)\n",
    "\n",
    "    status = response[\"PiiEntitiesDetectionJobProperties\"][\"JobStatus\"]\n",
    "    clear_output(wait=True)\n",
    "    print(f\"{current_time} : PII detection batch job: {status}\")\n",
    "    \n",
    "    if status == \"COMPLETED\" or status == \"FAILED\":\n",
    "        break\n",
    "     \n",
    "    time.sleep(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the batch job is complete, Comprehend will store PII detection/redaction result in the S3 bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_data_s3_file = response['PiiEntitiesDetectionJobProperties']['OutputDataConfig']['S3Uri'] + 'pii-test-data.csv.out'\n",
    "output_data_s3_file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's use the API for our sample dataset and let's check the response "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from io import BytesIO\n",
    "\n",
    "# The output filename is the input filename + \".out\"\n",
    "\n",
    "filename=\"pii-test-data.csv\"\n",
    "output_data_s3_file = response['PiiEntitiesDetectionJobProperties']['OutputDataConfig']['S3Uri'] + filename + '.out'\n",
    "output_data_s3_filepath=output_data_s3_file.replace(f's3://{data_bucket}/','')\n",
    "print(output_data_s3_filepath)\n",
    "\n",
    "f = BytesIO()\n",
    "s3.download_fileobj(data_bucket, output_data_s3_filepath, f)\n",
    "f.seek(0)\n",
    "arr = f.read().decode('UTF-8').split('\\r\\n')\n",
    "for a in arr:\n",
    "    print(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analyze text sentiment in real-time <a id=\"step4\"></a>\n",
    "Amazon Comprehend can be used to perform sentiment analysis. You can accurately analyze customer interactions, including social media posts, reviews, customer interaction transcripts to improve your products and services.\n",
    "\n",
    "This lab will only cover the real-time Sentiment analysis use case. For more information about Comprehend Sentiment Analysis, refers to this [doc](https://docs.aws.amazon.com/comprehend/latest/dg/how-sentiment.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"It is raining today in Seattle\"\n",
    "response = comprehend.detect_sentiment(Text=text, LanguageCode='en')\n",
    "print(json.dumps(response, sort_keys=True, indent=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sentiment determination returns the following values with confidence score:\n",
    "- Positive – The text expresses an overall positive sentiment.\n",
    "- Negative – The text expresses an overall negative sentiment.\n",
    "- Mixed – The text expresses both positive and negative sentiments.\n",
    "- Neutral – The text does not express either positive or negative sentiments.\n",
    "\n",
    "The above response shows that the sentiment of the input text is 'NEUTRAL' with a confidence score associated with each value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "\n",
    "We have tried using Comprehend PII API in this lab to detect PII entities and apply redaction logic. We also tried an example of using Comprehend Sentiment Analysis API to get the sentiment of the input text. "
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.m5.large",
  "kernelspec": {
   "display_name": "Python 3 (Data Science)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  },
  "vscode": {
   "interpreter": {
    "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}