{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Post-Processing Amazon Textract with Location-Aware Transformers**\n",
    "\n",
    "# Part 3: Implementing Human Review\n",
    "\n",
    "> *This notebook works well with the `Data Science 3.0 (Python 3)` kernel on SageMaker Studio - use the same as for NB1*\n",
    "\n",
    "In this final notebook, we'll set up the human review component of the OCR pipeline using [Amazon Augmented AI (A2I)](https://aws.amazon.com/augmented-ai/): Completing the demo pipeline.\n",
    "\n",
    "The A2I service shares a lot in common with SageMaker Ground Truth, with the main difference that A2I is designed for **near-real-time, single-example** annotation/review to support a live business process, while SMGT is oriented towards **offline, batch** annotation for building datasets.\n",
    "\n",
    "The two services both use the Liquid HTML templating language, and you might reasonably wonder: \"*Are we going to use the same custom boxes-plus-review template from earlier?*\"\n",
    "\n",
    "In fact, no we won't - for reasons we'll get to in a moment.\n",
    "\n",
    "First though, let's load the required libraries and configuration for the notebook as before:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dependencies and configuration\n",
    "\n",
    "The custom task template demonstrated in this notebook is a little more complex than the SageMaker Ground Truth one we saw in notebook 1, so is built with a [NodeJS](https://nodejs.org/en/)-based **toolchain** rather than edited as a raw HTML file.\n",
    "\n",
    "- If you're running this notebook in SageMaker Studio, you can install NodeJS by running the below.\n",
    "- If you're on a SageMaker Notebook Instance, check as it may already be installed - in which case you can skip this step.\n",
    "- If you're running on some other environment (like a local laptop), you probably want to install NodeJS via standard tools instead. [nvm](https://github.com/nvm-sh/nvm) is a helpful utility for managing multiple different Node versions on your system.\n",
    "- If you're not able to install NodeJS on your environment, don't worry - there's an alternative pre-built option (missing some features) mentioned later when we use it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Check if NodeJS installed:\n",
    "!node --version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Install NodeJS:\n",
    "NODE_VER = \"v18.15.0\"\n",
    "NODE_DISTRO = \"linux-x64\"\n",
    "!mkdir -p /usr/local/lib/nodejs\n",
    "!wget -c https://nodejs.org/dist/{NODE_VER}/node-{NODE_VER}-{NODE_DISTRO}.tar.xz -O - | tar -xJ -C /usr/local/lib/nodejs\n",
    "# Can't easily override PATH here, so instead we'll just symlink relevant executable files into a\n",
    "# folder that's already on the PATH:\n",
    "NODE_BIN_DIR = f\"/usr/local/lib/nodejs/node-{NODE_VER}-{NODE_DISTRO}/bin\"\n",
    "ONPATH_BIN_DIR = \"/usr/local/bin\"\n",
    "!ln -fs {NODE_BIN_DIR}/node {ONPATH_BIN_DIR}/node && \\\n",
    "    ln -fs {NODE_BIN_DIR}/npm {ONPATH_BIN_DIR}/npm && \\\n",
    "    ln -fs {NODE_BIN_DIR}/npx {ONPATH_BIN_DIR}/npx && \\\n",
    "    echo \"NodeJS {NODE_VER} installed!\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As before, once required libraries are installed, we can proceed with other imports and configuration:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "# Python Built-Ins:\n",
    "import json\n",
    "from logging import getLogger\n",
    "import os\n",
    "\n",
    "# External Dependencies:\n",
    "import boto3  # AWS SDK for Python\n",
    "import sagemaker  # High-level SDK for SageMaker\n",
    "\n",
    "# Local Dependencies:\n",
    "import util\n",
    "\n",
    "logger = getLogger()\n",
    "role = sagemaker.get_execution_role()\n",
    "s3 = boto3.resource(\"s3\")\n",
    "smclient = boto3.client(\"sagemaker\")\n",
    "\n",
    "# Manual configuration (check this matches notebook 1):\n",
    "bucket_name = sagemaker.Session().default_bucket()\n",
    "bucket_prefix = \"textract-transformers/\"\n",
    "print(f\"Working in bucket s3://{bucket_name}/{bucket_prefix}\")\n",
    "config = util.project.init(\"ocr-transformers-demo\")\n",
    "print(config)\n",
    "\n",
    "# Field configuration saved from first notebook:\n",
    "with open(\"data/field-config.json\", \"r\") as f:\n",
    "    fields = [\n",
    "        util.postproc.config.FieldConfiguration.from_dict(cfg)\n",
    "        for cfg in json.loads(f.read())\n",
    "    ]\n",
    "entity_classes = [f.name for f in fields]\n",
    "\n",
    "# S3 URIs per first notebook:\n",
    "raw_s3uri = f\"s3://{bucket_name}/{bucket_prefix}data/raw\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The rationale for a separate review template\n",
    "\n",
    "For many ML-powered processes, intercepting low-confidence predictions for human review is important for delivering efficient, accurate service.\n",
    "\n",
    "To deliver high-performing ML models sustainably, continuous collection of feedback for re-training is also important.\n",
    "\n",
    "In this section we'll detail some reasons **why**; although joining the two processes together might be ideal; this example will demonstrate a **separate prediction review workflow** from the training data collection."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tension between process execution and model improvement\n",
    "\n",
    "As we saw when setting up the pipeline in the last notebook, there's a **post-processing step** after the ML model - whose purpose is:\n",
    "\n",
    "- To consolidate consecutive `WORD`s of the same class into a single \"entity\" detection via a simple heuristic\n",
    "- To apply (configurable) business rules to consolidate entity detections into \"fields\" on the document (e.g. selecting a single value from multiple possible matches, etc).\n",
    "\n",
    "Both of these processes are implemented in a simple Python Lambda function, and so would be technically straightforward to port into the ML model endpoint itself (in [src/inference.py](src/inference.py)). However, it's the **second one** that's important.\n",
    "\n",
    "For any use case where there's a non-trivial **gap** between what the model is trained to estimate and what the business process consumes, there's a **tension** in the review process:\n",
    "\n",
    "1. Reviewing business process fields is efficient, but does not collect model training data (although it may help us understand overall accuracy)\n",
    "2. Reviewing the model inputs & outputs directly collects training data, but:\n",
    "    - Does not directly review the accuracy of the end-to-end business process, so requires complete trust in the post-processing rules\n",
    "    - May be inefficient, as the reviewer needs to collect more information than the business process absolutely requires (e.g. having to highlight every instance of `Provider Name` in the doc, when the business process just needs to know what the name is)\n",
    "3. Splitting the review into multiple stages collects training/accuracy data for both components (ML model and business rules), but requires even more time - especially if the hand-off between the review stages might be asynchronous\n",
    "\n",
    "In many cases the efficiency of the live process is most important for customer experience and cost management, and so approach 1 is taken (as we will in this example): With collection of additional model training data handled as an additional background task.\n",
    "\n",
    "In some cases it may be possible to fully close the gap to resolve the tension and make a single offline-annotation/online-review UI work for everybody... E.g. for the credit cards use case, we might be able to:\n",
    "\n",
    "- (Add effort) Move from word classification to a **sequence-to-sequence model**, to support more complex output processing (like OCR error correction, field format standardization, grouping words into matches, etc)... *OR*\n",
    "- (Reduce scope) **Focus only on use-cases** where:\n",
    "    - Each entity class only appears once in the document, *OR* most/every detection of multiple entities is equally important to the business process (may often be the case! E.g. on forms or other well-structured use cases) *AND*\n",
    "    - Errors in OCR transcription or the heuristic rules to group matched words together are rare enough *or unpredictable enough* that there's no value in a confidence-score-based review (E.g. if \"The OCR/business rules aren't making mistakes very often, and even when they do the confidence is still high - so our online review isn't helping these issues\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A small techical challenge\n",
    "\n",
    "So what if your use case for this model is:\n",
    "\n",
    "- Seeing **high enough OCR accuracy rates** from Textract, and\n",
    "- Enjoying good success with the heuristic for **joining classified words together** into multi-word entities based on the order Textract returned them, and\n",
    "- Either having only **one match per entity type** per document; or where it's important to **always return multiple matches** if they're present?\n",
    "\n",
    "Then maybe it would make sense roll your online review and training data collection into one process! By simply trusting the post-processing logic / OCR quality, and having reviewers use the bounding box tool.\n",
    "\n",
    "**However,** there's one final hurdle: At the time of writing, the Ground Truth/A2I bounding box annotator works only for individual images, not PDFs. This means you'd also need to either:\n",
    "\n",
    "- Restrict the pipeline to processing single-page documents/images, or\n",
    "- Implement a custom box task UI capable of working with PDFs also, or\n",
    "- Orchestrate around the problem by splitting and dispatching each document to multiple single-page A2I reviews."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### In summary\n",
    "\n",
    "For some use cases of technology like this, directly using the training data annotation UI for online review could be the most efficient option.\n",
    "\n",
    "But to avoid ignoring the (potentially large) set of viable use cases where it's not practical; and to avoid introducing complexity or workarounds for handling multi-page documents; this sample presents a separate tailored online review UI.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Develop the review task template\n",
    "\n",
    "Just as with SageMaker Ground Truth, a custom task UI template has been provided and we can preview it via the SageMaker SDK.\n",
    "\n",
    "The interfaces for building SMGT and A2I templates are generally very similar but in this particular case there are some differences here from our notebook 1:\n",
    "\n",
    "1. This template accepts the list of fields dynamically at run-time, so **no extra parameter substitutions** are required in the template itself\n",
    "1. The input to this stage of the pipeline is a little more complex than a simple image + Amazon Textract result URI: So we'll use an **example JSON file** and substitute the Textract URI to match your bucket and prefix (so the displayed file will not match the extracted field content)\n",
    "1. Since the custom template here is a little more complex, we use a **NodeJS-based toolchain** to build it rather than directly authoring a browser-ready HTML file. You can find more detailed information about the reasons and practicalities for this in the [review/README.md](review/README.md) file.\n",
    "\n",
    "First, you'll need to set up the custom UI project in the `review/` folder - installing the additional dependencies it requires:\n",
    "\n",
    "> ⚠️ **If you have problems** with the node/npm build process, first try re-running the cell. In some cases we've seen intermittent permissions errors that can be resolved by retrying.\n",
    ">\n",
    "> If it still won't work, you can instead fall back to the provided legacy straight-to-HTML template instead - by setting:\n",
    ">\n",
    "> ```python\n",
    "> ui_template_file = \"review/fields-validation-legacy.liquid.html\"  # (Already exists)\n",
    "> ```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!cd review && npm install"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, build the UI HTML template from source:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!cd review && npm run build\n",
    "ui_template_file = \"review/dist/index.html\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, prepare the dummy task JSON object for usefully previewing the UI:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Load the sample input from file:\n",
    "with open(\"review/task-input.example.json\", \"r\") as f:\n",
    "    sample_obj = json.loads(f.read())\n",
    "\n",
    "# Find any `a_pdf_s3uri`, so long as it exists in your account:\n",
    "textract_s3key_root = f\"{bucket_prefix}data/raw\"\n",
    "try:\n",
    "    a_pdf_s3obj = next(filter(\n",
    "        lambda o: o.key.endswith(\".pdf\"),\n",
    "        s3.Bucket(bucket_name).objects.filter(Prefix=textract_s3key_root)\n",
    "    ))\n",
    "    a_pdf_s3uri = f\"s3://{a_pdf_s3obj.bucket_name}/{a_pdf_s3obj.key}\"\n",
    "except StopIteration as e:\n",
    "    raise ValueError(\n",
    "        f\"Couldn't find any .pdf files in s3://{bucket_name}/{textract_s3key_root}\"\n",
    "    ) from e\n",
    "\n",
    "# Substitute the PDF URI in the sample input object:\n",
    "sample_obj[\"TaskObject\"] = a_pdf_s3uri"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, render the template:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ui_render_file = \"review/render.tmp.html\"\n",
    "with open(ui_template_file, \"r\") as fui:\n",
    "    with open(ui_render_file, \"w\") as frender:\n",
    "        ui_render_resp = smclient.render_ui_template(\n",
    "            UiTemplate={ \"Content\": fui.read() },\n",
    "            Task={ \"Input\": json.dumps(sample_obj) },\n",
    "            RoleArn=role,\n",
    "        )\n",
    "        frender.write(ui_render_resp[\"RenderedContent\"])\n",
    "\n",
    "if \"Errors\" in ui_render_resp:\n",
    "    if (ui_render_resp[\"Errors\"] and len(ui_render_resp[\"Errors\"])):\n",
    "        print(ui_render_resp[\"Errors\"])\n",
    "        raise ValueError(\"Template render returned errors\")\n",
    "\n",
    "print(f\"▶️ Open {ui_render_file} and click 'Trust HTML' to see the UI in action!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Opening [review/render.tmp.html](review/render.tmp.html) and clicking **Trust HTML** in the toolbar, you should see a view something similar to the below.\n",
    "\n",
    "In this UI, the model's detections are rendered as bounding boxes over the source document with the same class colours as the original annotation view. In the right panel, you can view and amend the detected values for each field or use the checkboxes to toggle whether the field is present in the document or not. Both single- and multi-value fields are supported, and the overall confidence of detection is shown as a bar graph for each field type.\n",
    "\n",
    "![](img/a2i-custom-template-demo.png \"Screenshot of custom review UI\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up the human review workflow\n",
    "\n",
    "Similarly to a SageMaker Ground Truth labelling job, we have 3 main concerns for setting up an A2I review workflow:\n",
    "\n",
    "- **Who's** doing the labelling\n",
    "- **What** the task will look like\n",
    "- **Where** the output reviews will be stored to once the review completes (i.e. location on Amazon S3)\n",
    "\n",
    "Our **workteam** from notebook 1 should already be set up.\n",
    "\n",
    "▶️ **Check** the workteam name below matches your setup, and run the cell to store the ARN:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "workteam_name = \"just-me\"  # TODO: Update this to match yours, if different\n",
    "\n",
    "workteam_arn = util.smgt.workteam_arn_from_name(workteam_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our **template** has been tested as above, so just needs to be registered with A2I.\n",
    "\n",
    "You can use the below code to register your template and store its ARN, but can also refer to the [A2I Console worker task templates list](https://console.aws.amazon.com/a2i/home?#/worker-task-templates)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "with open(ui_template_file, \"r\") as f:\n",
    "    create_template_resp = smclient.create_human_task_ui(\n",
    "        HumanTaskUiName=\"fields-validation-1\",  # (Can change this name as you like)\n",
    "        UiTemplate={\"Content\": f.read()},\n",
    "    )\n",
    "\n",
    "task_template_arn = create_template_resp[\"HumanTaskUiArn\"]\n",
    "print(f\"Created A2I task template:\\n{task_template_arn}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To finish setting up the \"workflow\" itself, we need 2 more pieces of information:\n",
    "\n",
    "- The **location in S3** where review outputs should be stored\n",
    "- An appropriate **execution role** which will give the A2I workflow to read input documents and write review results.\n",
    "\n",
    "These are determined by the **OCR pipeline solution stack**, because the reviews bucket is created by the pipeline with event triggers to resume the next stage when reviews are uploaded.\n",
    "\n",
    "The code below should be able to look up these parameters for you automatically:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "reviews_bucket_name = config.pipeline_reviews_bucket_name\n",
    "print(reviews_bucket_name)\n",
    "reviews_role_arn = config.a2i_execution_role_arn\n",
    "print(reviews_role_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, you may **find** your pipeline solution stack from the [AWS CloudFormation Console](https://console.aws.amazon.com/cloudformation/home?#/stacks) and click through to the stack detail page. From the **Outputs** tab, you should see the `A2IHumanReviewBucketName` and `A2IHumanReviewExecutionRoleArn` values as shown below.\n",
    "\n",
    "(You may also note the `A2IHumanReviewFlowParamName`, which we'll use in the next section)\n",
    "\n",
    "![](img/cfn-stack-outputs-a2i.png \"CloudFormation stack outputs for OCR pipeline\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once these values are populated, you're ready to create your review workflow by running the code below.\n",
    "\n",
    "Note that you can also manage flows via the [A2I Human Review Workflows Console](https://console.aws.amazon.com/a2i/home?#/human-review-workflows/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "create_flow_resp = smclient.create_flow_definition(\n",
    "    FlowDefinitionName=\"ocr-fields-validation-1\",  # (Can change this name as you like)\n",
    "    HumanLoopConfig={\n",
    "        \"WorkteamArn\": workteam_arn,\n",
    "        \"HumanTaskUiArn\": task_template_arn,\n",
    "        \"TaskTitle\": \"Review OCR Field Extractions\",\n",
    "        \"TaskDescription\": \"Review and correct credit card agreement field extractions\",\n",
    "        \"TaskCount\": 1,  # One reviewer per item\n",
    "        \"TaskAvailabilityLifetimeInSeconds\": 60 * 60,  # Availability timeout\n",
    "        \"TaskTimeLimitInSeconds\": 60 * 60,  # Working timeout\n",
    "    },\n",
    "    OutputConfig={\n",
    "        \"S3OutputPath\": f\"s3://{reviews_bucket_name}/reviews\",\n",
    "    },\n",
    "    RoleArn=reviews_role_arn,\n",
    ")\n",
    "\n",
    "print(f\"Created review workflow:\\n{create_flow_resp['FlowDefinitionArn']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Integrate with the OCR pipeline\n",
    "\n",
    "Once the human review workflow is created, the final integration step is to point the pipeline at it - just as we did for our SageMaker endpoint earlier.\n",
    "\n",
    "In code, this can be done as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(f\"Configuring pipeline with review workflow: {create_flow_resp['FlowDefinitionArn']}\")\n",
    "\n",
    "ssm = boto3.client(\"ssm\")\n",
    "ssm.put_parameter(\n",
    "    Name=config.a2i_review_flow_arn_param,\n",
    "    Overwrite=True,\n",
    "    Value=create_flow_resp[\"FlowDefinitionArn\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively through the console, you would follow these steps:\n",
    "\n",
    "▶️ **Check** the `A2IHumanReviewFlowParamName` output of your OCR pipeline stack in [CloudFormation](https://console.aws.amazon.com/cloudformation/home?#/stacks) (as we did above)\n",
    "\n",
    "▶️ **Open** the [AWS Systems Manager Parameter Store console](https://console.aws.amazon.com/systems-manager/parameters/?tab=Table) and **find the review flow parameter in the list**.\n",
    "\n",
    "▶️ **Click** on the name of the parameter to open its detail page, and then on the **Edit** button in the top right corner. Set the value to the **workflow ARN** (see previous code cell in this notebook) and save the changes.\n",
    "\n",
    "![](img/ssm-a2i-param-detail.png \"Screenshot of SSM parameter detail page for human workflow\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Final testing\n",
    "\n",
    "Your OCR pipeline should now be fully functional! Let's try it out:\n",
    "\n",
    "▶️ **Log in** to the labelling portal (URL available from the [SageMaker Ground Truth Workforces Console](https://console.aws.amazon.com/sagemaker/groundtruth?#/labeling-workforces) for your correct AWS Region)\n",
    "\n",
    "![](img/smgt-find-workforce-url.png \"Screenshot of SMGT console with workforce login URL\")\n",
    "\n",
    "▶️ **Upload** one of the sample documents to your pipeline's input bucket in Amazon S3, either using the code snippets below or drag and drop in the [Amazon S3 Console](https://console.aws.amazon.com/s3/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pdfpaths = []\n",
    "for currpath, dirs, files in os.walk(\"data/raw\"):\n",
    "    if \"/.\" in currpath or \"__\" in currpath:\n",
    "        continue\n",
    "    pdfpaths += [\n",
    "        os.path.join(currpath, f) for f in files\n",
    "        if f.lower().endswith(\".pdf\")\n",
    "    ]\n",
    "pdfpaths = sorted(pdfpaths)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "test_filepath = pdfpaths[14]\n",
    "test_s3uri = f\"s3://{config.pipeline_input_bucket_name}/{test_filepath}\"\n",
    "\n",
    "!aws s3 cp '{test_filepath}' '{test_s3uri}'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "▶️ **Open up** your \"Processing Pipeline\" state machine in the [AWS Step Functions Console](https://console.aws.amazon.com/states/home?#/statemachines)\n",
    "\n",
    "After a few seconds you should find that a Step Function execution is automatically triggered and (since we enabled so many fields that at least one is always missing) the example is eventually forwarded for human review in A2I.\n",
    "\n",
    "As you'll see from the `ModelResult` field in your final *Step Output*, this pipeline produces a rich but usefully-structured output - with good opportunities for onward integration into further Step Functions steps or external systems. You can find more information and sample solutions for integrating AWS Step Functions in the [Step Functions Developer Guide](https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html).\n",
    "\n",
    "![](img/sfn-statemachine-success.png \"Screenshot of successful Step Function execution with output JSON\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Conclusion\n",
    "\n",
    "In this worked example we showed how advanced, open-source language processing models specifically tailored for document understanding can be integrated with [Amazon Textract](https://aws.amazon.com/textract/): providing a trainable, ML-driven framework for tackling more niche or complex requirements where Textract's [built-in structure extraction tools](https://aws.amazon.com/textract/features/) may not fully solve the challenges out-of-the-box.\n",
    "\n",
    "The underlying principle of the model - augmenting multi-task neural text processing architectures with positional data - is highly extensible, with potential to tackle a wide range of use cases where joint understanding of the content and presentation of text can deliver better results than considering text alone.\n",
    "\n",
    "We demonstrated how an end-to-end process automation pipeline applying this technology might look: Developing and deploying the model with [Amazon SageMaker](https://aws.amazon.com/sagemaker/), building a serverless workflow with [AWS Step Functions](https://aws.amazon.com/step-functions/) and [AWS Lambda](https://aws.amazon.com/lambda/), and driving quality with human review of low-confidence documents through [Amazon Augmented AI](https://aws.amazon.com/augmented-ai/).\n",
    "\n",
    "Thanks for following along, and for more information, don't forget to check out:\n",
    "\n",
    "- The other published [Amazon Textract Examples](https://docs.aws.amazon.com/textract/latest/dg/other-examples.html) listed in the [Textract Developer Guide](https://docs.aws.amazon.com/textract/latest/dg/what-is.html)\n",
    "- The extensive repository of [Amazon SageMaker Examples](https://github.com/aws/amazon-sagemaker-examples) and usage documentation in the [SageMaker Python SDK User Guide](https://sagemaker.readthedocs.io/en/stable/) - as well as the [SageMaker Developer Guide](https://docs.aws.amazon.com/sagemaker/index.html)\n",
    "- The wide range of other open algorithms and models published by [HuggingFace Transformers](https://huggingface.co/transformers/), and their specific documentation on [using the library with SageMaker](https://huggingface.co/transformers/sagemaker.html)\n",
    "- The conversational AI and NLP area (and others) of Amazon's own [Amazon.Science](https://www.amazon.science/conversational-ai-natural-language-processing) blog\n",
    "\n",
    "Happy building!"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science 3.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-data-science-310-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}