{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "fb8b9566",
   "metadata": {},
   "source": [
    "# **Amazon Lookout for Equipment** - SDK Tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "424c147e",
   "metadata": {},
   "source": [
    "## Initialization\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab4f8f47",
   "metadata": {},
   "source": [
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2bbf64c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install lookoutequipment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2eb236d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.ticker as mtick\n",
    "import numpy as np\n",
    "import os\n",
    "import pandas as pd\n",
    "import sagemaker\n",
    "import sys\n",
    "import time\n",
    "\n",
    "from lookoutequipment import plot, dataset, model, evaluation, scheduler"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c776025",
   "metadata": {},
   "source": [
    "### Parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56d60b9b",
   "metadata": {},
   "source": [
    "<span style=\"color: white; background-color: OrangeRed; padding: 0px 15px 0px 15px; border-radius: 20px;\">**Note:** Update the value of the **bucket** and **prefix** variables below **before** running the following cell</span>\n",
    "\n",
    "Make sure the IAM role used to run your notebook has access to the chosen bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b09a33ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "bucket   = '<<YOUR-BUCKET>>'\n",
    "prefix   = '<<YOUR_PREFIX>>/' # Keep the trailing slash at the end\n",
    "\n",
    "plt.style.use('Solarize_Light2')\n",
    "plt.rcParams['lines.linewidth'] = 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "390c4f28",
   "metadata": {},
   "source": [
    "### Dataset preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1011425",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = dataset.load_dataset(dataset_name='expander', target_dir='expander-data')\n",
    "dataset.upload_dataset('expander-data', bucket, prefix)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62e1fe70",
   "metadata": {},
   "source": [
    "## Role definition\n",
    "--- \n",
    "Before you can run this notebook (for instance, from a SageMaker environment), you will need:\n",
    "* To allow SageMaker to run Lookout for Equipment API calls\n",
    "* To allow Amazon Lookout for Equipment to access your training data (located in the bucket and prefix defined in the previous cell)\n",
    "\n",
    "### Authorizing SageMaker to make Lookout for Equipment calls\n",
    "You need to ensure that this notebook instance has an IAM role which allows it to call the Amazon Lookout for Equipment APIs:\n",
    "\n",
    "1. In your IAM console, look for the SageMaker execution role endorsed by your notebook instance (a role with a name like `AmazonSageMaker-ExecutionRole-yyyymmddTHHMMSS`)\n",
    "2. On the `Permissions` tab, click on `Attach policies`\n",
    "3. In the Filter policies search field, look for `AmazonLookoutEquipmentFullAccess`, tick the checkbox next to it and click on `Attach policy`\n",
    "\n",
    "Your notebook has now the ability to call any Lookout for Equipment APIs.\n",
    "\n",
    "### Give access to your S3 data to Lookout for Equipment\n",
    "When Lookout for Equipment will run, it will try to access your S3 data at several occasions:\n",
    "\n",
    "* When ingesting the training data\n",
    "* At training time when accessing the label data\n",
    "* At inference time to run the input data and output the results\n",
    "\n",
    "To enable these access, you need to create a role that Lookout for Equipment can endorse by following these steps:\n",
    "\n",
    "1. Log in again to your [**IAM console**](https://console.aws.amazon.com/iamv2/home)\n",
    "2. On the left menu bar click on `Roles` and then on the `Create role` button located at the top right\n",
    "3. On the create role screen, selected `AWS Service` as the type of trusted entity\n",
    "4. In the following section (`Choose a use case`), locate `SageMaker` and click on the service name. Not all AWS services appear in these ready to configure use cases and this is why we are using SageMaker as the baseline for our new role. In the next steps, we will adjust the role created to configure it specifically for Amazon Lookout for Equipment.\n",
    "5. Click on the `Next` button until you reach the last step (`Review`): give a name and a description to your role (for instance `LookoutEquipmentS3AccessRole`)\n",
    "6. Click on `Create role`: your role is created and you are brought back to the list of existing role\n",
    "7. In the search bar, search for the role you just created and choose it from the returned result to see a summary of your role\n",
    "8. At the top of your screen, you will see a role ARN field: **copy this ARN and paste it in the following cell, replacing the `<<YOUR_ROLE_ARN>>` string below**\n",
    "9. Click on the cross at the far right of the `AmazonSageMakerFullAccess` managed policy to remove this permission for this role as we don't need it.\n",
    "10. Click on `Add inline policy` and then on the `JSON` tab. Then fill in the policy with the following document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d87b7ae8",
   "metadata": {},
   "outputs": [],
   "source": [
    "print((\n",
    "    '{\\n'\n",
    "    '    \"Version\": \"2012-10-17\",\\n'\n",
    "    '    \"Statement\": [\\n'\n",
    "    '        {\\n'\n",
    "    '            \"Effect\": \"Allow\",\\n'\n",
    "    '            \"Action\": [\\n'\n",
    "    '                \"s3:ListBucket\",\\n'\n",
    "    '                \"s3:GetObject\",\\n'\n",
    "    '                \"s3:PutObject\"\\n'\n",
    "    '            ],\\n'\n",
    "    '            \"Resource\": [\\n'\n",
    "    f'                \"arn:aws:s3:::{bucket}/*\",\\n'\n",
    "    f'                \"arn:aws:s3:::{bucket}\"\\n'\n",
    "    '            ]\\n'\n",
    "    '        }\\n'\n",
    "    '    ]\\n'\n",
    "    '}'\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cdf172e",
   "metadata": {},
   "source": [
    "11. Give a name to your policy (for instance: `LookoutEquipmentS3AccessPolicy`) and click on `Create policy`.\n",
    "12. On the `Trust relationships` tab, choose `Edit trust relationship`.\n",
    "13. Under policy document, replace the whole policy by the following document and click on the `Update Trust Policy` button on the bottom right:\n",
    "\n",
    "```json\n",
    "{\n",
    "  \"Version\": \"2012-10-17\",\n",
    "  \"Statement\": [\n",
    "    {\n",
    "      \"Sid\": \"\",\n",
    "      \"Effect\": \"Allow\",\n",
    "      \"Principal\": {\n",
    "        \"Service\": \"lookoutequipment.amazonaws.com\"\n",
    "      },\n",
    "      \"Action\": \"sts:AssumeRole\"\n",
    "    }\n",
    "  ]\n",
    "}\n",
    "```\n",
    "\n",
    "And you're done! When Amazon Lookout for Equipment will try to read the datasets you just uploaded in S3, it will request permissions from IAM by using the role we just created:\n",
    "1. The **trust policy** allows Lookout for Equipment to assume this role.\n",
    "2. The **inline policy** specifies that Lookout for Equipment is authorized to list and access the objects in the S3 bucket you created earlier.\n",
    "\n",
    "<span style=\"color: white; background-color: OrangeRed; padding: 0px 15px 0px 15px; border-radius: 20px;\">Don't forget to update the **role_arn** variable below with the ARN of the role you just create **before** running the following cell</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ffcbd896",
   "metadata": {},
   "outputs": [],
   "source": [
    "role_arn = '<<YOUR_ROLE_ARN>>'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c25e6b33",
   "metadata": {},
   "source": [
    "## Lookout for Equipment end-to-end walkthrough\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae506603",
   "metadata": {},
   "source": [
    "### Dataset creation and data ingestion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41227611",
   "metadata": {},
   "outputs": [],
   "source": [
    "lookout_dataset = dataset.LookoutEquipmentDataset(\n",
    "    dataset_name='my_dataset',\n",
    "    access_role_arn=role_arn,\n",
    "    component_root_dir=f's3://{bucket}/{prefix}training-data'\n",
    ")\n",
    "lookout_dataset.create()\n",
    "response = lookout_dataset.ingest_data(bucket, prefix + 'training-data/', wait=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ddb8d645",
   "metadata": {},
   "source": [
    "### Building an anomaly detection model\n",
    "#### Model training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d955521",
   "metadata": {},
   "outputs": [],
   "source": [
    "lookout_model = model.LookoutEquipmentModel(model_name='my_model', \n",
    "                                            dataset_name='my_dataset')\n",
    "lookout_model.set_time_periods(data['evaluation_start'],\n",
    "                               data['evaluation_end'],\n",
    "                               data['training_start'],\n",
    "                               data['training_end'])\n",
    "lookout_model.set_label_data(bucket=bucket, \n",
    "                             prefix=prefix + 'label-data/',\n",
    "                             access_role_arn=role_arn)\n",
    "lookout_model.set_target_sampling_rate(sampling_rate='PT30M')\n",
    "\n",
    "response = lookout_model.train()\n",
    "lookout_model.poll_model_training(sleep_time=300)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91d5cb36",
   "metadata": {},
   "source": [
    "#### Trained model evaluation overview"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6da1202",
   "metadata": {},
   "outputs": [],
   "source": [
    "LookoutDiagnostics = evaluation.LookoutEquipmentAnalysis(model_name='my_model', tags_df=data['data'])\n",
    "predicted_ranges = LookoutDiagnostics.get_predictions()\n",
    "labels_fname = os.path.join('expander-data', 'labels.csv')\n",
    "labeled_range = LookoutDiagnostics.get_labels(labels_fname)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "421eb03b",
   "metadata": {},
   "outputs": [],
   "source": [
    "TSViz = plot.TimeSeriesVisualization(timeseries_df=data['data'], data_format='tabular')\n",
    "TSViz.add_signal(['signal-028'])\n",
    "TSViz.add_labels(labeled_range)\n",
    "TSViz.add_predictions([predicted_ranges])\n",
    "TSViz.add_train_test_split(data['evaluation_start'])\n",
    "TSViz.add_rolling_average(60*24)\n",
    "TSViz.legend_format = {'loc': 'upper left', 'framealpha': 0.4, 'ncol': 3}\n",
    "fig, axis = TSViz.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ac41bb2",
   "metadata": {},
   "source": [
    "#### Plot signal distribution\n",
    "You might be curious about why Amazon Lookout for Equipment detected an anomalous event. Sometime, looking at a few of the time series is enough. But sometime, you need to dig deeper.\n",
    "\n",
    "The following function, aggregate the signal importance of every signals over the evaluation period and sum these contributions over time for each signal. Then, it takes the top 8 signals and plot two distributions: one with the values each signal takes during the normal periods (present in the evaluation range) and a second one with the values taken during all the anomalous events detected in the evaluation range. This will help you visualize any significant shift of values for the top contributing signals.\n",
    "\n",
    "You can also restrict these histograms over a specific range of time by setting the `start` and `end` arguments of the following function with datetime values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8bffac83",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = TSViz.plot_histograms(freq='5min', top_n=8)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f45ec96",
   "metadata": {},
   "source": [
    "### Scheduling inferences\n",
    "#### Preparing inferencing data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c38366a",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.prepare_inference_data(\n",
    "    root_dir='expander-data',\n",
    "    sample_data_dict=data,\n",
    "    bucket=bucket,\n",
    "    prefix=prefix,\n",
    "    start_date='2015-11-21 04:00:00',\n",
    "    num_sequences=12\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de8d0573",
   "metadata": {},
   "source": [
    "#### Configuring and starting a scheduler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "34e4af0a",
   "metadata": {},
   "outputs": [],
   "source": [
    "lookout_scheduler = scheduler.LookoutEquipmentScheduler(\n",
    "    scheduler_name='my_scheduler',\n",
    "    model_name='my_model'\n",
    ")\n",
    "\n",
    "scheduler_params = {\n",
    "    'input_bucket': bucket,\n",
    "    'input_prefix': prefix + 'inference-data/input/',\n",
    "    'output_bucket': bucket,\n",
    "    'output_prefix': prefix + 'inference-data/output/',\n",
    "    'role_arn': role_arn,\n",
    "    'upload_frequency': 'PT30M',\n",
    "    'delay_offset': None,\n",
    "    'timezone_offset': '+00:00',\n",
    "    'component_delimiter': '_',\n",
    "    'timestamp_format': 'yyyyMMddHHmmss'\n",
    "}\n",
    "\n",
    "lookout_scheduler.set_parameters(**scheduler_params)\n",
    "response = lookout_scheduler.create()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37d532ff",
   "metadata": {},
   "source": [
    "Let's now wait for the scheduler to generate the first execution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4fa455b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "execution_summaries = []\n",
    "\n",
    "while len(execution_summaries) == 0:\n",
    "    execution_summaries = lookout_scheduler.list_inference_executions()\n",
    "    \n",
    "    if len(execution_summaries) == 0:\n",
    "        print('WAITING FOR THE FIRST INFERENCE EXECUTION')\n",
    "        time.sleep(60)\n",
    "        \n",
    "    else:\n",
    "        print('FIRST INFERENCE EXECUTED\\n')\n",
    "        break\n",
    "            \n",
    "execution_summaries"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a355442d",
   "metadata": {},
   "source": [
    "#### Post-processing the inference results\n",
    "Make sure you have some inference results before you run the next cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8916ae07",
   "metadata": {},
   "outputs": [],
   "source": [
    "results_df = lookout_scheduler.get_predictions()\n",
    "results_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b74d1ed8",
   "metadata": {},
   "outputs": [],
   "source": [
    "event_details = pd.DataFrame(results_df.iloc[0, 1:]).reset_index()\n",
    "fig, ax = plot.plot_event_barh(event_details, fig_width=12)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fe014ec",
   "metadata": {},
   "source": [
    "## Cleanup\n",
    "---\n",
    "The next cell deletes all the artifacts created by this notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "410371e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.delete_dataset(dataset_name='my_dataset', delete_children=True, verbose=True)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}