{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "50f34553",
   "metadata": {},
   "source": [
    "Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.\n",
    "\n",
    "SPDX-License-Identifier: Apache-2.0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5471b7d7",
   "metadata": {},
   "source": [
    "# Explore the BATtle of the Attack Detection ALgorithms (BATADAL) data set\n",
    "\n",
    "The BATtle of the Attack Detection ALgorithms (BATADAL) data set is from a real-world, medium-sized water distribution system operated through programmable logic controllers and a supervisory control and data acquisition (SCADA) system. The data contains simulated SCADA observations with anomalies at the sensor level that mimics real world attacks on water management systems. It consists of multiple time series containing sensor values of 43 system variables, with each time step labeled as attack or normal operation. The objective is to identify the attacks/anomalies at the time step level."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22c51837",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "\n",
    "1. Load BATADAL data\n",
    "2. Examine data columns\n",
    "3. Visualize specific sensor time series\n",
    "  * Train with no anomalies\n",
    "  * Train with anomalies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f44f3d33",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from collections import defaultdict"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b400742d",
   "metadata": {},
   "source": [
    "### BATADAL data \n",
    "The data set contains hourly historical SCADA operations of a water distribution network. It contains three CSVs: Training Data set 1: year long simulation containing no anomalies, Training Data set 2: 6 month simulation with anomalies, and Test Data set: 3 month simulation with anomalies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c63c66d",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_no_anom = pd.read_csv(\"../../data/01_raw/iot/BATADAL_dataset03_train_no_anomaly.csv\")\n",
    "train_some_anom = pd.read_csv(\"../../data/01_raw/iot/BATADAL_dataset04_train_some_anomaly.csv\")\n",
    "test_with_anom = pd.read_csv(\"../../data/01_raw/iot/BATADAL_test_dataset_some_anomaly.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81e453a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# has leading white space\n",
    "train_some_anom.columns = train_some_anom.columns.str.strip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7adca29e",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_no_anom.shape, train_some_anom.shape, test_with_anom.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c64da1bc",
   "metadata": {},
   "source": [
    "### Data columns\n",
    "\n",
    "The columns contain info columns datetime and attack, and SCADA readings for 43 water system variables. Columns denoted as L_* are the tank water levels (in meters), columns denoted as P_* are the inlet and outlet pressure for the actuated valve and pumping stations, variables noted as F_* and S_* are the flow (in liters per second) and status of the actuated valve.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "167382a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_no_anom.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aed1b365",
   "metadata": {},
   "outputs": [],
   "source": [
    "SENSOR_COLS = [c for c in train_no_anom.columns if c not in [\"DATETIME\", \"ATT_FLAG\", \"timestamp\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a633a885",
   "metadata": {},
   "source": [
    "# Visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af7fbb15",
   "metadata": {},
   "source": [
    "# Train no anomaly\n",
    "\n",
    "Let's look at a few days of normal operation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "089898c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_no_anom[\"timestamp\"] = pd.to_datetime(train_no_anom[\"DATETIME\"], format=\"%d/%m/%y %H\")\n",
    "train_no_anom = train_no_anom.set_index([\"timestamp\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4dc154e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_no_anom.index.min(), train_no_anom.index.max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5bda904",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_no_anom['2014-03-07': '2014-03-11'][[c for c in SENSOR_COLS if \"L_T\" in c]].plot(figsize=(20,10), grid=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42d0cab6",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_no_anom['2014-03-07': '2014-03-11'][[c for c in SENSOR_COLS if \"P_\" in c]].plot(figsize=(20,10), grid=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4068becb",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_no_anom['2014-03-07': '2014-03-11'][[c for c in SENSOR_COLS if \"F_\" in c]].plot(figsize=(20,10), grid=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4badac3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_no_anom['2014-03-07': '2014-03-11'][[c for c in SENSOR_COLS if \"S_\" in c]].plot(figsize=(20,10), grid=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e79b9303",
   "metadata": {},
   "source": [
    "# Train with anomaly \n",
    "\n",
    "The anomalies are described in [Attacks_TrainingDataset2.png](http://www.batadal.net/images/Attacks_TrainingDataset2.png).\n",
    "\n",
    "Let's look at some details around attack ID 3. Attack 3 lasts for 60 hours from October 9, 2016 at 9 AM to October 11, 2016 at 8 PM. The tank T1 is non-empty during this time, but the attackers trick the system into thinking that L_T1 (tank water levels in m) is low. This results in pumps 1 and 2 (PU1 and PU2) to remain on, resulting in an overflow of tank 1.\n",
    "\n",
    "To hide this attack, the attackers use a polyline adjustment on L_T1. Let's see if we can see it visually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d646ad2b",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_some_anom[\"timestamp\"] = pd.to_datetime(train_some_anom[\"DATETIME\"], format=\"%d/%m/%y %H\")\n",
    "train_some_anom = train_some_anom.set_index([\"timestamp\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "308db220",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_some_anom['2016-10-08':'2016-10-12'][[\"F_PU1\", \"F_PU2\"]].plot(figsize=(20,10), grid=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "137a63d6",
   "metadata": {},
   "source": [
    "The attackers hide their behavior by masking the true levels of L_T1. In the plot, we see that L_T1 remains reasonable, and unless someone is physically at Tank 1, the anomaly behavior can't be seen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65ea9b67",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_some_anom['2016-10-08':'2016-10-12'][[\"S_PU1\", \"S_PU2\", \"L_T1\"]].plot(figsize=(20,10), grid=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73eff2d3",
   "metadata": {},
   "source": [
    "# References\n",
    "\n",
    "Riccardo Taormina and Stefano Galelli and Nils Ole Tippenhauer and Elad Salomons and Avi Ostfeld and Demetrios G. Eliades and Mohsen Aghashahi and Raanju Sundararajan and Mohsen Pourahmadi and M. Katherine Banks and B. M. Brentan and Enrique Campbell and G. Lima and D. Manzi and D. Ayala-Cabrera and M. Herrera and I. Montalvo and J. Izquierdo and E. Luvizotto and Sarin E. Chandy and Amin Rasekh and Zachary A. Barker and Bruce Campbell and M. Ehsan Shafiee and Marcio Giacomoni and Nikolaos Gatsis and Ahmad Taha and Ahmed A. Abokifa and Kelsey Haddad and Cynthia S. Lo and Pratim Biswas and M. Fayzul K. Pasha and Bijay Kc and Saravanakumar Lakshmanan Somasundaram and Mashor Housh and Ziv Ohar; \"The Battle Of The Attack Detection Algorithms: Disclosing Cyber Attacks On Water Distribution Networks.\" Journal of Water Resources Planning and Management, 144 (8), August 2018"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a904c545",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}