{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data exploration/visualization\n",
    "\n",
    "**SageMaker Studio Kernel**: Data Science\n",
    "\n",
    "The challenge we're trying to address here is to detect anomalies in the components of a Wind Turbine. Each wind turbine has many sensors that reads data like:\n",
    " - Internal & external temperature\n",
    " - Wind speed\n",
    " - Rotor speed\n",
    " - Air pressure\n",
    " - Voltage (or current) in the generator\n",
    " - Vibration in the GearBox (using an IMU -> Accelerometer + Gyroscope)\n",
    "\n",
    "So, depending on the types of the anomalies we want to detect, we need to select one or more features and then prepare a dataset that 'explains' the anomalies. We are interested in three types of anomalies:\n",
    " - Rotor speed (when the rotor is not in an expected speed)\n",
    " - Produced voltage (when the generator is not producing the expected voltage)\n",
    " - Gearbox vibration (when the vibration of the gearbox is far from the expected)\n",
    " \n",
    "All these three anomalies (or violations) depend on many variables while the turbine is working. Thus, in order to address that, let's use a ML model called [Autoencoder](https://en.wikipedia.org/wiki/Autoencoder), with correlated features. This model is unsupervised. It learns the latent representation of the dataset and tries to predict (regression) the same tensor given as input. The strategy then is to use a dataset collected from a normal turbine (without anomalies). The model will then learn **'what is a normal turbine'**. When the sensors readings of a malfunctioning turbine is used as input, the model will not be able to rebuild the input, predicting something with a high error and detected as an anomaly.\n",
    "\n",
    "The sequence of the sensors readings can be seen as a time-series dataset and therefore we observe a high correlation between neighbour samples. We can explore this by reformatting the data as a multidimensional tensor. We'll create a temporal encoding of six features in 10x10 steps of 250ms each. 250ms is the interval computed using 5 samples (the time interval between each sample is ~50ms). It means that we will create a tensor with a shape of 6x10x10.\n",
    "\n",
    "![Tensor](imgs/tensor.png)\n",
    "\n",
    "In the tensor above, each color is a different feature, encoded in 100 (10x10) timesteps (from the current reading to the past in a sliding window).\n",
    "\n",
    "Let's start preparing our dataset, then."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install this lib to improve data visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!pip install -U matplotlib==3.4.1 seaborn==0.11.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### And download the sensors (raw) data\n",
    "This dataset was created by mini wind turbines, 3D printed and assembled for experimenting with ML@Edge. If you're interested on building your own 3D printed mini wind turbines, please check the link to this project in the home of this workshop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p data\n",
    "!curl https://aws-ml-blog.s3.amazonaws.com/artifacts/monitor-manage-anomaly-detection-model-wind-turbine-fleet-sagemaker-neo/dataset_wind_turbine.csv.gz -o data/dataset_wind.csv.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's take a look on the data\n",
    "Loading the dataset using Pandas..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format='retina'\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from datetime import datetime\n",
    "## preprocessing is the data preparation script we'll use in our automated ML Pipeline\n",
    "## here, it will be just a loaded library\n",
    "import preprocessing as dataprep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parser = lambda date: datetime.strptime(date, '%Y-%m-%dT%H:%M:%S.%f+00:00')\n",
    "df = pd.read_csv('data/dataset_wind.csv.gz', compression=\"gzip\", sep=',', low_memory=False, parse_dates=[ 'eventTime'], date_parser=parser)\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Features:\n",
    "  - **nanoId**: id of the edge device that collected the data\n",
    "  - **turbineId**: id of the turbine that produced this data\n",
    "  - **arduino_timestamp**: timestamp of the arduino that was operating this turbine\n",
    "  - **nanoFreemem**: amount of free memory in bytes\n",
    "  - **eventTime**: timestamp of the row\n",
    "  - **rps**: rotation of the rotor in Rotations Per Second\n",
    "  - **voltage**: voltage produced by the generator in milivolts\n",
    "  - **qw, qx, qy, qz**: quaternion angular acceleration\n",
    "  - **gx, gy, gz**: gravity acceleration\n",
    "  - **ax, ay, az**: linear acceleration\n",
    "  - **gearboxtemp**: internal temperature\n",
    "  - **ambtemp**: external temperature\n",
    "  - **humidity**: air humidity\n",
    "  - **pressure**: air pressure\n",
    "  - **gas**: air quality\n",
    "  - **wind_speed_rps**: wind speed in Rotations Per Second"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## we will select the following features to prepare our dataset\n",
    "## with these features we have parameters for vibration, rotation and voltage\n",
    "quat=['qx', 'qy', 'qz', 'qw']\n",
    "rot=['wind_speed_rps', 'rps']\n",
    "volt=['voltage']\n",
    "features = quat + rot + volt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ploting the vibration data, just to have an idea"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[quat[:3]].iloc[1910:2000].plot(figsize=(20,10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now, plot the rotation of the turbine and the wind speed in RPS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[rot].iloc[1910:2000].plot(figsize=(20,10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finally, plot the voltage readings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[volt].iloc[1910:2000].plot(figsize=(20,10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data preparation\n",
    "The raw data for rotation is formated as angular acceleration using a Quaternion representation. We can convert it to Euler angles to make it easier to understand."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('now converting quat to euler...')\n",
    "roll,pitch,yaw = [], [], []\n",
    "for idx, row in df.iterrows():\n",
    "    r,p,y = dataprep.euler_from_quaternion(row['qx'], row['qy'], row['qz'], row['qw'])\n",
    "    roll.append(r)\n",
    "    pitch.append(p)\n",
    "    yaw.append(y)\n",
    "df['roll'] = roll\n",
    "df['pitch'] = pitch\n",
    "df['yaw'] = yaw"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Then, we can denoise and normalize the data to complete the process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train = df.copy()\n",
    "\n",
    "# select the features\n",
    "features = ['roll', 'pitch', 'yaw', 'wind_speed_rps', 'rps', 'voltage']\n",
    "\n",
    "# get the std for denoising\n",
    "raw_std = df_train[features].std()\n",
    "for f in features:\n",
    "    df_train[f] = dataprep.wavelet_denoise(df_train[f].values, 'db6', raw_std[f])#[:-1]\n",
    "\n",
    "# normalize\n",
    "training_std = df_train[features].std()\n",
    "training_mean = df_train[features].mean()\n",
    "df_train = (df_train[features] - training_mean) / training_std\n",
    "\n",
    "df_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Alright, this is our dataset. Let's just plot the original vs the prepared data\n",
    "**Original Data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[features][:2000].plot(figsize=(20,10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Denoised & Normalized Data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train[:2000].plot(figsize=(20,10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " > There is too much noise in the raw data, specially in the accelerometer + gyroscope readings  \n",
    " > This process is important to remove the impurity and make the model more efficient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "corr = df[features].corr()\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(15, 8))\n",
    "\n",
    "sns.heatmap(corr, annot=True, fmt=\"f\",\n",
    "            xticklabels=corr.columns.values,\n",
    "            yticklabels=corr.columns.values,\n",
    "            ax=ax)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, the linear correlation between **rps (rotation speed)** and **voltage** is high. We need to keep both, given the model needs to understand what is a normal relationship between these two + other features.\n",
    "\n",
    "Alright! Now you can start exercise #2: create a ML pipeline to train your model and then deploy it to the edge devices.\n",
    "\n",
    " > [Exercise 02](02%20-%20Training%20with%20Pytorch.ipynb)"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/datascience-1.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}