{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Running the fleet of Virtual Wind Turbines and Edge Devices\n", "\n", "**SageMaker Studio Kernel**: Data Science\n", "\n", "After deploying the inference app on the 5 IoT Core devices(EC2 instances, in this case), its time to set up the virtual fleet of turbines and the simulator app along with it. In this exercise you will run a local application written in Python3 that simulates 5 Wind Turbines. This application will interact with IoT Core devices via IoT MQTT to exchange information.\n", "\n", "Here you'll be the **Wind Turbine Farm Operator**. It's possible to visualize the data flowing from the sensors to the ML Model and analyze the anomalies. Also, you'll be able to inject noise (pressing some buttons) in the data to simulate potential anomalies with the equipment.\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
STEP-BY-STEPAPPLICATION ARCHITECTURE
\n", "\n", "The components of the Simulator are:\n", " - [Simulator](simulator/simulator.py): Program that launches the virtual wind turbines and the edge devices. It also maintains a simple dashboard to show the data which is being sent for anomaly detection to the detector.\n", " - [Turbine](simulator/turbine.py): Virtual Wind Turbine. It reads the raw data collected from the 3D Prited Mini Turbine and stream it as a circular buffer. It also has a graphical representation in **IPython Widgets** that is rendered by the Simulator/Dashboard. It contains all the business logic of maintaining the virtual turbine.\n", " - [MQTT Client](simulator/mqttclient.py): An abstraction layer for MQTT communication providing convenience methods to publish and subscribe to topics." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First, let us restore variables stored in the previous exercise" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%store -r project_id\n", "%store -r deployment_id" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Install some dependencies" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip install ipywidgets\n", "!pip install awsiotsdk" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Download the sensors data (streamed by the virtual wind turbines)\n", "\n", "The below cell checks if the data is already downloaded.\n", "# \n", "\n", " if not, then it downloads the data. Otherwise it skips the downloading." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import os.path\n", "\n", "if os.path.isfile('../../data/dataset_wind.csv.gz') is not True:\n", " print('file not present, downloading now..')\n", " os.system(\"mkdir -p ../../data\")\n", " os.system(\"curl https://aws-ml-blog.s3.amazonaws.com/artifacts/monitor-manage-anomaly-detection-model-wind-turbine-fleet-sagemaker-neo/dataset_wind_turbine.csv.gz -o ../../data/dataset_wind.csv.gz\")\n", " print('download complete !!')\n", "else:\n", " print('file already present, no need to download again !')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "project_name='<>'\n", "\n", "import boto3\n", "\n", "sm_client = boto3.client('sagemaker')\n", "project_id = sm_client.describe_project(ProjectName=project_name)['ProjectId']\n", "bucket_name = 'sagemaker-wind-turbine-farm-%s' % project_id" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Simulating The Wind Turbine Farm\n", "Now its time to run our simulator and start playing with the turbines and with the anomalies\n", " > After clicking on **Start**, each turbine will start buffering some data. It takes a few seconds but after completing this process, the application runs in real-time \n", " > Try to press some buttons while the simulation is running, to inject noise in the data and see some anomalies \n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import os\n", "import sys\n", "sys.path.insert(0, os.path.join(os.path.realpath('../../'),'simulator'))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import turbine\n", "import simulator\n", "import mqttclient\n", "from importlib import reload\n", "import logging\n", "\n", "logging.basicConfig(level=logging.INFO)\n", "\n", "reload(mqttclient)\n", "reload(turbine)\n", "reload(simulator)\n", "\n", "simulator = simulator.WindTurbineFarmSimulator(5)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "simulator.start()\n", "simulator.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Once your experiment is finished, stop the simulator. This will in turn also stops/halts all the turbines." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "simulator.halt()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " > If you want to experiment with the deployment process, with the wind farm running, go back to [Exercise #2](02-Training-with-Pytorch.ipynb#Building-the-Deployment-Package-SageMaker-Edge-Manager) (in the last cells), go back to lab02 notebook and deploy a new `version` of the model, create a new greengrass component version of the model and create a new deployment version deploying this new model version." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Cleanup\n", "Run the next cell only if you already finished exploring/hacking the content of the workshop. \n", "This code will delete all the resouces created so far, including the **SageMaker Project** you've created" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import boto3\n", "import time\n", "from shutil import rmtree\n", "\n", "iot_client = boto3.client('iot')\n", "sm_client = boto3.client('sagemaker')\n", "s3_resource = boto3.resource('s3')\n", "ggv2_client = boto3.client('greengrassv2')\n", "\n", "policy_name='WindTurbineFarmPolicy-%s' % project_id\n", "thing_group_name='WindTurbineFarm-%s' % project_id\n", "fleet_name='wind-turbine-farm-%s' % project_id\n", "\n", "account_id = boto3.client('sts').get_caller_identity().get('Account')\n", "region = boto3.session.Session().region_name\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#Cancel the deployment of greengrass v2\n", "ggv2_client.cancel_deployment(deploymentId=deployment_id)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#Delete the core devices:\n", "for i in range(5):\n", " ggv2_client.delete_core_device(coreDeviceThingName='edge-device-'+str(i))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Delete all files from the S3 Bucket\n", "s3_resource.Bucket(bucket_name).objects.all().delete()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# delete the ggv2 components\n", "ggv2_client.delete_component(arn='arn:aws:greengrass:{}:{}:components:aws.samples.windturbine.detector:versions:1.0.0'.format(region,account_id))\n", "ggv2_client.delete_component(arn='arn:aws:greengrass:{}:{}:components:aws.samples.windturbine.model:versions:1.0.0'.format(region,account_id))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# now deregister the devices from the fleet\n", "resp = sm_client.list_devices(DeviceFleetName=fleet_name)\n", "devices = [d['DeviceName'] for d in resp['DeviceSummaries']]\n", "if len(devices) > 0:\n", " sm_client.deregister_devices(DeviceFleetName=fleet_name, DeviceNames=devices)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# now deregister the devices from the fleet\n", "for i,cert_arn in enumerate(iot_client.list_targets_for_policy(policyName=policy_name)['targets']):\n", " for t in iot_client.list_principal_things(principal=cert_arn)['things']:\n", " iot_client.detach_thing_principal(thingName=t, principal=cert_arn)\n", " iot_client.detach_policy(policyName=policy_name, target=cert_arn)\n", " certificateId = cert_arn.split('/')[-1]\n", " iot_client.update_certificate(certificateId=certificateId, newStatus='INACTIVE')\n", " iot_client.delete_certificate(certificateId=certificateId)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "iot_client.delete_role_alias(roleAlias='SageMakerEdge-%s' % fleet_name)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "iot_client.delete_thing_group(thingGroupName=thing_group_name)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "if os.path.isdir('agent'): rmtree('agent')\n", "sm_client.delete_project(ProjectName=project_name)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Mission Complete! \n", "Congratulations! :)" ] } ], "metadata": { "instance_type": "ml.t3.medium", "kernelspec": { "display_name": "Python 3 (Data Science)", "language": "python", "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:eu-west-1:470317259841:image/datascience-1.0" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.10" } }, "nbformat": 4, "nbformat_minor": 4 }