{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Secure Operations with EC2 and AWS Systems Manger\n",
    "\n",
    "![SSM](../../docs/assets/images/ec2_ssm.png)\n",
    "\n",
    "In this session, we will be creating an EC2 instance using CloudFormation to show you how to automate your [Infrastructure as Code](https://en.wikipedia.org/wiki/Infrastructure_as_code). We will also be leveraging AWS Systems Manager to show you how to gain shell access to your EC2 instances without using a bastion host. To get started we willuse a pre-created CloudFormation template in yaml and use the [Python AWS SDK](https://aws.amazon.com/sdk-for-python/) to launch the template.\n",
    "\n",
    "#### [Amazon Elastic Compute Cloud](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html)\n",
    "Amazon Elastic Compute Cloud (Amazon EC2) provides scalable computing capacity in the Amazon Web Services (AWS) cloud. Using Amazon EC2 eliminates your need to invest in hardware up front, so you can develop and deploy applications faster. You can use Amazon EC2 to launch as many or as few virtual servers as you need, configure security and networking, and manage storage. Amazon EC2 enables you to scale up or down to handle changes in requirements or spikes in popularity, reducing your need to forecast traffic.\n",
    "\n",
    "#### [AWS Systems Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/what-is-systems-manager.html)\n",
    "AWS Systems Manager is an AWS service that you can use to view and control your infrastructure on AWS. Using the Systems Manager console, you can view operational data from multiple AWS services and automate operational tasks across your AWS resources. Systems Manager helps you maintain security and compliance by scanning your managed instances and reporting on (or taking corrective action on) any policy violations it detects.\n",
    "\n",
    "A managed instance is a machine that has been configured for use with Systems Manager. Systems Manager also helps you configure and maintain your managed instances. Supported machine types include Amazon EC2 instances, on-premises servers, and virtual machines (VMs), including VMs in other cloud environments. Supported operating system types include Windows Server, multiple distributions of Linux, and Raspbian.\n",
    "\n",
    "#### [CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html)\n",
    "AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less time managing those resources and more time focusing on your applications that run in AWS. You create a template that describes all the AWS resources that you want (like Amazon EC2 instances or Amazon RDS DB instances), and AWS CloudFormation takes care of provisioning and configuring those resources for you. You don't need to individually create and configure AWS resources and figure out what's dependent on what; AWS CloudFormation handles all of that.\n",
    "\n",
    "#### Further Recommended Reading\n",
    "* [Session Manager Benefits](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html#session-manager-benefits)\n",
    "* [Getting Started with Session Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-getting-started.html)\n",
    "* [Enable SSH and SCP through AWS Systems Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-getting-started-enable-ssh-connections.html)\n",
    "* [CDK](https://docs.aws.amazon.com/cdk/latest/guide/home.html)\n",
    "\n",
    "***In the variable `bucket` below replace the `{{FIXME}}` with your initials.***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "import botocore\n",
    "import json\n",
    "import time\n",
    "import pandas as pd\n",
    "\n",
    "cfn = boto3.client('cloudformation')\n",
    "s3 = boto3.client('s3')\n",
    "\n",
    "# General variables for the region and account id for the location of the resources being created\n",
    "session = boto3.session.Session()\n",
    "region = session.region_name\n",
    "account_id = boto3.client('sts').get_caller_identity().get('Account')\n",
    "\n",
    "stack_name = 'ec2-ssm-stack'\n",
    "bucket = 'ec2-ssm-demo-rr-{{FIXME}}'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Create S3 Bucket](https://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html)\n",
    "\n",
    "We will create an S3 bucket that will be used throughout the workshop for storing our data.\n",
    "\n",
    "[s3.create_bucket](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.create_bucket) boto3 documentation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_bucket(bucket):\n",
    "    import logging\n",
    "\n",
    "    try:\n",
    "        s3.create_bucket(Bucket=bucket, CreateBucketConfiguration={'LocationConstraint': region})\n",
    "    except botocore.exceptions.ClientError as e:\n",
    "        logging.error(e)\n",
    "        return 'Bucket {0} could not be created.'.format(bucket)\n",
    "    return 'Created {0} bucket.'.format(bucket)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "create_bucket(bucket)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload [CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/GettingStarted.html) template\n",
    "\n",
    "We will be launching a CloudFormation template to create our EC2 instance to show you how you might generate those resources in you account."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "demo_file = 'cfn/ec2_ssm.yaml'\n",
    "session.resource('s3').Bucket(bucket).Object(demo_file).upload_file(demo_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What does the CloudFormation template look like?\n",
    "A CloudFormation template will use either `.json` or `.yaml` to describe the resources you want created in your account."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat cfn/ec2_ssm.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's launch the template\n",
    "\n",
    "We will use the boto3 libraries for launching the CloudFormation template in the account. The default values will be used but you could easily substitute custom patameters when launching the template based on your environment byt passing in parameters like below.\n",
    "\n",
    "```json\n",
    "Parameters=[\n",
    "    {\n",
    "        'ParameterKey': 'EC2InstanceType',\n",
    "        'ParameterValue': t2.medium\n",
    "    }\n",
    "]    \n",
    "```\n",
    "\n",
    "[cfn.create_stack](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudformation.html#CloudFormation.Client.create_stack) boto3 documentation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cfn_template = 'https://s3.amazonaws.com/{0}/{1}'.format(bucket, demo_file)\n",
    "print(cfn_template)\n",
    "\n",
    "response = cfn.create_stack(\n",
    "    StackName=stack_name,\n",
    "    TemplateURL=cfn_template,\n",
    "    Capabilities = [\"CAPABILITY_NAMED_IAM\"],\n",
    ")\n",
    "\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wait for stack to complete\n",
    "\n",
    "We can call `describe_stacks` in the boto3 library to poll for the status of the stack completion. Once we see a `CREATE_COMPLETE` message the resources will have been successfully created.\n",
    "\n",
    "[cfn.describe_stacks](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudformation.html#CloudFormation.Client.describe_stacks) boto3 documentation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = cfn.describe_stacks(\n",
    "    StackName=stack_name\n",
    ")\n",
    "\n",
    "while response['Stacks'][0]['StackStatus'] != 'CREATE_COMPLETE':\n",
    "    print('Waiting...')\n",
    "    time.sleep(30)\n",
    "    response = cfn.describe_stacks(\n",
    "        StackName=stack_name\n",
    "    )\n",
    "\n",
    "outputs = response['Stacks'][0]['Outputs']\n",
    "\n",
    "for output in outputs:\n",
    "    if (output['OutputKey'] == 'SessionManagementInstanceUrl'):\n",
    "        session_instance_url = output['OutputValue']\n",
    "    if (output['OutputKey'] == 'SessionManagementListUrl'):\n",
    "        session_manager_list_url = output['OutputValue']\n",
    "        \n",
    "pd.set_option('display.max_colwidth', -1)\n",
    "pd.DataFrame(outputs, columns=[\"OutputKey\", \"OutputValue\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What the template created\n",
    "\n",
    "[VPC Design](https://aws.amazon.com/answers/networking/aws-single-vpc-design/)\n",
    "\n",
    "Amazon Virtual Private Cloud (Amazon VPC) offers a comprehensive set of virtual networking capabilities that provide AWS customers with many options for designing and implementing their AWS network. However, it can be difficult to decide on an ideal network design for your organization from among the various network strategies and configuration permutations, especially for customers who do not have dedicated network professionals to design, size, and manage their networks.\n",
    "\n",
    "[EC2 Instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html)\n",
    "\n",
    "Amazon Elastic Compute Cloud (Amazon EC2) provides scalable computing capacity in the Amazon Web Services (AWS) cloud. Using Amazon EC2 eliminates your need to invest in hardware up front, so you can develop and deploy applications faster. You can use Amazon EC2 to launch as many or as few virtual servers as you need, configure security and networking, and manage storage. Amazon EC2 enables you to scale up or down to handle changes in requirements or spikes in popularity, reducing your need to forecast traffic."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Shell Access to Instance\n",
    "\n",
    "Now you can run the cell below to generate the links to gain shell access to the EC2 instance you just launched in the CloudFormation template."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print (\"Session Manager Listing: {0}\".format(session_manager_list_url))\n",
    "print(\"Session Manager Instance URL: {0}\".format(session_instance_url))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Install AWS ParallelCluster\n",
    "AWS ParallelCluster is an AWS-supported open source cluster management tool that helps you to deploy and manage High Performance Computing (HPC) clusters in the AWS Cloud. Built on the open source CfnCluster project, AWS ParallelCluster enables you to quickly build an HPC compute environment in AWS. It automatically sets up the required compute resources and shared filesystem. You can use AWS ParallelCluster with a variety of batch schedulers, such as AWS Batch, SGE, Torque, and Slurm. AWS ParallelCluster facilitates quick start proof of concept deployments and production deployments. You can also build higher level workflows, such as a genomics portal that automates an entire DNA sequencing workflow, on top of AWS ParallelCluster.\n",
    "\n",
    "For more information follow [link](https://docs.aws.amazon.com/parallelcluster/latest/ug/what-is-aws-parallelcluster.html).\n",
    "\n",
    "\n",
    "Now we will upgrade to Python3 and install AWS ParallelCluster to show how you can leverage shell access in the same way you are used to accessing your instances with SSH. Run the commands below in the terminal to install AWS ParallelCluster.\n",
    "\n",
    "```bash\n",
    "sudo yum install -y python36 python36-devel python36-pip python36-setuptools python36-virtualenv\n",
    "\n",
    "sudo pip-3.6 install --upgrade pip\n",
    "\n",
    "pip install aws-parallelcluster --upgrade --user\n",
    "\n",
    "export PATH=/home/ssm-user/.local/bin/:$PATH\n",
    "\n",
    "pcluster -h\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point you would be ready to launch HPC clusters in AWS and learned to gain shell access to your EC2 instances without having to open SSH port 22 on your instances thereby decreasing you security blast radius in your research environment.\n",
    "\n",
    "### Cleanup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = cfn.delete_stack(StackName=stack_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "waiter = cfn.get_waiter('stack_delete_complete')\n",
    "waiter.wait(\n",
    "    StackName=stack_name\n",
    ")\n",
    "\n",
    "print('The wait is over for {0}'.format(stack_name))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!aws s3 rb s3://$bucket --force"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}