AWSTemplateFormatVersion: '2010-09-09' Description: This CloudFormation template creates the Axomo components. Authored by Mactores.(qs-1q18ntcq9) Metadata: AWS::CloudFormation::Interface: ParameterGroups: - Label: default: Network configuration Parameters: - VPCID - PrivateSubnet1AID - PrivateSubnet2AID - PublicSubnet1ID - PublicSubnet2ID - Label: default: Amazon SageMaker configuration Parameters: - SagemakerEndpointConfigInstanceType - SagemakerModelName - Label: default: AWS Glue configuration Parameters: - CrawlerDBname - Label: default: Amazon Athena configuration Parameters: - AthenaDBname - Label: default: DMS instance configuration Parameters: - DmsInstanceClass - CreateDMSRoles - Label: default: Amazon S3 configuration Parameters: - S3BucketName - Label: default: Components tag details Parameters: - ProjectTag - EnvTag - Label: default: AWS Quick Start configuration Parameters: - QSS3BucketName - QSS3BucketRegion - QSS3KeyPrefix ParameterLabels: QSS3BucketName: default: Quick Start S3 bucket name QSS3BucketRegion: default: Quick Start S3 bucket region QSS3KeyPrefix: default: Quick Start S3 key prefix VPCID: default: VPC ID PrivateSubnet1AID: default: Private subnet 1A ID in Availability Zone 1 PrivateSubnet2AID: default: Private subnet 2A ID in Availability Zone 2 PublicSubnet1ID: default: Public subnet 1 ID in Availability Zone 1 PublicSubnet2ID: default: Public subnet 2 ID in Availability Zone 2 CrawlerDBname: default: DB name for Glue crawler AthenaDBname: default: Athena DB name DmsInstanceClass: default: Instance class for DMS CreateDMSRoles: default: Conditional IAM roles creation for DMS S3BucketName: default: S3 bucket name ProjectTag: default: Value for project tag EnvTag: default: Value for environment tag SagemakerEndpointConfigInstanceType: default: Instance type for SageMaker EndpointConfig SagemakerModelName: default: Model name for SageMaker Parameters: QSS3BucketName: Description: >- The S3 bucket you created for your copy of Quick Start assets, if you decide to customize or extend the Quick Start for your own use. The bucket name can include numbers, lowercase letters, uppercase letters, and hyphens, but should not start or end with a hyphen. Type: String AllowedPattern: ^[0-9a-zA-Z]+([0-9a-zA-Z-]*[0-9a-zA-Z])*$ ConstraintDescription: Quick Start bucket name can include numbers, lowercase letters, uppercase letters, and hyphens (-). It cannot start or end with a hyphen (-). Default: aws-quickstart QSS3BucketRegion: Default: 'us-east-1' Description: 'The AWS Region where the Quick Start S3 bucket (QSS3BucketName) is hosted. When using your own bucket, you must specify this value.' Type: String QSS3KeyPrefix: Description: >- The S3 key name prefix used to simulate a folder for your copy of Quick Start assets, if you decide to customize or extend the Quick Start for your own use. This prefix can include numbers, lowercase letters, uppercase letters, hyphens, and forward slashes. Type: String AllowedPattern: ^[0-9a-zA-Z-/]*$ ConstraintDescription: Quick Start key prefix can include numbers, lowercase letters, uppercase letters, hyphens (-), and forward slash (/). Default: quickstart-mactores-axomo/ VPCID: Description: ID of your existing VPC for deployment. Type: AWS::EC2::VPC::Id PrivateSubnet1AID: Description: ID of existing Private subnet 1A in Availability Zone 1. Type: AWS::EC2::Subnet::Id PrivateSubnet2AID: Description: ID of existing Private subnet 2A in Availability Zone 2. Type: AWS::EC2::Subnet::Id PublicSubnet1ID: Description: ID of existing Public subnet 1 in Availability Zone 1. Type: AWS::EC2::Subnet::Id PublicSubnet2ID: Description: ID of existing Public subnet 2 in Availability Zone 2. Type: AWS::EC2::Subnet::Id S3BucketName: Description: S3 bucket name for the target location for Glue crawlers. Bucket name must contain only lowercase letters, numbers, periods and dashes. Type: String Default: '' ProjectTag: Description: Value for the Project tag to be associated with all the deployed components. Tags make it easier to manage, search for, and filter resources for Billing and tracking purposes. Type: String Default: Axomo EnvTag: Description: >- Value for the Environment tag to be associated with all the deployed components. Tags make it easier to manage, search for, and filter resources for Billing and tracking purposes. Can be Dev, Prd or any custom value Type: String Default: Dev DmsInstanceClass: Description: Instance class for the DMS ReplicationInstance. Type: String AllowedValues: - dms.t2.micro - dms.t2.small - dms.t2.medium - dms.t2.large - dms.c4.large - dms.c4.xlarge - dms.c4.2xlarge - dms.c4.4xlarge ConstraintDescription: 'Valid Values: dms.t2.micro | dms.t2.small | dms.t2.medium | dms.t2.large | dms.c4.large | dms.c4.xlarge | dms.c4.2xlarge | dms.c4.4xlarge' Default: dms.t2.micro CreateDMSRoles: Description: Specify True to create the IAM roles (dms-vpc-role and dms-cloudwatch-logs-role) required for DMS, if non-existent in your AWS account. Else, specify False. Type: String AllowedValues: - 'True' - 'False' ConstraintDescription: 'Valid Values: True | False' AthenaDBname: Description: Database name for the Athena NamedQuery. Type: String Default: el-db CrawlerDBname: Description: The name of the database in which the crawler's output is stored. Type: String Default: el-db-crawler SagemakerEndpointConfigInstanceType: Description: Instance Type for the Sagemaker Endpoint Config. Type: String AllowedValues: - ml.c4.2xlarge - ml.c4.4xlarge - ml.c4.8xlarge - ml.c4.large - ml.c4.xlarge - ml.c5.18xlarge - ml.c5.2xlarge - ml.c5.4xlarge - ml.c5.9xlarge - ml.c5.large - ml.c5.xlarge - ml.m4.10xlarge - ml.m4.16xlarge - ml.m4.2xlarge - ml.m4.4xlarge - ml.m4.xlarge - ml.m5.12xlarge - ml.m5.24xlarge - ml.m5.2xlarge - ml.m5.4xlarge - ml.m5.large - ml.m5.xlarge - ml.p2.16xlarge - ml.p2.8xlarge - ml.p2.xlarge - ml.p3.16xlarge - ml.p3.2xlarge - ml.p3.8xlarge - ml.t2.2xlarge - ml.t2.large - ml.t2.medium - ml.t2.xlarge ConstraintDescription: >- Allowed Values: ml.c4.2xlarge | ml.c4.4xlarge | ml.c4.8xlarge | ml.c4.large | ml.c4.xlarge | ml.c5.18xlarge | ml.c5.2xlarge | ml.c5.4xlarge | ml.c5.9xlarge | ml.c5.large | ml.c5.xlarge | ml.m4.10xlarge | ml.m4.16xlarge | ml.m4.2xlarge | ml.m4.4xlarge | ml.m4.xlarge | ml.m5.12xlarge | ml.m5.24xlarge | ml.m5.2xlarge | ml.m5.4xlarge | ml.m5.large | ml.m5.xlarge | ml.p2.16xlarge | ml.p2.8xlarge | ml.p2.xlarge | ml.p3.16xlarge | ml.p3.2xlarge | ml.p3.8xlarge | ml.t2.2xlarge | ml.t2.large | ml.t2.medium | ml.t2.xlarge Default: ml.m4.xlarge SagemakerModelName: Description: The name of the Amazon Sagemaker model that you want to host. Type: String Default: '' Conditions: UsingDefaultBucket: !Equals [!Ref QSS3BucketName, 'aws-quickstart'] CreateDMSRoles: !Equals - !Ref 'CreateDMSRoles' - 'True' SkipCreateDMSRoles: !Equals - !Ref 'CreateDMSRoles' - 'False' Mappings: RegionMap: us-east-1: ImageURI: 382416733822.dkr.ecr.us-east-1.amazonaws.com/linear-learner:latest us-east-2: ImageURI: 404615174143.dkr.ecr.us-east-2.amazonaws.com/linear-learner:latest us-west-1: ImageURI: 656441707956.dkr.ecr.us-west-1.amazonaws.com/el-dorado:latest us-west-2: ImageURI: 174872318107.dkr.ecr.us-west-2.amazonaws.com/linear-learner:latest ap-east-1: ImageURI: 656441707956.dkr.ecr.ap-east-1.amazonaws.com/el-dorado:latest ap-south-1: ImageURI: 656441707956.dkr.ecr.ap-south-1.amazonaws.com/el-dorado:latest ap-northeast-2: ImageURI: 656441707956.dkr.ecr.ap-northeast-2.amazonaws.com/el-dorado:latest ap-southeast-1: ImageURI: 656441707956.dkr.ecr.ap-southeast-1.amazonaws.com/el-dorado:latest ap-southeast-2: ImageURI: 656441707956.dkr.ecr.ap-southeast-2.amazonaws.com/el-dorado:latest ap-northeast-1: ImageURI: 656441707956.dkr.ecr.ap-northeast-1.amazonaws.com/el-dorado:latest ca-central-1: ImageURI: 656441707956.dkr.ecr.ca-central-1.amazonaws.com/el-dorado:latest eu-central-1: ImageURI: 656441707956.dkr.ecr.eu-central-1.amazonaws.com/el-dorado:latest eu-west-1: ImageURI: 438346466558.dkr.ecr.eu-west-1.amazonaws.com/linear-learner:latest eu-west-2: ImageURI: 656441707956.dkr.ecr.eu-west-2.amazonaws.com/el-dorado:latest eu-west-3: ImageURI: 656441707956.dkr.ecr.eu-west-3.amazonaws.com/el-dorado:latest eu-north-1: ImageURI: 656441707956.dkr.ecr.eu-north-1.amazonaws.com/el-dorado:latest me-south-1: ImageURI: 656441707956.dkr.ecr.me-south-1.amazonaws.com/el-dorado:latest sa-east-1: ImageURI: 656441707956.dkr.ecr.sa-east-1.amazonaws.com/el-dorado:latest Resources: DMSRolesStack: Condition: CreateDMSRoles Type: AWS::CloudFormation::Stack Properties: TemplateURL: !Sub - 'https://${S3Bucket}.s3.${S3Region}.${AWS::URLSuffix}/${QSS3KeyPrefix}templates/axomo-create-dms-roles.template.yaml' - S3Region: !If [UsingDefaultBucket, !Ref 'AWS::Region', !Ref QSS3BucketRegion] S3Bucket: !If [UsingDefaultBucket, !Sub '${QSS3BucketName}-${AWS::Region}', !Ref QSS3BucketName] GlueS3bucket: Type: AWS::S3::Bucket Properties: AccessControl: Private BucketEncryption: ServerSideEncryptionConfiguration: - ServerSideEncryptionByDefault: SSEAlgorithm: AES256 BucketName: !Ref 'S3BucketName' Tags: - Key: Project Value: !Ref 'ProjectTag' - Key: Env Value: !Ref 'EnvTag' AthenaNamedQuery: Type: AWS::Athena::NamedQuery Properties: Database: !Ref 'AthenaDBname' Description: Query description Name: !Ref 'S3BucketName' QueryString: !Join - '' - - 'SELECT * from ' - !Ref 'AthenaDBname' GlueJobRole: DependsOn: GlueS3bucket Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: - glue.amazonaws.com Action: - sts:AssumeRole Path: / Policies: - PolicyName: root PolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Action: - glue:* - s3:GetBucketLocation - s3:ListBucket - s3:ListAllMyBuckets - s3:GetBucketAcl - iam:ListRolePolicies - iam:GetRole - iam:GetRolePolicy - cloudwatch:PutMetricData Resource: - '*' - Effect: Allow Action: - s3:GetObject - s3:PutObject - s3:DeleteObject Resource: - !Join - '' - - 'arn:aws:s3:::' - !Ref 'S3BucketName' - /* - !Sub - arn:${AWS::Partition}:s3:::${S3Bucket}/* - S3Bucket: !If [UsingDefaultBucket, !Sub '${QSS3BucketName}-${AWS::Region}', !Ref QSS3BucketName] - Effect: Allow Action: - s3:GetObject Resource: - arn:aws:s3:::crawler-public* - !Join - '' - - 'arn:aws:s3:::' - !Ref 'S3BucketName' - !Sub - arn:${AWS::Partition}:s3:::${S3Bucket}/* - S3Bucket: !If [UsingDefaultBucket, !Sub '${QSS3BucketName}-${AWS::Region}', !Ref QSS3BucketName] - Effect: Allow Action: - logs:CreateLogGroup - logs:CreateLogStream - logs:PutLogEvents Resource: - !Join - '' - - arn:aws:logs:*:*:/ - !Ref 'S3BucketName' - /* - !Join - '' - - arn:aws:logs:*:*:/ - !Ref 'QSS3BucketName' - /* GlueJob: Type: AWS::Glue::Job Properties: Command: Name: glueetl ScriptLocation: !Ref 'GlueS3bucket' Description: Glue job Name: !Ref 'S3BucketName' Role: !Ref 'GlueJobRole' Tags: Project: !Ref 'ProjectTag' Env: !Ref 'EnvTag' GlueCrawler: Type: AWS::Glue::Crawler Properties: DatabaseName: !Ref 'CrawlerDBname' Description: Glue crawler Name: !Ref 'S3BucketName' Role: !Ref 'GlueJobRole' Tags: Project: !Ref 'ProjectTag' Env: !Ref 'EnvTag' Targets: S3Targets: - Path: !Ref 'GlueS3bucket' LambdaExecutionRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: - lambda.amazonaws.com Action: - sts:AssumeRole Path: / Policies: - PolicyName: root PolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Action: ecr:* Resource: '*' - Effect: Allow Action: ec2:* Resource: '*' - Effect: Allow Action: s3:GetObject Resource: - !Join - '' - - 'arn:aws:s3:::' - !Ref 'QSS3BucketName' - !Join - '' - - 'arn:aws:s3:::' - !Ref 'QSS3BucketName' - /* - Effect: Allow Action: s3:* Resource: - !Join - '' - - 'arn:aws:s3:::' - !Ref 'S3BucketName' - !Join - '' - - 'arn:aws:s3:::' - !Ref 'S3BucketName' - /* - Effect: Allow Action: sagemaker:InvokeEndpoint Resource: '*' - Effect: Allow Action: - logs:CreateLogGroup - logs:CreateLogStream - logs:PutLogEvents Resource: '*' Lambda: DependsOn: CopyS3objectsOnCreate Type: AWS::Lambda::Function Properties: Code: S3Bucket: !Ref 'GlueS3bucket' S3Key: !Join - '' - - !Ref 'QSS3KeyPrefix' - functions/packages/invokesagemaker/function.zip Description: Lambda function to invoke Sagemaker endpoint FunctionName: axomo-lambda-1 Handler: lambda_function.lambda_handler Role: !GetAtt 'LambdaExecutionRole.Arn' Runtime: python3.7 Tags: - Key: Project Value: !Ref 'ProjectTag' - Key: Env Value: !Ref 'EnvTag' Timeout: 60 VpcConfig: SecurityGroupIds: - !Ref 'SecurityGroup' SubnetIds: - !Ref 'PrivateSubnet1AID' - !Ref 'PrivateSubnet2AID' - !Ref 'PublicSubnet1ID' - !Ref 'PublicSubnet2ID' SecurityGroup: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: Allow lambda connectivity to VPC resources. SecurityGroupIngress: - IpProtocol: '-1' CidrIp: '0.0.0.0/0' Tags: - Key: Project Value: !Ref 'ProjectTag' - Key: Env Value: !Ref 'EnvTag' VpcId: !Ref 'VPCID' LambdaEmptyBucket: DependsOn: CopyS3objectsOnCreate Type: AWS::Lambda::Function Properties: Code: S3Bucket: !Ref 'GlueS3bucket' S3Key: !Join - '' - - !Ref 'QSS3KeyPrefix' - functions/packages/emptys3bucket/function.zip Description: Lambda function to empty the S3 bucket during stack deletion FunctionName: axomo-lambda-2 Handler: lambda_function.lambda_handler Role: !GetAtt 'LambdaExecutionRole.Arn' Runtime: python3.7 Tags: - Key: Project Value: !Ref 'ProjectTag' - Key: Env Value: !Ref 'EnvTag' Timeout: 60 cleanupBucketOnDelete: Type: Custom::cleanupbucket Properties: ServiceToken: !GetAtt 'LambdaEmptyBucket.Arn' BucketName: !Ref 'GlueS3bucket' LambdaCopyArtifacts: Type: AWS::Lambda::Function Properties: Code: ZipFile: !Join - '' - - "import json\n" - "import logging\n" - "import threading\n" - "import boto3\n" - "import cfnresponse\n" - "def copy_objects(source_bucket, dest_bucket, prefix, objects):\n" - " s3 = boto3.client('s3')\n" - " for o in objects:\n" - " key = prefix + o\n" - " copy_source = {\n" - " 'Bucket': source_bucket,\n" - " 'Key': key\n" - " }\n" - " print('copy_source: %s' % copy_source)\n" - " print('dest_bucket = %s'%dest_bucket)\n" - " print('key = %s' %key)\n" - " s3.copy_object(CopySource=copy_source, Bucket=dest_bucket, Key=key)\n" - "def delete_objects(bucket, prefix, objects):\n" - " s3 = boto3.client('s3')\n" - " objects = {'Objects': [{'Key': prefix + o} for o in objects]}\n" - " s3.delete_objects(Bucket=bucket, Delete=objects)\n" - "def timeout(event, context):\n" - " logging.error('Execution is about to time out, sending failure response\ \ to CloudFormation')\n" - " cfnresponse.send(event, context, cfnresponse.FAILED, {}, None)\n" - "def handler(event, context):\n" - " timer = threading.Timer((context.get_remaining_time_in_millis() /\ \ 1000.00) - 0.5, timeout, args=[event, context])\n" - " timer.start()\n" - " print('Received event: %s' % json.dumps(event))\n" - " status = cfnresponse.SUCCESS\n" - " try:\n" - " source_bucket = event['ResourceProperties']['SourceBucket']\n" - " dest_bucket = event['ResourceProperties']['DestBucket']\n" - " prefix = event['ResourceProperties']['Prefix']\n" - " objects = event['ResourceProperties']['Objects']\n" - " if event['RequestType'] == 'Delete':\n" - " delete_objects(dest_bucket, prefix, objects)\n" - " else:\n" - " copy_objects(source_bucket, dest_bucket, prefix, objects)\n" - " except Exception as e:\n" - " logging.error('Exception: %s' % e, exc_info=True)\n" - " status = cfnresponse.FAILED\n" - " finally:\n" - " timer.cancel()\n" - " cfnresponse.send(event, context, status, {}, None)\n" Description: Lambda function to copy artifacts from main bucket to new region s3 bucket FunctionName: axomo-lambda-3 Handler: index.handler Role: !GetAtt 'LambdaExecutionRole.Arn' Runtime: python3.7 Tags: - Key: Project Value: !Ref 'ProjectTag' - Key: Env Value: !Ref 'EnvTag' Timeout: 60 CopyS3objectsOnCreate: Type: Custom::copys3objects Properties: ServiceToken: !GetAtt 'LambdaCopyArtifacts.Arn' DestBucket: !Ref 'GlueS3bucket' SourceBucket: !Ref 'QSS3BucketName' Prefix: !Ref 'QSS3KeyPrefix' Objects: - scripts/model-ll.tar.gz - functions/packages/invokesagemaker/function.zip - functions/packages/emptys3bucket/function.zip BasicReplicationInstance1: Condition: CreateDMSRoles DependsOn: DMSRolesStack Type: AWS::DMS::ReplicationInstance Properties: PubliclyAccessible: false ReplicationInstanceClass: !Ref 'DmsInstanceClass' ReplicationInstanceIdentifier: !Ref 'S3BucketName' Tags: - Key: Project Value: !Ref 'ProjectTag' - Key: Env Value: !Ref 'EnvTag' BasicReplicationInstance2: Condition: SkipCreateDMSRoles Type: AWS::DMS::ReplicationInstance Properties: PubliclyAccessible: false ReplicationInstanceClass: !Ref 'DmsInstanceClass' ReplicationInstanceIdentifier: !Ref 'S3BucketName' Tags: - Key: Project Value: !Ref 'ProjectTag' - Key: Env Value: !Ref 'EnvTag' SagemakerExecutionRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: - sagemaker.amazonaws.com Action: - sts:AssumeRole Path: / Policies: - PolicyName: root PolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Action: s3:* Resource: - !Join - '' - - 'arn:aws:s3:::' - !Ref 'S3BucketName' - !Join - '' - - 'arn:aws:s3:::' - !Ref 'S3BucketName' - /* - !Join - '' - - 'arn:aws:s3:::' - !Ref 'QSS3BucketName' - !Join - '' - - 'arn:aws:s3:::' - !Ref 'QSS3BucketName' - /* - Effect: Allow Action: - ecr:BatchGetImage - ecr:GetDownloadUrlForLayer Resource: - '*' SagemakerModel: DependsOn: CopyS3objectsOnCreate Type: AWS::SageMaker::Model Properties: ExecutionRoleArn: !GetAtt 'SagemakerExecutionRole.Arn' ModelName: !Ref 'SagemakerModelName' PrimaryContainer: Image: !FindInMap - RegionMap - !Ref 'AWS::Region' - ImageURI ModelDataUrl: !Join - '' - - s3:// - !Ref 'GlueS3bucket' - / - !Ref 'QSS3KeyPrefix' - scripts/model-ll.tar.gz Tags: - Key: Project Value: !Ref 'ProjectTag' - Key: Env Value: !Ref 'EnvTag' SagemakerEndpointConfig: Type: AWS::SageMaker::EndpointConfig Properties: EndpointConfigName: !Ref 'SagemakerModelName' ProductionVariants: - InitialInstanceCount: 1 InitialVariantWeight: 1.0 InstanceType: !Ref 'SagemakerEndpointConfigInstanceType' ModelName: !GetAtt 'SagemakerModel.ModelName' VariantName: !GetAtt 'SagemakerModel.ModelName' Tags: - Key: Project Value: !Ref 'ProjectTag' - Key: Env Value: !Ref 'EnvTag' SagemakerEndpoint: Type: AWS::SageMaker::Endpoint Properties: EndpointConfigName: !GetAtt 'SagemakerEndpointConfig.EndpointConfigName' EndpointName: !Ref 'SagemakerModelName' Tags: - Key: Project Value: !Ref 'ProjectTag' - Key: Env Value: !Ref 'EnvTag' Outputs: S3bucketArn: Value: !GetAtt 'GlueS3bucket.Arn' Description: Amazon Resource Name (ARN) of the specified bucket. SagemakerEndpointConfigARN: Value: !Ref 'SagemakerEndpointConfig' Description: Amazon Resource Name (ARN) of the sagemaker endpoint configuration. SagemakerEndpointARN: Value: !Ref 'SagemakerEndpoint' Description: Amazon Resource Name (ARN) of the SagemakerEndpoint.