AWSTemplateFormatVersion: 2010-09-09 Description: Creates CodePipeline for Database Migration Service (qs-1qcbthe7t) Metadata: AWS::CloudFormation::Interface: ParameterGroups: - Label: default: Database Migration Service Endpoints Parameters: - DmsSourceArn - DmsTargetArn - DmsReplicationArn - TaskName - FullLoadOnly - Label: default: Target RDS Access and Credentials Parameters: - TargetEndpoint - TargetDbName - TargetDbUser - TargetDbPassword - Label: default: Source RDS Access and Credentials (optional) Parameters: - CreateSample - SourceEndpoint - SourceDbName - SourceDbUser - SourceDbPassword - Label: default: CodeBuild Configuration and Environment Parameters: - CodeBuildSecurityGroup - CodeBuildSubnet - CodeBuildVpc - Label: default: CodePipeline Source Configuration Parameters: - S3BucketName - S3BucketKey - Label: default: Migration Notifications Parameters: - MigNotify ParameterLabels: CreateSample: default: Create Sample Source? FullLoadOnly: default: Full Load Only? MigNotify: default: Email address TaskName: default: DMS Task Name CodeBuildVpc: default: VPC used by CodeBuild CodeBuildSubnet: default: SubnetID CodeBuildSecurityGroup: default: Security Group S3BucketName: default: S3 Bucket S3BucketKey: default: S3 Key TargetEndpoint: default: RDS Endpoint TargetDbName: default: Database Name TargetDbUser: default: Database Username TargetDbPassword: default: Database Password SourceDbPassword: default: Source Database Password SourceEndpoint: default: Source RDS Endpoint SourceDbName: default: Source Database Name SourceDbUser: default: Source Database Username DmsSourceArn: default: Source Endpoint Arn DmsTargetArn: default: Target Endpoint Arn DmsReplicationArn: default: Replication instance Arn Conditions: cTaskName: !Equals - !Ref TaskName - '' cCdc: !Equals - !Ref FullLoadOnly - 'no' cCreateSource: !Equals - !Ref CreateSample - 'yes' Parameters: CreateSample: Type: String Description: Please choose this option if you want to populate source database AllowedValues: - 'yes' - 'no' Default: 'no' FullLoadOnly: Type: String Description: This is for task to do only full loads and not CDC AllowedValues: - 'yes' - 'no' Default: 'no' MigNotify: Type: String Description: This email will get notifications on Migration Status CodeBuildSecurityGroup: Type: 'AWS::EC2::SecurityGroup::Id' Description: Security group used by CodeBuild. Please ensure that this has access to RDS endpoint CodeBuildSubnet: Type: 'AWS::EC2::Subnet::Id' Description: Private subnet id with NAT gateway where codebuild will be launched CodeBuildVpc: Type: 'AWS::EC2::VPC::Id' Description: VPC Id where codebuild will be launched. Subnet and security group should belong to this VPC. S3BucketKey: Type: String Description: The location of file which is used as source for CodePipeline S3BucketName: Type: String Description: The name of the S3 bucket where the code resides. This will be used as source for the CodePipeline TargetEndpoint: Type: String Description: RDS Target endpoint where the schema changes will take place TargetDbName: Type: String Description: RDS Target database name for deployments TargetDbPassword: Type: String Description: RDS Target database user password for deployment NoEcho: 'true' TargetDbUser: Type: String Description: RDS Target database user name SourceEndpoint: Type: String Description: RDS Source endpoint where the schema changes will take place SourceDbName: Type: String Description: RDS Source database name for deployments SourceDbPassword: Type: String Description: RDS Source database user password for deployment NoEcho: 'true' SourceDbUser: Type: String Description: RDS Source database user name TaskName: Type: String Description: Name of DMS task. If left blank stack name will be used. DmsSourceArn: Type: String Description: DMS Arn of source endpoint DmsTargetArn: Type: String Description: DMS Arn of target endpoint DmsReplicationArn: Type: String Description: DMS Replication Arn used for migration Resources: # create project for populating source DmsPipelineEvents: Type: 'AWS::Events::Rule' Properties: Description: EventRule EventPattern: source: - aws.codepipeline detail-type: - CodePipeline Pipeline Execution State Change detail: state: - FAILED pipeline: - !Ref DmsPipeline State: ENABLED Targets: - Arn: !Ref MigrationNotification Id: MigrationPipeline MigrationNotification: Type: 'AWS::SNS::Topic' Properties: Subscription: - Endpoint: !Ref MigNotify Protocol: email MigrationNotificationPolicy: Type: 'AWS::SNS::TopicPolicy' Properties: PolicyDocument: Statement: - Effect: Allow Principal: Service: events.amazonaws.com Action: 'sns:Publish' Resource: '*' Topics: - !Ref MigrationNotification DmsPipeline: Type: 'AWS::CodePipeline::Pipeline' Properties: ArtifactStore: Location: !Ref ArtifactStoreS3Location Type: S3 RoleArn: !GetAtt - DmsPipelineRole - Arn Stages: - Actions: - ActionTypeId: Category: Source Owner: AWS Provider: S3 Version: '1' Configuration: PollForSourceChanges: true S3Bucket: !Ref S3BucketName S3ObjectKey: !Ref S3BucketKey Name: Source OutputArtifacts: - Name: AppSource RunOrder: 1 Name: Source - !If - cCreateSource - Actions: - ActionTypeId: Category: Build Owner: AWS Provider: CodeBuild Version: '1' Configuration: ProjectName: !Ref CodeBuildSetupSample InputArtifacts: - Name: AppSource Name: SetupSource RunOrder: 1 Name: Sample - !Ref 'AWS::NoValue' - Actions: - ActionTypeId: Category: Build Owner: AWS Provider: CodeBuild Version: '1' Configuration: ProjectName: !Ref CodeBuildSetupTarget InputArtifacts: - Name: AppSource Name: SetupTarget OutputArtifacts: - Name: AppBuild RunOrder: 1 Name: Setup - Actions: - ActionTypeId: Category: Approval Owner: AWS Provider: Manual Version: '1' Configuration: NotificationArn: !Ref SnsApproval Name: ApprovalForDMS RunOrder: 1 Name: Approve - !If - cCdc - Actions: - ActionTypeId: Category: Build Owner: AWS Provider: CodeBuild Version: '1' Configuration: ProjectName: !Ref CodeBuildPreCdc InputArtifacts: - Name: AppSource Name: SetupTarget RunOrder: 1 Name: PreCDC - !Ref 'AWS::NoValue' DmsPipelineRole: Type: 'AWS::IAM::Role' Properties: AssumeRolePolicyDocument: Statement: - Action: - 'sts:AssumeRole' Effect: Allow Principal: Service: - codepipeline.amazonaws.com Version: 2012-10-17 Policies: - PolicyName: 'codepipeline-policy' PolicyDocument: Statement: - Action: - 's3:*' - 'sns:*' - 'sqs:*' Effect: Allow Resource: '*' - Action: - 'codebuild:BatchGetBuilds' - 'codebuild:StartBuild' Effect: Allow Resource: '*' CodeBuildSetupTarget: Type: 'AWS::CodeBuild::Project' Properties: Artifacts: Type: CODEPIPELINE Description: Target and DMS task setup Environment: ComputeType: BUILD_GENERAL1_SMALL EnvironmentVariables: - Name: PGPASSWORD Type: PARAMETER_STORE Value: !Ref TargetRdsPassword - Name: DB_NAME Type: PLAINTEXT Value: !Ref TargetDbName - Name: TARGET_ENDPOINT Type: PLAINTEXT Value: !Ref TargetEndpoint - Name: DB_USER Type: PLAINTEXT Value: !Ref TargetDbUser - Name: TASK_NAME Type: PLAINTEXT Value: !If [cTaskName, !Ref 'AWS::StackName', !Ref TaskName] - Name: SOURCE_ENDPOINT_ARN Type: PLAINTEXT Value: !Ref DmsSourceArn - Name: TARGET_ENDPOINT_ARN Type: PLAINTEXT Value: !Ref DmsTargetArn - Name: REPLICATION_INSTANCE_ARN Type: PLAINTEXT Value: !Ref DmsReplicationArn - Name: SNS_TOPIC Type: PLAINTEXT Value: !Ref SnsApproval - Name: NOTIFY_SNS Type: PLAINTEXT Value: !Ref MigrationNotification Image: 'aws/codebuild/standard:1.0' Type: LINUX_CONTAINER ServiceRole: !GetAtt - CodeBuildServiceRole - Arn Source: BuildSpec: setup-target-schema.yml Type: CODEPIPELINE Tags: - Key: Key1 Value: Value1 - Key: Key2 Value: Value2 TimeoutInMinutes: 60 VpcConfig: SecurityGroupIds: - !Ref CodeBuildSecurityGroup Subnets: - !Ref CodeBuildSubnet VpcId: !Ref CodeBuildVpc CodeBuildPreCdc: Type: 'AWS::CodeBuild::Project' Condition: cCdc Properties: Artifacts: Type: CODEPIPELINE Description: DMS project to run before Change Data Capture (CDC) starts Environment: ComputeType: BUILD_GENERAL1_SMALL EnvironmentVariables: - Name: PGPASSWORD Type: PARAMETER_STORE Value: !Ref TargetRdsPassword - Name: DB_NAME Type: PLAINTEXT Value: !Ref TargetDbName - Name: TARGET_ENDPOINT Type: PLAINTEXT Value: !Ref TargetEndpoint - Name: DB_USER Type: PLAINTEXT Value: !Ref TargetDbUser - Name: NOTIFY_SNS Type: PLAINTEXT Value: !Ref MigrationNotification - Name: TASK_NAME Type: PLAINTEXT Value: !If [cTaskName, !Ref 'AWS::StackName', !Ref TaskName] Image: 'aws/codebuild/standard:1.0' Type: LINUX_CONTAINER ServiceRole: !GetAtt - CodeBuildServiceRole - Arn Source: Type: CODEPIPELINE BuildSpec: pre-CDC-build.yml Tags: - Key: Key1 Value: Value1 - Key: Key2 Value: Value2 TimeoutInMinutes: 10 VpcConfig: SecurityGroupIds: - !Ref CodeBuildSecurityGroup Subnets: - !Ref CodeBuildSubnet VpcId: !Ref CodeBuildVpc CodeBuildSetupSample: Type: 'AWS::CodeBuild::Project' Condition: cCreateSource Properties: Artifacts: Type: CODEPIPELINE Description: DMS project to populate source database Environment: ComputeType: BUILD_GENERAL1_SMALL EnvironmentVariables: - Name: DB_PASSWORD Type: PARAMETER_STORE Value: !Ref SourceRdsPassword - Name: DB_NAME Type: PLAINTEXT Value: !Ref SourceDbName - Name: DB_HOST Type: PLAINTEXT Value: !Ref SourceEndpoint - Name: DB_USER Type: PLAINTEXT Value: !Ref SourceDbUser - Name: NOTIFY_SNS Type: PLAINTEXT Value: !Ref MigrationNotification Image: 'aws/codebuild/amazonlinux2-x86_64-standard:2.0' Type: LINUX_CONTAINER ServiceRole: !GetAtt - CodeBuildServiceRole - Arn Source: Type: CODEPIPELINE BuildSpec: | version: 0.2 phases: install: runtime-versions: python: 3.8 commands: - pip3 install awscli --upgrade - yum install git which -y - yum install -y http://yum.oracle.com/repo/OracleLinux/OL7/oracle/instantclient/x86_64/getPackage/oracle-instantclient18.3-basic-18.3.0.0.0-3.x86_64.rpm - yum install -y http://yum.oracle.com/repo/OracleLinux/OL7/oracle/instantclient/x86_64/getPackage/oracle-instantclient18.3-devel-18.3.0.0.0-3.x86_64.rpm - yum install -y http://yum.oracle.com/repo/OracleLinux/OL7/oracle/instantclient/x86_64/getPackage/oracle-instantclient18.3-sqlplus-18.3.0.0.0-3.x86_64.rpm - echo /usr/lib/oracle/18.3/client64/lib > /etc/ld.so.conf.d/oracle-instantclient18.3.conf && ldconfig - ls /etc/ld.so.conf.d/ - PATH=$PATH:/usr/lib/oracle/18.3/client64/bin - which sqlplus build: commands: - git clone https://github.com/aws-samples/aws-database-migration-samples.git - cd aws-database-migration-samples/oracle/sampledb/v1 - echo "drop index dms_sample.set_ev_id_tkholder_id_idx;" >> install-rds.sql - echo "create index dms_sample.set_ev_id_tkholder_id_idx on dms_sample.sporting_event_ticket(sporting_event_id,ticketholder_id);" >> install-rds.sql - echo "drop index dms_sample.se_start_date_fcn;" >> install-rds.sql - echo "create index dms_sample.se_start_date_fcn on dms_sample.sporting_event(start_date_time);" >> install-rds.sql - echo "alter table dms_sample.sport_location modify (id number);" >> install-rds.sql - echo "exit;" >> install-rds.sql - sqlplus $DB_USER/$DB_PASSWORD@$DB_HOST/$DB_NAME @install-rds.sql Tags: - Key: Created by Value: !Sub ${AWS::StackName} TimeoutInMinutes: 100 VpcConfig: SecurityGroupIds: - !Ref CodeBuildSecurityGroup Subnets: - !Ref CodeBuildSubnet VpcId: !Ref CodeBuildVpc TargetRdsPassword: Type: 'AWS::SSM::Parameter' Properties: Description: SSM Parameter for RDS Password. Name: !Sub 'DMS-${AWS::StackName}' Type: String Value: !Ref TargetDbPassword SourceRdsPassword: Type: 'AWS::SSM::Parameter' Condition : cCreateSource Properties: Description: SSM Parameter for Source RDS Password. Name: !Sub 'DMS-${AWS::StackName}-source' Type: String Value: !Ref SourceDbPassword CodeBuildServiceRole: Type: 'AWS::IAM::Role' Properties: AssumeRolePolicyDocument: Statement: - Action: - 'sts:AssumeRole' Effect: Allow Principal: Service: - codebuild.amazonaws.com Version: 2012-10-17 Policies: - PolicyName: !Sub 'Codebuild-${AWS::StackName}-${AWS::Region}-policy' PolicyDocument: Statement: - Action: - 'logs:CreateLogGroup' - 'logs:CreateLogStream' - 'logs:PutLogEvents' Effect: Allow Resource: - !Sub arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/codebuild/* - Action: - 's3:PutObject' - 's3:GetObject' - 's3:GetBucketVersioning' - 's3:GetObjectVersion' Effect: Allow Resource: - !Sub 'arn:aws:s3:::${ArtifactStoreS3Location}*' - Action: - 'dms:StartReplicationTask' - 'dms:CreateReplicationTask' - 'dms:CreateEventSubscription' - 'dms:Describe*' Effect: Allow Resource: '*' - Action: 'ssm:GetParameters' Effect: Allow Resource: - !Sub arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/${TargetRdsPassword} - !If [cCreateSource, !Sub 'arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/${SourceRdsPassword}', !Ref 'AWS::NoValue'] - Action: - 'sns:GetTopicAttributes' - 'sns:Subscribe' Effect: Allow Resource: !Ref SnsApproval - Action: - 'sns:Publish' Effect: Allow Resource: !Ref MigrationNotification - Action: - 'ec2:CreateNetworkInterface' - 'ec2:DescribeDhcpOptions' - 'ec2:DescribeNetworkInterfaces' - 'ec2:DeleteNetworkInterface' - 'ec2:DescribeSubnets' - 'ec2:DescribeSecurityGroups' - 'ec2:DescribeVpcs' Effect: Allow Resource: '*' - Action: - 'ec2:CreateNetworkInterfacePermission' Condition: StringEquals: 'ec2:AuthorizedService': codebuild.amazonaws.com 'ec2:Subnet': - !Sub arn:aws:ec2:${AWS::Region}:${AWS::AccountId}:subnet/${CodeBuildSubnet} Effect: Allow Resource: !Sub arn:aws:ec2:${AWS::Region}:${AWS::AccountId}:network-interface/* Version: 2012-10-17 ArtifactStoreS3Location: Type: 'AWS::S3::Bucket' Properties: Tags: - Key: Solution Value: !Sub 'Artifact store for - ${AWS::StackName}' VersioningConfiguration: Status: Enabled SnsApproval: Type: 'AWS::SNS::Topic' CodepipelineExecutionToken: Type: 'AWS::SSM::Parameter' Properties: Description: SSM Parameter for CodePipeline token. Name: !Sub 'pCodePipelineToken-${AWS::StackName}' Type: String Value: default MigrationLambdaRole: Type: 'AWS::IAM::Role' Properties: AssumeRolePolicyDocument: Statement: - Action: - 'sts:AssumeRole' Effect: Allow Principal: Service: - lambda.amazonaws.com Version: 2012-10-17 ManagedPolicyArns: - 'arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole' Policies: - PolicyName: !Sub 'lambda-${AWS::StackName}-${AWS::Region}-policy' PolicyDocument: Statement: - Action: 'codepipeline:PutApprovalResult' Effect: Allow Resource: '*' - Action: - 'sns:Publish' Effect: Allow Resource: !Ref MigrationNotification - Action: - 'ssm:GetParameter' - 'ssm:PutParameter' Effect: Allow Resource: !Sub arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/${CodepipelineExecutionToken} Version: 2012-10-17 MigrationLambdaFunction: Type: 'AWS::Lambda::Function' Properties: Code: ZipFile: | import boto3 import json import os ssm = boto3.client('ssm') sns = boto3.client('sns') codepipeline = boto3.client('codepipeline') ssm_parameter = os.environ['codepipeline_token'] pipeline_name = os.environ['pipeline_name'] task_name = os.environ['dms_task'].lower() topic = os.environ['notify_topic'] def lambda_handler(event, context): print('Received event: %s' % json.dumps(event, indent=2)) str_subject = event['Records'][0]['Sns']['Subject'] if 'APPROVAL NEEDED' in str_subject: print('This is a Codepipeline approval action') str_sns = event['Records'][0]['Sns']['Message'] sns_msg = json.loads(str_sns) pipeline = sns_msg['approval']['pipelineName'] stage = sns_msg['approval']['stageName'] action = sns_msg['approval']['actionName'] token = sns_msg['approval']['token'] approve_param ="pipelineName='%s',stageName='%s',actionName='%s',token='%s'" % ( pipeline , stage , action , token) print(approve_param) response = ssm.put_parameter(Name=ssm_parameter, Value=approve_param, Type='String', Overwrite=True ) elif 'DMS' in str_subject: print('This is a message from DMS') str_sns = event['Records'][0]['Sns']['Message'] if 'attempt' in str_sns: print(str_sns) print('Event notification nothing will be done') else: sns_msg = json.loads(str_sns) print(sns_msg['Event Message']) dms_status = sns_msg['Event Message'] if 'STOPPED_AFTER_FULL_LOAD' in dms_status: print('DMS task replication is stopped after full load, proceeding to put an approval in Codepipeline') result_pipeline('Approved') elif 'started' in dms_status: print('Lambda will do nothing at this step as the task is started') elif 'Create' in dms_status: print('Lambda will do nothing at this step as the task is created') elif 'FAIL' in dms_status.upper(): status = 'DMS task failed. Please check the task' print(status) subj = 'Status Update on DMS Task ' + task_name sns.publish(TopicArn = topic, Message = status, Subject = subj) result_pipeline('Rejected') else: status = 'DMS task did not stop or errored out after full load. Please check the task' print(status) subj = 'Status Update on DMS Task ' + task_name sns.publish(TopicArn = topic, Message = status, Subject = subj) result_pipeline('Rejected') else: print('This message is from neither Codepipeline Approval or DMS event. Nothing will be done') def result_pipeline(event): print('Getting Codepipeline parameters from SSM to put a %s' %(event)) codepipeline_params = ssm.get_parameter(Name=ssm_parameter)['Parameter']['Value'].split("'") print(codepipeline_params) result_reponse = codepipeline.put_approval_result( pipelineName=codepipeline_params[1], stageName=codepipeline_params[3], actionName=codepipeline_params[5], result={ 'summary': event, 'status': event }, token=codepipeline_params[7] ) print(result_reponse) Environment: Variables: codepipeline_token: !Ref CodepipelineExecutionToken pipeline_name: !Ref DmsPipeline dms_task: !If [cTaskName, !Ref 'AWS::StackName', !Ref TaskName] notify_topic: !Ref MigrationNotification Handler: index.lambda_handler Runtime: python3.6 Role: !GetAtt - MigrationLambdaRole - Arn Timeout: 300 CheckEndpointConnection: Type: 'Custom::DmsEndpointConnectionTester' Properties: ServiceToken: !GetAtt DmsEndpointConnTest.Arn SourceArn: !Ref DmsSourceArn TargetArn: !Ref DmsTargetArn ReplicationInstanceArn: !Ref DmsReplicationArn DmsEndpointConnTestRole: Type: 'AWS::IAM::Role' Properties: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Principal: Service: lambda.amazonaws.com Action: 'sts:AssumeRole' ManagedPolicyArns: - 'arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole' Path: / Policies: - PolicyName: dms-connection-access PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - 'dms:DescribeConnections' - 'dms:TestConnection' Resource: '*' DmsEndpointConnTest: Type: 'AWS::Lambda::Function' Properties: Description: Checks the connection between DMS replication instance and DMS endpoint Handler: index.lambda_handler Runtime: python3.7 Role: !GetAtt - DmsEndpointConnTestRole - Arn Timeout: 900 Code: ZipFile: | import cfnresponse import json import boto3 def lambda_handler(event, context): print('Received event: %s' % json.dumps(event, indent=2)) source_endpoint = event['ResourceProperties']['SourceArn'] target_endpoint = event['ResourceProperties']['TargetArn'] replication_inst = event['ResourceProperties']['ReplicationInstanceArn'] try: if (event['RequestType'] == 'Create') or (event['RequestType'] == 'Update'): print ('This is a %s event' %(event['RequestType'])) print('Checking connection for Source .....') source_result = check_connection(source_endpoint,replication_inst) print('Source result was %s' %(source_result)) if 'success' in source_result: print('Proceeding to check connection for Target ....') target_result = check_connection(target_endpoint,replication_inst) print('Target result was %s' %(target_result)) if 'success' in target_result: cfnresponse.send(event, context, cfnresponse.SUCCESS, {}, '') else: print('Target connection failed') cfnresponse.send(event, context, cfnresponse.FAILED, {}, '') else: print('Source connection failed') cfnresponse.send(event, context, cfnresponse.FAILED, {}, '') elif event['RequestType'] == 'Delete': print('Delete event nothing will be done') cfnresponse.send(event, context, cfnresponse.SUCCESS, {}, '') except Exception as e: print(e) cfnresponse.send(event, context, cfnresponse.FAILED, {}, '') def check_connection(endpoint,rep): dms = boto3.client('dms') dms.test_connection(ReplicationInstanceArn=rep,EndpointArn=endpoint) waiter = dms.get_waiter('test_connection_succeeds') waiter.wait( Filters=[ { 'Name': 'endpoint-arn', 'Values': [endpoint] }, { 'Name': 'replication-instance-arn', 'Values':[rep] } ] ) status_conn_api = dms.describe_connections( Filters=[ { 'Name': 'endpoint-arn', 'Values': [endpoint] }, { 'Name': 'replication-instance-arn', 'Values': [rep] } ] ) stat_task = status_conn_api['Connections'][0]['Status'] print('The connection test was %s' %(stat_task)) return (stat_task) LambdaPermissionGrantSNS: Type: 'AWS::Lambda::Permission' Properties: FunctionName: !GetAtt - MigrationLambdaFunction - Arn Action: 'lambda:InvokeFunction' Principal: sns.amazonaws.com SourceArn: !Ref SnsApproval SnsSubscription: Type: 'AWS::SNS::Subscription' Properties: Protocol: lambda Endpoint: !GetAtt - MigrationLambdaFunction - Arn TopicArn: !Ref SnsApproval SnsTopicPolicy: Type: 'AWS::SNS::TopicPolicy' Properties: PolicyDocument: Version: 2012-10-17 Id: __default_policy_ID Statement: - Sid: !Sub 'SNS-${AWS::StackName}-${AWS::Region}-policy' Effect: Allow Principal: AWS: '*' Action: - 'SNS:GetTopicAttributes' - 'SNS:SetTopicAttributes' - 'SNS:AddPermission' - 'SNS:RemovePermission' - 'SNS:DeleteTopic' - 'SNS:Subscribe' - 'SNS:ListSubscriptionsByTopic' - 'SNS:Publish' - 'SNS:Receive' Resource: '*' Condition: StringEquals: 'AWS:SourceOwner': !Ref 'AWS::AccountId' - Sid: dms-allow-publish Effect: Allow Principal: Service: dms.amazonaws.com Action: 'sns:Publish' Resource: !Ref SnsApproval Topics: - !Ref SnsApproval ArtifactStoreCleanUp: Type: 'Custom::S3CleanUp' Properties: DestBucket: !Ref ArtifactStoreS3Location ServiceToken: !GetAtt - S3CleanUpFunction - Arn S3CleanUpRole: Type: 'AWS::IAM::Role' Properties: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Action: 'sts:AssumeRole' Effect: Allow Principal: Service: lambda.amazonaws.com ManagedPolicyArns: - 'arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole' Path: / Policies: - PolicyDocument: Statement: - Action: - 's3:PutObject' - 's3:DeleteObject' - 's3:GetObject' - 's3:ListBucket' - 's3:ListBucketVersions' - 's3:DeleteObjectVersion' - 's3:GetObjectVersion' - 's3:GetBucketVersioning' Effect: Allow Resource: !Sub 'arn:aws:s3:::${ArtifactStoreS3Location}*' Version: 2012-10-17 PolicyName: S3CleanupPolicy S3CleanUpFunction: Type: 'AWS::Lambda::Function' Properties: Description: Empty the S3 Buckets while deleting the Stack Handler: index.lambda_handler Role: !GetAtt - S3CleanUpRole - Arn Runtime: python3.6 Timeout: 240 Code: ZipFile: | import boto3 import cfnresponse import json def lambda_handler(event, context): try: bucketcfn=event['ResourceProperties']['DestBucket'] responseData = {} print('Received event: %s' % json.dumps(event, indent=2)) if event['RequestType'] == 'Create': print('Create stack operation nothing will be done') print(bucketcfn) elif event['RequestType'] == 'Delete': s3 = boto3.resource('s3') bucket = s3.Bucket(bucketcfn) bucket.object_versions.all().delete() print('Delete stack in progress the bucket is emptied') elif event['RequestType'] == 'Update': print('Update stack') cfnresponse.send(event, context, cfnresponse.SUCCESS, {}, '') except Exception as e: print(e) cfnresponse.send(event, context, cfnresponse.FAILURE, {}, '') DmsTaskCleanupRole: Condition: cCreateSource Type: 'AWS::IAM::Role' Properties: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Principal: Service: lambda.amazonaws.com Action: 'sts:AssumeRole' ManagedPolicyArns: - 'arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole' Path: / Policies: - PolicyName: dms-connection-access PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - 'dms:Describe*' - 'dms:DeleteReplicationTask' - 'dms:StopReplicationTask' Resource: '*' - PolicyName: codebuild-cleanup PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - 'codebuild:StopBuild' - 'codebuild:BatchDeleteBuilds' - 'codebuild:ListBuildsForProject' - 'codebuild:BatchGetBuilds' Resource: - !Sub 'arn:${AWS::Partition}:codebuild:${AWS::Region}:${AWS::AccountId}:project/${CodeBuildSetupTarget}' - !If [cCreateSource, !Sub 'arn:${AWS::Partition}:codebuild:${AWS::Region}:${AWS::AccountId}:project/${CodeBuildSetupSample}', !Ref 'AWS::NoValue'] - !If [cCdc, !Sub 'arn:${AWS::Partition}:codebuild:${AWS::Region}:${AWS::AccountId}:project/${CodeBuildPreCdc}', !Ref 'AWS::NoValue'] DmsCleanupResource: Condition: cCreateSource Type: 'Custom::DmsCleanupResource' DependsOn: - CleanupVpc Properties: ServiceToken: !GetAtt - DmsCleanupFunction - Arn TaskName: !If [cTaskName, !Ref 'AWS::StackName', !Ref TaskName] SampleCodeBuild : !If [cCreateSource,!Ref CodeBuildSetupSample, !Ref 'AWS::NoValue'] PreCDC: !If [cCdc, !Ref CodeBuildPreCdc, !Ref 'AWS::NoValue'] SetupTarget: !Ref CodeBuildSetupTarget DmsCleanupFunction: Condition: cCreateSource Type: 'AWS::Lambda::Function' Properties: Description: Cleans up the DMS replication task Handler: index.lambda_handler Runtime: python3.7 Role: !GetAtt - DmsTaskCleanupRole - Arn Timeout: 900 Code: ZipFile: | import cfnresponse import json import boto3 import time import botocore dms = boto3.client('dms') codebuild = boto3.client('codebuild') def lambda_handler(event, context): task_name = event['ResourceProperties']['TaskName'].lower() sample = event['ResourceProperties']['SampleCodeBuild'] precdc = event['ResourceProperties']['PreCDC'] setup = event['ResourceProperties']['SetupTarget'] print('Received event: %s' % json.dumps(event, indent=2)) try: if event['RequestType'] == 'Delete': time.sleep(5) print('Delete event. Task and Codebuild will be stopped and deleted') print('The task name is %s' %(task_name)) if sample: cleanup_codebuild(sample) else: print('No sample') if precdc: cleanup_codebuild(precdc) else: print('Full Load only') cleanup_codebuild(setup) try: task_des = dms.describe_replication_tasks(Filters=[{'Name': 'replication-task-id','Values':[task_name]}]) task_arn = task_des['ReplicationTasks'][0]['ReplicationTaskArn'] print('The task arn is %s' %(task_arn)) dms.stop_replication_task(ReplicationTaskArn=task_arn) waiter_stop = dms.get_waiter('replication_task_stopped') waiter_stop.wait(Filters=[{'Name': 'replication-task-arn','Values':[task_arn]}]) dms.delete_replication_task(ReplicationTaskArn=task_arn) waiter = dms.get_waiter('replication_task_deleted') waiter.wait(Filters=[{'Name': 'replication-task-arn','Values':[task_arn]}]) except botocore.exceptions.ClientError as e: if e.response['Error']['Code'] == 'ResourceNotFoundFault': print('Task was not found. Proceeding with delete') else: raise cfnresponse.send(event, context, cfnresponse.SUCCESS, {}, '') else: print ('This is a %s event. Nothing will be done' %(event['RequestType'])) print ('Task name is %s' %(task_name)) cfnresponse.send(event, context, cfnresponse.SUCCESS, {}, '') except Exception as e: print(e) cfnresponse.send(event, context, cfnresponse.FAILED, {}, '') def cleanup_codebuild(proj): try: r = codebuild.list_builds_for_project(projectName=proj) if r['ids']: for build_id in r['ids']: print('Stopping build %s' %(build_id)) s = codebuild.stop_build(id=build_id) i = 0 while i < 10: time.sleep(6) i = i + 1 t = codebuild.batch_get_builds(ids=[build_id]) status = t['builds'][0]['buildStatus'] if status in ['STOPPED','SUCCEEDED']: print('Build stopped %s' %(build_id)) break print('Deleting build %s' %(build_id)) del_build = codebuild.batch_delete_builds(ids=[build_id]) except botocore.exceptions.ClientError as e: if e.response['Error']['Code'] == 'ResourceNotFoundFault': print('Build was not found. Proceeding with delete') else: print(e) raise CleanupVpcRole: Type: AWS::IAM::Role # Condition: CreateSample Properties: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Principal: Service: - lambda.amazonaws.com Action: - 'sts:AssumeRole' ManagedPolicyArns: - !Sub 'arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole' Path: / Policies: - PolicyName: ec2-read-access PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - 'ec2:DescribeNetworkInterfaces' - 'ec2:DeleteNetworkInterface' Resource: '*' CleanupVpc: Type: 'Custom::CleanupVpc' Properties: ServiceToken: !GetAtt - CleanupVpcFunction - Arn SecurityGroup: !Ref CodeBuildSecurityGroup CleanupVpcFunction: Type: 'AWS::Lambda::Function' # Condition: CreateSample Properties: Code: ZipFile: | import cfnresponse import boto3 import traceback import json import time ec2 = boto3.client('ec2') def lambda_handler(event, context): try: print('Received event: %s' % json.dumps(event, indent=2)) sg = event['ResourceProperties']['SecurityGroup'] if event['RequestType'] == 'Delete': time.sleep(60) print('Delete stack call, network interfaces will be cleaned up.') desc_net = ec2.describe_network_interfaces(Filters=[{'Name': 'group-id','Values': [sg]}]) for net in desc_net['NetworkInterfaces']: netid = net['NetworkInterfaceId'] print(netid) if net['Status'] == 'available': print(netid + ' is available') delete_net = ec2.delete_network_interface(NetworkInterfaceId=netid) elif event['RequestType'] == 'Update': print('Stack is getting updated nothing will be done') elif event['RequestType'] == 'Create': print('Stack is getting created nothing will be done') cfnresponse.send(event, context, cfnresponse.SUCCESS, {}, '') except: print(traceback.print_exc()) cfnresponse.send(event, context, cfnresponse.FAILED, {}, '') Role: !GetAtt - CleanupVpcRole - Arn Runtime: python3.7 Handler: index.lambda_handler Timeout: 900 Outputs: DmsPipeline: Description: Pipeline for DMS tasks Value: !Ref DmsPipeline DmsPipelineRole: Description: IAM Role used by CodePipeline Value: !Ref DmsPipelineRole SnsTopicPolicy: Description: SNS Topic Policy for DMS and CodePipeline Events Value: !Ref SnsTopicPolicy SnsSubscription: Description: SNS Subscription to Lambda function Value: !Ref SnsSubscription MigrationLambdaFunction: Description: Lambda function for Approval Stage Value: !Ref MigrationLambdaFunction MigrationLambdaRole: Description: IAM role used by Lambda function Value: !Ref MigrationLambdaRole CodepipelineExecutionToken: Description: Execution token for Codepipeline Approval stage Value: !Ref CodepipelineExecutionToken SnsApproval: Description: SNS Topic for DMS and CodePipeline events Value: !Ref SnsApproval ArtifactStoreS3Location: Description: S3 bucket for Codepipeline artifacts Value: !Ref ArtifactStoreS3Location CodeBuildServiceRole: Description: CodeBuild service role Value: !Ref CodeBuildServiceRole CodeBuildPreCdc: Description: PreCDC CodeBuild Project Value: !If [cCdc, !Ref CodeBuildPreCdc, 'none' ] CodeBuildSetupTarget: Description: SetupTarget CodeBuild Project Value: !Ref CodeBuildSetupTarget