AWSTemplateFormatVersion: "2010-09-09" Description: "Creates glue S3 buckets and upload scripts to it." Resources: CopyCMK: Type: AWS::KMS::Key Properties: KeyPolicy: Version: '2012-10-17' Id: key-copycmk Statement: - Sid: Enable IAM User Permissions Effect: Allow Principal: AWS: Fn::Join: - '' - - 'arn:aws:iam::' - Ref: AWS::AccountId - :root Action: kms:* Resource: '*' GlueScriptsS3Bucket: Type: "AWS::S3::Bucket" Properties: BucketKeyEnabled: True VersioningConfiguration: Status: Enabled BucketEncryption: ServerSideEncryptionConfiguration: - ServerSideEncryptionByDefault: SSEAlgorithm: 'aws:kms' KMSMasterKeyID: !Ref CopyCMK # Properties: # BucketName: !Sub 'aws-glue-scripts-${AWS::AccountId}-${AWS::Region}' GlueTempS3Bucket: Type: "AWS::S3::Bucket" Properties: BucketKeyEnabled: True VersioningConfiguration: Status: Enabled BucketEncryption: ServerSideEncryptionConfiguration: - ServerSideEncryptionByDefault: SSEAlgorithm: 'aws:kms' KMSMasterKeyID: !Ref CopyCMK # Properties: # BucketName: !Sub 'aws-glue-temporary-${AWS::AccountId}-${AWS::Region}' CopyArtifacts: Type: Custom::CopyArtifacts Properties: ServiceToken: !GetAtt 'CopyArtifactsFunction.Arn' DestBucket: !Ref 'GlueScriptsS3Bucket' DestPrefix: 'admin/' SourceBucket: !Ref 'QSS3BucketName' Prefix: !Sub '${QSS3KeyPrefix}assets/glue/scripts/' Objects: - business_aggregate_daily.py - business_aggregate_monthly.py - business_aggregate_weekly.py - business_daily_to_redshift.py - transform_clean_to_business_partition.py - transform_raw_to_clean.py - transform_raw_to_clean_london.py - import_demo_data_to_redshift.py CopyArtifactsRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: lambda.amazonaws.com Action: sts:AssumeRole ManagedPolicyArns: - arn:aws-cn:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole Path: / Policies: - PolicyName: lambda-copier PolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Action: - s3:GetObject Resource: - !Sub 'arn:aws-cn:s3:::${QSS3BucketName}/${QSS3KeyPrefix}*' - Effect: Allow Action: - s3:PutObject - s3:DeleteObject Resource: - !Sub 'arn:aws-cn:s3:::${GlueScriptsS3Bucket}/*' CopyArtifactsFunction: Type: AWS::Lambda::Function Properties: Description: Copies objects from a source S3 bucket to a destination Handler: index.handler Runtime: python3.7 Role: !GetAtt 'CopyArtifactsRole.Arn' Timeout: 240 Code: ZipFile: | import json import logging import threading import boto3 import cfnresponse def copy_objects(source_bucket, dest_bucket, prefix, dest_prefix, objects): s3 = boto3.client('s3') for o in objects: key = prefix + o copy_source = { 'Bucket': source_bucket, 'Key': key } dest_key = dest_prefix + o print('copy_source: %s' % copy_source) print('dest_bucket = %s'%dest_bucket) print('key = %s' %key) print('dest_key = %s' % dest_key) s3.copy_object(CopySource=copy_source, Bucket=dest_bucket, Key=dest_key) def delete_objects(bucket, prefix, objects): s3 = boto3.client('s3') objects = {'Objects': [{'Key': prefix + o} for o in objects]} s3.delete_objects(Bucket=bucket, Delete=objects) def timeout(event, context): logging.error('Execution is about to time out, sending failure response to CloudFormation') cfnresponse.send(event, context, cfnresponse.FAILED, {}, None) def handler(event, context): # make sure we send a failure to CloudFormation if the function # is going to timeout timer = threading.Timer((context.get_remaining_time_in_millis() / 1000.00) - 0.5, timeout, args=[event, context]) timer.start() print('Received event: %s' % json.dumps(event)) status = cfnresponse.SUCCESS try: source_bucket = event['ResourceProperties']['SourceBucket'] dest_bucket = event['ResourceProperties']['DestBucket'] dest_prefix = event['ResourceProperties']['DestPrefix'] prefix = event['ResourceProperties']['Prefix'] objects = event['ResourceProperties']['Objects'] if event['RequestType'] == 'Delete': delete_objects(dest_bucket, dest_prefix, objects) else: copy_objects(source_bucket, dest_bucket, prefix, dest_prefix, objects) except Exception as e: logging.error('Exception: %s' % e, exc_info=True) status = cfnresponse.FAILED finally: timer.cancel() cfnresponse.send(event, context, status, {}, None) Outputs: GlueScriptsS3Bucket: Value: !Ref GlueScriptsS3Bucket GlueTempS3Bucket: Value: !Ref GlueTempS3Bucket Parameters: QSS3BucketName: AllowedPattern: ^[0-9a-zA-Z]+([0-9a-zA-Z-]*[0-9a-zA-Z])*$ Default: aws-cn-quickstart-cn-northwest-1 Type: String QSS3KeyPrefix: AllowedPattern: ^[0-9a-zA-Z-/]*$ Default: quickstart-aws-utility-meter-data-analytics-platform-cn/ Type: String