--- AWSTemplateFormatVersion: "2010-09-09" Description: >- (WWPS-GLS-WF-CONTAINER-BUILD) Creates resources for building a Docker container image using CodeBuild, storing the image in ECR, and optionally creating a corresponding Batch Job Definition. It is recommended to name this stack "container-{ImageName}". Mappings: TagMap: default: architecture: "genomics-workflows" tags: - Key: "architecture" Value: "genomics-workflows" Parameters: ImageName: Description: (REQUIRED) Name of the container image (does not include tag). An ECR image repository will be created with this name. Type: String ImageTag: Description: (REQUIRED) Container image tag to use (e.g. the version of the tool you are building). The image will also be tagged "latest". Type: String GitRepoType: Description: > Git repository hosting provider. Type: String Default: GITHUB AllowedValues: - CODECOMMIT - GITHUB - BITBUCKET GitCloneUrlHttp: Description: > The HTTP clone url for the GitHub repository that has container source code. For example - http://github.com/user/repo.git Type: String ProjectBranch: Description: branch, tag, or commit to use Type: String Default: main ProjectPath: Description: > Relative path in the repository to enter for the build. For example - ./path/to/container Type: String Default: "." ProjectBuildSpecFile: Description: > Relative path to the project buildspec.yml file or equivalent. Leave blank to use the template defined buildspec script. Default: "buildspec.yml" Type: String CreateBatchJobDefinition: Description: > Create an AWS Batch Job Definition for the container Type: String Default: "No" AllowedValues: - "Yes" - "No" BatchJobDefinitionName: Description: > AWS Batch Job Definition name for the container. Defaults to ImageName if not specified. Type: String Default: "" ArtifactBucketName: Type: String Default: aws-genomics-workflows Description: >- S3 Bucket where artifacts and additions scripts are stored ArtifactBucketPrefix: Type: String Default: artifacts Description: >- Prefix in ArtifactBucketName where artifacts and additions scripts are stored Conditions: CreateBatchJobDefinitionTrue: Fn::Equals: - !Ref CreateBatchJobDefinition - "Yes" NoBatchJobDefinitionName: !Equals [ !Ref BatchJobDefinitionName, "" ] Resources: IAMCodeBuildRole: Type: AWS::IAM::Role Properties: Description: !Sub codebuild-service-role-${AWS::StackName}-${AWS::Region} Path: /service-role/ AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: codebuild.amazonaws.com Action: sts:AssumeRole Policies: - PolicyName: !Sub codebuild-basepolicy-${AWS::StackName}-${AWS::Region} PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Resource: - !Sub "arn:aws:codebuild:${AWS::Region}:${AWS::AccountId}:report-group/*" Action: - codebuild:CreateReportGroup - codebuild:CreateReport - codebuild:UpdateReport - codebuild:BatchPutTestCases - Effect: Allow Resource: - !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/codebuild/*" - !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/codebuild/*:*" Action: - logs:CreateLogGroup - logs:CreateLogStream - logs:PutLogEvents - Effect: Allow Resource: - !Sub arn:aws:codecommit:*:${AWS::AccountId}:* Action: - codecommit:GitPull - Effect: Allow Resource: # Constructing the ARN manually here since we are using a custom # resource to create the repository - !Sub "arn:aws:ecr:${AWS::Region}:${AWS::AccountId}:repository/${ImageName}" Action: - ecr:BatchCheckLayerAvailability - ecr:CompleteLayerUpload - ecr:InitiateLayerUpload - ecr:PutImage - ecr:UploadLayerPart - Effect: Allow Resource: - !Sub arn:aws:ecr:*:${AWS::AccountId}:repository/* Action: - ecr:DescribeRepositories - Effect: Allow Resource: "*" Action: - ecr:GetAuthorizationToken CodeBuildProject: Type: AWS::CodeBuild::Project Properties: Description: !Sub >- Builds the container image ${ImageName} Artifacts: Type: NO_ARTIFACTS Environment: Type: LINUX_CONTAINER Image: aws/codebuild/standard:1.0 ComputeType: BUILD_GENERAL1_LARGE PrivilegedMode: True EnvironmentVariables: - Name: AWS_ACCOUNT_ID Value: !Ref AWS::AccountId - Name: REGISTRY Value: !Sub "${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com" - Name: IMAGE_NAME Value: !Ref ImageName - Name: IMAGE_TAG Value: !Ref ImageTag - Name: PROJECT_BRANCH Value: !Ref ProjectBranch - Name: PROJECT_PATH Value: !Ref ProjectPath ServiceRole: !GetAtt IAMCodeBuildRole.Arn Source: Type: !Ref GitRepoType Location: !Ref GitCloneUrlHttp BuildSpec: !Ref ProjectBuildSpecFile Tags: !FindInMap ["TagMap", "default", "tags"] IAMLambdaExecutionRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Principal: Service: lambda.amazonaws.com Action: "sts:AssumeRole" Path: / ManagedPolicyArns: - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole - arn:aws:iam::aws:policy/service-role/AWSLambdaRole Policies: - PolicyName: !Sub codebuild-access-${AWS::Region} PolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Action: - "codebuild:StartBuild" - "codebuild:BatchGetBuilds" Resource: "*" - PolicyName: !Sub events-access-${AWS::Region} PolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Action: - events:DeleteRule - events:PutRule - events:PutTargets - events:RemoveTargets Resource: - !Sub arn:aws:events:*:${AWS::AccountId}:rule/* - PolicyName: !Sub lambda-access-${AWS::Region} PolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Action: - lambda:AddPermission - lambda:RemovePermission Resource: - !Sub arn:aws:lambda:*:${AWS::AccountId}:function:* - PolicyName: !Sub ecr-access-${AWS::Region} PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Resource: "*" Action: - "ecr:GetAuthorizationToken" - Effect: Allow Resource: !Sub "arn:aws:ecr:${AWS::Region}:${AWS::AccountId}:repository/*" Action: - "ecr:DescribeRepositories" - "ecr:CreateRepository" - Effect: Allow Resource: !Sub "arn:aws:ecr:${AWS::Region}:${AWS::AccountId}:repository/${ImageName}" Action: - "ecr:*LifecyclePolicy" - "ecr:DeleteRepository" - "ecr:BatchDeleteImage" CodeBuildInvocation: Type: Custom::CodeBuildInvocation Properties: ServiceToken: !GetAtt CodeBuildInvocationFunction.Arn BuildProject: !Ref CodeBuildProject # Need to explicitly define dependency on the ECR Custom Resource DependsOn: ECRRepositoryHandler CodeBuildInvocationFunction: Type: AWS::Lambda::Function Properties: Handler: lambda.handler Role: !GetAtt IAMLambdaExecutionRole.Arn Runtime: python3.7 Timeout: 60 Code: S3Bucket: !Ref ArtifactBucketName S3Key: !Sub ${ArtifactBucketPrefix}/lambda-codebuild.zip # ECR Repositories defined by CloudFormation currently do not support Deletion # or UpdateReplace Policies. This will cause failures when this stack or a # parent stack (if this template is nested) are deleted or updated. # # To work around this, we can use a custom resource that handles: # * pre-existing repositories on create and update events # * retaining repositories on delete events # # Preferred way to create an ECR Image Repository # # Leaving this here for reference # ECRRepository: # Type: "AWS::ECR::Repository" # Properties: # RepositoryName: !Ref ImageName # LifecyclePolicy: # LifecyclePolicyText: |- # { # "rules": [ # { # "rulePriority": 1, # "description": "Keep only one untagged image, expire all others", # "selection": { # "tagStatus": "untagged", # "countType": "imageCountMoreThan", # "countNumber": 1 # }, # "action": { # "type": "expire" # } # } # ] # } ECRRepositoryHandler: Type: Custom::ECRRepositoryHandler Properties: ServiceToken: !GetAtt ECRRepositoryHandlerFunction.Arn RepositoryName: !Ref ImageName DeletePolicy: Retain UpdateReplacePolicy: Retain LifecyclePolicy: LifecyclePolicyText: |- { "rules": [ { "rulePriority": 1, "description": "Keep only one untagged image, expire all others", "selection": { "tagStatus": "untagged", "countType": "imageCountMoreThan", "countNumber": 1 }, "action": { "type": "expire" } } ] } ECRRepositoryHandlerFunction: Type: AWS::Lambda::Function Properties: Handler: index.handler Role: !GetAtt IAMLambdaExecutionRole.Arn Runtime: python3.7 Timeout: 60 Code: # Breaking code out to a zip in S3 for code stalls the stack # S3Bucket: !Ref ArtifactBucketName # S3Key: !Sub ${ArtifactBucketPrefix}/lambda-ecr.zip ZipFile: | from time import sleep import boto3 import cfnresponse send, SUCCESS, FAILED = ( cfnresponse.send, cfnresponse.SUCCESS, cfnresponse.FAILED ) ecr = boto3.client('ecr') def wait(repo, until): until = until.lower() if until == "deleted": while True: try: sleep(1) ecr.describe_repositories(repositoryNames=[repo]) except ecr.exceptions.RepositoryNotFoundException: break if until == "exists": exists = False while not exists: try: sleep(1) exists = ecr.describe_repositories(repositoryNames=[repo])["repositories"] break except ecr.exceptions.RepositoryNotFoundException: exists = False def put_lifecycle_policy(repo, props): if props.get("LifecyclePolicy"): ecr.put_lifecycle_policy( repositoryName=repo, lifecyclePolicyText=props["LifecyclePolicy"]["LifecyclePolicyText"] ) def create(repo, props, event, context): # use existing repository if available, otherwise create try: ecr.create_repository(repositoryName=repo) wait(repo, "exists") put_lifecycle_policy(repo, props) except ecr.exceptions.RepositoryAlreadyExistsException: print(f"Repository '{repo}' already exists - CREATE ECR repository ignored") put_lifecycle_policy(repo, props) except Exception as e: send(event, context, FAILED, None) raise(e) def update(repo, props, event, context): # use existing repository if available update_policy = props.get("UpdateReplacePolicy") try: if update_policy and update_policy.lower() == "retain": put_lifecycle_policy(repo, props) else: # replace the repo delete(repo, props, event, context) create(repo, props, event, context) except Exception as e: send(event, context, FAILED, None) raise(e) def delete(repo, props, event, context): # retain repository if specified # otherwise force delete delete_policy = props.get("DeletePolicy") try: if delete_policy and not delete_policy.lower() == "retain": ecr.delete_repository(repositoryName=repo, force=True) wait(repo, "deleted") except Exception as e: send(event, context, FAILED, None) raise(e) def handler(event, context): props = event["ResourceProperties"] repo = props.get("RepositoryName") if event["RequestType"] in ("Create", "Update", "Delete"): action = globals()[event["RequestType"].lower()] action(repo, props, event, context) send(event, context, SUCCESS, None) else: # unhandled request type send(event, context, FAILED, None) BatchJobDef: Type: AWS::Batch::JobDefinition Condition: CreateBatchJobDefinitionTrue Properties: JobDefinitionName: Fn::If: - NoBatchJobDefinitionName - !Ref ImageName - !Ref BatchJobDefinitionName Type: container ContainerProperties: Image: !Sub ${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${ImageName}:${ImageTag} Vcpus: 8 Memory: 16000 Volumes: - Host: SourcePath: /opt/aws-cli Name: awscli MountPoints: - ContainerPath: /opt/aws-cli SourceVolume: awscli Outputs: CodeBuildProject: Value: !GetAtt CodeBuildProject.Arn Export: Name: !Sub ${AWS::StackName}-CodeBuildProject CodeBuildServiceRole: Value: !GetAtt IAMCodeBuildRole.Arn Export: Name: !Sub ${AWS::StackName}-CodeBuildServiceRole ContainerImage: Value: !Sub ${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${ImageName} Export: Name: !Sub ${AWS::StackName}-ECRImageRepository JobDefinition: Value: Fn::If: - CreateBatchJobDefinitionTrue - !Ref BatchJobDef - "-" Export: Name: !Sub ${AWS::StackName}-BatchJobDefinition ...