--- AWSTemplateFormatVersion: '2010-09-09' Description: >- (WWPS-GLS-WF-GWFCORE) Genomics workflow core execution environment Metadata: AWS::CloudFormation::Interface: ParameterGroups: - Label: default: Required Parameters: - S3BucketName - ExistingBucket - VpcId - SubnetIds - NumberOfSubnets - Label: default: Optional Parameters: - Namespace - DefaultCEMinvCpus - DefaultCEMaxvCpus - PriorityCEMinvCpus - PriorityCEMaxvCpus - CreateEFS - ExistingEFS - CreateFSx - FSxBucketName - FSxSubnetId - FSxStorageVolumeSize - FSxPerUnitStorageThroughput - FSxStorageType - BatchSpotBidPercentage - BatchComputeInstanceTypes - Label: default: Distribution Configuration Parameters: - ArtifactBucketName - ArtifactBucketPrefix - TemplateRootUrl ParameterLabels: S3BucketName: default: S3 Bucket Name ExistingBucket: default: Existing Bucket? VpcId: default: VPC ID SubnetIds: default: VPC Subnet IDs NumberOfSubnets: default: Number of Subnets DefaultCEMinvCpus: default: Default Min vCPU DefaultCEMaxvCpus: default: Default Max vCPU PriorityCEMinvCpus: default: High Priority Min vCPU PriorityCEMaxvCpus: default: High Priority Max vCPU CreateEFS: default: Create Amazon EFS File System ExistingEFS: default: Existing Amazon EFS File System ArtifactBucketName: default: Artifact S3 Bucket Name ArtifactBucketPrefix: default: Artifact S3 Prefix TemplateRootUrl: default: Template Root URL CreateFSx: default: Create Amazon FSx for Lustre File System FSxBucketName: default: A S3 bucket name that will be mounted on FSx FSxSubnetId: default: FSx for Lustre SubnetId FSxStorageVolumeSize: default: The size of FSx Storage Volume. FSxPerUnitStorageThroughput: default: The throughput for the FSx file system. FSxStorageType: default: The type of FS needed i.e. SSD or HDD or SCRATCH BatchSpotBidPercentage: default: The maximum percentage that an EC2 Spot Instance price can be when compared with the On-Demand price for that instance type before instances are launched. BatchComputeInstanceTypes: default: The list of instance types to be used for EC2 instances using Batch Compute Environments. Parameters: Namespace: Type: String Description: Optional namespace (e.g. project name) to use to label resources. If not specified the stack-name will be used. Default: "" VpcId: Type: AWS::EC2::VPC::Id Description: 'The VPC to create security groups and deploy AWS Batch to. NOTE: Must be the same VPC as the provided subnet IDs.' SubnetIds: Type: List Description: 'Subnets you want your batch compute environment to launch in. We recommend private subnets. NOTE: Must be from the VPC provided.' NumberOfSubnets: Type: Number Description: Number of subnets to launch into. Should correspond to the length of Subnet Ids MinValue: 1 MaxValue: 6 S3BucketName: Description: >- A S3 bucket name for storing analysis results. The bucket name must respect the S3 bucket naming conventions (can contain lowercase letters, numbers, periods and hyphens). The bucket must be in the same region this template is being deployed to. Type: String AllowedPattern: "((?=^.{3,63}$)(?!^(\\d+\\.)+\\d+$)(^(([a-z0-9]|[a-z0-9][a-z0-9\\-]*[a-z0-9])\\.)*([a-z0-9]|[a-z0-9][a-z0-9\\-]*[a-z0-9])$)|(^.{0}$))" ConstraintDescription: "Must respect AWS naming conventions" ExistingBucket: Description: Does this bucket already exist? Type: String AllowedValues: - "Yes" - "No" Default: "No" DefaultCEMinvCpus: Description: The minimum number of CPUs for the default Batch Compute Environment Type: Number Default: 0 MinValue: 0 DefaultCEMaxvCpus: Description: The maximum number of CPUs for the default Batch Compute Environment Type: Number Default: 100 MinValue: 0 PriorityCEMinvCpus: Description: The minimum number of CPUs for the high-priority Batch Compute Environment Type: Number Default: 0 MinValue: 0 PriorityCEMaxvCpus: Description: The maximum number of CPUs for the high-priority Batch Compute Environment Type: Number Default: 100 MinValue: 0 CreateEFS: Description: Create Amazon EFS File System, eg for shared data. Mounts to each subnet specified. Default is No. Type: String AllowedValues: - "Yes" - "No" Default: "No" ExistingEFS: Description: Specify existing EFS File System, eg for shared data. You should already have EFS Mounts to each subnet specified. If you select to create a new EFS file system, this parameter will be ignored. Type: String Default: "" CreateFSx: Description: Create Amazon FSx for Lustre File System. Default is No. Type: String AllowedValues: - "Yes" - "No" Default: "No" FSxBucketName: Description: >- A S3 bucket name that will be mounted on FSx Type: String Default: "" AllowedPattern: "((?=^.{3,63}$)(?!^(\\d+\\.)+\\d+$)(^(([a-z0-9]|[a-z0-9][a-z0-9\\-]*[a-z0-9])\\.)*([a-z0-9]|[a-z0-9][a-z0-9\\-]*[a-z0-9])$)|(^.{0}$))" ConstraintDescription: "Must respect AWS naming conventions" FSxSubnetId: Type: String Description: 'Subnet you want FSx for Lustre to launch in. We recommend a private subnet. NOTE: Must be from the VPC provided.' Default: "No" FSxStorageVolumeSize: Type: Number Description: The initial size of the FSx volume to be used in GB Default: 0 FSxPerUnitStorageThroughput: Type: Number Default: 0 Description: The throughput for the FSx file system. FSxStorageType: Type: String Description: The type of FS needed i.e. SSD/HDD/SCRATCH, all capitals. Default: SCRATCH AllowedValues: - "SSD" - "HDD" - "SCRATCH" ArtifactBucketName: Type: String Default: aws-genomics-workflows Description: >- S3 Bucket where distrubution artifacts and additions scripts are stored ArtifactBucketPrefix: Type: String Default: artifacts # dist: pin_version Description: >- Prefix in ArtifactBucketName where distribution artifacts and additions scripts are stored TemplateRootUrl: Type: String Description: >- Root URL for where nested templates are stored Default: https://aws-genomics-workflows.s3.amazonaws.com/templates # dist: pin_version ConstraintDescription: >- Must be a valid S3 HTTP URL AllowedPattern: "https://[a-z0-9-./]{0,}s3([a-z0-9-.]+)*\\.amazonaws\\.com/[a-z0-9-./]{3,}" BatchSpotBidPercentage: Type: Number Description: The maximum percentage that an EC2 Spot Instance price can be when compared with the On-Demand price for that instance type before instances are launched. Default: 100 BatchComputeInstanceTypes: Type: String Description: >- The list of instance types to be used for EC2 instances using Batch Compute Environments. This list works for the majority of genomics workflow tooling. If you have tooling that requires more compute resources, you can add larger (e.g. *.8xlarge and up) instance types. If you have workflows that read large amounts of data, but otherwise have modest compute resource needs, you can add "n" instances (e.g. c5n, m5n, r5n) - note check availability in your region. Default: "c4.large,m4.large,r4.large,c4.xlarge,m4.xlarge,r4.xlarge,c4.2xlarge,m4.2xlarge,r4.2xlarge,c4.4xlarge,m4.4xlarge,r4.4xlarge,c5.large,m5.large,r5.large,c5.xlarge,m5.xlarge,r5.xlarge,c5.2xlarge,m5.2xlarge,r5.2xlarge,c5.4xlarge,m5.4xlarge,r5.4xlarge" Conditions: NoNamespace: !Equals [ !Ref Namespace, "" ] EfsCondition: !Equals [ !Ref CreateEFS, "Yes"] ExistingEfsCondition: !And [ !Not [ !Equals [ !Ref ExistingEFS, "" ] ], !Not [ !Condition EfsCondition ] ] FsxCondition: !Equals [ !Ref CreateFSx, "Yes"] HasFSxBucket: !Not [!Equals ["", !Ref FSxBucketName]] Resources: S3Stack: Type: AWS::CloudFormation::Stack Properties: TemplateURL: !Sub ${TemplateRootUrl}/gwfcore/gwfcore-s3.template.yaml TimeoutInMinutes: 5 Parameters: Namespace: Fn::If: - NoNamespace - !Sub ${AWS::StackName} - !Ref Namespace S3BucketName: !Ref S3BucketName ExistingBucket: !Ref ExistingBucket IamStack: Type: AWS::CloudFormation::Stack DependsOn: S3Stack Properties: TemplateURL: !Sub ${TemplateRootUrl}/gwfcore/gwfcore-iam.template.yaml TimeoutInMinutes: 10 Parameters: Namespace: Fn::If: - NoNamespace - !Sub ${AWS::StackName} - !Ref Namespace S3BucketName: !Sub ${S3Stack.Outputs.BucketName} CodeStack: Type: AWS::CloudFormation::Stack Properties: TemplateURL: !Sub ${TemplateRootUrl}/gwfcore/gwfcore-code.template.yaml Parameters: Namespace: Fn::If: - NoNamespace - !Sub ${AWS::StackName} - !Ref Namespace InstallBucketName: !Sub ${S3Stack.Outputs.BucketName} ArtifactBucketName: !Ref ArtifactBucketName ArtifactBucketPrefix: !Ref ArtifactBucketPrefix EfsStack: Type: AWS::CloudFormation::Stack Condition: EfsCondition Properties: TemplateURL: !Sub ${TemplateRootUrl}/gwfcore/gwfcore-efs.template.yaml Parameters: VpcId: !Ref VpcId SubnetIds: !Join [",", !Ref SubnetIds ] NumberOfSubnets: !Ref NumberOfSubnets FsxStack: Type: AWS::CloudFormation::Stack Condition: FsxCondition Properties: TemplateURL: !Sub ${TemplateRootUrl}/gwfcore/gwfcore-fsx.template.yaml Parameters: VpcId: !Ref VpcId SubnetId: !Ref FSxSubnetId S3BucketName: !If [HasFSxBucket, !Ref FSxBucketName, AWS::NoValue] FSxStorageType: !Ref FSxStorageType FSxStorageVolumeSize: !Ref FSxStorageVolumeSize FSxPerUnitStorageThroughput: !Ref FSxPerUnitStorageThroughput LaunchTplStack: Type: AWS::CloudFormation::Stack DependsOn: - ParamEfsFileSystem - ParamFSxFileSystemID - ParamFSxFileSystemMount Properties: TemplateURL: !Sub ${TemplateRootUrl}/gwfcore/gwfcore-launch-template.template.yaml Parameters: Namespace: Fn::If: - NoNamespace - !Sub ${AWS::StackName} - !Ref Namespace BatchStack: Type: AWS::CloudFormation::Stack DependsOn: - IamStack - LaunchTplStack Properties: TemplateURL: !Sub ${TemplateRootUrl}/gwfcore/gwfcore-batch.template.yaml TimeoutInMinutes: 10 Parameters: Namespace: Fn::If: - NoNamespace - !Sub ${AWS::StackName} - !Ref Namespace LaunchTemplateId: !Sub ${LaunchTplStack.Outputs.LaunchTemplateId} VpcId: !Ref VpcId SubnetIds: !Join [",", !Ref SubnetIds ] DefaultCEMinvCpus: !Ref DefaultCEMinvCpus DefaultCEMaxvCpus: !Ref DefaultCEMaxvCpus PriorityCEMinvCpus: !Ref PriorityCEMinvCpus PriorityCEMaxvCpus: !Ref PriorityCEMaxvCpus BatchServiceRoleArn: !Sub ${IamStack.Outputs.BatchServiceRoleArn} Ec2InstanceProfileArn: !Sub ${IamStack.Outputs.BatchInstanceProfileArn} SpotFleetRoleArn: !Sub ${IamStack.Outputs.BatchSpotFleetRoleArn} FSxSubnetId: !Ref FSxSubnetId BatchSpotBidPercentage: !Ref BatchSpotBidPercentage BatchComputeInstanceTypes: !Ref BatchComputeInstanceTypes FSxSGIngressTCP: Condition: FsxCondition DependsOn: BatchStack Type: AWS::EC2::SecurityGroupIngress Properties: Description: "Allow TCP Connections for the Batch security group to FSx SG" GroupId: !Sub ${FsxStack.Outputs.FSxSecurityGroupId} SourceSecurityGroupId: !Sub ${BatchStack.Outputs.BatchSecurityGroupId} IpProtocol: tcp FromPort: 0 ToPort: 65535 ParamS3Bucket: Type: "AWS::SSM::Parameter" Properties: Name: Fn::Sub: - /gwfcore/${namespace}/s3-bucket - namespace: Fn::If: - NoNamespace - !Sub ${AWS::StackName} - !Ref Namespace Type: String #required Value: !Sub ${S3Stack.Outputs.BucketName} #required Description: GWFCore S3 Bucket ParamEfsFileSystem: Type: "AWS::SSM::Parameter" Properties: Name: Fn::Sub: - /gwfcore/${namespace}/efs-shared-file-system - namespace: Fn::If: - NoNamespace - !Sub ${AWS::StackName} - !Ref Namespace Type: String #required Value: !If [ EfsCondition, !GetAtt EfsStack.Outputs.EfsId, !If [ ExistingEfsCondition, !Ref ExistingEFS , 'none' ] ] # required Description: GWFCore EFS Shared File System ParamFSxFileSystemID: Type: "AWS::SSM::Parameter" Properties: Name: Fn::Sub: - /gwfcore/${namespace}/fsx-file-system - namespace: Fn::If: - NoNamespace - !Sub ${AWS::StackName} - !Ref Namespace Type: String #required Value: !If [ FsxCondition, !GetAtt FsxStack.Outputs.FSxId, 'none' ] # required Description: GWFCore FSx File System ID ParamFSxFileSystemMount: Type: "AWS::SSM::Parameter" Properties: Name: Fn::Sub: - /gwfcore/${namespace}/fsx-mount-name - namespace: Fn::If: - NoNamespace - !Sub ${AWS::StackName} - !Ref Namespace Type: String #required Value: !If [ FsxCondition, !GetAtt FsxStack.Outputs.FSxMount, 'none' ] # required Description: GWFCore FSx File System Mount Name ParamDefaultJobQueue: Type: "AWS::SSM::Parameter" Properties: Name: Fn::Sub: - /gwfcore/${namespace}/job-queue/default - namespace: Fn::If: - NoNamespace - !Sub ${AWS::StackName} - !Ref Namespace Type: String #required Value: !Sub ${BatchStack.Outputs.DefaultJobQueueArn} #required Description: GWFCore Default AWS Batch Job Queue ParamPriorityJobQueue: Type: "AWS::SSM::Parameter" Properties: Name: Fn::Sub: - /gwfcore/${namespace}/job-queue/priority - namespace: Fn::If: - NoNamespace - !Sub ${AWS::StackName} - !Ref Namespace Type: String #required Value: !Sub ${BatchStack.Outputs.PriorityJobQueueArn} #required Description: GWFCore Priority AWS Batch Job Queue ParamAdditionsRepoName: Type: "AWS::SSM::Parameter" Properties: Name: Fn::Sub: - /gwfcore/${namespace}/additions-repository/name - namespace: Fn::If: - NoNamespace - !Sub ${AWS::StackName} - !Ref Namespace Type: String #required Value: !Sub ${CodeStack.Outputs.EcsAdditionsRepoName} #required Description: GWFCore ECS Additions Repository Name ParamAdditionsRepoCloneUrlHttp: Type: "AWS::SSM::Parameter" Properties: Name: Fn::Sub: - /gwfcore/${namespace}/additions-repository/clone-url-http - namespace: Fn::If: - NoNamespace - !Sub ${AWS::StackName} - !Ref Namespace Type: String #required Value: !Sub ${CodeStack.Outputs.EcsAdditionsRepoCloneUrlHttp} #required Description: GWFCore ECS Additions Repository HTTP Clone Url ParamInstalledArtifactsBucket: Type: "AWS::SSM::Parameter" Properties: Name: Fn::Sub: - /gwfcore/${namespace}/installed-artifacts/bucket - namespace: Fn::If: - NoNamespace - !Sub ${AWS::StackName} - !Ref Namespace Type: String #required Value: !Sub ${CodeStack.Outputs.InstalledArtifactsBucket} #required Description: GWFCore S3 Bucket for Installed Artifacts ParamInstalledArtifactsPrefix: Type: "AWS::SSM::Parameter" Properties: Name: Fn::Sub: - /gwfcore/${namespace}/installed-artifacts/prefix - namespace: Fn::If: - NoNamespace - !Sub ${AWS::StackName} - !Ref Namespace Type: String #required Value: !Sub ${CodeStack.Outputs.InstalledArtifactsPrefix} #required Description: GWFCore S3 Prefix for Installed Artifacts ParamInstalledArtifactsS3RootUrl: Type: "AWS::SSM::Parameter" Properties: Name: Fn::Sub: - /gwfcore/${namespace}/installed-artifacts/s3-root-url - namespace: Fn::If: - NoNamespace - !Sub ${AWS::StackName} - !Ref Namespace Type: String #required Value: !Sub ${CodeStack.Outputs.InstalledArtifactsS3RootUrl} #required Description: GWFCore S3 Root Url for Installed Artifacts Outputs: S3BucketName: Value: !GetAtt S3Stack.Outputs.BucketName BatchJobRoleArn: Value: !Sub ${IamStack.Outputs.BatchJobRoleArn} DefaultJobQueueArn: Value: !Sub ${BatchStack.Outputs.DefaultJobQueueArn} PriorityJobQueueArn: Value: !Sub ${BatchStack.Outputs.PriorityJobQueueArn} ParamS3Bucket: Value: !Ref ParamS3Bucket ParamDefaultJobQueue: Value: !Ref ParamDefaultJobQueue ParamPriorityJobQueue: Value: !Ref ParamPriorityJobQueue ParamAdditionsRepoName: Value: !Ref ParamAdditionsRepoName ParamAdditionsRepoCloneUrlHttp: Value: !Ref ParamAdditionsRepoCloneUrlHttp ParamInstalledArtifactsBucket: Value: !Ref ParamInstalledArtifactsBucket ParamInstalledArtifactsPrefix: Value: !Ref ParamInstalledArtifactsPrefix ParamInstalledArtifactsS3RootUrl: Value: !Ref ParamInstalledArtifactsS3RootUrl FSxFileSystemID: Value: !If [FsxCondition, !GetAtt FsxStack.Outputs.FSxId, 'NA' ] FSxFileSystemMount: Value: !If [FsxCondition, !GetAtt FsxStack.Outputs.FSxMount, 'NA' ] FSxSecurityGroupId: Value: !If [FsxCondition, !GetAtt FsxStack.Outputs.FSxSecurityGroupId, 'NA' ]