--- AWSTemplateFormatVersion: 2010-09-09 Description: Amazon EFS performance tutorial - Creates multiple Auto Scaling groups & mounts an existing EFS file system Metadata: Authors: Description: Darryl Osborne (darrylo@amazon.com) License: Description: 'Copyright 2018 Amazon.com, Inc. and its affiliates. All Rights Reserved. SPDX-License-Identifier: MIT-0' AWS::CloudFormation::Interface: ParameterGroups: - Label: default: File System Parameters Parameters: - ElasticFileSystem - Label: default: EC2 Parameters Parameters: - KeyName - SecurityGroup - InstanceType0 - InstanceType1 - InstanceType2 - NumberOfSubnets - Subnet ParameterLabels: ElasticFileSystem: default: Amazon EFS File System InstanceType0: default: Instance Type 0 InstanceType1: default: Instance Type 1 InstanceType2: default: Instance Type 2 KeyName: default: Existing Key Pair NumberOfSubnets: default: Number of subnets SecurityGroup: default: Security Group Subnet: default: Subnets Parameters: ElasticFileSystem: AllowedPattern: ^(fs-)([a-z0-9]{8})$ Description: The Amazon EFS file system id. Type: String KeyName: AllowedPattern: ^([a-zA-Z0-9 @.`~!#$%^&*()_+,\\-])*$ ConstraintDescription: Must be letters (upper or lower), numbers, and special characters. Description: Name of an EC2 KeyPair. Your EC2 instances will launch with this KeyPair. Type: AWS::EC2::KeyPair::KeyName InstanceType0: AllowedValues: - t2.nano - t2.micro - t2.small - t2.medium - t2.large - t2.xlarge - t2.2xlarge - m3.medium - m3.large - m3.xlarge - m3.2xlarge - m4.large - m4.xlarge - m4.2xlarge - m4.4xlarge - m4.10xlarge - m4.16xlarge - m5.large - m5.xlarge - m5.2xlarge - m5.4xlarge - m5.12xlarge - m5.24xlarge - c3.large - c3.xlarge - c3.2xlarge - c3.4xlarge - c3.8xlarge - c4.large - c4.xlarge - c4.2xlarge - c4.4xlarge - c4.8xlarge - c5.large - c5.xlarge - c5.2xlarge - c5.4xlarge - c5.9xlarge - c5.18xlarge - r3.large - r3.xlarge - r3.2xlarge - r3.4xlarge - r3.8xlarge - r4.large - r4.xlarge - r4.2xlarge - r4.4xlarge - r4.8xlarge - r4.16xlarge - x1.16xlarge - x1.32xlarge - x1e.xlarge - x1e.2xlarge - x1e.4xlarge - x1e.8xlarge - x1e.16xlarge - x1e.32xlarge - d2.xlarge - d2.2xlarge - d2.4xlarge - d2.8xlarge - h1.2xlarge - h1.4xlarge - h1.8xlarge - h1.16xlarge - i2.xlarge - i2.2xlarge - i2.4xlarge - i2.8xlarge - i3.large - i3.xlarge - i3.2xlarge - i3.4xlarge - i3.8xlarge - i3.16xlarge - f1.2xlarge - f1.16xlarge - g2.2xlarge - g2.8xlarge - g3.4xlarge - g3.8xlarge - g3.16xlarge - p2.xlarge - p2.8xlarge - p2.16xlarge - p3.2xlarge - p3.8xlarge - p3.16xlarge ConstraintDescription: Must be a valid Amazon EC2 instance type. Default: t2.micro Description: The Amazon EC2 instance type for Auto Scaling group 0. Type: String InstanceType1: AllowedValues: - t2.nano - t2.micro - t2.small - t2.medium - t2.large - t2.xlarge - t2.2xlarge - m3.medium - m3.large - m3.xlarge - m3.2xlarge - m4.large - m4.xlarge - m4.2xlarge - m4.4xlarge - m4.10xlarge - m4.16xlarge - m5.large - m5.xlarge - m5.2xlarge - m5.4xlarge - m5.12xlarge - m5.24xlarge - c3.large - c3.xlarge - c3.2xlarge - c3.4xlarge - c3.8xlarge - c4.large - c4.xlarge - c4.2xlarge - c4.4xlarge - c4.8xlarge - c5.large - c5.xlarge - c5.2xlarge - c5.4xlarge - c5.9xlarge - c5.18xlarge - r3.large - r3.xlarge - r3.2xlarge - r3.4xlarge - r3.8xlarge - r4.large - r4.xlarge - r4.2xlarge - r4.4xlarge - r4.8xlarge - r4.16xlarge - x1.16xlarge - x1.32xlarge - x1e.xlarge - x1e.2xlarge - x1e.4xlarge - x1e.8xlarge - x1e.16xlarge - x1e.32xlarge - d2.xlarge - d2.2xlarge - d2.4xlarge - d2.8xlarge - h1.2xlarge - h1.4xlarge - h1.8xlarge - h1.16xlarge - i2.xlarge - i2.2xlarge - i2.4xlarge - i2.8xlarge - i3.large - i3.xlarge - i3.2xlarge - i3.4xlarge - i3.8xlarge - i3.16xlarge - f1.2xlarge - f1.16xlarge - g2.2xlarge - g2.8xlarge - g3.4xlarge - g3.8xlarge - g3.16xlarge - p2.xlarge - p2.8xlarge - p2.16xlarge - p3.2xlarge - p3.8xlarge - p3.16xlarge ConstraintDescription: Must be a valid Amazon EC2 instance type. Default: m4.large Description: The Amazon EC2 instance type for Auto Scaling group 1. Type: String InstanceType2: AllowedValues: - t2.nano - t2.micro - t2.small - t2.medium - t2.large - t2.xlarge - t2.2xlarge - m3.medium - m3.large - m3.xlarge - m3.2xlarge - m4.large - m4.xlarge - m4.2xlarge - m4.4xlarge - m4.10xlarge - m4.16xlarge - m5.large - m5.xlarge - m5.2xlarge - m5.4xlarge - m5.12xlarge - m5.24xlarge - c3.large - c3.xlarge - c3.2xlarge - c3.4xlarge - c3.8xlarge - c4.large - c4.xlarge - c4.2xlarge - c4.4xlarge - c4.8xlarge - c5.large - c5.xlarge - c5.2xlarge - c5.4xlarge - c5.9xlarge - c5.18xlarge - r3.large - r3.xlarge - r3.2xlarge - r3.4xlarge - r3.8xlarge - r4.large - r4.xlarge - r4.2xlarge - r4.4xlarge - r4.8xlarge - r4.16xlarge - x1.16xlarge - x1.32xlarge - x1e.xlarge - x1e.2xlarge - x1e.4xlarge - x1e.8xlarge - x1e.16xlarge - x1e.32xlarge - d2.xlarge - d2.2xlarge - d2.4xlarge - d2.8xlarge - h1.2xlarge - h1.4xlarge - h1.8xlarge - h1.16xlarge - i2.xlarge - i2.2xlarge - i2.4xlarge - i2.8xlarge - i3.large - i3.xlarge - i3.2xlarge - i3.4xlarge - i3.8xlarge - i3.16xlarge - f1.2xlarge - f1.16xlarge - g2.2xlarge - g2.8xlarge - g3.4xlarge - g3.8xlarge - g3.16xlarge - p2.xlarge - p2.8xlarge - p2.16xlarge - p3.2xlarge - p3.8xlarge - p3.16xlarge ConstraintDescription: Must be a valid Amazon EC2 instance type. Default: c5.2xlarge Description: The Amazon EC2 instance type for Auto Scaling group 2. Type: String NumberOfSubnets: AllowedValues: - 1 - 2 - 3 - 4 - 5 - 6 Default: 3 Description: Number of subnets. This must match your selections in the list of Subnets below. Type: String SecurityGroup: Description: Select the EC2 security group that allows SSH access to the instance and NFS access to the EFS file system. Type: AWS::EC2::SecurityGroup::Id Subnet: Description: Select existing public subnets. Type: List Conditions: NumberOfSubnets1: !Equals [ 1, !Ref NumberOfSubnets ] NumberOfSubnets2: !Equals [ 2, !Ref NumberOfSubnets ] NumberOfSubnets3: !Equals [ 3, !Ref NumberOfSubnets ] NumberOfSubnets4: !Equals [ 4, !Ref NumberOfSubnets ] NumberOfSubnets5: !Equals [ 5, !Ref NumberOfSubnets ] NumberOfSubnets6: !Equals [ 6, !Ref NumberOfSubnets ] Subnet0: !Or - !Condition NumberOfSubnets1 - !Condition NumberOfSubnets2 - !Condition NumberOfSubnets3 - !Condition NumberOfSubnets4 - !Condition NumberOfSubnets5 - !Condition NumberOfSubnets6 Subnet1: !Or - !Condition NumberOfSubnets2 - !Condition NumberOfSubnets3 - !Condition NumberOfSubnets4 - !Condition NumberOfSubnets5 - !Condition NumberOfSubnets6 Subnet2: !Or - !Condition NumberOfSubnets3 - !Condition NumberOfSubnets4 - !Condition NumberOfSubnets5 - !Condition NumberOfSubnets6 Subnet3: !Or - !Condition NumberOfSubnets4 - !Condition NumberOfSubnets5 - !Condition NumberOfSubnets6 Subnet4: !Or - !Condition NumberOfSubnets5 - !Condition NumberOfSubnets6 Subnet5: !Condition NumberOfSubnets6 Mappings: RegionMap: us-east-1: AMI: ami-97785bed us-east-2: AMI: ami-f63b1193 us-west-2: AMI: ami-f2d3638a us-west-1: AMI: ami-824c4ee2 ca-central-1: AMI: ami-a954d1cd eu-west-1: AMI: ami-d834aba1 eu-west-2: AMI: ami-403e2524 eu-west-3: AMI: ami-8ee056f3 eu-central-1: AMI: ami-5652ce39 ap-southeast-1: AMI: ami-68097514 ap-northeast-2: AMI: ami-863090e8 ap-northeast-1: AMI: ami-ceafcba8 ap-southeast-2: AMI: ami-942dd1f6 ap-south-1: AMI: ami-531a4c3c sa-east-1: AMI: ami-84175ae8 Resources: InstanceProfile: Type: AWS::IAM::InstanceProfile Properties: Path: / Roles: - !Ref InstanceRole InstanceRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Principal: Service: - ec2.amazonaws.com Action: - sts:AssumeRole Path: / Policies: - PolicyName: efs-performance-tutorial PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - ec2:CreateTags - ec2:DescribeAvailabilityZones - ec2:DescribeNetworkInterfaceAttribute - ec2:DescribeNetworkInterfaces - ec2:DescribeSecurityGroups - ec2:DescribeSubnets - ec2:DescribeVpcAttribute - ec2:DescribeVpcs - elasticfilesystem:Describe* - kms:ListAliases Resource: '*' AutoScalingGroup0: Type: AWS::AutoScaling::AutoScalingGroup Properties: Cooldown: 60 HealthCheckGracePeriod: 120 HealthCheckType: EC2 LaunchConfigurationName: !Ref LaunchConfiguration0 MaxSize: 1 MinSize: 1 Tags: - Key: Name Value: 'EFS Performance Tutorial - Launching...' PropagateAtLaunch: true VPCZoneIdentifier: !If [ NumberOfSubnets1, [ !Select [ 0, !Ref Subnet ] ], !If [ NumberOfSubnets2, [ !Select [ 0, !Ref Subnet ], !Select [ 1, !Ref Subnet ] ], !If [ NumberOfSubnets3, [ !Select [ 0, !Ref Subnet ], !Select [ 1, !Ref Subnet ], !Select [ 2, !Ref Subnet ] ], !If [ NumberOfSubnets4, [ !Select [ 0, !Ref Subnet ], !Select [ 1, !Ref Subnet ], !Select [ 2, !Ref Subnet ], !Select [ 3, !Ref Subnet ] ], !If [ NumberOfSubnets5, [ !Select [ 0, !Ref Subnet ], !Select [ 1, !Ref Subnet ], !Select [ 2, !Ref Subnet ], !Select [ 3, !Ref Subnet ], !Select [ 4, !Ref Subnet ] ], [ !Select [ 0, !Ref Subnet ], !Select [ 1, !Ref Subnet ], !Select [ 2, !Ref Subnet ], !Select [ 3, !Ref Subnet ], !Select [ 4, !Ref Subnet ], !Select [ 5, !Ref Subnet ] ] ] ] ] ] ] CreationPolicy: ResourceSignal: Count: 0 Timeout: PT5M LaunchConfiguration0: Type: AWS::AutoScaling::LaunchConfiguration Metadata: AWS::CloudFormation::Init: configSets: install_all_utilities: - tools-install - smallfile-install - fpart-install - automake-1.14-install - mcp-install tools-install: commands: misctools-install: command: yum --enablerepo=epel install nload sysstat ioping -y ignoreErrors: false devtools-install: command: yum groupinstall "Development Tools" -y smallfile-install: files: /tmp/smallfile-install.sh: content: !Sub | #!/bin/bash -xe cd /home/ec2-user/ git clone https://github.com/bengland2/smallfile.git mode: 000777 owner: ec2-user group: ec2-user commands: smallfile-install: command: su -l ec2-user /tmp/smallfile-install.sh ignoreErrors: false fpart-install: files: /tmp/fpart-install.sh: content: !Sub | #!/bin/bash -xe cd /home/ec2-user/ git clone https://github.com/martymac/fpart.git cd /home/ec2-user/fpart autoreconf -i ./configure make sudo make install mode: 000777 owner: ec2-user group: ec2-user commands: fpart-install: command: su -l ec2-user /tmp/fpart-install.sh ignoreErrors: false automake-1.14-install: files: /tmp/automake-1.14-install.sh: content: !Sub | #!/bin/bash -xe cd /home/ec2-user wget http://ftp.gnu.org/gnu/automake/automake-1.14.tar.gz tar xvzf automake-1.14.tar.gz cd /home/ec2-user/automake-1.14 ./configure make sudo make install mode: 000777 owner: ec2-user group: ec2-user commands: automake-remove: command: yum remove automake -y ignoreErrors: false automake-1.14-install: command: su -l ec2-user /tmp/automake-1.14-install.sh ignoreErrors: false mcp-install: files: /tmp/mcp-install.sh: content: !Sub | #!/bin/bash -xe cd /home/ec2-user/ git clone https://github.com/pkolano/mutil.git wget http://ftp.gnu.org/gnu/coreutils/coreutils-8.22.tar.xz tar xvf coreutils-8.22.tar.xz cd /home/ec2-user/coreutils-8.22 patch -p1 < /home/ec2-user/mutil/patch/coreutils-8.22.patch ./configure make sudo cp src/cp /usr/local/bin/mcp sudo cp src/md5sum /usr/local/bin/msum mode: 000777 owner: ec2-user group: ec2-user commands: mcp-install: command: su -l ec2-user /tmp/mcp-install.sh ignoreErrors: false Properties: BlockDeviceMappings: - DeviceName: /dev/xvda Ebs: DeleteOnTermination: true VolumeSize: 10 VolumeType: gp2 - DeviceName: /dev/xvdf Ebs: DeleteOnTermination: true VolumeSize: 20 VolumeType: gp2 IamInstanceProfile: !Ref InstanceProfile ImageId: !FindInMap [ RegionMap, !Ref 'AWS::Region', AMI ] InstanceMonitoring: true InstanceType: !Ref InstanceType0 KeyName: !Ref KeyName SecurityGroups: - !Ref SecurityGroup UserData: "Fn::Base64": !Sub | #cloud-config repo_update: true repo_upgrade: all packages: - parallel - git - gcc runcmd: - instanceid=$(curl -s http://169.254.169.254/latest/meta-data/instance-id) - efs_directory=/efs - mkdir -p $efs_directory - chown ec2-user:ec2-user $efs_directory - echo "${ElasticFileSystem}.efs.${AWS::Region}.amazonaws.com:/ $efs_directory nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 0 0" >> /etc/fstab - mount -a -t nfs4 - mkdir -p $efs_directory/tutorial/touch/$instanceid - mkdir -p $efs_directory/tutorial/dd/$instanceid - mkdir -p $efs_directory/tutorial/cp/$instanceid - mkdir -p $efs_directory/tutorial/rsync/$instanceid - mkdir -p $efs_directory/tutorial/mcp/$instanceid - mkdir -p $efs_directory/tutorial/parallelcp/$instanceid - mkdir -p $efs_directory/tutorial/fpsync/$instanceid - mkdir -p $efs_directory/tutorial/parallelcpio/$instanceid - chown ec2-user:ec2-user $efs_directory/tutorial/ -R - ebs_directory=/ebs - mkdir $ebs_directory - mkfs -t ext4 /dev/xvdf - mount /dev/xvdf $ebs_directory - mkdir -p $ebs_directory/tutorial/dd/$instanceid - mkdir -p $ebs_directory/tutorial/smallfile - mkdir -p $ebs_directory/tutorial/data-1m - chown ec2-user:ec2-user $ebs_directory/tutorial/ -R - /opt/aws/bin/cfn-init --configsets install_all_utilities --verbose --stack ${AWS::StackName} --resource LaunchConfiguration0 --region ${AWS::Region} - python /home/ec2-user/smallfile/smallfile_cli.py --operation create --threads 10 --file-size 1024 --file-size-distribution exponential --files 500 --same-dir N --dirs-per-dir 1024 --hash-into-dirs Y --files-per-dir 10240 --top $ebs_directory/tutorial/smallfile - cp -R $ebs_directory/tutorial/smallfile/file_srcdir $ebs_directory/tutorial/data-1m/ #- rm $ebs_directory/tutorial/smallfile/ -r - aws ec2 create-tags --resources $instanceid --tags "Key=Name,Value=EFS Performance Tutorial - Ready" --region ${AWS::Region} AutoScalingGroup1: Type: AWS::AutoScaling::AutoScalingGroup Properties: Cooldown: 60 HealthCheckGracePeriod: 120 HealthCheckType: EC2 LaunchConfigurationName: !Ref LaunchConfiguration1 MaxSize: 1 MinSize: 1 Tags: - Key: Name Value: 'EFS Performance Tutorial - Launching...' PropagateAtLaunch: true VPCZoneIdentifier: !If [ NumberOfSubnets1, [ !Select [ 0, !Ref Subnet ] ], !If [ NumberOfSubnets2, [ !Select [ 0, !Ref Subnet ], !Select [ 1, !Ref Subnet ] ], !If [ NumberOfSubnets3, [ !Select [ 0, !Ref Subnet ], !Select [ 1, !Ref Subnet ], !Select [ 2, !Ref Subnet ] ], !If [ NumberOfSubnets4, [ !Select [ 0, !Ref Subnet ], !Select [ 1, !Ref Subnet ], !Select [ 2, !Ref Subnet ], !Select [ 3, !Ref Subnet ] ], !If [ NumberOfSubnets5, [ !Select [ 0, !Ref Subnet ], !Select [ 1, !Ref Subnet ], !Select [ 2, !Ref Subnet ], !Select [ 3, !Ref Subnet ], !Select [ 4, !Ref Subnet ] ], [ !Select [ 0, !Ref Subnet ], !Select [ 1, !Ref Subnet ], !Select [ 2, !Ref Subnet ], !Select [ 3, !Ref Subnet ], !Select [ 4, !Ref Subnet ], !Select [ 5, !Ref Subnet ] ] ] ] ] ] ] CreationPolicy: ResourceSignal: Count: 0 Timeout: PT5M LaunchConfiguration1: Type: AWS::AutoScaling::LaunchConfiguration Metadata: AWS::CloudFormation::Init: configSets: install_all_utilities: - tools-install - smallfile-install - fpart-install - automake-1.14-install - mcp-install tools-install: commands: misctools-install: command: yum --enablerepo=epel install nload sysstat ioping -y ignoreErrors: false devtools-install: command: yum groupinstall "Development Tools" -y smallfile-install: files: /tmp/smallfile-install.sh: content: !Sub | #!/bin/bash -xe cd /home/ec2-user/ git clone https://github.com/bengland2/smallfile.git mode: 000777 owner: ec2-user group: ec2-user commands: smallfile-install: command: su -l ec2-user /tmp/smallfile-install.sh ignoreErrors: false fpart-install: files: /tmp/fpart-install.sh: content: !Sub | #!/bin/bash -xe cd /home/ec2-user/ git clone https://github.com/martymac/fpart.git cd /home/ec2-user/fpart autoreconf -i ./configure make sudo make install mode: 000777 owner: ec2-user group: ec2-user commands: fpart-install: command: su -l ec2-user /tmp/fpart-install.sh ignoreErrors: false automake-1.14-install: files: /tmp/automake-1.14-install.sh: content: !Sub | #!/bin/bash -xe cd /home/ec2-user wget http://ftp.gnu.org/gnu/automake/automake-1.14.tar.gz tar xvzf automake-1.14.tar.gz cd /home/ec2-user/automake-1.14 ./configure make sudo make install mode: 000777 owner: ec2-user group: ec2-user commands: automake-remove: command: yum remove automake -y ignoreErrors: false automake-1.14-install: command: su -l ec2-user /tmp/automake-1.14-install.sh ignoreErrors: false mcp-install: files: /tmp/mcp-install.sh: content: !Sub | #!/bin/bash -xe cd /home/ec2-user/ git clone https://github.com/pkolano/mutil.git wget http://ftp.gnu.org/gnu/coreutils/coreutils-8.22.tar.xz tar xvf coreutils-8.22.tar.xz cd /home/ec2-user/coreutils-8.22 patch -p1 < /home/ec2-user/mutil/patch/coreutils-8.22.patch ./configure make sudo cp src/cp /usr/local/bin/mcp sudo cp src/md5sum /usr/local/bin/msum mode: 000777 owner: ec2-user group: ec2-user commands: mcp-install: command: su -l ec2-user /tmp/mcp-install.sh ignoreErrors: false Properties: BlockDeviceMappings: - DeviceName: /dev/xvda Ebs: DeleteOnTermination: true VolumeSize: 10 VolumeType: gp2 - DeviceName: /dev/xvdf Ebs: DeleteOnTermination: true VolumeSize: 20 VolumeType: gp2 IamInstanceProfile: !Ref InstanceProfile ImageId: !FindInMap [ RegionMap, !Ref 'AWS::Region', AMI ] InstanceMonitoring: true InstanceType: !Ref InstanceType1 KeyName: !Ref KeyName SecurityGroups: - !Ref SecurityGroup UserData: "Fn::Base64": !Sub | #cloud-config repo_update: true repo_upgrade: all packages: - parallel - git - gcc runcmd: - instanceid=$(curl -s http://169.254.169.254/latest/meta-data/instance-id) - efs_directory=/efs - mkdir -p $efs_directory - chown ec2-user:ec2-user $efs_directory - echo "${ElasticFileSystem}.efs.${AWS::Region}.amazonaws.com:/ $efs_directory nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 0 0" >> /etc/fstab - mount -a -t nfs4 - mkdir -p $efs_directory/tutorial/touch/$instanceid - mkdir -p $efs_directory/tutorial/dd/$instanceid - mkdir -p $efs_directory/tutorial/cp/$instanceid - mkdir -p $efs_directory/tutorial/rsync/$instanceid - mkdir -p $efs_directory/tutorial/mcp/$instanceid - mkdir -p $efs_directory/tutorial/parallelcp/$instanceid - mkdir -p $efs_directory/tutorial/fpsync/$instanceid - mkdir -p $efs_directory/tutorial/parallelcpio/$instanceid - chown ec2-user:ec2-user $efs_directory/tutorial/ -R - ebs_directory=/ebs - mkdir $ebs_directory - mkfs -t ext4 /dev/xvdf - mount /dev/xvdf $ebs_directory - mkdir -p $ebs_directory/tutorial/dd/$instanceid - mkdir -p $ebs_directory/tutorial/smallfile - mkdir -p $ebs_directory/tutorial/data-1m - chown ec2-user:ec2-user $ebs_directory/tutorial/ -R - /opt/aws/bin/cfn-init --configsets install_all_utilities --verbose --stack ${AWS::StackName} --resource LaunchConfiguration0 --region ${AWS::Region} - python /home/ec2-user/smallfile/smallfile_cli.py --operation create --threads 10 --file-size 1024 --file-size-distribution exponential --files 500 --same-dir N --dirs-per-dir 1024 --hash-into-dirs Y --files-per-dir 10240 --top $ebs_directory/tutorial/smallfile - cp -R $ebs_directory/tutorial/smallfile/file_srcdir $ebs_directory/tutorial/data-1m/ #- rm $ebs_directory/tutorial/smallfile/ -r - aws ec2 create-tags --resources $instanceid --tags "Key=Name,Value=EFS Performance Tutorial - Ready" --region ${AWS::Region} AutoScalingGroup2: Type: AWS::AutoScaling::AutoScalingGroup Properties: Cooldown: 60 HealthCheckGracePeriod: 120 HealthCheckType: EC2 LaunchConfigurationName: !Ref LaunchConfiguration2 MaxSize: 1 MinSize: 1 Tags: - Key: Name Value: 'EFS Performance Tutorial - Launching...' PropagateAtLaunch: true VPCZoneIdentifier: !If [ NumberOfSubnets1, [ !Select [ 0, !Ref Subnet ] ], !If [ NumberOfSubnets2, [ !Select [ 0, !Ref Subnet ], !Select [ 1, !Ref Subnet ] ], !If [ NumberOfSubnets3, [ !Select [ 0, !Ref Subnet ], !Select [ 1, !Ref Subnet ], !Select [ 2, !Ref Subnet ] ], !If [ NumberOfSubnets4, [ !Select [ 0, !Ref Subnet ], !Select [ 1, !Ref Subnet ], !Select [ 2, !Ref Subnet ], !Select [ 3, !Ref Subnet ] ], !If [ NumberOfSubnets5, [ !Select [ 0, !Ref Subnet ], !Select [ 1, !Ref Subnet ], !Select [ 2, !Ref Subnet ], !Select [ 3, !Ref Subnet ], !Select [ 4, !Ref Subnet ] ], [ !Select [ 0, !Ref Subnet ], !Select [ 1, !Ref Subnet ], !Select [ 2, !Ref Subnet ], !Select [ 3, !Ref Subnet ], !Select [ 4, !Ref Subnet ], !Select [ 5, !Ref Subnet ] ] ] ] ] ] ] CreationPolicy: ResourceSignal: Count: 0 Timeout: PT5M LaunchConfiguration2: Type: AWS::AutoScaling::LaunchConfiguration Metadata: AWS::CloudFormation::Init: configSets: install_all_utilities: - tools-install - smallfile-install - fpart-install - automake-1.14-install - mcp-install tools-install: commands: misctools-install: command: yum --enablerepo=epel install nload sysstat ioping -y ignoreErrors: false devtools-install: command: yum groupinstall "Development Tools" -y smallfile-install: files: /tmp/smallfile-install.sh: content: !Sub | #!/bin/bash -xe cd /home/ec2-user/ git clone https://github.com/bengland2/smallfile.git mode: 000777 owner: ec2-user group: ec2-user commands: smallfile-install: command: su -l ec2-user /tmp/smallfile-install.sh ignoreErrors: false fpart-install: files: /tmp/fpart-install.sh: content: !Sub | #!/bin/bash -xe cd /home/ec2-user/ git clone https://github.com/martymac/fpart.git cd /home/ec2-user/fpart autoreconf -i ./configure make sudo make install mode: 000777 owner: ec2-user group: ec2-user commands: fpart-install: command: su -l ec2-user /tmp/fpart-install.sh ignoreErrors: false automake-1.14-install: files: /tmp/automake-1.14-install.sh: content: !Sub | #!/bin/bash -xe cd /home/ec2-user wget http://ftp.gnu.org/gnu/automake/automake-1.14.tar.gz tar xvzf automake-1.14.tar.gz cd /home/ec2-user/automake-1.14 ./configure make sudo make install mode: 000777 owner: ec2-user group: ec2-user commands: automake-remove: command: yum remove automake -y ignoreErrors: false automake-1.14-install: command: su -l ec2-user /tmp/automake-1.14-install.sh ignoreErrors: false mcp-install: files: /tmp/mcp-install.sh: content: !Sub | #!/bin/bash -xe cd /home/ec2-user/ git clone https://github.com/pkolano/mutil.git wget http://ftp.gnu.org/gnu/coreutils/coreutils-8.22.tar.xz tar xvf coreutils-8.22.tar.xz cd /home/ec2-user/coreutils-8.22 patch -p1 < /home/ec2-user/mutil/patch/coreutils-8.22.patch ./configure make sudo cp src/cp /usr/local/bin/mcp sudo cp src/md5sum /usr/local/bin/msum mode: 000777 owner: ec2-user group: ec2-user commands: mcp-install: command: su -l ec2-user /tmp/mcp-install.sh ignoreErrors: false Properties: BlockDeviceMappings: - DeviceName: /dev/xvda Ebs: DeleteOnTermination: true VolumeSize: 10 VolumeType: gp2 - DeviceName: /dev/xvdf Ebs: DeleteOnTermination: true VolumeSize: 40 VolumeType: gp2 IamInstanceProfile: !Ref InstanceProfile ImageId: !FindInMap [ RegionMap, !Ref 'AWS::Region', AMI ] InstanceMonitoring: true InstanceType: !Ref InstanceType2 KeyName: !Ref KeyName SecurityGroups: - !Ref SecurityGroup UserData: "Fn::Base64": !Sub | #cloud-config repo_update: true repo_upgrade: all packages: - parallel - git - gcc runcmd: - instanceid=$(curl -s http://169.254.169.254/latest/meta-data/instance-id) - efs_directory=/efs - mkdir -p $efs_directory - chown ec2-user:ec2-user $efs_directory - echo "${ElasticFileSystem}.efs.${AWS::Region}.amazonaws.com:/ $efs_directory nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 0 0" >> /etc/fstab - mount -a -t nfs4 - mkdir -p $efs_directory/tutorial/touch/$instanceid - mkdir -p $efs_directory/tutorial/dd/$instanceid - mkdir -p $efs_directory/tutorial/cp/$instanceid - mkdir -p $efs_directory/tutorial/rsync/$instanceid - mkdir -p $efs_directory/tutorial/mcp/$instanceid - mkdir -p $efs_directory/tutorial/parallelcp/$instanceid - mkdir -p $efs_directory/tutorial/fpsync/$instanceid - mkdir -p $efs_directory/tutorial/parallelcpio/$instanceid - chown ec2-user:ec2-user $efs_directory/tutorial/ -R - ebs_directory=/ebs - mkdir $ebs_directory - mkfs -t ext4 /dev/xvdf - mount /dev/xvdf $ebs_directory - mkdir -p $ebs_directory/tutorial/dd/$instanceid - mkdir -p $ebs_directory/tutorial/smallfile - mkdir -p $ebs_directory/tutorial/data-1m - chown ec2-user:ec2-user $ebs_directory/tutorial/ -R - /opt/aws/bin/cfn-init --configsets install_all_utilities --verbose --stack ${AWS::StackName} --resource LaunchConfiguration0 --region ${AWS::Region} - python /home/ec2-user/smallfile/smallfile_cli.py --operation create --threads 10 --file-size 1024 --file-size-distribution exponential --files 500 --same-dir N --dirs-per-dir 1024 --hash-into-dirs Y --files-per-dir 10240 --top $ebs_directory/tutorial/smallfile - cp -R $ebs_directory/tutorial/smallfile/file_srcdir $ebs_directory/tutorial/data-1m/ #- rm $ebs_directory/tutorial/smallfile/ -r - aws ec2 create-tags --resources $instanceid --tags "Key=Name,Value=EFS Performance Tutorial - Ready" --region ${AWS::Region}