AWSTemplateFormatVersion: "2010-09-09" Description: (SO8019) - This template is used for setting up a ZooKeeper cluster. (qs-1s4ldl6ti) Metadata: LICENSE: Apache License Version 2.0 AWS::CloudFormation::Interface: ParameterGroups: - Label: default: Network configuration Parameters: - PrivateSubnetID1 - PrivateSubnetID2 - Label: default: EC2 configuration Parameters: - ZookeeperServerSecurityGroupID - KeyPairName - Label: default: ZooKeeper configuration Parameters: - ZookeeperNodeName1 - ZookeeperNodeName2 - ZookeeperNodeName3 - ZookeeperInstanceType - ZookeeperVersion - ZookeeperNodeCount - CloudWatchAgentURL - RootStackName - DeviceName - VolumeSize - VolumeType - Iops - InstanceRoleArn - LatestAmiId - Label: default: AWS Quick Start configuration Parameters: - QSS3BucketName - QSS3KeyPrefix ParameterLabels: PrivateSubnetID1: default: Private subnet ID 1 PrivateSubnetID2: default: Private subnet ID 2 ZookeeperServerSecurityGroupID: default: ZooKeeper server security group KeyPairName: default: Key pair name ZookeeperNodeName1: default: ZooKeeper node name 1 ZookeeperNodeName2: default: ZooKeeper node name 2 ZookeeperNodeName3: default: ZooKeeper node name 3 ZookeeperInstanceType: default: ZooKeeper instance type ZookeeperVersion: default: ZooKeeper version ZookeeperNodeCount: default: ZooKeeper node count CloudWatchAgentURL: default: CloudWatch agent URL RootStackName: default: Prefix name of ZooKeeper cluster DeviceName: default: Device name VolumeSize: default: Volume size VolumeType: default: Volume type Iops: default: IOPS InstanceRoleArn: default: Instance role arn for aws or aws-cn QSS3BucketName: default: S3 bucket name for the Quick Start assets QSS3KeyPrefix: default: S3 key prefix for the Quick Start assets LatestAmiId: default: Reference the latest Amazon Linux AMI in a CloudFormation template Parameters: PrivateSubnetID1: Description: Private subnet ID 1. Type: String PrivateSubnetID2: Description: Private subnet ID 2. Type: String ZookeeperServerSecurityGroupID: Description: ID of the ZooKeeper server access security group (e.g., sg-7f16e910). Type: AWS::EC2::SecurityGroup::Id KeyPairName: Type: AWS::EC2::KeyPair::KeyName Description: Public/private key pairs allow you to securely connect to your instance after it launches. ZookeeperNodeName1: Default: ZookeeperNode1 Description: ZooKeeper node name 1. Type: String ZookeeperNodeName2: Default: ZookeeperNode2 Description: ZooKeeper node name 2. Type: String ZookeeperNodeName3: Default: ZookeeperNode3 Description: ZooKeeper node name 3. Type: String ZookeeperInstanceType: Description: Amazon EC2 instance type for the ZooKeeper nodes. Type: String Default: i3.2xlarge AllowedValues: - t2.micro - t2.small - t2.medium - t2.large - m4.large - m4.xlarge - m4.2xlarge - m4.4xlarge - m4.10xlarge - m5.large - m5.xlarge - m5.2xlarge - m5.4xlarge - c4.large - c4.xlarge - c4.2xlarge - c4.4xlarge - c4.8xlarge - r5.large - r5.xlarge - r5.2xlarge - r5.4xlarge - r5.8xlarge - i3.large - i3.xlarge - i3.2xlarge - i3.4xlarge - i3.8xlarge - i3.16xlarge ZookeeperVersion: AllowedValues: - '3.5.10' - '3.6.3' - '3.7.1' Default: '3.5.10' Description: ZooKeeper version (3.5.10). Type: String ZookeeperNodeCount: Default: 3 Description: ZooKeeper node count. Type: Number CloudWatchAgentURL: Description: For each download link, you can download and install the CloudWatch agent using the command line. Type: String RootStackName: Type: String Description: Prefix name of ZooKeeper cluster. DeviceName: Description: The device name (for example, /dev/sdh or xvdh). Type: String Default: /dev/xvdh VolumeSize: Type: String Description: EBS volume size (data) to be attached to node in GBs. Default: 500 VolumeType: Type: String Description: EBS volume type (data) to be attached to node in GBs [gp2,gp3,st1], one volume for data storage is mounted automatically by CloudFormation stack. Default: gp2 AllowedValues: - gp2 - gp3 - st1 - io1 Iops: Type: String Description: IOPS of EBS volume when io1 type is chosen. Otherwise ignored. Default: 1000 InstanceRoleArn: Description: For aws or aws-cn. Type: String Default: aws QSS3BucketName: AllowedPattern: ^[0-9a-zA-Z]+([0-9a-zA-Z-]*[0-9a-zA-Z])*$ ConstraintDescription: Quick Start bucket name can include numbers, lowercase letters, uppercase letters, and hyphens (-). It cannot start or end with a hyphen (-). Default: aws-quickstart Description: S3 bucket name for the Quick Start assets. This string can include numbers, lowercase letters, uppercase letters, and hyphens (-). It cannot start or end with a hyphen (-). Type: String QSS3KeyPrefix: AllowedPattern: ^[0-9a-zA-Z-/.]*$ ConstraintDescription: Quick Start key prefix can include numbers, lowercase letters, uppercase letters, hyphens (-), and forward slashes (/). Default: quickstart-clickhouse-cluster/ Description: S3 key prefix for the Quick Start assets. Quick Start key prefix can include numbers, lowercase letters, uppercase letters, hyphens (-), and forward slashes (/). Type: String LatestAmiId: Description: Reference the latest Amazon Linux AMI in a CloudFormation template. Type: AWS::SSM::Parameter::Value Default: /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2 Conditions: UseIops: Fn::Equals: [!Ref VolumeType,"io1"] Resources: ZookeeperMainLogGroup: Type: AWS::Logs::LogGroup Properties: LogGroupName: "ZookeeperLogGroup" RetentionInDays: 180 ZookeeperInstanceRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Statement: - Effect: Allow Principal: Service: - !Sub ec2.${AWS::URLSuffix} Action: - sts:AssumeRole Path: / Policies: - PolicyDocument: Statement: - Effect: Allow Action: - logs:CreateLogGroup - logs:CreateLogStream - logs:PutLogEvents - logs:DescribeLogStreams Resource: !Sub arn:${InstanceRoleArn}:logs:${AWS::Region}:${AWS::AccountId}:log-group:${ZookeeperMainLogGroup}:* - Effect: Allow Action: - s3:PutObjectAcl - s3:GetObjectAcl - s3:PutObject - s3:GetObject - s3:DeleteObject - s3:RestoreObject - s3:GetObjectVersion - s3:DeleteObjectVersion - s3:GetObjectRetention - s3:PutObjectRetention - s3:GetObjectVersionAcl - s3:PutObjectVersionAcl - s3:CreateBucket - s3:ListAllMyBuckets - s3:ListBucket - s3:PutBucketVersioning - s3:ListBucketVersions - s3:PutBucketLogging - s3:PutBucketAcl - s3:GetBucketAcl - s3:PutBucketPolicy - s3:GetBucketPolicy - s3:GetBucketLocation - s3:PutEncryptionConfiguration - s3:GetEncryptionConfiguration - s3:GetBucketVersioning - s3:GetLifecycleConfiguration - s3:GetIntelligentTieringConfiguration - s3:PutIntelligentTieringConfiguration - s3:PutLifecycleConfiguration Resource: !Sub arn:${InstanceRoleArn}:s3:::*/* - Effect: Allow Action: - ec2:CreateTags - ec2:DeleteTags Resource: !Sub arn:${InstanceRoleArn}:ec2:*:${AWS::AccountId}:instance/* - Effect: Allow Action: - ec2:DescribeTags Resource: '*' - Effect: Allow Action: - cloudwatch:PutDashboard - cloudwatch:DeleteAlarms - cloudwatch:DeleteInsightRules - cloudwatch:StartMetricStreams - cloudwatch:UntagResource - cloudwatch:ListTagsForResource - cloudwatch:GetDashboard - cloudwatch:GetInsightRuleReport - cloudwatch:DisableInsightRules - cloudwatch:DescribeAlarms - cloudwatch:GetMetricStream - cloudwatch:DeleteDashboards - cloudwatch:DescribeAlarmHistory - cloudwatch:StopMetricStreams - cloudwatch:DisableAlarmActions - cloudwatch:DeleteMetricStream - cloudwatch:SetAlarmState - cloudwatch:EnableInsightRules - cloudwatch:PutCompositeAlarm - cloudwatch:PutMetricStream - cloudwatch:PutInsightRule - cloudwatch:TagResource - cloudwatch:PutMetricAlarm - cloudwatch:EnableAlarmActions Resource: - !Sub arn:${InstanceRoleArn}:cloudwatch::${AWS::AccountId}:dashboard/* - !Sub arn:${InstanceRoleArn}:cloudwatch:*:${AWS::AccountId}:alarm:* - !Sub arn:${InstanceRoleArn}:cloudwatch:*:${AWS::AccountId}:insight-rule/* - !Sub arn:${InstanceRoleArn}:cloudwatch:*:${AWS::AccountId}:metric-stream/* - Effect: Allow Action: - cloudwatch:DescribeInsightRules - cloudwatch:PutMetricData - cloudwatch:GetMetricData - cloudwatch:ListMetricStreams - cloudwatch:DescribeAlarmsForMetric - cloudwatch:ListDashboards - cloudwatch:PutAnomalyDetector - cloudwatch:GetMetricStatistics - cloudwatch:GetMetricWidgetImage - cloudwatch:DeleteAnomalyDetector - cloudwatch:ListMetrics - cloudwatch:DescribeAnomalyDetectors Resource: '*' PolicyName: ZooKeeper-Instance-Role ZookeeperInstanceProfile: Type: AWS::IAM::InstanceProfile Properties: Path: / Roles: - !Ref ZookeeperInstanceRole ZookeeperInstance1: Type: AWS::EC2::Instance Metadata: Comment: Install zookeeper 1 AWS::CloudFormation::Authentication: S3AccessCreds: type: S3 roleName: !Ref ZookeeperInstanceRole AWS::CloudFormation::Init: configSets: default: - 01_setupCfnHup - 02_config-amazon-cloudwatch-agent - 03_restart_amazon-cloudwatch-agent UpdateEnvironment: - 02_config-amazon-cloudwatch-agent - 03_restart_amazon-cloudwatch-agent # Definition of json configuration of AmazonCloudWatchAgent, you can change the configuration below. 02_config-amazon-cloudwatch-agent: files: '/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json': content: !Sub | { "agent": { "metrics_collection_interval": 1, "logfile": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log" }, "metrics": { "append_dimensions": { "AutoScalingGroupName": "${!aws:AutoScalingGroupName}", "ImageId": "${!aws:ImageId}", "InstanceId": "${!aws:InstanceId}", "InstanceType": "${!aws:InstanceType}" }, "namespace": "ZookeeperNamespace", "metrics_collected": { "mem": { "measurement": [ "mem_used_percent" ] }, "cpu": { "resources": [ "*" ], "measurement": [ {"name": "cpu_usage_idle", "rename": "CPU_USAGE_IDLE", "unit": "Percent"}, {"name": "cpu_usage_nice", "unit": "Percent"}, {"name": "cpu_usage_user", "unit": "Percent"}, {"name": "cpu_usage_system", "unit": "Percent"} ], "totalcpu": true, "metrics_collection_interval": 10 }, "disk": { "resources": [ "/" ], "measurement": [ "used_percent", "total", "used" ], "ignore_file_system_types": [ "sysfs", "devtmpfs" ], "metrics_collection_interval": 60 }, "diskio": { "resources": [ "*" ], "measurement": [ "reads", "writes", "read_time", "write_time", "io_time" ], "metrics_collection_interval": 60 }, "swap": { "measurement": [ "swap_used_percent" ] }, "net": { "resources": [ "eth0" ], "measurement": [ "bytes_sent", "bytes_recv", "drop_in", "drop_out" ] } } }, "logs": { "force_flush_interval": 5, "logs_collected": { "files": { "collect_list": [ { "file_path": "/usr/local/apache-zookeeper-3.5.9-bin/logs/zookeeper--server-ip-*.internal.out", "log_group_name": "${ZookeeperMainLogGroup}", "log_stream_name": "${ZookeeperNodeName1}", "timestamp_format": "%Y-%m-%d %H:%M:%S", "timezone": "UTC" } ] } } } } # Invoke amazon-cloudwatch-agent-ctl to restart the AmazonCloudWatchAgent. 03_restart_amazon-cloudwatch-agent: commands: 01_stop_service: command: /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a stop 02_start_service: command: /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json -s # Cfn-hup setting, it is to monitor the change of metadata. # When there is change in the contents of json file in the metadata section, cfn-hup will call cfn-init to restart the AmazonCloudWatchAgent. 01_setupCfnHup: files: '/etc/cfn/cfn-hup.conf': content: !Sub | [main] stack=${AWS::StackId} region=${AWS::Region} interval=1 mode: '000400' owner: root group: root '/etc/cfn/hooks.d/amazon-cloudwatch-agent-auto-reloader.conf': content: !Sub | [cfn-auto-reloader-hook] triggers=post.update path=Resources.ZookeeperInstance1.Metadata.AWS::CloudFormation::Init.02_config-amazon-cloudwatch-agent action=/opt/aws/bin/cfn-init -v --stack ${AWS::StackId} --resource ZookeeperInstance1 --region ${AWS::Region} --configsets UpdateEnvironment runas=root mode: '000400' owner: root group: root "/lib/systemd/system/cfn-hup.service": content: | [Unit] Description=cfn-hup daemon [Service] Type=simple ExecStart=/opt/aws/bin/cfn-hup Restart=always [Install] WantedBy=multi-user.target commands: 01enable_cfn_hup: command: | systemctl enable cfn-hup.service 02start_cfn_hup: command: | systemctl start cfn-hup.service Properties: KeyName: !Ref KeyPairName IamInstanceProfile: !Ref ZookeeperInstanceProfile #ImageId: !FindInMap [AWSRegion2AMI, !Ref 'AWS::Region', HVM64] ImageId: !Ref LatestAmiId InstanceType: !Ref ZookeeperInstanceType SubnetId: !Ref PrivateSubnetID1 SecurityGroupIds: - !Ref ZookeeperServerSecurityGroupID BlockDeviceMappings: - DeviceName: !Ref DeviceName Ebs: VolumeSize: !Ref VolumeSize VolumeType: !Ref VolumeType Iops: !If [UseIops,!Ref Iops,!Ref AWS::NoValue] DeleteOnTermination: true UserData: !Base64 Fn::Sub: | #!/bin/bash yum update -y # This script below is to install AmazonCloudWatchAgent, restart AmazonCloudWatchAgent and tell the result to cloudformation. rpm -Uvh ${CloudWatchAgentURL} /opt/aws/bin/cfn-init -v --stack ${AWS::StackId} --resource ZookeeperInstance1 --region ${AWS::Region} --configsets default sudo mkfs.xfs ${DeviceName} sudo mkdir -p /data/ sudo mount ${DeviceName} /data sudo mkdir -p /data/zookeeper/{data,logs} echo "${DeviceName} /data xfs defaults 0 0" >> /etc/fstab cd /home/ec2-user/ aws s3 cp s3://${QSS3BucketName}/${QSS3KeyPrefix}scripts/zookeeper-install.sh ./ --region ${AWS::Region} aws s3 cp s3://${QSS3BucketName}/${QSS3KeyPrefix}scripts/find-zookeeper-node.py ./ --region ${AWS::Region} chmod +x zookeeper-install.sh ./zookeeper-install.sh ${ZookeeperVersion} > zk.log echo "1" >> /data/zookeeper/data/myid aws ec2 create-tags --region ${AWS::Region} --resources `curl http://169.254.169.254/latest/meta-data/instance-id` --tags Key=${RootStackName}-zookeeper-cluster,Value="server.1=`curl http://169.254.169.254/latest/meta-data/local-ipv4`:2888:3888" flag=1800 while((flag > 0)) do echo `aws ec2 describe-tags --filters Name=key,Values=${RootStackName}-zookeeper-cluster --region ${AWS::Region}` > instancelist count=`awk -v RS="@#$j" '{print gsub(/instance/,"&")}' instancelist` if (( $count >= ${ZookeeperNodeCount} )) then python3 find-zookeeper-node.py instancelist result while read line do echo $line echo $line >> /usr/local/apache-zookeeper-${ZookeeperVersion}-bin/conf/zoo.cfg done < result break fi echo $flag let flag-- sleep 1 done rm -rf /home/ec2-user/openjdk-8u41-b04-linux-x64-14_jan_2020.tar.gz rm -rf /home/ec2-user/apache-zookeeper-${ZookeeperVersion}-bin.tar.gz rm -rf /home/ec2-user/zookeeper-install.sh rm -rf /home/ec2-user/find-zookeeper-node.py rm -rf /home/ec2-user/instancelist rm -rf /home/ec2-user/result echo "/usr/local/apache-zookeeper-${ZookeeperVersion}-bin/bin/zkServer.sh start" > /home/ec2-user/zk-start.sh chmod +x /home/ec2-user/zk-start.sh echo "/home/ec2-user/zk-start.sh" >> /etc/rc.d/rc.local chmod +x /etc/rc.d/rc.local /usr/local/apache-zookeeper-${ZookeeperVersion}-bin/bin/zkServer.sh start /opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackId} --resource ZookeeperInstance1 --region ${AWS::Region} Tags: - Key: Name Value: !Ref ZookeeperNodeName1 CreationPolicy: ResourceSignal: Count: 1 Timeout: "PT120M" ZookeeperInstance2: Type: AWS::EC2::Instance Metadata: Comment: Install zookeeper 2 AWS::CloudFormation::Authentication: S3AccessCreds: type: S3 roleName: !Ref ZookeeperInstanceRole AWS::CloudFormation::Init: configSets: default: - 01_setupCfnHup - 02_config-amazon-cloudwatch-agent - 03_restart_amazon-cloudwatch-agent UpdateEnvironment: - 02_config-amazon-cloudwatch-agent - 03_restart_amazon-cloudwatch-agent # Definition of json configuration of AmazonCloudWatchAgent, you can change the configuration below. 02_config-amazon-cloudwatch-agent: files: '/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json': content: !Sub | { "agent": { "metrics_collection_interval": 1, "logfile": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log" }, "metrics": { "append_dimensions": { "AutoScalingGroupName": "${!aws:AutoScalingGroupName}", "ImageId": "${!aws:ImageId}", "InstanceId": "${!aws:InstanceId}", "InstanceType": "${!aws:InstanceType}" }, "namespace": "ZookeeperNamespace", "metrics_collected": { "mem": { "measurement": [ "mem_used_percent" ] }, "cpu": { "resources": [ "*" ], "measurement": [ {"name": "cpu_usage_idle", "rename": "CPU_USAGE_IDLE", "unit": "Percent"}, {"name": "cpu_usage_nice", "unit": "Percent"}, {"name": "cpu_usage_user", "unit": "Percent"}, {"name": "cpu_usage_system", "unit": "Percent"} ], "totalcpu": true, "metrics_collection_interval": 10 }, "disk": { "resources": [ "/" ], "measurement": [ "used_percent", "total", "used" ], "ignore_file_system_types": [ "sysfs", "devtmpfs" ], "metrics_collection_interval": 60 }, "diskio": { "resources": [ "*" ], "measurement": [ "reads", "writes", "read_time", "write_time", "io_time" ], "metrics_collection_interval": 60 }, "swap": { "measurement": [ "swap_used_percent" ] }, "net": { "resources": [ "eth0" ], "measurement": [ "bytes_sent", "bytes_recv", "drop_in", "drop_out" ] } } }, "logs": { "force_flush_interval": 5, "logs_collected": { "files": { "collect_list": [ { "file_path": "/usr/local/apache-zookeeper-3.5.9-bin/logs/zookeeper--server-ip-*.internal.out", "log_group_name": "${ZookeeperMainLogGroup}", "log_stream_name": "${ZookeeperNodeName2}", "timestamp_format": "%Y-%m-%d %H:%M:%S", "timezone": "UTC" } ] } } } } # Invoke amazon-cloudwatch-agent-ctl to restart the AmazonCloudWatchAgent. 03_restart_amazon-cloudwatch-agent: commands: 01_stop_service: command: /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a stop 02_start_service: command: /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json -s # Cfn-hup setting, it is to monitor the change of metadata. # When there is change in the contents of json file in the metadata section, cfn-hup will call cfn-init to restart the AmazonCloudWatchAgent. 01_setupCfnHup: files: '/etc/cfn/cfn-hup.conf': content: !Sub | [main] stack=${AWS::StackId} region=${AWS::Region} interval=1 mode: '000400' owner: root group: root '/etc/cfn/hooks.d/amazon-cloudwatch-agent-auto-reloader.conf': content: !Sub | [cfn-auto-reloader-hook] triggers=post.update path=Resources.ZookeeperInstance2.Metadata.AWS::CloudFormation::Init.02_config-amazon-cloudwatch-agent action=/opt/aws/bin/cfn-init -v --stack ${AWS::StackId} --resource ZookeeperInstance2 --region ${AWS::Region} --configsets UpdateEnvironment runas=root mode: '000400' owner: root group: root "/lib/systemd/system/cfn-hup.service": content: | [Unit] Description=cfn-hup daemon [Service] Type=simple ExecStart=/opt/aws/bin/cfn-hup Restart=always [Install] WantedBy=multi-user.target commands: 01enable_cfn_hup: command: | systemctl enable cfn-hup.service 02start_cfn_hup: command: | systemctl start cfn-hup.service Properties: KeyName: !Ref KeyPairName IamInstanceProfile: !Ref ZookeeperInstanceProfile #ImageId: !FindInMap [AWSRegion2AMI, !Ref 'AWS::Region', HVM64] ImageId: !Ref LatestAmiId InstanceType: !Ref ZookeeperInstanceType SubnetId: !Ref PrivateSubnetID1 SecurityGroupIds: - !Ref ZookeeperServerSecurityGroupID BlockDeviceMappings: - DeviceName: !Ref DeviceName Ebs: VolumeSize: !Ref VolumeSize VolumeType: !Ref VolumeType Iops: !If [UseIops,!Ref Iops,!Ref AWS::NoValue] DeleteOnTermination: true UserData: !Base64 Fn::Sub: | #!/bin/bash yum update -y # This script below is to install AmazonCloudWatchAgent, restart AmazonCloudWatchAgent and tell the result to cloudformation. rpm -Uvh ${CloudWatchAgentURL} /opt/aws/bin/cfn-init -v --stack ${AWS::StackId} --resource ZookeeperInstance2 --region ${AWS::Region} --configsets default sudo mkfs.xfs ${DeviceName} sudo mkdir -p /data/ sudo mount ${DeviceName} /data sudo mkdir -p /data/zookeeper/{data,logs} echo "${DeviceName} /data xfs defaults 0 0" >> /etc/fstab cd /home/ec2-user/ aws s3 cp s3://${QSS3BucketName}/${QSS3KeyPrefix}scripts/zookeeper-install.sh ./ --region ${AWS::Region} aws s3 cp s3://${QSS3BucketName}/${QSS3KeyPrefix}scripts/find-zookeeper-node.py ./ --region ${AWS::Region} chmod +x zookeeper-install.sh ./zookeeper-install.sh ${ZookeeperVersion} > zk.log echo "2" >> /data/zookeeper/data/myid aws ec2 create-tags --region ${AWS::Region} --resources `curl http://169.254.169.254/latest/meta-data/instance-id` --tags Key=${RootStackName}-zookeeper-cluster,Value="server.2=`curl http://169.254.169.254/latest/meta-data/local-ipv4`:2888:3888" flag=1800 while((flag > 0)) do echo `aws ec2 describe-tags --filters Name=key,Values=${RootStackName}-zookeeper-cluster --region ${AWS::Region}` > instancelist count=`awk -v RS="@#$j" '{print gsub(/instance/,"&")}' instancelist` if (( $count >= ${ZookeeperNodeCount} )) then python3 find-zookeeper-node.py instancelist result while read line do echo $line echo $line >> /usr/local/apache-zookeeper-${ZookeeperVersion}-bin/conf/zoo.cfg done < result break fi echo $flag let flag-- sleep 1 done rm -rf /home/ec2-user/openjdk-8u41-b04-linux-x64-14_jan_2020.tar.gz rm -rf /home/ec2-user/apache-zookeeper-${ZookeeperVersion}-bin.tar.gz rm -rf /home/ec2-user/zookeeper-install.sh rm -rf /home/ec2-user/find-zookeeper-node.py rm -rf /home/ec2-user/instancelist rm -rf /home/ec2-user/result echo "/usr/local/apache-zookeeper-${ZookeeperVersion}-bin/bin/zkServer.sh start" > /home/ec2-user/zk-start.sh chmod +x /home/ec2-user/zk-start.sh echo "/home/ec2-user/zk-start.sh" >> /etc/rc.d/rc.local chmod +x /etc/rc.d/rc.local /usr/local/apache-zookeeper-${ZookeeperVersion}-bin/bin/zkServer.sh start /opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackId} --resource ZookeeperInstance2 --region ${AWS::Region} Tags: - Key: Name Value: !Ref ZookeeperNodeName2 CreationPolicy: ResourceSignal: Count: 1 Timeout: "PT120M" ZookeeperInstance3: Type: AWS::EC2::Instance Metadata: Comment: Install zookeeper 3 AWS::CloudFormation::Authentication: S3AccessCreds: type: S3 roleName: !Ref ZookeeperInstanceRole AWS::CloudFormation::Init: configSets: default: - 01_setupCfnHup - 02_config-amazon-cloudwatch-agent - 03_restart_amazon-cloudwatch-agent UpdateEnvironment: - 02_config-amazon-cloudwatch-agent - 03_restart_amazon-cloudwatch-agent # Definition of json configuration of AmazonCloudWatchAgent, you can change the configuration below. 02_config-amazon-cloudwatch-agent: files: '/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json': content: !Sub | { "agent": { "metrics_collection_interval": 1, "logfile": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log" }, "metrics": { "append_dimensions": { "AutoScalingGroupName": "${!aws:AutoScalingGroupName}", "ImageId": "${!aws:ImageId}", "InstanceId": "${!aws:InstanceId}", "InstanceType": "${!aws:InstanceType}" }, "namespace": "ZookeeperNamespace", "metrics_collected": { "mem": { "measurement": [ "mem_used_percent" ] }, "cpu": { "resources": [ "*" ], "measurement": [ {"name": "cpu_usage_idle", "rename": "CPU_USAGE_IDLE", "unit": "Percent"}, {"name": "cpu_usage_nice", "unit": "Percent"}, {"name": "cpu_usage_user", "unit": "Percent"}, {"name": "cpu_usage_system", "unit": "Percent"} ], "totalcpu": true, "metrics_collection_interval": 10 }, "disk": { "resources": [ "/" ], "measurement": [ "used_percent", "total", "used" ], "ignore_file_system_types": [ "sysfs", "devtmpfs" ], "metrics_collection_interval": 60 }, "diskio": { "resources": [ "*" ], "measurement": [ "reads", "writes", "read_time", "write_time", "io_time" ], "metrics_collection_interval": 60 }, "swap": { "measurement": [ "swap_used_percent" ] }, "net": { "resources": [ "eth0" ], "measurement": [ "bytes_sent", "bytes_recv", "drop_in", "drop_out" ] } } }, "logs": { "force_flush_interval": 5, "logs_collected": { "files": { "collect_list": [ { "file_path": "/usr/local/apache-zookeeper-3.5.9-bin/logs/zookeeper--server-ip-*.internal.out", "log_group_name": "${ZookeeperMainLogGroup}", "log_stream_name": "${ZookeeperNodeName3}", "timestamp_format": "%Y-%m-%d %H:%M:%S", "timezone": "UTC" } ] } } } } # Invoke amazon-cloudwatch-agent-ctl to restart the AmazonCloudWatchAgent. 03_restart_amazon-cloudwatch-agent: commands: 01_stop_service: command: /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a stop 02_start_service: command: /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json -s # Cfn-hup setting, it is to monitor the change of metadata. # When there is change in the contents of json file in the metadata section, cfn-hup will call cfn-init to restart the AmazonCloudWatchAgent. 01_setupCfnHup: files: '/etc/cfn/cfn-hup.conf': content: !Sub | [main] stack=${AWS::StackId} region=${AWS::Region} interval=1 mode: '000400' owner: root group: root '/etc/cfn/hooks.d/amazon-cloudwatch-agent-auto-reloader.conf': content: !Sub | [cfn-auto-reloader-hook] triggers=post.update path=Resources.ZookeeperInstance3.Metadata.AWS::CloudFormation::Init.02_config-amazon-cloudwatch-agent action=/opt/aws/bin/cfn-init -v --stack ${AWS::StackId} --resource ZookeeperInstance3 --region ${AWS::Region} --configsets UpdateEnvironment runas=root mode: '000400' owner: root group: root "/lib/systemd/system/cfn-hup.service": content: | [Unit] Description=cfn-hup daemon [Service] Type=simple ExecStart=/opt/aws/bin/cfn-hup Restart=always [Install] WantedBy=multi-user.target commands: 01enable_cfn_hup: command: | systemctl enable cfn-hup.service 02start_cfn_hup: command: | systemctl start cfn-hup.service Properties: KeyName: !Ref KeyPairName IamInstanceProfile: !Ref ZookeeperInstanceProfile #ImageId: !FindInMap [AWSRegion2AMI, !Ref 'AWS::Region', HVM64] ImageId: !Ref LatestAmiId InstanceType: !Ref ZookeeperInstanceType SubnetId: !Ref PrivateSubnetID2 SecurityGroupIds: - !Ref ZookeeperServerSecurityGroupID BlockDeviceMappings: - DeviceName: !Ref DeviceName Ebs: VolumeSize: !Ref VolumeSize VolumeType: !Ref VolumeType Iops: !If [UseIops,!Ref Iops,!Ref AWS::NoValue] DeleteOnTermination: true UserData: !Base64 Fn::Sub: | #!/bin/bash yum update -y # This script below is to install AmazonCloudWatchAgent, restart AmazonCloudWatchAgent and tell the result to cloudformation. rpm -Uvh ${CloudWatchAgentURL} /opt/aws/bin/cfn-init -v --stack ${AWS::StackId} --resource ZookeeperInstance3 --region ${AWS::Region} --configsets default sudo mkfs.xfs ${DeviceName} sudo mkdir -p /data/ sudo mount ${DeviceName} /data sudo mkdir -p /data/zookeeper/{data,logs} echo "${DeviceName} /data xfs defaults 0 0" >> /etc/fstab cd /home/ec2-user/ aws s3 cp s3://${QSS3BucketName}/${QSS3KeyPrefix}scripts/zookeeper-install.sh ./ --region ${AWS::Region} aws s3 cp s3://${QSS3BucketName}/${QSS3KeyPrefix}scripts/find-zookeeper-node.py ./ --region ${AWS::Region} chmod +x zookeeper-install.sh ./zookeeper-install.sh ${ZookeeperVersion} > zk.log echo "3" >> /data/zookeeper/data/myid aws ec2 create-tags --region ${AWS::Region} --resources `curl http://169.254.169.254/latest/meta-data/instance-id` --tags Key=${RootStackName}-zookeeper-cluster,Value="server.3=`curl http://169.254.169.254/latest/meta-data/local-ipv4`:2888:3888" flag=1800 while((flag > 0)) do echo `aws ec2 describe-tags --filters Name=key,Values=${RootStackName}-zookeeper-cluster --region ${AWS::Region}` > instancelist count=`awk -v RS="@#$j" '{print gsub(/instance/,"&")}' instancelist` if (( $count >= ${ZookeeperNodeCount} )) then python3 find-zookeeper-node.py instancelist result while read line do echo $line echo $line >> /usr/local/apache-zookeeper-${ZookeeperVersion}-bin/conf/zoo.cfg done < result break fi echo $flag let flag-- sleep 1 done rm -rf /home/ec2-user/openjdk-8u41-b04-linux-x64-14_jan_2020.tar.gz rm -rf /home/ec2-user/apache-zookeeper-${ZookeeperVersion}-bin.tar.gz rm -rf /home/ec2-user/zookeeper-install.sh rm -rf /home/ec2-user/find-zookeeper-node.py rm -rf /home/ec2-user/instancelist rm -rf /home/ec2-user/result echo "/usr/local/apache-zookeeper-${ZookeeperVersion}-bin/bin/zkServer.sh start" > /home/ec2-user/zk-start.sh chmod +x /home/ec2-user/zk-start.sh echo "/home/ec2-user/zk-start.sh" >> /etc/rc.d/rc.local chmod +x /etc/rc.d/rc.local /usr/local/apache-zookeeper-${ZookeeperVersion}-bin/bin/zkServer.sh start /opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackId} --resource ZookeeperInstance3 --region ${AWS::Region} Tags: - Key: Name Value: !Ref ZookeeperNodeName3 CreationPolicy: ResourceSignal: Count: 1 Timeout: "PT120M" Outputs: ZookeeperInstanceID1: Description: ZooKeeper instance Value: !Ref ZookeeperInstance1 ZookeeperPrivateIp1: Value: !GetAtt ZookeeperInstance1.PrivateIp ZookeeperInstanceID2: Description: ZooKeeper instance Value: !Ref ZookeeperInstance2 ZookeeperPrivateIp2: Value: !GetAtt ZookeeperInstance2.PrivateIp ZookeeperInstanceID3: Description: ZooKeeper instance Value: !Ref ZookeeperInstance3 ZookeeperPrivateIp3: Value: !GetAtt ZookeeperInstance3.PrivateIp ZookeeperMainLogGroup: Value: !Ref ZookeeperMainLogGroup ImageId: Value: !Ref LatestAmiId