AWSTemplateFormatVersion: 2010-09-09 Description: Template to create an Aerospike cluster on existing vpc (qs-1rpniq8qr) Metadata: AWSAMIRegionMap: Filters: AEROSPIKEAMI: product-code: ermcxk7wxadxl2w4kemoshqg0 QuickStartDocumentation: EntrypointName: "Parameters for deploying into an existing VPC" Order: Index a LintSpellExclude: - Aerospike - IDs - Linux - Security Group - namespace - Namespace - keypair - sdg - Location - "aerospike.conf" - cloudwatch - aerospike - configuration - .conf - QSS3BucketRegion - QSS3BucketName - QSS3KeyPrefix AWS::CloudFormation::Interface: ParameterGroups: - Label: default: VPC network configuration Parameters: - KeyPairName - VPCID - VPCCIDR - Tenancy - AccessCIDR - PublicSubnet1ID - PublicSubnet2ID - PrivateSubnet1ID - PrivateSubnet2ID - PrivateSubnet3ID - Label: default: Aerospike configuration Parameters: - NamespaceFile - FeatureKeyFile - NumberOfInstances - EnableCloudWatch - InstanceType - EBS - Label: default: Enable monitoring stack Parameters: - MonitoringStack - Label: default: Linux bastion configuration Parameters: - BastionAMIOS - BastionInstanceType - NumBastionHosts - Label: default: AWS Quick Start configuration Parameters: - QSS3BucketName - QSS3KeyPrefix - QSS3BucketRegion ParameterLabels: KeyPairName: default: EC2 key pair VPCID: default: VPC ID VPCCIDR: default: CIDR for the VPC Tenancy: default: Tenancy AccessCIDR: default: Access CIDR PublicSubnet1ID: default: Public subnet 1 ID PublicSubnet2ID: default: Public subnet 2 ID PrivateSubnet1ID: default: Private subnet 1 ID PrivateSubnet2ID: default: Private subnet 2 ID PrivateSubnet3ID: default: Private subnet 3 ID NumberOfInstances: default: Number of instances EnableCloudWatch: default: Enable CloudWatch logging InstanceType: default: Instance type EBS: default: EBS volume size BastionAMIOS: default: Bastion AMI operating system BastionInstanceType: default: Bastion instance type NumBastionHosts: default: Number of bastion hosts NamespaceFile: default: Namespace file FeatureKeyFile: default: Feature key file QSS3BucketName: default: Quick Start S3 bucket name QSS3KeyPrefix: default: Quick Start S3 key prefix QSS3BucketRegion: default: Quick Start S3 bucket Region MonitoringStack: default: Enable monitoring stack Parameters: BastionInstanceType: Default: t3.micro Type: String Description: Type of EC2 instance for the bastion hosts. AllowedValues: - t2.nano - t2.micro - t2.small - t2.medium - t2.large - t3.micro - t3.small - t3.medium - t3.large - m3.large - m3.xlarge - m3.2xlarge - m4.large - m4.xlarge - m4.2xlarge - m4.4xlarge NumBastionHosts: AllowedValues: - '1' - '2' - '3' - '4' Default: '1' Description: Number of bastion hosts to create. Type: String BastionAMIOS: Default: Amazon-Linux2-HVM Description: Linux distribution for the AMI to be used for the bastion instances. Type: String KeyPairName: Description: Name of the key pair to be used to connect to your EC2 instances by using SSH. Type: 'AWS::EC2::KeyPair::KeyName' ConstraintDescription: Specify the name of the key pair that you use to log in. VPCID: Description: ID of the VPC that you're deploying this cluster into. Type: 'AWS::EC2::VPC::Id' VPCCIDR: Description: VPC CIDR block in the format x.x.0.0/16. Type: String Default: 10.0.0.0/16 ConstraintDescription: Must be in format x.x.0.0/16. AllowedPattern: '^([0-9]{1,3}\.){2}([0]{1}.)[0]{1}(\/[16]{2})$' AccessCIDR: AllowedPattern: "^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\\/([0-9]|[1-2][0-9]|3[0-2]))$" Description: "Block of CIDR IP addresses that can access Aerospike. A value of 0.0.0.0/0 allows access from any IP address." Type: String Default: 0.0.0.0/0 PublicSubnet1ID: Type: String Default: "" Description: "ID of public subnet 1." PublicSubnet2ID: Type: String Default: "" Description: "ID of public subnet 2." PrivateSubnet1ID: Type: String Default: "" Description: "ID of private subnet 1." PrivateSubnet2ID: Type: String Default: "" Description: "ID of private subnet 2." PrivateSubnet3ID: Type: String Default: "" Description: "(Optional) ID of private subnet 3." Tenancy: Description: Tenancy of the Aerospike instances. Choose "dedicated" to change the host from shared (default). Type: String Default: default AllowedValues: - default - dedicated NumberOfInstances: Description: Number of instances to provision for the Aerospike cluster. Type: Number Default: '1' MinValue: '1' MaxValue: '15' EnableCloudWatch: Description: >- Choose "yes" to enable CloudWatch logging for the Aerospike cluster, which adds basic Aerospike metrics to Amazon CloudWatch. This incurs CloudWatch expenses. Type: String Default: 'no' AllowedValues: - 'yes' - 'no' InstanceType: Description: Type of EC2 instance to deploy for the Aerospike cluster. Type: String Default: m5d.xlarge AllowedValues: - t2.micro - t2.small - t2.medium - t2.large - t3.micro - t3.small - t3.medium - t3.large - t3.xlarge - t3.2xlarge - m5.large - m5.xlarge - m5.2xlarge - m5.4xlarge - m5.8xlarge - m5.12xlarge - m5.24xlarge - m5.metal - m5d.large - m5d.xlarge - m5d.2xlarge - m5d.4xlarge - m5d.8xlarge - m5d.12xlarge - m5d.16xlarge - m5d.24xlarge - m5d.metal - c5.large - c5.xlarge - c5.2xlarge - c5.4xlarge - c5.9xlarge - c5.18xlarge - c5.24xlarge - c5.metal - c5d.large - c5d.xlarge - c5d.2xlarge - c5d.4xlarge - c5d.9xlarge - c5d.18xlarge - r5.large - r5.xlarge - r5.2xlarge - r5.4xlarge - r5.8xlarge - r5.12xlarge - r5.16xlarge - r5.24xlarge - r5.metal - r5d.large - r5d.xlarge - r5d.2xlarge - r5d.4xlarge - r5d.8xlarge - r5d.12xlarge - r5d.16xlarge - r5d.24xlarge - r5d.metal - i3.large - i3.xlarge - i3.2xlarge - i3.4xlarge - i3.8xlarge - i3.16xlarge - i3.metal - i3en.large - i3en.xlarge - i3en.2xlarge - i3en.3xlarge - i3en.6xlarge - i3en.12xlarge - i3en.24xlarge - i3en.metal EBS: Description: | Size of the EBS SSD volume in GB. The volume attaches under /dev/sdg. Limit of 16000. Enter 0 if you do not want to use EBS. Type: Number Default: '50' MinValue: '0' MaxValue: '16000' NamespaceFile: Description: >- (Optional) Location of your namespace-definition file. Must be publically downloadable. Appends to aerospike.conf. Type: String Default: '' FeatureKeyFile: Description: Base64 encoding of the feature key file. This is the Aerospike key to use for this deployment. Keeping this blank works only for single node cluster. Type: String Default: "" QSS3BucketName: AllowedPattern: "^[0-9a-zA-Z]+([0-9a-zA-Z-]*[0-9a-zA-Z])*$" ConstraintDescription: The Quick Start bucket name can include numbers, lowercase letters, uppercase letters, and hyphens (-). It cannot start or end with a hyphen (-). Default: aws-quickstart Description: Name of the S3 bucket for your copy of the Quick Start assets. Keep the default name unless you are customizing the template. Changing the name updates code references to point to a new Quick Start location. This name can include numbers, lowercase letters, uppercase letters, and hyphens, but do not start or end with a hyphen (-). See https://aws-quickstart.github.io/option1.html. Type: String QSS3BucketRegion: Default: 'us-east-1' Description: 'AWS Region where the Quick Start S3 bucket (QSS3BucketName) is hosted. Keep the default Region unless you are customizing the template. Changing this Region updates code references to point to a new Quick Start location. When using your own bucket, specify the Region. See https://aws-quickstart.github.io/option1.html.' Type: String QSS3KeyPrefix: AllowedPattern: "^[0-9a-zA-Z-/]*$" ConstraintDescription: The Quick Start S3 key prefix can include numbers, lowercase letters, uppercase letters, hyphens (-), and forward slashes (/). The prefix should end with a forward slash (/). Default: quickstart-aerospike/ Description: S3 key prefix that is used to simulate a directory for your copy of the Quick Start assets. Keep the default prefix unless you are customizing the template. Changing this prefix updates code references to point to a new Quick Start location. This prefix can include numbers, lowercase letters, uppercase letters, hyphens (-), and forward slashes (/). End with a forward slash. See https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html and https://aws-quickstart.github.io/option1.html. Type: String MonitoringStack: Type: String AllowedValues: [ "Prometheus + Grafana + asbench", "None" ] Default: "Prometheus + Grafana + asbench" Description: "Enable monitoring stack (prometheus and grafana) and asbench." Conditions: UsingDefaultBucket: !Equals [!Ref QSS3BucketName, 'aws-quickstart'] TrialLicense: !Equals [!Ref FeatureKeyFile, ''] NotUsingEBS: !Equals - !Ref EBS - 0 3AZDeployment: !Not [!Equals [!Ref PrivateSubnet3ID, ""]] 2AZDeployment: !Or - !Not [!Equals [!Ref PrivateSubnet2ID, ""]] - !Not [!Equals [!Ref PrivateSubnet3ID, ""]] MonitoringStackEnabled: !Equals [!Ref MonitoringStack, "Prometheus + Grafana + asbench"] Mappings: AWSAMIRegionMap: AMI: AEROSPIKEAMI: aerospike-amazonlinux2-20220508100353 us-east-1: AEROSPIKEAMI: ami-05f5709c7c092d892 us-east-2: AEROSPIKEAMI: ami-0ded916282885082d us-west-1: AEROSPIKEAMI: ami-0d3b55865039748fd us-west-2: AEROSPIKEAMI: ami-07a9aaaf28175bbef ca-central-1: AEROSPIKEAMI: ami-05251da23e7307086 eu-west-1: AEROSPIKEAMI: ami-05cfcd1145303a472 eu-west-2: AEROSPIKEAMI: ami-0ee5f72a638057fdd eu-west-3: AEROSPIKEAMI: ami-0fd069c2b52922e34 eu-north-1: AEROSPIKEAMI: ami-0b5663b8bbccd2de7 eu-central-1: AEROSPIKEAMI: ami-0a73b33b34f77b5a3 ap-southeast-1: AEROSPIKEAMI: ami-0d2589a63abcac68d ap-southeast-2: AEROSPIKEAMI: ami-0524a4c7c0b9c3940 ap-south-1: AEROSPIKEAMI: ami-06705fdd006924ad6 ap-northeast-1: AEROSPIKEAMI: ami-02c8c5857559980fe ap-northeast-2: AEROSPIKEAMI: ami-0a528e959d8a686f5 ap-northeast-3: AEROSPIKEAMI: ami-0c88f98d79afe460a sa-east-1: AEROSPIKEAMI: ami-0279ec5b44c36973b Resources: BastionStack: Metadata: { cfn-lint: { config: { ignore_checks: [ W9901 ] } } } Type: 'AWS::CloudFormation::Stack' Properties: TemplateURL: !Sub - 'https://${S3Bucket}.s3.${S3Region}.${AWS::URLSuffix}/${QSS3KeyPrefix}submodules/quickstart-linux-bastion/templates/linux-bastion.template' - S3Region: !If [UsingDefaultBucket, !Ref 'AWS::Region', !Ref QSS3BucketRegion] S3Bucket: !If [UsingDefaultBucket, !Sub '${QSS3BucketName}-${AWS::Region}', !Ref QSS3BucketName] Parameters: BastionInstanceType: !Ref BastionInstanceType NumBastionHosts: !Ref NumBastionHosts BastionHostName: 'Bastion host' BastionAMIOS: !Ref BastionAMIOS EnableTCPForwarding: 'true' KeyPairName: !Ref KeyPairName PublicSubnet1ID: !Ref PublicSubnet1ID PublicSubnet2ID: !If [2AZDeployment, !Ref PublicSubnet2ID, !Ref "AWS::NoValue"] QSS3BucketName: !Ref QSS3BucketName QSS3KeyPrefix: !Sub '${QSS3KeyPrefix}submodules/quickstart-linux-bastion/' QSS3BucketRegion: !Ref QSS3BucketRegion RemoteAccessCIDR: !Ref AccessCIDR VPCID: !Ref VPCID PsuedoRandom: Type: 'AWS::EC2::SecurityGroup' Properties: GroupDescription: 'Do NOT USE. Do NOT DELETE. Used for random name suffix for CFN circular dependency workaround.' ClusterRole: Type: 'AWS::IAM::Role' Properties: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Principal: Service: - ec2.amazonaws.com - autoscaling.amazonaws.com Action: - 'sts:AssumeRole' Path: / Policies: - PolicyName: AerospikeClusterPolicy PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - 'ec2:DescribeInstances' - 'ec2:DescribeVpcAttribute' Resource: '*' - PolicyName: root PolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Resource: - '*' Action: - autoscaling:DescribeAutoScalingGroups - autoscaling:DescribeAutoScalingInstances - autoscaling:DescribeLaunchConfigurations - PolicyName: AerospikeCloudWatchPolicy PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: 'cloudwatch:PutMetricData' Resource: '*' - PolicyName: AerospikeSQSPolicy PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - sqs:GetQueueAttributes - sqs:SendMessageBatch - sqs:SendMessage - sqs:SendMessageBatch - sqs:GetQueueUrl Resource: !GetAtt - MigrationSQS - Arn - PolicyName: AerospikeAutoScalingPolicy PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - autoscaling:CompleteLifecycleAction - autoscaling:DescribeAutoScalingGroups - autoscaling:RecordLifecycleActionHeartbeat Resource: !Sub - 'arn:${AWS::Partition}:autoscaling:${AWS::Region}:${AWS::AccountId}:autoScalingGroup:*:autoScalingGroupName/${AutoScalingGroupName}' - AutoScalingGroupName: !Sub 'AerospikeCluster-${PsuedoRandom.GroupId}' - Effect: Allow Action: - ec2:DescribeInstances - ec2:DescribeVpcAttribute Resource: "*" ClusterInstanceProfile: Type: 'AWS::IAM::InstanceProfile' Properties: Path: / Roles: - !Ref ClusterRole MigrationSQS: Type: 'AWS::SQS::Queue' Properties: ReceiveMessageWaitTimeSeconds: 10 MigrationHook: Type: 'AWS::AutoScaling::LifecycleHook' Properties: AutoScalingGroupName: !Ref ClusterGroup LifecycleTransition: 'autoscaling:EC2_INSTANCE_TERMINATING' NotificationTargetARN: !GetAtt - MigrationSQS - Arn RoleARN: !GetAtt - ClusterRole - Arn ClusterGroup: Type: 'AWS::AutoScaling::AutoScalingGroup' Properties: AutoScalingGroupName: !Sub 'AerospikeCluster-${PsuedoRandom.GroupId}' LaunchConfigurationName: !Ref LaunchConfig DesiredCapacity: !Ref NumberOfInstances MinSize: '1' MaxSize: '15' VPCZoneIdentifier: !If - 3AZDeployment - [ !Ref PrivateSubnet1ID, !Ref PrivateSubnet2ID, !Ref PrivateSubnet3ID ] - [ !Ref PrivateSubnet1ID, !Ref PrivateSubnet2ID ] Tags: - Key: StackID Value: !Ref 'AWS::StackId' PropagateAtLaunch: true - Key: Name Value: !Ref 'AWS::StackName' PropagateAtLaunch: true LaunchConfig: Type: 'AWS::AutoScaling::LaunchConfiguration' Metadata: 'AWS::CloudFormation::Init': config: files: /opt/aerospike/cft_scripts/aerospike_init: content: !Sub - | #!/bin/bash # ${Stub} echo ${FeatureKeyFileBase64} | base64 --decode > /etc/aerospike/features.conf rm -rf /etc/aerospike/aerospike.conf cp -r /opt/aerospike/cft_scripts/aerospike_mesh.conf /etc/aerospike mv /etc/aerospike/aerospike_mesh.conf /etc/aerospike/aerospike.conf - Stub: A FeatureKeyFileBase64: !If [ TrialLicense, "IyBnZW5lcmF0ZWQgMjAyMS0wMy0xMCAwNDowNDoyOAoKZmVhdHVyZS1rZXktdmVyc2lvbiAgICAgICAgICAgICAgICAgIDIKc2VyaWFsLW51bWJlciAgICAgICAgICAgICAgICAgICAgICAgIDU2ODk1MzA1OAoKYWNjb3VudC1uYW1lICAgICAgICAgICAgICAgICAgICAgRXZhbHVhdGlvbl9MaWNlbnNlCgphY2NvdW50LUlEICAgICAgICAgICAgICAgICAgICAgICBBZXJvc3Bpa2VfXzQzNTcxMTM3NAoKYXNkYi1jaGFuZ2Utbm90aWZpY2F0aW9uICAgICAgICAgdHJ1ZQphc2RiLWNsdXN0ZXItbm9kZXMtbGltaXQgICAgICAgICAxCmFzZGItY29tcHJlc3Npb24gICAgICAgICAgICAgICAgIHRydWUKYXNkYi1lbmNyeXB0aW9uLWF0LXJlc3QgICAgICAgICAgdHJ1ZQphc2RiLWxkYXAgICAgICAgICAgICAgICAgICAgICAgICB0cnVlCmFzZGItcG1lbSAgICAgICAgICAgICAgICAgICAgICAgIHRydWUKYXNkYi1zdHJvbmctY29uc2lzdGVuY3kgICAgICAgICAgdHJ1ZQptZXNnLWptcy1jb25uZWN0b3IgICAgICAgICAgICAgICB0cnVlCm1lc2cta2Fma2EtY29ubmVjdG9yICAgICAgICAgICAgIHRydWUKcHJlc3RvLWNvbm5lY3RvciAgICAgICAgICAgICAgICAgdHJ1ZQpyYWYtcmVhbHRpbWUtYW5hbHlzaXMtZnJhbWV3b3JrICB0cnVlCgotLS0tLSBTSUdOQVRVUkUgLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCk1FVUNJRXhGN3VSYzBZdURvb3J6ZkNUR2NhdGgrQVB6U0doUi94M21pN0RuRFB6a0FpRUF1cjkzRVcwb2tlNVoKUWZ5c1l4dzZCb0tKRUI1STVlV2xXdDU2UzlxeGNySWsKLS0tLS0gRU5EIE9GIFNJR05BVFVSRSAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQo=", !Ref FeatureKeyFile ] mode: '000744' owner: root group: root /opt/aerospike/cft_scripts/aerospike_cluster: content: !Sub - | #!/bin/bash echo ClusterInstancesScriptStart > /var/log/awsuserdatascript PRIVATE_IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4) CONF=/etc/aerospike/aerospike.conf sed -i "s/port 3000/port 3000\n\t\taccess-address $PRIVATE_IP virtual\n/g" $CONF ###Point to all instances using the mesh-address config option sleep 60 # wait for AWS to provision FILTER=Name=tag-key,Values=StackID PRIVATEIP=$(aws ec2 describe-instances --filter $FILTER Name=tag-value,Values=${AWS::StackId} --output=text --region=${REGION} | grep PRIVATEIPADDRESSES | awk '{print $4}') echo $PRIVATEIP >> /var/log/awsuserdatascript sed -i '/.*mesh-seed-address-port/d' $CONF for i in $PRIVATEIP; do sed -i "/interval/i \\\t\tmesh-seed-address-port $i 3002\n" $CONF done # Check if namespace file in input CODE=$(curl -Is ${NamespaceFile} | head -n 1 | cut -d$' ' -f2) if [ "$CODE" != "200" ]; then echo 'Namespace File not found' >> /var/log/awsuserdatascript else sed -i '/namespace test/,$d' $CONF curl -s ${NamespaceFile} >> $CONF; fi systemctl restart aerospike echo OtherInstancesScriptFinish >> /var/log/awsuserdatascript (crontab -l 2>/dev/null; echo '*/5 * * * * /opt/aerospike/poll_sqs') | crontab - if [[ ${EnableCloudWatch} == "yes" ]]; then (crontab -l 2>/dev/null; echo '*/5 * * * * /opt/aerospike/cloudwatch') | crontab - fi - REGION: !Sub "${AWS::Region}" S3Bucket: !If [UsingDefaultBucket, !Sub '${QSS3BucketName}-${AWS::Region}', !Ref QSS3BucketName] mode: '000744' owner: root group: root /opt/aerospike/cloudwatch: content: !Sub - | #!/bin/bash METRICS=$(asinfo -v stats -l) NAMESPACE=aerospike INSTANCE=$(curl 169.254.169.254/latest/meta-data/instance-id) CLUSTER="${AWS::StackId}" for L in $METRICS; do # The ! in the following is to allow for a literal $ { in !Sub - https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-sub.html LINE=(${!L//=/ }) case ${!LINE[0]} in cluster_integrity) if [[ ${!LINE[1]} != "true" ]]; then INTEGRITY_ERROR=1 else INTEGRITY_ERROR=0 fi ;; total-bytes-memory) TBM=${!LINE[1]} ;; used-bytes-memory) UBM=${!LINE[1]} ;; total-bytes-disk) TBD=${!LINE[1]} ;; used-bytes-disk) UBD=${!LINE[1]} ;; objects) OBJECTS=${!LINE[1]} ;; *) continue ;; esac done FM=$(expr $TBM - $UBM) FD=$(expr $TBD - $UBD) # Submit metrics aws cloudwatch --region ${REGION} put-metric-data --dimensions Cluster=$CLUSTER,Instance=$INSTANCE --namespace $NAMESPACE --value $INTEGRITY_ERROR --metric-name 'Cluster Integrity' aws cloudwatch --region ${REGION} put-metric-data --dimensions Cluster=$CLUSTER,Instance=$INSTANCE --namespace $NAMESPACE --value $FM --metric-name 'Free Memory' --unit 'Bytes' aws cloudwatch --region ${REGION} put-metric-data --dimensions Cluster=$CLUSTER,Instance=$INSTANCE --namespace $NAMESPACE --value $FD --metric-name 'Free Disk' --unit 'Bytes' aws cloudwatch --region ${REGION} put-metric-data --dimensions Cluster=$CLUSTER,Instance=$INSTANCE --namespace $NAMESPACE --value $OBJECTS --metric-name 'Number of Objects' --unit 'Count' - REGION: !Sub "${AWS::Region}" mode: '000744' owner: root group: root /opt/aerospike/poll_sqs: content: !Sub - | #!/bin/bash # This script will prevent autoscaling from terminating # this instance until ASD migrations are completed set -e MYIP=$(curl 169.254.169.254/latest/meta-data/local-ipv4) MYNODE=$(curl 169.254.169.254/latest/meta-data/instance-id) # CFT QUEUE=${MigrationSQS} # CFT HOSTNAMES=$(aws ec2 describe-vpc-attribute --vpc-id=${VPCID} --region=${REGION} --attribute=enableDnsHostnames --output=text | grep ENABLEDNSHOSTNAMES | awk '{print $2}') if [[ "$HOSTNAMES" == "True" ]]; then CLUSTER=$(aws ec2 describe-instances --filter Name=tag-key,Values=StackID Name=tag-value,Values='${AWS::StackId} --output=text --region=${REGION} | grep PRIVATEIPADDRESSES | awk '{print $4}') else CLUSTER=$(aws ec2 describe-instances --filter Name=tag-key,Values=StackID Name=tag-value,Values=${AWS::StackId} --output=text --region=${REGION} | grep PRIVATEIPADDRESSES | awk '{print $3}') fi # Find SQS message with termination message FOUND=false MESSAGE=$(aws sqs receive-message --region ${REGION} --queue-url $QUEUE --wait-time-seconds 10 --visibility-timeout 2 ) BODY=$(echo $MESSAGE | jq '.Messages[0] .Body') RECEIPT=$(echo $MESSAGE | jq --raw-output '.Messages[0].ReceiptHandle') LIFECYCLE=$(eval echo $BODY | jq --raw-output '.LifecycleTransition') INSTANCE=$(eval echo $BODY | jq --raw-output '.EC2InstanceId') if [[ "$LIFECYCLE" == "autoscaling:EC2_INSTANCE_TERMINATING" ]] && [[ "$INSTANCE" == "$MYNODE" ]]; then TOKEN=$(eval echo $BODY | jq --raw-output '.LifecycleActionToken') HOOK=$(eval echo $BODY | jq --raw-output '.LifecycleHookName') ASG=$(eval echo $BODY | jq --raw-output '.AutoScalingGroupName') FOUND=true aws sqs delete-message --region ${REGION} --queue-url $QUEUE --receipt-handle $RECEIPT fi # If not not found, exit if [[ $FOUND == false ]]; then exit 0 fi # stop aerospike /etc/init.d/aerospike stop # give time for cluster to react sleep 10 # Find first node that's not myself for I in $CLUSTER; do if [[ $I == $MYIP ]]; then continue; fi NODE=$I break done # Grab migration info MIGRATIONS=$(asadm -h $NODE -e 'show statistics namespace' | grep migrate-[rt]x-partitions-remaining | awk '{print !$1}') DONE=true # check every node's migration status for STAT in $MIGRATIONS; do if [[ "$STAT" != '0' ]]; then $DONE=false break; fi done # if migrations not done, pause ASG actions. Otherwise, continue autoscaling termination. if [[ $DONE == false ]]; then aws autoscaling record-lifecycle-action-heartbeat --region ${REGION} --lifecycle-action-token $TOKEN --auto-scaling-group-name $ASG --lifecycle-hook-name $HOOK else aws autoscaling complete-lifecycle-action --region ${REGION} --lifecycle-action-token $TOKEN --lifecycle-hook-name $HOOK --auto-scaling-group-name $ASG --lifecycle-action-result CONTINUE fi - REGION: !Sub "${AWS::Region}" mode: '000744' owner: root group: root /opt/aerospike/cft_scripts/aerospike_start: content: !Sub - | #!/bin/bash systemctl start ${Service} - Service: aerospike mode: '000744' owner: root group: root /opt/aerospike/cft_scripts/prometheus_exporter: content: | #!/bin/bash wget https://www.aerospike.com/download/monitoring/aerospike-prometheus-exporter/latest/artifact/tgz -O aerospike-prometheus-exporter.tgz tar -xvzf aerospike-prometheus-exporter.tgz ./usr/bin/aerospike-prometheus-exporter --config ./etc/aerospike-prometheus-exporter/ape.toml > /dev/null 2>&1 & mode: '000744' owner: root group: root /opt/aerospike/cft_scripts/zeroize_disk: content: | #!/bin/bash devices=$(lsblk -dn | awk '{print $1}') echo 'devices : ' $devices for i in $devices; do echo 'zeroizing device' $i sudo blkdiscard -z --length 8MiB /dev/$i done mode: '000744' owner: root group: root commands: 00_aerospike_init: command: /opt/aerospike/cft_scripts/aerospike_init cwd: /opt/aerospike/cft_scripts 01_form_asd_cluster: command: /opt/aerospike/cft_scripts/aerospike_cluster cwd: /opt/aerospike/cft_scripts 02_zeroize_disk: command: /opt/aerospike/cft_scripts/zeroize_disk cwd: /opt/aerospike/cft_scripts 03_aerospike_start: command: /opt/aerospike/cft_scripts/aerospike_start cwd: /opt/aerospike/cft_scripts 04_prometheus_exporter: command: /opt/aerospike/cft_scripts/prometheus_exporter cwd: /opt/aerospike/cft_scripts Properties: InstanceType: !Ref InstanceType KeyName: !Ref KeyPairName BlockDeviceMappings: !If - NotUsingEBS - !Ref 'AWS::NoValue' - - DeviceName: /dev/sdg Ebs: VolumeSize: !Ref EBS VolumeType: gp2 IamInstanceProfile: !Ref ClusterInstanceProfile ImageId: !FindInMap - AWSAMIRegionMap - !Ref 'AWS::Region' - AEROSPIKEAMI AssociatePublicIpAddress: true PlacementTenancy: !Ref Tenancy SecurityGroups: - !GetAtt - InstanceSecurityGroup - GroupId UserData: Fn::Base64: !Sub - | #!/bin/bash -xe yum update -y aws-cfn-bootstrap yum install -y jq /opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource LaunchConfig --region ${Region} /opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackName} --resource ClusterGroup --region ${Region} - Region: !Sub "${AWS::Region}" InstanceSecurityGroup: Type: 'AWS::EC2::SecurityGroup' Properties: GroupDescription: Enable ports to access Aerospike VpcId: !Ref VPCID SecurityGroupIngress: - IpProtocol: tcp FromPort: 3000 ToPort: 3000 CidrIp: !Ref VPCCIDR - IpProtocol: tcp FromPort: 8081 ToPort: 8081 CidrIp: !Ref VPCCIDR - IpProtocol: tcp FromPort: 22 ToPort: 22 SourceSecurityGroupId: !GetAtt - BastionStack - Outputs.BastionSecurityGroupID - IpProtocol: icmp FromPort: -1 ToPort: -1 CidrIp: !Ref VPCCIDR - IpProtocol: tcp FromPort: 9145 ToPort: 9145 CidrIp: !Ref VPCCIDR Tags: - Key: StackID Value: !Ref 'AWS::StackId' InstanceSecurityGroupIngress: Type: 'AWS::EC2::SecurityGroupIngress' Properties: GroupId: !GetAtt - InstanceSecurityGroup - GroupId IpProtocol: tcp FromPort: 3001 ToPort: 3004 SourceSecurityGroupId: !GetAtt - InstanceSecurityGroup - GroupId MonitoringBastionStack: Metadata: { cfn-lint: { config: { ignore_checks: [ W9901 ] } } } Condition: MonitoringStackEnabled Type: 'AWS::CloudFormation::Stack' Properties: TemplateURL: !Sub - 'https://${S3Bucket}.s3.${S3Region}.${AWS::URLSuffix}/${QSS3KeyPrefix}templates/monitoring-bastion-template.yaml' - S3Region: !If [UsingDefaultBucket, !Ref 'AWS::Region', !Ref QSS3BucketRegion] S3Bucket: !If [UsingDefaultBucket, !Sub '${QSS3BucketName}-${AWS::Region}', !Ref QSS3BucketName] Parameters: BastionInstanceType: !Ref BastionInstanceType NumBastionHosts: !Ref NumBastionHosts BastionHostName: 'Monitoring Bastion host' BastionAMIOS: !Ref BastionAMIOS EnableTCPForwarding: 'true' KeyPairName: !Ref KeyPairName PublicSubnet1ID: !Ref PublicSubnet1ID PublicSubnet2ID: !If [2AZDeployment, !Ref PublicSubnet2ID, !Ref "AWS::NoValue"] QSS3BucketName: !Ref QSS3BucketName QSS3KeyPrefix: !Ref QSS3KeyPrefix QSS3BucketRegion: !Ref QSS3BucketRegion RemoteAccessCIDR: !Ref AccessCIDR VPCID: !Ref VPCID AerospikeClusterAutoscalingGroup: !Ref ClusterGroup Outputs: BastionHostIP: Description: IP for bastion host. Value: !GetAtt "BastionStack.Outputs.EIP1" MonitoringBastionHostIP: Description: IP for monitoring bastion host. Value: !If [ MonitoringStackEnabled,!GetAtt "MonitoringBastionStack.Outputs.EIP1", "No monitoring bastion" ] AutoscalingID: Description: The Autoscaling Group ID that is used to deploy your cluster Value: !Ref ClusterGroup Postdeployment: Description: To test your deployment, see the procedure in the deployment guide. Value: https://fwd.aws/Pa8Yw? Rules: TrialSingleNode: RuleCondition: !Equals [ !Ref FeatureKeyFile, '' ] Assertions: - Assert: !Equals [!Ref NumberOfInstances, '1'] AssertDescription: "If you are not providing your own license you may only launch a single node."