AWSTemplateFormatVersion: "2010-09-09" Transform: "AWS::Serverless-2016-10-31" Description: Cribl LogStream free distributed deployment (arm64)(qs-1tgmlqpeq) Parameters: workerCount: Description: (Required) Number of desired Cribl worker nodes. Type: String vpcId: Description: (Required) Existing VPC ID. Type: 'AWS::EC2::VPC::Id' subnetIds: Description: (Required) Select 2 subnet IDs in different Availability Zones. Subnets must be in the VPC specified in VPC ID. Type: 'List' webAccessCidr: Type: String AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/([0-9]|[1-2][0-9]|3[0-2]))$ ConstraintDescription: CIDR block parameter must be in the form "x.x.x.x/x". Description: (Required) CIDR IP range permitted to access the Cribl Stream web console. We recommend you set this value to a trusted IP range. leaderInstanceType: Description: EC2 instance type for the Cribl Stream leader. Type: String Default: c6g.2xlarge AllowedValues: - c6g.large - c6g.xlarge - c6g.2xlarge - c6g.4xlarge - c6gd.large - c6gd.xlarge - c6gd.2xlarge - c6gd.4xlarge - m6g.large - m6g.xlarge - m6g.2xlarge - m6g.4xlarge - m6gd.large - m6gd.xlarge - m6gd.2xlarge - m6gd.4xlarge ConstraintDescription: Must contain a valid instance type. workerInstanceType: Description: EC2 instance type for Cribl Stream workers. Type: String Default: c6g.2xlarge AllowedValues: - c6g.large - c6g.xlarge - c6g.2xlarge - c6g.4xlarge - c6gd.large - c6gd.xlarge - c6gd.2xlarge - c6gd.4xlarge - m6g.large - m6g.xlarge - m6g.2xlarge - m6g.4xlarge - m6gd.large - m6gd.xlarge - m6gd.2xlarge - m6gd.4xlarge ConstraintDescription: Must contain valid instance type. AdditionalPolicies: Default: "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore,arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy" Description: Additional policy ARNs to attach to the IAM role used by the Cribl Stream instances, in a comma-separated list format. Do not remove default policies. Type: CommaDelimitedList Metadata: cfn-lint: config: configure_rules: RuleId: E9007 ignore_checks: - E9007 ignore_reasons: - "A combination of Serverless Transform and metadata not being passed through (yet) means that we need to globally exclude E9007 until an upstream workaround is available." AWS::CloudFormation::Interface: ParameterGroups: - Label: default: Leader instance configuration Parameters: - leaderInstanceType - Label: default: Worker instance configuration Parameters: - workerInstanceType - workerCount - Label: default: Network configuration Parameters: - vpcId - subnetIds - webAccessCidr - Label: default: Advanced settings Parameters: - AdditionalPolicies ParameterLabels: leaderInstanceType: default: Cribl Stream leader EC2 instance type workerInstanceType: default: Cribl Stream workers EC2 instance type workerCount: default: Number of Cribl worker nodes vpcId: default: VPC ID subnetIds: default: Subnet IDs webAccessCidr: default: Web access CIDR AdditionalPolicies: default: Additional policies Rules: SubnetsInVPC: Assertions: - Assert: !EachMemberIn - !ValueOfAll - AWS::EC2::Subnet::Id - VpcId - !RefAll "AWS::EC2::VPC::Id" AssertDescription: All subnets must in the VPC Mappings: RegionMap: ap-northeast-1: ARM64: ami-05713725331d7da87 ap-northeast-2: ARM64: ami-0d0cbd4bb503a8392 ap-south-1: ARM64: ami-08c37554a74085233 ap-southeast-1: ARM64: ami-0001ccc0fa2f4bb58 ap-southeast-2: ARM64: ami-030a28fd0f4425aa0 ca-central-1: ARM64: ami-049e89b91860c0c76 eu-central-1: ARM64: ami-0a54bb2bd5e987705 eu-north-1: ARM64: ami-024f442825535976f eu-west-1: ARM64: ami-0fc642da59aaa3054 eu-west-2: ARM64: ami-03af1bbaa5159b909 eu-west-3: ARM64: ami-00b4befb784285e18 sa-east-1: ARM64: ami-06f2bd6f4bc75e1e4 us-east-1: ARM64: ami-0d674664089a2cc30 us-east-2: ARM64: ami-0e691c58d8cbfddfd us-west-1: ARM64: ami-0d277970fb2218635 us-west-2: ARM64: ami-0be98fa4b8a367d2b Resources: ec2leaderSecurityGroup: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: Cribl LogStream Access VpcId: !Ref vpcId SecurityGroupIngress: - IpProtocol: tcp FromPort: 9000 ToPort: 9000 CidrIp: !Ref webAccessCidr Description: UI access to the LogStream leader instance - IpProtocol: tcp FromPort: 4200 ToPort: 4200 CidrIp: !Ref webAccessCidr #SourceSecurityGroupId: !GetAtt ec2WorkerSecurityGroup.GroupId Description: Cluster communications between leader and workers SecurityGroupEgress: - IpProtocol: "-1" CidrIp: 0.0.0.0/0 Description: Egress access LeaderLoadBalancerExternal: Type: AWS::ElasticLoadBalancingV2::LoadBalancer Properties: Type: application Scheme: internet-facing IpAddressType: ipv4 SecurityGroups: - !Ref ec2leaderSecurityGroup Subnets: - !Select [0, !Ref 'subnetIds'] - !Select [1, !Ref 'subnetIds'] LeaderLoadBalancerInternal: Type: AWS::ElasticLoadBalancingV2::LoadBalancer Properties: Type: network Scheme: internal IpAddressType: ipv4 Subnets: - !Select [0, !Ref 'subnetIds'] - !Select [1, !Ref 'subnetIds'] LeaderCriblTargetGroup: Type: AWS::ElasticLoadBalancingV2::TargetGroup DependsOn: LeaderLoadBalancerExternal Properties: HealthCheckPort: '9000' HealthCheckProtocol: HTTP Port: 9000 Protocol: HTTP TargetType: instance VpcId: !Ref vpcId CriblLeaderWebListener: Type: AWS::ElasticLoadBalancingV2::Listener Properties: DefaultActions: - Type: forward TargetGroupArn: !Ref LeaderCriblTargetGroup LoadBalancerArn: !Ref LeaderLoadBalancerExternal Port: 9000 Protocol: HTTP WorkerCriblTargetGroup: Type: AWS::ElasticLoadBalancingV2::TargetGroup DependsOn: LeaderLoadBalancerInternal Properties: HealthCheckPort: '4200' HealthCheckProtocol: TCP Port: 4200 Protocol: TCP TargetType: instance VpcId: !Ref vpcId CriblWorkerAdminListener: Type: AWS::ElasticLoadBalancingV2::Listener Properties: DefaultActions: - Type: forward TargetGroupArn: !Ref WorkerCriblTargetGroup LoadBalancerArn: !Ref LeaderLoadBalancerInternal Port: 4200 Protocol: TCP WorkerCriblTCPJSONTargetGroup: Type: AWS::ElasticLoadBalancingV2::TargetGroup DependsOn: LeaderLoadBalancerInternal Properties: HealthCheckPort: '10001' HealthCheckProtocol: TCP Port: 10001 Protocol: TCP TargetType: instance VpcId: !Ref vpcId CriblWorkerTCPJSONListener: Type: AWS::ElasticLoadBalancingV2::Listener Properties: DefaultActions: - Type: forward TargetGroupArn: !Ref WorkerCriblTCPJSONTargetGroup LoadBalancerArn: !Ref LeaderLoadBalancerInternal Port: 10001 Protocol: TCP WorkerCriblTCPRAWTargetGroup: Type: AWS::ElasticLoadBalancingV2::TargetGroup DependsOn: LeaderLoadBalancerInternal Properties: HealthCheckPort: '7777' HealthCheckProtocol: TCP Port: 7777 Protocol: TCP TargetType: instance VpcId: !Ref vpcId CriblWorkerTCPRAWListener: Type: AWS::ElasticLoadBalancingV2::Listener Properties: DefaultActions: - Type: forward TargetGroupArn: !Ref WorkerCriblTCPRAWTargetGroup LoadBalancerArn: !Ref LeaderLoadBalancerInternal Port: 7777 Protocol: TCP CriblLeaderWebListenerRule1: Type: AWS::ElasticLoadBalancingV2::ListenerRule Properties: Actions: - Type: forward TargetGroupArn: !Ref LeaderCriblTargetGroup Conditions: - Field: http-request-method HttpRequestMethodConfig: Values: - GET_OR_HEAD ListenerArn: !Ref CriblLeaderWebListener Priority: 1 ec2WorkerSecurityGroup: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: Cribl LogStream Ingress and Egress for Ports VpcId: !Ref vpcId SecurityGroupIngress: - IpProtocol: tcp FromPort: 10001 ToPort: 10001 CidrIp: !Ref webAccessCidr Description: TCP-raw Data Collection Ingress - IpProtocol: tcp FromPort: 7777 ToPort: 7777 CidrIp: !Ref webAccessCidr Description: TCP-json Data Collection Ingress SecurityGroupEgress: - IpProtocol: "-1" CidrIp: 0.0.0.0/0 Description: Egress access PrivateLinklInterfaceEndpoint: Type: AWS::EC2::VPCEndpointService Properties: NetworkLoadBalancerArns: - !Ref LeaderLoadBalancerInternal AcceptanceRequired: false s3DefaultDestinationBucket: Type: AWS::S3::Bucket Properties: PublicAccessBlockConfiguration: BlockPublicAcls: true BlockPublicPolicy: true IgnorePublicAcls: true RestrictPublicBuckets: true Tags: - Key: Name Value: Cribl LogStream default destination bucket LogstreamRole: Type: AWS::IAM::Role Properties: Path: !Sub "/logstream/${AWS::StackName}/" Description: Cribl LogStream IAM role AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Principal: Service: - ec2.amazonaws.com Action: - sts:AssumeRole ManagedPolicyArns: !Ref AdditionalPolicies Policies: - PolicyName: S3Destinations PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - s3:PutObject - s3:GetObject - s3:ListBucket - s3:GetBucketLocation Resource: - !Sub ${s3DefaultDestinationBucket.Arn} - !Sub ${s3DefaultDestinationBucket.Arn}/* - PolicyName: S3Sources PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - s3:GetObject - s3:GetBucketLocation Resource: - !Sub ${s3DefaultDestinationBucket.Arn} - !Sub ${s3DefaultDestinationBucket.Arn}/* - PolicyName: KinesisSources PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - kinesis:GetRecords - kinesis:GetShardIterator - kinesis:ListShards NotResource: "*" - PolicyName: SecretsManagerRead PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - secretsmanager:GetSecretValue Resource: - !Ref clusterAuthToken Tags: - Key: Name Value: Cribl LogStream default IAM role LogstreamInstanceProfile: Type: AWS::IAM::InstanceProfile Properties: Path: !Sub "/logstream/${AWS::StackName}/" Roles: - !Ref LogstreamRole ec2LeaderAutoScalingGroup: Type: AWS::AutoScaling::AutoScalingGroup Properties: VPCZoneIdentifier: - !Select [0, !Ref 'subnetIds'] - !Select [1, !Ref 'subnetIds'] TargetGroupARNs: - !Ref LeaderCriblTargetGroup - !Ref WorkerCriblTargetGroup DesiredCapacity: "1" MaxSize: "1" MinSize: "0" HealthCheckType: EC2 LaunchTemplate: LaunchTemplateId: !Ref ec2leaderlaunchTemplate Version: !GetAtt ec2leaderlaunchTemplate.LatestVersionNumber Tags: - Key: Name Value: Cribl LogStream Leader Nodes ASG PropagateAtLaunch: false ec2LeaderAutoScalingGroupLifecycleHook: Type: AWS::AutoScaling::LifecycleHook Properties: AutoScalingGroupName: !Ref ec2LeaderAutoScalingGroup DefaultResult: ABANDON HeartbeatTimeout: 3600 LifecycleHookName: cribl-logstream-leader-shutdown LifecycleTransition: "autoscaling:EC2_INSTANCE_TERMINATING" ec2leaderlaunchTemplate: Type: AWS::EC2::LaunchTemplate Metadata: cfn-lint: config: ignore_checks: - E9101 Properties: LaunchTemplateData: InstanceInitiatedShutdownBehavior: terminate ImageId: !FindInMap [RegionMap, !Ref "AWS::Region", ARM64] InstanceType: !Ref leaderInstanceType IamInstanceProfile: Arn: !GetAtt LogstreamInstanceProfile.Arn SecurityGroupIds: - !Ref ec2leaderSecurityGroup UserData: !Base64 Fn::Sub: - | #cloud-config runcmd: - export AWS_DEFAULT_REGION=${Region} - /usr/local/bin/configure_logstream.sh -m master -b ${s3DefaultDestinationBucket} -u ${clusterAuthToken} - sleep 10 - cloud-init query -f "$(cat /opt/cribl_build/users.json.j2)" > /opt/cribl/local/cribl/auth/users.json - chown -R cribl:cribl /opt/cribl - clusterAuthToken: !Ref clusterAuthToken Region: !Ref AWS::Region TagSpecifications: - ResourceType: instance Tags: - Key: Name Value: Cribl LogStream Leader Node leaderShutdownEventRule: Type: AWS::Events::Rule Properties: Description: "Cribl LogStream Leader Shutdown Event" EventPattern: source: - "aws.autoscaling" detail-type: - "EC2 Instance-terminate Lifecycle Action" detail: AutoScalingGroupName: - !Ref ec2LeaderAutoScalingGroup State: "ENABLED" Targets: - Arn: !GetAtt functionLeaderShutdown.Arn Id: "WorkerShutdownFunction" lamdbaPermissionLeaderShutdownEventRule: Type: AWS::Lambda::Permission Properties: FunctionName: !Ref functionLeaderShutdown Action: "lambda:InvokeFunction" Principal: "events.amazonaws.com" SourceArn: !GetAtt workerShutdownEventRule.Arn functionLeaderShutdown: Type: AWS::Serverless::Function Properties: Description: Cribl LogStream Leader Shutdown InlineCode: | import boto3 import logging import json import time import os ssm = boto3.client('ssm') autoscaling = boto3.client('autoscaling') # Logging log = logging.getLogger() log.setLevel(logging.DEBUG) def handler(event, context): log.debug("Received event {}".format(json.dumps(event))) command = ssm.send_command( InstanceIds=[event['detail']['EC2InstanceId'],], DocumentName='AWS-RunShellScript', Comment='Gracefully terminate Cribl LogStream leader node', Parameters={"commands":["systemctl disable cribl","systemctl daemon-reload","systemctl stop cribl"],"workingDirectory":[""],"executionTimeout":["3600"]}, CloudWatchOutputConfig={ 'CloudWatchOutputEnabled': True } ) command_id = command['Command']['CommandId'] for x in range(0,59): time.sleep(10) response = ssm.get_command_invocation( CommandId=command_id, InstanceId=event['detail']['EC2InstanceId'], ) if response.get('Status') == 'Success': break else: continue autoscaling.complete_lifecycle_action( LifecycleHookName=event['detail']['LifecycleHookName'], AutoScalingGroupName=event['detail']['AutoScalingGroupName'], LifecycleActionToken=event['detail']['LifecycleActionToken'], LifecycleActionResult='CONTINUE', InstanceId=event['detail']['EC2InstanceId'] ) Policies: - Statement: - Action: ["ssm:SendCommand"] Effect: Allow Resource: "*" Condition: StringEquals: "aws:ResourceTag/aws:ec2launchtemplate:id": !Ref ec2leaderlaunchTemplate - Action: ["ssm:SendCommand"] Effect: Allow Resource: !Sub "arn:${AWS::Partition}:ssm:${AWS::Region}::document/AWS-RunShellScript" - Action: ["autoscaling:CompleteLifecycleAction"] Effect: Allow Resource: !Sub "arn:${AWS::Partition}:autoscaling:${AWS::Region}:${AWS::AccountId}:autoScalingGroup:*:autoScalingGroupName/${ec2WorkersAutoScalingGroup}" - Action: ["ssm:GetCommandInvocation"] Effect: Allow Resource: "*" Runtime: python3.7 Timeout: 600 Handler: index.handler clusterAuthToken: Type: 'AWS::SecretsManager::Secret' Properties: Description: "Cribl Logstream Cluster Auth Token" GenerateSecretString: PasswordLength: 30 ExcludePunctuation: true ec2WorkersAutoScalingGroup: Type: AWS::AutoScaling::AutoScalingGroup Properties: DesiredCapacity: !Ref workerCount MaxSize: !Ref workerCount MinSize: !Ref workerCount HealthCheckType: EC2 LaunchTemplate: LaunchTemplateId: !Ref ec2WorkerslaunchTemplate Version: !GetAtt ec2WorkerslaunchTemplate.LatestVersionNumber VPCZoneIdentifier: - !Select [0, !Ref subnetIds] - !Select [1, !Ref subnetIds] Tags: - Key: Name Value: Cribl LogStream Worker Nodes ASG PropagateAtLaunch: false ec2WorkersAutoScalingGroupLifecycleHook: Type: AWS::AutoScaling::LifecycleHook Properties: AutoScalingGroupName: !Ref ec2WorkersAutoScalingGroup DefaultResult: ABANDON HeartbeatTimeout: 3600 LifecycleHookName: cribl-logstream-worker-shutdown LifecycleTransition: "autoscaling:EC2_INSTANCE_TERMINATING" ec2WorkerslaunchTemplate: Type: AWS::EC2::LaunchTemplate Properties: LaunchTemplateData: InstanceInitiatedShutdownBehavior: terminate ImageId: !FindInMap [RegionMap, !Ref "AWS::Region", ARM64] InstanceType: !Ref workerInstanceType IamInstanceProfile: Arn: !GetAtt LogstreamInstanceProfile.Arn SecurityGroupIds: - !Ref ec2WorkerSecurityGroup UserData: !Base64 Fn::Sub: - | #cloud-config runcmd: - export AWS_DEFAULT_REGION=${Region} - /usr/local/bin/configure_logstream.sh -m worker -u ${clusterAuthToken} -H ${LeaderLoadBalancerInternal.DNSName} -p 4200 - sed -i 's/false/true/g' /opt/cribl/local/_system/instance.yml - sleep 60 - systemctl restart cribl - clusterAuthToken: !Ref clusterAuthToken Region: !Ref AWS::Region TagSpecifications: - ResourceType: instance Tags: - Key: Name Value: Cribl LogStream Worker Instance workerShutdownEventRule: Type: AWS::Events::Rule Properties: Description: "Cribl LogStream Worker Shutdown Event" EventPattern: source: - "aws.autoscaling" detail-type: - "EC2 Instance-terminate Lifecycle Action" detail: AutoScalingGroupName: - !Ref ec2WorkersAutoScalingGroup State: "ENABLED" Targets: - Arn: !GetAtt functionWorkerShutdown.Arn Id: "WorkerShutdownFunction" lamdbaPermissionWorkerShutdownEventRule: Type: AWS::Lambda::Permission Properties: FunctionName: !Ref functionWorkerShutdown Action: "lambda:InvokeFunction" Principal: "events.amazonaws.com" SourceArn: !GetAtt workerShutdownEventRule.Arn functionWorkerShutdown: Type: AWS::Serverless::Function Properties: Description: Cribl LogStream Worker Shutdown InlineCode: | import boto3 import logging import json import time import os ssm = boto3.client('ssm') autoscaling = boto3.client('autoscaling') # Logging log = logging.getLogger() log.setLevel(logging.DEBUG) def handler(event, context): log.debug("Received event {}".format(json.dumps(event))) command = ssm.send_command( InstanceIds=[event['detail']['EC2InstanceId'],], DocumentName='AWS-RunShellScript', Comment='Gracefully terminate Cribl LogStream worker node', Parameters={"commands":["systemctl disable cribl","systemctl daemon-reload","systemctl stop cribl"],"workingDirectory":[""],"executionTimeout":["3600"]}, CloudWatchOutputConfig={ 'CloudWatchOutputEnabled': True } ) command_id = command['Command']['CommandId'] for x in range(0,59): time.sleep(10) response = ssm.get_command_invocation( CommandId=command_id, InstanceId=event['detail']['EC2InstanceId'], ) if response.get('Status') == 'Success': break else: continue autoscaling.complete_lifecycle_action( LifecycleHookName=event['detail']['LifecycleHookName'], AutoScalingGroupName=event['detail']['AutoScalingGroupName'], LifecycleActionToken=event['detail']['LifecycleActionToken'], LifecycleActionResult='CONTINUE', InstanceId=event['detail']['EC2InstanceId'] ) Policies: - Statement: - Action: ["ssm:SendCommand"] Effect: Allow Resource: "*" Condition: StringEquals: "aws:ResourceTag/aws:ec2launchtemplate:id": !Ref ec2WorkerslaunchTemplate - Action: ["ssm:SendCommand"] Effect: Allow Resource: !Sub "arn:${AWS::Partition}:ssm:${AWS::Region}::document/AWS-RunShellScript" - Action: ["autoscaling:CompleteLifecycleAction"] Effect: Allow Resource: !Sub "arn:${AWS::Partition}:autoscaling:${AWS::Region}:${AWS::AccountId}:autoScalingGroup:*:autoScalingGroupName/${ec2WorkersAutoScalingGroup}" - Action: ["ssm:GetCommandInvocation"] Effect: Allow Resource: "*" Runtime: python3.7 Timeout: 600 Handler: index.handler Outputs: logstreamWebUrlPublic: Value: !Sub http://${LeaderLoadBalancerExternal.DNSName}:9000/login Description: Cribl Stream web console URL. logstreamWebAccessCreds: Value: "admin / Leader EC2 Instance ID" Description: Default web access credentials. Postdeployment: Description: See the deployment guide for postdeployment steps. Value: https://fwd.aws/9Jq7J?