AWSTemplateFormatVersion: "2010-09-09" Description: Nested stack of Cisco Secure Firewall Cloud Native Quickstart entrypoints that deployes all CSFCN resources (qs-1s3afab27). Metadata: cfn-lint: { config: { ignore_checks: [ W9002, W9003, W9004, W9006 ] } } Parameters: QSS3BucketName: AllowedPattern: ^[0-9a-zA-Z]+([0-9a-zA-Z-]*[0-9a-zA-Z])*$ ConstraintDescription: Quick Start bucket name can include numbers, lowercase letters, uppercase letters, and hyphens (-). It cannot start or end with a hyphen (-). Default: aws-quickstart Description: S3 bucket name for the Quick Start assets. This string can include numbers, lowercase letters, uppercase letters, and hyphens (-). It cannot start or end with a hyphen (-). Type: String QSS3KeyPrefix: AllowedPattern: ^[0-9a-zA-Z-/.]*$ ConstraintDescription: Quick Start key prefix can include numbers, lowercase letters, uppercase letters, hyphens (-), periods (.) and forward slashes (/). Default: quickstart-cisco-secure-firewall-cloud-native/ Description: The S3 key prefix for the Quick Start assets. Quick Start key prefix can include numbers, lowercase letters, uppercase letters, hyphens (-), periods (.) and forward slashes (/). Type: String QSS3BucketRegion: Default: 'us-east-1' Description: AWS Region where the Quick Start S3 bucket (QSS3BucketName) is hosted. If you use your own bucket, you must specify this value. Type: String VPCID: Type: "AWS::EC2::VPC::Id" Description: ID of your existing VPC (e.g., vpc-0343606e). PublicSubnet1ID: Type: "AWS::EC2::Subnet::Id" Description: ID of the public DMZ subnet 1 (CNFW default 1), located in Availability Zone 1. PrivateSubnet1ID: Type: "AWS::EC2::Subnet::Id" Description: ID of the private subnet 1, located in Availability Zone 1. Must have "kubernetes.io/role/internal-elb" tag attached. PublicSubnet2ID: Type: "AWS::EC2::Subnet::Id" Description: ID of the public DMZ subnet 2 (CNFW default 2), located in Availability Zone 2. PrivateSubnet2ID: Type: "AWS::EC2::Subnet::Id" Description: ID of the private subnet 2, located in Availability Zone 2. Must have "kubernetes.io/role/internal-elb" tag attached. PublicSubnetRouteTableID: Description: ID of VPC public subnet route table (e.g., rtb-0b9ce78da592f11e0). Type: String CNFWOutsideSubnet1CIDR: AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(1[6-9]|2[0-8]))$ ConstraintDescription: CIDR block parameter must be in the form x.x.x.x/16–28. Description: CIDR block for the CNFW outside subnet 1, located in Availability Zone 1. Type: String CNFWInsideSubnet1CIDR: AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(1[6-9]|2[0-8]))$ ConstraintDescription: CIDR block parameter must be in the form x.x.x.x/16–28. Description: CIDR block for the CNFW inside subnet 1, located in Availability Zone 1. Type: String CNFWOutsideSubnet2CIDR: AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(1[6-9]|2[0-8]))$ ConstraintDescription: CIDR block parameter must be in the form x.x.x.x/16–28. Description: CIDR block for the CNFW outside subnet 2, located in Availability Zone 2. Type: String CNFWInsideSubnet2CIDR: AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(1[6-9]|2[0-8]))$ ConstraintDescription: CIDR block parameter must be in the form x.x.x.x/16–28. Description: CIDR block for the CNFW inside subnet 2, located in Availability Zone 2. Type: String EKSClusterName: Type: String MaxLength: 26 # Issue https://github.com/aws-quickstart/quickstart-amazon-eks/issues/314 KubernetesVersion: Type: String Default: 1.19 AllowedValues: - 1.19 Description: The desired Kubernetes version for your cluster. KeyPairName: Description: Name of an existing key pair, which allows you to securely connect to your instance after it launches. Type: AWS::EC2::KeyPair::KeyName AdditionalEKSAdminUserArn: Type: String Default: "" Description: "(Optional) IAM user Amazon Resource Name (ARN) to be granted administrative access to the EKS cluster." AdditionalEKSAdminRoleArn: Type: String Default: "" Description: "(Optional) IAM role Amazon Resource Name (ARN) to be granted administrative access to the EKS cluster." EKSClusterLoggingTypes: Type: String Default: "" AllowedPattern: ^$|^\s*(api|audit|authenticator|controllerManager|scheduler)\s*(,\s*(api|audit|authenticator|controllerManager|scheduler)\s*)*$ ConstraintDescription: Comma separated list of allowed values - api, audit, authenticator, controllerManager and scheduler. Description: | Cluster control plane logging options (api, audit, authenticator, controllerManager and scheduler); comma separated. ControllerConfigurationTier: Description: Instance tier to be used by controller workers. Default: "vCPU4-c5.2xlarge" Type: String AllowedValues: [ "vCPU4-m5.xlarge", "vCPU4-c5.2xlarge" ] ConstraintDescription: must specify vCPU4-m5.xlarge or vCPU4-c5.2xlarge EnforcerConfigurationTier: Description: Instance tier to be used by enforcer workers. Default: "vCPU4-c5.2xlarge" Type: String AllowedValues: [ "vCPU4-m5.xlarge", "vCPU4-c5.2xlarge" ] ConstraintDescription: must specify vCPU4-m5.xlarge or vCPU4-c5.2xlarge ControlNodeGroupMaxSize: Type: Number Default: 2 Description: The maximum number of control plane worker nodes. ControlNodeGroupDesiredSize: Type: Number Default: 1 Description: The desired number of control plane worker nodes. StorageType: Type: String AllowedValues: [ local, efs ] Default: local Description: The type of a storage used to keep logs and deployments files. LoadBalancerController: Type: String AllowedValues: [ Enabled, Disabled ] Default: Enabled Description: Enable or Disable load balancer controller. EnforcerNodeGroupMaxSize: Type: Number Default: 5 Description: The maximum number of data plane worker nodes. EnforcerNodeGroupDesiredSize: Type: Number Default: 1 Description: The desired number of data plane worker nodes. EIPAttachmentMode: Type: String Default: "outside" AllowedValues: - none - outside - inside - both Description: CNFW interfaces to attach Elastic IPs. EnforcerCacheType: Type: String Default: "none" AllowedValues: - none - elasticache Description: Type of External Cache to use for CNFW. EnforcerCacheNodeType: Type: String Default: "cache.m5.large" AllowedValues: [ cache.m5.xlarge, cache.m5.large ] Description: EC2 instance type to use as Cache node, check region availaibility before selection. EnforcerCacheAuthToken: Type: String NoEcho: true MaxLength: 128 Default: "" AllowedPattern: "([a-zA-Z0-9!&#$^><-]{16,128})?" ConstraintDescription: Must be blank or contain at least 16 and up to 128 alphanumeric or allowed special characters. Description: | Blank or 16-128 alphanumeric characters. Only permitted special characters are !, &, #, $, ^, <, >, and -. Takes effect only if in-transit encryption is enabled. EnforcerCacheStorageKey: Type: String NoEcho: true MaxLength: 64 Default: "" AllowedPattern: "^([A-F0-9]{64})?$" ConstraintDescription: Must be a string of length 64 consisting of uppercase hexadecimal characters (A-F,0-9). Description: | User-provided AES-256 key that is 64 hexadecimal characters (A-F,0-9) used to encrypt sensitive cache data before it leaves CNFW. This can be generated with the command "openssl enc -aes-256-cbc -k password -P -md sha256" or any other AES key generation methods. Required when 'elasticache' cache type is specified. EnforcerCacheTransitEncryption: Type: String AllowedValues: [ Enabled, Disabled ] Default: Enabled Description: Cannot be changed after creation, update requires replacement. EnforcerCachePort: Type: Number Default: 6379 MinValue: 1 MaxValue: 65535 Description: Port number to use for communicating with Redis server. SecureFirewallDataplane: Type: String AllowedValues: [ Enabled, Disabled ] Default: Enabled Description: This is enabled by default for a full deployment with control plane and data plane components. To deploy control plane components only (for multi-tenancy), disable this setting. FirewallVersion: Type: String Default: v1.1.1 AllowedPattern: ".+" Description: Helm chart version of the product. SystemNamespace: Type: String Default: sfcn-system AllowedPattern: ".+" Description: The namespace where product components are installed. SmartLicenseToken: Type: String MaxLicenseCount: Type: Number EnforcerServiceRoles: Type: String Default: default AllowedPattern: "[a-zA-Z0-9-]{3,15}(,[a-zA-Z0-9-]{3,15})*" AllowedValues: [ "default", "default,vpnredirector" ] ConstraintDescription: | Role should be alphanumeric and can contain the special character, hyphen (-). Role must be at least 3 characters and upto 15 characters long. Description: | The service role of the data plane node. Leave the setting as "default" if you selected 1 data plane node. Each additional service role requires a corresponding data plane node. EnforcerAutoscaling: Type: String AllowedValues: [ Enabled, Disabled ] Default: Disabled Description: Enable the default Enforcer autoscaling. Mappings: ConfigurationTierMap: "vCPU4-m5.xlarge": instanceType: m5.xlarge hugePagesSize: "128" "vCPU4-c5.2xlarge": instanceType: c5.2xlarge hugePagesSize: "128" Rules: EnforcerElasticacheRule: RuleCondition: !Equals [ !Ref EnforcerCacheType, elasticache ] Assertions: - Assert: !Equals [ !Ref SecureFirewallDataplane, Enabled ] AssertDescription: Cache type can be set to "elasticache" only if firewall data plane is "Enabled". - Assert: !Not [ !Equals [ !Ref EnforcerCacheStorageKey, "" ] ] AssertDescription: Cache storage key must not be empty when cache type is set to "elasticache". Conditions: UsingDefaultBucket: !Equals [ !Ref QSS3BucketName, 'aws-quickstart' ] EnableEFS: !Equals [ !Ref StorageType, "efs" ] EnableLB: !Equals [ !Ref LoadBalancerController, "Enabled" ] EnforcerCacheTypeIsElastiCache: !Equals [ !Ref EnforcerCacheType, elasticache ] Resources: CloudFormationKubernetesVPCRoleExists: Type: Custom::CliQuery Properties: ServiceToken: !Sub 'arn:${AWS::Partition}:lambda:${AWS::Region}:${AWS::AccountId}:function:eks-quickstart-ResourceReader' AwsCliCommand: "iam list-roles --query 'Roles[?RoleName==`CloudFormation-Kubernetes-VPC`].RoleName | {RoleName: [0]}'" IdField: 'RoleName' IamStack: Type: "AWS::CloudFormation::Stack" Properties: TemplateURL: !Sub - 'https://${S3Bucket}.s3.${S3Region}.${AWS::URLSuffix}/${QSS3KeyPrefix}submodules/quickstart-amazon-eks/templates/amazon-eks-iam.template.yaml' - S3Region: !If [ UsingDefaultBucket, !Ref 'AWS::Region', !Ref QSS3BucketRegion ] S3Bucket: !If [ UsingDefaultBucket, !Sub '${QSS3BucketName}-${AWS::Region}', !Ref QSS3BucketName ] Parameters: QSS3BucketName: !Ref QSS3BucketName CreateBastionRole: "Disabled" BastionIAMRoleName: "" CloudFormationKubernetesVPCRoleExists: !Ref CloudFormationKubernetesVPCRoleExists VPCCIDR: Type: Custom::CliQuery Properties: ServiceToken: !Sub 'arn:${AWS::Partition}:lambda:${AWS::Region}:${AWS::AccountId}:function:eks-quickstart-ResourceReader' AwsCliCommand: !Sub "ec2 describe-vpcs --vpc-id ${VPCID} --query 'Vpcs[0].{CidrBlock: CidrBlock}'" IdField: 'CidrBlock' UbuntuAMI: Type: Custom::CliQuery Properties: ServiceToken: !Sub 'arn:${AWS::Partition}:lambda:${AWS::Region}:${AWS::AccountId}:function:eks-quickstart-ResourceReader' AwsCliCommand: !Sub "ec2 describe-images --filters 'Name=name,Values=ubuntu-eks/k8s_${KubernetesVersion}/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20210423.1' --query '{Value: Images[*].[ImageId][0][0]}'" IdField: 'Value' # Inside/Outside subnets are created for the first two Availability Zones of the current region. # Each subnet uses a /20 netmask (4096 IPv4 addresses). # Creating subnets for all AZs dynamically is not supported by vanilla CloudFormation templates. # # If you are wondering why `kubernetes.io/role/elb` and similar tags are assigned for the subnets, refer to: # https://aws.amazon.com/ru/premiumsupport/knowledge-center/eks-vpc-subnet-discovery/ InsideSubnetsRouteTable: Type: AWS::EC2::RouteTable Properties: VpcId: !Ref VPCID Tags: - Key: Name Value: Inside Subnets - Key: Network Value: Private OutsideSubnetAZ1: Type: AWS::EC2::Subnet Properties: VpcId: !Ref VPCID CidrBlock: !Ref CNFWOutsideSubnet1CIDR MapPublicIpOnLaunch: true AvailabilityZone: !Select [ 0, !GetAZs "" ] Tags: - Key: Name Value: !Sub [ "${pref}_outside_${az}", { pref: !Ref EKSClusterName, az: !Select [ 0, !GetAZs "" ] } ] - Key: !Sub "kubernetes.io/cluster/${EKSClusterName}" Value: "shared" - Key: kubernetes.io/role/elb Value: "1" OutsideSubnetAZ1RouteTableAssociation: Type: AWS::EC2::SubnetRouteTableAssociation Properties: SubnetId: !Ref OutsideSubnetAZ1 RouteTableId: !Ref PublicSubnetRouteTableID InsideSubnetAZ1: Type: AWS::EC2::Subnet Properties: VpcId: !Ref VPCID CidrBlock: !Ref CNFWInsideSubnet1CIDR AvailabilityZone: !Select [ 0, !GetAZs "" ] Tags: - Key: Name Value: !Sub [ "${pref}_inside_${az}", { pref: !Ref EKSClusterName, az: !Select [ 0, !GetAZs "" ] } ] - Key: !Sub "kubernetes.io/cluster/${EKSClusterName}" Value: "shared" - Key: kubernetes.io/role/internal-elb Value: "1" InsideSubnetAZ1RouteTableAssociation: Type: AWS::EC2::SubnetRouteTableAssociation Properties: SubnetId: !Ref InsideSubnetAZ1 RouteTableId: !Ref InsideSubnetsRouteTable OutsideSubnetAZ2: Type: AWS::EC2::Subnet Properties: VpcId: !Ref VPCID CidrBlock: !Ref CNFWOutsideSubnet2CIDR MapPublicIpOnLaunch: true AvailabilityZone: !Select [ 1, !GetAZs "" ] Tags: - Key: Name Value: !Sub [ "${pref}_outside_${az}", { pref: !Ref EKSClusterName, az: !Select [ 1, !GetAZs "" ] } ] - Key: !Sub "kubernetes.io/cluster/${EKSClusterName}" Value: "shared" - Key: kubernetes.io/role/elb Value: "1" OutsideSubnetAZ2RouteTableAssociation: Type: AWS::EC2::SubnetRouteTableAssociation Properties: SubnetId: !Ref OutsideSubnetAZ2 RouteTableId: !Ref PublicSubnetRouteTableID InsideSubnetAZ2: Type: AWS::EC2::Subnet Properties: VpcId: !Ref VPCID CidrBlock: !Ref CNFWInsideSubnet2CIDR AvailabilityZone: !Select [ 1, !GetAZs "" ] Tags: - Key: Name Value: !Sub [ "${pref}_inside_${az}", { pref: !Ref EKSClusterName, az: !Select [ 1, !GetAZs "" ] } ] - Key: !Sub "kubernetes.io/cluster/${EKSClusterName}" Value: "shared" - Key: kubernetes.io/role/internal-elb Value: "1" InsideSubnetAZ2RouteTableAssociation: Type: AWS::EC2::SubnetRouteTableAssociation Properties: SubnetId: !Ref InsideSubnetAZ2 RouteTableId: !Ref InsideSubnetsRouteTable # Security Groups DefaultSecurityGroupId: Type: Custom::CliQuery Properties: ServiceToken: !Sub 'arn:${AWS::Partition}:lambda:${AWS::Region}:${AWS::AccountId}:function:eks-quickstart-ResourceReader' AwsCliCommand: !Sub 'ec2 describe-security-groups --filters "Name=group-name,Values=default" "Name=vpc-id,Values=${VPCID}" --query "SecurityGroups[0]"' IdField: 'GroupId' ControlPlaneSecurityGroup: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: EKS cluster control plane security group. VpcId: !Ref VPCID Tags: - Key: Name Value: !Sub "${EKSClusterName}-control-plane" - Key: !Sub "kubernetes.io/cluster/${EKSClusterName}" Value: "owned" SecurityGroupIngress: - Description: "Allow pods to communicate with the EKS cluster API." IpProtocol: tcp FromPort: 443 ToPort: 443 CidrIp: 0.0.0.0/0 SecurityGroupEgress: - Description: "Allow cluster egress access to the Internet." IpProtocol: "-1" FromPort: 0 ToPort: 0 CidrIp: 0.0.0.0/0 ControlPlaneSecurityGroupIngress: Type: AWS::EC2::SecurityGroupIngress Properties: Description: Allow SG members to access k8s api. GroupId: !Ref ControlPlaneSecurityGroup SourceSecurityGroupId: !Ref ControlPlaneSecurityGroup IpProtocol: tcp FromPort: 443 ToPort: 443 ControlPlaneSecurityGroupIngressForWorkers: Type: AWS::EC2::SecurityGroupIngress Properties: Description: "Allow pods running on workers to send communication to cluster primary security group." GroupId: !Ref ControlPlaneSecurityGroup SourceSecurityGroupId: !Ref WorkersSecurityGroup IpProtocol: "-1" FromPort: 0 ToPort: 65535 WorkersSecurityGroup: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: EKS cluster worker nodes security group. VpcId: !Ref VPCID Tags: - Key: Name Value: !Sub "${EKSClusterName}-workers" - Key: !Sub "kubernetes.io/cluster/${EKSClusterName}" Value: "owned" SecurityGroupIngress: - Description: "Allow SSH access." IpProtocol: tcp FromPort: 22 ToPort: 22 CidrIp: 0.0.0.0/0 - Description: "Allow workers pods to receive communication from the cluster control plane." SourceSecurityGroupId: !Ref ControlPlaneSecurityGroup IpProtocol: "-1" FromPort: 0 ToPort: 0 SecurityGroupEgress: - Description: "Allow all egress." IpProtocol: "-1" FromPort: 0 ToPort: 0 CidrIp: 0.0.0.0/0 WorkersSecurityGroupIngress: Type: AWS::EC2::SecurityGroupIngress Properties: Description: "Allow inter-worker communication." GroupId: !Ref WorkersSecurityGroup SourceSecurityGroupId: !Ref WorkersSecurityGroup IpProtocol: "-1" FromPort: 0 ToPort: 0 WorkersSecurityGroup0: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: Security group for default interface of cluster workers. VpcId: !Ref VPCID Tags: - Key: Name Value: !Sub "${EKSClusterName}-workers-0" SecurityGroupIngress: - Description: "Allow connections from first CNFW interface." IpProtocol: "-1" FromPort: 0 ToPort: 65535 SourceSecurityGroupId: !Ref WorkersSecurityGroup1 - Description: "Allow connections from second CNFW interface" IpProtocol: "-1" FromPort: 0 ToPort: 65535 SourceSecurityGroupId: !Ref WorkersSecurityGroup2 - Description: "Allow connections from third CNFW interface" IpProtocol: "-1" FromPort: 0 ToPort: 65535 SourceSecurityGroupId: !Ref WorkersSecurityGroup3 WorkersSecurityGroup1: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: Security group for the first additional interface of cluster workers. VpcId: !Ref VPCID Tags: - Key: Name Value: !Sub "${EKSClusterName}-workers-1" WorkersSecurityGroup2: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: Security group for the second additional interface of enforcement point workers. That interface is designed to be used as an outside facing interface. VpcId: !Ref VPCID Tags: - Key: Name Value: !Sub "${EKSClusterName}-workers-2" SecurityGroupIngress: - Description: "Allow all ingress connections." IpProtocol: "-1" FromPort: 0 ToPort: 0 CidrIp: 0.0.0.0/0 SecurityGroupEgress: - Description: "Allow all egress connections." IpProtocol: "-1" FromPort: 0 ToPort: 0 CidrIp: 0.0.0.0/0 WorkersSecurityGroup3: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: Security group for the third additional interface of enforcement point workers. That interface is designed to be used as an inside facing interface. VpcId: !Ref VPCID Tags: - Key: Name Value: !Sub "${EKSClusterName}-workers-3" EFSSecurityGroup: Condition: EnableEFS Type: AWS::EC2::SecurityGroup Properties: GroupDescription: Security group that allows inbound NFS traffic for Amazon EFS mount points GroupName: !Sub "${EKSClusterName}-efs" VpcId: !Ref VPCID EFSSecurityGroupIngress: Condition: EnableEFS Type: AWS::EC2::SecurityGroupIngress Properties: Description: Allows inbound NFS traffic from the CIDR of cluster VPC GroupId: !Ref EFSSecurityGroup SourceSecurityGroupId: !Ref WorkersSecurityGroup IpProtocol: tcp FromPort: 2049 ToPort: 2049 # IAM EC2WorkersRole: Type: "AWS::IAM::Role" Properties: AssumeRolePolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Principal: Service: - ec2.amazonaws.com Action: - "sts:AssumeRole" ManagedPolicyArns: - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonEKSWorkerNodePolicy" - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonEKS_CNI_Policy" - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly" Policies: # Allow nodes to signal CF resources from user-data - PolicyName: cfn-signal PolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Action: - cloudformation:SignalResource Resource: !Sub "arn:${AWS::Partition}:cloudformation:${AWS::Region}:${AWS::AccountId}:stack/${AWS::StackName}*/*" # Allow route53ingress controller to manage route53 records - PolicyName: !Sub "${EKSClusterName}-workers-route53-policy" PolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Action: - "route53:ChangeResourceRecordSets" - "route53:ListResourceRecordSets" - "route53:ListHostedZones" Resource: - !Sub "arn:${AWS::Partition}:route53:::hostedzone/*" # Allow workers to describe EIPs, interfaces and regions - PolicyName: !Sub "${EKSClusterName}-workers-describe-eip-nic-regions-policy" PolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Action: - "ec2:DescribeAddresses" - "ec2:DescribeNetworkInterfaceAttribute" - "ec2:DescribeNetworkInterfaces" - "ec2:DescribeRegions" Resource: - "*" # Allow workers to associate EIP - PolicyName: !Sub "${EKSClusterName}-workers-eip-policy" PolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Action: - "ec2:AssociateAddress" - "ec2:AllocateAddress" Resource: - !Sub "arn:${AWS::Partition}:ec2:${AWS::Region}:${AWS::AccountId}:elastic-ip/*" - !Sub "arn:${AWS::Partition}:ec2:${AWS::Region}:${AWS::AccountId}:network-interface/*" # Allow worker nodes to manage additional ENIs - PolicyName: !Sub "${EKSClusterName}-workers-eni-policy" PolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Action: - "ec2:CreateTags" - "ec2:DescribeSubnets" - "ec2:AttachNetworkInterface" - "ec2:CreateNetworkInterface" - "ec2:ModifyNetworkInterfaceAttribute" Resource: - !Sub "arn:${AWS::Partition}:ec2:${AWS::Region}:${AWS::AccountId}:elastic-ip/*" - !Sub "arn:${AWS::Partition}:ec2:${AWS::Region}:${AWS::AccountId}:network-interface/*" - !Sub "arn:${AWS::Partition}:ec2:${AWS::Region}:${AWS::AccountId}:subnet/*" # Allow worker nodes to perform CA API calls - PolicyName: !Sub "${EKSClusterName}-workers-autoscaler-policy" PolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Action: - "autoscaling:DescribeAutoScalingGroups" - "autoscaling:DescribeAutoScalingInstances" - "autoscaling:DescribeLaunchConfigurations" - "autoscaling:DescribeTags" - "autoscaling:SetDesiredCapacity" - "autoscaling:TerminateInstanceInAutoScalingGroup" - "ec2:DescribeLaunchTemplateVersions" Resource: - "*" - !Sub "arn:${AWS::Partition}:autoscaling:${AWS::Region}:${AWS::AccountId}:autoScalingGroup:*:autoScalingGroupName/*" # Allow worker nodes to update Route Tables for ipv4subnetpool integration - PolicyName: !Sub "${EKSClusterName}-workers-route-table-policy" PolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Action: - "ec2:DescribeRouteTables" - "ec2:CreateRoute" - "ec2:ReplaceRoute" - "ec2:DeleteRoute" Resource: - !Sub "arn:${AWS::Partition}:ec2:${AWS::Region}:${AWS::AccountId}:route-table/*" EFSDriverRole: Condition: EnableEFS DependsOn: ClusterOIDCProvider Type: "AWS::IAM::Role" Properties: AssumeRolePolicyDocument: !Sub - | { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:${AWS::Partition}:iam::${AWS::AccountId}:oidc-provider/${OIDCIssuerURLWithoutProtocol}" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "${OIDCIssuerURLWithoutProtocol}:sub": "system:serviceaccount:kube-system:efs-csi-controller-sa" } } } ] } - OIDCIssuerURLWithoutProtocol: !Join [ '', !Split [ 'https://', !GetAtt EKSControlPlane.Outputs.OIDCIssuerURL ] ] Policies: - PolicyName: !Sub "${EKSClusterName}-efs-csi-driver-policy" PolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Action: - "elasticfilesystem:DescribeAccessPoints" - "elasticfilesystem:DescribeFileSystems" Resource: - !Sub "arn:${AWS::Partition}:elasticfilesystem:${AWS::Region}:${AWS::AccountId}:access-point/*" - !Sub "arn:${AWS::Partition}:elasticfilesystem:${AWS::Region}:${AWS::AccountId}:file-system/*" - Effect: Allow Action: - "elasticfilesystem:CreateAccessPoint" Resource: - !Sub "arn:${AWS::Partition}:elasticfilesystem:${AWS::Region}:${AWS::AccountId}:file-system/*" Condition: StringLike: "aws:RequestTag/efs.csi.aws.com/cluster": "true" - Effect: Allow Action: - "elasticfilesystem:DeleteAccessPoint" Resource: - !Sub "arn:${AWS::Partition}:elasticfilesystem:${AWS::Region}:${AWS::AccountId}:access-point/*" Condition: StringEquals: "aws:ResourceTag/efs.csi.aws.com/cluster": "true" # EKS Cluster EKSControlPlane: Type: AWS::CloudFormation::Stack DependsOn: ControlPlaneSecurityGroupIngress Properties: TemplateURL: !Sub - 'https://${S3Bucket}.s3.${S3Region}.${AWS::URLSuffix}/${QSS3KeyPrefix}submodules/quickstart-amazon-eks/templates/amazon-eks-controlplane.template.yaml' - S3Region: !If [ UsingDefaultBucket, !Ref 'AWS::Region', !Ref QSS3BucketRegion ] S3Bucket: !If [ UsingDefaultBucket, !Sub '${QSS3BucketName}-${AWS::Region}', !Ref QSS3BucketName ] Parameters: EKSClusterName: !Ref EKSClusterName KubernetesVersion: !Ref KubernetesVersion SecurityGroupIds: !Ref ControlPlaneSecurityGroup SubnetIds: !Join - "," - - !Ref PublicSubnet1ID - !Ref PublicSubnet2ID - !Ref PrivateSubnet1ID - !Ref PrivateSubnet2ID RoleArn: !Sub "arn:${AWS::Partition}:iam::${AWS::AccountId}:role/eks-quickstart-ControlPlane" CustomNodeRole: !Ref EC2WorkersRole FunctionRoleArn: !GetAtt IamStack.Outputs.KubernetesAdminRoleArn AdditionalEKSAdminUserArn: !Ref AdditionalEKSAdminUserArn AdditionalEKSAdminRoleArn: !Ref AdditionalEKSAdminRoleArn EKSPublicAccessEndpoint: "Enabled" EKSPrivateAccessEndpoint: "Disabled" EKSEncryptSecrets: "Disabled" EKSClusterLoggingTypes: !Ref EKSClusterLoggingTypes IamOidcProvider: "Disabled" # Remove OIDCProviderResource, DeleteOldClusterOIDCProviderIfExists, ClusterOIDCProviderRemover and ClusterOIDCRemoverRole # functions after v1.1+. They were added for an upgrade compatibility with a current amazon eks quickstart. OIDCProviderResource: Type: Custom::CliQuery Properties: ServiceToken: !Sub 'arn:${AWS::Partition}:lambda:${AWS::Region}:${AWS::AccountId}:function:eks-quickstart-ResourceReader' AwsCliCommand: !Sub - | iam list-open-id-connect-providers --query "{Arn: OpenIDConnectProviderList[?contains(Arn,'${OIDCIssuer}')] | [0].Arn || ''}" - OIDCIssuer: !Join [ '', !Split [ 'https://', !GetAtt EKSControlPlane.Outputs.OIDCIssuerURL ] ] IdField: Arn DeleteOldClusterOIDCProviderIfExists: Type: Custom::ClusterOIDCProviderRemover Properties: ServiceToken: !GetAtt ClusterOIDCProviderRemover.Arn ProviderArn: !GetAtt OIDCProviderResource.Arn ClusterOIDCProviderRemover: Type: AWS::Lambda::Function Properties: Runtime: python3.7 Handler: index.lambda_handler MemorySize: 128 Role: !GetAtt ClusterOIDCRemoverRole.Arn Timeout: 30 Code: ZipFile: | import boto3 import cfnresponse def lambda_handler(event, context): data = {} try: provider_arn = event['ResourceProperties']["ProviderArn"] if event['RequestType'] == 'Create': if provider_arn: resp = boto3.client("iam").delete_open_id_connect_provider(OpenIDConnectProviderArn=provider_arn) data["Reason"] = f"Provider with ARN {provider_arn} deleted" if provider_arn else "Provider not present" cfnresponse.send(event, context, cfnresponse.SUCCESS, data, provider_arn) except Exception as e: data["Reason"] = str(e) cfnresponse.send(event, context, cfnresponse.FAILED, data, "") ClusterOIDCRemoverRole: Type: AWS::IAM::Role Properties: Path: / AssumeRolePolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Principal: { Service: [ lambda.amazonaws.com ] } Action: [ sts:AssumeRole ] Policies: - PolicyName: root PolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Action: [ iam:DeleteOpenIDConnectProvider ] Resource: !Sub "arn:${AWS::Partition}:iam::${AWS::AccountId}:oidc-provider/*" - Effect: Allow Action: [ logs:CreateLogGroup, logs:CreateLogStream, logs:PutLogEvents ] Resource: !Sub "arn:${AWS::Partition}:logs:*:*:*" ClusterOIDCProvider: Type: AWS::IAM::OIDCProvider DependsOn: DeleteOldClusterOIDCProviderIfExists Properties: ClientIdList: - sts.amazonaws.com ThumbprintList: - 9e99a48a9960b14926bb7f3b02e22da2b0ab7280 Url: !GetAtt EKSControlPlane.Outputs.OIDCIssuerURL EKSLoadBalancerController: Condition: EnableLB Type: AWS::CloudFormation::Stack Properties: TemplateURL: !Sub - 'https://${S3Bucket}.s3.${S3Region}.${AWS::URLSuffix}/${QSS3KeyPrefix}submodules/quickstart-amazon-eks/templates/amazon-eks-load-balancer-controller.template.yaml' - S3Region: !If [ UsingDefaultBucket, !Ref 'AWS::Region', !Ref QSS3BucketRegion ] S3Bucket: !If [ UsingDefaultBucket, !Sub '${QSS3BucketName}-${AWS::Region}', !Ref QSS3BucketName ] Parameters: VpcId: !Ref VPCID EksClusterName: !Ref EKSClusterName ControllerCPULimit: "100m" ControllerMemoryLimit: "80Mi" ControllerReplicaCount: 1 ControllerHostNetwork: Enabled OIDCIssuerURLWithoutProtocol: !Join [ '', !Split [ 'https://', !GetAtt EKSControlPlane.Outputs.OIDCIssuerURL ] ] # EFS EFSFileSystem: Condition: EnableEFS Type: AWS::EFS::FileSystem Properties: Encrypted: true LifecyclePolicies: - TransitionToIA: AFTER_14_DAYS FileSystemTags: - Key: Name Value: !Ref EKSClusterName MountTargetAZ1: Condition: EnableEFS Type: AWS::EFS::MountTarget DependsOn: EFSSecurityGroupIngress Properties: FileSystemId: !Ref EFSFileSystem SubnetId: !Ref PrivateSubnet1ID SecurityGroups: - !Ref EFSSecurityGroup MountTargetAZ2: Condition: EnableEFS Type: AWS::EFS::MountTarget DependsOn: EFSSecurityGroupIngress Properties: FileSystemId: !Ref EFSFileSystem SubnetId: !Ref PrivateSubnet2ID SecurityGroups: - !Ref EFSSecurityGroup # Node groups NodeInstanceProfile: Type: "AWS::IAM::InstanceProfile" Properties: Path: / Roles: - !Ref EC2WorkersRole CPWorkersStack: Type: "AWS::CloudFormation::Stack" DependsOn: [ CalicoCNI ] Properties: TemplateURL: !Sub - 'https://${S3Bucket}.s3.${S3Region}.${AWS::URLSuffix}/${QSS3KeyPrefix}submodules/quickstart-amazon-eks-nodegroup/templates/amazon-eks-nodegroup.template.yaml' - S3Region: !If [ UsingDefaultBucket, !Ref 'AWS::Region', !Ref QSS3BucketRegion ] S3Bucket: !If [ UsingDefaultBucket, !Sub '${QSS3BucketName}-${AWS::Region}', !Ref QSS3BucketName ] Parameters: Subnet1ID: !Ref PublicSubnet1ID Subnet2ID: !Ref PublicSubnet2ID NodeInstanceType: !FindInMap [ ConfigurationTierMap, !Ref ControllerConfigurationTier, instanceType ] NumberOfNodes: !Ref ControlNodeGroupDesiredSize MaxNumberOfNodes: !Ref ControlNodeGroupMaxSize NodeGroupName: "cpworkers" EKSClusterName: !GetAtt EKSControlPlane.Outputs.EKSName NodeGroupType: "Unmanaged" LaunchTemplateId: !Ref CPLaunchTemplate LaunchTemplateVersion: !GetAtt CPLaunchTemplate.LatestVersionNumber CPLaunchTemplate: Type: "AWS::EC2::LaunchTemplate" Properties: LaunchTemplateName: !Sub "${EKSClusterName}-cpworkers" LaunchTemplateData: MetadataOptions: HttpPutResponseHopLimit: 2 IamInstanceProfile: Arn: !GetAtt NodeInstanceProfile.Arn KeyName: !Ref KeyPairName ImageId: !GetAtt UbuntuAMI.Value InstanceType: !FindInMap [ ConfigurationTierMap, !Ref ControllerConfigurationTier, instanceType ] NetworkInterfaces: - DeviceIndex: 0 AssociatePublicIpAddress: true DeleteOnTermination: true Groups: - !Ref WorkersSecurityGroup - !Ref WorkersSecurityGroup0 TagSpecifications: - ResourceType: instance Tags: - Key: Name Value: !Sub "${EKSClusterName}-cpworker" UserData: !Base64 "Fn::Sub": | Content-Type: multipart/mixed; boundary="MIMEBOUNDARY" MIME-Version: 1.0 --MIMEBOUNDARY Content-Type: text/cloud-config; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="cloud-config.txt" #cloud-config cloud_final_modules: - [scripts-user, always] --MIMEBOUNDARY Content-Disposition: attachment; filename="03-configure-instance.sh" Content-Transfer-Encoding: 7bit Content-Type: text/x-shellscript Mime-Version: 1.0 #!/bin/bash echo "setting hugepages and doing modprobe" swapoff -a cd /etc && sudo sed -i.backup '/swap/s/^/#/g' fstab && cd ~ HUGEPGSZ=`cat /proc/meminfo | grep Hugepagesize | cut -d : -f 2 | tr -d ' '` clear_huge_pages() { echo > .echo_tmp for d in /sys/devices/system/node/node? ; do echo "echo 0 > $d/hugepages/hugepages-${!HUGEPGSZ}/nr_hugepages" >> .echo_tmp done echo "Removing currently reserved hugepages" chmod 755 .echo_tmp sh .echo_tmp rm -f .echo_tmp remove_mnt_huge } remove_mnt_huge() { echo "Unmounting /mnt/huge and removing directory" grep -s '/mnt/huge' /proc/mounts > /dev/null if [[ $? -eq 0 && -d /mnt/huge ]] ; then umount /mnt/huge rm -R /mnt/huge fi } set_numa_pages() { clear_huge_pages NEW_PAGES=$1 echo "The new page is $NEW_PAGES" echo "#! /bin/bash" > .echo_tmp for d in /sys/devices/system/node/node? ; do node=$(basename $d) echo "Setting $NEW_PAGES pages for $node" if [ ! -e $d/hugepages/hugepages-${!HUGEPGSZ}/nr_hugepages ]; then echo "echo $NEW_PAGES > $d/hugepages/hugepages-${!HUGEPGSZ}/nr_hugepages" >> .echo_tmp else temp=$(cat $d/hugepages/hugepages-${!HUGEPGSZ}/nr_hugepages) if [ "$temp" != "$NEW_PAGES" ]; then echo "echo $NEW_PAGES > $d/hugepages/hugepages-${!HUGEPGSZ}/nr_hugepages" >> .echo_tmp fi fi done chmod 755 .echo_tmp echo "Reserving hugepages" sh ./.echo_tmp rm -f .echo_tmp create_mnt_huge } create_mnt_huge() { echo "Creating /mnt/huge and mounting as hugetlbfs" mkdir -p /mnt/huge grep '/mnt/huge' /proc/mounts if [ $? -ne 0 ] ; then echo "trying to mount" mount -t hugetlbfs nodev /mnt/huge echo "finished mounting hugetlbfs" fi echo "done mounting things" } display_current_setting() { for d in /sys/devices/system/node/node? ; do node=$(basename $d) echo "huge pages for $node: " cat $d/hugepages/hugepages-${!HUGEPGSZ}/nr_hugepages done } set_numa_pages 128 modprobe igb_uio wc_activate=1 modprobe dm_crypt # need to create tap_nlp ahead of time and set rp_filter off across the board. # NOTE: could (should?) be moved into init container along with interface attachment # logic, but that may require some permissions and stuff. ip tuntap add tap_nlp mode tap ip link set tap_nlp up echo 0 >/proc/sys/net/ipv4/conf/all/rp_filter echo 0 >/proc/sys/net/ipv4/conf/tap_nlp/rp_filter # override default docker ulimits according to STO's recommendation sudo bash -c "cat > /etc/docker/daemon.json" < /root/.docker/config.json" < /root/.docker/config.json" < .echo_tmp for d in /sys/devices/system/node/node? ; do echo "echo 0 > $d/hugepages/hugepages-${!HUGEPGSZ}/nr_hugepages" >> .echo_tmp done echo "Removing currently reserved hugepages" chmod 755 .echo_tmp sh .echo_tmp rm -f .echo_tmp remove_mnt_huge } remove_mnt_huge() { echo "Unmounting /mnt/huge and removing directory" grep -s '/mnt/huge' /proc/mounts > /dev/null if [[ $? -eq 0 && -d /mnt/huge ]] ; then umount /mnt/huge rm -R /mnt/huge fi } set_numa_pages() { clear_huge_pages NEW_PAGES=$1 echo "The new page is $NEW_PAGES" echo "#! /bin/bash" > .echo_tmp for d in /sys/devices/system/node/node? ; do node=$(basename $d) echo "Setting $NEW_PAGES pages for $node" if [ ! -e $d/hugepages/hugepages-${!HUGEPGSZ}/nr_hugepages ]; then echo "echo $NEW_PAGES > $d/hugepages/hugepages-${!HUGEPGSZ}/nr_hugepages" >> .echo_tmp else temp=$(cat $d/hugepages/hugepages-${!HUGEPGSZ}/nr_hugepages) if [ "$temp" != "$NEW_PAGES" ]; then echo "echo $NEW_PAGES > $d/hugepages/hugepages-${!HUGEPGSZ}/nr_hugepages" >> .echo_tmp fi fi done chmod 755 .echo_tmp echo "Reserving hugepages" sh ./.echo_tmp rm -f .echo_tmp create_mnt_huge } create_mnt_huge() { echo "Creating /mnt/huge and mounting as hugetlbfs" mkdir -p /mnt/huge grep '/mnt/huge' /proc/mounts if [ $? -ne 0 ] ; then echo "trying to mount" mount -t hugetlbfs nodev /mnt/huge echo "finished mounting hugetlbfs" fi echo "done mounting things" } display_current_setting() { for d in /sys/devices/system/node/node? ; do node=$(basename $d) echo "huge pages for $node: " cat $d/hugepages/hugepages-${!HUGEPGSZ}/nr_hugepages done } set_numa_pages ${HugePagesSize} modprobe igb_uio wc_activate=1 modprobe dm_crypt # need to create tap_nlp ahead of time and set rp_filter off across the board. # NOTE: could (should?) be moved into init container along with interface attachment # logic, but that may require some permissions and stuff. ip tuntap add tap_nlp mode tap ip link set tap_nlp up echo 0 >/proc/sys/net/ipv4/conf/all/rp_filter echo 0 >/proc/sys/net/ipv4/conf/tap_nlp/rp_filter # override default docker ulimits according to STO's recommendation sudo bash -c "cat > /etc/docker/daemon.json" < /mnt/coredump_helper #!/usr/bin/python3 import sys import os import os.path import subprocess import io import grp import logging from datetime import datetime DEFAULT_LIMIT=41943040 # 40 MB default MULTIPLIER = 1024 LOG_FILE= '/var/log/core.log' def parsePodInfo(pid): env = {} with open('/proc/%s/environ' % pid) as fd: for envspec in fd.read().split('\000'): if not envspec: continue try: varname, varval = envspec.split('=', 1) env[varname] = varval except Exception: continue return env def getSize(start_path = '.'): total_size = 0 for dirpath, dirnames, filenames in os.walk(start_path): for f in filenames: fp = os.path.join(dirpath, f) # skip if it is symbolic link if not os.path.islink(fp): total_size += os.path.getsize(fp) return total_size def writeCoreDump(env): coreLimit = DEFAULT_LIMIT now = datetime.fromtimestamp(int(timeStamp)) date = now.strftime("%Y%m%d") time = now.strftime("%H:%M:%S") key = "POD_NAME" if key in env: podName = env["POD_NAME"] podNamespace = env["POD_NAMESPACE"] corePath = env["COREDUMP_PATH"] coreLimitString = env["CORE_SIZE_LIMIT"].rstrip('MB') coreLimit = MULTIPLIER * MULTIPLIER * int(coreLimitString) logging.info("Core detectected in EP pod: %s ,Namespace: %s , dumping to path: %s", podName, podNamespace, corePath) else: podName = "HOST" podNamespace = "NONE" corePath = "/tmp/dumps/" logging.info("Core detectected in Host, dumping core to %s",corePath) os.makedirs(corePath,exist_ok=True) coreFile = os.path.join(corePath, '-'.join(["core",process_name, podNamespace, podName,date,time, pid, signum]) + ".gz") if getSize(corePath) < coreLimit: core = open(coreFile, 'w') p = subprocess.Popen("gzip", shell=True, stdin=sys.stdin, stdout=core, stderr=core) core.close() logging.info("Successfully dumped core to %s", coreFile) else: logging.error("Not enough space left in %s. Core will not be generated.",corePath) # Start of the script try: logging.basicConfig(filename=LOG_FILE,level=logging.DEBUG) if len(sys.argv) != 5: logging.info('Usage: %s