AWSTemplateFormatVersion: "2010-09-09" Description: Deploy Lenses on an AWS EC2 instance into an existing VPC.. **WARNING** This template creates Amazon EC2 instances and related resources. You will be billed for the AWS resources used if you create a stack from this template. (qs-1rrj29p8i) Metadata: QuickStartDocumentation: EntrypointName: Launch into an existing VPC Order: "2" AWS::CloudFormation::Interface: ParameterGroups: - Label: default: Network configuration Parameters: - VPC - SubnetId - Label: default: Lenses network Parameters: - LensesLocation - Label: default: Lenses (required) Parameters: - InstanceType - Label: default: Amazon MSK (required) Parameters: - MSKClusterARN - MSKSecurityGroup - Label: default: Amazon MSK SASL/SCRAM (optional) Parameters: - MSKSecretName - MSKSecretDecryptionKMSKey - Label: default: Services (optional) Parameters: - SchemaRegistryURLs - ConnectURLs - Label: default: Monitoring Parameters: - CloudWatchMetrics - Label: default: Lenses storage Parameters: - LensesStorage - StoragePostgresHostname - StoragePostgresUsername - StoragePostgresPassword - StoragePostgresDatabase ParameterLabels: CloudWatchMetrics: default: Cloudwatch metrics ConnectURLs: default: Connect JSON (optional) LensesStorage: default: Lenses storage type MSKClusterARN: default: Cluster ARN MSKSecretDecryptionKMSKey: defaukt: KMS SymmetricKey name MSKSecretName: default: MSK associated MSKSecretNameSecret MSKSecurityGroup: default: Cluster security group SchemaRegistryURLs: default: Schema registry JSON (optional) StoragePostgresDatabase: default: Postgres database StoragePostgresHostname: default: Postgres hostname StoragePostgresPassword: default: Postgres password StoragePostgresUsername: default: Postgres username SubnetId: default: Subnet ID of VPC Parameters: CloudWatchMetrics: Type: String Default: "No" AllowedValues: - "Yes" - "No" Description: Select *Yes* to forward Lenses metrics to CloudWatch. ConnectURLs: Type: String Description: Connect nodes. For more information, see https://docs.lenses.io/install_setup/configuration/lenses-config.html#kafka-connect[Configuration^]. InstanceType: Type: String Default: t2.large AllowedValues: - t2.large - t2.xlarge - t3.medium - t3.large - t3.xlarge - m4.large - m4.xlarge - m4.2xlarge - m4.4xlarge - m4.10xlarge - m4.16xlarge - m5.large - m5.xlarge - m5.2xlarge - m5.4xlarge - c4.large - c4.xlarge - c4.2xlarge - c4.4xlarge - c4.8xlarge - c5.large - c5.xlarge - c5.2xlarge - c5.4xlarge - c5.9xlarge - r4.large - r4.xlarge - r4.2xlarge - r4.4xlarge - r4.8xlarge - r4.16xlarge Description: Lenses EC2 instance type. LensesLocation: Type: String AllowedPattern: (\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})/(\d{1,2}) ConstraintDescription: Entry must be a valid IP range in CIDR format `x.x.x.x/x`. Description: Enter the inbound IP address range in CIDR format `x.x.x.x/x`. To allow one specific IP address, set to `x.x.x.x/32`. To allow all IP addresses, enter `0.0.0.0/0`. MaxLength: "18" MinLength: "9" LensesStorage: Type: String Default: local AllowedValues: - local - postgres Description: Storage mode. MSKClusterARN: Type: String Description: MSK cluster Amazon Resource Name (ARN). MSKSecretDecryptionKMSKey: Type: String Description: The AWS Key Management Service (AWS KMS) symmetric customer master key (CMK) used to decrypt the secret. MSKSecretName: Type: String Description: SASL/SCRAM secret name. MSKSecurityGroup: Type: String Description: MSK cluster security group ID. SchemaRegistryURLs: Type: String Description: Schema registry nodes. For more information, see https://docs.lenses.io/4.1/configuration/schema-registry/[Schema Registry^]. StoragePostgresDatabase: Type: String Default: "" Description: If storage is PostgreSQL, enter the database name. StoragePostgresHostname: Type: String Default: "" Description: If storage is PostgreSQL, enter the hostname. StoragePostgresPassword: Type: String Default: "" Description: If storage is PostgreSQL, enter the password. NoEcho: "true" StoragePostgresUsername: Type: String Default: "" Description: PostgreSQL username. Required if storage is PostgreSQL. SubnetId: Type: "AWS::EC2::Subnet::Id" ConstraintDescription: Must be an existing subnet in the selected VPC. Description: Subnet ID of an existing subnet (for the primary network) in your VPC. VPC: Type: "AWS::EC2::VPC::Id" Description: The VPC to which the Amazon MSK cluster is deployed. Mappings: RegionMap: us-east-1: linux: ami-0a89d582e2885cc40 us-east-2: linux: ami-08e1e91a6180f7b40 us-west-1: linux: ami-0dc730e43318102a7 us-west-2: linux: ami-0554f12e3965e6206 ca-central-1: linux: ami-0d16afe7fc8cbc1ea eu-central-1: linux: ami-0eb2f69f2f1e01577 eu-west-1: linux: ami-063674ec4de45960a eu-west-2: linux: ami-0c7f6e8ec13bea848 eu-west-3: linux: ami-0e0dad81c27dc9627 ap-southeast-1: linux: ami-0438e994d42560b20 ap-southeast-2: linux: ami-09b9ae5de50665d97 ap-south-1: linux: ami-0ad5e7422e7894dc3 ap-northeast-1: linux: ami-0eb9ae48e0d96372a ap-northeast-2: linux: ami-077d4256e6236e625 sa-east-1: linux: ami-09a710b2cc3a5bed2 Resources: CloudWatchLogGroup: Type: "AWS::Logs::LogGroup" Properties: LogGroupName: !Join - "-" - - quickstart-lenses - "Fn::Select": - 4 - "Fn::Split": - "-" - "Fn::Select": - 2 - "Fn::Split": - / - Ref: "AWS::StackId" RetentionInDays: 90 LensesSecurityGroup: Type: "AWS::EC2::SecurityGroup" Properties: GroupName: LensesEc2SecurityGroup GroupDescription: Enable HTTP access for Lenses SecurityGroupEgress: - CidrIp: "0.0.0.0/0" IpProtocol: "-1" SecurityGroupIngress: - Description: Access Lenses on the internet. CidrIp: !Ref LensesLocation FromPort: 9991 IpProtocol: tcp ToPort: 9991 - Description: Allow traffic for Amazon MSK. IpProtocol: "-1" SourceSecurityGroupId: !Ref MSKSecurityGroup Tags: - Key: Name Value: quickstart-lenses VpcId: !Ref VPC MSKSecurityIngress: Type: "AWS::EC2::SecurityGroupIngress" Properties: GroupId: !Ref MSKSecurityGroup IpProtocol: "-1" SourceSecurityGroupId: !GetAtt LensesSecurityGroup.GroupId EC2Role: Type: "AWS::IAM::Role" Properties: AssumeRolePolicyDocument: Statement: - Effect: Allow Principal: Service: - ec2.amazonaws.com Action: - "sts:AssumeRole" Path: / Policies: - PolicyName: LensesMSKAccessPolicy PolicyDocument: Statement: - Effect: Allow Action: - "kafka:Describe*" - "kafka:List*" - "kafka:Get*" - "acm-pca:IssueCertificate" - "acm-pca:GetCertificate" Resource: - Ref: MSKClusterARN - "Fn::Join": - "" - - "arn:aws:acm-pca:" - Ref: "AWS::Region" - ":" - Ref: "AWS::AccountId" - ":certificate-authority/*" - PolicyName: LensesLoggingPolicy PolicyDocument: Statement: - Effect: Allow Action: - "logs:CreateLogGroup" - "logs:CreateLogStream" - "logs:PutLogEvents" Resource: - "Fn::GetAtt": - CloudWatchLogGroup - Arn - "Fn::Join": - "" - - "arn:aws:logs:" - Ref: "AWS::Region" - ":" - Ref: "AWS::AccountId" - ":log-group:" - Ref: CloudWatchLogGroup - ":log-stream:" - "*" - PolicyName: LensesMSKAuthSecretAccessPolicy PolicyDocument: Statement: - Effect: Allow Action: - "secretsmanager:GetSecretValue" - "secretsmanager:DescribeSecret" - "kms:Decrypt" Resource: - "Fn::Join": - "" - - "arn:aws:secretsmanager:" - Ref: "AWS::Region" - ":" - Ref: "AWS::AccountId" - ":secret:" - Ref: MSKSecretName - "*" - "Fn::Join": - "" - - "arn:aws:kms:" - Ref: "AWS::Region" - ":" - Ref: "AWS::AccountId" - ":key/" - Ref: MSKSecretDecryptionKMSKey - PolicyName: LensesCloudwatchMetricDataPolicy PolicyDocument: Statement: - Effect: Allow Action: - "cloudwatch:PutMetricData" Resource: "*" RoleName: !Join - "-" - - quickstart-lenses-ec2 - "Fn::Select": - 4 - "Fn::Split": - "-" - "Fn::Select": - 2 - "Fn::Split": - / - Ref: "AWS::StackId" Tags: - Key: Name Value: quickstart-lenses IAMProfile: Type: "AWS::IAM::InstanceProfile" Properties: InstanceProfileName: !Join - "-" - - quickstart-lenses-ec2-iam-profile - "Fn::Select": - 4 - "Fn::Split": - "-" - "Fn::Select": - 2 - "Fn::Split": - / - Ref: "AWS::StackId" Path: / Roles: - Ref: EC2Role Lenses: Type: "AWS::EC2::Instance" Properties: BlockDeviceMappings: - DeviceName: /dev/sdf Ebs: DeleteOnTermination: false VolumeSize: 20 IamInstanceProfile: !Ref IAMProfile ImageId: !FindInMap - RegionMap - Ref: "AWS::Region" - linux InstanceType: !Ref InstanceType SecurityGroupIds: - "Fn::GetAtt": - LensesSecurityGroup - GroupId SubnetId: !Ref SubnetId Tags: - Key: Name Value: quickstart-lenses UserData: !Base64 "Fn::Join": - "" - - "Fn::Sub": | #!/bin/bash set -o errexit cd /opt MSK_CLUSTER_ARN="${MSKClusterARN}" MSK_SECRET_NAME="${MSKSecretName}" AWS_REGION="${AWS::Region}" LENSES_STORAGE_TYPE="${LensesStorage}" LENSES_STORAGE_POSTGRES_HOSTNAME="${StoragePostgresHostname}" LENSES_STORAGE_POSTGRES_USERNAME="${StoragePostgresUsername}" LENSES_STORAGE_POSTGRES_PASSWORD="${StoragePostgresPassword}" LENSES_STORAGE_POSTGRES_DATABASE="${StoragePostgresDatabase}" CloudWatchCustomMetrics="${CloudWatchMetrics}" CLOUDWATCH_LOG_GROUP="${CloudWatchLogGroup}" SCHEMA_REGISTRY_URLS='${SchemaRegistryURLs}' CONNECT_URLS='${ConnectURLs}' CLUSTER_PROTOCOL="PLAINTEXT" CONN_PROTOCOL="PLAINTEXT" JSON_CLUSTER_TRANSIT=".BootstrapBrokerString" KEY_ALIAS="lenses-io" # Create base Lenses Configuration cat << EOF > /opt/lenses/lenses.conf lenses.port=9991 lenses.secret.file=security.conf lenses.sql.state.dir="/mnt/persistent/kafka-streams-state" lenses.storage.directory="/mnt/persistent/storage" lenses.license.file=license.json EOF # Create Lenses License cat << EOF > /opt/lenses/license.json EOF - | EC2_INSTANCE_ID="`wget -q -O - http://169.254.169.254/latest/meta-data/instance-id`" ADMIN_PASS=$EC2_INSTANCE_ID # Create Lenses Security configuration cat << EOF > /opt/lenses/security.conf lenses.security.user="admin" lenses.security.password="${ADMIN_PASS}" EOF if [ "${LENSES_STORAGE_TYPE}" == "postgres" ]; then cat << EOF >> /opt/lenses/lenses.conf lenses.storage.postgres.database: "${LENSES_STORAGE_POSTGRES_DATABASE}" lenses.storage.postgres.host: "${LENSES_STORAGE_POSTGRES_HOSTNAME}" lenses.storage.postgres.port: 5432 lenses.storage.postgres.username: "${LENSES_STORAGE_POSTGRES_USERNAME}" EOF cat << EOF >> /opt/lenses/security.conf lenses.storage.postgres.password: "${LENSES_STORAGE_POSTGRES_PASSWORD}" EOF fi if [ ! -z "$MSK_CLUSTER_ARN" ]; then CLUSTER_PROTOCOL=$(/usr/local/bin/aws kafka describe-cluster --cluster-arn $MSK_CLUSTER_ARN --region $AWS_REGION | jq -r '.ClusterInfo.EncryptionInfo.EncryptionInTransit.ClientBroker') echo "[INFO] MSK Brokers Transit: $CLUSTER_PROTOCOL" case $CLUSTER_PROTOCOL in PLAINTEXT) ;; TLS | TLS_PLAINTEXT) # TLS support JSON_CLUSTER_TRANSIT=".BootstrapBrokerStringTls" CONN_PROTOCOL="SSL" mkdir -p /var/private/ssl cp /usr/lib/jvm/java-1.8.0/jre/lib/security/cacerts /var/private/ssl/client.truststore.jks SASL_SCRAM_ENABLED="$(aws kafka describe-cluster \ --cluster-arn "${MSK_CLUSTER_ARN}" \ --region "${AWS_REGION}" | jq -r '.ClusterInfo.ClientAuthentication.Sasl.Scram.Enabled')" if [[ "${SASL_SCRAM_ENABLED}" == "true" ]]; then sasl_scram_secret="$(aws secretsmanager get-secret-value --secret-id "${MSK_SECRET_NAME}" --region "${AWS_REGION}" | jq -r '.SecretString')" sasl_cram_username="$(echo "${sasl_scram_secret}" | jq -r '.username')" sasl_cram_password="$(echo "${sasl_scram_secret}" | jq -r '.password')" cat >/etc/users_jaas.conf<> /opt/lenses/lenses.conf fi echo "lenses.kafka.settings.client.security.protocol=${CONN_PROTOCOL}" >> /opt/lenses/lenses.conf echo 'lenses.kafka.settings.client.ssl.truststore.location=/var/private/ssl/client.truststore.jks' >> /opt/lenses/lenses.conf CLUSTER_PRIVATE_CA_ARN=$(/usr/local/bin/aws kafka describe-cluster --cluster-arn $MSK_CLUSTER_ARN --region $AWS_REGION | jq -r '.ClusterInfo.ClientAuthentication.Tls.CertificateAuthorityArnList | .[0]') # TLS support with Authentication if [ "$CLUSTER_PRIVATE_CA_ARN" != "null" ]; then keytool -genkey -keystore /var/private/ssl/client.keystore.jks \ -validity 300 -storepass $ADMIN_PASS \ -keypass $ADMIN_PASS \ -dname "CN=$KEY_ALIAS" \ -alias $KEY_ALIAS \ -storetype pkcs12 keytool -keystore /var/private/ssl/client.keystore.jks \ -certreq -file client-cert-sign-request \ -alias $KEY_ALIAS \ -storepass $ADMIN_PASS \ -keypass $ADMIN_PASS # Remove NEW word (AWS recommendation) in order to issue-certificate sed -i 's/NEW //g' client-cert-sign-request # Create a certificate request CERTIFICATE_REQUEST_ARN=$(/usr/local/bin/aws acm-pca issue-certificate --region $AWS_REGION --certificate-authority-arn $CLUSTER_PRIVATE_CA_ARN --csr file://client-cert-sign-request --signing-algorithm "SHA256WITHRSA" --validity Value=300,Type="DAYS" | jq -r '.CertificateArn') # Wait Certificate request to finish sleep 20 # Get certification for the new Certificate Request /usr/local/bin/aws acm-pca get-certificate --region $AWS_REGION --certificate-authority-arn $CLUSTER_PRIVATE_CA_ARN --certificate-arn $CERTIFICATE_REQUEST_ARN >> result.json CERTIFICATE=$(cat result.json | jq -r '.Certificate') CERTIFICATE_CHAIN=$(cat result.json | jq -r '.CertificateChain') # Create Signed certificate from ACM cat << EOF > /var/private/ssl/signed-certificate-from-acm $CERTIFICATE $CERTIFICATE_CHAIN EOF yes | keytool -keystore /var/private/ssl/client.keystore.jks \ -import -file /var/private/ssl/signed-certificate-from-acm \ -alias $KEY_ALIAS \ -storepass $ADMIN_PASS \ -keypass $ADMIN_PASS # Create the keystore configuration echo lenses.kafka.settings.client.ssl.keystore.location="/var/private/ssl/client.keystore.jks" >> /opt/lenses/lenses.conf echo lenses.kafka.settings.client.ssl.keystore.password="\"$ADMIN_PASS"\" >> /opt/lenses/lenses.conf echo lenses.kafka.settings.client.ssl.key.password="\"$ADMIN_PASS"\" >> /opt/lenses/lenses.conf # append empty line echo "" >> /opt/lenses/lenses.conf # Cleanup rm result.json fi ;; *) echo "[ERROR] Uknown MSK Protocol" exit 1 ;; esac # Create the connection string for the brokers BOOTSTRAP_BROKERS=($(/usr/local/bin/aws kafka get-bootstrap-brokers --cluster-arn $MSK_CLUSTER_ARN --region $AWS_REGION | jq -r "$JSON_CLUSTER_TRANSIT" | tr "," "\n")) for BROKER in "${BOOTSTRAP_BROKERS[@]}" do BROKERS="${BROKERS:+$BROKERS,}$CONN_PROTOCOL://$BROKER" done # Create the connection string for the zookeepers ZOOKEEPER_CONN=($(/usr/local/bin/aws kafka describe-cluster --cluster-arn $MSK_CLUSTER_ARN --region $AWS_REGION | jq -r '.ClusterInfo.ZookeeperConnectString' | tr "," "\n")) echo $ZOOKEEPER_CONN for ZK in ${ZOOKEEPER_CONN[@]}; do ZOOKEEPERS="${ZOOKEEPERS:+$ZOOKEEPERS, }{url:\"$ZK\"}" done ZOOKEEPERS="[$ZOOKEEPERS]" IS_OPEN_MONITORING_ENABLED=$(/usr/local/bin/aws kafka describe-cluster --cluster-arn $MSK_CLUSTER_ARN --region $AWS_REGION | jq -r '.ClusterInfo.OpenMonitoring.Prometheus.JmxExporter.EnabledInBroker') fi if [ -z "$BROKERS" ]; then echo "[ERROR] Unable to configure Kafka Brokers." exit 1 fi # Append Lenses configuration echo lenses.kafka.brokers="\"$BROKERS"\" >> /opt/lenses/lenses.conf if [ ! -z "$ZOOKEEPERS" ]; then echo lenses.zookeeper.hosts="$ZOOKEEPERS" >> /opt/lenses/lenses.conf fi if [ ! -z "$SCHEMA_REGISTRY_URLS" ]; then echo lenses.schema.registry.urls="$SCHEMA_REGISTRY_URLS" >> /opt/lenses/lenses.conf fi if [ ! -z "$CONNECT_URLS" ]; then echo lenses.kafka.connect.clusters="$CONNECT_URLS" >> /opt/lenses/lenses.conf fi if $IS_OPEN_MONITORING_ENABLED; then MSK_NODES=($(/usr/local/bin/aws kafka list-nodes --cluster-arn $MSK_CLUSTER_ARN --region $AWS_REGION | jq -r ".NodeInfoList[] | @base64")) for NODE in "${MSK_NODES[@]}" do BROKER_ID=$(echo $NODE | base64 --decode | jq -r ${1} '.BrokerNodeInfo.BrokerId') BROKER_ENDPOINT=$(echo $NODE | base64 --decode | jq -r ${1} ".BrokerNodeInfo.Endpoints[]") KAFKA_METRICS_PORT="${KAFKA_METRICS_PORT:+$KAFKA_METRICS_PORT,}{id: $BROKER_ID, url:\"http://$BROKER_ENDPOINT:11001/metrics\"}" done # Append to Lenses conf echo lenses.kafka.metrics={type: "AWS", port: [$KAFKA_METRICS_PORT]} >> /opt/lenses/lenses.conf fi # Start Lenses Service systemctl restart lenses-io systemctl enable lenses-io.service # Create configuration for AWS Cloudwatch Logging mkdir -p /etc/awslogs cat << EOF > /etc/awslogs/awscli.conf [plugins] cwlogs = cwlogs [default] region = $AWS_REGION EOF cat /dev/null > /etc/awslogs/awslogs.conf cat << EOF > /etc/awslogs/awslogs.conf [general] state_file= /var/awslogs/state/agent-state [/opt/lenses/logs/lenses] file = /opt/lenses/logs/lenses.log log_group_name = $CLOUDWATCH_LOG_GROUP log_stream_name = $EC2_INSTANCE_ID/lenses.log [/opt/lenses/logs/lenses-warn] file = /opt/lenses/logs/lenses-warn.log log_group_name = $CLOUDWATCH_LOG_GROUP log_stream_name = $EC2_INSTANCE_ID/lenses-warn.log [/opt/lenses/logs/lenses-metrics] file = /opt/lenses/logs/metrics.log log_group_name = $CLOUDWATCH_LOG_GROUP log_stream_name = $EC2_INSTANCE_ID/lenses-metrics.log EOF systemctl restart awslogsd systemctl enable awslogsd.service echo "AWS_DEFAULT_REGION=${AWS_REGION}" >> /etc/environment if [ "${CloudWatchCustomMetrics}" == "Yes" ] && [ -e /opt/awscloudwatch.py ]; then systemctl enable --now cloudwatch_metrics.service fi Outputs: FQDN: Description: FQDN for the Lenses EC2 instance. Value: !Join - ":" - - "Fn::GetAtt": - Lenses - PublicDnsName - "9991" Postdeployment: Description: See the deployment guide for post-deployment steps. Value: https://aws.amazon.com/quickstart/?quickstart-all.sort-by=item.additionalFields.sortDate&quickstart-all.sort-order=desc&awsm.page-quickstart-all=5