--- AWSTemplateFormatVersion: 2010-09-09 Description: Creates the Amazon FSx for Windows File Server environment for the FSx File Gateway workshop. Metadata: Authors: Description: Darryl Osborne (darrylo@amazon.com) AWS::CloudFormation::Interface: ParameterGroups: - Label: default: Workshop Environment Parameters: - VpcCidr - AvailabilityZones - InstanceType - KeyPair - LatestAmzn2AmiId - LatestWS2019AmiId ParameterLabels: AvailabilityZones: default: Availability Zones InstanceType: default: Instance type KeyPair: default: Key pair LatestAmzn2AmiId: default: Latest Amazon Linux 2 AMI (do not change) LatestWS2019AmiId: default: Latest Microsoft Server 2019 Base AMI (do not change) VpcCidr: default: VPC CIDR Parameters: AvailabilityZones: Description: Select two (2) Availability Zones (AZ). One public subnet and one private subnet will be created in each AZ. Type: List InstanceType: AllowedValues: - m5.large Default: m5.large Type: String KeyPair: Type: AWS::EC2::KeyPair::KeyName LatestAmzn2AmiId: Type: 'AWS::SSM::Parameter::Value' Default: '/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2' LatestWS2019AmiId: Type: 'AWS::SSM::Parameter::Value' Default: '/aws/service/ami-windows-latest/Windows_Server-2019-English-Full-Base' VpcCidr: AllowedValues: - 10.0.0.0/16 - 173.31.0.0/16 - 192.168.0.0/16 Default: 10.0.0.0/16 Description: Select the private IPv4 CIDR for the VPC. Type: String Resources: AttachInternetGateway: Type: AWS::EC2::VPCGatewayAttachment Properties: InternetGatewayId: !Ref InternetGateway VpcId: !Ref Vpc InternetGateway: Type: AWS::EC2::InternetGateway Properties: Tags: - Key: Name Value: !Join [ '', [ 'VPC IGW | ', !Ref 'AWS::StackName' ] ] PrivateRouteTable: Type: AWS::EC2::RouteTable Properties: Tags: - Key: Name Value: !Join [ '', [ 'Private Route Table | ', !Ref 'AWS::StackName' ] ] - Key: Network Value: Private VpcId: !Ref Vpc PrivateRouteTableAssociation0: Type: AWS::EC2::SubnetRouteTableAssociation Properties: SubnetId: !Ref PrivateSubnet0 RouteTableId: !Ref PrivateRouteTable PrivateRouteTableAssociation1: Type: AWS::EC2::SubnetRouteTableAssociation Properties: SubnetId: !Ref PrivateSubnet1 RouteTableId: !Ref PrivateRouteTable PrivateSubnet0: Type: AWS::EC2::Subnet Properties: AvailabilityZone: !Select [ 0, !Ref AvailabilityZones ] CidrBlock: !Select [ 1, !Cidr [ !GetAtt Vpc.CidrBlock, 3, 8 ]] Tags: - Key: Name Value: !Join [ '', [ 'Private Subnet 0 | ', !Ref 'AWS::StackName' ] ] - Key: SubnetType Value: Private VpcId: !Ref Vpc PrivateSubnet1: Type: AWS::EC2::Subnet Properties: AvailabilityZone: !Select [ 1, !Ref AvailabilityZones ] CidrBlock: !Select [ 2, !Cidr [ !GetAtt Vpc.CidrBlock, 3, 8 ] ] Tags: - Key: Name Value: !Join [ '', [ 'Private Subnet 1 | ', !Ref 'AWS::StackName' ] ] - Key: SubnetType Value: Private VpcId: !Ref Vpc PublicRoute: Type: AWS::EC2::Route Properties: RouteTableId: !Ref PublicRouteTable DestinationCidrBlock: 0.0.0.0/0 GatewayId: !Ref InternetGateway PublicRouteTable: Type: AWS::EC2::RouteTable Properties: Tags: - Key: Name Value: !Join [ '', [ 'Public Route Table | ', !Ref 'AWS::StackName' ] ] - Key: Network Value: Public VpcId: !Ref Vpc PublicRouteTableAssociation0: Type: AWS::EC2::SubnetRouteTableAssociation Properties: SubnetId: !Ref PublicSubnet0 RouteTableId: !Ref PublicRouteTable PublicRouteTableAssociation1: Type: AWS::EC2::SubnetRouteTableAssociation Properties: SubnetId: !Ref PublicSubnet1 RouteTableId: !Ref PublicRouteTable PublicSubnet0: Type: AWS::EC2::Subnet Properties: AvailabilityZone: !Select [ 0, !Ref AvailabilityZones ] CidrBlock: !Select [ 0, !Cidr [ !GetAtt Vpc.CidrBlock, 2, 4 ] ] MapPublicIpOnLaunch: true Tags: - Key: Name Value: !Join [ '', [ 'Public Subnet 0 | ', !Ref 'AWS::StackName' ] ] - Key: SubnetType Value: Public VpcId: !Ref Vpc PublicSubnet1: Type: AWS::EC2::Subnet Properties: AvailabilityZone: !Select [ 1, !Ref AvailabilityZones ] CidrBlock: !Select [ 1, !Cidr [ !GetAtt Vpc.CidrBlock, 2, 4 ] ] MapPublicIpOnLaunch: true Tags: - Key: Name Value: !Join [ '', [ 'Public Subnet 1 | ', !Ref 'AWS::StackName' ] ] - Key: SubnetType Value: Public VpcId: !Ref Vpc Vpc: Type: AWS::EC2::VPC Properties: CidrBlock: !Ref VpcCidr EnableDnsHostnames: true EnableDnsSupport: true Tags: - Key: Name Value: !Join [ '', [ 'VPC | ', !Ref 'AWS::StackName' ] ] ClientSecurityGroup: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: Security group for Amazon EC2 instances SecurityGroupIngress: - IpProtocol: tcp FromPort: 22 ToPort: 22 CidrIp: 0.0.0.0/0 - IpProtocol: tcp FromPort: 3389 ToPort: 3389 CidrIp: 0.0.0.0/0 Tags: - Key: Name Value: "Instance security group" VpcId: !Ref Vpc ClientSecurityGroupIngress: Type: AWS::EC2::SecurityGroupIngress Properties: GroupId: !Ref ClientSecurityGroup IpProtocol: -1 SourceSecurityGroupId: !Ref ClientSecurityGroup FileSystemSecurityGroup: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: Security group for Amazon FSx for Windows File Server SecurityGroupIngress: - IpProtocol: tcp FromPort: 445 ToPort: 445 SourceSecurityGroupId: !Ref ClientSecurityGroup - IpProtocol: tcp FromPort: 5985 ToPort: 5985 SourceSecurityGroupId: !Ref ClientSecurityGroup - IpProtocol: tcp FromPort: 445 ToPort: 445 CidrIp: 10.0.0.0/16 - IpProtocol: tcp FromPort: 445 ToPort: 445 CidrIp: 172.31.0.0/16 - IpProtocol: tcp FromPort: 445 ToPort: 445 CidrIp: 192.168.0.0/16 - IpProtocol: tcp FromPort: 5985 ToPort: 5985 CidrIp: 10.0.0.0/16 - IpProtocol: tcp FromPort: 5985 ToPort: 5985 CidrIp: 172.31.0.0/16 - IpProtocol: tcp FromPort: 5985 ToPort: 5985 CidrIp: 192.168.0.0/16 Tags: - Key: Name Value: "File system security group" VpcId: !Ref Vpc FileSystemSecurityGroupIngress: Type: AWS::EC2::SecurityGroupIngress Properties: GroupId: !Ref FileSystemSecurityGroup IpProtocol: -1 SourceSecurityGroupId: !Ref FileSystemSecurityGroup Directory: DependsOn: Password Type: AWS::DirectoryService::MicrosoftAD Properties: Edition: Standard Name: example.com Password: !Join ['', ['{{resolve:secretsmanager:', !Ref Password, ':SecretString:password}}' ]] VpcSettings: SubnetIds: - !Ref PrivateSubnet0 - !Ref PrivateSubnet1 VpcId: !Ref Vpc Password: Type: AWS::SecretsManager::Secret Properties: GenerateSecretString: SecretStringTemplate: !Sub '{"username": "admin@example.com"}' GenerateStringKey: password PasswordLength: 32 ExcludePunctuation: true DhcpOptions: Type: AWS::EC2::DHCPOptions Properties: DomainName: example.com DomainNameServers: !GetAtt Directory.DnsIpAddresses DhcpOptionsAssociation: Type: AWS::EC2::VPCDHCPOptionsAssociation Properties: DhcpOptionsId: !Ref DhcpOptions VpcId: !Ref Vpc WindowsFileSystem: Type: AWS::FSx::FileSystem Properties: FileSystemType: WINDOWS StorageCapacity: 4000 StorageType: SSD SubnetIds: - !Ref PrivateSubnet0 SecurityGroupIds: - !Ref FileSystemSecurityGroup Tags: - Key: Name Value: SAZ2 WindowsConfiguration: ActiveDirectoryId: !Ref Directory ThroughputCapacity: 64 DeploymentType: SINGLE_AZ_2 Instance0: DependsOn: Directory Type: AWS::EC2::Instance Properties: IamInstanceProfile: !Ref InstanceProfile ImageId: !Ref LatestWS2019AmiId InstanceType: !Ref InstanceType KeyName: !Ref KeyPair Monitoring: true SecurityGroupIds: - !Ref ClientSecurityGroup SubnetId: !Ref PublicSubnet0 Tags: - Key: Name Value: FSx Workshop Windows Instance UserData: Fn::Base64: !Sub | # disable ie security (windows) function Disable-ieESC { $AdminKey = "HKLM:\SOFTWARE\Microsoft\Active Setup\Installed Components\{A509B1A7-37EF-4b3f-8CFC-4F3A74704073}" $UserKey = "HKLM:\SOFTWARE\Microsoft\Active Setup\Installed Components\{A509B1A8-37EF-4b3f-8CFC-4F3A74704073}" Set-ItemProperty -Path $AdminKey -Name "IsInstalled" -Value 0 Set-ItemProperty -Path $UserKey -Name "IsInstalled" -Value 0 Stop-Process -Name Explorer Write-Host "IE Enhanced Security Configuration (ESC) has been disabled." -ForegroundColor Green } Disable-ieESC # install DiskSpd (windows) $path = "C:\Tools\DiskSpd-2.0.21a" $url = "https://github.com/microsoft/diskspd/releases/download/v2.0.21a/DiskSpd.zip" $destination = "C:\Tools\DiskSpd-2.0.21a.zip" $download = New-Object -Typename System.Net.WebClient New-Item -Type Directory -Path $path $download.DownloadFile($url,$destination) $extract = New-Object -ComObject Shell.Application $files = $extract.Namespace($destination).Items() $extract.NameSpace($path).CopyHere($files) Instance1: DependsOn: [ Directory, WindowsFileSystem ] Type: AWS::EC2::Instance Properties: IamInstanceProfile: !Ref InstanceProfile ImageId: !Ref LatestAmzn2AmiId InstanceType: !Ref InstanceType KeyName: !Ref KeyPair Monitoring: true SecurityGroupIds: - !Ref ClientSecurityGroup SubnetId: !Ref PublicSubnet0 Tags: - Key: Name Value: FSx Workshop Linux Instance UserData: Fn::Base64: !Sub | #cloud-config repo_update: true repo_upgrade: all packages: - cifs-utils - jq runcmd: - curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip" - unzip awscli-bundle.zip - ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws - sudo export PATH=/usr/local/bin:$PATH - amazon-linux-extras install -y epel - yum groupinstall -y "Development Tools" - yum install -y fpart parallel iftop iperf3 hping3 tree nload git libaio-devel openmpi openmpi-devel - cd /home/ec2-user - git clone https://github.com/bengland2/smallfile.git - cd /home/ec2-user - git clone https://github.com/hpc/ior.git - export PATH=$PATH:/usr/lib64/openmpi/bin - export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib64/openmpi/lib/ - cd ior - ./bootstrap - ./configure - make - sudo cp src/ior /usr/local/bin - mkdir -p /fsx - availability_zone=$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone) - region=${!availability_zone:0:-1} - echo -e "\n\n${!region}\ntext\n" | aws configure - ip_address=$(aws fsx describe-file-systems --file-system-ids ${WindowsFileSystem} --query 'FileSystems[].[WindowsConfiguration.PreferredFileServerIp]' --region ${!region} --output text) - password=$(aws secretsmanager get-secret-value --secret-id ${Password} --region ${!region} --output json | jq --raw-output .SecretString | jq -r ."password") - mount -t cifs -o vers=3.0,sec=ntlmsspi,user=admin@example.com,password=${!password} //${!ip_address}/share /fsx - aws s3 cp s3://darrylsosborne-us-west-2/workshop/fsx-file-gateway/sample-data/ /fsx/. --recursive - yum update -y - reboot FSxDomainJoinInstallDFS: DependsOn: Directory Type: AWS::SSM::Document Properties: DocumentType: Command Content: schemaVersion: "2.2" description: Join instances to an AWS Directory Service domain parameters: directoryId: type: String description: (Required) ID of the AWS Managed Microsoft AD (i.e. d-0123456789) directoryName: type: String description: (Required) Directory name (i.e. example.com) dnsIpAddresses: type: StringList default: [] description: "(Optional) AWS Managed Microsoft AD DNS IP addresses (i.e. 172.31.0.11, 172.31.0.12) Required when DHCP is not configured. Learn more at http://docs.aws.amazon.com/directoryservice/latest/simple-ad/join_get_dns_addresses.html" mainSteps: - action: aws:domainJoin name: joinDomain inputs: directoryId: "{{ directoryId }}" directoryName: "{{ directoryName }}" dnsIpAddresses: "{{ dnsIpAddresses }}" - action: aws:runPowerShellScript name: installDfsMgmt inputs: runCommand: - Install-WindowsFeature -Name FS-FileServer, FS-DFS-namespace -IncludeManagementTools; - Install-WindowsFeature -Name RSAT-DFS-Mgmt-Con - New-Item -Type Directory -Path "C:\DFS" Association: DependsOn: Directory Type: AWS::SSM::Association Properties: Name: !Ref FSxDomainJoinInstallDFS Parameters: directoryId: [!Ref Directory] directoryName: [!GetAtt ActiveDirectory.Name] dnsIpAddresses: - !Select [ 0, !GetAtt ActiveDirectory.DnsIpAddrs ] - !Select [ 1, !GetAtt ActiveDirectory.DnsIpAddrs ] Targets: - Key: InstanceIds Values: [ !Ref Instance0 ] InstanceProfile: DependsOn: Directory Type: AWS::IAM::InstanceProfile Properties: Path: '/' Roles: - !Ref InstanceRole InstanceRole: DependsOn: Directory Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Principal: Service: - ec2.amazonaws.com Action: - sts:AssumeRole ManagedPolicyArns: - arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM - arn:aws:iam::aws:policy/AmazonFSxReadOnlyAccess Path: "/" Policies: - PolicyName: secretsmanager PolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Action: - secretsmanager:GetSecretValue Resource: arn:aws:secretsmanager:*:*:secret:* LambdaExecutionRole: DependsOn: Directory Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: - lambda.amazonaws.com Action: - sts:AssumeRole ManagedPolicyArns: - 'arn:aws:iam::aws:policy/AWSLambdaExecute' - 'arn:aws:iam::aws:policy/AWSDirectoryServiceReadOnlyAccess' Path: "/" Policies: - PolicyName: root PolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Action: - logs:CreateLogGroup - logs:CreateLogStream - logs:PutLogEvents Resource: arn:aws:logs:*:*:* - Effect: Allow Action: - ec2:DescribeImages Resource: "*" ActiveDirectoryFunction: DependsOn: Directory Type: AWS::Lambda::Function Properties: Code: S3Bucket: !Sub solution-references-${AWS::Region} S3Key: fsx/dfs/ActiveDirectoryCustomResource.zip Handler: ad.handler Runtime: nodejs12.x Timeout: 30 Role: !GetAtt LambdaExecutionRole.Arn ActiveDirectory: DependsOn: Directory Type: Custom::ActiveDirectory Properties: ServiceToken: !GetAtt ActiveDirectoryFunction.Arn Region: !Ref 'AWS::Region' ID: !Ref Directory Outputs: VpcId: Value: !Ref Vpc VpcCidr: Value: !Ref VpcCidr