Parameters: EventTableTTL: Description: The amount of time in days that current event data is cached in Dynamodb tables Type: Number Default: 1 JobTableTTL: Description: The amount of time in days that current job data is cached in Dynamodb tables Type: Number Default: 1 EventStreamTTL: Description: The amount of time in days that current event data is cached in Kinesis streams Type: Number Default: 1 JobStreamTTL: Description: The amount of time in days that current job data is cached in Kinesis streams Type: Number Default: 1 MetricStreamTTL: Description: The amount of time in days that current alarm data is cached in Kinesis streams Type: Number Default: 1 UseStreams: Type: String Default: No AllowedValues: - Yes - No Mappings: SourceCode: General: S3Bucket: "rodeolabz" KeyPrefix: "vodtk/2-mediaconvert-job-progress-metrics/pipeline-base" Resources: EventTable: Type: AWS::DynamoDB::Table Properties: TableName: !Sub "${AWS::StackName}-EventTable" AttributeDefinitions: - AttributeName: id AttributeType: S - AttributeName: timestamp AttributeType: N - AttributeName: jobId AttributeType: S KeySchema: - AttributeName: id KeyType: HASH - AttributeName: timestamp KeyType: RANGE GlobalSecondaryIndexes: - IndexName: jobId-timestamp-index KeySchema: - AttributeName: jobId KeyType: HASH - AttributeName: timestamp KeyType: RANGE Projection: ProjectionType: ALL ProvisionedThroughput: ReadCapacityUnits: 5 WriteCapacityUnits: 5 ProvisionedThroughput: ReadCapacityUnits: 5 WriteCapacityUnits: 5 TimeToLiveSpecification: AttributeName: timestampTTL Enabled: true JobTable: Type: AWS::DynamoDB::Table Properties: TableName: !Sub "${AWS::StackName}-JobTable" AttributeDefinitions: - AttributeName: id AttributeType: S - AttributeName: status AttributeType: S - AttributeName: createdAt AttributeType: N KeySchema: - AttributeName: id KeyType: HASH GlobalSecondaryIndexes: - IndexName: status-createdAt-index KeySchema: - AttributeName: status KeyType: HASH - AttributeName: createdAt KeyType: RANGE Projection: ProjectionType: INCLUDE NonKeyAttributes: - progressMetrics - analysis - filters.jobId - filters.queueName - filters.region - id - status - eventTimes ProvisionedThroughput: ReadCapacityUnits: 5 WriteCapacityUnits: 5 - IndexName: id-createdAt-index KeySchema: - AttributeName: id KeyType: HASH - AttributeName: createdAt KeyType: RANGE Projection: ProjectionType: INCLUDE NonKeyAttributes: - progressMetrics - analysis - filters.jobId - filters.queueName - filters.region - id - status - eventTimes ProvisionedThroughput: ReadCapacityUnits: 5 WriteCapacityUnits: 5 ProvisionedThroughput: ReadCapacityUnits: 10 WriteCapacityUnits: 20 TimeToLiveSpecification: AttributeName: timestampTTL Enabled: true EventLambdaRole: Type: "AWS::IAM::Role" Properties: RoleName: !Sub "${AWS::StackName}-EventLambdaRole" AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Principal: Service: - lambda.amazonaws.com Action: - sts:AssumeRole Policies: - PolicyName: !Sub "${AWS::StackName}-EventLambdaPolicy" PolicyDocument: Statement: - Effect: Allow Action: - logs:CreateLogGroup - logs:CreateLogStream - logs:PutLogEvents Resource: - !Join ["", ["arn:aws:logs:", Ref: "AWS::Region", ":", Ref: "AWS::AccountId", ":log-group:/aws/lambda/*"]] - Effect: Allow Action: "kinesis:*" Resource: - Fn::GetAtt: - EventStream - Arn - Fn::GetAtt: - "JobStream" - "Arn" - Fn::GetAtt: - "MetricStream" - "Arn" - Effect: Allow Action: - dynamodb:GetItem - dynamodb:PutItem - dynamodb:UpdateItem - dynamodb:Scan Resource: - !Join ["", ["arn:aws:dynamodb:", Ref: "AWS::Region", ":", Ref: "AWS::AccountId", ":table/",Ref: "JobTable"]] - !Join ["", ["arn:aws:dynamodb:", Ref: "AWS::Region", ":", Ref: "AWS::AccountId", ":table/",Ref: "EventTable"]] - PolicyName: !Sub "${AWS::StackName}-EventLambdaMediainfoPolicy" PolicyDocument: Statement: - Effect: Allow Action: - s3:GetObjectVersionTorrent - s3:GetObjectAcl - s3:GetObject - s3:GetObjectTorrent - s3:GetObjectVersionTagging - s3:GetObjectVersionAcl - s3:GetObjectTagging - s3:GetObjectVersionForReplication - s3:GetObjectVersion - s3:ListMultipartUploadParts Resource: "arn:aws:s3:::*/*" - Effect: Allow Action: - s3:ListBucketByTags - s3:GetLifecycleConfiguration - s3:ListBucketMultipartUploads - s3:GetBucketTagging - s3:GetInventoryConfiguration - s3:GetBucketWebsite - s3:ListBucketVersions - s3:GetBucketLogging - s3:ListBucket - s3:GetAccelerateConfiguration - s3:GetBucketVersioning - s3:GetBucketAcl - s3:GetBucketNotification - s3:GetBucketPolicy - s3:GetReplicationConfiguration - s3:GetBucketRequestPayment - s3:GetBucketCORS - s3:GetAnalyticsConfiguration - s3:GetMetricsConfiguration - s3:GetBucketLocation - s3:GetIpConfiguration Resource: "arn:aws:s3:::*" - Effect: Allow Action: - s3:ListAllMyBuckets - s3:HeadBucket - s3:ListObjects Resource: "*" EventInvokeLambda: Type: AWS::Lambda::Permission Properties: FunctionName: !GetAtt EventCollectorLambda.Arn Action: lambda:InvokeFunction Principal: events.amazonaws.com SourceArn: Fn::GetAtt: - "AllMediaConvertEventRule" - "Arn" AllMediaConvertEventRule: Type: "AWS::Events::Rule" Properties: Description: catches MediaConvert Events State: "ENABLED" EventPattern: source: - "aws.mediaconvert" Targets: - Arn: Fn::GetAtt: - "EventCollectorLambda" - "Arn" Id: "TargetEventCollectorLambda" EventCollectorLambda: Type: AWS::Lambda::Function Properties: FunctionName: !Sub ${AWS::StackName}-EventCollector Description: Collect events, update status, make metrics Handler: event_collector.lambda_handler Role: !GetAtt EventLambdaRole.Arn Code: S3Bucket: !Join ["-", [!FindInMap ["SourceCode", "General", "S3Bucket"], Ref: "AWS::Region"]] S3Key: !Join ["/", [!FindInMap ["SourceCode", "General", "KeyPrefix"], "event_collector.zip"]] Environment: Variables: EventTable: !Ref EventTable JobTable: !Ref JobTable EventStream: !Ref EventStream JobStream: !Ref JobStream MetricStream: !Ref MetricStream EventTableTTL: !Ref EventTableTTL JobTableTTL: !Ref JobTableTTL EventStreamTTL: !Ref EventStreamTTL JobStreamTTL: !Ref JobStreamTTL MetricStreamTTL: !Ref MetricStreamTTL Runtime: python3.6 Timeout: 120 EventStream: Type: "AWS::Kinesis::Stream" Properties: Name: !Sub "${AWS::StackName}-EventStream" ShardCount: 1 JobStream: Type: "AWS::Kinesis::Stream" Properties: Name: !Sub "${AWS::StackName}-JobStream" ShardCount: 1 MetricStream: Type: "AWS::Kinesis::Stream" Properties: Name: !Sub "${AWS::StackName}-MetricStream" ShardCount: 1 StreamConsumerRole: Type: "AWS::IAM::Role" Properties: RoleName: !Sub "${AWS::StackName}-StreamConsumerRole" AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Principal: Service: - lambda.amazonaws.com - firehose.amazonaws.com Action: - sts:AssumeRole Policies: - PolicyName: !Sub "${AWS::StackName}-StreamConsumerPolicy" PolicyDocument: Statement: - Effect: Allow Action: "kinesis:*" Resource: - Fn::GetAtt: - EventStream - Arn - Fn::GetAtt: - "JobStream" - "Arn" - Fn::GetAtt: - "MetricStream" - "Arn" APILambdaRole: Type: "AWS::IAM::Role" Properties: RoleName: !Sub "${AWS::StackName}-APILambdaRole" AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Principal: Service: - lambda.amazonaws.com Action: - sts:AssumeRole Policies: - PolicyName: !Sub "${AWS::StackName}-APILambdaPolicy" PolicyDocument: Statement: - Effect: Allow Action: - logs:CreateLogGroup - logs:CreateLogStream - logs:PutLogEvents Resource: - !Join ["", ["arn:aws:logs:", Ref: "AWS::Region", ":", Ref: "AWS::AccountId", ":log-group:/aws/lambda/*"]] - Effect: Allow Action: - dynamodb:GetItem - dynamodb:Scan - dynamodb:BatchGetItem - dynamodb:DescribeTable - dynamodb:GetItem - dynamodb:ListTables - dynamodb:Query - dynamodb:Scan - dynamodb:DescribeReservedCapacity - dynamodb:DescribeReservedCapacityOfferings - dynamodb:ListTagsOfResource - dynamodb:DescribeTimeToLive - dynamodb:DescribeLimits - dynamodb:ListGlobalTables Resource: - "*" # - !Join ["", ["arn:aws:dynamodb:", Ref: "AWS::Region", ":", Ref: "AWS::AccountId", ":table/",Ref: "JobTable"]] # - !Join ["", ["arn:aws:dynamodb:", Ref: "AWS::Region", ":", Ref: "AWS::AccountId", ":table/",Ref: "EventTable"]] Outputs: EventTable: Description: The name of the Table to get Event data from Value: !Ref EventTable JobTable: Description: The name of the Table to get Job data from Value: !Ref JobTable EventStreamARN: Description: The ARN of the stream to get Event data from Value: !GetAtt EventStream.Arn JobStreamARN: Description: The ARN of the stream to get Job data from Value: !GetAtt JobStream.Arn MetricStreamARN: Description: The ARN of the stream to get Metric data from Value: !GetAtt MetricStream.Arn StreamConsumerRoleArn: Description: The ARN of the role to use to grant consumers access to the streams Value: !GetAtt StreamConsumerRole.Arn APILambdaRoleArn: Description: The ARN of the role to use to grant APIs access to Dyanmodb tables Value: !GetAtt APILambdaRole.Arn