The following steps describe how to create an AWS Lambda function to automatically normalize and upload any new file matching the pattern s3://{{ this.DATA_BUCKET_NAME }}/{{ prefix }}*{{ suffix }} to AMC. Each file will upload into the (previously defined) dataset, {{s3_trigger_dataset_id}}, so be sure that dataset exists and every input file complies with the schema defined for that dataset. This procedure assumes you are using MacOS or Linux and have installed and configured the AWS Command Line Interface to interact with your AWS account.
| Target dataset: |
|
| S3 key filter prefix: |
|
| S3 key filter suffix: |
|
$ aws iam create-role --role-name AmcUploaderS3TriggerRole --assume-role-policy-document file://trust_policy.json
$ aws iam put-role-policy --role-name AmcUploaderS3TriggerRole --policy-name AmcUploaderS3TriggerRolePolicy --policy-document file://permissions_policy.json
$ ROLE_ARN=$(aws iam get-role --role-name AmcUploaderS3TriggerRole --query 'Role.Arn' --output text)
$ pip3 install --target ./package requests
$ cd package
$ zip -r ../amc-uploader-s3-trigger.zip .
$ cd ..
$ zip amc-uploader-s3-trigger.zip lambda_function.py
$ rm -rf package
$ aws lambda create-function --function-name AmcUploaderS3Trigger --runtime python3.10 --role ${ROLE_ARN} --zip-file fileb://amc-uploader-s3-trigger.zip --handler lambda_function.lambda_handler --region
{{ AWS_REGION }}
$ FUNCTION_ARN=$(aws lambda get-function --function-name AmcUploaderS3Trigger --query 'Configuration.FunctionArn' --region {{ AWS_REGION }})
$ aws lambda add-permission --region {{ AWS_REGION }} --function-name AmcUploaderS3Trigger --action "lambda:InvokeFunction" --principal s3.amazonaws.com --source-arn arn:aws:s3:::{{ DATA_BUCKET_NAME}} --statement-id s3-trigger
$ aws s3api put-bucket-notification-configuration --region {{ AWS_REGION }} --bucket {{ DATA_BUCKET_NAME}} --notification-configuration '{
"LambdaFunctionConfigurations": [
{
"LambdaFunctionArn": '${FUNCTION_ARN}',
"Events": ["s3:ObjectCreated:*"],
"Filter": {
"Key": {
"FilterRules": [
{
"Name": "prefix",
"Value": "{{ prefix }}"
},
{
"Name": "suffix",
"Value": "{{ suffix }}"
}
]
}
}
}
]
}'
$ rm amc-uploader-s3-trigger.zip lambda_function.py permissions_policy.json trust_policy.json
$ aws lambda delete-function --function-name AmcUploaderS3Trigger --region {{ AWS_REGION }}
$ aws iam delete-role-policy --policy-name AmcUploaderS3TriggerRolePolicy --role-name AmcUploaderS3TriggerRole
$ aws iam delete-role --role-name AmcUploaderS3TriggerRole
Click
To setup this request as an Amazon S3 triggered Lambda function, click
Invalid dataset. Verify that your dataset definition is complete.
{{ endpoint_request(endpoint).status }}