# SageMaker Ground Truth Recipe This is a sample template for AWS SageMaker GroundTruth Lambdas to be used Custom Labeling Jobs ## Requirements * AWS CLI already configured with Administrator permission * [Python 3 installed](https://www.python.org/downloads/) * [Docker installed](https://www.docker.com/community-edition) ## Setup process ### Local development **Invoking function locally using a local sample payload** Testing GtRecipePreHumanTaskFunction Lambda ```bash sam local invoke GtRecipePreHumanTaskFunction --event resources/pre_human_task_test_event.json ``` Testing GtRecipeAnnotationConsolidationFunction Lambda Following function will fail due to permission issue. Modify annotation_consolidation_test_event.json before trying. Change "roleArn" to your ARN Change "s3Uri" to point to a JSON file. A sample S3 file is located here ```bash sam local invoke GtRecipeAnnotationConsolidationFunction --event resources/annotation_consolidation_test_event.json ``` Note : You can also test GtRecipeAnnotationConsolidationFunction in AWS UI Console after deployment ## Packaging and deployment AWS Lambda Python runtime requires a flat folder with all dependencies including the application. SAM will use `CodeUri` property to know where to look up for both application and dependencies: ```yaml ... GtRecipePreHumanTaskFunction: Type: AWS::Serverless::Function Properties: CodeUri: aws_sagemaker_ground_truth_sample_lambda/ ... ``` Firstly, we need a `S3 bucket` where we can upload our Lambda functions packaged as ZIP before we deploy anything - If you don't have a S3 bucket to store code artifacts then this is a good time to create one: ```bash aws s3 mb s3://BUCKET_NAME ``` Next, run the following command to package our Lambda function to S3: ```bash sam package \ --output-template-file packaged.yaml \ --s3-bucket REPLACE_THIS_WITH_YOUR_S3_BUCKET_NAME ``` Next, the following command will create a Cloudformation Stack and deploy your SAM resources. ```bash sam deploy \ --template-file packaged.yaml \ --stack-name gt-recipe \ --capabilities CAPABILITY_IAM ``` > **See [Serverless Application Model (SAM) HOWTO Guide](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-quick-start.html) for more details in how to get started.** After deployment is complete you can run the following command to retrieve the API Gateway Endpoint URL: ```bash aws cloudformation describe-stacks \ --stack-name gt-recipe ``` ## Fetch, tail, and filter Lambda function logs To simplify troubleshooting, SAM CLI has a command called sam logs. sam logs lets you fetch logs generated by your Lambda function from the command line. In addition to printing the logs on the terminal, this command has several nifty features to help you quickly find the bug. `NOTE`: This command works for all AWS Lambda functions; not just the ones you deploy using SAM. ```bash sam logs -n GtRecipePreHumanTaskFunction --stack-name gt-recipe --tail ``` You can find more information and examples about filtering Lambda function logs in the [SAM CLI Documentation](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-logging.html). ## Testing Next, we install test dependencies and we run `pytest` against our `tests` folder to run our initial unit tests: ```bash pip install pytest pytest-mock --user python -m pytest tests/ -v ``` ## Cleanup In order to delete our Serverless Application recently deployed you can use the following AWS CLI Command: ```bash aws cloudformation delete-stack --stack-name gt-recipe ``` ## Bringing to the next level Here are a few things you can try to get more acquainted with building serverless applications using SAM: ### Learn how SAM Build can help you with dependencies * Build the project with ``sam build --use-container`` * Invoke with ``sam local invoke GtRecipePreHumanTaskFunction --event resources/annotation_consolidation_test_event.json `` * Update tests ### Step-through debugging * **[Enable step-through debugging docs for supported runtimes]((https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-using-debugging.html))** Next, you can use AWS Serverless Application Repository to deploy ready to use Apps that go beyond hello world samples and learn how authors developed their applications: [AWS Serverless Application Repository main page](https://aws.amazon.com/serverless/serverlessrepo/) # Appendix ## Building the project [AWS Lambda requires a flat folder](https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html) with the application as well as its dependencies in deployment package. When you make changes to your source code or dependency manifest, run the following command to build your project local testing and deployment: ```bash sam build ``` If your dependencies contain native modules that need to be compiled specifically for the operating system running on AWS Lambda, use this command to build inside a Lambda-like Docker container instead: ```bash sam build --use-container ``` By default, this command writes built artifacts to `.aws-sam/build` folder. ## SAM and AWS CLI commands All commands used throughout this document ```bash # Invoke function locally with event.json as an input sam local invoke GtRecipePreHumanTaskFunction --event resources/pre_human_task_test_event.json # Create S3 bucket aws s3 mb s3://BUCKET_NAME # Package Lambda function defined locally and upload to S3 as an artifact sam package \ --output-template-file packaged.yaml \ --s3-bucket REPLACE_THIS_WITH_YOUR_S3_BUCKET_NAME # Deploy SAM template as a CloudFormation stack sam deploy \ --template-file packaged.yaml \ --stack-name gt-recipe \ --capabilities CAPABILITY_IAM # Describe Output section of CloudFormation stack previously created aws cloudformation describe-stacks \ --stack-name gt-recipe # Tail Lambda function Logs using Logical name defined in SAM Template sam logs -n GtRecipePreHumanTaskFunction --stack-name gt-recipe --tail ```