{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Building an Automated Pipeline for Amazon Forecast\n", "When a file is put in S3 with data for training, build a pipeline that automatically performs data import, training, forecasting, and exporting of predictions in Amazon Forecast.\n", "We use Step Functions and Lambda." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Configure SageMaker role\n", "Attach policies for the services you use and also set up trust relationships, as shown in the figure below." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![IAMroles_Permissions](https://user-images.githubusercontent.com/27226946/89102049-7d18e380-d440-11ea-91a6-6ab2c7e63870.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![IAMroles_TrustRelationships](https://user-images.githubusercontent.com/27226946/89102054-84d88800-d440-11ea-9199-0583be09aa1c.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Edit trust relationship should be written as follows." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "{\n", " \"Version\": \"2012-10-17\",\n", " \"Statement\": [\n", " {\n", " \"Effect\": \"Allow\",\n", " \"Principal\": {\n", " \"Service\": \"sagemaker.amazonaws.com\"\n", " },\n", " \"Action\": \"sts:AssumeRole\"\n", " },\n", " {\n", " \"Effect\": \"Allow\",\n", " \"Principal\": {\n", " \"Service\": \"forecast.amazonaws.com\"\n", " },\n", " \"Action\": \"sts:AssumeRole\"\n", " },\n", " {\n", " \"Effect\": \"Allow\",\n", " \"Principal\": {\n", " \"Service\": \"lambda.amazonaws.com\"\n", " },\n", " \"Action\": \"sts:AssumeRole\"\n", " },\n", " {\n", " \"Effect\": \"Allow\",\n", " \"Principal\": {\n", " \"Service\": \"events.amazonaws.com\"\n", " },\n", " \"Action\": \"sts:AssumeRole\"\n", " },\n", " {\n", " \"Effect\": \"Allow\",\n", " \"Principal\": {\n", " \"Service\": \"states.amazonaws.com\"\n", " },\n", " \"Action\": \"sts:AssumeRole\"\n", " }\n", " ]\n", "}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Getting SageMaker role" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "from sagemaker import get_execution_role\n", "\n", "role_sm = get_execution_role()" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'arn:aws:iam::805433377179:role/service-role/AmazonSageMaker-ExecutionRole-20200716T084970'" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "role_sm" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 1.make Lambda function\n", "We will create a Lambda function. The files we will use are located in lambdas/." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "import boto3" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "lambda_ = boto3.client('lambda')" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " adding: datasetimport.py (deflated 54%)\n" ] }, { "data": { "text/plain": [ "{'ResponseMetadata': {'RequestId': 'b45fbdeb-4c54-4706-aa18-5184e4ceff20',\n", " 'HTTPStatusCode': 201,\n", " 'HTTPHeaders': {'date': 'Sat, 08 Aug 2020 05:50:40 GMT',\n", " 'content-type': 'application/json',\n", " 'content-length': '893',\n", " 'connection': 'keep-alive',\n", " 'x-amzn-requestid': 'b45fbdeb-4c54-4706-aa18-5184e4ceff20'},\n", " 'RetryAttempts': 0},\n", " 'FunctionName': 'datasetimport',\n", " 'FunctionArn': 'arn:aws:lambda:us-east-1:805433377179:function:datasetimport',\n", " 'Runtime': 'python3.7',\n", " 'Role': 'arn:aws:iam::805433377179:role/service-role/AmazonSageMaker-ExecutionRole-20200716T084970',\n", " 'Handler': 'datasetimport.lambda_handler',\n", " 'CodeSize': 563,\n", " 'Description': '',\n", " 'Timeout': 900,\n", " 'MemorySize': 3008,\n", " 'LastModified': '2020-08-08T05:50:40.778+0000',\n", " 'CodeSha256': 'PoxpcyDxzFemgk0SYn6f9o5ovhfiFrt0/6ql8O4JS58=',\n", " 'Version': '$LATEST',\n", " 'TracingConfig': {'Mode': 'PassThrough'},\n", " 'RevisionId': 'a72e6c64-145e-44b1-a257-b1dfd80cd91b',\n", " 'State': 'Active',\n", " 'LastUpdateStatus': 'Successful'}" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "!rm -f lambdas/createdatasetimport/datasetimport.zip\n", "!cd lambdas/createdatasetimport; zip -r datasetimport .\n", "\n", "zip_file = open(\"lambdas/createdatasetimport/datasetimport.zip\", \"rb\").read()\n", "\n", "\n", "lambda_.create_function(\n", " FunctionName=\"datasetimport\",\n", " Runtime=\"python3.7\",\n", " Role=role_sm,\n", " Handler=\"datasetimport.lambda_handler\",\n", " Code={\"ZipFile\": zip_file},\n", " Timeout=60*15,\n", " MemorySize=3008\n", ")" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " adding: getstatusimport.py (deflated 46%)\n" ] }, { "data": { "text/plain": [ "{'ResponseMetadata': {'RequestId': '4b18b01c-e16e-49c0-b96c-e98fc546d614',\n", " 'HTTPStatusCode': 201,\n", " 'HTTPHeaders': {'date': 'Sat, 08 Aug 2020 05:50:41 GMT',\n", " 'content-type': 'application/json',\n", " 'content-length': '899',\n", " 'connection': 'keep-alive',\n", " 'x-amzn-requestid': '4b18b01c-e16e-49c0-b96c-e98fc546d614'},\n", " 'RetryAttempts': 0},\n", " 'FunctionName': 'getstatusimport',\n", " 'FunctionArn': 'arn:aws:lambda:us-east-1:805433377179:function:getstatusimport',\n", " 'Runtime': 'python3.7',\n", " 'Role': 'arn:aws:iam::805433377179:role/service-role/AmazonSageMaker-ExecutionRole-20200716T084970',\n", " 'Handler': 'getstatusimport.lambda_handler',\n", " 'CodeSize': 387,\n", " 'Description': '',\n", " 'Timeout': 900,\n", " 'MemorySize': 3008,\n", " 'LastModified': '2020-08-08T05:50:41.497+0000',\n", " 'CodeSha256': '4N7tY/qqjlNGRU9NEkinRSOSPECeFJT+uB15WGceE28=',\n", " 'Version': '$LATEST',\n", " 'TracingConfig': {'Mode': 'PassThrough'},\n", " 'RevisionId': '9fc8bb6b-fe49-48f6-94e2-76f3edfa7243',\n", " 'State': 'Active',\n", " 'LastUpdateStatus': 'Successful'}" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "!rm -f lambdas/GetStatusImport/getstatusimport.zip\n", "!cd lambdas/GetStatusImport; zip -r getstatusimport .\n", "\n", "zip_file = open(\"lambdas/GetStatusImport/getstatusimport.zip\", \"rb\").read()\n", "\n", "lambda_.create_function(\n", " FunctionName=\"getstatusimport\",\n", " Runtime=\"python3.7\",\n", " Role=role_sm,\n", " Handler=\"getstatusimport.lambda_handler\",\n", " Code={\"ZipFile\": zip_file},\n", " Timeout=60*15,\n", " MemorySize=3008\n", ")" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " adding: predictor.py (deflated 56%)\n" ] }, { "data": { "text/plain": [ "{'ResponseMetadata': {'RequestId': '8daf3893-6ea8-4af8-ae72-ea3f12cd75a0',\n", " 'HTTPStatusCode': 201,\n", " 'HTTPHeaders': {'date': 'Sat, 08 Aug 2020 05:50:42 GMT',\n", " 'content-type': 'application/json',\n", " 'content-length': '881',\n", " 'connection': 'keep-alive',\n", " 'x-amzn-requestid': '8daf3893-6ea8-4af8-ae72-ea3f12cd75a0'},\n", " 'RetryAttempts': 0},\n", " 'FunctionName': 'predictor',\n", " 'FunctionArn': 'arn:aws:lambda:us-east-1:805433377179:function:predictor',\n", " 'Runtime': 'python3.7',\n", " 'Role': 'arn:aws:iam::805433377179:role/service-role/AmazonSageMaker-ExecutionRole-20200716T084970',\n", " 'Handler': 'predictor.lambda_handler',\n", " 'CodeSize': 645,\n", " 'Description': '',\n", " 'Timeout': 900,\n", " 'MemorySize': 3008,\n", " 'LastModified': '2020-08-08T05:50:42.562+0000',\n", " 'CodeSha256': 'ip8vS4n9cWDNFEfKspM/g8DXIi/it/l1ZcqgcsXsc2c=',\n", " 'Version': '$LATEST',\n", " 'TracingConfig': {'Mode': 'PassThrough'},\n", " 'RevisionId': '0b7c6138-0747-4cb2-8d6c-a4da51330ab1',\n", " 'State': 'Active',\n", " 'LastUpdateStatus': 'Successful'}" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "!rm -f lambdas/createpredictor/predictor.zip\n", "!cd lambdas/createpredictor; zip -r predictor .\n", "\n", "zip_file = open(\"lambdas/createpredictor/predictor.zip\", \"rb\").read()\n", "\n", "lambda_.create_function(\n", " FunctionName=\"predictor\",\n", " Runtime=\"python3.7\",\n", " Role=role_sm,\n", " Handler=\"predictor.lambda_handler\",\n", " Code={\"ZipFile\": zip_file},\n", " Timeout=60*15,\n", " MemorySize=3008\n", ")" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " adding: getstatuspredictor.py (deflated 46%)\n" ] }, { "data": { "text/plain": [ "{'ResponseMetadata': {'RequestId': '091a4a42-111c-4da6-99b8-c33d17bed07b',\n", " 'HTTPStatusCode': 201,\n", " 'HTTPHeaders': {'date': 'Sat, 08 Aug 2020 05:50:43 GMT',\n", " 'content-type': 'application/json',\n", " 'content-length': '908',\n", " 'connection': 'keep-alive',\n", " 'x-amzn-requestid': '091a4a42-111c-4da6-99b8-c33d17bed07b'},\n", " 'RetryAttempts': 0},\n", " 'FunctionName': 'getstatuspredictor',\n", " 'FunctionArn': 'arn:aws:lambda:us-east-1:805433377179:function:getstatuspredictor',\n", " 'Runtime': 'python3.7',\n", " 'Role': 'arn:aws:iam::805433377179:role/service-role/AmazonSageMaker-ExecutionRole-20200716T084970',\n", " 'Handler': 'getstatuspredictor.lambda_handler',\n", " 'CodeSize': 382,\n", " 'Description': '',\n", " 'Timeout': 900,\n", " 'MemorySize': 3008,\n", " 'LastModified': '2020-08-08T05:50:43.596+0000',\n", " 'CodeSha256': 'JnAo+fVvxxjk1WGjnqUdNW4ozQR0CmGLeLFJJW4DBLQ=',\n", " 'Version': '$LATEST',\n", " 'TracingConfig': {'Mode': 'PassThrough'},\n", " 'RevisionId': 'c23e3bfc-eeb0-46fd-8d6b-a80bb761be1f',\n", " 'State': 'Active',\n", " 'LastUpdateStatus': 'Successful'}" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "!rm -f lambdas/GetStatusPredictor/getstatuspredictor.zip\n", "!cd lambdas/GetStatusPredictor; zip -r getstatuspredictor .\n", "\n", "zip_file = open(\"lambdas/GetStatusPredictor/getstatuspredictor.zip\", \"rb\").read()\n", "\n", "lambda_.create_function(\n", " FunctionName=\"getstatuspredictor\",\n", " Runtime=\"python3.7\",\n", " Role=role_sm,\n", " Handler=\"getstatuspredictor.lambda_handler\",\n", " Code={\"ZipFile\": zip_file},\n", " Timeout=60*15,\n", " MemorySize=3008\n", ")" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " adding: forecast.py (deflated 45%)\n" ] }, { "data": { "text/plain": [ "{'ResponseMetadata': {'RequestId': '8a38d610-48b8-4378-9140-1799796c6c7f',\n", " 'HTTPStatusCode': 201,\n", " 'HTTPHeaders': {'date': 'Sat, 08 Aug 2020 05:50:44 GMT',\n", " 'content-type': 'application/json',\n", " 'content-length': '878',\n", " 'connection': 'keep-alive',\n", " 'x-amzn-requestid': '8a38d610-48b8-4378-9140-1799796c6c7f'},\n", " 'RetryAttempts': 0},\n", " 'FunctionName': 'forecast',\n", " 'FunctionArn': 'arn:aws:lambda:us-east-1:805433377179:function:forecast',\n", " 'Runtime': 'python3.7',\n", " 'Role': 'arn:aws:iam::805433377179:role/service-role/AmazonSageMaker-ExecutionRole-20200716T084970',\n", " 'Handler': 'forecast.lambda_handler',\n", " 'CodeSize': 374,\n", " 'Description': '',\n", " 'Timeout': 900,\n", " 'MemorySize': 3008,\n", " 'LastModified': '2020-08-08T05:50:44.543+0000',\n", " 'CodeSha256': 'CvuC6H1Ce7+QbYW8JHYuRJDdhVVizyDfjNeOr1Chb/4=',\n", " 'Version': '$LATEST',\n", " 'TracingConfig': {'Mode': 'PassThrough'},\n", " 'RevisionId': 'd702c495-93df-48d0-9610-44a84a43e45a',\n", " 'State': 'Active',\n", " 'LastUpdateStatus': 'Successful'}" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "!rm -f lambdas/createforecast/forecast.zip\n", "!cd lambdas/createforecast; zip -r forecast .\n", "\n", "zip_file = open(\"lambdas/createforecast/forecast.zip\", \"rb\").read()\n", "\n", "lambda_.create_function(\n", " FunctionName=\"forecast\",\n", " Runtime=\"python3.7\",\n", " Role=role_sm,\n", " Handler=\"forecast.lambda_handler\",\n", " Code={\"ZipFile\": zip_file},\n", " Timeout=60*15,\n", " MemorySize=3008\n", ")" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " adding: getstatusforecast.py (deflated 47%)\n" ] }, { "data": { "text/plain": [ "{'ResponseMetadata': {'RequestId': 'f958c6bc-7c73-404e-8bf9-dbdfd6b7cb1c',\n", " 'HTTPStatusCode': 201,\n", " 'HTTPHeaders': {'date': 'Sat, 08 Aug 2020 05:50:45 GMT',\n", " 'content-type': 'application/json',\n", " 'content-length': '905',\n", " 'connection': 'keep-alive',\n", " 'x-amzn-requestid': 'f958c6bc-7c73-404e-8bf9-dbdfd6b7cb1c'},\n", " 'RetryAttempts': 0},\n", " 'FunctionName': 'getstatusforecast',\n", " 'FunctionArn': 'arn:aws:lambda:us-east-1:805433377179:function:getstatusforecast',\n", " 'Runtime': 'python3.7',\n", " 'Role': 'arn:aws:iam::805433377179:role/service-role/AmazonSageMaker-ExecutionRole-20200716T084970',\n", " 'Handler': 'getstatusforecast.lambda_handler',\n", " 'CodeSize': 374,\n", " 'Description': '',\n", " 'Timeout': 900,\n", " 'MemorySize': 3008,\n", " 'LastModified': '2020-08-08T05:50:45.439+0000',\n", " 'CodeSha256': '/zOy8Vh2h6IOAxpWKIsdEVmFBzkd4PhdLnO0e40FPyQ=',\n", " 'Version': '$LATEST',\n", " 'TracingConfig': {'Mode': 'PassThrough'},\n", " 'RevisionId': '1170e79a-9e8e-4f53-ae81-81530437f17c',\n", " 'State': 'Active',\n", " 'LastUpdateStatus': 'Successful'}" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "!rm -f lambdas/GetStatusForecast/getstatusforecast.zip\n", "!cd lambdas/GetStatusForecast; zip -r getstatusforecast .\n", "\n", "zip_file = open(\"lambdas/GetStatusForecast/getstatusforecast.zip\", \"rb\").read()\n", "\n", "lambda_.create_function(\n", " FunctionName=\"getstatusforecast\",\n", " Runtime=\"python3.7\",\n", " Role=role_sm,\n", " Handler=\"getstatusforecast.lambda_handler\",\n", " Code={\"ZipFile\": zip_file},\n", " Timeout=60*15,\n", " MemorySize=3008\n", ")" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " adding: forecastexportjob.py (deflated 53%)\n" ] }, { "data": { "text/plain": [ "{'ResponseMetadata': {'RequestId': 'b282734e-be76-4071-8773-f402505d7fcf',\n", " 'HTTPStatusCode': 201,\n", " 'HTTPHeaders': {'date': 'Sat, 08 Aug 2020 05:50:46 GMT',\n", " 'content-type': 'application/json',\n", " 'content-length': '905',\n", " 'connection': 'keep-alive',\n", " 'x-amzn-requestid': 'b282734e-be76-4071-8773-f402505d7fcf'},\n", " 'RetryAttempts': 0},\n", " 'FunctionName': 'forecastexportjob',\n", " 'FunctionArn': 'arn:aws:lambda:us-east-1:805433377179:function:forecastexportjob',\n", " 'Runtime': 'python3.7',\n", " 'Role': 'arn:aws:iam::805433377179:role/service-role/AmazonSageMaker-ExecutionRole-20200716T084970',\n", " 'Handler': 'forecastexportjob.lambda_handler',\n", " 'CodeSize': 508,\n", " 'Description': '',\n", " 'Timeout': 900,\n", " 'MemorySize': 3008,\n", " 'LastModified': '2020-08-08T05:50:46.474+0000',\n", " 'CodeSha256': 'saHaLFRozjXJiN8TkSn0N9Rmo0Vveic+Urj2LzdKWh0=',\n", " 'Version': '$LATEST',\n", " 'TracingConfig': {'Mode': 'PassThrough'},\n", " 'RevisionId': 'b76bf9cd-1c8f-4de3-8136-c65272bc237b',\n", " 'State': 'Active',\n", " 'LastUpdateStatus': 'Successful'}" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "!rm -f lambdas/createforecastexportjob/forecastexportjob.zip\n", "!cd lambdas/createforecastexportjob; zip -r forecastexportjob .\n", "\n", "zip_file = open(\"lambdas/createforecastexportjob/forecastexportjob.zip\", \"rb\").read()\n", "\n", "lambda_.create_function(\n", " FunctionName=\"forecastexportjob\",\n", " Runtime=\"python3.7\",\n", " Role=role_sm,\n", " Handler=\"forecastexportjob.lambda_handler\",\n", " Code={\"ZipFile\": zip_file},\n", " Timeout=60*15,\n", " MemorySize=3008\n", ")" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " adding: getstatusforecastexportjob.py (deflated 47%)\n" ] }, { "data": { "text/plain": [ "{'ResponseMetadata': {'RequestId': 'b2978cbf-0db6-4129-88ec-ef2c1b027da0',\n", " 'HTTPStatusCode': 201,\n", " 'HTTPHeaders': {'date': 'Sat, 08 Aug 2020 05:50:47 GMT',\n", " 'content-type': 'application/json',\n", " 'content-length': '932',\n", " 'connection': 'keep-alive',\n", " 'x-amzn-requestid': 'b2978cbf-0db6-4129-88ec-ef2c1b027da0'},\n", " 'RetryAttempts': 0},\n", " 'FunctionName': 'getstatusforecastexportjob',\n", " 'FunctionArn': 'arn:aws:lambda:us-east-1:805433377179:function:getstatusforecastexportjob',\n", " 'Runtime': 'python3.7',\n", " 'Role': 'arn:aws:iam::805433377179:role/service-role/AmazonSageMaker-ExecutionRole-20200716T084970',\n", " 'Handler': 'getstatusforecastexportjob.lambda_handler',\n", " 'CodeSize': 406,\n", " 'Description': '',\n", " 'Timeout': 900,\n", " 'MemorySize': 3008,\n", " 'LastModified': '2020-08-08T05:50:47.451+0000',\n", " 'CodeSha256': 'vexcWxFlnRrXANyWYRS+dDmcn6gFvyUX9WC9sApFD4E=',\n", " 'Version': '$LATEST',\n", " 'TracingConfig': {'Mode': 'PassThrough'},\n", " 'RevisionId': 'df40e2c6-2dc6-4948-b7a6-2f81269cb0ca',\n", " 'State': 'Active',\n", " 'LastUpdateStatus': 'Successful'}" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "!rm -f lambdas/GetStatusForecastExportJob/getstatusforecastexportjob.zip\n", "!cd lambdas/GetStatusForecastExportJob; zip -r getstatusforecastexportjob .\n", "\n", "zip_file = open(\"lambdas/GetStatusForecastExportJob/getstatusforecastexportjob.zip\", \"rb\").read()\n", "\n", "lambda_.create_function(\n", " FunctionName=\"getstatusforecastexportjob\",\n", " Runtime=\"python3.7\",\n", " Role=role_sm,\n", " Handler=\"getstatusforecastexportjob.lambda_handler\",\n", " Code={\"ZipFile\": zip_file},\n", " Timeout=60*15,\n", " MemorySize=3008\n", ")" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " adding: notifyuser.py (deflated 12%)\n" ] }, { "data": { "text/plain": [ "{'ResponseMetadata': {'RequestId': '1b0ccd82-cc2a-409d-9f0f-389c7c1533ac',\n", " 'HTTPStatusCode': 201,\n", " 'HTTPHeaders': {'date': 'Sat, 08 Aug 2020 05:50:48 GMT',\n", " 'content-type': 'application/json',\n", " 'content-length': '884',\n", " 'connection': 'keep-alive',\n", " 'x-amzn-requestid': '1b0ccd82-cc2a-409d-9f0f-389c7c1533ac'},\n", " 'RetryAttempts': 0},\n", " 'FunctionName': 'notifyuser',\n", " 'FunctionArn': 'arn:aws:lambda:us-east-1:805433377179:function:notifyuser',\n", " 'Runtime': 'python3.7',\n", " 'Role': 'arn:aws:iam::805433377179:role/service-role/AmazonSageMaker-ExecutionRole-20200716T084970',\n", " 'Handler': 'notifyuser.lambda_handler',\n", " 'CodeSize': 263,\n", " 'Description': '',\n", " 'Timeout': 900,\n", " 'MemorySize': 3008,\n", " 'LastModified': '2020-08-08T05:50:48.430+0000',\n", " 'CodeSha256': '1tZH3rulcoyzKrdIvZSNIBVPTAe+/dn78gtSbuknHkI=',\n", " 'Version': '$LATEST',\n", " 'TracingConfig': {'Mode': 'PassThrough'},\n", " 'RevisionId': '50de254a-e4ab-44a4-bc7a-a259d1d08c79',\n", " 'State': 'Active',\n", " 'LastUpdateStatus': 'Successful'}" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "!rm -f lambdas/NotifyUser/notifyuser.zip\n", "!cd lambdas/NotifyUser; zip -r notifyuser .\n", "\n", "zip_file = open(\"lambdas/NotifyUser/notifyuser.zip\", \"rb\").read()\n", "\n", "lambda_.create_function(\n", " FunctionName=\"notifyuser\",\n", " Runtime=\"python3.7\",\n", " Role=role_sm,\n", " Handler=\"notifyuser.lambda_handler\",\n", " Code={\"ZipFile\": zip_file},\n", " Timeout=60*15,\n", " MemorySize=3008\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 2. Step Functins: Create state machine\n", "The definition of the state machine is created, and the creation of the state machine is performed based on the definition." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "import sagemaker" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "sagemaker_session = sagemaker.Session()" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'us-east-1'" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sagemaker_session.boto_region_name" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [], "source": [ "sts = boto3.client('sts')\n", "id_info = sts.get_caller_identity()" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'805433377179'" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "id_info['Account']" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "import json" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "def_sfn={\n", " \"Comment\": \"Amazon Forecast example of the Amazon States Language using an AWS Lambda Function\",\n", " \"StartAt\": \"datasetimport\",\n", " \"States\": {\n", " \"datasetimport\": {\n", " \"Type\": \"Task\",\n", " \"InputPath\":\"$\",\n", " \"Resource\": \"arn:aws:lambda:\" + sagemaker_session.boto_region_name + \":\" + id_info['Account'] + \":function:datasetimport\",\n", " \"ResultPath\":\"$\",\n", " \"Next\": \"GetStatusImport\"\n", " },\n", " \"GetStatusImport\": {\n", " \"Type\": \"Task\",\n", " \"InputPath\":\"$\",\n", " \"Resource\": \"arn:aws:lambda:\" + sagemaker_session.boto_region_name + \":\" + id_info['Account'] + \":function:getstatusimport\",\n", " \"ResultPath\":\"$\",\n", " \"Next\": \"CheckStatusImport\"\n", " },\n", " \"CheckStatusImport\": {\n", " \"Type\": \"Choice\",\n", " \"InputPath\":\"$\",\n", " \"Choices\": [\n", " {\n", " \"Variable\": \"$.is_active_import\",\n", " \"BooleanEquals\": True,\n", " \"Next\": \"predictor\"\n", " }\n", " ],\n", " \"Default\": \"SleepCheckStatusImport\"\n", " },\n", " \"SleepCheckStatusImport\": {\n", " \"Type\": \"Wait\",\n", " \"Seconds\": 300,\n", " \"Next\": \"GetStatusImport\"\n", " },\n", " \"predictor\": {\n", " \"Type\": \"Task\",\n", " \"InputPath\":\"$\",\n", " \"Resource\": \"arn:aws:lambda:\" + sagemaker_session.boto_region_name + \":\" + id_info['Account'] + \":function:predictor\",\n", " \"ResultPath\":\"$\",\n", " \"Next\": \"GetStatusPredictor\"\n", " },\n", " \"GetStatusPredictor\": {\n", " \"Type\": \"Task\",\n", " \"InputPath\":\"$\",\n", " \"Resource\": \"arn:aws:lambda:\" + sagemaker_session.boto_region_name + \":\" + id_info['Account'] + \":function:getstatuspredictor\",\n", " \"ResultPath\":\"$\",\n", " \"Next\": \"CheckStatusPredictor\"\n", " },\n", " \"CheckStatusPredictor\": {\n", " \"Type\": \"Choice\",\n", " \"InputPath\":\"$\",\n", " \"Choices\": [\n", " {\n", " \"Variable\": \"$.is_active_predictor\",\n", " \"BooleanEquals\": True,\n", " \"Next\": \"forecast\"\n", " }\n", " ],\n", " \"Default\": \"SleepCheckStatusPredictor\"\n", " },\n", " \"SleepCheckStatusPredictor\": {\n", " \"Type\": \"Wait\",\n", " \"Seconds\": 300,\n", " \"Next\": \"GetStatusPredictor\"\n", " },\n", " \"forecast\": {\n", " \"Type\": \"Task\",\n", " \"InputPath\":\"$\",\n", " \"Resource\": \"arn:aws:lambda:\" + sagemaker_session.boto_region_name + \":\" + id_info['Account'] + \":function:forecast\",\n", " \"ResultPath\":\"$\",\n", " \"Next\": \"GetStatusForecast\"\n", " },\n", " \"GetStatusForecast\": {\n", " \"Type\": \"Task\",\n", " \"InputPath\":\"$\",\n", " \"Resource\": \"arn:aws:lambda:\" + sagemaker_session.boto_region_name + \":\" + id_info['Account'] + \":function:getstatusforecast\",\n", " \"ResultPath\":\"$\",\n", " \"Next\": \"CheckStatusForecast\"\n", " }, \n", " \"CheckStatusForecast\": {\n", " \"Type\": \"Choice\",\n", " \"InputPath\":\"$\",\n", " \"Choices\": [\n", " {\n", " \"Variable\": \"$.is_active_forecast\",\n", " \"BooleanEquals\": True,\n", " \"Next\": \"forecastexportjob\"\n", " }\n", " ],\n", " \"Default\": \"SleepCheckStatusForecast\"\n", " },\n", " \"SleepCheckStatusForecast\": {\n", " \"Type\": \"Wait\",\n", " \"Seconds\": 300,\n", " \"Next\": \"GetStatusForecast\"\n", " },\n", " \"forecastexportjob\": {\n", " \"Type\": \"Task\",\n", " \"InputPath\":\"$\",\n", " \"Resource\": \"arn:aws:lambda:\" + sagemaker_session.boto_region_name + \":\" + id_info['Account'] + \":function:forecastexportjob\",\n", " \"ResultPath\":\"$\",\n", " \"Next\": \"GetStatusForecastExportjob\"\n", " },\n", " \"GetStatusForecastExportjob\": {\n", " \"Type\": \"Task\",\n", " \"InputPath\":\"$\",\n", " \"Resource\": \"arn:aws:lambda:\" + sagemaker_session.boto_region_name + \":\" + id_info['Account'] + \":function:getstatusforecastexportjob\",\n", " \"ResultPath\":\"$\",\n", " \"Next\": \"CheckStatusExport\"\n", " }, \n", " \"CheckStatusExport\": {\n", " \"Type\": \"Choice\",\n", " \"InputPath\":\"$\",\n", " \"Choices\": [\n", " {\n", " \"Variable\": \"$.is_active_export\",\n", " \"BooleanEquals\": True,\n", " \"Next\": \"NotifyUser\"\n", " }\n", " ],\n", " \"Default\": \"SleepCheckStatusExport\"\n", " },\n", " \"SleepCheckStatusExport\": {\n", " \"Type\": \"Wait\",\n", " \"Seconds\": 300,\n", " \"Next\": \"GetStatusForecastExportjob\"\n", " },\n", " \"NotifyUser\": {\n", " \"Type\": \"Task\",\n", " \"InputPath\":\"$\",\n", " \"Resource\": \"arn:aws:lambda:\" + sagemaker_session.boto_region_name + \":\" + id_info['Account'] + \":function:notifyuser\",\n", " \"ResultPath\":\"$\",\n", " \"End\": True\n", " }\n", " }\n", "}\n" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [], "source": [ "with open('./definition.json', 'w') as f:\n", " json.dump(def_sfn, f, indent=2, ensure_ascii=False)" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [], "source": [ "import boto3\n", "sfn = boto3.client('stepfunctions')" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'stateMachineArn': 'arn:aws:states:us-east-1:805433377179:stateMachine:demo-forecast',\n", " 'creationDate': datetime.datetime(2020, 8, 8, 5, 50, 56, 173000, tzinfo=tzlocal()),\n", " 'ResponseMetadata': {'RequestId': '78b2c992-e93e-4ef2-9bba-9f53b6a455a3',\n", " 'HTTPStatusCode': 200,\n", " 'HTTPHeaders': {'x-amzn-requestid': '78b2c992-e93e-4ef2-9bba-9f53b6a455a3',\n", " 'content-type': 'application/x-amz-json-1.0',\n", " 'content-length': '118'},\n", " 'RetryAttempts': 0}}" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sfn.create_state_machine(\n", " name=\"demo-forecast\",\n", " definition=open(\"definition.json\").read(),\n", " roleArn=role_sm\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 3.AWS CloudTrail :create trail \n", "https://docs.aws.amazon.com/step-functions/latest/dg/tutorial-cloudwatch-events-s3.html\n", "\n", "Configure CloudTrail and CloudWatch Events to run Step Functions by triggering S3 object placement as described in this guide.\n", "\n", "https://boto3.amazonaws.com/v1/documentation/api/1.9.42/reference/services/cloudtrail.html#CloudTrail.Client.create_trail" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "boto3==1.14.16\n" ] } ], "source": [ "! pip freeze | grep boto3" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [], "source": [ "cloudtrail = boto3.client('cloudtrail')" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "805433377179\n" ] } ], "source": [ "sts = boto3.client('sts')\n", "id_info = sts.get_caller_identity()\n", "print(id_info['Account'])" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [], "source": [ "bucket_name = 'demo-forecast-' + id_info['Account']" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'demo-forecast-805433377179'" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "bucket_name" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [], "source": [ "output_trail_bucket = bucket_name + '-trail'" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'ResponseMetadata': {'RequestId': '72B0723FF5865BB6',\n", " 'HostId': 'V7JrH3tzZ1f0uNEsxfZNb3DwtkvjdK8pXC4qKisMxwnuqGKUm8AR71xzezA/VZli9E8gyr0kZOE=',\n", " 'HTTPStatusCode': 200,\n", " 'HTTPHeaders': {'x-amz-id-2': 'V7JrH3tzZ1f0uNEsxfZNb3DwtkvjdK8pXC4qKisMxwnuqGKUm8AR71xzezA/VZli9E8gyr0kZOE=',\n", " 'x-amz-request-id': '72B0723FF5865BB6',\n", " 'date': 'Sat, 08 Aug 2020 05:51:02 GMT',\n", " 'location': '/demo-forecast-805433377179-trail',\n", " 'content-length': '0',\n", " 'server': 'AmazonS3'},\n", " 'RetryAttempts': 0},\n", " 'Location': '/demo-forecast-805433377179-trail'}" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s3 = boto3.client('s3')\n", "s3.create_bucket(Bucket=output_trail_bucket)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## set bucket policy\n", "setting policy with boto3 \n", "https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-example-bucket-policies.html" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [], "source": [ "import json" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [], "source": [ "bucket_policy = {\n", " \"Version\": \"2012-10-17\",\n", " \"Statement\": [\n", " {\n", " \"Sid\": \"AWSCloudTrailAclCheck20150319\",\n", " \"Effect\": \"Allow\",\n", " \"Principal\": {\n", " \"Service\": \"cloudtrail.amazonaws.com\"\n", " },\n", " \"Action\": \"s3:GetBucketAcl\",\n", " \"Resource\": \"arn:aws:s3:::\" + bucket_name + \"-trail\"\n", " },\n", " {\n", " \"Sid\": \"AWSCloudTrailWrite20150319\",\n", " \"Effect\": \"Allow\",\n", " \"Principal\": {\n", " \"Service\": \"cloudtrail.amazonaws.com\"\n", " },\n", " \"Action\": \"s3:PutObject\",\n", " \"Resource\": \"arn:aws:s3:::\" + bucket_name + \"-trail/AWSLogs/\" + id_info['Account'] + \"/*\",\n", " \"Condition\": {\n", " \"StringEquals\": {\n", " \"s3:x-amz-acl\": \"bucket-owner-full-control\"\n", " }\n", " }\n", " }\n", " ]\n", "}\n", "\n", "# Convert the policy from JSON dict to string\n", "bucket_policy = json.dumps(bucket_policy)" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'ResponseMetadata': {'RequestId': 'C959349CFB2936F4',\n", " 'HostId': 'vTZwu7SKmG+IrE1gxQDyAW7u96qzSD/mxBl8wK1WuZD8+8Ix10PJR80NfeHEVIJ38RE6h5Dh2ec=',\n", " 'HTTPStatusCode': 204,\n", " 'HTTPHeaders': {'x-amz-id-2': 'vTZwu7SKmG+IrE1gxQDyAW7u96qzSD/mxBl8wK1WuZD8+8Ix10PJR80NfeHEVIJ38RE6h5Dh2ec=',\n", " 'x-amz-request-id': 'C959349CFB2936F4',\n", " 'date': 'Sat, 08 Aug 2020 05:51:06 GMT',\n", " 'server': 'AmazonS3'},\n", " 'RetryAttempts': 0}}" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Set the new policy\n", "s3 = boto3.client('s3')\n", "s3.put_bucket_policy(Bucket=output_trail_bucket, Policy=bucket_policy)" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'Name': 'forecast-trail',\n", " 'S3BucketName': 'demo-forecast-805433377179-trail',\n", " 'IncludeGlobalServiceEvents': True,\n", " 'IsMultiRegionTrail': False,\n", " 'TrailARN': 'arn:aws:cloudtrail:us-east-1:805433377179:trail/forecast-trail',\n", " 'LogFileValidationEnabled': True,\n", " 'IsOrganizationTrail': False,\n", " 'ResponseMetadata': {'RequestId': 'db78ab21-5808-40dc-981f-efd1b47aa45e',\n", " 'HTTPStatusCode': 200,\n", " 'HTTPHeaders': {'x-amzn-requestid': 'db78ab21-5808-40dc-981f-efd1b47aa45e',\n", " 'content-type': 'application/x-amz-json-1.1',\n", " 'content-length': '272',\n", " 'date': 'Sat, 08 Aug 2020 05:51:05 GMT'},\n", " 'RetryAttempts': 0}}" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "cloudtrail.create_trail(\n", " Name='forecast-trail',\n", " S3BucketName=output_trail_bucket,\n", " EnableLogFileValidation=True\n", ")" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'TrailARN': 'arn:aws:cloudtrail:us-east-1:805433377179:trail/forecast-trail',\n", " 'EventSelectors': [{'ReadWriteType': 'All',\n", " 'IncludeManagementEvents': True,\n", " 'DataResources': [{'Type': 'AWS::S3::Object',\n", " 'Values': ['arn:aws:s3:::demo-forecast-805433377179/input']}],\n", " 'ExcludeManagementEventSources': []}],\n", " 'ResponseMetadata': {'RequestId': '3fb976be-ca56-4270-8660-c5a36f2efc86',\n", " 'HTTPStatusCode': 200,\n", " 'HTTPHeaders': {'x-amzn-requestid': '3fb976be-ca56-4270-8660-c5a36f2efc86',\n", " 'content-type': 'application/x-amz-json-1.1',\n", " 'content-length': '290',\n", " 'date': 'Sat, 08 Aug 2020 05:51:06 GMT'},\n", " 'RetryAttempts': 0}}" ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" } ], "source": [ "cloudtrail.put_event_selectors(\n", " TrailName='forecast-trail',\n", " EventSelectors=[\n", " {\n", " 'ReadWriteType': 'All',\n", " 'IncludeManagementEvents': True,\n", " 'DataResources': [\n", " {\n", " 'Type': 'AWS::S3::Object',\n", " 'Values': [\n", " f'arn:aws:s3:::{bucket_name}/input',\n", " ]\n", " },\n", " ]\n", " },\n", " ]\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### enable logging\n", "https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudtrail.html#CloudTrail.Client.start_logging" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'ResponseMetadata': {'RequestId': '97173bec-db69-44aa-9697-d03ba43c8608',\n", " 'HTTPStatusCode': 200,\n", " 'HTTPHeaders': {'x-amzn-requestid': '97173bec-db69-44aa-9697-d03ba43c8608',\n", " 'content-type': 'application/x-amz-json-1.1',\n", " 'content-length': '2',\n", " 'date': 'Sat, 08 Aug 2020 05:51:14 GMT'},\n", " 'RetryAttempts': 0}}" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "cloudtrail.start_logging(Name='forecast-trail')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 4.CloudWatch Event: build a rule\n", "https://boto3.amazonaws.com/v1/documentation/api/1.9.42/reference/services/events.html\n", "\n", "\n", "put_rule(): create rule \n", "https://boto3.amazonaws.com/v1/documentation/api/1.9.42/reference/services/events.html#CloudWatchEvents.Client.put_rule\n" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'demo-forecast-805433377179'" ] }, "execution_count": 37, "metadata": {}, "output_type": "execute_result" } ], "source": [ "bucket_name" ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [], "source": [ "cwe = boto3.client('events')" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [], "source": [ "ep_str ='{\"source\":[\"aws.s3\"], \\\n", " \"detail-type\":[\"AWS API Call via CloudTrail\"], \\\n", " \"detail\":{\"eventSource\":[\"s3.amazonaws.com\"], \\\n", " \"eventName\":[\"PutObject\", \"CompleteMultipartUpload\"], \\\n", " \"requestParameters\":{\"bucketName\":[\"'+ bucket_name + '\"]}}}'" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'RuleArn': 'arn:aws:events:us-east-1:805433377179:rule/demo-forecast',\n", " 'ResponseMetadata': {'RequestId': '864dd3e4-805f-457a-93bd-e447e6fa9ed7',\n", " 'HTTPStatusCode': 200,\n", " 'HTTPHeaders': {'x-amzn-requestid': '864dd3e4-805f-457a-93bd-e447e6fa9ed7',\n", " 'content-type': 'application/x-amz-json-1.1',\n", " 'content-length': '70',\n", " 'date': 'Sat, 08 Aug 2020 05:51:21 GMT'},\n", " 'RetryAttempts': 0}}" ] }, "execution_count": 40, "metadata": {}, "output_type": "execute_result" } ], "source": [ "cwe.put_rule(\n", " Name='demo-forecast',\n", " EventPattern=ep_str,\n", " State='ENABLED'\n", ")" ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'FailedEntryCount': 0,\n", " 'FailedEntries': [],\n", " 'ResponseMetadata': {'RequestId': '4d6db0e4-4c38-4cdf-85e8-721248701413',\n", " 'HTTPStatusCode': 200,\n", " 'HTTPHeaders': {'x-amzn-requestid': '4d6db0e4-4c38-4cdf-85e8-721248701413',\n", " 'content-type': 'application/x-amz-json-1.1',\n", " 'content-length': '41',\n", " 'date': 'Sat, 08 Aug 2020 05:51:21 GMT'},\n", " 'RetryAttempts': 0}}" ] }, "execution_count": 41, "metadata": {}, "output_type": "execute_result" } ], "source": [ "cwe.put_targets(\n", " Rule='demo-forecast',\n", " Targets=[\n", " {\n", " 'Id': 'forecast',\n", " 'Arn': \"arn:aws:states:\" + sagemaker_session.boto_region_name + \":\" + id_info['Account'] + \":stateMachine:demo-forecast\",\n", " 'RoleArn': role_sm\n", " }\n", " ]\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 5.Put additional file into S3\n", "When upload a file to S3, Step Functions runs.\n", "\n", "It may not work if you run it all at once. wait about 30 seconds. the S3 file can be uploaded repeatedly." ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [], "source": [ "s3 = boto3.resource('s3')\n", "bucket = s3.Bucket(bucket_name)\n", "\n", "bucket.upload_file('./output/tr_target_add_20091201_20101209.csv', 'input/tr_target_add_20091201_20101209.csv')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 6.Next\n", "When Step Functions is completed, the prediction results are output to S3, import the data source in QuickSight using the same procedure as before and visualize it." ] } ], "metadata": { "kernelspec": { "display_name": "conda_python3", "language": "python", "name": "conda_python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.10" } }, "nbformat": 4, "nbformat_minor": 4 }