{ "cells": [ { "cell_type": "markdown", "id": "4f5f2d42", "metadata": {}, "source": [ "# [모듈 1.1] 패키징: 서비스 카탈로그 포트폴리오 및 프러덕트\n", "## 부제: Package CloudFormation Templates\n", "\n", "노트북은 필수 리소스를 생성하기 위한 다음 활동을 안내합니다.\n", "\n", "1. 사전 참고\n", "2. S3 버킷 설정\n", "3. 소스 코드 준비\n", " - 3.1. Source code zip file 생성\n", " - 3.2 소스 코드 zip 파일을 S3 버킷에 업로드\n", "4. package-cfn.yaml 배포 파일 확인\n", "5. package-cfn.yaml (CloudFormation 템플릿) 을 배포\n", " - 최종적으로 아래와 같은 파일을 S3에 업로딩 합니다.\n", " - amazon-sagemaker-reusable-components/sm-project-sc-portfolio.yaml\n", " - 카탈로그 포트폴리오, 프러덕트 생성 YAML 파일 \n", " - amazon-sagemaker-reusable-components/amazon-sagemaker-reusable-components.zip\n", " - 카탈로그 포트폴리오, 프러덕트 관련된 파일을 압축 함.\n", " - amazon-sagemaker-reusable-components/project-s3-fs-ingestion.yaml\n", " - 세이지 메이커 파이프라인 프로젝트 생성 YAML 파일\n", " - amazon-sagemaker-reusable-components/seed-code/s3-fs-ingestion-v1.0.zip\n", " - 세이지 메이커 파이프라인 프로젝트 생성시에 CodeCommit에 등록될 소스 코드\n", " \n", "---" ] }, { "cell_type": "markdown", "id": "1005574c", "metadata": {}, "source": [ "# 1. 사전 참고" ] }, { "cell_type": "markdown", "id": "1729e0ee", "metadata": {}, "source": [ "## 1.1 참고: 코드 수정 (원문 대비 변경 사항. 확인하시고 스킵하세요.)\n", "- cfn-templates/sm-project-sc-portfolio.yaml 파일을 아래와 같이 수정 함.\n", " - Amazon SageMaker execution role 이름을 얻는데 에러가 발생을 하여 수정" ] }, { "cell_type": "markdown", "id": "5f252a68", "metadata": {}, "source": [ "Before:\n", "```YAML\n", "AmazonSageMakerExecutionRoleName:\n", " Description: Name of the Amazon SageMaker execution role\n", " Value: !Select [2, !Split ['/', !Ref SCPortfolioPrincipalRoleArn ] ] \n", "```\n", "After:\n", "```YAML\n", " AmazonSageMakerExecutionRoleName:\n", " Description: Name of the Amazon SageMaker execution role\n", " Value: !Select [1, !Split ['/', !Ref SCPortfolioPrincipalRoleArn ] ] \n", "```\n" ] }, { "cell_type": "markdown", "id": "dcbcaf6d", "metadata": {}, "source": [ "## 2. S3 버킷 설정\n", "- 디폴트 버킷을 사용 합니다. 사용자 정의 버킷을 사용시에는 아래 코드를 수정해서 입력 해주시기 바랍니다." ] }, { "cell_type": "code", "execution_count": 1, "id": "5348803b", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "bucket: sagemaker-us-east-1-569441333767\n", "bucket_region: us-east-1\n" ] } ], "source": [ "import sagemaker\n", "Use_Custome_Bucket = None\n", "if Use_Custome_Bucket is None:\n", " bucket = sagemaker.Session().default_bucket()\n", " bucket_region = sagemaker.Session().boto_region_name\n", "else: \n", " bucket = '<>' # 사용자 버킷을 이용하세요.\n", " bucket_region = '<>' # 위의 버킷 리젼을 입력 하세요.\n", "\n", "print(\"bucket: \", bucket)\n", "print(\"bucket_region: \", bucket_region)" ] }, { "cell_type": "markdown", "id": "1e77bc96", "metadata": {}, "source": [ "# 3. 소스 코드 준비" ] }, { "cell_type": "markdown", "id": "00f460e1", "metadata": {}, "source": [ "## 3.1. Source code zip file 생성\n", "\n", "- 제공된 Git 리파지토리에서 yaml, yml, sh, py 파일을 amazon-sagemaker-reusable-components.zip 로 압축하여 소스 코드를 준비 합니다.\n", "원문을 아래와 같이 `*.py` 를 추가 했습니다. 이렇지 않은 경우 에러 발생 합니다.\n", "- 참고:\n", " - 원문: zip -r amazon-sagemaker-reusable-components.zip . -i \"*.yaml\" \"*.yml\" \"*.sh\"\n", " - 수정: zip -r amazon-sagemaker-reusable-components.zip . -i \"*.yaml\" \"*.yml\" \"*.sh\" \"*.py\"" ] }, { "cell_type": "code", "execution_count": 2, "id": "1d556cca", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " adding: package-cfn.sh (deflated 66%)\n", " adding: sm-project-sc-portfolio.yaml (deflated 77%)\n", " adding: buildspec-package-cfn.yml (deflated 18%)\n", " adding: package-cfn.yaml (deflated 69%)\n", " adding: project-seed-code/s3-fs-ingestion/build.py (deflated 63%)\n", " adding: project-seed-code/s3-fs-ingestion/pipeline/pipeline.py (deflated 70%)\n", " adding: project-seed-code/s3-fs-ingestion/functions/start_fs_ingestion.py (deflated 58%)\n", " adding: project-seed-code/s3-fs-ingestion/buildspec.yml (deflated 54%)\n", " adding: prerequsite/create-sm-notebook.yaml (deflated 59%)\n", " adding: cfn-templates/project-s3-fs-ingestion.yaml (deflated 72%)\n", " adding: cfn-templates/sm-project-sc-portfolio.yaml (deflated 76%)\n", " adding: cfn-templates/.ipynb_checkpoints/sm-project-sc-portfolio-checkpoint.yaml (deflated 76%)\n" ] } ], "source": [ "%%sh\n", "\n", "# cd amazon-sagemaker-reusable-components \n", "rm -f amazon-sagemaker-reusable-components.zip\n", "# zip -r amazon-sagemaker-reusable-components.zip . -i \"*.yaml\" \"*.yml\" \"*.sh\"\n", "zip -r amazon-sagemaker-reusable-components.zip . -i \"*.yaml\" \"*.yml\" \"*.sh\" \"*.py\"" ] }, { "cell_type": "markdown", "id": "e9ee4b75", "metadata": {}, "source": [ "## 3.2 소스 코드 zip 파일을 S3 버킷에 업로드\n", "- 소스 코드 zip 파일 `amazon-sagemaker-reusable-components.zip`을 S3 버킷에 업로드합니다." ] }, { "cell_type": "code", "execution_count": 3, "id": "71055fde", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "upload: ./amazon-sagemaker-reusable-components.zip to s3://sagemaker-us-east-1-569441333767/amazon-sagemaker-reusable-components/amazon-sagemaker-reusable-components.zip\n" ] } ], "source": [ "# S3_BUCKET_NAME=\n", "! aws s3 cp amazon-sagemaker-reusable-components.zip s3://{bucket}/amazon-sagemaker-reusable-components/" ] }, { "cell_type": "code", "execution_count": 4, "id": "86704cd1", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2022-02-07 01:35:49 0 569441333767/sagemaker/us-east-1/offline-store/FG-abalone-07-01-25-47-c99ecc2d-1644197747/FG-abalone-07-01-25-47-c99ecc2d2022-02-07T01:35:47.967Z.txt\n", "2022-02-07 03:13:29 19868 amazon-sagemaker-reusable-components/amazon-sagemaker-reusable-components.zip\n", "2022-02-07 01:08:51 15598 amazon-sagemaker-reusable-components/project-s3-fs-ingestion.yaml\n", "2022-02-07 01:08:49 4326 amazon-sagemaker-reusable-components/seed-code/s3-fs-ingestion-v1.0.zip\n", "2022-02-07 01:08:50 7205 amazon-sagemaker-reusable-components/sm-project-sc-portfolio.yaml\n" ] } ], "source": [ "! aws s3 ls {bucket} --recursive\n", "# ! aws s3 rm s3://{bucket} --recursive" ] }, { "cell_type": "markdown", "id": "dbf6a808", "metadata": {}, "source": [ "# 4. package-cfn.yaml 배포 파일 확인" ] }, { "cell_type": "markdown", "id": "1a2c328b", "metadata": {}, "source": [ "### 4.1 CF 의 Output 확인\n", "- SMProjectSCPortfolioS3Uri 생성\n", "- SMProjectSCPortfolioDeployLink 생성\n", "- StartBuildCLICommand 생성" ] }, { "cell_type": "code", "execution_count": 5, "id": "707526ea", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[94mOutputs\u001b[39;49;00m:\n", " \u001b[94mSMProjectSCPortfolioS3Uri\u001b[39;49;00m:\n", " \u001b[94mDescription\u001b[39;49;00m: S3 URI for SageMaker projects as Service Catalog portfolio deployment stack\n", " \u001b[94mValue\u001b[39;49;00m: \u001b[36m!Sub\u001b[39;49;00m \u001b[33m'\u001b[39;49;00m\u001b[33mhttps://s3.${AWS::Region}.amazonaws.com/${S3BucketName}/amazon-sagemaker-reusable-components/sm-project-sc-portfolio.yaml\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", "\n", " \u001b[94mSMProjectSCPortfolioDeployLink\u001b[39;49;00m:\n", " \u001b[94mDescription\u001b[39;49;00m: Link to open CloudFormation deployment of Service Catalog portfolio with SageMaker projects\n", " \u001b[94mValue\u001b[39;49;00m: \u001b[36m!Sub\u001b[39;49;00m \u001b[33m'\u001b[39;49;00m\u001b[33mhttps://console.aws.amazon.com/cloudformation/home?region=${AWS::Region}#/stacks/new?templateURL=https://s3.${AWS::Region}.amazonaws.com/${S3BucketName}/amazon-sagemaker-reusable-components/sm-project-sc-portfolio.yaml\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", "\n", " \u001b[94mStartBuildCLICommand\u001b[39;49;00m:\n", " \u001b[94mDescription\u001b[39;49;00m: CLI to start CodeBuild build\n", " \u001b[94mValue\u001b[39;49;00m: \u001b[36m!Sub\u001b[39;49;00m \u001b[33m'\u001b[39;49;00m\u001b[33maws\u001b[39;49;00m\u001b[31m \u001b[39;49;00m\u001b[33mcodebuild\u001b[39;49;00m\u001b[31m \u001b[39;49;00m\u001b[33mstart-build\u001b[39;49;00m\u001b[31m \u001b[39;49;00m\u001b[33m--project-name\u001b[39;49;00m\u001b[31m \u001b[39;49;00m\u001b[33m${CfnTemplatePackageProject}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n" ] } ], "source": [ "!pygmentize \"package-cfn.yaml\" | sed -n 13,24p" ] }, { "cell_type": "markdown", "id": "993f9bc5", "metadata": {}, "source": [ "### 4.2 리소스 확인" ] }, { "cell_type": "markdown", "id": "2c46dc08", "metadata": {}, "source": [ "#### StartBuildLambdaExecutionRole 생성\n", "- 람다 함수(예: amazon-sagemaker-reusable-compone-StartBuildLambda-FI8vkTm6S8AT) 를 실행하는 역할 생성\n" ] }, { "cell_type": "code", "execution_count": 6, "id": "d2fbdc8e", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[94mResources\u001b[39;49;00m:\n", "\n", " \u001b[94mStartBuildLambdaExecutionRole\u001b[39;49;00m: \n", " \u001b[94mType\u001b[39;49;00m: \u001b[33m'\u001b[39;49;00m\u001b[33mAWS::IAM::Role\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " \u001b[94mProperties\u001b[39;49;00m:\n", " \u001b[94mAssumeRolePolicyDocument\u001b[39;49;00m:\n", " \u001b[94mVersion\u001b[39;49;00m: 2012-10-17\n", " \u001b[94mStatement\u001b[39;49;00m:\n", " - \u001b[94mEffect\u001b[39;49;00m: Allow\n", " \u001b[94mPrincipal\u001b[39;49;00m:\n", " \u001b[94mService\u001b[39;49;00m:\n", " - lambda.amazonaws.com\n", " \u001b[94mAction\u001b[39;49;00m:\n", " - \u001b[33m'\u001b[39;49;00m\u001b[33msts:AssumeRole\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " \u001b[94mPath\u001b[39;49;00m: /\n", " \u001b[94mPolicies\u001b[39;49;00m:\n", " - \u001b[94mPolicyName\u001b[39;49;00m: InlinePolicy\n", " \u001b[94mPolicyDocument\u001b[39;49;00m:\n", " \u001b[94mVersion\u001b[39;49;00m: \u001b[33m'\u001b[39;49;00m\u001b[33m2012-10-17\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " \u001b[94mStatement\u001b[39;49;00m:\n", " - \u001b[94mSid\u001b[39;49;00m: CodeBuildPermission\n", " \u001b[94mEffect\u001b[39;49;00m: Allow\n", " \u001b[94mAction\u001b[39;49;00m:\n", " - codebuild:StartBuild\n", " \u001b[94mResource\u001b[39;49;00m: \u001b[36m!GetAtt\u001b[39;49;00m CfnTemplatePackageProject.Arn \n", " \u001b[94mManagedPolicyArns\u001b[39;49;00m:\n", " - \u001b[33m'\u001b[39;49;00m\u001b[33marn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n" ] } ], "source": [ "!pygmentize \"package-cfn.yaml\" | sed -n 26,52p" ] }, { "cell_type": "markdown", "id": "56f85176", "metadata": {}, "source": [ "### 4.3. StartBuildLambda 함수 생성\n", "- `cb.start_build(projectName=event['ResourceProperties']['ProjectName'])` 실행" ] }, { "cell_type": "code", "execution_count": 7, "id": "ee955cdd", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " \u001b[94mStartBuildLambda\u001b[39;49;00m:\n", " \u001b[94mType\u001b[39;49;00m: AWS::Lambda::Function\n", " \u001b[94mProperties\u001b[39;49;00m:\n", " \u001b[94mReservedConcurrentExecutions\u001b[39;49;00m: 1\n", " \u001b[94mCode\u001b[39;49;00m:\n", " \u001b[94mZipFile\u001b[39;49;00m: |\n", " \u001b[31m# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.\u001b[39;49;00m\n", " \u001b[31m# SPDX-License-Identifier: MIT-0\u001b[39;49;00m\n", " \u001b[31mimport json\u001b[39;49;00m\n", " \u001b[31mimport boto3\u001b[39;49;00m\n", " \u001b[31mimport cfnresponse\u001b[39;49;00m\n", "\n", " \u001b[31mcb = boto3.client(\"codebuild\")\u001b[39;49;00m\n", "\n", " \u001b[31mdef lambda_handler(event, context):\u001b[39;49;00m\n", " \u001b[31mtry:\u001b[39;49;00m\n", " \u001b[31mresponse_status = cfnresponse.SUCCESS\u001b[39;49;00m\n", "\n", " \u001b[31mif 'RequestType' in event and event['RequestType'] == 'Create':\u001b[39;49;00m\n", " \u001b[31mcb.start_build(projectName=event['ResourceProperties']['ProjectName'])\u001b[39;49;00m\n", " \u001b[31m \u001b[39;49;00m\n", " \u001b[31mcfnresponse.send(event, context, response_status, {}, '')\u001b[39;49;00m\n", "\n", " \u001b[31mexcept Exception as e:\u001b[39;49;00m\n", " \u001b[31mprint(str(e))\u001b[39;49;00m\n", " \u001b[31mcfnresponse.send(event, context, cfnresponse.FAILED, {}, physicalResourceId=event.get('PhysicalResourceId'), reason=str(e))\u001b[39;49;00m\n", "\n", " \u001b[94mDescription\u001b[39;49;00m: Start CodeBuild project\n", " \u001b[94mHandler\u001b[39;49;00m: index.lambda_handler\n", " \u001b[94mMemorySize\u001b[39;49;00m: 128\n", " \u001b[94mRole\u001b[39;49;00m: \u001b[36m!GetAtt\u001b[39;49;00m StartBuildLambdaExecutionRole.Arn \n", " \u001b[94mRuntime\u001b[39;49;00m: python3.8\n", " \u001b[94mTimeout\u001b[39;49;00m: 120\n" ] } ], "source": [ "!pygmentize \"package-cfn.yaml\" | sed -n 54,86p" ] }, { "cell_type": "markdown", "id": "a83feda6", "metadata": {}, "source": [ "### 4.4. StartBuild 및 CodeBuildServiceRole 생성\n", "- CodeBuild 를 위한 커스텀 StartBuild 및 역할 생성" ] }, { "cell_type": "code", "execution_count": 8, "id": "0fb689b3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " \u001b[94mStartBuild\u001b[39;49;00m:\n", " \u001b[94mType\u001b[39;49;00m: Custom::StartBuild\n", " \u001b[94mProperties\u001b[39;49;00m:\n", " \u001b[94mServiceToken\u001b[39;49;00m: \u001b[36m!GetAtt\u001b[39;49;00m StartBuildLambda.Arn\n", " \u001b[94mProjectName\u001b[39;49;00m: \u001b[36m!Ref\u001b[39;49;00m CfnTemplatePackageProject\n", "\n", " \u001b[94mCodeBuildServiceRole\u001b[39;49;00m:\n", " \u001b[94mType\u001b[39;49;00m: \u001b[33m'\u001b[39;49;00m\u001b[33mAWS::IAM::Role\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " \u001b[94mProperties\u001b[39;49;00m:\n", " \u001b[94mAssumeRolePolicyDocument\u001b[39;49;00m:\n", " \u001b[94mVersion\u001b[39;49;00m: 2012-10-17\n", " \u001b[94mStatement\u001b[39;49;00m:\n", " - \u001b[94mEffect\u001b[39;49;00m: Allow\n", " \u001b[94mPrincipal\u001b[39;49;00m:\n", " \u001b[94mService\u001b[39;49;00m: codebuild.amazonaws.com\n", " \u001b[94mAction\u001b[39;49;00m: \u001b[33m'\u001b[39;49;00m\u001b[33msts:AssumeRole\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " \u001b[94mPath\u001b[39;49;00m: \u001b[33m'\u001b[39;49;00m\u001b[33m/service-role/\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " \u001b[94mPolicies\u001b[39;49;00m:\n", " - \u001b[94mPolicyName\u001b[39;49;00m: CodeBuildServiceRoleInLinePolicy\n", " \u001b[94mPolicyDocument\u001b[39;49;00m:\n", " \u001b[94mVersion\u001b[39;49;00m: 2012-10-17\n", " \u001b[94mStatement\u001b[39;49;00m:\n", " -\n", " \u001b[94mAction\u001b[39;49;00m:\n", " - \u001b[33m'\u001b[39;49;00m\u001b[33mlogs:CreateLogGroup\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " - \u001b[33m'\u001b[39;49;00m\u001b[33mlogs:CreateLogStream\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " - \u001b[33m'\u001b[39;49;00m\u001b[33mlogs:PutLogEvents\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " \u001b[94mResource\u001b[39;49;00m: \u001b[33m'\u001b[39;49;00m\u001b[33m*\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " \u001b[94mEffect\u001b[39;49;00m: \u001b[33m'\u001b[39;49;00m\u001b[33mAllow\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " -\n", " \u001b[94mAction\u001b[39;49;00m:\n", " - \u001b[33m'\u001b[39;49;00m\u001b[33ms3:*Object\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " - \u001b[33m'\u001b[39;49;00m\u001b[33ms3:GetObjectVersion\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " - \u001b[33m'\u001b[39;49;00m\u001b[33ms3:GetBucketAcl\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " - \u001b[33m'\u001b[39;49;00m\u001b[33ms3:GetBucketLocation\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " - \u001b[33m'\u001b[39;49;00m\u001b[33ms3:ListBucket\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " - \u001b[33m'\u001b[39;49;00m\u001b[33ms3:PutObjectTagging\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " - \u001b[33m'\u001b[39;49;00m\u001b[33ms3:CreateBucket\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " \u001b[94mResource\u001b[39;49;00m: \n", " - \u001b[36m!Sub\u001b[39;49;00m \u001b[33m'\u001b[39;49;00m\u001b[33marn:aws:s3:::${S3BucketName}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " - \u001b[36m!Sub\u001b[39;49;00m \u001b[33m'\u001b[39;49;00m\u001b[33marn:aws:s3:::${S3BucketName}/*\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " \u001b[94mEffect\u001b[39;49;00m: \u001b[33m'\u001b[39;49;00m\u001b[33mAllow\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " -\n", " \u001b[94mAction\u001b[39;49;00m:\n", " - \u001b[33m'\u001b[39;49;00m\u001b[33mcodebuild:CreateReportGroup\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " - \u001b[33m'\u001b[39;49;00m\u001b[33mcodebuild:CreateReport\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " - \u001b[33m'\u001b[39;49;00m\u001b[33mcodebuild:UpdateReport\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " - \u001b[33m'\u001b[39;49;00m\u001b[33mcodebuild:BatchPutTestCases\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " - \u001b[33m'\u001b[39;49;00m\u001b[33mcodebuild:BatchPutCodeCoverages\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " \u001b[94mResource\u001b[39;49;00m: \u001b[36m!Sub\u001b[39;49;00m \u001b[33m'\u001b[39;49;00m\u001b[33marn:aws:codebuild:${AWS::Region}:${AWS::AccountId}:report-group/*\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " \u001b[94mEffect\u001b[39;49;00m: \u001b[33m'\u001b[39;49;00m\u001b[33mAllow\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", "\n" ] } ], "source": [ "!pygmentize \"package-cfn.yaml\" | sed -n 88,139p" ] }, { "cell_type": "markdown", "id": "7092f166", "metadata": {}, "source": [ "### 4.5. CfnTemplatePackageProject 생성\n", "- 이 빌드 프로젝트는 buildspec-package-cfn.yml 실행 함." ] }, { "cell_type": "code", "execution_count": 9, "id": "4424024d", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " \u001b[94mCfnTemplatePackageProject\u001b[39;49;00m:\n", " \u001b[94mType\u001b[39;49;00m: AWS::CodeBuild::Project\n", " \u001b[94mProperties\u001b[39;49;00m:\n", " \u001b[94mDescription\u001b[39;49;00m: \u001b[36m!Sub\u001b[39;49;00m \u001b[33m'\u001b[39;49;00m\u001b[33mPackaging\u001b[39;49;00m\u001b[31m \u001b[39;49;00m\u001b[33mCFN\u001b[39;49;00m\u001b[31m \u001b[39;49;00m\u001b[33mtemplates\u001b[39;49;00m\u001b[31m \u001b[39;49;00m\u001b[33minto\u001b[39;49;00m\u001b[31m \u001b[39;49;00m\u001b[33m${S3BucketName}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " \u001b[94mServiceRole\u001b[39;49;00m: \u001b[36m!GetAtt\u001b[39;49;00m CodeBuildServiceRole.Arn\n", " \u001b[94mArtifacts\u001b[39;49;00m: \n", " \u001b[94mType\u001b[39;49;00m: NO_ARTIFACTS\n", " \u001b[94mEnvironment\u001b[39;49;00m:\n", " \u001b[94mType\u001b[39;49;00m: \u001b[33m'\u001b[39;49;00m\u001b[33mLINUX_CONTAINER\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " \u001b[94mComputeType\u001b[39;49;00m: \u001b[33m'\u001b[39;49;00m\u001b[33mBUILD_GENERAL1_SMALL\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " \u001b[94mImage\u001b[39;49;00m: \u001b[33m'\u001b[39;49;00m\u001b[33maws/codebuild/amazonlinux2-x86_64-standard:3.0\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " \u001b[94mEnvironmentVariables\u001b[39;49;00m:\n", " - \u001b[94mName\u001b[39;49;00m: \u001b[33m'\u001b[39;49;00m\u001b[33mS3_BUCKET_NAME\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " \u001b[94mValue\u001b[39;49;00m: \u001b[36m!Ref\u001b[39;49;00m S3BucketName\n", " - \u001b[94mName\u001b[39;49;00m: \u001b[33m'\u001b[39;49;00m\u001b[33mDEPLOYMENT_REGION\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " \u001b[94mValue\u001b[39;49;00m: \u001b[36m!Ref\u001b[39;49;00m AWS::Region\n", " \u001b[94mSource\u001b[39;49;00m:\n", " \u001b[94mType\u001b[39;49;00m: S3\n", " \u001b[94mLocation\u001b[39;49;00m: \u001b[36m!Sub\u001b[39;49;00m \u001b[33m'\u001b[39;49;00m\u001b[33m${S3BucketName}/amazon-sagemaker-reusable-components/amazon-sagemaker-reusable-components.zip\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " \u001b[94mBuildSpec\u001b[39;49;00m: buildspec-package-cfn.yml\n", " \u001b[94mLogsConfig\u001b[39;49;00m:\n", " \u001b[94mCloudWatchLogs\u001b[39;49;00m:\n", " \u001b[94mStatus\u001b[39;49;00m: \u001b[33m'\u001b[39;49;00m\u001b[33mENABLED\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n", " \u001b[94mTimeoutInMinutes\u001b[39;49;00m: 5\n" ] } ], "source": [ "!pygmentize \"package-cfn.yaml\" | sed -n 140,163p" ] }, { "cell_type": "markdown", "id": "864d45e3", "metadata": {}, "source": [ "### 4.6. buildspec-package-cfn.yml 확인\n", "```\n", "./package-cfn.sh ${S3_BUCKET_NAME} ${DEPLOYMENT_REGION} 실행 함\n", "```" ] }, { "cell_type": "code", "execution_count": 10, "id": "dff692b4", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[94mversion\u001b[39;49;00m: 0.2\n", "\n", "\u001b[94mphases\u001b[39;49;00m:\n", " \u001b[94mbuild\u001b[39;49;00m:\n", " \u001b[94mcommands\u001b[39;49;00m:\n", " - env\n", " - echo Packaging Cloudformation and uploading to S3...\n", " - ./package-cfn.sh ${S3_BUCKET_NAME} ${DEPLOYMENT_REGION}\n" ] } ], "source": [ "!pygmentize \"buildspec-package-cfn.yml\" " ] }, { "cell_type": "markdown", "id": "76fa9e29", "metadata": {}, "source": [ "### 4.7. package-cfn.sh 확인\n", "```\n", "./package-cfn.sh ${S3_BUCKET_NAME} ${DEPLOYMENT_REGION} 실행 함\n", "```\n", "\n", "- 이 셀은 아래의 주요한 파일을 Zip 으로 압축하고 S3에 업로딩 합니다.\n", " - (1) sm-project-sc-portfolio.yaml (카탈로그 포트폴리오 정의 파일)\n", " - (2) project-s3-fs-ingestion.yaml (카탈로그 프러덕트 정의 파일)\n", " - (3) 카탈로그 프러덕트 Seed 코드 (Pipeline 정의 등) 추가\n", " - (4) 지정된 S3 버킷에 위의 결과 파일을 업로드\n", "\n", "\n", "- 아래 코드를 스키밍 하시고 실행 결과는 추후에 확인 함." ] }, { "cell_type": "code", "execution_count": 11, "id": "d067b58f", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[37m#!/bin/bash\u001b[39;49;00m\n", "\n", "\u001b[37m# Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\u001b[39;49;00m\n", "\u001b[37m# SPDX-License-Identifier: MIT-0\u001b[39;49;00m\n", "\n", "\u001b[36mset\u001b[39;49;00m -e\n", "\n", "\u001b[37m# This script will package the CloudFormation in ${CFN_TEMPLATE_DIR} directory and upload it \u001b[39;49;00m\n", "\u001b[37m# to Amazon S3 in preparation for deployment using the AWS CloudFromation service. \u001b[39;49;00m\n", "\u001b[37m# \u001b[39;49;00m\n", "\u001b[37m# This script exists because Service Catalog products, when using relative references to cloudformation templates are \u001b[39;49;00m\n", "\u001b[37m# not properly packaged by the AWS cli. Also the full stack, due to 2 levels of Service Catalog deployment will not \u001b[39;49;00m\n", "\u001b[37m# always package properly using the AWS cli.\u001b[39;49;00m\n", "\n", "\u001b[37m# This script treats the templates as source code and packages them, putting the results into a 'build' subdirectory.\u001b[39;49;00m\n", "\n", "\u001b[37m# This script assumes a Linux or MacOSX environment and relies on the following software packages being installed:\u001b[39;49;00m\n", "\u001b[37m# . - AWS Command Line Interface (CLI)\u001b[39;49;00m\n", "\u001b[37m# . - sed\u001b[39;49;00m\n", "\u001b[37m# . - Python 3 / pip3\u001b[39;49;00m\n", "\u001b[37m# . - zip\u001b[39;49;00m\n", "\n", "\u001b[37m# PLEASE NOTE this script will store all resources to an Amazon S3 bucket s3://${CFN_BUCKET_NAME}/${PROJECT_NAME}\u001b[39;49;00m\n", "\u001b[31mCFN_BUCKET_NAME\u001b[39;49;00m=\u001b[31m$1\u001b[39;49;00m\n", "\u001b[31mDEPLOYMENT_REGION\u001b[39;49;00m=\u001b[31m$2\u001b[39;49;00m\n", "\u001b[31mPROJECT_NAME\u001b[39;49;00m=\u001b[33m\"amazon-sagemaker-reusable-components\"\u001b[39;49;00m\n", "\u001b[31mCFN_TEMPLATE_DIR\u001b[39;49;00m=\u001b[33m\"cfn-templates\"\u001b[39;49;00m\n", "\u001b[31mSEED_CODE_DIR\u001b[39;49;00m=\u001b[33m\"project-seed-code\"\u001b[39;49;00m\n", "\u001b[31mCFN_OUTPUT_DIR\u001b[39;49;00m=\u001b[33m\"\u001b[39;49;00m\u001b[33mbuild/\u001b[39;49;00m\u001b[33m${\u001b[39;49;00m\u001b[31mDEPLOYMENT_REGION\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m\n", "\u001b[31mSEED_CODE_OUTPUT_DIR\u001b[39;49;00m=\u001b[33m\"\u001b[39;49;00m\u001b[33mbuild/\u001b[39;49;00m\u001b[33m${\u001b[39;49;00m\u001b[31mDEPLOYMENT_REGION\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\u001b[33m/seed-code\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m\n", "\u001b[31mSOURCE_CODE_ZIP_NAME\u001b[39;49;00m=\u001b[33m\"amazon-sagemaker-reusable-components.zip\"\u001b[39;49;00m\n", "\n", "\u001b[37m# files that need to be scrubbed with sed to replace < S3 BUCKET LOCATION > with an actual S3 bucket name\u001b[39;49;00m\n", "\u001b[31mSELF_PACKAGE_LIST\u001b[39;49;00m=\u001b[33m\"sm-project-sc-portfolio.yaml project-s3-fs-ingestion.yaml\"\u001b[39;49;00m\n", "\n", "\u001b[37m# files to be packaged using `aws cloudformation package`\u001b[39;49;00m\n", "\u001b[31mAWS_PACKAGE_LIST\u001b[39;49;00m=\u001b[33m\"sm-project-sc-portfolio.yaml\"\u001b[39;49;00m\n", "\n", "\u001b[37m# files that wont be uploaded by `aws cloudformation package`\u001b[39;49;00m\n", "\u001b[31mUPLOAD_LIST\u001b[39;49;00m=\u001b[33m\"sm-project-sc-portfolio.yaml project-s3-fs-ingestion.yaml\"\u001b[39;49;00m \n", "\n", "\u001b[37m# Check that S3 bucket exists, if not create a new one\u001b[39;49;00m\n", "\u001b[34mif\u001b[39;49;00m aws s3 ls s3://\u001b[33m${\u001b[39;49;00m\u001b[31mCFN_BUCKET_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m \u001b[34m2\u001b[39;49;00m>&\u001b[34m1\u001b[39;49;00m | grep NoSuchBucket\n", "\u001b[34mthen\u001b[39;49;00m\n", " \u001b[36mecho\u001b[39;49;00m Creating Amazon S3 bucket \u001b[33m${\u001b[39;49;00m\u001b[31mCFN_BUCKET_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\n", " aws s3 mb s3://\u001b[33m${\u001b[39;49;00m\u001b[31mCFN_BUCKET_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m --region \u001b[31m$DEPLOYMENT_REGION\u001b[39;49;00m\n", "\u001b[34mfi\u001b[39;49;00m\n", "\u001b[36mecho\u001b[39;49;00m \u001b[33m\"\u001b[39;49;00m\u001b[33mPreparing content for publication to Amazon S3 s3://\u001b[39;49;00m\u001b[33m${\u001b[39;49;00m\u001b[31mCFN_BUCKET_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\u001b[33m/\u001b[39;49;00m\u001b[33m${\u001b[39;49;00m\u001b[31mPROJECT_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m\n", "\n", "\u001b[37m## clean away any previous builds of the CFN\u001b[39;49;00m\n", "rm -fr \u001b[33m${\u001b[39;49;00m\u001b[31mCFN_OUTPUT_DIR\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\n", "rm -fr \u001b[33m${\u001b[39;49;00m\u001b[31mSEED_CODE_OUTPUT_DIR\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\n", "mkdir -p \u001b[33m${\u001b[39;49;00m\u001b[31mCFN_OUTPUT_DIR\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\n", "mkdir -p \u001b[33m${\u001b[39;49;00m\u001b[31mSEED_CODE_OUTPUT_DIR\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\n", "rm -f build/*-\u001b[33m${\u001b[39;49;00m\u001b[31mDEPLOYMENT_REGION\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m.zip\n", "cp \u001b[33m${\u001b[39;49;00m\u001b[31mCFN_TEMPLATE_DIR\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m/*.yaml \u001b[33m${\u001b[39;49;00m\u001b[31mCFN_OUTPUT_DIR\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\n", "\n", "\u001b[37m# Zip the source code\u001b[39;49;00m\n", "\u001b[36mecho\u001b[39;49;00m \u001b[33m\"Zipping the CloudFormation templates and buildspec files\"\u001b[39;49;00m\n", "rm -f \u001b[33m${\u001b[39;49;00m\u001b[31mSOURCE_CODE_ZIP_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\n", "zip -r \u001b[33m${\u001b[39;49;00m\u001b[31mSOURCE_CODE_ZIP_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m . -i \u001b[33m\"*.yaml\"\u001b[39;49;00m \u001b[33m\"*.yml\"\u001b[39;49;00m \u001b[33m\"*.sh\"\u001b[39;49;00m\n", "\n", "\u001b[37m## Zip the MLOps project seed code for\u001b[39;49;00m\n", "\u001b[36mecho\u001b[39;49;00m \u001b[33m\"Zipping MLOps project seed code\"\u001b[39;49;00m\n", "(\u001b[36mcd\u001b[39;49;00m \u001b[33m${\u001b[39;49;00m\u001b[31mSEED_CODE_DIR\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m/s3-fs-ingestion/ && zip -r ../../\u001b[33m${\u001b[39;49;00m\u001b[31mSEED_CODE_OUTPUT_DIR\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m/s3-fs-ingestion-v1.0.zip .)\n", "\n", "\u001b[37m## publish files to target AWS regions\u001b[39;49;00m\n", "\u001b[36mecho\u001b[39;49;00m \u001b[33m\"\u001b[39;49;00m\u001b[33mPublishing CloudFormation to \u001b[39;49;00m\u001b[33m${\u001b[39;49;00m\u001b[31mDEPLOYMENT_REGION\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m\n", "\u001b[36mecho\u001b[39;49;00m \u001b[33m\"\u001b[39;49;00m\u001b[33mClearing the project directory for \u001b[39;49;00m\u001b[33m${\u001b[39;49;00m\u001b[31mPROJECT_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\u001b[33m in \u001b[39;49;00m\u001b[33m${\u001b[39;49;00m\u001b[31mCFN_BUCKET_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\u001b[33m...\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m\n", "\n", "aws s3 rm \u001b[33m\\\u001b[39;49;00m\n", " s3://\u001b[33m${\u001b[39;49;00m\u001b[31mCFN_BUCKET_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m/\u001b[33m${\u001b[39;49;00m\u001b[31mPROJECT_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m/ \u001b[33m\\\u001b[39;49;00m\n", " --recursive \u001b[33m\\\u001b[39;49;00m\n", " --region \u001b[33m${\u001b[39;49;00m\u001b[31mDEPLOYMENT_REGION\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\n", "\n", "\u001b[36mecho\u001b[39;49;00m \u001b[33m\"\u001b[39;49;00m\u001b[33mSelf-packaging the Cloudformation templates: \u001b[39;49;00m\u001b[33m${\u001b[39;49;00m\u001b[31mSELF_PACKAGE_LIST\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m\n", "\u001b[34mfor\u001b[39;49;00m fname \u001b[34min\u001b[39;49;00m \u001b[33m${\u001b[39;49;00m\u001b[31mSELF_PACKAGE_LIST\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m;\n", "\u001b[34mdo\u001b[39;49;00m\n", " sed -ie \u001b[33m\"\u001b[39;49;00m\u001b[33ms/< S3_CFN_STAGING_PATH >/\u001b[39;49;00m\u001b[33m${\u001b[39;49;00m\u001b[31mPROJECT_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\u001b[33m/\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m \u001b[33m${\u001b[39;49;00m\u001b[31mCFN_OUTPUT_DIR\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m/\u001b[33m${\u001b[39;49;00m\u001b[31mfname\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\n", " sed -ie \u001b[33m\"\u001b[39;49;00m\u001b[33ms/< S3_CFN_STAGING_BUCKET >/\u001b[39;49;00m\u001b[33m${\u001b[39;49;00m\u001b[31mCFN_BUCKET_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\u001b[33m/\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m \u001b[33m${\u001b[39;49;00m\u001b[31mCFN_OUTPUT_DIR\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m/\u001b[33m${\u001b[39;49;00m\u001b[31mfname\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\n", " sed -ie \u001b[33m\"\u001b[39;49;00m\u001b[33ms/< S3_CFN_STAGING_BUCKET_PATH >/\u001b[39;49;00m\u001b[33m${\u001b[39;49;00m\u001b[31mCFN_BUCKET_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\u001b[33m\\/\u001b[39;49;00m\u001b[33m${\u001b[39;49;00m\u001b[31mPROJECT_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\u001b[33m/\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m \u001b[33m${\u001b[39;49;00m\u001b[31mCFN_OUTPUT_DIR\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m/\u001b[33m${\u001b[39;49;00m\u001b[31mfname\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\n", "\u001b[34mdone\u001b[39;49;00m\n", "\n", "\u001b[36mecho\u001b[39;49;00m \u001b[33m\"\u001b[39;49;00m\u001b[33mPackaging Cloudformation templates: \u001b[39;49;00m\u001b[33m${\u001b[39;49;00m\u001b[31mAWS_PACKAGE_LIST\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m\n", "\u001b[34mfor\u001b[39;49;00m fname \u001b[34min\u001b[39;49;00m \u001b[33m${\u001b[39;49;00m\u001b[31mAWS_PACKAGE_LIST\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m;\n", "\u001b[34mdo\u001b[39;49;00m\n", " \u001b[36mpushd\u001b[39;49;00m \u001b[33m${\u001b[39;49;00m\u001b[31mCFN_OUTPUT_DIR\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\n", " aws cloudformation package \u001b[33m\\\u001b[39;49;00m\n", " --template-file \u001b[33m${\u001b[39;49;00m\u001b[31mfname\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m \u001b[33m\\\u001b[39;49;00m\n", " --s3-bucket \u001b[33m${\u001b[39;49;00m\u001b[31mCFN_BUCKET_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m \u001b[33m\\\u001b[39;49;00m\n", " --s3-prefix \u001b[33m${\u001b[39;49;00m\u001b[31mPROJECT_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m \u001b[33m\\\u001b[39;49;00m\n", " --output-template-file \u001b[33m${\u001b[39;49;00m\u001b[31mfname\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m-packaged \u001b[33m\\\u001b[39;49;00m\n", " --region \u001b[33m${\u001b[39;49;00m\u001b[31mDEPLOYMENT_REGION\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\n", " \u001b[36mpopd\u001b[39;49;00m\n", "\u001b[34mdone\u001b[39;49;00m\n", "\n", "\u001b[37m# copy source code .zip file to the S3 bucket\u001b[39;49;00m\n", "aws s3 cp \u001b[33m${\u001b[39;49;00m\u001b[31mSOURCE_CODE_ZIP_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m s3://\u001b[33m${\u001b[39;49;00m\u001b[31mCFN_BUCKET_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m/\u001b[33m${\u001b[39;49;00m\u001b[31mPROJECT_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m/\n", "\n", "\u001b[37m# copy all seed-code .zip files from ${SEED_CODE_OUTPUT_DIR} to S3\u001b[39;49;00m\n", "aws s3 cp \u001b[33m${\u001b[39;49;00m\u001b[31mSEED_CODE_OUTPUT_DIR\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m s3://\u001b[33m${\u001b[39;49;00m\u001b[31mCFN_BUCKET_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m/\u001b[33m${\u001b[39;49;00m\u001b[31mPROJECT_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m/seed-code/ --recursive\n", "\n", "\u001b[37m# put an object tag servicecatalog:provisioning=true for AmazonSageMakerServiceCatalogProductsLaunchRole access\u001b[39;49;00m\n", "\u001b[34mfor\u001b[39;49;00m fname \u001b[34min\u001b[39;49;00m \u001b[33m${\u001b[39;49;00m\u001b[31mSEED_CODE_OUTPUT_DIR\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m/*\n", "\u001b[34mdo\u001b[39;49;00m\n", " \u001b[36mecho\u001b[39;49;00m \u001b[33m\"\u001b[39;49;00m\u001b[33mSet servicecatalog:provisioning=true tag to object: \u001b[39;49;00m\u001b[33m${\u001b[39;49;00m\u001b[31mfname\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m\n", " aws s3api put-object-tagging \u001b[33m\\\u001b[39;49;00m\n", " --bucket \u001b[33m${\u001b[39;49;00m\u001b[31mCFN_BUCKET_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m \u001b[33m\\\u001b[39;49;00m\n", " --key \u001b[33m${\u001b[39;49;00m\u001b[31mPROJECT_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m/seed-code/\u001b[34m$(\u001b[39;49;00mbasename \u001b[31m$fname\u001b[39;49;00m\u001b[34m)\u001b[39;49;00m \u001b[33m\\\u001b[39;49;00m\n", " --tagging \u001b[33m'TagSet=[{Key=servicecatalog:provisioning,Value=true}]'\u001b[39;49;00m\n", "\u001b[34mdone\u001b[39;49;00m\n", "\n", "\u001b[37m# push files to S3, note this does not 'package' the templates\u001b[39;49;00m\n", "\u001b[36mecho\u001b[39;49;00m \u001b[33m\"\u001b[39;49;00m\u001b[33mCopying cloudformation templates and files to S3: \u001b[39;49;00m\u001b[33m${\u001b[39;49;00m\u001b[31mUPLOAD_LIST\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m\n", "\u001b[34mfor\u001b[39;49;00m fname \u001b[34min\u001b[39;49;00m \u001b[33m${\u001b[39;49;00m\u001b[31mUPLOAD_LIST\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m;\n", "\u001b[34mdo\u001b[39;49;00m\n", " \u001b[34mif\u001b[39;49;00m [ -f \u001b[33m${\u001b[39;49;00m\u001b[31mCFN_OUTPUT_DIR\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m/\u001b[33m${\u001b[39;49;00m\u001b[31mfname\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m-packaged ]; \u001b[34mthen\u001b[39;49;00m\n", " aws s3 cp \u001b[33m${\u001b[39;49;00m\u001b[31mCFN_OUTPUT_DIR\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m/\u001b[33m${\u001b[39;49;00m\u001b[31mfname\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m-packaged s3://\u001b[33m${\u001b[39;49;00m\u001b[31mCFN_BUCKET_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m/\u001b[33m${\u001b[39;49;00m\u001b[31mPROJECT_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m/\u001b[33m${\u001b[39;49;00m\u001b[31mfname\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\n", " \u001b[34melse\u001b[39;49;00m\n", " aws s3 cp \u001b[33m${\u001b[39;49;00m\u001b[31mCFN_OUTPUT_DIR\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m/\u001b[33m${\u001b[39;49;00m\u001b[31mfname\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m s3://\u001b[33m${\u001b[39;49;00m\u001b[31mCFN_BUCKET_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m/\u001b[33m${\u001b[39;49;00m\u001b[31mPROJECT_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m/\u001b[33m${\u001b[39;49;00m\u001b[31mfname\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\n", " \u001b[34mfi\u001b[39;49;00m\n", "\n", " \u001b[36mecho\u001b[39;49;00m \u001b[33m\"\u001b[39;49;00m\u001b[33mTo validate template \u001b[39;49;00m\u001b[33m${\u001b[39;49;00m\u001b[31mfname\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\u001b[33m:\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m\n", " \u001b[36mecho\u001b[39;49;00m \u001b[33m\"\u001b[39;49;00m\u001b[33maws cloudformation validate-template --template-url https://s3.\u001b[39;49;00m\u001b[33m${\u001b[39;49;00m\u001b[31mDEPLOYMENT_REGION\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\u001b[33m.amazonaws.com/\u001b[39;49;00m\u001b[33m${\u001b[39;49;00m\u001b[31mCFN_BUCKET_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\u001b[33m/\u001b[39;49;00m\u001b[33m${\u001b[39;49;00m\u001b[31mPROJECT_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\u001b[33m/\u001b[39;49;00m\u001b[33m${\u001b[39;49;00m\u001b[31mfname\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m\n", "\n", " \u001b[36mecho\u001b[39;49;00m \u001b[33m\"To deploy stack execute:\"\u001b[39;49;00m\n", " \u001b[36mecho\u001b[39;49;00m \u001b[33m\"\u001b[39;49;00m\u001b[33maws cloudformation create-stack --template-url https://s3.\u001b[39;49;00m\u001b[33m${\u001b[39;49;00m\u001b[31mDEPLOYMENT_REGION\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\u001b[33m.amazonaws.com/\u001b[39;49;00m\u001b[33m${\u001b[39;49;00m\u001b[31mCFN_BUCKET_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\u001b[33m/\u001b[39;49;00m\u001b[33m${\u001b[39;49;00m\u001b[31mPROJECT_NAME\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\u001b[33m/\u001b[39;49;00m\u001b[33m${\u001b[39;49;00m\u001b[31mfname\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\u001b[33m --region \u001b[39;49;00m\u001b[33m${\u001b[39;49;00m\u001b[31mDEPLOYMENT_REGION\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\u001b[33m --stack-name --disable-rollback --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM --parameters ParameterKey=,ParameterValue=\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m \n", "\n", "\u001b[34mdone\u001b[39;49;00m\n", "\n", "\u001b[36mecho\u001b[39;49;00m ==================================================\n", "\u001b[36mecho\u001b[39;49;00m \u001b[33m\"Publication complete\"\u001b[39;49;00m\n" ] } ], "source": [ "!pygmentize \"package-cfn.sh\" " ] }, { "cell_type": "markdown", "id": "d3250b04", "metadata": {}, "source": [ "# 5. package-cfn.yaml (CloudFormation 템플릿) 을 배포\n", "- sm-project-sc-portfolio.yaml 생성\n", "- seed-code/s3-fs-ingestion-v1.0.zip 생성" ] }, { "cell_type": "markdown", "id": "f88b7e6a", "metadata": {}, "source": [ "## 5.1. 클라우드 포메이션 실행\n", "- 약 3분 정도 소요 됩니다.\n", "- Cloud Formation 콘솔로 가서 진행 사항을 확인할 수 있습니다." ] }, { "cell_type": "code", "execution_count": 12, "id": "f04b4fe6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Waiting for changeset to be created..\n", "Waiting for stack create/update to complete\n", "Successfully created/updated stack - amazon-sagemaker-reusable-components-package-cfn\n" ] } ], "source": [ "# Deploy the CloudFormation template:\n", "STACK_NAME='amazon-sagemaker-reusable-components-package-cfn'\n", "! aws cloudformation deploy \\\n", " --template-file package-cfn.yaml \\\n", " --stack-name {STACK_NAME} \\\n", " --capabilities CAPABILITY_NAMED_IAM \\\n", " --parameter-overrides \\\n", " S3BucketName={bucket}" ] }, { "cell_type": "code", "execution_count": 13, "id": "8fcc3a98", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n", "\u001b[0m\u001b[0m| DescribeStacks |\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0m+--------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0m| \u001b[1m\u001b[34mSMProjectSCPortfolioS3Uri\u001b[0m | \u001b[1m\u001b[34mhttps://s3.us-east-1.amazonaws.com/sagemaker-us-east-1-569441333767/amazon-sagemaker-reusable-components/sm-project-sc-portfolio.yaml\u001b[0m |\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0m| \u001b[1m\u001b[34mSMProjectSCPortfolioDeployLink\u001b[0m| \u001b[1m\u001b[34mhttps://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?templateURL=https://s3.us-east-1.amazonaws.com/sagemaker-us-east-1-569441333767/amazon-sagemaker-reusable-components/sm-project-sc-portfolio.yaml\u001b[0m |\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0m| \u001b[1m\u001b[34mStartBuildCLICommand\u001b[0m | \u001b[1m\u001b[34maws codebuild start-build --project-name CfnTemplatePackageProject-Zix2iOsi4G5U\u001b[0m |\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0m+--------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0m" ] } ], "source": [ "! aws cloudformation describe-stacks \\\n", " --stack-name {STACK_NAME} \\\n", " --output table \\\n", " --query \"Stacks[0].Outputs[*].[OutputKey, OutputValue]\"" ] }, { "cell_type": "markdown", "id": "8657c18b", "metadata": {}, "source": [ "## 5.2. CloudFormation 템플릿 실행 결과\n", "- 두개의 중요한 Output 이 출력 됩니다.\n", " - 카탈로그 포트폴리오 정의하는 CF: `sm-project-sc-portfolio.yaml` \n", " - 카탈로그 포트폴리오 정의 링크: SMProjectSCPortfolioDeployLink" ] }, { "cell_type": "markdown", "id": "c490fc83", "metadata": {}, "source": [ "예시: \n", "```\n", "| SMProjectSCPortfolioS3Uri | https://s3.us-east-1.amazonaws.com/sagemaker-us-east-1-569441333767/amazon-sagemaker-reusable-components/sm-project-sc-portfolio.yaml |\n", "| SMProjectSCPortfolioDeployLink| https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?templateURL=https://s3.us-east-1.amazonaws.com/sagemaker-us-east-1-569441333767/amazon-sagemaker-reusable-components/sm-project-sc-portfolio.yaml |\n", "| StartBuildCLICommand | aws codebuild start-build --project-name CfnTemplatePackageProject-ZGoGzOIfOy4h \n", "+--------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n", "```" ] }, { "cell_type": "markdown", "id": "e6ff322f", "metadata": {}, "source": [ "## 5.2. package-cfn.yaml (CloudFormation 템플릿) 실행 과정\n", "- 아래와 같이 CodeBuild (예: \"CfnTemplatePackageProject-acyQl0hj9nBO\") 코드 빌드 프로젝트가 실행이 되었습니다." ] }, { "cell_type": "markdown", "id": "665284b8", "metadata": {}, "source": [ "![cfn-template.png](img2/cfn-template.png)" ] }, { "cell_type": "markdown", "id": "d166399a", "metadata": {}, "source": [ "## 5.3. \"package-cfn.sh\" 실행 결과\n", "- 위의 CodeBuild는 실질적으로 \"package-cfn.sh\" 를 실행 하였습니다.\n", "- 아래의 로그는 다음과 같이 Zip 파일 생성 작업을 합니다.\n", " - (1) sm-project-sc-portfolio.yaml (카탈로그 포트폴리오 정의 파일) \n", " - (2) project-s3-fs-ingestion.yaml (카탈로그 프러덕트 정의 파일)\n", " - (3) 카탈로그 프러덕트 Seed 코드 (Pipeline 정의 등) 추가\n", "\n", "```\n", "[Container] 2022/02/04 05:49:26 Running command ./package-cfn.sh ${S3_BUCKET_NAME} ${DEPLOYMENT_REGION}\n", "127\tPreparing content for publication to Amazon S3 s3://sagemaker-ap-northeast-2-XXXXXXXXX/amazon-sagemaker-reusable-components\n", "128\tZipping the CloudFormation templates and buildspec files\n", "129\t adding: build/ap-northeast-2/project-s3-fs-ingestion.yaml (deflated 72%)\n", "130\t adding: build/ap-northeast-2/sm-project-sc-portfolio.yaml (deflated 76%)\n", "131\t adding: package-cfn.sh (deflated 66%)\n", "132\t adding: buildspec-package-cfn.yml (deflated 18%)\n", "133\t adding: cfn-templates/project-s3-fs-ingestion.yaml (deflated 72%)\n", "134\t adding: cfn-templates/.ipynb_checkpoints/sm-project-sc-portfolio-checkpoint.yaml (deflated 76%)\n", "135\t adding: cfn-templates/sm-project-sc-portfolio.yaml (deflated 76%)\n", "136\t adding: .ipynb_checkpoints/package-cfn-checkpoint.yaml (deflated 69%)\n", "137\t adding: .ipynb_checkpoints/buildspec-package-cfn-checkpoint.yml (deflated 18%)\n", "138\t adding: project-seed-code/s3-fs-ingestion/buildspec.yml (deflated 54%)\n", "139\t adding: package-cfn.yaml (deflated 69%)\n", "140\tZipping MLOps project seed code\n", "141\t adding: pipeline/ (stored 0%)\n", "142\t adding: pipeline/pipeline.py (deflated 70%)\n", "143\t adding: build.py (deflated 63%)\n", "144\t adding: functions/ (stored 0%)\n", "145\t adding: functions/start_fs_ingestion.py (deflated 58%)\n", "146\t adding: buildspec.yml (deflated 54%)\n", "147\tPublishing CloudFormation to ap-northeast-2\n", "```" ] }, { "cell_type": "markdown", "id": "28f5dcf7", "metadata": {}, "source": [ "![cfn-template-build-log.png](img2/cfn-template-build-log.png)" ] }, { "cell_type": "markdown", "id": "9a8b94e9", "metadata": {}, "source": [ "## 5.4. S3에 생성된 파일 확인" ] }, { "cell_type": "code", "execution_count": 14, "id": "9bfc6cdc", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2022-02-07 03:13:29 19868 amazon-sagemaker-reusable-components/amazon-sagemaker-reusable-components.zip\n", "2022-02-07 01:08:51 15598 amazon-sagemaker-reusable-components/project-s3-fs-ingestion.yaml\n", "2022-02-07 01:08:49 4326 amazon-sagemaker-reusable-components/seed-code/s3-fs-ingestion-v1.0.zip\n", "2022-02-07 01:08:50 7205 amazon-sagemaker-reusable-components/sm-project-sc-portfolio.yaml\n" ] } ], "source": [ "! aws s3 ls s3://{bucket}/amazon-sagemaker-reusable-components/ --recursive" ] }, { "cell_type": "markdown", "id": "867a592e", "metadata": {}, "source": [ "# 6. 변수 저장\n", "- 다음 노트북에서 사용할 변수 저장" ] }, { "cell_type": "code", "execution_count": 15, "id": "d71f3bbd", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Stored 'bucket' (str)\n", "Stored 'bucket_region' (str)\n" ] } ], "source": [ "%store bucket\n", "%store bucket_region" ] }, { "cell_type": "markdown", "id": "6827d76a", "metadata": {}, "source": [ "# 7. Next\n", "- 다음 노트북 \"2.1 Deploy-Catalog-Portfolio-Product.ipynb\" 로 이동 하세요." ] }, { "cell_type": "markdown", "id": "9e5d82db", "metadata": {}, "source": [ "---\n", "# A. 트러블 슈팅 커맨드 및 에러 케이스" ] }, { "cell_type": "markdown", "id": "2fe28ea5", "metadata": {}, "source": [ "## A.1. 클라우드 포메이션 스택 진행 상황, 삭제 및 기타 명령어" ] }, { "cell_type": "code", "execution_count": null, "id": "1bb9d87e", "metadata": {}, "outputs": [], "source": [ "# 스택 이벤트 실행 내역 보여주기\n", "! aws cloudformation describe-stack-events --stack-name amazon-sagemaker-reusable-components-package-cfn" ] }, { "cell_type": "code", "execution_count": null, "id": "1fe043ed", "metadata": {}, "outputs": [], "source": [ "# 스택 삭제\n", "! aws cloudformation delete-stack --stack-name amazon-sagemaker-reusable-components-package-cfn" ] }, { "cell_type": "markdown", "id": "89004ce7", "metadata": {}, "source": [ "## A.2 S3 CLI 관련 명령어" ] }, { "cell_type": "code", "execution_count": null, "id": "bf1193b9", "metadata": {}, "outputs": [], "source": [ "! aws s3 ls https://s3.{bucket_region}.amazonaws.com/{bucket}/amazon-sagemaker-reusable-components/sm-project-sc-portfolio.yaml\n", "# ! aws s3 ls https://s3.ap-southeast-2.amazonaws.com/sagemaker-ap-southeast-2-057716757052/amazon-sagemaker-reusable-components/sm-project-sc-portfolio.yaml" ] }, { "cell_type": "code", "execution_count": null, "id": "1989940d", "metadata": {}, "outputs": [], "source": [ "! aws s3 ls {bucket} --recursive" ] }, { "cell_type": "code", "execution_count": null, "id": "63fc0bb8", "metadata": {}, "outputs": [], "source": [ "! aws s3 rm s3://{bucket} --recursive" ] }, { "cell_type": "markdown", "id": "bdc877a1", "metadata": {}, "source": [ "## A.3. 클라우드 포메이션 실행시 권한 에러\n", "- 아래와 같은 필수 권한을 세이지 메이커 노트북을 실행하는 역할에 추가 해주세요.\n", " - \"arn:aws:iam::aws:policy/AmazonSageMakerFullAccess\"\n", " - \"arn:aws:iam::aws:policy/AmazonS3FullAccess\"\n", " - \"arn:aws:iam::aws:policy/IAMFullAccess\" \n", " - \"arn:aws:iam::aws:policy/AWSCodeBuildAdminAccess\" \n", " - \"arn:aws:iam::aws:policy/AWSServiceCatalogAdminFullAccess\"\n", " - \"arn:aws:iam::aws:policy/AWSCloudFormationFullAccess\"\n", " - \"arn:aws:iam::aws:policy/AWSLambda_FullAccess\"\n", "\n", "\n" ] }, { "cell_type": "markdown", "id": "5a79a3a9", "metadata": {}, "source": [ "## A.4. 에러: seed-code/s3-fs-ingestion-v1.0.zip 생성시 전부 아티펙트를 포함하지 않음.\n", "```\n", "Zipping MLOps project seed code\n", "147\t adding: .ipynb_checkpoints/ (stored 0%)\n", "148\t adding: .ipynb_checkpoints/buildspec-checkpoint.yml (deflated 54%)\n", "149\t adding: buildspec.yml (deflated 54%)\n", "```\n", "정상 케이스\n", "```\n", "Zipping MLOps project seed code\n", " adding: LICENSE (deflated 39%)\n", " adding: pipeline/ (stored 0%)\n", " adding: pipeline/pipeline.py (deflated 70%)\n", " adding: buildspec.yml (deflated 54%)\n", " adding: .ipynb_checkpoints/ (stored 0%)\n", " adding: .ipynb_checkpoints/buildspec-checkpoint.yml (deflated 54%)\n", " adding: build.py (deflated 63%)\n", " adding: README.md (stored 0%)\n", " adding: functions/ (stored 0%)\n", " adding: functions/start_fs_ingestion.py (deflated 58%)\n", "```\n", "\n", "### 해결\n", "- Before\n", " - zip -r amazon-sagemaker-reusable-components.zip . -i \"*.yaml\" \"*.yml\" \"*.sh\"\n", "- After\n", " - zip -r amazon-sagemaker-reusable-components.zip . -i \"*.yaml\" \"*.yml\" \"*.sh\" \"*.py\"" ] }, { "cell_type": "code", "execution_count": null, "id": "e5669651", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "conda_python3", "language": "python", "name": "conda_python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.13" } }, "nbformat": 4, "nbformat_minor": 5 }