{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Amazon Personalize ハンズオン\n", "* 本ハンズオンは Personalize でレコメンデーションするために必要な工程を一通り体験するハンズオンです\n", "* 実行環境の前提として SageMaker Notebook Instance で Jupyter Notebook を使用します\n", "* 構成は下記の通り\n", "\n", "## 手順概要\n", "\n", "0. SageMaker Notebook Instance を立ち上げ、Jupyter Notebook を開いてこのノートブックを開く\n", "1. Personalize にデータを取り込むための S3 バケットを作成する\n", "2. S3 のバケットポリシーを設定し、 Amazon Personalize のサービスがアップロードしたデータにアクセスできるようにする\n", "3. Amazon Personalize の実行ロールを作成し、Amazon Personalize のサービスが使えるようにする\n", "4. S3 にデータをアップロードする\n", "5. S3 から Personalize へデータをインポートするためにスキーマ定義を行い、インポートする\n", "6. ソリューションとバージョンを作成する(学習)\n", "7. 作成したソリューションバージョンからキャンペーンを作成する（ホスティング）\n", "8. 作成したキャンペーンでレコメンデーション機能を試す" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "

" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 環境準備" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 本ハンズオンで利用するライブラリの読み込み" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip install jsonlines\n", "import boto3, json, time, jsonlines\n", "from datetime import date,datetime\n", "import pandas as pd\n", "import numpy as np\n", "print(f'Current boto3 Version ={boto3.__version__}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "上記セルを実行して、boto3 が 1.20.21 未満の場合、以下のセルのコメントアウトを解除してから実行してください。実行が完了したら、上にあるメニューから [Kernel] -> [Restart] を選択してカーネルを再起動してください。" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# !pip install -U \"boto3==1.20.21\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 学習データ保存用S3バケットの作成\n", "#### 概要\n", "* Personalizeがデータインポート時、推論結果出力時に利用するS3バケットの作成と設定を行う\n", "* 作成したバケットに対して、Amazon Personlize のサービスがアクセスできるようバケットポリシーを設定する\n", "\n", "* 本ハンズオンでは personlize-handson-${yourname}-YYYY-MM-DD-hh-mm-ss というバケット名を作成して、利用する\n", "* 下記セルの yourname 変数に任意の名前を入力する\n", "\n", "#### 設定値を変数に格納する" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# 各設定値を変数に格納する\n", "\n", "#------下記変数をご自身の名前で必ず設定ください------#\n", "#----------------------------------------------------#\n", "yourname = 'gokazu' # <- ここに記入をお願いします\n", "#----------------------------------------------------#\n", "#----------------------------------------------------#\n", "\n", "# region は東京リージョンを利用します\n", "region = boto3.session.Session().region_name\n", "\n", "# 説明分の通りのバケット名を bucket_name 変数に格納します\n", "timestamp = datetime.now().strftime('-%Y-%m-%d-%H-%M-%S')\n", "bucket_name = 'personalize-handson-' + yourname + timestamp\n", "print(f'本ハンズオンで利用する\\nバケットは \"{bucket_name}\"')\n", "print(f'リージョンは \"{region}\"\\nです')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### bucket 作成\n", "boto3 の s3 [client](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.BucketPolicy.put) の [create_bucket](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.create_bucket) メソッドを利用して作成する" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "s3_client = boto3.client('s3')\n", "location = {'LocationConstraint': region}\n", "\n", "if region == 'us-east-1':\n", " response = s3_client.create_bucket(\n", " Bucket = bucket_name\n", " )\n", "else:\n", " response = s3_client.create_bucket(\n", " Bucket = bucket_name,\n", " CreateBucketConfiguration = location\n", " )\n", "print(json.dumps(response, indent=2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### バケットポリシーの設定\n", "* Amazon Personalize のサービスが作成した S3 バケットにアクセスできるようバケットポリシーを設定する\n", "* バケットポリシーは [BucketPolicy](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.ServiceResource.BucketPolicy) の [put](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.BucketPolicy.put) メソッドで json 形式で設定する" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# bucket policy 設定\n", "bucket_policy = boto3.resource('s3').BucketPolicy(bucket_name)\n", "\n", "policy = {\n", " \"Version\": \"2012-10-17\",\n", " \"Id\": \"PersonalizeS3Bucket AccessPolicy\",\n", " \"Statement\": [\n", " {\n", " \"Sid\": \"PersonalizeS3BucketAccessPolicy\",\n", " \"Effect\": \"Allow\",\n", " \"Principal\": {\n", " \"Service\": \"personalize.amazonaws.com\"\n", " },\n", " \"Action\": [\n", " \"s3:GetObject\",\n", " \"s3:PutObject\",\n", " \"s3:ListBucket\"\n", " ],\n", " \"Resource\": [\n", " \"arn:aws:s3:::\" + bucket_name,\n", " \"arn:aws:s3:::\" + bucket_name +\"/*\"\n", " ]\n", " }\n", " ]\n", "}\n", "\n", "policy = json.dumps(policy)\n", "\n", "response = bucket_policy.put(\n", " Policy=policy\n", ")\n", "print(json.dumps(response, indent=2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### key_prefix の設定\n", "本ハンズオンで利用する bucket の key_prefix を設定する" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# 後続処理で利用するS3のprefixを指定\n", "s3_key_prefix = 'personalize-demo' + timestamp + '/'\n", "print(f'今日のハンズオンで利用する key prefix は \"{s3_key_prefix}\" です')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Personalize実行ロールの作成\n", "* [Amazon Personalize 用の IAM ロールの作成](https://docs.aws.amazon.com/ja_jp/personalize/latest/dg/aws-personalize-set-up-permissions.html#set-up-required-permissions)パートの手順に従ってPersonalizeの実行ロールを作成する\n", "* IAM ロールの作成は、 boto3 の [create_role](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/iam.html#IAM.Client.create_role) メソッドを利用して作成する\n", "* 作成した IAM ロールに AmazonPersonalizeFullAccess ポリシーを [attach_role_policy](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/iam.html#IAM.Client.attach_role_policy) でアタッチする\n", "* 作成した Personalize ロールの ARN を変数に格納する" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ロール作成\n", "iam = boto3.client('iam')\n", "\n", "assume_role_policy_document = {\n", " 'Version': '2012-10-17',\n", " 'Statement': [{'Sid': '','Effect': 'Allow','Principal': {'Service': 'personalize.amazonaws.com'},'Action': 'sts:AssumeRole'}]\n", "}\n", "\n", "response = iam.create_role(\n", " RoleName = 'personalizeRole' + timestamp,\n", " AssumeRolePolicyDocument = json.dumps(assume_role_policy_document),\n", " Description='using Amazon Personalize handson '+ timestamp.replace('-',''),\n", " MaxSessionDuration=3600*12 # 12時間\n", ")\n", "role_arn = response['Role']['Arn']\n", "role_name = response['Role']['RoleName']\n", "\n", "print(f'作成した role の\\narn は \"{role_arn}\"')\n", "print(f'名前は \"{role_name}\"\\nです')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Policy attach\n", "response = iam.attach_role_policy(\n", " RoleName=role_name,\n", " PolicyArn='arn:aws:iam::aws:policy/service-role/AmazonPersonalizeFullAccess'\n", ")\n", "print(json.dumps(response, indent=2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Personalize boto3 Clientの設定" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Personalize を操作するための client を生成\n", "personalize = boto3.client(service_name='personalize')\n", "personalize_runtime = boto3.client(service_name='personalize-runtime')\n", "personalize_events = boto3.client(service_name='personalize-events')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## データの確認" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### サンプルデータ\n", "* Movielensの10万件の評価データを Personalize で利用できるように事前にデータを加工済\n", "* どのようなデータ形式になっているかを確認する\n", "* 処理内容については本ハンズオンでは扱わないが、[こちら](./data_preparation.ipynb)のノートブックに記載" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Interaction Data\n", "* ユーザがいつどの映画を見たのかを示すデータ\n", "* レコメンドに評価の低かった映画は出したくないので、レートが4以上のものにフィルタしている\n", "* 必須項目は USER_ID,ITEM_ID,TIMESTAMP" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "pd.read_csv('./data/interaction.csv').sort_values(['USER_ID','TIMESTAMP']).reset_index().drop(columns='index').head(50)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Item Data\n", "* アイテムの属性を表すデータ\n", "* 複数の属性を含む場合は一つの列に|(パイプ)で区切って格納する\n", "* 必須項目は ITEM_ID と、1 つ以上の属性情報(今回は GENRE )" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "pd.read_csv('./data/item.csv').head(50)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### User Data\n", "* ユーザの属性を表すデータ\n", "* 必須項目は USER_ID と、1 つ以上の属性情報\n", "* このデータでは年齢、性別、職業、郵便番号を含む" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "pd.read_csv('./data/user.csv').head(50)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Interaction Data, Item Data, User Data の 3 つを S3 にアップロードする\n", "* 確認した 3 ファイルを事前に設定したバケット、key_prefix にアップロードする\n", "* アップロードには [upload_file](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Object.upload_file) を用いる(アップロードするためのメソッドは多種あるがアップロードされればどれでもよい）" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "boto3.Session().resource('s3').Bucket(bucket_name).Object(s3_key_prefix + 'interaction.csv').upload_file('./data/interaction.csv')\n", "boto3.Session().resource('s3').Bucket(bucket_name).Object(s3_key_prefix + 'item.csv').upload_file('./data/item.csv')\n", "boto3.Session().resource('s3').Bucket(bucket_name).Object(s3_key_prefix + 'user.csv').upload_file('./data/user.csv')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Personalize 環境の準備\n", "\n", "まずはデータをインポートするためにデータのスキーマ定義とデータセットグループの作成を行う" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### インタラクションスキーマの定義\n", "* インタラクションデータの csv に沿ったデータスキーマを設定\n", "* 必須項目はアップロードした csv 同様、USER_ID, ITEM_ID, TIME_STAMP の 3 種\n", "\n", "インタラクションスキーマの[参考情報](https://docs.aws.amazon.com/personalize/latest/dg/schema-examples-interactions.html)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "now = datetime.now()\n", "time_str = now.strftime(\"-%Y-%m-%d-%H-%M-%S\")\n", "\n", "interaction_schema = {\n", " \"type\": \"record\",\n", " \"name\": \"Interactions\",\n", " \"namespace\": \"com.amazonaws.personalize.schema\",\n", " \"fields\": [\n", " {\n", " \"name\": \"USER_ID\",\n", " \"type\": \"string\"\n", " },\n", " {\n", " \"name\": \"ITEM_ID\",\n", " \"type\": \"string\"\n", " },\n", " {\n", " \"name\": \"EVENT_TYPE\",\n", " \"type\": \"string\"\n", " },\n", " {\n", " \"name\": \"EVENT_VALUE\",\n", " \"type\": [\n", " \"float\",\n", " \"null\"\n", " ]\n", " },\n", " {\n", " \"name\": \"TIMESTAMP\",\n", " \"type\": \"long\"\n", " }\n", " ],\n", " \"version\": \"1.0\"\n", "}\n", "\n", "create_interaction_schema_response = personalize.create_schema(\n", " name = \"DEMO-movielens-interaction-schema\" + time_str,\n", " schema = json.dumps(interaction_schema)\n", ")\n", "\n", "interaction_schema_arn = create_interaction_schema_response['schemaArn']\n", "print(json.dumps(create_interaction_schema_response, indent=2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### アイテムスキーマの定義\n", "\n", "* アイテムデータの csv に沿ったデータスキーマを設定\n", "* 必須項目はアップロードした csv 同様、ITEM_ID の他、1 つ以上のメタデータ情報\n", "\n", "アイテムスキーマの[参考情報](https://docs.aws.amazon.com/personalize/latest/dg/schema-examples-items.html)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "now = datetime.now()\n", "time_str = now.strftime(\"-%Y-%m-%d-%H-%M-%S\")\n", "\n", "item_schema = {\n", " \"type\": \"record\",\n", " \"name\": \"Items\",\n", " \"namespace\": \"com.amazonaws.personalize.schema\",\n", " \"fields\": [\n", " {\n", " \"name\": \"ITEM_ID\",\n", " \"type\": \"string\"\n", " },\n", " {\n", " \"name\": \"GENRE\",\n", " \"type\": \"string\",\n", " \"categorical\": True\n", " }\n", " ],\n", " \"version\": \"1.0\"\n", "}\n", "\n", "create_item_schema_response = personalize.create_schema(\n", " name = \"DEMO-movielens-item-schema\" + time_str,\n", " schema = json.dumps(item_schema)\n", ")\n", "\n", "item_schema_arn = create_item_schema_response['schemaArn']\n", "print(json.dumps(create_item_schema_response, indent=2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### ユーザースキーマの定義\n", "\n", "* ユーザデータの csv に沿ったデータスキーマを設定\n", "* 必須項目はアップロードした csv 同様、USER_ID の他、1 つ以上のメタデータ情報\n", "\n", "ユーザスキーマの[参考情報](https://docs.aws.amazon.com/personalize/latest/dg/schema-examples-users.html)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "now = datetime.now()\n", "time_str = now.strftime(\"-%Y-%m-%d-%H-%M-%S\")\n", "\n", "user_schema = {\n", " \"type\": \"record\",\n", " \"name\": \"Users\",\n", " \"namespace\": \"com.amazonaws.personalize.schema\",\n", " \"fields\": [\n", " {\n", " \"name\": \"USER_ID\",\n", " \"type\": \"string\"\n", " },\n", " {\n", " \"name\": \"AGE\",\n", " \"type\": \"int\"\n", " },\n", " {\n", " \"name\": \"GENDER\",\n", " \"type\": \"string\",\n", " \"categorical\": True\n", " },\n", " {\n", " \"name\": \"JOB\",\n", " \"type\": \"string\",\n", " \"categorical\": True\n", " },\n", " {\n", " \"name\": \"ZIP\",\n", " \"type\": \"string\",\n", " \"categorical\": True\n", " }\n", " ],\n", " \"version\": \"1.0\"\n", "}\n", "\n", "create__user_schema_response = personalize.create_schema(\n", " name = \"DEMO-movielens-user-schema\" + time_str,\n", " schema = json.dumps(user_schema)\n", ")\n", "\n", "user_schema_arn = create__user_schema_response['schemaArn']\n", "print(json.dumps(create__user_schema_response, indent=2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### データグループの作成\n", "* インタラクションデータ、アイテムデータ、ユーザデータを束ねるデータグループを作成\n", "* [参考情報](https://docs.aws.amazon.com/personalize/latest/dg/data-prep-ds-group.html)\n", "* [create_dataset_group](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/personalize.html#Personalize.Client.create_dataset_group) メソッドはデータグループを作成するためのジョブを実行する非同期メソッド\n", "* 完了を確認するために、別途 [describe_dataset_group](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/personalize.html#Personalize.Client.describe_dataset_group) を定期的に呼び出してジョブの実行結果を確認しにいく\n", "\n", "ただし、データグループ作成自体はすぐに完了（秒レベル）する" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "create_dataset_group_response = personalize.create_dataset_group(\n", " name = \"DEMO-movielens-dataset-group\" + timestamp\n", ")\n", "\n", "dataset_group_arn = create_dataset_group_response['datasetGroupArn']\n", "print(json.dumps(create_dataset_group_response, indent=2))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%time\n", "status = None\n", "max_time = time.time() + 3*60*60 # 3 hours\n", "while time.time() < max_time:\n", " describe_dataset_group_response = personalize.describe_dataset_group(\n", " datasetGroupArn = dataset_group_arn\n", " )\n", " status = describe_dataset_group_response[\"datasetGroup\"][\"status\"] \n", " if status == \"ACTIVE\" or status == \"CREATE FAILED\":\n", " print('!')\n", " print(f\"CreateDatasetGroup: {status}\")\n", " break\n", " else:\n", " print('.',end='')\n", " time.sleep(1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### インタラクションデータセットの作成\n", "データセットグループの中にインタラクションデータセットを作成する \n", "必要な情報は下記 4 つ\n", "* データセットのタイプ（この場合はインタラクションデータであることを指定）\n", "* 親となるデータセットグループの arn\n", "* 事前に定義したインタラクションデータのスキーマの arn\n", "* 作成するデータセットのユニークな名前\n", "\n", "詳細 : [create_dataset](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/personalize.html#Personalize.Client.create_dataset)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "now = datetime.now()\n", "time_str = now.strftime(\"-%Y-%m-%d-%H-%M-%S\")\n", "\n", "interaction_dataset_type = \"Interactions\"\n", "create_interaction_dataset_response = personalize.create_dataset(\n", " datasetType = interaction_dataset_type,\n", " datasetGroupArn = dataset_group_arn,\n", " name = 'DEMO-movielens-interactions',\n", " schemaArn = interaction_schema_arn\n", ")\n", "\n", "interaction_dataset_arn = create_interaction_dataset_response['datasetArn']\n", "print(json.dumps(create_interaction_dataset_response, indent=2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### インタラクションデータセットのインポート\n", "作成したインタラクションデータセットにインタラクションデータをインポートする(ただし事前に dataset が作成し終えていないとエラーで落ちるため、dataset の作成が完了するまで待つ処理を入れている) \n", "必須項目は下記 4 つ\n", "* インポートジョブ名\n", "* インタラクションデータセットの arn\n", "* インポートするデータの S3 URI\n", "* インポートジョブを実行するロールの arn\n", "\n", "[create_dataset_import_job](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/personalize.html#Personalize.Client.create_dataset_import_job) はジョブを実行するだけの非同期メソッドのため、完了確認は別途行う" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "while True:\n", " status = personalize.describe_dataset(datasetArn=interaction_dataset_arn)['dataset']['status']\n", " if status == 'ACTIVE':\n", " print('!')\n", " print(status)\n", " break\n", " else:\n", " print('.',end='')\n", " time.sleep(5)\n", "\n", "now = datetime.now()\n", "time_str = now.strftime(\"-%Y-%m-%d-%H-%M-%S\")\n", "\n", "create_interaction_dataset_import_job_response = personalize.create_dataset_import_job(\n", " jobName = \"DEMO-interaction-dataset-import-job\" + time_str,\n", " datasetArn = interaction_dataset_arn,\n", " dataSource = {\n", " \"dataLocation\": \"s3://{}/{}\".format(bucket_name, s3_key_prefix + 'interaction.csv')\n", " },\n", " roleArn = role_arn\n", ")\n", "\n", "interaction_dataset_import_job_arn = create_interaction_dataset_import_job_response['datasetImportJobArn']\n", "print(json.dumps(create_interaction_dataset_import_job_response, indent=2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### ユーザデータセットの作成\n", "\n", "データセットグループの中にユーザデータセットを作成する \n", "必要な情報は下記 4 つ\n", "* データセットのタイプ（この場合はユーザデータであることを指定）\n", "* 親となるデータセットグループの arn\n", "* 事前に定義したユーザデータのスキーマの arn\n", "* 作成するデータセットのユニークな名前\n", "\n", "詳細 : [create_dataset](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/personalize.html#Personalize.Client.create_dataset)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "user_dataset_type = \"Users\"\n", "create_user_dataset_response = personalize.create_dataset(\n", " datasetType = user_dataset_type,\n", " datasetGroupArn = dataset_group_arn,\n", " name = 'DEMO-movielens-user',\n", " schemaArn = user_schema_arn\n", ")\n", "\n", "user_dataset_arn = create_user_dataset_response['datasetArn']\n", "print(json.dumps(create_user_dataset_response, indent=2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### ユーザーデータセットのインポート\n", "\n", "作成したユーザデータセットにインタラクションデータをインポートする(ただし事前に dataset が作成し終えていないとエラーで落ちるため、dataset の作成が完了するまで待つ処理を入れている) \n", "必須項目は下記 4 つ\n", "* インポートジョブ名\n", "* ユーザデータセットの arn\n", "* インポートするデータの S3 URI\n", "* インポートジョブを実行するロールの arn\n", "\n", "[create_dataset_import_job](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/personalize.html#Personalize.Client.create_dataset_import_job) はジョブを実行するだけの非同期メソッドのため、完了確認は別途行う" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "while True:\n", " status = personalize.describe_dataset(datasetArn=user_dataset_arn)['dataset']['status']\n", " if status == 'ACTIVE':\n", " print('!')\n", " print(status)\n", " break\n", " else:\n", " print('.',end='')\n", " time.sleep(5)\n", "\n", "now = datetime.now()\n", "time_str = now.strftime(\"-%Y-%m-%d-%H-%M-%S\")\n", "\n", "create_user_dataset_import_job_response = personalize.create_dataset_import_job(\n", " jobName = \"DEMO-user-dataset-import-job\" + time_str,\n", " datasetArn = user_dataset_arn,\n", " dataSource = {\n", " \"dataLocation\": \"s3://{}/{}\".format(bucket_name, s3_key_prefix + 'user.csv')\n", " },\n", " roleArn = role_arn\n", ")\n", "\n", "user_dataset_import_job_arn = create_user_dataset_import_job_response['datasetImportJobArn']\n", "print(json.dumps(create_user_dataset_import_job_response, indent=2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### アイテムデータセットの作成\n", "\n", "データセットグループの中にアイテムデータセットを作成する\n", "必要な情報は下記 4 つ\n", "* データセットのタイプ（この場合はアイテムデータであることを指定）\n", "* 親となるデータセットグループの arn\n", "* 事前に定義したアイテムデータのスキーマの arn\n", "* 作成するデータセットのユニークな名前\n", "\n", "詳細 : [create_dataset](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/personalize.html#Personalize.Client.create_dataset)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "item_dataset_type = \"Items\"\n", "create_item_dataset_response = personalize.create_dataset(\n", " datasetType = item_dataset_type,\n", " datasetGroupArn = dataset_group_arn,\n", " name = 'DEMO-movielens-item',\n", " schemaArn = item_schema_arn\n", ")\n", "\n", "item_dataset_arn = create_item_dataset_response['datasetArn']\n", "print(json.dumps(create_item_dataset_response, indent=2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### アイテムデータセットのインポート\n", "\n", "作成したアイテムデータセットにインタラクションデータをインポートする(ただし事前に dataset が作成し終えていないとエラーで落ちるため、dataset の作成が完了するまで待つ処理を入れている) \n", "必須項目は下記 4 つ\n", "* インポートジョブ名\n", "* アイテムデータセットの arn\n", "* インポートするデータの S3 URI\n", "* インポートジョブを実行するロールの arn\n", "\n", "[create_dataset_import_job](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/personalize.html#Personalize.Client.create_dataset_import_job) はジョブを実行するだけの非同期メソッドのため、完了確認は別途行う" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "while True:\n", " status = personalize.describe_dataset(datasetArn=item_dataset_arn)['dataset']['status']\n", " if status == 'ACTIVE':\n", " print('!')\n", " print(status)\n", " break\n", " else:\n", " print('.',end='')\n", " time.sleep(5)\n", "\n", "now = datetime.now()\n", "time_str = now.strftime(\"-%Y-%m-%d-%H-%M-%S\")\n", "\n", "create_item_dataset_import_job_response = personalize.create_dataset_import_job(\n", " jobName = \"DEMO-item-dataset-import-job\" + time_str,\n", " datasetArn = item_dataset_arn,\n", " dataSource = {\n", " \"dataLocation\": \"s3://{}/{}\".format(bucket_name, s3_key_prefix + 'item.csv')\n", " },\n", " roleArn = role_arn\n", ")\n", "\n", "item_dataset_import_job_arn = create_item_dataset_import_job_response['datasetImportJobArn']\n", "print(json.dumps(create_item_dataset_import_job_response, indent=2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### インタラクションデータのインポート完了を待つ\n", "* create_dataset_import_job はインポートジョブをキックするだけの非同期メソッド\n", "* 終わるまでソリューションのバージョンの作成が出来ないため、終了を確認する処理を行う\n", "* describe_dataset_import_job でジョブのステータスが確認できるので、 ACTIVE になるまで待つ\n", "* [describe_dataset_import_job](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/personalize.html#Personalize.Client.describe_dataset_import_job) に必要な引数はデータセットインポートジョブの arn のみ\n", "* 下記セルはインタラクションデータセットのインポートジョブの完了確認だが、アイテムもユーザも同様\n", "* 15分程度待つ" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%time\n", "\n", "status = None\n", "max_time = time.time() + 3*60*60 # 3 hours\n", "while time.time() < max_time:\n", " describe_dataset_import_job_response = personalize.describe_dataset_import_job(\n", " datasetImportJobArn = interaction_dataset_import_job_arn\n", " )\n", " \n", " dataset_import_job = describe_dataset_import_job_response[\"datasetImportJob\"]\n", " status = dataset_import_job[\"status\"]\n", " \n", " if status == \"ACTIVE\" or status == \"CREATE FAILED\":\n", " print('!')\n", " print(f\"LatestDatasetImportJobRun: {status}\")\n", " break\n", " else:\n", " print('.',end='')\n", " \n", " time.sleep(60)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### ユーザーデータのインポート完了を待つ" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%time\n", "\n", "status = None\n", "max_time = time.time() + 3*60*60 # 3 hours\n", "while time.time() < max_time:\n", " describe_dataset_import_job_response = personalize.describe_dataset_import_job(\n", " datasetImportJobArn = user_dataset_import_job_arn\n", " )\n", " \n", " dataset_import_job = describe_dataset_import_job_response[\"datasetImportJob\"]\n", " status = dataset_import_job[\"status\"]\n", " \n", " if status == \"ACTIVE\" or status == \"CREATE FAILED\":\n", " print('!')\n", " print(f\"LatestDatasetImportJobRun: {status}\")\n", " break\n", " else:\n", " print('.',end='')\n", " \n", " time.sleep(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### アイテムデータのインポート完了を待つ" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%time\n", "\n", "status = None\n", "max_time = time.time() + 3*60*60 # 3 hours\n", "while time.time() < max_time:\n", " describe_dataset_import_job_response = personalize.describe_dataset_import_job(\n", " datasetImportJobArn = item_dataset_import_job_arn\n", " )\n", " \n", " dataset_import_job = describe_dataset_import_job_response[\"datasetImportJob\"]\n", " status = dataset_import_job[\"status\"]\n", " \n", " if status == \"ACTIVE\" or status == \"CREATE FAILED\":\n", " print('!')\n", " print(f\"LatestDatasetImportJobRun: {status}\")\n", " break\n", " else:\n", " print('.',end='')\n", " \n", " time.sleep(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### ソリューションの作成\n", "* 各ソリューションのトレーニング完了までには一定時間掛かるため、レシピ毎にトレーニングジョブを並列で起動し、後続のステップですべてのトレーニングジョブが完了するのを待つ\n", "* ソリューションを作成するにあたって、各ソリューションに使うレシピの arn が必要なので最初にレシピの arn を取得する" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import boto3\n", "paginator = personalize.get_paginator('list_recipes')\n", "recipe_arns = {}\n", "for page in paginator.paginate():\n", " for recipe in page['recipes']:\n", " recipe_arns[recipe['name']] = recipe['recipeArn']\n", "print(recipe_arns)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### ソリューションの作成（user-personalization）\n", "* user-personalization レシピを使ったソリューションを [create_solution](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/personalize.html#Personalize.Client.create_solution)メソッドを使って作成する\n", "* create_solution メソッドの必須項目は下記の通り\n", " * ソリューションのユニークな名前\n", " * データセットグループの arn\n", " * AutoMLを利用しないときはレシピの arn\n", "* performHPO は現状デフォルトで False だが、念の為明示的に False を指定\n", " * HPO は時間がかかるため、本ハンズオンでは使用しない\n", "* この段階ではあくまでソリューションの箱を作るだけで実際に学習を開始するのは次の処理" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "now = datetime.now()\n", "time_str = now.strftime(\"-%Y-%m-%d-%H-%M-%S\")\n", "\n", "user_personalization_create_solution_response = personalize.create_solution(\n", " name = \"DEMO-movielens-user-personalization\" + time_str,\n", " datasetGroupArn = dataset_group_arn,\n", " recipeArn = 'arn:aws:personalize:::recipe/aws-user-personalization',\n", " performHPO = False\n", ")\n", "\n", "user_personalization_solution_arn = user_personalization_create_solution_response['solutionArn']\n", "print(json.dumps(user_personalization_create_solution_response, indent=2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### user-personalization ソリューションのバージョンの作成\n", "* **時間のかかる処理をキックします**\n", "* 非同期メソッドの実行なのでバージョン作成の完了確認は別途実行\n", "* [create_solution_verion](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/personalize.html#Personalize.Client.create_solution_version) の実行には作成したソリューションの arn を指定するだけでよい" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "user_personalization_create_solution_version_response = personalize.create_solution_version(\n", " solutionArn=user_personalization_solution_arn\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "user_personalization_solution_version_arn = user_personalization_create_solution_version_response['solutionVersionArn']\n", "print(json.dumps(user_personalization_create_solution_version_response, indent=2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### personalized-ranking ソリューション/ソリューションバージョンの作成" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "now = datetime.now()\n", "time_str = now.strftime(\"-%Y-%m-%d-%H-%M-%S\")\n", "# ソリューション作成\n", "personalized_ranking_create_solution_response = personalize.create_solution(\n", " name = \"DEMO-movielens-personalized-ranking\" + time_str,\n", " datasetGroupArn = dataset_group_arn,\n", " recipeArn = recipe_arns[\"aws-personalized-ranking\"],\n", " performHPO = False\n", ")\n", "personalized_ranking_solution_arn = personalized_ranking_create_solution_response['solutionArn']\n", "print(json.dumps(personalized_ranking_create_solution_response, indent=2))\n", "# ソリューションバージョン作成\n", "personalized_ranking_create_solution_version_response = personalize.create_solution_version(solutionArn=personalized_ranking_solution_arn)\n", "personalized_ranking_solution_version_arn = personalized_ranking_create_solution_version_response['solutionVersionArn']\n", "print(json.dumps(personalized_ranking_create_solution_version_response, indent=2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### sims ソリューション/ソリューションバージョンの作成" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "now = datetime.now()\n", "time_str = now.strftime(\"-%Y-%m-%d-%H-%M-%S\")\n", "# ソリューション作成\n", "sims_create_solution_response = personalize.create_solution(\n", " name = \"DEMO-movielens-sims\" + time_str,\n", " datasetGroupArn = dataset_group_arn,\n", " recipeArn = recipe_arns[\"aws-sims\"],\n", " performHPO = False\n", ")\n", "sims_solution_arn = sims_create_solution_response['solutionArn']\n", "print(json.dumps(sims_create_solution_response, indent=2))\n", "# ソリューションバージョン作成\n", "sims_create_solution_version_response = personalize.create_solution_version(solutionArn=sims_solution_arn)\n", "sims_solution_version_arn = sims_create_solution_version_response['solutionVersionArn']\n", "print(json.dumps(sims_create_solution_version_response, indent=2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### affinity ソリューション/ソリューションバージョンの作成" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "now = datetime.now()\n", "time_str = now.strftime(\"-%Y-%m-%d-%H-%M-%S\")\n", "# ソリューション作成\n", "affinity_create_solution_response = personalize.create_solution(\n", " name = \"DEMO-movielens-affinity\" + time_str,\n", " datasetGroupArn = dataset_group_arn,\n", " recipeArn = recipe_arns[\"aws-item-affinity\"],\n", " performHPO = False\n", ")\n", "affinity_solution_arn = affinity_create_solution_response['solutionArn']\n", "print(json.dumps(affinity_create_solution_response, indent=2))\n", "# ソリューションバージョン作成\n", "affinity_create_solution_version_response = personalize.create_solution_version(solutionArn=affinity_solution_arn)\n", "affinity_solution_version_arn = affinity_create_solution_version_response['solutionVersionArn']\n", "print(json.dumps(affinity_create_solution_version_response, indent=2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### attribute affinity ソリューション/ソリューションバージョンの作成" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "now = datetime.now()\n", "time_str = now.strftime(\"-%Y-%m-%d-%H-%M-%S\")\n", "\n", "attribute_affinity_create_solution_response = personalize.create_solution(\n", " name = \"DEMO-movielens-attribute-affinity\" + time_str,\n", " datasetGroupArn = dataset_group_arn,\n", " recipeArn = recipe_arns[\"aws-item-attribute-affinity\"],\n", " performHPO = False\n", ")\n", "# ソリューション作成\n", "attribute_affinity_solution_arn = attribute_affinity_create_solution_response['solutionArn']\n", "print(json.dumps(attribute_affinity_solution_arn, indent=2))\n", "# ソリューションバージョン作成\n", "attribute_affinity_create_solution_version_response = personalize.create_solution_version(solutionArn=attribute_affinity_solution_arn)\n", "attribute_affinity_solution_version_arn = attribute_affinity_create_solution_version_response['solutionVersionArn']\n", "print(json.dumps(attribute_affinity_create_solution_version_response, indent=2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### user-personalization レシピのトレーニング完了を待つ\n", "* ソリューションのバージョン作成は非同期メソッドのため完了を確認する処理を別途行う\n", "* [describe_solution_version](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/personalize.html#Personalize.Client.describe_solution_version) でステータスを取得可能\n", "* 引数はソリューションバージョンの arn のみ\n", "\n", "**40 分くらい待ちます**" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%time\n", "\n", "status = None\n", "max_time = time.time() + 3*60*60 # 3 hours\n", "while time.time() < max_time:\n", " describe_solution_version_response = personalize.describe_solution_version(\n", " solutionVersionArn = user_personalization_solution_version_arn\n", " )\n", " status = describe_solution_version_response[\"solutionVersion\"][\"status\"]\n", " \n", " if status == \"ACTIVE\" or status == \"CREATE FAILED\":\n", " print('!')\n", " print(f\"user-personalize solution version : {status}\")\n", " break\n", " else:\n", " print('.',end='')\n", " \n", " time.sleep(60)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### personalized-ranking レシピのトレーニング完了を待つ" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%time\n", "\n", "status = None\n", "max_time = time.time() + 3*60*60 # 3 hours\n", "while time.time() < max_time:\n", " describe_solution_version_response = personalize.describe_solution_version(\n", " solutionVersionArn = personalized_ranking_solution_version_arn\n", " )\n", " status = describe_solution_version_response[\"solutionVersion\"][\"status\"]\n", " \n", " if status == \"ACTIVE\" or status == \"CREATE FAILED\":\n", " print('!')\n", " print(\"personalize-ranking solution version : {}\".format(status))\n", " break\n", " else:\n", " print('.',end='')\n", " \n", " time.sleep(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### sims レシピのトレーニング完了を待つ" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%time\n", "\n", "status = None\n", "max_time = time.time() + 3*60*60 # 3 hours\n", "while time.time() < max_time:\n", " describe_solution_version_response = personalize.describe_solution_version(\n", " solutionVersionArn = sims_solution_version_arn\n", " )\n", " status = describe_solution_version_response[\"solutionVersion\"][\"status\"]\n", " \n", " if status == \"ACTIVE\" or status == \"CREATE FAILED\":\n", " print('!')\n", " print(\"sims solution version : {}\".format(status))\n", " break\n", " else:\n", " print('.',end='')\n", " \n", " time.sleep(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### affinity レシピのトレーニング完了を待つ" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%time\n", "\n", "status = None\n", "max_time = time.time() + 3*60*60 # 3 hours\n", "while time.time() < max_time:\n", " describe_solution_version_response = personalize.describe_solution_version(\n", " solutionVersionArn = affinity_solution_version_arn\n", " )\n", " status = describe_solution_version_response[\"solutionVersion\"][\"status\"]\n", " \n", " if status == \"ACTIVE\" or status == \"CREATE FAILED\":\n", " print('!')\n", " print(\"sims solution version : {}\".format(status))\n", " break\n", " else:\n", " print('.',end='')\n", " \n", " time.sleep(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### attribute affinity レシピのトレーニング完了を待つ" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%time\n", "\n", "status = None\n", "max_time = time.time() + 3*60*60 # 3 hours\n", "while time.time() < max_time:\n", " describe_solution_version_response = personalize.describe_solution_version(\n", " solutionVersionArn = attribute_affinity_solution_version_arn\n", " )\n", " status = describe_solution_version_response[\"solutionVersion\"][\"status\"]\n", " \n", " if status == \"ACTIVE\" or status == \"CREATE FAILED\":\n", " print('!')\n", " print(\"sims solution version : {}\".format(status))\n", " break\n", " else:\n", " print('.',end='')\n", " \n", " time.sleep(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### user-personalization ソリューションメトリックの取得\n", "* 学習したソリューションのバージョンのメトリックを [get_solution_metrics](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/personalize.html#Personalize.Client.get_solution_metrics) で取得することが可能\n", "* 引数はソリューションバージョンの arn のみ\n", "* metrics についての概要は [URL](https://docs.aws.amazon.com/ja_jp/personalize/latest/dg/working-with-training-metrics.html) を参考" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "get_solution_metrics_response = personalize.get_solution_metrics(\n", " solutionVersionArn = user_personalization_solution_version_arn\n", ")\n", "\n", "print(json.dumps(get_solution_metrics_response, indent=2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### personalized-ranking ソリューションメトリックの取得" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "get_solution_metrics_response = personalize.get_solution_metrics(\n", " solutionVersionArn = personalized_ranking_solution_version_arn\n", ")\n", "\n", "print(json.dumps(get_solution_metrics_response, indent=2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### sims ソリューションメトリックの取得" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "get_solution_metrics_response = personalize.get_solution_metrics(\n", " solutionVersionArn = sims_solution_version_arn\n", ")\n", "\n", "print(json.dumps(get_solution_metrics_response, indent=2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### affinity ソリューションメトリックの取得" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "get_solution_metrics_response = personalize.get_solution_metrics(\n", " solutionVersionArn = affinity_solution_version_arn\n", ")\n", "\n", "print(json.dumps(get_solution_metrics_response, indent=2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### attribute affinity ソリューションメトリックの取得" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "get_solution_metrics_response = personalize.get_solution_metrics(\n", " solutionVersionArn = attribute_affinity_solution_version_arn\n", ")\n", "\n", "print(json.dumps(get_solution_metrics_response, indent=2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### キャンペーンの作成\n", "* 各キャンペーン作成完了までには一定時間掛かるため、キャンペーンの作成をを並列実行し、後続のステップですべてのキャンペーン作成完了を待つ\n", "* キャンペーンを作成することでレコメンデーションができるようになる" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### user-personalization キャンペーンの作成\n", "* キャンペーンの作成は [create_campaign](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/personalize.html#Personalize.Client.create_campaign) メソッドを使う(非同期メソッドのため完了は別途確認)\n", "* 必須項目は 3 つ\n", " * キャンペーンのユニークな名前\n", " * 適用するソリューションバージョンの arn\n", " * 最低限確保する TPS ( 1 秒あたりにさばけるトランザクション量）\n", " * [exploration](https://docs.aws.amazon.com/ja_jp/personalize/latest/dg/native-recipe-new-item-USER_PERSONALIZATION.html) の設定について" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "now = datetime.now()\n", "time_str = now.strftime(\"-%Y-%m-%d-%H-%M-%S\")\n", "\n", "create_user_personalization_campaign_response = personalize.create_campaign(\n", " name = \"DEMO-campaign-user-personalization\" + time_str,\n", " solutionVersionArn = user_personalization_solution_version_arn,\n", " minProvisionedTPS = 1,\n", " campaignConfig = { \n", " \"itemExplorationConfig\": { \n", " \"explorationWeight\" : \"0.3\",\n", " \"explorationItemAgeCutOff\" : \"30.0\",\n", " }\n", " },\n", ")\n", "\n", "user_personalization_campaign_arn = create_user_personalization_campaign_response['campaignArn']\n", "print(json.dumps(create_user_personalization_campaign_response, indent=2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### sims キャンペーンの作成" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "now = datetime.now()\n", "time_str = now.strftime(\"-%Y-%m-%d-%H-%M-%S\")\n", "\n", "create_sims_campaign_response = personalize.create_campaign(\n", " name = \"DEMO-campaign-sims\" + time_str,\n", " solutionVersionArn = sims_solution_version_arn,\n", " minProvisionedTPS = 1\n", ")\n", "\n", "sims_campaign_arn = create_sims_campaign_response['campaignArn']\n", "print(json.dumps(create_sims_campaign_response, indent=2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### personalized-ranking キャンペーンの作成" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "now = datetime.now()\n", "time_str = now.strftime(\"-%Y-%m-%d-%H-%M-%S\")\n", "\n", "create_personalized_ranking_campaign_response = personalize.create_campaign(\n", " name = \"DEMO-campaign-personalized-ranking\" + time_str,\n", " solutionVersionArn = personalized_ranking_solution_version_arn,\n", " minProvisionedTPS = 1\n", ")\n", "\n", "personalized_ranking_campaign_arn = create_personalized_ranking_campaign_response['campaignArn']\n", "print(json.dumps(create_personalized_ranking_campaign_response, indent=2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### user-personalization キャンペーンの作成完了待ち\n", "* キャンペーン作成のステータスは [describe_campaign](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/personalize.html#Personalize.Client.describe_campaign) で取得できる\n", "* キャンペーンの arn を指定するのみ\n", "* キャンペーンの作成が完了するまで 10 分程度かかる" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%time\n", "\n", "status = None\n", "max_time = time.time() + 3*60*60 # 3 hours\n", "while time.time() < max_time:\n", " describe_campaign_response = personalize.describe_campaign(\n", " campaignArn = user_personalization_campaign_arn\n", " )\n", " status = describe_campaign_response[\"campaign\"][\"status\"]\n", " \n", " if status == \"ACTIVE\" or status == \"CREATE FAILED\":\n", " print('!')\n", " print(f\"LatestDatasetImportJobRun: {status}\")\n", " break\n", " else:\n", " print('.',end='')\n", " \n", " time.sleep(60)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### sims キャンペーンの作成完了待ち" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%time\n", "\n", "status = None\n", "max_time = time.time() + 3*60*60 # 3 hours\n", "while time.time() < max_time:\n", " describe_campaign_response = personalize.describe_campaign(\n", " campaignArn = sims_campaign_arn\n", " )\n", " status = describe_campaign_response[\"campaign\"][\"status\"]\n", " \n", " if status == \"ACTIVE\" or status == \"CREATE FAILED\":\n", " print('!')\n", " print(\"LatestDatasetImportJobRun: {}\".format(status))\n", " break\n", " else:\n", " print('.',end='')\n", " \n", " time.sleep(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### personalizad-ranking キャンペーンの作成完了待ち" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%time\n", "\n", "status = None\n", "max_time = time.time() + 3*60*60 # 3 hours\n", "while time.time() < max_time:\n", " describe_campaign_response = personalize.describe_campaign(\n", " campaignArn = personalized_ranking_campaign_arn\n", " )\n", " status = describe_campaign_response[\"campaign\"][\"status\"]\n", " \n", " if status == \"ACTIVE\" or status == \"CREATE FAILED\":\n", " print('!')\n", " print(\"LatestDatasetImportJobRun: {}\".format(status))\n", " break\n", " else:\n", " print('.',end='')\n", " \n", " time.sleep(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### APIの呼び出し\n", "* ここからは、実際にレコメンデーションをする\n", "* レコメンデーションは boto3 の personalize-runtime を用いる" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# 事前にアイテムとユーザのデータを読み込む\n", "item_data = pd.read_csv('./ml-100k/u.item', sep='|', encoding='latin-1', header=None)\n", "item_data = item_data.drop(columns=3)\n", "item_data = item_data.rename(columns={0:'ITEM_ID', 1:'TITLE', 2:'RELEASE', 4:'IMDB_URL', 5:'unknown', 6:'Action', 7:'Adventure', 8:'Animation', 9:\"Children's\", 10:'Comedy', 11:'Crime', 12:'Documentary', 13:'Drama', 14:'Fantasy', 15:'Film-Noir', 16:'Horror', 17:'Musical', 18:'Mystery', 19:'Romance', 20:'Sci-Fi', 21:'Thriller', 22:'War', 23:'Western'})\n", "item_data.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "user_data = pd.read_csv('./ml-100k/u.user', sep='|', names=['USER_ID', 'AGE', 'GENDER', 'JOB', 'ZIP'])\n", "user_data.head(11)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### ユーザー向けレコメンデーションAPIの呼び出し\n", "* ユーザ ID が 10 の人には何をレコメンデーションするべきか、というサンプル\n", "* [get_recommendations](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/personalize-runtime.html#PersonalizeRuntime.Client.get_recommendations) メソッドに、 USER-PERSONALIZATION レシピを使用しているキャンペーンの arn と、ユーザ ID を指定する\n", "* レコメンデーションを返してくれるアイテム数はデフォルトで 25 だが、最大で 500 まで増やせる" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "get_recommendations_response = personalize_runtime.get_recommendations(\n", " campaignArn = user_personalization_campaign_arn,\n", " userId = '10'\n", ")\n", "\n", "item_list = get_recommendations_response['itemList']\n", "title_list = [item_data.loc[item_data['ITEM_ID'] == np.int(item['itemId'])].values[0][1] for item in item_list]\n", "\n", "print(\"Recommendations: {}\".format(json.dumps(title_list, indent=2)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 類似アイテムAPIの呼び出し\n", "\n", "* [get_recommendations](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/personalize-runtime.html#PersonalizeRuntime.Client.get_recommendations) メソッドを sims のキャンペーンの arn を指定し、類似アイテムの元となる itemId を指定して呼び出す" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "get_recommendations_response = personalize_runtime.get_recommendations(\n", " campaignArn = sims_campaign_arn,\n", " itemId = '30' # ItemId が 30 の商品の類似商品を出す\n", ")\n", "\n", "item_list = get_recommendations_response['itemList']\n", "title_list = [item_data.loc[item_data['ITEM_ID'] == np.int(item['itemId'])].values[0][1] for item in item_list]\n", "\n", "print(item_data.loc[item_data['ITEM_ID'] == 30].TITLE.values[0])\n", "print(\"Recommendations: {}\".format(json.dumps(title_list, indent=2)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Personalized Ranking APIの呼び出し\n", "* 勧めたい商品が複数あったときにどの優先順位で勧めればよいかを知る\n", "* [get_personalized_ranking](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/personalize-runtime.html#PersonalizeRuntime.Client.get_personalized_ranking) を利用する\n", "* キャンペーンの arn, ユーザ ID, 優先順位を知りたいアイテムのリストを引数に呼び出す" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "input_item_list = [str(i) for i in range(1,11)]\n", "\n", "get_personalized_ranking_response = personalize_runtime.get_personalized_ranking(\n", " campaignArn = personalized_ranking_campaign_arn,\n", " userId = '20', # UserId が 20 の人にとってのランキング\n", " inputList = input_item_list\n", ")\n", "\n", "item_list = get_personalized_ranking_response['personalizedRanking']\n", "item_id_list = [item['itemId'] for item in item_list]\n", "\n", "print(\"PersonalizedRanking: {}\".format(json.dumps(item_id_list, indent=2)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### リアルタイムイベント処理\n", "#### EventTrackerの作成\n", "* リアルタイムのお客様のアクションを取り込んでレコメンデーションを行うための、EventTracker を作成する\n", "* [create_event_tracker](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/personalize.html#Personalize.Client.create_event_tracker) メソッドで作成できる\n", "* データセットグループの arn と、ユニークな名前を指定して実行する\n", "* 30 秒程度で完了する" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "create_event_tracker_response = personalize.create_event_tracker(\n", " datasetGroupArn = dataset_group_arn,\n", " name = 'DEMO-event-tracker'\n", ")\n", "\n", "print(json.dumps(create_event_tracker_response, indent=2))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "event_tracker_arn = create_event_tracker_response['eventTrackerArn']" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%time\n", "\n", "status = None\n", "max_time = time.time() + 3*60*60 # 3 hours\n", "while time.time() < max_time:\n", " describe_event_tracker_response = personalize.describe_event_tracker(\n", " eventTrackerArn = event_tracker_arn\n", " )\n", " status = describe_event_tracker_response[\"eventTracker\"][\"status\"]\n", " \n", " if status == \"ACTIVE\" or status == \"CREATE FAILED\":\n", " print('!')\n", " print(f'status:{status}')\n", " break\n", " print('.',end = '')\n", " time.sleep(5)\n", "tracking_id = create_event_tracker_response['trackingId']\n", "print(f'tracking_id: {tracking_id}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### レコメンデーションのリストを取得\n", "* 比較のため、まずは Event Tracker を使わずにレコメンデーションする" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "get_recommendations_response = personalize_runtime.get_recommendations(\n", " campaignArn = user_personalization_campaign_arn,\n", " userId = '10'\n", ")\n", "\n", "item_list = get_recommendations_response['itemList']\n", "title_list = [item_data.loc[item_data['ITEM_ID'] == np.int(item['itemId'])].values[0][1] for item in item_list]\n", "\n", "print(\"Recommendations: {}\".format(json.dumps(title_list, indent=2)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### リアルタイムイベントを Feed\n", "* ユーザのリアルタイムイベントは [put_event](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/personalize-events.html#PersonalizeEvents.Client.put_events) メソッドで Feed することができる\n", "* 今回は user_id が 10 の人が item_id が 215 の映画について 5 というレートをつけたイベントを feed する" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# USER_ID が 10 のユーザ\n", "user_data[user_data['USER_ID']==10]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ITEM_ID が 215 の映画\n", "item_data[item_data['ITEM_ID']==215]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import uuid\n", "from datetime import datetime\n", "\n", "now = datetime.now()\n", "\n", "sentAt = str(now.timestamp())\n", "int(now.timestamp())\n", "\n", "session_id = str(uuid.uuid4())\n", "event_id = str(uuid.uuid4())" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "tracking_id = create_event_tracker_response['trackingId']\n", "print(tracking_id)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "personalize_events.put_events(\n", " trackingId = tracking_id,\n", " userId = '10',\n", " sessionId = session_id,\n", " eventList = [\n", " {\n", " \"eventId\": event_id,\n", " \"sentAt\": int(now.timestamp()),\n", " \"eventType\": \"RATING\",\n", " \"properties\": json.dumps(\n", " {\n", " 'itemId': '215',\n", " 'eventValue': 5\n", " })\n", " }\n", " ]\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### レコメンデーションのリストを再取得\n", "* put_event すると結果が一部変わる（ことがある）" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "get_recommendations_response = personalize_runtime.get_recommendations(\n", " campaignArn = user_personalization_campaign_arn,\n", " userId = '10'\n", ")\n", "\n", "item_list = get_recommendations_response['itemList']\n", "title_list = [item_data.loc[item_data['ITEM_ID'] == np.int(item['itemId'])].values[0][1] for item in item_list]\n", "\n", "print(\"Recommendations: {}\".format(json.dumps(title_list, indent=2)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 推論フィルターの利用\n", "#### 推論フィルターの作成\n", "ここではジャンルでの絞り込みができる推論フィルターを作成\n", "* [create_filter](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/personalize.html#Personalize.Client.create_filter) メソッドを使う\n", " * filterExpression 引数にクエリを記載する\n", " * クエリの書き方は [URL](https://docs.aws.amazon.com/personalize/latest/dg/filter-expressions.html) を参照\n", "* フィルタ作成も非同期メソッドのため、確認は [describe_filter](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/personalize.html#Personalize.Client.describe_filter) を利用する\n", " * 作成に 1 ~ 2 分かかる\n", "* filter の expression に変数を用いることが可能になった\n", " * フィルタを作成する際に ($GENRE) のように推論時にいろいろな値を入れられるようにする変数を事前に宣言し、推論する際に filterValues 引数に変数に格納する値を指定する" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "filter_expression = 'INCLUDE ItemID WHERE Items.GENRE IN ($GENRE)'\n", "\n", "create_filter_response = personalize.create_filter(\n", " datasetGroupArn = dataset_group_arn,\n", " filterExpression = filter_expression,\n", " name = 'genre_filter_action'\n", ")\n", "filter_arn = create_filter_response['filterArn']\n", "print(json.dumps(create_filter_response, indent=2))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%time\n", "\n", "status = None\n", "max_time = time.time() + 3*60*60 # 3 hours\n", "while time.time() < max_time:\n", " describe_filter_response = personalize.describe_filter(\n", " filterArn = filter_arn\n", " )\n", " status = describe_filter_response[\"filter\"][\"status\"]\n", " print('.',end='')\n", " \n", " if status == \"ACTIVE\" or status == \"CREATE FAILED\":\n", " print('!')\n", " print(f\"Filter: {status}\")\n", " break\n", " \n", " time.sleep(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 推論フィルターの有無で結果を比較\n", "* [get_recommendations](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/personalize-runtime.html#PersonalizeRuntime.Client.get_recommendations) メソッドを呼び出す際に、filterArn 引数に作成したフィルタの arn を指定する" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "# 推論フィルタ無し\n", "get_recommendations_response = personalize_runtime.get_recommendations(\n", " campaignArn = user_personalization_campaign_arn,\n", " userId = '10'\n", ")\n", "\n", "item_list = get_recommendations_response['itemList']\n", "title_list = [item_data.loc[item_data['ITEM_ID'] == np.int(item['itemId'])].values[0][1] for item in item_list]\n", "\n", "print(\"Recommendations: {}\".format(json.dumps(title_list, indent=2)))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "item_data[item_data['TITLE'].isin(title_list)]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "# 推論フィルタあり ( Action のみ)\n", "get_recommendations_response = personalize_runtime.get_recommendations(\n", " campaignArn = user_personalization_campaign_arn,\n", " userId = '10',\n", " filterArn = filter_arn,\n", " filterValues={ \"GENRE\": \"\\\"Action\\\"\"} # Action のみに絞る\n", ")\n", "\n", "item_list = get_recommendations_response['itemList']\n", "title_list = [item_data.loc[item_data['ITEM_ID'] == np.int(item['itemId'])].values[0][1] for item in item_list]\n", "\n", "print(\"Recommendations: {}\".format(json.dumps(title_list, indent=2)))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "item_data[item_data['TITLE'].isin(title_list)]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## ここから先はハンズオン完了後余裕のある方向け\n", "### バッチ推論\n", "* アプリ通知やメール配信向けにバッチ処理でレコメンドする機能がある\n", "* 予め S3 に推論対象を保存しておき [create_batch_inference_job](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/personalize.html#Personalize.Client.create_batch_inference_job) でソリューションバージョンと推論対象を設定して実行する\n", "* 本ハンズオンでは User-Personalization とバッチ推論のみの Item-Affinity, Item-Attribute-Affinity を対象とする" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 推論用入力データの作成" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "user_list = []\n", "user_list.append({\"userId\": \"3\"})\n", "user_list.append({\"userId\": \"10\"})\n", "user_list.append({\"userId\": \"15\"})\n", "print(user_list)\n", "\n", "with jsonlines.open('batch_infrence_input.json', mode='w') as writer:\n", " writer.write_all(user_list)\n", "\n", "!head batch_infrence_input.json\n", "boto3.Session().resource('s3').Bucket(bucket_name).Object(s3_key_prefix + 'batch-inference-input/batch_infrence_input.json').upload_file('batch_infrence_input.json')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### バッチ推論ジョブの実行" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "now = datetime.now()\n", "time_str = now.strftime(\"-%Y-%m-%d-%H-%M-%S\")\n", "create_batch_inference_job_response = personalize.create_batch_inference_job (\n", " solutionVersionArn = user_personalization_solution_version_arn,\n", " jobName = \"userpersonalization-batch-inference-job\" + time_str,\n", " roleArn = role_arn,\n", " jobInput = \n", " {\"s3DataSource\": {\"path\": \"s3://\" + bucket_name + \"/\" + s3_key_prefix + \"batch-inference-input/batch_infrence_input.json\"}},\n", " jobOutput = \n", " {\"s3DataDestination\": {\"path\": \"s3://\" + bucket_name + \"/\" + s3_key_prefix + \"batch-inference-output/\"}},\n", " numResults = 100\n", ")\n", "batchInferenceJobArn = create_batch_inference_job_response['batchInferenceJobArn']\n", "describe_batch_inference_job_response = personalize.describe_batch_inference_job(\n", " batchInferenceJobArn = batchInferenceJobArn\n", " )\n", "\n", "describe_batch_inference_job_response" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%time\n", "\n", "# 待つ\n", "status = None\n", "max_time = time.time() + 3*60*60 # 3 hours\n", "while time.time() < max_time:\n", " describe_batch_inference_job_response = personalize.describe_batch_inference_job(\n", " batchInferenceJobArn = batchInferenceJobArn\n", " )\n", " status = describe_batch_inference_job_response[\"batchInferenceJob\"][\"status\"]\n", " \n", " if status == \"ACTIVE\" or status == \"CREATE FAILED\":\n", " print('!')\n", " print(f\"status: {status}\")\n", " break\n", " else:\n", " print('.',end='')\n", " time.sleep(60)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### バッチ推論結果の取得・表示" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "output_path = describe_batch_inference_job_response['batchInferenceJob']['jobOutput']['s3DataDestination']['path'] + 'batch_infrence_input.json.out'\n", "!aws s3 cp {output_path} .\n", "!head batch_infrence_input.json.out" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### affniate バッチセグメントジョブの実行" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "item_list = []\n", "for i in range(1,101):\n", " item_list.append({\"itemId\": str(i)})\n", "\n", "with jsonlines.open('affinity_input.json', mode='w') as writer:\n", " writer.write_all(item_list)\n", "\n", "!head affinity_input.json\n", "boto3.Session().resource('s3').Bucket(bucket_name).Object(s3_key_prefix + 'affinity-input/affinity_input.json').upload_file('affinity_input.json')\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "now = datetime.now()\n", "time_str = now.strftime(\"-%Y-%m-%d-%H-%M-%S\")\n", "create_batch_segment_job_response = personalize.create_batch_segment_job(\n", " jobName = \"affinity-batch-segment-job\" + time_str,\n", " solutionVersionArn = affinity_solution_version_arn,\n", " numResults=100,\n", " jobInput={'s3DataSource': {\"path\": \"s3://\" + bucket_name + \"/\" + s3_key_prefix + \"affinity-input/affinity_input.json\"}},\n", " jobOutput = {\"s3DataDestination\": {\"path\": \"s3://\" + bucket_name + \"/\" + s3_key_prefix + \"affinity-output/\"}},\n", " roleArn=role_arn\n", ")\n", "batchSegmentJobArn = create_batch_segment_job_response['batchSegmentJobArn']\n", "describe_batch_segment_job_response = personalize.describe_batch_segment_job(\n", " batchSegmentJobArn = batchSegmentJobArn\n", " )\n", "\n", "describe_batch_segment_job_response" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%time\n", "\n", "# 待つ\n", "status = None\n", "max_time = time.time() + 3*60*60 # 3 hours\n", "while time.time() < max_time:\n", " describe_batch_segment_job_response = personalize.describe_batch_segment_job(\n", " batchSegmentJobArn = batchSegmentJobArn\n", " )\n", " status = describe_batch_segment_job_response[\"batchSegmentJob\"][\"status\"]\n", " \n", " if status == \"ACTIVE\" or status == \"CREATE FAILED\":\n", " print('!')\n", " print(f\"status: {status}\")\n", " break\n", " else:\n", " print('.',end='')\n", " time.sleep(60)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "output_path = describe_batch_segment_job_response['batchSegmentJob']['jobOutput']['s3DataDestination']['path'] + 'affinity_input.json.out'\n", "!aws s3 cp {output_path} .\n", "!head affinity_input.json.out" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### attribute affniate バッチセグメントジョブの実行\n", "* input の形式は [Developer Guide](https://docs.aws.amazon.com/personalize/latest/dg/batch-data-upload.html)に例の記載あり。Schema 情報に合わせる。\n", "* 条件式も記載できる(AND/ORなど)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "genre_list_work = []\n", "for genres in pd.read_csv('./data/item.csv')['GENRE'].unique().tolist():\n", " temp = genres.split('|')\n", " for t in temp:\n", " genre_list_work.append(t)\n", "genre_list_work = list(set(genre_list_work))[:10]\n", "print(genre_list_work)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "genre_list = []\n", "for genre in genre_list_work:\n", " genre_list.append({\"itemAttributes\": \"Items.GENRE = \\\"\"+genre + \"\\\"\"})\n", " \n", "\n", "with jsonlines.open('attribute_affinity_input.json', mode='w') as writer:\n", " writer.write_all(genre_list)\n", "\n", "!head attribute_affinity_input.json\n", "boto3.Session().resource('s3').Bucket(bucket_name).Object(s3_key_prefix + 'attribute-affinity-input/attribute_affinity_input.json').upload_file('attribute_affinity_input.json')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "now = datetime.now()\n", "time_str = now.strftime(\"-%Y-%m-%d-%H-%M-%S\")\n", "create_batch_segment_job_response = personalize.create_batch_segment_job(\n", " jobName = \"attribute-affinity-batch-segment-job\" + time_str,\n", " solutionVersionArn = attribute_affinity_solution_version_arn,\n", " numResults=10,\n", " jobInput={'s3DataSource': {\"path\": \"s3://\" + bucket_name + \"/\" + s3_key_prefix + \"attribute-affinity-input/attribute_affinity_input.json\"}},\n", " jobOutput = {\"s3DataDestination\": {\"path\": \"s3://\" + bucket_name + \"/\" + s3_key_prefix + \"attribute-affinity-output/\"}},\n", " roleArn=role_arn\n", ")\n", "batchSegmentJobArn = create_batch_segment_job_response['batchSegmentJobArn']\n", "describe_batch_segment_job_response = personalize.describe_batch_segment_job(\n", " batchSegmentJobArn = batchSegmentJobArn\n", " )\n", "\n", "describe_batch_segment_job_response" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%time\n", "\n", "# 待つ\n", "status = None\n", "max_time = time.time() + 3*60*60 # 3 hours\n", "while time.time() < max_time:\n", " describe_batch_segment_job_response = personalize.describe_batch_segment_job(\n", " batchSegmentJobArn = batchSegmentJobArn\n", " )\n", " status = describe_batch_segment_job_response[\"batchSegmentJob\"][\"status\"]\n", " \n", " if status == \"ACTIVE\" or status == \"CREATE FAILED\":\n", " print('!')\n", " print(f\"status: {status}\")\n", " break\n", " else:\n", " print('.',end='')\n", " time.sleep(60)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "output_path = describe_batch_segment_job_response['batchSegmentJob']['jobOutput']['s3DataDestination']['path'] + 'attribute_affinity_input.json.out'\n", "!aws s3 cp {output_path} .\n", "!head attribute_affinity_input.json.out" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# personalize.describe_schema(schemaArn=item_schema_arn)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## お片付け\n", "* 作ってきた手順とは逆の順番で削除していく \n", " いきなりデータセットグループを削除、などはできない" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "personalize.delete_filter(filterArn = filter_arn)\n", "while True:\n", " try:\n", " status = personalize.describe_filter(filterArn=filter_arn)['filter']['status']\n", " print('.',end='')\n", " time.sleep(1)\n", " except:\n", " print('!')\n", " print('deleted')\n", " break" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "personalize.delete_event_tracker(eventTrackerArn=event_tracker_arn)\n", "while True:\n", " try:\n", " status = personalize.describe_event_tracker(eventTrackerArn=event_tracker_arn)['eventTracker']['status']\n", " print('.',end='')\n", " time.sleep(1)\n", " except:\n", " print('!')\n", " print('deleted')\n", " break" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "personalize.delete_campaign(campaignArn = user_personalization_campaign_arn)\n", "personalize.delete_campaign(campaignArn = personalized_ranking_campaign_arn)\n", "personalize.delete_campaign(campaignArn = sims_campaign_arn)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "while True:\n", " try:\n", " status = personalize.describe_campaign(campaignArn = user_personalization_campaign_arn)['campaign']['status']\n", " print('.',end='')\n", " time.sleep(1)\n", " except:\n", " print('!')\n", " print('deleted')\n", " break\n", "while True:\n", " try:\n", " status = personalize.describe_campaign(campaignArn = personalized_ranking_campaign_arn)['campaign']['status']\n", " print('.',end='')\n", " time.sleep(1)\n", " except:\n", " print('!')\n", " print('deleted')\n", " break\n", "while True:\n", " try:\n", " status = personalize.describe_campaign(campaignArn = sims_campaign_arn)['campaign']['status']\n", " print('.',end='')\n", " time.sleep(1)\n", " except:\n", " print('!')\n", " print('deleted')\n", " break" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "personalize.delete_solution(solutionArn=user_personalization_solution_arn)\n", "personalize.delete_solution(solutionArn=personalized_ranking_solution_arn)\n", "personalize.delete_solution(solutionArn=sims_solution_arn)\n", "personalize.delete_solution(solutionArn=affinity_solution_arn)\n", "personalize.delete_solution(solutionArn=attribute_affinity_solution_arn)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "while True:\n", " try:\n", " status = personalize.describe_solution(solutionArn=user_personalization_solution_arn)['solution']['status']\n", " print('.',end='')\n", " time.sleep(1)\n", " except:\n", " print('!')\n", " print('deleted')\n", " break\n", "while True:\n", " try:\n", " status = personalize.describe_solution(solutionArn=personalized_ranking_solution_arn)['solution']['status']\n", " print('.',end='')\n", " time.sleep(1)\n", " except:\n", " print('!')\n", " print('deleted')\n", " break\n", "while True:\n", " try:\n", " status = personalize.describe_solution(solutionArn=sims_solution_arn)['solution']['status']\n", " print('.',end='')\n", " time.sleep(1)\n", " except:\n", " print('!')\n", " print('deleted')\n", " break \n", "while True:\n", " try:\n", " status = personalize.describe_solution(solutionArn=affinity_solution_arn)['solution']['status']\n", " print('.',end='')\n", " time.sleep(1)\n", " except:\n", " print('!')\n", " print('deleted')\n", " break \n", "while True:\n", " try:\n", " status = personalize.describe_solution(solutionArn=attribute_affinity_solution_arn)['solution']['status']\n", " print('.',end='')\n", " time.sleep(1)\n", " except:\n", " print('!')\n", " print('deleted')\n", " break " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "personalize.delete_dataset(datasetArn=interaction_dataset_arn)\n", "personalize.delete_dataset(datasetArn=user_dataset_arn)\n", "personalize.delete_dataset(datasetArn=item_dataset_arn)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "while True:\n", " try:\n", " status = personalize.describe_dataset(datasetArn=interaction_dataset_arn)['dataset']['status']\n", " print('.',end='')\n", " time.sleep(1)\n", " except:\n", " print('!')\n", " print('deleted')\n", " break\n", "while True:\n", " try:\n", " status = personalize.describe_dataset(datasetArn=user_dataset_arn)['dataset']['status']\n", " print('.',end='')\n", " time.sleep(1)\n", " except:\n", " print('!')\n", " print('deleted')\n", " break\n", "while True:\n", " try:\n", " status = personalize.describe_dataset(datasetArn=item_dataset_arn)['dataset']['status']\n", " print('.',end='')\n", " time.sleep(1)\n", " except:\n", " print('!')\n", " print('deleted')\n", " break" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "personalize.delete_dataset_group(datasetGroupArn=dataset_group_arn)\n", "while True:\n", " try:\n", " status = personalize.describe_dataset_group(datasetGroupArn=dataset_group_arn)['datasetGroup']['status']\n", " print('.',end='')\n", " time.sleep(1)\n", " except:\n", " print('!')\n", " print('deleted')\n", " break" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "personalize.delete_schema(schemaArn=interaction_schema_arn)\n", "personalize.delete_schema(schemaArn=user_schema_arn)\n", "personalize.delete_schema(schemaArn=item_schema_arn)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "iam.detach_role_policy(\n", " RoleName=role_name,\n", " PolicyArn='arn:aws:iam::aws:policy/service-role/AmazonPersonalizeFullAccess'\n", ")\n", "iam.delete_role(\n", " RoleName = role_name\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!aws s3 rb s3://{bucket_name} --force" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "conda_python3", "language": "python", "name": "conda_python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.13" } }, "nbformat": 4, "nbformat_minor": 4 }