{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# モデル品質モニタリングのステップA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "このノートブックを実行する時のヒント：   \n",
    "- KernelはPython3（Data Science）で動作確認をしています。\n",
    "- デフォルトではSageMakerのデフォルトBucketを利用します。必要に応じて変更することも可能です。\n",
    "- 実際に動かさなくても出力を確認できるようにセルのアウトプットを残しています。きれいな状態から実行したい場合は、右クリックメニューから \"Clear All Outputs\"を選択して出力をクリアしてから始めてください。\n",
    "- 作成されたスケジュールはSageMaker Studioの`SageMaker resource` （左側ペインの一番下）のEndpointメニューからも確認可能"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color: orange; font-size: 120%; \">モデル名は実行前に設定変更が必要です</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# step-0-train-model.ipynb でトレーニングしたモデルの名前に変更してください\n",
    "model_name = 'nyctaxi-xgboost-regression-2022-12-15-10-58-19-model'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "複数のノートブックで共通で使用する変数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# エンドポイント名を指定する\n",
    "endpoint_name = 'nyctaxi-xgboost-endpoint'\n",
    "\n",
    "# エンドポイントConfigの名前を指定する\n",
    "endpoint_config_name = f'{endpoint_name}-config'\n",
    "\n",
    "# データ品質のモニタリングスケジュールの名前を指定する\n",
    "model_quality_monitoring_schedule = f'{endpoint_name}-model-quality-schedule'\n",
    "\n",
    "# SageMaker default bucketをModel Monitorのバケットとして使用\n",
    "# それ以外のバケットを使用している場合はここで指定する\n",
    "import sagemaker\n",
    "bucket = sagemaker.Session().default_bucket()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "モニタリング結果を保管するための、ベースラインやレポートのS3上のPrefixを設定します"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ベースラインの出力先Prefixを設定する\n",
    "baseline_prefix = 'model_monitor/model_quality_baseline'\n",
    "\n",
    "# 時系列での可視化のために、複数のレポートに共通するPrefixを設定する\n",
    "report_prefix = 'model_monitor/model_quality_monitoring_report'\n",
    "\n",
    "# Ground Truthをアップロードする先のPrefixを指定します\n",
    "ground_truth_prefix = 'model_monitor/model_quality_ground_truth'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A1. 推論エンドポイントにデータキャプチャの設定を行う"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "データキャプチャの設定を入れた新しい推論エンドポイントを作成する場合のコードサンプルです。  \n",
    "<span style=\"color: orange; \">データ品質ですでにOUTPUT設定付きのエンドポイントを作成している場合はこのセルの実行は不要なので、A2から進めてください</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "import time\n",
    "import sagemaker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sm_client = boto3.client('sagemaker')\n",
    "\n",
    "# Create endpoint config\n",
    "create_endpoint_config_response = sm_client.create_endpoint_config(\n",
    "    EndpointConfigName=endpoint_config_name,\n",
    "    ProductionVariants=[{\n",
    "        'InstanceType': 'ml.t2.medium',\n",
    "        'InitialVariantWeight': 1,\n",
    "        'InitialInstanceCount': 1,\n",
    "        'ModelName': model_name,\n",
    "        'VariantName': 'AllTraffic'}],\n",
    "    # Set data capture config\n",
    "    DataCaptureConfig={\n",
    "        'EnableCapture': True,\n",
    "        'InitialSamplingPercentage': 100,\n",
    "        'DestinationS3Uri': f's3://{bucket}/model_monitor/endpoint-data-capture',\n",
    "        'CaptureOptions': [{'CaptureMode': 'Input'}, {'CaptureMode': 'Output'}],\n",
    "        'CaptureContentTypeHeader': {\n",
    "            'CsvContentTypes': ['text/csv'],\n",
    "            'JsonContentTypes': ['application/json']\n",
    "        }\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def wait_for_endpoint_creation(endpoint_name):\n",
    "    sm_client = boto3.client('sagemaker')\n",
    "    \n",
    "    # Check endpoint creation status\n",
    "    resp = sm_client.describe_endpoint(EndpointName=endpoint_name)\n",
    "    status = resp['EndpointStatus']\n",
    "    while status=='Creating':\n",
    "        print(\"Status: \" + status)\n",
    "        time.sleep(60)\n",
    "        resp = sm_client.describe_endpoint(EndpointName=endpoint_name)\n",
    "        status = resp['EndpointStatus']\n",
    "        \n",
    "    print('Finished!', status)\n",
    "\n",
    "    \n",
    "# Create endpoint\n",
    "create_endpoint_response = sm_client.create_endpoint(\n",
    "    EndpointName=endpoint_name,\n",
    "    EndpointConfigName=endpoint_config_name\n",
    ")\n",
    "wait_for_endpoint_creation(endpoint_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color: orange; \">エンドポイントのデプロイに10分程度の時間がかかります</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A2. ベースラインを作成する\n",
    "<span style=\"color: orange; \">ベースラインの計算には5分程度の時間がかかります</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "import time\n",
    "import sagemaker\n",
    "from datetime import datetime\n",
    "\n",
    "import pandas as pd\n",
    "from sagemaker import get_execution_role, session, Session\n",
    "from sagemaker.model_monitor import ModelQualityMonitor\n",
    "from sagemaker.model_monitor.dataset_format import DatasetFormat\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "job_timestamp = datetime.now().strftime('%Y-%m-%d-%H-%M-%S')\n",
    "baseline_job_name = \"model-quality-baseline-job-{}\".format(job_timestamp)\n",
    "\n",
    "role = get_execution_role()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Job Name:  model-quality-baseline-job-2022-12-16-10-27-07\n",
      "Inputs:  [{'InputName': 'baseline_dataset_input', 'AppManaged': False, 'S3Input': {'S3Uri': 's3://sagemaker-ap-northeast-1-370828233696/model_monitor/model_quality_baseline_input/', 'LocalPath': '/opt/ml/processing/input/baseline_dataset_input', 'S3DataType': 'S3Prefix', 'S3InputMode': 'File', 'S3DataDistributionType': 'FullyReplicated', 'S3CompressionType': 'None'}}]\n",
      "Outputs:  [{'OutputName': 'monitoring_output', 'AppManaged': False, 'S3Output': {'S3Uri': 's3://sagemaker-ap-northeast-1-370828233696/model_monitor/model_quality_baseline/', 'LocalPath': '/opt/ml/processing/output', 'S3UploadMode': 'EndOfJob'}}]\n",
      "........................................................................!CPU times: user 366 ms, sys: 21.7 ms, total: 388 ms\n",
      "Wall time: 6min 2s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "model_quality_monitor = ModelQualityMonitor(\n",
    "    role=role,\n",
    "    instance_count=1,\n",
    "    instance_type='ml.m5.xlarge',\n",
    "    volume_size_in_gb=20,\n",
    "    max_runtime_in_seconds=1800,\n",
    ")\n",
    "\n",
    "job = model_quality_monitor.suggest_baseline(\n",
    "    job_name=baseline_job_name,\n",
    "    baseline_dataset=f's3://{bucket}/model_monitor/model_quality_baseline_input/',\n",
    "    dataset_format=DatasetFormat.csv(header=True),\n",
    "    output_s3_uri = f's3://{bucket}/{baseline_prefix}/',\n",
    "    problem_type='Regression',\n",
    "    inference_attribute= \"pred\", # データセットに含まれる（テスト時の）推論結果を保持したカラム名\n",
    "    # probability_attribute= \"probability\", # データセットに含まれる（テスト時の）確信度を保持したカラム名（Regressionでは不要）\n",
    "    ground_truth_attribute= \"pickup_count\" # データセットに含まれる（テスト時の）ラベルを保持したカラム名\n",
    ")\n",
    "job.wait(logs=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 作成されたベースラインを確認する"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'version': 0.0,\n",
       " 'dataset': {'item_count': 2688,\n",
       "  'evaluation_time': '2022-12-16T10:32:50.057Z'},\n",
       " 'regression_metrics': {'mae': {'value': 38.29501488095238,\n",
       "   'standard_deviation': 0.21881824869307076},\n",
       "  'mse': {'value': 2809.991443452381, 'standard_deviation': 21.11839953830601},\n",
       "  'rmse': {'value': 53.00935241495015,\n",
       "   'standard_deviation': 0.19907891114454224},\n",
       "  'r2': {'value': 0.9499197226669486,\n",
       "   'standard_deviation': 0.0006458841438745311}}}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_quality_monitor.latest_baselining_job.baseline_statistics().body_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'version': 0.0,\n",
       " 'regression_constraints': {'mae': {'threshold': 38.29501488095238,\n",
       "   'comparison_operator': 'GreaterThanThreshold'},\n",
       "  'mse': {'threshold': 2809.991443452381,\n",
       "   'comparison_operator': 'GreaterThanThreshold'},\n",
       "  'rmse': {'threshold': 53.00935241495015,\n",
       "   'comparison_operator': 'GreaterThanThreshold'},\n",
       "  'r2': {'threshold': 0.9499197226669486,\n",
       "   'comparison_operator': 'LessThanThreshold'}}}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_quality_monitor.latest_baselining_job.suggested_constraints().body_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A3. スケジュールの作成"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "from sagemaker import model_monitor\n",
    "from sagemaker.model_monitor import Constraints, EndpointInput\n",
    "\n",
    "bucket = sagemaker.Session().default_bucket()\n",
    "model_constraints = Constraints.from_s3_uri(f's3://{bucket}/{baseline_prefix}/constraints.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_quality_monitor = ModelQualityMonitor(\n",
    "    role=sagemaker.get_execution_role(),\n",
    "    instance_count=1,\n",
    "    instance_type='ml.m5.xlarge',\n",
    "    volume_size_in_gb=20,\n",
    "    max_runtime_in_seconds=1800,\n",
    ")\n",
    "\n",
    "# ここでは、推論からGround Truthの収集までに1時間から3時間かかることを想定したオフセットをセットする\n",
    "# テスト目的ですぐにGround Truthをセットしてアップロードする場合は、end_time_offest=\"-PT0H\"を指定する\n",
    "endpont_input = EndpointInput(\n",
    "        endpoint_name=endpoint_name,\n",
    "        destination=\"/opt/ml/processing/input/endpoint\",\n",
    "        start_time_offset=\"-PT3H\",\n",
    "        end_time_offset=\"-PT0H\",\n",
    "        inference_attribute=\"0\",\n",
    ")\n",
    "\n",
    "model_quality_monitor.create_monitoring_schedule(\n",
    "    monitor_schedule_name=model_quality_monitoring_schedule,\n",
    "    output_s3_uri=f's3://{bucket}/{report_prefix}',\n",
    "    constraints=model_constraints,\n",
    "    schedule_cron_expression=model_monitor.CronExpressionGenerator.hourly(),\n",
    "    enable_cloudwatch_metrics=True,\n",
    "    endpoint_input=endpont_input,\n",
    "    ground_truth_input=f's3://{bucket}/{ground_truth_prefix}',\n",
    "    problem_type='Regression',\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "作成されたスケジュールを確認"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'MonitoringScheduleArn': 'arn:aws:sagemaker:ap-northeast-1:370828233696:monitoring-schedule/nyctaxi-xgboost-endpoint-model-quality-schedule',\n",
       " 'MonitoringScheduleName': 'nyctaxi-xgboost-endpoint-model-quality-schedule',\n",
       " 'MonitoringScheduleStatus': 'Pending',\n",
       " 'MonitoringType': 'ModelQuality',\n",
       " 'CreationTime': datetime.datetime(2022, 12, 16, 10, 33, 11, 742000, tzinfo=tzlocal()),\n",
       " 'LastModifiedTime': datetime.datetime(2022, 12, 16, 10, 33, 11, 849000, tzinfo=tzlocal()),\n",
       " 'MonitoringScheduleConfig': {'ScheduleConfig': {'ScheduleExpression': 'cron(0 * ? * * *)'},\n",
       "  'MonitoringJobDefinitionName': 'model-quality-job-definition-2022-12-16-10-33-11-430',\n",
       "  'MonitoringType': 'ModelQuality'},\n",
       " 'EndpointName': 'nyctaxi-xgboost-endpoint',\n",
       " 'LastMonitoringExecutionSummary': {'MonitoringScheduleName': 'nyctaxi-xgboost-endpoint-model-quality-schedule',\n",
       "  'ScheduledTime': datetime.datetime(2022, 12, 16, 9, 0, tzinfo=tzlocal()),\n",
       "  'CreationTime': datetime.datetime(2022, 12, 16, 9, 8, 25, 258000, tzinfo=tzlocal()),\n",
       "  'LastModifiedTime': datetime.datetime(2022, 12, 16, 9, 8, 27, 131000, tzinfo=tzlocal()),\n",
       "  'MonitoringExecutionStatus': 'Failed',\n",
       "  'EndpointName': 'nyctaxi-xgboost-endpoint',\n",
       "  'FailureReason': 'No S3 objects found under S3 URL \"s3://sagemaker-ap-northeast-1-370828233696/model_monitor/model_quality_ground_truth/2022/12/16/08\" given in input data source. Please ensure that the bucket exists in the selected region (ap-northeast-1), that objects exist under that S3 prefix, and that the role \"arn:aws:iam::370828233696:role/user1_ds_dev_exec_role\" has \"s3:ListBucket\" permissions on bucket \"sagemaker-ap-northeast-1-370828233696\".'},\n",
       " 'ResponseMetadata': {'RequestId': '9180fb13-e8d2-443f-b22e-5d61046210a0',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': '9180fb13-e8d2-443f-b22e-5d61046210a0',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '1349',\n",
       "   'date': 'Fri, 16 Dec 2022 10:33:11 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sm_client = boto3.client('sagemaker')\n",
    "sm_client.describe_monitoring_schedule(MonitoringScheduleName=model_quality_monitoring_schedule)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:ap-northeast-1:102112518831:image/datascience-1.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}