{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Amazon SageMaker モニタリング Part3(モデルドリフトとFeature Attribution)\n", "\n", "このノートブックを実行する時のヒント：\n", "\n", "- KernelはPython3（Data Science）で動作確認をしています。\n", "- デフォルトではSageMakerのデフォルトBucketを利用します。必要に応じて変更することも可能です。\n", "- 実際に動かさなくても出力を確認できるようにセルのアウトプットを残しています。きれいな状態から実行したい場合は、右クリックメニューから \"Clear All Outputs\"を選択して出力をクリアしてから始めてください。\n", "- 作成されたスケジュールはSageMaker StudioのSageMaker resource （左側ペインの一番下）のEndpointメニューからも確認可能です。\n", "\n", "## 前提\n", "### 使用するデータの背景\n", "米国国勢調査局の 人口動態調査（Current Population Survey: CPS）- 社会経済年次補助（Annual Social and Economic Supplement: ASEC）を利用します。 CPS は、一般の労働者を含むグループの収入やその他の特性に関する月次の調査です。 また、年次社会経済補足（CPS-ASEC） は、毎年2月、3月、4月に実施され、15歳以上の人の労働経験、収入、非現金給付、移住に関する情報を提供しています。\n", "\n", "https://health.gov/healthypeople/objectives-and-data/data-sources-and-methods/data-sources/current-population-survey-annual-social-and-economic-supplement-cps-asec\n", "\n", "### データ項目\n", "- A_AGE: 年齢(数値)\n", "- A_FTLF: 現在、フルタイムか(カテゴリ)\n", "- A_HGA: 学歴(カテゴリ)\n", "- A_HSCOL: 現在、高校/大学で就学しているか(カテゴリ)\n", "- A_MARITL: 婚姻状況(カテゴリ)\n", "- A_SEX: 性別(カテゴリ)\n", "- A_UNMEM: 労働組合員か(カテゴリ)\n", "- A_USLHRS: 労働時間(数値)\n", "- NOEMP: 雇用主の企業総従業員数(カテゴリ)\n", "- PENATVTY: 出身国(カテゴリ)\n", "- PRCITSHP: 市民権の記録(カテゴリ)\n", "- SEOTR: 自営業か(カテゴリ)\n", "- WKSWORK: 何週間働いたか(数値)\n", "- PTOTVAL: 総収入額(数値)、推論対象\n", "\n", "### 運用時の課題\n", "50000ドル以上の収入が得られる状況は年々変化することが想定される。また、学習時に重視していた特徴量が重要でなくなるなどの環境の変化も想定される。これらをモニタリングして、学習データの見直し、学習アルゴリズムの見直し、再学習を自動で行えるようにしたい。\n", "\n", "### データ加工\n", "1. 18歳以上のデータに限定\n", "2. PTOTVALが50000ドル以上であれば1、そうでなければ0(変数名はPTOTVAL_Over50000)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## ステップ0: モニタリング前の準備\n", "### 環境準備" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import boto3\n", "import copy\n", "import json\n", "import os\n", "import random\n", "import requests\n", "import time\n", "import numpy as np\n", "import pandas as pd\n", "\n", "from datetime import datetime, timedelta\n", "\n", "from sagemaker import get_execution_role, image_uris, Session\n", "from sagemaker.clarify import (\n", " BiasConfig,\n", " DataConfig,\n", " ModelConfig,\n", " ModelPredictedLabelConfig,\n", " SHAPConfig,\n", ")\n", "from sagemaker.estimator import Estimator\n", "from sagemaker.image_uris import retrieve\n", "from sagemaker.inputs import TrainingInput\n", "from sagemaker.model import Model\n", "from sagemaker.model_monitor import (\n", " BiasAnalysisConfig,\n", " CronExpressionGenerator,\n", " DataCaptureConfig,\n", " EndpointInput,\n", " ExplainabilityAnalysisConfig,\n", " ModelBiasMonitor,\n", " ModelExplainabilityMonitor,\n", ")\n", "from sagemaker.s3 import S3Downloader, S3Uploader\n", "from sagemaker.utils import name_from_base" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "RoleArn: arn:aws:iam::036661559124:role/service-role/AmazonSageMaker-ExecutionRole-20220417T231709\n", "AWS region: ap-northeast-1\n", "Bucket: sagemaker-ap-northeast-1-036661559124\n", "S3 key: s3://sagemaker-ap-northeast-1-036661559124/sagemaker/blackbelt-part3-sample\n", "Capture path: s3://sagemaker-ap-northeast-1-036661559124/sagemaker/blackbelt-part3-sample/datacapture\n", "Ground truth path: s3://sagemaker-ap-northeast-1-036661559124/sagemaker/blackbelt-part3-sample/ground_truth_data/2023-01-04-12-11-59\n", "Report path: s3://sagemaker-ap-northeast-1-036661559124/sagemaker/blackbelt-part3-sample/reports\n", "Baseline results uri: s3://sagemaker-ap-northeast-1-036661559124/sagemaker/blackbelt-part3-sample/baselining\n", "Endpoint name: bb-example-xgboost-classification-model-2023-01-04-12-11-59-829\n", "Training dataset s3 path: s3://sagemaker-ap-northeast-1-036661559124/sagemaker/blackbelt-part3-sample/sample_data\n" ] } ], "source": [ "role = get_execution_role()\n", "print(f\"RoleArn: {role}\")\n", "\n", "sagemaker_session = Session()\n", "sagemaker_client = sagemaker_session.sagemaker_client\n", "sagemaker_runtime_client = sagemaker_session.sagemaker_runtime_client\n", "\n", "region = sagemaker_session.boto_region_name\n", "print(f\"AWS region: {region}\")\n", "\n", "bucket = Session().default_bucket()\n", "print(f\"Bucket: {bucket}\")\n", "prefix = \"sagemaker/blackbelt-part3-sample\"\n", "s3_key = f\"s3://{bucket}/{prefix}\"\n", "print(f\"S3 key: {s3_key}\")\n", "\n", "s3_capture_upload_path = f\"{s3_key}/datacapture\"\n", "ground_truth_upload_path = f\"{s3_key}/ground_truth_data/{datetime.now():%Y-%m-%d-%H-%M-%S}\"\n", "s3_report_path = f\"{s3_key}/reports\"\n", "\n", "print(f\"Capture path: {s3_capture_upload_path}\")\n", "print(f\"Ground truth path: {ground_truth_upload_path}\")\n", "print(f\"Report path: {s3_report_path}\")\n", "\n", "baseline_prefix = 'baselining/baseline'\n", "baseline_results_uri = f\"{s3_key}/baselining\"\n", "print(f\"Baseline results uri: {baseline_results_uri}\")\n", "\n", "endpoint_name = name_from_base(f\"bb-example-xgboost-classification-model-\")\n", "print(f'Endpoint name: {endpoint_name}')\n", "endpoint_instance_count = 1\n", "endpoint_instance_type = \"ml.m5.large\"\n", "schedule_expression = CronExpressionGenerator.hourly()\n", "\n", "training_data_prefix = 'sample_data'\n", "training_dataset_uri = f\"{s3_key}/{training_data_prefix}\"\n", "print(f'Training dataset s3 path: {training_dataset_uri}')\n", "training_instance_type = \"ml.m5.xlarge\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### データ準備\n", "米国国勢調査局のデータをダウンロードします。" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "def get_csv_file(year):\n", " if year in [2019, 2020, 2021, 2022]:\n", " url = f\"https://api.census.gov/data/{year}/cps/asec/mar?get=A_AGE,A_FTLF,A_HGA,A_HSCOL,A_MARITL,A_SEX,A_UNMEM,A_USLHRS,NOEMP,PENATVTY,PRCITSHP,SEOTR,WKSWORK,PTOTVAL\"\n", " else:\n", " url = \"\"\n", "\n", " try:\n", " print(f\"Downloading {year} file...\")\n", " response = requests.request(\"GET\", url, timeout=(3.0, 3600)).json()\n", " df = pd.DataFrame(response[1:], columns=response[0])\n", " df.to_csv(f'cps_{year}.csv')\n", " except json.JSONDecodeError:\n", " pass\n", " \n", "def get_multi_records(years_list):\n", " for year in years_list:\n", " get_csv_file(year)" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Downloading 2019 file...\n", "Downloading 2020 file...\n", "Downloading 2021 file...\n", "Downloading 2022 file...\n" ] } ], "source": [ "years = [2019, 2020, 2021, 2022]\n", "get_multi_records(years)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### データ加工\n", "18歳以上のデータを抽出し、年収が50000ドル以上かどうかのフラグ変数を作成します。加工後のデータをS3へアップロードします。" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "def data_processing(csv_name):\n", " df = pd.read_csv(csv_name, index_col=0)\n", " df = df[df['A_AGE'] >= 18]\n", " df['PTOTVAL_Over50000'] = np.select([df['PTOTVAL']>50000], [1], default=0)\n", " df = df.drop(['PTOTVAL'], axis=1)\n", " col_list = ['PTOTVAL_Over50000'] + df.columns.tolist()[:-1]\n", " df = df[col_list]\n", " \n", " return df" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "cps_2019_df = data_processing('cps_2019.csv')\n", "cps_2019_df.to_csv(\"train.csv\", header=False, index=False)\n", "cps_2020_df = data_processing('cps_2020.csv')\n", "cps_2020_df.to_csv(\"validation.csv\", header=False, index=False)\n", "\n", "boto3.Session().resource('s3').Bucket(bucket).Object(os.path.join(f'{prefix}/{training_data_prefix}', 'train/data.csv')).upload_file('train.csv')\n", "boto3.Session().resource('s3').Bucket(bucket).Object(os.path.join(f'{prefix}/{training_data_prefix}', 'validation/data.csv')).upload_file('validation.csv')\n", "\n", "train_input = TrainingInput(f'{training_dataset_uri}/train/data.csv', content_type=\"csv\")\n", "validation_input = TrainingInput(f'{training_dataset_uri}/validation/data.csv', content_type=\"csv\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "ベースラインの作成に必要なデータを作成し、S3へアップロードします。またヘッダー情報を取得します。" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "baseline_uri: s3://sagemaker-ap-northeast-1-036661559124/sagemaker/blackbelt-part3-sample/baselining/baseline/baseline_with_header.csv\n" ] } ], "source": [ "baseline_file = 'baseline_with_header.csv'\n", "baseline_dataset = cps_2020_df.sample(n=1000)\n", "baseline_dataset.to_csv(baseline_file, header=True, index=False)\n", "boto3.Session().resource('s3').Bucket(bucket).Object(os.path.join(f'{prefix}/{baseline_prefix}', baseline_file)).upload_file(baseline_file)\n", "baseline_uri = f'{s3_key}/{baseline_prefix}/{baseline_file}'\n", "print(f'baseline_uri: {baseline_uri}')" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "with open('baseline_with_header.csv') as f:\n", " headers_line = f.readline().rstrip()\n", "all_headers = headers_line.split(\",\")\n", "label_header = all_headers[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### モデル準備\n", "ビルトインアルゴリズムのXGBoostを使用してモデルを作成します。" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "collapsed": true, "jupyter": { "outputs_hidden": true }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2023-01-04 12:14:15 Starting - Starting the training job...\n", "2023-01-04 12:14:29 Starting - Preparing the instances for training............\n", "2023-01-04 12:16:40 Downloading - Downloading input data\n", "2023-01-04 12:16:40 Training - Training image download completed. Training in progress..\u001b[34m[2023-01-04 12:16:41.892 ip-10-0-203-120.ap-northeast-1.compute.internal:7 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None\u001b[0m\n", "\u001b[34m[2023-01-04:12:16:41:INFO] Imported framework sagemaker_xgboost_container.training\u001b[0m\n", "\u001b[34m[2023-01-04:12:16:41:INFO] Failed to parse hyperparameter objective value binary:logistic to Json.\u001b[0m\n", "\u001b[34mReturning the value itself\u001b[0m\n", "\u001b[34m[2023-01-04:12:16:41:INFO] No GPUs detected (normal if no gpus installed)\u001b[0m\n", "\u001b[34m[2023-01-04:12:16:41:INFO] Running XGBoost Sagemaker in algorithm mode\u001b[0m\n", "\u001b[34m[2023-01-04:12:16:41:INFO] Determined delimiter of CSV input is ','\u001b[0m\n", "\u001b[34m[2023-01-04:12:16:41:INFO] Determined delimiter of CSV input is ','\u001b[0m\n", "\u001b[34m[2023-01-04:12:16:41:INFO] files path: /opt/ml/input/data/train\u001b[0m\n", "\u001b[34m[2023-01-04:12:16:41:INFO] Determined delimiter of CSV input is ','\u001b[0m\n", "\u001b[34m[2023-01-04:12:16:42:INFO] files path: /opt/ml/input/data/validation\u001b[0m\n", "\u001b[34m[2023-01-04:12:16:42:INFO] Determined delimiter of CSV input is ','\u001b[0m\n", "\u001b[34m[2023-01-04:12:16:42:INFO] Single node training.\u001b[0m\n", "\u001b[34m[2023-01-04:12:16:42:INFO] Train matrix has 132868 rows and 13 columns\u001b[0m\n", "\u001b[34m[2023-01-04:12:16:42:INFO] Validation matrix has 117763 rows\u001b[0m\n", "\u001b[34m[2023-01-04 12:16:42.082 ip-10-0-203-120.ap-northeast-1.compute.internal:7 INFO json_config.py:91] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.\u001b[0m\n", "\u001b[34m[12:16:42] WARNING: ../src/learner.cc:1115: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\u001b[0m\n", "\u001b[34m[0]#011train-logloss:0.61257#011validation-logloss:0.61642\u001b[0m\n", "\u001b[34m[1]#011train-logloss:0.55789#011validation-logloss:0.56494\u001b[0m\n", "\u001b[34m[2]#011train-logloss:0.51880#011validation-logloss:0.52843\u001b[0m\n", "\u001b[34m[3]#011train-logloss:0.49021#011validation-logloss:0.50201\u001b[0m\n", "\u001b[34m[4]#011train-logloss:0.46840#011validation-logloss:0.48225\u001b[0m\n", "\u001b[34m[5]#011train-logloss:0.45182#011validation-logloss:0.46709\u001b[0m\n", "\u001b[34m[6]#011train-logloss:0.43852#011validation-logloss:0.45499\u001b[0m\n", "\u001b[34m[7]#011train-logloss:0.42843#011validation-logloss:0.44599\u001b[0m\n", "\u001b[34m[8]#011train-logloss:0.42050#011validation-logloss:0.43904\u001b[0m\n", "\u001b[34m[9]#011train-logloss:0.41425#011validation-logloss:0.43351\u001b[0m\n", "\u001b[34m[10]#011train-logloss:0.40904#011validation-logloss:0.42892\u001b[0m\n", "\u001b[34m[11]#011train-logloss:0.40503#011validation-logloss:0.42547\u001b[0m\n", "\u001b[34m[12]#011train-logloss:0.40169#011validation-logloss:0.42265\u001b[0m\n", "\u001b[34m[13]#011train-logloss:0.39877#011validation-logloss:0.42016\u001b[0m\n", "\u001b[34m[14]#011train-logloss:0.39649#011validation-logloss:0.41828\u001b[0m\n", "\u001b[34m[15]#011train-logloss:0.39485#011validation-logloss:0.41693\u001b[0m\n", "\u001b[34m[16]#011train-logloss:0.39321#011validation-logloss:0.41580\u001b[0m\n", "\u001b[34m[17]#011train-logloss:0.39182#011validation-logloss:0.41470\u001b[0m\n", "\u001b[34m[18]#011train-logloss:0.39089#011validation-logloss:0.41410\u001b[0m\n", "\u001b[34m[19]#011train-logloss:0.38970#011validation-logloss:0.41318\u001b[0m\n", "\u001b[34m[20]#011train-logloss:0.38878#011validation-logloss:0.41250\u001b[0m\n", "\u001b[34m[21]#011train-logloss:0.38814#011validation-logloss:0.41210\u001b[0m\n", "\u001b[34m[22]#011train-logloss:0.38739#011validation-logloss:0.41154\u001b[0m\n", "\u001b[34m[23]#011train-logloss:0.38684#011validation-logloss:0.41119\u001b[0m\n", "\u001b[34m[24]#011train-logloss:0.38640#011validation-logloss:0.41091\u001b[0m\n", "\u001b[34m[25]#011train-logloss:0.38587#011validation-logloss:0.41068\u001b[0m\n", "\u001b[34m[26]#011train-logloss:0.38552#011validation-logloss:0.41049\u001b[0m\n", "\u001b[34m[27]#011train-logloss:0.38518#011validation-logloss:0.41030\u001b[0m\n", "\u001b[34m[28]#011train-logloss:0.38475#011validation-logloss:0.41003\u001b[0m\n", "\u001b[34m[29]#011train-logloss:0.38448#011validation-logloss:0.40993\u001b[0m\n", "\u001b[34m[30]#011train-logloss:0.38421#011validation-logloss:0.40975\u001b[0m\n", "\u001b[34m[31]#011train-logloss:0.38398#011validation-logloss:0.40967\u001b[0m\n", "\u001b[34m[32]#011train-logloss:0.38378#011validation-logloss:0.40952\u001b[0m\n", "\u001b[34m[33]#011train-logloss:0.38360#011validation-logloss:0.40941\u001b[0m\n", "\u001b[34m[34]#011train-logloss:0.38335#011validation-logloss:0.40937\u001b[0m\n", "\u001b[34m[35]#011train-logloss:0.38307#011validation-logloss:0.40926\u001b[0m\n", "\u001b[34m[36]#011train-logloss:0.38277#011validation-logloss:0.40912\u001b[0m\n", "\u001b[34m[37]#011train-logloss:0.38254#011validation-logloss:0.40897\u001b[0m\n", "\u001b[34m[38]#011train-logloss:0.38241#011validation-logloss:0.40883\u001b[0m\n", "\u001b[34m[39]#011train-logloss:0.38237#011validation-logloss:0.40880\u001b[0m\n", "\u001b[34m[40]#011train-logloss:0.38215#011validation-logloss:0.40875\u001b[0m\n", "\u001b[34m[41]#011train-logloss:0.38203#011validation-logloss:0.40871\u001b[0m\n", "\u001b[34m[42]#011train-logloss:0.38186#011validation-logloss:0.40871\u001b[0m\n", "\u001b[34m[43]#011train-logloss:0.38177#011validation-logloss:0.40871\u001b[0m\n", "\u001b[34m[44]#011train-logloss:0.38167#011validation-logloss:0.40868\u001b[0m\n", "\u001b[34m[45]#011train-logloss:0.38142#011validation-logloss:0.40855\u001b[0m\n", "\u001b[34m[46]#011train-logloss:0.38127#011validation-logloss:0.40851\u001b[0m\n", "\u001b[34m[47]#011train-logloss:0.38119#011validation-logloss:0.40845\u001b[0m\n", "\u001b[34m[48]#011train-logloss:0.38110#011validation-logloss:0.40846\u001b[0m\n", "\u001b[34m[49]#011train-logloss:0.38102#011validation-logloss:0.40848\u001b[0m\n", "\u001b[34m[50]#011train-logloss:0.38090#011validation-logloss:0.40848\u001b[0m\n", "\u001b[34m[51]#011train-logloss:0.38077#011validation-logloss:0.40841\u001b[0m\n", "\u001b[34m[52]#011train-logloss:0.38061#011validation-logloss:0.40836\u001b[0m\n", "\u001b[34m[53]#011train-logloss:0.38050#011validation-logloss:0.40834\u001b[0m\n", "\u001b[34m[54]#011train-logloss:0.38043#011validation-logloss:0.40832\u001b[0m\n", "\u001b[34m[55]#011train-logloss:0.38036#011validation-logloss:0.40833\u001b[0m\n", "\u001b[34m[56]#011train-logloss:0.38021#011validation-logloss:0.40836\u001b[0m\n", "\u001b[34m[57]#011train-logloss:0.38016#011validation-logloss:0.40831\u001b[0m\n", "\u001b[34m[58]#011train-logloss:0.38009#011validation-logloss:0.40831\u001b[0m\n", "\u001b[34m[59]#011train-logloss:0.38001#011validation-logloss:0.40837\u001b[0m\n", "\u001b[34m[60]#011train-logloss:0.37993#011validation-logloss:0.40833\u001b[0m\n", "\u001b[34m[61]#011train-logloss:0.37982#011validation-logloss:0.40838\u001b[0m\n", "\u001b[34m[62]#011train-logloss:0.37972#011validation-logloss:0.40841\u001b[0m\n", "\u001b[34m[63]#011train-logloss:0.37966#011validation-logloss:0.40839\u001b[0m\n", "\u001b[34m[64]#011train-logloss:0.37953#011validation-logloss:0.40835\u001b[0m\n", "\u001b[34m[65]#011train-logloss:0.37944#011validation-logloss:0.40831\u001b[0m\n", "\u001b[34m[66]#011train-logloss:0.37932#011validation-logloss:0.40834\u001b[0m\n", "\u001b[34m[67]#011train-logloss:0.37929#011validation-logloss:0.40835\u001b[0m\n", "\u001b[34m[68]#011train-logloss:0.37922#011validation-logloss:0.40840\u001b[0m\n", "\u001b[34m[69]#011train-logloss:0.37918#011validation-logloss:0.40841\u001b[0m\n", "\u001b[34m[70]#011train-logloss:0.37915#011validation-logloss:0.40841\u001b[0m\n", "\u001b[34m[71]#011train-logloss:0.37907#011validation-logloss:0.40838\u001b[0m\n", "\u001b[34m[72]#011train-logloss:0.37898#011validation-logloss:0.40837\u001b[0m\n", "\u001b[34m[73]#011train-logloss:0.37897#011validation-logloss:0.40838\u001b[0m\n", "\u001b[34m[74]#011train-logloss:0.37883#011validation-logloss:0.40836\u001b[0m\n", "\u001b[34m[75]#011train-logloss:0.37875#011validation-logloss:0.40832\u001b[0m\n", "\u001b[34m[76]#011train-logloss:0.37865#011validation-logloss:0.40831\u001b[0m\n", "\u001b[34m[77]#011train-logloss:0.37861#011validation-logloss:0.40833\u001b[0m\n", "\u001b[34m[78]#011train-logloss:0.37852#011validation-logloss:0.40828\u001b[0m\n", "\u001b[34m[79]#011train-logloss:0.37849#011validation-logloss:0.40829\u001b[0m\n", "\u001b[34m[80]#011train-logloss:0.37840#011validation-logloss:0.40829\u001b[0m\n", "\u001b[34m[81]#011train-logloss:0.37838#011validation-logloss:0.40829\u001b[0m\n", "\u001b[34m[82]#011train-logloss:0.37832#011validation-logloss:0.40832\u001b[0m\n", "\u001b[34m[83]#011train-logloss:0.37826#011validation-logloss:0.40833\u001b[0m\n", "\u001b[34m[84]#011train-logloss:0.37821#011validation-logloss:0.40830\u001b[0m\n", "\u001b[34m[85]#011train-logloss:0.37810#011validation-logloss:0.40830\u001b[0m\n", "\u001b[34m[86]#011train-logloss:0.37800#011validation-logloss:0.40823\u001b[0m\n", "\u001b[34m[87]#011train-logloss:0.37795#011validation-logloss:0.40822\u001b[0m\n", "\u001b[34m[88]#011train-logloss:0.37791#011validation-logloss:0.40821\u001b[0m\n", "\u001b[34m[89]#011train-logloss:0.37779#011validation-logloss:0.40821\u001b[0m\n", "\u001b[34m[90]#011train-logloss:0.37774#011validation-logloss:0.40821\u001b[0m\n", "\u001b[34m[91]#011train-logloss:0.37770#011validation-logloss:0.40820\u001b[0m\n", "\u001b[34m[92]#011train-logloss:0.37768#011validation-logloss:0.40817\u001b[0m\n", "\u001b[34m[93]#011train-logloss:0.37766#011validation-logloss:0.40818\u001b[0m\n", "\u001b[34m[94]#011train-logloss:0.37760#011validation-logloss:0.40816\u001b[0m\n", "\u001b[34m[95]#011train-logloss:0.37757#011validation-logloss:0.40816\u001b[0m\n", "\u001b[34m[96]#011train-logloss:0.37752#011validation-logloss:0.40816\u001b[0m\n", "\u001b[34m[97]#011train-logloss:0.37737#011validation-logloss:0.40811\u001b[0m\n", "\u001b[34m[98]#011train-logloss:0.37731#011validation-logloss:0.40816\u001b[0m\n", "\u001b[34m[99]#011train-logloss:0.37729#011validation-logloss:0.40816\u001b[0m\n", "\u001b[34m[100]#011train-logloss:0.37721#011validation-logloss:0.40818\u001b[0m\n", "\u001b[34m[101]#011train-logloss:0.37716#011validation-logloss:0.40819\u001b[0m\n", "\u001b[34m[102]#011train-logloss:0.37710#011validation-logloss:0.40817\u001b[0m\n", "\u001b[34m[103]#011train-logloss:0.37706#011validation-logloss:0.40820\u001b[0m\n", "\u001b[34m[104]#011train-logloss:0.37706#011validation-logloss:0.40821\u001b[0m\n", "\u001b[34m[105]#011train-logloss:0.37703#011validation-logloss:0.40821\u001b[0m\n", "\u001b[34m[106]#011train-logloss:0.37702#011validation-logloss:0.40821\u001b[0m\n", "\u001b[34m[107]#011train-logloss:0.37700#011validation-logloss:0.40818\u001b[0m\n", "\u001b[34m[108]#011train-logloss:0.37695#011validation-logloss:0.40817\u001b[0m\n", "\u001b[34m[109]#011train-logloss:0.37689#011validation-logloss:0.40817\u001b[0m\n", "\u001b[34m[110]#011train-logloss:0.37684#011validation-logloss:0.40816\u001b[0m\n", "\u001b[34m[111]#011train-logloss:0.37682#011validation-logloss:0.40815\u001b[0m\n", "\u001b[34m[112]#011train-logloss:0.37675#011validation-logloss:0.40819\u001b[0m\n", "\u001b[34m[113]#011train-logloss:0.37671#011validation-logloss:0.40820\u001b[0m\n", "\u001b[34m[114]#011train-logloss:0.37669#011validation-logloss:0.40819\u001b[0m\n", "\u001b[34m[115]#011train-logloss:0.37666#011validation-logloss:0.40816\u001b[0m\n", "\u001b[34m[116]#011train-logloss:0.37661#011validation-logloss:0.40819\u001b[0m\n", "\u001b[34m[117]#011train-logloss:0.37656#011validation-logloss:0.40819\u001b[0m\n", "\u001b[34m[118]#011train-logloss:0.37648#011validation-logloss:0.40818\u001b[0m\n", "\u001b[34m[119]#011train-logloss:0.37647#011validation-logloss:0.40817\u001b[0m\n", "\u001b[34m[120]#011train-logloss:0.37647#011validation-logloss:0.40818\u001b[0m\n", "\u001b[34m[121]#011train-logloss:0.37640#011validation-logloss:0.40819\u001b[0m\n", "\u001b[34m[122]#011train-logloss:0.37637#011validation-logloss:0.40819\u001b[0m\n", "\u001b[34m[123]#011train-logloss:0.37627#011validation-logloss:0.40815\u001b[0m\n", "\u001b[34m[124]#011train-logloss:0.37621#011validation-logloss:0.40816\u001b[0m\n", "\u001b[34m[125]#011train-logloss:0.37616#011validation-logloss:0.40816\u001b[0m\n", "\u001b[34m[126]#011train-logloss:0.37611#011validation-logloss:0.40818\u001b[0m\n", "\u001b[34m[127]#011train-logloss:0.37608#011validation-logloss:0.40820\u001b[0m\n", "\u001b[34m[128]#011train-logloss:0.37604#011validation-logloss:0.40824\u001b[0m\n", "\u001b[34m[129]#011train-logloss:0.37598#011validation-logloss:0.40822\u001b[0m\n", "\u001b[34m[130]#011train-logloss:0.37595#011validation-logloss:0.40822\u001b[0m\n", "\u001b[34m[131]#011train-logloss:0.37595#011validation-logloss:0.40823\u001b[0m\n", "\u001b[34m[132]#011train-logloss:0.37591#011validation-logloss:0.40819\u001b[0m\n", "\u001b[34m[133]#011train-logloss:0.37587#011validation-logloss:0.40819\u001b[0m\n", "\u001b[34m[134]#011train-logloss:0.37586#011validation-logloss:0.40818\u001b[0m\n", "\u001b[34m[135]#011train-logloss:0.37585#011validation-logloss:0.40821\u001b[0m\n", "\u001b[34m[136]#011train-logloss:0.37582#011validation-logloss:0.40824\u001b[0m\n", "\u001b[34m[137]#011train-logloss:0.37578#011validation-logloss:0.40822\u001b[0m\n", "\u001b[34m[138]#011train-logloss:0.37577#011validation-logloss:0.40819\u001b[0m\n", "\u001b[34m[139]#011train-logloss:0.37576#011validation-logloss:0.40820\u001b[0m\n", "\u001b[34m[140]#011train-logloss:0.37571#011validation-logloss:0.40819\u001b[0m\n", "\u001b[34m[141]#011train-logloss:0.37566#011validation-logloss:0.40823\u001b[0m\n", "\u001b[34m[142]#011train-logloss:0.37563#011validation-logloss:0.40823\u001b[0m\n", "\u001b[34m[143]#011train-logloss:0.37556#011validation-logloss:0.40825\u001b[0m\n", "\u001b[34m[144]#011train-logloss:0.37550#011validation-logloss:0.40827\u001b[0m\n", "\u001b[34m[145]#011train-logloss:0.37550#011validation-logloss:0.40828\u001b[0m\n", "\u001b[34m[146]#011train-logloss:0.37547#011validation-logloss:0.40827\u001b[0m\n", "\u001b[34m[147]#011train-logloss:0.37547#011validation-logloss:0.40828\u001b[0m\n", "\u001b[34m[148]#011train-logloss:0.37546#011validation-logloss:0.40827\u001b[0m\n", "\u001b[34m[149]#011train-logloss:0.37542#011validation-logloss:0.40827\u001b[0m\n", "\u001b[34m[150]#011train-logloss:0.37542#011validation-logloss:0.40827\u001b[0m\n", "\u001b[34m[151]#011train-logloss:0.37539#011validation-logloss:0.40825\u001b[0m\n", "\u001b[34m[152]#011train-logloss:0.37538#011validation-logloss:0.40829\u001b[0m\n", "\u001b[34m[153]#011train-logloss:0.37537#011validation-logloss:0.40830\u001b[0m\n", "\u001b[34m[154]#011train-logloss:0.37535#011validation-logloss:0.40832\u001b[0m\n", "\u001b[34m[155]#011train-logloss:0.37526#011validation-logloss:0.40831\u001b[0m\n", "\u001b[34m[156]#011train-logloss:0.37526#011validation-logloss:0.40831\u001b[0m\n", "\u001b[34m[157]#011train-logloss:0.37518#011validation-logloss:0.40831\u001b[0m\n", "\u001b[34m[158]#011train-logloss:0.37517#011validation-logloss:0.40833\u001b[0m\n", "\u001b[34m[159]#011train-logloss:0.37513#011validation-logloss:0.40833\u001b[0m\n", "\u001b[34m[160]#011train-logloss:0.37511#011validation-logloss:0.40834\u001b[0m\n", "\u001b[34m[161]#011train-logloss:0.37507#011validation-logloss:0.40836\u001b[0m\n", "\u001b[34m[162]#011train-logloss:0.37506#011validation-logloss:0.40832\u001b[0m\n", "\u001b[34m[163]#011train-logloss:0.37505#011validation-logloss:0.40832\u001b[0m\n", "\u001b[34m[164]#011train-logloss:0.37505#011validation-logloss:0.40834\u001b[0m\n", "\u001b[34m[165]#011train-logloss:0.37498#011validation-logloss:0.40831\u001b[0m\n", "\u001b[34m[166]#011train-logloss:0.37497#011validation-logloss:0.40833\u001b[0m\n", "\u001b[34m[167]#011train-logloss:0.37494#011validation-logloss:0.40834\u001b[0m\n", "\u001b[34m[168]#011train-logloss:0.37494#011validation-logloss:0.40836\u001b[0m\n", "\u001b[34m[169]#011train-logloss:0.37490#011validation-logloss:0.40834\u001b[0m\n", "\u001b[34m[170]#011train-logloss:0.37482#011validation-logloss:0.40834\u001b[0m\n", "\u001b[34m[171]#011train-logloss:0.37481#011validation-logloss:0.40835\u001b[0m\n", "\u001b[34m[172]#011train-logloss:0.37476#011validation-logloss:0.40832\u001b[0m\n", "\u001b[34m[173]#011train-logloss:0.37476#011validation-logloss:0.40832\u001b[0m\n", "\u001b[34m[174]#011train-logloss:0.37476#011validation-logloss:0.40831\u001b[0m\n", "\u001b[34m[175]#011train-logloss:0.37472#011validation-logloss:0.40831\u001b[0m\n", "\u001b[34m[176]#011train-logloss:0.37471#011validation-logloss:0.40830\u001b[0m\n", "\u001b[34m[177]#011train-logloss:0.37469#011validation-logloss:0.40829\u001b[0m\n", "\u001b[34m[178]#011train-logloss:0.37469#011validation-logloss:0.40822\u001b[0m\n", "\u001b[34m[179]#011train-logloss:0.37469#011validation-logloss:0.40825\u001b[0m\n", "\u001b[34m[180]#011train-logloss:0.37466#011validation-logloss:0.40826\u001b[0m\n", "\u001b[34m[181]#011train-logloss:0.37463#011validation-logloss:0.40827\u001b[0m\n", "\u001b[34m[182]#011train-logloss:0.37462#011validation-logloss:0.40829\u001b[0m\n", "\u001b[34m[183]#011train-logloss:0.37460#011validation-logloss:0.40830\u001b[0m\n", "\u001b[34m[184]#011train-logloss:0.37460#011validation-logloss:0.40827\u001b[0m\n", "\u001b[34m[185]#011train-logloss:0.37459#011validation-logloss:0.40829\u001b[0m\n", "\u001b[34m[186]#011train-logloss:0.37457#011validation-logloss:0.40831\u001b[0m\n", "\u001b[34m[187]#011train-logloss:0.37451#011validation-logloss:0.40830\u001b[0m\n", "\u001b[34m[188]#011train-logloss:0.37444#011validation-logloss:0.40828\u001b[0m\n", "\u001b[34m[189]#011train-logloss:0.37442#011validation-logloss:0.40829\u001b[0m\n", "\u001b[34m[190]#011train-logloss:0.37440#011validation-logloss:0.40835\u001b[0m\n", "\u001b[34m[191]#011train-logloss:0.37440#011validation-logloss:0.40834\u001b[0m\n", "\u001b[34m[192]#011train-logloss:0.37438#011validation-logloss:0.40834\u001b[0m\n", "\u001b[34m[193]#011train-logloss:0.37439#011validation-logloss:0.40832\u001b[0m\n", "\u001b[34m[194]#011train-logloss:0.37435#011validation-logloss:0.40835\u001b[0m\n", "\u001b[34m[195]#011train-logloss:0.37434#011validation-logloss:0.40835\u001b[0m\n", "\u001b[34m[196]#011train-logloss:0.37429#011validation-logloss:0.40831\u001b[0m\n", "\u001b[34m[197]#011train-logloss:0.37428#011validation-logloss:0.40833\u001b[0m\n", "\u001b[34m[198]#011train-logloss:0.37426#011validation-logloss:0.40829\u001b[0m\n", "\u001b[34m[199]#011train-logloss:0.37421#011validation-logloss:0.40835\u001b[0m\n", "\u001b[34m[200]#011train-logloss:0.37418#011validation-logloss:0.40835\u001b[0m\n", "\u001b[34m[201]#011train-logloss:0.37418#011validation-logloss:0.40834\u001b[0m\n", "\u001b[34m[202]#011train-logloss:0.37415#011validation-logloss:0.40828\u001b[0m\n", "\u001b[34m[203]#011train-logloss:0.37414#011validation-logloss:0.40826\u001b[0m\n", "\u001b[34m[204]#011train-logloss:0.37408#011validation-logloss:0.40829\u001b[0m\n", "\u001b[34m[205]#011train-logloss:0.37406#011validation-logloss:0.40829\u001b[0m\n", "\u001b[34m[206]#011train-logloss:0.37406#011validation-logloss:0.40832\u001b[0m\n", "\u001b[34m[207]#011train-logloss:0.37406#011validation-logloss:0.40835\u001b[0m\n", "\u001b[34m[208]#011train-logloss:0.37403#011validation-logloss:0.40833\u001b[0m\n", "\u001b[34m[209]#011train-logloss:0.37397#011validation-logloss:0.40835\u001b[0m\n", "\u001b[34m[210]#011train-logloss:0.37397#011validation-logloss:0.40836\u001b[0m\n", "\u001b[34m[211]#011train-logloss:0.37389#011validation-logloss:0.40836\u001b[0m\n", "\u001b[34m[212]#011train-logloss:0.37387#011validation-logloss:0.40838\u001b[0m\n", "\u001b[34m[213]#011train-logloss:0.37385#011validation-logloss:0.40840\u001b[0m\n", "\u001b[34m[214]#011train-logloss:0.37383#011validation-logloss:0.40837\u001b[0m\n", "\u001b[34m[215]#011train-logloss:0.37381#011validation-logloss:0.40836\u001b[0m\n", "\u001b[34m[216]#011train-logloss:0.37380#011validation-logloss:0.40837\u001b[0m\n", "\u001b[34m[217]#011train-logloss:0.37376#011validation-logloss:0.40838\u001b[0m\n", "\u001b[34m[218]#011train-logloss:0.37372#011validation-logloss:0.40836\u001b[0m\n", "\u001b[34m[219]#011train-logloss:0.37368#011validation-logloss:0.40836\u001b[0m\n", "\u001b[34m[220]#011train-logloss:0.37368#011validation-logloss:0.40836\u001b[0m\n", "\u001b[34m[221]#011train-logloss:0.37367#011validation-logloss:0.40840\u001b[0m\n", "\u001b[34m[222]#011train-logloss:0.37361#011validation-logloss:0.40840\u001b[0m\n", "\u001b[34m[223]#011train-logloss:0.37357#011validation-logloss:0.40841\u001b[0m\n", "\u001b[34m[224]#011train-logloss:0.37353#011validation-logloss:0.40839\u001b[0m\n", "\u001b[34m[225]#011train-logloss:0.37353#011validation-logloss:0.40837\u001b[0m\n", "\u001b[34m[226]#011train-logloss:0.37350#011validation-logloss:0.40843\u001b[0m\n", "\u001b[34m[227]#011train-logloss:0.37348#011validation-logloss:0.40842\u001b[0m\n", "\u001b[34m[228]#011train-logloss:0.37344#011validation-logloss:0.40846\u001b[0m\n", "\u001b[34m[229]#011train-logloss:0.37340#011validation-logloss:0.40847\u001b[0m\n", "\u001b[34m[230]#011train-logloss:0.37339#011validation-logloss:0.40846\u001b[0m\n", "\u001b[34m[231]#011train-logloss:0.37339#011validation-logloss:0.40847\u001b[0m\n", "\u001b[34m[232]#011train-logloss:0.37334#011validation-logloss:0.40845\u001b[0m\n", "\u001b[34m[233]#011train-logloss:0.37332#011validation-logloss:0.40848\u001b[0m\n", "\u001b[34m[234]#011train-logloss:0.37329#011validation-logloss:0.40851\u001b[0m\n", "\u001b[34m[235]#011train-logloss:0.37328#011validation-logloss:0.40854\u001b[0m\n", "\u001b[34m[236]#011train-logloss:0.37324#011validation-logloss:0.40859\u001b[0m\n", "\u001b[34m[237]#011train-logloss:0.37320#011validation-logloss:0.40857\u001b[0m\n", "\u001b[34m[238]#011train-logloss:0.37317#011validation-logloss:0.40857\u001b[0m\n", "\u001b[34m[239]#011train-logloss:0.37314#011validation-logloss:0.40856\u001b[0m\n", "\u001b[34m[240]#011train-logloss:0.37313#011validation-logloss:0.40858\u001b[0m\n", "\u001b[34m[241]#011train-logloss:0.37313#011validation-logloss:0.40856\u001b[0m\n", "\u001b[34m[242]#011train-logloss:0.37308#011validation-logloss:0.40856\u001b[0m\n", "\u001b[34m[243]#011train-logloss:0.37305#011validation-logloss:0.40855\u001b[0m\n", "\u001b[34m[244]#011train-logloss:0.37303#011validation-logloss:0.40858\u001b[0m\n", "\u001b[34m[245]#011train-logloss:0.37303#011validation-logloss:0.40858\u001b[0m\n", "\u001b[34m[246]#011train-logloss:0.37302#011validation-logloss:0.40863\u001b[0m\n", "\u001b[34m[247]#011train-logloss:0.37299#011validation-logloss:0.40864\u001b[0m\n", "\u001b[34m[248]#011train-logloss:0.37299#011validation-logloss:0.40862\u001b[0m\n", "\u001b[34m[249]#011train-logloss:0.37298#011validation-logloss:0.40859\u001b[0m\n", "\u001b[34m[250]#011train-logloss:0.37295#011validation-logloss:0.40862\u001b[0m\n", "\u001b[34m[251]#011train-logloss:0.37289#011validation-logloss:0.40865\u001b[0m\n", "\u001b[34m[252]#011train-logloss:0.37286#011validation-logloss:0.40864\u001b[0m\n", "\u001b[34m[253]#011train-logloss:0.37285#011validation-logloss:0.40866\u001b[0m\n", "\u001b[34m[254]#011train-logloss:0.37285#011validation-logloss:0.40867\u001b[0m\n", "\u001b[34m[255]#011train-logloss:0.37284#011validation-logloss:0.40865\u001b[0m\n", "\u001b[34m[256]#011train-logloss:0.37282#011validation-logloss:0.40867\u001b[0m\n", "\u001b[34m[257]#011train-logloss:0.37282#011validation-logloss:0.40864\u001b[0m\n", "\u001b[34m[258]#011train-logloss:0.37279#011validation-logloss:0.40864\u001b[0m\n", "\u001b[34m[259]#011train-logloss:0.37278#011validation-logloss:0.40866\u001b[0m\n", "\u001b[34m[260]#011train-logloss:0.37276#011validation-logloss:0.40861\u001b[0m\n", "\u001b[34m[261]#011train-logloss:0.37270#011validation-logloss:0.40863\u001b[0m\n", "\u001b[34m[262]#011train-logloss:0.37266#011validation-logloss:0.40865\u001b[0m\n", "\u001b[34m[263]#011train-logloss:0.37264#011validation-logloss:0.40869\u001b[0m\n", "\u001b[34m[264]#011train-logloss:0.37263#011validation-logloss:0.40867\u001b[0m\n", "\u001b[34m[265]#011train-logloss:0.37259#011validation-logloss:0.40867\u001b[0m\n", "\u001b[34m[266]#011train-logloss:0.37255#011validation-logloss:0.40873\u001b[0m\n", "\u001b[34m[267]#011train-logloss:0.37253#011validation-logloss:0.40873\u001b[0m\n", "\u001b[34m[268]#011train-logloss:0.37251#011validation-logloss:0.40874\u001b[0m\n", "\u001b[34m[269]#011train-logloss:0.37251#011validation-logloss:0.40872\u001b[0m\n", "\u001b[34m[270]#011train-logloss:0.37249#011validation-logloss:0.40869\u001b[0m\n", "\u001b[34m[271]#011train-logloss:0.37249#011validation-logloss:0.40869\u001b[0m\n", "\u001b[34m[272]#011train-logloss:0.37246#011validation-logloss:0.40871\u001b[0m\n", "\u001b[34m[273]#011train-logloss:0.37241#011validation-logloss:0.40869\u001b[0m\n", "\u001b[34m[274]#011train-logloss:0.37236#011validation-logloss:0.40868\u001b[0m\n", "\u001b[34m[275]#011train-logloss:0.37232#011validation-logloss:0.40866\u001b[0m\n", "\u001b[34m[276]#011train-logloss:0.37228#011validation-logloss:0.40869\u001b[0m\n", "\u001b[34m[277]#011train-logloss:0.37227#011validation-logloss:0.40869\u001b[0m\n", "\u001b[34m[278]#011train-logloss:0.37223#011validation-logloss:0.40871\u001b[0m\n", "\u001b[34m[279]#011train-logloss:0.37221#011validation-logloss:0.40874\u001b[0m\n", "\u001b[34m[280]#011train-logloss:0.37218#011validation-logloss:0.40871\u001b[0m\n", "\u001b[34m[281]#011train-logloss:0.37217#011validation-logloss:0.40873\u001b[0m\n", "\u001b[34m[282]#011train-logloss:0.37214#011validation-logloss:0.40872\u001b[0m\n", "\u001b[34m[283]#011train-logloss:0.37208#011validation-logloss:0.40874\u001b[0m\n", "\u001b[34m[284]#011train-logloss:0.37206#011validation-logloss:0.40872\u001b[0m\n", "\u001b[34m[285]#011train-logloss:0.37205#011validation-logloss:0.40875\u001b[0m\n", "\u001b[34m[286]#011train-logloss:0.37201#011validation-logloss:0.40874\u001b[0m\n", "\u001b[34m[287]#011train-logloss:0.37202#011validation-logloss:0.40874\u001b[0m\n", "\u001b[34m[288]#011train-logloss:0.37197#011validation-logloss:0.40875\u001b[0m\n", "\u001b[34m[289]#011train-logloss:0.37195#011validation-logloss:0.40875\u001b[0m\n", "\u001b[34m[290]#011train-logloss:0.37193#011validation-logloss:0.40876\u001b[0m\n", "\u001b[34m[291]#011train-logloss:0.37190#011validation-logloss:0.40876\u001b[0m\n", "\u001b[34m[292]#011train-logloss:0.37189#011validation-logloss:0.40874\u001b[0m\n", "\u001b[34m[293]#011train-logloss:0.37186#011validation-logloss:0.40875\u001b[0m\n", "\u001b[34m[294]#011train-logloss:0.37186#011validation-logloss:0.40874\u001b[0m\n", "\u001b[34m[295]#011train-logloss:0.37186#011validation-logloss:0.40875\u001b[0m\n", "\u001b[34m[296]#011train-logloss:0.37179#011validation-logloss:0.40875\u001b[0m\n", "\u001b[34m[297]#011train-logloss:0.37175#011validation-logloss:0.40880\u001b[0m\n", "\u001b[34m[298]#011train-logloss:0.37174#011validation-logloss:0.40880\u001b[0m\n", "\u001b[34m[299]#011train-logloss:0.37174#011validation-logloss:0.40880\u001b[0m\n", "\u001b[34m[300]#011train-logloss:0.37172#011validation-logloss:0.40877\u001b[0m\n", "\u001b[34m[301]#011train-logloss:0.37170#011validation-logloss:0.40876\u001b[0m\n", "\u001b[34m[302]#011train-logloss:0.37166#011validation-logloss:0.40878\u001b[0m\n", "\u001b[34m[303]#011train-logloss:0.37164#011validation-logloss:0.40879\u001b[0m\n", "\u001b[34m[304]#011train-logloss:0.37164#011validation-logloss:0.40880\u001b[0m\n", "\u001b[34m[305]#011train-logloss:0.37164#011validation-logloss:0.40881\u001b[0m\n", "\u001b[34m[306]#011train-logloss:0.37161#011validation-logloss:0.40879\u001b[0m\n", "\u001b[34m[307]#011train-logloss:0.37160#011validation-logloss:0.40881\u001b[0m\n", "\u001b[34m[308]#011train-logloss:0.37160#011validation-logloss:0.40879\u001b[0m\n", "\u001b[34m[309]#011train-logloss:0.37157#011validation-logloss:0.40879\u001b[0m\n", "\u001b[34m[310]#011train-logloss:0.37157#011validation-logloss:0.40881\u001b[0m\n", "\u001b[34m[311]#011train-logloss:0.37156#011validation-logloss:0.40882\u001b[0m\n", "\u001b[34m[312]#011train-logloss:0.37156#011validation-logloss:0.40882\u001b[0m\n", "\u001b[34m[313]#011train-logloss:0.37156#011validation-logloss:0.40881\u001b[0m\n", "\u001b[34m[314]#011train-logloss:0.37154#011validation-logloss:0.40880\u001b[0m\n", "\u001b[34m[315]#011train-logloss:0.37149#011validation-logloss:0.40881\u001b[0m\n", "\u001b[34m[316]#011train-logloss:0.37149#011validation-logloss:0.40880\u001b[0m\n", "\u001b[34m[317]#011train-logloss:0.37149#011validation-logloss:0.40880\u001b[0m\n", "\u001b[34m[318]#011train-logloss:0.37147#011validation-logloss:0.40879\u001b[0m\n", "\u001b[34m[319]#011train-logloss:0.37144#011validation-logloss:0.40880\u001b[0m\n", "\u001b[34m[320]#011train-logloss:0.37144#011validation-logloss:0.40880\u001b[0m\n", "\u001b[34m[321]#011train-logloss:0.37143#011validation-logloss:0.40881\u001b[0m\n", "\u001b[34m[322]#011train-logloss:0.37142#011validation-logloss:0.40884\u001b[0m\n", "\u001b[34m[323]#011train-logloss:0.37139#011validation-logloss:0.40882\u001b[0m\n", "\u001b[34m[324]#011train-logloss:0.37136#011validation-logloss:0.40886\u001b[0m\n", "\u001b[34m[325]#011train-logloss:0.37135#011validation-logloss:0.40888\u001b[0m\n", "\u001b[34m[326]#011train-logloss:0.37132#011validation-logloss:0.40886\u001b[0m\n", "\u001b[34m[327]#011train-logloss:0.37131#011validation-logloss:0.40884\u001b[0m\n", "\u001b[34m[328]#011train-logloss:0.37126#011validation-logloss:0.40886\u001b[0m\n", "\u001b[34m[329]#011train-logloss:0.37125#011validation-logloss:0.40887\u001b[0m\n", "\u001b[34m[330]#011train-logloss:0.37121#011validation-logloss:0.40891\u001b[0m\n", "\u001b[34m[331]#011train-logloss:0.37116#011validation-logloss:0.40891\u001b[0m\n", "\u001b[34m[332]#011train-logloss:0.37116#011validation-logloss:0.40890\u001b[0m\n", "\u001b[34m[333]#011train-logloss:0.37114#011validation-logloss:0.40888\u001b[0m\n", "\u001b[34m[334]#011train-logloss:0.37112#011validation-logloss:0.40889\u001b[0m\n", "\u001b[34m[335]#011train-logloss:0.37110#011validation-logloss:0.40889\u001b[0m\n", "\u001b[34m[336]#011train-logloss:0.37108#011validation-logloss:0.40891\u001b[0m\n", "\u001b[34m[337]#011train-logloss:0.37107#011validation-logloss:0.40893\u001b[0m\n", "\u001b[34m[338]#011train-logloss:0.37107#011validation-logloss:0.40892\u001b[0m\n", "\u001b[34m[339]#011train-logloss:0.37107#011validation-logloss:0.40893\u001b[0m\n", "\u001b[34m[340]#011train-logloss:0.37105#011validation-logloss:0.40893\u001b[0m\n", "\u001b[34m[341]#011train-logloss:0.37103#011validation-logloss:0.40892\u001b[0m\n", "\u001b[34m[342]#011train-logloss:0.37102#011validation-logloss:0.40893\u001b[0m\n", "\u001b[34m[343]#011train-logloss:0.37101#011validation-logloss:0.40896\u001b[0m\n", "\u001b[34m[344]#011train-logloss:0.37101#011validation-logloss:0.40894\u001b[0m\n", "\u001b[34m[345]#011train-logloss:0.37101#011validation-logloss:0.40893\u001b[0m\n", "\u001b[34m[346]#011train-logloss:0.37092#011validation-logloss:0.40895\u001b[0m\n", "\u001b[34m[347]#011train-logloss:0.37092#011validation-logloss:0.40892\u001b[0m\n", "\u001b[34m[348]#011train-logloss:0.37090#011validation-logloss:0.40896\u001b[0m\n", "\u001b[34m[349]#011train-logloss:0.37090#011validation-logloss:0.40897\u001b[0m\n", "\u001b[34m[350]#011train-logloss:0.37089#011validation-logloss:0.40896\u001b[0m\n", "\u001b[34m[351]#011train-logloss:0.37089#011validation-logloss:0.40898\u001b[0m\n", "\u001b[34m[352]#011train-logloss:0.37088#011validation-logloss:0.40899\u001b[0m\n", "\u001b[34m[353]#011train-logloss:0.37086#011validation-logloss:0.40900\u001b[0m\n", "\u001b[34m[354]#011train-logloss:0.37084#011validation-logloss:0.40899\u001b[0m\n", "\u001b[34m[355]#011train-logloss:0.37084#011validation-logloss:0.40899\u001b[0m\n", "\u001b[34m[356]#011train-logloss:0.37080#011validation-logloss:0.40903\u001b[0m\n", "\u001b[34m[357]#011train-logloss:0.37080#011validation-logloss:0.40902\u001b[0m\n", "\u001b[34m[358]#011train-logloss:0.37080#011validation-logloss:0.40902\u001b[0m\n", "\u001b[34m[359]#011train-logloss:0.37077#011validation-logloss:0.40902\u001b[0m\n", "\u001b[34m[360]#011train-logloss:0.37077#011validation-logloss:0.40901\u001b[0m\n", "\u001b[34m[361]#011train-logloss:0.37077#011validation-logloss:0.40899\u001b[0m\n", "\u001b[34m[362]#011train-logloss:0.37077#011validation-logloss:0.40899\u001b[0m\n", "\u001b[34m[363]#011train-logloss:0.37075#011validation-logloss:0.40900\u001b[0m\n", "\u001b[34m[364]#011train-logloss:0.37075#011validation-logloss:0.40901\u001b[0m\n", "\u001b[34m[365]#011train-logloss:0.37073#011validation-logloss:0.40905\u001b[0m\n", "\u001b[34m[366]#011train-logloss:0.37071#011validation-logloss:0.40908\u001b[0m\n", "\u001b[34m[367]#011train-logloss:0.37070#011validation-logloss:0.40906\u001b[0m\n", "\u001b[34m[368]#011train-logloss:0.37069#011validation-logloss:0.40910\u001b[0m\n", "\u001b[34m[369]#011train-logloss:0.37069#011validation-logloss:0.40908\u001b[0m\n", "\u001b[34m[370]#011train-logloss:0.37066#011validation-logloss:0.40911\u001b[0m\n", "\u001b[34m[371]#011train-logloss:0.37066#011validation-logloss:0.40911\u001b[0m\n", "\u001b[34m[372]#011train-logloss:0.37066#011validation-logloss:0.40910\u001b[0m\n", "\u001b[34m[373]#011train-logloss:0.37065#011validation-logloss:0.40907\u001b[0m\n", "\u001b[34m[374]#011train-logloss:0.37065#011validation-logloss:0.40909\u001b[0m\n", "\u001b[34m[375]#011train-logloss:0.37065#011validation-logloss:0.40910\u001b[0m\n", "\u001b[34m[376]#011train-logloss:0.37063#011validation-logloss:0.40908\u001b[0m\n", "\u001b[34m[377]#011train-logloss:0.37062#011validation-logloss:0.40908\u001b[0m\n", "\u001b[34m[378]#011train-logloss:0.37058#011validation-logloss:0.40910\u001b[0m\n", "\u001b[34m[379]#011train-logloss:0.37058#011validation-logloss:0.40912\u001b[0m\n", "\u001b[34m[380]#011train-logloss:0.37056#011validation-logloss:0.40910\u001b[0m\n", "\u001b[34m[381]#011train-logloss:0.37054#011validation-logloss:0.40910\u001b[0m\n", "\u001b[34m[382]#011train-logloss:0.37054#011validation-logloss:0.40916\u001b[0m\n", "\u001b[34m[383]#011train-logloss:0.37051#011validation-logloss:0.40915\u001b[0m\n", "\u001b[34m[384]#011train-logloss:0.37051#011validation-logloss:0.40913\u001b[0m\n", "\u001b[34m[385]#011train-logloss:0.37047#011validation-logloss:0.40908\u001b[0m\n", "\u001b[34m[386]#011train-logloss:0.37045#011validation-logloss:0.40908\u001b[0m\n", "\u001b[34m[387]#011train-logloss:0.37042#011validation-logloss:0.40910\u001b[0m\n", "\u001b[34m[388]#011train-logloss:0.37041#011validation-logloss:0.40912\u001b[0m\n", "\u001b[34m[389]#011train-logloss:0.37036#011validation-logloss:0.40914\u001b[0m\n", "\u001b[34m[390]#011train-logloss:0.37033#011validation-logloss:0.40914\u001b[0m\n", "\u001b[34m[391]#011train-logloss:0.37033#011validation-logloss:0.40915\u001b[0m\n", "\u001b[34m[392]#011train-logloss:0.37032#011validation-logloss:0.40915\u001b[0m\n", "\u001b[34m[393]#011train-logloss:0.37030#011validation-logloss:0.40912\u001b[0m\n", "\u001b[34m[394]#011train-logloss:0.37028#011validation-logloss:0.40911\u001b[0m\n", "\u001b[34m[395]#011train-logloss:0.37028#011validation-logloss:0.40909\u001b[0m\n", "\u001b[34m[396]#011train-logloss:0.37027#011validation-logloss:0.40913\u001b[0m\n", "\u001b[34m[397]#011train-logloss:0.37021#011validation-logloss:0.40914\u001b[0m\n", "\u001b[34m[398]#011train-logloss:0.37020#011validation-logloss:0.40912\u001b[0m\n", "\u001b[34m[399]#011train-logloss:0.37020#011validation-logloss:0.40913\u001b[0m\n", "\u001b[34m[400]#011train-logloss:0.37019#011validation-logloss:0.40917\u001b[0m\n", "\u001b[34m[401]#011train-logloss:0.37018#011validation-logloss:0.40917\u001b[0m\n", "\u001b[34m[402]#011train-logloss:0.37018#011validation-logloss:0.40918\u001b[0m\n", "\u001b[34m[403]#011train-logloss:0.37018#011validation-logloss:0.40919\u001b[0m\n", "\u001b[34m[404]#011train-logloss:0.37018#011validation-logloss:0.40916\u001b[0m\n", "\u001b[34m[405]#011train-logloss:0.37016#011validation-logloss:0.40913\u001b[0m\n", "\u001b[34m[406]#011train-logloss:0.37016#011validation-logloss:0.40916\u001b[0m\n", "\u001b[34m[407]#011train-logloss:0.37013#011validation-logloss:0.40914\u001b[0m\n", "\u001b[34m[408]#011train-logloss:0.37009#011validation-logloss:0.40919\u001b[0m\n", "\u001b[34m[409]#011train-logloss:0.37007#011validation-logloss:0.40921\u001b[0m\n", "\u001b[34m[410]#011train-logloss:0.37007#011validation-logloss:0.40921\u001b[0m\n", "\u001b[34m[411]#011train-logloss:0.37007#011validation-logloss:0.40920\u001b[0m\n", "\u001b[34m[412]#011train-logloss:0.37005#011validation-logloss:0.40921\u001b[0m\n", "\u001b[34m[413]#011train-logloss:0.37002#011validation-logloss:0.40924\u001b[0m\n", "\u001b[34m[414]#011train-logloss:0.37002#011validation-logloss:0.40925\u001b[0m\n", "\u001b[34m[415]#011train-logloss:0.37002#011validation-logloss:0.40922\u001b[0m\n", "\u001b[34m[416]#011train-logloss:0.37000#011validation-logloss:0.40921\u001b[0m\n", "\u001b[34m[417]#011train-logloss:0.37000#011validation-logloss:0.40921\u001b[0m\n", "\u001b[34m[418]#011train-logloss:0.36998#011validation-logloss:0.40920\u001b[0m\n", "\u001b[34m[419]#011train-logloss:0.36998#011validation-logloss:0.40923\u001b[0m\n", "\u001b[34m[420]#011train-logloss:0.36995#011validation-logloss:0.40928\u001b[0m\n", "\u001b[34m[421]#011train-logloss:0.36994#011validation-logloss:0.40929\u001b[0m\n", "\u001b[34m[422]#011train-logloss:0.36992#011validation-logloss:0.40929\u001b[0m\n", "\u001b[34m[423]#011train-logloss:0.36990#011validation-logloss:0.40927\u001b[0m\n", "\u001b[34m[424]#011train-logloss:0.36990#011validation-logloss:0.40927\u001b[0m\n", "\u001b[34m[425]#011train-logloss:0.36990#011validation-logloss:0.40928\u001b[0m\n", "\u001b[34m[426]#011train-logloss:0.36990#011validation-logloss:0.40926\u001b[0m\n", "\u001b[34m[427]#011train-logloss:0.36987#011validation-logloss:0.40929\u001b[0m\n", "\u001b[34m[428]#011train-logloss:0.36987#011validation-logloss:0.40927\u001b[0m\n", "\u001b[34m[429]#011train-logloss:0.36981#011validation-logloss:0.40929\u001b[0m\n", "\u001b[34m[430]#011train-logloss:0.36981#011validation-logloss:0.40925\u001b[0m\n", "\u001b[34m[431]#011train-logloss:0.36981#011validation-logloss:0.40923\u001b[0m\n", "\u001b[34m[432]#011train-logloss:0.36980#011validation-logloss:0.40921\u001b[0m\n", "\u001b[34m[433]#011train-logloss:0.36980#011validation-logloss:0.40923\u001b[0m\n", "\u001b[34m[434]#011train-logloss:0.36977#011validation-logloss:0.40928\u001b[0m\n", "\u001b[34m[435]#011train-logloss:0.36975#011validation-logloss:0.40928\u001b[0m\n", "\u001b[34m[436]#011train-logloss:0.36975#011validation-logloss:0.40927\u001b[0m\n", "\u001b[34m[437]#011train-logloss:0.36971#011validation-logloss:0.40926\u001b[0m\n", "\u001b[34m[438]#011train-logloss:0.36964#011validation-logloss:0.40927\u001b[0m\n", "\u001b[34m[439]#011train-logloss:0.36961#011validation-logloss:0.40930\u001b[0m\n", "\u001b[34m[440]#011train-logloss:0.36960#011validation-logloss:0.40929\u001b[0m\n", "\u001b[34m[441]#011train-logloss:0.36958#011validation-logloss:0.40929\u001b[0m\n", "\u001b[34m[442]#011train-logloss:0.36955#011validation-logloss:0.40936\u001b[0m\n", "\u001b[34m[443]#011train-logloss:0.36953#011validation-logloss:0.40937\u001b[0m\n", "\u001b[34m[444]#011train-logloss:0.36952#011validation-logloss:0.40936\u001b[0m\n", "\u001b[34m[445]#011train-logloss:0.36950#011validation-logloss:0.40934\u001b[0m\n", "\u001b[34m[446]#011train-logloss:0.36951#011validation-logloss:0.40936\u001b[0m\n", "\u001b[34m[447]#011train-logloss:0.36950#011validation-logloss:0.40936\u001b[0m\n", "\u001b[34m[448]#011train-logloss:0.36950#011validation-logloss:0.40936\u001b[0m\n", "\u001b[34m[449]#011train-logloss:0.36950#011validation-logloss:0.40938\u001b[0m\n", "\u001b[34m[450]#011train-logloss:0.36950#011validation-logloss:0.40937\u001b[0m\n", "\u001b[34m[451]#011train-logloss:0.36950#011validation-logloss:0.40937\u001b[0m\n", "\u001b[34m[452]#011train-logloss:0.36950#011validation-logloss:0.40935\u001b[0m\n", "\u001b[34m[453]#011train-logloss:0.36947#011validation-logloss:0.40939\u001b[0m\n", "\u001b[34m[454]#011train-logloss:0.36944#011validation-logloss:0.40936\u001b[0m\n", "\u001b[34m[455]#011train-logloss:0.36942#011validation-logloss:0.40934\u001b[0m\n", "\u001b[34m[456]#011train-logloss:0.36939#011validation-logloss:0.40937\u001b[0m\n", "\u001b[34m[457]#011train-logloss:0.36939#011validation-logloss:0.40940\u001b[0m\n", "\u001b[34m[458]#011train-logloss:0.36937#011validation-logloss:0.40937\u001b[0m\n", "\u001b[34m[459]#011train-logloss:0.36936#011validation-logloss:0.40935\u001b[0m\n", "\u001b[34m[460]#011train-logloss:0.36936#011validation-logloss:0.40938\u001b[0m\n", "\u001b[34m[461]#011train-logloss:0.36933#011validation-logloss:0.40939\u001b[0m\n", "\u001b[34m[462]#011train-logloss:0.36932#011validation-logloss:0.40938\u001b[0m\n", "\u001b[34m[463]#011train-logloss:0.36930#011validation-logloss:0.40938\u001b[0m\n", "\u001b[34m[464]#011train-logloss:0.36929#011validation-logloss:0.40938\u001b[0m\n", "\u001b[34m[465]#011train-logloss:0.36927#011validation-logloss:0.40939\u001b[0m\n", "\u001b[34m[466]#011train-logloss:0.36926#011validation-logloss:0.40943\u001b[0m\n", "\u001b[34m[467]#011train-logloss:0.36922#011validation-logloss:0.40944\u001b[0m\n", "\u001b[34m[468]#011train-logloss:0.36921#011validation-logloss:0.40940\u001b[0m\n", "\u001b[34m[469]#011train-logloss:0.36919#011validation-logloss:0.40940\u001b[0m\n", "\u001b[34m[470]#011train-logloss:0.36919#011validation-logloss:0.40942\u001b[0m\n", "\u001b[34m[471]#011train-logloss:0.36919#011validation-logloss:0.40940\u001b[0m\n", "\u001b[34m[472]#011train-logloss:0.36918#011validation-logloss:0.40938\u001b[0m\n", "\u001b[34m[473]#011train-logloss:0.36916#011validation-logloss:0.40936\u001b[0m\n", "\u001b[34m[474]#011train-logloss:0.36915#011validation-logloss:0.40938\u001b[0m\n", "\u001b[34m[475]#011train-logloss:0.36911#011validation-logloss:0.40939\u001b[0m\n", "\u001b[34m[476]#011train-logloss:0.36911#011validation-logloss:0.40939\u001b[0m\n", "\u001b[34m[477]#011train-logloss:0.36911#011validation-logloss:0.40938\u001b[0m\n", "\u001b[34m[478]#011train-logloss:0.36910#011validation-logloss:0.40939\u001b[0m\n", "\u001b[34m[479]#011train-logloss:0.36910#011validation-logloss:0.40942\u001b[0m\n", "\u001b[34m[480]#011train-logloss:0.36909#011validation-logloss:0.40946\u001b[0m\n", "\u001b[34m[481]#011train-logloss:0.36906#011validation-logloss:0.40947\u001b[0m\n", "\u001b[34m[482]#011train-logloss:0.36906#011validation-logloss:0.40949\u001b[0m\n", "\u001b[34m[483]#011train-logloss:0.36904#011validation-logloss:0.40952\u001b[0m\n", "\u001b[34m[484]#011train-logloss:0.36902#011validation-logloss:0.40951\u001b[0m\n", "\u001b[34m[485]#011train-logloss:0.36900#011validation-logloss:0.40951\u001b[0m\n", "\u001b[34m[486]#011train-logloss:0.36900#011validation-logloss:0.40950\u001b[0m\n", "\u001b[34m[487]#011train-logloss:0.36899#011validation-logloss:0.40951\u001b[0m\n", "\u001b[34m[488]#011train-logloss:0.36897#011validation-logloss:0.40949\u001b[0m\n", "\u001b[34m[489]#011train-logloss:0.36897#011validation-logloss:0.40947\u001b[0m\n", "\u001b[34m[490]#011train-logloss:0.36897#011validation-logloss:0.40946\u001b[0m\n", "\u001b[34m[491]#011train-logloss:0.36897#011validation-logloss:0.40946\u001b[0m\n", "\u001b[34m[492]#011train-logloss:0.36896#011validation-logloss:0.40943\u001b[0m\n", "\u001b[34m[493]#011train-logloss:0.36894#011validation-logloss:0.40944\u001b[0m\n", "\u001b[34m[494]#011train-logloss:0.36894#011validation-logloss:0.40943\u001b[0m\n", "\u001b[34m[495]#011train-logloss:0.36892#011validation-logloss:0.40943\u001b[0m\n", "\u001b[34m[496]#011train-logloss:0.36891#011validation-logloss:0.40943\u001b[0m\n", "\u001b[34m[497]#011train-logloss:0.36891#011validation-logloss:0.40941\u001b[0m\n", "\u001b[34m[498]#011train-logloss:0.36891#011validation-logloss:0.40941\u001b[0m\n", "\u001b[34m[499]#011train-logloss:0.36890#011validation-logloss:0.40943\u001b[0m\n", "\u001b[34m[500]#011train-logloss:0.36887#011validation-logloss:0.40948\u001b[0m\n", "\u001b[34m[501]#011train-logloss:0.36887#011validation-logloss:0.40951\u001b[0m\n", "\u001b[34m[502]#011train-logloss:0.36887#011validation-logloss:0.40952\u001b[0m\n", "\u001b[34m[503]#011train-logloss:0.36885#011validation-logloss:0.40956\u001b[0m\n", "\u001b[34m[504]#011train-logloss:0.36884#011validation-logloss:0.40955\u001b[0m\n", "\u001b[34m[505]#011train-logloss:0.36884#011validation-logloss:0.40952\u001b[0m\n", "\u001b[34m[506]#011train-logloss:0.36881#011validation-logloss:0.40955\u001b[0m\n", "\u001b[34m[507]#011train-logloss:0.36878#011validation-logloss:0.40956\u001b[0m\n", "\u001b[34m[508]#011train-logloss:0.36875#011validation-logloss:0.40957\u001b[0m\n", "\u001b[34m[509]#011train-logloss:0.36876#011validation-logloss:0.40956\u001b[0m\n", "\u001b[34m[510]#011train-logloss:0.36875#011validation-logloss:0.40952\u001b[0m\n", "\u001b[34m[511]#011train-logloss:0.36871#011validation-logloss:0.40954\u001b[0m\n", "\u001b[34m[512]#011train-logloss:0.36871#011validation-logloss:0.40955\u001b[0m\n", "\u001b[34m[513]#011train-logloss:0.36868#011validation-logloss:0.40959\u001b[0m\n", "\u001b[34m[514]#011train-logloss:0.36868#011validation-logloss:0.40958\u001b[0m\n", "\u001b[34m[515]#011train-logloss:0.36867#011validation-logloss:0.40956\u001b[0m\n", "\u001b[34m[516]#011train-logloss:0.36867#011validation-logloss:0.40959\u001b[0m\n", "\u001b[34m[517]#011train-logloss:0.36864#011validation-logloss:0.40960\u001b[0m\n", "\u001b[34m[518]#011train-logloss:0.36862#011validation-logloss:0.40959\u001b[0m\n", "\u001b[34m[519]#011train-logloss:0.36862#011validation-logloss:0.40957\u001b[0m\n", "\u001b[34m[520]#011train-logloss:0.36862#011validation-logloss:0.40957\u001b[0m\n", "\u001b[34m[521]#011train-logloss:0.36862#011validation-logloss:0.40959\u001b[0m\n", "\u001b[34m[522]#011train-logloss:0.36861#011validation-logloss:0.40957\u001b[0m\n", "\u001b[34m[523]#011train-logloss:0.36860#011validation-logloss:0.40958\u001b[0m\n", "\u001b[34m[524]#011train-logloss:0.36860#011validation-logloss:0.40958\u001b[0m\n", "\u001b[34m[525]#011train-logloss:0.36858#011validation-logloss:0.40956\u001b[0m\n", "\u001b[34m[526]#011train-logloss:0.36858#011validation-logloss:0.40953\u001b[0m\n", "\u001b[34m[527]#011train-logloss:0.36857#011validation-logloss:0.40953\u001b[0m\n", "\u001b[34m[528]#011train-logloss:0.36856#011validation-logloss:0.40953\u001b[0m\n", "\u001b[34m[529]#011train-logloss:0.36856#011validation-logloss:0.40956\u001b[0m\n", "\u001b[34m[530]#011train-logloss:0.36855#011validation-logloss:0.40955\u001b[0m\n", "\u001b[34m[531]#011train-logloss:0.36853#011validation-logloss:0.40953\u001b[0m\n", "\u001b[34m[532]#011train-logloss:0.36851#011validation-logloss:0.40955\u001b[0m\n", "\u001b[34m[533]#011train-logloss:0.36850#011validation-logloss:0.40954\u001b[0m\n", "\u001b[34m[534]#011train-logloss:0.36850#011validation-logloss:0.40953\u001b[0m\n", "\u001b[34m[535]#011train-logloss:0.36842#011validation-logloss:0.40958\u001b[0m\n", "\u001b[34m[536]#011train-logloss:0.36841#011validation-logloss:0.40960\u001b[0m\n", "\u001b[34m[537]#011train-logloss:0.36841#011validation-logloss:0.40960\u001b[0m\n", "\u001b[34m[538]#011train-logloss:0.36838#011validation-logloss:0.40964\u001b[0m\n", "\u001b[34m[539]#011train-logloss:0.36838#011validation-logloss:0.40963\u001b[0m\n", "\u001b[34m[540]#011train-logloss:0.36836#011validation-logloss:0.40964\u001b[0m\n", "\u001b[34m[541]#011train-logloss:0.36834#011validation-logloss:0.40961\u001b[0m\n", "\u001b[34m[542]#011train-logloss:0.36834#011validation-logloss:0.40960\u001b[0m\n", "\u001b[34m[543]#011train-logloss:0.36830#011validation-logloss:0.40961\u001b[0m\n", "\u001b[34m[544]#011train-logloss:0.36828#011validation-logloss:0.40963\u001b[0m\n", "\u001b[34m[545]#011train-logloss:0.36827#011validation-logloss:0.40965\u001b[0m\n", "\u001b[34m[546]#011train-logloss:0.36826#011validation-logloss:0.40963\u001b[0m\n", "\u001b[34m[547]#011train-logloss:0.36826#011validation-logloss:0.40967\u001b[0m\n", "\u001b[34m[548]#011train-logloss:0.36822#011validation-logloss:0.40971\u001b[0m\n", "\u001b[34m[549]#011train-logloss:0.36816#011validation-logloss:0.40971\u001b[0m\n", "\u001b[34m[550]#011train-logloss:0.36816#011validation-logloss:0.40968\u001b[0m\n", "\u001b[34m[551]#011train-logloss:0.36814#011validation-logloss:0.40969\u001b[0m\n", "\u001b[34m[552]#011train-logloss:0.36811#011validation-logloss:0.40972\u001b[0m\n", "\u001b[34m[553]#011train-logloss:0.36811#011validation-logloss:0.40971\u001b[0m\n", "\u001b[34m[554]#011train-logloss:0.36810#011validation-logloss:0.40971\u001b[0m\n", "\u001b[34m[555]#011train-logloss:0.36808#011validation-logloss:0.40970\u001b[0m\n", "\u001b[34m[556]#011train-logloss:0.36808#011validation-logloss:0.40970\u001b[0m\n", "\u001b[34m[557]#011train-logloss:0.36808#011validation-logloss:0.40971\u001b[0m\n", "\u001b[34m[558]#011train-logloss:0.36807#011validation-logloss:0.40967\u001b[0m\n", "\u001b[34m[559]#011train-logloss:0.36807#011validation-logloss:0.40968\u001b[0m\n", "\u001b[34m[560]#011train-logloss:0.36806#011validation-logloss:0.40971\u001b[0m\n", "\u001b[34m[561]#011train-logloss:0.36806#011validation-logloss:0.40971\u001b[0m\n", "\u001b[34m[562]#011train-logloss:0.36804#011validation-logloss:0.40968\u001b[0m\n", "\u001b[34m[563]#011train-logloss:0.36804#011validation-logloss:0.40973\u001b[0m\n", "\u001b[34m[564]#011train-logloss:0.36802#011validation-logloss:0.40970\u001b[0m\n", "\u001b[34m[565]#011train-logloss:0.36802#011validation-logloss:0.40972\u001b[0m\n", "\u001b[34m[566]#011train-logloss:0.36802#011validation-logloss:0.40971\u001b[0m\n", "\u001b[34m[567]#011train-logloss:0.36800#011validation-logloss:0.40971\u001b[0m\n", "\u001b[34m[568]#011train-logloss:0.36800#011validation-logloss:0.40972\u001b[0m\n", "\u001b[34m[569]#011train-logloss:0.36799#011validation-logloss:0.40972\u001b[0m\n", "\u001b[34m[570]#011train-logloss:0.36794#011validation-logloss:0.40974\u001b[0m\n", "\u001b[34m[571]#011train-logloss:0.36794#011validation-logloss:0.40974\u001b[0m\n", "\u001b[34m[572]#011train-logloss:0.36793#011validation-logloss:0.40975\u001b[0m\n", "\u001b[34m[573]#011train-logloss:0.36793#011validation-logloss:0.40978\u001b[0m\n", "\u001b[34m[574]#011train-logloss:0.36793#011validation-logloss:0.40980\u001b[0m\n", "\u001b[34m[575]#011train-logloss:0.36794#011validation-logloss:0.40979\u001b[0m\n", "\u001b[34m[576]#011train-logloss:0.36793#011validation-logloss:0.40977\u001b[0m\n", "\u001b[34m[577]#011train-logloss:0.36792#011validation-logloss:0.40979\u001b[0m\n", "\u001b[34m[578]#011train-logloss:0.36790#011validation-logloss:0.40978\u001b[0m\n", "\u001b[34m[579]#011train-logloss:0.36790#011validation-logloss:0.40979\u001b[0m\n", "\u001b[34m[580]#011train-logloss:0.36788#011validation-logloss:0.40978\u001b[0m\n", "\u001b[34m[581]#011train-logloss:0.36788#011validation-logloss:0.40976\u001b[0m\n", "\u001b[34m[582]#011train-logloss:0.36788#011validation-logloss:0.40978\u001b[0m\n", "\u001b[34m[583]#011train-logloss:0.36787#011validation-logloss:0.40975\u001b[0m\n", "\u001b[34m[584]#011train-logloss:0.36786#011validation-logloss:0.40977\u001b[0m\n", "\u001b[34m[585]#011train-logloss:0.36785#011validation-logloss:0.40975\u001b[0m\n", "\u001b[34m[586]#011train-logloss:0.36784#011validation-logloss:0.40979\u001b[0m\n", "\u001b[34m[587]#011train-logloss:0.36784#011validation-logloss:0.40978\u001b[0m\n", "\u001b[34m[588]#011train-logloss:0.36782#011validation-logloss:0.40983\u001b[0m\n", "\u001b[34m[589]#011train-logloss:0.36782#011validation-logloss:0.40982\u001b[0m\n", "\u001b[34m[590]#011train-logloss:0.36782#011validation-logloss:0.40981\u001b[0m\n", "\u001b[34m[591]#011train-logloss:0.36780#011validation-logloss:0.40981\u001b[0m\n", "\u001b[34m[592]#011train-logloss:0.36780#011validation-logloss:0.40979\u001b[0m\n", "\u001b[34m[593]#011train-logloss:0.36779#011validation-logloss:0.40977\u001b[0m\n", "\u001b[34m[594]#011train-logloss:0.36779#011validation-logloss:0.40975\u001b[0m\n", "\u001b[34m[595]#011train-logloss:0.36777#011validation-logloss:0.40975\u001b[0m\n", "\u001b[34m[596]#011train-logloss:0.36774#011validation-logloss:0.40976\u001b[0m\n", "\u001b[34m[597]#011train-logloss:0.36773#011validation-logloss:0.40979\u001b[0m\n", "\u001b[34m[598]#011train-logloss:0.36772#011validation-logloss:0.40978\u001b[0m\n", "\u001b[34m[599]#011train-logloss:0.36772#011validation-logloss:0.40978\u001b[0m\n", "\u001b[34m[600]#011train-logloss:0.36769#011validation-logloss:0.40979\u001b[0m\n", "\u001b[34m[601]#011train-logloss:0.36768#011validation-logloss:0.40976\u001b[0m\n", "\u001b[34m[602]#011train-logloss:0.36766#011validation-logloss:0.40980\u001b[0m\n", "\u001b[34m[603]#011train-logloss:0.36763#011validation-logloss:0.40982\u001b[0m\n", "\u001b[34m[604]#011train-logloss:0.36761#011validation-logloss:0.40984\u001b[0m\n", "\u001b[34m[605]#011train-logloss:0.36760#011validation-logloss:0.40983\u001b[0m\n", "\u001b[34m[606]#011train-logloss:0.36758#011validation-logloss:0.40985\u001b[0m\n", "\u001b[34m[607]#011train-logloss:0.36758#011validation-logloss:0.40985\u001b[0m\n", "\u001b[34m[608]#011train-logloss:0.36758#011validation-logloss:0.40985\u001b[0m\n", "\u001b[34m[609]#011train-logloss:0.36757#011validation-logloss:0.40987\u001b[0m\n", "\u001b[34m[610]#011train-logloss:0.36755#011validation-logloss:0.40987\u001b[0m\n", "\u001b[34m[611]#011train-logloss:0.36754#011validation-logloss:0.40991\u001b[0m\n", "\u001b[34m[612]#011train-logloss:0.36748#011validation-logloss:0.40989\u001b[0m\n", "\u001b[34m[613]#011train-logloss:0.36747#011validation-logloss:0.40988\u001b[0m\n", "\u001b[34m[614]#011train-logloss:0.36746#011validation-logloss:0.40989\u001b[0m\n", "\u001b[34m[615]#011train-logloss:0.36744#011validation-logloss:0.40992\u001b[0m\n", "\u001b[34m[616]#011train-logloss:0.36743#011validation-logloss:0.40993\u001b[0m\n", "\u001b[34m[617]#011train-logloss:0.36743#011validation-logloss:0.40991\u001b[0m\n", "\u001b[34m[618]#011train-logloss:0.36742#011validation-logloss:0.40993\u001b[0m\n", "\u001b[34m[619]#011train-logloss:0.36742#011validation-logloss:0.40994\u001b[0m\n", "\u001b[34m[620]#011train-logloss:0.36742#011validation-logloss:0.40993\u001b[0m\n", "\u001b[34m[621]#011train-logloss:0.36741#011validation-logloss:0.40994\u001b[0m\n", "\u001b[34m[622]#011train-logloss:0.36741#011validation-logloss:0.40993\u001b[0m\n", "\u001b[34m[623]#011train-logloss:0.36741#011validation-logloss:0.40993\u001b[0m\n", "\u001b[34m[624]#011train-logloss:0.36739#011validation-logloss:0.40991\u001b[0m\n", "\u001b[34m[625]#011train-logloss:0.36739#011validation-logloss:0.40990\u001b[0m\n", "\u001b[34m[626]#011train-logloss:0.36739#011validation-logloss:0.40989\u001b[0m\n", "\u001b[34m[627]#011train-logloss:0.36738#011validation-logloss:0.40993\u001b[0m\n", "\u001b[34m[628]#011train-logloss:0.36733#011validation-logloss:0.40994\u001b[0m\n", "\u001b[34m[629]#011train-logloss:0.36733#011validation-logloss:0.40996\u001b[0m\n", "\u001b[34m[630]#011train-logloss:0.36732#011validation-logloss:0.40998\u001b[0m\n", "\u001b[34m[631]#011train-logloss:0.36732#011validation-logloss:0.40999\u001b[0m\n", "\u001b[34m[632]#011train-logloss:0.36732#011validation-logloss:0.40998\u001b[0m\n", "\u001b[34m[633]#011train-logloss:0.36731#011validation-logloss:0.40999\u001b[0m\n", "\u001b[34m[634]#011train-logloss:0.36730#011validation-logloss:0.40997\u001b[0m\n", "\u001b[34m[635]#011train-logloss:0.36730#011validation-logloss:0.40998\u001b[0m\n", "\u001b[34m[636]#011train-logloss:0.36727#011validation-logloss:0.40999\u001b[0m\n", "\u001b[34m[637]#011train-logloss:0.36725#011validation-logloss:0.41002\u001b[0m\n", "\u001b[34m[638]#011train-logloss:0.36724#011validation-logloss:0.41000\u001b[0m\n", "\u001b[34m[639]#011train-logloss:0.36724#011validation-logloss:0.40999\u001b[0m\n", "\u001b[34m[640]#011train-logloss:0.36724#011validation-logloss:0.41002\u001b[0m\n", "\u001b[34m[641]#011train-logloss:0.36724#011validation-logloss:0.41001\u001b[0m\n", "\u001b[34m[642]#011train-logloss:0.36724#011validation-logloss:0.40999\u001b[0m\n", "\u001b[34m[643]#011train-logloss:0.36724#011validation-logloss:0.41000\u001b[0m\n", "\u001b[34m[644]#011train-logloss:0.36723#011validation-logloss:0.40996\u001b[0m\n", "\u001b[34m[645]#011train-logloss:0.36723#011validation-logloss:0.40993\u001b[0m\n", "\u001b[34m[646]#011train-logloss:0.36719#011validation-logloss:0.40994\u001b[0m\n", "\u001b[34m[647]#011train-logloss:0.36717#011validation-logloss:0.40994\u001b[0m\n", "\u001b[34m[648]#011train-logloss:0.36717#011validation-logloss:0.40993\u001b[0m\n", "\u001b[34m[649]#011train-logloss:0.36715#011validation-logloss:0.40996\u001b[0m\n", "\u001b[34m[650]#011train-logloss:0.36714#011validation-logloss:0.40995\u001b[0m\n", "\u001b[34m[651]#011train-logloss:0.36711#011validation-logloss:0.40997\u001b[0m\n", "\u001b[34m[652]#011train-logloss:0.36709#011validation-logloss:0.40999\u001b[0m\n", "\u001b[34m[653]#011train-logloss:0.36707#011validation-logloss:0.41001\u001b[0m\n", "\u001b[34m[654]#011train-logloss:0.36705#011validation-logloss:0.41004\u001b[0m\n", "\u001b[34m[655]#011train-logloss:0.36705#011validation-logloss:0.41005\u001b[0m\n", "\u001b[34m[656]#011train-logloss:0.36705#011validation-logloss:0.41007\u001b[0m\n", "\u001b[34m[657]#011train-logloss:0.36705#011validation-logloss:0.41009\u001b[0m\n", "\u001b[34m[658]#011train-logloss:0.36700#011validation-logloss:0.41012\u001b[0m\n", "\u001b[34m[659]#011train-logloss:0.36700#011validation-logloss:0.41011\u001b[0m\n", "\u001b[34m[660]#011train-logloss:0.36700#011validation-logloss:0.41011\u001b[0m\n", "\u001b[34m[661]#011train-logloss:0.36698#011validation-logloss:0.41009\u001b[0m\n", "\u001b[34m[662]#011train-logloss:0.36698#011validation-logloss:0.41009\u001b[0m\n", "\u001b[34m[663]#011train-logloss:0.36692#011validation-logloss:0.41013\u001b[0m\n", "\u001b[34m[664]#011train-logloss:0.36686#011validation-logloss:0.41014\u001b[0m\n", "\u001b[34m[665]#011train-logloss:0.36686#011validation-logloss:0.41014\u001b[0m\n", "\u001b[34m[666]#011train-logloss:0.36685#011validation-logloss:0.41013\u001b[0m\n", "\u001b[34m[667]#011train-logloss:0.36685#011validation-logloss:0.41013\u001b[0m\n", "\u001b[34m[668]#011train-logloss:0.36684#011validation-logloss:0.41014\u001b[0m\n", "\u001b[34m[669]#011train-logloss:0.36684#011validation-logloss:0.41014\u001b[0m\n", "\u001b[34m[670]#011train-logloss:0.36679#011validation-logloss:0.41018\u001b[0m\n", "\u001b[34m[671]#011train-logloss:0.36674#011validation-logloss:0.41017\u001b[0m\n", "\u001b[34m[672]#011train-logloss:0.36670#011validation-logloss:0.41019\u001b[0m\n", "\u001b[34m[673]#011train-logloss:0.36669#011validation-logloss:0.41019\u001b[0m\n", "\u001b[34m[674]#011train-logloss:0.36669#011validation-logloss:0.41019\u001b[0m\n", "\u001b[34m[675]#011train-logloss:0.36668#011validation-logloss:0.41017\u001b[0m\n", "\u001b[34m[676]#011train-logloss:0.36668#011validation-logloss:0.41017\u001b[0m\n", "\u001b[34m[677]#011train-logloss:0.36667#011validation-logloss:0.41016\u001b[0m\n", "\u001b[34m[678]#011train-logloss:0.36666#011validation-logloss:0.41015\u001b[0m\n", "\u001b[34m[679]#011train-logloss:0.36664#011validation-logloss:0.41020\u001b[0m\n", "\u001b[34m[680]#011train-logloss:0.36664#011validation-logloss:0.41021\u001b[0m\n", "\u001b[34m[681]#011train-logloss:0.36664#011validation-logloss:0.41021\u001b[0m\n", "\u001b[34m[682]#011train-logloss:0.36664#011validation-logloss:0.41021\u001b[0m\n", "\u001b[34m[683]#011train-logloss:0.36664#011validation-logloss:0.41019\u001b[0m\n", "\u001b[34m[684]#011train-logloss:0.36664#011validation-logloss:0.41018\u001b[0m\n", "\u001b[34m[685]#011train-logloss:0.36661#011validation-logloss:0.41018\u001b[0m\n", "\u001b[34m[686]#011train-logloss:0.36661#011validation-logloss:0.41019\u001b[0m\n", "\u001b[34m[687]#011train-logloss:0.36661#011validation-logloss:0.41021\u001b[0m\n", "\u001b[34m[688]#011train-logloss:0.36661#011validation-logloss:0.41021\u001b[0m\n", "\u001b[34m[689]#011train-logloss:0.36661#011validation-logloss:0.41019\u001b[0m\n", "\u001b[34m[690]#011train-logloss:0.36658#011validation-logloss:0.41018\u001b[0m\n", "\u001b[34m[691]#011train-logloss:0.36657#011validation-logloss:0.41016\u001b[0m\n", "\u001b[34m[692]#011train-logloss:0.36656#011validation-logloss:0.41019\u001b[0m\n", "\u001b[34m[693]#011train-logloss:0.36656#011validation-logloss:0.41017\u001b[0m\n", "\u001b[34m[694]#011train-logloss:0.36655#011validation-logloss:0.41019\u001b[0m\n", "\u001b[34m[695]#011train-logloss:0.36655#011validation-logloss:0.41017\u001b[0m\n", "\u001b[34m[696]#011train-logloss:0.36655#011validation-logloss:0.41019\u001b[0m\n", "\u001b[34m[697]#011train-logloss:0.36653#011validation-logloss:0.41019\u001b[0m\n", "\u001b[34m[698]#011train-logloss:0.36652#011validation-logloss:0.41021\u001b[0m\n", "\u001b[34m[699]#011train-logloss:0.36652#011validation-logloss:0.41022\u001b[0m\n", "\u001b[34m[700]#011train-logloss:0.36651#011validation-logloss:0.41025\u001b[0m\n", "\u001b[34m[701]#011train-logloss:0.36648#011validation-logloss:0.41024\u001b[0m\n", "\u001b[34m[702]#011train-logloss:0.36648#011validation-logloss:0.41024\u001b[0m\n", "\u001b[34m[703]#011train-logloss:0.36646#011validation-logloss:0.41024\u001b[0m\n", "\u001b[34m[704]#011train-logloss:0.36646#011validation-logloss:0.41025\u001b[0m\n", "\u001b[34m[705]#011train-logloss:0.36645#011validation-logloss:0.41026\u001b[0m\n", "\u001b[34m[706]#011train-logloss:0.36644#011validation-logloss:0.41025\u001b[0m\n", "\u001b[34m[707]#011train-logloss:0.36642#011validation-logloss:0.41023\u001b[0m\n", "\u001b[34m[708]#011train-logloss:0.36642#011validation-logloss:0.41025\u001b[0m\n", "\u001b[34m[709]#011train-logloss:0.36642#011validation-logloss:0.41027\u001b[0m\n", "\u001b[34m[710]#011train-logloss:0.36640#011validation-logloss:0.41026\u001b[0m\n", "\u001b[34m[711]#011train-logloss:0.36639#011validation-logloss:0.41023\u001b[0m\n", "\u001b[34m[712]#011train-logloss:0.36639#011validation-logloss:0.41019\u001b[0m\n", "\u001b[34m[713]#011train-logloss:0.36638#011validation-logloss:0.41019\u001b[0m\n", "\u001b[34m[714]#011train-logloss:0.36637#011validation-logloss:0.41019\u001b[0m\n", "\u001b[34m[715]#011train-logloss:0.36637#011validation-logloss:0.41021\u001b[0m\n", "\u001b[34m[716]#011train-logloss:0.36636#011validation-logloss:0.41020\u001b[0m\n", "\u001b[34m[717]#011train-logloss:0.36636#011validation-logloss:0.41021\u001b[0m\n", "\u001b[34m[718]#011train-logloss:0.36629#011validation-logloss:0.41026\u001b[0m\n", "\u001b[34m[719]#011train-logloss:0.36628#011validation-logloss:0.41027\u001b[0m\n", "\u001b[34m[720]#011train-logloss:0.36627#011validation-logloss:0.41025\u001b[0m\n", "\u001b[34m[721]#011train-logloss:0.36627#011validation-logloss:0.41024\u001b[0m\n", "\u001b[34m[722]#011train-logloss:0.36626#011validation-logloss:0.41024\u001b[0m\n", "\u001b[34m[723]#011train-logloss:0.36624#011validation-logloss:0.41027\u001b[0m\n", "\u001b[34m[724]#011train-logloss:0.36624#011validation-logloss:0.41026\u001b[0m\n", "\u001b[34m[725]#011train-logloss:0.36623#011validation-logloss:0.41025\u001b[0m\n", "\u001b[34m[726]#011train-logloss:0.36622#011validation-logloss:0.41026\u001b[0m\n", "\u001b[34m[727]#011train-logloss:0.36622#011validation-logloss:0.41026\u001b[0m\n", "\u001b[34m[728]#011train-logloss:0.36622#011validation-logloss:0.41028\u001b[0m\n", "\u001b[34m[729]#011train-logloss:0.36621#011validation-logloss:0.41030\u001b[0m\n", "\u001b[34m[730]#011train-logloss:0.36619#011validation-logloss:0.41033\u001b[0m\n", "\u001b[34m[731]#011train-logloss:0.36616#011validation-logloss:0.41035\u001b[0m\n", "\u001b[34m[732]#011train-logloss:0.36617#011validation-logloss:0.41036\u001b[0m\n", "\u001b[34m[733]#011train-logloss:0.36615#011validation-logloss:0.41034\u001b[0m\n", "\u001b[34m[734]#011train-logloss:0.36615#011validation-logloss:0.41032\u001b[0m\n", "\u001b[34m[735]#011train-logloss:0.36614#011validation-logloss:0.41033\u001b[0m\n", "\u001b[34m[736]#011train-logloss:0.36614#011validation-logloss:0.41035\u001b[0m\n", "\u001b[34m[737]#011train-logloss:0.36613#011validation-logloss:0.41033\u001b[0m\n", "\u001b[34m[738]#011train-logloss:0.36613#011validation-logloss:0.41031\u001b[0m\n", "\u001b[34m[739]#011train-logloss:0.36613#011validation-logloss:0.41031\u001b[0m\n", "\u001b[34m[740]#011train-logloss:0.36611#011validation-logloss:0.41034\u001b[0m\n", "\u001b[34m[741]#011train-logloss:0.36611#011validation-logloss:0.41034\u001b[0m\n", "\u001b[34m[742]#011train-logloss:0.36610#011validation-logloss:0.41031\u001b[0m\n", "\u001b[34m[743]#011train-logloss:0.36608#011validation-logloss:0.41029\u001b[0m\n", "\u001b[34m[744]#011train-logloss:0.36607#011validation-logloss:0.41033\u001b[0m\n", "\u001b[34m[745]#011train-logloss:0.36607#011validation-logloss:0.41032\u001b[0m\n", "\u001b[34m[746]#011train-logloss:0.36607#011validation-logloss:0.41029\u001b[0m\n", "\u001b[34m[747]#011train-logloss:0.36606#011validation-logloss:0.41034\u001b[0m\n", "\u001b[34m[748]#011train-logloss:0.36605#011validation-logloss:0.41033\u001b[0m\n", "\u001b[34m[749]#011train-logloss:0.36605#011validation-logloss:0.41033\u001b[0m\n", "\u001b[34m[750]#011train-logloss:0.36605#011validation-logloss:0.41036\u001b[0m\n", "\u001b[34m[751]#011train-logloss:0.36604#011validation-logloss:0.41037\u001b[0m\n", "\u001b[34m[752]#011train-logloss:0.36604#011validation-logloss:0.41036\u001b[0m\n", "\u001b[34m[753]#011train-logloss:0.36604#011validation-logloss:0.41035\u001b[0m\n", "\u001b[34m[754]#011train-logloss:0.36603#011validation-logloss:0.41031\u001b[0m\n", "\u001b[34m[755]#011train-logloss:0.36602#011validation-logloss:0.41036\u001b[0m\n", "\u001b[34m[756]#011train-logloss:0.36601#011validation-logloss:0.41036\u001b[0m\n", "\u001b[34m[757]#011train-logloss:0.36601#011validation-logloss:0.41036\u001b[0m\n", "\u001b[34m[758]#011train-logloss:0.36601#011validation-logloss:0.41033\u001b[0m\n", "\u001b[34m[759]#011train-logloss:0.36601#011validation-logloss:0.41032\u001b[0m\n", "\u001b[34m[760]#011train-logloss:0.36601#011validation-logloss:0.41034\u001b[0m\n", "\u001b[34m[761]#011train-logloss:0.36601#011validation-logloss:0.41033\u001b[0m\n", "\u001b[34m[762]#011train-logloss:0.36600#011validation-logloss:0.41033\u001b[0m\n", "\u001b[34m[763]#011train-logloss:0.36595#011validation-logloss:0.41035\u001b[0m\n", "\u001b[34m[764]#011train-logloss:0.36595#011validation-logloss:0.41035\u001b[0m\n", "\u001b[34m[765]#011train-logloss:0.36595#011validation-logloss:0.41036\u001b[0m\n", "\u001b[34m[766]#011train-logloss:0.36592#011validation-logloss:0.41035\u001b[0m\n", "\u001b[34m[767]#011train-logloss:0.36591#011validation-logloss:0.41038\u001b[0m\n", "\u001b[34m[768]#011train-logloss:0.36591#011validation-logloss:0.41037\u001b[0m\n", "\u001b[34m[769]#011train-logloss:0.36590#011validation-logloss:0.41038\u001b[0m\n", "\u001b[34m[770]#011train-logloss:0.36590#011validation-logloss:0.41042\u001b[0m\n", "\u001b[34m[771]#011train-logloss:0.36590#011validation-logloss:0.41038\u001b[0m\n", "\u001b[34m[772]#011train-logloss:0.36590#011validation-logloss:0.41041\u001b[0m\n", "\u001b[34m[773]#011train-logloss:0.36590#011validation-logloss:0.41038\u001b[0m\n", "\u001b[34m[774]#011train-logloss:0.36590#011validation-logloss:0.41036\u001b[0m\n", "\u001b[34m[775]#011train-logloss:0.36589#011validation-logloss:0.41035\u001b[0m\n", "\u001b[34m[776]#011train-logloss:0.36589#011validation-logloss:0.41038\u001b[0m\n", "\u001b[34m[777]#011train-logloss:0.36588#011validation-logloss:0.41038\u001b[0m\n", "\u001b[34m[778]#011train-logloss:0.36588#011validation-logloss:0.41036\u001b[0m\n", "\u001b[34m[779]#011train-logloss:0.36586#011validation-logloss:0.41038\u001b[0m\n", "\u001b[34m[780]#011train-logloss:0.36585#011validation-logloss:0.41037\u001b[0m\n", "\u001b[34m[781]#011train-logloss:0.36584#011validation-logloss:0.41034\u001b[0m\n", "\u001b[34m[782]#011train-logloss:0.36582#011validation-logloss:0.41034\u001b[0m\n", "\u001b[34m[783]#011train-logloss:0.36582#011validation-logloss:0.41032\u001b[0m\n", "\n", "2023-01-04 12:17:41 Uploading - Uploading generated training model\u001b[34m[784]#011train-logloss:0.36582#011validation-logloss:0.41031\u001b[0m\n", "\u001b[34m[785]#011train-logloss:0.36582#011validation-logloss:0.41031\u001b[0m\n", "\u001b[34m[786]#011train-logloss:0.36581#011validation-logloss:0.41033\u001b[0m\n", "\u001b[34m[787]#011train-logloss:0.36581#011validation-logloss:0.41038\u001b[0m\n", "\u001b[34m[788]#011train-logloss:0.36581#011validation-logloss:0.41040\u001b[0m\n", "\u001b[34m[789]#011train-logloss:0.36581#011validation-logloss:0.41039\u001b[0m\n", "\u001b[34m[790]#011train-logloss:0.36580#011validation-logloss:0.41036\u001b[0m\n", "\u001b[34m[791]#011train-logloss:0.36580#011validation-logloss:0.41037\u001b[0m\n", "\u001b[34m[792]#011train-logloss:0.36579#011validation-logloss:0.41038\u001b[0m\n", "\u001b[34m[793]#011train-logloss:0.36579#011validation-logloss:0.41039\u001b[0m\n", "\u001b[34m[794]#011train-logloss:0.36577#011validation-logloss:0.41037\u001b[0m\n", "\u001b[34m[795]#011train-logloss:0.36576#011validation-logloss:0.41040\u001b[0m\n", "\u001b[34m[796]#011train-logloss:0.36574#011validation-logloss:0.41046\u001b[0m\n", "\u001b[34m[797]#011train-logloss:0.36573#011validation-logloss:0.41046\u001b[0m\n", "\u001b[34m[798]#011train-logloss:0.36573#011validation-logloss:0.41046\u001b[0m\n", "\u001b[34m[799]#011train-logloss:0.36573#011validation-logloss:0.41048\u001b[0m\n", "\n", "2023-01-04 12:17:52 Completed - Training job completed\n", "Training seconds: 92\n", "Billable seconds: 92\n" ] } ], "source": [ "container = retrieve(\"xgboost\", region, version=\"1.5-1\")\n", "xgb = Estimator(\n", " container,\n", " role,\n", " instance_count=1,\n", " instance_type=training_instance_type,\n", " disable_profiler=True,\n", " sagemaker_session=sagemaker_session,\n", ")\n", "\n", "xgb.set_hyperparameters(\n", " max_depth=5,\n", " eta=0.2,\n", " gamma=4,\n", " min_child_weight=6,\n", " subsample=0.8,\n", " objective=\"binary:logistic\",\n", " num_round=800,\n", ")\n", "\n", "xgb.fit({'train': train_input, 'validation': validation_input})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 推論データ準備\n", "推論用データとして学習データと同様の加工を行い、説明変数のみからなる推論用データと正解データのデータを作成します。" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The test dataset contains 114158 examples and 14 columns.\n", "\n" ] } ], "source": [ "# test_data = data_processing('cps_2021.csv')\n", "test_data = data_processing('cps_2022.csv')\n", "\n", "num_examples, num_columns = test_data.shape\n", "print(\n", " f\"The test dataset contains {num_examples} examples and {num_columns} columns.\\n\"\n", ")\n", "\n", "ground_truth_label, features = test_data.iloc[:, :1], test_data.iloc[:, 1:]\n", "\n", "features.to_csv('testdata.csv', header=False, index=False)\n", "\n", "dataset_type = \"text/csv\"\n", "test_dataset = 'testdata.csv'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 推論エンドポイント作成\n", "作成したモデルを用いてエンドポイントをデプロイします。" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model file is stored in s3://sagemaker-ap-northeast-1-036661559124/sagemaker-xgboost-2023-01-04-12-14-14-700/output/model.tar.gz\n" ] } ], "source": [ "model_url = xgb.model_data\n", "print(f\"Model file is stored in {model_url}\")" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model name: xgb-2023-01-04-1218\n", "Endpoint name: bb-example-xgboost-classification-model-2023-01-04-12-11-59-829\n" ] } ], "source": [ "model_name = f'xgb-{datetime.now():%Y-%m-%d-%H%M}'\n", "print(\"Model name: \", model_name)\n", "print(\"Endpoint name: \", endpoint_name)" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "XGBoost image uri: 354813040037.dkr.ecr.ap-northeast-1.amazonaws.com/sagemaker-xgboost:1.5-1\n", "Deploying model xgb-2023-01-04-1218 to endpoint bb-example-xgboost-classification-model-2023-01-04-12-11-59-829\n", "-----!" ] } ], "source": [ "image_uri = image_uris.retrieve(\"xgboost\", region, \"1.5-1\")\n", "\n", "print(f\"XGBoost image uri: {image_uri}\")\n", "model = Model(\n", " role=role,\n", " name=model_name,\n", " image_uri=image_uri,\n", " model_data=model_url,\n", " sagemaker_session=sagemaker_session,\n", ")\n", "\n", "data_capture_config = DataCaptureConfig(\n", " enable_capture=True,\n", " sampling_percentage=100,\n", " destination_s3_uri=s3_capture_upload_path,\n", ")\n", "\n", "print(f\"Deploying model {model_name} to endpoint {endpoint_name}\")\n", "model.deploy(\n", " initial_instance_count=endpoint_instance_count,\n", " instance_type=endpoint_instance_type,\n", " endpoint_name=endpoint_name,\n", " data_capture_config=data_capture_config,\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 推論\n", "作成したエンドポイントを使用して推論を実行します。Feature Attributionでドリフトを検知したい場合はコメントアウトされているpayloadを使用してください。" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Sending test traffic to the endpoint bb-example-xgboost-classification-model-2023-01-04-12-11-59-829. \n", "Please wait.................................................\n", "Done!\n" ] } ], "source": [ "print(f\"Sending test traffic to the endpoint {endpoint_name}. \\nPlease wait\", end=\"\")\n", "test_dataset_size = 0 # record the number of rows in data we're sending for inference\n", "with open(test_dataset, \"r\") as f:\n", " for row in f:\n", " if 400 < test_dataset_size and test_dataset_size < 450:\n", " payload = row.rstrip(\"\\n\")\n", " # 下記を有効にするとFeature Attributionのドリフトが検知できる\n", " # payload = '100,100,100,100,100,100,100,100,100,100,100,100,100'\n", " response = sagemaker_runtime_client.invoke_endpoint(\n", " EndpointName=endpoint_name,\n", " Body=payload,\n", " ContentType=dataset_type,\n", " InferenceId=str(test_dataset_size),\n", " )\n", " prediction = response[\"Body\"].read()\n", " print(\".\", end=\"\", flush=True)\n", " time.sleep(0.5)\n", " test_dataset_size += 1\n", "\n", "print()\n", "print(\"Done!\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### キャプチャデータの確認\n", "キャプチャされたデータを確認します。\n", "\n", "もし600秒では足りず、エラーになった場合は時間をおいて再度実行してみてください。\n", "\n", "こちらのコードはキャプチャデータの確認のみが目的のため、実行が完了していなくても、先に進むことができます。" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Waiting 600 seconds for captures to show up.................................................\n", "Found Capture Files:\n", "s3://sagemaker-ap-northeast-1-036661559124/sagemaker/blackbelt-part3-sample/datacapture/bb-example-xgboost-classification-model-2023-01-04-12-11-59-829/AllTraffic/2023/01/04/12/21-01-974-079e84cc-98a0-41fa-842a-f18c59da9013.jsonl\n" ] } ], "source": [ "print(\"Waiting 600 seconds for captures to show up\", end=\"\")\n", "for _ in range(600):\n", " capture_files = sorted(S3Downloader.list(f\"{s3_capture_upload_path}/{endpoint_name}\"))\n", " if capture_files:\n", " break\n", " print(\".\", end=\"\", flush=True)\n", " time.sleep(1)\n", "print()\n", "print(\"Found Capture Files:\")\n", "print(\"\\n \".join(capture_files[-5:]))" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\"captureData\":{\"endpointInput\":{\"observedContentType\":\"text/csv\",\"mode\":\"INPUT\",\"data\":\"77,0,44,0,1,2,0,-1,0,57,1,0,0\",\"encoding\":\"CSV\"},\"endpointOutput\":{\"observedContentType\":\"text/csv; charset=utf-8\",\"mode\":\"OUTPUT\",\"data\":\"0.2509244382381439\\n\",\"encoding\":\"CSV\"}},\"eventMetadata\":{\"eventId\":\"a608eba5-9aef-4580-8dba-a2b35aa94324\",\"inferenceId\":\"449\",\"inferenceTime\":\"2023-01-04T12:21:26Z\"},\"eventVersion\":\"0\"}\n" ] } ], "source": [ "capture_file = S3Downloader.read_file(capture_files[-1]).split(\"\\n\")[-10:-1]\n", "print(capture_file[-1])" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"captureData\": {\n", " \"endpointInput\": {\n", " \"observedContentType\": \"text/csv\",\n", " \"mode\": \"INPUT\",\n", " \"data\": \"77,0,44,0,1,2,0,-1,0,57,1,0,0\",\n", " \"encoding\": \"CSV\"\n", " },\n", " \"endpointOutput\": {\n", " \"observedContentType\": \"text/csv; charset=utf-8\",\n", " \"mode\": \"OUTPUT\",\n", " \"data\": \"0.2509244382381439\\n\",\n", " \"encoding\": \"CSV\"\n", " }\n", " },\n", " \"eventMetadata\": {\n", " \"eventId\": \"a608eba5-9aef-4580-8dba-a2b35aa94324\",\n", " \"inferenceId\": \"449\",\n", " \"inferenceTime\": \"2023-01-04T12:21:26Z\"\n", " },\n", " \"eventVersion\": \"0\"\n", "}\n" ] } ], "source": [ "print(json.dumps(json.loads(capture_file[-1]), indent=2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 正解データのアップロード\n", "正解データをアップロードします。正解データがない場合、バイアスのスケジュールジョブは失敗します。" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "# import random\n", "\n", "def ground_truth_with_id(inference_id):\n", " # random.seed(inference_id)\n", " # rand = random.random()\n", "\n", " return {\n", " \"groundTruthData\": {\n", " # ランダムなデータを正解として用意する場合のサンプルコード\n", " # \"data\": \"1\" if rand < 0.7 else \"0\",\n", " \"data\": str(ground_truth_label.iat[inference_id,0]),\n", " \"encoding\": \"CSV\",\n", " },\n", " \"eventMetadata\": {\n", " \"eventId\": str(inference_id),\n", " },\n", " \"eventVersion\": \"0\",\n", " }\n", "\n", "\n", "def upload_ground_truth(upload_time):\n", " records = [ground_truth_with_id(i) for i in range(test_dataset_size)]\n", " fake_records = [json.dumps(r) for r in records]\n", " data_to_upload = \"\\n\".join(fake_records)\n", " target_s3_uri = f\"{ground_truth_upload_path}/{upload_time:%Y/%m/%d/%H/%M%S}.jsonl\"\n", " print(f\"Uploading {len(fake_records)} records to\", target_s3_uri)\n", " S3Uploader.upload_string_as_file_body(data_to_upload, target_s3_uri)" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Uploading 114158 records to s3://sagemaker-ap-northeast-1-036661559124/sagemaker/blackbelt-part3-sample/ground_truth_data/2023-01-04-12-11-59/2023/01/04/12/2231.jsonl\n" ] } ], "source": [ "upload_ground_truth(datetime.now())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## ステップ1: モデルバイアス\n", "### ベースライン作成\n", "ベースラインの作成には10分程度かかります。" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [], "source": [ "model_bias_monitor = ModelBiasMonitor(\n", " role=role,\n", " sagemaker_session=sagemaker_session,\n", " max_runtime_in_seconds=1800,\n", ")" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [], "source": [ "model_bias_baselining_job_result_uri = f\"{baseline_results_uri}/model_bias\"\n", "model_bias_data_config = DataConfig(\n", " s3_data_input_path=baseline_uri,\n", " s3_output_path=model_bias_baselining_job_result_uri,\n", " label=label_header,\n", " headers=all_headers,\n", " dataset_type=dataset_type,\n", ")" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [], "source": [ "model_bias_config = BiasConfig(\n", " label_values_or_threshold=[1], \n", " facet_name='A_SEX',\n", " group_name='A_HGA'\n", ")" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [], "source": [ "model_predicted_label_config = ModelPredictedLabelConfig(\n", " probability_threshold=0.7,\n", ")" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [], "source": [ "model_config = ModelConfig(\n", " model_name=model_name,\n", " instance_count=endpoint_instance_count,\n", " instance_type=endpoint_instance_type,\n", " content_type=dataset_type,\n", " accept_type=dataset_type,\n", ")" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Job Name: baseline-suggestion-job-2023-01-04-12-22-36-206\n", "Inputs: [{'InputName': 'dataset', 'AppManaged': False, 'S3Input': {'S3Uri': 's3://sagemaker-ap-northeast-1-036661559124/sagemaker/blackbelt-part3-sample/baselining/baseline/baseline_with_header.csv', 'LocalPath': '/opt/ml/processing/input/data', 'S3DataType': 'S3Prefix', 'S3InputMode': 'File', 'S3DataDistributionType': 'FullyReplicated', 'S3CompressionType': 'None'}}, {'InputName': 'analysis_config', 'AppManaged': False, 'S3Input': {'S3Uri': 's3://sagemaker-ap-northeast-1-036661559124/sagemaker/blackbelt-part3-sample/baselining/model_bias/analysis_config.json', 'LocalPath': '/opt/ml/processing/input/config', 'S3DataType': 'S3Prefix', 'S3InputMode': 'File', 'S3DataDistributionType': 'FullyReplicated', 'S3CompressionType': 'None'}}]\n", "Outputs: [{'OutputName': 'analysis_result', 'AppManaged': False, 'S3Output': {'S3Uri': 's3://sagemaker-ap-northeast-1-036661559124/sagemaker/blackbelt-part3-sample/baselining/model_bias', 'LocalPath': '/opt/ml/processing/output', 'S3UploadMode': 'EndOfJob'}}]\n", "ModelBiasMonitor baselining job: baseline-suggestion-job-2023-01-04-12-22-36-206\n" ] } ], "source": [ "model_bias_monitor.suggest_baseline(\n", " model_config=model_config,\n", " data_config=model_bias_data_config,\n", " bias_config=model_bias_config,\n", " model_predicted_label_config=model_predicted_label_config,\n", ")\n", "print(f\"ModelBiasMonitor baselining job: {model_bias_monitor.latest_baselining_job_name}\")" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "...................................................................................................!\n", "ModelBiasMonitor suggested constraints: s3://sagemaker-ap-northeast-1-036661559124/sagemaker/blackbelt-part3-sample/baselining/model_bias/analysis.json\n", "{\n", " \"version\": \"1.0\",\n", " \"post_training_bias_metrics\": {\n", " \"label\": \"PTOTVAL_Over50000\",\n", " \"facets\": {\n", " \"A_SEX\": [\n", " {\n", " \"value_or_threshold\": \"1\",\n", " \"metrics\": [\n", " {\n", " \"name\": \"AD\",\n", " \"description\": \"Accuracy Difference (AD)\",\n", " \"value\": 0.047995482778091514\n", " },\n", " {\n", " \"name\": \"CDDPL\",\n", " \"description\": \"Conditional Demographic Disparity in Predicted Labels (CDDPL)\",\n", " \"value\": -0.35173994883458676\n", " },\n", " {\n", " \"name\": \"DAR\",\n", " \"description\": \"Difference in Acceptance Rates (DAR)\",\n", " \"value\": -0.18103448275862066\n", " },\n", " {\n", " \"name\": \"DCA\",\n", " \"description\": \"Difference in Conditional Acceptance (DCA)\",\n", " \"value\": 0.6393678160919543\n", " },\n", " {\n", " \"name\": \"DCR\",\n", " \"description\": \"Difference in Conditional Rejection (DCR)\",\n", " \"value\": -0.10960185448777915\n", " },\n", " {\n", " \"name\": \"DI\",\n", " \"description\": \"Disparate Impact (DI)\",\n", " \"value\": 2.586783988957902\n", " },\n", " {\n", " \"name\": \"DPPL\",\n", " \"description\": \"Difference in Positive Proportions in Predicted Labels (DPPL)\",\n", " \"value\": -0.14732230458409923\n", " },\n", " {\n", " \"name\": \"DRR\",\n", " \"description\": \"Difference in Rejection Rates (DRR)\",\n", " \"value\": -0.10581386566583206\n", " },\n", " {\n", " \"name\": \"FT\",\n", " \"description\": \"Flip Test (FT)\",\n", " \"value\": 0.12215320910973085\n", " },\n", " {\n", " \"name\": \"GE\",\n", " \"description\": \"Generalized Entropy (GE)\",\n", " \"value\": 0.12748753989027944\n", " },\n", " {\n", " \"name\": \"RD\",\n", " \"description\": \"Recall Difference (RD)\",\n", " \"value\": -0.20676359546951556\n", " },\n", " {\n", " \"name\": \"SD\",\n", " \"description\": \"Specificity Difference (SD)\",\n", " \"value\": 0.0006634232640424509\n", " },\n", " {\n", " \"name\": \"TE\",\n", " \"description\": \"Treatment Equality (TE)\",\n", " \"value\": 6.041666666666667\n", " }\n", " ]\n", " },\n", " {\n", " \"value_or_threshold\": \"2\",\n", " \"metrics\": [\n", " {\n", " \"name\": \"AD\",\n", " \"description\": \"Accuracy Difference (AD)\",\n", " \"value\": -0.047995482778091514\n", " },\n", " {\n", " \"name\": \"CDDPL\",\n", " \"description\": \"Conditional Demographic Disparity in Predicted Labels (CDDPL)\",\n", " \"value\": 0.46273994883458675\n", " },\n", " {\n", " \"name\": \"DAR\",\n", " \"description\": \"Difference in Acceptance Rates (DAR)\",\n", " \"value\": 0.18103448275862066\n", " },\n", " {\n", " \"name\": \"DCA\",\n", " \"description\": \"Difference in Conditional Acceptance (DCA)\",\n", " \"value\": -0.6393678160919543\n", " },\n", " {\n", " \"name\": \"DCR\",\n", " \"description\": \"Difference in Conditional Rejection (DCR)\",\n", " \"value\": 0.10960185448777915\n", " },\n", " {\n", " \"name\": \"DI\",\n", " \"description\": \"Disparate Impact (DI)\",\n", " \"value\": 0.3865804041886214\n", " },\n", " {\n", " \"name\": \"DPPL\",\n", " \"description\": \"Difference in Positive Proportions in Predicted Labels (DPPL)\",\n", " \"value\": 0.14732230458409923\n", " },\n", " {\n", " \"name\": \"DRR\",\n", " \"description\": \"Difference in Rejection Rates (DRR)\",\n", " \"value\": 0.10581386566583206\n", " },\n", " {\n", " \"name\": \"FT\",\n", " \"description\": \"Flip Test (FT)\",\n", " \"value\": -0.09671179883945841\n", " },\n", " {\n", " \"name\": \"GE\",\n", " \"description\": \"Generalized Entropy (GE)\",\n", " \"value\": 0.12748753989027944\n", " },\n", " {\n", " \"name\": \"RD\",\n", " \"description\": \"Recall Difference (RD)\",\n", " \"value\": 0.20676359546951556\n", " },\n", " {\n", " \"name\": \"SD\",\n", " \"description\": \"Specificity Difference (SD)\",\n", " \"value\": -0.0006634232640424509\n", " },\n", " {\n", " \"name\": \"TE\",\n", " \"description\": \"Treatment Equality (TE)\",\n", " \"value\": -6.041666666666667\n", " }\n", " ]\n", " }\n", " ]\n", " },\n", " \"label_value_or_threshold\": \"1\"\n", " },\n", " \"pre_training_bias_metrics\": {\n", " \"label\": \"PTOTVAL_Over50000\",\n", " \"facets\": {\n", " \"A_SEX\": [\n", " {\n", " \"value_or_threshold\": \"1\",\n", " \"metrics\": [\n", " {\n", " \"name\": \"CDDL\",\n", " \"description\": \"Conditional Demographic Disparity in Labels (CDDL)\",\n", " \"value\": -0.27982989438014333\n", " },\n", " {\n", " \"name\": \"CI\",\n", " \"description\": \"Class Imbalance (CI)\",\n", " \"value\": 0.034\n", " },\n", " {\n", " \"name\": \"DPL\",\n", " \"description\": \"Difference in Positive Proportions in Labels (DPL)\",\n", " \"value\": -0.2086131568092715\n", " },\n", " {\n", " \"name\": \"JS\",\n", " \"description\": \"Jensen-Shannon Divergence (JS)\",\n", " \"value\": 0.024821569781261953\n", " },\n", " {\n", " \"name\": \"KL\",\n", " \"description\": \"Kullback-Liebler Divergence (KL)\",\n", " \"value\": 0.09503394708171636\n", " },\n", " {\n", " \"name\": \"KS\",\n", " \"description\": \"Kolmogorov-Smirnov Distance (KS)\",\n", " \"value\": 0.2086131568092715\n", " },\n", " {\n", " \"name\": \"LP\",\n", " \"description\": \"L-p Norm (LP)\",\n", " \"value\": 0.29502355564913696\n", " },\n", " {\n", " \"name\": \"TVD\",\n", " \"description\": \"Total Variation Distance (TVD)\",\n", " \"value\": 0.2086131568092715\n", " }\n", " ]\n", " },\n", " {\n", " \"value_or_threshold\": \"2\",\n", " \"metrics\": [\n", " {\n", " \"name\": \"CDDL\",\n", " \"description\": \"Conditional Demographic Disparity in Labels (CDDL)\",\n", " \"value\": 0.2918298943801434\n", " },\n", " {\n", " \"name\": \"CI\",\n", " \"description\": \"Class Imbalance (CI)\",\n", " \"value\": -0.034\n", " },\n", " {\n", " \"name\": \"DPL\",\n", " \"description\": \"Difference in Positive Proportions in Labels (DPL)\",\n", " \"value\": 0.2086131568092715\n", " },\n", " {\n", " \"name\": \"JS\",\n", " \"description\": \"Jensen-Shannon Divergence (JS)\",\n", " \"value\": 0.024821569781261953\n", " },\n", " {\n", " \"name\": \"KL\",\n", " \"description\": \"Kullback-Liebler Divergence (KL)\",\n", " \"value\": 0.1061385749752419\n", " },\n", " {\n", " \"name\": \"KS\",\n", " \"description\": \"Kolmogorov-Smirnov Distance (KS)\",\n", " \"value\": 0.2086131568092715\n", " },\n", " {\n", " \"name\": \"LP\",\n", " \"description\": \"L-p Norm (LP)\",\n", " \"value\": 0.29502355564913696\n", " },\n", " {\n", " \"name\": \"TVD\",\n", " \"description\": \"Total Variation Distance (TVD)\",\n", " \"value\": 0.2086131568092715\n", " }\n", " ]\n", " }\n", " ]\n", " },\n", " \"label_value_or_threshold\": \"1\"\n", " }\n", "}\n" ] } ], "source": [ "model_bias_monitor.latest_baselining_job.wait(logs=False)\n", "model_bias_constraints = model_bias_monitor.suggested_constraints()\n", "print()\n", "print(f\"ModelBiasMonitor suggested constraints: {model_bias_constraints.file_s3_uri}\")\n", "print(S3Downloader.read_file(model_bias_constraints.file_s3_uri))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### スケジュールジョブ作成" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model bias monitoring schedule: monitoring-schedule-2023-01-04-12-30-54-730\n" ] } ], "source": [ "model_bias_analysis_config = None\n", "if not model_bias_monitor.latest_baselining_job:\n", " model_bias_analysis_config = BiasAnalysisConfig(\n", " model_bias_config,\n", " headers=all_headers,\n", " label=label_header,\n", " )\n", "model_bias_monitor.create_monitoring_schedule(\n", " analysis_config=model_bias_analysis_config,\n", " output_s3_uri=s3_report_path,\n", " endpoint_input=EndpointInput(\n", " endpoint_name=endpoint_name,\n", " destination=\"/opt/ml/processing/input/endpoint\",\n", " start_time_offset=\"-PT1H\",\n", " end_time_offset=\"-PT0H\",\n", " probability_threshold_attribute=0.7,\n", " ),\n", " ground_truth_input=ground_truth_upload_path,\n", " schedule_cron_expression=schedule_expression,\n", " enable_cloudwatch_metrics=True,\n", ")\n", "print(f\"Model bias monitoring schedule: {model_bias_monitor.monitoring_schedule_name}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## ステップ2: Feature Attribution\n", "### ベースライン作成\n", "ベースラインの作成には10分程度かかります。" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [], "source": [ "model_explainability_monitor = ModelExplainabilityMonitor(\n", " role=role,\n", " sagemaker_session=sagemaker_session,\n", " max_runtime_in_seconds=1800,\n", ")" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [], "source": [ "model_explainability_baselining_job_result_uri = f\"{baseline_results_uri}/model_explainability\"\n", "model_explainability_data_config = DataConfig(\n", " s3_data_input_path=baseline_uri,\n", " s3_output_path=model_explainability_baselining_job_result_uri,\n", " label=label_header,\n", " headers=all_headers,\n", " dataset_type=dataset_type,\n", ")" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [], "source": [ "baseline_dataframe = pd.read_csv(baseline_uri).drop(label_header, axis=1)\n", "shap_baseline = [list(baseline_dataframe.mean())]\n", "\n", "shap_config = SHAPConfig(\n", " baseline=shap_baseline,\n", " num_samples=100,\n", " agg_method=\"mean_abs\",\n", " save_local_shap_values=True,\n", ")" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Job Name: baseline-suggestion-job-2023-01-04-12-30-55-986\n", "Inputs: [{'InputName': 'dataset', 'AppManaged': False, 'S3Input': {'S3Uri': 's3://sagemaker-ap-northeast-1-036661559124/sagemaker/blackbelt-part3-sample/baselining/baseline/baseline_with_header.csv', 'LocalPath': '/opt/ml/processing/input/data', 'S3DataType': 'S3Prefix', 'S3InputMode': 'File', 'S3DataDistributionType': 'FullyReplicated', 'S3CompressionType': 'None'}}, {'InputName': 'analysis_config', 'AppManaged': False, 'S3Input': {'S3Uri': 's3://sagemaker-ap-northeast-1-036661559124/sagemaker/blackbelt-part3-sample/baselining/model_explainability/analysis_config.json', 'LocalPath': '/opt/ml/processing/input/config', 'S3DataType': 'S3Prefix', 'S3InputMode': 'File', 'S3DataDistributionType': 'FullyReplicated', 'S3CompressionType': 'None'}}]\n", "Outputs: [{'OutputName': 'analysis_result', 'AppManaged': False, 'S3Output': {'S3Uri': 's3://sagemaker-ap-northeast-1-036661559124/sagemaker/blackbelt-part3-sample/baselining/model_explainability', 'LocalPath': '/opt/ml/processing/output', 'S3UploadMode': 'EndOfJob'}}]\n", "ModelExplainabilityMonitor baselining job: baseline-suggestion-job-2023-01-04-12-30-55-986\n" ] } ], "source": [ "model_explainability_monitor.suggest_baseline(\n", " data_config=model_explainability_data_config,\n", " model_config=model_config,\n", " explainability_config=shap_config,\n", ")\n", "print(\n", " f\"ModelExplainabilityMonitor baselining job: {model_explainability_monitor.latest_baselining_job_name}\"\n", ")" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ ".........................................................................................................!\n", "ModelExplainabilityMonitor suggested constraints: s3://sagemaker-ap-northeast-1-036661559124/sagemaker/blackbelt-part3-sample/baselining/model_explainability/analysis.json\n", "{\n", " \"version\": \"1.0\",\n", " \"explanations\": {\n", " \"kernel_shap\": {\n", " \"label0\": {\n", " \"global_shap_values\": {\n", " \"A_AGE\": 0.03900713880041894,\n", " \"A_FTLF\": 0.009864065651959351,\n", " \"A_HGA\": 0.08227859137445104,\n", " \"A_HSCOL\": 0.006078339682850803,\n", " \"A_MARITL\": 0.020923815390634877,\n", " \"A_SEX\": 0.0464811787751795,\n", " \"A_UNMEM\": 0.00656392329159415,\n", " \"A_USLHRS\": 0.09165896714888844,\n", " \"NOEMP\": 0.029412844725952524,\n", " \"PENATVTY\": 0.03568894808061845,\n", " \"PRCITSHP\": 0.008444406567974315,\n", " \"SEOTR\": 0.008066975367524291,\n", " \"WKSWORK\": 0.10033051463598301\n", " },\n", " \"expected_value\": 0.05636756867170334\n", " }\n", " }\n", " }\n", "}\n" ] } ], "source": [ "model_explainability_monitor.latest_baselining_job.wait(logs=False)\n", "model_explainability_constraints = model_explainability_monitor.suggested_constraints()\n", "print()\n", "print(\n", " f\"ModelExplainabilityMonitor suggested constraints: {model_explainability_constraints.file_s3_uri}\"\n", ")\n", "print(S3Downloader.read_file(model_explainability_constraints.file_s3_uri))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### スケジュールジョブ作成" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [], "source": [ "model_explainability_analysis_config = None\n", "if not model_explainability_monitor.latest_baselining_job:\n", " headers_without_label_header = copy.deepcopy(all_headers)\n", " headers_without_label_header.remove(label_header)\n", " model_explainability_analysis_config = ExplainabilityAnalysisConfig(\n", " explainability_config=shap_config,\n", " model_config=model_config,\n", " headers=headers_without_label_header,\n", " )\n", "model_explainability_monitor.create_monitoring_schedule(\n", " analysis_config = model_explainability_analysis_config,\n", " output_s3_uri=s3_report_path,\n", " endpoint_input=endpoint_name,\n", " schedule_cron_expression=schedule_expression,\n", " enable_cloudwatch_metrics=True,\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## ステップ3: リソース削除" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Deleting Monitoring Schedule with name: monitoring-schedule-2023-01-04-12-30-54-730\n" ] } ], "source": [ "model_bias_monitor.delete_monitoring_schedule()" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Deleting Monitoring Schedule with name: monitoring-schedule-2023-01-04-12-39-44-618\n" ] } ], "source": [ "model_explainability_monitor.delete_monitoring_schedule()" ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [], "source": [ "from sagemaker.predictor import Predictor\n", "\n", "predictor = Predictor(endpoint_name, sagemaker_session=sagemaker_session)\n", "predictor.delete_endpoint(delete_endpoint_config=False)" ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [], "source": [ "model.delete_model()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "instance_type": "ml.t3.medium", "kernelspec": { "display_name": "Python 3 (Data Science)", "language": "python", "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:ap-northeast-1:102112518831:image/datascience-1.0" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.10" } }, "nbformat": 4, "nbformat_minor": 4 }