{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "dbf512e7-7891-4379-84e3-fc93e5dad9d6" }, "source": [ "# Integrate SageMaker Model with Watson Open Scale" ] }, { "cell_type": "markdown", "metadata": { "id": "e68acbdc-c539-4215-8dc8-22245a24445a" }, "source": [ "Contents\n", "- Setup\n", "- Binding machine learning engine\n", "- Subscriptions\n", "- Performance monitor, scoring and payload logging\n", "- Quality monitor and feedback logging\n", "- Fairness, Drift monitoring and explanations" ] }, { "cell_type": "markdown", "metadata": { "id": "d9f4b9cc-8a90-437b-baf8-75c64297c083" }, "source": [ "### Required packages installation" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "id": "3ca194ad-47b1-4c76-b3c6-765d2f5d5843" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Successfully installed boto3-1.26.60 botocore-1.29.60 contextlib2-21.6.0 dill-0.3.6 multiprocess-0.70.14 pathos-0.3.0 pox-0.3.2 ppft-1.7.6.6 protobuf3-to-dict-0.1.5 sagemaker-2.131.0 schema-0.7.5 smdebug_rulesconfig-1.0.1\n", "Successfully installed ibm-watson-openscale-3.0.27\n", "Requirement already satisfied: six>=1.5 in /opt/conda/envs/Python-3.10/lib/python3.10/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.30.0,>=1.29.60->boto3) (1.16.0)\n", "Successfully installed pandas-1.2.5\n" ] } ], "source": [ "!pip install sagemaker --no-cache | tail -n 1\n", "!pip install --upgrade ibm-watson-openscale --no-cache | tail -n 1\n", "!pip install --upgrade boto3 --no-cache | tail -n 1\n", "!pip install -U pandas==1.2.5 | tail -n 1" ] }, { "cell_type": "markdown", "metadata": { "id": "56540e9f-69a7-4586-a1f8-4f28dfd85c78" }, "source": [ "### Model creation using [Amazon SageMaker](https://aws.amazon.com/sagemaker/)" ] }, { "cell_type": "markdown", "metadata": { "id": "1df3327f-ed1c-48f0-bc2f-7f8138e2dff0" }, "source": [ "- Run the notebook ( RI-SageMaker-Deploy-Wstudio.ipynb ) to train a SageMaker model and create deployment endpoint for online inference" ] }, { "cell_type": "markdown", "metadata": { "id": "c0d92147-4956-4019-acfe-972708792d0a" }, "source": [ "### Update the below cell with IBM DB2 credentials like DATABASE, HOSTNAME, USERNAME, PASSWORD & PORT. " ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "id": "52f7d27b-25e1-416c-af74-948adf7ccdfa" }, "outputs": [], "source": [ "SCHEMA_NAME='LAB-1-DATA'\n", "TABLE_NAME='DATA-LAB-1-WOS'\n", "DATABASE=''\n", "HOSTNAME=''\n", "USERNAME=''\n", "PASSWORD=''\n", "PORT=" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "id": "9f614fda82a44b2d8a9c578a22a7aa2f" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "--2023-01-31 07:42:27-- https://ibm-aws-immersion-day.s3.us-east-2.amazonaws.com/publicdata/Data-region-RI-SM.csv\n", "Resolving ibm-aws-immersion-day.s3.us-east-2.amazonaws.com (ibm-aws-immersion-day.s3.us-east-2.amazonaws.com)... 52.219.97.74\n", "Connecting to ibm-aws-immersion-day.s3.us-east-2.amazonaws.com (ibm-aws-immersion-day.s3.us-east-2.amazonaws.com)|52.219.97.74|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 36731 (36K) [text/csv]\n", "Saving to: ‘Data-region-RI-SM.csv’\n", "\n", "Data-region-RI-SM.c 100%[===================>] 35.87K --.-KB/s in 0.001s \n", "\n", "2023-01-31 07:42:27 (45.3 MB/s) - ‘Data-region-RI-SM.csv’ saved [36731/36731]\n", "\n" ] } ], "source": [ "!wget \"https://ibm-aws-immersion-day.s3.us-east-2.amazonaws.com/publicdata/Data-region-RI-SM.csv\"" ] }, { "cell_type": "markdown", "metadata": { "id": "232c82c79e8948f1a3e188a1ba66fb6b" }, "source": [ "### Update the URL along with username & password for Cloud Pak for Data" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "id": "7ea1e5a385f74a01929746e1bb8dff66" }, "outputs": [], "source": [ "from ibm_cloud_sdk_core.authenticators import IAMAuthenticator,CloudPakForDataAuthenticator\n", "from ibm_watson_openscale import APIClient\n", "\n", "service_credentials = {\n", " \"url\": \"\",\n", " \"username\": \"\",\n", " \"password\": \"\"\n", " }\n", "\n", "authenticator = CloudPakForDataAuthenticator(\n", " url=service_credentials['url'],\n", " username=service_credentials['username'],\n", " password=service_credentials['password'],\n", " disable_ssl_verification=True\n", " )" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "id": "30d87b5ae2d34c1c8f0bc9ec55092b87" }, "outputs": [ { "data": { "text/plain": [ "'3.0.26'" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from ibm_watson_openscale import *\n", "from ibm_watson_openscale.supporting_classes.enums import *\n", "from ibm_watson_openscale.supporting_classes import *\n", "\n", "import urllib3\n", "urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)\n", "\n", "wos_client = APIClient(\n", " service_url=service_credentials['url'],\n", " authenticator=authenticator\n", ")\n", "wos_client.version" ] }, { "cell_type": "markdown", "metadata": { "id": "7fc1dfb7-d953-4578-a6d8-09e047f5a087" }, "source": [ "Create schema for data mart." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "id": "0bf2a4e0-eae4-4b06-8ed2-8de52efe5b93" }, "outputs": [ { "data": { "text/html": [ "\n", " \n", "

Data Marts

\n", " \n", " \n", " \n", "
namedescriptioninternal_databasestatuscreated_atid
NoneNoneFalseactive2022-12-19 11:00:56.742000+00:0000000000-0000-0000-0000-000000000000
\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "wos_client.data_marts.show()" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "id": "c46281b7-c27e-47ad-9046-140a99d9d702" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Using existing datamart 00000000-0000-0000-0000-000000000000\n" ] } ], "source": [ "data_marts = wos_client.data_marts.list().result.data_marts\n", "if len(data_marts) == 0:\n", " if DB_CREDENTIALS is not None:\n", " if SCHEMA_NAME is None: \n", " print(\"Please specify the SCHEMA_NAME and rerun the cell\")\n", "\n", " print('Setting up external datamart')\n", " added_data_mart_result = wos_client.data_marts.add(\n", " background_mode=False,\n", " name=\"WOS SGM Data Mart\",\n", " description=\"Data Mart created for WOS SGM\",\n", " database_configuration=DatabaseConfigurationRequest(\n", " database_type=DatabaseType.POSTGRESQL, # For DB2 use DatabaseType.DB2\n", " credentials=PrimaryStorageCredentialsLong(\n", " hostname=DB_CREDENTIALS['hostname'],\n", " username=DB_CREDENTIALS['username'],\n", " password=DB_CREDENTIALS['password'],\n", " db=DB_CREDENTIALS['database'],\n", " port=DB_CREDENTIALS['port'],\n", " ssl=False,\n", " certificate_base64=False\n", " ),\n", " location=LocationSchemaName(\n", " schema_name= SCHEMA_NAME\n", " )\n", " )\n", " ).result\n", " else:\n", " print('Setting up internal datamart')\n", " added_data_mart_result = wos_client.data_marts.add(\n", " background_mode=False,\n", " name=\"WOS SGM Data Mart\",\n", " description=\"Data Mart created for WOS SGM\", \n", " internal_database = True).result\n", " \n", " data_mart_id = added_data_mart_result.metadata.id\n", " \n", "else:\n", " data_mart_id=data_marts[0].metadata.id\n", " print('Using existing datamart {}'.format(data_mart_id))" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "id": "f4633150-a92a-4698-8499-cb0548d46f0c" }, "outputs": [ { "data": { "text/plain": [ "{'metadata': {'id': '00000000-0000-0000-0000-000000000000',\n", " 'crn': 'crn:v1:bluemix:public:aiopenscale:us-south:a/na:00000000-0000-0000-0000-000000000000:data_mart:00000000-0000-0000-0000-000000000000',\n", " 'url': '/v2/data_marts/00000000-0000-0000-0000-000000000000',\n", " 'created_at': '2022-12-19T11:00:56.742000Z',\n", " 'created_by': 'admin'},\n", " 'entity': {'service_instance_crn': 'N/A',\n", " 'internal_database': False,\n", " 'database_configuration': {'database_type': 'db2',\n", " 'instance_id': 'ip-10-0-22-85.us-east-2.compute.internal',\n", " 'name': 'manual-ip-10-0-22-85.us-east-2.compute.internal',\n", " 'credentials': {'secret_id': '782809e9-c182-499f-a922-2d9208968770'},\n", " 'location': {'schema_name': 'LAB-1-DATA'}},\n", " 'status': {'state': 'active'}}}" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "wos_client.data_marts.get(data_mart_id).result.to_dict()" ] }, { "cell_type": "markdown", "metadata": { "id": "c9311a7f-779c-4726-ad00-8fae71efccc6" }, "source": [ "\n", "## 2. Bind machine learning engines" ] }, { "cell_type": "markdown", "metadata": { "id": "9664669a-723c-43fa-a5a7-09fef8479d98" }, "source": [ "### Bind `SageMaker` machine learning engine\n", "\n", "#### Provide SageMaker credentials using following fields:\n", "- `access_key_id`\n", "- `secret_access_key`\n", "- `region`" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "id": "d3a3d477-d543-49ba-8979-e50e87a4ee43" }, "outputs": [], "source": [ "SAGEMAKER_ENGINE_CREDENTIALS = {\n", " 'access_key_id': '', \n", " 'secret_access_key': '', \n", " 'region': 'us-east-2'}" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "id": "2dc3c3d4-f6c0-4d94-842c-c64a985d9f6e" }, "outputs": [], "source": [ "SERVICE_PROVIDER_NAME = \"AWS SGM Machine Learning\"\n", "SERVICE_PROVIDER_DESCRIPTION = \"Added by AWS IBM Integration\"" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "id": "57c0684c-ecb8-492f-b752-236bc529fe0c" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Deleted existing service_provider for WML instance: a164cb69-145c-4015-9d4e-7bc5d84fd20c\n" ] } ], "source": [ "service_providers = wos_client.service_providers.list().result.service_providers\n", "for service_provider in service_providers:\n", " service_instance_name = service_provider.entity.name\n", " if service_instance_name == SERVICE_PROVIDER_NAME:\n", " service_provider_id = service_provider.metadata.id\n", " wos_client.service_providers.delete(service_provider_id)\n", " print(\"Deleted existing service_provider for WML instance: {}\".format(service_provider_id))" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "id": "5509cf40-9af8-4e19-a43d-ff817709ee9b" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "=================================================================================\n", "\n", " Waiting for end of adding service provider b426a9dc-b15b-420c-99bf-1b509b40f5ae \n", "\n", "=================================================================================\n", "\n", "\n", "\n", "active\n", "\n", "-----------------------------------------------\n", " Successfully finished adding service provider \n", "-----------------------------------------------\n", "\n", "\n", "Service Provider id b426a9dc-b15b-420c-99bf-1b509b40f5ae\n" ] } ], "source": [ "added_service_provider_result=wos_client.service_providers.add(\n", " name=SERVICE_PROVIDER_NAME,\n", " description=\"AWS Service Provider\",\n", " service_type=ServiceTypes.AMAZON_SAGEMAKER,\n", " credentials=SageMakerCredentials(\n", " access_key_id=SAGEMAKER_ENGINE_CREDENTIALS['access_key_id'],\n", " secret_access_key=SAGEMAKER_ENGINE_CREDENTIALS['secret_access_key'],\n", " region=SAGEMAKER_ENGINE_CREDENTIALS['region']\n", " ),\n", " background_mode=False\n", " ).result\n", "\n", "\n", "\n", "service_provider_id = added_service_provider_result.metadata.id\n", "print(\"Service Provider id \", service_provider_id)" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "id": "0c7f2e0a-b14d-4d5a-8cf4-d8bcb02a6ef9", "scrolled": true }, "outputs": [ { "data": { "text/html": [ "\n", " \n", "

Service Providers

\n", " \n", " \n", " \n", "
instance_idstatusnameservice_typecreated_atid
NoneactiveAWS SGM Machine Learningamazon_sagemaker2023-01-31 07:42:29.457000+00:00b426a9dc-b15b-420c-99bf-1b509b40f5ae
\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "wos_client.service_providers.show()" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "id": "8f9f1fa8-4ebc-485c-955d-867ec44c7215" }, "outputs": [ { "data": { "text/plain": [ "{'resources': [{'metadata': {'guid': 'f87b94db698b412f3953cb42f15dbbaa',\n", " 'url': 'linear-learner-2023-01-16-06-26-51-232',\n", " 'created_at': '2023-01-16T06:26:51.872Z',\n", " 'modified_at': '2023-01-16T06:30:35.707Z'},\n", " 'entity': {'name': 'linear-learner-2023-01-16-06-26-51-232',\n", " 'deployment_rn': 'arn:aws:sagemaker:us-east-2:481118440516:endpoint/linear-learner-2023-01-16-06-26-51-232',\n", " 'type': 'online',\n", " 'scoring_endpoint': {'url': 'linear-learner-2023-01-16-06-26-51-232'},\n", " 'asset': {'asset_id': 'f87b94db698b412f3953cb42f15dbbaa',\n", " 'asset_rn': 'arn:aws:sagemaker:us-east-2:481118440516:model/linear-learner-2023-01-16-06-26-51-232',\n", " 'url': 's3://sagemaker-us-east-2-481118440516/linear-learner-2023-01-16-06-22-37-086/output/model.tar.gz',\n", " 'name': 'linear-learner-2023-01-16-06-26-51-232',\n", " 'asset_type': 'model',\n", " 'created_at': '2023-01-16T06:26:51.552Z'},\n", " 'asset_properties': {'asset_revision': '1673850635707'}}}],\n", " 'count': 1}" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "asset_deployment_details = wos_client.service_providers.list_assets(data_mart_id=data_mart_id, service_provider_id=service_provider_id).result\n", "asset_deployment_details" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "id": "ad240686-addf-4a17-9aab-11fe14fc7eff" }, "outputs": [ { "data": { "text/plain": [ "{'metadata': {'guid': 'f87b94db698b412f3953cb42f15dbbaa',\n", " 'url': 'linear-learner-2023-01-16-06-26-51-232',\n", " 'created_at': '2023-01-16T06:26:51.872Z',\n", " 'modified_at': '2023-01-16T06:30:35.707Z'},\n", " 'entity': {'name': 'linear-learner-2023-01-16-06-26-51-232',\n", " 'deployment_rn': 'arn:aws:sagemaker:us-east-2:481118440516:endpoint/linear-learner-2023-01-16-06-26-51-232',\n", " 'type': 'online',\n", " 'scoring_endpoint': {'url': 'linear-learner-2023-01-16-06-26-51-232'},\n", " 'asset': {'asset_id': 'f87b94db698b412f3953cb42f15dbbaa',\n", " 'asset_rn': 'arn:aws:sagemaker:us-east-2:481118440516:model/linear-learner-2023-01-16-06-26-51-232',\n", " 'url': 's3://sagemaker-us-east-2-481118440516/linear-learner-2023-01-16-06-22-37-086/output/model.tar.gz',\n", " 'name': 'linear-learner-2023-01-16-06-26-51-232',\n", " 'asset_type': 'model',\n", " 'created_at': '2023-01-16T06:26:51.552Z'},\n", " 'asset_properties': {'asset_revision': '1673850635707'}}}" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "deployment_id='' # 'resources': [{'metadata': {'guid' value from previous cell will be deployment_id.\n", "for model_asset_details in asset_deployment_details['resources']:\n", " if model_asset_details['metadata']['guid']==deployment_id:\n", " break\n", "model_asset_details" ] }, { "cell_type": "markdown", "metadata": { "id": "4138ac46-c946-49ae-a200-3a7db5d6fe3f" }, "source": [ "\n", "## 3. Subscriptions" ] }, { "cell_type": "markdown", "metadata": { "id": "c116a828-bdfa-416c-b11b-b33ff4488686" }, "source": [ "### Add subscriptions" ] }, { "cell_type": "markdown", "metadata": { "id": "3983194d-5524-4414-a1c3-7d5d514aad3b" }, "source": [ "List available deployments.\n", "\n", "**Note:** Depending on number of assets it may take some time." ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "id": "090b820e-5d68-4588-8346-1bca073e4d16" }, "outputs": [], "source": [ "aws_asset = Asset(\n", " asset_id=model_asset_details['entity']['asset']['asset_id'],\n", " name=model_asset_details['entity']['asset']['name'],\n", " url=model_asset_details['entity']['asset']['url'],\n", " asset_type=model_asset_details['entity']['asset']['asset_type'] if 'asset_type' in model_asset_details['entity']['asset'] else 'model',\n", " problem_type=ProblemType.MULTICLASS_CLASSIFICATION,\n", " input_data_type=InputDataType.STRUCTURED,\n", " )" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "id": "230d13f2-76ba-4e6e-b85c-721844f878ab" }, "outputs": [], "source": [ "from ibm_watson_openscale.base_classes.watson_open_scale_v2 import ScoringEndpointRequest\n", "deployment_scoring_endpoint = model_asset_details['entity']['scoring_endpoint']\n", "scoring_endpoint = ScoringEndpointRequest(url = model_asset_details['entity']['scoring_endpoint']['url'] )" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "id": "255492a9-132c-42cd-ada3-d4f754fc7f1f" }, "outputs": [], "source": [ "deployment = AssetDeploymentRequest(\n", " deployment_id=model_asset_details['metadata']['guid'],\n", " url=model_asset_details['metadata']['url'],\n", " name=model_asset_details['entity']['name'],\n", " deployment_type=model_asset_details['entity']['type'],\n", " scoring_endpoint = scoring_endpoint\n", " )" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "id": "63b6a579-7870-4e5c-b7a7-f54c6028ed13" }, "outputs": [], "source": [ "training_data_reference = TrainingDataReference(type=TrainingDataReferenceType.DB2,\n", " location=DB2TrainingDataReferenceLocation(\n", " schema_name=SCHEMA_NAME,\n", " table_name=TABLE_NAME),\n", " connection=DB2TrainingDataReferenceConnection(\n", " hostname = HOSTNAME,\n", " username=USERNAME,\n", " password=PASSWORD,\n", " database_name=DATABASE,\n", " port=PORT\n", " )\n", " )" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "id": "bba9276c-b1fa-4145-9d84-dab20a0ffc08" }, "outputs": [], "source": [ "feature_columns = ['REGION','TOTAL_CASES']\n", "categorical_columns = ['REGION']" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "id": "47e4eb79-b4e3-4d20-8c3e-c487347b4b75" }, "outputs": [], "source": [ "asset_properties = AssetPropertiesRequest(\n", " label_column=\"RISK_INDEX\",\n", " prediction_field='predicted_label',\n", " probability_fields=['score'],\n", " training_data_reference=training_data_reference,\n", " training_data_schema=None,\n", " input_data_schema=None,\n", " output_data_schema=None,\n", " feature_fields=feature_columns,\n", " categorical_fields=categorical_columns\n", " )" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "id": "f69a719a-0351-4bfc-949a-829f5802f031" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "=============================================================================\n", "\n", " Waiting for end of adding subscription f26582b5-3112-4f0f-b00b-64cc07d0f6fa \n", "\n", "=============================================================================\n", "\n", "\n", "\n", "preparing\n", "active\n", "\n", "-------------------------------------------\n", " Successfully finished adding subscription \n", "-------------------------------------------\n", "\n", "\n" ] }, { "data": { "text/plain": [ "'f26582b5-3112-4f0f-b00b-64cc07d0f6fa'" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "subscription_details = wos_client.subscriptions.add(\n", " data_mart_id=data_mart_id,\n", " service_provider_id=service_provider_id,\n", " asset=aws_asset,\n", " deployment=deployment,\n", " asset_properties=asset_properties,\n", " background_mode=False\n", ").result\n", "subscription_id = subscription_details.metadata.id\n", "subscription_id" ] }, { "cell_type": "markdown", "metadata": { "id": "a01b405e-11a1-4e8e-8d03-0e572b0710d0" }, "source": [ "#### List subscriptions" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "id": "3092d51d-af01-4f89-8f64-c36ee54e3ff5" }, "outputs": [ { "data": { "text/html": [ "\n", " \n", "

Subscriptions

\n", " \n", " \n", " \n", "
asset_idasset_namedata_mart_iddeployment_iddeployment_nameservice_provider_idstatuscreated_atid
f87b94db698b412f3953cb42f15dbbaalinear-learner-2023-01-16-06-26-51-23200000000-0000-0000-0000-000000000000f87b94db698b412f3953cb42f15dbbaalinear-learner-2023-01-16-06-26-51-232b426a9dc-b15b-420c-99bf-1b509b40f5aeactive2023-01-31 07:42:35.201000+00:00f26582b5-3112-4f0f-b00b-64cc07d0f6fa
\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "wos_client.subscriptions.show()" ] }, { "cell_type": "markdown", "metadata": { "id": "faaf3457-5ec4-4621-a887-abd25832bf26" }, "source": [ "\n", "## 4. Performance metrics, scoring and payload logging" ] }, { "cell_type": "markdown", "metadata": { "id": "f2ab922c-6e90-40c2-a439-5b85d17ba3dc" }, "source": [ "### Score the risk index model and measure response time" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "id": "22cf94ec-5c58-41c5-af6a-1c5cbfff7392" }, "outputs": [], "source": [ "import requests\n", "import time\n", "import json\n", "import boto3" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "id": "7c0055b0-ae39-4355-a01e-e4c156dad9bd" }, "outputs": [ { "data": { "text/plain": [ "{'metadata': {'id': 'f26582b5-3112-4f0f-b00b-64cc07d0f6fa',\n", " 'crn': 'crn:v1:bluemix:public:aiopenscale:us-south:a/na:00000000-0000-0000-0000-000000000000:subscription:f26582b5-3112-4f0f-b00b-64cc07d0f6fa',\n", " 'url': '/v2/subscriptions/f26582b5-3112-4f0f-b00b-64cc07d0f6fa',\n", " 'created_at': '2023-01-31T07:42:35.201000Z',\n", " 'created_by': 'admin'},\n", " 'entity': {'data_mart_id': '00000000-0000-0000-0000-000000000000',\n", " 'service_provider_id': 'b426a9dc-b15b-420c-99bf-1b509b40f5ae',\n", " 'asset': {'asset_id': 'f87b94db698b412f3953cb42f15dbbaa',\n", " 'url': 's3://sagemaker-us-east-2-481118440516/linear-learner-2023-01-16-06-22-37-086/output/model.tar.gz',\n", " 'name': 'linear-learner-2023-01-16-06-26-51-232',\n", " 'asset_type': 'model',\n", " 'problem_type': 'multiclass',\n", " 'input_data_type': 'structured'},\n", " 'asset_properties': {'training_data_reference': {'secret_id': '3271ffbe-4294-47fa-b34f-f109bb08d923'},\n", " 'output_data_schema': {'type': 'struct',\n", " 'fields': [{'metadata': {'columnInfo': {'columnLength': 128},\n", " 'modeling_role': 'record-id',\n", " 'primary_key': True},\n", " 'name': 'scoring_id',\n", " 'nullable': False,\n", " 'type': 'string'},\n", " {'metadata': {'modeling_role': 'record-timestamp'},\n", " 'name': 'scoring_timestamp',\n", " 'nullable': False,\n", " 'type': 'timestamp'},\n", " {'metadata': {},\n", " 'name': 'deployment_id',\n", " 'nullable': False,\n", " 'type': 'string'},\n", " {'metadata': {},\n", " 'name': 'asset_revision',\n", " 'nullable': True,\n", " 'type': 'string'}]},\n", " 'label_column': 'RISK_INDEX',\n", " 'prediction_field': 'predicted_label',\n", " 'feature_fields': ['REGION', 'TOTAL_CASES'],\n", " 'categorical_fields': ['REGION'],\n", " 'probability_fields': ['score']},\n", " 'deployment': {'deployment_id': 'f87b94db698b412f3953cb42f15dbbaa',\n", " 'url': 'linear-learner-2023-01-16-06-26-51-232',\n", " 'name': 'linear-learner-2023-01-16-06-26-51-232',\n", " 'deployment_type': 'online',\n", " 'scoring_endpoint': {'url': 'linear-learner-2023-01-16-06-26-51-232'}},\n", " 'status': {'state': 'active'}}}" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "subscription_details=wos_client.subscriptions.get(subscription_id).result.to_dict()\n", "subscription_details" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "id": "63c05eb3-614e-4a76-aafe-3220db2e0aae" }, "outputs": [], "source": [ "endpoint_name = subscription_details['entity']['deployment']['name']\n", "\n", "payload = \"0,100\"" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "id": "a115e98f-96bb-464e-a797-d67184b13c3d" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"predictions\": [\n", " {\n", " \"score\": [\n", " 0.8418549299240112,\n", " 0.023631207644939423,\n", " 0.13451385498046875\n", " ],\n", " \"predicted_label\": 0\n", " }\n", " ]\n", "}\n" ] } ], "source": [ "runtime = boto3.client('sagemaker-runtime',\n", " region_name=SAGEMAKER_ENGINE_CREDENTIALS['region'],\n", " aws_access_key_id=SAGEMAKER_ENGINE_CREDENTIALS['access_key_id'],\n", " aws_secret_access_key=SAGEMAKER_ENGINE_CREDENTIALS['secret_access_key'])\n", "\n", "start_time = time.time()\n", "response = runtime.invoke_endpoint(EndpointName=endpoint_name, ContentType='text/csv', Body=payload)\n", "response_time = int((time.time() - start_time)*1000)\n", "result = json.loads(response['Body'].read().decode())\n", "\n", "print(json.dumps(result, indent=2))" ] }, { "cell_type": "markdown", "metadata": { "id": "2602aa1b-9d31-4605-a88e-8eee828db7b4" }, "source": [ "### Store the request and response in payload logging table" ] }, { "cell_type": "markdown", "metadata": { "id": "07b8ddb7-cd3b-458c-b820-6d2b924d54ae" }, "source": [ "#### Transform the model's input and output to the format compatible with OpenScale standard." ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "id": "e736b382-ca1a-4281-a64b-5e92c789d94c" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Payload data set id: bc1d0f0a-877d-422d-97d1-89a455ad0aa7\n" ] } ], "source": [ "import time\n", "\n", "time.sleep(5)\n", "payload_data_set_id = None\n", "payload_data_set_id = wos_client.data_sets.list(type=DataSetTypes.PAYLOAD_LOGGING, \n", " target_target_id=subscription_id, \n", " target_target_type=TargetTypes.SUBSCRIPTION).result.data_sets[0].metadata.id\n", "if payload_data_set_id is None:\n", " print(\"Payload data set not found. Please check subscription status.\")\n", "else:\n", " print(\"Payload data set id: \", payload_data_set_id)" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "id": "88146788-c6b5-4cdf-b145-617b74d73e87" }, "outputs": [], "source": [ "values = [float(s) for s in payload.split(',')]\n", "\n", "request_data = {'fields': feature_columns, \n", " 'values': values}\n", "\n", "response_data = {'fields': list(result['predictions'][0]),\n", " 'values': [list(x.values()) for x in result['predictions']]}" ] }, { "cell_type": "markdown", "metadata": { "id": "9705e93b-b5d4-47b7-950d-1768d9be26fb" }, "source": [ "#### Store the payload using Python SDK" ] }, { "cell_type": "markdown", "metadata": { "id": "0074022a-4407-4e2a-ba31-5dd556df9add" }, "source": [ "**Hint:** You can embed payload logging code into your custom deployment so it is logged automatically each time you score the model." ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "id": "c4f10e7e-d201-40cd-8efb-3cfd0f0218b2" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Performing explicit payload logging.....\n", "\n", "\n", "==========================================================================================\n", "\n", " Waiting for end of storing records with request id: 1f2cb328-d81e-4141-b02a-0d35c41871e1 \n", "\n", "==========================================================================================\n", "\n", "\n", "\n", "pending\n", "active\n", "\n", "---------------------------------------\n", " Successfully finished storing records \n", "---------------------------------------\n", "\n", "\n", "Number of records in the payload logging table: 1\n" ] } ], "source": [ "import uuid\n", "from ibm_watson_openscale.supporting_classes.payload_record import PayloadRecord\n", "\n", "print(\"Performing explicit payload logging.....\")\n", "wos_client.data_sets.store_records(data_set_id=payload_data_set_id, background_mode=False,request_body=[PayloadRecord(\n", " scoring_id=str(uuid.uuid4()),\n", " request=request_data,\n", " response=response_data,\n", " response_time=460\n", ")])\n", "time.sleep(5)\n", "pl_records_count = wos_client.data_sets.get_records_count(payload_data_set_id)\n", "print(\"Number of records in the payload logging table: {}\".format(pl_records_count))" ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "id": "9af476d8-8b2c-4ead-9731-47e140ed7d08" }, "outputs": [ { "data": { "text/html": [ "\n", " \n", "

Data Set bc1d0f0a-877d-422d-97d1-89a455ad0aa7 Records

\n", " \n", " \n", " \n", "
asset_revisionscoring_idpredicted_labelscoring_timestampprediction_probabilityscoreTOTAL_CASESREGIONdeployment_id
Nonec3594b77-285a-44f3-8a08-7eca514af7d3-102023-01-31T07:42:50.797Z0.8418549299240112[0.8418549299240112, 0.023631207644939423, 0.13451385498046875]100.00.0f87b94db698b412f3953cb42f15dbbaa
\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "wos_client.data_sets.show_records(data_set_id=payload_data_set_id)" ] }, { "cell_type": "markdown", "metadata": { "id": "4aa064bc-5ffe-42d9-82d0-ac2a194779a5" }, "source": [ "\n", "## 5. Feedback logging & quality (accuracy) monitoring" ] }, { "cell_type": "markdown", "metadata": { "id": "56281cbf-4e04-48f4-a020-69aed704acf5" }, "source": [ "### Enable quality monitoring" ] }, { "cell_type": "markdown", "metadata": { "id": "918e6572-cda9-4b9b-872c-bc0e4dcf9cb2" }, "source": [ "You need to provide the monitoring `threshold` and `min_records` (minimal number of feedback records)." ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "id": "83edd2a5-3ea2-4dbb-a20e-d9c0f22534c0" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "===================================================================================\n", "\n", " Waiting for end of monitor instance creation 1a4119ad-12f6-4ce9-a2ae-b14048a26656 \n", "\n", "===================================================================================\n", "\n", "\n", "\n", "active\n", "\n", "---------------------------------------\n", " Monitor instance successfully created \n", "---------------------------------------\n", "\n", "\n" ] } ], "source": [ "import time\n", "\n", "time.sleep(10)\n", "target = Target(\n", " target_type=TargetTypes.SUBSCRIPTION,\n", " target_id=subscription_id\n", ")\n", "parameters = {\n", " \"min_feedback_data_size\": 10\n", "}\n", "thresholds = [\n", " {\n", " \"metric_id\": \"area_under_roc\",\n", " \"type\": \"lower_limit\",\n", " \"value\": .80\n", " }\n", " ]\n", "quality_monitor_details = wos_client.monitor_instances.create(\n", " data_mart_id=data_mart_id,\n", " background_mode=False,\n", " monitor_definition_id=wos_client.monitor_definitions.MONITORS.QUALITY.ID,\n", " target=target,\n", " parameters=parameters,\n", " thresholds=thresholds\n", ").result" ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "id": "b1f80e2c-8152-4669-868a-69334090ecac" }, "outputs": [ { "data": { "text/plain": [ "'1a4119ad-12f6-4ce9-a2ae-b14048a26656'" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "quality_monitor_instance_id = quality_monitor_details.metadata.id\n", "quality_monitor_instance_id" ] }, { "cell_type": "markdown", "metadata": { "id": "2b8a5bc2-fa1a-4175-8692-a6a720fb1869" }, "source": [ "### Feedback records logging" ] }, { "cell_type": "markdown", "metadata": { "id": "d18ff39a-ad0b-48ea-a878-59cf36ba5510" }, "source": [ "Feedback records are used to evaluate your model. The predicted values are compared to real values (feedback records)." ] }, { "cell_type": "markdown", "metadata": { "id": "4076732f-a929-4c3a-98fa-ad4ce4af2bce" }, "source": [ "You can check the schema of feedback table using below method." ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "id": "5088583c-013c-4982-a238-ea7d7236019a" }, "outputs": [ { "data": { "text/plain": [ "'4c12c48a-e4ad-47d1-83bc-b6af0c5a7099'" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "feedback_dataset_id = None\n", "feedback_dataset = wos_client.data_sets.list(type=DataSetTypes.FEEDBACK, \n", " target_target_id=subscription_id, \n", " target_target_type=TargetTypes.SUBSCRIPTION).result\n", "feedback_dataset_id = feedback_dataset.data_sets[0].metadata.id\n", "if feedback_dataset_id is None:\n", " print(\"Feedback data set not found. Please check quality monitor status.\")\n", "feedback_dataset_id" ] }, { "cell_type": "markdown", "metadata": { "id": "240c773e-aada-4777-b219-f1ad2e789030" }, "source": [ "The feedback records can be send to feedback table using below code." ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "id": "1b0ae8d0-9d65-4988-bd2c-eee60cbf34c2" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/tmp/1000670000/ipykernel_213/3079094294.py:8: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.\n", "Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n", " data = pd.read_csv('https://ibm-aws-immersion-day.s3.us-east-2.amazonaws.com/publicdata/Data-region-RI-SM-Feedback.csv',header=0,dtype=np.float)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "==========================================================================================\n", "\n", " Waiting for end of storing records with request id: 280946b0-94e3-4e9c-b266-12c199504f33 \n", "\n", "==========================================================================================\n", "\n", "\n", "\n", "active\n", "\n", "---------------------------------------\n", " Successfully finished storing records \n", "---------------------------------------\n", "\n", "\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import requests\n", "import pandas as pd\n", "import numpy as np\n", "import time\n", "\n", "time.sleep(10) #It gives enough time for dataset creation\n", "\n", "data = pd.read_csv('https://ibm-aws-immersion-day.s3.us-east-2.amazonaws.com/publicdata/Data-region-RI-SM-Feedback.csv',header=0,dtype=np.float)\n", "feedback_columns = data.columns.tolist()\n", "feedback_records = data.values.tolist()\n", "\n", "payload_scoring = [{\"fields\": feedback_columns, \"values\": feedback_records}]\n", "wos_client.data_sets.store_records(feedback_dataset_id, request_body=payload_scoring, background_mode=False)" ] }, { "cell_type": "code", "execution_count": 36, "metadata": { "id": "4851e875-2da5-4abf-85a1-1d882bd099a9" }, "outputs": [ { "data": { "text/html": [ "\n", " \n", "

Schema of 4c12c48a-e4ad-47d1-83bc-b6af0c5a7099 data set

\n", " \n", " \n", " \n", "
nametypenullable
REGIONdoubleTrue
TOTAL_CASESdoubleTrue
RISK_INDEXdoubleTrue
record_idstringFalse
record_timestamptimestampFalse
transaction_idstringTrue
_original_predictionintegerTrue
_original_probability{'containsNull': True, 'elementType': 'double', 'type': 'array'}True
_debiased_predictionintegerTrue
_debiased_probability{'containsNull': True, 'elementType': 'double', 'type': 'array'}True
\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "wos_client.data_sets.print_records_schema(data_set_id=feedback_dataset_id)" ] }, { "cell_type": "code", "execution_count": 37, "metadata": { "id": "6a6f4849-8bc3-49bc-bc33-2a4d5a32b170" }, "outputs": [ { "data": { "text/plain": [ "12" ] }, "execution_count": 37, "metadata": {}, "output_type": "execute_result" } ], "source": [ "wos_client.data_sets.get_records_count(data_set_id=feedback_dataset_id)" ] }, { "cell_type": "code", "execution_count": 38, "metadata": { "id": "f13f9028-baec-4012-aea7-6e1f8e2a5d3d" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "========================================================================\n", "\n", " Waiting for end of monitoring run ae938061-3a37-4c27-8b03-adc7c9239110 \n", "\n", "========================================================================\n", "\n", "\n", "\n", "running\n", "finished\n", "\n", "---------------------------\n", " Successfully finished run \n", "---------------------------\n", "\n", "\n" ] } ], "source": [ "run_details = wos_client.monitor_instances.run(monitor_instance_id=quality_monitor_instance_id, background_mode=False).result" ] }, { "cell_type": "code", "execution_count": 39, "metadata": { "id": "917b7ae9-6a6b-4a27-a4fd-83d0adfd2429" }, "outputs": [ { "data": { "text/html": [ "\n", " \n", "

1a4119ad-12f6-4ce9-a2ae-b14048a26656 Monitor Runs Metrics from: 2023-01-24 07:43:56.818707 till: 2023-01-31 07:43:56.818720

\n", " \n", " \n", " \n", "
tsidmeasurement_idvaluelower_limitupper_limittagsmonitor_definition_idmonitor_instance_idrun_idtarget_typetarget_id
2023-01-31 07:43:40.257000+00:00accuracy90ef22f8-af95-41a7-9552-6b048ddc3b740.8333333333333334NoneNone['model_type:original']quality1a4119ad-12f6-4ce9-a2ae-b14048a26656ae938061-3a37-4c27-8b03-adc7c9239110subscriptionf26582b5-3112-4f0f-b00b-64cc07d0f6fa
2023-01-31 07:43:40.257000+00:00weighted_true_positive_rate90ef22f8-af95-41a7-9552-6b048ddc3b740.8333333333333333NoneNone['model_type:original']quality1a4119ad-12f6-4ce9-a2ae-b14048a26656ae938061-3a37-4c27-8b03-adc7c9239110subscriptionf26582b5-3112-4f0f-b00b-64cc07d0f6fa
2023-01-31 07:43:40.257000+00:00weighted_precision90ef22f8-af95-41a7-9552-6b048ddc3b740.888888888888889NoneNone['model_type:original']quality1a4119ad-12f6-4ce9-a2ae-b14048a26656ae938061-3a37-4c27-8b03-adc7c9239110subscriptionf26582b5-3112-4f0f-b00b-64cc07d0f6fa
2023-01-31 07:43:40.257000+00:00log_loss90ef22f8-af95-41a7-9552-6b048ddc3b740.7617815096623222NoneNone['model_type:original']quality1a4119ad-12f6-4ce9-a2ae-b14048a26656ae938061-3a37-4c27-8b03-adc7c9239110subscriptionf26582b5-3112-4f0f-b00b-64cc07d0f6fa
2023-01-31 07:43:40.257000+00:00weighted_recall90ef22f8-af95-41a7-9552-6b048ddc3b740.8333333333333333NoneNone['model_type:original']quality1a4119ad-12f6-4ce9-a2ae-b14048a26656ae938061-3a37-4c27-8b03-adc7c9239110subscriptionf26582b5-3112-4f0f-b00b-64cc07d0f6fa
2023-01-31 07:43:40.257000+00:00weighted_f_measure90ef22f8-af95-41a7-9552-6b048ddc3b740.837037037037037NoneNone['model_type:original']quality1a4119ad-12f6-4ce9-a2ae-b14048a26656ae938061-3a37-4c27-8b03-adc7c9239110subscriptionf26582b5-3112-4f0f-b00b-64cc07d0f6fa
2023-01-31 07:43:40.257000+00:00weighted_false_positive_rate90ef22f8-af95-41a7-9552-6b048ddc3b740.08333333333333333NoneNone['model_type:original']quality1a4119ad-12f6-4ce9-a2ae-b14048a26656ae938061-3a37-4c27-8b03-adc7c9239110subscriptionf26582b5-3112-4f0f-b00b-64cc07d0f6fa
\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "time.sleep(5)\n", "wos_client.monitor_instances.show_metrics(monitor_instance_id=quality_monitor_instance_id)" ] }, { "cell_type": "markdown", "metadata": { "id": "e8da7566-a9ce-4839-a653-b779f810636e" }, "source": [ "\n", "## 6. Get the logged data" ] }, { "cell_type": "markdown", "metadata": { "id": "0cfcfeca-03ac-4177-a23a-2a9a559b0358" }, "source": [ "### Payload logging" ] }, { "cell_type": "markdown", "metadata": { "id": "b504a117-b371-4ca4-acc6-ed138ce0f35e" }, "source": [ "#### Print schema of payload_logging table" ] }, { "cell_type": "code", "execution_count": 40, "metadata": { "id": "5b0b0e39-b19b-477d-b2fb-1056f2afaf1d" }, "outputs": [ { "data": { "text/html": [ "\n", " \n", "

Schema of bc1d0f0a-877d-422d-97d1-89a455ad0aa7 data set

\n", " \n", " \n", " \n", "
nametypenullable
scoring_idstringFalse
scoring_timestamptimestampFalse
deployment_idstringFalse
asset_revisionstringTrue
REGIONdoubleTrue
TOTAL_CASESdoubleTrue
score{'containsNull': True, 'elementType': 'double', 'type': 'array'}True
predicted_labelintegerTrue
prediction_probabilitydoubleTrue
\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "wos_client.data_sets.print_records_schema(data_set_id=payload_data_set_id)" ] }, { "cell_type": "markdown", "metadata": { "id": "a7d3bdd2-68a5-45b3-89ee-6507eebcc704" }, "source": [ "\n", "## 7. Fairness, Drift monitoring and explanations" ] }, { "cell_type": "markdown", "metadata": { "id": "2fdf5f1e-a892-45e8-89fb-a96655a88767" }, "source": [ "### Get payload data" ] }, { "cell_type": "code", "execution_count": 41, "metadata": { "id": "bcb445f6-f77c-458d-b5be-bd51c9f9ff32" }, "outputs": [], "source": [ "scoring_data_filename='Data-region-RI-SM-Scoring.csv'\n", "scoring_data_filename_json='Data-region-RI-SM-Scoring.json'" ] }, { "cell_type": "code", "execution_count": 42, "metadata": { "id": "cb7481fb90ee4ddb8ec6a5854b6444d9" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "rm: cannot remove 'Data-region-RI-SM-Scoring.csv': No such file or directory\n", "rm: cannot remove 'Data-region-RI-SM-Scoring.json': No such file or directory\n", "--2023-01-31 07:43:58-- https://ibm-aws-immersion-day.s3.us-east-2.amazonaws.com/publicdata/Data-region-RI-SM-Scoring.json\n", "Resolving ibm-aws-immersion-day.s3.us-east-2.amazonaws.com (ibm-aws-immersion-day.s3.us-east-2.amazonaws.com)... 52.219.94.122\n", "Connecting to ibm-aws-immersion-day.s3.us-east-2.amazonaws.com (ibm-aws-immersion-day.s3.us-east-2.amazonaws.com)|52.219.94.122|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 106 [application/json]\n", "Saving to: ‘Data-region-RI-SM-Scoring.json’\n", "\n", "Data-region-RI-SM-S 100%[===================>] 106 --.-KB/s in 0s \n", "\n", "2023-01-31 07:43:58 (3.85 MB/s) - ‘Data-region-RI-SM-Scoring.json’ saved [106/106]\n", "\n", "--2023-01-31 07:43:59-- https://ibm-aws-immersion-day.s3.us-east-2.amazonaws.com/publicdata/Data-region-RI-SM-Scoring.csv\n", "Resolving ibm-aws-immersion-day.s3.us-east-2.amazonaws.com (ibm-aws-immersion-day.s3.us-east-2.amazonaws.com)... 52.219.94.122\n", "Connecting to ibm-aws-immersion-day.s3.us-east-2.amazonaws.com (ibm-aws-immersion-day.s3.us-east-2.amazonaws.com)|52.219.94.122|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 94 [text/csv]\n", "Saving to: ‘Data-region-RI-SM-Scoring.csv’\n", "\n", "Data-region-RI-SM-S 100%[===================>] 94 --.-KB/s in 0s \n", "\n", "2023-01-31 07:43:59 (3.21 MB/s) - ‘Data-region-RI-SM-Scoring.csv’ saved [94/94]\n", "\n" ] } ], "source": [ "!rm Data-region-RI-SM-Scoring.csv\n", "!rm Data-region-RI-SM-Scoring.json\n", "!wget \"https://ibm-aws-immersion-day.s3.us-east-2.amazonaws.com/publicdata/Data-region-RI-SM-Scoring.json\"\n", "!wget \"https://ibm-aws-immersion-day.s3.us-east-2.amazonaws.com/publicdata/Data-region-RI-SM-Scoring.csv\"" ] }, { "cell_type": "code", "execution_count": 43, "metadata": { "id": "f7b1c646f0ab444d828d0ac7ba39538f" }, "outputs": [], "source": [ "data=pd.read_csv('https://ibm-aws-immersion-day.s3.us-east-2.amazonaws.com/publicdata/Data-region-RI-SM-Scoring.csv')" ] }, { "cell_type": "code", "execution_count": 44, "metadata": { "id": "cb0f95d405a04f44a9f14f0b01d42045" }, "outputs": [], "source": [ "import io\n", "from io import StringIO\n", "csv_file = io.StringIO()\n", "# by default sagemaker expects comma seperated\n", "data.to_csv(csv_file, sep=\",\", header=False, index=False)\n", "scoring_data_filename = csv_file.getvalue()" ] }, { "cell_type": "code", "execution_count": 45, "metadata": { "id": "98ccab660c7743a99c8ae5694326551a" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"predictions\": [\n", " {\n", " \"score\": [\n", " 0.02799299918115139,\n", " 0.7118191719055176,\n", " 0.2601878345012665\n", " ],\n", " \"predicted_label\": 1\n", " },\n", " {\n", " \"score\": [\n", " 0.8067101836204529,\n", " 0.02732161432504654,\n", " 0.16596820950508118\n", " ],\n", " \"predicted_label\": 0\n", " },\n", " {\n", " \"score\": [\n", " 0.010750409215688705,\n", " 0.19208674132823944,\n", " 0.7971628308296204\n", " ],\n", " \"predicted_label\": 2\n", " },\n", " {\n", " \"score\": [\n", " 0.03177503123879433,\n", " 0.7168406248092651,\n", " 0.25138434767723083\n", " ],\n", " \"predicted_label\": 1\n", " },\n", " {\n", " \"score\": [\n", " 0.7829863429069519,\n", " 0.029666010290384293,\n", " 0.18734769523143768\n", " ],\n", " \"predicted_label\": 0\n", " },\n", " {\n", " \"score\": [\n", " 0.060710735619068146,\n", " 0.26483166217803955,\n", " 0.6744576096534729\n", " ],\n", " \"predicted_label\": 2\n", " },\n", " {\n", " \"score\": [\n", " 0.041998133063316345,\n", " 0.7258813977241516,\n", " 0.23212049901485443\n", " ],\n", " \"predicted_label\": 1\n", " },\n", " {\n", " \"score\": [\n", " 0.3273279070854187,\n", " 0.27560198307037354,\n", " 0.39707013964653015\n", " ],\n", " \"predicted_label\": 2\n", " },\n", " {\n", " \"score\": [\n", " 0.05508704483509064,\n", " 0.7314295172691345,\n", " 0.21348345279693604\n", " ],\n", " \"predicted_label\": 1\n", " },\n", " {\n", " \"score\": [\n", " 0.7493403553962708,\n", " 0.03281030058860779,\n", " 0.21784931421279907\n", " ],\n", " \"predicted_label\": 0\n", " },\n", " {\n", " \"score\": [\n", " 0.826548159122467,\n", " 0.025272492319345474,\n", " 0.1481793373823166\n", " ],\n", " \"predicted_label\": 0\n", " }\n", " ]\n", "}\n" ] } ], "source": [ "sm_runtime = boto3.client('sagemaker-runtime',\n", " region_name=SAGEMAKER_ENGINE_CREDENTIALS['region'],\n", " aws_access_key_id=SAGEMAKER_ENGINE_CREDENTIALS['access_key_id'],\n", " aws_secret_access_key=SAGEMAKER_ENGINE_CREDENTIALS['secret_access_key'])\n", "\n", "\n", "scoring_response = sm_runtime.invoke_endpoint(EndpointName = endpoint_name,\n", " ContentType = 'text/csv',\n", " Body = scoring_data_filename)\n", " \n", "result = json.loads(scoring_response['Body'].read().decode())\n", "print(json.dumps(result, indent=2))" ] }, { "cell_type": "code", "execution_count": 46, "metadata": { "id": "6da9de22-6714-42c6-a65d-173e7f4c64d9" }, "outputs": [], "source": [ "f = open(scoring_data_filename_json,\"r\")\n", "payload_values = json.load(f)\n", "request_data = {'fields': feature_columns, \n", " 'values': payload_values}\n", "\n", "response_data = {'fields': list(result['predictions'][0]),\n", " 'values': [list(x.values()) for x in result['predictions']]}\n" ] }, { "cell_type": "code", "execution_count": 47, "metadata": { "id": "6b2162d3-9d90-44a2-9426-23839cc02071" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Performing explicit payload logging.....\n", "Number of records in the payload logging table: 13\n" ] } ], "source": [ "import uuid\n", "from ibm_watson_openscale.supporting_classes.payload_record import PayloadRecord\n", "\n", "print(\"Performing explicit payload logging.....\")\n", "wos_client.data_sets.store_records(data_set_id=payload_data_set_id, request_body=[PayloadRecord(\n", " scoring_id=str(uuid.uuid4()),\n", " request=request_data,\n", " response=response_data,\n", " response_time=460\n", ")])\n", "time.sleep(5)\n", "pl_records_count = wos_client.data_sets.get_records_count(payload_data_set_id)\n", "print(\"Number of records in the payload logging table: {}\".format(pl_records_count))" ] }, { "cell_type": "code", "execution_count": 48, "metadata": { "id": "e0b7bc5d-ddd1-4e7d-8e5a-14ea4192eaf1" }, "outputs": [ { "data": { "text/html": [ "\n", " \n", "

Data Set bc1d0f0a-877d-422d-97d1-89a455ad0aa7 Records

\n", " \n", " \n", " \n", "
asset_revisionscoring_idpredicted_labelscoring_timestampprediction_probabilityscoreTOTAL_CASESREGIONdeployment_id
None84164fa7-4f67-4884-be52-0e419970ef89-11.02023-01-31T07:43:59.979Z0.7118191719055176[0.02799299918115139, 0.7118191719055176, 0.2601878345012665]10575.01.0f87b94db698b412f3953cb42f15dbbaa
None84164fa7-4f67-4884-be52-0e419970ef89-100.02023-01-31T07:43:59.979Z0.7493403553962708[0.7493403553962708, 0.03281030058860779, 0.21784931421279907]200.02.0f87b94db698b412f3953cb42f15dbbaa
None84164fa7-4f67-4884-be52-0e419970ef89-110.02023-01-31T07:43:59.979Z0.826548159122467[0.826548159122467, 0.025272492319345474, 0.1481793373823166]1400.00.0f87b94db698b412f3953cb42f15dbbaa
None84164fa7-4f67-4884-be52-0e419970ef89-12nan2023-01-31T07:43:59.979Znan[]350.00.0f87b94db698b412f3953cb42f15dbbaa
None84164fa7-4f67-4884-be52-0e419970ef89-20.02023-01-31T07:43:59.979Z0.8067101836204529[0.8067101836204529, 0.02732161432504654, 0.16596820950508118]2100.02.0f87b94db698b412f3953cb42f15dbbaa
None84164fa7-4f67-4884-be52-0e419970ef89-32.02023-01-31T07:43:59.979Z0.7971628308296204[0.010750409215688705, 0.19208674132823944, 0.7971628308296204]649.00.0f87b94db698b412f3953cb42f15dbbaa
None84164fa7-4f67-4884-be52-0e419970ef89-41.02023-01-31T07:43:59.979Z0.7168406248092651[0.03177503123879433, 0.7168406248092651, 0.25138434767723083]10023.01.0f87b94db698b412f3953cb42f15dbbaa
None84164fa7-4f67-4884-be52-0e419970ef89-50.02023-01-31T07:43:59.979Z0.7829863429069519[0.7829863429069519, 0.029666010290384293, 0.18734769523143768]1750.02.0f87b94db698b412f3953cb42f15dbbaa
None84164fa7-4f67-4884-be52-0e419970ef89-62.02023-01-31T07:43:59.979Z0.6744576096534729[0.060710735619068146, 0.26483166217803955, 0.6744576096534729]977.00.0f87b94db698b412f3953cb42f15dbbaa
None84164fa7-4f67-4884-be52-0e419970ef89-71.02023-01-31T07:43:59.979Z0.7258813977241516[0.041998133063316345, 0.7258813977241516, 0.23212049901485443]5900.01.0f87b94db698b412f3953cb42f15dbbaa
\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "wos_client.data_sets.show_records(payload_data_set_id)" ] }, { "cell_type": "markdown", "metadata": { "id": "11a99b20-05d3-4725-8531-363ea5906f29" }, "source": [ "### Enable and run fairness monitoring" ] }, { "cell_type": "code", "execution_count": 49, "metadata": { "id": "d9e3bd57-5661-452b-9d99-c30773c597f5" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "===================================================================================\n", "\n", " Waiting for end of monitor instance creation fefa92dc-2985-43e9-b132-23b670c2043b \n", "\n", "===================================================================================\n", "\n", "\n", "\n", "active\n", "\n", "---------------------------------------\n", " Monitor instance successfully created \n", "---------------------------------------\n", "\n", "\n" ] }, { "data": { "text/plain": [ "'fefa92dc-2985-43e9-b132-23b670c2043b'" ] }, "execution_count": 49, "metadata": {}, "output_type": "execute_result" } ], "source": [ "target = Target(\n", " target_type=TargetTypes.SUBSCRIPTION,\n", " target_id=subscription_id\n", "\n", ")\n", "parameters = {\n", " \"features\": [\n", " {\"feature\": \"REGION\",\n", " \"majority\": [[1,1]],\n", " \"minority\": [[0,0]],\n", " \"threshold\": 0.95\n", " }\n", " ],\n", " \"favourable_class\": [0],\n", " \"unfavourable_class\": [2],\n", " \"min_records\": 10\n", "}\n", "\n", "fairness_monitor_details = wos_client.monitor_instances.create(\n", " data_mart_id=data_mart_id,\n", " background_mode=False,\n", " monitor_definition_id=wos_client.monitor_definitions.MONITORS.FAIRNESS.ID,\n", " target=target,\n", " parameters=parameters).result\n", "fairness_monitor_instance_id =fairness_monitor_details.metadata.id\n", "fairness_monitor_instance_id" ] }, { "cell_type": "markdown", "metadata": { "id": "aca0c5f9-cacc-457c-9e5a-a5814e3a96e9" }, "source": [ "### Run fairness monitor" ] }, { "cell_type": "code", "execution_count": 50, "metadata": { "id": "e4e9ecd0-52cc-4253-bacd-786144006fd0" }, "outputs": [ { "data": { "text/html": [ "\n", " \n", "

fefa92dc-2985-43e9-b132-23b670c2043b Monitor Runs Metrics from: 2023-01-24 07:44:30.775333 till: 2023-01-31 07:44:30.775348

\n", " \n", " \n", " \n", "
tsidmeasurement_idvaluelower_limitupper_limittagsmonitor_definition_idmonitor_instance_idrun_idtarget_typetarget_id
2023-01-31 07:44:15.913442+00:00fairness_valuef6441a45-0451-4c97-8b96-b61ee598e2ae100.080.0None['feature:REGION', 'fairness_metric_type:fairness', 'feature_value:0-0']fairnessfefa92dc-2985-43e9-b132-23b670c2043b44e7591d-f300-4ae4-9f4c-6b77980755e3subscriptionf26582b5-3112-4f0f-b00b-64cc07d0f6fa
\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "time.sleep(20)\n", "#Note: When you create fairness monitor, initial run is also created\n", "wos_client.monitor_instances.show_metrics(monitor_instance_id=fairness_monitor_instance_id)" ] }, { "cell_type": "markdown", "metadata": { "id": "a7975b8f-ce14-4c70-818c-e1b622005f02" }, "source": [ "### Enable and run Drift monitoring" ] }, { "cell_type": "markdown", "metadata": { "id": "10145390-cedd-460d-aec0-787ada2d9c4c" }, "source": [ "#### We will have to upload the model manually in the OpenScale UI for setting up Drift Metrics" ] }, { "cell_type": "markdown", "metadata": { "id": "9a1b191c-3fbe-4273-a1f0-926dc78674f1" }, "source": [ "### Enable Explainability and run explanation on sample record. " ] }, { "cell_type": "code", "execution_count": 51, "metadata": { "id": "f3111eff-6c2a-458a-b5ad-bfd1c3703499" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "===================================================================================\n", "\n", " Waiting for end of monitor instance creation f0f59f48-8959-4904-99de-900380e9bbb8 \n", "\n", "===================================================================================\n", "\n", "\n", "\n", "preparing\n", "active\n", "\n", "---------------------------------------\n", " Monitor instance successfully created \n", "---------------------------------------\n", "\n", "\n" ] } ], "source": [ "target = Target(\n", " target_type=TargetTypes.SUBSCRIPTION,\n", " target_id=subscription_id\n", ")\n", "parameters = {\n", " \"enabled\": True\n", "}\n", "explainability_details = wos_client.monitor_instances.create(\n", " data_mart_id=data_mart_id,\n", " background_mode=False,\n", " monitor_definition_id=wos_client.monitor_definitions.MONITORS.EXPLAINABILITY.ID,\n", " target=target,\n", " parameters=parameters\n", ").result" ] }, { "cell_type": "markdown", "metadata": { "id": "d2fefc3f-3d85-47fe-a107-ee23b429e595" }, "source": [ "Getting a `transaction_id` to run explanation on" ] }, { "cell_type": "code", "execution_count": 52, "metadata": { "id": "7043112c-755f-41d4-bdf7-48fba0e0396e" }, "outputs": [ { "data": { "text/plain": [ "'f0f59f48-8959-4904-99de-900380e9bbb8'" ] }, "execution_count": 52, "metadata": {}, "output_type": "execute_result" } ], "source": [ "explainability_monitor_id = explainability_details.metadata.id\n", "explainability_monitor_id" ] }, { "cell_type": "code", "execution_count": 53, "metadata": { "id": "dab932e7-41ce-413c-b56d-49163dd64ae8" }, "outputs": [ { "data": { "text/html": [ "\n", " \n", "

Data Set bc1d0f0a-877d-422d-97d1-89a455ad0aa7 Records

\n", " \n", " \n", " \n", "
asset_revisionscoring_idpredicted_labelscoring_timestampprediction_probabilityscoreTOTAL_CASESREGIONdeployment_id
None84164fa7-4f67-4884-be52-0e419970ef89-11.02023-01-31T07:43:59.979Z0.7118191719055176[0.02799299918115139, 0.7118191719055176, 0.2601878345012665]10575.01.0f87b94db698b412f3953cb42f15dbbaa
None84164fa7-4f67-4884-be52-0e419970ef89-100.02023-01-31T07:43:59.979Z0.7493403553962708[0.7493403553962708, 0.03281030058860779, 0.21784931421279907]200.02.0f87b94db698b412f3953cb42f15dbbaa
None84164fa7-4f67-4884-be52-0e419970ef89-110.02023-01-31T07:43:59.979Z0.826548159122467[0.826548159122467, 0.025272492319345474, 0.1481793373823166]1400.00.0f87b94db698b412f3953cb42f15dbbaa
None84164fa7-4f67-4884-be52-0e419970ef89-12nan2023-01-31T07:43:59.979Znan[]350.00.0f87b94db698b412f3953cb42f15dbbaa
None84164fa7-4f67-4884-be52-0e419970ef89-20.02023-01-31T07:43:59.979Z0.8067101836204529[0.8067101836204529, 0.02732161432504654, 0.16596820950508118]2100.02.0f87b94db698b412f3953cb42f15dbbaa
None84164fa7-4f67-4884-be52-0e419970ef89-32.02023-01-31T07:43:59.979Z0.7971628308296204[0.010750409215688705, 0.19208674132823944, 0.7971628308296204]649.00.0f87b94db698b412f3953cb42f15dbbaa
None84164fa7-4f67-4884-be52-0e419970ef89-41.02023-01-31T07:43:59.979Z0.7168406248092651[0.03177503123879433, 0.7168406248092651, 0.25138434767723083]10023.01.0f87b94db698b412f3953cb42f15dbbaa
None84164fa7-4f67-4884-be52-0e419970ef89-50.02023-01-31T07:43:59.979Z0.7829863429069519[0.7829863429069519, 0.029666010290384293, 0.18734769523143768]1750.02.0f87b94db698b412f3953cb42f15dbbaa
None84164fa7-4f67-4884-be52-0e419970ef89-62.02023-01-31T07:43:59.979Z0.6744576096534729[0.060710735619068146, 0.26483166217803955, 0.6744576096534729]977.00.0f87b94db698b412f3953cb42f15dbbaa
None84164fa7-4f67-4884-be52-0e419970ef89-71.02023-01-31T07:43:59.979Z0.7258813977241516[0.041998133063316345, 0.7258813977241516, 0.23212049901485443]5900.01.0f87b94db698b412f3953cb42f15dbbaa
None84164fa7-4f67-4884-be52-0e419970ef89-82.02023-01-31T07:43:59.979Z0.39707013964653015[0.3273279070854187, 0.27560198307037354, 0.39707013964653015]971.02.0f87b94db698b412f3953cb42f15dbbaa
None84164fa7-4f67-4884-be52-0e419970ef89-91.02023-01-31T07:43:59.979Z0.7314295172691345[0.05508704483509064, 0.7314295172691345, 0.21348345279693604]1090.01.0f87b94db698b412f3953cb42f15dbbaa
Nonec3594b77-285a-44f3-8a08-7eca514af7d3-10.02023-01-31T07:42:50.797Z0.8418549299240112[0.8418549299240112, 0.023631207644939423, 0.13451385498046875]100.00.0f87b94db698b412f3953cb42f15dbbaa
\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "wos_client.data_sets.show_records(data_set_id=payload_data_set_id,limit=50)" ] }, { "cell_type": "code", "execution_count": 54, "metadata": { "id": "7f69deca-872d-4ca6-b4fa-4c642c949df0" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Running explanations on scoring IDs: ['84164fa7-4f67-4884-be52-0e419970ef89-1']\n", "------------------------------------------------------------------------\n", "| Warning: |\n", "| After November 2022, the explanation_tasks method must include the |\n", "| subscription_id parameter for future IBM Watson OpenScale releases. |\n", "------------------------------------------------------------------------\n", "{\n", " \"metadata\": {\n", " \"explanation_task_ids\": [\n", " \"904a9081-c474-42c9-bacc-433d4161cfdf\"\n", " ],\n", " \"created_by\": \"1000330999\",\n", " \"created_at\": \"2023-01-31T07:44:41.701637Z\"\n", " }\n", "}\n" ] }, { "data": { "text/plain": [ "'904a9081-c474-42c9-bacc-433d4161cfdf'" ] }, "execution_count": 54, "metadata": {}, "output_type": "execute_result" } ], "source": [ "payload_data = wos_client.data_sets.get_list_of_records(limit=1,data_set_id=payload_data_set_id,output_type='pandas').result\n", "scoring_ids=payload_data['scoring_id'].tolist()\n", "print(\"Running explanations on scoring IDs: {}\".format(scoring_ids))\n", "explanation_types = [\"lime\", \"contrastive\"]\n", "result = wos_client.monitor_instances.explanation_tasks(scoring_ids=scoring_ids, explanation_types=explanation_types).result\n", "print(result)\n", "explanation_task_id=result.to_dict()['metadata']['explanation_task_ids'][0]\n", "explanation_task_id" ] }, { "cell_type": "code", "execution_count": 55, "metadata": { "id": "97976891-089b-4b3a-adcd-fc8bc55d42e5" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "----------------------------------------------------------------------------\n", "| Warning: |\n", "| After November 2022, the get_explanation_tasks method must include the |\n", "| subscription_id parameter for future IBM Watson OpenScale releases. |\n", "----------------------------------------------------------------------------\n" ] }, { "data": { "text/plain": [ "{'metadata': {'explanation_task_id': '904a9081-c474-42c9-bacc-433d4161cfdf',\n", " 'created_by': '1000330999',\n", " 'created_at': '2023-01-31T07:44:41.701637Z'},\n", " 'entity': {'status': {'state': 'in_progress'},\n", " 'scoring_id': '84164fa7-4f67-4884-be52-0e419970ef89-1'}}" ] }, "execution_count": 55, "metadata": {}, "output_type": "execute_result" } ], "source": [ "wos_client.monitor_instances.get_explanation_tasks(explanation_task_id=explanation_task_id).result.to_dict()" ] }, { "cell_type": "markdown", "metadata": { "id": "e8efa91e-375f-4eb1-b137-249c30787f9f" }, "source": [ "You can now view the [OpenScale Dashboard](https://cpd-zen-46.apps.cp-deployer46-d01.ibmworkshops.com/aiopenscale/insights?serviceInstanceNamespace=zen-46&serviceInstanceDisplayName=openscale-defaultinstance). Click on the tile for the Risk Index AWS model to see Fairness, Explainability, Accuracy, and Performance monitors. " ] }, { "cell_type": "markdown", "metadata": { "id": "df62295c-bd84-49e1-9d54-fe0d1367ef64" }, "source": [ "### In this Notebook, we have learnt how to configure the metrics for monitoring the SageMaker model using Watson OpenScale. Fairness, Quality & Explainability metrics have been programmatically configuered using this notebook. Drift model has to be configuered manually using the steps listed in the documentation. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "92bd4e099cfe4b5bbf94c7f7be6cfe32" }, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "0419824b50d6491f8bde9216b7fdc32f" }, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": { "id": "a233fab5-3712-4fe9-988e-0a3ddc379d9f" }, "source": [ "---" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.6" }, "varInspector": { "cols": { "lenName": 16, "lenType": 16, "lenVar": 40 }, "kernels_config": { "python": { "delete_cmd_postfix": "", "delete_cmd_prefix": "del ", "library": "var_list.py", "varRefreshCmd": "print(var_dic_list())" }, "r": { "delete_cmd_postfix": ") ", "delete_cmd_prefix": "rm(", "library": "var_list.r", "varRefreshCmd": "cat(var_dic_list()) " } }, "types_to_exclude": [ "module", "function", "builtin_function_or_method", "instance", "_Feature" ], "window_display": false } }, "nbformat": 4, "nbformat_minor": 1 }