{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "THIS EXERCISE IS BEING DEVELOPED (will be added before Wrap-up Section)\n", "\n", "# 3a. Exercise\n", "\n", "### >>> Your turn (challenge)! \n", "\n", "- Add some other metrics to the evaluation report (from [Scikit-Learn](https://scikit-learn.org/stable/modules/model_evaluation.html#classification-metrics)) \n", "\n", "- Create plots and save them in reports to be viewed inside Studio!\n", "\n", "We will use this to show the metrics inside the Pipeline.\n", "\n", "It will be something like this (but for the Processing):\n", "\n", "\"studio-plots.png\"\n", "\n", "https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines-studio-view-execution.html" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# YOUR SOLUTION HERE\n", "\n", "\n", "\n", "\n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you've completed the challenge exercise above, let's load and update the the evaluation code in S3:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you'd like to save some plots, [check the hints and solution here!](./solutions/a-hint.md)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%store -r s3_evaluation_code_uri_with_experiments" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Update the `s3_evaluation_code_uri` variable to point to the new script\n", "s3_evaluation_code_uri = s3_evaluation_code_uri_with_experiments" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Let's store the S3 URI where our evaluation script was saved for later" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%store s3_evaluation_code_uri" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Solution - Exercise 3a.\n", "\n", "If you want, just run the cells below:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%writefile evaluate_with_experiments.py\n", "\"\"\"Evaluation script for measuring model accuracy.\"\"\"\n", "import argparse, os, subprocess, sys\n", "import json\n", "import os\n", "import tarfile\n", "import logging\n", "import pickle\n", "\n", "import pandas as pd\n", "import xgboost\n", "\n", "\n", "logger = logging.getLogger()\n", "logger.setLevel(logging.INFO)\n", "logger.addHandler(logging.StreamHandler())\n", "\n", "# May need to import additional metrics depending on what you are measuring.\n", "# See https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-model-quality-metrics.html\n", "from sklearn.metrics import classification_report, roc_auc_score, accuracy_score\n", "\n", "\n", "def pip_install(package):\n", " logger.info(f\"Pip installing `{package}`\")\n", " subprocess.call([sys.executable, \"-m\", \"pip\", \"install\", package])\n", "\n", "\n", "if __name__ == \"__main__\":\n", " pip_install(\"sagemaker-experiments==0.1.31\")\n", " \n", " # Instantiate SM Experiment Tracker\n", " from smexperiments.tracker import Tracker\n", " tracker = Tracker.load()\n", " \n", " \n", " model_path = \"/opt/ml/processing/model/model.tar.gz\"\n", " with tarfile.open(model_path) as tar:\n", " tar.extractall(path=\"..\")\n", "\n", " logger.debug(\"Loading xgboost model.\")\n", " model = pickle.load(open(\"xgboost-model\", \"rb\"))\n", "\n", " logger.info(\"Loading test input data\")\n", " test_path = \"/opt/ml/processing/test/test-dataset.csv\"\n", " df = pd.read_csv(test_path, header=None)\n", "\n", " logger.debug(\"Reading test data.\")\n", " y_test = df.iloc[:, 0].to_numpy()\n", " df.drop(df.columns[0], axis=1, inplace=True)\n", " X_test = xgboost.DMatrix(df.values)\n", "\n", " logger.info(\"Performing predictions against test data.\")\n", " predictions_probs = model.predict(X_test)\n", " predictions = predictions_probs.round()\n", "\n", " logger.info(\"Creating classification evaluation report\")\n", " acc = accuracy_score(y_test, predictions)\n", " auc = roc_auc_score(y_test, predictions_probs)\n", "\n", " # The metrics reported can change based on the model used, but it must be a specific name per (https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-model-quality-metrics.html)\n", " report_dict = {\n", " \"binary_classification_metrics\": {\n", " \"accuracy\": {\n", " \"value\": acc,\n", " \"standard_deviation\": \"NaN\",\n", " },\n", " \"auc\": {\"value\": auc, \"standard_deviation\": \"NaN\"},\n", " },\n", " }\n", "\n", " logger.info(\"Classification report:\\n{}\".format(report_dict))\n", "\n", " evaluation_output_path = os.path.join(\n", " \"/opt/ml/processing/evaluation\", \"evaluation.json\"\n", " )\n", " logger.info(\"Saving classification report to {}\".format(evaluation_output_path))\n", "\n", " with open(evaluation_output_path, \"w\") as f:\n", " f.write(json.dumps(report_dict))\n", " \n", " logger.info(\"Creating and logging plots to Studio\")\n", " tracker.log_precision_recall(y_test, predictions_probs, title=\"Precision-recall for predicting Churn\", output_artifact=True)\n", " tracker.log_roc_curve(y_test, predictions_probs, title=\"ROC Curve for predicting Churn\", output_artifact=True)\n", " tracker.log_confusion_matrix(y_test, predictions, title=\"Confusion matrix for predicting Churn\", output_artifact=True)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Observe that we called the following methods after instantiating a `tracker` object:\n", "- `tracker.log_precision_recall(...)`\n", "- `tracker.log_roc_curve(...)`\n", "- `tracker.log_confusion_matrix(...)`\n", " \n", "These plots will be visible in the 6-Pipelines lab! " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%store -r docker_image_name\n", "%store -r s3uri_model\n", "%store -r s3url_test" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "entrypoint = \"evaluate_with_experiments.py\"" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sagemaker.processing import (\n", " ProcessingInput,\n", " ProcessingOutput,\n", " ScriptProcessor,\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import sagemaker\n", "sm_sess = sagemaker.session.Session()\n", "role = sagemaker.get_execution_role()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Processing step for evaluation\n", "processor = ScriptProcessor(\n", " image_uri=docker_image_name,\n", " command=[\"python3\"],\n", " instance_type=\"ml.m5.xlarge\",\n", " instance_count=1,\n", " base_job_name=\"CustomerChurn/eval-script\",\n", " sagemaker_session=sm_sess,\n", " role=role,\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from time import strftime, gmtime\n", "# Helper to create timestamps\n", "create_date = lambda: strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "processor.run(\n", " code=entrypoint,\n", " inputs=[\n", " sagemaker.processing.ProcessingInput(\n", " source=s3uri_model,\n", " destination=\"/opt/ml/processing/model\",\n", " ),\n", " sagemaker.processing.ProcessingInput(\n", " source=s3url_test,\n", " destination=\"/opt/ml/processing/test\",\n", " ),\n", " ],\n", " outputs=[\n", " sagemaker.processing.ProcessingOutput(\n", " output_name=\"evaluation\", source=\"/opt/ml/processing/evaluation\"\n", " ),\n", " ],\n", " job_name=f\"Experiments-CustomerChurnEval-{create_date()}\"\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's get the S3 URI location for the new evaluation python script:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "for proc_in in processor.latest_job.inputs:\n", " if proc_in.input_name == \"code\":\n", " s3_evaluation_code_uri_with_experiments = proc_in.source \n", " \n", "s3_evaluation_code_uri_with_experiments" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Save this location for later:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%store s3_evaluation_code_uri_with_experiments" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can now go back to [the main notebook](../evaluation.ipynb)." ] } ], "metadata": { "instance_type": "ml.t3.medium", "kernelspec": { "display_name": "Python 3 (Data Science)", "language": "python", "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-2:429704687514:image/datascience-1.0" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.10" } }, "nbformat": 4, "nbformat_minor": 4 }