{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Amazon SageMaker Model Governance - Model Cards\n" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n", "\n", "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-west-2/sagemaker_model_governance|model_card.ipynb)\n", "\n", "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This notebook walks you through the features of Amazon SageMaker Model Cards. For more information, see [Model Cards](https://docs.aws.amazon.com/sagemaker/latest/dg/model-cards.html) in the _Amazon SageMaker Developer Guide_.\n", "\n", "Amazon SageMaker Model Cards give you the ability to create a centralized, customizable fact-sheet to document critical details about your machine learning (ML) models. Use model cards to keep a record of model information, such as intended uses, risk ratings, training details, evaluation metrics, and more for streamlined governance and reporting. \n", "\n", "In this example, you create a binary classification model along with a model card to document model details along the way. Learn how to create, read, update, delete, and export model cards using the Amazon SageMaker Python SDK.\n", "\n", "---\n", "## Contents\n", "\n", "1. [Setup](#Setup)\n", "1. [Prepare a Binary Classification Model](#Model)\n", "1. [Create Model Card](#ModelCard)\n", "1. [Update Model Card](#Update)\n", "1. [Load Model Card](#Load)\n", "1. [List Model Card History](#ListHistory)\n", "1. [Export Model Card](#Export)\n", "1. [Cleanup](#Cleanup)\n", "\n", "---\n", "## Setup\n", "To begin, you must specify the following information:\n", "- The IAM role ARN used to give SageMaker training and hosting access to your data. The following example uses the SageMaker execution role.\n", "- The SageMaker session used to manage interactions with Amazon SageMaker Model Card API methods.\n", "- The S3 URI (`bucket` and `prefix`) where you want to store training artifacts, models, and any exported model card PDFs. This S3 bucket should be in the same Region as your Notebook Instance, training, and hosting configurations. The following example uses the default SageMaker S3 bucket and creates a default SageMaker S3 bucket if one does not already exist.\n", "- The S3 session used to manage interactions with Amazon S3 storage." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "! pip install --upgrade sagemaker" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import boto3\n", "from sagemaker.session import Session\n", "from sagemaker import get_execution_role\n", "\n", "role = get_execution_role()\n", "\n", "sagemaker_session = Session()\n", "\n", "bucket = sagemaker_session.default_bucket()\n", "prefix = \"model-card-sample-notebook\"\n", "\n", "region = sagemaker_session.boto_region_name\n", "s3 = boto3.client(\"s3\", region_name=region)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Next, import the necessary Python libraries." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import io\n", "import os\n", "import numpy as np\n", "from six.moves.urllib.parse import urlparse\n", "from pprint import pprint\n", "import boto3\n", "import sagemaker\n", "from sagemaker.image_uris import retrieve\n", "import sagemaker.amazon.common as smac\n", "from sagemaker.model_card import (\n", " ModelCard,\n", " ModelOverview,\n", " ObjectiveFunction,\n", " Function,\n", " TrainingDetails,\n", " IntendedUses,\n", " BusinessDetails,\n", " EvaluationJob,\n", " AdditionalInformation,\n", " Metric,\n", " MetricGroup,\n", " ModelCardStatusEnum,\n", " ObjectiveFunctionEnum,\n", " FacetEnum,\n", " RiskRatingEnum,\n", " MetricTypeEnum,\n", " EvaluationMetricTypeEnum,\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Prepare a Model\n", "The following code creates an example binary classification model trained on a synthetic dataset. The target variable (0 or 1) is the second variable in the tuple.\n", "\n", "### 1. Prepare the training data\n", "The code will upload example data to your S3 bucket." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# synthetic data\n", "raw_data = (\n", " (0.5, 0),\n", " (0.75, 0),\n", " (1.0, 0),\n", " (1.25, 0),\n", " (1.50, 0),\n", " (1.75, 0),\n", " (2.0, 0),\n", " (2.25, 1),\n", " (2.5, 0),\n", " (2.75, 1),\n", " (3.0, 0),\n", " (3.25, 1),\n", " (3.5, 0),\n", " (4.0, 1),\n", " (4.25, 1),\n", " (4.5, 1),\n", " (4.75, 1),\n", " (5.0, 1),\n", " (5.5, 1),\n", ")\n", "training_data = np.array(raw_data).astype(\"float32\")\n", "labels = training_data[:, 1]\n", "\n", "# upload data to S3 bucket\n", "buf = io.BytesIO()\n", "smac.write_numpy_to_dense_tensor(buf, training_data, labels)\n", "buf.seek(0)\n", "boto3.resource(\"s3\").Bucket(bucket).Object(os.path.join(prefix, \"train\")).upload_fileobj(buf)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2. Train a model\n", "Train a binary classification model with the training data from the previous step." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "s3_train_data = f\"s3://{bucket}/{prefix}/train\"\n", "output_location = f\"s3://{bucket}/{prefix}/output\"\n", "container = retrieve(\"linear-learner\", sagemaker_session.boto_session.region_name)\n", "estimator = sagemaker.estimator.Estimator(\n", " container,\n", " role=role,\n", " instance_count=1,\n", " instance_type=\"ml.m4.xlarge\",\n", " output_path=output_location,\n", " sagemaker_session=sagemaker_session,\n", ")\n", "estimator.set_hyperparameters(feature_dim=2, mini_batch_size=10, predictor_type=\"binary_classifier\")\n", "estimator.fit({\"train\": s3_train_data})\n", "print(f\"Training job name: {estimator.latest_training_job.name}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3. Create a model" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "model_name = \"model-card-test-model\"\n", "model = estimator.create_model(name=model_name)\n", "container_def = model.prepare_container_def()\n", "sagemaker_session.create_model(model_name, role, container_def)\n", "print(f\"Model name: {model_name}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Create Model Card\n", "Document your binary classification model details in an Amazon SageMaker Model Card using the SageMaker Python SDK.\n", "\n", "### 1. Auto-collect model details\n", "Automatically collect basic model information like model ID, training environment, and the model output S3 URI. Add additional model information such as a description, problem type, algorithm type, model creator, and model owner." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "model_overview = ModelOverview.from_model_name(\n", " model_name=model_name,\n", " sagemaker_session=sagemaker_session,\n", " model_description=\"This is an example binary classification model used for a Python SDK demo of Amazon SageMaker Model Cards.\",\n", " problem_type=\"Binary Classification\",\n", " algorithm_type=\"Logistic Regression\",\n", " model_creator=\"DEMO-ModelCard\",\n", " model_owner=\"DEMO-ModelCard\",\n", ")\n", "print(f\"Model id: {model_overview.model_id}\")\n", "print(f\"Model training images: {model_overview.inference_environment.container_image}\")\n", "print(f\"Model: {model_overview.model_artifact}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2. Auto-collect training details\n", "Automatically collect basic training information like training ID, training environment, and training metrics. Add additional training information such as objective function details and training observations." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "objective_function = ObjectiveFunction(\n", " function=Function(\n", " function=ObjectiveFunctionEnum.MINIMIZE,\n", " facet=FacetEnum.LOSS,\n", " ),\n", " notes=\"This is an example objective function.\",\n", ")\n", "training_details = TrainingDetails.from_model_overview(\n", " model_overview=model_overview,\n", " sagemaker_session=sagemaker_session,\n", " objective_function=objective_function,\n", " training_observations=\"Add model training observations here.\",\n", ")\n", "print(f\"Training job id: {training_details.training_job_details.training_arn}\")\n", "print(\n", " f\"Training image: {training_details.training_job_details.training_environment.container_image}\"\n", ")\n", "print(\"Training Metrics: \")\n", "pprint(\n", " [\n", " {\"name\": i.name, \"value\": i.value}\n", " for i in training_details.training_job_details.training_metrics\n", " ]\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3. Collect evaluation details\n", "Add evaluation observations, datasets, and metrics." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "manual_metric_group = MetricGroup(\n", " name=\"binary classification metrics\",\n", " metric_data=[Metric(name=\"accuracy\", type=MetricTypeEnum.NUMBER, value=0.5)],\n", ")\n", "example_evaluation_job = EvaluationJob(\n", " name=\"Example evaluation job\",\n", " evaluation_observation=\"Evaluation observations.\",\n", " datasets=[\"s3://path/to/evaluation/data\"],\n", " metric_groups=[manual_metric_group],\n", ")\n", "evaluation_details = [example_evaluation_job]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### (Optional) 3.1 Parse metrics from existing evaluation report\n", "If you have existing evaluation reports generated by [SageMaker Clarify](https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-processing-job-run.html) or [SageMaker Model Monitor](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-model-quality.html), upload them to S3 and provide an S3 URI to automatically parse evaluation metrics. To add your own generic model card evaluation report, provide a report in the [evaluation results JSON format](https://docs.aws.amazon.com/sagemaker/latest/dg/model-cards-json-schema.html). See the example JSON files in the `./example_metrics` folder for reference.\n", "##### Collect metrics from a JSON format evaluation report" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "report_type = \"clarify_bias.json\"\n", "example_evaluation_job.add_metric_group_from_json(\n", " f\"example_metrics/{report_type}\", EvaluationMetricTypeEnum.CLARIFY_BIAS\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Collect metrics from S3" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# upload metric file to s3\n", "with open(f\"example_metrics/{report_type}\", \"rb\") as metrics:\n", " s3.upload_fileobj(\n", " metrics,\n", " Bucket=bucket,\n", " Key=f\"{prefix}/{report_type}\",\n", " ExtraArgs={\"ContentType\": \"application/json\"},\n", " )\n", "\n", "metric_s3_url = f\"s3://{bucket}/{prefix}/{report_type}\"\n", "example_evaluation_job.add_metric_group_from_s3(\n", " session=sagemaker_session.boto_session,\n", " s3_url=metric_s3_url,\n", " metric_type=EvaluationMetricTypeEnum.CLARIFY_BIAS,\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4. Collect additional details\n", "Add the intended uses of your model and business details and any additional information that you want to include in your model card. For more information on intended uses and business details, see [Model Cards](https://docs.aws.amazon.com/sagemaker/latest/dg/model-cards.html) in the `Amazon SageMaker Developer Guide`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "intended_uses = IntendedUses(\n", " purpose_of_model=\"Test model card.\",\n", " intended_uses=\"Not used except this test.\",\n", " factors_affecting_model_efficiency=\"No.\",\n", " risk_rating=RiskRatingEnum.LOW,\n", " explanations_for_risk_rating=\"Just an example.\",\n", ")\n", "business_details = BusinessDetails(\n", " business_problem=\"The business problem that your model is used to solve.\",\n", " business_stakeholders=\"The stakeholders who have the interest in the business that your model is used for.\",\n", " line_of_business=\"Services that the business is offering.\",\n", ")\n", "additional_information = AdditionalInformation(\n", " ethical_considerations=\"Your model ethical consideration.\",\n", " caveats_and_recommendations=\"Your model's caveats and recommendations.\",\n", " custom_details={\"custom details1\": \"details value\"},\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 5. Initialize a model card\n", "Initialize a model card with the information collected in the previous steps." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "model_card_name = \"sample-notebook-model-card\"\n", "my_card = ModelCard(\n", " name=model_card_name,\n", " status=ModelCardStatusEnum.DRAFT,\n", " model_overview=model_overview,\n", " training_details=training_details,\n", " intended_uses=intended_uses,\n", " business_details=business_details,\n", " evaluation_details=evaluation_details,\n", " additional_information=additional_information,\n", " sagemaker_session=sagemaker_session,\n", ")\n", "my_card.create()\n", "print(f\"Model card {my_card.name} is successfully created with id {my_card.arn}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Update Model Card\n", "After creating a model card, you can update the model card information. Updating a model card creates a new model card version." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "my_card.model_overview.model_description = \"the model is updated.\"\n", "my_card.update()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Load Model Card\n", "Load an existing model card with the model card name." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "my_card2 = ModelCard.load(\n", " name=model_card_name,\n", " sagemaker_session=sagemaker_session,\n", ")\n", "print(f\"Model id: {my_card2.arn}\")\n", "print(f\"Model description: {my_card.model_overview.model_description}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## List Model Card History\n", "Track the model card history by listing historical versions." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "history = my_card.get_version_history()\n", "assert len(history) == 2 # one for creation and one for update" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Export Model Card\n", "Share the model card by exporting it to a PDF file.\n", "\n", "### 1. Create an export job" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "s3_output_path = f\"s3://{bucket}/{prefix}/export\"\n", "pdf_s3_url = my_card.export_pdf(s3_output_path=s3_output_path)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### (optional) List export jobs\n", "Check all the export jobs for this model card." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "my_card.list_export_jobs()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2. Download the exported model card PDF\n", "The downloaded PDF is stored in the same directory as this notebook by default.\n", "\n", "#### Parse the bucket and key of the exported PDF" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "parsed_url = urlparse(pdf_s3_url)\n", "pdf_bucket = parsed_url.netloc\n", "pdf_key = parsed_url.path.lstrip(\"/\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Download" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "file_name = parsed_url.path.split(\"/\")[-1]\n", "s3.download_file(Filename=file_name, Bucket=pdf_bucket, Key=pdf_key)\n", "print(f\"{file_name} is downloaded to \\n{os.path.join(os.getcwd(), file_name)}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Cleanup\n", "Delete the following resources:\n", "1. The model card\n", "2. The exported model card PDF\n", "3. The example binary classification model" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "my_card.delete()\n", "\n", "s3.delete_object(Bucket=pdf_bucket, Key=pdf_key)\n", "\n", "sagemaker_session.delete_model(model_name)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Notebook CI Test Results\n", "\n", "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n", "\n", "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-east-1/sagemaker_model_governance|model_card.ipynb)\n", "\n", "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-east-2/sagemaker_model_governance|model_card.ipynb)\n", "\n", "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-west-1/sagemaker_model_governance|model_card.ipynb)\n", "\n", "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ca-central-1/sagemaker_model_governance|model_card.ipynb)\n", "\n", "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/sa-east-1/sagemaker_model_governance|model_card.ipynb)\n", "\n", "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-west-1/sagemaker_model_governance|model_card.ipynb)\n", "\n", "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-west-2/sagemaker_model_governance|model_card.ipynb)\n", "\n", "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-west-3/sagemaker_model_governance|model_card.ipynb)\n", "\n", "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-central-1/sagemaker_model_governance|model_card.ipynb)\n", "\n", "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-north-1/sagemaker_model_governance|model_card.ipynb)\n", "\n", "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-southeast-1/sagemaker_model_governance|model_card.ipynb)\n", "\n", "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-southeast-2/sagemaker_model_governance|model_card.ipynb)\n", "\n", "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-northeast-1/sagemaker_model_governance|model_card.ipynb)\n", "\n", "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-northeast-2/sagemaker_model_governance|model_card.ipynb)\n", "\n", "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-south-1/sagemaker_model_governance|model_card.ipynb)\n" ] } ], "metadata": { "instance_type": "ml.t3.medium", "kernelspec": { "display_name": "Python 3 (Data Science 3.0)", "language": "python", "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-data-science-310-v1" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.6" }, "vscode": { "interpreter": { "hash": "1571133684af95a48d70cfb4aef2840ed1e20d7c2d2a63af1685000148425678" } } }, "nbformat": 4, "nbformat_minor": 4 }