{ "cells": [ { "cell_type": "markdown", "id": "de7e4dca-3653-4bf6-80a0-d964492d1d91", "metadata": {}, "source": [ "# Track an experiment while training a Pytorch model with a SageMaker Training Job\n", "***\n", "To execute this notebook in SageMaker Studio, you should select the **`PyTorch 1.12 Python 3.8 CPU Optimizer`** image.\n", "***\n", "\n", "This notebook shows how you can use the SageMaker SDK to track a Machine Learning experiment using a Pytorch model trained in a SageMaker Training Job with Script mode, where you will provide the model script file.\n", "\n", "We introduce two concepts in this notebook -\n", "\n", "* *Experiment:* An experiment is a collection of runs. When you initialize a run in your training loop, you include the name of the experiment that the run belongs to. Experiment names must be unique within your AWS account. \n", "* *Run:* A run consists of all the inputs, parameters, configurations, and results for one iteration of model training. Initialize an experiment run for tracking a training job with Run(). \n", "\n", "\n", "\n", "You can track artifacts for experiments, including datasets, algorithms, hyperparameters and metrics. Experiments executed on SageMaker such as SageMaker training jobs are automatically tracked and any existen SageMaker experiment on your AWS account is automatically migrated to the new UI version.\n", "\n", "In this notebook we will demonstrate the capabilities through an MNIST handwritten digits classification example. The notebook is organized as follow:\n", "\n", "1. Train a Convolutional Neural Network (CNN) Model and log the model training metrics\n", "1. Tune the hyperparameters that configures the number of hidden channels and the optimized in the model. Track teh parameter's configuration, resulting model loss and accuracy and automatically plot a confusion matrix using the Experiments capabilities of the SageMaker SDK.\n", "1. Analyse your model results and plot graphs comparing your model different runs generated from the tunning step 3.\n", "\n", "## Runtime\n", "This notebook takes approximately 20 minutes to run.\n", "\n", "## Contents\n", "1. [Install modules](#Install-modules)\n", "1. [Setup](#Setup)\n", "1. [Create model training script](#Create-model-training-script)\n", "1. [Train model with Run context](#Train-model-with-Run-context)\n", "1. [Contact](#Contact)" ] }, { "cell_type": "markdown", "id": "1141d3f8-45ed-4a56-8651-8964446befac", "metadata": {}, "source": [ "## Install modules\n", "\n", "Let's ensure we have the latest SageMaker SDK available, including the SageMaker Experiments functionality" ] }, { "cell_type": "code", "execution_count": null, "id": "ad635414-188d-4459-9e05-231f82f8054e", "metadata": { "jp-MarkdownHeadingCollapsed": true, "scrolled": true, "tags": [] }, "outputs": [], "source": [ "# update boto3 and sagemaker to ensure latest SDK version\n", "%pip install --upgrade pip\n", "%pip install --upgrade boto3\n", "%pip install --upgrade sagemaker\n", "%pip install torch\n", "%pip install torchvision\n", "%pip install --upgrade seaborn" ] }, { "cell_type": "markdown", "id": "3368d208-aebb-4844-bf27-2b2e373ef3d2", "metadata": { "tags": [] }, "source": [ "## Setup\n", "\n", "Import required libraries and set logging and experiment configuration\n", "\n", "SageMaker Experiments now provides the `Run` class that allows you to create a new experiment run. " ] }, { "cell_type": "code", "execution_count": null, "id": "5c4aa28d-25e3-4ade-a19e-2f6a36099570", "metadata": { "tags": [] }, "outputs": [], "source": [ "from sagemaker import get_execution_role\n", "from sagemaker.experiments.run import Run, load_run\n", "from sagemaker.pytorch import PyTorch\n", "from sagemaker.session import Session\n", "from sagemaker.utils import name_from_base\n", "import datetime" ] }, { "cell_type": "code", "execution_count": null, "id": "3877f90a-d680-4f9e-badf-d2bb4d19783a", "metadata": { "tags": [] }, "outputs": [], "source": [ "role = get_execution_role()\n", "region = Session().boto_session.region_name" ] }, { "cell_type": "markdown", "id": "d9dc0054-d7dd-4ec8-b1e9-0b292fc7b1c0", "metadata": {}, "source": [ "## Check model training script\n", "* Optional Step: check *`mnist.py`* using the cell below, the pytorch script file to train our model." ] }, { "cell_type": "code", "execution_count": null, "id": "20c6e08a-92d3-4819-a080-4858337813cf", "metadata": { "tags": [] }, "outputs": [], "source": [ "%load ./script/mnist.py" ] }, { "cell_type": "markdown", "id": "0342b174-1d33-4b27-a7cd-28571f1a1507", "metadata": {}, "source": [ "The cell above implements the code necessary to train our PyTorch model in SageMaker, using the SageMaker PyTorch image. It uses the `load_run` function to automatically detect the experiment configuration and `run.log_parameter`, `run.log_parameters`, `run.log_file`, `run.log_metric` and `run.log_confusion_matrix` to track the model training" ] }, { "cell_type": "markdown", "id": "cc1913d9-fe5f-4cf1-aca6-c6f6c12bd21c", "metadata": {}, "source": [ "## Train model with Run context\n", "\n", "Let's now train the model with passing the experiement run context to the training job\n", "\n", "For detailed explanation of API run, refer to source code [here](https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/experiments/run.py)" ] }, { "cell_type": "code", "execution_count": null, "id": "83c6d1dc-9d7e-4dbb-9957-2a5fb16f8527", "metadata": { "tags": [] }, "outputs": [], "source": [ "# set new experiment configuration\n", "experiment_name = \"training-job-experiment\"\n", "run_name_base = \"run-example\"\n", "run_name = name_from_base(run_name_base, short=True)\n", "print(f\"Experiment name: {experiment_name}\\nRun name: {run_name}\")" ] }, { "cell_type": "code", "execution_count": null, "id": "93f266e0-d73d-452c-a3ed-51b0fc48075d", "metadata": { "scrolled": true, "tags": [] }, "outputs": [], "source": [ "%%time\n", "# Start training job with experiment setting\n", "with Run(experiment_name=experiment_name, run_name=run_name, sagemaker_session=Session()) as run:\n", " estmator = PyTorch(\n", " entry_point=\"mnist.py\",\n", " source_dir=\"script\",\n", " role=role,\n", " model_dir=False,\n", " framework_version=\"1.12\",\n", " py_version=\"py38\",\n", " instance_type=\"ml.c5.xlarge\",\n", " instance_count=1,\n", " hyperparameters={\"epochs\": 8, \"hidden_channels\": 5, \"optimizer\": \"adam\"},\n", " keep_alive_period_in_seconds=3600,\n", " )\n", "\n", " estmator.fit()" ] }, { "cell_type": "markdown", "id": "259c4fca-5128-4213-8065-cb68bd50b973", "metadata": { "tags": [] }, "source": [ "Checking the SageMaker Experiments UI, you can observe the Experiment run, populated with the metrics and parameters logged. We can also see the automatically generated outputs for the model data\n", "\n", "\n", "\n", "\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "id": "61561b3e-0f87-4d53-8369-83a193121f34", "metadata": {}, "source": [ "## Run multiple experiments\n", "\n", "You can now create multiple runs of your experiment by varying a few parameters. Feel free to play with the parameters. " ] }, { "cell_type": "code", "execution_count": null, "id": "c8e4f80b-01ec-415b-8e80-507bbeadc402", "metadata": { "scrolled": true, "tags": [] }, "outputs": [], "source": [ "%%time\n", "# Start training job with experiment setting\n", "\n", "hidden_channels = [5, 10]\n", "optimizer = [\"adam\", \"sgd\"]\n", "\n", "for h in hidden_channels:\n", " for j in optimizer:\n", " now = f\"{datetime.datetime.now():%Y-%m-%d-%H-%M-%S}\"\n", " run_name_n = f\"{run_name_base}-{j}-{h}-{now}\"\n", " with Run(experiment_name=experiment_name, run_name=run_name_n) as run:\n", " print(\"hidden_channels-\", h, \" optimizer-\", j)\n", " estmator = PyTorch(\n", " entry_point=\"./script/mnist.py\",\n", " role=role,\n", " model_dir=False,\n", " framework_version=\"1.12\",\n", " py_version=\"py38\",\n", " instance_type=\"ml.c5.xlarge\",\n", " instance_count=1,\n", " hyperparameters={\n", " \"epochs\": 10,\n", " \"hidden_channels\": h,\n", " \"optimizer\": j,\n", " },\n", " keep_alive_period_in_seconds=1200, # use warm pool\n", " )\n", "\n", " estmator.fit(wait=True)" ] }, { "cell_type": "markdown", "id": "95f58280-c71b-4ad7-b084-7e65e79673ea", "metadata": {}, "source": [ "## Compare the performance through Experiment UI\n", "\n", "In the SageMaker Experiments UI, you can compare the different runs and analyze the metrics for those runs \n", "\n", "\n", "\n" ] }, { "cell_type": "markdown", "id": "4f4a2125-3fd9-48b4-92eb-6b0408e0f5ce", "metadata": {}, "source": [ "## Bonus Point: customized analysis \n", "\n", "Besides all the built-in analysis withn Experiments, you can also customize your analysis and plot based on available metrics and parameter!\n", "\n", "Below, we have provided you an example for your reference. \n", "\n", "This analsis is built on top of [Experiment Analysis API](https://sagemaker.readthedocs.io/en/stable/api/training/analytics.html)" ] }, { "cell_type": "code", "execution_count": null, "id": "43c03b10-cc26-4c12-8310-08d84ec5d292", "metadata": { "tags": [] }, "outputs": [], "source": [ "import matplotlib as mpl\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "import seaborn.objects as so\n", "from sagemaker.analytics import ExperimentAnalytics\n" ] }, { "cell_type": "code", "execution_count": null, "id": "71e349de-5771-4404-8d1d-45c807638e6d", "metadata": { "tags": [] }, "outputs": [], "source": [ "def analyze_experiment(experiment_name:str ,parameter_names: str, metric_names: str, stat_name: str = \"Last\"):\n", " re_expr = (\n", " f\"(?:{'|'.join([f'{k}.*- {stat_name}' for k in metric_names] + parameter_names + ['DisplayName'])})\"\n", " )\n", "\n", " trial_component_analytics = ExperimentAnalytics(\n", " experiment_name=experiment_name,\n", " parameter_names=parameter_names,\n", " )\n", " df = trial_component_analytics.dataframe()\n", " # df = df[df[\"SourceArn\"].isna()]\n", " df = df.filter(regex=re_expr)\n", "\n", " # join the categorical parameters\n", " df_temp = df[parameter_names].select_dtypes(\"object\")\n", "\n", " cat_col_name = \"_\".join(df_temp.columns.values)\n", "\n", " if len(df_temp.columns) > 1:\n", " df.loc[:, cat_col_name] = df_temp.astype(str).apply(\"_\".join, axis=1)\n", " df = df.drop(columns=df_temp.columns.values)\n", "\n", " ordinal_params = df[parameter_names].select_dtypes(\"number\").columns.tolist()\n", " df_plot = df.melt(id_vars=[\"DisplayName\"] + ordinal_params + [cat_col_name])\n", " df_plot[[\"Dataset\", \"Metrics\"]] = (\n", " df_plot.variable.str.split(\" - \").str[0].str.split(\":\", expand=True)\n", " )\n", " f = plt.Figure(\n", " figsize=(8, 6 * len(ordinal_params)),\n", " facecolor=\"w\",\n", " layout=\"constrained\",\n", " frameon=True,\n", " )\n", " f.suptitle(\"Experiment Analysis\")\n", " sf = f.subfigures(1, len(ordinal_params))\n", "\n", " if isinstance(sf, mpl.figure.SubFigure):\n", " sf = [sf]\n", "\n", " for k, p in zip(sf, ordinal_params):\n", "\n", " (\n", " so.Plot(\n", " df_plot,\n", " y=\"value\",\n", " x=p,\n", " color=cat_col_name,\n", " )\n", " .facet(col=\"Dataset\", row=\"Metrics\")\n", " .add(so.Dot())\n", " .share(y=False)\n", " .limit(y=(0, None))\n", " .on(k)\n", " .plot()\n", " )\n", " return f" ] }, { "cell_type": "code", "execution_count": null, "id": "013de6d2-ce25-4079-911b-bebca8429d29", "metadata": { "tags": [] }, "outputs": [], "source": [ "parameter_names = [\"hidden_channels\", \"optimizer\"]\n", "metric_names = [\"accuracy\", \"loss\"]\n", "stat_name = \"Last\"\n", "\n", "\n", "analyze_experiment(experiment_name, parameter_names, metric_names)\n" ] }, { "cell_type": "code", "execution_count": null, "id": "5e24a511-813d-468d-9c55-019c9f114b07", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "forced_instance_type": "ml.t3.medium", "forced_lcc_arn": "", "instance_type": "ml.m5.large", "kernelspec": { "display_name": "Python 3 (PyTorch 1.12 Python 3.8 CPU Optimized)", "language": "python", "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/pytorch-1.12-cpu-py38" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.13" } }, "nbformat": 4, "nbformat_minor": 5 }