{ "cells": [ { "cell_type": "markdown", "id": "80b9b865", "metadata": {}, "source": [ "# Serving JPMML-based Tree-based models on Amazon SageMaker\n" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n", "\n", "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-west-2/inference|structured|realtime|byoc|byoc-mme-java|JPMML_Models_SageMaker.ipynb)\n", "\n", "---" ] }, { "cell_type": "markdown", "id": "80b9b865", "metadata": {}, "source": [ "\n", "This example notebook demonstrates how to bring your own java-based container and serve PMML based Models on Amazon SageMaker for Inference.\n", "\n", "The parts handling the PMML stuff (aka the model, loading the model and predicting) were inspired from [here](https://github.com/hkropp/jpmml-iris-example/blob/master/src/main/resources/sample/Iris.csv) and [here](https://henning.kropponline.de/2015/09/06/jpmml-example-random-forest/).\n", "\n", "This example shows serving a pre-trained random forest model (PMML-based) on Amazon SageMaker using Bring your own container.\n", "SageMaker provides the ability to bring your own model in the format of the Docker containers. More information and examples on how to bring your own algorithms can be found [here](https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-inference-code.html)" ] }, { "cell_type": "markdown", "id": "9875c8f1", "metadata": {}, "source": [ "Update the SageMaker Python SDK" ] }, { "cell_type": "code", "execution_count": null, "id": "15c39090", "metadata": { "tags": [] }, "outputs": [], "source": [ "!pip install --upgrade sagemaker" ] }, { "cell_type": "code", "execution_count": null, "id": "d462bdf3-fe9b-4a23-a5fc-f1fe7769afc8", "metadata": { "tags": [] }, "outputs": [], "source": [ "!pip install sagemaker-studio-image-build" ] }, { "cell_type": "markdown", "id": "e8c32f72", "metadata": {}, "source": [ "Create a SageMaker session and get a IAM role." ] }, { "cell_type": "code", "execution_count": null, "id": "50c4ff16", "metadata": { "tags": [] }, "outputs": [], "source": [ "import boto3\n", "import numpy as np\n", "import os\n", "import pandas as pd\n", "import re\n", "import sagemaker\n", "\n", "from sagemaker.utils import S3DataConfig\n", "from datetime import datetime\n", "\n", "\n", "import shutil\n", "import tarfile\n", "\n", "\n", "role = sagemaker.get_execution_role()\n", "sm_session = sagemaker.Session()\n", "bucket_name = sm_session.default_bucket()\n", "prefix = \"demo-multimodel-endpoint\"\n", "\n", "bucket_name" ] }, { "attachments": {}, "cell_type": "markdown", "id": "bb03ecd8", "metadata": {}, "source": [ "## When should I build my own algorithm container?\n", "You may not need to create a container to bring your own code to Amazon SageMaker. When you are using a framework such as Apache MXNet or TensorFlow that has direct support in SageMaker, you can simply supply the Python code that implements your algorithm using the SDK entry points for that framework. This set of supported frameworks is regularly added to, so you should check the current list to determine whether your algorithm is written in one of these common machine learning environments.\n", "\n", "Even if there is direct SDK support for your environment or framework, you may find it more effective to build your own container. If the code that implements your algorithm is quite complex, or you need special additions to the framework, building your own container may be the right choice.\n", "\n", "Some reasons to build an already supported framework container are:\n", "\n", "- A specific version isn't supported.\n", "- Configure and install your dependencies and environment.\n", "- Use a different training/hosting solution than provided.\n", "\n", "This walkthrough shows that it is quite straightforward to build your own container. So you can still use SageMaker even if your use case is not covered by the deep learning containers that we've built for you.\n", "\n", "## Permissions\n", "Running this notebook requires permissions in addition to the normal SageMakerFullAccess permissions. This is because it creates new repositories on Amazon ECR. The easiest way to add these permissions is simply to add the managed policy AmazonEC2ContainerRegistryFullAccess to the role that you used to start your notebook instance. There's no need to restart your notebook instance when you do this, the new permissions will be available immediately.\n", "\n", "## The example\n", "In this example we show how to package a custom Java Spring-boot container serving PMML-based Random Forest Tree model on SageMaker. \n", "\n", "## Part 1: Packaging and Uploading your Algorithm for use with Amazon SageMaker\n", "### An overview of Docker\n", "If you're familiar with Docker already, you can skip ahead to the next section.\n", "\n", "For many data scientists, Docker containers are a new technology. But they are not difficult and can significantly simply the deployment of your software packages.\n", "\n", "Docker provides a simple way to package arbitrary code into an image that is totally self-contained. Once you have an image, you can use Docker to run a container based on that image. Running a container is just like running a program on the machine except that the container creates a fully self-contained environment for the program to run. Containers are isolated from each other and from the host environment, so the way your program is set up is the way it runs, no matter where you run it.\n", "\n", "Docker is more powerful than environment managers like conda or virtualenv because (a) it is completely language independent and (b) it comprises your whole operating environment, including startup commands, and environment variable.\n", "\n", "A Docker container is like a virtual machine, but it is much lighter weight. For example, a program running in a container can start in less than a second and many containers can run simultaneously on the same physical or virtual machine instance.\n", "\n", "Docker uses a simple file called a Dockerfile to specify how the image is assembled. An example is provided below. You can build your Docker images based on Docker images built by yourself or by others, which can simplify things quite a bit.\n", "\n", "Docker has become very popular in programming and devops communities due to its flexibility and its well-defined specification of how code can be run in its containers. It is the underpinning of many services built in the past few years, such as Amazon ECS.\n", "\n", "Amazon SageMaker uses Docker to allow users to train and deploy arbitrary algorithms.\n", "\n", "In Amazon SageMaker, Docker containers are invoked in a one way for training and another, slightly different, way for hosting. The following sections outline how to build containers for the SageMaker environment.\n", "\n", "Some helpful links:\n", "\n", "- [Docker home page](http://www.docker.com/)\n", "- [Getting started with Docker](https://docs.docker.com/get-started/)\n", "- [Dockerfile reference](https://docs.docker.com/engine/reference/builder/)\n", "- [docker run reference](https://docs.docker.com/engine/reference/run/)\n", "\n", "## How Amazon SageMaker runs your Docker container\n", "Because you can run the same image in training or hosting, Amazon SageMaker runs your container with the argument train or serve. How your container processes this argument depends on the container.\n", "\n", "- In this example, we don't define a ENTRYPOINT in the Dockerfile, so Docker runs the command train at training time and serve at serving time. In this example, we define these as executable Python scripts, but they could be any program that we want to start in that environment.\n", "- If you specify a program as a ENTRYPOINT in the Dockerfile, that program will be run at startup and its first argument will be train or serve. The program can then look at that argument and decide what to do.\n", "- If you are building separate containers for training and hosting (or building only for one or the other), you can define a program as a ENTRYPOINT in the Dockerfile and ignore (or verify) the first argument passed in.\n", "\n", "### Running your container during hosting\n", "Hosting has a very different model than training because hosting is responding to inference requests that come in via HTTP. In this example, we use TensorFlow Serving, however the hosting solution can be customized. One example is the Python serving stack within the scikit learn example.\n", "\n", "Amazon SageMaker uses two URLs in the container:\n", "\n", "/ping receives GET requests from the infrastructure. Your program returns 200 if the container is up and accepting requests.\n", "/invocations is the endpoint that receives client inference POST requests. The format of the request and the response is up to the algorithm. If the client supplied ContentType and Accept headers, these are passed in as well.\n", "The container has the model files in the same place that they were written to during training:\n", "\n", "\n", " /opt/ml\n", " `-- model\n", " `-- \n", "\n", "## The Dockerfile\n", " \n", "The Dockerfile describes the image that we want to build. You can think of it as describing the complete operating system installation of the system that you want to run. A Docker container running is quite a bit lighter than a full operating system, however, because it takes advantage of Linux on the host machine for the basic operations.\n", "\n", "For the Python science stack, we start from an official TensorFlow docker image and run the normal tools to install TensorFlow Serving. Then we add the code that implements our specific algorithm to the container and set up the right environment for it to run under.\n", "\n", "Let's look at the Dockerfile for this example." ] }, { "cell_type": "code", "execution_count": null, "id": "c5ab458c", "metadata": { "tags": [] }, "outputs": [], "source": [ "!cat Dockerfile" ] }, { "cell_type": "markdown", "id": "4d19d38e", "metadata": {}, "source": [ "### Building and registering the container\n", "The following shell code shows how to build the container image using docker build and push the container image to ECR using docker push. This code is also available as the shell script container/build-and-push.sh, which you can run as build-and-push.sh sagemaker-tf-cifar10-example to build the image sagemaker-tf-cifar10-example.\n", "\n", "This code looks for an ECR repository in the account you're using and the current default region (if you're using a SageMaker notebook instance, this is the region where the notebook instance was created). If the repository doesn't exist, the script will create it." ] }, { "cell_type": "code", "execution_count": null, "id": "02c38671", "metadata": { "tags": [] }, "outputs": [], "source": [ "%%sh\n", "\n", "# The name of our algorithm\n", "algorithm_name=random_forest\n", "\n", "#cd sagemaker-byoc-pmml-example\n", "\n", "account=$(aws sts get-caller-identity --query Account --output text)\n", "\n", "# Get the region defined in the current configuration (default to us-west-2 if none defined)\n", "region=$(aws configure get region)\n", "region=${region:-us-east-2}\n", "\n", "fullname=\"${account}.dkr.ecr.${region}.amazonaws.com/${algorithm_name}:latest\"\n", "\n", "# If the repository doesn't exist in ECR, create it.\n", "aws ecr describe-repositories --repository-names \"${algorithm_name}\" > /dev/null 2>&1\n", "\n", "if [ $? -ne 0 ]\n", "then\n", " aws ecr create-repository --repository-name \"${algorithm_name}\" > /dev/null\n", "fi\n", "\n", "# Get the login command from ECR and execute it directly\n", "aws ecr get-login-password --region ${region}|docker login --username AWS --password-stdin ${fullname}\n", "\n", "#sm-docker build .\n", "# Build the docker image locally with the image name and then push it to ECR\n", "# with the full name.\n", "\n", "docker build -t ${algorithm_name} .\n", "docker tag ${algorithm_name} ${fullname}\n", "\n", "docker push ${fullname}" ] }, { "cell_type": "markdown", "id": "47d55160-20ac-42aa-b5a6-2dadf8fc834e", "metadata": { "tags": [] }, "source": [ "### Upload model artifacts to S3\n", "\n", "In this example we will use pre-trained XGBoost based model defined in pmml format. First, we will convert the models to tar.gz format and then upload them to S3." ] }, { "cell_type": "code", "execution_count": null, "id": "3b893d84", "metadata": { "tags": [] }, "outputs": [], "source": [ "account = sm_session.boto_session.client(\"sts\").get_caller_identity()[\"Account\"]\n", "region = sm_session.boto_session.region_name\n", "serving_image = \"{}.dkr.ecr.{}.amazonaws.com/random_forest:latest\".format(\n", " account,\n", " region\n", " # serving_image = \"{}.dkr.ecr.{}.amazonaws.com/sagemaker-studio-d-v8zbzuweo1qc:ds-fetch-user\".format(\n", " # account, region\n", ")\n", "\n", "serving_image" ] }, { "cell_type": "code", "execution_count": null, "id": "686c9bbc-98f2-4a58-af85-bd2db1982f67", "metadata": {}, "outputs": [], "source": [ "import tarfile\n", "\n", "with tarfile.open(\"data/iris_rf_1.tar.gz\", \"w:gz\") as tar:\n", " tar.add(\"data/iris_rf_1.pmml\", arcname=\".\")\n", "\n", "with tarfile.open(\"data/iris_rf_2.tar.gz\", \"w:gz\") as tar:\n", " tar.add(\"data/iris_rf_2.pmml\", arcname=\".\")" ] }, { "cell_type": "code", "execution_count": null, "id": "d0d2d434-362e-4e51-ac79-da6995b15da3", "metadata": { "tags": [] }, "outputs": [], "source": [ "from botocore.client import ClientError\n", "import os\n", "\n", "s3 = boto3.resource(\"s3\")\n", "try:\n", " s3.meta.client.head_bucket(Bucket=bucket_name)\n", "except ClientError:\n", " s3.create_bucket(Bucket=bucket_name, CreateBucketConfiguration={\"LocationConstraint\": region})\n", "\n", "models = {\"iris_rf_2.tar.gz\", \"iris_rf_1.tar.gz\"}\n", "\n", "for model in models:\n", " key = os.path.join(prefix, model)\n", " with open(\"data/\" + model, \"rb\") as file_obj:\n", " s3.Bucket(bucket_name).Object(key).upload_fileobj(file_obj)" ] }, { "cell_type": "markdown", "id": "e2cc9de3", "metadata": {}, "source": [ "### Define Amazon SageMaker Model\n", "\n", "Next, we define an Amazon SageMaker Model that defines the deployed model we will serve from an Amazon SageMaker Endpoint.\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "id": "b3878305", "metadata": { "tags": [] }, "outputs": [], "source": [ "model_url = \"https://s3-{}.amazonaws.com/{}/{}/\".format(region, bucket_name, prefix)\n", "\n", "serving_container_def = {\"Image\": serving_image, \"ModelDataUrl\": model_url, \"Mode\": \"MultiModel\"}\n", "model_name = \"pmml-random-forest\"\n", "\n", "create_model_response = sm_session.create_model(\n", " name=model_name, role=role, container_defs=serving_container_def\n", ")\n", "\n", "print(create_model_response)" ] }, { "cell_type": "markdown", "id": "b63f72a8", "metadata": {}, "source": [ "Next, we set the name of the Amaozn SageMaker hosted service endpoint configuration.\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "id": "096bf89a", "metadata": { "tags": [] }, "outputs": [], "source": [ "endpoint_config_name = f\"{model_name}-endpoint-config\"\n", "print(endpoint_config_name)" ] }, { "cell_type": "markdown", "id": "12270761", "metadata": {}, "source": [ "Next, we create the Amazon SageMaker hosted service endpoint configuration that uses one instance of ml.p3.2xlarge to serve the model.\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "id": "dca6b2c8", "metadata": { "tags": [] }, "outputs": [], "source": [ "epc = sm_session.create_endpoint_config(\n", " name=endpoint_config_name,\n", " model_name=model_name,\n", " initial_instance_count=1,\n", " instance_type=\"ml.m5.large\",\n", ")\n", "print(epc)" ] }, { "cell_type": "markdown", "id": "3b9a8bb0", "metadata": {}, "source": [ "Next we specify the Amazon SageMaker endpoint name for the endpoint used to serve the model.\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "id": "30fc65e1", "metadata": { "tags": [] }, "outputs": [], "source": [ "endpoint_name = f\"{model_name}-endpoint-{datetime.now().strftime('%Y%m-%d%H-%M%S')}\"\n", "print(endpoint_name)" ] }, { "cell_type": "markdown", "id": "d237ce48", "metadata": {}, "source": [ "Next, we create the Amazon SageMaker endpoint using the endpoint configuration we created above.\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "id": "953daaad", "metadata": { "tags": [] }, "outputs": [], "source": [ "ep = sm_session.create_endpoint(\n", " endpoint_name=endpoint_name, config_name=endpoint_config_name, wait=True\n", ")\n", "print(ep)" ] }, { "cell_type": "markdown", "id": "c81b1d6a", "metadata": {}, "source": [ "Now that the Amazon SageMaker endpoint is in service, we will use the endpoint to do inference.\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "id": "d5db63bd", "metadata": {}, "outputs": [], "source": [ "import boto3\n", "import base64\n", "import json\n", "\n", "\n", "client = boto3.client(\"sagemaker-runtime\")\n", "\n", "payload = '{\"data\": [{ \"features\": [\"5.1\",\"3.5\",\"1.4\",\"0.2\",\"Iris-setosa\"]}]}'\n", "\n", "response = client.invoke_endpoint(\n", " EndpointName=endpoint_name,\n", " ContentType=\"application/json\",\n", " TargetModel=\"iris_rf_1.tar.gz\", # this is the rest of the S3 path where the model artifacts are located\n", " Body=payload,\n", ")\n", "body = response[\"Body\"].read()\n", "print(body)" ] }, { "cell_type": "markdown", "id": "c4ca414a-e7d9-48b8-99c7-e4a23042e1d9", "metadata": {}, "source": [ "### Add models to the endpoint\n", "We can add more models to the endpoint without having to update the endpoint. To demonstrate hosting multiple models behind the endpoint, this model is duplicated 10 times with a slightly different name in S3. In a more realistic scenario, these could be 10 new different models." ] }, { "cell_type": "code", "execution_count": null, "id": "38fc18f2-9880-4985-802e-c1fccd25a32a", "metadata": { "tags": [] }, "outputs": [], "source": [ "with tarfile.open(\"data/iris_rf.tar.gz\", \"w:gz\") as tar:\n", " tar.add(\"data/iris_rf.pmml\", arcname=\".\")\n", "\n", "file = \"data/iris_rf.tar.gz\"\n", "\n", "for x in range(0, 30):\n", " s3_file_name = \"demo-subfolder/iris_rf_{}.tar.gz\".format(x)\n", " key = os.path.join(prefix, s3_file_name)\n", " with open(file, \"rb\") as file_obj:\n", " s3.Bucket(bucket_name).Object(key).upload_fileobj(file_obj)\n", " models.add(s3_file_name)\n", "\n", "print(\"Number of models: {}\".format(len(models)))\n", "print(\"Models: {}\".format(models))" ] }, { "cell_type": "markdown", "id": "759825c4-703f-4343-8449-24f298076ac7", "metadata": {}, "source": [ "After uploading the SqueezeNet models to S3, we will invoke the endpoint 100 times, randomly choosing from one of the 12 models behind the S3 prefix for each invocation, and keeping a count of the label with the highest probability on each invoke response." ] }, { "cell_type": "code", "execution_count": null, "id": "7822ba43-9e05-4e95-b948-202208897ad9", "metadata": { "tags": [] }, "outputs": [], "source": [ "%%time\n", "\n", "import random\n", "from collections import defaultdict\n", "\n", "results = defaultdict(int)\n", "\n", "for x in range(0, 30):\n", " target_model = random.choice(tuple(models))\n", " response = client.invoke_endpoint(\n", " EndpointName=endpoint_name,\n", " ContentType=\"application/json\",\n", " TargetModel=target_model,\n", " Body=payload,\n", " )\n", "\n", " # results[json.loads(response[\"Body\"]] += 1\n", "\n", "# print(*results.items(), sep=\"\\n\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Notebook CI Test Results\n", "\n", "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n", "\n", "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-east-1/inference|structured|realtime|byoc|byoc-mme-java|JPMML_Models_SageMaker.ipynb)\n", "\n", "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-east-2/inference|structured|realtime|byoc|byoc-mme-java|JPMML_Models_SageMaker.ipynb)\n", "\n", "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-west-1/inference|structured|realtime|byoc|byoc-mme-java|JPMML_Models_SageMaker.ipynb)\n", "\n", "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ca-central-1/inference|structured|realtime|byoc|byoc-mme-java|JPMML_Models_SageMaker.ipynb)\n", "\n", "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/sa-east-1/inference|structured|realtime|byoc|byoc-mme-java|JPMML_Models_SageMaker.ipynb)\n", "\n", "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-west-1/inference|structured|realtime|byoc|byoc-mme-java|JPMML_Models_SageMaker.ipynb)\n", "\n", "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-west-2/inference|structured|realtime|byoc|byoc-mme-java|JPMML_Models_SageMaker.ipynb)\n", "\n", "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-west-3/inference|structured|realtime|byoc|byoc-mme-java|JPMML_Models_SageMaker.ipynb)\n", "\n", "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-central-1/inference|structured|realtime|byoc|byoc-mme-java|JPMML_Models_SageMaker.ipynb)\n", "\n", "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-north-1/inference|structured|realtime|byoc|byoc-mme-java|JPMML_Models_SageMaker.ipynb)\n", "\n", "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-southeast-1/inference|structured|realtime|byoc|byoc-mme-java|JPMML_Models_SageMaker.ipynb)\n", "\n", "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-southeast-2/inference|structured|realtime|byoc|byoc-mme-java|JPMML_Models_SageMaker.ipynb)\n", "\n", "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-northeast-1/inference|structured|realtime|byoc|byoc-mme-java|JPMML_Models_SageMaker.ipynb)\n", "\n", "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-northeast-2/inference|structured|realtime|byoc|byoc-mme-java|JPMML_Models_SageMaker.ipynb)\n", "\n", "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-south-1/inference|structured|realtime|byoc|byoc-mme-java|JPMML_Models_SageMaker.ipynb)\n" ] } ], "metadata": { "instance_type": "ml.t3.medium", "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.16" }, "vscode": { "interpreter": { "hash": "397704579725e15f5c7cb49fe5f0341eb7531c82d19f2c29d197e8b64ab5776b" } } }, "nbformat": 4, "nbformat_minor": 5 }