{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## Time Series Classification Using AWS Built-in TensorFlow Framework" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. SPDX-License-Identifier: MIT-0`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "This notebook uses the NASA turbofan jet engine maintenace data, which is available [here](https://ti.arc.nasa.gov/m/project/prognostic-repository/CMAPSSData.zip), to demostrate `TensorFlow` modeling using the AWS pre-built TensorFlow image. \n", "\n", "Built-in framework is one of methods using SageMaker services: \n", "\n", " \n", "\n", "[Source: AWS re:Invent 2020: Implementing MLOps practices with Amazon SageMaker](https://www.youtube.com/watch?v=8ZpE-9LnaJk)\n", "\n", "Please note when you use the built-in TensorFlow image, you will need to develop your own training script. \n", "\n", "The detailed steps are presented in the following sections. \n", "\n", "\n", "1. [Data preprocessing](#1-data-preprocessing) \n", " -[Retrieve data](#retrieve-data-from-nasa-website) \n", " -[Label data](#label-data) \n", " -[Reshape data](#reshape-data)\n", "2. [Create TensorFlow model](#2-create-tensorflow-model) \n", " -[Create training script](#create-script) \n", " -[Define TensorFlow model](#define-tensorflow-estimator)\n", "3. [Train the model](#3-train-the-model)\n", "4. [Deploy the model](#4-deploy-the-trained-model-to-an-endpoint)\n", "5. [Invoke the endpoint](#5-invoke-the-endpoint)\n", "6. [Delete the endpoint](#6-delete-the-endpoint)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Import libraries" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import os\n", "import zipfile\n", "import urllib\n", "import boto3\n", "\n", "import numpy as np\n", "import pandas as pd\n", "import sagemaker\n", "import sklearn\n", "from sklearn import metrics\n", "from sklearn.metrics import confusion_matrix\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.preprocessing import StandardScaler\n", "from sagemaker import get_execution_role\n", "from sagemaker.tensorflow import TensorFlow" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Get SageMaker service role\n", "try:\n", " role = get_execution_role() # runing from SageMaker Notebook\n", "except ValueError:\n", " iam = boto3.client('iam') #runing from local machine\n", " role = iam.get_role(\n", " RoleName='AmazonSageMaker-ExecutionRole')['Role']['Arn']\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#Use default bucket\n", "sess = sagemaker.Session()\n", "bucket = sess.default_bucket() \n", "# Use your own bucket\n", "#bucket=\"\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1 Data preprocessing\n", "\n", " ### Retrieve Data from NASA website" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Get data from NASA website\n", "\n", "data_folder = 'data'\n", "if not os.path.exists(data_folder):\n", " os.makedirs(data_folder)\n", "urllib.request.urlretrieve('https://ti.arc.nasa.gov/m/project/prognostic-repository/CMAPSSData.zip',\n", " os.path.join(data_folder, 'CMAPSSData.zip'))\n", "\n", "with zipfile.ZipFile(os.path.join(data_folder, 'CMAPSSData.zip'), \"r\") as zip_ref:\n", " zip_ref.extractall(data_folder)\n", "\n", "columns = ['id', 'cycle', 'setting1', 'setting2', 'setting3', 's1', 's2', 's3', 's4', 's5', 's6',\n", " 's7', 's8', 's9', 's10', 's11', 's12', 's13', 's14', 's15', 's16', 's17', 's18', 's19', 's20', 's21']\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# read data\n", "for i in range(1, 5):\n", " df = pd.read_csv(\n", " 'data/train_FD{:03d}.txt'.format(i), delimiter=' ', header=None)\n", " df.drop(df.columns[[26, 27]], axis=1, inplace=True)\n", " df.columns = columns\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ### Label data \n", "Generate target for the turbines. 14 cycles before the engine failure is defined as 'failure' (1). \n", "The data records the full life of engines. To create label for the data, it defines the last 14 cycles as failure. In this way, the model can predict engine failue 13 cycles in advance. \n", "The diagram depicts the appraoch. \n", "" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Data Labeling - generate target.\n", "rul = pd.DataFrame(df.groupby('id')['cycle'].max()).reset_index()\n", "rul.columns = ['id', 'max']\n", "df = df.merge(rul, on=['id'], how='left')\n", "df['RUL'] = df['max'] - df['cycle']\n", "df.drop('max', axis=1, inplace=True)\n", "df['target'] = df['RUL'].apply(lambda x: 1 if x <= 14 else 0)\n", "df.head()\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df.shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Split training and testing data" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "trainID, testID = train_test_split(df.id.unique(), test_size=0.2)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "train = df[df['id'].isin(trainID)]\n", "\n", "test = df[df['id'].isin(testID)]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "train.columns" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Get numerical columns and use scaler to transform the data" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "scaler = StandardScaler()\n", "num_cols = ['setting1', 'setting2', 'setting3', 's1', 's2', 's3',\n", " 's4', 's5', 's6', 's7', 's8', 's9', 's10', 's11', 's12', 's13', 's14',\n", " 's15', 's16', 's17', 's18', 's19', 's20', 's21']\n", "scaler.fit(train[num_cols])\n", "train[num_cols] = scaler.transform(train[num_cols])\n", "test[num_cols] = scaler.transform(test[num_cols])\n", "train.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Reshape data \n", "For time seriees analysis, the 2-dimension data (rows are timestamps, columns are features) needs to be reconstructed as 3-dimension data. For every sample, the sequence length defines how many time steps will be in this sample. A single sample is a 2-D data with a window of sequence length by features. Sequence length is also called 'history size' or 'look back window'. \n", "\n", "" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#For time serries analysis\n", "sequence_length = 30" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# get train id that has record more than the sequence_length\n", "id_count1 = train.groupby(\"id\").count()\n", "id_count1 = id_count1.reset_index()\n", "train_id = id_count1[id_count1['s1'] > sequence_length]\n", "train = train[train['id'].isin(train_id['id'])]\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# get test id that has record more than tne sequence_length\n", "id_count2 = test.groupby(\"id\").count()\n", "id_count2 = id_count2.reset_index()\n", "test_id = id_count2[id_count2['s1'] > sequence_length]\n", "test = test[test['id'].isin(test_id['id'])]\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are multiple engines for the dataset. To model engines, same engine's data need to be grouped and processed together. The following functions are used to reshape the data of multiple engines. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Target size indicates if the model will predict future data. If the target size is 0, it means it use previous sequence length of data to predict current target. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "target_size=0\n", "history_size=sequence_length" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def reconstruct_data_singleID(dataset, target, start_index, end_index, history_size,\n", " target_size, single_step=True):\n", " data = []\n", " labels = []\n", "\n", " start_index = start_index + history_size\n", " if end_index is None:\n", " end_index = len(dataset) - target_size\n", "\n", " for i in range(start_index, end_index):\n", " indices = range(i-history_size, i)\n", " data.append(dataset[indices])\n", "\n", " if single_step:\n", " labels.append([target[i+target_size]])\n", " else:\n", " labels.append([target[i:i+target_size]])\n", "\n", " return np.array(data), np.array(labels)\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def reconstruct_data_mutipleID(raw_df):\n", " x_data = []\n", " y_data = []\n", " for id in raw_df['id'].unique():\n", " df = raw_df[raw_df['id'] == id]\n", " x_, y_ = reconstruct_data_singleID(df[num_cols].values, df['target'].values, 0, None, history_size,\n", " target_size, single_step=True)\n", " x_data.append(x_)\n", " y_data.append(y_)\n", " target = np.concatenate(y_data).astype(np.float32)\n", " fea_data = np.concatenate(x_data).astype(np.float32)\n", " return fea_data, target\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "x_train, y_train = reconstruct_data_mutipleID(train)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# function to reshape features into (samples, time steps, features)\n", "def get_sequence(df, seq_length, feature_cols):\n", " data_array = df[feature_cols].values\n", " num_elements = data_array.shape[0]\n", " for start, stop in zip(range(0, num_elements-seq_length), range(seq_length, num_elements)):\n", " yield data_array[start:stop, :]\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def get_targets(df, seq_length, target):\n", " data_array = df[target].values\n", " num_elements = data_array.shape[0]\n", " return data_array[seq_length:num_elements, :]\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# generator for the sequences\n", "seq_gen1 = (list(get_sequence(train[train['id'] == id], sequence_length, num_cols))\n", " for id in train['id'].unique())\n", "# generate sequences and convert to numpy array\n", "X_train = np.concatenate(list(seq_gen1)).astype(np.float32)\n", "X_train.shape" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# generate targets\n", "target1 = [get_targets(train[train['id'] == id], sequence_length, ['target'])\n", " for id in train['id'].unique()]\n", "y_train = np.concatenate(target1).astype(np.float32)\n", "y_train.shape\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "seq_gen2 = (list(get_sequence(test[test['id'] == id], sequence_length, num_cols))\n", " for id in test['id'].unique())\n", "# generate sequences and convert to numpy array\n", "X_test = np.concatenate(list(seq_gen2)).astype(np.float32)\n", "print(X_test.shape)\n", "# generate targets\n", "target2 = [get_targets(test[test['id'] == id], sequence_length, ['target'])\n", " for id in test['id'].unique()]\n", "y_test = np.concatenate(target2).astype(np.float32)\n", "print(y_test.shape)\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from numpy import save\n", "save('x_train.npy',X_train)\n", "save('y_train.npy', y_train)\n", "save('x_val.npy', X_test)\n", "save('y_val.npy', y_test)\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "x_test1 = np.load('x_val.npy')\n", "y_test1 = np.load('y_val.npy')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "y_test2=y_test1[-30:-10]\n", "x_test2 = x_test1[-30:-10]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "save('x_test1.npy',x_test1)\n", "save('y_test1.npy', y_test1)\n", "save('x_test2.npy', x_test2)\n", "save('y_test2.npy', y_test2)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#Save data to S3\n", "x_train_filename = 'train/x_train.npy'\n", "y_train_filename = 'train/y_train.npy'\n", "x_val_filename = 'train/x_val.npy'\n", "y_val_filename = 'train/y_val.npy'\n", "x_test1_filename = 'test/x_test1.npy'\n", "y_test1_filename = 'test/y_test1.npy'\n", "x_test2_filename = 'test/x_test2.npy'\n", "y_test2_filename = 'test/y_test2.npy'\n", "\n", "s3 = boto3.resource('s3')\n", "\n", "s3.meta.client.upload_file('./x_train.npy', bucket, x_train_filename)\n", "s3.meta.client.upload_file('./y_train.npy', bucket, y_train_filename)\n", "s3.meta.client.upload_file('./x_val.npy', bucket, x_val_filename)\n", "s3.meta.client.upload_file('./y_val.npy', bucket, y_val_filename)\n", "s3.meta.client.upload_file('./x_test1.npy', bucket, x_test1_filename)\n", "s3.meta.client.upload_file('./y_test1.npy', bucket, y_test1_filename)\n", "s3.meta.client.upload_file('./x_test2.npy', bucket, x_test2_filename)\n", "s3.meta.client.upload_file('./y_test2.npy', bucket, y_test2_filename)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2 Create TensorFlow Model\n", "\n", "The `sagemaker.tensorflow.TensorFlow` estimator handles locating the script mode container, uploading your script to a S3 location and creating a SageMaker training job. Let's call out a couple important parameters here:\n", "\n", "* `py_version` is set to `'py3'` to indicate that we are using script mode with built-in TensorFlow image.\n", "\n", "* `distribution` is used to configure the distributed training setup. It's required only if you are doing distributed training either across a cluster of instances or across multiple GPUs. Here we are using parameter servers as the distributed training schema. SageMaker training jobs run on homogeneous clusters. To make parameter server more performant in the SageMaker setup, we run a parameter server on every instance in the cluster, so there is no need to specify the number of parameter servers to launch. Script mode also supports distributed training with [Horovod](https://github.com/horovod/horovod). You can find the full documentation on how to configure `distribution` [here](https://github.com/aws/sagemaker-python-sdk/tree/master/src/sagemaker/tensorflow#distributed-training). " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Create script " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%writefile tf_sdk_train.py\n", "# Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n", "#\n", "# Licensed under the Apache License, Version 2.0 (the \"License\"). You\n", "# may not use this file except in compliance with the License. A copy of\n", "# the License is located at\n", "#\n", "# http://aws.amazon.com/apache2.0/\n", "#\n", "# or in the \"license\" file accompanying this file. This file is\n", "# distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n", "# ANY KIND, either express or implied. See the License for the specific\n", "# language governing permissions and limitations under the License.import tensorflow as tf\n", "\n", "import tensorflow as tf\n", "from tensorflow import keras\n", "from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping\n", "from tensorflow.keras.models import Sequential\n", "from tensorflow.keras.layers import Dense, Dropout, LSTM, Activation\n", "import argparse\n", "import os\n", "import numpy as np\n", "import pandas as pd\n", "import json\n", "\n", "\n", "def model(x_train, y_train, x_test, y_test):\n", " \"\"\"Generate a simple model\"\"\"\n", "\n", " nb_features = x_train.shape[2]\n", " sequence_length = x_train.shape[1]\n", " nb_out = y_train.shape[1]\n", " \n", " model = Sequential()\n", " \n", " model.add(LSTM(\n", " input_shape=(sequence_length, nb_features),\n", " units=100,\n", " return_sequences=True))\n", " model.add(Dropout(0.2))\n", " model.add(LSTM(\n", " units=50,\n", " return_sequences=False))\n", " model.add(Dropout(0.2))\n", " model.add(Dense(units=nb_out, activation='sigmoid'))\n", "\n", " model.compile(loss='binary_crossentropy',\n", " optimizer='RMSProp', metrics=[tf.keras.metrics.AUC()])\n", "\n", " model.fit(x_train, y_train)\n", " model.evaluate(x_test, y_test)\n", "\n", " return model\n", "\n", "\n", "def _load_training_data(base_dir):\n", " \"\"\"Load training data\"\"\"\n", " x_train = np.load(os.path.join(base_dir, 'x_train.npy'))\n", " y_train = np.load(os.path.join(base_dir, 'y_train.npy'))\n", " return x_train, y_train\n", "\n", "\n", "def _load_testing_data(base_dir):\n", " \"\"\"Load testing data\"\"\"\n", " x_test = np.load(os.path.join(base_dir, 'x_val.npy'))\n", " y_test = np.load(os.path.join(base_dir, 'y_val.npy'))\n", " return x_test, y_test\n", "\n", "\n", "def _parse_args():\n", " parser = argparse.ArgumentParser()\n", "\n", " # Data, model, and output directories\n", " # model_dir is always passed in from SageMaker. By default this is a S3 path under the default bucket.\n", " parser.add_argument('--model_dir', type=str)\n", " parser.add_argument('--sm-model-dir', type=str, default=os.environ.get('SM_MODEL_DIR'))\n", " parser.add_argument('--train', type=str, default=os.environ.get('SM_CHANNEL_TRAINING'))\n", " parser.add_argument('--hosts', type=list, default=json.loads(os.environ.get('SM_HOSTS')))\n", " parser.add_argument('--current-host', type=str, default=os.environ.get('SM_CURRENT_HOST'))\n", "\n", " return parser.parse_known_args()\n", "\n", "\n", "if __name__ == \"__main__\":\n", " args, unknown = _parse_args()\n", "\n", " train_data, train_labels = _load_training_data(args.train)\n", " eval_data, eval_labels = _load_testing_data(args.train)\n", "\n", " tf_classifier = model(train_data, train_labels, eval_data, eval_labels)\n", "\n", " if args.current_host == args.hosts[0]:\n", " # save model to an S3 directory with version number '00000001'\n", " tf_classifier.save(os.path.join(args.sm_model_dir, '000000001'), 'tf_sm_model.h5')\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Define TensorFlow estimator" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "tf_estimator = TensorFlow(entry_point='tf_sdk_train.py',\n", " role=role,\n", " instance_count=2,\n", " instance_type='ml.p3.2xlarge',\n", " framework_version='2.1.0',\n", " py_version='py3',\n", " distribution={'parameter_server': {'enabled': True}})" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "training_data_uri=f's3://{bucket}/train'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3 Train the model\n", "\n", "To start a training job, call `estimator.fit(training_data_uri)`.\n", "\n", "An S3 location is used here as the input. `fit` creates a default channel named `'training'`, which points to this S3 location. In the training script we can then access the training data from the location stored in `SM_CHANNEL_TRAINING`. `fit` accepts a couple other types of input as well. See the API doc [here](https://sagemaker.readthedocs.io/en/stable/estimators.html#sagemaker.estimator.EstimatorBase.fit) for details.\n", "\n", "When training starts, the TensorFlow container executes training script tf_sdk_train.py, passing `hyperparameters` and `model_dir` from the estimator as script arguments. Because we didn't define either in this example, no hyperparameters are passed, and `model_dir` defaults to `s3:///`, so the script execution is as follows:\n", "```bash\n", "python tf_sdk_train.py --model_dir s3:///\n", "```\n", "When training is complete, the training job will upload the saved model for TensorFlow serving." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "tf_estimator.fit(training_data_uri)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4 Deploy the trained model to an endpoint\n", "\n", "The `deploy()` method creates a SageMaker model, which is then deployed to an endpoint to serve prediction requests in real time. We will use the TensorFlow Serving container for the endpoint, because we trained with script mode. This serving container runs an implementation of a web server that is compatible with SageMaker hosting protocol. The [Using your own inference code]() document explains how SageMaker runs inference containers.\n", "\n", "Deployed the trained TensorFlow 2.1 model to an endpoint." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "predictor = tf_estimator.deploy(\n", " initial_instance_count=1, instance_type='ml.p2.xlarge')\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 5 Invoke the endpoint\n", "\n", "Let's download the training data and use that as input for inference." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Invoke Endpoint: Option 1-using SDK" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "predictions = predictor.predict(x_test1[:50])\n", "for i in range(0, 50):\n", " prediction = np.argmax(predictions['predictions'][i])\n", " label = y_test1[i]\n", " print('prediction is {}, label is {}, matched: {}'.format(prediction, label, prediction == label))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "endpoint_name=predictor.endpoint\n", "endpoint_name" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Invoke Endpoint: Option 2-using Boto3" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import json\n", "import botocore\n", "ENDPOINT_NAME = predictor.endpoint\n", "\n", "config = botocore.config.Config(read_timeout=80)\n", "runtime = boto3.client('runtime.sagemaker', config=config)\n", "data = x_test1[0:50]\n", "payload = json.dumps(data.tolist())\n", "\n", "response = runtime.invoke_endpoint(EndpointName=ENDPOINT_NAME,\n", " ContentType='application/json',\n", " Body=payload)\n", "result = json.loads(response['Body'].read().decode())\n", "res = result['predictions']" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "for i in range(0, 50):\n", " prediction = np.argmax(result['predictions'][i])\n", " label = y_test1[i]\n", " print('prediction is {}, label is {}, matched: {}'.format(prediction, label, prediction == label))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 6. Delete the endpoint\n", "\n", "Let's delete the endpoint we just created to prevent incurring any extra costs." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "sagemaker.Session().delete_endpoint(endpoint_name)" ] } ], "metadata": { "interpreter": { "hash": "be583636f02b8176d07e25c3aa14c0b4b500e6c84fbd1a028f1ae57a974e092a" }, "kernelspec": { "display_name": "Python 3.8.12 ('py38')", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.12" }, "orig_nbformat": 4 }, "nbformat": 4, "nbformat_minor": 2 }