{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "![MLU Logo](../data/MLU_Logo.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Machine Learning Accelerator - Tabular Data - Lecture 3\n", "\n", "\n", "## AutoGluon\n", "\n", "In this notebook, we use __AutoGluon__ to predict the __Outcome Type__ field of our review dataset.\n", "\n", "\n", "[AutoGluon](https://auto.gluon.ai/stable/index.html) implements many of the best practices that we have discussed in this class, and more! In particular, it sets itself apart from other AutoML solutions by having excellent automated feature engineering that can handle text data and missing values without any hand-coded solutions (See their [paper](https://arxiv.org/abs/2003.06505) for details). It is too new to be in an existing Sagemaker kernel, so let's install it.\n", "\n", "1. Set up AutoGluon\n", "2. Read the datasets\n", "3. Train a classifier with AutoGluon\n", "4. Model evaluation\n", "5. Clean up model artifacts\n", "\n", "__Austin Animal Center Dataset__:\n", "\n", "In this exercise, we are working with pet adoption data from __Austin Animal Center__. We have two datasets that cover intake and outcome of animals. Intake data is available from [here](https://data.austintexas.gov/Health-and-Community-Services/Austin-Animal-Center-Intakes/wter-evkm) and outcome is from [here](https://data.austintexas.gov/Health-and-Community-Services/Austin-Animal-Center-Outcomes/9t4d-g238). \n", "\n", "In order to work with a single table, we joined the intake and outcome tables using the \"Animal ID\" column and created a single __review.csv__ file. We also didn't consider animals with multiple entries to the facility to keep our dataset simple. If you want to see the original datasets and the merged data with multiple entries, they are available under data/review folder: Austin_Animal_Center_Intakes.csv, Austin_Animal_Center_Outcomes.csv and Austin_Animal_Center_Intakes_Outcomes.csv.\n", "\n", "__Dataset schema:__ \n", "- __Pet ID__ - Unique ID of pet\n", "- __Outcome Type__ - State of pet at the time of recording the outcome (0 = not placed, 1 = placed). This is the field to predict.\n", "- __Sex upon Outcome__ - Sex of pet at outcome\n", "- __Name__ - Name of pet \n", "- __Found Location__ - Found location of pet before entered the center\n", "- __Intake Type__ - Circumstances bringing the pet to the center\n", "- __Intake Condition__ - Health condition of pet when entered the center\n", "- __Pet Type__ - Type of pet\n", "- __Sex upon Intake__ - Sex of pet when entered the center\n", "- __Breed__ - Breed of pet \n", "- __Color__ - Color of pet \n", "- __Age upon Intake Days__ - Age of pet when entered the center (days)\n", "- __Age upon Outcome Days__ - Age of pet at outcome (days))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Set up AutoGluon\n", "(Go to top)" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[33mWARNING: You are using pip version 21.3.1; however, version 22.3.1 is available.\n", "You should consider upgrading via the '/home/ec2-user/anaconda3/envs/pytorch_p39/bin/python -m pip install --upgrade pip' command.\u001b[0m\n", "Note: you may need to restart the kernel to use updated packages.\n" ] } ], "source": [ "%pip install -q -r ../requirements.txt" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Read the dataset\n", "(Go to top)\n", "\n", "Let's read the dataset into a dataframe, using Pandas, and split the dataset into train and test sets (AutoGluon will handle the validation itself)." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "\n", "df = pd.read_csv('../data/review/review_dataset.csv')" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "from sklearn.model_selection import train_test_split\n", "\n", "train_data, test_data = train_test_split(df, test_size=0.1, shuffle=True, random_state=23)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Train a classifier with AutoGluon\n", "(Go to top)\n", "\n", "We can run AutoGluon with a short snippet. For fitting, we just call the __.fit()__ function. In this exercise, we used the data frame objects, but this tool also accepts the raw csv files as input. To use this tool with simple csv files, you can follow the code snippet below.\n", "\n", "```python\n", "from autogluon.tabular import TabularDataset, TabularPredictor\n", "\n", "train_data = TabularDataset(file_path='path_to_dataset/train.csv')\n", "test_data = TabularDataset(file_path='path_to_dataset/test.csv')\n", "\n", "predictor = TabularPredictor(label='label_column').fit(train_data)\n", "test_predictions = predictor.predict(test_data)\n", "```\n", "\n", "We have our separate __data frames__ for training and test data, so we work with them below. We grab the first 10000 data points for a quick demo. You can also pass the full dataset." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "scrolled": true, "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "No path specified. Models will be saved in: \"AutogluonModels/ag-20230126_225442/\"\n", "Beginning AutoGluon training ...\n", "AutoGluon will save models to \"AutogluonModels/ag-20230126_225442/\"\n", "AutoGluon Version: 0.6.1\n", "Python Version: 3.9.13\n", "Operating System: Linux\n", "Platform Machine: x86_64\n", "Platform Version: #1 SMP Fri Oct 14 01:16:24 UTC 2022\n", "Train Data Rows: 10000\n", "Train Data Columns: 12\n", "Label Column: Outcome Type\n", "Preprocessing data ...\n", "AutoGluon infers your prediction problem is: 'binary' (because only two unique label-values observed).\n", "\t2 unique label values: [1.0, 0.0]\n", "\tIf 'binary' is not the correct problem_type, please manually specify the problem_type parameter during predictor init (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])\n", "Selected class <--> label mapping: class 1 = 1, class 0 = 0\n", "Using Feature Generators to preprocess the data ...\n", "Fitting AutoMLPipelineFeatureGenerator...\n", "\tAvailable Memory: 13890.67 MB\n", "\tTrain Data (Original) Memory Usage: 6.86 MB (0.0% of available memory)\n", "\tInferring data type of each feature based on column values. Set feature_metadata_in to manually specify special dtypes of the features.\n", "\tStage 1 Generators:\n", "\t\tFitting AsTypeFeatureGenerator...\n", "\tStage 2 Generators:\n", "\t\tFitting FillNaFeatureGenerator...\n", "\tStage 3 Generators:\n", "\t\tFitting IdentityFeatureGenerator...\n", "\t\tFitting CategoryFeatureGenerator...\n", "\t\t\tFitting CategoryMemoryMinimizeFeatureGenerator...\n", "\t\tFitting TextSpecialFeatureGenerator...\n", "\t\t\tFitting BinnedFeatureGenerator...\n", "\t\t\tFitting DropDuplicatesFeatureGenerator...\n", "\t\tFitting TextNgramFeatureGenerator...\n", "\t\t\tFitting CountVectorizer for text features: ['Found Location']\n", "\t\t\tCountVectorizer fit with vocabulary size = 198\n", "\tStage 4 Generators:\n", "\t\tFitting DropUniqueFeatureGenerator...\n", "\tUnused Original Features (Count: 1): ['Pet ID']\n", "\t\tThese features were not used to generate any of the output features. Add a feature generator compatible with these features to utilize them.\n", "\t\tFeatures can also be unused if they carry very little information, such as being categorical but having almost entirely unique values or being duplicates of other features.\n", "\t\tThese features do not need to be present at inference time.\n", "\t\t('object', []) : 1 | ['Pet ID']\n", "\tTypes of features in original data (raw dtype, special dtypes):\n", "\t\t('int', []) : 2 | ['Age upon Intake Days', 'Age upon Outcome Days']\n", "\t\t('object', []) : 8 | ['Sex upon Outcome', 'Name', 'Intake Type', 'Intake Condition', 'Pet Type', ...]\n", "\t\t('object', ['text']) : 1 | ['Found Location']\n", "\tTypes of features in processed data (raw dtype, special dtypes):\n", "\t\t('category', []) : 8 | ['Sex upon Outcome', 'Name', 'Intake Type', 'Intake Condition', 'Pet Type', ...]\n", "\t\t('category', ['text_as_category']) : 1 | ['Found Location']\n", "\t\t('int', []) : 2 | ['Age upon Intake Days', 'Age upon Outcome Days']\n", "\t\t('int', ['binned', 'text_special']) : 12 | ['Found Location.char_count', 'Found Location.word_count', 'Found Location.capital_ratio', 'Found Location.lower_ratio', 'Found Location.digit_ratio', ...]\n", "\t\t('int', ['text_ngram']) : 199 | ['__nlp__.183', '__nlp__.183 and', '__nlp__.183 in', '__nlp__.183 in austin', '__nlp__.1st', ...]\n", "\t1.0s = Fit runtime\n", "\t11 features in original data used to generate 222 features in processed data.\n", "\tTrain Data (Processed) Memory Usage: 4.4 MB (0.0% of available memory)\n", "Data preprocessing and feature engineering runtime = 1.03s ...\n", "AutoGluon will gauge predictive performance using evaluation metric: 'accuracy'\n", "\tTo change this, specify the eval_metric parameter of Predictor()\n", "Automatically generating train/validation split with holdout_frac=0.1, Train Rows: 9000, Val Rows: 1000\n", "Fitting 13 L1 models ...\n", "Fitting model: KNeighborsUnif ...\n", "\t0.663\t = Validation score (accuracy)\n", "\t23.67s\t = Training runtime\n", "\t0.37s\t = Validation runtime\n", "Fitting model: KNeighborsDist ...\n", "\t0.668\t = Validation score (accuracy)\n", "\t0.09s\t = Training runtime\n", "\t0.38s\t = Validation runtime\n", "Fitting model: LightGBMXT ...\n", "\t0.85\t = Validation score (accuracy)\n", "\t2.26s\t = Training runtime\n", "\t0.02s\t = Validation runtime\n", "Fitting model: LightGBM ...\n", "\t0.849\t = Validation score (accuracy)\n", "\t1.15s\t = Training runtime\n", "\t0.02s\t = Validation runtime\n", "Fitting model: RandomForestGini ...\n", "\t0.851\t = Validation score (accuracy)\n", "\t2.37s\t = Training runtime\n", "\t0.14s\t = Validation runtime\n", "Fitting model: RandomForestEntr ...\n", "\t0.847\t = Validation score (accuracy)\n", "\t2.58s\t = Training runtime\n", "\t0.14s\t = Validation runtime\n", "Fitting model: CatBoost ...\n", "\t0.855\t = Validation score (accuracy)\n", "\t7.17s\t = Training runtime\n", "\t0.03s\t = Validation runtime\n", "Fitting model: ExtraTreesGini ...\n", "\t0.838\t = Validation score (accuracy)\n", "\t2.32s\t = Training runtime\n", "\t0.15s\t = Validation runtime\n", "Fitting model: ExtraTreesEntr ...\n", "\t0.84\t = Validation score (accuracy)\n", "\t2.44s\t = Training runtime\n", "\t0.16s\t = Validation runtime\n", "Fitting model: NeuralNetFastAI ...\n", "No improvement since epoch 2: early stopping\n", "\t0.831\t = Validation score (accuracy)\n", "\t13.52s\t = Training runtime\n", "\t0.04s\t = Validation runtime\n", "Fitting model: XGBoost ...\n", "\t0.854\t = Validation score (accuracy)\n", "\t7.83s\t = Training runtime\n", "\t0.02s\t = Validation runtime\n", "Fitting model: NeuralNetTorch ...\n", "\t0.849\t = Validation score (accuracy)\n", "\t13.39s\t = Training runtime\n", "\t0.03s\t = Validation runtime\n", "Fitting model: LightGBMLarge ...\n", "\t0.847\t = Validation score (accuracy)\n", "\t1.96s\t = Training runtime\n", "\t0.02s\t = Validation runtime\n", "Fitting model: WeightedEnsemble_L2 ...\n", "\t0.871\t = Validation score (accuracy)\n", "\t0.9s\t = Training runtime\n", "\t0.0s\t = Validation runtime\n", "AutoGluon training complete, total runtime = 85.07s ... Best model: \"WeightedEnsemble_L2\"\n", "TabularPredictor saved. To load, use: predictor = TabularPredictor.load(\"AutogluonModels/ag-20230126_225442/\")\n" ] } ], "source": [ "from autogluon.tabular import TabularDataset, TabularPredictor\n", "\n", "k = 10000 # grab less data for a quick demo\n", "#k = train_data.shape[0] # grad the whole dataset\n", "\n", "predictor = TabularPredictor(label='Outcome Type').fit(train_data.head(k))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also summarize what happened during fit." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "*** Summary of fit() ***\n", "Estimated performance of each model:\n", " model score_val pred_time_val fit_time pred_time_val_marginal fit_time_marginal stack_level can_infer fit_order\n", "0 WeightedEnsemble_L2 0.871 0.512801 26.287463 0.002498 0.902044 2 True 14\n", "1 CatBoost 0.855 0.027408 7.169063 0.027408 7.169063 1 True 7\n", "2 XGBoost 0.854 0.016263 7.829739 0.016263 7.829739 1 True 11\n", "3 RandomForestGini 0.851 0.141809 2.365973 0.141809 2.365973 1 True 5\n", "4 LightGBMXT 0.850 0.017072 2.259549 0.017072 2.259549 1 True 3\n", "5 LightGBM 0.849 0.018099 1.154111 0.018099 1.154111 1 True 4\n", "6 NeuralNetTorch 0.849 0.029052 13.390344 0.029052 13.390344 1 True 12\n", "7 LightGBMLarge 0.847 0.017192 1.960757 0.017192 1.960757 1 True 13\n", "8 RandomForestEntr 0.847 0.144485 2.581486 0.144485 2.581486 1 True 6\n", "9 ExtraTreesEntr 0.840 0.159155 2.438493 0.159155 2.438493 1 True 9\n", "10 ExtraTreesGini 0.838 0.145048 2.324290 0.145048 2.324290 1 True 8\n", "11 NeuralNetFastAI 0.831 0.042973 13.518711 0.042973 13.518711 1 True 10\n", "12 KNeighborsDist 0.668 0.375591 0.085545 0.375591 0.085545 1 True 2\n", "13 KNeighborsUnif 0.663 0.372251 23.665409 0.372251 23.665409 1 True 1\n", "Number of models trained: 14\n", "Types of models trained:\n", "{'KNNModel', 'RFModel', 'XTModel', 'XGBoostModel', 'TabularNeuralNetTorchModel', 'CatBoostModel', 'WeightedEnsembleModel', 'LGBModel', 'NNFastAiTabularModel'}\n", "Bagging used: False \n", "Multi-layer stack-ensembling used: False \n", "Feature Metadata (Processed):\n", "(raw dtype, special dtypes):\n", "('category', []) : 8 | ['Sex upon Outcome', 'Name', 'Intake Type', 'Intake Condition', 'Pet Type', ...]\n", "('category', ['text_as_category']) : 1 | ['Found Location']\n", "('int', []) : 2 | ['Age upon Intake Days', 'Age upon Outcome Days']\n", "('int', ['binned', 'text_special']) : 12 | ['Found Location.char_count', 'Found Location.word_count', 'Found Location.capital_ratio', 'Found Location.lower_ratio', 'Found Location.digit_ratio', ...]\n", "('int', ['text_ngram']) : 199 | ['__nlp__.183', '__nlp__.183 and', '__nlp__.183 in', '__nlp__.183 in austin', '__nlp__.1st', ...]\n", "Plot summary of models saved to file: AutogluonModels/ag-20230126_225442/SummaryOfModels.html\n", "*** End of fit() summary ***\n" ] }, { "data": { "text/plain": [ "{'model_types': {'KNeighborsUnif': 'KNNModel',\n", " 'KNeighborsDist': 'KNNModel',\n", " 'LightGBMXT': 'LGBModel',\n", " 'LightGBM': 'LGBModel',\n", " 'RandomForestGini': 'RFModel',\n", " 'RandomForestEntr': 'RFModel',\n", " 'CatBoost': 'CatBoostModel',\n", " 'ExtraTreesGini': 'XTModel',\n", " 'ExtraTreesEntr': 'XTModel',\n", " 'NeuralNetFastAI': 'NNFastAiTabularModel',\n", " 'XGBoost': 'XGBoostModel',\n", " 'NeuralNetTorch': 'TabularNeuralNetTorchModel',\n", " 'LightGBMLarge': 'LGBModel',\n", " 'WeightedEnsemble_L2': 'WeightedEnsembleModel'},\n", " 'model_performance': {'KNeighborsUnif': 0.663,\n", " 'KNeighborsDist': 0.668,\n", " 'LightGBMXT': 0.85,\n", " 'LightGBM': 0.849,\n", " 'RandomForestGini': 0.851,\n", " 'RandomForestEntr': 0.847,\n", " 'CatBoost': 0.855,\n", " 'ExtraTreesGini': 0.838,\n", " 'ExtraTreesEntr': 0.84,\n", " 'NeuralNetFastAI': 0.831,\n", " 'XGBoost': 0.854,\n", " 'NeuralNetTorch': 0.849,\n", " 'LightGBMLarge': 0.847,\n", " 'WeightedEnsemble_L2': 0.871},\n", " 'model_best': 'WeightedEnsemble_L2',\n", " 'model_paths': {'KNeighborsUnif': 'AutogluonModels/ag-20230126_225442/models/KNeighborsUnif/',\n", " 'KNeighborsDist': 'AutogluonModels/ag-20230126_225442/models/KNeighborsDist/',\n", " 'LightGBMXT': 'AutogluonModels/ag-20230126_225442/models/LightGBMXT/',\n", " 'LightGBM': 'AutogluonModels/ag-20230126_225442/models/LightGBM/',\n", " 'RandomForestGini': 'AutogluonModels/ag-20230126_225442/models/RandomForestGini/',\n", " 'RandomForestEntr': 'AutogluonModels/ag-20230126_225442/models/RandomForestEntr/',\n", " 'CatBoost': 'AutogluonModels/ag-20230126_225442/models/CatBoost/',\n", " 'ExtraTreesGini': 'AutogluonModels/ag-20230126_225442/models/ExtraTreesGini/',\n", " 'ExtraTreesEntr': 'AutogluonModels/ag-20230126_225442/models/ExtraTreesEntr/',\n", " 'NeuralNetFastAI': 'AutogluonModels/ag-20230126_225442/models/NeuralNetFastAI/',\n", " 'XGBoost': 'AutogluonModels/ag-20230126_225442/models/XGBoost/',\n", " 'NeuralNetTorch': 'AutogluonModels/ag-20230126_225442/models/NeuralNetTorch/',\n", " 'LightGBMLarge': 'AutogluonModels/ag-20230126_225442/models/LightGBMLarge/',\n", " 'WeightedEnsemble_L2': 'AutogluonModels/ag-20230126_225442/models/WeightedEnsemble_L2/'},\n", " 'model_fit_times': {'KNeighborsUnif': 23.665409326553345,\n", " 'KNeighborsDist': 0.08554530143737793,\n", " 'LightGBMXT': 2.259549140930176,\n", " 'LightGBM': 1.1541106700897217,\n", " 'RandomForestGini': 2.3659727573394775,\n", " 'RandomForestEntr': 2.5814859867095947,\n", " 'CatBoost': 7.169063091278076,\n", " 'ExtraTreesGini': 2.3242902755737305,\n", " 'ExtraTreesEntr': 2.438493490219116,\n", " 'NeuralNetFastAI': 13.51871109008789,\n", " 'XGBoost': 7.829739093780518,\n", " 'NeuralNetTorch': 13.39034390449524,\n", " 'LightGBMLarge': 1.960756778717041,\n", " 'WeightedEnsemble_L2': 0.9020442962646484},\n", " 'model_pred_times': {'KNeighborsUnif': 0.3722507953643799,\n", " 'KNeighborsDist': 0.37559080123901367,\n", " 'LightGBMXT': 0.017071962356567383,\n", " 'LightGBM': 0.018099308013916016,\n", " 'RandomForestGini': 0.14180850982666016,\n", " 'RandomForestEntr': 0.1444845199584961,\n", " 'CatBoost': 0.027408123016357422,\n", " 'ExtraTreesGini': 0.1450481414794922,\n", " 'ExtraTreesEntr': 0.15915513038635254,\n", " 'NeuralNetFastAI': 0.04297327995300293,\n", " 'XGBoost': 0.01626276969909668,\n", " 'NeuralNetTorch': 0.029051542282104492,\n", " 'LightGBMLarge': 0.017191648483276367,\n", " 'WeightedEnsemble_L2': 0.0024983882904052734},\n", " 'num_bag_folds': 0,\n", " 'max_stack_level': 2,\n", " 'num_classes': 2,\n", " 'model_hyperparams': {'KNeighborsUnif': {'weights': 'uniform'},\n", " 'KNeighborsDist': {'weights': 'distance'},\n", " 'LightGBMXT': {'learning_rate': 0.05, 'extra_trees': True},\n", " 'LightGBM': {'learning_rate': 0.05},\n", " 'RandomForestGini': {'n_estimators': 300,\n", " 'max_leaf_nodes': 15000,\n", " 'n_jobs': -1,\n", " 'random_state': 0,\n", " 'bootstrap': True,\n", " 'criterion': 'gini'},\n", " 'RandomForestEntr': {'n_estimators': 300,\n", " 'max_leaf_nodes': 15000,\n", " 'n_jobs': -1,\n", " 'random_state': 0,\n", " 'bootstrap': True,\n", " 'criterion': 'entropy'},\n", " 'CatBoost': {'iterations': 10000,\n", " 'learning_rate': 0.05,\n", " 'random_seed': 0,\n", " 'allow_writing_files': False,\n", " 'eval_metric': 'Accuracy'},\n", " 'ExtraTreesGini': {'n_estimators': 300,\n", " 'max_leaf_nodes': 15000,\n", " 'n_jobs': -1,\n", " 'random_state': 0,\n", " 'bootstrap': True,\n", " 'criterion': 'gini'},\n", " 'ExtraTreesEntr': {'n_estimators': 300,\n", " 'max_leaf_nodes': 15000,\n", " 'n_jobs': -1,\n", " 'random_state': 0,\n", " 'bootstrap': True,\n", " 'criterion': 'entropy'},\n", " 'NeuralNetFastAI': {'layers': None,\n", " 'emb_drop': 0.1,\n", " 'ps': 0.1,\n", " 'bs': 'auto',\n", " 'lr': 0.01,\n", " 'epochs': 'auto',\n", " 'early.stopping.min_delta': 0.0001,\n", " 'early.stopping.patience': 20,\n", " 'smoothing': 0.0},\n", " 'XGBoost': {'n_estimators': 10000,\n", " 'learning_rate': 0.1,\n", " 'n_jobs': -1,\n", " 'proc.max_category_levels': 100,\n", " 'objective': 'binary:logistic',\n", " 'booster': 'gbtree'},\n", " 'NeuralNetTorch': {'num_epochs': 500,\n", " 'epochs_wo_improve': 20,\n", " 'activation': 'relu',\n", " 'embedding_size_factor': 1.0,\n", " 'embed_exponent': 0.56,\n", " 'max_embedding_dim': 100,\n", " 'y_range': None,\n", " 'y_range_extend': 0.05,\n", " 'dropout_prob': 0.1,\n", " 'optimizer': 'adam',\n", " 'learning_rate': 0.0003,\n", " 'weight_decay': 1e-06,\n", " 'proc.embed_min_categories': 4,\n", " 'proc.impute_strategy': 'median',\n", " 'proc.max_category_levels': 100,\n", " 'proc.skew_threshold': 0.99,\n", " 'use_ngram_features': False,\n", " 'num_layers': 4,\n", " 'hidden_size': 128,\n", " 'max_batch_size': 512,\n", " 'use_batchnorm': False,\n", " 'loss_function': 'auto'},\n", " 'LightGBMLarge': {'learning_rate': 0.03,\n", " 'num_leaves': 128,\n", " 'feature_fraction': 0.9,\n", " 'min_data_in_leaf': 5},\n", " 'WeightedEnsemble_L2': {'use_orig_features': False,\n", " 'max_base_models': 25,\n", " 'max_base_models_per_type': 5,\n", " 'save_bag_folds': True}},\n", " 'leaderboard': model score_val pred_time_val fit_time \\\n", " 0 WeightedEnsemble_L2 0.871 0.512801 26.287463 \n", " 1 CatBoost 0.855 0.027408 7.169063 \n", " 2 XGBoost 0.854 0.016263 7.829739 \n", " 3 RandomForestGini 0.851 0.141809 2.365973 \n", " 4 LightGBMXT 0.850 0.017072 2.259549 \n", " 5 LightGBM 0.849 0.018099 1.154111 \n", " 6 NeuralNetTorch 0.849 0.029052 13.390344 \n", " 7 LightGBMLarge 0.847 0.017192 1.960757 \n", " 8 RandomForestEntr 0.847 0.144485 2.581486 \n", " 9 ExtraTreesEntr 0.840 0.159155 2.438493 \n", " 10 ExtraTreesGini 0.838 0.145048 2.324290 \n", " 11 NeuralNetFastAI 0.831 0.042973 13.518711 \n", " 12 KNeighborsDist 0.668 0.375591 0.085545 \n", " 13 KNeighborsUnif 0.663 0.372251 23.665409 \n", " \n", " pred_time_val_marginal fit_time_marginal stack_level can_infer \\\n", " 0 0.002498 0.902044 2 True \n", " 1 0.027408 7.169063 1 True \n", " 2 0.016263 7.829739 1 True \n", " 3 0.141809 2.365973 1 True \n", " 4 0.017072 2.259549 1 True \n", " 5 0.018099 1.154111 1 True \n", " 6 0.029052 13.390344 1 True \n", " 7 0.017192 1.960757 1 True \n", " 8 0.144485 2.581486 1 True \n", " 9 0.159155 2.438493 1 True \n", " 10 0.145048 2.324290 1 True \n", " 11 0.042973 13.518711 1 True \n", " 12 0.375591 0.085545 1 True \n", " 13 0.372251 23.665409 1 True \n", " \n", " fit_order \n", " 0 14 \n", " 1 7 \n", " 2 11 \n", " 3 5 \n", " 4 3 \n", " 5 4 \n", " 6 12 \n", " 7 13 \n", " 8 6 \n", " 9 9 \n", " 10 8 \n", " 11 10 \n", " 12 2 \n", " 13 1 }" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "predictor.fit_summary()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. Model evaluation\n", "(Go to top)\n", "\n", "Next, we load a separate test data to demonstrate how to make predictions on new examples at inference time." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Evaluation: accuracy on test data: 0.860299507801864\n", "Evaluations on test data:\n", "{\n", " \"accuracy\": 0.860299507801864,\n", " \"balanced_accuracy\": 0.8471309800400696,\n", " \"mcc\": 0.71864266414641,\n", " \"f1\": 0.8847217421361908,\n", " \"precision\": 0.8316815597075549,\n", " \"recall\": 0.944988000738416\n", "}\n" ] }, { "data": { "text/plain": [ "{'accuracy': 0.860299507801864,\n", " 'balanced_accuracy': 0.8471309800400696,\n", " 'mcc': 0.71864266414641,\n", " 'f1': 0.8847217421361908,\n", " 'precision': 0.8316815597075549,\n", " 'recall': 0.944988000738416}" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# First predictions\n", "y_pred = predictor.predict(test_data.head(k))\n", "\n", "# Then, evaluations\n", "predictor.evaluate_predictions(y_true=test_data['Outcome Type'],\n", " y_pred=y_pred,\n", " auxiliary_metrics=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see the performance of each individual trained model on the test data:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
modelscore_testscore_valpred_time_testpred_time_valfit_timepred_time_test_marginalpred_time_val_marginalfit_time_marginalstack_levelcan_inferfit_order
0WeightedEnsemble_L20.8603000.8711.5319760.51280126.2874630.0111850.0024980.9020442True14
1RandomForestEntr0.8556920.8470.3382840.1444852.5814860.3382840.1444852.5814861True6
2CatBoost0.8540160.8550.0433160.0274087.1690630.0433160.0274087.1690631True7
3RandomForestGini0.8534930.8510.4346000.1418092.3659730.4346000.1418092.3659731True5
4LightGBM0.8509790.8490.0815780.0180991.1541110.0815780.0180991.1541111True4
5XGBoost0.8497220.8540.1316520.0162637.8297390.1316520.0162637.8297391True11
6LightGBMLarge0.8451150.8470.0802090.0171921.9607570.0802090.0171921.9607571True13
7NeuralNetTorch0.8440670.8490.1770990.02905213.3903440.1770990.02905213.3903441True12
8LightGBMXT0.8434390.8500.0672750.0170722.2595490.0672750.0170722.2595491True3
9ExtraTreesEntr0.8413450.8400.4520110.1591552.4384930.4520110.1591552.4384931True9
10ExtraTreesGini0.8370510.8380.4111520.1450482.3242900.4111520.1450482.3242901True8
11NeuralNetFastAI0.8358990.8310.2105940.04297313.5187110.2105940.04297313.5187111True10
12KNeighborsDist0.6586030.6683.2481150.3755910.0855453.2481150.3755910.0855451True2
13KNeighborsUnif0.6569270.6633.2586860.37225123.6654093.2586860.37225123.6654091True1
\n", "
" ], "text/plain": [ " model score_test score_val pred_time_test pred_time_val \\\n", "0 WeightedEnsemble_L2 0.860300 0.871 1.531976 0.512801 \n", "1 RandomForestEntr 0.855692 0.847 0.338284 0.144485 \n", "2 CatBoost 0.854016 0.855 0.043316 0.027408 \n", "3 RandomForestGini 0.853493 0.851 0.434600 0.141809 \n", "4 LightGBM 0.850979 0.849 0.081578 0.018099 \n", "5 XGBoost 0.849722 0.854 0.131652 0.016263 \n", "6 LightGBMLarge 0.845115 0.847 0.080209 0.017192 \n", "7 NeuralNetTorch 0.844067 0.849 0.177099 0.029052 \n", "8 LightGBMXT 0.843439 0.850 0.067275 0.017072 \n", "9 ExtraTreesEntr 0.841345 0.840 0.452011 0.159155 \n", "10 ExtraTreesGini 0.837051 0.838 0.411152 0.145048 \n", "11 NeuralNetFastAI 0.835899 0.831 0.210594 0.042973 \n", "12 KNeighborsDist 0.658603 0.668 3.248115 0.375591 \n", "13 KNeighborsUnif 0.656927 0.663 3.258686 0.372251 \n", "\n", " fit_time pred_time_test_marginal pred_time_val_marginal \\\n", "0 26.287463 0.011185 0.002498 \n", "1 2.581486 0.338284 0.144485 \n", "2 7.169063 0.043316 0.027408 \n", "3 2.365973 0.434600 0.141809 \n", "4 1.154111 0.081578 0.018099 \n", "5 7.829739 0.131652 0.016263 \n", "6 1.960757 0.080209 0.017192 \n", "7 13.390344 0.177099 0.029052 \n", "8 2.259549 0.067275 0.017072 \n", "9 2.438493 0.452011 0.159155 \n", "10 2.324290 0.411152 0.145048 \n", "11 13.518711 0.210594 0.042973 \n", "12 0.085545 3.248115 0.375591 \n", "13 23.665409 3.258686 0.372251 \n", "\n", " fit_time_marginal stack_level can_infer fit_order \n", "0 0.902044 2 True 14 \n", "1 2.581486 1 True 6 \n", "2 7.169063 1 True 7 \n", "3 2.365973 1 True 5 \n", "4 1.154111 1 True 4 \n", "5 7.829739 1 True 11 \n", "6 1.960757 1 True 13 \n", "7 13.390344 1 True 12 \n", "8 2.259549 1 True 3 \n", "9 2.438493 1 True 9 \n", "10 2.324290 1 True 8 \n", "11 13.518711 1 True 10 \n", "12 0.085545 1 True 2 \n", "13 23.665409 1 True 1 " ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "predictor.leaderboard(test_data, silent=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. Clean up model artifacts\n", "(Go to top)" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "!rm -r AutogluonModels" ] } ], "metadata": { "kernelspec": { "display_name": "conda_pytorch_p39", "language": "python", "name": "conda_pytorch_p39" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.13" } }, "nbformat": 4, "nbformat_minor": 4 }