{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "![MLU Logo](../data/MLU_Logo.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Machine Learning Accelerator - Natural Language Processing - Lecture 2\n", "\n", "## Tree-Based Models for a Classification Problem, and Hyperparameter Tuning\n", "\n", "We continue to work with our review dataset to see how Tree-based classifiers (Decision Tree, Random Forest), along with efficient optimization techniques (GridSearch, RandomizedSearch), perform to predict the __isPositive__ field of our review dataset (that is very similar to the final project dataset).\n", "\n", "1. Reading the dataset\n", "2. Exploratory data analysis\n", "3. Stop word removal and stemming\n", "4. Train - Validation Split\n", "5. Data processing with Pipeline and ColumnTransform\n", "6. Fit the Decision Tree classifier Find more details on the __DecisionTreeClassifier__ here: https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html \n", "7. Test the classifier\n", "8. Fit and test the Random Forest classifier Find more details on the __RandomForestClassifier__ here: https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html\n", "9. Hyperparameter Tuning\n", " * Find more details on the __GridSearchCV__ here: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html\n", " * Find more details on the __RandomizedSearchCV__ here: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html\n", "10. Ideas for improvement\n", "\n", "Overall dataset schema:\n", "* __reviewText:__ Text of the review\n", "* __summary:__ Summary of the review\n", "* __verified:__ Whether the purchase was verified (True or False)\n", "* __time:__ UNIX timestamp for the review\n", "* __log_votes:__ Logarithm-adjusted votes log(1+votes)\n", "* __isPositive:__ Whether the review is positive or negative (1 or 0)\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Note: you may need to restart the kernel to use updated packages.\n" ] } ], "source": [ "%pip install -q -r ../requirements.txt" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Reading the dataset\n", "(Go to top)\n", "\n", "We will use the __pandas__ library to read our dataset." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The shape of the dataset is: (70000, 6)\n" ] } ], "source": [ "import pandas as pd\n", "\n", "df = pd.read_csv('../data/examples/AMAZON-REVIEW-DATA-CLASSIFICATION.csv')\n", "\n", "print('The shape of the dataset is:', df.shape)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's look at the first 10 rows of the dataset. " ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
reviewTextsummaryverifiedtimelog_votesisPositive
0PURCHASED FOR YOUNGSTER WHO\\nINHERITED MY \"TOO...IDEAL FOR BEGINNER!True13618368000.0000001.0
1unable to open or useTwo StarsTrue14526432000.0000000.0
2Waste of money!!! It wouldn't load to my system.Dont buy it!True14332896000.0000000.0
3I attempted to install this OS on two differen...I attempted to install this OS on two differen...True15189120000.0000000.0
4I've spent 14 fruitless hours over the past tw...Do NOT Download.True14419296001.0986120.0
5I purchased the home and business because I wa...Quicken home and business not for amaturesTrue13353120000.0000000.0
6The download doesn't take long at all. And it'...Great!True13779936000.0000001.0
7This program is positively wonderful for word ...Terrific for practice.False11583648002.3978951.0
8Fantastic protection!! Great customer support!!Five StarsTrue14784768000.0000001.0
9Obviously Win 7 now the last great operating s...Five StarsTrue14714784000.0000001.0
\n", "
" ], "text/plain": [ " reviewText \\\n", "0 PURCHASED FOR YOUNGSTER WHO\\nINHERITED MY \"TOO... \n", "1 unable to open or use \n", "2 Waste of money!!! It wouldn't load to my system. \n", "3 I attempted to install this OS on two differen... \n", "4 I've spent 14 fruitless hours over the past tw... \n", "5 I purchased the home and business because I wa... \n", "6 The download doesn't take long at all. And it'... \n", "7 This program is positively wonderful for word ... \n", "8 Fantastic protection!! Great customer support!! \n", "9 Obviously Win 7 now the last great operating s... \n", "\n", " summary verified time \\\n", "0 IDEAL FOR BEGINNER! True 1361836800 \n", "1 Two Stars True 1452643200 \n", "2 Dont buy it! True 1433289600 \n", "3 I attempted to install this OS on two differen... True 1518912000 \n", "4 Do NOT Download. True 1441929600 \n", "5 Quicken home and business not for amatures True 1335312000 \n", "6 Great! True 1377993600 \n", "7 Terrific for practice. False 1158364800 \n", "8 Five Stars True 1478476800 \n", "9 Five Stars True 1471478400 \n", "\n", " log_votes isPositive \n", "0 0.000000 1.0 \n", "1 0.000000 0.0 \n", "2 0.000000 0.0 \n", "3 0.000000 0.0 \n", "4 1.098612 0.0 \n", "5 0.000000 0.0 \n", "6 0.000000 1.0 \n", "7 2.397895 1.0 \n", "8 0.000000 1.0 \n", "9 0.000000 1.0 " ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.head(10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Exploratory data analysis\n", "(Go to top)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's look at the range and distribution of log_votes" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "1.0 43692\n", "0.0 26308\n", "Name: isPositive, dtype: int64" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[\"isPositive\"].value_counts()" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "reviewText 11\n", "summary 14\n", "verified 0\n", "time 0\n", "log_votes 0\n", "isPositive 0\n", "dtype: int64\n" ] } ], "source": [ "print(df.isna().sum())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can check the number of missing values for each columm below." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "reviewText 11\n", "summary 14\n", "verified 0\n", "time 0\n", "log_votes 0\n", "isPositive 0\n", "dtype: int64\n" ] } ], "source": [ "print(df.isna().sum())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We have missing values in our text fields." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Text Processing: Stop words removal and stemming\n", "(Go to top)\n", "\n", "We will apply the text processing methods discussed in the class. " ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[nltk_data] Downloading package punkt to /home/ec2-user/nltk_data...\n", "[nltk_data] Package punkt is already up-to-date!\n", "[nltk_data] Downloading package stopwords to\n", "[nltk_data] /home/ec2-user/nltk_data...\n", "[nltk_data] Package stopwords is already up-to-date!\n" ] }, { "data": { "text/plain": [ "True" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Install the library and functions\n", "import nltk\n", "\n", "nltk.download('punkt')\n", "nltk.download('stopwords')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We will create the stop word removal and text cleaning processes below. NLTK library provides a list of common stop words. We will use the list, but remove some of the words from that list (because those words are actually useful to understand the sentiment in the sentence)." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "import nltk, re\n", "from nltk.corpus import stopwords\n", "from nltk.stem import SnowballStemmer\n", "from nltk.tokenize import word_tokenize\n", "\n", "# Let's get a list of stop words from the NLTK library\n", "stop = stopwords.words('english')\n", "\n", "# These words are important for our problem. We don't want to remove them.\n", "excluding = ['against', 'not', 'don', \"don't\",'ain', 'aren', \"aren't\", 'couldn', \"couldn't\",\n", " 'didn', \"didn't\", 'doesn', \"doesn't\", 'hadn', \"hadn't\", 'hasn', \"hasn't\", \n", " 'haven', \"haven't\", 'isn', \"isn't\", 'mightn', \"mightn't\", 'mustn', \"mustn't\",\n", " 'needn', \"needn't\",'shouldn', \"shouldn't\", 'wasn', \"wasn't\", 'weren', \n", " \"weren't\", 'won', \"won't\", 'wouldn', \"wouldn't\"]\n", "\n", "# New stop word list\n", "stop_words = [word for word in stop if word not in excluding]\n", "\n", "snow = SnowballStemmer('english')\n", "\n", "def process_text(texts): \n", " final_text_list=[]\n", " for sent in texts:\n", " \n", " # Check if the sentence is a missing value\n", " if isinstance(sent, str) == False:\n", " sent = \"\"\n", " \n", " filtered_sentence=[]\n", " \n", " sent = sent.lower() # Lowercase \n", " sent = sent.strip() # Remove leading/trailing whitespace\n", " sent = re.sub('\\s+', ' ', sent) # Remove extra space and tabs\n", " sent = re.compile('<.*?>').sub('', sent) # Remove HTML tags/markups:\n", " \n", " for w in word_tokenize(sent):\n", " # We are applying some custom filtering here, feel free to try different things\n", " # Check if it is not numeric and its length>2 and not in stop words\n", " if(not w.isnumeric()) and (len(w)>2) and (w not in stop_words): \n", " # Stem and add to filtered list\n", " filtered_sentence.append(snow.stem(w))\n", " final_string = \" \".join(filtered_sentence) #final string of cleaned words\n", " \n", " final_text_list.append(final_string)\n", " \n", " return final_text_list" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. Train - Validation Split\n", "(Go to top)\n", "\n", "Let's split our dataset into training (90%) and validation (10%)." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "from sklearn.model_selection import train_test_split\n", "\n", "X_train, X_val, y_train, y_val = train_test_split(df[[\"reviewText\", \"summary\", \"time\", \"log_votes\"]],\n", " df[\"isPositive\"],\n", " test_size=0.10,\n", " shuffle=True,\n", " random_state=324\n", " )" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Processing the reviewText fields\n", "Processing the summary fields\n" ] } ], "source": [ "print(\"Processing the reviewText fields\")\n", "X_train[\"reviewText\"] = process_text(X_train[\"reviewText\"].tolist())\n", "X_val[\"reviewText\"] = process_text(X_val[\"reviewText\"].tolist())\n", "\n", "print(\"Processing the summary fields\")\n", "X_train[\"summary\"] = process_text(X_train[\"summary\"].tolist())\n", "X_val[\"summary\"] = process_text(X_val[\"summary\"].tolist())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Our process_text() method in section 3 uses empty string for missing values." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. Data processing with Pipeline and ColumnTransform\n", "(Go to top)\n", "\n", "In the previous examples, we have seen how to use pipeline to prepare a data field for our machine learning model. This time, we will focus on multiple fields: numeric and text fields. Find more details on __Decision Trees__ here: https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html\n", "\n", " * For the numerical features pipeline, the __numerical_processor__ below, we use a MinMaxScaler (don't have to scale features when using Decision Trees, but it's a good idea to see how to use more data transforms). If different processing is desired for different numerical features, different pipelines should be built - just like shown below for the two text features.\n", " * For the text features pipeline, the __text_processor__ below, we use CountVectorizer() for the text fields.\n", " \n", "The selective preparations of the dataset features are then put together into a collective ColumnTransformer, to be finally used in a Pipeline along with an estimator. This ensures that the transforms are performed automatically on the raw data when fitting the model and when making predictions, such as when evaluating the model on a validation dataset via cross-validation or making predictions on a test dataset in the future." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "# Grab model features/inputs and target/output\n", "numerical_features = ['time',\n", " 'log_votes']\n", "\n", "text_features = ['summary',\n", " 'reviewText']\n", "\n", "model_features = numerical_features + text_features\n", "model_target = 'isPositive'" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
Pipeline(steps=[('data_preprocessing',\n",
       "                 ColumnTransformer(transformers=[('numerical_pre',\n",
       "                                                  Pipeline(steps=[('num_scaler',\n",
       "                                                                   MinMaxScaler())]),\n",
       "                                                  ['time', 'log_votes']),\n",
       "                                                 ('text_pre_0',\n",
       "                                                  Pipeline(steps=[('text_vect_0',\n",
       "                                                                   CountVectorizer(binary=True,\n",
       "                                                                                   max_features=50))]),\n",
       "                                                  'summary'),\n",
       "                                                 ('text_pre_1',\n",
       "                                                  Pipeline(steps=[('text_vect_1',\n",
       "                                                                   CountVectorizer(binary=True,\n",
       "                                                                                   max_features=150))]),\n",
       "                                                  'reviewText')])),\n",
       "                ('decision_tree',\n",
       "                 DecisionTreeClassifier(max_depth=10, min_samples_leaf=15))])
ColumnTransformer(transformers=[('numerical_pre',\n",
       "                                 Pipeline(steps=[('num_scaler',\n",
       "                                                  MinMaxScaler())]),\n",
       "                                 ['time', 'log_votes']),\n",
       "                                ('text_pre_0',\n",
       "                                 Pipeline(steps=[('text_vect_0',\n",
       "                                                  CountVectorizer(binary=True,\n",
       "                                                                  max_features=50))]),\n",
       "                                 'summary'),\n",
       "                                ('text_pre_1',\n",
       "                                 Pipeline(steps=[('text_vect_1',\n",
       "                                                  CountVectorizer(binary=True,\n",
       "                                                                  max_features=150))]),\n",
       "                                 'reviewText')])
['time', 'log_votes']
MinMaxScaler()
summary
CountVectorizer(binary=True, max_features=50)
reviewText
CountVectorizer(binary=True, max_features=150)
DecisionTreeClassifier(max_depth=10, min_samples_leaf=15)
" ], "text/plain": [ "Pipeline(steps=[('data_preprocessing',\n", " ColumnTransformer(transformers=[('numerical_pre',\n", " Pipeline(steps=[('num_scaler',\n", " MinMaxScaler())]),\n", " ['time', 'log_votes']),\n", " ('text_pre_0',\n", " Pipeline(steps=[('text_vect_0',\n", " CountVectorizer(binary=True,\n", " max_features=50))]),\n", " 'summary'),\n", " ('text_pre_1',\n", " Pipeline(steps=[('text_vect_1',\n", " CountVectorizer(binary=True,\n", " max_features=150))]),\n", " 'reviewText')])),\n", " ('decision_tree',\n", " DecisionTreeClassifier(max_depth=10, min_samples_leaf=15))])" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.impute import SimpleImputer\n", "from sklearn.preprocessing import MinMaxScaler\n", "from sklearn.feature_extraction.text import CountVectorizer\n", "from sklearn.pipeline import Pipeline\n", "from sklearn.compose import ColumnTransformer\n", "from sklearn.tree import DecisionTreeClassifier\n", "\n", "### COLUMN_TRANSFORMER ###\n", "##########################\n", "\n", "# Preprocess the numerical features\n", "numerical_processor = Pipeline([\n", " ('num_scaler', MinMaxScaler()) # this can be skipped for trees\n", "])\n", "\n", "# Preprocess 1st text feature\n", "text_processor_0 = Pipeline([\n", " ('text_vect_0', CountVectorizer(binary=True, max_features=50))\n", "])\n", "\n", "# Preprocess 2nd text feature (larger vocabulary)\n", "text_precessor_1 = Pipeline([\n", " ('text_vect_1', CountVectorizer(binary=True, max_features=150))\n", "])\n", "\n", "# Combine all data preprocessors from above (add more, if you choose to define more!)\n", "# For each processor/step specify: a name, the actual process, and finally the features to be processed\n", "data_preprocessor = ColumnTransformer([\n", " ('numerical_pre', numerical_processor, numerical_features),\n", " ('text_pre_0', text_processor_0, text_features[0]),\n", " ('text_pre_1', text_precessor_1, text_features[1])\n", "]) \n", "\n", "### PIPELINE ###\n", "################\n", "\n", "# Pipeline desired all data transformers, along with an estimator at the end\n", "# Later you can set/reach the parameters using the names issued - for hyperparameter tuning, for example\n", "pipeline = Pipeline([\n", " ('data_preprocessing', data_preprocessor),\n", " ('decision_tree', DecisionTreeClassifier(max_depth = 10,\n", " min_samples_leaf = 15))\n", "])\n", "\n", "# Visualize the pipeline\n", "# This will come in handy especially when building more complex pipelines, stringing together multiple preprocessing steps\n", "from sklearn import set_config\n", "set_config(display='diagram')\n", "pipeline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 6. Fit the Decision Tree classifier\n", "(Go to top)\n", "\n", "We train our model by using __.fit()__ on our training dataset. " ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
Pipeline(steps=[('data_preprocessing',\n",
       "                 ColumnTransformer(transformers=[('numerical_pre',\n",
       "                                                  Pipeline(steps=[('num_scaler',\n",
       "                                                                   MinMaxScaler())]),\n",
       "                                                  ['time', 'log_votes']),\n",
       "                                                 ('text_pre_0',\n",
       "                                                  Pipeline(steps=[('text_vect_0',\n",
       "                                                                   CountVectorizer(binary=True,\n",
       "                                                                                   max_features=50))]),\n",
       "                                                  'summary'),\n",
       "                                                 ('text_pre_1',\n",
       "                                                  Pipeline(steps=[('text_vect_1',\n",
       "                                                                   CountVectorizer(binary=True,\n",
       "                                                                                   max_features=150))]),\n",
       "                                                  'reviewText')])),\n",
       "                ('decision_tree',\n",
       "                 DecisionTreeClassifier(max_depth=10, min_samples_leaf=15))])
ColumnTransformer(transformers=[('numerical_pre',\n",
       "                                 Pipeline(steps=[('num_scaler',\n",
       "                                                  MinMaxScaler())]),\n",
       "                                 ['time', 'log_votes']),\n",
       "                                ('text_pre_0',\n",
       "                                 Pipeline(steps=[('text_vect_0',\n",
       "                                                  CountVectorizer(binary=True,\n",
       "                                                                  max_features=50))]),\n",
       "                                 'summary'),\n",
       "                                ('text_pre_1',\n",
       "                                 Pipeline(steps=[('text_vect_1',\n",
       "                                                  CountVectorizer(binary=True,\n",
       "                                                                  max_features=150))]),\n",
       "                                 'reviewText')])
['time', 'log_votes']
MinMaxScaler()
summary
CountVectorizer(binary=True, max_features=50)
reviewText
CountVectorizer(binary=True, max_features=150)
DecisionTreeClassifier(max_depth=10, min_samples_leaf=15)
" ], "text/plain": [ "Pipeline(steps=[('data_preprocessing',\n", " ColumnTransformer(transformers=[('numerical_pre',\n", " Pipeline(steps=[('num_scaler',\n", " MinMaxScaler())]),\n", " ['time', 'log_votes']),\n", " ('text_pre_0',\n", " Pipeline(steps=[('text_vect_0',\n", " CountVectorizer(binary=True,\n", " max_features=50))]),\n", " 'summary'),\n", " ('text_pre_1',\n", " Pipeline(steps=[('text_vect_1',\n", " CountVectorizer(binary=True,\n", " max_features=150))]),\n", " 'reviewText')])),\n", " ('decision_tree',\n", " DecisionTreeClassifier(max_depth=10, min_samples_leaf=15))])" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Fit the Pipeline to training data\n", "pipeline.fit(X_train, y_train.values)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 7. Test the classifier\n", "(Go to top)\n", "\n", "Let's evaluate the performance of the trained classifier. We use __.predict()__ this time. " ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[1840 765]\n", " [ 716 3679]]\n", " precision recall f1-score support\n", "\n", " 0.0 0.72 0.71 0.71 2605\n", " 1.0 0.83 0.84 0.83 4395\n", "\n", " accuracy 0.79 7000\n", " macro avg 0.77 0.77 0.77 7000\n", "weighted avg 0.79 0.79 0.79 7000\n", "\n", "Accuracy (validation): 0.7884285714285715\n" ] } ], "source": [ "from sklearn.metrics import confusion_matrix, classification_report, accuracy_score\n", "\n", "# Use the fitted pipeline to make predictions on the validation dataset\n", "val_predictions = pipeline.predict(X_val)\n", "print(confusion_matrix(y_val.values, val_predictions))\n", "print(classification_report(y_val.values, val_predictions))\n", "print(\"Accuracy (validation):\", accuracy_score(y_val.values, val_predictions))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8. Fit and test the Random Forest classifier\n", "(Go to top)\n", "\n", "This time, we will use the Random Forest classifier. Let's update our pipeline for that" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
Pipeline(steps=[('data_preprocessing',\n",
       "                 ColumnTransformer(transformers=[('numerical_pre',\n",
       "                                                  Pipeline(steps=[('num_scaler',\n",
       "                                                                   MinMaxScaler())]),\n",
       "                                                  ['time', 'log_votes']),\n",
       "                                                 ('text_pre_0',\n",
       "                                                  Pipeline(steps=[('text_vect_0',\n",
       "                                                                   CountVectorizer(binary=True,\n",
       "                                                                                   max_features=50))]),\n",
       "                                                  'summary'),\n",
       "                                                 ('text_pre_1',\n",
       "                                                  Pipeline(steps=[('text_vect_1',\n",
       "                                                                   CountVectorizer(binary=True,\n",
       "                                                                                   max_features=150))]),\n",
       "                                                  'reviewText')])),\n",
       "                ('decision_tree',\n",
       "                 RandomForestClassifier(max_depth=10, min_samples_leaf=15,\n",
       "                                        n_estimators=150))])
ColumnTransformer(transformers=[('numerical_pre',\n",
       "                                 Pipeline(steps=[('num_scaler',\n",
       "                                                  MinMaxScaler())]),\n",
       "                                 ['time', 'log_votes']),\n",
       "                                ('text_pre_0',\n",
       "                                 Pipeline(steps=[('text_vect_0',\n",
       "                                                  CountVectorizer(binary=True,\n",
       "                                                                  max_features=50))]),\n",
       "                                 'summary'),\n",
       "                                ('text_pre_1',\n",
       "                                 Pipeline(steps=[('text_vect_1',\n",
       "                                                  CountVectorizer(binary=True,\n",
       "                                                                  max_features=150))]),\n",
       "                                 'reviewText')])
['time', 'log_votes']
MinMaxScaler()
summary
CountVectorizer(binary=True, max_features=50)
reviewText
CountVectorizer(binary=True, max_features=150)
RandomForestClassifier(max_depth=10, min_samples_leaf=15, n_estimators=150)
" ], "text/plain": [ "Pipeline(steps=[('data_preprocessing',\n", " ColumnTransformer(transformers=[('numerical_pre',\n", " Pipeline(steps=[('num_scaler',\n", " MinMaxScaler())]),\n", " ['time', 'log_votes']),\n", " ('text_pre_0',\n", " Pipeline(steps=[('text_vect_0',\n", " CountVectorizer(binary=True,\n", " max_features=50))]),\n", " 'summary'),\n", " ('text_pre_1',\n", " Pipeline(steps=[('text_vect_1',\n", " CountVectorizer(binary=True,\n", " max_features=150))]),\n", " 'reviewText')])),\n", " ('decision_tree',\n", " RandomForestClassifier(max_depth=10, min_samples_leaf=15,\n", " n_estimators=150))])" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.ensemble import RandomForestClassifier\n", "\n", "pipeline = Pipeline([\n", " ('data_preprocessing', data_preprocessor),\n", " ('decision_tree', RandomForestClassifier(n_estimators=150,\n", " max_depth = 10,\n", " min_samples_leaf = 15))\n", "])\n", "\n", "# Fit the Pipeline to training data\n", "pipeline.fit(X_train, y_train.values)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's get the predictions on our validation data." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[1738 867]\n", " [ 411 3984]]\n", " precision recall f1-score support\n", "\n", " 0.0 0.81 0.67 0.73 2605\n", " 1.0 0.82 0.91 0.86 4395\n", "\n", " accuracy 0.82 7000\n", " macro avg 0.82 0.79 0.80 7000\n", "weighted avg 0.82 0.82 0.81 7000\n", "\n", "Accuracy (validation): 0.8174285714285714\n" ] } ], "source": [ "from sklearn.metrics import confusion_matrix, classification_report, accuracy_score\n", "\n", "# Use the fitted pipeline to make predictions on the validation dataset\n", "val_predictions = pipeline.predict(X_val)\n", "print(confusion_matrix(y_val.values, val_predictions))\n", "print(classification_report(y_val.values, val_predictions))\n", "print(\"Accuracy (validation):\", accuracy_score(y_val.values, val_predictions))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 9. Hyperparameter Tuning\n", "(Go to top)\n", "\n", "Let's try different parameter values and see how the __DecisionTreeClassifier__ model performs under some combinations of parameters.\n", "\n", "__Warning__: The number of hyperparameters tuned, along with the cross-validations, can greatly increase training time! Especially if trying hyperparameters tuning on the __RandomForestClassifier__ instead of the lower performing __DecisionTreeClassifier__ that we showcase below for speed! Similar tuning on a __RandomForestClassifier__ model can take more minutes to hours!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 9.1 GridSearchCV\n", "\n", "Find more details on the __GridSearchCV__ here:\n", "https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Fitting 5 folds for each of 6 candidates, totalling 30 fits\n", "Best parameters: {'decision_tree__max_depth': 30, 'decision_tree__min_samples_leaf': 5}\n", "Best score: 0.8414761904761905\n" ] } ], "source": [ "from sklearn.model_selection import GridSearchCV\n", "\n", "### PIPELINE GRID_SEARCH ###\n", "############################\n", "\n", "# Parameter grid for GridSearch\n", "param_grid={'decision_tree__max_depth': [10, 20, 30],#, 15, 25, 35, 45, 55, 75], \n", " 'decision_tree__min_samples_leaf': [5, 10],#, 15, 30],\n", " }\n", "\n", "grid_search = GridSearchCV(pipeline, # Base model\n", " param_grid, # Parameters to try\n", " cv = 5, # Apply 5-fold cross validation\n", " verbose = 1, # Print summary\n", " n_jobs = -1 # Use all available processors\n", " )\n", "\n", "# Fit the GridSearch to our training data\n", "grid_search.fit(X_train, y_train)\n", "\n", "print(\"Best parameters: \", grid_search.best_params_)\n", "print(\"Best score: \", grid_search.best_score_)" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[1982 623]\n", " [ 468 3927]]\n", " precision recall f1-score support\n", "\n", " 0.0 0.81 0.76 0.78 2605\n", " 1.0 0.86 0.89 0.88 4395\n", "\n", " accuracy 0.84 7000\n", " macro avg 0.84 0.83 0.83 7000\n", "weighted avg 0.84 0.84 0.84 7000\n", "\n", "Accuracy (validation): 0.8441428571428572\n" ] } ], "source": [ "from sklearn.metrics import confusion_matrix, classification_report, accuracy_score\n", "\n", "# Use the fitted pipeline to make predictions on the validation dataset\n", "val_predictions = grid_search.best_estimator_.predict(X_val)\n", "print(confusion_matrix(y_val.values, val_predictions))\n", "print(classification_report(y_val.values, val_predictions))\n", "print(\"Accuracy (validation):\", accuracy_score(y_val.values, val_predictions))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 8.2 RandomizedSearchCV\n", "\n", "Find more details on the __RandomizedSearchCV__ here:\n", "https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Fitting 5 folds for each of 6 candidates, totalling 30 fits\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages/sklearn/model_selection/_search.py:292: UserWarning: The total space of parameters 6 is smaller than n_iter=10. Running 6 iterations. For exhaustive searches, use GridSearchCV.\n", " warnings.warn(\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Best parameters: {'decision_tree__min_samples_leaf': 5, 'decision_tree__max_depth': 30}\n", "Best score: 0.8424920634920635\n" ] } ], "source": [ "from sklearn.model_selection import RandomizedSearchCV\n", "\n", "# Parameter grid for GridSearch\n", "param_grid={'decision_tree__max_depth': [10, 20, 30],#, 15, 25, 35, 45, 55, 75], \n", " 'decision_tree__min_samples_leaf': [5, 10],#, 15, 30],\n", " }\n", "\n", "random_search = RandomizedSearchCV(pipeline, # Base model\n", " param_grid, # Parameters to try\n", " cv = 5, # Apply 5-fold cross validation\n", " verbose = 1, # Print summary\n", " n_jobs = -1 # Use all available processors\n", " )\n", "\n", "# Fit the GridSearch to our training data\n", "random_search.fit(X_train, y_train)\n", "\n", "print(\"Best parameters: \", random_search.best_params_)\n", "print(\"Best score: \", random_search.best_score_)" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[1983 622]\n", " [ 469 3926]]\n", " precision recall f1-score support\n", "\n", " 0.0 0.81 0.76 0.78 2605\n", " 1.0 0.86 0.89 0.88 4395\n", "\n", " accuracy 0.84 7000\n", " macro avg 0.84 0.83 0.83 7000\n", "weighted avg 0.84 0.84 0.84 7000\n", "\n", "Accuracy (validation): 0.8441428571428572\n" ] } ], "source": [ "from sklearn.metrics import confusion_matrix, classification_report, accuracy_score\n", "\n", "# Use the fitted pipeline to make predictions on the validation dataset\n", "val_predictions = random_search.best_estimator_.predict(X_val)\n", "print(confusion_matrix(y_val.values, val_predictions))\n", "print(classification_report(y_val.values, val_predictions))\n", "print(\"Accuracy (validation):\", accuracy_score(y_val.values, val_predictions))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 10. Ideas for improvement\n", "(Go to top)\n", "\n", "**Preprocessing**: We can usually improve performance with some additional work. You can try the following:\n", "* Change the feature extractor to TF, TF-IDF. Also experiment with different vocabulary size.\n", "* Come up with some other features such as having certain punctuations, all-capitalized words or some words that might be useful in this problem.\n", "\n", "**Hyperparameter Tuning**: Always a good idea to try other parameter ranges and/or combinations of parameters. If training time is a priority, try __RandomizedSearchCV__ instead of __GridSearchCV__, it's much faster and with almost as good results. " ] } ], "metadata": { "kernelspec": { "display_name": "conda_pytorch_p39", "language": "python", "name": "conda_pytorch_p39" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.13" } }, "nbformat": 4, "nbformat_minor": 2 }