{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Introduction\n", "\n", "In this workshop, we will go through the steps of training, debugging, deploying and monitoring a **network traffic classification model**.\n", "\n", "For training our model we will be using datasets CSE-CIC-IDS2018 by CIC and ISCX which are used for security testing and malware prevention.\n", "These datasets include a huge amount of raw network traffic logs, plus pre-processed data where network connections have been reconstructed and relevant features have been extracted using CICFlowMeter, a tool that outputs network connection features as CSV files. Each record is classified as benign traffic, or it can be malicious traffic, with a total number of 15 classes.\n", "\n", "Starting from this featurized dataset, we have executed additional pre-processing for the purpose of this lab:\n", "\n", "\n", "Class are represented and have been encoded as follows (train + validation):\n", "\n", "\n", "| Label | Encoded | N. records |\n", "|:-------------------------|:-------:|-----------:|\n", "| Benign | 0 | 1000000 |\n", "| Bot | 1 | 200000 |\n", "| DoS attacks-GoldenEye | 2 | 40000 |\n", "| DoS attacks-Slowloris | 3 | 10000 |\n", "| DDoS attacks-LOIC-HTTP | 4 | 300000 |\n", "| Infilteration | 5 | 150000 |\n", "| DDOS attack-LOIC-UDP | 6 | 1730 |\n", "| DDOS attack-HOIC | 7 | 300000 |\n", "| Brute Force -Web | 8 | 611 |\n", "| Brute Force -XSS | 9 | 230 |\n", "| SQL Injection | 10 | 87 |\n", "| DoS attacks-SlowHTTPTest | 11 | 100000 |\n", "| DoS attacks-Hulk | 12 | 250000 |\n", "| FTP-BruteForce | 13 | 150000 |\n", "| SSH-Bruteforce | 14 | 150000 | \n", "\n", "The final pre-processed dataset has been saved to a public Amazon S3 bucket for your convenience, and will represent the inputs to the training processes.\n", "\n", "### Let's get started!\n", "\n", "First, we set some variables, including the AWS region we are working in, the IAM (Identity and Access Management) execution role of the notebook instance and the Amazon S3 bucket where we will store data, models, outputs, etc. We will use the Amazon SageMaker default bucket for the selected AWS region, and then define a key prefix to make sure all objects have share the same prefix for easier discoverability." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import os\n", "import boto3\n", "import sagemaker\n", "\n", "region = boto3.Session().region_name\n", "role = sagemaker.get_execution_role()\n", "sagemaker_session = sagemaker.Session()\n", "bucket_name = sagemaker.Session().default_bucket()\n", "prefix = 'aim362'\n", "os.environ[\"AWS_REGION\"] = region\n", "\n", "print(region)\n", "print(role)\n", "print(bucket_name)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we can copy the dataset from the public Amazon S3 bucket to the Amazon SageMaker default bucket used in this workshop. To do this, we will leverage on the AWS Python SDK (boto3) as follows:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import boto3\n", "\n", "s3 = boto3.resource('s3')\n", "\n", "source_bucket_name = \"endtoendmlapp\"\n", "source_bucket_prefix = \"aim362/data/\"\n", "source_bucket = s3.Bucket(source_bucket_name)\n", "\n", "for s3_object in source_bucket.objects.filter(Prefix=source_bucket_prefix):\n", " copy_source = {\n", " 'Bucket': source_bucket_name,\n", " 'Key': s3_object.key\n", " }\n", " print('Copying {0} ...'.format(s3_object.key))\n", " s3.Bucket(bucket_name).copy(copy_source, s3_object.key)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's download some of the data to the notebook to quickly explore the dataset structure:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "train_file_path = 's3://' + bucket_name + '/' + prefix + '/data/train/0.part'\n", "val_file_path = 's3://' + bucket_name + '/' + prefix + '/data/val/0.part'" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!mkdir -p data/train/ data/val/\n", "!aws s3 cp {train_file_path} data/train/\n", "!aws s3 cp {val_file_path} data/val/" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "pd.options.display.max_columns = 100\n", "\n", "df = pd.read_csv('data/train/0.part')\n", "df.head(10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Training and Debugging\n", "\n", "The network traffic classification model will be trained using the Amazon SageMaker framework container for XGBoost (https://github.com/aws/sagemaker-xgboost-container). Using XGBoost as a framework provides more flexibility than using it as a built-in algorithm as it enables more advanced scenarios that allow pre-processing and post-processing scripts or any kind of custom logic to be incorporated into your training script.\n", "\n", "First, we will execute basic training to make sure our training script works as expected and we are able to fit the model successfully, and then we will go through the steps for enabling debugging using Amazon SageMaker Debugger." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Basic Training\n", "\n", "We will execute the training script in local mode while building our model: local mode is a functionality enabled by the Amazon SageMaker Python SDK that allows running the same training code and container that will be used in Amazon SageMaker locally on the notebook instance, in order to speed-up experimentation and quickly fix errors before running training with Amazon SageMaker training.\n", "\n", "For local mode training, we can re-use the training and validation files downloaded on the notebook instance in the previous steps, as local file inputs." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's take a look at our training script." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pygmentize source_dir/train_xgboost_no_debug.py" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The script parses arguments that are passed when the XGBoost Docker container code invokes the script for execution. These arguments represent the hyperparameters that you specify when strarting the training job plus the location of training and validation data; this behavior, named Script Mode execution, is enabled by a library that is installed in the XGBoost container (sagemaker-containers, https://github.com/aws/sagemaker-containers) and facilitates the development of SageMaker-compatible Docker containers.\n", "\n", "Then, we load training and validation data and execute XGBoost training with the provided parameters." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Once we have our script ready, we can leverage on the XGBoost estimator of the Amazon SageMaker Python SDK to start training locally." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sagemaker.xgboost import XGBoost\n", "\n", "hyperparameters = {\n", " \"max_depth\": \"3\",\n", " \"eta\": \"0.1\",\n", " \"gamma\": \"6\",\n", " \"min_child_weight\": \"6\",\n", " \"silent\": \"0\",\n", " \"objective\": \"multi:softmax\",\n", " \"num_class\": \"15\",\n", " \"num_round\": \"10\"\n", "}\n", "\n", "entry_point='train_xgboost_no_debug.py'\n", "source_dir='source_dir/'\n", "output_path = 's3://{0}/{1}/output/'.format(bucket_name, prefix)\n", "code_location = 's3://{0}/{1}/code'.format(bucket_name, prefix)\n", "\n", "estimator = XGBoost(\n", " base_job_name=\"nw-traffic-classification-xgb\",\n", " entry_point=entry_point,\n", " source_dir=source_dir,\n", " output_path=output_path,\n", " code_location=code_location,\n", " hyperparameters=hyperparameters,\n", " train_instance_type=\"local\", # Specifying local as instance type to run local-mode training\n", " train_instance_count=1,\n", " framework_version=\"0.90-2\",\n", " py_version=\"py3\",\n", " role=role\n", ")\n", "\n", "train_config = 'file://data/train/'\n", "val_config = 'file://data/val/'\n", "\n", "estimator.fit({'train': train_config, 'validation': val_config })" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In order to make sure that our code works for inference, we can deploy the trained model locally and execute some inferences." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "predictor = estimator.deploy(initial_instance_count=1,\n", " instance_type='local') # Using local-mode deployment" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sagemaker.predictor import csv_serializer, json_deserializer\n", "from sagemaker.predictor import RealTimePredictor\n", "\n", "predictor.content_type = 'text/csv'\n", "predictor.serializer = csv_serializer\n", "predictor.deserializer = json_deserializer\n", "\n", "# We expect 4 - DDoS attacks-LOIC-HTTP as the predicted class for this instance.\n", "test_values = [80,1056736,3,4,20,964,20,0,6.666666667,11.54700538,964,0,241.0,482.0,931.1691850999999,6.6241710320000005,176122.6667,431204.4454,1056315,2,394,197.0,275.77164469999997,392,2,1056733,352244.3333,609743.1115,1056315,24,0,0,0,0,72,92,2.8389304419999997,3.78524059,0,964,123.0,339.8873763,115523.4286,0,0,1,1,0,0,0,1,1.0,140.5714286,6.666666667,241.0,0.0,0.0,0.0,0.0,0.0,0.0,3,20,4,964,8192,211,1,20,0.0,0.0,0,0,0.0,0.0,0,0,20,2,2018,1,0,1,0]\n", "result = predictor.predict(test_values)\n", "print(result)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally, let's gracefully stop the deployed local endpoint." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "predictor.delete_endpoint()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Results\n", "During training, we have seen that both the train-merror and validation-merror are decreasing, although we don't have details on the accuracy per-class (we will address this later). We have also successfully deployed the model to a local endpoint and executed inferences." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Debugging" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### LossNotDecreasing\n", "\n", "Once we are confident our training script is working as expected and there are no major errors preventing its execution, we can enable debugging.\n", "\n", "During training, we will save the state of the tensors using Amazon SageMaker debugging features, and then analyze debugging outputs with jobs that are run while the training job is executed. For XGBoost, Amazon SageMaker debugging supports saving evaluation metrics, lebels and predictions, feature importances, and SHAP values.\n", "\n", "First, we need to modify our training script to enable Amazon SageMaker debugging. Note that this is required for the XGBoost framework, whilst for MXNet and Tensorflow debugging works also with no code changes.\n", "\n", "We created a Hook object which we pass as a callback function when creating a Booster. The Hook object is created by loading a JSON configuration that is available in a specific path in the Docker container (opt/ml/input/config/debughookconfig.json); this file is generated by Amazon SageMaker from the CreateTrainingJob() API call configuration. Note that Amazon SageMaker debugging is highly configurable, you can choose exactly what to save.\n", "\n", "Let's look at the modified script:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pygmentize source_dir/train_xgboost.py" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The modified script allows to **capture tensors** and **save to Amazon S3**, but doing this will not cause any debug analysis to run. In order to analyze debug outputs we need to configure the XGBoost estimator to define **a collection of rules that will be run while the training job is executed**." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We are enabling a built-in (1P) debug rule named **LossNotDecreasing** which checks if the loss is not decreasing across step. In this scenario, we have chosen to run this rule at every step on the validation-merror metric values: this means that the new merror values must always go down at each step.\n", "\n", "When the estimator fit() method is called, Amazon SageMaker will start two jobs: a **Training Job**, where we also capture and save tensors, and a debug **Processing Job** (powered by **Amazon SageMaker Processing Jobs**), which will run in parallel and analyze tensor data to check if the rule conditions are met.\n", "\n", "Note that we are passing the **Wait=False** parameter to the fit() method to avoid waiting for the training job to complete and just fire and forget the API call." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sagemaker.xgboost import XGBoost\n", "from sagemaker.debugger import Rule, rule_configs, DebuggerHookConfig, CollectionConfig\n", "\n", "hyperparameters = {\n", " \"max_depth\": \"10\",\n", " \"eta\": \"0.2\",\n", " \"gamma\": \"1\",\n", " \"min_child_weight\": \"6\",\n", " \"silent\": \"0\",\n", " \"objective\": \"multi:softmax\",\n", " \"num_class\": \"15\",\n", " \"num_round\": \"20\"\n", "}\n", "\n", "entry_point='train_xgboost.py'\n", "source_dir='source_dir/'\n", "output_path = 's3://{0}/{1}/output/'.format(bucket_name, prefix)\n", "debugger_output_path = 's3://{0}/{1}/output/debug'.format(bucket_name, prefix) # Path where we save debug outputs\n", "code_location = 's3://{0}/{1}/code'.format(bucket_name, prefix)\n", "\n", "hook_config = DebuggerHookConfig(\n", " s3_output_path=debugger_output_path,\n", " hook_parameters={\n", " \"save_interval\": \"1\"\n", " },\n", " collection_configs=[\n", " CollectionConfig(\"hyperparameters\"),\n", " CollectionConfig(\"metrics\"),\n", " CollectionConfig(\"predictions\"),\n", " CollectionConfig(\"labels\"),\n", " CollectionConfig(\"feature_importance\")\n", " ]\n", ")\n", "\n", "estimator = XGBoost(\n", " base_job_name=\"nw-traffic-classification-xgb\",\n", " entry_point=entry_point,\n", " source_dir=source_dir,\n", " output_path=output_path,\n", " code_location=code_location,\n", " hyperparameters=hyperparameters,\n", " train_instance_type=\"ml.m5.4xlarge\",\n", " train_instance_count=1,\n", " framework_version=\"0.90-2\",\n", " py_version=\"py3\",\n", " role=role,\n", " \n", " # Initialize your hook.\n", " debugger_hook_config=hook_config,\n", " \n", " # Initialize your rules. These will read data for analyses from the path specified\n", " # for the hook\n", " rules=[Rule.sagemaker(rule_configs.loss_not_decreasing(),\n", " rule_parameters={\n", " # Rule does not use the default losses collection,\n", " # but uses a regex to look for specific tensor values\n", " \"use_losses_collection\": \"False\",\n", " \"tensor_regex\": \"validation-merror\",\n", " # Num steps is used to specify when to evaluate this rule (every num_steps)\n", " \"num_steps\" : \"1\"}\n", " )]\n", ")\n", "\n", "train_config = sagemaker.session.s3_input('s3://{0}/{1}/data/train/'.format(\n", " bucket_name, prefix), content_type='text/csv')\n", "val_config = sagemaker.session.s3_input('s3://{0}/{1}/data/val/'.format(\n", " bucket_name, prefix), content_type='text/csv')\n", "\n", "estimator.fit({'train': train_config, 'validation': val_config }, wait=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Once the training job has started, we can check its debug configuration and status:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import time\n", "client = estimator.sagemaker_session.sagemaker_client\n", "\n", "description = client.describe_training_job(TrainingJobName=estimator.latest_training_job.name)\n", "print('Debug Hook configuration: ')\n", "print(description['DebugHookConfig'])\n", "print()\n", "print('Debug rules configuration: ')\n", "print(description['DebugRuleConfigurations'])\n", "print()\n", "print('Training job status')\n", "print(description['TrainingJobStatus'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also get all the logs for the training job being executed:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "sagemaker_session.logs_for_job(estimator.latest_training_job.name)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "At the same time, we can check the status of the rule execution job as follows. Note that this requires some time, so you might be interested in looking at the SageMaker Debugger documentation while this runs: https://github.com/awslabs/sagemaker-debugger." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import time\n", "\n", "client = estimator.sagemaker_session.sagemaker_client\n", "\n", "iterate = True\n", "while(iterate):\n", " description = client.describe_training_job(TrainingJobName=estimator.latest_training_job.name)\n", " eval_status = description['DebugRuleEvaluationStatuses'][0]\n", " print(eval_status)\n", " if eval_status['RuleEvaluationStatus'] != 'InProgress':\n", " iterate = False\n", " else:\n", " time.sleep(60)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The rule execution job raised an error since the rule evaluation condition is met. Let's review the configuration and logs of the rule execution job, executed by Amazon SageMaker Processing Jobs:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "processing_job_arn = eval_status['RuleEvaluationJobArn']\n", "processing_job_name = processing_job_arn[processing_job_arn.rfind('/') + 1 :]\n", "print(processing_job_name)\n", "\n", "client = estimator.sagemaker_session.sagemaker_client\n", "descr = client.describe_processing_job(ProcessingJobName=processing_job_name)\n", "descr" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "sagemaker_session.logs_for_processing_job(descr['ProcessingJobName'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Results\n", "We can see that the condition is being met at step 7 when validation-merror is not decreasing. When this happens, we might be interested in stopping training earlier. You can also leverage on Amazon CloudWatch Events to detect the rule condition met event and take specific actions automatically. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Debugging - Confusion" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As another example of using a first party (1P) rule provided by Amazon SageMaker debugging, let us again train and use a 1P rule `Confusion` to monitor the training job in realtime.\n", "\n", "During training, `Confusion` Rule job will monitor whether you are running into a situation where the ratio of on-diagonal and off-diagonal values in the confusion matrix is not within a specified range. In other words, this rule evaluates the goodness of a confusion matrix for a classification problem. It creates a matrix of size `category_no` $\\times$ `category_no` and populates it with data coming from (`y`, `y_hat`) pairs. For each (`y`, `y_hat`) pairs the count in `confusion[y][y_hat]` is incremented by 1. Once the matrix is fully populated, the ratio of data on- and off-diagonal will be evaluated according to:\n", "\n", "- For elements on the diagonal:\n", "\n", "$$ \\frac{ \\text{confusion}_{ii} }{ \\sum_j \\text{confusion}_{jj} } \\geq \\text{min_diag} $$\n", "\n", "- For elements off the diagonal:\n", "\n", "$$ \\frac{ \\text{confusion}_{ji} }{ \\sum_j \\text{confusion}_{ji} } \\leq \\text{max_off_diag} $$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Please note that in this case we are setting the `start_step` and `end_step` rule parameters, to make sure the rule is evaluated only during the latest steps." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sagemaker.xgboost import XGBoost\n", "from sagemaker.debugger import Rule, rule_configs, DebuggerHookConfig\n", "\n", "hyperparameters = {\n", " \"max_depth\": \"10\",\n", " \"eta\": \"0.2\",\n", " \"gamma\": \"1\",\n", " \"min_child_weight\": \"6\",\n", " \"silent\": \"0\",\n", " \"objective\": \"multi:softmax\",\n", " \"num_class\": \"15\",\n", " \"num_round\": \"20\"\n", "}\n", "\n", "entry_point='train_xgboost.py'\n", "source_dir='source_dir/'\n", "output_path = 's3://{0}/{1}/output/'.format(bucket_name, prefix)\n", "debugger_output_path = 's3://{0}/{1}/output/debug'.format(bucket_name, prefix)\n", "code_location = 's3://{0}/{1}/code'.format(bucket_name, prefix)\n", "\n", "hook_config = DebuggerHookConfig(\n", " s3_output_path=debugger_output_path,\n", " hook_parameters={\n", " \"save_interval\": \"1\"\n", " },\n", " collection_configs=[\n", " CollectionConfig(\"hyperparameters\"),\n", " CollectionConfig(\"metrics\"),\n", " CollectionConfig(\"predictions\"),\n", " CollectionConfig(\"labels\"),\n", " CollectionConfig(\"feature_importance\")\n", " ]\n", ")\n", "\n", "estimator = XGBoost(\n", " base_job_name=\"nw-traffic-classification-xgb\",\n", " entry_point=entry_point,\n", " source_dir=source_dir,\n", " output_path=output_path,\n", " code_location=code_location,\n", " hyperparameters=hyperparameters,\n", " train_instance_type=\"ml.m5.4xlarge\",\n", " train_instance_count=1,\n", " framework_version=\"0.90-2\",\n", " py_version=\"py3\",\n", " role=role,\n", " \n", " # Initialize your hook.\n", " debugger_hook_config=hook_config,\n", " \n", " # Initialize your rules. These will read data for analyses from the path specified\n", " # for the hook\n", " rules=[Rule.sagemaker(rule_configs.confusion(),\n", " rule_parameters={\n", " \"category_no\": \"15\",\n", " \"min_diag\": \"0.7\",\n", " \"max_off_diag\": \"0.3\",\n", " \"start_step\": \"17\",\n", " \"end_step\": \"19\"}\n", " )]\n", ")\n", "\n", "train_config = sagemaker.session.s3_input('s3://{0}/{1}/data/train/'.format(\n", " bucket_name, prefix), content_type='text/csv')\n", "val_config = sagemaker.session.s3_input('s3://{0}/{1}/data/val/'.format(\n", " bucket_name, prefix), content_type='text/csv')\n", "\n", "estimator.fit({'train': train_config, 'validation': val_config }, wait=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Agan, let's review the training job status, configuration and logs:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import time\n", "client = estimator.sagemaker_session.sagemaker_client\n", "\n", "description = client.describe_training_job(TrainingJobName=estimator.latest_training_job.name)\n", "print('Debug Hook configuration: ')\n", "print(description['DebugHookConfig'])\n", "print()\n", "print('Debug rules configuration: ')\n", "print(description['DebugRuleConfigurations'])\n", "print()\n", "print('Training job status')\n", "print(description['TrainingJobStatus'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "sagemaker_session.logs_for_job(estimator.latest_training_job.name)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then, we can wait for the rule execution to complete:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import time\n", "\n", "client = estimator.sagemaker_session.sagemaker_client\n", "\n", "iterate = True\n", "while(iterate):\n", " description = client.describe_training_job(TrainingJobName=estimator.latest_training_job.name)\n", " eval_status = description['DebugRuleEvaluationStatuses'][0]\n", " print(eval_status)\n", " if eval_status['RuleEvaluationStatus'] != 'InProgress':\n", " iterate = False\n", " else:\n", " time.sleep(60)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's review the rule execution job, executed by Amazon SageMaker Processing:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "processing_job_arn = eval_status['RuleEvaluationJobArn']\n", "processing_job_name = processing_job_arn[processing_job_arn.rfind('/') + 1 :]\n", "print(processing_job_name)\n", "\n", "client = estimator.sagemaker_session.sagemaker_client\n", "descr = client.describe_processing_job(ProcessingJobName=processing_job_name)\n", "descr" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "sagemaker_session.logs_for_processing_job(descr['ProcessingJobName'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's also make sure the training job is completed." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import time\n", "\n", "client = estimator.sagemaker_session.sagemaker_client\n", "\n", "iterate = True\n", "while(iterate):\n", " description = client.describe_training_job(TrainingJobName=estimator.latest_training_job.name)\n", " training_job_status = description['TrainingJobStatus']\n", " print(training_job_status)\n", " if training_job_status != 'InProgress':\n", " iterate = False\n", " else:\n", " time.sleep(60)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Results\n", "We can see that the condition is being met and this gives evidences that our confusion matrix is not matching our thresholds.\n", "Let's review the confusion matrix by analyzing the debug outputs in next section." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Analyzing Debug Outputs\n", "\n", "In this section we will see how you can use the SDK to manually analyze debug outputs.\n", "\n", "First thing is creating a trial, which is the construct that allows accessing to tensors for a single training run." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip install smdebug" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from smdebug.trials import create_trial\n", "\n", "s3_output_path = description[\"DebugHookConfig\"][\"S3OutputPath\"] + '/' + estimator.latest_training_job.name + '/debug-output/'\n", "print(s3_output_path)\n", "trial = create_trial(s3_output_path)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we can print the list of all the tensors that were saved." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "trial.tensor_names()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Given a specific tensor, we can ask at which steps we have data for the tensor. In this case, we have data for all steps since the frequency was set to 1." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "trial.tensor(\"validation-merror\").steps()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also get the value of a specific tensor for a specific step as numpy.array" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "trial.tensor(\"train-merror\").value(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Performance metrics" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also create a simple function that visualizes the training and validation errors as the training progresses. We expect each training errors to get smaller over time, as the system converges to a good solution. Now, remember that this is an interactive analysis - we are showing these tensors to give an idea of the data." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "\n", "# Define a function that, for the given tensor name, walks through all \n", "# the iterations for which we have data and fetches the value.\n", "# Returns the set of steps and the values\n", "def get_data(trial, tname):\n", " tensor = trial.tensor(tname)\n", " steps = tensor.steps()\n", " vals = [tensor.value(s) for s in steps]\n", " return steps, vals" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "metrics_to_plot = [\"train-merror\", \"validation-merror\"]\n", "for metric in metrics_to_plot:\n", " steps, data = get_data(trial, metric)\n", " plt.plot(steps, data, label=metric)\n", "plt.xlabel('Iteration')\n", "plt.ylabel('Classification error')\n", "plt.legend()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Feature importance" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also visualize the feature importances as determined by xgboost.get_fscore(). Note that feature importances with zero values are not included here (which means that those features were not used in any split conditions).\n", "\n", "For more information on the metrics related to feature importance in XGBoost, please visit: https://towardsdatascience.com/be-careful-when-interpreting-your-features-importance-in-xgboost-6e16132588e7\n", "\n", "`weight` is the number of times a feature is used to split the data across all trees
\n", "`gain` represents fractional contribution of each feature to the model based on the total gain of this feature's splits. Higher percentage means a more important predictive feature
\n", "`cover` is a metric of the number of observation related to this feature
\n", "`total_gain` is the total gain across all splits the feature is used in
\n", "`total_cover` is the total coverage across all splits the feature is used in
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", " \n", "def plot_feature_importance(trial, collection_name, step, metric):\n", " feature_importance_tensors = trial.collection(collection_name).tensor_names\n", "\n", " feature_names = []\n", " feature_values = []\n", " \n", " plt.subplots(figsize=(18,7))\n", " \n", " for tensor_name in feature_importance_tensors:\n", " if tensor_name.find('/' + metric) >= 0:\n", " index = tensor_name.rfind('/')\n", " feature_name = tensor_name[index+1:]\n", " feature_names.append(feature_name)\n", " tensor = trial.tensor(tensor_name)\n", " value_at_step = tensor.value(step)[0]\n", " feature_values.append(value_at_step)\n", "\n", " pos = range(len(feature_values))\n", " plt.bar(pos, feature_values, color='g')\n", " plt.xlabel('Features', fontsize=16)\n", " plt.ylabel('Feature Importance ({0})'.format(metric), fontsize=16)\n", " plt.xticks(pos, feature_names)\n", " plt.show()\n", " " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot_feature_importance(trial, \"feature_importance\", 19, \"gain\")\n", "plot_feature_importance(trial, \"feature_importance\", 19, \"cover\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Confusion Matrix\n", "\n", "Finally, since we were logging labels and predictions, we can visualize the confusion matrix of the last step." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "import numpy as np\n", "from sklearn.metrics import confusion_matrix\n", "from IPython.display import display, clear_output\n", "\n", "fig, ax = plt.subplots(figsize=(15,10))\n", "step = 19\n", "\n", "cm = confusion_matrix(\n", " trial.tensor('labels').value(step),\n", " trial.tensor('predictions').value(step)\n", ")\n", "normalized_cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n", "sns.heatmap(normalized_cm, ax=ax, annot=cm, fmt='')\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can now move to Deploy and Monitor to see how to deploy this model and monitor its inference performance over time using Amazon SageMaker Model Monitor." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## References\n", "\n", "A Realistic Cyber Defense Dataset (CSE-CIC-IDS2018) https://registry.opendata.aws/cse-cic-ids2018/" ] } ], "metadata": { "kernelspec": { "display_name": "conda_python3", "language": "python", "name": "conda_python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.5" } }, "nbformat": 4, "nbformat_minor": 4 }