{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![MLU Logo](../../data/MLU_Logo.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a name=\"0\">Machine Learning Accelerator - Natural Language Processing - Lecture 3</a>\n",
    "\n",
    "## Neural Networks with PyTorch\n",
    "\n",
    "In this notebook, we will build, train and validate a Neural Network using PyTorch.\n",
    "1. <a href=\"#1\">Implementing a neural network with PyTorch</a>\n",
    "2. <a href=\"#2\">Loss Functions</a>\n",
    "3. <a href=\"#3\">Training</a>\n",
    "4. <a href=\"#4\">Example - Binary Classification</a>\n",
    "5. <a href=\"#5\">Natural Language Processing Context</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -q -r ../../requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. <a name=\"1\">Implementing a neural network with PyTorch</a>\n",
    "(<a href=\"#0\">Go to top</a>)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's implement a simple neural network with two hidden layers of size 64 and 128 using the sequential mode (Adding things in sequence). We will have 3 inputs, 2 hidden layers and 1 output layer. Some drop-outs attached to the hidden layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-09T14:42:26.315306Z",
     "start_time": "2021-01-09T14:42:25.876374Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sequential(\n",
      "  (0): Linear(in_features=3, out_features=64, bias=True)\n",
      "  (1): Tanh()\n",
      "  (2): Dropout(p=0.4, inplace=False)\n",
      "  (3): Linear(in_features=64, out_features=64, bias=True)\n",
      "  (4): Tanh()\n",
      "  (5): Dropout(p=0.3, inplace=False)\n",
      "  (6): Linear(in_features=64, out_features=1, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "\n",
    "net = nn.Sequential(\n",
    "        nn.Linear(in_features=3,       # Input size of 3 is expected\n",
    "                  out_features=64),    # Linear layer-1 with 64 units\n",
    "        nn.Tanh(),                     # Tanh activation is applied\n",
    "        nn.Dropout(p=.4),              # Apply random 40% drop-out to layer_1\n",
    "        nn.Linear(64, 64),             # Linear layer-2 with 64 units  \n",
    "        nn.Tanh(),                     # Tanh activation is applied\n",
    "        nn.Dropout(p=.3),              # Apply random 30% drop-out to layer_2\n",
    "        nn.Linear(64,1))               # Output layer with single unit\n",
    "\n",
    "print(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can initialize the weights of the network with 'initialize()' function. We prefer to use the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-09T14:42:26.323790Z",
     "start_time": "2021-01-09T14:42:26.316902Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Sequential(\n",
       "  (0): Linear(in_features=3, out_features=64, bias=True)\n",
       "  (1): Tanh()\n",
       "  (2): Dropout(p=0.4, inplace=False)\n",
       "  (3): Linear(in_features=64, out_features=64, bias=True)\n",
       "  (4): Tanh()\n",
       "  (5): Dropout(p=0.3, inplace=False)\n",
       "  (6): Linear(in_features=64, out_features=1, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def xavier_init_weights(m):\n",
    "    if type(m) == nn.Linear:\n",
    "        torch.nn.init.xavier_uniform_(m.weight)\n",
    "\n",
    "net.apply(xavier_init_weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at our layers and dropouts on them. We can easily access them wth net[layer_index]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-09T14:42:26.329940Z",
     "start_time": "2021-01-09T14:42:26.326430Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Linear(in_features=3, out_features=64, bias=True)\n",
      "Tanh()\n",
      "Dropout(p=0.4, inplace=False)\n",
      "Linear(in_features=64, out_features=64, bias=True)\n",
      "Tanh()\n"
     ]
    }
   ],
   "source": [
    "print(net[0])\n",
    "print(net[1])\n",
    "print(net[2])\n",
    "print(net[3])\n",
    "print(net[4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. <a name=\"2\">Loss Functions</a>\n",
    "(<a href=\"#0\">Go to top</a>)\n",
    "\n",
    "We can select [loss functions](https://d2l.ai/chapter_linear-networks/linear-regression.html#loss-function) according to our problem. A full list of supported `Loss` functions in PyTorch are available [here](https://pytorch.org/docs/stable/nn.html#loss-functions). \n",
    "\n",
    "Let's go over some popular loss functions and see how to call a built-in loss function:\n",
    "\n",
    "\n",
    "__Binary Cross-entropy Loss:__ A common used loss function for binary classification. \n",
    "\n",
    "```python\n",
    "loss = nn.BCELoss()\n",
    "```\n",
    "\n",
    "__Categorical Cross-entropy Loss:__ A common used loss function for multi-class classification. \n",
    "\n",
    "```python\n",
    "loss = nn.CrossEntropyLoss()\n",
    "```\n",
    "\n",
    "__MSE Loss:__ One of the most common loss functions for regression problems. \n",
    "\n",
    "```python\n",
    "loss = nn.MSELoss()\n",
    "```\n",
    "\n",
    "__L1 Loss:__ This is similar to L2 loss. It measures the abolsute difference between target values (y) and predictions (p).\n",
    "$$\n",
    "\\mathrm{L1 loss} = \\frac{1}{2} \\sum_{examples}|y - p|\n",
    "$$\n",
    "In pytorch, we can use it with `L1Loss`:\n",
    "```python\n",
    "loss = nn.L1Loss()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. <a name=\"3\">Training</a>\n",
    "(<a href=\"#0\">Go to top</a>)\n",
    "\n",
    "`torch.optim` module provides necessary optimization algorithms for neural networks. We can use the following `Optimizers` to train a network using [Stochastic Gradient Descent (SGD)](https://d2l.ai/chapter_optimization/sgd.html) method and learning rate of 0.001.\n",
    "\n",
    "```python\n",
    "from torch import optim\n",
    "optimizer = optim.SGD(net.parameters(), lr=0.001)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. <a name=\"4\">Example - Binary Classification</a>\n",
    "(<a href=\"#0\">Go to top</a>)\n",
    "\n",
    "Let's train a neural network on a random dataset. We have two classes and will learn to classify them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-09T14:42:26.750569Z",
     "start_time": "2021-01-09T14:42:26.332404Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_circles\n",
    "\n",
    "X, y = make_circles(n_samples=750, shuffle=True, random_state=42, noise=0.05, factor=0.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-09T14:42:27.482480Z",
     "start_time": "2021-01-09T14:42:26.752160Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZsAAAEcCAYAAAARLRmYAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABCUElEQVR4nO3de3wTdbo/8M9MeiG0qTRQamkKeNYjICrCKiiuqNBS3AUKW1uqKMcqVQEFRdciRdkKXS3uglgUFlHZFV5IeyoiyHLb4+KKgnrw/EQugpfSK600rZRQekm+vz/ShKSZSSbJTDJJn/fr5UvaTCZPp2me+d6eL8cYYyCEEEIUxAc7AEIIIeGPkg0hhBDFUbIhhBCiOEo2hBBCFEfJhhBCiOIo2RBCCFEcJRtCCCGKo2RDiJfGjx+PG264ASNHjsRNN92EnJwcbNmyBRaLxeNzq6urMWTIEHR2dioaY6BehxCpIoIdACGhaN26dRg7dixaWlrwxRdfoKioCN988w1eeumlYIdGiCpRy4YQP+h0OkyYMAGvvvoqtm3bhlOnTuFf//oXpk2bhlGjRuGOO+5ASUmJ/fj7778fAHDzzTdj5MiR+Prrr1FZWYlZs2ZhzJgxGDNmDJ5++mmcP3/e/pz169fj9ttvx8iRI5Geno7PP/8cAGCxWLB+/XqkpqZizJgxWLBgAZqbm0Vfh5CgYoQQr9x1113s4MGDLt+/44472ObNm9mhQ4fYyZMnmdlsZidOnGC33nor27dvH2OMsaqqKnbNNdewjo4O+/MqKirYp59+ytra2lhjYyO777772PLlyxljjP3www9s3Lhx7OzZs/bnnzlzhjHG2DvvvMOysrJYXV0da2trY88//zx76qmnRF+HkGCilg0hMunfvz9++eUXjBkzBkOGDAHP8xg6dCh+97vf4YsvvhB93qBBg3DbbbchKioKer0eubm5+PLLLwEAGo0G7e3t+OGHH9DR0QGDwYCBAwcCALZu3YqnnnoKV155JaKiovD4449jz549NE5DVInGbAiRSX19Pa644gr8v//3//DnP/8Zp0+fRkdHB9rb2zFp0iTR5zU2NmL58uX46quvYDKZwBhDXFwcAGsiWrx4MUpKSvD999/jN7/5DRYtWoTExETU1tZi3rx54PnL94w8z6OxsVHxn5UQb1HLhhAZfPPNN6ivr8evf/1rPP3005gwYQIOHDiA//3f/0VOTg5YV3F1juNcnvuXv/wFHMfhww8/xJEjR/DKK6/YjweAKVOmYMuWLfj444/BcRz+/Oc/AwCuvPJKvPnmm/jqq6/s/x09ehSJiYmCr0NIMFGyIcQPFy5cwMcff4yFCxdi6tSpGDJkCEwmE6644gpER0fjm2++wc6dO+3H6/V68DyPqqoq+/dMJhN69+6NuLg41NfXY8OGDfbHfvzxR3z++edob29HVFQUoqOjodFoAAD33nsvXn31VdTU1AAAjEYj9u/fL/o6hAQTxxjtZ0OIN8aPH49z585Bo9GA53lcffXVmDp1KnJycqDRaLB7924UFxejubkZo0ePRnJyMs6fP29vkaxevRpbtmxBZ2cnNmzYgJiYGOTn5+Onn37CwIEDkZGRgY0bN+KTTz7ByZMnsWTJEvzwww+IjIzEyJEj8eKLLyIxMREWiwV/+9vf8N5776GhoQF9+/bFb3/7WyxcuFDwdW688cYgXjXS01GyIYQQojjqRiOEEKI41cxGa2pqwrPPPovKykpERUVh0KBBePHFF6HX652OM5vNWL58Of7973+D4zg88sgjyMrKClLUhBBCpFBNy4bjOMyePRt79uzBjh07kJKSYu/jdrRjxw5UVlZi79692Lp1K0pKSlBdXR2EiAkhhEilmmTTp08fjBkzxv71jTfeiNraWpfjdu3ahaysLPA8D71ej9TUVOzevTuQoRJCCPGSapKNI4vFgi1btmD8+PEuj9XV1WHAgAH2r5OSknD27NlAhkcIIcRLqkw2y5YtQ+/eve3FBAkhhIQ21UwQsCkuLsaZM2ewbt06pzIcNklJSaitrcUNN9wAwLWlI0VTkwkWi7pnfPftG4vGxgvBDsMjilNeFKd8QiFGIDTi5HkO8fExfp1DVclm1apV+Pbbb7F+/XpERUUJHjNp0iSUlZVh4sSJaG5uxv79+7F582avXsdiYapPNgBCIkaA4pQbxSmfUIgRCJ04/aGabrTTp09j3bp1aGhoQE5ODjIyMjBv3jwAQF5eHo4ePQoAyMjIgMFgwMSJE5GdnY158+YhJSUlmKETQgjxoEdWEGhsvKD6O4mEBB1+/rkl2GF4RHHKi+KUTyjECIRGnDzPoW/fWL/OoapuNEIIkRNjDE1NP6O9/RIAdd5gNjTwsFgswQ4DAKDRRCA2tg+0Wv/GZ4RQsiGEhK0LF34Bx3FITDSA41QzauAkIoJHZ2fwkw1jDB0d7Whu/hkAZE846rz6hBAig9bWC9Dp+qg20agJx3GIiopGnz4JuHChWfbz02+AEBK2LBYzNBrqwPFGZGQUzGb5txanZEMICWu0a6l3lLpelGwIISTI7rlnCn788XtFX2PNmleRlTUVv/nNTYq/lhBKNoQQ0gPcfvudWLNmPa68Mikor0+dmYQQ4qC8PAJFRdGoqeGQnMxQUNCGzEx5xjC+/fYbvP76aly8eBEAMG/eAowdO9bpmC1bNuGf/9wLs7kTUVHReOaZRfjP/xyCS5cuYfnypaio+BEaTQQGDhyEZcteRmVlBYqKCnHp0iVYLGbcffcU3HffAy6vPWLEjbL8DL6iZEMIIV3KyyOwcGEvtLZaxy2qqzksXNgLwCW/E875879g8eI/oKhoBa6/fgTMZjNMJpPLcZMm/Q733mstQvzll4fxyisvYf36jTh8+HO0tLRg06ayrvOdBwC8//5/49Zbb8ODD852+r7aULIhhJAuRUXR9kRj09rKoago2u9k8+23RzF48FW4/voRAACNRoO4uDiX47777gTeffcdnD//C3ieR1VVJQDg6qv/E5WVFfjLX4oxcuSvMXbsbwAAN944Eq+/vhodHR0YNeomjBp1k19xKoXGbAghpEtNjfBMLLHve0NKZbCOjg48/3w+5s9/Gu++W4q//KUE7e3tAIDkZAM2by7DzTePwVdfHcaDD96LtrY23HnnBKxd+xaSkw3YtGkjli17we9YlUDJhhBCuiQnCycEse974/rrb0BFxU/49ttvAABms9mly6u9vQ1msxn9+ycCAN5/v8z+WENDPXheg3Hj7sT8+U+jubkJLS3nUV1dBb2+L3772ynIzc3D8ePH/I5VCdSNRgghXQoK2pzGbABAq7VOEvBXXNwVKCpagZKSVbh0qRUcx2PevAW49dZb7cfExMTi4YcfRV7eLCQmXolbbrk8eeCHH77HunVrAFgXq95//4Po1y8Bf//729i7dzciIyPAcRwWLHha8PVfffUVHDjwMYzGRjz55DzExV2BTZtK/f65pKKqzyoVCpVgAYpTbhSnfBISdDh69FtceeUgr56n5Gw0IWqpjebo7NkzTteNqj4TQojMMjM7FU0uPRWN2RBCCFEcJRtCCCGKo2RDCCFEcapKNsXFxRg/fjyGDBmCU6dOCR5TUlKCW2+9FRkZGcjIyEBhYWGAoySEEOItVU0QmDBhAmbNmoWZM2e6PW7atGnIz88PUFSEEEL8papkc9NN6iyzQAghxD+q6kaT6qOPPsKUKVPw0EMP4euvvw52OIQQ4hel97P55ZdmPPPMfNx77+/xX/+Vg8WL/4CmpibFXk+Iqlo2UuTk5OCxxx5DZGQkDh48iLlz52LXrl2Ij4+XfA5/FycFSkKCLtghSEJxyovilA/P84iI8O6eOrJsK7TL/gi+phqWZANan/8jOrJmKBThZRqN97FKFRGhwQMPPIhf/9rae1RSsgrr169BQcFSweN5npf99xtyySYhIcH+79tuuw1JSUk4ffo0Ro8eLfkcVEFAPhSnvChO+SQk6GCxWLxanR9dXoqYhU+Aa20FAGiqqxDz5ONosTC0ZWb7HZO7/WzMZmusSuxnExOjw4gRo+zXYtiw67BtW7notbFYLE6/3x5ZQaC+vh6JidYidSdOnEBNTQ2uuuqqIEdFCAkHMUWF9kRjw7W2Iqao0O9ko5b9bCwWC7ZtK8dvfjPOr5/HW6pKNsuXL8fevXtx7tw55Obmok+fPvjoo4+Ql5eH+fPn4/rrr8fKlStx7Ngx8DyPyMhIrFixwqm1QwghvuJrqr36vjfUsp/NqlWvoHdvLTJlaKl5Q1XJZsmSJViyZInL99988037v4uLiwMZEiGkB7EkG6CprhL8vr+82c9mzZo3MWTIUJw79zOmTbsbwOX9bL766kscOnQQ69e/jr/97T3ceecEXHfdDfjii0PYtGkjPvroQ7zwwjLB869Z8yqqqytRXLwKPB/Y+WEhORuNEEKUYCpYCqbVOn2PabUwiQykeyPY+9n89a+v47vvTuCll/6CqKgov38eb6mqZUMIIcFkG5eJKSq0z0YzFSyVZXJAMPez+fHHH/Duu+8gJWUgHnvsIQBAUtIAvPTSn/3+uaSi/WxUKhRm+wAUp9woTvn4up9NoPWU/WyoG40QQojiKNkQQghRHCUbQkhY64EjBX5R6npRsiGEhC2e18Bspi2evdHR0Q6NRv65Y5RsCCFhS6uNRUtLMxhT1wC8GjHG0N7ehubmnxEb20f289PUZ0JI2IqNvQJNTT+jvr4agDq703ieh8WijmSo0URAp4uHVhsj+7kp2RBCwhbHcdDr+wc7DLdCYRq5HKgbjRBCiOIo2RBCCFEcJRtCCCGKo2RDCCFEcZRsCCGEKI6SDSEBFl1eCv2o4eiXeAX0o4Yjurw02CERojhKNoT4wdvEEV1eCt3CJ6CprgLHGDTVVdAtfOLy8zZvdjpfTP5CSkwkLNA6G0J8ZEsctj3rbYkDgOj+J7EFz4rucQ8AeHo+NBcv2s+nfWcDuK7jpJyfELWilg0hPoopKhRMHLEFzwoeH11eCs5oFHyMr6m2JpyuRGM/X7fjuNZW6B5/1OcWDnXhkWBRVbIpLi7G+PHjMWTIEJw6dUrwGLPZjMLCQqSmpiItLQ1lZWWCxxGiNL6mWvD7nNEo2P0VW/CsS/KwYfHxoudzOb/Z7Nz1JkF0eSn6DhkM3ZzZzl14c/MQk79Q8nnsunX3UdIinqgq2UyYMAGbN29GcnKy6DE7duxAZWUl9u7di61bt6KkpATV1dL+SAmxsd/h949Dv6R49OsfJ/qhKdQaiMlfCIiUYucAaDe+5fyhvmCuaKsGAMAAS7JBcvxOXW9uRJeXou9Qa5Lhm4yuLSXGoN34lteJC488Ij7uRIgAVSWbm266CUlJSW6P2bVrF7KyssDzPPR6PVJTU7F79+4ARUjCQXR5KXTz51g/LGFtKXCA4Iem4ID+nNlOYylCuG6JiGtvd398cxNMBUu9+jk8tYRssfNG1yTj9NqMSUpcNoLdfRKTH+m5Qm6CQF1dHQYMGGD/OikpCWfPnvXqHP7upR0oCQm6YIcgiarj3LwZKCgAKiuRMHAgUFQELMkHOjoED+daWxG3JN96TGOj8DEKhMkNHIi4xx4G5syW/hzGkHDz9dafaeZM1wNeXgZ0G1MSo6mplv57FElyXp0jQNQWj5hQidMfIZds5NDYeAEWizrLjduESiVYNcfZfbYYzpwBy8sDWlvdJgzW2KhIQhF9Pa0WLYueR9vPLUjgecCbcvNdP1NLyyWXGWr9Kisl/xzmZANM695CTFEh+JpqWJIN9pZW9+/FJBugqa4SPIdRRe8FNb83HYVCnDzP+X2TrqpuNCmSkpJQW1tr/7qurg5XXnllECMiaiU2W8yTQCUaBsASr0fLypLLicKHfU26d2HZxpjExpRc4tBq0ZaW7tpdOH8OdAvmuozNtKWlA717u5zD225A0rOEXLKZNGkSysrKYLFYYDQasX//fqSnpwc7LKJCorPFAhyHGBavR+N3Fc4tkkGDfDqX7Wd1GmNy99pd/5kNKWhZWYLofXtcE3NHB7j2dufvtbYiet8eYP16mA0pYBxnPwet/SHuqCrZLF++HOPGjcPZs2eRm5uL3/3udwCAvLw8HD16FACQkZEBg8GAiRMnIjs7G/PmzUNKSkowwyYq5c3sLink7njlmptcv1lUBKbVen0u288q1JoTfG0A0GhgKliKtsxsydOuAYCvrgIKCmAqWIpz9b/AeOQYJRriEceYxLZ2GKExG/moMc7o8lLrOEN1FcBxTjPDWLevpWJRUWidOQvR+/ZYP5h5HpzZ7HocpLeczIYUGI8cc/peQoIO521jJx5aJ46v2bJ2A9oys9Gvf5xXLTem1aJlZQliigoFx2HcPjcyEi2vrRVMNPbfgcNYT6ATkhrfm0JCIc4eOWZDiDvdu5E4xsA468ev2ZCC1gcf9qqFwgBY9Hq0rH4DpuKVMB45hnP1v6BlzV9dWiBMq0Vr7mxr9xIAptHYx2VYVJTLsWJjHG2Z2TAeOYaWtRsktXJYvB5tmdnWKducd52EtvEeU8FSr1tUXEcHYhdbqyU4rkXqO3Tw5anltA6HdOmRs9FI+HFszQgtXASs3T+9tr/v9bkbT1a4fM92ly50924qdhOfF3f63V+DxceDa2kB5zBtm2m1uPCnFfbjfGm18TXV9tfSPf6oYItNDNdkdJn1J7R41ZbUqLut56JuNJUKhaY1ENw43XWXueNNV5elaxA/UDxdT3dJq1/iFT4lG8fuPG/PwQBYDCmSuuAYx+Fc/S9ex+cr+huSD3WjkR6je8mYmPyFLt1lUonWJxP4+tK03/sYsTJsXWxCA/NSJkSwbt1s3bvzfJlUwUsc6/Hm3FQwNPxQsiGqF5O/ELq5eU5jANqNb0madSVGKLG4dL8B1mm+IcLTuAvTatH64MNupyx7O3Zjm9XmCYuMdBmjEksoHvf8ISGJutFUKhSa1oDycUaXl0I3N8+n7iF3LPF6sJgYe3eU2MyvUOv6cexmY/HxALNOsfZmRlh0eSl0c2ZL7mpkgMduTEtMDFi83toKcqiS4PganmbGCc3ec4f+huQjRzcaJRuVCoU3IKB8nH2HDAbf5KZasg9YVBRaVr/h9MGrHzVclg84f6nl9y52PYRImU4udZzMbEgBX1MteD5vE79arqUnoRAnjdmQsBZdXgrOTaLx5XaB8bxLogGEu496cgkWwesRGek6hRvSxsuktpJsLU0hci/SJYFFyYYEjadB4JiiQsmD+ZIxJtiV1JaZjZaVJVSCpYvg9XhtLVpnznKaZCB36R9LsgFtaemCY2ptaVSWKpRRN5pKhULTGvA9TpeKzLjcZw/A4wp6sW4ZT901ge4W85baf+/edK95iwFgej04kf13aMwmeOToRqNFnSQoxCoy6+Y9Algsft0xmw0pwqVqenC3mFyk1FDzZh2Ty/nd7GYqdYo1USfqRiNBIVqRWUKiYZGR1hlNQjQa6zqUhvNoeeNNezcQBg3q0d1ichEbN2Eajb27zV6yh+O86u70mKAkTLEm6kXJhgQFi4/37viu/yy9Y6xjB//1kGC/fuusXPvXjgsgUVFBiUYGYhMpWtb81b7Q1LGGHNPrJZ1XUlLyoowOUR9KNiTgostLwbV410fN2f5j1vUZpuKVaM2dbS92yTQatObOhql4pezxkstsEwcwaJCkiRQXilaAeWiRMAAsJsbja7N4aYmLqBNNEFCpUBg0BHyL099BZl8G+cP5egaDN3FGl5cidvGzl6exc5x9F1EWr7cXEu0+YaQ7ofVRcsUYTKEQJ00QICHJm426lHg+Cay2zGyPCSK6vBSMt7aAxMZuuPZ2qhwdwqgbjQScpIKREO/Hp8V94SW6vBS6BXPBmy54nCTguP01FeoMLZRsiOw8fRBIKfbIAUBUlHXmmQOavhx+YooKwbW3SzrWkmygQp0hSlXJ5qeffsKMGTOQnp6OGTNmoKKiwuWYkpIS3HrrrcjIyEBGRgYKCwsDHygRJfhB8Pij6HdlH/TrH4d+SfGI3rIJTKv1OAOJa28H0+loVX+Yk9otyqKiYCpYah3/EVijFVNEnwVqpqoxm6VLl+K+++5DRkYGtm/fjhdeeAF///vfXY6bNm0a8vPzgxAh8URwsabjlFWzGVGf/Evyoj+uqUlwp0wSPizJBmkTRhhDxBeHROvl8TXVTlWvMXAgohc9TzcnKqGalk1jYyOOHz+OyZMnAwAmT56M48ePw+hmRTEJHltXGXjeqatMyl2qN6vLvV2PQ0KDY1crd9HkcXo0AHAdHdC+s0G8hFGfeKdWNc6coe41FVFNy6aurg6JiYnQdL3pNBoN+vfvj7q6Oui7LQz76KOP8OmnnyIhIQFPPPEERo4c6dVr+TuFL1ASEnTBDkHY5s3A0/OBixcBAJrqKsQ9PR/Q9QIGDgTOnJHtpXiOk+06qPZ6dhP2cXZ7/3BGIxAVBfTqBZhMbp/q7kaF5zlAoHst7uVlwGMP+xZrgITK79wfqkk2UuXk5OCxxx5DZGQkDh48iLlz52LXrl2I9+IOmNbZ+Ee/6Dlouj4o7C5ehHnRczAVLIVuwVzJA742ooU1jUack+E6qPl6OhKLs7w8AkVF0aip4ZCczFBQ0IbMzM4gRGjlz/UUfP+0t8PcPxHGn+qsx3i5FosBQGOj8HuoslKW95BSQuG9GVb72SQlJaG+vh7mrv59s9mMhoYGJCUlOR2XkJCAyK4ZSrfddhuSkpJw+vTpgMfbk4l1ldm/3ynfh2BPmOZcXh6BUaNikJgYi8GDrV93f3zhwl6orubBGIfqah4LF/ZyOS5UeHz/wMftqTnhdg/rQ12xaqCaZNO3b18MGzYMO3fuBADs3LkTw4YNc+lCq6+vt//7xIkTqKmpwVVXXRXQWHs6d5tbxS5+FlzXlr/e4ACnfVKAnjHNuXsiOXMGLomkqCgara3O16a1lUNRUXSgw5WFlM3RnPbTkXpisWIocm+6Q3yimmQDAH/84x+xadMmpKenY9OmTfZpzXl5eTh69CgAYOXKlZg8eTKmTp2KJUuWYMWKFUhISAhm2D2O2C6O3EWT2501PWIsrKc5O7ZgRo2KsXeNCSWSxYuj7cdWVwt/WtbUhOanqNRdUW2FVFvWbnC5EfEG19Tk83OJfKg2mkqpvR/XNsVUU1MNS594cKYLXo/TdKfkxmZKXk8p4ynl5RGYP78XOjouf2hGRjJ0dAC+bQMHGAwWHDnifkBdKf5eT8cpypZkA0wFS+03FkKPRXxxCNqNb0nagro72jDPf3KM2VCyUalQeAMC1jjNKQP93r3RtkunUi0Zpa6nrRvMsXWi1TKsXHkJmZmd9kRkbZ34llSEOL5GMCh1Pf3dwbU7pd9XcgiFv3VKNj6iZCOfhAQdGM97fcfJoqLAYmLBNTe53NkqQanrOWpUDKqrXXuj4+Ot41ZNTWJJxh8MvXsz9OplPX8wZqcpdT3FZqExdG0x0N4GzmRye0Vt70ROo8HFWbmq33YiFP7Ww2o2GgkN3eueYfNmr2eMMY0GLavfQON3FfYNt9R852kjNOYiNm7S1MShqYmHMqPTHC5e5GE0hsfsNEeiO7gC4JuM4D0kGtuxHACYzdC+t5kWdaoEtWxUSo13O0JdHIB190yus8NpzEZ03QzHoeWNNwOeXPy5nuXlEVi8ONqllaLVMvA8g8mklns2BoMhMK2cQLds/ME0Gusun13/txhSFG9Je0ONf+vdUcuGBJRQ3TMA4C+aAMZg0eut3R0cJ373yZhq/siFdG+95OdHY+HCXoKtlNZWDiaT2DhMMFhbOQsWhG4rx9v1NVJwZrO1tdP1f6oSHRyUbIhkvJs7Tq6jw1p2RKNxO35jMaQoEZoshBZPbtwY6TI12ZnQY8GdktzezmHOnF5ISorFPfdoXbr+hLoD1cKX9TW+pHaqEh141I2mUmpsWvdLineu4OwlBqA1d3ZQBmylXE+xwf7Q5tyhGRXFwBicpmD7MrMtEO9PsW5bGwaA6fXgjEaf0jsDcK7hvD8hykKNf+vdUTcaCSw/Eg1g/ciL3rdHnlgUEKqLJN1z/pna2zmnRAOotxqBUyuH42CJ11u7am2LftduQOPJCt9byxxHXWkBRMmGeGTfTkAGUjfKCobkZOHWLsepuxUsB7UmWlsVgXP1v6Dxuwo0nnSdwWgqWOpbVxpj1JUWQJRsiFtOO2/KcD41F9YsKGiDVtv9Y8va7RQTwxC8gX/liSXaUODPhBM13/yEG0o2xC2xGWiOJA/kqrywZmZmJ1auvITevS1wWBoIgPO0zUoIcU2aWq11unSo8qcrjCpCBw4lG+KWXDtvMo1G9WVDAGvCuXRJaNW/OruZvGf92axdgwwGgyWoZW/kEFNU6PNvhzNdoHGbAKFkQ9ySo9uLabVoWfNX1ScaAMjPj4YPOySEHMY4aDTWsZqiomhVTX/2lj9dYVx7O43bBAglG+LCl/3hu2McBwaE1FYB5eUR2LgxEuKtmEC1bgIzfmI2c2FR7sbfGyIatwkMSjbEidOEAMbAG41eT3m2Fk2MR8vaDaqteya2twxjSiUUaRMMoqKA3NwOGAwWScfLRa3Tn6VwNxtNyhVU86SVcELJhjgRmhDg7ccvB4A3GlVbEkSoUsCCBb1ENynzD4NGw5Cb24G1ay+5zHaLjGTQ6y3gOOv4ydtvA8XFbThyxCR4vJIJqLqaC8nWjT83MwwAZzKp8n0abijZECfedCl4+tjjWluhe/xR1f0hC+2O2d6uxFYAAMDBbObw3nuRAICVKy/BYLicXF577RJOnjShvv4CjhwxYebMy8+0zY5zPH7cOLOHdT/+JCMOc+f2Qn5+6LVwRBd28rzLFXFsY9qqSav1xiicSEo2//d//4eNGzfi008/dXls/fr1sgdFgkesS6H7trxMq0Vr7mygd2+35+PMZtX9IQdjAaOtmyozsxNHjlxOLp5mgXU//r//uxVvvHEJGo0yLRzGOGzcGBlyLRyxAp6cxWK/hWAALPF6sHi961xDqpWmOI/J5oMPPsAjjzyCw4cPY9GiRXjkkUdgclh0sG7dOtmC+emnnzBjxgykp6djxowZqKiocDnGbDajsLAQqampSEtLQ1lZmWyvT8T3h2998GF72RDboH/n6FsArdbjaITa/pCDtYBRriSXmdmJNWtcu9isLR7/X4Mx9Y7fdN9PyXYT0720jdCkFg4Ai4kB19wkeG6aKKAsj8lm/fr12LBhA9auXYv9+/cjPj4es2bNwvnz1gJ2ctbxXLp0Ke677z7s2bMH9913H1544QWXY3bs2IHKykrs3bsXW7duRUlJCaqr6U0il+5/tLbEYipeCeORY2h5400AgG7ObOjm5gGNjfbNqty9E9T0hyxcKUB5ciY5WxebXm+bSGCtdCAXNZav6T55pftWAY6lbcTmr/M11eKtd1rgqSiPyaa+vh433HADAKBXr14oLi7G6NGjMXPmTJw7dw4cJ8+bsrGxEcePH8fkyZMBAJMnT8bx48dhNBqdjtu1axeysrLA8zz0ej1SU1Oxe/duWWIgVo5/tI6zyWLyF0I3Z7a9dE33rQTcJRw1zfhxHAtRbsA9MKv0rWNPjv/Jo08f9ZWvEZy80toK3dw89Osfh37949B36GBEl5eKvt9sW5CzyEiXx2iBp7I8Jpt+/fq5dGfl5+cjLS0NM2fORGenPCuP6+rqkJiYCE1X81ej0aB///6oq6tzOW7AgAH2r5OSknD27FlZYiDiostLoX1ng7RqAd274aKiwJlMLl0fwWQbC5HpXkmA8qv0hSY6yEW56+I70S2jGbOnWt5ohG7BXLSlpVvnkTtgUVH2HTqZTud6HlrgqSiPo4Djx4/Hzp078fjjjzt9f/78+YiOjsaqVasUC04p/u7LECgJCa5/EEHz8jJJh3GDBgFFRUBBAVBZCej14M6fB9dkbaFqqqsQ9/R8QNcLTlOvAkDoeg4cCJw5o8zrMcZh0CCgooIDIH33Sam/95oaHwOTwGjkPcYR8PenxF8W196O3h9uQ/d+RY4xxOl6AQk6oEl43EZTUx2UvztV/a0rxGOyyc/PBwB8+eWXuPnmm50ee/TRR5GcnCxLIElJSaivr4fZbIZGo4HZbEZDQwOSkpJcjqutrbV37XVv6UhBm6d5r9+ZMx5bNUyrRcui59E2cSowcSqArj3lGxudD7x4EeZFz8HYdUwgiF3PRYsiMH9+L5c9XuRSWcnw888XJB/vze89OVl4s7f4eAuamzk/F6gyrFsn3hoLxvszetHzbjdTc8S6xhKddHTY33f6ZAM0AjvPmpMNMAb451Lb37qQgG6e9sQTT+CVV15BR0cHAOD8+fN48sknUVJS4lcANn379sWwYcOwc+dOAMDOnTsxbNgw6PV6p+MmTZqEsrIyWCwWGI1G7N+/H+np6bLEQIR56vZyV5ZGrOtDLRMGMjM7odMpd+Oh5Mw3oYkOkZEMv/zib6IBAPXNSOu+ZbQvV9a2tbnYrEs1VyUPdZKTzfbt23Hy5Encc889KCsrw5QpUxAXF4dt27bJFswf//hHbNq0Cenp6di0aRMKC639p3l5eTh69CgAICMjAwaDARMnTkR2djbmzZuHlBT17msfDjxW1XXoC+/O3UCtWhiNyrRqlC7d333Rp15vQWcnYLHI8/OocUaabfKKxZAi+p70NAHcthGg0KxLNZZWChcc82Lu8qVLl5CVlYXvv/8e99xzD5Ytk9aPrzbUjeadfolXuMw8685sSIHxyDGX7wvtI8+02oD/Ybu7nklJsTCb5f5gZRgyxIJ///uiV8+S+nu31XKrqeGQnMyQltaJ996LlHXCgMFgwZEjwhv5BPv9KfaelLrSyPYeBKw3U7Yp0WI3TUoK9rWUIqDdaCdOnEBmZiZSUlLwxhtv4NChQ1i4cKF9vQ0JX1LWH/DVVYIzzcTW7ajpDtLLOqMScfjuOx79+18u9OkLoYKhQrXdNm6UN9FERqp7QzV/W8ZcaytiFz/rdt0OkZfkls2YMWPwzDPPICsrCwBw8eJFLF++HAcPHsSBAwcUDVJu1LKRLrq8FLoFc8G1t0s63nbHGPHFIWj//o71k1yjQefV1yDi+1P2r1tn5cJUvFLh6C9zdz1HjRIeaJeTVsuQk9OBffsiUF1t3UvGbAYMBuuHum0g3jFOW1JxTCJaLUOvXgxNTXLHe/nvIT6e4U9/anM7VTvY70+h9yWLigKLjbVWKpdArBUk1kpXSrCvpRRytGwkJ5uqqirBsZF//vOfmDBhgl9BBBolG+n0o4YLztpxx7HIoeP3un/dmjs7YAnH3fUU+lCX3iEjHccxwYF7rZbZ1+EkJOiwbl0rioqiu6pQC8Ugf2y283ZPfmKC/f6MLi+Fbv4ccF0TlgDrVekcMgwRp0567Pa1HS94dTnOWoUgQIJ9LaUIaLIJJ5RspHPXNw7495HHNBqcqxNe7yA3T9fTcQyE56HAGI4n1isaG8uhrY15mIqtVLKxckx+YoL1/owuL7WOsXRVsejOmytDLRvpAjpmQ3omsb5xDvB/mbkygyU+cayuHJxtoa2tmAsXIHHNj3I3S2rdSM2pNprIMd68I1m8nqY/BxAlG+KWWOl2wLU2mtd82G46EJRZGyPnOZXae+cyNU57FqqN5ium1eLCn1aofvJKOAmtTStIwNn+8HRzZvvXZQaBMZtZuX6cUTkFBW0CYzj+Ut+HtzvB2obBHTkWAjMATK/HhaIV9vc2JZfAoJYN8agtM1t8J0QRzPE/jQadQ4aBde2ayGDdV6Rz9C3yBysD1/L94Uj851J6Maqv5FgIzOL1aDxZQQkmCCjZEEkEy3u4OZ4DAI0GLWs34FxdE1qffBqIjr5cnddkUvWahszMTpw8KbygMfS531ZaiQrVcvD2PShEbOM0ojxKNkQS2+JMS7xe8kw0xy2hxfYiUXtJd4Mh3Fo2DLm5HcjN7YDrR7X1MTUmGkB4gXBr7mzRMUUhaiqT1NNQsiGStWVmW7fV9eI5toSi9oKcYoR39QzNBBQVxbB27SUUF7ehuLgNubkd0GisHZsajTXRFBerr/vMUfeN/TpH3wLW6/LW5JaYWNHfDuM4mmkWRJRsiFd8SQ58dRVYvHDJG7XfaXYvdmkwWDBunBk87zQqBfUmIGtser0Fq1c7d48VF7ehru4CGhouoK7uguoTTXcx+Quhm5sHvsl4ea9Sixnt4+4E6zYtn3EcWh98mMZqgohmoxGvWET2AQHEF8lxANDSYt2x07G8SIisacjM7PTYtVReHoF583rJVnFZDlIWZ4aq6PJSaDe+5bo1eWsrIn78AS1vvIm4l5eBVVYGrcAmcUYtG+IVsf3bWVSU2/5zrqMDLCY2bNc0ZGZ24vXXLyE6Wg2tHeW2olaLmKJC0XVefHWV9X1VUWHvbguX91koo2RDvNKWmY2W19Y6TRSw6PVoWf0GTMUr0bKyRPQjlmsyBrWUu9IyMztRVWXtlmpouIC1a63db8okHfFzGgwMR46YwirRRJeXQj9qOPolXgH9qOH2TdAEqXSxcE9HyYZ4rS0zG43fVeBcw3mAMTSerABgLdqpmzNb9Hkc0KNKudtK4Kxde0lgkoFUQi0k6145QuuA1LpGxh9OZWq63j9uSyWpqAwSuYySDfFZdHkpMHgw+vWPg25unr1mlZRRi1CY9iwXx0kG3nWvWWeP2VpItgkKa9dewr//fREnT5pcHgvHrjPBafOMiV9Fjgv7G5lQRFWfVUrtlWCFduD0FgNgMaQEpEtNTdezvDwCBQXR3bajFq5h3NBwIVBheSWQ19PdTrGi2wRoNOD+9jf8PHGqorHJQU3vTTFhU/W5tbUVTz75JNLS0jBp0iR8/PHHgscdPnwYI0aMQEZGBjIyMuwbuZHAk6MoIgf0mC41R7bqBLaxHbGFo+G3oNSz7mMz0eWlbqfHi1Z/NpuBRx7pUe8rtVNFsnnrrbcQExODffv2Yd26dViyZAlMJuFSIb/61a+wfft2bN++HWVlZQGOlNjIuRiTa22F7vFHnT5gehKhhaPhOPbiidDYjG7ObHD19b5Nsbh4EbEFz8odJvGRKpLNP/7xD+Tk5AAABg8ejOuuuw6ffPJJkKMi7viyGNPdaAVnNjtNHojJX+hyhxuunBeOImzHXjwRHJsBwHe0+1wzmzMaw/q9E0pUkWxqa2uRnJxs/zopKQlnz54VPLaiogLTp09HVlYWtm3bFqgQSTeCRRE5zn1CgfTJA9qNbznf4YZ5V5tt5prFgrCbtiyVEqWLOKDHTERRu4BUEJg+fTpqa2sFH/vss88kn2f48OE4cOAAdDodqqqqkJubi8TERIwdO9arePwd6AqUhARdsEMQ99jDgK4XUFAAVFYCAweCKyqyfn3mjN+nF1oZHvfyMuvr+kjV19NBj41TrwcaG+U9JwBNTbXqr6na45NDQJKNpxbIgAEDUFNTA71eDwCoq6vDmDFjXI6Ljb2cJFJSUpCamoojR454nWxoNppMJk5FwsyZTnH2e+ABxbYJY5WVOOfjNQmJ64meHWdfxiR3tYjNQhNiTjbAqOJrGgq/87CZjTZp0iRs3boVgLWb7OjRo7j99ttdjmtoaIBtpnZzczMOHjyIoUOHBjRW4p63YzksKgqMl/Y2VHvRTuIfrkn6XjNiicZl+WuI1N/rCVSRbB5++GGcP38eaWlpePTRR/Hiiy/aWzGrV6/Gli1bAAB79+7F5MmTkZGRgfvvvx9Tp05FampqMEMn3ZgKlrpU3O2O8fzl+mir35C0Hwl9aKiX0HRlX8iyE6dD/T0MGhRW9fdCHS3qVKlQaFoDwnHG3TMVUZ/8S/zuk+Nwrv4X+9f9+seJLGm0kmPhZyhfTzWyxSm0uJdptV59yNs21+O7ytA4jtdJ3agPsLaSW1a/YX/dULuWahY23WgkfESXlyLq4L/dfjhIvYNVT7F+IsbfHVid1tagqwxN16xGsyEFLWs3oGXtBmmt35hYasWoGCUbIhv7Xa6bQoi27jDHrhe4GbOxVxmYPyespz6HKm92YBXqbhOte9Y1WUg3Nw8xRYVozZkJi17vdnEn1yx9zIcEHm2eRmTjqYQN02jQsrIEAJy7XrqKKrpryXAdHdDNzQMAuntVEbHN9Lq3Xrt3t2mqq6y/T5FefM5oBG802o/VvrcZLStLEPHFIWjf2SD4XqEJJOpGLRsiG3eL8phWi5Y1f0VbZrboSnFPo2gcY6LVBeQapCbeaUtLd92CWWAyh1gLRrS2Wfevu7rmTMUrrZv0SXhNoi6UbIhsxO4sbS0aW4vEn5XigtUFHn8Uujmzfao4QEnKd9HlpdC+t9llQJ/xPHRz85yup9vNzroRu+mwvW9MxSvR8sabYbvra7iiZENkI1jCxqFFY+Nvd4dLdQGzWfRO2B3Bwo9hXhZHTqK1zEwmlzp3bjc7k8jxfdOWmQ3jkWO07XMIoWRDZNOWmY2WlSUe7ziFkhIg/+wzvrrq8iSEwYNdkoi/M6l6OiktVK61Fdq/vS26H43gcwDqJgtDNEGAyKotM9vjXabtcdvaCrE1NoLf77YOwxP74PWZM9AtfMLp9b2ZSUVciU0OcD3Q4v3JGYPZkAK+phqWZENANtgjyqKWDQkKWzeIu+4VsyHFOgag0djXXbQ++LC0NRcQH2S2EevOo1lN0pgKloJFRXk8zpcWq8WQQt1kYYZaNiSoRKfOdn3YCOkcfcvlFedw/TBzN43asdViKlgquPqdumvE2dbGoLoKOo0GcLOmypG730n3x+h3EJ6oZUOCSmxSgbsPG3urSKMR/ADzVL3ANgNNNzcPrJfWuliQZjV5nJkXXV4K3fw59psDoYkZYjwdRzPLwh+1bEhQOY3feNs/L/Gu2oZxHNrS0p1aM1yT0dpVp9f36HEBsUWXEV8cgql4JQAgdvGz4Do6ZH9td61YEj4o2ZCgkzKpAHAo2NiVlMBxoivQBTEmuPqcg3XFevcJBD2BYxFMl+vCGLQb30Ln6FvQlpkNrsko++uzyEjqMushqBuNhAShNTHgeY9VBxx52pZa6WnP0eWl6DtkMPr1j0O//nHoO9R1OrbSYvIXol9SvDWGxCugm/eIvQimEI4xRa4JA2CJ16PltbU9Krn3ZNSyISFBcE2M2QxLTCxguiDbGh2+ugr9Eq+QfbqtbbzDsRuKMxqhWzAXQGBaUy5bPzAmqWVoW6/kjssgP88DERHg2tuFn8BxaPyuQkLUJFxQy4aEBLG1L9xFk+QS9FLYytw7VpqWo6RNTFGh4HgH197uV8tBamwx+Qvd7jHkiVjrhwGw9I4BNBrnByIi0DpzFlj373eh6eU9DyUbEhLcrYmxVS6Qezs8rqMDsU8vkKWkjbuFor4uIpVabie6vBTajW/5nGjEnmfbb4bp9S7bSnDt7Yjetwcta/7q9WxDEp4o2ZCQ4GmKdFtmNiyGFNlfl7toEixpE7v4Wa/GX9zdydseiy4vBQYPltyCklpuJ6ao0KuqC1LZFlu6q8QgtYQRCX+qSDbbt2/HlClTcO2112LTpk1ujy0tLUVaWhpSU1Px4osvwuJLKQwScqR8aAkmJIXi4ZqM4JuM9kkHfNf4i1iCaEtLF4yFwTom0nfoYOjmzwHOnBFspQh1l4lVUraNO/UdMhh9hw6WpeKyC4fuMU+VGKhoJgEAjjEFbnm8dOrUKfA8j/Xr1+OGG27A/fffL3hcVVUVZs6ciQ8++AB9+vRBXl4epkyZgmnTpnn1eo2NF2CxBP3HdisU9iUH1Bdn9+nR3tZe8/U4G7PDmpGY/IXQ/v0d+3ogpwH0rv97Orfju9Tp+VFRQHu7bBMjGKzrXdrS0hG9b4/odXP0c8N5AK5rdICuat9BbsGo7b0pJhTi5HkOffvG+ncOmWLxyzXXXIOrr74avJvtgQFgz549SE1NhV6vB8/zyMrKwq5duwIUJQkF3e+i3XWt2SoHWGJixe/ovSyNb+tSislfaF3T07XKXnBtj4TzcRA+lpMz0URGomXtBhiPHIOpeCWMR455nnQxaJD9n9RVRqRQRbKRqq6uDgMGDLB/PWDAANTV1QUxIqJ2poKlLuXqga4y9r1jcK7+FzT+VIv2cXe6JBwGeLdoFJfL4YhtXRxsQj9j6/3/ZU8MLqV8BBIx02qBoiKn71FXGfEkIOtspk+fjtraWsHHPvvsM2hEpkcqxd/mYKAkJOiCHYIkqo7zsYeBObMFH9LUVF+O/cDHwObNQEEBcOYM4OVWBgCAqChopkxG3NPz/QxaOUItrN7/sw+9E3TWn//p+cDFi9bHmoxA797AnDnArl1AZSUwcCC4oiJg5kwkBDx676n6vekgVOL0R0CSzbZt22Q5T1JSklPSqq2tRVJSktfnoTEb+YRCnHpDimBlaXOyAUbH2CdOBSZOhX7UcEn7tDi9g3gerTNnIXrHTmi6Pqy95e34kFznZ5WVOPdzC/SLnnON/eJFmHfsdKldlgCo/vceCu9NIDTiDJsxG6nS09Oxf/9+GI1GWCwWlJWV4e677w52WETlTAVLrXfoDtyt9fBm3Yt9TMVigfadDV7N/LLHAuv4UWuucAtMyvOlEEtktlljtJkcUZIqks3OnTsxbtw47N69G6tXr8a4cePw/fffAwBWr16NLVu2AABSUlIwd+5cZGdnY+LEiTAYDJg6dWowQychoC0zG1i/XvIAttTV7d4M+guNldg2hGtZuwGNJyus1ZX79pX02k7n0utFV+p7jMMh6dJmckRJqpj6HGjUjSafcIxTcCovfO/iEtvK2ixQWj9h74dgeXkuizXF4rBNMdbNzZM0xmTR68F6xwhu5+DNFOZQ+L2HQoxAaMTZ47rRCAkEoam8rbmz7dtUS8EA+3PFZrQJdk/NnOn02pZ4vdPmbvY4urXQpLQ+mFaLC0UrRGeN0RRmoiRq2ahUKNztAD0vTqmTBxxbLWLPEWzZ+BinYKskMhJMpwPX1CR7FetQ+L2HQoxAaMRJLRtCAkxKSZzukw982fraW4KtktfWovFkBa19IapA+9kQ4gWhbaztJV5EtrX2a+trL2OjhELUipINIV4S+lA3FXv/HEJ6EupGI4QQojhKNoQQQhRHyYYQQojiKNkQQghRHCUbQgghiqNkQwghRHGUbAghhCiOkg0hhBDFUbIhhBCiOEo2hBBCFEfJhhBCiOIo2RBCCFEcJRtCCCGKU0Wy2b59O6ZMmYJrr70WmzZtEj3u8OHDGDFiBDIyMpCRkYGsrKwARkkIIcRXqthiYNiwYVi1ahXWr1/v8dhf/epXeP/99wMQFSGEELmoItlcc801AACeV0VDixBCiMxC7tO9oqIC06dPR1ZWFrZt2xbscAghhEgQkJbN9OnTUVtbK/jYZ599Bo1GI+k8w4cPx4EDB6DT6VBVVYXc3FwkJiZi7NixXsXTt2+sV8cHS0KCLtghSEJxyovilE8oxAiETpz+CEiykasFEht7OUmkpKQgNTUVR44c8TrZNDZegMXCZIlJKQkJOvz8c0uww/CI4pQXxSmfUIgRCI04eZ7z+yY9pLrRGhoawJg1STQ3N+PgwYMYOnRokKMihBDiiSomCOzcuRMrVqzA+fPn8c9//hPr16/H22+/jauvvhqrV69G//79ce+992Lv3r3YsmULIiIiYDabkZGRgdTU1GCHTwghxAOO2ZoKPQh1o8mH4pQXxSmfUIgRCI04e1w3GiGEkNBEyYYQQojiKNkQQghRHCUbQgghiqNkQwghRHGUbAghhCiOkg0hhBDFUbIhhBCiOEo2hBBCFEfJhhBCiOIo2RBCCFEcJRtCCCGKo2RDCCFEcZRsCCGEKI6SDSGEEMVRsiGEEKI4SjaEEEIUR8mGEEKI4ijZEEIIUVxEsAMAgMLCQnz++eeIiopC7969UVBQgOuvv17w2NLSUrz55ptgjGHcuHFYsmQJeJ5yJiGEqJkqPqXHjRuHHTt24MMPP8Sjjz6Kp556SvC4qqoqrFmzBlu3bsXevXtx5swZfPjhhwGOlhBCiLdU0bK566677P++8cYbcfbsWVgsFpcWy549e5Camgq9Xg8AyMrKwvvvv49p06Z59Xo8z/kdcyBQnPKiOOUVCnGGQoyA+uOUIz5VJBtHmzdvxp133inYNVZXV4cBAwbYvx4wYADq6uq8fo34+Bi/YgyUvn1jgx2CJBSnvChO+YRCjEDoxOmPgCSb6dOno7a2VvCxzz77DBqNBgDw0UcfYceOHdi8eXMgwiKEEBIgAUk227Zt83jMvn37sGrVKmzcuBH9+vUTPCYpKckpadXW1iIpKUm2OAkhhChDFRMEPv74Y7z00kt46623YDAYRI9LT0/H/v37YTQaYbFYUFZWhrvvvjuAkRJCCPEFxxhjwQ7illtuQWRkpH3gHwA2btyI+Ph4rF69Gv3798e9994LAHjvvfewYcMGAMBtt92GF154wd4NRwghRJ1UkWwIIYSEN1V0oxFCCAlvlGwIIYQojpINIYQQxVGyIYQQoriwTzaFhYWYNGkSpk6dipycHBw9elT02NLSUqSlpSE1NRUvvvgiLBZLwOLcvn07pkyZgmuvvRabNm0SPe7w4cMYMWIEMjIykJGRgaysrIDFCEiPEwju9WxtbcWTTz6JtLQ0TJo0CR9//LHgccG4nj/99BNmzJiB9PR0zJgxAxUVFS7HmM1mFBYWIjU1FWlpaSgrK1M8Ll/iLCkpwa233mq/foWFhQGNsbi4GOPHj8eQIUNw6tQpwWPUcC2lxBnsa9nU1IS8vDykp6djypQpePzxx2E0Gl2O8/l6sjD3P//zP6y9vd3+7wkTJggeV1lZyW6//XbW2NjIzGYze+ihh9i2bdsCFud3333HTp8+zf7whz+wd999V/S4Q4cOsenTpwcsru6kxhns61lSUsIWL17MGGPsp59+YmPHjmUXLlxwOS4Y1/OBBx5gH3zwAWOMsQ8++IA98MADLsds27aNPfTQQ8xsNrPGxkZ2++23s6qqKtXF+dprr7GXX345oHE5+vLLL1ltbS2766672HfffSd4jBqupZQ4g30tm5qa2KFDh+xfv/zyy+y5555zOc7X6xn2LZu77roLkZGRAJyLfHbnWOST53lkZWVh165dAYvzmmuuwdVXX6367RKkxhns6/mPf/wDOTk5AIDBgwfjuuuuwyeffBKw1xfT2NiI48ePY/LkyQCAyZMn4/jx4y53kLt27UJWVhZ4noder0dqaip2796tujiD7aabbvJYRSTY1xKQFmew9enTB2PGjLF/feONNwqWGfP1eqr7k01mgSjyGQgVFRWYPn06srKyJJUCCoZgX8/a2lokJyfbv05KSsLZs2cFjw3k9ayrq0NiYqJ9IbJGo0H//v1drk336+cu/mDGCVhrGk6ZMgUPPfQQvv7664DFKFWwr6U31HItLRYLtmzZgvHjx7s85uv1VF3VZ2+FSpFPqXF6Mnz4cBw4cAA6nQ5VVVXIzc1FYmIixo4dq6o4leYpTqmUvp7hLicnB4899hgiIyNx8OBBzJ07F7t27UJ8fHywQws5arqWy5YtQ+/evXH//ffLds6QTzahUuRTrjvm2NjLpchTUlKQmpqKI0eOyPbhKFecwb6eAwYMQE1Njb0EUl1dnVMXgY3S17O7pKQk1NfXw2w2Q6PRwGw2o6GhweXa2K7fDTfcYI/f8W5SaVLjTEhIsP/7tttuQ1JSEk6fPo3Ro0cHLFZPgn0tpVLLtSwuLsaZM2ewbt06wV4gX69n2HejhVuRz4aGBrCuCkPNzc04ePAghg4dGuSoXAX7ek6aNAlbt24FYO0mO3r0KG6//XaX4wJ9Pfv27Ythw4Zh586dAICdO3di2LBhTnUBbfGXlZXBYrHAaDRi//79SE9PVywuX+Osr6+3//vEiROoqanBVVddFbA4pQj2tZRKDddy1apV+Pbbb/H6668jKipK8Bhfr2fY10YLlSKfO3fuxIoVK3D+/HlERkZCq9Xi7bffxtVXX+0U56ZNm7BlyxZERETAbDYjIyMDeXl5AYnRmziB4F7PixcvYtGiRThx4gR4nscf/vAHpKamAkDQr+cPP/yARYsW4fz584iLi0NxcTH+4z/+A3l5eZg/fz6uv/56mM1mvPjiizh48CAAIC8vDzNmzFA0Ll/izM/Px7Fjx8DzPCIjIzF//nzccccdAYtx+fLl2Lt3L86dO4f4+Hj06dMHH330kequpZQ4g30tT58+jcmTJ2Pw4MHo1asXAMBgMOD111+X5XqGfbIhhBASfGHfjUYIIST4KNkQQghRHCUbQgghiqNkQwghRHGUbAghhCiOkg0hhBDFUbIhREV27dqFnJwcjBgxAg888ECwwyFENiFfroaQcNKnTx/MmjULP/74Iw4fPhzscAiRDbVsCAmwyspKjB49GseOHQNgLVMyZswYHD58GGPHjsVvf/tbJCYmBjlKQuRFyYaQABs4cCCeeeYZPPPMM2htbcXixYvx+9//XrBQKCHhgpINIUGQnZ2NQYMGITs7Gw0NDXjqqaeCHRIhiqJkQ0iQZGdn49SpU3jggQdEK+wSEi4o2RASBCaTCX/6059wzz33oKSkBM3NzcEOiRBFUbIhJAiKioowfPhwFBUV4c4778TSpUsBAGazGW1tbejs7ITFYkFbWxs6OjqCHC0h/qMtBggJsP3796OwsBA7duxAnz59YDKZMG3aNDzxxBPo7OzEc88953T89OnT8fLLLwcpWkLkQcmGEEKI4qgbjRBCiOIo2RBCCFEcJRtCCCGKo2RDCCFEcZRsCCGEKI6SDSGEEMVRsiGEEKI4SjaEEEIUR8mGEEKI4v4/QJTjpHe8PGkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "def plot_dataset(X, y, title):\n",
    "    \n",
    "    # Activate Seaborn visualization\n",
    "    sns.set()\n",
    "    \n",
    "    # Plot both classes: Class1->Blue, Class2->Red\n",
    "    plt.scatter(X[y==1, 0], X[y==1, 1], c='blue', label=\"class 1\")\n",
    "    plt.scatter(X[y==0, 0], X[y==0, 1], c='red', label=\"class 2\")\n",
    "    plt.legend(loc='upper right')\n",
    "    plt.xlabel('x1')\n",
    "    plt.ylabel('x2')\n",
    "    plt.xlim(-2, 2)\n",
    "    plt.ylim(-2, 2)\n",
    "    plt.title(title)\n",
    "    plt.show()\n",
    "    \n",
    "plot_dataset(X, y, title=\"Dataset\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Importing the necessary libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-09T14:42:28.491779Z",
     "start_time": "2021-01-09T14:42:28.489635Z"
    }
   },
   "outputs": [],
   "source": [
    "import time\n",
    "from torch.nn import BCELoss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are creating the network below. We will have two hidden layers. Since the data seems easily seperable, we can have a small network (2 hidden layers) with 10 units at each layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-09T14:42:28.498956Z",
     "start_time": "2021-01-09T14:42:28.494975Z"
    }
   },
   "outputs": [],
   "source": [
    "net = nn.Sequential(nn.Linear(in_features=2, out_features=10),\n",
    "                    nn.ReLU(),\n",
    "                    nn.Linear(10, 10),\n",
    "                    nn.ReLU(),\n",
    "                    nn.Linear(10, 1),\n",
    "                    nn.Sigmoid())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's define the training parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-09T14:42:28.504886Z",
     "start_time": "2021-01-09T14:42:28.500985Z"
    }
   },
   "outputs": [],
   "source": [
    "batch_size = 4           # How many samples to use for each weight update \n",
    "epochs = 50              # Total number of iterations\n",
    "learning_rate = 0.01     # Learning rate\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "epochs = 50   # Total number of iterations\n",
    "lr = 0.01     # Learning rate\n",
    "\n",
    "# Define the loss. As we used sigmoid in the last layer, we use `nn.BCELoss`.\n",
    "# Otherwise we could have made use of `nn.BCEWithLogitsLoss`.\n",
    "loss = BCELoss(reduction='none')\n",
    "\n",
    "# Define the optimizer, SGD with learning rate\n",
    "optimizer = torch.optim.SGD(net.parameters(), lr=lr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-09T14:42:28.512093Z",
     "start_time": "2021-01-09T14:42:28.506620Z"
    }
   },
   "outputs": [],
   "source": [
    "# Split the dataset into two parts: 80%-20% split\n",
    "X_train, X_val = X[0:int(len(X)*0.8), :], X[int(len(X)*0.8):, :]\n",
    "y_train, y_val = y[:int(len(X)*0.8)], y[int(len(X)*0.8):]\n",
    "\n",
    "# Use PyTorch DataLoaders to load the data in batches\n",
    "train_dataset = torch.utils.data.TensorDataset(torch.tensor(X_train, dtype=torch.float32),\n",
    "                                               torch.tensor(y_train, dtype=torch.float32))\n",
    "train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size)\n",
    "\n",
    "# Move validation dataset on CPU/GPU device\n",
    "X_val = torch.tensor(X_val, dtype=torch.float32).to(device)\n",
    "y_val = torch.tensor(y_val, dtype=torch.float32).to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start the training process. We will have training and validation sets and print our losses at each step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-09T14:42:32.187608Z",
     "start_time": "2021-01-09T14:42:28.513937Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0. Train_loss 0.691907 Validation_loss 0.676005 Seconds 0.082086\n",
      "Epoch 1. Train_loss 0.670287 Validation_loss 0.655867 Seconds 0.077845\n",
      "Epoch 2. Train_loss 0.643775 Validation_loss 0.618162 Seconds 0.076957\n",
      "Epoch 3. Train_loss 0.595045 Validation_loss 0.548945 Seconds 0.077757\n",
      "Epoch 4. Train_loss 0.498955 Validation_loss 0.408997 Seconds 0.077910\n",
      "Epoch 5. Train_loss 0.303406 Validation_loss 0.193668 Seconds 0.076993\n",
      "Epoch 6. Train_loss 0.125799 Validation_loss 0.080758 Seconds 0.077985\n",
      "Epoch 7. Train_loss 0.057654 Validation_loss 0.042393 Seconds 0.077412\n",
      "Epoch 8. Train_loss 0.032992 Validation_loss 0.026434 Seconds 0.077835\n",
      "Epoch 9. Train_loss 0.021789 Validation_loss 0.018319 Seconds 0.077515\n",
      "Epoch 10. Train_loss 0.015715 Validation_loss 0.013642 Seconds 0.077841\n",
      "Epoch 11. Train_loss 0.012024 Validation_loss 0.010670 Seconds 0.076876\n",
      "Epoch 12. Train_loss 0.009590 Validation_loss 0.008659 Seconds 0.077558\n",
      "Epoch 13. Train_loss 0.007895 Validation_loss 0.007228 Seconds 0.077840\n",
      "Epoch 14. Train_loss 0.006661 Validation_loss 0.006167 Seconds 0.076993\n",
      "Epoch 15. Train_loss 0.005727 Validation_loss 0.005353 Seconds 0.078901\n",
      "Epoch 16. Train_loss 0.005001 Validation_loss 0.004713 Seconds 0.077746\n",
      "Epoch 17. Train_loss 0.004424 Validation_loss 0.004198 Seconds 0.077517\n",
      "Epoch 18. Train_loss 0.003954 Validation_loss 0.003775 Seconds 0.077761\n",
      "Epoch 19. Train_loss 0.003560 Validation_loss 0.003421 Seconds 0.077350\n",
      "Epoch 20. Train_loss 0.003228 Validation_loss 0.003124 Seconds 0.077678\n",
      "Epoch 21. Train_loss 0.002946 Validation_loss 0.002869 Seconds 0.077460\n",
      "Epoch 22. Train_loss 0.002704 Validation_loss 0.002650 Seconds 0.077357\n",
      "Epoch 23. Train_loss 0.002494 Validation_loss 0.002459 Seconds 0.077229\n",
      "Epoch 24. Train_loss 0.002313 Validation_loss 0.002288 Seconds 0.077835\n",
      "Epoch 25. Train_loss 0.002154 Validation_loss 0.002138 Seconds 0.077294\n",
      "Epoch 26. Train_loss 0.002015 Validation_loss 0.002005 Seconds 0.077167\n",
      "Epoch 27. Train_loss 0.001892 Validation_loss 0.001887 Seconds 0.078338\n",
      "Epoch 28. Train_loss 0.001782 Validation_loss 0.001781 Seconds 0.076799\n",
      "Epoch 29. Train_loss 0.001683 Validation_loss 0.001686 Seconds 0.077582\n",
      "Epoch 30. Train_loss 0.001593 Validation_loss 0.001599 Seconds 0.077930\n",
      "Epoch 31. Train_loss 0.001513 Validation_loss 0.001521 Seconds 0.076850\n",
      "Epoch 32. Train_loss 0.001439 Validation_loss 0.001449 Seconds 0.076659\n",
      "Epoch 33. Train_loss 0.001372 Validation_loss 0.001383 Seconds 0.077665\n",
      "Epoch 34. Train_loss 0.001310 Validation_loss 0.001323 Seconds 0.076882\n",
      "Epoch 35. Train_loss 0.001253 Validation_loss 0.001268 Seconds 0.076400\n",
      "Epoch 36. Train_loss 0.001201 Validation_loss 0.001216 Seconds 0.077991\n",
      "Epoch 37. Train_loss 0.001152 Validation_loss 0.001169 Seconds 0.077189\n",
      "Epoch 38. Train_loss 0.001107 Validation_loss 0.001124 Seconds 0.076929\n",
      "Epoch 39. Train_loss 0.001065 Validation_loss 0.001083 Seconds 0.077906\n",
      "Epoch 40. Train_loss 0.001026 Validation_loss 0.001044 Seconds 0.076958\n",
      "Epoch 41. Train_loss 0.000990 Validation_loss 0.001008 Seconds 0.077863\n",
      "Epoch 42. Train_loss 0.000956 Validation_loss 0.000974 Seconds 0.078232\n",
      "Epoch 43. Train_loss 0.000924 Validation_loss 0.000943 Seconds 0.077750\n",
      "Epoch 44. Train_loss 0.000894 Validation_loss 0.000913 Seconds 0.077740\n",
      "Epoch 45. Train_loss 0.000865 Validation_loss 0.000885 Seconds 0.078359\n",
      "Epoch 46. Train_loss 0.000838 Validation_loss 0.000858 Seconds 0.077139\n",
      "Epoch 47. Train_loss 0.000813 Validation_loss 0.000833 Seconds 0.077528\n",
      "Epoch 48. Train_loss 0.000789 Validation_loss 0.000809 Seconds 0.077115\n",
      "Epoch 49. Train_loss 0.000767 Validation_loss 0.000787 Seconds 0.077799\n"
     ]
    }
   ],
   "source": [
    "train_losses = []\n",
    "val_losses = []\n",
    "for epoch in range(epochs):\n",
    "    start = time.time()\n",
    "    training_loss = 0\n",
    "    # Build a training loop, to train the network\n",
    "    for idx, (data, target) in enumerate(train_loader):\n",
    "        # zero the parameter gradients\n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        data = data.to(device)\n",
    "        target = target.to(device).view(-1, 1)\n",
    "        \n",
    "        output = net(data)\n",
    "        L = loss(output, target).sum()\n",
    "        training_loss += L.item()\n",
    "        L.backward()\n",
    "        optimizer.step()\n",
    "    \n",
    "    # Get validation predictions\n",
    "    val_predictions = net(X_val)\n",
    "    # Calculate the validation loss\n",
    "    val_loss = torch.sum(loss(val_predictions, y_val.view(-1, 1))).item()\n",
    "    \n",
    "    # Take the average losses\n",
    "    training_loss = training_loss / len(y_train)\n",
    "    val_loss = val_loss / len(y_val)\n",
    "    \n",
    "    train_losses.append(training_loss)\n",
    "    val_losses.append(val_loss)\n",
    "    \n",
    "    end = time.time()\n",
    "    print(\"Epoch %s. Train_loss %f Validation_loss %f Seconds %f\" % \\\n",
    "          (epoch, training_loss, val_loss, end-start))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see the training and validation loss plots below. Losses go down as the training process continues as expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-09T14:42:32.401843Z",
     "start_time": "2021-01-09T14:42:32.189298Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAEcCAYAAAAydkhNAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA1V0lEQVR4nO3dd3gc1dX48e/Mrnq3tG5yt9GhuggbQyAhCaGZmgRITAKkEEIoKUAgIYQSIKGlkGBKgLz4jcGQwO+FQJwAIaGYanCBUK5t3BuWZFlWscruzO+PGckrWbIlWdp6Ps+jZ2fvTjnXRWfvvXPnWq7ropRSSnXHjncASimlEpcmCaWUUj3SJKGUUqpHmiSUUkr1SJOEUkqpHmmSUEop1SNNEkolIBFxRWRSvONQSpOESnkiskZEvhDvOJRKRpoklFJK9SgY7wCUihcRyQJuBc7yi/4CXGWMaRGRMuAh4CjAAd4HjjbGOCJyFfB9oBDYBFxkjHmhy7kPB54Eyo0xEb/si8ANxpjJInIYcCdwALATeAK4zBjT2k2cLwLzjDEP+O+/AZxvjDnKf78/8AfgUKAK+Lkx5i/+Z7OAO4DRwA7gt8aYO/bhj02lGW1JqHT2M+BwYCowBTgMuMb/7HJgAxAChgFXA66ICHAJMMMYUwAcD6zpemJjzBtAI/D5qOKzgUf87QjwI6AMOAI4BriorxUQkTzgef+8Q4HZwN0icpC/y4PAd/1YDwb+3ddrqPSmLQmVzr4GXGqM2QogIjcA9wE/B9qAEcBYY8xK4BV/nwiQBRwoIlXGmDV7OP98vF/az4tIATALuALAGPNO1H5rROQ+4Gjgd32sw8nAGmPM//jvF4vIE8AZeK2fNj/WZcaYWqC2j+dXaU5bEiqdjQTWRr1f65cB3A6sBJ4TkVUi8hMAP2H8ELge2Coij4rISLr3CPAlv1vrS8BiY8xaABGpEJFnRGSLiOwAfonXquirscBMEdne/oOX/Ib7n38ZLzmtFZGXROSIflxDpTFNEiqdbcL7JdtujF+GMabeGHO5MWYCcApwmYgc43/2iD8eMBZw8cY1dmOM+QAv8ZxI564mgHuAj4D9jDGFeN1ZVg9xNgK5Ue+HR22vB14yxhRH/eQbY77nx7DIGHMaXlfUk3jjLkr1mnY3qXSRISLZUe/DeN1B14jIIrxf9tcC8wBE5GS8X+If4w34RoCIPyZRDrwKNOMNOu/py9YjeIPcR+B9w29X4J+3wR94/h7eoHN3luK1SB7Aa+l8G/jE/+wZ4BYROQd41C+bCjT4sZ8JPGOMqfNbLJE9xKrUbrQlodLFArxf6O0/1wM3AW8D7wLvAYv9MoD9gH/h/bJ9HbjbGPMi3njELUA1sAXvG/rVe7jufOCzwL+NMdVR5VfgtS7qgfuBx/Zwjt8CrXiJYS7wcPsHxph64Djgq3itoC14LZssf5dz8MY8dgAXAl/fw3WU2o2liw4ppZTqibYklFJK9UiThFJKqR5pklBKKdUjTRJKKaV6lEq3wGYBM4DN6G1+SinVWwG8pwssAlq6fphKSWIG/qMTlFJK9dmngYVdC1MpSWwGqK1txHH6fltvaWk+NTUNAx5UokvXekP61l3rnV72Vm/btigpyQP/d2hXqZQkIgCO4/YrSbQfm47Std6QvnXXeqeXXta72256HbhWSinVo5i1JESkAu+RAqVADXCuMWZFl33+F5gcVTQZON0Y87dYxamUUmqXWHY33QvMMcbME5Gv4z23P3pBFowx57Zvi8gUvAVSno1hjEqlLdd1qa2torW1Ge95h6ll61Ybx3HiHUbMefV2yczMpqQkhGX19LDh7sUkSYjIUKASONYvmg/cJSIhY0xPT778NvCwMWa3W7KUUgOvoaEOy7IYNmwUlpV6PdHBoE04nH5JIhi0aWsLs317NQ0NdRQUFPft+MEJazejgY3ta/0aYyIisskv3y1JiEgm3hMyv9DXC5WW5vc7yFCooN/HJrN0rTekb927q3dNzSZKS4cRDKbS/SydBYOpl/x6IyMjSElJKdu2fUIoNLpPxybqv4bTgXXGmKV9PbCmpqFfdzCEQgVUVdX3+bhkl671hvSte0/1bmtrw3VT99t2OrckwmEH17VpbW3b7e/etq09frmOVVpdD5SLSADAfx3pl3fnW8CfYhQbH6zZxgW//Bdrt6TfLwylovW1v1olj/7+3cakJWGM2SoiS/EWhZ/nvy7pbjxCREbhzfw7OxaxAYwozSPsONzx6BKu+Oo0xg5Pzy4IpRLFd75zHm1tbYTDbaxfv47x4ycCUFEhXH31db06x5NPPk5LSwtf+crX9rjfwoUvsWzZUi6++Af7HHe7m2++nv33P4Avf/krA3bOeIlld9OFwFwRuRaoBc4FEJEFwLXGmLf9/c4DnjbGbItVYCUFWfzye0fyk7te0UShVAK4//65AGzevInzzz+Hhx56ZLd9wuHwHsdPTj/9jF5d66ijjuaoo47uX6BpIGZJwhjzETCzm/JZXd7fHKuYog0vzePKsyu57ZEl3D5/CVfMnsq44YXxCEUp1YMzzjiFk08+jXfeWcTIkeVccMFFXH/9z2hsbKS1tZVPfepILrrIaxE8+OB97Ny5k0su+SELFjzNv/71LPn5Baxa9TEFBfncdNNtlJaWsWDB07z22ivcdNNtLF78Nr///W848MCDeP/99wCLG274JePGjQfgvvvm8O9/P09hYRHTph3KO+8s4sEH/9zr+D/88H1+97s7aG7eSXZ2Dj/84RUccMBB1NZu4/rrr6G2tgaA6dMP4/vfv5z33lvGb397G47jEg6HOe+8b3HssScM+J/rniTqwHVMua1NNJoPKCs5gKvOnsatjyzhjvlLufyrUxk/QhOFSj+vvreZhe92+yiffXbU5BEceciIfh9fXV3NH/5wHwAtLS3ceutvyc3NJRwOc9lll/DGG69x+OGf2u24Dz98n4cems+wYcO59dabePzxx/judy/ebb/Vqz/m6quv5corf8bcuQ8yd+6DXHfdTSxc+DKvvbaQhx6aT1ZWFtdcc1Wf4m5ra+NnP7uSn/70WmbMmMnbb7/Fz352JY899iTPPfcPhg8fzp133g3Ajh07AHj44bmcddbZnHDCSbiuS0ND7J89lZ73g3UR2WT45PHbaHn5T5QWZnLV2dPIzQ7y60eXsnrzjniHp5SKcsIJJ3VsO47D3XffyXnnzebb3/46q1Z9zIoVy7s9bvLkKQwbNhyAgw46mE2bNnS735gxY6mo2N/f7xA2bvT2W7LkbT7/+S+Qk5ODbduceOJJ3R7fk3Xr1pKRkcGMGV6HyvTph5GRkcG6dWs56KBDWLToTebMuZNXX32F3NxcACorpzNv3kM89NADfPDB+xQUxL4bXFsSQGDsVIqPOoPtCx/HbWmi9JgLufLsadz2yBLueHQpP/1aJaOG9n/+hVLJ5shD9u3b/mDKzc3p2H7ssYepr9/BH//4EFlZWdx66820tnY//zYzM6tj27YDRCLdLzvTeT+7Yz/Xdffp7q+ejrcsOPjgyfzP/zzMokVv8uyzC5g37yHuuedBzjrrbI488jMsWvQmv/vdbcyYcTgXXHBRv2PoD21J4N0aNuTo2WQdMZvwmnfY+c/fUpoDV549jcwMm98/8S71Ta3xDlMp1UV9fT2lpWVkZWVRVbWVhQtfGrRrVVZO5z//+RfNzc04jsOzzy7o0/Fjx46jtbWVxYu9e3QWL36bcDjM6NFj2bRpI3l5+XzhC8dz6aU/wpiPcByHdevWUl4+itNP/zJnnjmbDz98fzCqtkfakoiSecjxWFn5NL/0IE1/v53SEy/jki8dwq0PL+GeJ//LZV+ZSjCgeVWpRHHmmV/l5z+/im9+82yGDh3GoYfOGLRrHXXU0bz33rt84xuzKSsLcdBBh1Bf3/Pcqvvvv5d58+Z2vL/yyqu5+ebbOg1c33TTrWRkZLBkyTs8+ug8AoEgruvw4x//FNu2efzxR1m8+B0yMoJkZGTyox/9eNDq1xPLdVPmQV7jgNUDMeM6vGYJO1+Yg104lJxZP+aN1c088MyHfK6ynHOOk4GNOs7SddYxpG/de6r3li1rGT58bBwiio2BmHHd1NRIbm4ejuNwyy03UlYWinn3T19F17u7v+OoGdfjgTW7HT/4ISaf4Lhp5Jx4OTufvZOmp27i8NN/zobDxvDPt9YxOpTPZ6eVxztEpVQc3HjjdWzZsomWlhZEDuBrXzt37wclOU0SPQiOPIDck6+i6ambaXnzr5zx2fPZUN3Aw88vZ0RpLjKmJN4hKqVi7Fe/uiPeIcScdrDvQSA0nsxDjiO84lXc6lVceOpBhIpzmPN//6V6+854h6eUUoNOk8ReZE47BSuniObXHiYnK8D3z5hMxHH5/RPv0dwajnd4Sik1qDRJ7IWVmUPWYWfgbF1FeOUbDB+Sy/dOO4iNVQ08/dqaeIenlFKDSpNELwQrjsQuG0fLW3/FbWvh4AmlHCohXlqySVsTSqmUpkmiFyzLJutTX8NtrKV12d8BOO6wMTS1hAft+TZKKZUINEn0UnD4fgQnHk7rsn/g1FczqbyIiSMLef7t9f2al6GU6tlll13Kk08+0anMdV3OPPNUli5d3ONxN998PU888RjgrSfx2GMPd7vfggVPc801V+41jpdffpEPPvhvx/uPPvqAG264pjdV6LVLLrmAV199ZUDPOZA0SfRB1swzAYuWN/8CeK2Jqu3NLFlRHd/AlEoxJ510KgsWPN2pbMmSdwgEAkydWtmrc5x++hl7XXBob1555cVOj8LYf/8Due66m/bpnMlG50n0gZ1fSuaUE2ld/BThzcdQWbEfZUXZPLdoHYdKKN7hKTVg2pa/Spt5eVDOnSGfIaPiyD3u85nPfJbf/OYWVq9exfjxEwD4+9//xqxZp/Dxxyv59a9vobl5J62trZx66hc566zdF7KMXk+ira2NO+64nXfeeZtQaChjxozr2K+n87355ussXPgyb7/9Fk8//RRf+crZDBs2nDlz7uxYQ+If/3iG+fP/jGVZjBw5iiuvvJqSkiEsWPA0zz//TwoKCndbv6K33njjNe677y4cx6G4uIQf//hqRo0azbp1a7j55hv8Z0hFOPHEUzj77HN45ZUXuf/+e/yHF4b50Y+upLJyeq+v1xNNEn2UOXUWbeYVWl57hNwvXscXpo/m0RdWsGrTDiaM1LUnlBoIGRkZHHvsCfzjH09z0UU/oKmpkVdeeYkLL7yEvLw8fve7u8nMzKSpqYkLLjiPww47omNhoO489dQTbNq0iT//+S+Ew2Euvvg7jBjhPeV2xIgR3Z5v5swjOOqoz3RahrT94XwAq1at5N577+LBB+dRVlbG/fffw29/ezu/+MWvAPjwww+YO3fv61d0p7Z2GzfddC1/+MMfGT9+As888yQ33HAN998/l//3/x7niCOO5BvfOB/YtfbEAw/cx+WX/4QpU6YRiURobh6YuVyaJPrICmaRNfMsmv99L+GVr/PpyTN5auEqnlu0jgtPOzje4Sk1IDIqjtzrt/3BdtJJp3HFFZdywQUX88ILzzN58hRCoaFs21bDXXfdwsqVy7Esm+rqKlauXL7HJLF48TvMmnUywWCQYDDI8cefyLvvLgWgubm5z+fzzvk2RxxxJGVlXuvgtNO+xDe+satF03X9ikWL3ux13d9//79MnFjR0YqaNetUfv3rW2lqamTq1GnMmXMnbW1tVFZO72gtHHrodO6667d87nNf4PDDP8WECZN6fb090TGJfghOnIlVEKLt4zfJyQrymSkjefujKmrqmuMdmlIpY7/9KigtLePNN19nwYK/cdJJpwLeEqJDhpTypz89zNy58znggINobd3zo/z39CDT/pzPOye7rQ8R/TYzM7Nje0/rV/RwdnpauuKznz2Ge+55kPLyUcyb9xA33ngtAN///uX85CfXEgxm8POf/4S//e3/+nC9nsUsSYhIhYi8LiLL/df9etjvLBF5T0T+678Oi1WMvWVZFsFxlUQ2foDbupNjDh0FwAvvdL/SlVKqf0466VT+9Kc/sn79Oo466mgAGhrqGTp0GMFgkFWrVrJs2dK9nmf69Bn8859/JxwO09LSzPPP/7Pjsz2dLy8vr8clQw89dAavv/4qNTXejStPP/0k06cf1v/KRjnooMmsXLmctWvXAN7Yx377Cbm5eWzYsJ4hQ0qZNesUvvnN7/DBB97A+rp1a5g4cRJnnTWb4447kQ8//GBAYolld9O9wBxjzDwR+TpwH/D56B1EZDpwPfB5Y8wWESkCul9mKs6C4yppe+9ZwuvfpWziTKbvH+KlZRs55chx5GRpL55SA+HYY09kzpzfc9ppXyIjIwOA8877NjfeeC3PPfcPysvLmTp12l7Pc+qpX2LVqpWcc85ZDB06jKlTD2Xz5o17Pd/xx8/i5ptv4D//eaFj4LrdhAkT+e53L+ZHP7rYH7gu58c/vrpf9fzlL6/vtCLe7bffyTXX/IIbbvgZkUiE4uISrr32RgD+/e/nee65f5KREcSyLH7wg8sBuOeeu9iwYR2BQJD8/Hx++tNr+xVLVzFZT0JEhgLLgVJjTEREAkANsJ8xpipqv4eBF4wxf+rHZcYxQOtJ9IbrODTO+wGB8gPJOeZ7rNq0g5v+921mH7Mfx84Y3efrx0u6rqkA6Vt3XU8ivezrehKx6m4aDWw0xkQA/NdNfnm0A4EJIvKyiCwWkWtEpP+Lyg4iy7YJjp1GeN0y3EgbE0YWMmlUkU6uU0qllETrFwkCk4FjgUzgn8A64H97ewI/I/ZLKFTQp/0bpxzJJ+Zl8hvXkjtxGmceU8Gv5i5i5ScNHDl5ZL/jiLW+1juVpGvdu6v31q02wWBq38uS6vXrSXu9bdvu87/5WCWJ9UC5iASiuptG+uXR1gKPG2NagBYReQo4jD4kiVh1NwG4+eMhmEXNsoU0Fk5i4rB8SguzWbBwFRUjkuOXT7p2uUD61r2nejuOQ1tbZLc7dlJFunc3ua6L4zi7/d1HdTd1KyZp1RizFVgKzPaLZgNLoscjfI8Ax4mIJSIZwDHAsljE2B9WMJPg6EMIr1mC6zrYtsVB40v4eGMdTuqsHa7SRPtMXZWaIpEwth3o83GxbHtdCFwqIsuBS/33iMgC/64mgEeBrcAHeEnlfeDBGMbYZ8Fxlbg763C2rgJgYnkRjc1httQ0xTkypfomJyef+vrtuG76fdtOda7rUF9fS05O37vjYzYmYYz5CJjZTfmsqG0HuMz/SQrBMVPAChBes5jAsElMKi8CYOXGOkaW5cU5OqV6Lz+/iNraKj75ZAOQei1h27ZxnPRLgF69XTIzs8nPL+rz8Yk2cJ10rKw8AiP3p23NO2QedibDhuSSlx1k5cY6PjMleQavlbIsiyFDhsY7jEGjY1D9k55D/QMsOK4St+4TnO2bsC2LieVFfLyxLt5hKaXUPtMkMQCC47zn24fXeIuhTCovYnNNEw072+IZllJK7TNNEgPAzivBDk3olCQAVm3S1oRSKrlpkhggwXGVOFWrcRq2MX5EIbZlsVK7nJRSSU6TxAAJjve7nNYuJiszwOhh+azcoElCKZXcNEkMkEDxSOyi4YTXLAG8LqdVm3cQScNb7pRSqUOTxAAKjqsksukj3JZGJpYX0trmsGFrY7zDUkqpftMkMYCC4w8FN0J43bJOk+qUUipZaZIYQHZoPFZOEeF1yygtzKY4P1PnSyilkpomiQFkWTZ2aBxO7UYsy2JSeZG2JJRSSU2TxACzi0fg1H2C6zhMKi+iuq6Z2vqEXIFVKaX2SpPEALOLR0CkDbehhomjvHEJ7XJSSiUrTRIDzC4eAYCzfTNjhxUQDNja5aSUSlqaJAZYdJIIBmzGjSjQloRSKmlpkhhgdnYBVlY+Tt1mwJtUt/aTetrCkThHppRSfadJYhBYxcNxtu9KEuGIy9otDXGOSiml+k6TxCAIFI/oSBITdVKdUiqJaZIYBHbxCNydO3BbGinKy2RocY4mCaVUUtIkMQg6Bq/rtgBea2LlxjpcN/XWDVZKpbaYrXEtIhXAXKAUqAHONcas6LLP9cBFwCa/6FVjzMWxinGg2EW77nAKDJ3IpFFFvP7+FqrqmhlanBPn6JRSqvdiliSAe4E5xph5IvJ14D7g893s97/GmCtiGNeAswrLwA50GrwG+HhDnSYJpVRSiUl3k4gMBSqB+X7RfKBSREKxuH6sWXYQu3AYznavu6m8LI/szAArdTlTpVSSidWYxGhgozEmAuC/bvLLu/qqiLwrIs+JyBExim/A2VG3wdq2xYSRhXysK9UppZJMLLubeuNe4GZjTJuIHAs8JSIHGGNqenuC0tL8fl88FCro97FdbRs5lu3r3qVsSA5WIMh+Y4bw7BtrKCvLx7KsAbvOQBjIeiebdK271ju97Eu9Y5Uk1gPlIhIwxkREJACM9Ms7GGO2RG0/LyLrgYOBl3p7oZqaBhyn73cRhUIFVFXV9/m4nrRlloITZuuq1djFw8kOWjS3Rli7oZa87IwBu86+Guh6J5N0rbvWO73srd62be3xy3VMupuMMVuBpcBsv2g2sMQYUxW9n4iUR21PBcYBJhYxDrRdt8F6XU5DCrMBqN2hjw1XSiWPWHY3XQjMFZFrgVrgXAARWQBca4x5G/iliBwKRIBW4Jzo1kUysYuGA95tsIydRklBFgDb6psZNbT/XWJKKRVLMUsSxpiPgJndlM+K2j4vVvEMNisrDyunsGPwekhHktCWhFIqeeiM60FkF48g4ieJovxMLEu7m5RSyUWTxCCyi0bg+nMlArZNcX4W2+qb4xyVUkr1niaJQWQXj8BtacBp9u4sGFKQpetdK6WSiiaJQRS9Sh1ASUEW27S7SSmVRDRJDKKuSWJIYTa19S36NFilVNLQJDGIrPxSCAQ7tSRa2iI0tYTjHJlSSvWOJolBZNk2dtHwTi0J0DuclFLJQ5PEILOLR3Q8DbZE50oopZKMJolBZhePwK2vwo20RU2o09tglVLJQZPEILOLhoPr4OzYqhPqlFJJR5PEILOLRwL+UqY6oU4plWQ0SQwyuzjqQX/ohDqlVHLRJDHIrIxsrLySToPXOqFOKZUsNEnEgHeHk06oU0olH00SMWAXeUnCdd2OCXU7dUKdUioJaJKIAbt4BLTtxN1Z1zGhTruclFLJQJNEDEQPXuuEOqVUMtEkEQPRD/rTCXVKqWSiSSIGrLwSCGbhbN+sE+qUUklFk0QMWJaNXTwcp26LTqhTSiWVYKwuJCIVwFygFKgBzjXGrOhhXwGWAHcbY66IVYyDyS4aQWTrx4BOqFNKJY9YtiTuBeYYYyqAOcB93e0kIgH/sydjF9rgs/OH4DZu67gNVpOEUioZxCRJiMhQoBKY7xfNBypFJNTN7j8BngGWxyK2WLHySsCJ4LY0MKQwm207dEKdUirxxaq7aTSw0RgTATDGRERkk19e1b6TiEwGjgc+B/y8PxcqLc3vd5ChUEG/j92bhuHD2QqUZLUxekQhLYvWk1uQQ35OxqBds7cGs96JLl3rrvVOL/tS75iNSeyNiGQA9wPf9JNIv85TU9OA4/T9G3ooVEBVVX2/rtkb4bA3ia5mw0YyrKEArFhVzaih/U9qA2Gw653I0rXuWu/0srd627a1xy/XsRqTWA+U++MN7eMOI/3ydiOAicACEVkD/BD4joj8MUYxDio7txgAt2n7rlnXOi6hlEpwMWlJGGO2ishSYDYwz39dYoypitpnHVDW/l5ErgfyU+XuJiu3CACnsZYhI3RCnVIqOcTy7qYLgUtFZDlwqf8eEVkgItNjGEdcWMFMyMrDbdquE+qUUkkjZmMSxpiPgJndlM/qYf/rBzumWLNzi3GbtndMqNPbYJVSia7XSUJEPgesMcasFpERwC1ABLjaGLNlsAJMJVZuMU7TdsCbUKfdTUqpRNeX7qa78ZICwK+BDMAFUmJgORasvGLcxu0AOqFOKZUU+tLdVG6MWSciQby5DGOBVmDToESWguzcEsJNdbiuQ0lBNu+t8mZgW5YV79CUUqpbfWlJ7BCRYcDRwAfGmAa/PP6zwZKElVsMbgS3uYEhhbpCnVIq8fWlJfEHYBGQiTeHAeBI4KMBjillWXnFALiNtZQU7FqhLjdb86xSKjH1uiVhjLkV+AJwpDHmUb94I3D+YASWinZNqKvTCXVKqaTQp1tgjTEdD93z73aKGGNeHvCoUpTlJwmnqZYhIysAqNU7nJRSCazXLQkReUlEjvS3rwIeBeaLyNWDFVyqaZ91HT2hbptOqFNKJbC+DFwfDLzhb38H+CxwOP7MabV3ViADK7sAt1En1CmlkkNfuptswBWRiYBljPkQQERKBiWyFGX5s67BmyuhE+qUUomsL0liIXAX3tNa/w/ATxjVgxBXyrLyOs+63ljdGN+AlFJqD/rS3fQNYDvwLnC9X7Y/cOeARpTirJzoloSuUKeUSmy9bkkYY2qAq7uU/X3AI0pxdl6xN+vacTpNqNO5EkqpRNSXB/xlANcA5+AtGLQJ+DNwszGmdXDCSz3erGsHt3kHJQXt60rohDqlVGLqS3fTbXiT6S4EpvivnwduHYS4UlbHrOvoFer0NlilVILqy8D1mcAUv9sJwIjIYmAZ8KMBjyxF2bnezWBu43aGlAwHdEKdUipx9aUl0dOjSvURpn3QsYypTqhTSiWBvrQk/go8LSI3AOvwHhV+DfCXwQgsVUXPutYJdUqpRNeXJHElXlKYgzdwvRHv0RxZgxBXyrLsIFZOYafFh3RCnVIqUfXlFthW4Fr/BwARyQYa8RLIHolIBTAXKAVqgHONMSu67PNNvPENBwgA9xtjft/bGJOFt4xpLaAT6pRSia0vYxLdcen9mMS9wBxjTAVea+S+bvZ5Am9wfCrwKeByEZm8jzEmnM6P5tAJdUqpxLWvSQK8RLFHIjIUqATm+0XzgUoRCUXvZ4zZYYxpP18uu9bRTil21FrX7RPqmnSFOqVUAtprd5OIfH4PH2f28jqjgY3GmAiAMSYiIpv88qou1zsV+BUwEfipMea9Xl4jaVi5xbjNO3CdCKX+XImaumbydEKdUirB9GZM4sG9fL5uIAJpZ4z5G/A3ERkDPCkiC4wxprfHl5bm9/vaoVBBv4/tix1Dh1PtugzJdZg4dggAYayYXb+reF03EaRr3bXe6WVf6r3XJGGMGd/vs++yHigXkYDfigjg3SG1fg/XXScibwEnA71OEjU1DThO33uoQqECqqrq+3xcf4SdHACq128gkFsOwKr1tUwY1v8E11+xrHeiSde6a73Ty97qbdvWHr9cD8SYxF4ZY7YCS4HZftFsYIkxpmtX0/5R22XA54DU625qfzRH43YKcjPIDNpU1+ltsEqpxNOnNa730YXAXBG5FqgFzgUQkQXAtcaYt4HvishxQBveXVN3GWOei2GMMRG91nXQsigtyqZmhyYJpVTiiVmSMMZ8BMzspnxW1HZaPAPKyikEy+q4Dba0MJsabUkopRJQTLqbVGeWHcDKLtyVJLQloZRKUJok4sTKK8bx50qUFWVT39RGS2skvkEppVQXmiTiJHrWdcdcCW1NKKUSjCaJOLFzSzp1N4EmCaVU4tEkESdWXjHuzh24TrijJaG3wSqlEo0miThpvw3WbaqjOD+LgG3pHU5KqYSjSSJO7KgkYdsWQwqztLtJKZVwNEnESfus6/Z1JUoLs6mu2xnHiJRSaneaJOKko7up4zbYHO1uUkolHE0ScWJlF4Jld7rDqa6hlbawE9/AlFIqiiaJOLFsGyu3qGNCXWlhNi7oetdKqYSiSSKOrNxi3J3bAW/WNaBdTkqphKJJIo7s3F3LmJZqklBKJSBNEnEU/WiOkoIsLEtnXSulEosmiTiy8opxm+txI20EAzbF+Vk661oplVA0ScRR9Kxr8MYltLtJKZVINEnEkZ1bAqDrSiilEpYmiTiycosAcKIeGb5tRwsRR+dKKKUSgyaJOLLy/JZE1OJDjuuyvb41jlEppdQumiTiyMrOByug60oopRJWMFYXEpEKYC5QCtQA5xpjVnTZ5+fAV4Gw/3O1MebZWMUYa5blz7qOesgfQHXdTipGF8cxMqWU8sSyJXEvMMcYUwHMAe7rZp+3gBnGmCnAt4DHRCQnhjHGnJUXNaGuUCfUKaUSS0yShIgMBSqB+X7RfKBSRELR+xljnjXGNPlv3wUsvJZHyopexjQzI0BhXqZ2NymlEkasuptGAxuNMREAY0xERDb55VU9HHMu8LExZkNfLlRamt/vIEOhgn4f21/VpSEaPjEd1x5emsuOpnBMY4lHvRNFutZd651e9qXeMRuT6AsRORq4ETi2r8fW1DTgOG6frxkKFVBVVd/n4/ZVi5WLs7OBrZtrsIKZFOZmsv6T+pjFEq96J4J0rbvWO73srd62be3xy3WsxiTWA+UiEgDwX0f65Z2IyBHAPOB0Y4yJUXxxYxeUAeDUew2qsqJsana04Lh9T3RKKTXQYpIkjDFbgaXAbL9oNrDEGNOpq0lEZgCPAWcYYxbHIrZ4s0tHA+BUrwW8wetwxKG+UedKKKXiL5Z3N10IXCoiy4FL/feIyAIRme7vczeQA9wnIkv9n0NiGGPM2cUjIBAkUrMO2DVXQh/0p5RKBDEbkzDGfATM7KZ8VtT2jFjFkygsO4g9ZHRHS6IsakLdxPKieIamlFI64zoRBErHEqlei+u6OldCKZVQNEkkALtsDLQ24TZUk5MVJC87qN1NSqmEoEkiAQTKxgIQqd41LqET6pRSiUCTRAKwh4wCy8KpXgN4dzhpd5NSKhFokkgAVjALu3hkpzucquuacXWuhFIqzjRJJAi7dMyuO5wKs2lpi9DYHI5zVEqpdKdJIkEEysbiNm3HaaqjtMh78K12OSml4k2TRIKw/cFrp2Ztx1yJ6rqd8QxJKaU0SSSKQOkYwLvDqWOFOm1JKKXiTJNEgrCy8rAKQjg1a8nLDpKVGaBab4NVSsWZJokEEigdQ6R6HZZlUaa3wSqlEoAmiQRil43F3fEJbmuTN6FOk4RSKs40SSSQQJk/LlGzXmddK6USgiaJBGKXjQO8tSXKCrNpbA6zs0XnSiil4keTRAKxc4uxcgqJ1KzVO5yUUglBk0SCscvGei0Jf0Ldlm1NcY5IKZXONEkkmEDpWJzaTYwuzSInK8i7H9fEOySlVBrTJJFg7LKx4DrYOzYxZVIpS1dWE3GceIellEpTmiQSzK61JdZyaEWIhp1tLF9fF+eolFLpKmZrXItIBTAXKAVqgHONMSu67HMc8EvgEOAPxpgrYhVforAKyiAjB6d6LQfP/DQZQZvFy6s4YGxJvENTSqWhWLYk7gXmGGMqgDnAfd3sswr4DnB7DONKKJZlEygbQ6RmLVmZAQ4eP4TFy6twdG0JpVQcxCRJiMhQoBKY7xfNBypFJBS9nzFmpTFmCZDWkwPs0rE4NRtwHYfKihC19S2s2Vwf77CUUmkoVi2J0cBGY0wEwH/d5JerLgJlYyHSilO3mSmTygjYFouXV8U7LKVUGorZmESslJbm9/vYUKhgACPpv1b3ADa8CHktnzC8Yn8OmVTG0pXVXHjGFCzLGvDrJUq94yFd6671Ti/7Uu9YJYn1QLmIBIwxEREJACP98gFVU9OA4/S9/z4UKqCqKjG6dFy3EAJBtq82NA+v5JBxJfx5eRXLPtxCeaj/SbA7iVTvWEvXumu908ve6m3b1h6/XMeku8kYsxVYCsz2i2YDS4wx2ofSDcsOYg8ZjVOzDoBpFSEs0C4npVTMxfLupguBS0VkOXCp/x4RWSAi0/3to0RkA3AZ8F0R2SAix8cwxoQRKB1LpHotrutSnJ/FhPJC3tEkoZSKsZiNSRhjPgJmdlM+K2p7ITAqVjElMrtsDHz0Im59NVZhiEMrhvKX/6ykevtOyopz4h2eUipN6IzrBBUIjQcgvH4ZAJUVZYB2OSmlYkuTRIKyy8YRGCG0vvMUbmsTQ0tyGRXK1y4npVRMaZJIUJZlkXX4bNzmBlqXPAPAoRJi5YY66hpb4xydUipdaJJIYIHQOIIVn6L1vedwdmylsiKECyxZoa0JpVRsaJJIcFkzzgDbpuXNvzAqlMfQ4hwdl1BKxYwmiQRn55WQOeUkwqvfJrJlOZUVIT5cU0tTc1u8Q1NKpQFNEkkgc8oJWHlDaHl9PpUVpUQclyUrquMdllIqDWiSSAJWMIusw87AqV7D6Mb3KS/L49EXVrC5pjHeoSmlUpwmiSQRnHQ4dmg8bYse5/unCwHb4jePLaOuoSXeoSmlUpgmiSRhWTZZR8zGbdpO4Zr/8IMzp1C/s5Xf/fVdmlvTevkNpdQg0iSRRILDKwhOmEHrsn8wtiDMRacfzPqtDdzz5PtEHCfe4SmlUpAmiSSTddhZgEPT32/joKJGzjm+gvdW1fDnZw2uLnGqlBpgmiSSjF0YIufEy6GtmaanbuSIjA85+YgxvLxsM8+8tibe4SmlUowmiSQUHHkAuV/+BYGRB9Ly6jxOaF7AZw8s4v9eWc2zb63Triel1IBJueVL04WdU0jOCT+k7b3naHnrr3wpZy3Bscfw2L9X8uKSjZz26fEcdsAw7EFY7lQplT60JZHELMsmc/IJ5J76M7BtTml4nBsmr2Z4YDt//NsHXPent3jHVOlYhVKq37QlkQICQyeQ9+Vf0PzawxSveJVvuQ47x47mpfqx/M+T23hmWBknHzGWg8eXkpUZiHe4SqkkokkiRViZOeR89nycw84kvPIN7OULOYGFHDckwEfNY3nlmdE84g5jxKgRHDy+lIMnDKG8LC/eYSulEpwmiRRj5xaROfl4MicfT6R6LW3LF3Lgytc50F4FQN2OAj5+awj/fj3EtswRDJ10AGUlBZSH8igvy6MoLxNLxzGUUj5NEiksUDaWQNlYsmZ+hUjVapytH1O6dRWFW1ZS2bQWAGf1M9R+nEuNU8CbkQJ22EWQHyKzdDi5xWUUlJQwpCiXkoIsSgqyCAZ0GEupdBKzJCEiFcBcoBSoAc41xqzosk8A+D1wAuACtxhjHohVjKnKCgQJDt8Phu8HQA7gNNXhbF1F1s7NWOvXkl+7hfGNm8gIr4BWYLP347gWDW4W1U4Oa9xsdtr5RIK5OBk5WFm52Fn5BHPyycjNJzM3j4ycPLJyc8nOzSMnO5Oc7CDZmQECtiYXpZJRLFsS9wJzjDHzROTrwH3A57vs8zVgErAfXjJZIiL/MsasiWGcacHOLcIeN40hoc8QqarvKHdbd+Ls2IpTX0Xrjm201NVi1ddS0LidguYdBNs+ITOyk4zWNi+Z1Pd8jTbXZoebQZUbpI0gYYJE7CARKwPHysCxM3DtIK6dgRsIgh3ECmRAIAMrEAQ7gBUMYtkZ2MEgViCIHQhiBQLYHdtBAsGA9xoIYnd8FvB+gkECdgA7GCAQCGIFbGw7SCAQoKU1TMRxsC1Lu9iU6kFMkoSIDAUqgWP9ovnAXSISMsZEL7P2FeB+Y4wDVInIk8CZwO2xiFN5A+Dt3VQZQE9D224kjNvaBC2NhJsb2LljB61NjbQ1NxFu2UmkeSeR1mbctmZoa4ZIG8FIKxmRNmynjYDbTMBpw45ECBAhQJiAGyFoDf5EQMf/2YjXUnLwftwur2DhYOMCLjau/1nHj+W9ElWGRacyuuyDxe6fdSQo71re+87702mbjmMtwPX3J+pYOp2HqGtYBDMChMNOx/uOz6L2oVPS7PJ5dBxYWH6dLbr7vOt5o85p7b6PFXWc2/FR1/2ir7NrY7dE3+V9dnYGzS3hzrF12c/yr2tFx9x5526Ko47v6ctGp2t03sftcv2e4t/TNeyMLMbP+BQZGZndX38fxKolMRrYaIyJABhjIiKyyS+PThJjgLVR79f5+6gEYwWCWDmFkFNIJpA5fGDO67oORMLghHHCbUTCbbS1homE2wi3thKJeOVOOEIk0oYbjhAJt+E6EdxIBMeJ4EQi4Dg4kTCu4227bgTXccCJeNdwHDKCFq0tbeA6/o/rvToOLg6W4wJR5a6XCnCjynGxXO8XruW6u8qI2t+rGDadz+H9V49+7+1r0X4OOr+67Z9773ft25FSui3v+nnnX/9d9+u6T/f7R+9rayMsIazLzGLi9CMG/LwpN3BdWprf72NDoYIBjCR5pGu91cBpn7DpOC5ueyJxnI6U4jpe4mvfdqO3/WOjJ33u2nb9vOsf0WXf3Y7dSzntUXR83qXl6tIlZj+G6HN0PVfH/t23gjvXa/fr7dqMeuN0fw26/TOCYGYWh08Y1+31Yd/+j8cqSawHykUk4LciAsBIvzzaOmAssMh/37VlsVc1NQ04jrv3HbsIhQqoqtpDB3uKStd6Q/rWPbnq3VO3T98P6229+3nFuOupbnurt21be/xyHZNbTowxW4GlwGy/aDawpMt4BMBfge+IiC0iIeB04IlYxKiUUmp3sbwv8ULgUhFZDlzqv0dEFojIdH+fPwOrgBXAG8AvjDGrYhijUkqpKDEbkzDGfATM7KZ8VtR2BPherGJSSim1ZzrDSSmlVI80SSillOqRJgmllFI9SqV5EgHwbufqr305Npmla70hfeuu9U4ve6p31GfdLjZjpdCqZUcBr8Q7CKWUSlKfBhZ2LUylJJEFzMB7fmkkzrEopVSyCAAj8CYxt3T9MJWShFJKqQGmA9dKKaV6pElCKaVUjzRJKKWU6pEmCaWUUj3SJKGUUqpHmiSUUkr1SJOEUkqpHqXSYzn6TUQqgLlAKVADnGuMWRHfqAaeiNwBfBkYBxxijPmvX56y9ReRUrx1SibiTRRaCXzXGFOVyvUGEJEngfGAAzQAlxpjlqZ6vduJyHXA9fj/1tOh3iKyBmj2fwCuMsY8uy9115aE515gjjGmApgD3BfneAbLk8Bn2H1J2FSuvwvcZowRY8xk4GPgFv+zVK43wHnGmCnGmGnAHcCf/PJUrzciUgkcjrckcruUr7fvDGPMVP/nWb+s33VP+yQhIkOBSmC+XzQfqPSXT00pxpiFxphO64qnev2NMduMMS9GFb0BjE31egMYY+qi3hYBTjrUW0Sy8H4RXoT3JSHl/53vyb7WPe2TBDAa2Oivite+Ot4mvzwdpE39RcTGW/nwb6RJvUXkARFZB9wMnEd61PsXwDxjzOqosnSod7uHReRdEblbRIrZx7prklDp5A94ffN3xTuQWDHGnG+MGQNcDdwe73gGm4gcgfegz7vjHUucfNoYMwXvz8BiAP6ta5KA9UC5iAQA/NeRfnk6SIv6+4P2+wFfMcY4pEm92xlj/gx8DthAatf7aGB/YLU/iDsKeBbvxoVUrjcA7d3JxpgWvER5JPv4bz3tk4QxZiuwFJjtF80GlhhjquIWVAylQ/1F5GbgUOB0/z9PytdbRPJFZHTU+1OAbUBK19sYc4sxZqQxZpwxZhxeUjzeGPMXUrjeACKSJyJF/rYFfBVYuq//1vUWWM+FwFwRuRaoBc6NczyDQkR+D3wJGA78S0RqjDEHkcL1F5GD8LpalgOviQjAamPMF0nhegN5wF9FJA9vfZVtwCnGGFdEUrnee5Lq9R4GPOG3FALAB3iD97APddf1JJRSSvUo7bublFJK9UyThFJKqR5pklBKKdUjTRJKKaV6pElCKaVUjzRJKJVgRMQVkUnxjkMp0HkSSu2VP3N3GN58g3YPGWMuiU9ESsWOJgmleucUY8y/4h2EUrGmSUKpfhKRbwDfARbjzWDdDFxsjHnB/3wk3nP8j8Kb8XyrMeZ+/7MAcBXwbWAo3ozw06Me5f4FEfkHUAY8AlxijNGZryrmdExCqX0zE1iF98v8OuD/icgQ/7P5eM8OGgmcAfxSRI7xP7sM7xk6s4BC4FtAU9R5T8Z7kucU4Czg+MGthlLd05aEUr3zpIiEo97/GGjDe2De7/xv+Y+JyOXASSLyIl4L4mRjTDOwVEQeAM4BXgDOB640xhj/fMu6XO8WY8x2YLuI/AeYCvxzUGqm1B5oklCqd07vOibhdzdt7NINtBav5TAS2GaMqe/y2XR/ezTeUqo92RK13QTk9zNupfaJdjcptW/K/ccytxuDt+rXJmCIiBR0+Wyjv70eb40DpRKatiSU2jdDge+LyN3A6cABwAJjTI2IvAb8SkSuACrwBqm/7h/3AHCjiHwArAQOwWuV1MS6AkrtiSYJpXrnaRGJnifxPPAU8CbeinfVwCfAGVG/6Gfj3d20Ce8Z/tcZY573P/sNkAU8hzfo/RHwxcGuhFJ9petJKNVP/pjE+caYo+Idi1KDRccklFJK9UiThFJKqR5pd5NSSqkeaUtCKaVUjzRJKKWU6pEmCaWUUj3SJKGUUqpHmiSUUkr1SJOEUkqpHv1/2dP86vj8Fg8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "plt.plot(train_losses, label=\"Training Loss\")\n",
    "plt.plot(val_losses, label=\"Validation Loss\")\n",
    "plt.title(\"Loss values\")\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Loss\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. <a name=\"5\">Natural Language Processing Context</a>\n",
    "(<a href=\"#0\">Go to top</a>)\n",
    "\n",
    "If we want to use the same type of architecture for text classification, we need to apply some feature extraction methods first. For example: We can get TF-IDF vectors of text fields. After that, we can use neural networks on those features. \n",
    "\n",
    "We will also look at __more advanced neural network architrectures__ such as __Recurrent Neural Networks (RNNs)__, __Long Short-Term Memory networks (LSTMs)__ and __Transformers__. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_pytorch_p39",
   "language": "python",
   "name": "conda_pytorch_p39"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}