{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Kubeflow Fairing Introduction\n",
    "\n",
    "Kubeflow Fairing is a Python package that streamlines the process of `building`, `training`, and `deploying` machine learning (ML) models in a hybrid cloud environment. By using Kubeflow Fairing and adding a few lines of code, you can run your ML training job locally or in the cloud, directly from Python code or a Jupyter notebook. After your training job is complete, you can use Kubeflow Fairing to deploy your trained model as a prediction endpoint.\n",
    "\n",
    "\n",
    "# How does Kubeflow Fairing work\n",
    "\n",
    "Kubeflow Fairing \n",
    "1. Packages your Jupyter notebook, Python function, or Python file as a Docker image\n",
    "2. Deploys and runs the training job on Kubeflow or AI Platform. \n",
    "3. Deploy your trained model as a prediction endpoint on Kubeflow after your training job is complete.\n",
    "\n",
    "\n",
    "# Goals of Kubeflow Fairing project\n",
    "\n",
    "- Easily package ML training jobs: Enable ML practitioners to easily package their ML model training code, and their code’s dependencies, as a Docker image.\n",
    "- Easily train ML models in a hybrid cloud environment: Provide a high-level API for training ML models to make it easy to run training jobs in the cloud, without needing to understand the underlying infrastructure.\n",
    "- Streamline the process of deploying a trained model: Make it easy for ML practitioners to deploy trained ML models to a hybrid cloud environment.\n",
    "\n",
    "\n",
    "> Note: Before fairing workshop, please read `README.md` under `02_01_fairing_introduction`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install latest Fairing from github repository\n",
    "!pip install kubeflow-fairing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check fairing is installed \n",
    "!pip show kubeflow-fairing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Example\n",
    "\n",
    "If you see any issues, please restart notebook. It's probably because of new installed packages.\n",
    "\n",
    "Click `Kernel` -> `Restart & Clear Output`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "from kubeflow import fairing\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "\n",
    "def train():\n",
    "    # Genrating random linear data \n",
    "    # There will be 50 data points ranging from 0 to 50 \n",
    "    x = np.linspace(0, 50, 50) \n",
    "    y = np.linspace(0, 50, 50) \n",
    "\n",
    "    # Adding noise to the random linear data \n",
    "    x += np.random.uniform(-4, 4, 50) \n",
    "    y += np.random.uniform(-4, 4, 50) \n",
    "\n",
    "    n = len(x) # Number of data points \n",
    "\n",
    "    X = tf.placeholder(\"float\") \n",
    "    Y = tf.placeholder(\"float\")\n",
    "    W = tf.Variable(np.random.randn(), name = \"W\") \n",
    "    b = tf.Variable(np.random.randn(), name = \"b\") \n",
    "    learning_rate = 0.01\n",
    "    training_epochs = 1000\n",
    "    \n",
    "    # Hypothesis \n",
    "    y_pred = tf.add(tf.multiply(X, W), b) \n",
    "\n",
    "    # Mean Squared Error Cost Function \n",
    "    cost = tf.reduce_sum(tf.pow(y_pred-Y, 2)) / (2 * n)\n",
    "\n",
    "    # Gradient Descent Optimizer \n",
    "    optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost) \n",
    "\n",
    "    # Global Variables Initializer \n",
    "    init = tf.global_variables_initializer() \n",
    "\n",
    "\n",
    "    sess = tf.Session()\n",
    "    sess.run(init) \n",
    "      \n",
    "    # Iterating through all the epochs \n",
    "    for epoch in range(training_epochs): \n",
    "          \n",
    "        # Feeding each data point into the optimizer using Feed Dictionary \n",
    "        for (_x, _y) in zip(x, y): \n",
    "            sess.run(optimizer, feed_dict = {X : _x, Y : _y}) \n",
    "          \n",
    "        # Displaying the result after every 50 epochs \n",
    "        if (epoch + 1) % 50 == 0: \n",
    "            # Calculating the cost a every epoch \n",
    "            c = sess.run(cost, feed_dict = {X : x, Y : y}) \n",
    "            print(\"Epoch\", (epoch + 1), \": cost =\", c, \"W =\", sess.run(W), \"b =\", sess.run(b)) \n",
    "      \n",
    "    # Storing necessary values to be used outside the Session \n",
    "    training_cost = sess.run(cost, feed_dict ={X: x, Y: y}) \n",
    "    weight = sess.run(W) \n",
    "    bias = sess.run(b) \n",
    "\n",
    "    print('Weight: ', weight, 'Bias: ', bias)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Local training for development\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Remote training\n",
    "\n",
    "We will show you how to remotely run training job in kubernetes cluster. You can use `ECR` as your container image registry."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Authenticate ECR\n",
    "# This command retrieves a token that is valid for a specified registry for 12 hours, \n",
    "# and then it prints a docker login command with that authorization token. \n",
    "# Then we executate this command to login ECR\n",
    "\n",
    "REGION='us-west-2'\n",
    "!eval $(aws ecr get-login --no-include-email --region=$REGION)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an ECR repository in the same region\n",
    "# If you receive \"RepositoryAlreadyExistsException\" error, it means the repository already\n",
    "# exists. You can move to the next step\n",
    "!aws ecr create-repository --repository-name fairing-job --region=$REGION"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setting up AWS Elastic Container Registry (ECR) for storing output containers\n",
    "# You can use any docker container registry istead of ECR\n",
    "AWS_ACCOUNT_ID=fairing.cloud.aws.guess_account_id()\n",
    "AWS_REGION='us-west-2'\n",
    "DOCKER_REGISTRY = '{}.dkr.ecr.{}.amazonaws.com'.format(AWS_ACCOUNT_ID, AWS_REGION)\n",
    "\n",
    "fairing.config.set_builder('append', base_image='tensorflow/tensorflow:1.14.0-py3', registry=DOCKER_REGISTRY, push=True)\n",
    "fairing.config.set_deployer('job')\n",
    "    \n",
    "if __name__ == '__main__':\n",
    "    remote_train = fairing.config.fn(train)\n",
    "    remote_train()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}