{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Kubeflow Primitives Examples "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Kubeflow supports many machine learning and deep learnin frameworks. Katib provides hyper parameter tunning \n",
    "\n",
    "We will run Katib Experiments using `TFjob` and `PyTorchJob` "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tensorlow Job (TFJob)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl create -f tfjob-example.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you check manifest, you will see\n",
    "\n",
    "\n",
    "```yaml\n",
    "  parameters:\n",
    "    - name: --learning_rate\n",
    "      parameterType: double\n",
    "      feasibleSpace:\n",
    "        min: \"0.01\"\n",
    "        max: \"0.05\"\n",
    "    - name: --batch_size\n",
    "      parameterType: int\n",
    "      feasibleSpace:\n",
    "        min: \"100\"\n",
    "        max: \"200\"\n",
    "```\n",
    "\n",
    "\n",
    "This job generates 2 hyperparameters, parameter type and range are also listed.\n",
    "\n",
    "* --learning-rate - type: double\n",
    "* --batch_size - type: int"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The demo should start an experiment and run three jobs with different parameters. You can run following command to check job status.\n",
    "\n",
    "When the `spec.Status.Condition` changes to Completed, the experiment is finished.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl describe experiment tfjob-example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Monitoring\n",
    "\n",
    "You can monitor your results in the Katib UI. If you installed Kubeflow using the deployment guide, you can access the Katib UI at `https://<your kubeflow endpoint>/katib/`\n",
    "\n",
    "For `random-experiment`, please go to `HP (HypterParameter)` -> `Monitor` -> `random-experiment`.\n",
    "\n",
    "![katib-experiment-selection.png](./images/katib-experiment-selection.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pick up best parameters in from results\n",
    "\n",
    "Once you click job and go the detail page, you will see different combination of parameters and accuracy.\n",
    "\n",
    "\n",
    "| trialName  | Validation-accuracy \t| accuracy \t| --lr \t| --num-layers \t| --optimizer|\n",
    "|----------------------------|----------|----------|----------------------|---|------|\n",
    "| random-experiment-rfwwbnsd | 0.974920 | 0.984844 | 0.013831565266960293 | 4 | sgd  |\n",
    "| random-experiment-vxgwlgqq | 0.113854 | 0.116646 | 0.024225789898529138 | 4 | ftrl |\n",
    "| random-experiment-wclrwlcq | 0.979697 | 0.998437 | 0.021916171239020756 | 4 | sgd  |\n",
    "| random-experiment-7lsc4pwb | 0.113854 | 0.115312 | 0.024163810384272653 | 5 | ftrl |\n",
    "| random-experiment-86vv9vgv | 0.963475 | 0.971562 | 0.02943228249244735  | 3 | adam |\n",
    "| random-experiment-jh884cxz | 0.981091 | 0.999219 | 0.022372025623908262 | 2 | sgd  |\n",
    "| random-experiment-sgtwhrgz | 0.980693 | 0.997969 | 0.016641686851083654 | 4 | sgd  |\n",
    "| random-experiment-c6vvz6dv | 0.980792 | 0.998906 | 0.0264125850165842   | 3 | sgd  |\n",
    "| random-experiment-vqs2xmfj | 0.113854 | 0.105313 | 0.026629394628228185 | 4 | ftrl |\n",
    "| random-experiment-bv8lsh2m | 0.980195 | 0.999375 | 0.021769570793012488 | 2 | sgd  |\n",
    "| random-experiment-7vbnqc7z | 0.113854 | 0.102188 | 0.025079750575740783 | 4 | ftrl |\n",
    "| random-experiment-kwj9drmg | 0.979498 | 0.995469 | 0.014985919312945063 | 4 | sgd  |\n",
    "\n",
    "\n",
    "![katib-experiment-result.png](./images/katib-experiment-result.png)\n",
    "\n",
    "You can also click trail name to check Trial data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PyTorch Job (PyTorchJob)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl create -f pytorchjob-example.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl describe experiment pytorchjob-example"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}