{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# HyperParameter Tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hyperparameter optimization or tuning chooses a set of optimal hyperparameters, parameters that control the learning process, for a learning algorithm. The set of hyperparameters yield an optimal model that minimizes a pre-defined loss function on given test data. \n",
    "\n",
    "There are many approaches for HPO: \n",
    "- grid search\n",
    "- random search\n",
    "- bayesian optimization\n",
    "- gradient-based optimization\n",
    "- evolutionary optimization\n",
    "- population based training\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Katib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [Katib](https://github.com/kubeflow/katib) project is inspired by Google vizier. Katib is a scalable and flexible hyperparameter tuning framework and is tightly integrated with Kubernetes. It does not depend on any specific deep learning framework (such as TensorFlow, MXNet, or PyTorch).\n",
    "\n",
    "\n",
    "* Two versions: v1alpha1 and v1alpha2\n",
    "* Across both versions, there is support for random search, grid search, hyperband, bayesian optimization and NAS\n",
    "* One metric is the objective metric, can have a goal for optimization value (no multimetric optimization)\n",
    "* Int, double, discrete, and categorical parameter ranges, no options for scaling type (unless that is what “step” means)\n",
    "* Option for early stopping (not part of the StudyJob (HPO) job request parameters)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Note: In current Kubeflow version, katib is installed by default. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HyperParameter Examples "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will run three Katib Experiments using `random search`, `grid-search`, `bayesian` and `hyperband`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl create -f random-search-example.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you check manifest, you will see\n",
    "\n",
    "\n",
    "```yaml\n",
    "parameters:\n",
    "- name: --lr\n",
    "  parameterType: double\n",
    "  feasibleSpace:\n",
    "    min: \"0.01\"\n",
    "    max: \"0.03\"\n",
    "- name: --num-layers\n",
    "  parameterType: int\n",
    "  feasibleSpace:\n",
    "    min: \"2\"\n",
    "    max: \"5\"\n",
    "- name: --optimizer\n",
    "  parameterType: categorical\n",
    "  feasibleSpace:\n",
    "    list:\n",
    "    - sgd\n",
    "    - adam\n",
    "    - ftrl\n",
    "```\n",
    "\n",
    "\n",
    "This job generates 3 hyperparameters, parameter type and range are also listed.\n",
    "\n",
    "* --lr (Learning Rate) - type: double\n",
    "* --num-layers (Number of NN Layer) - type: int\n",
    "* --optimizer (optimizer) - type: categorical"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The demo should start an experiment and run three jobs with different parameters. You can run following command to check job status.\n",
    "\n",
    "When the `spec.Status.Condition` changes to Completed, the experiment is finished.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl describe experiment random-example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Monitoring\n",
    "\n",
    "You can monitor your results in the Katib UI. If you installed Kubeflow using the deployment guide, you can access the Katib UI at `https://<your kubeflow endpoint>/katib/`\n",
    "\n",
    "For `random-experiment`, please go to `HP (HypterParameter)` -> `Monitor` -> `random-experiment`.\n",
    "\n",
    "![katib-experiment-selection.png](./images/katib-experiment-selection.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pick up best parameters in from results\n",
    "\n",
    "Once you click job and go the detail page, you will see different combination of parameters and accuracy.\n",
    "\n",
    "\n",
    "| trialName  | Validation-accuracy \t| accuracy \t| --lr \t| --num-layers \t| --optimizer|\n",
    "|----------------------------|----------|----------|----------------------|---|------|\n",
    "| random-experiment-rfwwbnsd | 0.974920 | 0.984844 | 0.013831565266960293 | 4 | sgd  |\n",
    "| random-experiment-vxgwlgqq | 0.113854 | 0.116646 | 0.024225789898529138 | 4 | ftrl |\n",
    "| random-experiment-wclrwlcq | 0.979697 | 0.998437 | 0.021916171239020756 | 4 | sgd  |\n",
    "| random-experiment-7lsc4pwb | 0.113854 | 0.115312 | 0.024163810384272653 | 5 | ftrl |\n",
    "| random-experiment-86vv9vgv | 0.963475 | 0.971562 | 0.02943228249244735  | 3 | adam |\n",
    "| random-experiment-jh884cxz | 0.981091 | 0.999219 | 0.022372025623908262 | 2 | sgd  |\n",
    "| random-experiment-sgtwhrgz | 0.980693 | 0.997969 | 0.016641686851083654 | 4 | sgd  |\n",
    "| random-experiment-c6vvz6dv | 0.980792 | 0.998906 | 0.0264125850165842   | 3 | sgd  |\n",
    "| random-experiment-vqs2xmfj | 0.113854 | 0.105313 | 0.026629394628228185 | 4 | ftrl |\n",
    "| random-experiment-bv8lsh2m | 0.980195 | 0.999375 | 0.021769570793012488 | 2 | sgd  |\n",
    "| random-experiment-7vbnqc7z | 0.113854 | 0.102188 | 0.025079750575740783 | 4 | ftrl |\n",
    "| random-experiment-kwj9drmg | 0.979498 | 0.995469 | 0.014985919312945063 | 4 | sgd  |\n",
    "\n",
    "\n",
    "![katib-experiment-result.png](./images/katib-experiment-result.png)\n",
    "\n",
    "You can also click trail name to check Trial data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Note: All rest examples are different optimization algorithms.  \n",
    "> The way to submit the job and check job lifecycle is same as random-search-example we did."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Grid Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl create -f grid-example.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl describe experiment grid-example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bayesian\n",
    "\n",
    "BayesOpt: A toolbox for bayesian optimization, experimental design and stochastic bandits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl create -f bayesopt-example.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl describe experiment bayesopt-example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyperband"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl create -f hyperband-example.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl describe experiment hyperband-example"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}