{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Architecture Search"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Developing neural network models often requires significant architecture engineering. User can sometimes get by with **transfer learning**, but if user really want the best possible performance it’s usually best to design your own network. This requires specialised skills and is challenging in general. \n",
    "\n",
    "We may not even know the limits of the current state-of-the-art techniques! It’s a lot of trial and error and the experimentation itself is time consuming and expensive.\n",
    "\n",
    "This is where **NAS** comes in. NAS is an algorithm that searches for the best **neural network architecture**. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Katib AutoML/NAS Support"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Katib is a scalable Kubernetes-native general AutoML platform that can\n",
    "support a range of AutoML algorithms including both hyperparameter tuning and neural architecture search. \n",
    "\n",
    "AutoML algorithms share the common ground that they run in an iterative manner. The user first defines the search space, metrics target and maximum iterations. The algorithm searches for the optimal solution until the target metrics or the maximum number of iterations is reached. However, they may vary in terms of their internal mechanisms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NAS Examples "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl apply -f nasjob-example-RL.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl describe experiment nas-rl-example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Monitoring\n",
    "\n",
    "You can monitor your results in the Katib UI. If you installed Kubeflow using the deployment guide, you can access the Katib UI at `https://<your kubeflow endpoint>/katib/`\n",
    "\n",
    "For this job, please go to `NAS (HypterParameter)` -> `Monitor` -> `nas-rl-example`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pick up best parameters in from results\n",
    "\n",
    "Once you click job and go the detail page, you will see different combination of parameters and accuracy.\n",
    "\n",
    "\n",
    "| trialName  | Validation-accuracy \t| accuracy \t| --lr \t| --num-layers \t| --optimizer|\n",
    "|----------------------------|----------|----------|----------------------|---|------|\n",
    "| random-experiment-rfwwbnsd | 0.974920 | 0.984844 | 0.013831565266960293 | 4 | sgd  |\n",
    "| random-experiment-vxgwlgqq | 0.113854 | 0.116646 | 0.024225789898529138 | 4 | ftrl |\n",
    "| random-experiment-wclrwlcq | 0.979697 | 0.998437 | 0.021916171239020756 | 4 | sgd  |\n",
    "| random-experiment-7lsc4pwb | 0.113854 | 0.115312 | 0.024163810384272653 | 5 | ftrl |\n",
    "| random-experiment-86vv9vgv | 0.963475 | 0.971562 | 0.02943228249244735  | 3 | adam |\n",
    "| random-experiment-jh884cxz | 0.981091 | 0.999219 | 0.022372025623908262 | 2 | sgd  |\n",
    "| random-experiment-sgtwhrgz | 0.980693 | 0.997969 | 0.016641686851083654 | 4 | sgd  |\n",
    "| random-experiment-c6vvz6dv | 0.980792 | 0.998906 | 0.0264125850165842   | 3 | sgd  |\n",
    "| random-experiment-vqs2xmfj | 0.113854 | 0.105313 | 0.026629394628228185 | 4 | ftrl |\n",
    "| random-experiment-bv8lsh2m | 0.980195 | 0.999375 | 0.021769570793012488 | 2 | sgd  |\n",
    "| random-experiment-7vbnqc7z | 0.113854 | 0.102188 | 0.025079750575740783 | 4 | ftrl |\n",
    "| random-experiment-kwj9drmg | 0.979498 | 0.995469 | 0.014985919312945063 | 4 | sgd  |\n",
    "\n",
    "\n",
    "![katib-experiment-result.png](./images/katib-experiment-result.png)\n",
    "\n",
    "You can also click trail name to check Trial data."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}