{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multilingual Transcription Pipeline\n",
    "In this notebook, you will:\n",
    "1. Deploy the trained model to a sagemaker endpoint\n",
    "2. Create transcription pipeline that feeds model predictions into Amazon Transcribe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import libraries and load AWS credentials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U sagemaker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "from sagemaker.estimator import Estimator\n",
    "from sagemaker.serializers import JSONSerializer\n",
    "from sagemaker.predictor import Predictor\n",
    "import json\n",
    "import tarfile\n",
    "import os\n",
    "import pandas as pd\n",
    "import boto3\n",
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "role = sagemaker.get_execution_role()\n",
    "sess = sagemaker.session.Session()\n",
    "account_id = boto3.client('sts').get_caller_identity().get('Account')\n",
    "region = boto3.session.Session().region_name\n",
    "\n",
    "bucket = sess.default_bucket()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Insert your training job ID here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_job_id = '<insert-training-job-id>'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "image_uri = f'{account_id}.dkr.ecr.{region}.amazonaws.com/spoken-language-detection'\n",
    "model_path = f's3://{bucket}/models/{training_job_id}/output/model.tar.gz'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = sagemaker.Model(\n",
    "    image_uri=image_uri,\n",
    "    model_data=model_path,\n",
    "    role=role\n",
    ")\n",
    "\n",
    "model.deploy(1, 'ml.m4.xlarge')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor = Predictor(model.endpoint_name, serializer=JSONSerializer())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Detect language and run transcription jobs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download sample audio clips from [Audio Lingua](audio-lingua.eu)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir samples\n",
    "!wget -O samples/french-sample.wav https://audio-lingua.eu/spip.php?article7143\n",
    "!wget -O samples/english-sample.wav https://audio-lingua.eu/spip.php?article6968\n",
    "!wget -O samples/russian-sample.wav https://audio-lingua.eu/spip.php?article7109\n",
    "!wget -O samples/spanish-sample.wav https://audio-lingua.eu/spip.php?article7103\n",
    "!wget -O samples/italian-sample.wav https://audio-lingua.eu/spip.php?article7139"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show files to be transcribed from \"samples\" folder.  Each file is a different language."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "files = [os.path.join('samples', f) for f in os.listdir('samples')]\n",
    "files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loop through the files, upload each to s3, predict the language, and pass the predicted language and audio file to Transcribe.  Go to Transcribe in the AWS console to see the transcription jobs.  **Make sure that your role has access to Transcribe (attach AmazonTranscribeFullAccess policy to your role)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lang_code_dict = {\n",
    "    'en' : 'en-US',\n",
    "    'es' : 'es-ES',\n",
    "    'it' : 'it-IT',\n",
    "    'fr' : 'fr-FR',\n",
    "    'ru' : 'ru-RU',\n",
    "    'de' : 'de-DE'\n",
    "}\n",
    "\n",
    "transcribe = boto3.client('transcribe')\n",
    "\n",
    "for f in files:\n",
    "    s3_path = sess.upload_data(f, key_prefix='samples')\n",
    "    \n",
    "    pred = predictor.predict([s3_path])\n",
    "    pred = json.loads(pred)[0]\n",
    "\n",
    "    print('Detected language : {}'.format(pred))\n",
    "    job_name = f.split('/')[-1].split('.wav')[0]\n",
    "\n",
    "    transcribe.start_transcription_job(\n",
    "        TranscriptionJobName=job_name,\n",
    "        Media={'MediaFileUri': s3_path},\n",
    "        MediaFormat='wav',\n",
    "        LanguageCode=lang_code_dict[pred]\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Be sure to delete the endpoint after evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor.delete_endpoint()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}