{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c4a68ee1-09fb-4d8f-a614-d137c731d243",
   "metadata": {},
   "source": [
    "# Unsupervised evaluation of LLMs using LLMs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ded26b0e",
   "metadata": {},
   "source": [
    "_License information_\n",
    "\n",
    "    Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.\n",
    "    SPDX-License-Identifier: MIT-0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecaa6385-94c8-43d1-8f25-9abf3b9f02c1",
   "metadata": {},
   "source": [
    "This notebook shows how to use an LLM to evaluate the work of other LLMs.\n",
    "\n",
    "In this simple example, we will load a canned summarization dataset from Hugging Face. We will obtain summaries from two LLMs, Falcon 40B Instruct BF16 and Flan T5 XL. We will ask Anthropic's Claude model to evaluate those summaries, along with the ground truth summary."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ea04ab4-09e7-4dee-b725-5e31bb6f04e6",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "You will need an [API key](https://docs.anthropic.com/claude/docs/getting-access-to-claude) to use Claude. In the future you can use [Amazon Bedrock](https://aws.amazon.com/bedrock/) instead, as it offers Claude as a supported model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "4edc13f1-283c-43a4-9e61-4461850231cf",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "claude_api_key = '' ## Enter your API key here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1850362f-d73b-4fac-904b-fb00f621e66d",
   "metadata": {
    "tags": []
   },
   "source": [
    "You will also need to install several Python modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "298f4860-c7c4-4575-9590-08a50862f703",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f892d3a-4dcc-4699-a763-9f04e3b7db55",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! pip install -U anthropic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "218f5f95-aa09-4028-8342-fdebce701f63",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! pip install -U pydantic==1.10"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de6ac61a-49d6-4bbb-a2ad-1e19784ea35d",
   "metadata": {},
   "source": [
    "Finally, define the names of the SageMaker endpoints you're using for Falcon and Flan-T5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "bd36946a-d42a-4113-9575-6062a3605fad",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "t5_ep_name = '' # Enter the name of the SageMaker inference endpoint you deployed with Flan-T5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "922690e3-e1e5-46d8-b7f3-aa8006ca5be4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "falcon_ep_name = '' # Enter the name of the SageMaker inference endpoint you deployed with Falcon 40B"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cae7a3d-ce06-4655-a203-8f6e13aa15b7",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Dataset\n",
    "\n",
    "We'll use the cnn_dailymail dataset. We'll only process five samples to save time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "57890bee-eaee-41a4-bb11-455475df0e6f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.8/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.24.3\n",
      "  warnings.warn(f\"A NumPy version >={np_minversion} and <{np_maxversion}\"\n",
      "Found cached dataset cnn_dailymail (/root/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0/1b3c71476f6d152c31c1730e83ccb08bcf23e348233f4fcc11e182248e6bf7de)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "29e891a9f3d148beb24e9d36b0e4cbb5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "dataset = load_dataset('cnn_dailymail', '3.0.0')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0bc6de13-a24f-40fb-bb1f-a72b17cd3000",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['article', 'highlights', 'id'],\n",
       "        num_rows: 287113\n",
       "    })\n",
       "    validation: Dataset({\n",
       "        features: ['article', 'highlights', 'id'],\n",
       "        num_rows: 13368\n",
       "    })\n",
       "    test: Dataset({\n",
       "        features: ['article', 'highlights', 'id'],\n",
       "        num_rows: 11490\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "ba4ce648-d68e-487f-b317-48ca451cee8e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'LONDON, England (Reuters) -- Harry Potter star Daniel Radcliffe gains access to a reported £20 million ($41.1 million) fortune as he turns 18 on Monday, but he insists the money won\\'t cast a spell on him. Daniel Radcliffe as Harry Potter in \"Harry Potter and the Order of the Phoenix\" To the disappointment of gossip columnists around the world, the young actor says he has no plans to fritter his cash away on fast cars, drink and celebrity parties. \"I don\\'t plan to be one of those people who, as soon as they turn 18, suddenly buy themselves a massive sports car collection or something similar,\" he told an Australian interviewer earlier this month. \"I don\\'t think I\\'ll be particularly extravagant. \"The things I like buying are things that cost about 10 pounds -- books and CDs and DVDs.\" At 18, Radcliffe will be able to gamble in a casino, buy a drink in a pub or see the horror film \"Hostel: Part II,\" currently six places below his number one movie on the UK box office chart. Details of how he\\'ll mark his landmark birthday are under wraps. His agent and publicist had no comment on his plans. \"I\\'ll definitely have some sort of party,\" he said in an interview. \"Hopefully none of you will be reading about it.\" Radcliffe\\'s earnings from the first five Potter films have been held in a trust fund which he has not been able to touch. Despite his growing fame and riches, the actor says he is keeping his feet firmly on the ground. \"People are always looking to say \\'kid star goes off the rails,\\'\" he told reporters last month. \"But I try very hard not to go that way because it would be too easy for them.\" His latest outing as the boy wizard in \"Harry Potter and the Order of the Phoenix\" is breaking records on both sides of the Atlantic and he will reprise the role in the last two films.  Watch I-Reporter give her review of Potter\\'s latest » . There is life beyond Potter, however. The Londoner has filmed a TV movie called \"My Boy Jack,\" about author Rudyard Kipling and his son, due for release later this year. He will also appear in \"December Boys,\" an Australian film about four boys who escape an orphanage. Earlier this year, he made his stage debut playing a tortured teenager in Peter Shaffer\\'s \"Equus.\" Meanwhile, he is braced for even closer media scrutiny now that he\\'s legally an adult: \"I just think I\\'m going to be more sort of fair game,\" he told Reuters. E-mail to a friend . Copyright 2007 Reuters. All rights reserved.This material may not be published, broadcast, rewritten, or redistributed.'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset['train']['article'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8bb890e9-5b7b-4602-a39c-c66f58620293",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Harry Potter star Daniel Radcliffe gets £20M fortune as he turns 18 Monday .\\nYoung actor says he has no plans to fritter his cash away .\\nRadcliffe's earnings from first five Potter films have been held in trust fund .\""
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset['train']['highlights'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "1d366892-6b94-432e-8291-9dfa03ed779b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([143884,  60192,  54750, 117176, 138558])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "num_to_eval = 5\n",
    "eval_idxs = np.random.randint(low=0, high=len(dataset['train']), size=num_to_eval)\n",
    "eval_idxs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "d5bda61f-ed8a-4e2c-bd8e-92976095919b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "docs_to_summarize = [dataset['train']['article'][i] for i in eval_idxs]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "2376fd18-c725-47f6-865c-8e8f0cf0e381",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(docs_to_summarize)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "492b972a-b408-4bae-8c1c-c21aa24ac42e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "docs_gt = [dataset['train']['highlights'][i] for i in eval_idxs]# ## Get summaries "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a741a9f5-051b-41cd-bcd5-75f22e6f046a",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Get summaries "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e23632f-2efa-46d8-b533-4b33027eda62",
   "metadata": {},
   "source": [
    "### Flan-T5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c2c7d02d-6dce-4379-b9f0-cc0c4a95fe9d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "c4d6348e-5f2b-4b2e-a754-3af0df8cdb4e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.serializers import JSONSerializer\n",
    "import sagemaker\n",
    "\n",
    "t5_ep = t5_ep_name\n",
    "t5_predictor = sagemaker.predictor.Predictor(t5_ep)\n",
    "t5_predictor.serializer = JSONSerializer()\n",
    "t5_predictor.content_type = \"application/json\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "3366dabc-43e2-45ea-ac9b-0ab124f8cd23",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "def query_t5(predictor, doc):\n",
    "    payload = {\n",
    "        \"text_inputs\": f\"Write a short summary for this text: \\\"{doc}\\\"\",\n",
    "        \"max_length\": 5000,\n",
    "        \"do_sample\": True,\n",
    "        \"top_k\": 10,\n",
    "    }\n",
    "    response = json.loads(predictor.predict(payload))\n",
    "    return response['generated_texts'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "d57eb662-c6ea-45e5-a85d-fbcba94c56ef",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "t5_sums = []\n",
    "for doc in docs_to_summarize:\n",
    "    response = query_t5(t5_predictor, doc)\n",
    "    t5_sums.append(response)\n",
    "    time.sleep(2)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "bf52736b-6efc-44b8-b21c-046d81993d4f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Jury selection will begin in the trial of Andre Leteve, who is accused of gunning down his two boys in 2010. He is accused of doing it as an act of vengeance against his wife Laurie, with whom he was locked in a bitter divorce battle.'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t5_sums[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca1ea9f6-eefc-4dbb-a145-b8ceb6d678e6",
   "metadata": {},
   "source": [
    "### Falcon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "5f150f87-1129-4f48-9797-7430d8d3029a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.serializers import JSONSerializer\n",
    "import sagemaker\n",
    "\n",
    "falcon_ep = falcon_ep_name\n",
    "falcon_predictor = sagemaker.predictor.Predictor(falcon_ep)\n",
    "falcon_predictor.serializer = JSONSerializer()\n",
    "falcon_predictor.content_type = \"application/json\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "4049fa31-c0c4-4bdc-bfad-39a0dc5d2060",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "def query_falcon(predictor, doc):\n",
    "    payload = {\n",
    "        \"inputs\": f\"\\\"{doc[:950]}\\\". Summarize the article above:\",\n",
    "        \"max_new_tokens\": 300,\n",
    "        \"return_full_text\": False\n",
    "    }\n",
    "    response = json.loads(predictor.predict(payload))\n",
    "    return response[0]['generated_text']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "f594b27b-e42d-4b21-838d-94f5439a65c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "falcon_sums = []\n",
    "for doc in docs_to_summarize:\n",
    "    response = query_falcon(falcon_predictor, doc)\n",
    "    falcon_sums.append(response)\n",
    "    time.sleep(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "93c79085-b54e-461e-8440-a57246d6cebb",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\nThe article above is about an Arizona man named Andre Leteve who is accused of murdering his'"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "falcon_sums[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa341ff9-3615-482c-af71-c47b5f8be003",
   "metadata": {},
   "source": [
    "## Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "8db497fb-96ee-450c-b0af-79398d21ba54",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from anthropic import Anthropic, HUMAN_PROMPT, AI_PROMPT\n",
    "\n",
    "anthropic = Anthropic(api_key=claude_api_key)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "46ef5253-30cb-4e08-aed8-eaa9e6e002ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt_eng_base = '''You will be given a summmary of a news article. Your task is to evaluate the summary in four dimensions, accuracy, coherence, factuality, and completeness. Provide a score of 1-5 in each dimension, with 5 being the best score.\n",
    "\n",
    "Original discussion: [DISCUSSION]\n",
    "\n",
    "Summary: [SUMMARY]\n",
    "\n",
    "Evaluation form (scores only):\n",
    "\n",
    "- Coherence: \n",
    "- Accuracy:\n",
    "- Factuality:\n",
    "- Completeness:\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "fc6f1d57-ab20-437b-9c66-c14a053b0468",
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_prompt(search, context, prompt_eng_base):\n",
    "    search = search.replace(\"\\\"\", \"'\")\n",
    "    context = context.replace(\"\\\"\", \"'\")\n",
    "    prompt = prompt_eng_base.replace('[DISCUSSION]', context)\n",
    "    prompt = prompt.replace('[SUMMARY]', search)\n",
    "    return prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "27b46b4d-c3c9-4287-ae79-cb95e24faea2",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_eval(prompt):\n",
    "    completion = anthropic.completions.create(\n",
    "        model=\"claude-1\",\n",
    "        max_tokens_to_sample=300,\n",
    "        prompt=f\"{HUMAN_PROMPT} {prompt} {AI_PROMPT}\",\n",
    "    )\n",
    "    return completion.completion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50b23f9a-de5f-4cb1-b1d4-c45f82188ef1",
   "metadata": {},
   "source": [
    "Example output:\n",
    "    \n",
    "    \"\\nSummary Scores:\\n\\nCoherence: 5\\nAccuracy: 5 \\nFactuality: 5\\nCompleteness: 2\\n\\nComments: The summary is coherent, accurate and factual in reporting the key details about Daniel Radcliffe gaining access to his trust fund as he turns 18, and his stated plans to not spend extravagantly. However, the summary lacks details on Radcliffe's acting work beyond the Harry Potter series, his upcoming films My Boy Jack and December Boys, as well as his stage debut in Equus. The summary is incomplete in covering the breadth of topics discussed in the original report. Overall, this is a decent high-level summary but could be more complete.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "46225833-2899-43b6-83e3-bcd3cecce5c1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import re\n",
    "def parse_result(r):\n",
    "    m = re.search(\"Accuracy: (\\d)\", r)\n",
    "    if m is None:\n",
    "        accuracy = 0\n",
    "    else:\n",
    "        accuracy = m.group(1)\n",
    "        \n",
    "    m = re.search(\"Coherence: (\\d)\", r)\n",
    "    if m is None:\n",
    "        coherence = 0\n",
    "    else:\n",
    "        coherence = m.group(1)\n",
    "        \n",
    "    m = re.search(\"Factuality: (\\d)\", r)\n",
    "    if m is None:\n",
    "        factuality = 0\n",
    "    else:\n",
    "        factuality = m.group(1)\n",
    "        \n",
    "    m = re.search(\"Completeness: (\\d)\", r)\n",
    "    if m is None:\n",
    "        completeness = 0\n",
    "    else:\n",
    "        completeness = m.group(1)\n",
    "    \n",
    "    return accuracy, coherence, factuality, completeness"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "7733355c-cc47-455c-bc2c-3317ee38267e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "result_map = []\n",
    "for doc, t5_sum, falcon_sum, gt in zip(docs_to_summarize, t5_sums, falcon_sums, docs_gt):\n",
    "    p_t5 = make_prompt(t5_sum, doc, prompt_eng_base)\n",
    "    p_falcon = make_prompt(falcon_sum, doc, prompt_eng_base)\n",
    "    p_gt = make_prompt(gt, doc, prompt_eng_base)\n",
    "    \n",
    "    r_t5 = get_eval(p_t5)\n",
    "    time.sleep(5)\n",
    "    r_falcon = get_eval(p_falcon)\n",
    "    time.sleep(5)\n",
    "    r_gt = get_eval(p_gt)\n",
    "    time.sleep(5)\n",
    "    \n",
    "    metrics_t5 = parse_result(r_t5)\n",
    "    metrics_falcon = parse_result(r_falcon)\n",
    "    metrics_gt = parse_result(r_gt)\n",
    "    \n",
    "    result_map.append({\n",
    "        'doc': doc,\n",
    "        't5': {\n",
    "            'summary': t5_sum,\n",
    "            'eval': r_t5,\n",
    "            'accuracy': metrics_t5[0],\n",
    "            'coherence': metrics_t5[1],\n",
    "            'factuality': metrics_t5[2],\n",
    "            'completeness': metrics_t5[3],\n",
    "        },\n",
    "        'falcon': {\n",
    "            'summary': falcon_sum,\n",
    "            'eval': r_falcon,\n",
    "            'accuracy': metrics_falcon[0],\n",
    "            'coherence': metrics_falcon[1],\n",
    "            'factuality': metrics_falcon[2],\n",
    "            'completeness': metrics_falcon[3],\n",
    "        },\n",
    "        'gt': {\n",
    "            'summary': gt,\n",
    "            'eval': r_gt,\n",
    "            'accuracy': metrics_gt[0],\n",
    "            'coherence': metrics_gt[1],\n",
    "            'factuality': metrics_gt[2],\n",
    "            'completeness': metrics_gt[3],\n",
    "        }\n",
    "        \n",
    "    })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "b1b138f4-62e9-4337-86a0-e00e35a74164",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'doc': \"By . Daily Mail Reporter . PUBLISHED: . 12:03 EST, 10 September 2012 . | . UPDATED: . 13:03 EST, 10 September 2012 . Mug shot: Andre Leteve suffered facial injuries after trying to kill himself following the murder of his sons . An Arizona man who stands accused of murdering his two sons at the height of a bitter divorce battle with his wife will stand trial beginning this week. Jury selection is set to begin in the trial of Andre Leteve, who is accused of gunning down his two young sons Alec, 5, and Asher, 1, in Scottsdale on March 31, 2010. Prosecutors are out to prove that Leteve, who was 39 at the time, shot his sons as an act of vengeance against his estranged wife Laurie, with whom he was locked in a bitter divorce battle. Their marriage reportedly soured after she discovered that he had cheated on her with prostitutes. A court document obtained by The Arizona Republic read: 'The state claims one of the motives for killing the sons was revenge because Laurie, [his] wife and the children's mother, left him after finding out that [he] had sex with multiple prostitutes.' The document went on: 'Laurie was dating someone else at the time of the shootings. The other motive stems from the defendant's poor finances at the time of the shooting.' Leteve reportedly has difficulty speaking after he tried to kill himself. Cops said that moments after killing his Alec and Asher, Leteve stuck the gun under his chin and pulled the trigger, but the bullet went through his nose and he survived. Scroll down for video . Shattered: Prosecutors are out to prove that Leteve shot his sons as an act of vengeance against his wife Laurie, pictured, with whom he was locked in a bitter divorce battle . In a 911 call made moments after the shootings, Leteve can be heard saying: 'I just shot and killed my sons and shot myself.' Leteve was taken to a hospital for treatment of his facial injuries, and taken into custody after his release. KTVK reported that Leteve told detectives that he killed the boys because their mother was preparing to move them with her to Florida. Relatives . told the network that the divorce was not a messy one, and that there . were no indications that Leteve would commit such an act. Murdered: Leteve is accused of gunning down his two young sons Alec, 5, left, and Asher, 1, right, in Scottsdale, Arizona, on March 31, 2010 . Defense attorney Greg Parzych told The Republic: 'We don't feel comfortable talking about our theories of the case on the brink of jury selection, other than to say that for the past two years, we have been working diligently and thoroughly to ensure that Mr Leteve gets the best possible trial he can.' 100 witnesses are reportedly expected to testify. Opening statements are slated to begin on October 9. If convicted, Leteve could face the death penalty. Watch video here .\",\n",
       " 't5': {'summary': 'Jury selection will begin in the trial of Andre Leteve, who is accused of gunning down his two boys in 2010. He is accused of doing it as an act of vengeance against his wife Laurie, with whom he was locked in a bitter divorce battle.',\n",
       "  'eval': ' Here are the scores for the summary:\\n\\nCoherence: 5\\nAccuracy: 5 \\nFactuality: 5\\nCompleteness: 3',\n",
       "  'accuracy': '5',\n",
       "  'coherence': '5',\n",
       "  'factuality': '5',\n",
       "  'completeness': '3'},\n",
       " 'falcon': {'summary': '\\nThe article above is about an Arizona man named Andre Leteve who is accused of murdering his',\n",
       "  'eval': ' Here are the scores for the summary:\\n\\nCoherence: 5\\nAccuracy: 4  \\nFactuality: 5\\nCompleteness: 2',\n",
       "  'accuracy': '4',\n",
       "  'coherence': '5',\n",
       "  'factuality': '5',\n",
       "  'completeness': '2'},\n",
       " 'gt': {'summary': 'Andre Leteve charged with murdering sons Alec, 5, and Asher, 1, on March 31, 2010 .\\nLeteve attempted suicide after the murders but survived .\\nProsecutors to argue that Leteve shots his sons as an act of revenge against estranged wife .',\n",
       "  'eval': ' Here are the scores I would assign to the summary:\\n\\n- Coherence: 4\\n- Accuracy: 5 \\n- Factuality: 4\\n- Completeness: 3',\n",
       "  'accuracy': '5',\n",
       "  'coherence': '4',\n",
       "  'factuality': '4',\n",
       "  'completeness': '3'}}"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result_map[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "108c1e95-ef63-499e-96d6-56d5287ef313",
   "metadata": {},
   "source": [
    "## Results\n",
    "\n",
    "*Note*: Do not take these results as a comprehensive model evaluation. The purpose of this notebook is to illustrate the evaluation technique. We did not spend any time on model tuning or prompt engineering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "e173fd58-ea90-4ef0-9889-71e4c5e255d2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "4cf60863-1e82-47f4-a9cf-c5051c0ddbd0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "result_values = []\n",
    "result_types = []\n",
    "result_models = []\n",
    "result_docs = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "cd70e840-f4a2-4b95-ada0-86492b77548f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "for r in result_map:\n",
    "    for m in ['t5', 'falcon', 'gt']:\n",
    "        for t in ['accuracy', 'coherence', 'factuality', 'completeness']:\n",
    "            result_docs.append(r['doc'])\n",
    "            result_values.append(r[m][t])\n",
    "            result_types.append(t)\n",
    "            result_models.append(m)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "b4a43e69-6196-44b5-ae41-46c7f25145d5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "df = pd.DataFrame({'doc': result_docs, 'value': result_values, 'type': result_types, 'model': result_models})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "9aec6f63-a568-45b1-87bd-e998a858361d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>doc</th>\n",
       "      <th>value</th>\n",
       "      <th>type</th>\n",
       "      <th>model</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>By . Daily Mail Reporter . PUBLISHED: . 12:03 ...</td>\n",
       "      <td>5</td>\n",
       "      <td>accuracy</td>\n",
       "      <td>t5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>By . Daily Mail Reporter . PUBLISHED: . 12:03 ...</td>\n",
       "      <td>5</td>\n",
       "      <td>coherence</td>\n",
       "      <td>t5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>By . Daily Mail Reporter . PUBLISHED: . 12:03 ...</td>\n",
       "      <td>5</td>\n",
       "      <td>factuality</td>\n",
       "      <td>t5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>By . Daily Mail Reporter . PUBLISHED: . 12:03 ...</td>\n",
       "      <td>3</td>\n",
       "      <td>completeness</td>\n",
       "      <td>t5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>By . Daily Mail Reporter . PUBLISHED: . 12:03 ...</td>\n",
       "      <td>4</td>\n",
       "      <td>accuracy</td>\n",
       "      <td>falcon</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 doc value          type  \\\n",
       "0  By . Daily Mail Reporter . PUBLISHED: . 12:03 ...     5      accuracy   \n",
       "1  By . Daily Mail Reporter . PUBLISHED: . 12:03 ...     5     coherence   \n",
       "2  By . Daily Mail Reporter . PUBLISHED: . 12:03 ...     5    factuality   \n",
       "3  By . Daily Mail Reporter . PUBLISHED: . 12:03 ...     3  completeness   \n",
       "4  By . Daily Mail Reporter . PUBLISHED: . 12:03 ...     4      accuracy   \n",
       "\n",
       "    model  \n",
       "0      t5  \n",
       "1      t5  \n",
       "2      t5  \n",
       "3      t5  \n",
       "4  falcon  "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "c05c2124-adbf-453e-9110-d09af6909c60",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df['value'] = df['value'].astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "5a865da3-7006-413b-b06c-d3cd00d88e83",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='type', ylabel='value'>"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6wAAAHlCAYAAADx83HJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAyZklEQVR4nO3deYBVdf0//ucMsyCggIA7irkQuXwUwdzSRMsV0Swzs/yYZtrn+5EyIxXLHUFtcUsUzT72cylzQbBVzTTNErFSMzUh1FJBRFkEZmDO7w+/zjc+goLO3HuYeTz+OXPvPee8XucO73vmyVluTVEURQAAAKBkaqvdAAAAACyPwAoAAEApCawAAACUksAKAABAKQmsAAAAlJLACgAAQCnVVbuBlTVnzoK0tPgGHgAAgI6ktrYmvXt3X+5rq01gbWkpBFYAAIBOxCnBAAAAlJLACgAAQCkJrAAAAJTSanMNKwAAQJktXbokc+bMypIlTdVupZTq6hrSu3e/dOmy8jFUYAUAAGgDc+bMSteu3dK9+3qpqampdjulUhRFFiyYmzlzZqVv3/VXejmnBAMAALSBJUua0r37WsLqctTU1KR797VW+eizwAoAANBGhNUVey/vjcAKAABAKQmsAAAAncR5552Zq676/krN+8lPDs/DD/+hnTt6ZwIrAAAApSSwAgAAUEq+1gYAAKBkPvnJ4fnEJz6VX/7yZ/nnP1/IXnt9PF/60n/lvPPOyl/+8qd86ENb55xzxmattdbK737324wff3leeWVmNt98y5x88qkZMGDTJMnTT/8tY8eek+effz4777xr/vd9jx544P5MmHBFXnrpXxkw4AM5+eRTs/nmW1Rhi5fPEVYAAIASuvfee/Ld716eG2+8NQ88cH9OPvnEfOlLX86dd96VomjJT396U557bkbOPHN0Ro48KZMn35Wdd9413/jGV9Pc3Jzm5uaceurJ2Wef/fPzn9+TPffcK/fee0/r+p966m85//yz8/Wvn5Y777w7I0Z8IqecclKamlbtq2fak8AKAABQQp/85Kez9tp90q/fOvmP/9guH/rQ1tlyyw+moaEhu+/+0TzzzFO5555fZ+edd8vQoTulrq4un/nM57J48eI89tif88QTj2XJkiU57LAjUldXlz333DuDBm3Vuv5Jk27LiBGfyFZbbZ0uXbpkv/0OTH19fZ544rEqbvWynBIMAABQQr17r936c2Nj17c9fuONhXnllVlZb731W5+vra3NOuusm1demZXa2tr067fOMt9/uu6667X+/NJLL+bnP5+cW275cetzzc3NeeWVWe21SausYoF12LBhaWhoSGNjY5Lk5JNPzkc+8pFKlQcAAOhw+vbtl2ef/Xvr46IoMnPmy+nbt19qamoya9bMFEXRGlpnznwpG264UZJknXXWzec//4UcddQxVel9ZVT0lOBLLrkkEydOzMSJE4VVAACA92nYsL3z+9//LlOm/DFLlizJjTf+f6mvb8g22/xHtt5623Tp0iU333xTlixZkt/+9p789a9PtC570EGHZOLEW/PEE4+nKIosXLgwDz74u7zxxoIqbtGynBJcYVOnTsmkSbdl+PBDMnjwkGq3A3RQPmsAoHPYeOMB+eY3z8n3vndhZs2amS22GJhx476T+vr6JMmYMRdm3LhzM2HCFdl5512zxx7DWpf94Ac/lFGjRue7370gL7zwXBobG7PNNttlu+22r9bmvE1NURRFJQoNGzYsPXr0SFEU2WGHHXLSSSdlrbXWqkTpUjn++OPzzDPPZIsttsj48eOr3Q7QQfmsAYDKe+KJv2aDDTapdhul9q9/zchWW31opeev2BHW66+/Puuvv36amppy3nnn5eyzz85FF1200svPnj0/LS0Vydbtat68Ba3TWbPmVbkboKPyWQMAldfS0pIlS1qq3UaptbS0vO1vk9ramvTp02O581fsGtb113/zzlUNDQ054ogjMnXq1EqVBgAAYDVUkcD6xhtvZN68N1N0URT52c9+lkGDBlWiNAAAAKupipwSPHv27Pz3f/93li5dmpaWlmy22WY544wzKlEaAACA1VRFAmv//v1z++23V6IUAAAAHURFv4cVAAAAVpbACgAAQClV7GttAAAAOpM11+qaro31bb7eRYubM2/uojZfbxkJrAAAAO2ga2N9jhh1fZuv94YLPpt5effAes01V+bzn/9C6uvrc801V+a2236avn37JUm22eY/8rWvfaPNe2trAisAAEAHdO21E/KZz3wu9fVvHuXdd98D8n/+z1eq29QqElgBAAA6mG9/e1yS5IQTvpCamtpsv/0O6dKlS5W7WnVuugQAANDBvHW67xVX/CA//OEN6dGjR+6++1c56qjD89Wv/lcef/wvVe5w5TjCCgAA0MEdfPChOeqoY1JXV5eHH34op5zytVx//c3p2bNXtVt7R46wAgAAdHB9+vRNXd2bxyuHDt0p66yzbqZNe7bKXb07gRUAAKAD6tatexYsmJ8kmTVrZuvzzzzzVF566cVsvPEm1WptpTklGOgUpk6dkkmTbsvw4Ydk8OAh1W6HNuR3W26d7ffT2bYXeGeLFjfnhgs+2y7rXRmHH/7ZnHji8Wls7Jr11lsv//znC6mt7ZL6+vp885tnpU+fvm3eW1sTWIFO4eabb8j06dOyaNFCf0R2MH635dbZfj+dbXuBdzZv7qKV+r7U9vKFLxyXL3zhuKrVbwtOCQY6hYULFy0zpePwuy23zvb76WzbC9DeBFYAAABKSWAFAACglARWAAAASklgBQAAoJQEVgAAAErJ19oAAAC0g949G1LX0Njm613StDhzXm961/nuu+/eXHnlZWloaMhZZ43JxhsPeNs8U6dOyeWXX5xrrvlRm/fZFgRWAACAdlDX0JhHLji2zde7w6irk7x7YJ048dYcc8zxGTZs7zbvoVIEVgAAgA7mkku+nb/85dE899yM3Hbbzenbt1+ee25GmpubsuGG/XPqqd/KWmut9bblHnjg/vzgB1dlyZIlqa2tyejRZ2XzzbfIQw89mCuvvCwtLS3p1at3vv7107LRRv0zdeqUXHLJd/KhD22VJ554LElNzjprTAYM2LRNtkNgBQAA6GBOPPFrefrpp/KZz3wuu+76kbz22mvp1atXkuSqq76f66//n5xwwn8vs8xzz83IuHHn5vLLJ6R//43T1NSUJUuaM2fOqzn33G/l0kuvyqabfiCTJ9+es846PRMm/E+SZPr0Z3Paad/KqFGj8z//c03+53+uyRlnnNsm2yGwAgAAdHC/+MXk/OpXv8iSJc1ZuHBR+vff+G3zPPzwH7LTTru0vtbQ0JCGhoZMnfpINttsy2y66QeSJPvvf1C+/e1xeeONBUmSjTfeJFtu+cEkyVZbbZMHHri/zfoWWAEAADqwP//50dx++y254oofpHfv3vnVr36RO+64dTlzFitYQ5GamhWvv+HfbixVW1ubpUuXvq9+/52vtQEAAOjA5s2bl+7de6Rnz55pamrKnXfesdz5dtxx5zz00IN5/vnnkiRNTU15440F2WqrbfP3vz+dGTP+kST5+c8nZ4stBqZbt+7t3rsjrAAAAO1gSdPi/3tH37Zf76rYaadd8qtf/TxHHPHJrLPOOvngBwflr3994m3z9e+/cUaNGp0zzjg1S5e2pEuX2owefVY222zznH762TnrrNFZunRpevXqnW9965y22px3JLACALDKpk6dkkmTbsvw4Ydk8OAh1W4HSunN70p996+faS+XXXZV689nn33+cucZPHjIMt/Buttuu2e33XZ/23w77bRLdtppl3dd/n8/fr8EVgAAVtnNN9+Q6dOnZdGihQIr0G5cwwoAwCpbuHDRMlOA9iCwAgAAUEoCKwAAAKUksAIAAFBKAisAAACl5C7BAAAA7WCtno1pbGho8/UubmrK3NdX7btYV1cCKwAAQDtobGjIf147ss3X+8OjL07y3gPrNddcmc9//gupr69vu6baiVOCAQAAOpFrr52Q5ubmarexUhxhBQAA6IDuvffuXHXV99PY2Jg999w7V131/RxwwEFJkhNO+EJqampz6aVXZs0116xypysmsAIAAHQwc+a8mgsuGJMrr7w2/ftvnB//+PokyciRJ+fOO+/IFVf8IN26datyl+/OKcEAAAAdzBNPPJYttxyY/v03TpIccMCIKnf03gisAAAAHUxRFKmpqal2G++bwAoAANDBbLXVNnnqqb/lhReeT5L87GeTWl/r1q17FiyYX63WVolrWAEAANrB4qam//sVNG2/3nez9tp9cvLJp+brXx+ZXr16ZZdddk9dXV26du2aww//bE488fg0NnZ10yUAAIDOaO7ri/N+vi/1/dppp50zbNjeSZI777wjgwZtldra2nzhC8flC184rmp9rQqBFdrI1KlTMmnSbRk+/JAMHjyk2u0AANDJ3XzzTfnNb+7O0qVLstZaPfONb5xe7ZZWWacOrGuu1TVdG+srWrNLl5rWab9+lT30vmhxc+bNXVTRmp3JzTffkOnTp2XRooUCK8vo3bMhdQ2NFa1Zrc+apiVNaahrqFi9pLqfq4ubmv7v/54DQPkcddQxOeqoY6rdxvvSqQNr18b6HDHq+orWfOWVeUmSl16ZV/HaN1zw2cyLwNpeFi5ctMwU3lLX0JhHLji2ojUXz3m5dVrJ2juMujr/ee3IitVLkpfnzmqdVrr2m9clCawA/D8d5e687aEoilVexl2CAQAA2kBdXUMWLJj7noJZR1cURRYsmJu6VTwTq1MfYQUAAGgrvXv3y5w5szJ//mvVbqWU6uoa0rt3v1Vbpp16AQAA6FS6dKlL377rV7uNDsUpwQAAAJSSwAoAAEApCawAAACUksAKAABAKQmsAAAAlJLACgAAQCkJrAAAAJSSwAoAAEApCawAAACUksAKAABAKQmsAAAAlFJdtRsAOp811+qaro31Fa3ZpUtN67RfvzUrWhvKorONvUWLmzNv7qKK1gSgbQmsQMV1bazPEaOur2jNV16ZlyR56ZV5Fa99wwWfrWg9WJHOOPbmRWAFWJ05JRgAAIBSElgBAAAoJYEVAACAUhJYAQAAKCWBFQAAgFISWAEAACglgRUAAIBSElgBAAAoJYEVAACAUhJYAQAAKCWBFQAAgFISWAEAACiligfWyy67LAMHDszTTz9d6dIAAACsRioaWJ944on86U9/ygYbbFDJsgAAAKyGKhZYm5qacvbZZ+eMM85ITU1NpcpSRVOnTslZZ43O1KlTqt0KAACwGqqrVKGLL744Bx10UPr37/+elu/Tp0cbd9Q59eu3ZsVq3Xbbj/PMM89kyZKm7LPPnhWrWy1dutS0Tiv5PkNnV1Nfu8y00oz38mpZ0lzx30+19gVNS5rSUNdQsXpJdfd7TUua01BXX9GaQHVUJLA++uijeeyxx3LyySe/53XMnj0/LS1FG3bVOf/ImDVrXsVqzZu3oHVaybrVsnRp0TrtDNv7fnTGsUf76bntupn35CtZc1DfqtRfncZ7Zxt7tXX1eeSCYytac/Gcl1unlay9w6ir85/XjqxYvSR5ee6s1mmla//w6ItXq7EHvLPa2poVHqCsyH9HP/zww5k2bVr22muvDBs2LC+99FKOOeaY/O53v6tEeQA6sDU2WjPrfGzTrLFR5wpjANAZVOQI63HHHZfjjjuu9fGwYcMyfvz4bLnllpUoDwAAwGrI97ACAABQShW76dK/u+eee6pRFgAAgNWII6wAAACUksAKAABAKQmsAAAAlJLACgAAQCkJrAAAAJSSwAoAAEApCawAAACUksAKAABAKQmsAAAAlJLACgAAQCkJrAAAAJRSXbUb6GxqutQvM62kliXN6ddvzYrV69KlpnVaybpJ0rSkKQ11DRWtWc3tXdzUlLmvL65oTcqtsa52mSkAwOpIYK2wHhsMzoKXHkv39bapeO3auvo8csGxFau3eM7LrdNK1k2SHUZdnf+8dmRFa748d1brtNK1f3j0xUkEVv6fj2/eO7+d/nr22LRntVsBAHjPBNYKa+zZP409+1e7DaCDG9SvWwb161btNgAA3hfnigEAAFBKAisAAAClJLACAABQSgIrAAAApSSwAgAAUEoCKwAAAKUksAIAAFBKAisAAAClJLACAABQSgIrAAAApSSwAgAAUEoCKwAAAKUksAIAAFBKAivQKdR0qV9mClSGsQfA+yGwAp1Cjw0Gp77HeumxweBqtwKdirEHwPtRV+0GACqhsWf/NPbsX+02oNMx9gB4PxxhBQAAoJQEVgAAAEpJYAUAAKCUBFYAAABKSWAFAACglARWAAAASklgBQAAoJQEVgAAAEpJYAUAAKCUBFYAAABKSWAFAACglARWAAAASklgBQAAoJQEVgAAgNXE1KlTctZZozN16pRqt1IRddVugI6rsa52mSkA0HHU1NcuMwUq4+abb8j06dOyaNHCDB48pNrttDufMLSbj2/eOx/o3TUf37x3tVsBANpYz23XTeO63dNz23Wr3Qp0KgsXLlpm2tE5wkq7GdSvWwb161btNgCAdrDGRmtmjY3WrHYbQAfnCCsAAAClJLACAABQSgIrAAAApSSwAgAAUEoCKwAAAKUksAIAAFBKAisAAAClJLACAABQSgIrAAAApSSwAgAAUEoCKwAAAKUksAIAAFBKAisAALBamzp1Ss46a3SmTp1S7VZoY3XVbgA6ipr62mWmAHQ+jXW1y0yByrj55hsyffq0LFq0MIMHD6l2O7Qhn6bQRnpuu24a1+2entuuW+1WAKiSj2/eOx/o3TUf37x3tVuBTmXhwkXLTOk4HGGFNrLGRmtmjY3WrHYbAFTRoH7dMqhft2q3AdBhOMIKAABAKQmsAAAAlJLACgAAQCkJrAAAAJSSwAoAAEApCawAAACUksAKAABAKQmsAAAAlJLACgAAQCkJrAAAAJSSwAoAAEAp1VWq0Je//OW88MILqa2tTbdu3fLNb34zgwYNqlR5AAAAVjMVC6zjxo3LmmuumSS56667ctppp+W2226rVHkAAABWMxU7JfitsJok8+fPT01NTaVKAwBApzJ16pScddboTJ06pdqtwPtSsSOsSTJ69Og88MADKYoiV1999Sot26dPj3bqCjqGfv3WfPeZgDZn7EF1GHvv7LbbfpxnnnkmS5Y0ZZ999qx2O+2uS5ea1mlH/7fRmbY1qXBgPe+885Ikt99+ey644IJMmDBhpZedPXt+WlqKNu2nM/yC6TxmzZpX7RZWmrFHR2LsQXWsTmOvGubNW9A67Qzv1dKlReu0o29vR9zW2tqaFR6grMpdgg8++OD84Q9/yJw5c6pRHgAAgNVARQLrggUL8uKLL7Y+vueee9KzZ8/06tWrEuUBAABYDVXklOCFCxdm5MiRWbhwYWpra9OzZ8+MHz/ejZcAAABYoYoE1r59++YnP/lJJUoBAADQQVTlGlYAAAB4NwIrAAAApSSwAgAAUEoCKwAAAKUksAIAAFBKAisAAAClJLACAABQSgIrAAAApSSwAgAAUEoCKwAAAKVUV+0GAAAAVldr9WxMY0NDxep16VLTOu3Xb82K1U2SxU1Nmfv64orWXKXA2tzcnD//+c+ZOXNm9t9//7zxxhtJkm7durVLcwAAAGXW2NCQ/7x2ZMXqvTx3Vuu0knWT5IdHX5ykpIH1qaeeygknnJCGhoa8/PLL2X///fPwww/ntttuy/e+9712bBEAAIDOaKWvYT3zzDNz4okn5he/+EXq6t7MuUOHDs0jjzzSbs0BAADQea10YP373/+eESNGJElqat48b7pbt25ZvLiyh4QBAADoHFY6sG644YZ5/PHHl3nuL3/5SzbeeOM2bwoAAABW+hrWkSNH5ktf+lIOP/zwNDc358orr8xNN92Uc845pz37AwAAoJNa6SOse+65ZyZMmJBXX301Q4cOzT//+c9ceuml2W233dqzPwAAADqpVfpam6222ipbbbVVe/UCAAAArVY6sF588cUrfG3kyMp+/w8AAAAd30oH1pdeemmZx7NmzcrDDz+cvffeu82bAgAAgJUOrOeff/7bnrvvvvty5513tmlDAAAAkKzCTZeWZ7fddstdd93VVr0AAABAq5U+wvr8888v83jhwoWZPHly1l9//TZvCgAAAFY6sH7sYx9LTU1NiqJIkqyxxhoZNGhQxo4d227NAQAA0HmtdGD929/+1p59AAAAwDLe1zWsAAAA0F7e8QjrHnvskZqamnddyb333ttW/QAAQIez5lpd07WxvmL1unSpaZ3267dmxeomyaLFzZk3d1FFa9JxvWNgvfDCCyvVBwAAdFhdG+tzxKjrK1bvlVfmJUleemVeResmyQ0XfDbzIrDSNt4xsO64446V6gMAAACWsdI3XUqSJ598MlOmTMmcOXNa7xacJCNHjmzzxgAAAOjcVvqmSz/+8Y/zmc98Jg899FAmTJiQp59+Otdee22ee+659uwPAACATmqlA+vVV1+dq6++Opdffnm6du2ayy+/PBdffHHq6lbpIC0AAACslJUOrLNnz86QIUPeXKi2Ni0tLdljjz3ym9/8pt2aAwAAoPNa6cOj6623Xp5//vn0798/AwYMyN13353evXunvr5yt+cGAACg81jpwHrsscdm2rRp6d+/f7785S9n5MiRaW5uzujRo9uzPwAAADqplQ6sTz75ZIYPH54k2WOPPfLHP/4xzc3N6d69e7s1BwAAQOe1SndM+vKXv5xu3brlwAMPzPDhw7Ppppu2V18AAAB0cit906XRo0fnvvvuyxlnnJEXX3wxhx12WD7xiU/k2muvbc/+AAAA6KRWOrAmb94deNddd83555+fyZMnp1evXrngggvaqzcAAAA6sVU6JXjBggW56667cuedd+aPf/xjhg4dmrFjx7ZXbwAAAHRiKx1YTzzxxNx///350Ic+lAMOOCBjx47N2muv3Z69AQAA0ImtdGDdeuutc8opp2SDDTZoz34AAAAgySoE1uOOO649+wAAADqAliXN6ddvzYrW7NKlpnVa6dq0r1W6hhUAAOCd1NbV55ELjq1ozcVzXm6dVrr2DqOurmi9zmaV7hIMAAAAlSKwAgAAUEoCKwAAAKUksAIAAFBKAisAAAClJLACAABQSgIrAAAApSSwAgAAUEoCKwAAAKUksAIAAFBKAisAAAClJLACAABQSgIrAAAApSSwAgAAUEoCKwAAAKUksAIAQAdT06V+mSkdR0197TLTjq5zbCUAAHQiPTYYnPoe66XHBoOr3QptrOe266Zx3e7pue261W6lIuqq3QAAANC2Gnv2T2PP/tVug3awxkZrZo2N1qx2GxXjCCsAAAClJLACAABQSgIrAAAApSSwAgAAUEoCKwAAAKUksAIAAFBKAisAAAClJLACAABQSgIrAAAApSSwAgAAUEp1lSgyZ86cjBo1Ks8991waGhqyySab5Oyzz87aa69difIAAACshipyhLWmpibHHntsfvnLX2bSpEnp379/LrrookqUBgAAYDVVkcDaq1evfPjDH259vN122+Vf//pXJUoDAACwmqr4NawtLS258cYbM2zYsEqXBgAAYDVSkWtY/90555yTbt265cgjj1yl5fr06dFOHUHH0K/fmtVuATolYw+qw9jj3zXW1S4zpf1UeuxVNLCOGzcuM2bMyPjx41Nbu2r/mGbPnp+WlqJN+/FBR0cya9a8arew0ow9OhJjD6pjdRp7ifHX3j6+ee/8dvrr2WPTntVupcNrj7FXW1uzwgOUFQus3/3ud/P444/nqquuSkNDQ6XKAgAAHdygft0yqF+3ardBO6hIYH3mmWcyfvz4DBgwIIcffniSZKONNsrll19eifIAAACshioSWLfYYos89dRTlSgFAABAB+GqZAAAAEpJYAUAAKCUBFYAAABKSWAFAACglARWAAAASklgBQAAoJQEVgAAAEpJYAUAAKCUBFYAAABKSWAFAACglARWAAAASklgBQAAoJQEVgAAAEpJYAUAAKCUBFYAAABKSWAFAACglARWAAAASklgBQAAoJQEVgAAAEpJYAUAAKCUBFYAAABKSWAFAACglARWAAAASklgBQAAoJQEVgAAAEpJYAUAAKCUBFYAAABKSWAFAACglARWAAAASklgBQAAoJQEVgAAAEpJYAUAAKCUBFYAAABKSWAFAACglARWAAAASklgBQAAoJQEVgAAAEpJYAUAAKCUBFYAAABKSWAFAACglARWAAAASklgBQAAoJQEVgAAAEpJYAUAAKCUBFYAAABKSWAFAACglARWAAAASklgBQAAoJQEVgAAAEpJYAUAAKCUBFYAAABKSWAFAACglARWAAAASklgBQAAoJQEVgAAAEpJYAUAAKCUBFYAAABKSWAFAACglARWAAAASklgBQAAoJQEVgAAAEpJYAUAAKCUBFYAAABKSWAFAACglARWAAAASklgBQAAoJQEVgAAAEpJYAUAAKCUBFYAAABKSWAFAACglARWAAAASklgBQAAoJQEVgAAAEpJYAUAAKCUKhJYx40bl2HDhmXgwIF5+umnK1ESAACA1VxFAutee+2V66+/PhtuuGElygEAANAB1FWiyJAhQypRBgAAgA6kIoG1LfTp06PaLUCp9eu3ZrVbgE7J2IPqMPagOio99labwDp79vy0tBRtuk4fdHQks2bNq3YLK83YoyMx9qA6Vqexlxh/dBztMfZqa2tWeIDSXYIBAAAoJYEVAACAUqpIYD333HOz++6756WXXsrRRx+dAw44oBJlAQAAWI1V5BrW008/PaeffnolSgEAANBBOCUYAACAUhJYAQAAKCWBFQAAgFISWAEAACglgRUAAIBSElgBAAAoJYEVAACAUhJYAQAAKCWBFQAAgFISWAEAACglgRUAAIBSElgBAAAoJYEVAACAUhJYAQAAKCWBFQAAgFISWAEAACglgRUAAIBSElgBAAAoJYEVAACAUhJYAQAAKCWBFQAAgFISWAEAACglgRUAAIBSElgBAAAoJYEVAACAUhJYAQAAKCWBFQAAgFISWAEAACglgRUAAIBSElgBAAAoJYEVAACAUhJYAQAAKCWBFQAAgFISWAEAACglgRUAAIBSElgBAAAoJYEVAACAUhJYAQAAKCWBFQAAgFISWAEAACglgRUAAIBSElgBAAAoJYEVAACAUhJYAQAAKCWBFQAAgFISWAEAACglgRUAAIBSElgBAAAoJYEVAACAUhJYAQAAKCWBFQAAgFISWAEAACglgRUAAIBSElgBAAAoJYEVAACAUhJYAQAAKCWBFQAAgFISWAEAACglgRUAAIBSElgBAAAoJYEVAACAUhJYAQAAKCWBFQAAgFISWAEAACglgRUAAIBSElgBAAAoJYEVAACAUhJYAQAAKCWBFQAAgFISWAEAACglgRUAAIBSElgBAAAoJYEVAACAUhJYAQAAKKWKBdbp06fn05/+dPbZZ598+tOfzj/+8Y9KlQYAAGA1VLHAesYZZ+SII47IL3/5yxxxxBH51re+VanSAAAArIbqKlFk9uzZ+etf/5prr702SXLggQfmnHPOyauvvpq11157pdZRW1vTLr317d29XdZbVg1r9al2CxXTt8fK/dvqKNprjLQXY6/jMvbKzdjruIy98utM468zjb2kc42/9hh777TOmqIoijav+L88/vjj+cY3vpE777yz9bn9998/F154Ybbaaqv2Lg8AAMBqyE2XAAAAKKWKBNb1118/L7/8cpYuXZokWbp0aWbOnJn111+/EuUBAABYDVUksPbp0yeDBg3K5MmTkySTJ0/OoEGDVvr6VQAAADqfilzDmiTPPvtsTjnllMydOzdrrbVWxo0blw984AOVKA0AAMBqqGKBFQAAAFaFmy4BAABQSgIrAAAApSSwAgAAUEoCKwAAAKUksAIAAFBKAitACb3wwgv58Ic/XO02oMO56667st9+++Xggw/OtGnTVmnZuXPnZsKECe+7h/89vkeMGJFFixYlSX74wx9m9uzZ77sGdES33nprTjzxxHed78knn8zPfvazCnREJQisVNySJUuq3QJ0GkuXLq12C1AqN910U0488cTcfvvtq/x98HPnzs3VV1/d5j1NnDgxXbt2TZJcd911Aiu8T08++WR+8YtfVLsN2khdtRug+r72ta9l+vTpaW5uzsYbb5wxY8akZ8+e+elPf5rrrrsuSVJfX58rr7wyffv2zW9+85tceumlWbJkSWprazN27Nj06NEjhx56aP7whz8kefN/j996/NbPRx55ZB588MEcdNBBGTBgQL73ve9l8eLFWbp0aY4//vgccMABSZKXX3455557bv7xj38kSQ488MAcfPDBOfTQQ3P33XensbExSVqXGT58eOXfNHgPHn300VxwwQVZsGBBkmTUqFFZa621ct555+WNN95It27dMnr06Gy77baty3z3u9/Nb3/72yxcuDDnnXdehgwZkiT57W9/myuuuCJNTU2pr6/Pqaeemu222y5/+MMfMmbMmAwZMiSPPfZYTjjhhGyyySYZM2ZM5syZk+bm5hx11FE59NBDkyQDBw7MV7/61fz617/Oa6+9llGjRmWfffZZYb+77bZbpk2btsL1QZmNGTMmjzzySKZPn54bbrgh66yzznL3f0mWuw88++yzM2/evIwYMSJrrLFGbrrppgwbNizjx4/PlltumSTLPB43blz++Mc/prm5Ob17986YMWOy4YYbvq2vgQMHZurUqbnuuusyc+bMnHjiiWlsbMxFF12Uo48+OrfeemvWWWedJMm5556bvn375vjjj6/QuwbLtyr7tLf+FjzssMNy//33Z9GiRbnoooty00035c9//nO6du2a73//++nXr19uvfXWTJo0KT169MiMGTPSq1evXHjhhVl33XXf1sNtt92WG264IUuXLk2PHj1y5plnpnfv3rnkkksyf/78jBgxIkOHDs3pp5+eP//5z7nooota+z3xxBPz0Y9+tLW3ww8/fJX2t9OmTcupp56ahQsXpqWlJYccckiOOeaY3HXXXbn44otTW1ubpUuX5pvf/KYzpt6vgk5v9uzZrT9/5zvfKS688MLioYceKvbee+9i5syZRVEUxfz584tFixYV06ZNK3bZZZdi+vTpRVEUxeLFi4t58+YVzz//fLHjjju2ruffHz///PPFlltuWdx5552tr7/22mvFkiVLiqIoilmzZhUf+chHitdee60oiqI48sgjiwkTJrytv6985SvFrbfeWhRFUbzwwgvFrrvuWixevLit3w5oF3PmzCl22WWX4pFHHimKoiiWLFlSzJo1q9hjjz2KBx54oCiKonjwwQeLPfbYo1i8eHHruLnnnnuKoiiKiRMnFp/+9KeLoiiKGTNmFIcddlgxb968oiiK4umnny722GOPoiiK4qGHHio++MEPFlOnTi2Koiiam5uLQw45pPj73/9eFEVRzJs3r/j4xz/e+njLLbcsfvSjHxVFURRTpkwpdttttxX2+9prr73r+qDsjjzyyNZxtbz9X1EUK9wH/u99XVEUxZ577lk89dRTy3387+v/yU9+UnzlK18piqJ423q23HLLYv78+ctd34UXXlhceumlRVEUxYIFC4qddtqpeOWVV97nuwDvz3vdp/3mN78piqIoJkyYUOywww7FX//616IoiuKMM84ovvOd7xRFURS33HJLsc022xTPPvtsURRFcemllxb//d//3fraWz8//PDDxRe/+MXWvwXvvffe1v3kv89XFEXx+uuvFyNGjChefvnloiiK4uWXXy4+8pGPFK+//vp73t+ec845xWWXXdZa462/Y4cPH148/PDDre/LW8vy3jnCSiZOnJhJkyalubk5b7zxRgYMGJClS5dmxIgR6devX5Kke/fuSZIHH3wwu+++ewYMGJAkaWhoSENDQ1577bV3rNHY2Jj99tuv9fGrr76a0047LTNmzEiXLl3y+uuvZ/r06dliiy3y6KOP5tprr22dd+21106SfO5zn8v555+fQw45JDfeeGMOPfTQNDQ0tOE7Ae3nT3/6UzbbbLMMHjw4SdKlS5fMnj079fX12WWXXZIkO++8c+rr6zN9+vR079493bp1y5577pkk2W677TJu3Lgkyf3335/nnnsun/3sZ1vXv2TJkrzyyitJkk022STbb799kuQf//hHnn322Zx00kmt8zY3N2fatGnZbLPNkiT7779/a42ZM2dm8eLFy+23Z8+e+fvf//6u64PVxfL2f0ly7733LncfuKruu+++3HDDDXnjjTfe8+Uwn/3sZ3PEEUfk+OOPz8SJE7PrrrumT58+72ld0Fbe6z7tox/9aJJkq622ynrrrZdBgwa1Pn7wwQdb17/DDju0nrL/qU99arln091zzz3529/+lk996lNJkqIoMnfu3OX2++ijj+aFF17IF7/4xdbnampqMmPGjPTu3fs97W+HDh2acePGpbm5OR/+8Iez0047JUl22mmnjB07Nvvuu29233331rMveO8E1k5uypQpufHGG3PTTTdl7bXXzqRJk/KTn/xkhfMXRbHc5+vq6pZ5bfHixcu8vsYaa6Smpqb18Zlnnplhw4blsssuS01NTfbZZ5+3LfO/DR48OEuXLs0jjzyS22+/PTfffPPKbCKUwvLGTlEUy4yLt7z13L//h0xtbe0yf/B+5CMfyQUXXPC2ZZ999tl069ZtmRq9e/fOxIkTV9jbW6fZd+nSJcmbO+MVjfWVWR+sDlZ1/7ciXbp0SUtLS+vjt/Zl//znP3P++efnpz/9afr375+pU6fm5JNPXuX1r7/++tlmm21y991354YbbsjZZ5+9yuuAttYW+7R/f9ylS5cV3nNhRestiiKHHnpoRo4cuVL9Dhw4MNdff/3bXnvhhRfe0/52n332yXbbbZcHHnggEyZMyC233JKLLroop512Wp566qk89NBDGTlyZI4++ugcdthh79ojK+amS53c3Llz06NHj/Tq1StNTU255ZZbkiR77rlnJk6c2HrEZsGCBWlqaspuu+2W++67r/X60qampsyfPz99+/ZNc3NzZsyYkSSZPHnyO9adN29eNtxww9TU1OSBBx5oXa579+7Zfvvt88Mf/rB13ldffbX158997nM56aSTst1222X99ddvq7cB2t3222+fZ599No8++miSN2+G1Ldv3zQ1NeWhhx5Kkjz00ENZsmRJ61GeFdl1111z//3355lnnml97i9/+cty5910003TtWvX3H777a3PPfvss5k/f/4q9/v666+/5/VB2axo/5eseB/Yo0ePLFq0aJk/ZjfeeOM89thjSZLf//73rcvMnz8/9fX16devX1paWnLTTTetVF/du3fPvHnzlnnuyCOPzJgxY1JXV9d69gRUU1vu05Zn6tSprX9r3nrrrcu9BnTYsGGZOHFiXnrppdYeHn/88SRJjx49lhlH22+/fWbMmNHaW/LmfnNF/zn7lnfa386YMSP9+vXLJz7xifzXf/1X6+fAtGnTMnDgwBx11FE56KCDWp/nvXOEtZPbfffdc8cdd2S//fbLuuuum6233jqPPfZYdtxxxxx33HE5+uijU1NTk4aGhowfPz4DBgzIOeeck69+9atZunRpunTpkrFjx2bgwIEZPXp0jj766Gy44YbvenH51772tZx11lmZMGFCBg4cmIEDB7a+dtFFF+Wss87KgQcemNra2hx44IE57rjjkiQHHHBAzj777BxxxBHt+r5AW+vVq1cuvfTSjB07Nm+88UZqa2vzjW98I5dccskyN6i4+OKL3/VU9wEDBuTCCy/M6NGjs2jRojQ3N2fw4MHL3KzpLXV1dRk/fnzGjBmTa665Ji0tLenTp0++973vvad+d9lll/e0PiibFe3/kqxwH9i3b98MHz48w4cPT8+ePXPTTTdl5MiROeWUU3LzzTdn8ODB2WCDDZK8eSOlfffdNwcccEA22GCDDB06NFOmTHnXvj7/+c/ntNNOS9euXfPtb387m2++eXbcccc0Njba91EabblPW56hQ4fm0ksvzTPPPNN606XlzfOVr3wlJ5xwQpYuXZrm5ubsu+++2XrrrbPzzjvnBz/4QQ466KDsuOOOOf300/P9738/F154YcaMGZPm5ub0798/48ePf8c+3ml/+/Of/zyTJk1KfX19ampqctpppyVJvv3tb7de8vbWTah4f2qKd/uvBSiRKVOm5Mwzz8ykSZOWe3oIAHQ0zz//fD7zmc/k17/+ddZYY41qtwPt6tZbb829996bSy65pNqtUBKOsLLaOO200/Lggw9m3LhxwioAncLFF1+cW265JaeccoqwCnRKjrACAABQSm66BAAAQCkJrAAAAJSSwAoAAEApCawAAACUksAKAO1s2LBhefDBB6vdBgCsdgRWAAAASklgBYB29PWvfz3/+te/cvzxx2f77bfPtttumx/96EfLzDN8+PDcddddSZKBAwfmuuuuy1577ZUPf/jDGTduXFpaWlrn/elPf5r99tsvQ4cOzTHHHJN//vOfFd0eAKgkgRUA2tGFF16YDTbYIOPHj8+jjz6asWPH5o477mh9/W9/+1tmzpyZ3XffvfW5X//617nlllty22235Z577sktt9ySJLnrrrty5ZVX5rLLLsvvf//77LDDDvna175W8W0CgEoRWAGggvbee+/MmDEj//jHP5IkEydOzH777ZeGhobWeb74xS+mV69e2WCDDfL5z38+kydPTpLcdNNNOe6447LZZpulrq4uxx9/fJ588klHWQHosARWAKighoaG7LvvvrnjjjvS0tKSyZMnZ8SIEcvMs/7667f+vOGGG2bmzJlJkn/9618ZM2ZMhgwZkiFDhmTHHXdMURR5+eWXK7oNAFApddVuAAA6m0MOOSSjRo3KDjvskDXWWCPbb7/9Mq+/+OKL2WKLLZK8GVLXWWedJG8G2eOPPz4HHXRQxXsGgGpwhBUA2lnfvn3z/PPPtz7efvvtU1tbm7Fjxy43fF5zzTV5/fXX8+KLL+a6667L/vvvnyQ5/PDDc9VVV+WZZ55JksybNy8///nPK7MRAFAFAisAtLPjjjsuV1xxRYYMGZJrrrkmSTJixIg8/fTTbzsdOEn22muvfOITn8jBBx+cj370o/nkJz+ZJPnYxz6WY489NieddFIGDx6cAw88MPfdd19FtwUAKqmmKIqi2k0AQGdz++2358c//nFuvPHGZZ4fOHBgfvWrX2WTTTapUmcAUB6OsAJAhS1cuDA33HBDPv3pT1e7FQAoNYEVACro/vvvz84775w+ffrkwAMPrHY7AFBqTgkGAACglBxhBQAAoJQEVgAAAEpJYAUAAKCUBFYAAABKSWAFAACglP5/AIyv9/m2Z08AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 1152x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.set(rc={\"figure.figsize\":(16, 8)})\n",
    "sns.barplot(data=df, x='type', y='value', hue='model')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03d516c3-3173-4ab4-9a9b-9f858d2f76bf",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   }
  ],
  "instance_type": "ml.m5.large",
  "kernelspec": {
   "display_name": "Python 3 (Data Science 2.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-data-science-38"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}