{ "cells": [ { "cell_type": "markdown", "id": "71a329f0", "metadata": {}, "source": [ "# Serve FlanT5-XXL on SageMaker with Faster Transformers using DJL container.\n" ] }, { "cell_type": "markdown", "id": "950a40c2", "metadata": {}, "source": [ "---\n", "\n", "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n", "\n", "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-west-2/inference|generativeai|llm-workshop|lab5-flan-t5-xxl|flant5-xxl-fastertransformer-no-code.ipynb)\n", "\n", "---" ] }, { "cell_type": "markdown", "id": "cff05ea4", "metadata": {}, "source": [ "\n", "In this notebook, we explore how to host a large language model on SageMaker using the latest container that packages some of the most popular open source libraries for model parallel inference like Faster Transfomers, DeepSpeed and Hugging Face Accelerate. We use DJLServing as the model serving solution in this example. DJLServing is a high-performance universal model serving solution powered by the Deep Java Library (DJL) that is programming language agnostic. To learn more about DJL and DJLServing, you can refer to our recent blog post (https://aws.amazon.com/blogs/machine-learning/deploy-bloom-176b-and-opt-30b-on-amazon-sagemaker-with-large-model-inference-deep-learning-containers-and-deepspeed/).\n", "\n", "Language models have recently exploded in both size and popularity. In 2018, BERT-large entered the scene and, with its 340M parameters and novel transformer architecture, set the standard on NLP task accuracy. Within just a few years, state-of-the-art NLP model size has grown by more than 500x with models such as OpenAI’s 175 billion parameter GPT-3 and similarly sized open source Bloom 176B raising the bar on NLP accuracy. This increase in the number of parameters is driven by the simple and empirically-demonstrated positive relationship between model size and accuracy: more is better. With easy access from models zoos such as Hugging Face and improved accuracy in NLP tasks such as classification and text generation, practitioners are increasingly reaching for these large models. However, deploying them can be a challenge because of their size.\n", "\n", "SageMaker has rolled out Deep Learning containers container which now provides users with the ability to leverage the managed serving capabilities and help to provide the un-differentiated heavy lifting.\n", "\n", "In this notebook, we deploy the open source Flan-T5-XXL model across GPUs on a ml.g5.12xlarge instance. The model is loaded using split layer or tensor model partitioning through [Faster Transformers](https://github.com/NVIDIA/FasterTransformer). You can also quantize the model weights to int8 thereby greatly reducing the memory footprint of the model from the initial FP32. See this [blog post](https://huggingface.co/blog/hf-bitsandbytes-integration) from Hugging Face for additional information.\n", "\n", "FlanT5 is available at [Huggingface Model Hub - Model Weights](https://huggingface.co/google/flan-t5-xxl) . For optimizations the model weights can be compiled ahead of time as well by following the steps from [Compile Model](https://github.com/NVIDIA/FasterTransformer/blob/main/examples/pytorch/t5/utils/huggingface_t5_ckpt_convert.py)" ] }, { "cell_type": "markdown", "id": "58b618e7-b049-4d44-8974-8fd1bbeb9a50", "metadata": {}, "source": [ "## Step 1: Install, import the required libraries; set some variables" ] }, { "cell_type": "code", "execution_count": null, "id": "67fa3208", "metadata": { "tags": [] }, "outputs": [], "source": [ "%pip install sagemaker boto3 awscli --upgrade --quiet" ] }, { "cell_type": "code", "execution_count": null, "id": "ec9ac353", "metadata": { "tags": [] }, "outputs": [], "source": [ "import boto3\n", "import jinja2\n", "import sagemaker\n", "from pathlib import Path\n", "from sagemaker import Model, image_uris, serializers, deserializers\n", "\n", "role = sagemaker.get_execution_role() # execution role for the endpoint\n", "sess = sagemaker.session.Session() # sagemaker session for interacting with different AWS APIs\n", "region = sess._region_name # region name of the current SageMaker Studio environment\n", "account_id = sess.account_id() # account_id of the current SageMaker Studio environment\n", "jinja_env = jinja2.Environment()" ] }, { "cell_type": "markdown", "id": "81deac79", "metadata": {}, "source": [ "## Step 2: Start preparing model artifacts\n", "In LMI container, we expect some artifacts to help setting up the model\n", "- serving.properties (required): Defines the model server settings\n", "\n", "This is a configuration file to indicate to LMI Indicator which model parallelization and inference optimization libraries you would like to use. Depending on your need, you can set the appropriate configuration.\n", "\n", "There are a few options specified here. Lets go through them in turn
\n", "1. `engine` - specifies the engine that will be used for this workload. In this case we'll be hosting a model using the **FasterTransformer**\n", "2. `option.entryPoint` - specifies the entrypoint code that will be used to host the model. `djl_python.fastertransformer` refers to the `fastertransformer.py` module from [djl_python repo](https://github.com/deepjavalibrary/djl-serving/tree/master/engines/python/setup/djl_python). \n", "3. `option.s3url`: Set this to the URI of the Amazon S3 bucket that contains the model. When this is set, the container leverages [s5cmd](https://github.com/peak/s5cmd) to download the model from s3. This enables faster deployments by utilizing optimized approach within the DJL inference container to transfer the model from S3 into the hosting instance \n", "4. `option.tensor_parallel_degree`: Set to the number of GPU devices over which DeepSpeed needs to partition the model. This parameter also controls the number of workers per model which will be started up when DJL serving runs. As an example if we have a 8 GPU machine and we are creating 8 partitions then we will have 1 worker per model to serve the requests.\n", "\n", "\n", "For more information on the available options, please refer to the [SageMaker Large Model Inference Documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints-large-model-configuration.html)\n", "\n", "\n", "If you want to download the model from huggingface.co, you can set `option.modelid`. The model id of a pre-trained model hosted inside a model repository on huggingface.co (https://huggingface.co/models). The container uses this model id to download the corresponding model repository on huggingface.co. In this example we will leverage the option.modelid to download from Hugging Face Hub\n", "\n", "\n", "Below files are optional\n", "- model.py (optional): A python file to define the core inference logic\n", "- requirements.txt (optional): Any additional pip wheel need to install\n" ] }, { "cell_type": "code", "execution_count": null, "id": "0710406f-1586-4604-a9d9-20253becec0b", "metadata": { "tags": [] }, "outputs": [], "source": [ "!rm -r code_flant5_ft\n", "!mkdir -p code_flant5_ft" ] }, { "cell_type": "code", "execution_count": null, "id": "b011bf5f", "metadata": { "tags": [] }, "outputs": [], "source": [ "%%writefile code_flant5_ft/serving.properties\n", "engine = FasterTransformer\n", "option.entryPoint = djl_python.fastertransformer\n", "option.s3url = {{s3url}}\n", "option.tensor_parallel_degree = 4" ] }, { "cell_type": "markdown", "id": "5838ae36-7e46-4311-9c5b-00021e19819d", "metadata": {}, "source": [ "### Define a variable to contain the s3url of the location that has the model" ] }, { "cell_type": "code", "execution_count": null, "id": "68d11a3b-c9ea-4139-909b-02aeed1186ba", "metadata": { "tags": [] }, "outputs": [], "source": [ "pretrained_model_location = (\n", " f\"s3://sagemaker-example-files-prod-{region}/models/hf-large-model-djl-ds/flant5-xxl/\"\n", ")\n", "print(f\"Pretrained model will be downloaded from ---- > {pretrained_model_location}\")" ] }, { "cell_type": "code", "execution_count": null, "id": "313ff195-2022-4553-85a6-503cfdf71285", "metadata": { "tags": [] }, "outputs": [], "source": [ "# we plug in the appropriate model location into our `serving.properties` file based on the region in which this notebook is running\n", "template = jinja_env.from_string(Path(\"code_flant5_ft/serving.properties\").open().read())\n", "Path(\"code_flant5_ft/serving.properties\").open(\"w\").write(\n", " template.render(s3url=pretrained_model_location)\n", ")\n", "!pygmentize code_flant5_ft/serving.properties | cat -n" ] }, { "cell_type": "code", "execution_count": null, "id": "ccc82315-1070-4a4f-8d5a-9b9e97e6acc5", "metadata": { "tags": [] }, "outputs": [], "source": [ "!rm -f model.tar.gz\n", "!tar czvf model.tar.gz -C code_flant5_ft ." ] }, { "cell_type": "markdown", "id": "2e58cf33", "metadata": {}, "source": [ "## Step 3: To create the end point the steps are:\n", "\n", "1. Create the Model using the Image container and the Model Tarball uploaded earlier\n", "2. Create the endpoint config using the following key parameters\n", "\n", " a) Instance Type is ml.g5.12xlarge\n", " \n", " b) ContainerStartupHealthCheckTimeoutInSeconds is 3600 to ensure health check starts after the model is ready \n", "3. Create the end point using the endpoint config create\n", "\n", "#### Create the Model\n", "Use the image URI for the DJL container and the s3 location to which the tarball was uploaded.\n", "\n", "The container downloads the model into the `/tmp` space on the container because SageMaker maps the `/tmp` to the Amazon Elastic Block Store (Amazon EBS) volume that is mounted when we specify the endpoint creation parameter VolumeSizeInGB. It leverages `s5cmd`(https://github.com/peak/s5cmd) which offers a very fast download speed and hence extremely useful when downloading large models.\n", "\n", "For instances like p4dn, which come pre-built with the volume instance, we can continue to leverage the `/tmp` on the container. The size of this mount is large enough to hold the model.\n" ] }, { "cell_type": "markdown", "id": "4d955679", "metadata": {}, "source": [ "### Getting the container image URI\n", "\n", "Available frameworks are:\n", "- djl-deepspeed (0.20.0, 0.21.0)\n", "- djl-fastertransformer (0.21.0)\n", "\n", "[Deep learning containers for large model inference](https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints-large-model-dlc.html) has more details about each version and framework in the container." ] }, { "cell_type": "code", "execution_count": null, "id": "7a174b36", "metadata": { "tags": [] }, "outputs": [], "source": [ "inference_image_uri = image_uris.retrieve(\n", " framework=\"djl-fastertransformer\", region=sess.boto_session.region_name, version=\"0.21.0\"\n", ")\n", "print(f\"Image going to be used is ---- > {inference_image_uri}\")" ] }, { "cell_type": "markdown", "id": "072c58a5-28bf-4748-abb8-12fe43b4e3d4", "metadata": {}, "source": [ "#### Create a TAR Ball and upload to S3\n", "\n", "This mainly contains the serving.properties" ] }, { "cell_type": "code", "execution_count": null, "id": "38b1e5ca", "metadata": { "tags": [] }, "outputs": [], "source": [ "s3_code_prefix = \"flants/code\"\n", "bucket = sess.default_bucket() # bucket to house artifacts\n", "code_artifact = sess.upload_data(\"model.tar.gz\", bucket, s3_code_prefix)\n", "print(f\"S3 Code or Model tar ball uploaded to --- > {code_artifact}\")\n", "env = {\"HUGGINGFACE_HUB_CACHE\": \"/tmp\", \"TRANSFORMERS_CACHE\": \"/tmp\"}\n", "\n", "model = Model(image_uri=inference_image_uri, model_data=code_artifact, env=env, role=role)" ] }, { "cell_type": "markdown", "id": "004f39f6", "metadata": {}, "source": [ "#### Create SageMaker endpoint\n", "\n", "You need to specify the instance to use and endpoint names" ] }, { "cell_type": "code", "execution_count": null, "id": "8e0e61cd", "metadata": { "tags": [] }, "outputs": [], "source": [ "instance_type = \"ml.g5.12xlarge\"\n", "endpoint_name = sagemaker.utils.name_from_base(\"lmi-model\")\n", "print(endpoint_name)\n", "model.deploy(\n", " initial_instance_count=1,\n", " instance_type=instance_type,\n", " endpoint_name=endpoint_name,\n", " container_startup_health_check_timeout=3600,\n", ")\n", "\n", "# our requests and responses will be in json format so we specify the serializer and the deserializer\n", "predictor = sagemaker.Predictor(\n", " endpoint_name=endpoint_name,\n", " sagemaker_session=sess,\n", " serializer=serializers.JSONSerializer(),\n", " deserializer=deserializers.JSONDeserializer(),\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "e442d051-9e2f-4026-b29a-2e5dd0ff7819", "metadata": { "tags": [] }, "outputs": [], "source": [ "# our requests and responses will be in json format so we specify the serializer and the deserializer\n", "predictor = sagemaker.Predictor(\n", " endpoint_name=endpoint_name,\n", " sagemaker_session=sess,\n", " serializer=serializers.JSONSerializer(),\n", " deserializer=deserializers.JSONDeserializer(),\n", ")" ] }, { "cell_type": "markdown", "id": "73c96600-56f1-44b9-af9f-d396cc8f4189", "metadata": {}, "source": [ "#### While you wait for the endpoint to be created, you can read more about:\n", "- [Deep Learning containers for large model inference](https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints-large-model-dlc.html)\n", "- [FasterTransformer](https://github.com/NVIDIA/FasterTransformer) \n", "- [Realtime Endpoint for Large Models - FasterTransformer](https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints-large-model-tutorials-fastertransformer.html)\n", " \n", "## Step 4: Invoke the endpoint. \n", "\n", "This is a generative model so we pass in a Text as a prompt and Model will complete the sentence and return the results.\n", "\n", "You can pass a batch of prompts as input to the model. This is done by setting `inputs` to the list of prompts. The model then returns a result for each prompt. The text generation can be configured using appropriate parameters. These `parameters` need to be passed to the endpoint as a dictionary of `kwargs`. \n", "\n", "The below code sample illustrates the invocation of the endpoint using a batch of prompts and also sets some parameters. \n", " " ] }, { "cell_type": "markdown", "id": "bb63ee65", "metadata": {}, "source": [ "Here's a list of default arguments that's used by the model for inference. You can pass specific values based on the use case - \n", "```\n", "default_args = dict(\n", " inputs_embeds=None,\n", " beam_width=1,\n", " max_seq_len=200,\n", " top_k=1,\n", " top_p=0.0,\n", " beam_search_diversity_rate=0.0,\n", " temperature=1.0,\n", " len_penalty=0.0,\n", " repetition_penalty=1.0,\n", " presence_penalty=None,\n", " min_length=0,\n", " random_seed=0,\n", " is_return_output_log_probs=False,\n", " is_return_cum_log_probs=False,\n", " is_return_cross_attentions=False,\n", " bad_words_list=None,\n", " stop_words_list=None\n", " )\n", "```\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "id": "2bcef095", "metadata": { "tags": [] }, "outputs": [], "source": [ "predictor.predict(\n", " {\n", " \"inputs\": [\"Amazon.com is the best \", \"Large model inference is\"],\n", " \"parameters\": {\"min_length\": 20, \"max_seq_len\": 200, \"temperature\": 0.1},\n", " }\n", ")" ] }, { "cell_type": "markdown", "id": "6343ae98-409f-4234-b328-9040c2016e17", "metadata": {}, "source": [ "### Examples of prompt engineering for 'Zero Shot' NLP tasks\n", "Here are some examples of how you can use the deployed model for zero-shot NLP tasks." ] }, { "cell_type": "markdown", "id": "2f8cf100-aec2-4feb-bbfb-cba06137a3d9", "metadata": {}, "source": [ "#### Common Sense Reasoning" ] }, { "cell_type": "code", "execution_count": null, "id": "b54cacd8-f5aa-49aa-a0f9-1283ce7693e1", "metadata": { "tags": [] }, "outputs": [], "source": [ "predictor.predict(\n", " {\n", " \"inputs\": [\n", " \"The world cup has kicked off in Los Angeles, United States.\\n\\nBased on the paragraph above can we conclude that \\”The world cup takes place in United States.\\”?\\n\\n[\\”yes\\”, \\”no\\”]\"\n", " ],\n", " \"parameters\": {\"temperature\": 0.7},\n", " }\n", ")" ] }, { "cell_type": "markdown", "id": "53e7ecde-e1f9-420e-8909-da44f7db9b39", "metadata": {}, "source": [ "#### Text/Sentiment Classification" ] }, { "cell_type": "code", "execution_count": null, "id": "026f2d88-ea03-452f-9540-a6203154ec97", "metadata": { "tags": [] }, "outputs": [], "source": [ "predictor.predict(\n", " {\n", " \"inputs\": [\n", " \"Review:\\nThis movie is so great and once again dazzles and delights us\\nIs this movie review sentence negative or positive?\\nOPTIONS:\\n-positive \\n-negative\"\n", " ],\n", " \"parameters\": {},\n", " }\n", ")" ] }, { "cell_type": "markdown", "id": "8fda82b9-6a42-4f37-8267-78924c222164", "metadata": {}, "source": [ "#### Translation" ] }, { "cell_type": "code", "execution_count": null, "id": "8821289f-a030-403f-a678-e85fd621a976", "metadata": { "tags": [] }, "outputs": [], "source": [ "predictor.predict(\n", " {\"inputs\": [\"My name is Arthur\\n\\nTranslate to German\"], \"parameters\": {\"temperature\": 0.7}}\n", ")" ] }, { "cell_type": "markdown", "id": "c9df301b-0fb0-4b79-af14-24d83d5a694c", "metadata": {}, "source": [ "#### Article Generation" ] }, { "cell_type": "code", "execution_count": null, "id": "9461af2e-ea31-49ff-922c-1b137cb76764", "metadata": { "tags": [] }, "outputs": [], "source": [ "predictor.predict(\n", " {\n", " \"inputs\": [\n", " \"Title: \\”University has new facility coming up“\\\\nGiven the above title of an imaginary article, imagine the article.\\n\"\n", " ],\n", " \"parameters\": {\"max_seq_len\": 200, \"temperature\": 0.7},\n", " }\n", ")" ] }, { "cell_type": "markdown", "id": "7f89e2bd-2c07-4649-a07d-ebc7e38867aa", "metadata": {}, "source": [ "#### Abstractive Question Answering" ] }, { "cell_type": "code", "execution_count": null, "id": "a8ad4c49-f845-4477-8e7c-4a028da0d38f", "metadata": { "tags": [] }, "outputs": [], "source": [ "context = \"\"\"\n", "Customer: Hi there, I'm having a problem with my iPhone.\n", "Agent: Hi! I'm sorry to hear that. What's happening?\n", "Customer: The phone is not charging properly, and the battery seems to be draining very quickly. I've tried different charging cables and power adapters, but the issue persists.\n", "Agent: Hmm, that's not good. Let's try some troubleshooting steps. Can you go to Settings, then Battery, and see if there are any apps that are using up a lot of battery life?\n", "Customer: Yes, there are some apps that are using up a lot of battery.\n", "Agent: Okay, try force quitting those apps by swiping up from the bottom of the screen and then swiping up on the app to close it.\n", "Customer: I did that, but the issue is still there.\n", "Agent: Alright, let's try resetting your iPhone's settings to their default values. This won't delete any of your data. Go to Settings, then General, then Reset, and then choose Reset All Settings.\n", "Customer: Okay, I did that. What's next?\n", "Agent: Now, let's try restarting your iPhone. Press and hold the power button until you see the \"slide to power off\" option. Slide to power off, wait a few seconds, and then turn your iPhone back on.\n", "Customer: Alright, I restarted it, but it's still not charging properly.\n", "Agent: I see. It looks like we need to run a diagnostic test on your iPhone. Please visit the nearest Apple Store or authorized service provider to get your iPhone checked out.\n", "Customer: Do I need to make an appointment?\n", "Agent: Yes, it's always best to make an appointment beforehand so you don't have to wait in line. You can make an appointment online or by calling the Apple Store or authorized service provider.\n", "Customer: Okay, will I have to pay for the repairs?\n", "Agent: That depends on whether your iPhone is covered under warranty or not. If it is, you won't have to pay anything. However, if it's not covered under warranty, you will have to pay for the repairs.\n", "Customer: How long will it take to get my iPhone back?\n", "Agent: It depends on the severity of the issue, but it usually takes 1-2 business days.\n", "Customer: Can I track the repair status online?\n", "Agent: Yes, you can track the repair status online or by calling the Apple Store or authorized service provider.\n", "Customer: Alright, thanks for your help.\n", "Agent: No problem, happy to help. Is there anything else I can assist you with?\n", "Customer: No, that's all for now.\n", "Agent: Alright, have a great day and good luck with your iPhone!\n", "\"\"\"" ] }, { "cell_type": "markdown", "id": "fc53211b-790f-486b-b761-c811a5e8b134", "metadata": {}, "source": [ "#### Q1" ] }, { "cell_type": "code", "execution_count": null, "id": "e55f2a5b-eb57-4b88-a285-f95e53ebda6d", "metadata": { "tags": [] }, "outputs": [], "source": [ "query = (\n", " \"What troubleshooting steps were suggested to the customer to fix their iPhone charging issue?\"\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "a808fb9f-bab5-4b8c-b2d1-b5fe8fbf9d6a", "metadata": { "tags": [] }, "outputs": [], "source": [ "predictor.predict(\n", " {\"inputs\": [f\"{context}\\n{query}\"], \"parameters\": {\"max_seq_len\": 200, \"temperature\": 0.7}}\n", ")" ] }, { "cell_type": "markdown", "id": "d800ec99-e432-4eca-8cef-70bdd252c1a9", "metadata": {}, "source": [ "#### Q2" ] }, { "cell_type": "code", "execution_count": null, "id": "3a3b7a23-a596-426b-a52e-c7ed189d22b4", "metadata": { "tags": [] }, "outputs": [], "source": [ "query = \"Was resetting the iPhone to its default settings able to solve the charging issue and battery drain problem?\"" ] }, { "cell_type": "code", "execution_count": null, "id": "8a696e9d-2c03-475e-9f3d-11306fdf2f2f", "metadata": { "tags": [] }, "outputs": [], "source": [ "predictor.predict(\n", " {\"inputs\": [f\"{context}\\n{query}\"], \"parameters\": {\"max_seq_len\": 200, \"temperature\": 0.7}}\n", ")" ] }, { "cell_type": "markdown", "id": "c1cd9042", "metadata": {}, "source": [ "## Clean up the environment" ] }, { "cell_type": "code", "execution_count": null, "id": "3d674b41", "metadata": { "tags": [] }, "outputs": [], "source": [ "sess.delete_endpoint(endpoint_name)\n", "sess.delete_endpoint_config(endpoint_name)\n", "model.delete_model()" ] }, { "cell_type": "markdown", "id": "1e710115", "metadata": {}, "source": [ "## Notebook CI Test Results\n", "\n", "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n", "\n", "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-east-1/inference|generativeai|llm-workshop|lab5-flan-t5-xxl|flant5-xxl-fastertransformer-no-code.ipynb)\n", "\n", "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-east-2/inference|generativeai|llm-workshop|lab5-flan-t5-xxl|flant5-xxl-fastertransformer-no-code.ipynb)\n", "\n", "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-west-1/inference|generativeai|llm-workshop|lab5-flan-t5-xxl|flant5-xxl-fastertransformer-no-code.ipynb)\n", "\n", "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ca-central-1/inference|generativeai|llm-workshop|lab5-flan-t5-xxl|flant5-xxl-fastertransformer-no-code.ipynb)\n", "\n", "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/sa-east-1/inference|generativeai|llm-workshop|lab5-flan-t5-xxl|flant5-xxl-fastertransformer-no-code.ipynb)\n", "\n", "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-west-1/inference|generativeai|llm-workshop|lab5-flan-t5-xxl|flant5-xxl-fastertransformer-no-code.ipynb)\n", "\n", "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-west-2/inference|generativeai|llm-workshop|lab5-flan-t5-xxl|flant5-xxl-fastertransformer-no-code.ipynb)\n", "\n", "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-west-3/inference|generativeai|llm-workshop|lab5-flan-t5-xxl|flant5-xxl-fastertransformer-no-code.ipynb)\n", "\n", "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-central-1/inference|generativeai|llm-workshop|lab5-flan-t5-xxl|flant5-xxl-fastertransformer-no-code.ipynb)\n", "\n", "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-north-1/inference|generativeai|llm-workshop|lab5-flan-t5-xxl|flant5-xxl-fastertransformer-no-code.ipynb)\n", "\n", "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-southeast-1/inference|generativeai|llm-workshop|lab5-flan-t5-xxl|flant5-xxl-fastertransformer-no-code.ipynb)\n", "\n", "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-southeast-2/inference|generativeai|llm-workshop|lab5-flan-t5-xxl|flant5-xxl-fastertransformer-no-code.ipynb)\n", "\n", "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-northeast-1/inference|generativeai|llm-workshop|lab5-flan-t5-xxl|flant5-xxl-fastertransformer-no-code.ipynb)\n", "\n", "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-northeast-2/inference|generativeai|llm-workshop|lab5-flan-t5-xxl|flant5-xxl-fastertransformer-no-code.ipynb)\n", "\n", "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-south-1/inference|generativeai|llm-workshop|lab5-flan-t5-xxl|flant5-xxl-fastertransformer-no-code.ipynb)\n" ] } ], "metadata": { "availableInstances": [ { "_defaultOrder": 0, "_isFastLaunch": true, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 4, "name": "ml.t3.medium", "vcpuNum": 2 }, { "_defaultOrder": 1, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.t3.large", "vcpuNum": 2 }, { "_defaultOrder": 2, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.t3.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 3, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.t3.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 4, "_isFastLaunch": true, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.m5.large", "vcpuNum": 2 }, { "_defaultOrder": 5, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.m5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 6, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.m5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 7, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.m5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 8, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.m5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 9, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.m5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 10, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.m5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 11, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.m5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 12, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.m5d.large", "vcpuNum": 2 }, { "_defaultOrder": 13, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.m5d.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 14, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.m5d.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 15, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.m5d.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 16, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.m5d.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 17, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.m5d.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 18, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.m5d.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 19, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.m5d.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 20, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": true, "memoryGiB": 0, "name": "ml.geospatial.interactive", "supportedImageNames": [ "sagemaker-geospatial-v1-0" ], "vcpuNum": 0 }, { "_defaultOrder": 21, "_isFastLaunch": true, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 4, "name": "ml.c5.large", "vcpuNum": 2 }, { "_defaultOrder": 22, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.c5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 23, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.c5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 24, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.c5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 25, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 72, "name": "ml.c5.9xlarge", "vcpuNum": 36 }, { "_defaultOrder": 26, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 96, "name": "ml.c5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 27, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 144, "name": "ml.c5.18xlarge", "vcpuNum": 72 }, { "_defaultOrder": 28, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.c5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 29, "_isFastLaunch": true, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.g4dn.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 30, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.g4dn.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 31, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.g4dn.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 32, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.g4dn.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 33, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.g4dn.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 34, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.g4dn.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 35, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 61, "name": "ml.p3.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 36, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 244, "name": "ml.p3.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 37, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 488, "name": "ml.p3.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 38, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 768, "name": "ml.p3dn.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 39, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.r5.large", "vcpuNum": 2 }, { "_defaultOrder": 40, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.r5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 41, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.r5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 42, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.r5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 43, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.r5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 44, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.r5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 45, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 512, "name": "ml.r5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 46, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 768, "name": "ml.r5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 47, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.g5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 48, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.g5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 49, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.g5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 50, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.g5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 51, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.g5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 52, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.g5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 53, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.g5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 54, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 768, "name": "ml.g5.48xlarge", "vcpuNum": 192 }, { "_defaultOrder": 55, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 1152, "name": "ml.p4d.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 56, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 1152, "name": "ml.p4de.24xlarge", "vcpuNum": 96 } ], "kernelspec": { "display_name": "Python 3 (Data Science 3.0)", "language": "python", "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/sagemaker-data-science-310-v1" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.6" } }, "nbformat": 4, "nbformat_minor": 5 }