{ "cells": [ { "cell_type": "markdown", "id": "dab554c6", "metadata": {}, "source": [ "# Deploy Stable Diffusion on a SageMaker GPU Single Model Endpoint with Triton" ] }, { "cell_type": "markdown", "id": "5640df9b", "metadata": {}, "source": [ "In this notebook we will host Stable Diffusion SageMaker GPU Multi-Model Endpoints (MME GPU) powered by NVIDIA Triton Inference Server.\n", "\n", "Skip to:\n", "1. [Installs and imports](#installs)\n", "2. [Save a pretrained model](#modelartifact)\n", "3. [Packaging a conda environment, extending Sagemaker Triton container](#condaenv)\n", "4. [Local testing of Triton model repository](#local)\n", "5. [Deploy to SageMaker Real-Time Endpoint](#deploy)\n", "6. [Analyze endpoint logs](#logs)\n", "7. [Clean up](#cleanup)\n", "------\n", "------" ] }, { "cell_type": "markdown", "id": "fdbf35ff", "metadata": {}, "source": [ "### Part 1 - Installs and imports " ] }, { "cell_type": "code", "execution_count": 1, "id": "69df6cd4", "metadata": { "collapsed": true, "jupyter": { "outputs_hidden": true }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Looking in indexes: https://pypi.org/simple, https://pip.repos.neuron.amazonaws.com\n", "Collecting nvidia-pyindex\n", " Downloading nvidia-pyindex-1.0.9.tar.gz (10 kB)\n", " Preparing metadata (setup.py) ... \u001b[?25ldone\n", "\u001b[?25hBuilding wheels for collected packages: nvidia-pyindex\n", " Building wheel for nvidia-pyindex (setup.py) ... \u001b[?25ldone\n", "\u001b[?25h Created wheel for nvidia-pyindex: filename=nvidia_pyindex-1.0.9-py3-none-any.whl size=8418 sha256=149f25543fe93332f3642f3b88c06e3565aa4aed23f8790d641a38de9c71b37d\n", " Stored in directory: /home/ec2-user/.cache/pip/wheels/0e/62/68/8bb6aafc3cb47e3468055aebc10d004b55da43563d748aac9c\n", "Successfully built nvidia-pyindex\n", "Installing collected packages: nvidia-pyindex\n", "Successfully installed nvidia-pyindex-1.0.9\n", "Looking in indexes: https://pypi.org/simple, https://pip.repos.neuron.amazonaws.com, https://pypi.ngc.nvidia.com\n", "Collecting tritonclient[http]\n", " Downloading tritonclient-2.31.0-py3-none-manylinux1_x86_64.whl (11.7 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m11.7/11.7 MB\u001b[0m \u001b[31m195.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n", "\u001b[?25hRequirement already satisfied: numpy>=1.19.1 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from tritonclient[http]) (1.23.5)\n", "Collecting python-rapidjson>=0.9.1\n", " Downloading python_rapidjson-1.10-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.7 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m257.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting geventhttpclient<=2.0.2,>=1.4.4\n", " Downloading geventhttpclient-2.0.2-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (100 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m100.1/100.1 kB\u001b[0m \u001b[31m357.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hRequirement already satisfied: aiohttp>=3.8.1 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from tritonclient[http]) (3.8.3)\n", "Requirement already satisfied: aiosignal>=1.1.2 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from aiohttp>=3.8.1->tritonclient[http]) (1.3.1)\n", "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from aiohttp>=3.8.1->tritonclient[http]) (4.0.2)\n", "Requirement already satisfied: yarl<2.0,>=1.0 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from aiohttp>=3.8.1->tritonclient[http]) (1.8.2)\n", "Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from aiohttp>=3.8.1->tritonclient[http]) (2.1.1)\n", "Requirement already satisfied: attrs>=17.3.0 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from aiohttp>=3.8.1->tritonclient[http]) (22.2.0)\n", "Requirement already satisfied: multidict<7.0,>=4.5 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from aiohttp>=3.8.1->tritonclient[http]) (6.0.4)\n", "Requirement already satisfied: frozenlist>=1.1.1 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from aiohttp>=3.8.1->tritonclient[http]) (1.3.3)\n", "Collecting brotli\n", " Downloading Brotli-1.0.9-cp39-cp39-manylinux1_x86_64.whl (357 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m357.2/357.2 kB\u001b[0m \u001b[31m405.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hRequirement already satisfied: six in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from geventhttpclient<=2.0.2,>=1.4.4->tritonclient[http]) (1.16.0)\n", "Requirement already satisfied: certifi in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from geventhttpclient<=2.0.2,>=1.4.4->tritonclient[http]) (2022.12.7)\n", "Requirement already satisfied: gevent>=0.13 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from geventhttpclient<=2.0.2,>=1.4.4->tritonclient[http]) (22.10.2)\n", "Requirement already satisfied: zope.interface in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from gevent>=0.13->geventhttpclient<=2.0.2,>=1.4.4->tritonclient[http]) (5.5.2)\n", "Requirement already satisfied: setuptools in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from gevent>=0.13->geventhttpclient<=2.0.2,>=1.4.4->tritonclient[http]) (65.6.3)\n", "Requirement already satisfied: greenlet>=2.0.0 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from gevent>=0.13->geventhttpclient<=2.0.2,>=1.4.4->tritonclient[http]) (2.0.1)\n", "Requirement already satisfied: zope.event in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from gevent>=0.13->geventhttpclient<=2.0.2,>=1.4.4->tritonclient[http]) (4.6)\n", "Requirement already satisfied: idna>=2.0 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from yarl<2.0,>=1.0->aiohttp>=3.8.1->tritonclient[http]) (3.4)\n", "Installing collected packages: brotli, python-rapidjson, tritonclient, geventhttpclient\n", "Successfully installed brotli-1.0.9 geventhttpclient-2.0.2 python-rapidjson-1.10 tritonclient-2.31.0\n", "Looking in indexes: https://pypi.org/simple, https://pip.repos.neuron.amazonaws.com, https://pypi.ngc.nvidia.com\n", "Requirement already satisfied: sagemaker in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (2.132.0)\n", "Collecting sagemaker\n", " Downloading sagemaker-2.140.1.tar.gz (684 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m684.5/684.5 kB\u001b[0m \u001b[31m120.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25h Preparing metadata (setup.py) ... \u001b[?25ldone\n", "\u001b[?25hRequirement already satisfied: ipywidgets in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (8.0.4)\n", "Collecting ipywidgets\n", " Downloading ipywidgets-8.0.5-py3-none-any.whl (138 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m138.3/138.3 kB\u001b[0m \u001b[31m368.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hRequirement already satisfied: pillow in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (9.2.0)\n", "Collecting pillow\n", " Downloading Pillow-9.4.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.3 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.3/3.3 MB\u001b[0m \u001b[31m314.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hRequirement already satisfied: numpy in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (1.23.5)\n", "Collecting numpy\n", " Downloading numpy-1.24.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m17.3/17.3 MB\u001b[0m \u001b[31m400.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n", "\u001b[?25hCollecting transformers\n", " Downloading transformers-4.27.2-py3-none-any.whl (6.8 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.8/6.8 MB\u001b[0m \u001b[31m269.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting accelerate\n", " Downloading accelerate-0.17.1-py3-none-any.whl (212 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m212.8/212.8 kB\u001b[0m \u001b[31m390.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting diffusers\n", " Downloading diffusers-0.14.0-py3-none-any.whl (737 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m737.4/737.4 kB\u001b[0m \u001b[31m426.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hRequirement already satisfied: attrs<23,>=20.3.0 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from sagemaker) (22.2.0)\n", "Requirement already satisfied: boto3<2.0,>=1.26.28 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from sagemaker) (1.26.71)\n", "Requirement already satisfied: google-pasta in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from sagemaker) (0.2.0)\n", "Requirement already satisfied: protobuf<4.0,>=3.1 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from sagemaker) (3.20.2)\n", "Requirement already satisfied: protobuf3-to-dict<1.0,>=0.1.5 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from sagemaker) (0.1.5)\n", "Requirement already satisfied: smdebug_rulesconfig==1.0.1 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from sagemaker) (1.0.1)\n", "Requirement already satisfied: importlib-metadata<5.0,>=1.4.0 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from sagemaker) (4.13.0)\n", "Requirement already satisfied: packaging>=20.0 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from sagemaker) (21.3)\n", "Requirement already satisfied: pandas in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from sagemaker) (1.4.4)\n", "Requirement already satisfied: pathos in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from sagemaker) (0.3.0)\n", "Requirement already satisfied: schema in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from sagemaker) (0.7.5)\n", "Requirement already satisfied: ipython>=6.1.0 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from ipywidgets) (7.32.0)\n", "Requirement already satisfied: widgetsnbextension~=4.0 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from ipywidgets) (4.0.5)\n", "Requirement already satisfied: traitlets>=4.3.1 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from ipywidgets) (5.8.1)\n", "Requirement already satisfied: jupyterlab-widgets~=3.0 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from ipywidgets) (3.0.5)\n", "Collecting tokenizers!=0.11.3,<0.14,>=0.11.1\n", " Downloading tokenizers-0.13.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.6 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.6/7.6 MB\u001b[0m \u001b[31m290.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hRequirement already satisfied: pyyaml>=5.1 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from transformers) (5.4.1)\n", "Requirement already satisfied: regex!=2019.12.17 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from transformers) (2022.10.31)\n", "Requirement already satisfied: filelock in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from transformers) (3.6.0)\n", "Requirement already satisfied: tqdm>=4.27 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from transformers) (4.63.2)\n", "Collecting huggingface-hub<1.0,>=0.11.0\n", " Downloading huggingface_hub-0.13.3-py3-none-any.whl (199 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m199.8/199.8 kB\u001b[0m \u001b[31m409.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hRequirement already satisfied: requests in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from transformers) (2.28.1)\n", "Requirement already satisfied: psutil in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from accelerate) (5.9.4)\n", "Requirement already satisfied: torch>=1.4.0 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from accelerate) (1.13.1)\n", "Requirement already satisfied: s3transfer<0.7.0,>=0.6.0 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from boto3<2.0,>=1.26.28->sagemaker) (0.6.0)\n", "Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from boto3<2.0,>=1.26.28->sagemaker) (1.0.1)\n", "Requirement already satisfied: botocore<1.30.0,>=1.29.71 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from boto3<2.0,>=1.26.28->sagemaker) (1.29.71)\n", "Requirement already satisfied: typing-extensions>=3.7.4.3 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from huggingface-hub<1.0,>=0.11.0->transformers) (4.4.0)\n", "Requirement already satisfied: zipp>=0.5 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from importlib-metadata<5.0,>=1.4.0->sagemaker) (3.11.0)\n", "Requirement already satisfied: pygments in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets) (2.14.0)\n", "Requirement already satisfied: backcall in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets) (0.2.0)\n", "Requirement already satisfied: jedi>=0.16 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets) (0.18.2)\n", "Requirement already satisfied: matplotlib-inline in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets) (0.1.6)\n", "Requirement already satisfied: pickleshare in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets) (0.7.5)\n", "Requirement already satisfied: decorator in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets) (5.1.1)\n", "Requirement already satisfied: pexpect>4.3 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets) (4.8.0)\n", "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets) (3.0.36)\n", "Requirement already satisfied: setuptools>=18.5 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets) (65.6.3)\n", "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from packaging>=20.0->sagemaker) (3.0.9)\n", "Requirement already satisfied: six in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from protobuf3-to-dict<1.0,>=0.1.5->sagemaker) (1.16.0)\n", "Requirement already satisfied: python-dateutil>=2.8.1 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from pandas->sagemaker) (2.8.2)\n", "Requirement already satisfied: pytz>=2020.1 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from pandas->sagemaker) (2022.7)\n", "Requirement already satisfied: ppft>=1.7.6.6 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from pathos->sagemaker) (1.7.6.6)\n", "Requirement already satisfied: pox>=0.3.2 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from pathos->sagemaker) (0.3.2)\n", "Requirement already satisfied: multiprocess>=0.70.14 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from pathos->sagemaker) (0.70.14)\n", "Requirement already satisfied: dill>=0.3.6 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from pathos->sagemaker) (0.3.6)\n", "Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from requests->transformers) (2022.12.7)\n", "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from requests->transformers) (1.26.8)\n", "Requirement already satisfied: idna<4,>=2.5 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from requests->transformers) (3.4)\n", "Requirement already satisfied: charset-normalizer<3,>=2 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from requests->transformers) (2.1.1)\n", "Requirement already satisfied: contextlib2>=0.5.5 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from schema->sagemaker) (21.6.0)\n", "Requirement already satisfied: parso<0.9.0,>=0.8.0 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from jedi>=0.16->ipython>=6.1.0->ipywidgets) (0.8.3)\n", "Requirement already satisfied: ptyprocess>=0.5 in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from pexpect>4.3->ipython>=6.1.0->ipywidgets) (0.7.0)\n", "Requirement already satisfied: wcwidth in /home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=6.1.0->ipywidgets) (0.2.5)\n", "Building wheels for collected packages: sagemaker\n", " Building wheel for sagemaker (setup.py) ... \u001b[?25ldone\n", "\u001b[?25h Created wheel for sagemaker: filename=sagemaker-2.140.1-py2.py3-none-any.whl size=925381 sha256=97c8fb5f4d4ba29952d0ad0f62139c5ac22ae0c1f6f692445eea9010e265059f\n", " Stored in directory: /tmp/pip-ephem-wheel-cache-h3zyi23b/wheels/f3/6c/41/d279aee7906606981bb84976ffb75ee2e986f783cf7cc53bc3\n", "Successfully built sagemaker\n", "Installing collected packages: tokenizers, pillow, numpy, huggingface-hub, accelerate, transformers, ipywidgets, diffusers, sagemaker\n", " Attempting uninstall: pillow\n", " Found existing installation: Pillow 9.2.0\n", " Uninstalling Pillow-9.2.0:\n", " Successfully uninstalled Pillow-9.2.0\n", " Attempting uninstall: numpy\n", " Found existing installation: numpy 1.23.5\n", " Uninstalling numpy-1.23.5:\n", " Successfully uninstalled numpy-1.23.5\n", " Attempting uninstall: ipywidgets\n", " Found existing installation: ipywidgets 8.0.4\n", " Uninstalling ipywidgets-8.0.4:\n", " Successfully uninstalled ipywidgets-8.0.4\n", " Attempting uninstall: sagemaker\n", " Found existing installation: sagemaker 2.132.0\n", " Uninstalling sagemaker-2.132.0:\n", " Successfully uninstalled sagemaker-2.132.0\n", "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n", "numba 0.56.4 requires numpy<1.24,>=1.18, but you have numpy 1.24.2 which is incompatible.\u001b[0m\u001b[31m\n", "\u001b[0mSuccessfully installed accelerate-0.17.1 diffusers-0.14.0 huggingface-hub-0.13.3 ipywidgets-8.0.5 numpy-1.24.2 pillow-9.4.0 sagemaker-2.140.1 tokenizers-0.13.2 transformers-4.27.2\n" ] } ], "source": [ "!pip install nvidia-pyindex\n", "!pip install tritonclient[http]\n", "!pip install -U sagemaker ipywidgets pillow numpy transformers accelerate diffusers" ] }, { "cell_type": "code", "execution_count": 2, "id": "44c48876", "metadata": {}, "outputs": [], "source": [ "import boto3\n", "import sagemaker\n", "from sagemaker import get_execution_role\n", "\n", "import tritonclient.http as httpclient\n", "from tritonclient.utils import *\n", "import time\n", "from PIL import Image\n", "import numpy as np\n", "\n", "# variables\n", "s3_client = boto3.client(\"s3\")\n", "ts = time.strftime(\"%Y-%m-%d-%H-%M-%S\", time.gmtime())\n", "\n", "# sagemaker variables\n", "role = get_execution_role()\n", "sm_client = boto3.client(service_name=\"sagemaker\")\n", "runtime_sm_client = boto3.client(\"sagemaker-runtime\")\n", "sagemaker_session = sagemaker.Session(boto_session=boto3.Session())\n", "bucket = sagemaker_session.default_bucket()" ] }, { "cell_type": "markdown", "id": "7fda567c-afaa-4a7e-a947-98f1b15cb358", "metadata": {}, "source": [ "### Part 2 - Save pretrained model " ] }, { "cell_type": "markdown", "id": "d30d8b41-ffc3-49c6-831f-b70e08746284", "metadata": {}, "source": [ "We save a pretrained Stable Diffusion Model to the Triton model repository directory." ] }, { "cell_type": "code", "execution_count": 3, "id": "8e5c517d-872b-478c-a349-16264427e6b9", "metadata": { "collapsed": true, "jupyter": { "outputs_hidden": true }, "scrolled": true, "tags": [] }, "outputs": [ { "data": { "application/json": { "ascii": false, "bar_format": null, "colour": null, "elapsed": 0.006927967071533203, "initial": 0, "n": 0, "ncols": null, "nrows": null, "postfix": null, "prefix": "Downloading (…)p16/model_index.json", "rate": null, "total": 543, "unit": "B", "unit_divisor": 1000, "unit_scale": true }, "application/vnd.jupyter.widget-view+json": { "model_id": "2ec832e396214ec19c07790e53cb825a", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Downloading (…)p16/model_index.json: 0%| | 0.00/543 [00:00, ?B/s]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/json": { "ascii": false, "bar_format": null, "colour": null, "elapsed": 0.009253740310668945, "initial": 0, "n": 0, "ncols": null, "nrows": null, "postfix": null, "prefix": "Fetching 16 files", "rate": null, "total": 16, "unit": "it", "unit_divisor": 1000, "unit_scale": false }, "application/vnd.jupyter.widget-view+json": { "model_id": "fb2b187b326b4e6587f41aa6b0817eb0", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Fetching 16 files: 0%| | 0/16 [00:00, ?it/s]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/json": { "ascii": false, "bar_format": null, "colour": null, "elapsed": 0.009071826934814453, "initial": 0, "n": 0, "ncols": null, "nrows": null, "postfix": null, "prefix": "Downloading (…)rocessor_config.json", "rate": null, "total": 342, "unit": "B", "unit_divisor": 1000, "unit_scale": true }, "application/vnd.jupyter.widget-view+json": { "model_id": "0838eb3ed1b34a5981e4ba07bbe4f022", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Downloading (…)rocessor_config.json: 0%| | 0.00/342 [00:00, ?B/s]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/json": { "ascii": false, "bar_format": null, "colour": null, "elapsed": 0.009196996688842773, "initial": 0, "n": 0, "ncols": null, "nrows": null, "postfix": null, "prefix": "Downloading (…)_checker/config.json", "rate": null, "total": 4628, "unit": "B", "unit_divisor": 1000, "unit_scale": true }, "application/vnd.jupyter.widget-view+json": { "model_id": "58bb5c9650654bc1b156cedd78c67aee", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Downloading (…)_checker/config.json: 0%| | 0.00/4.63k [00:00, ?B/s]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/json": { "ascii": false, "bar_format": null, "colour": null, "elapsed": 0.013409137725830078, "initial": 0, "n": 0, "ncols": null, "nrows": null, "postfix": null, "prefix": "Downloading pytorch_model.bin", "rate": null, "total": 246184375, "unit": "B", "unit_divisor": 1000, "unit_scale": true }, "application/vnd.jupyter.widget-view+json": { "model_id": "2c858090171b427d8dadfe3cc23fdf24", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Downloading pytorch_model.bin: 0%| | 0.00/246M [00:00, ?B/s]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/json": { "ascii": false, "bar_format": null, "colour": null, "elapsed": 0.01302957534790039, "initial": 0, "n": 0, "ncols": null, "nrows": null, "postfix": null, "prefix": "Downloading (…)cheduler_config.json", "rate": null, "total": 307, "unit": "B", "unit_divisor": 1000, "unit_scale": true }, "application/vnd.jupyter.widget-view+json": { "model_id": "41a3ce1abb3d4c58b2d6f9a501420d99", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Downloading (…)cheduler_config.json: 0%| | 0.00/307 [00:00, ?B/s]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/json": { "ascii": false, "bar_format": null, "colour": null, "elapsed": 0.012043476104736328, "initial": 0, "n": 0, "ncols": null, "nrows": null, "postfix": null, "prefix": "Downloading (…)_encoder/config.json", "rate": null, "total": 572, "unit": "B", "unit_divisor": 1000, "unit_scale": true }, "application/vnd.jupyter.widget-view+json": { "model_id": "619fefa6944248bd8e937782f5cdbf57", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Downloading (…)_encoder/config.json: 0%| | 0.00/572 [00:00, ?B/s]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/json": { "ascii": false, "bar_format": null, "colour": null, "elapsed": 0.011409282684326172, "initial": 0, "n": 0, "ncols": null, "nrows": null, "postfix": null, "prefix": "Downloading (…)nfig-checkpoint.json", "rate": null, "total": 209, "unit": "B", "unit_divisor": 1000, "unit_scale": true }, "application/vnd.jupyter.widget-view+json": { "model_id": "918a058862d945bdb427a49673d462cc", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Downloading (…)nfig-checkpoint.json: 0%| | 0.00/209 [00:00, ?B/s]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/json": { "ascii": false, "bar_format": null, "colour": null, "elapsed": 0.014387369155883789, "initial": 0, "n": 0, "ncols": null, "nrows": null, "postfix": null, "prefix": "Downloading pytorch_model.bin", "rate": null, "total": 608098599, "unit": "B", "unit_divisor": 1000, "unit_scale": true }, "application/vnd.jupyter.widget-view+json": { "model_id": "c25dcc5002774f61a73cda860a141bae", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Downloading pytorch_model.bin: 0%| | 0.00/608M [00:00, ?B/s]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/json": { "ascii": false, "bar_format": null, "colour": null, "elapsed": 0.025308609008789062, "initial": 0, "n": 0, "ncols": null, "nrows": null, "postfix": null, "prefix": "Downloading (…)tokenizer/merges.txt", "rate": null, "total": 524619, "unit": "B", "unit_divisor": 1000, "unit_scale": true }, "application/vnd.jupyter.widget-view+json": { "model_id": "05c990a1642a41b0bfa4b8733e3823d0", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Downloading (…)tokenizer/merges.txt: 0%| | 0.00/525k [00:00, ?B/s]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/json": { "ascii": false, "bar_format": null, "colour": null, "elapsed": 0.0052874088287353516, "initial": 0, "n": 0, "ncols": null, "nrows": null, "postfix": null, "prefix": "Downloading (…)cial_tokens_map.json", "rate": null, "total": 472, "unit": "B", "unit_divisor": 1000, "unit_scale": true }, "application/vnd.jupyter.widget-view+json": { "model_id": "0a32c72f7d3f4646baec839a78b1a6ad", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Downloading (…)cial_tokens_map.json: 0%| | 0.00/472 [00:00, ?B/s]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/json": { "ascii": false, "bar_format": null, "colour": null, "elapsed": 0.0070040225982666016, "initial": 0, "n": 0, "ncols": null, "nrows": null, "postfix": null, "prefix": "Downloading (…)okenizer_config.json", "rate": null, "total": 788, "unit": "B", "unit_divisor": 1000, "unit_scale": true }, "application/vnd.jupyter.widget-view+json": { "model_id": "dde92df13545498e828d2fe27b3476e2", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Downloading (…)okenizer_config.json: 0%| | 0.00/788 [00:00, ?B/s]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/json": { "ascii": false, "bar_format": null, "colour": null, "elapsed": 0.012964248657226562, "initial": 0, "n": 0, "ncols": null, "nrows": null, "postfix": null, "prefix": "Downloading (…)587/unet/config.json", "rate": null, "total": 772, "unit": "B", "unit_divisor": 1000, "unit_scale": true }, "application/vnd.jupyter.widget-view+json": { "model_id": "f7cf604062864ca4ae16d76a4cba8a6c", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Downloading (…)587/unet/config.json: 0%| | 0.00/772 [00:00, ?B/s]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/json": { "ascii": false, "bar_format": null, "colour": null, "elapsed": 0.016084909439086914, "initial": 0, "n": 0, "ncols": null, "nrows": null, "postfix": null, "prefix": "Downloading (…)tokenizer/vocab.json", "rate": null, "total": 1059962, "unit": "B", "unit_divisor": 1000, "unit_scale": true }, "application/vnd.jupyter.widget-view+json": { "model_id": "8506bb3c35554d16b5a7f5074b2080ff", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Downloading (…)tokenizer/vocab.json: 0%| | 0.00/1.06M [00:00, ?B/s]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/json": { "ascii": false, "bar_format": null, "colour": null, "elapsed": 0.012598514556884766, "initial": 0, "n": 0, "ncols": null, "nrows": null, "postfix": null, "prefix": "Downloading (…)7587/vae/config.json", "rate": null, "total": 550, "unit": "B", "unit_divisor": 1000, "unit_scale": true }, "application/vnd.jupyter.widget-view+json": { "model_id": "57b4b144f0564165a78de834b606bb94", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Downloading (…)7587/vae/config.json: 0%| | 0.00/550 [00:00, ?B/s]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/json": { "ascii": false, "bar_format": null, "colour": null, "elapsed": 0.010919332504272461, "initial": 0, "n": 0, "ncols": null, "nrows": null, "postfix": null, "prefix": "Downloading (…)on_pytorch_model.bin", "rate": null, "total": 1719312805, "unit": "B", "unit_divisor": 1000, "unit_scale": true }, "application/vnd.jupyter.widget-view+json": { "model_id": "5be75dfa574c452d89754c6bc04c98c7", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Downloading (…)on_pytorch_model.bin: 0%| | 0.00/1.72G [00:00, ?B/s]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/json": { "ascii": false, "bar_format": null, "colour": null, "elapsed": 0.006277561187744141, "initial": 0, "n": 0, "ncols": null, "nrows": null, "postfix": null, "prefix": "Downloading (…)on_pytorch_model.bin", "rate": null, "total": 167399505, "unit": "B", "unit_divisor": 1000, "unit_scale": true }, "application/vnd.jupyter.widget-view+json": { "model_id": "c8e96737b88a4902926e819e2e76fdae", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Downloading (…)on_pytorch_model.bin: 0%| | 0.00/167M [00:00, ?B/s]" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config[\"id2label\"]` will be overriden.\n", "/home/ec2-user/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages/transformers/models/clip/feature_extraction_clip.py:28: FutureWarning: The class CLIPFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use CLIPImageProcessor instead.\n", " warnings.warn(\n" ] } ], "source": [ "import diffusers\n", "import torch \n", "\n", "pipeline = diffusers.StableDiffusionPipeline.from_pretrained(\"CompVis/stable-diffusion-v1-4\",\n", " cache_dir='hf_cache',\n", " torch_dtype=torch.float16,\n", " revision=\"fp16\")" ] }, { "cell_type": "code", "execution_count": 4, "id": "0cd50657-e4e9-4532-9763-473d1a0d9d38", "metadata": {}, "outputs": [], "source": [ "pipeline.save_pretrained('model_repo_0/pipeline_0/stable_diff')" ] }, { "cell_type": "markdown", "id": "95d58e0e", "metadata": {}, "source": [ "### Part 3 - Packaging a conda environment, extending Sagemaker Triton container " ] }, { "cell_type": "markdown", "id": "8973a7d2", "metadata": {}, "source": [ "When using the Triton Python backend (which our Stable Diffusion model will run on), you can include your own environment and dependencies. The recommended way to do this is to use [conda pack](https://conda.github.io/conda-pack/) to generate a conda environment archive in `tar.gz` format, and point to it in the `config.pbtxt` file of the models that should use it, adding the snippet: \n", "\n", "```\n", "parameters: {\n", " key: \"EXECUTION_ENV_PATH\",\n", " value: {string_value: \"path_to_your_env.tar.gz\"}\n", "}\n", "\n", "```\n", "You can use a different environment per model, or the same for all models (read more on this [here](https://github.com/triton-inference-server/python_backend#creating-custom-execution-environments)). Since we are hosting many versions of the same model, only perhaps personalized (fine-tuned) for different end-users, we don't want the overhead of including a different environment for each one, which would also increase the model S3 download and GPU load times. \n", "\n", "Therefore, we will create the packaged environment once, and include it directly in our container image, where all hosted models can access it. In order to do this, we will extend the public SageMaker Triton container image. \n", "\n", "Let's start by creating the conda environment with the necessary dependencies; this script will output a `stablediff_env.tar.gz` file." ] }, { "cell_type": "code", "execution_count": 5, "id": "5f523d03", "metadata": { "collapsed": true, "jupyter": { "outputs_hidden": true }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Collecting package metadata (current_repodata.json): done\n", "Solving environment: done\n", "\n", "\n", "==> WARNING: A newer version of conda exists. <==\n", " current version: 22.9.0\n", " latest version: 23.1.0\n", "\n", "Please update conda by running\n", "\n", " $ conda update -n base -c conda-forge conda\n", "\n", "\n", "\n", "## Package Plan ##\n", "\n", " environment location: /home/ec2-user/anaconda3/envs/stablediff_env\n", "\n", " added / updated specs:\n", " - python=3.8\n", "\n", "\n", "The following packages will be downloaded:\n", "\n", " package | build\n", " ---------------------------|-----------------\n", " openssl-3.1.0 | h0b41bf4_0 2.5 MB conda-forge\n", " pip-23.0.1 | pyhd8ed1ab_0 1.3 MB conda-forge\n", " python-3.8.16 |he550d4f_1_cpython 21.8 MB conda-forge\n", " setuptools-67.6.0 | pyhd8ed1ab_0 566 KB conda-forge\n", " wheel-0.40.0 | pyhd8ed1ab_0 54 KB conda-forge\n", " ------------------------------------------------------------\n", " Total: 26.2 MB\n", "\n", "The following NEW packages will be INSTALLED:\n", "\n", " _libgcc_mutex conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge None\n", " _openmp_mutex conda-forge/linux-64::_openmp_mutex-4.5-2_gnu None\n", " bzip2 conda-forge/linux-64::bzip2-1.0.8-h7f98852_4 None\n", " ca-certificates conda-forge/linux-64::ca-certificates-2022.12.7-ha878542_0 None\n", " ld_impl_linux-64 conda-forge/linux-64::ld_impl_linux-64-2.40-h41732ed_0 None\n", " libffi conda-forge/linux-64::libffi-3.4.2-h7f98852_5 None\n", " libgcc-ng conda-forge/linux-64::libgcc-ng-12.2.0-h65d4601_19 None\n", " libgomp conda-forge/linux-64::libgomp-12.2.0-h65d4601_19 None\n", " libnsl conda-forge/linux-64::libnsl-2.0.0-h7f98852_0 None\n", " libsqlite conda-forge/linux-64::libsqlite-3.40.0-h753d276_0 None\n", " libuuid conda-forge/linux-64::libuuid-2.32.1-h7f98852_1000 None\n", " libzlib conda-forge/linux-64::libzlib-1.2.13-h166bdaf_4 None\n", " ncurses conda-forge/linux-64::ncurses-6.3-h27087fc_1 None\n", " openssl conda-forge/linux-64::openssl-3.1.0-h0b41bf4_0 None\n", " pip conda-forge/noarch::pip-23.0.1-pyhd8ed1ab_0 None\n", " python conda-forge/linux-64::python-3.8.16-he550d4f_1_cpython None\n", " readline conda-forge/linux-64::readline-8.1.2-h0f457ee_0 None\n", " setuptools conda-forge/noarch::setuptools-67.6.0-pyhd8ed1ab_0 None\n", " tk conda-forge/linux-64::tk-8.6.12-h27826a3_0 None\n", " wheel conda-forge/noarch::wheel-0.40.0-pyhd8ed1ab_0 None\n", " xz conda-forge/linux-64::xz-5.2.6-h166bdaf_0 None\n", "\n", "\n", "\n", "Downloading and Extracting Packages\n", "pip-23.0.1 | 1.3 MB | ##################################### | 100% \n", "wheel-0.40.0 | 54 KB | ##################################### | 100% \n", "python-3.8.16 | 21.8 MB | ##################################### | 100% \n", "setuptools-67.6.0 | 566 KB | ##################################### | 100% \n", "openssl-3.1.0 | 2.5 MB | ##################################### | 100% \n", "Preparing transaction: done\n", "Verifying transaction: done\n", "Executing transaction: done\n", "#\n", "# To activate this environment, use\n", "#\n", "# $ conda activate stablediff_env\n", "#\n", "# To deactivate an active environment, use\n", "#\n", "# $ conda deactivate\n", "\n", "Retrieving notices: ...working... done\n", "Looking in indexes: https://pypi.org/simple, https://pip.repos.neuron.amazonaws.com, https://pypi.ngc.nvidia.com\n", "Collecting torch\n", " Downloading torch-2.0.0-cp38-cp38-manylinux1_x86_64.whl (619.9 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m619.9/619.9 MB\u001b[0m \u001b[31m201.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n", "\u001b[?25hCollecting typing-extensions\n", " Downloading typing_extensions-4.5.0-py3-none-any.whl (27 kB)\n", "Collecting nvidia-cuda-runtime-cu11==11.7.99\n", " Downloading nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl (849 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m849.3/849.3 kB\u001b[0m \u001b[31m428.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting nvidia-cufft-cu11==10.9.0.58\n", " Downloading nvidia_cufft_cu11-10.9.0.58-py3-none-manylinux1_x86_64.whl (168.4 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m168.4/168.4 MB\u001b[0m \u001b[31m202.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n", "\u001b[?25hCollecting nvidia-cudnn-cu11==8.5.0.96\n", " Downloading nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl (557.1 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m557.1/557.1 MB\u001b[0m \u001b[31m175.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n", "\u001b[?25hCollecting networkx\n", " Downloading networkx-3.0-py3-none-any.whl (2.0 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.0/2.0 MB\u001b[0m \u001b[31m257.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting nvidia-cuda-nvrtc-cu11==11.7.99\n", " Downloading nvidia_cuda_nvrtc_cu11-11.7.99-2-py3-none-manylinux1_x86_64.whl (21.0 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m21.0/21.0 MB\u001b[0m \u001b[31m201.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n", "\u001b[?25hCollecting nvidia-cuda-cupti-cu11==11.7.101\n", " Downloading nvidia_cuda_cupti_cu11-11.7.101-py3-none-manylinux1_x86_64.whl (11.8 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m11.8/11.8 MB\u001b[0m \u001b[31m179.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n", "\u001b[?25hCollecting nvidia-cublas-cu11==11.10.3.66\n", " Downloading nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl (317.1 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m317.1/317.1 MB\u001b[0m \u001b[31m133.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n", "\u001b[?25hCollecting nvidia-cusolver-cu11==11.4.0.1\n", " Downloading nvidia_cusolver_cu11-11.4.0.1-2-py3-none-manylinux1_x86_64.whl (102.6 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m102.6/102.6 MB\u001b[0m \u001b[31m234.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n", "\u001b[?25hCollecting nvidia-cusparse-cu11==11.7.4.91\n", " Downloading nvidia_cusparse_cu11-11.7.4.91-py3-none-manylinux1_x86_64.whl (173.2 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m173.2/173.2 MB\u001b[0m \u001b[31m168.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n", "\u001b[?25hCollecting triton==2.0.0\n", " Downloading triton-2.0.0-1-cp38-cp38-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (63.2 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m63.2/63.2 MB\u001b[0m \u001b[31m216.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n", "\u001b[?25hCollecting nvidia-nccl-cu11==2.14.3\n", " Downloading nvidia_nccl_cu11-2.14.3-py3-none-manylinux1_x86_64.whl (177.1 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m177.1/177.1 MB\u001b[0m \u001b[31m191.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n", "\u001b[?25hCollecting sympy\n", " Downloading sympy-1.11.1-py3-none-any.whl (6.5 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.5/6.5 MB\u001b[0m \u001b[31m222.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting nvidia-nvtx-cu11==11.7.91\n", " Downloading nvidia_nvtx_cu11-11.7.91-py3-none-manylinux1_x86_64.whl (98 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m98.6/98.6 kB\u001b[0m \u001b[31m356.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting nvidia-curand-cu11==10.2.10.91\n", " Downloading nvidia_curand_cu11-10.2.10.91-py3-none-manylinux1_x86_64.whl (54.6 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m54.6/54.6 MB\u001b[0m \u001b[31m215.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n", "\u001b[?25hCollecting filelock\n", " Downloading filelock-3.10.0-py3-none-any.whl (9.9 kB)\n", "Collecting jinja2\n", " Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m133.1/133.1 kB\u001b[0m \u001b[31m357.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hRequirement already satisfied: setuptools in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch) (67.6.0)\n", "Requirement already satisfied: wheel in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch) (0.40.0)\n", "Collecting lit\n", " Downloading lit-16.0.0.tar.gz (144 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m145.0/145.0 kB\u001b[0m \u001b[31m335.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25h Preparing metadata (setup.py) ... \u001b[?25ldone\n", "\u001b[?25hCollecting cmake\n", " Downloading cmake-3.26.0-py2.py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (24.0 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m24.0/24.0 MB\u001b[0m \u001b[31m233.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n", "\u001b[?25hCollecting MarkupSafe>=2.0\n", " Downloading MarkupSafe-2.1.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)\n", "Collecting mpmath>=0.19\n", " Downloading mpmath-1.3.0-py3-none-any.whl (536 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m536.2/536.2 kB\u001b[0m \u001b[31m396.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hBuilding wheels for collected packages: lit\n", " Building wheel for lit (setup.py) ... \u001b[?25ldone\n", "\u001b[?25h Created wheel for lit: filename=lit-16.0.0-py3-none-any.whl size=93586 sha256=d2796febc55a97d95ae3f9860523ed2662a2300f278677b68a367fd007b15e34\n", " Stored in directory: /tmp/pip-ephem-wheel-cache-fa9s6ykl/wheels/30/c8/c0/9d24f63f30f4879b8efb718a11b2191ef5c8510259d1d6c392\n", "Successfully built lit\n", "Installing collected packages: mpmath, lit, cmake, typing-extensions, sympy, nvidia-nvtx-cu11, nvidia-nccl-cu11, nvidia-cusparse-cu11, nvidia-curand-cu11, nvidia-cufft-cu11, nvidia-cuda-runtime-cu11, nvidia-cuda-nvrtc-cu11, nvidia-cuda-cupti-cu11, nvidia-cublas-cu11, networkx, MarkupSafe, filelock, nvidia-cusolver-cu11, nvidia-cudnn-cu11, jinja2, triton, torch\n", "Successfully installed MarkupSafe-2.1.2 cmake-3.26.0 filelock-3.10.0 jinja2-3.1.2 lit-16.0.0 mpmath-1.3.0 networkx-3.0 nvidia-cublas-cu11-11.10.3.66 nvidia-cuda-cupti-cu11-11.7.101 nvidia-cuda-nvrtc-cu11-11.7.99 nvidia-cuda-runtime-cu11-11.7.99 nvidia-cudnn-cu11-8.5.0.96 nvidia-cufft-cu11-10.9.0.58 nvidia-curand-cu11-10.2.10.91 nvidia-cusolver-cu11-11.4.0.1 nvidia-cusparse-cu11-11.7.4.91 nvidia-nccl-cu11-2.14.3 nvidia-nvtx-cu11-11.7.91 sympy-1.11.1 torch-2.0.0 triton-2.0.0 typing-extensions-4.5.0\n", "Looking in indexes: https://pypi.org/simple, https://pip.repos.neuron.amazonaws.com, https://pypi.ngc.nvidia.com\n", "Collecting transformers\n", " Downloading transformers-4.27.2-py3-none-any.whl (6.8 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.8/6.8 MB\u001b[0m \u001b[31m204.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting ftfy\n", " Downloading ftfy-6.1.1-py3-none-any.whl (53 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m53.1/53.1 kB\u001b[0m \u001b[31m262.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting scipy\n", " Downloading scipy-1.10.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m34.5/34.5 MB\u001b[0m \u001b[31m263.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n", "\u001b[?25hCollecting accelerate\n", " Downloading accelerate-0.17.1-py3-none-any.whl (212 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m212.8/212.8 kB\u001b[0m \u001b[31m389.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting packaging>=20.0\n", " Downloading packaging-23.0-py3-none-any.whl (42 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m42.7/42.7 kB\u001b[0m \u001b[31m247.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting tqdm>=4.27\n", " Downloading tqdm-4.65.0-py3-none-any.whl (77 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m77.1/77.1 kB\u001b[0m \u001b[31m306.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting tokenizers!=0.11.3,<0.14,>=0.11.1\n", " Downloading tokenizers-0.13.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.6 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.6/7.6 MB\u001b[0m \u001b[31m311.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting requests\n", " Downloading requests-2.28.2-py3-none-any.whl (62 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.8/62.8 kB\u001b[0m \u001b[31m293.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting numpy>=1.17\n", " Downloading numpy-1.24.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m17.3/17.3 MB\u001b[0m \u001b[31m351.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n", "\u001b[?25hCollecting regex!=2019.12.17\n", " Downloading regex-2022.10.31-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (772 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m772.3/772.3 kB\u001b[0m \u001b[31m421.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting huggingface-hub<1.0,>=0.11.0\n", " Downloading huggingface_hub-0.13.3-py3-none-any.whl (199 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m199.8/199.8 kB\u001b[0m \u001b[31m382.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting pyyaml>=5.1\n", " Downloading PyYAML-6.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (701 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m701.2/701.2 kB\u001b[0m \u001b[31m413.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hRequirement already satisfied: filelock in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from transformers) (3.10.0)\n", "Collecting wcwidth>=0.2.5\n", " Downloading wcwidth-0.2.6-py2.py3-none-any.whl (29 kB)\n", "Requirement already satisfied: torch>=1.4.0 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from accelerate) (2.0.0)\n", "Collecting psutil\n", " Downloading psutil-5.9.4-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (280 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m280.2/280.2 kB\u001b[0m \u001b[31m397.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hRequirement already satisfied: typing-extensions>=3.7.4.3 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from huggingface-hub<1.0,>=0.11.0->transformers) (4.5.0)\n", "Requirement already satisfied: nvidia-cusolver-cu11==11.4.0.1 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from torch>=1.4.0->accelerate) (11.4.0.1)\n", "Requirement already satisfied: nvidia-cublas-cu11==11.10.3.66 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from torch>=1.4.0->accelerate) (11.10.3.66)\n", "Requirement already satisfied: nvidia-cuda-nvrtc-cu11==11.7.99 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from torch>=1.4.0->accelerate) (11.7.99)\n", "Requirement already satisfied: nvidia-cuda-cupti-cu11==11.7.101 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from torch>=1.4.0->accelerate) (11.7.101)\n", "Requirement already satisfied: nvidia-cufft-cu11==10.9.0.58 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from torch>=1.4.0->accelerate) (10.9.0.58)\n", "Requirement already satisfied: nvidia-cuda-runtime-cu11==11.7.99 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from torch>=1.4.0->accelerate) (11.7.99)\n", "Requirement already satisfied: sympy in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from torch>=1.4.0->accelerate) (1.11.1)\n", "Requirement already satisfied: jinja2 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from torch>=1.4.0->accelerate) (3.1.2)\n", "Requirement already satisfied: networkx in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from torch>=1.4.0->accelerate) (3.0)\n", "Requirement already satisfied: nvidia-cusparse-cu11==11.7.4.91 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from torch>=1.4.0->accelerate) (11.7.4.91)\n", "Requirement already satisfied: nvidia-nvtx-cu11==11.7.91 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from torch>=1.4.0->accelerate) (11.7.91)\n", "Requirement already satisfied: nvidia-cudnn-cu11==8.5.0.96 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from torch>=1.4.0->accelerate) (8.5.0.96)\n", "Requirement already satisfied: nvidia-nccl-cu11==2.14.3 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from torch>=1.4.0->accelerate) (2.14.3)\n", "Requirement already satisfied: triton==2.0.0 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from torch>=1.4.0->accelerate) (2.0.0)\n", "Requirement already satisfied: nvidia-curand-cu11==10.2.10.91 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from torch>=1.4.0->accelerate) (10.2.10.91)\n", "Requirement already satisfied: setuptools in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.4.0->accelerate) (67.6.0)\n", "Requirement already satisfied: wheel in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.4.0->accelerate) (0.40.0)\n", "Requirement already satisfied: cmake in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from triton==2.0.0->torch>=1.4.0->accelerate) (3.26.0)\n", "Requirement already satisfied: lit in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from triton==2.0.0->torch>=1.4.0->accelerate) (16.0.0)\n", "Collecting certifi>=2017.4.17\n", " Downloading certifi-2022.12.7-py3-none-any.whl (155 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m155.3/155.3 kB\u001b[0m \u001b[31m380.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting urllib3<1.27,>=1.21.1\n", " Downloading urllib3-1.26.15-py2.py3-none-any.whl (140 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m140.9/140.9 kB\u001b[0m \u001b[31m374.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting idna<4,>=2.5\n", " Downloading idna-3.4-py3-none-any.whl (61 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m61.5/61.5 kB\u001b[0m \u001b[31m320.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting charset-normalizer<4,>=2\n", " Downloading charset_normalizer-3.1.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (195 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m195.9/195.9 kB\u001b[0m \u001b[31m380.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hRequirement already satisfied: MarkupSafe>=2.0 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from jinja2->torch>=1.4.0->accelerate) (2.1.2)\n", "Requirement already satisfied: mpmath>=0.19 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from sympy->torch>=1.4.0->accelerate) (1.3.0)\n", "Installing collected packages: wcwidth, tokenizers, urllib3, tqdm, regex, pyyaml, psutil, packaging, numpy, idna, ftfy, charset-normalizer, certifi, scipy, requests, huggingface-hub, transformers, accelerate\n", "Successfully installed accelerate-0.17.1 certifi-2022.12.7 charset-normalizer-3.1.0 ftfy-6.1.1 huggingface-hub-0.13.3 idna-3.4 numpy-1.24.2 packaging-23.0 psutil-5.9.4 pyyaml-6.0 regex-2022.10.31 requests-2.28.2 scipy-1.10.1 tokenizers-0.13.2 tqdm-4.65.0 transformers-4.27.2 urllib3-1.26.15 wcwidth-0.2.6\n", "Looking in indexes: https://pypi.org/simple, https://pip.repos.neuron.amazonaws.com, https://pypi.ngc.nvidia.com\n", "Collecting diffusers==0.9.0\n", " Downloading diffusers-0.9.0-py3-none-any.whl (453 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m453.1/453.1 kB\u001b[0m \u001b[31m60.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hRequirement already satisfied: filelock in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from diffusers==0.9.0) (3.10.0)\n", "Collecting importlib-metadata\n", " Downloading importlib_metadata-6.1.0-py3-none-any.whl (21 kB)\n", "Requirement already satisfied: regex!=2019.12.17 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from diffusers==0.9.0) (2022.10.31)\n", "Requirement already satisfied: numpy in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from diffusers==0.9.0) (1.24.2)\n", "Requirement already satisfied: huggingface-hub>=0.10.0 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from diffusers==0.9.0) (0.13.3)\n", "Requirement already satisfied: requests in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from diffusers==0.9.0) (2.28.2)\n", "Collecting Pillow\n", " Downloading Pillow-9.4.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.3 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.3/3.3 MB\u001b[0m \u001b[31m325.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hRequirement already satisfied: pyyaml>=5.1 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from huggingface-hub>=0.10.0->diffusers==0.9.0) (6.0)\n", "Requirement already satisfied: packaging>=20.9 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from huggingface-hub>=0.10.0->diffusers==0.9.0) (23.0)\n", "Requirement already satisfied: typing-extensions>=3.7.4.3 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from huggingface-hub>=0.10.0->diffusers==0.9.0) (4.5.0)\n", "Requirement already satisfied: tqdm>=4.42.1 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from huggingface-hub>=0.10.0->diffusers==0.9.0) (4.65.0)\n", "Collecting zipp>=0.5\n", " Downloading zipp-3.15.0-py3-none-any.whl (6.8 kB)\n", "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from requests->diffusers==0.9.0) (1.26.15)\n", "Requirement already satisfied: idna<4,>=2.5 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from requests->diffusers==0.9.0) (3.4)\n", "Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from requests->diffusers==0.9.0) (2022.12.7)\n", "Requirement already satisfied: charset-normalizer<4,>=2 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from requests->diffusers==0.9.0) (3.1.0)\n", "Installing collected packages: zipp, Pillow, importlib-metadata, diffusers\n", "Successfully installed Pillow-9.4.0 diffusers-0.9.0 importlib-metadata-6.1.0 zipp-3.15.0\n", "Looking in indexes: https://pypi.org/simple, https://pip.repos.neuron.amazonaws.com, https://pypi.ngc.nvidia.com\n", "Requirement already satisfied: transformers[onnxruntime] in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (4.27.2)\n", "Requirement already satisfied: requests in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from transformers[onnxruntime]) (2.28.2)\n", "Requirement already satisfied: huggingface-hub<1.0,>=0.11.0 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from transformers[onnxruntime]) (0.13.3)\n", "Requirement already satisfied: regex!=2019.12.17 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from transformers[onnxruntime]) (2022.10.31)\n", "Requirement already satisfied: tqdm>=4.27 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from transformers[onnxruntime]) (4.65.0)\n", "Requirement already satisfied: numpy>=1.17 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from transformers[onnxruntime]) (1.24.2)\n", "Requirement already satisfied: pyyaml>=5.1 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from transformers[onnxruntime]) (6.0)\n", "Requirement already satisfied: filelock in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from transformers[onnxruntime]) (3.10.0)\n", "Requirement already satisfied: packaging>=20.0 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from transformers[onnxruntime]) (23.0)\n", "Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from transformers[onnxruntime]) (0.13.2)\n", "Collecting onnxruntime-tools>=1.4.2\n", " Downloading onnxruntime_tools-1.7.0-py3-none-any.whl (212 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m212.7/212.7 kB\u001b[0m \u001b[31m33.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting onnxruntime>=1.4.0\n", " Downloading onnxruntime-1.14.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.7 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.7/5.7 MB\u001b[0m \u001b[31m262.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hRequirement already satisfied: typing-extensions>=3.7.4.3 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from huggingface-hub<1.0,>=0.11.0->transformers[onnxruntime]) (4.5.0)\n", "Collecting coloredlogs\n", " Downloading coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m46.0/46.0 kB\u001b[0m \u001b[31m261.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hRequirement already satisfied: sympy in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from onnxruntime>=1.4.0->transformers[onnxruntime]) (1.11.1)\n", "Collecting flatbuffers\n", " Downloading flatbuffers-23.3.3-py2.py3-none-any.whl (26 kB)\n", "Collecting protobuf\n", " Downloading protobuf-4.22.1-cp37-abi3-manylinux2014_x86_64.whl (302 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m302.4/302.4 kB\u001b[0m \u001b[31m406.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting onnx\n", " Downloading onnx-1.13.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.5 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m13.5/13.5 MB\u001b[0m \u001b[31m207.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n", "\u001b[?25hRequirement already satisfied: psutil in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from onnxruntime-tools>=1.4.2->transformers[onnxruntime]) (5.9.4)\n", "Collecting py3nvml\n", " Downloading py3nvml-0.2.7-py3-none-any.whl (55 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m55.5/55.5 kB\u001b[0m \u001b[31m288.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting py-cpuinfo\n", " Downloading py_cpuinfo-9.0.0-py3-none-any.whl (22 kB)\n", "Requirement already satisfied: charset-normalizer<4,>=2 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from requests->transformers[onnxruntime]) (3.1.0)\n", "Requirement already satisfied: idna<4,>=2.5 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from requests->transformers[onnxruntime]) (3.4)\n", "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from requests->transformers[onnxruntime]) (1.26.15)\n", "Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from requests->transformers[onnxruntime]) (2022.12.7)\n", "Collecting humanfriendly>=9.1\n", " Downloading humanfriendly-10.0-py2.py3-none-any.whl (86 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m86.8/86.8 kB\u001b[0m \u001b[31m314.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting protobuf\n", " Downloading protobuf-3.20.3-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.0/1.0 MB\u001b[0m \u001b[31m419.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting xmltodict\n", " Downloading xmltodict-0.13.0-py2.py3-none-any.whl (10.0 kB)\n", "Requirement already satisfied: mpmath>=0.19 in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from sympy->onnxruntime>=1.4.0->transformers[onnxruntime]) (1.3.0)\n", "Installing collected packages: py-cpuinfo, flatbuffers, xmltodict, protobuf, humanfriendly, py3nvml, onnx, coloredlogs, onnxruntime-tools, onnxruntime\n", "Successfully installed coloredlogs-15.0.1 flatbuffers-23.3.3 humanfriendly-10.0 onnx-1.13.1 onnxruntime-1.14.1 onnxruntime-tools-1.7.0 protobuf-3.20.3 py-cpuinfo-9.0.0 py3nvml-0.2.7 xmltodict-0.13.0\n", "Looking in indexes: https://pypi.org/simple, https://pip.repos.neuron.amazonaws.com, https://pypi.ngc.nvidia.com\n", "Collecting conda-pack\n", " Downloading conda-pack-0.6.0.tar.gz (43 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m43.2/43.2 kB\u001b[0m \u001b[31m20.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25h Preparing metadata (setup.py) ... \u001b[?25ldone\n", "\u001b[?25hRequirement already satisfied: setuptools in /home/ec2-user/anaconda3/envs/stablediff_env/lib/python3.8/site-packages (from conda-pack) (67.6.0)\n", "Building wheels for collected packages: conda-pack\n", " Building wheel for conda-pack (setup.py) ... \u001b[?25ldone\n", "\u001b[?25h Created wheel for conda-pack: filename=conda_pack-0.6.0-py2.py3-none-any.whl size=30883 sha256=3f0e3e26da8675aa44912ffec3586d112dc67d28f535a7d6d9b634fb06488940\n", " Stored in directory: /tmp/pip-ephem-wheel-cache-p_rd2z3g/wheels/56/1b/9e/0da27a4c18349d8f048a8fe87d763d75d3098384e9fa285e45\n", "Successfully built conda-pack\n", "Installing collected packages: conda-pack\n", "Successfully installed conda-pack-0.6.0\n", "Collecting packages...\n", "Packing environment at '/home/ec2-user/anaconda3/envs/stablediff_env' to 'stablediff_env.tar.gz'\n", "[########################################] | 100% Completed | 3min 7.9s\n" ] } ], "source": [ "!cd docker && bash conda_dependencies.sh " ] }, { "cell_type": "markdown", "id": "551fa2a6-3562-49aa-88bc-e35c2cf75c79", "metadata": {}, "source": [ "Now, we get the correct URI for the SageMaker Triton container image. Check out all the available Deep Learning Container images that AWS maintains [here](https://github.com/aws/deep-learning-containers/blob/master/available_images.md). " ] }, { "cell_type": "code", "execution_count": 15, "id": "b9b707c2-f78e-452a-9e99-4860232bd76b", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "785573368785.dkr.ecr.us-east-1.amazonaws.com/sagemaker-tritonserver:22.12-py3\n" ] } ], "source": [ "# account mapping for SageMaker Triton Image\n", "account_id_map = {\n", " \"us-east-1\": \"785573368785\",\n", " \"us-east-2\": \"007439368137\",\n", " \"us-west-1\": \"710691900526\",\n", " \"us-west-2\": \"301217895009\",\n", " \"eu-west-1\": \"802834080501\",\n", " \"eu-west-2\": \"205493899709\",\n", " \"eu-west-3\": \"254080097072\",\n", " \"eu-north-1\": \"601324751636\",\n", " \"eu-south-1\": \"966458181534\",\n", " \"eu-central-1\": \"746233611703\",\n", " \"ap-east-1\": \"110948597952\",\n", " \"ap-south-1\": \"763008648453\",\n", " \"ap-northeast-1\": \"941853720454\",\n", " \"ap-northeast-2\": \"151534178276\",\n", " \"ap-southeast-1\": \"324986816169\",\n", " \"ap-southeast-2\": \"355873309152\",\n", " \"cn-northwest-1\": \"474822919863\",\n", " \"cn-north-1\": \"472730292857\",\n", " \"sa-east-1\": \"756306329178\",\n", " \"ca-central-1\": \"464438896020\",\n", " \"me-south-1\": \"836785723513\",\n", " \"af-south-1\": \"774647643957\",\n", "}\n", "\n", "\n", "\n", "region = boto3.Session().region_name\n", "if region not in account_id_map.keys():\n", " raise (\"UNSUPPORTED REGION\")\n", "\n", "base = \"amazonaws.com.cn\" if region.startswith(\"cn-\") else \"amazonaws.com\"\n", "\n", "\n", "triton_image_uri = (\n", " \"{account_id}.dkr.ecr.{region}.{base}/sagemaker-tritonserver:22.12-py3\".format(\n", " account_id=account_id_map[region], region=region, base=base\n", " )\n", ")\n", "\n", "print(triton_image_uri)" ] }, { "cell_type": "code", "execution_count": 21, "id": "33a5d2e1", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "us-east-1\n" ] } ], "source": [ "print(region)" ] }, { "cell_type": "markdown", "id": "d498436a-9ee9-46aa-b7f7-e23ea073f710", "metadata": {}, "source": [ "We then build our extended image, which does nothing more than to copy the packaged environment into the container. Let's check out the Dockerfile." ] }, { "cell_type": "code", "execution_count": 16, "id": "9f77e62f-f7bb-4e65-9f8e-07f037a3ccfe", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "ARG BASE_IMAGE\n", "\n", "FROM $BASE_IMAGE\n", "\n", "#Install any additional libraries\n", "RUN echo \"Adding conda package to Docker image\"\n", "RUN mkdir -p /home/condpackenv/\n", "\n", "# Copy conda env\n", "COPY stablediff_env.tar.gz /home/condpackenv/stablediff_env.tar.gz" ] } ], "source": [ "!cat docker/Dockerfile" ] }, { "cell_type": "code", "execution_count": 18, "id": "f90ec85f-a0ff-4174-9cca-c06105d3055a", "metadata": {}, "outputs": [], "source": [ "# Change this var to change the name of new container image\n", "new_image_name = 'sagemaker-tritonserver-byoc'" ] }, { "cell_type": "markdown", "id": "a75dff8c-54fd-4f2c-b18e-ee5440aa0978", "metadata": {}, "source": [ "We catch the docker build process' output so that we can easily capture the output container image URI, and check for build errors." ] }, { "cell_type": "code", "execution_count": 22, "id": "348f5d52-3a04-48b3-82d7-8138293f5a6f", "metadata": {}, "outputs": [], "source": [ "%%capture build_output\n", "!cd docker && bash build_and_push.sh \"$new_image_name\" 22.12 \"$triton_image_uri\" \"$region\"" ] }, { "cell_type": "code", "execution_count": 23, "id": "3bfea1b4-7056-40a9-959c-ada1a7b325d9", "metadata": {}, "outputs": [], "source": [ "if 'Error response from daemon' in str(build_output):\n", " print(build_output)\n", " raise SystemExit('\\n\\n!!There was an error with the container build!!')\n", "else:\n", " extended_triton_image_uri = str(build_output).strip().split('\\n')[-1]" ] }, { "cell_type": "markdown", "id": "3a28fb2c-09a5-40a1-bf29-076d8e6fe18b", "metadata": {}, "source": [ "If the previous cell failed, check the docker build logs to understand the error problem, and read the possible resolution in the next cell" ] }, { "cell_type": "code", "execution_count": 11, "id": "289313a4-ac2b-45b9-89ec-efa0f1de81e8", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.\n", "Configure a credential helper to remove this warning. See\n", "https://docs.docker.com/engine/reference/commandline/login/#credentials-store\n", "\n", "Login Succeeded\n" ] } ], "source": [ "\"\"\"\n", "If the cell above fails (check out the build_output) because of missing permissions to pull the public Triton base container image,\n", "uncomment the commands in this cell, run them and retry the build\n", "\"\"\"\n", "mapped_region_account = account_id_map[region]\n", "!aws ecr get-login-password --region \"$region\" | docker login --username AWS --password-stdin \"$mapped_region_account\".dkr.ecr.\"$region\".amazonaws.com" ] }, { "cell_type": "markdown", "id": "b0c5d92f", "metadata": {}, "source": [ "----\n", "----" ] }, { "cell_type": "markdown", "id": "7b63de28", "metadata": { "jp-MarkdownHeadingCollapsed": true, "tags": [] }, "source": [ "### Part 4 - Local testing of Triton model repository " ] }, { "cell_type": "markdown", "id": "ab3e6815", "metadata": {}, "source": [ "Now you can test the model repository and validate it is working. Let's run the Triton docker container locally and invoke the model to check this." ] }, { "cell_type": "code", "execution_count": null, "id": "9b971890", "metadata": {}, "outputs": [], "source": [ "repo_name = \"model_repo_0\"" ] }, { "cell_type": "markdown", "id": "7fce2bd8-376a-4b44-b2aa-8415b45205e7", "metadata": {}, "source": [ "We are running the Triton container in detached model with the `-d` flag so that it runs in the background. " ] }, { "cell_type": "code", "execution_count": null, "id": "4c92b181", "metadata": {}, "outputs": [], "source": [ "!docker run --gpus=all -d --shm-size=4G --rm -p8000:8000 -p8001:8001 -p8002:8002 -v$(pwd)/$repo_name:/model_repository $extended_triton_image_uri tritonserver --model-repository=/model_repository --exit-on-error=false\n", "time.sleep(90)" ] }, { "cell_type": "code", "execution_count": null, "id": "1b90d62b", "metadata": {}, "outputs": [], "source": [ "CONTAINER_ID=!docker container ls -q\n", "FIRST_CONTAINER_ID = CONTAINER_ID[0]" ] }, { "cell_type": "code", "execution_count": null, "id": "30bbf319", "metadata": {}, "outputs": [], "source": [ "!echo $FIRST_CONTAINER_ID" ] }, { "cell_type": "code", "execution_count": null, "id": "77fc4caf", "metadata": {}, "outputs": [], "source": [ "!docker logs $FIRST_CONTAINER_ID" ] }, { "cell_type": "markdown", "id": "c580605b", "metadata": {}, "source": [ "