{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Create and Query ML Lineage between SageMaker - Models, Inference Endpoints, Feature Store, Processing Jobs and Datasources\n", "\n", "---\n", "\n", "#### Note: Please set kernel to Python 3 (Data Science) and select instance to ml.t3.medium\n", "\n", "\n", "
πŸ’‘ Quick Start \n", "ML Lineage racking from datasource to model endpoint, The challenge of reproducibility and lineage in machine learning (ML) is three-fold: code lineage, data lineage, and model lineage. Source version control is a standard for managing changes to code. For data lineage, most data storage services support versioning, which gives you the ability to track datasets at a given point in time. Model lineage combines code lineage, data lineage, and ML-specific information such as Docker containers used for training and deployment, model hyperparameters, and more.\n", "\n", "\n", " Click here for a comprehensive ML lineage concepts\n", "\n", "
\n", "\n", "Feature engineering is expensive and time-consuming, leading customers to adopt a feature store\n", "for managing features across teams and models. Unfortunately, ML lineage solutions have yet to\n", "adapt to this new concept of feature management. To achieve the full benefits of feature reuse,\n", "customers need to be able to answer fundamental questions about features. For example, how\n", "was this feature group built? What models are using this feature group? What features does my\n", "model depend on? What features are built with this data source?\n", "\n", "---\n", "\n", "Amazon SageMaker ML Lineage Tracking creates and stores information about the steps of a machine learning (ML) workflow from data preparation to model deployment. With the tracking information you can reproduce the workflow steps, track model and dataset lineage, and establish model governance and audit standards. \n", "\n", "\n", " Read about ML Lineage tracking with Feature store\n", "\n", "\n", "#### With SageMaker Lineage Tracking data and feature store, scientists and builders can do the following:\n", "---\n", "##### 1. Build confidence for reuse of existing features.\n", "\n", "##### 2. Avoid re-inventing features that are based on the same raw data as existing features.\n", "\n", "##### 3. Troubleshooting and auditing models and model predictions.\n", "\n", "##### 4. Manage features proactively.\n", "\n", "---\n", "\n", "## Contents\n", "\n", "1. [Notebook Preparation](#Notebook-Preparation)\n", " 1. [Imports](#Imports)\n", " 1. [Check git submodules](#Check-git-submodules)\n", " 1. [Check and update Sagemaker version](#Check-and-update-Sagemaker-version)\n", " 1. [Logging Settings](#Logging-Settings)\n", " 1. [Module Configurations](#Module-Configurations)\n", " 1. [Load peristed variables from previous modules](#Load-peristed-variables-from-previous-modules)\n", "1. [ML Lineage Creation](#ML-Lineage-Creation) \n", " 1. [Create ML Lineage](#Create-ML-Lineage)\n", " 1. [Verify ML Lineage](#Verify-ML-Lineage)\n", " 1. [ML Lineage Graph](#ML-Lineage-Graph)\n", "1. [ML Lineage Querying](#ML-Lineage-Querying)\n", " 1. [What ML lineage relationships can you infer from this model's endpoint?](#A.)\n", " 1. [What feature groups were used to train this model?](#B.)\n", " 1. [What models were trained using this feature group?](#C.)\n", " 1. [What feature groups were populated with data from this datasource?](#D.)\n", " 1. [What datasources were used to populate a feature group?](#E.)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Notebook Preparation" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Imports" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import sagemaker \n", "from sagemaker.feature_store.feature_group import FeatureGroup\n", "from sagemaker import get_execution_role\n", "import pandas as pd\n", "import logging\n", "import os\n", "import json\n", "import sys\n", "from pathlib import Path\n", "\n", "path = Path(os.path.abspath(os.getcwd()))\n", "package_dir = f'{str(path.parent)}/ml-lineage-helper'\n", "print(package_dir) \n", "sys.path.append(package_dir)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Check git submodules\n", "\n", "##### Check to confirm that the submodule ml-lineage-helper and all the files underneath are present as shown below. If not, please continue with the next instruction to update them.\n", "\n", "![check submodule](../images/m8_nb1-check-submodules.png \"check submodules\")\n", "\n", "---\n", "\n", "##### Run the following command in a terminal under the [./amazon-sagemaker-feature-store-end-to-end-workshop] folder path to update the missing submodules. \n", "\n", "git submodule update --init --recursive\n", "\n", "![run submodule update](../images/m8_nb1-run-submodules-update.png \"run submodule update\")\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%load_ext autoreload\n", "%autoreload 2\n", "from ml_lineage_helper import *\n", "from ml_lineage_helper.query_lineage import QueryLineage" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Check and update Sagemaker version" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "if sagemaker.__version__ < '2.48.1':\n", " subprocess.check_call([sys.executable, '-m', 'pip', 'install', 'sagemaker==2.48.1'])\n", " importlib.reload(sagemaker)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Logging Settings" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "logger = logging.getLogger('__name__')\n", "logger.setLevel(logging.DEBUG)\n", "logger.addHandler(logging.StreamHandler())\n", "logger.info(f'Using SageMaker version: {sagemaker.__version__}')\n", "logger.info(f'Using Pandas version: {pd.__version__}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Module Configurations " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Sagemaker session\n", "sess = sagemaker.Session()\n", "\n", "# Sagemaker Region\n", "region=sess.boto_region_name\n", "print(region)\n", "\n", "# IAM role for executing the processing job.\n", "iam_role = sagemaker.get_execution_role()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Load peristed variables from previous modules" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Retrieve Estimator parameters\n", "%store -r training_jobName\n", "print(training_jobName)\n", "\n", "# Retrieve FG names\n", "%store -r customers_feature_group_name\n", "print(customers_feature_group_name)\n", "%store -r products_feature_group_name\n", "print(products_feature_group_name)\n", "%store -r orders_feature_group_name\n", "print(orders_feature_group_name)\n", "\n", "# Retrieve Orders Datasource\n", "%store -r orders_datasource\n", "print(orders_datasource)\n", "\n", "# Retrieve Processing Job\n", "%store -r processing_job_name\n", "print(processing_job_name)\n", "%store -r processing_job_description\n", "print(processing_job_description)\n", "\n", "# Retrieve Endpoint Name\n", "%store -r endpoint_name\n", "print(endpoint_name)\n", "\n", "# Retrieve Query String\n", "%store -r query_string\n", "print(query_string)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## ML Lineage Creation\n", "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
πŸ’‘ Why is feature lineage important? \n", "

Lineage tracking can tie together a SageMaker Processing job, the raw data being processed, the processing code, the query you used against the Feature Store to fetch your training and test sets, the training and test data in S3, and the training code into a lineage represented as a DAG.

\n", "
\n", "\n", "![ML Lineage Tracking 1](../images/m8_nb1_ml-lineage-tracking-1.png \"ML Lineage Tracking 1\")\n", "\n", "##### Imagine trying to manually track all of this for a large team, or multiple teams or even multiple business units. Lineage tracking and querying helps make this more manageable and helps organizations move to ML at scale" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
πŸ’‘ What relationships are important to track? \n", "

The diagram below shows a sample set of ML lifecycle steps, artifacts, and associations that are\n", "typically needed for model lineage when using a feature store, including:

\n", "
\n", "\n", "---\n", "\n", "![ML Lineage Tracking 2](../images/m8_nb1_ml-lineage-tracking-2.png \"ML Lineage Tracking 2\")\n", "\n", "---\n", "\n", "#### Data source:\n", "##### ML features depend on raw data sources like an operational data store, or a set of CSV files in Amazon S3.\n", "\n", "#### Feature pipeline:\n", "##### Production-worthy features are typically built using a feature pipeline that takes a set of raw data sources, performs feature transformations, and ingests resulting features into the feature store. Lineage tracking can help by associating those pipelines with their data sources and their target feature groups.\n", "\n", "#### Feature sets:\n", "##### Once features are in a feature store, data scientists query it to retrieve data for training and validation of a model. You can use lineage tracking to associate the feature store query with the produced dataset. This provides granular detail into which features were used and what feature history was selected across multiple feature groups.\n", "\n", "#### Training job:\n", "##### As the ML lifecycle matures to adopt the use of a feature store, model lineage can associate training with specific features and feature groups\n", "\n", "#### Model: \n", "##### In addition to relating models to hosting endpoints, they can be linked to their corresponding training job, and indirectly to feature groups.\n", "\n", "#### Endpoint: \n", "##### Lastly, for online models, specific endpoints can be associated with the models they are hosting, completing the end to end chain from data sources to endpoints providing predictions.\n", "\n", "---\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
πŸ’‘ Tip\n", "

An end-to-end lineage solution needs to give you the means to access information about parameters, versioning, data sourcess and their respective associations to understand all aspects that went in to training the model.

\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Clear (Delete) existing ML Lineage" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
πŸ’‘ Warning!!!\n", "

Executing the [delete_lineage_data()] method will remove all Lineage among the associated artifacts used.

\n", "

Please DO NOT UNCOMMENT AND EXECUTE the following code unless you absolutely understand of the consequences

\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "endpoint_name" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# sagemakersession = SageMakerSession(bucket_name=sess.default_bucket(),\n", "# region=region,\n", "# role_name=iam_role,\n", "# aws_profile_name=\"default\",\n", "# )\n", "# ml_lineage = MLLineageHelper(sagemaker_session=sagemakersession, sagemaker_model_name_or_model_s3_uri=endpoint_name)\n", "# ml_lineage.delete_lineage_data()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Create ML Lineage\n", "---\n", "\n", "Lineage tracking can tie together a SageMaker Processing job, the raw data being processed, the processing code, the query you used against the Feature Store to fetch your training and test sets, the training and test data in S3, and the training code into a lineage represented as a DAG.\n", "\n", "---\n", "\n", "Many of the inputs are optional, but in this example we assume:\n", "1. You started with a raw data source\n", "2. You used SageMaker Data Wrangler to process the raw data and ingest it into the orders Feature Group.\n", "3. You queried the Feature Store to create training and test datasets.\n", "4. You trained a model in SageMaker on your training and test datasets." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Model name is same as endpoint name in this example\n", "ml_lineage = MLLineageHelper()\n", "lineage = ml_lineage.create_ml_lineage(training_jobName, model_name=endpoint_name, query=query_string,\n", " feature_group_names=[customers_feature_group_name,\n", " products_feature_group_name,\n", " orders_feature_group_name], \n", " sagemaker_processing_job_description=processing_job_description\n", " )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Verify ML Lineage" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Print the ML Lineage\n", "lineage" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### ML Lineage Graph" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
πŸ’‘ Tip\n", "

Given the number of components that are part of a model’s lineage, you may want to inspect the lineage of not only the model, but any object associated with the model, With a graph as the underlying data structure that supports lineage, you should have the flexibility to traverse an entity’s lineage from different focal points. You should be able to find the entire lineage of a model and all the components involved in creating it.

\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Visual Representation of the ML Lineage\n", "ml_lineage.graph()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "---\n", "## ML Lineage Querying\n", "---\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
πŸ’‘ What ML lineage relationships can you infer using this module? \n", "

Feature mangement, auditing and trouble shooting

\n", "
\n", "\n", "---\n", "\n", "![ML Lineage Tracking 3](../images/m8_nb1_ml-lineage-tracking-3.png \"ML Lineage Tracking 3\")\n", "\n", "---\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### A.\n", "
πŸ’‘ What ML lineage relationships can you infer from this model's endpoint?\n", "

Query ML Lineage by SageMaker Model Name or SageMaker Inference Endpoint

\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "lineageObject = MLLineageHelper(sagemaker_model_name_or_model_s3_uri=endpoint_name)\n", "lineageObject.df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### B.\n", "
πŸ’‘ What feature groups were used to train this model?\n", "

Given a SageMaker Model Name or artifact ARN, you can find associated Feature Groups

\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "query_lineage = QueryLineage()\n", "query_lineage.get_feature_groups_from_model(endpoint_name)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### C.\n", "
πŸ’‘ What models were trained using this feature group?\n", "

Given a Feature Group ARN, and find associated SageMaker Models

\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "feature_group = FeatureGroup(name=orders_feature_group_name, sagemaker_session=sess)\n", "query_lineage.get_models_from_feature_group(feature_group.describe()['FeatureGroupArn'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### D.\n", "
πŸ’‘ What feature groups were populated with data from this datasource?\n", "

Given a data source's S3 URI or Artifact ARN, you can find associated SageMaker Feature Groups

\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "query_lineage.get_feature_groups_from_data_source(orders_datasource, 3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### E.\n", "
πŸ’‘ What datasources were used to populate a feature group?\n", "

Given a Feature Group ARN, and find associated data sources

\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "orders_feature_group = FeatureGroup(name=orders_feature_group_name, sagemaker_session=sess)\n", "orders_feature_group_arn = orders_feature_group.describe()['FeatureGroupArn']\n", "print(orders_feature_group_arn)\n", "query_lineage.get_data_sources_from_feature_group(orders_feature_group_arn, max_depth=2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---" ] } ], "metadata": { "instance_type": "ml.t3.medium", "kernelspec": { "display_name": "Python 3 (Data Science)", "language": "python", "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/datascience-1.0" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.10" } }, "nbformat": 4, "nbformat_minor": 4 }