{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Create and Query ML Lineage between SageMaker - Models, Inference Endpoints, Feature Store, Processing Jobs and Datasources\n", "\n", "---\n", "\n", "#### Note: Please set kernel to Python 3 (Data Science) and select instance to ml.t3.medium\n", "\n", "\n", "
Lineage tracking can tie together a SageMaker Processing job, the raw data being processed, the processing code, the query you used against the Feature Store to fetch your training and test sets, the training and test data in S3, and the training code into a lineage represented as a DAG.
\n", "The diagram below shows a sample set of ML lifecycle steps, artifacts, and associations that are\n", "typically needed for model lineage when using a feature store, including:
\n", "An end-to-end lineage solution needs to give you the means to access information about parameters, versioning, data sourcess and their respective associations to understand all aspects that went in to training the model.
\n", "Executing the [delete_lineage_data()] method will remove all Lineage among the associated artifacts used.
\n", "Please DO NOT UNCOMMENT AND EXECUTE the following code unless you absolutely understand of the consequences
\n", "Given the number of components that are part of a modelβs lineage, you may want to inspect the lineage of not only the model, but any object associated with the model, With a graph as the underlying data structure that supports lineage, you should have the flexibility to traverse an entityβs lineage from different focal points. You should be able to find the entire lineage of a model and all the components involved in creating it.
\n", "Feature mangement, auditing and trouble shooting
\n", "Query ML Lineage by SageMaker Model Name or SageMaker Inference Endpoint
\n", "Given a SageMaker Model Name or artifact ARN, you can find associated Feature Groups
\n", "Given a Feature Group ARN, and find associated SageMaker Models
\n", "Given a data source's S3 URI or Artifact ARN, you can find associated SageMaker Feature Groups
\n", "Given a Feature Group ARN, and find associated data sources
\n", "