{ "cells": [ { "cell_type": "markdown", "id": "7cef6e80", "metadata": {}, "source": [ "Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.\n", "SPDX-License-Identifier: Apache-2.0" ] }, { "cell_type": "markdown", "id": "9cdd0912", "metadata": {}, "source": [ "# Building an Identity Graph Application on Amazon Neptune\n", "\n", "This notebook shows how Amazon Neptune can be used to build an identity graph for applications in marketing and targeted advertising using a smaller version of the dataset found in the Amazon Web Services blog post, [Building a customer identity graph with Amazon Neptune](https://aws.amazon.com/blogs/database/building-a-customer-identity-graph-with-amazon-neptune/).\n", " \n", " - [Background](#Background)\n", " - [Getting Started](#Getting-Started)\n", " - [Cross-Device Graphs](#Cross-Device-Graphs)\n", " - [Targeted Promotions](#Targeted-Promotions)\n", " - [Audience Segmentation](#Audience-Segmentation)\n", " - [Conclusion](#Conclusion)\n", " - [What's Next?](#What's-Next?)\n", "\n", "## Background\n", "\n", "An identity graph provides a single unified view of customers and prospects by linking multiple identifiers such as cookies, device identifiers, IP addresses, email IDs, and internal enterprise IDs to a known person or anonymous profile using privacy-compliant methods. Typically, identity graphs are part of a larger identity resolution architecture. Identity resolution is the process of matching human identity across a set of devices used by the same person or a household of persons for the purposes of building a representative identity, or known attributes, for targeted advertising.\n", "\n", "The following notebook walks you through a sample solution for identity graph and how it can be used within a larger identity resolution architecture using an open dataset and the use of a graph database, Amazon Neptune. In this notebook, we also show a number of data visualizations that allow one to better understand the structure of an identity graph and the aspects of an identity resolution dataset and use case. We also showcase some additional use cases that can be exposed using this particular dataset.\n", "\n", "\n", "\n", "## Getting Started\n", "\n", "The dataset used in this notebook is derived from the **CIKM Cup 2016 Track 1: Cross-Device Entity Linking Challenge**. This dataset contains anonymized clickstream data for a set of anonymized user IDs representing the same user across multiple devices, as well as hashed site URLs and HTML titles those users visited. For example 'http://www.amazon.com' could be represented as 'c94174b63350fd53' in this dataset and 'http://www.amazon.com/gp/bestsellers' could be represented as 'c94174b63350fd53/1e8deebfc8e36e85/2215e2a3f89eeba7' where each of the directories in the url are hashed as well. In a similar method, usernames, cookies, and device IDs are also hashed. To make this dataset more interesting to use, we have added additional features to this data using the [python faker](https://faker.readthedocs.io/en/master/) library. Combining the CIKM dataset with the manufactured data, we have stitched the data together to create the following graph data model:\n", "\n", "\"drawing\"\n", "\n", "This data model could be used as the starting point for your own identity graph. In most cases, an organization will have data related to website visits, click events, IP addresses, and some combination of anonymous users and registered (signed-on) users. Anonymous users in this data model are represented as \"transient IDs\" and known users are represented as \"persistent IDs\". The process of linking anonymous users to known users is accomplished through a process called [identity (or entity) resolution](https://en.wikipedia.org/wiki/Record_linkage#Entity_resolution). We have assumed here that identity resolution has already taken place and we are storing the relationships between the anonymous users and their resolved known users. The process of identity resolution can involve both deterministic patterns (such as matching users coming from the same public IP address within a certain time frame) or through more probabilistic means (using machine learning).\n", "\n", "### Loading the Identity Graph\n", "\n", "To get started with our Identity Graph, we will load a set of data from the data model shown above into our Neptune cluster. Run the following two cells to download and load this data. It will take approximately 5-6 minutes to complete this next step. This will load approximately 120,000 clickstream events with their corresponding user information. The first cell will download the data from S3 into your notebook instance. The second cell uses the `%seed` command to load this data to your Neptune cluster.\n", "\n", "***NOTE: If using this notebook in a locally installed deployment of graph-notebook, you may need to copy the contents of the first cell below, add `sudo` to the beginning of each line, and run these commands in a bash terminal window. Otherwise, you may get 'permission denied' errors. If running this notebook in a Neptune Notebook or SageMaker Notebook instance, this will work as is.***" ] }, { "cell_type": "code", "execution_count": null, "id": "0bd2ef12", "metadata": {}, "outputs": [], "source": [ "%%bash\n", "\n", "GRAPH_NOTEBOOK_INSTALL_PATH=`pip show graph-notebook | grep \"Location\" | cut -d ' ' -f2`\n", "mkdir $GRAPH_NOTEBOOK_INSTALL_PATH/graph_notebook/seed/queries/propertygraph/gremlin/identity\n", "cd $GRAPH_NOTEBOOK_INSTALL_PATH/graph_notebook/seed/queries/propertygraph/gremlin/identity\n", "curl -s \"https://aws-admartech-samples.s3.amazonaws.com/identity-graph-notebook-data/idgraph.zip\" > \\\n", " ./idgraph.zip\n", "unzip ./idgraph.zip\n", "rm -f ./idgraph.zip" ] }, { "cell_type": "code", "execution_count": null, "id": "d6412360", "metadata": { "scrolled": true }, "outputs": [], "source": [ "%seed --model Property_Graph --language gremlin --dataset identity --run" ] }, { "cell_type": "markdown", "id": "1f401538", "metadata": {}, "source": [ "### Reviewing the Dataset\n", "\n", "Before proceeding, let's check that the dataset loaded correctly. In the following two queries, we want to count each of the vertices and edges by their corresponding vertex or edge label (the type of vertex or edge from the graph data model picture above). You should get an output that matches the following (**NOTE: you may see more than this if you have previously loaded data into your Neptune cluster.**):\n", "\n", "For vertices:\n", "\n", "{'persistentId': 166, 'website': 41326, 'transientId': 671, 'identityGroup': 50, 'IP': 171, 'websiteGroup': 3758}" ] }, { "cell_type": "code", "execution_count": null, "id": "977e5d07", "metadata": { "scrolled": true }, "outputs": [], "source": [ "%%gremlin\n", "\n", "g.V().groupCount().by(label)" ] }, { "cell_type": "markdown", "id": "6dbafe06", "metadata": {}, "source": [ "For edges:\n", "\n", "{'member': 166, 'visited': 117808, 'links_to': 41326, 'uses': 676, 'has_identity': 671}" ] }, { "cell_type": "code", "execution_count": null, "id": "7225ab83", "metadata": { "scrolled": true }, "outputs": [], "source": [ "%%gremlin\n", "\n", "g.E().groupCount().by(label)" ] }, { "cell_type": "markdown", "id": "37505003", "metadata": {}, "source": [ "## Example Use Cases and Applications\n", "\n", "In the following sections we will showcase the many ways in which identity graphs are used. Many of these use cases focus around better understanding user behavior as users interact with an online platform. These insights provide you with a means of servicing customers in a targeted manner or converting potential product interest into sales opportunities.\n", "\n", "\n", "### Cross-Device Graphs\n", "\n", "**Advertisers want to find out information about user interests to provide an accurate targeting. The data should be based on the activity of the user across all devices.**\n", "\n", "Suppose you are hosting a web platform and collecting clickstream data as users browse your site or use your mobile app. In the majority of situations, users using your platform will be anonymous (or non-registered or logged in users). However, these anonymous users may be linked to other known users in that have used our platform before. We can join (or resolve) the identity of the anonymous user with attributes we know about existing users to make some assumptions (based off of known user behavior and heuristics) in order to know more about this anonymous user. We can then use this information to target the user with advertising, special offers, discounts, etc.\n", "\n", "Let's use an example where we have an anonymous user id ('ed4982b00e323383583f30236e5b1f11'). We want to know more about this user and if they are linked to other users on our platform. This anonymous user is considered a \"transient ID\" in our graph data model (see picture above). Assuming this user does not have a link to a known user, or \"persistent ID\", how might we find connections from this transient ID to other known user IDs? Looking at the data model, you can see that \"transient ID\" vertices in our graph are connected to \"IP Address\" vertices. We can traverse across \"IP Address\" vertices to get to other linked \"transient IDs\" that might be linked to a known user. Let's do that in the following graph query. Run the following cell, and then click on the Graph tab to see an output displaying the path from the anonymous user to the known user:" ] }, { "cell_type": "code", "execution_count": null, "id": "1cfefc6a", "metadata": { "scrolled": true }, "outputs": [], "source": [ "%%gremlin \n", "\n", "g.V('ed4982b00e323383583f30236e5b1f11').\n", " out('uses'). // traverse the users edge to get to IP addresses\n", " in('uses'). // go from the IP address vertex to other associated transient IDs\n", " in('has_identity'). //go from the found transient IDs to known users (persistent IDs)\n", " dedup(). // remove duplicate persistent IDs\n", " path() // show a path from the unknown anonymous user to a known user" ] }, { "cell_type": "markdown", "id": "f0cb21c1", "metadata": {}, "source": [ "As you can see in the output above, we have found one known user linked to this anonymous user via the same used public IP address. We can take this a step further by looking at other linked known users that exist in the same household in order to determine common attributes that we may want to use in targeting an offer or ad to this anonymous user. Let's look at the associated household context for this known user by building on to the query from above. Again, click on the Graph tab after the query is executed." ] }, { "cell_type": "code", "execution_count": null, "id": "8951b837", "metadata": {}, "outputs": [], "source": [ "%%gremlin\n", "\n", "g.V('ed4982b00e323383583f30236e5b1f11').\n", " out('uses'). // traverse the users edge to get to IP addresses\n", " in('uses'). // go from the IP address vertex to other associated transient IDs\n", " in('has_identity'). //go from the found transient IDs to known users (persistent IDs)\n", " dedup(). // remove duplicate persistent IDs\n", " in('member').as('household'). //found the household associated with the found known user\n", " out('member'). // traverse back to other known users in this household\n", " out('has_identity'). //find other associated anonymous users associated to the known users\n", " out('visited','uses'). //look at the other websites/products browed by those anonymous users\n", " path().from('household') //display the graph from household to associated clickstream events" ] }, { "cell_type": "markdown", "id": "8fb7bf2f", "metadata": {}, "source": [ "The graph above shows an entire \"household subgraph\" containing all known users in a household and the associated web activity. This contains useful information regarding what product pages or website subpages a given household is browsing and can be used to target offers to individuals in this household." ] }, { "cell_type": "markdown", "id": "7cb72bf8", "metadata": {}, "source": [ "### Targeted Promotions\n", "\n", "**Ecommerce publishers want to convince undecided users to purchase the product by offering them discount codes as soon as they have met certain criteria. Find all users who have visited product page at least X times in the last 30 days, but did not buy anything (have not visited thank you page).**\n", "\n", "Another method for looking at user behavior is to determine what users are interested in a specific product but haven't converted into a purchase. This can be captured in clickstream data by looking at clickstream events related to a product page and an associated \"thank you\" page (or an event showing a conversion to a purchase). In the following scenario we are going to assume a product page with the web url of 'c94174b63350fd53' (remember, these are hashed values - this could be http://www.amazon.com) and a thank you page with the url of 'c94174b63350fd53/1e8deebfc8e36e85/b5509c3fb28c4e4f'. Given this thank you page, let's traverse our graph to determine what users have looked at our website 'c94174b63350fd53' but have not made a purchase (or have not browsed to 'c94174b63350fd53/1e8deebfc8e36e85/b5509c3fb28c4e4f'). \n", "\n", "Let's first look at the users who have made a purchase. Run the following query and click on the Graph tab to view the results." ] }, { "cell_type": "code", "execution_count": null, "id": "f9261530", "metadata": {}, "outputs": [], "source": [ "%%gremlin\n", "\n", "g.V().has('url','c94174b63350fd53'). //find product page\n", " out('links_to').has('url','c94174b63350fd53/1e8deebfc8e36e85/b5509c3fb28c4e4f'). //find thank you page\n", " in('visited'). //traverse to users who have browsed the thank you page\n", " in('has_identity'). //traverse to the associated known users (persistent IDs)\n", " path()" ] }, { "cell_type": "markdown", "id": "77c545c7", "metadata": {}, "source": [ "Now, let's look at the converse and find all users that have browsed our site but have not converted to a sale (or viewed the thank you page). We can run the following query looking for users that have landed on our product page, viewed other pages on the site, and yet have not viewed the thank you page. Run the following query and view the Graph tab to see the results." ] }, { "cell_type": "code", "execution_count": null, "id": "fb564bc2", "metadata": { "scrolled": false }, "outputs": [], "source": [ "%%gremlin\n", "\n", "g.V().has('url','c94174b63350fd53'). //find product page\n", " out('links_to'). //find other views but not the thank you page\n", " where(not(has('url','c94174b63350fd53/1e8deebfc8e36e85/b5509c3fb28c4e4f'))).\n", " in('visited'). //get transient IDs from these views but you have not viewed the thank you page\n", " where(not(out('visited').has('url','c94174b63350fd53/1e8deebfc8e36e85/b5509c3fb28c4e4f'))).\n", " in('has_identity').dedup(). //fetch associated unique known users\n", " path()" ] }, { "cell_type": "markdown", "id": "f18b9cb5", "metadata": {}, "source": [ "The graph above will show all associated persistent IDs (known users), transient IDs (associated anonymous users), and the other sites they have viewed on our site. We can use this data to determine user behavior and make offers or discounts to entice a product conversion." ] }, { "cell_type": "markdown", "id": "2b9b149e", "metadata": {}, "source": [ "### Audience Segmentation\n", "\n", "**Advertisers want to generate audiences for demand side platform (DSP) targeting. The specific audience could be the users who are interested in specific car brands.**\n", "\n", "Using our dataset, let's assume a website group is associated with a given brand (could be http://www.amazon.com). In this particular case, let's assume it is a brand of automobile. Given our brand, we may want to look at the entire associated audience and extract certain characteristics/demographics from that audience. We can do this by looking at the entire subgraph related to the brand. Let's use the website group 'c94174b63350fd53' to model this pattern. Starting with 'c94174b63350fd53', let's traverse the graph and extract all associated known users. First, let's see how many clickstream event types and the size of the users associated with the brand subgraph. Run the following query below to see these statistics." ] }, { "cell_type": "code", "execution_count": null, "id": "36b4f98d", "metadata": {}, "outputs": [], "source": [ "%%gremlin\n", "\n", "g.V().has('url','c94174b63350fd53').\n", " project('subpages','audience_size').\n", " by(out('links_to').count()).\n", " by(out('links_to').in('visited').count())" ] }, { "cell_type": "markdown", "id": "6b1ff022", "metadata": {}, "source": [ "Next, let's take a look at the entire brand subgraph. Run the following query. This will display all associated transient/persistent IDs, website subpages, and brands (website groups) associated with the targeted brand. Run the following query and open the Graph tab to see the results." ] }, { "cell_type": "code", "execution_count": null, "id": "9a5584b8", "metadata": {}, "outputs": [], "source": [ "%%gremlin\n", "\n", "g.V().has('url','c94174b63350fd53').out('links_to').in('visited').in('has_identity').dedup().path()" ] }, { "cell_type": "markdown", "id": "82a88c23", "metadata": {}, "source": [ "\n", "The goal of this query is to find all user devices (transient IDs) that interacted with any of the subpages on that domain, given one website subpage vertex.\n", "\n", "In other words, we are interested in all user device or cookie identifiers (also the ones that did not explicitly interacted with the brand, but belongs to the same user), that showed up on any of the domain pages. This information is useful for the purposes of retargeting.\n" ] }, { "cell_type": "markdown", "id": "d89b5179", "metadata": {}, "source": [ "## Conclusion\n", "\n", "Identity graphs are a key component in being able to analyze consumer behavior and provide deep insights on how best to interact with potential customers. In this notebook, we walked through three potential use cases for building an identity graph. Cross-device graphs discuss how to unify user IDs across multiple devices and how to build household subgraphs. Targeted promotions showcase an example of isolating a subgraph of undecided consumers for targeted discounts. Lastly, audience segmentation discusses ways to identify a set of unique users based on all interactions with the brand through its websites. The patterns displayed above are just a small subset of how identity graphs can be leveraged." ] }, { "cell_type": "markdown", "id": "823d103e", "metadata": {}, "source": [ "## What's Next?\n", "\n", "To build an identity graph solution that incorporates Neptune, we recommend the following resources:\n", " \n", "- [Getting Started with Amazon Neptune](https://pages.awscloud.com/AWS-Learning-Path-Getting-Started-with-Amazon-Neptune_2020_LP_0009-DAT.html) is a video-based learning path that shows you how to create and connect to a Neptune database, choose a data model and query language, author and tune graph queries, and integrate Neptune with other Amazon Web services.\n", "- Before you begin designing your database, consult the [Amazon Web Services Reference Architectures for Using Graph Databases](https://github.com/aws-samples/aws-dbs-refarch-graph/) GitHub repo, where you can browse examples of reference deployment architectures, and learn more about building a graph data model and choosing a query language.\n", "- For links to documentation, blog posts, videos, and code repositories with samples and tools, see the [Amazon Neptune developer resources](https://aws.amazon.com/neptune/developer-resources/).\n", "- Neptune ML makes it possible to build and train useful machine learning models on large graphs in hours instead of weeks. To find out how to set up and use a graph neural network, see [Using Amazon Neptune ML for machine learning on graphs](https://docs.aws.amazon.com/neptune/latest/userguide/machine-learning.html).\n", "- [Identity Graphs on Amazon Web Services](https://aws.amazon.com/advertising-marketing/identity-graph/) showcases Amazon Web Services solutions specifically designed for identity graphs, focusing on advertising and marketing.\n", "- Cox Automotive scales digital personalization using an identity graph powered by Amazon Neptune with this [blog post](https://aws.amazon.com/blogs/database/cox-automotive-scales-digital-personalization-using-an-identity-graph-powered-by-amazon-neptune/) and [presentation](https://youtu.be/I7_b1xkQ7Dc)." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.5" } }, "nbformat": 4, "nbformat_minor": 5 }