{
"cells": [
{
"cell_type": "markdown",
"id": "74f20a33",
"metadata": {},
"source": [
"Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.\n",
"SPDX-License-Identifier: Apache-2.0"
]
},
{
"cell_type": "markdown",
"id": "75d9a24c",
"metadata": {},
"source": [
"# Building a Fraud Graph Application on Amazon Neptune\n",
"\n",
"This notebook shows how Amazon Neptune can be used to build a fraud graph for a fraud detection solution. It includes a credit card transaction data model that connects accounts, merchants and transactions, and Gremlin queries that help identify fraud rings (first-party fraud) and instances of identity theft (third-party fraud).\n",
"\n",
" - [Background](#Background)\n",
" - [Getting Started](#Getting-Started)\n",
" - [Fraud Rings](#Fraud-Rings)\n",
" - [Identity Theft](#Identity-Theft)\n",
" - [Building a Fraud Detection Solution](#Building-a-Fraud-Detection-Solution)\n",
" - [Conclusion](#Conclusion)\n",
" - [What's Next?](#What's-Next?)\n",
"\n",
"## Background\n",
"\n",
"Fraud hides itself in isolation. It exploits our failure to assess a transaction in the context of other recent transactions: patterns of fraudulent behaviour emerge only when we connect many seemingly discrete data points and events.\n",
"\n",
"A fraud graph connects the entities participating in retail and financial transactions: entities such as accounts, transactions and merchants. By relating accounts and connecting transactions we improve our chances to detect and prevent fraud. With Amazon Neptune we can connect account and transaction information and query it whenever a new account or transaction is submitted to the system. Using queries that find patterns in our data that we know to be _indicative_ of fraud, we can evaluate each transaction in the context of other transactions and accounts, and thereby determine whether constellations of data in the fraud graph represent fraudulent activity.\n",
"\n",
"The examples in this use case show how we can identify fraud rings (first-party fraud) and instances of identity theft (third-party fraud) in a credit card dataset.\n",
"\n",
" - **Fraud ring** A group of people who give false information when applying for a credit card, with the intention of purchasing goods and services without repaying the debt.\n",
" - **Identity theft** When an account holder's personal details are used without their permission to purchase goods or services.\n",
" \n",
"Most of the queries in this notebook are _graph local_ queries. A graph local query takes as its starting point an individual entity - an account, or transaction, for example - and from there explores the neighboring parts of the graph in order to compute a result or discover a local constellation of connected data. Fraud detection solutions use graph local queries when accounts are created or modified, or new transactions are submitted to the system, to identify fraudulent behaviour associated with the account or transaction."
]
},
{
"cell_type": "markdown",
"id": "a80202f4",
"metadata": {},
"source": [
"## Getting Started\n",
"\n",
"In this section we'll load the fraud graph and set some visualization options. We'll then use some Gremlin queries to inspect the data model used throughout the solution."
]
},
{
"cell_type": "markdown",
"id": "051ba5d9",
"metadata": {},
"source": [
"### Load data\n",
"\n",
"The cell below loads the example fraud graph into your Neptune cluster. When you run the cell you will be prompted to select a `Source type`, a `Data Model`, and a `Data set`. Select `samples`, `Property_Graph`, and `fraud_graph`, respectively. The graph takes about 5 minutes to load."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fd618aea",
"metadata": {},
"outputs": [],
"source": [
"%seed --model Property_Graph --language gremlin --dataset fraud_graph --run"
]
},
{
"cell_type": "markdown",
"id": "4191948d",
"metadata": {},
"source": [
"### Set visualization options\n",
"\n",
"The command below configures the visualization to use specific colours and icons for the different parts of the data model."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "bd2d7120",
"metadata": {},
"outputs": [],
"source": [
"%%graph_notebook_vis_options\n",
"\n",
"{\n",
" \"groups\": {\n",
" \"Account\": {\n",
" \"shape\": \"icon\",\n",
" \"icon\": {\n",
" \"face\": \"FontAwesome\",\n",
" \"code\": \"\\uf2bb\",\n",
" \"color\": \"red\"\n",
" }\n",
" },\n",
" \"Transaction\": {\n",
" \"shape\": \"icon\",\n",
" \"icon\": {\n",
" \"face\": \"FontAwesome\",\n",
" \"code\": \"\\uf155\",\n",
" \"color\": \"green\"\n",
" }\n",
" },\n",
" \"Merchant\": {\n",
" \"shape\": \"icon\",\n",
" \"icon\": {\n",
" \"face\": \"FontAwesome\",\n",
" \"code\": \"\\uf290\",\n",
" \"color\": \"orange\"\n",
" }\n",
" },\n",
" \"DateOfBirth\": {\n",
" \"shape\": \"icon\",\n",
" \"icon\": {\n",
" \"face\": \"FontAwesome\",\n",
" \"code\": \"\\uf1fd\",\n",
" \"color\": \"blue\"\n",
" }\n",
" },\n",
" \"EmailAddress\": {\n",
" \"shape\": \"icon\",\n",
" \"icon\": {\n",
" \"face\": \"FontAwesome\",\n",
" \"code\": \"\\uf1fa\",\n",
" \"color\": \"blue\"\n",
" }\n",
" },\n",
" \"Address\": {\n",
" \"shape\": \"icon\",\n",
" \"icon\": {\n",
" \"face\": \"FontAwesome\",\n",
" \"code\": \"\\uf015\",\n",
" \"color\": \"blue\"\n",
" }\n",
" },\n",
" \"IpAddress\": {\n",
" \"shape\": \"icon\",\n",
" \"icon\": {\n",
" \"face\": \"FontAwesome\",\n",
" \"code\": \"\\uf109\",\n",
" \"color\": \"blue\"\n",
" }\n",
" },\n",
" \"PhoneNumber\": {\n",
" \"shape\": \"icon\",\n",
" \"icon\": {\n",
" \"face\": \"FontAwesome\",\n",
" \"code\": \"\\uf095\",\n",
" \"color\": \"blue\"\n",
" }\n",
" }\n",
" },\n",
" \"edges\": {\n",
" \"color\": {\n",
" \"inherit\": false\n",
" },\n",
" \"smooth\": {\n",
" \"enabled\": true,\n",
" \"type\": \"straightCross\"\n",
" },\n",
" \"arrows\": {\n",
" \"to\": {\n",
" \"enabled\": false,\n",
" \"type\": \"arrow\"\n",
" }\n",
" },\n",
" \"font\": {\n",
" \"face\": \"courier new\"\n",
" }\n",
" },\n",
" \"interaction\": {\n",
" \"hover\": true,\n",
" \"hoverConnectedEdges\": true,\n",
" \"selectConnectedEdges\": false\n",
" },\n",
" \"physics\": {\n",
" \"minVelocity\": 0.75,\n",
" \"barnesHut\": {\n",
" \"centralGravity\": 0.1,\n",
" \"gravitationalConstant\": -50450,\n",
" \"springLength\": 95,\n",
" \"springConstant\": 0.04,\n",
" \"damping\": 0.09,\n",
" \"avoidOverlap\": 0.1\n",
" },\n",
" \"solver\": \"barnesHut\",\n",
" \"enabled\": true,\n",
" \"adaptiveTimestep\": true,\n",
" \"stabilization\": {\n",
" \"enabled\": true,\n",
" \"iterations\": 1\n",
" }\n",
" }\n",
"}"
]
},
{
"cell_type": "markdown",
"id": "e1331640",
"metadata": {},
"source": [
"### Data model\n",
"\n",
"The fraud graph included in this example models credit card accounts, account holder information, merchants, and the transactions performed when an account holder purchases goods or services from a merchant.\n",
"\n",
"#### Account and features\n",
"\n",
"An `Account` has a number of features, including physical `Address`, `IpAddress`, `DateOfBirth` of the account holder, `EmailAddress`, and contact `PhoneNumber`. An account holder can have multiple email addresses and phone numbers.\n",
"\n",
"In many graph data models these features of the account holder would be modelled as properties of the account. But with fraud detection it's important to be able to link accounts based on shared features, and to find related accounts at query time based on one or more shared features. Hence, our fraud detection application graph data model stores each feature as a separate vertex. Multiple accounts that share the same feature value - the same physical address, for example - are connected to the single vertex representing that feature value. For more details on modelling shared features as vertices, see [Relating entities through their attributes at query time](https://github.com/aws-samples/aws-dbs-refarch-graph/tree/master/src/graph-data-modelling#relating-entities-through-their-attributes-at-query-time).\n",
"\n",
"The following query shows a single account and its associated features. After running the query, click the `Graph` tab to see a visualization of the results. Note that the account holder shown in the results has two email addresses and two phone numbers."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "73d41e45",
"metadata": {},
"outputs": [],
"source": [
"%%gremlin -g type -p v,inV,outV\n",
"\n",
"g.V('account-4398046521820').\n",
" in('FEATURE_OF_ACCOUNT').\n",
" path().\n",
" by(\n",
" project('type', 'value').\n",
" by(label).\n",
" by(valueMap('account_number', 'first_name', 'last_name', 'value'))\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "21209a97",
"metadata": {},
"source": [
"The next query shows two accounts that share the same feature - a date of birth. After running the query, click the `Graph` tab to see a visualization of the results."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "83b4b9f8",
"metadata": {},
"outputs": [],
"source": [
"%%gremlin -g type -p v,inV,outV\n",
"\n",
"g.V('account-8853').\n",
" in('FEATURE_OF_ACCOUNT').\n",
" out('FEATURE_OF_ACCOUNT').\n",
" simplePath().\n",
" path().\n",
" by(\n",
" project('type', 'value').\n",
" by(label).\n",
" by(valueMap('account_number', 'value', 'amount', 'created', 'name'))\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "b67e8229",
"metadata": {},
"source": [
"#### Transaction\n",
"\n",
"When an `Account` holder purchases goods or services from a `Merchant` using their credit card, we create a new `Transaction` vertex in the graph with `amount` (numeric) and `created` (timestamp) properties.\n",
"\n",
"A transaction is associated with an account and a merchant. Each transaction is also associated with one or more features, such as `IpAddress` or `PhoneNumber`, captured at the time the transaction was submitted to the system. As with features associated with account holders, we store each feature value as a separate vertex.\n",
"\n",
"The following query shows a single transaction between an account and a merchant, together with the IP address from which the transaction was submitted. After running the query, click the `Graph` tab to see a visualization of the results."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8bd63b87",
"metadata": {},
"outputs": [],
"source": [
"%%gremlin -g type -p v,outV,inV\n",
"\n",
"g.V('account-8698').\n",
" in('ACCOUNT').limit(1).\n",
" union(\n",
" out('MERCHANT'),\n",
" in('FEATURE_OF_TRANSACTION')\n",
" ).\n",
" path().\n",
" by(\n",
" project('type', 'value').\n",
" by(label).\n",
" by(valueMap('account_number', 'created', 'amount', 'name', 'value'))\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "0fd59869",
"metadata": {},
"source": [
"#### All transactions for an account\n",
"\n",
"The following query shows all the transactions for an account. Some things to notice here:\n",
"\n",
" - There are several examples of the account making multiple payments to the same merchant. \n",
" - Most transactions are submitted from the IP address associated with the account. \n",
" - A few transactions use a different IP address. \n",
" - One of the transactions was submitted over the phone."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cdc5fe70",
"metadata": {},
"outputs": [],
"source": [
"%%gremlin -g type -p v,outV,inV\n",
"\n",
"g.V('account-8698').\n",
" in('ACCOUNT').\n",
" union(\n",
" out('MERCHANT'),\n",
" in('FEATURE_OF_TRANSACTION')\n",
" ).\n",
" path().\n",
" by(\n",
" project('type', 'value').\n",
" by(label).\n",
" by(valueMap('account_number', 'created', 'amount', 'name', 'value'))\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "946eaa1e",
"metadata": {},
"source": [
"#### Shared households\n",
"\n",
"Account holders who live at the same address will often have a number of shared features, including physical address, IP address and phone number. \n",
"\n",
"The following query shows accounts that share several of these features. Notice how accounts that belong to the same household form small, discrete components."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3f9c55df",
"metadata": {},
"outputs": [],
"source": [
"%%gremlin -g type -p v,outV,inV\n",
"\n",
"g.V().hasLabel('IpAddress').\n",
" where(\n",
" out('FEATURE_OF_ACCOUNT').count().is(eq(2))).\n",
" where(\n",
" out('FEATURE_OF_ACCOUNT').\n",
" in('FEATURE_OF_ACCOUNT').hasLabel('Address').dedup().count().is(eq(1)) \n",
" ).\n",
" limit(5).\n",
" out('FEATURE_OF_ACCOUNT').\n",
" in('FEATURE_OF_ACCOUNT').\n",
" simplePath().\n",
" path().\n",
" by(\n",
" project('type', 'value').\n",
" by(label).\n",
" by(valueMap('account_number', 'value', 'amount', 'created', 'name'))\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "d25cabc2",
"metadata": {},
"source": [
"## Fraud Rings\n",
"\n",
"Fraud rings are examples of first-party fraud. A fraud ring consists of a group of people who give false information when applying for a service, such as a credit card, in order to abuse the line of credit made available to them. The members of a fraud ring may act as \"good citizens\" for a while, but at some point they will coordinate their actions to leverage the credit, with no intention of paying off the debt. \n",
"\n",
"Accounts belonging to a fraud ring often willingly or inadvertently share features such as IP addresses or physical addresses. These shared features can be used to identify activities that can indicate the presence of a fraud ring.\n",
"\n",
"### Account with many shared features (possible fraud ring)\n",
"\n",
"The following query shows an account that is linked to a number of other accounts by way of some shared features. Note that there are more accounts here than are typically encountered in a shared household."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b45b1bbd",
"metadata": {},
"outputs": [],
"source": [
"%%gremlin -g type -p v,inV,outV\n",
"\n",
"g.V('account-4398046519460').\n",
" in('FEATURE_OF_ACCOUNT').\n",
" out('FEATURE_OF_ACCOUNT').\n",
" simplePath().\n",
" path().\n",
" by(\n",
" project('type', 'value').\n",
" by(label).\n",
" by(valueMap('account_number', 'value', 'amount', 'created', 'name'))\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "bf7fa7ff",
"metadata": {},
"source": [
"### Extended fraud ring\n",
"\n",
"We can extend the scope of the previous to find linked accounts two hops from the starting account. The size and complexity of this account network is suggestive of a fraud ring:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ad826ffb",
"metadata": {},
"outputs": [],
"source": [
"%%gremlin -g type -p v,inV,outV,inV,outV\n",
"\n",
"g.V('account-4398046519460').\n",
" emit().\n",
" repeat(\n",
" in('FEATURE_OF_ACCOUNT').\n",
" out('FEATURE_OF_ACCOUNT').\n",
" simplePath()\n",
" ).times(2). \n",
" path().\n",
" by(\n",
" project('type', 'value').\n",
" by(label).\n",
" by(valueMap('account_number', 'value'))\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "5562ce96",
"metadata": {},
"source": [
"### Fraud ring and transactions\n",
"\n",
"By further modifying the query, we can find all the transactions linked to the accounts in this presumed fraud ring.\n",
"\n",
"Run the following query and then click the `Graph` tab to see a visualization of the results. If you grab the accounts and tease them apart you'll see that there is one account (0010-9951-1628-1609) and its transactions that are linked to a member of the supposed fraud ring only by way of a shared date of birth: this is likely a legitimate account."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "bf3ed52a",
"metadata": {},
"outputs": [],
"source": [
"%%gremlin -g type -p v,inV,outV,inV,outV,inV,outV\n",
"\n",
"g.V('account-4398046519460').\n",
" emit().\n",
" repeat(\n",
" in('FEATURE_OF_ACCOUNT').\n",
" out('FEATURE_OF_ACCOUNT').\n",
" simplePath()\n",
" ).times(2). \n",
" in('ACCOUNT').\n",
" out('MERCHANT').\n",
" path().\n",
" by(\n",
" project('type', 'value').\n",
" by(label).\n",
" by(valueMap('account_number', 'value', 'amount', 'created', 'name'))\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "7730a878",
"metadata": {},
"source": [
"Here's a similar query, but instead of starting with an account, we start with a specific transaction. The results are similar, however, showing the constellation of accounts and other transactions associated with this transaction."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1d4e345e",
"metadata": {},
"outputs": [],
"source": [
"%%gremlin -g type -p v,inV,outV,inV,outV,inV,outV\n",
"\n",
"g.V('05d69fc8a55e4b648318d460e748839a').\n",
" out('ACCOUNT').as('account').\n",
" emit().\n",
" repeat(\n",
" in('FEATURE_OF_ACCOUNT').\n",
" out('FEATURE_OF_ACCOUNT').\n",
" simplePath()\n",
" ).times(2). \n",
" in('ACCOUNT').\n",
" out('MERCHANT').\n",
" path().from('account').\n",
" by(\n",
" project('type', 'value').\n",
" by(label).\n",
" by(valueMap('account_number', 'value', 'amount', 'created', 'name'))\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "5379766f",
"metadata": {},
"source": [
"## Find all potential fraud rings\n",
"\n",
"The following query finds all accounts that appear to share features with several other accounts. Unlike most of the account- or transaction-centric graph local interactive queries in this notebook, this is more of an analytics query, which surveys all accounts in the database. The `where()` step helps identify accounts that share features with other accounts.\n",
"\n",
"The results here show how difficult it is to identify the boundaries of a fraud ring. Real-world data is tangled and messy: something as simple as a shared date of birth may inadvertently associate an innocent account holder with a fraud ring. Nonetheless we can easily spot clusters of accounts that warrant further investigation."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a2427dc3",
"metadata": {},
"outputs": [],
"source": [
"%%gremlin -g type -p v,inV,outV,inV,outV\n",
"\n",
"g.V().hasLabel('Account').\n",
" where(\n",
" in('FEATURE_OF_ACCOUNT').\n",
" out('FEATURE_OF_ACCOUNT').\n",
" dedup().\n",
" count().\n",
" is(gt(5))\n",
" ).\n",
" emit().\n",
" repeat(\n",
" in('FEATURE_OF_ACCOUNT').\n",
" out('FEATURE_OF_ACCOUNT').\n",
" simplePath()\n",
" ).\n",
" times(2). \n",
" path().\n",
" by(\n",
" project('type', 'value').\n",
" by(label).\n",
" by(valueMap('account_number', 'value'))\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "f9766a53",
"metadata": {},
"source": [
"Filtering by IP address, we find several accounts that might serve as starting points for more detailed fraud ring investigations:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5fcfb1f3",
"metadata": {},
"outputs": [],
"source": [
"%%gremlin -g type -p v,inV,outV,inV,outV,inV,outV\n",
"\n",
"g.V().hasLabel('Account').\n",
" where(\n",
" in('FEATURE_OF_ACCOUNT').\n",
" out('FEATURE_OF_ACCOUNT').\n",
" dedup().\n",
" count().\n",
" is(gt(5))\n",
" ).\n",
" emit().\n",
" repeat(\n",
" in('FEATURE_OF_ACCOUNT').hasLabel('IpAddress').\n",
" out('FEATURE_OF_ACCOUNT').\n",
" simplePath()\n",
" ).\n",
" times(2).dedup().\n",
" in('ACCOUNT').in('FEATURE_OF_TRANSACTION').hasLabel('IpAddress').\n",
" path().\n",
" by(\n",
" project('type', 'value').\n",
" by(label).\n",
" by(valueMap('account_number', 'value', 'amount', 'created'))\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "881dd97b",
"metadata": {},
"source": [
"## Identity Theft\n",
"\n",
"Identity theft is an example of third-party fraud. With identity theft, an individual or group of people steal the account and personal details of one or more account holders, and use them to purchase goods and services.\n",
"\n",
"Transactions committed via identity theft are typically concentrated in a short window of time. The transactions are often associated with phone numbers or IP addresses not normally seen in the account holder's transaction history, and may reflect purchases and purchase amounts quite uncharacteristic of the account holder.\n",
"\n",
"### Single victim, multiple unusual transaction features\n",
"\n",
"The following query shows multiple transactions for a single account over a 2-day period. The transactions originate from several IP addresses and phone numbers not associated with the account. These are perhaps fraudulent transactions. Note that there is one transaction, for 971 dollars, that was issued from an IP address associated with the account: this is likely a legitimate transaction."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a46ff6ae",
"metadata": {},
"outputs": [],
"source": [
"%%gremlin -g type\n",
"\n",
"g.V('account-17592186055331').\n",
" union(\n",
" in('ACCOUNT').\n",
" and(\n",
" has('created', gte(datetime('2021-01-10'))), \n",
" has('created', lte(datetime('2021-01-12')))\n",
" ).\n",
" in('FEATURE_OF_TRANSACTION'),\n",
" in('FEATURE_OF_ACCOUNT')\n",
" ).\n",
" path().\n",
" by(\n",
" project('type', 'value').\n",
" by(label).\n",
" by(valueMap('account_number', 'value', 'amount', 'created', 'name'))\n",
" ) "
]
},
{
"cell_type": "markdown",
"id": "d7d153c1",
"metadata": {},
"source": [
"Here's a similar query, but instead of starting with an account, we start with a specific transaction. The results are similar, however, showing the constellation of other transactions and features associated with the starting transaction."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5af039f4",
"metadata": {},
"outputs": [],
"source": [
"%%gremlin -g type\n",
"\n",
"g.V('a5fdd9f7b64c48d2823da969d885373d').\n",
" out('ACCOUNT').as('account').\n",
" union(\n",
" in('ACCOUNT').\n",
" and(\n",
" has('created', gte(datetime('2021-01-10'))), \n",
" has('created', lte(datetime('2021-01-12')))\n",
" ).\n",
" in('FEATURE_OF_TRANSACTION'),\n",
" in('FEATURE_OF_ACCOUNT')\n",
" ).\n",
" path().from('account').\n",
" by(\n",
" project('type', 'value').\n",
" by(label).\n",
" by(valueMap('account_number', 'value', 'amount', 'created', 'name'))\n",
" ) "
]
},
{
"cell_type": "markdown",
"id": "4e28e6e6",
"metadata": {},
"source": [
"### Multiple victims, same transaction features\n",
"\n",
"Another pattern of identity theft is characterised by multiple transactions being issued in a short space of time for many different accounts - all from the same IP address.\n",
"\n",
"The following query shows an example of identity theft comprising transactions submitted from the same IP address that target multiple victims. \n",
"\n",
"The query starts from a particular account (0028-5873-0233-1601). We see several transactions issued from an IP address associated with this starting account: these are likely legitimate transactions. But the results also show one transaction for this account and several transactions for many other accounts, all issued in a short space of time from an IP address not associated with the starting account. These are likely fraudulent transactions committed via identity theft."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2aa9c9ab",
"metadata": {},
"outputs": [],
"source": [
"%%gremlin -g type -p v,inV,outV,inV,outV\n",
"\n",
"g.V('account-28587302331601').\n",
" in('ACCOUNT').\n",
" and(\n",
" has('created', gte(datetime('2021-01-03'))), \n",
" has('created', lte(datetime('2021-01-07')))\n",
" ).\n",
" in('FEATURE_OF_TRANSACTION').\n",
" out('FEATURE_OF_TRANSACTION').\n",
" and(\n",
" has('created', gte(datetime('2021-01-03'))), \n",
" has('created', lte(datetime('2021-01-07')))\n",
" ).\n",
" union(\n",
" out('ACCOUNT'),\n",
" out('MERCHANT')\n",
" ).\n",
" path().\n",
" by(\n",
" project('type', 'value').\n",
" by(label).\n",
" by(valueMap('account_number', 'value', 'amount', 'created', 'name'))\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "b80f79f6",
"metadata": {},
"source": [
"Here's a similar query, but instead of starting with an account, we start with a specific transaction. The results are similar, however, showing the constellation of other transactions and features associated with the starting transaction."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "44044260",
"metadata": {},
"outputs": [],
"source": [
"%%gremlin -g type -p v,inV,outV,inV,outV\n",
"\n",
"g.V('7dbf0290893441a699a30362d4158d4c').\n",
" out('ACCOUNT').as('account').\n",
" in('ACCOUNT').\n",
" and(\n",
" has('created', gte(datetime('2021-01-03'))), \n",
" has('created', lte(datetime('2021-01-07')))\n",
" ).\n",
" in('FEATURE_OF_TRANSACTION').\n",
" out('FEATURE_OF_TRANSACTION').\n",
" and(\n",
" has('created', gte(datetime('2021-01-03'))), \n",
" has('created', lte(datetime('2021-01-07')))\n",
" ).\n",
" union(\n",
" out('ACCOUNT'),\n",
" out('MERCHANT')\n",
" ).\n",
" path().from('account').\n",
" by(\n",
" project('type', 'value').\n",
" by(label).\n",
" by(valueMap('account_number', 'value', 'amount', 'created', 'name'))\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "16403581",
"metadata": {},
"source": [
"## Find all potential identity theft\n",
"\n",
"The following query finds all transactions that were made from an IP address not associated with any accounts, but which was used in at least 4 transactions. Like the query that finds all potential fraud rings, this is more of an analytics query, which surveys all transactions in the database. The `where()` step helps tune the sensitivity of the query."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f82b2adf",
"metadata": {},
"outputs": [],
"source": [
"%%gremlin -g type -p v,inV,outV,inV,outV\n",
"\n",
"g.E().hasLabel('FEATURE_OF_TRANSACTION').\n",
" outV().as ('feature').\n",
" where(\n",
" and(\n",
" out('FEATURE_OF_ACCOUNT').count().is(eq(0)),\n",
" out('FEATURE_OF_TRANSACTION').count().is(gte(4))\n",
" ) \n",
" ).\n",
" out('FEATURE_OF_TRANSACTION').\n",
" union(\n",
" out('ACCOUNT'),\n",
" out('MERCHANT'),\n",
" in('FEATURE_OF_TRANSACTION')\n",
" ).\n",
" path().from('feature').\n",
" by(\n",
" project('type', 'value').\n",
" by(label).\n",
" by(valueMap('account_number', 'value', 'amount', 'created', 'name'))\n",
" )\n",
" \n",
" "
]
},
{
"cell_type": "markdown",
"id": "32461e0c",
"metadata": {},
"source": [
"## Building a Fraud Detection Solution\n",
"\n",
"A fraud graph is but one building block in a fraud detection solution. To define the queries used to detect unusual and potentially fraudulent activities, we must draw on the insights of Subject Matter Experts (SMEs). To help decide whether a particular event or constellation of data that we have found represents fraud, we require access to an expert predictive or decision-making process: this can vary, from an SME conducting further investigations into potential instances of fraud highlighted by the graph, to a machine learning model (ML) hosted by a service such as Amazon SageMaker scoring the results of a graph query.\n",
"\n",
"### Formulating fraud detection graph queries\n",
"\n",
"Neptune can very quickly find instances of data that match _known_ patterns of potentially fraudulent behaviour. But how do we determine what these patterns look like in the first place?\n",
"\n",
"This is a task best accomplished by an SME reviewing historical data containing known instances of fraud to identify the kinds of structure that characterise fraud. The patterns identified by the SME are then encoded as graph queries (the kinds of graph queries shown throughout this notebook) that can be run repeatedly against existing and new graph data.\n",
"\n",
"### Evaluating fraud detection graph query results\n",
"\n",
"The results of a fraud detection graph query help us understand a transaction in the context of other data present in the system. But these query results are rarely sufficient in and of themselves to determine whether a transaction or series of transactions is indeed fraudulent. The results of the queries we've seen thoughout this notebook represent _potential_ instances of fraudulent behaviour. To determine whether the data that we have matched in the graph should really be considered fraud, we require access to an expert decision-making process. In some circumstances this might entail an SME further investigating the accounts and transactions highlighted by a graph query. In other cases, the solution can use a predictive ML model to score a set of results. \n",
"\n",
"### Fraud detection architectures\n",
"\n",
"Amazon Neptune can be used with other Amazon Web services to build fraud detection solutions, as shown in [this diagram](https://d1.awsstatic.com/products/Neptune/fraud_graph_neptune.ffeb117372fb1e120fc6f986126120dcb3ddde86.png). You can load data directly into Neptune using query APIs, or from relational databases using the Amazon Web Services Database Migration Service. Neptune also supports bulk loading data from Amazon S3. Neptune can then be used in conjunction with Amazon SageMaker to train predictive models.\n"
]
},
{
"cell_type": "markdown",
"id": "cf1625da",
"metadata": {},
"source": [
"## Conclusion\n",
"\n",
"This notebook has shown how you can use Amazon Neptune to create a fraud graph as part of a fraud detection solution. We've used a credit card dataset with account- and transaction-centric queries to find instances of potentially fraudulent behaviour. Real-world transaction data is tangled, messy and exceptional: in order to determine what patterns of fraud look like, and whether the results of a graph query truly represent fraud, we've suggested that the data model and queries be driven by SME insights, and that query results be assessed by an expert or predictive capability such as an ML model on Amazon SageMaker."
]
},
{
"cell_type": "markdown",
"id": "f3255a1f",
"metadata": {},
"source": [
"## What's Next?\n",
"\n",
"The examples in this notebook show how to develop a fraud graph data model and accompanying queries. To build a fraud detection solution that incorporates Neptune, we recommend the following resources:\n",
"\n",
" - [Getting Started with Amazon Neptune](https://pages.awscloud.com/AWS-Learning-Path-Getting-Started-with-Amazon-Neptune_2020_LP_0009-DAT.html) is a video-based learning path that shows you how to create and connect to a Neptune database, choose a data model and query language, author and tune graph queries, and integrate Neptune with other Amazon Web services.\n",
" - Before you begin designing your database, consult the [Amazon Web Services Reference Architectures for Using Graph Databases](https://github.com/aws-samples/aws-dbs-refarch-graph/) GitHub repo, where you can browse examples of reference deployment architectures, and learn more about building a graph data model and choosing a query language.\n",
" - For links to documentation, blog posts, videos, and code repositories with samples and tools, see the [Amazon Neptune developer resources](https://aws.amazon.com/neptune/developer-resources/).\n",
" - Neptune ML makes it possible to build and train useful machine learning models on large graphs in hours instead of weeks. To find out how to set up and use a graph neural network, see [Using Amazon Neptune ML for machine learning on graphs](https://docs.aws.amazon.com/neptune/latest/userguide/machine-learning.html).\n",
" "
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.5"
}
},
"nbformat": 4,
"nbformat_minor": 5
}