{ "cells": [ { "cell_type": "markdown", "id": "d5d54120", "metadata": {}, "source": [ "# Identifying Fraud Rings Using Social Network Analytics\n", "\n", "Within the financial industry, an organization can expect to lose 3-6%, and up to 10%, of its [business to fraudulent activities](http://www.crowe.ie/wp-content/uploads/2019/08/The-Financial-Cost-of-Fraud-2019.pdf). Fraudulent activities not only impact financial aspects, but victims often have negative views of the company, leading to negative market sentiment. Overall, these fraudulent activities have a significant impact on a business, in terms of both consumer confidence and bottom-line revenue. Due to the impact of these illicit activities on the bottom line, companies expend significant time and money to detect and prevent fraud. \n", "\n", "When dealing with fraud, there are two main components to a robust fraud system: fraud detection and fraud prevention. In the fraud detection component of a system, the main goals are to develop a system and methodology that allows for the rapid discovery of fraudulent activities. This usually involves a posterior evaluation of data, such as transactions, users, credit cards, etc. to determine what patterns or combinations represent actual fraud. This process usually involves a human-in-the-loop system where automated processes flag likely or potential fraudulent activities, which are then evaluated by an expert in the domain to determine the legitimacy, or illegitimacy, of the activities flagged. The output of this process is a set of known and evolving patterns of fraud that are fed into a fraud prevention system. Generally, this consists of a real-time system that compares a transaction, or a set of transactions, against the known fraudulent patterns identified by the fraud detection system. The objective of this fraud prevention system is to reduce and prevent fraudulent activities from occurring in the first place. \n", "\n", "\n", "## Challenges of Detecting Fraud\n", "\n", "When dealing with fraud, it is often helpful to understand some challenges of finding fraudulent activities when looking into data. Often this is aided by first understanding the definition and nature of fraud. While there are many definitions of fraud, my favorite is:\n", "\n", "[Fraud is an uncommon, well-considered, imperceptibly concealed, time-evolving and often carefully organized crime which appears in many types of forms .](https://www.amazon.com/Analytics-Descriptive-Predictive-Network-Techniques/dp/1119133122)\n", "\n", "\n", "This definition highlights the complex nature of the problems we must address when working on fraud systems. First, fraud is *uncommon*. Within any system of recorded transactions, only a small fraction of these transactions consist of fraudulent or illicit activities. The sparse nature of these illicit activities complicates the nature of identifying these activities. Second, fraud is *well-considered* and *imperceptibly concealed,* meaning that fraudulent activities are rarely impulsive activities. Most fraudulent activities, at least at scale, involve multiple parties colluding together to perform actions specifically designed to exploit weaknesses in the system and elude detection. Finally, fraud is *time-evolving*. Fraudsters are continuously evolving and adapting their techniques as detection and prevention improve in an endless game of hide and seek.\n", "\n", "With these challenges in mind, many fraud detection systems take a multi-faceted approach to identifying illicit activities. In this notebook, we will focus on the use of social networks implicitly found in fraud data to identify fraud rings committing illicit transactions using a guilt-by-association approach.\n", "\n", "Fraud rings are a common issue within the industry and consist of multiple individuals or entities colluding to defraud a system. These rings may consist of family members, acquaintances, or even buyer/seller pairs that span both sides of a transaction. These rings exist across a wide array of industries, but an important aspect of these fraud rings, at least as it is related to graph analysis, is that they are strongly linked groups of entities. " ] }, { "cell_type": "markdown", "id": "d1bdbb3e", "metadata": {}, "source": [ "## Creating a fraud graph\n", "\n", "In this section we'll load a fraud graph and set some visualization options. We'll then use some openCypher queries to inspect the data model used throughout the solution.\n", "\n", "### Load data\n", "The cell below loads the example fraud graph into your Neptune cluster. When you run the cell below, a graph for an example Fraud dataset will load, which will take about 5 minutes." ] }, { "cell_type": "code", "execution_count": null, "id": "b0589d25", "metadata": {}, "outputs": [], "source": [ "%seed --model Property_Graph --dataset fraud_graph --run" ] }, { "attachments": {}, "cell_type": "markdown", "id": "50febfd8", "metadata": {}, "source": [ "Now we will install the library we will be using later on in this notebook. " ] }, { "cell_type": "code", "execution_count": null, "id": "e3987422", "metadata": {}, "outputs": [], "source": [ "pip install igraph -q" ] }, { "cell_type": "markdown", "id": "43fdc65f", "metadata": {}, "source": [ "### Set visualization and configuration options\n", "\n", "The cell below configures the visualization to use specific colors and icons for the different parts of the data model." ] }, { "cell_type": "code", "execution_count": null, "id": "854a23c6", "metadata": {}, "outputs": [], "source": [ "%%graph_notebook_vis_options\n", "\n", "{\n", " \"groups\": {\n", " \"Account\": {\n", " \"shape\": \"icon\",\n", " \"icon\": {\n", " \"face\": \"FontAwesome\",\n", " \"code\": \"\\uf2bb\",\n", " \"color\": \"red\"\n", " }\n", " },\n", " \"Transaction\": {\n", " \"shape\": \"icon\",\n", " \"icon\": {\n", " \"face\": \"FontAwesome\",\n", " \"code\": \"\\uf155\",\n", " \"color\": \"green\"\n", " }\n", " },\n", " \"Merchant\": {\n", " \"shape\": \"icon\",\n", " \"icon\": {\n", " \"face\": \"FontAwesome\",\n", " \"code\": \"\\uf290\",\n", " \"color\": \"orange\"\n", " }\n", " },\n", " \"DateOfBirth\": {\n", " \"shape\": \"icon\",\n", " \"icon\": {\n", " \"face\": \"FontAwesome\",\n", " \"code\": \"\\uf1fd\",\n", " \"color\": \"blue\"\n", " }\n", " },\n", " \"EmailAddress\": {\n", " \"shape\": \"icon\",\n", " \"icon\": {\n", " \"face\": \"FontAwesome\",\n", " \"code\": \"\\uf1fa\",\n", " \"color\": \"blue\"\n", " }\n", " },\n", " \"Address\": {\n", " \"shape\": \"icon\",\n", " \"icon\": {\n", " \"face\": \"FontAwesome\",\n", " \"code\": \"\\uf015\",\n", " \"color\": \"blue\"\n", " }\n", " },\n", " \"IpAddress\": {\n", " \"shape\": \"icon\",\n", " \"icon\": {\n", " \"face\": \"FontAwesome\",\n", " \"code\": \"\\uf109\",\n", " \"color\": \"blue\"\n", " }\n", " },\n", " \"PhoneNumber\": {\n", " \"shape\": \"icon\",\n", " \"icon\": {\n", " \"face\": \"FontAwesome\",\n", " \"code\": \"\\uf095\",\n", " \"color\": \"blue\"\n", " }\n", " }\n", " },\n", " \"edges\": {\n", " \"color\": {\n", " \"inherit\": false\n", " },\n", " \"smooth\": {\n", " \"enabled\": true,\n", " \"type\": \"straightCross\"\n", " },\n", " \"arrows\": {\n", " \"to\": {\n", " \"enabled\": false,\n", " \"type\": \"arrow\"\n", " }\n", " },\n", " \"font\": {\n", " \"face\": \"courier new\"\n", " }\n", " },\n", " \"interaction\": {\n", " \"hover\": true,\n", " \"hoverConnectedEdges\": true,\n", " \"selectConnectedEdges\": false\n", " },\n", " \"physics\": {\n", " \"minVelocity\": 0.75,\n", " \"barnesHut\": {\n", " \"centralGravity\": 0.1,\n", " \"gravitationalConstant\": -50450,\n", " \"springLength\": 95,\n", " \"springConstant\": 0.04,\n", " \"damping\": 0.09,\n", " \"avoidOverlap\": 0.1\n", " },\n", " \"solver\": \"barnesHut\",\n", " \"enabled\": true,\n", " \"adaptiveTimestep\": true,\n", " \"stabilization\": {\n", " \"enabled\": true,\n", " \"iterations\": 1\n", " }\n", " }\n", "}" ] }, { "cell_type": "markdown", "id": "5b4ce1c2", "metadata": {}, "source": [ "### Data model\n", "The fraud graph included in this example models credit card accounts, account holder information, merchants, and the transactions performed when an account holder purchases goods or services from a merchant.\n", "\n", "**Account and features**\n", "\n", "An Account has a number of features, including physical Address, IpAddress, DateOfBirth of the account holder, EmailAddress, and contact PhoneNumber. An account holder can have multiple email addresses and phone numbers.\n", "\n", "In many graph data models, these features of the account holder would be modelled as properties of the account. With fraud detection, it's important to be able to link accounts based on shared features, and to find related accounts at query time based on one or more shared features. Hence, our fraud detection application graph data model stores each feature as a separate vertex. Multiple accounts that share the same feature value - the same physical address, for example - are connected to the single vertex representing that feature value. \n", "\n", "The following query shows a single account and its associated features. After running the query, click the Graph tab to see a visualization of the results." ] }, { "cell_type": "markdown", "id": "59e92609", "metadata": {}, "source": [ "### What does my fraud graph look like for an account?" ] }, { "cell_type": "code", "execution_count": null, "id": "e9c3a2b3", "metadata": {}, "outputs": [], "source": [ "%%oc -d value -l 20\n", "MATCH p=(n)-[]-()\n", "WHERE id(n)='account-4398046519460' \n", "RETURN p\n", "LIMIT 10" ] }, { "cell_type": "markdown", "id": "c882feb8", "metadata": {}, "source": [ "### What if I only want to look at the account properties, not the transactions?\n", "\n", "While the transaction information is very important, but it is often overwhelming, especially on very active accounts. Often we want to look at only account features, to see how that specific account is connected to features of that account." ] }, { "cell_type": "code", "execution_count": null, "id": "ad0e5c6c", "metadata": {}, "outputs": [], "source": [ "%%oc -d value -l 20\n", "MATCH p=(n)-[:FEATURE_OF_ACCOUNT]-()\n", "WHERE id(n)='account-4398046519460' \n", "RETURN p\n", "LIMIT 10" ] }, { "cell_type": "markdown", "id": "9c68588e", "metadata": {}, "source": [ "### What else connects to these same features?\n", "\n", "After isolating only the account features, we may want to extend our exploration out further to see if we can find other accounts that share these features." ] }, { "cell_type": "code", "execution_count": null, "id": "316b36ca", "metadata": {}, "outputs": [], "source": [ "%%oc -d value -l 20\n", "MATCH p=(n)-[:FEATURE_OF_ACCOUNT*1..2]-()\n", "WHERE id(n)='account-4398046519460' \n", "RETURN p\n", "LIMIT 10" ] }, { "cell_type": "markdown", "id": "e6695649", "metadata": {}, "source": [ "This is starting to show some interesting information. As we can see above, there are four other accounts that all share the same birthday and one that shares the same phone number. Having a shared birthday does not seem too suspicious, as there are only so many, but sharing a birthday and sharing a phone number seems unlikely.\n", "\n", "### Find accounts with shared features?\n", "\n", "Let's take a look at a visualization of all accounts that share a feature, other than a `DateOfBirth`, with another account." ] }, { "cell_type": "code", "execution_count": null, "id": "a5058a53", "metadata": {}, "outputs": [], "source": [ "%%oc -d value -l 20\n", "MATCH p=(n:Account)<-[:FEATURE_OF_ACCOUNT]-(f)-[:FEATURE_OF_ACCOUNT]->(t:Account)\n", "WHERE NOT f:DateOfBirth\n", "RETURN p" ] }, { "cell_type": "markdown", "id": "fdab6844", "metadata": {}, "source": [ "The results show a lot of shared features. In fact, if we start to examine this data, it begins to look a lot like a social network. Using social network analytics on top of this type of graph allows us to find many interesting insights that are not apparent in the original transaction graph.\n", "\n", "## Finding Fraud Rings in your social graph" ] }, { "cell_type": "markdown", "id": "839b0ce5", "metadata": {}, "source": [ "When analyzing graph data, a frequent requirement is to infer information or connections from the underlying graph into a graph that is optimized to run analytics and algorithms. Many graphs in Neptune contain a rich collection of entities and attributes, such as our transaction graph, that while useful for transactional queries, are not required to perform analytical tasks.\n", "\n", "### Inferring a social graph from your fraud graph\n", "\n", "In this example, we are going to create a social graph by inferring that any accounts that are connected by a feature, other than `DateOfBirth`, are connected within this social network. We will then use this social network to search for rings of fraudulent accounts within a social graph.\n", "\n", "Up until now we have been working with Amazon Neptune directly. For this analysis we are going to leverage an integration between Neptune and Pandas DataFrames, supplied by [AWS SDK for pandas](https://github.com/aws/aws-sdk-pandas), to read and write data from Neptune and the [iGraph](https://igraph.org/) library to perform network analysis and run graph algorithms on top of this data.\n", "\n", "**Note:** The AWS SDK for pandas library is suitable for use with small-medium sized graphs in Neptune as large as a few 100 million edges depending on the Neptune instance size.\n", "\n", "Running the cell below will retrieve that needed data from Neptune, in this case an [edge list](https://en.wikipedia.org/wiki/Edge_list), and load it into a Pandas DataFrame that we will use for later analysis. " ] }, { "cell_type": "code", "execution_count": null, "id": "0e42b967", "metadata": {}, "outputs": [], "source": [ "import awswrangler as wr\n", "import pandas as pd\n", "import igraph as ig\n", "import graph_notebook as gn\n", "from graph_notebook.configuration.generate_config import AuthModeEnum\n", "\n", "# Get the configuration information for the notebook\n", "config = gn.configuration.get_config.get_config()\n", "iam=True if config.auth_mode==AuthModeEnum.IAM else False\n", "\n", "# Retrieve Data from neptune\n", "client = wr.neptune.connect(config.host, config.port, iam_enabled=iam)\n", "query = \"\"\"MATCH (s:Account)<-[:FEATURE_OF_ACCOUNT]-(f)-[:FEATURE_OF_ACCOUNT]->(t:Account) \n", "WHERE NOT f:DateOfBirth RETURN id(s) as source, id(t) as target\"\"\"\n", "df = wr.neptune.execute_opencypher(client, query)\n", "display(df.head(10))" ] }, { "cell_type": "markdown", "id": "cc0f52c2", "metadata": {}, "source": [ "### Detecting strongly linked communities\n", "\n", "Fraud rings are a common issue within the industry and consist of multiple individuals or entities colluding to defraud a system. These rings may consist of family members, acquaintances, or even buyer/seller pairs that span both sides of a transaction. These rings exist across a wide array of industries, but an important aspect of these fraud rings, at least as it is related to graph analysis, is that they are strongly linked groups of entities. \n", "\n", "In networking analysis, there is an entire class of graph algorithms, known as community detection algorithms, that evaluates how groups of nodes are connected or partitioned from one another. Use community detection algorithms to determine strongly linked groups of entities. While there are a large number of these algorithms, the most commonly used community detection algorithms are Weakly Connected Components, Louvain, and Label Propagation. \n", "\n", "#### Finding strongly linked communities\n", "\n", "Now that we have created our social graph, the first step in analyzing this graph for fraud rings is to find the communities of entities that are highly connected to each other, which may represent collusion. Running the cell below will run the Weakly Connected Components algorithm on the data, which will assign a unique component value to all vertices in the graph that are connected by a path, regardless of the edge direction." ] }, { "cell_type": "code", "execution_count": null, "id": "b11d766b", "metadata": {}, "outputs": [], "source": [ "# Create a graph from the results returned\n", "g = ig.Graph.TupleList(df.itertuples(index=False), directed=True, weights=False)\n", "\n", "# Run WCC on the data\n", "wcc = g.clusters(mode=\"weak\")\n", "print(f'Their are {len(wcc)} communities in this data set.')\n", "\n", "giant=wcc.giant()\n", "print(f'The largest community has {len(giant.vs)} accounts.')\n", "\n", "print(f'The size histogram for these clusters is:')\n", "print(wcc.size_histogram())\n" ] }, { "cell_type": "markdown", "id": "ef834545", "metadata": {}, "source": [ "Running this algorithm we see that our dataset contains 123 communities with the largest one having 18 accounts. From looking at the size histogram, we can also see that there are two clusters which have an anomalously large number of accounts. Since this is often a sign of collusion between accounts, let us further examine the largest of these communities.\n", "\n", "### Identifying most important members in a community\n", "\n", "Once a set of communities are identified, the next step involves identifying the most important entities in one or more of those communities. This is accomplished using a centrality algorithm, which will identify the most important entities, enabling analysts to start their investigation with the “big fish” in the potential ring. There are many different centrality algorithms, each providing a slightly different method for determining the importance of a node. The most commonly used ones are: Degree, PageRank, Betweenness, Closeness, and Eigenvector.\n", "\n", "For our analysis below we will run [PageRank](https://en.wikipedia.org/wiki/PageRank), which is an algorithm that was originally developed to rank web pages but is now commonly applied to many other problems. PageRank returns a value that represents the relative importance of an account within a community based on its relationships and the importance of the corresponding accounts. In the end, the higher the PageRank value returned, the more important the account is inside the community." ] }, { "cell_type": "code", "execution_count": null, "id": "ab56de60", "metadata": {}, "outputs": [], "source": [ "pg = giant.pagerank()\n", "print('\\n'.join(map(str, pg)))" ] }, { "cell_type": "markdown", "id": "34ae6284", "metadata": {}, "source": [ "Great, we have now analyzed our inferred social network to identify the strongly connected communities within that network and prioritized the most important accounts within the anomalous community. Now that we have done this analysis we need to save this data back into our original graph to enable real-time investigations by trained fraud analysts. \n", "\n", "### Storing the risk values back into the graph\n", "\n", "To store our data back into our original graph, we can once again use our AWS SDK for pandas integration to save a Pandas DataFrame into Neptune. To accomplish this, we will first construct a DataFrame consisting of the account id, as well as the community/component value, as well as the associated PageRank value. We then save this DataFrame back to Neptune using the [`to_property_graph()`](https://aws-sdk-pandas.readthedocs.io/en/stable/stubs/awswrangler.neptune.to_property_graph.html#awswrangler.neptune.to_property_graph) method. " ] }, { "cell_type": "code", "execution_count": null, "id": "fcd4dcbb", "metadata": {}, "outputs": [], "source": [ "rows=[]\n", "for idx, c in enumerate(wcc):\n", " for item in c:\n", " rows.append({'~id': str(g.vs[item]['name']), 'component(single)': idx})\n", "\n", "for idx, v in enumerate(giant.vs):\n", " r = next(s for s in rows if s['~id'] == v['name'])\n", " r['pg(single)'] = pg[idx]\n", "\n", "new_df=pd.DataFrame(rows, columns=['~id','component(single)', 'pg(single)'])\n", "res = wr.neptune.to_property_graph(client, new_df, use_header_cardinality=True, batch_size=100)\n", "print(\"Save Complete\")" ] }, { "cell_type": "markdown", "id": "301adab7", "metadata": {}, "source": [ "## Analyzing the results\n", "\n", "\n", "A critical and ongoing part of any fraud workflow is to have a mechanism to enable analysts to investigate and prove/disprove that a potentially fraudulent activity exists. " ] }, { "cell_type": "markdown", "id": "d9ffa22f", "metadata": {}, "source": [ "### Find the most important accounts to examine first\n", "\n", "A common workflow for a skilled fraud analyst is to start by retrieving a list of prioritized accounts to look at based on the “risk” associated with the entity. In the query below, we will find the top 5 most important accounts to examine." ] }, { "cell_type": "code", "execution_count": null, "id": "2b1ec9f1", "metadata": {}, "outputs": [], "source": [ "%%oc -d value -l 20\n", "\n", "MATCH (a:Account) WHERE a.pg IS NOT NULL RETURN id(a), a.component, a.pg ORDER BY a.pg DESC LIMIT 5" ] }, { "cell_type": "markdown", "id": "ad105702", "metadata": {}, "source": [ "### Explore their connections\n", "The ability to visually explore graphs is a powerful tool that helps fraud analysts understand how certain account are connected. Let's take a look at the most important account information and see the graph of the surrounding connections. " ] }, { "cell_type": "code", "execution_count": null, "id": "fc0a9eb4", "metadata": { "scrolled": false }, "outputs": [], "source": [ "%%oc -d value -l 20\n", "\n", "MATCH p=(a)-[*1..2]-()\n", "WHERE id(a)='account-4398046511937'\n", "RETURN p" ] }, { "cell_type": "markdown", "id": "6d60c51f", "metadata": {}, "source": [ "### Mark as Fraud/Not Fraud\n", "\n", "Visual inspection, combined with the domain expertise of a fraud analyst, is a critical factor in being able to determine if anomalous patterns in a graph represent actual fraud or legitimate activity. Expert analysts are skilled in looking at the patterns of transactions and connections and the structural connections between items to determine the legitimacy of an account/transaction. Once they have made this determination, they will often flag these accounts/transactions as fraudulent in the graph to aid in future investigations.\n", "\n", "Let's mark the account above as fraudulent by setting the `isFraud` property to `True`" ] }, { "cell_type": "code", "execution_count": null, "id": "e4b36504", "metadata": {}, "outputs": [], "source": [ "%%oc -d value -l 20\n", "MATCH (a)\n", "WHERE id(a)='account-4398046511937'\n", "SET a.isFraud=True\n", "RETURN a" ] }, { "cell_type": "markdown", "id": "1b11de2f", "metadata": {}, "source": [ "### Find all fraudulent accounts within five hops \n", "\n", "Now that we have completed our investigation of `account-4398046511937` let's take a look at another account from our list above `account-21990232559534`. In addition to looking at the connections for an account, as shown above, another common use of graphs when analyzing anomalous activity is to look how closely an account is connected to a known fraudulent account. \n", "\n", "Let's take a look at this new account and see if it is connected to any marked fraudulent accounts." ] }, { "cell_type": "code", "execution_count": null, "id": "74dffa64", "metadata": {}, "outputs": [], "source": [ "%%oc -d value -l 20\n", "\n", "MATCH p=(a)-[:FEATURE_OF_ACCOUNT|ACCOUNT*1..5]-(b)\n", "WHERE id(a)='account-21990232559534'\n", "AND b.isFraud=True\n", "RETURN p LIMIT 1" ] }, { "cell_type": "markdown", "id": "b37c65a0", "metadata": {}, "source": [ "Wow, that account shares a phone number with a known fraudster so it looks suspicious, we should pass this account on to our downstream processes for action. \n", "\n", "## Conclusion\n", "\n", "This notebook has shown how you can use Amazon Neptune and AWS SDK for pandas to run analytics on your data to detect fraud rings. We've used a credit card dataset with account- and transaction-centric queries to infer a social network from this data and then use that social network to look for tightly connected groups of individuals. We then identified the most influential people within these groups and stored this information within our graph. Using this information we were able to explore the connections around the most influential people to identify other potentially fraudulent accounts.\n", "\n", "Combating fraud is an ongoing challenge for any organization. The faster a team can identify fraud and the more they do, the more efficient anti-fraud systems become, preventing significant financial losses. Finding and understanding fraud rings is a problem that requires the ability to query, analyze, and explore the connections between accounts, transactions, and account features. Combining the ability to query a graph with the ability to run network analysis and graph algorithms on top of that data enables us to derive novel insights from this data. " ] } ], "metadata": { "kernelspec": { "display_name": "Python 3.8.10 64-bit ('3.8.10')", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.10" }, "vscode": { "interpreter": { "hash": "a974f8b3b9f8334bb022754c2deb5119d965db26467015db28878a5b98c93519" } } }, "nbformat": 4, "nbformat_minor": 5 }