{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Demo Notebook for Metrics Correlation Model Testing\n", "\n", "#### [Download notebook](https://github.com/opensearch-project/opensearch-py-ml/blob/main/docs/source/examples/demo_metrics_correlation.ipynb)\n", "\n", "\n", "## Introduction\n", "\n", "This notebook contains an introduction to the [metrics correlation](https://opensearch.org/docs/latest/ml-commons-plugin/algorithms/#metrics-correlation) algorithm released in OpenSearch 2.7. We will work through an example using data from the [Server Machine Dataset](https://github.com/NetManAIOps/OmniAnomaly) and include commentary on the objective, configuration, and output of the algorithm itself." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 0: Imports\n", "\n", "Please install the following packages from the terminal if you haven't already. They can be also installed from the notebook by uncommenting and executing the line in the second code block." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": { "is_executing": true } }, "outputs": [], "source": [ "%load_ext autoreload\n", "%autoreload 2" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "# !pip install pandas matplotlib numpy opensearch-py" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "pycharm": { "is_executing": true } }, "outputs": [], "source": [ "# import this to stop opensearch-py-ml from yelling every time a DataFrame connection made\n", "import warnings\n", "warnings.filterwarnings('ignore')" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import opensearch_py_ml as oml\n", "from opensearchpy import OpenSearch\n", "\n", "# Import standard test settings for consistent results\n", "from opensearch_py_ml.conftest import *" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 1: Set up clients and define helper functions\n" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "CLUSTER_URL = 'https://localhost:9200'\n", "\n", "def get_os_client(cluster_url = CLUSTER_URL,\n", " username='admin',\n", " password='admin'):\n", " \"\"\"\n", " Get OpenSearch client\n", " :param cluster_url: cluster URL like https://ml-te-netwo-1s12ba42br23v-ff1736fa7db98ff2.elb.us-west-2.amazonaws.com:443\n", " :return: OpenSearch client\n", " \"\"\"\n", " client = OpenSearch(\n", " hosts=[cluster_url],\n", " http_auth=(username, password),\n", " verify_certs=False,\n", " timeout = 30\n", " )\n", " return client\n" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "client = get_os_client()" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "#connect to ml_common client with OpenSearch client\n", "from opensearch_py_ml.ml_commons import MLCommonClient\n", "ml_client = MLCommonClient(client)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 2: Preparing data\n" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", " | 0 | \n", "1 | \n", "... | \n", "998 | \n", "999 | \n", "
---|---|---|---|---|---|
0 | \n", "0.023229 | \n", "0.019744 | \n", "... | \n", "0.025552 | \n", "0.024390 | \n", "
1 | \n", "0.910256 | \n", "0.910256 | \n", "... | \n", "0.912821 | \n", "0.912821 | \n", "
2 | \n", "0.338983 | \n", "0.339925 | \n", "... | \n", "0.067797 | \n", "0.068738 | \n", "
3 | \n", "0.011496 | \n", "0.012064 | \n", "... | \n", "0.013625 | \n", "0.014334 | \n", "
4 | \n", "0.013699 | \n", "0.013699 | \n", "... | \n", "0.027397 | \n", "0.041096 | \n", "
5 | \n", "0.150411 | \n", "0.166988 | \n", "... | \n", "0.131524 | \n", "0.131827 | \n", "
6 | \n", "0.405081 | \n", "0.399555 | \n", "... | \n", "0.484677 | \n", "0.481303 | \n", "
7 | \n", "0.005224 | \n", "0.008237 | \n", "... | \n", "0.007500 | \n", "0.009327 | \n", "
8 | \n", "0.009096 | \n", "0.007172 | \n", "... | \n", "0.011545 | \n", "0.009620 | \n", "
9 rows × 1000 columns
\n", "