{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "### 电池单体一致性差报警\n", "\n", "- **问题描述**：预测未来两周内会不会出现 “电池单体一致性差报警”。\n", "- **问题分析**：将所有车辆数据进行聚合分析，设置最长时间窗口(两周)，并提取窗口内的特征。未来两周内如果有报警，标签为1，否则为0。将问题转化为一个Time Series问题，输出为二分类概率。\n", "- **特征向量**：在该示意代码中考虑的特征包括7个维度，分别为：“采集时间”，“总电压(V)”，“总电流(A)”，“电池单体电压最高值(V)”，“电池单体电压最低值(V)”， “最高温度值(℃)”， “最低温度值(℃)”\n", "\n", "\n", "#### 注意：该示例代码中数据为人造数据，并非真实数据，其车架号等信息均为随机生成的，仅作示例使用，客户可根据自己的数据集合调整算法，并进行部署。\n", "\n", "#### 目录\n", "\n", "1. [第三方依赖库安装](#第三方依赖库安装)\n", "1. [数据可视化](#数据可视化)\n", "1. [数据集划分](#数据集划分)\n", "1. [基于Xgboost分类模型进行电池单体一致性差预测](#基于Xgboost分类模型进行电池单体一致性差预测)\n", " 1. [Xgboost模型搭建与训练](#Xgboost模型搭建与训练)\n", " 1. [Xgboost模型部署](#Xgboost模型部署)\n", " 1. [Xgboost模型测试](#Xgboost模型测试)\n", "1. [资源回收](#资源回收)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 第三方依赖库下载" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# In China, please use Tsing University opentuna sources to speedup the downloading (because of the exists of Great Firewall of China)\n", "# ! pip install -i https://opentuna.cn/pypi/web/simple --upgrade pip==20.3.1\n", "# ! pip install -i https://opentuna.cn/pypi/web/simple pandas==1.1.5\n", "# ! pip install -i https://opentuna.cn/pypi/web/simple seaborn==0.11.0\n", "# ! pip install -i https://opentuna.cn/pypi/web/simple --upgrade sagemaker==2.18.0\n", "\n", "! pip install --upgrade pip==20.3.1\n", "! pip install pandas==1.1.5\n", "! pip install seaborn==0.11.0\n", "! pip install --upgrade sagemaker==2.18.0" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 数据可视化\n", "\n", "数据集合中包含了“车架号VIN”，“采集时间Date”，“总电压(V)”，“总电流(A)”，“电池单体电压最高值(V)”，“电池单体电压最低值(V)”， “最高温度值(℃)”， “最低温度值(℃)”。最后一列的Label表征数据在该天是否发生了电池单体一致性差报警，0表示没有报警，1表示发生报警。" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "\n", "data_path = './series_samples.csv'\n", "df = pd.read_csv(data_path)\n", "df.head(30)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import seaborn as sns\n", "import matplotlib.pyplot as plt\n", "\n", "# 计算相关性系数\n", "df.corr()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# 联合分布可视化\n", "sns.pairplot(df)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 数据集划分\n", "\n", "在基于Xgboost的分类算法中，我们选取历史14天的样本并聚合在一起形成一个 14*6 维度的特征向量，基于该向量预测未来14天内是否出现电池一致性差报警。数据集划分分为以下几步完成：\n", "\n", "- 将历史特征聚合成一个大的向量，并将未来14天内的电池一致性报警标签进行聚合，举例说明：对于第x天而言，第x-14天（含）到第x-1天（含）这共计14天的特征（维度为6）聚合成一个84维度的特征向量；若第x天（含）到第x+13天（含）这未来14天出现电池单体一致性报警，则该特征向量对应的分类标签为1；反之为0. 通过滑窗操作对所有数据执行上述操作，提取所有的训练/验证样本\n", "- 提取了所有样本之后按照4:1的比例进行训练集/测试集划分\n", "- 划分数据集之后将数据集上传至S3桶" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "\n", "vins = df['VIN'].unique()\n", "df = df.fillna(-1)\n", "\n", "context_length = 14 # 特征向量为历史14天特征组合而成\n", "predict_length = 14 # 预测未来两周内是否会出现电池一致性差报警\n", "\n", "all_samples = list()\n", "positive, negative = 0, 0\n", "\n", "for vin in vins:\n", " sequence = df.loc[df.loc[:,'VIN'] == vin].values[:, 2:]\n", " \n", " for index in range(len(sequence)):\n", " if index < context_length:\n", " continue\n", " \n", " feats = sequence[index - context_length:index, :-1].flatten()\n", " label = 1.0 if 1.0 in sequence[index: index + predict_length, -1] else 0.0\n", " \n", " sample = np.concatenate(([label], feats), axis=0)\n", " all_samples.append(sample)\n", " \n", " if label == 0:\n", " negative += 1\n", " else:\n", " positive += 1\n", "\n", "all_samples = np.array(all_samples)\n", "np.savetxt(\"xgboost_samples_label_first.csv\", all_samples, delimiter=\",\")\n", "print('Positive samples = {}'.format(positive))\n", "print('Negative samples = {}'.format(negative))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# 按照4:1比例划分训练集和验证集\n", "train_val_ratio = 4.0\n", "np.random.shuffle(all_samples)\n", "index_split = int(len(all_samples) * train_val_ratio / (1.0 + train_val_ratio))\n", "\n", "train_data = all_samples[0:index_split, :]\n", "val_data = all_samples[index_split:, :]\n", "\n", "print('train_data shape = {}'.format(train_data.shape))\n", "print('val_data shape = {}'.format(val_data.shape))\n", "\n", "np.savetxt(\"xgboost_train.csv\", train_data, delimiter=\",\")\n", "np.savetxt(\"xgboost_val.csv\", val_data, delimiter=\",\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# 将划分好的训练集/验证集上传至S3桶\n", "import boto3\n", "import sagemaker\n", "account_id = boto3.client('sts').get_caller_identity()['Account']\n", "region = boto3.Session().region_name\n", "\n", "# bucket name SHOULD BE SAME with the training bucket name in deployment phase\n", "bucket_name = 'bev-bms-train-{}-{}'.format(region, account_id)\n", "\n", "# create bucket\n", "s3 = boto3.client(\"s3\")\n", "existing_buckets = [b[\"Name\"] for b in s3.list_buckets()['Buckets']]\n", "\n", "if bucket_name in existing_buckets:\n", " print('Bucket {} already exists'.format(bucket_name))\n", "else:\n", " s3.create_bucket(Bucket=bucket_name)\n", "\n", "# upload train/val dataset to S3 bucket\n", "s3_client = boto3.Session().resource('s3')\n", "s3_client.Bucket(bucket_name).Object('train/xgboost_train.csv').upload_file('xgboost_train.csv')\n", "s3_client.Bucket(bucket_name).Object('val/xgboost_val.csv').upload_file('xgboost_val.csv')\n", "\n", "\n", "# s3_input_train = sagemaker.s3_input(s3_data='s3://{}/train'.format(bucket_name), content_type='csv')\n", "# s3_input_val = sagemaker.s3_input(s3_data='s3://{}/val/'.format(bucket_name), content_type='csv')\n", "\n", "s3_input_train = sagemaker.inputs.TrainingInput(s3_data='s3://{}/train'.format(bucket_name), content_type='csv')\n", "s3_input_val = sagemaker.inputs.TrainingInput(s3_data='s3://{}/val'.format(bucket_name), content_type='csv')\n", "\n", "print('s3_input_train = {}'.format(s3_input_train))\n", "print('s3_input_val = {}'.format(s3_input_val))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 基于Xgboost分类模型进行电池单体一致性差预测\n", "\n", "该示例代码中采用的是AWS自带的xgboost分类算法，用户可以根据实际情况选择其他类型的算法或者自己定义算法进行训练和部署。\n", "\n", "参考资料：\n", "\n", "- https://docs.aws.amazon.com/zh_cn/sagemaker/latest/dg/xgboost.html\n", "- https://github.com/aws/amazon-sagemaker-examples" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Xgboost模型搭建与训练" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import boto3\n", "import sagemaker\n", "from sagemaker import get_execution_role\n", "\n", "container = sagemaker.image_uris.retrieve(framework='xgboost', region=boto3.Session().region_name, version='1.0-1')\n", "\n", "sess = sagemaker.Session()\n", "role = get_execution_role()\n", "print('Container = {}'.format(container))\n", "print('Role = {}'.format(role))\n", "\n", "xgb = sagemaker.estimator.Estimator(container,\n", " role, \n", " instance_count=1, \n", " instance_type='ml.m4.xlarge',\n", " sagemaker_session=sess,\n", " output_path='s3://{}/{}'.format(bucket_name, 'xgboost-models'),\n", " )\n", "\n", "xgb.set_hyperparameters(max_depth=7, # 树的最大深度，值越大，树越复杂, 可以用来控制过拟合，典型值是3-10\n", " eta=0.1, # 学习率， 典型值0.01 - 0.3\n", " gamma=4, # 指定了一个结点被分割时，所需要的最小损失函数减小的大小\n", " min_child_weight=6,\n", " subsample=0.8, # 样本的采样率，如果设置成0.5，那么Xgboost会随机选择一半的样本作为训练集\n", " objective='binary:logistic',\n", " num_round=200)\n", "\n", "xgb.fit({'train': s3_input_train, 'validation': s3_input_val})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Xgboost模型部署\n", "\n", "将模型部署为一个Runtime Endpoint，供用户调用进行前向推理计算" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Sagemaker Endpoint名字需要与solution部署时的Endpoint名一致\n", "sm_endpoint_name = 'battery-consistency-bias-alarm-prediction-endpoint'\n", "\n", "xgb_predictor = xgb.deploy(\n", " endpoint_name=sm_endpoint_name,\n", " initial_instance_count=1,\n", " instance_type='ml.m4.xlarge'\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Xgboost模型测试\n", "\n", "模型的性能指标有以下几个：\n", "\n", "- 真阳(True Positive): 实际上是正例的数据被分类为正例\n", "- 假阳(False Positive): 实际上是反例的数据被分类为正例\n", "- 真阴(True Negative): 实际上是反例的数据被分类为反例\n", "- 假阴(False Negative): 实际上是正例的数据被分类为反例\n", "- 召回率(Recall): Recall = TPR = TP / (TP + FN), 衡量的数据集中所有的正样本被模型区分出来的比例\n", "- 精确率(Persion): Persion = TP / (TP + FP), 衡量的模型区分出来的正样本中真正为正样本的比例\n", "- 假阳率(False Positive Rate, FPR): FPR = FP / (FP + TN), 衡量的是 被错分的反例 在所有反例样本中的占比\n", "\n", "ROC曲线：当选择的阈值越小，其TPR越大，FPR越大\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# 验证集合性能分析\n", "xgb_predictor.serializer = sagemaker.serializers.CSVSerializer()\n", "\n", "gt_y = list()\n", "pred_y = list()\n", "\n", "for i in range(len(val_data)):\n", " feed_feats = val_data[i, 1:]\n", " \n", " prob = xgb_predictor.predict(feed_feats).decode('utf-8') # format: string\n", " prob = np.float(prob) # format: np.float\n", " \n", " gt_y.append(val_data[i, 0])\n", " pred_y.append(prob)\n", " \n", "from sklearn.metrics import roc_curve, auc, confusion_matrix, f1_score, precision_score, recall_score\n", "from pandas.tseries.offsets import *\n", "import matplotlib.pyplot as plt\n", "\n", "def draw_roc(test_Y, pred):\n", " fpr, tpr, thresholds = roc_curve(test_Y, pred, pos_label=1)\n", " auc_score = auc(fpr, tpr)\n", " print('Auccuracy = {}'.format(auc_score))\n", "\n", " plt.figure()\n", " lw = 2\n", " plt.plot(fpr, tpr, color='darkorange', lw=lw, label='ROC curve (area = %0.2f)' % auc_score)\n", " plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')\n", " plt.xlim([0.0, 1.0])\n", " plt.ylim([0.0, 1.05])\n", " plt.xlabel('False Positive Rate')\n", " plt.ylabel('True Positive Rate')\n", " plt.title('Receiver operating characteristic example')\n", " plt.legend(loc=\"lower right\")\n", " plt.show()\n", " \n", "\n", "def p_r_f1(test_Y, pred, thres=0.5):\n", " pred = [p>=thres and 1 or 0 for p in pred]\n", " cm = confusion_matrix(test_Y, pred)\n", " precision = precision_score(test_Y, pred)\n", " recall = recall_score(test_Y, pred)\n", " f1 = f1_score(test_Y, pred)\n", " print('Confusion Matrix = \\n{}'.format(cm))\n", " print('Precision = {}'.format(precision))\n", " print('recall = {}'.format(recall))\n", " print('f1_score = {}'.format(f1))\n", "\n", "draw_roc(gt_y, pred_y)\n", "p_r_f1(gt_y, pred_y, thres=0.15)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Sagemaker Endpoint调用测试\n", "import boto3\n", "import base64\n", "import json\n", "import time\n", "import numpy as np\n", "\n", "runtime_sm_client = boto3.client(service_name='sagemaker-runtime')\n", "\n", "test_feats = np.array([\n", " 364.61335034013604, -0.2457482993197279, 3.80116836734694, 3.7861037414965986, 24.78316326530612, \n", " 23.18324829931973, 360.41936658172546, -0.2934109938114305, 3.757078631234074, 3.74262977793957, \n", " 24.97961412449945, 23.499089916272297, 363.06764705882347, 0.05309597523219842, 3.7850770123839013,\n", " 3.770754643962848, 25.468653250773997, 23.89357585139319, 369.6609137055837, 0.3319796954314729, \n", " 3.8541254532269758, 3.838007251631617, 23.976069615663523, 22.530094271211023, 364.7919594594594,\n", " -0.003918918918919656, 3.80265472972973, 3.788666891891892, 23.415540540540537, 21.92297297297297, \n", " 353.11501792114694, 0.07193548387096703, 3.680777060931899, 3.667754121863799, 26.293906810035843, \n", " 24.433691756272406, 355.7949381989405, -0.17374926427310172, 3.708674220129488, 3.696100941730429, \n", " 25.93878752207181, 24.459976456739263, 349.36256842105263, 0.8924631578947366, 3.64051747368421, \n", " 3.6289330526315786, 25.335157894736838, 23.581052631578952, 372.0972686733557, -1.7828874024526196,\n", " 3.8795345596432553, 3.864828874024527, 25.26365663322185, 23.782608695652176, 348.5370820668693, \n", " 6.805876393110436, 3.632681864235056, 3.6181013171225933, 24.893617021276604, 23.38804457953394,\n", " 348.93208923208914, -3.038738738738739, 3.635883311883312, 3.624791076791076, 26.18876018876019, \n", " 24.41269841269841, 359.3844774273346, 0.13546691403834246, 3.7457195423624, 3.7334737167594314, \n", " 26.749845392702536, 25.212739641311067, 392.0038461538462, 0.0, 4.085240384615385, \n", " 4.071028846153847, 18.39423076923077, 17.0, 364.1829856584093, 0.1546284224250319, \n", " 3.79564667535854, 3.7821192959582786, 24.286179921773144, 22.885919165580184])\n", "\n", "payload = ''\n", "for k, value in enumerate(test_feats):\n", " if k == len(test_feats) - 1:\n", " payload += str(value)\n", " else:\n", " payload += (str(value) + ',')\n", "\n", "print('Payload = \\n{}'.format(payload))\n", "\n", "t1 = time.time()\n", "\n", "response = runtime_sm_client.invoke_endpoint(\n", " EndpointName=sm_endpoint_name,\n", " ContentType='text/csv', # The MIME type of the input data in the request body.\n", " Body=payload)\n", "\n", "t2 = time.time()\n", "\n", "print('Time cost = {}'.format(t2 - t1))\n", "\n", "predicted_prob = response['Body'].read().decode()\n", "print('Predicted prob = {}'.format(predicted_prob))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 资源回收" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# xgb_predictor.delete_endpoint()" ] } ], "metadata": { "kernelspec": { "display_name": "conda_mxnet_p36", "language": "python", "name": "conda_mxnet_p36" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.10" } }, "nbformat": 4, "nbformat_minor": 4 }