{ "cells": [ { "cell_type": "markdown", "id": "f0f1843d-5f5c-425f-a5d3-aec7b58727c2", "metadata": {}, "source": [ "# CLI Environment Setup\n", "Setup of the Managed KX environment beta. \n", "\n", "## Prerequisits\n", "- Region: **us-east-1**\n", "- Beta AWS CLI for Managed KX installed \n", "- AWS Account in which to install Managed KX \n", " - Account has been allow listed for Managed KX beta \n", "- S3 bucket for data and code\n", " - Grant GetObject* to the service principal \"finspace.amazonaws.com\" \n", "- Admin access to the AWS Account \n", "- KMS key for use by the environment\n", "- File from Welcome packet\n", " - example database file: hdb.tar.gz\n", " - example code for clusters: code.zip\n", "\n", "## Managed kdb Insights Archtecture\n", "\n", "\n", "## Steps\n", "- Step 0: Check CLI \n", "- Step 1: Create Envirironment \n", "- Step 2: Entitle S3 with DedicatedServiceAccountId \n", "- Step 3: Create Database \n", "- Step 4: Create Changeset \n", "- Step 5: Create Cluster for Database \n", "- Step 6: Roles, Policies, Permissions\n", "- Step 7: Use q\n", "- Step 8: Clean Up\n", "\n", "## Hints\n", "### 1: use jq to parse json output\n", "```\n", "aws finspace get-kx-cluster --environment-id=$ENV_ID --cluster-name=\\$CLUSTER_NAME | jq -r '.status'\n", "aws finspace get-kx-environment --environment-id=$ENV_ID | jq -r '.environment.status'\n", "```" ] }, { "cell_type": "code", "execution_count": null, "id": "b6428a5b-6430-4fe1-87d5-9cf85bff5aa5", "metadata": {}, "outputs": [], "source": [ "# these environment variables represent prerequisits\n", "export ACCOUNT_ID=612841383594\n", "export KMS_KEY_ARN=\"arn:aws:kms:us-east-1:612841383594:key/bbfad1fa-9e38-47f1-986d-33fb976a9ec4\"\n", "\n", "export S3_BUCKET=\"kdb-demo-612841383594-kms\"\n", "#export S3_BUCKET=\"kdb-demo-612841383594\"\n", "\n", "export TODAY=`date +'%Y%m%d'`\n", "\n", "export ENV_NAME=\"Managed_kdb_$TODAY\"\n", "\n", "export DB_NAME=\"welcomedb\"\n", "\n", "export CLUSTER_NAME=\"hdb-cluster-$DB_NAME\"\n", "\n", "# kdb username to be created and allowed to connect to cluster\n", "export KDB_USERNAME=\"bob\"\n", "\n", "# role used by users for connecting to clusters\n", "export ROLE_NAME=kdb-cluster-user\n" ] }, { "cell_type": "markdown", "id": "8a01f348-ea29-461f-8fe5-8437fd714aaa", "metadata": {}, "source": [ "## Credentials\n", "You can define the environment variables for AWS Credentials below or define these values in your ~/.aws/credentials file, under a profile such as \\[default\\]" ] }, { "cell_type": "code", "execution_count": null, "id": "850939b8-6c03-4e7f-869e-2fbd3774380c", "metadata": {}, "outputs": [], "source": [ "# Credentials\n", "export AWS_ACCESS_KEY_ID=ASIAY5MBRM2VJLJ26FHD\n", "export AWS_SECRET_ACCESS_KEY=dV1b3/CSyQHyjZO1SX0C4CNGlEKOKFL942IgljqF\n", "export AWS_SESSION_TOKEN=IQoJb3JpZ2luX2VjEPz//////////wEaCXVzLWVhc3QtMSJHMEUCIDSQM0XjZhf3GlfaHptuvOrdl3ifm+5cXQkatayVlVm3AiEAq8wdcI05PilqS8C+onUZSWhTsQFLcL04UwP1tXhD7CUqnwIIRRAAGgw2MTI4NDEzODM1OTQiDLVmMMTxNzCRh2+8pSr8ATGQ1jKHlf3JUCLvCGdUz83Kxw42BgxW2jZslAvIYfZZiNUixq5Nn07rVFtbelHadILxY6gbz6Yqw4MFclIBXCzL8qlktlUmQQI4T03uei7t/cxK5nmwoxal0RyTBRrj2HbhJSOT1umRMjtKY2DEzAAhVnfaX6fFA4+Khq9keKGJ4V6riERVLrlwUzgufSKBBTISkgPC4gfxbWIUfFHiEW23wwB9VhgQQpkVvhEr8I4HT6KDyMaHw6/VCjxIB6SY2oOkCXkCXd1Kw8bwHRpETUYz28UlaJud6JGOC5XETEuCk9Q+2xmhhv5uDkG0PZh1BHxxbt8f2jMksDPNqjDArvyjBjqdAcg6Zwg8HvO4DJUk+TnWGphV2aI8tpXSF6n+qP1gG/yzkzAu3mY42pICqItuxXyBex6wzd5k2Rl0eRogK43yJScnxe+h5EUHCqcHrzjp6b9BCIDDv7nxi//RJAf3iMqGpgSfiFx8XcqTs8n7YnoERNWGdoCgHycRgRG66VP4JWQRxR58qCfnT5JjGoe5z9pJ6tLMNMZ1yZBYTHawYKo=\n" ] }, { "cell_type": "markdown", "id": "4919908e-a8b2-48d2-a031-ed61e29bd82d", "metadata": {}, "source": [ "## Step 0: Check CLI\n", "Check help for the finspace API: create_kx_environment function, if it does not exist you do not have the beta CLI installed correctly." ] }, { "cell_type": "code", "execution_count": null, "id": "7d35ecf7-af9d-4f35-a286-f1e36e621850", "metadata": {}, "outputs": [], "source": [ "#aws finspace update-kx-cluster-databases help\n" ] }, { "cell_type": "markdown", "id": "0917fa9b-eeed-41c8-8d2d-231c5cd9390a", "metadata": {}, "source": [ "## Step 1: Create Environment" ] }, { "cell_type": "code", "execution_count": null, "id": "14ed42a5-3cde-42bd-aa21-ed6151baeecc", "metadata": {}, "outputs": [], "source": [ "aws finspace create-kx-environment \\\n", "--name $ENV_NAME \\\n", "--description \"Managed kdb Insights environment\" \\\n", "--kms-key-id $KMS_KEY_ARN\n" ] }, { "cell_type": "code", "execution_count": null, "id": "7587e59a-b899-4da4-bf62-060fd3c3a37d", "metadata": {}, "outputs": [], "source": [ "#\n", "# Assign value for environmentId to an environment variable\n", "# -----------------------------------------------------------\n", "#\n", "export ENV_ID=\"stceohfhtzkcdw4vyhodsi\"\n" ] }, { "cell_type": "markdown", "id": "9f3783e9-f093-4666-af2d-2d48ec67c515", "metadata": {}, "source": [ "### Confirm Creation\n", "Confirm creation with get-kx-environment, confirm status is **CREATED** before continuing" ] }, { "cell_type": "code", "execution_count": null, "id": "ff9c109e-6477-4586-bf1d-6ca8cbba8303", "metadata": {}, "outputs": [], "source": [ "# Get will show status as CREATE_REQUESTED then CREATING\n", "aws finspace get-kx-environment --environment-id=$ENV_ID\n", "\n", "# Repeat get until status is CREATED\n" ] }, { "cell_type": "markdown", "id": "78371625-eaa7-46c5-94c0-007d5eadb4e6", "metadata": {}, "source": [ "## Step 2: Entitle S3 and KMS Key Use\n", "\n", "S3 buckets are access using the service principal, you grant a specific account and environment access to the bucket. You can use wildcards, such as tof environment, so any Managed kdb Insights environment from an account has access. The S3 bucket is used for staging code and data for the environment.\n", "\n", "### S3 Permission\n", "Example of code and data access to the same S3 bucket.\n", "\n", "```\n", "{\n", " \"Version\": \"2012-10-17\",\n", " \"Statement\": [\n", " {\n", " \"Effect\": \"Allow\",\n", " \"Principal\": {\n", " \"Service\": \"finspace.amazonaws.com\"\n", " },\n", " \"Action\": [\n", " \"s3:GetObject\",\n", " \"s3:GetObjectTagging\",\n", " \"s3:ListBucket\"\n", " ],\n", " \"Resource\": [\n", " \"arn:aws:s3:::S3_BUCKET/*\",\n", " \"arn:aws:s3:::S3_BUCKET\"\n", " ],\n", " \"Condition\": {\n", " \"StringEquals\": {\n", " \"aws:SourceAccount\": \"ACCOUNT_ID\"\n", " },\n", " \"ArnEquals\": {\n", " \"aws:SourceArn\": \"arn:aws:finspace:us-east-1:ACCOUNT_ID:kxEnvironment/ENV_ID/*\"\n", " }\n", " }\n", " }\n", " ]\n", "}\n", "\n", "```\n", "\n", "### KMS Key Use\n", "Be sure the environment has access to use the KMS key given in environment creation.\n", "\n", "```\n", "\"Statement\": [\n", " {\n", " \"Sid\": \"Enable Managed kdb Insights Access\",\n", " \"Effect\": \"Allow\",\n", " \"Principal\": {\n", " \"Service\": \"finspace.amazonaws.com\"\n", " },\n", " \"Action\": [\n", " \"kms:Encrypt\",\n", " \"kms:Decrypt\",\n", " \"kms:GenerateDataKey\"\n", " ],\n", " \"Resource\": \"arn:aws:kms:us-east-1:ACCOUNT_ID:key/KEY_ID\",\n", " \"Condition\": {\n", " \"StringEquals\": {\n", " \"aws:SourceAccount\": \"ACCOUNT_ID\"\n", " },\n", " \"ArnLike\": {\n", " \"aws:SourceArn\": \"arn:aws:finspace:us-east-1:ACCOUNT_ID:kxEnvironment/ENV_ID/*\"\n", " }\n", " }\n", " }\n", " ]\n", "```\n" ] }, { "cell_type": "markdown", "id": "8bb07713-de18-400c-be17-4a0f3b7a0686", "metadata": {}, "source": [ "## Step 3: Create Database\n", "First need to create a database in Managed KX, will then put data into it with create-kx-changeset." ] }, { "cell_type": "code", "execution_count": null, "id": "f21dd403-c692-4869-a0e4-5b7e5710faaa", "metadata": {}, "outputs": [], "source": [ "#aws finspace create-kx-database help\n" ] }, { "cell_type": "code", "execution_count": null, "id": "9479ae8f-1516-4b36-aac3-30132756193b", "metadata": {}, "outputs": [], "source": [ "aws finspace create-kx-database --environment-id=$ENV_ID --database-name=$DB_NAME\n" ] }, { "cell_type": "markdown", "id": "10bdb77a-577c-43bf-98e5-d506185e8e34", "metadata": {}, "source": [ "### Confirm Creation\n", "You can always get information about the database with get-kx-datbase. In future betas the databaseArn will be used to grant access to the database." ] }, { "cell_type": "code", "execution_count": null, "id": "cc4be724-ffbf-47ed-9142-9d1882503a17", "metadata": {}, "outputs": [], "source": [ "aws finspace get-kx-database --environment-id=$ENV_ID --database-name=$DB_NAME\n" ] }, { "cell_type": "markdown", "id": "390b99a6-97ed-482b-b3c3-2fd36cd7b0c9", "metadata": {}, "source": [ "## Step 4: Create Changeset\n", "\n", "**PERMISSIONS** \n", "Be sure the S3 bucket being used has granted the Managed KX environment access to the bucket you will be using. \n", "\n", "Sub-Steps: \n", "a. Extract the provided hdb tarball (hdb.tar.gz) \n", "b. Copy the hdb contents to S3 with s3 sync \n", "c. Ingest data into database with create-kx-changeset. \n" ] }, { "cell_type": "markdown", "id": "b8afabd5-a43c-4fc3-a8c2-2144b556f4fc", "metadata": {}, "source": [ "### a. extract hdb from tarball" ] }, { "cell_type": "code", "execution_count": null, "id": "a5ad40fc-88e5-4e03-9529-5a8ba5961037", "metadata": {}, "outputs": [], "source": [ "tar -zxf hdb.tar.gz\n", "ls -l hdb" ] }, { "cell_type": "markdown", "id": "18279e99-0ad0-4227-a1e3-3bc6e9c0d6c5", "metadata": {}, "source": [ "### b. Copy hdb contents to S3\n", "confirm contents on S3" ] }, { "cell_type": "code", "execution_count": null, "id": "391c115e-2eff-4907-9e0b-9ba36de58506", "metadata": {}, "outputs": [], "source": [ "aws s3 sync ./hdb s3://$S3_BUCKET/data/hdb\n", "echo \"S3 Contents....\"\n", "aws s3 ls s3://$S3_BUCKET/data/hdb/" ] }, { "cell_type": "markdown", "id": "9ed73085-e193-4aaf-b18d-9d7c82852ded", "metadata": {}, "source": [ "### c. Ingest data with create-kx-changeset\n", "\n", "**IF YOU GET AN ERROR** confirm your S3 permissions to the dedicatedServiceAccountId!" ] }, { "cell_type": "code", "execution_count": null, "id": "b46b21e3-38fc-4d63-8d2e-d2720e91c6ef", "metadata": {}, "outputs": [], "source": [ "#aws finspace create-kx-changeset help\n" ] }, { "cell_type": "code", "execution_count": null, "id": "ea2d1603-f071-40bb-8292-140727c87e2f", "metadata": {}, "outputs": [], "source": [ "aws finspace create-kx-changeset \\\n", "--environment-id $ENV_ID \\\n", "--database-name $DB_NAME \\\n", "--change-requests \"[{\\\n", " \\\"changeType\\\": \\\"PUT\\\",\\\n", " \\\"s3Path\\\": \\\"s3://$S3_BUCKET/data/hdb/\\\",\\\n", " \\\"dbPath\\\": \\\"/\\\"\\\n", " }]\"" ] }, { "cell_type": "code", "execution_count": null, "id": "36adf246-b06e-4f06-a917-99d4c075d97b", "metadata": {}, "outputs": [], "source": [ "#\n", "# Assign the value for changesetId to an environment variable\n", "# -----------------------------------------------------------\n", "#\n", "export CHANGESET_ID=\"isRIOfpd2lyrudnX5paQQA\"" ] }, { "cell_type": "markdown", "id": "9b643a97-a111-4ddb-87cb-68c839d39bf0", "metadata": {}, "source": [ "### Confirm Completion of Changeset Creation\n", "check status with get-kx-changeset, monitor for status to be **COMPLETED**\n", "\n", "**Errors?** \n", "- Did you copy the value of changesetId into the environment variable CHANGESET_ID?" ] }, { "cell_type": "code", "execution_count": null, "id": "0e92d07e-d3b8-4a47-816c-50952551aefa", "metadata": {}, "outputs": [], "source": [ "# Get will first show status as IN_PROGRESS\n", "\n", "aws finspace get-kx-changeset \\\n", " --environment-id $ENV_ID \\\n", " --database-name $DB_NAME \\\n", " --changeset-id $CHANGESET_ID\n", " \n", "# Repeat get until status is COMPLETED\n", " " ] }, { "cell_type": "markdown", "id": "b36233cb-ba72-4358-97ba-8fd54bb3a87c", "metadata": { "tags": [] }, "source": [ "## Step 5: Create Cluster for Database\n", "1. Copy code.zip to the s3 bucket path code. \n", "2. create cluster and have it used code.zip\n", "\n", "**IMPORTANT**\n", "- You need to know values for VPC_ID, SECURITY_GROUP, and SUBNET_IDS for your environment. \n", "\n", "**Errors?** \n", "- Did you entitle the bucket with service principal so it can access Code.zip? \n", "- Did you get the right network parameters?\n", "\n", "### An error occurred (ValidationException) when calling the CreateKxCluster operation: Availability zone(s)\n", "This error means the AZs of the subnet(s) for the cluster do not match the subnets of the environment. Check the subnets of the environment from the console and what AZs the subnets used are in." ] }, { "cell_type": "code", "execution_count": null, "id": "175f8677-0197-40c1-8bc1-dbd26057b2b6", "metadata": {}, "outputs": [], "source": [ "# ZIP code\n", "#zip -r -X code.zip code -x '*.ipynb_checkpoints*'\n", "\n", "# Copy provided code.zip to S3 bucket\n", "aws s3 cp ./code.zip s3://$S3_BUCKET/code/\n", "\n", "# confirm its on S3\n", "echo \"S3 Contents....\"\n", "aws s3 ls s3://$S3_BUCKET/code/" ] }, { "cell_type": "markdown", "id": "9eb1d91a-0a9c-4668-91a2-72d872f38327", "metadata": {}, "source": [ "### Network Settings\n", "Confirm that your subnets are in the same AZs as your Managed KX environment." ] }, { "cell_type": "code", "execution_count": null, "id": "43428071-9b5a-48e6-9b7b-9ae8daa8ae81", "metadata": {}, "outputs": [], "source": [ "export VPC_ID=\"vpc-0e702dec545865b11\"\n", "export SECURITY_GROUP=\"sg-018111774e795682d\"\n", "\n", "export SUBNET_IDS=\"\\\"subnet-0f97cae6600859c17\\\"\" # use1-az4\n", "export AZ_ID=\"use1-az4\"\n", "\n", "export VPC_CONFIG=\"{\\\"vpcId\\\":\\\"$VPC_ID\\\",\\\"securityGroupIds\\\":[\\\"$SECURITY_GROUP\\\"],\\\"subnetIds\\\":[$SUBNET_IDS],\\\"ipAddressType\\\":\\\"IP_V4\\\"}\"\n", "export CAPACITY_CONFIG=\"{\\\"nodeType\\\": \\\"kx.s.xlarge\\\", \\\"nodeCount\\\": 2 }\"\n", "export DATABASES_CONFIG=\"[{\\\"databaseName\\\": \\\"$DB_NAME\\\", \\\"cacheConfigurations\\\": [{\\\"dbPaths\\\":[\"\\\"/\\\"\"], \\\"cacheType\\\":\\\"CACHE_1000\\\"}]}]\"\n", "export CACHE_CONFIG=\"[{\\\"type\\\": \\\"CACHE_1000\\\", \\\"size\\\":1200}]\"\n", "export CMD_LINE=\"[{\\\"key\\\": \\\"s\\\", \\\"value\\\": \\\"4\\\"}, \\\n", " {\\\"key\\\":\\\"dbname\\\", \\\"value\\\":\\\"$DB_NAME\\\"}, \\\n", " {\\\"key\\\": \\\"codebase\\\", \\\"value\\\": \\\"code\\\"}]\" \n" ] }, { "cell_type": "code", "execution_count": null, "id": "08fd4001-1e76-4c21-b40d-7023a6d39dbe", "metadata": {}, "outputs": [], "source": [ "echo $VPC_CONFIG\n", "echo $CAPACITY_CONFIG\n", "echo $DATABASES_CONFIG\n", "echo $CACHE_CONFIG\n", "echo $S3_BUCKET\n", "echo $AZ_ID\n", "echo $CMD_LINE" ] }, { "cell_type": "code", "execution_count": null, "id": "c1c01e53-d78b-4ed2-a4a8-d855874833aa", "metadata": {}, "outputs": [], "source": [ "aws finspace create-kx-cluster \\\n", " --environment-id $ENV_ID \\\n", " --cluster-name $CLUSTER_NAME \\\n", " --cluster-type HDB \\\n", " --release-label \"1.0\" \\\n", " --capacity-configuration \"$CAPACITY_CONFIG\" \\\n", " --cache-storage-configurations \"$CACHE_CONFIG\" \\\n", " --code s3Bucket=$S3_BUCKET,s3Key=code/code.zip \\\n", " --vpc-configuration \"$VPC_CONFIG\" \\\n", " --az-mode SINGLE \\\n", " --availability-zone-id $AZ_ID \\\n", " --databases \"$DATABASES_CONFIG\" \\\n", " --initialization-script=\"code/init.q\" \\\n", " --command-line-arguments \"$CMD_LINE\"\n" ] }, { "cell_type": "markdown", "id": "369c2b82-f065-47ff-b333-8e063073b640", "metadata": {}, "source": [ "### Confirm Creation of Cluster\n", "When cluster creation is complete, status will read as **CREATED**. \n", "For connecting to the cluster from q, use the connectionString in the returned json of get-kx-cluster. " ] }, { "cell_type": "code", "execution_count": null, "id": "922398a4-6550-4385-94eb-288e5779085e", "metadata": {}, "outputs": [], "source": [ "# Get will show status as PENDING or CREATING\n", "\n", "aws finspace get-kx-cluster --environment-id=$ENV_ID --cluster-name=$CLUSTER_NAME \n", " \n", "# Repeat get until status is RUNNING" ] }, { "cell_type": "markdown", "id": "19883ffa-d793-4ece-9788-7772acf4874d", "metadata": {}, "source": [ "# Step 6: Roles, Policies, Permissions\n", "\n", "You will need to create a role with the correct policy. In this case the role-name is **kdb-cluster-user** with policy **kdb-connect-cluster** . \n", "Use the role's ARN when creating a kx-user\n", "\n", "## Example\n", "**Role Name:** kdb-cluster-user \n", "**ARN:** arn:aws:iam::ACCOUNT_ID:role/kdb-cluster-user \n", "\n", "### Policy\n", "**Name:** kdb-connect-cluster \n", "```\n", "{\n", " \"Version\": \"2012-10-17\",\n", " \"Statement\": [\n", " {\n", " \"Effect\": \"Allow\",\n", " \"Action\": \"finspace:ConnectKxCluster\",\n", " \"Resource\": \"arn:aws:finspace:us-east-1:ACCOUNT_ID:kxEnvironment/ENV_ID/*\"\n", " },\n", " {\n", " \"Effect\": \"Allow\",\n", " \"Action\": \"finspace:MountKxDatabase\",\n", " \"Resource\": \"arn:aws:finspace:us-east-1:ACCOUNT_ID:kxEnvironment/ENV_ID/*\"\n", " },\n", " {\n", " \"Effect\": \"Allow\",\n", " \"Action\": \"finspace:GetKxConnectionString\",\n", " \"Resource\": \"arn:aws:finspace:us-east-1:ACCOUNT_ID:kxEnvironment/ENV_ID/*\"\n", " },\n", " {\n", " \"Effect\": \"Allow\",\n", " \"Action\": \"finspace:GetKxUser\",\n", " \"Resource\": \"arn:aws:finspace:us-east-1:ACCOUNT_ID:kxEnvironment/ENV_ID/*\"\n", " }\n", " \n", " ]\n", "}\n", "```\n", "\n", "### Trust Policy\n", "```\n", "{\n", " \"Version\": \"2012-10-17\",\n", " \"Statement\": [\n", " {\n", " \"Sid\": \"Statement1\",\n", " \"Effect\": \"Allow\",\n", " \"Principal\": {\n", " \"Service\": \"prod.finspacekx.aws.internal\",\n", " \"AWS\": \"arn:aws:iam::ACCOUNT_ID:root\"\n", " },\n", " \"Action\": \"sts:AssumeRole\"\n", " }\n", " ]\n", "}\n", "```" ] }, { "cell_type": "markdown", "id": "075b18e9-b7a2-4e8e-9849-8899fcd72d32", "metadata": {}, "source": [ "## Create the KX User" ] }, { "cell_type": "code", "execution_count": null, "id": "ea64b096-cc69-46b4-83de-e76dca60451c", "metadata": {}, "outputs": [], "source": [ "# confirm IAM role exists\n", "export IAM_ROLE=$(aws iam get-role --role-name $ROLE_NAME | jq -r '.Role''.Arn')\n", "echo $IAM_ROLE" ] }, { "cell_type": "code", "execution_count": null, "id": "1b8f6547-4094-4499-84df-363d94ff8249", "metadata": {}, "outputs": [], "source": [ "# if the user exists, delete first, then create\n", "if aws finspace get-kx-user --environment-id $ENV_ID --user-name $KDB_USERNAME\n", "then\n", " echo \"exists, will delete\"\n", " # if so delete\n", " aws finspace delete-kx-user --environment-id $ENV_ID --user-name $KDB_USERNAME\n", "fi\n", "\n", "aws finspace create-kx-user --environment-id $ENV_ID --iam-role $IAM_ROLE --user-name $KDB_USERNAME\n", "echo \"created\"" ] }, { "cell_type": "markdown", "id": "75f09254-ece5-4444-8ecc-f3333f9845d7", "metadata": {}, "source": [ "## Use KX User ARN to get Connection String" ] }, { "cell_type": "code", "execution_count": null, "id": "f1a672ed-f1d4-4593-a7da-7204fb2b0c1b", "metadata": {}, "outputs": [], "source": [ "#aws finspace get-kx-user --environment-id $ENV_ID --user-name $KDB_USERNAME\n", "\n", "# userArn for the kx-user\n", "export USER_ARN=$(aws finspace get-kx-user --environment-id $ENV_ID --user-name $KDB_USERNAME | jq -r .userArn)\n", "echo $USER_ARN" ] }, { "cell_type": "code", "execution_count": null, "id": "9781e649-dd24-493d-aeda-c621476988c1", "metadata": {}, "outputs": [], "source": [ "CREDS=$(aws sts assume-role --role-arn $IAM_ROLE --role-session-name \"Connect-to-kdb-cluster\" | jq -r .Credentials)\n", "echo $CREDS" ] }, { "cell_type": "code", "execution_count": null, "id": "81e04197-16a1-47b7-a938-53a40ca19390", "metadata": {}, "outputs": [], "source": [ "# original identity\n", "aws sts get-caller-identity" ] }, { "cell_type": "code", "execution_count": null, "id": "7682c9bd-7fb6-443a-99ed-9ceb5d996838", "metadata": {}, "outputs": [], "source": [ "# set to assumed credentials\n", "\n", "# SAVE existing creds\n", "export PREV_AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID\n", "export PREV_AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY\n", "export PREV_AWS_SESSION_TOKEN=$AWS_SESSION_TOKEN\n", "\n", "export AWS_ACCESS_KEY_ID=$(echo $CREDS | jq -r '.AccessKeyId')\n", "export AWS_SECRET_ACCESS_KEY=$(echo $CREDS | jq -r '.SecretAccessKey')\n", "export AWS_SESSION_TOKEN=$(echo $CREDS | jq -r '.SessionToken')" ] }, { "cell_type": "code", "execution_count": null, "id": "3f2a97c0-9836-44b7-8b2f-6be0e61cf211", "metadata": {}, "outputs": [], "source": [ "aws sts get-caller-identity" ] }, { "cell_type": "code", "execution_count": null, "id": "e95b0f14-cdf1-4191-85e4-5242d2ba3a77", "metadata": {}, "outputs": [], "source": [ "echo $CLUSTER_NAME" ] }, { "cell_type": "code", "execution_count": null, "id": "42715a9c-403a-4754-b6da-69a45fd12078", "metadata": {}, "outputs": [], "source": [ "aws finspace get-kx-connection-string --environment-id $ENV_ID --user-arn $USER_ARN --cluster-name $CLUSTER_NAME" ] }, { "cell_type": "code", "execution_count": null, "id": "be403688-6d37-4a0d-8fb6-1722b7001f67", "metadata": {}, "outputs": [], "source": [ "# return to original keys\n", "export AWS_ACCESS_KEY_ID=$PREV_AWS_ACCESS_KEY_ID\n", "export AWS_SECRET_ACCESS_KEY=$PREV_AWS_SECRET_ACCESS_KEY\n", "export AWS_SESSION_TOKEN=$PREV_AWS_SESSION_TOKEN\n" ] }, { "cell_type": "markdown", "id": "60b3b2ad-665a-42fb-8aa2-b56c662e8337", "metadata": { "tags": [] }, "source": [ "# Step 7: Use Q\n", "From a q process with network reachability to the cluster, connect to the cluster with the value in connectionString above.\n", "\n", "```\n", "q) kx_conn:\"CONNECTION_STRING\"\n", "q) c:hopen(`$kx_conn)\n", "q) c \"tables[]\"\n", ",`example\n", "q) c(meta; `example)\n", "c | t f a\n", "------| -----\n", "date | d\n", "sym | s p\n", "time | p\n", "number| j\n", "q) c \"select counts:count i by date from example\"\n", "date | counts\n", "----------| -------\n", "2023.01.29| 1000000\n", "2023.01.30| 1000000\n", "2023.01.31| 1000000\n", "2023.02.01| 1000000\n", "2023.02.02| 1000000\n", "2023.02.03| 1000000\n", "2023.02.04| 1000000\n", "2023.02.05| 1000000\n", "2023.02.06| 1000000\n", "2023.02.07| 1000000\n", "q) c \"\\\"Rows: \\\", {reverse\\\",\\\"sv 0N 3#reverse string x}count example\"\n", "\"Rows: 10,000,000\"\n", "\n", "```" ] }, { "cell_type": "markdown", "id": "b695d4f0-c6bb-4521-b6f5-5931ecc87f7d", "metadata": {}, "source": [ "# Step 8: Clean Up\n", "Execute service calls in order: \n", "a. Delete the created Cluster \n", "b. Delete the created Database \n", "c. Delete the created environment " ] }, { "cell_type": "markdown", "id": "764f20ec-4b99-4ff0-8fe8-6643a96573ff", "metadata": {}, "source": [ "## Step a: Delete Cluster" ] }, { "cell_type": "code", "execution_count": null, "id": "5800b11c-f7da-4c19-af0d-9fab1e0095c4", "metadata": {}, "outputs": [], "source": [ "aws finspace delete-kx-cluster --environment-id=$ENV_ID --cluster-name=$CLUSTER_NAME" ] }, { "cell_type": "markdown", "id": "27168980-a999-4bee-8072-dd6bda214c29", "metadata": {}, "source": [ "### Confirm Deletion of Cluster\n", "When cluster is deleted call will return 'Unable to find cluster'" ] }, { "cell_type": "code", "execution_count": null, "id": "ef1c1afe-a81f-4eee-ac72-02aae3b296b1", "metadata": { "tags": [] }, "outputs": [], "source": [ "# Get will show status as DELETING\n", "aws finspace get-kx-cluster --environment-id=$ENV_ID --cluster-name=$CLUSTER_NAME\n", "\n", "# eventually will be unable to find cluster once deleted" ] }, { "cell_type": "markdown", "id": "bb6f848e-d92c-4e07-965e-d4a691aa2557", "metadata": {}, "source": [ "## Step b: Delete Database" ] }, { "cell_type": "code", "execution_count": null, "id": "ccf09ed5-08f9-49ae-9d91-919ea1a5bde5", "metadata": {}, "outputs": [], "source": [ "aws finspace delete-kx-database --environment-id=$ENV_ID --database-name $DB_NAME" ] }, { "cell_type": "markdown", "id": "78152096-5c38-494f-b599-030219ae65f0", "metadata": {}, "source": [ "### Confirm Deletion of Database\n", "When deleted the resource (e.g. database) will not be found.\n", "\n", "```\n", "An error occurred (ResourceNotFoundException) when calling the GetKxDatabase operation: KX database welcomedb not found\n", "```" ] }, { "cell_type": "code", "execution_count": null, "id": "475207ca-1783-4b1c-a421-115f0fd2cad5", "metadata": {}, "outputs": [], "source": [ "aws finspace get-kx-database --environment-id=$ENV_ID --database-name $DB_NAME" ] }, { "cell_type": "markdown", "id": "9446e5df-c2db-478d-9c88-6282a85e644c", "metadata": {}, "source": [ "## Step c: Delete Environment\n", "\n", "### Common Errors\n", "\n", "**Cluster Still Deleting**\n", "```\n", "An error occurred (ValidationException) when calling the DeleteKxEnvironment operation: Can't delete this KX Environment with id clb2oiucxtzamq65dqlasm. It still contains active clusters. Please delete all clusters before deleting the environment.\n", "\n", "```" ] }, { "cell_type": "code", "execution_count": null, "id": "3b00c8d9-d2a9-41c9-adeb-f65e3fa1a067", "metadata": {}, "outputs": [], "source": [ "aws finspace delete-kx-environment --environment-id=$ENV_ID" ] }, { "cell_type": "markdown", "id": "fdb39887-3c38-45c7-93e8-c0ff351f5736", "metadata": {}, "source": [ "### Confirm Deletion of Environment\n", "Monitor status of environment until status is **DELETED**" ] }, { "cell_type": "code", "execution_count": null, "id": "3f47e518-f09b-4f24-8263-eeee7da2eec5", "metadata": {}, "outputs": [], "source": [ "aws finspace get-kx-environment --environment-id=$ENV_ID" ] }, { "cell_type": "code", "execution_count": null, "id": "b70d0963-93af-44db-b5a2-bdbd92c0b382", "metadata": {}, "outputs": [], "source": [ "date" ] }, { "cell_type": "code", "execution_count": null, "id": "ee057b81-3bce-4807-a1ca-71022ef46418", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Bash", "language": "bash", "name": "bash" }, "language_info": { "codemirror_mode": "shell", "file_extension": ".sh", "mimetype": "text/x-sh", "name": "bash" } }, "nbformat": 4, "nbformat_minor": 5 }