{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "[![AWS SDK for pandas](_static/logo.png \"AWS SDK for pandas\")](https://github.com/aws/aws-sdk-pandas)\n", "\n", "# 21 - Global Configurations\n", "\n", "[awswrangler](https://github.com/aws/aws-sdk-pandas) has two ways to set global configurations that will override the regular default arguments configured in functions signatures.\n", "\n", "- **Environment variables**\n", "- **wr.config**\n", "\n", "*P.S. Check the [function API doc](https://aws-sdk-pandas.readthedocs.io/en/3.2.1/api.html) to see if your function has some argument that can be configured through Global configurations.*\n", "\n", "*P.P.S. One exception to the above mentioned rules is the `botocore_config` property. It cannot be set through environment variables\n", "but only via `wr.config`. It will be used as the `botocore.config.Config` for all underlying `boto3` calls.\n", "The default config is `botocore.config.Config(retries={\"max_attempts\": 5}, connect_timeout=10, max_pool_connections=10)`.\n", "If you only want to change the retry behavior, you can use the environment variables `AWS_MAX_ATTEMPTS` and `AWS_RETRY_MODE`.\n", "(see [Boto3 documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html#using-environment-variables))*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Environment Variables" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "env: WR_DATABASE=default\n", "env: WR_CTAS_APPROACH=False\n", "env: WR_MAX_CACHE_SECONDS=900\n", "env: WR_MAX_CACHE_QUERY_INSPECTIONS=500\n", "env: WR_MAX_REMOTE_CACHE_ENTRIES=50\n", "env: WR_MAX_LOCAL_CACHE_ENTRIES=100\n" ] } ], "source": [ "%env WR_DATABASE=default\n", "%env WR_CTAS_APPROACH=False\n", "%env WR_MAX_CACHE_SECONDS=900\n", "%env WR_MAX_CACHE_QUERY_INSPECTIONS=500\n", "%env WR_MAX_REMOTE_CACHE_ENTRIES=50\n", "%env WR_MAX_LOCAL_CACHE_ENTRIES=100" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import awswrangler as wr\n", "import botocore" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
foo
01
\n", "
" ], "text/plain": [ " foo\n", "0 1" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "wr.athena.read_sql_query(\"SELECT 1 AS FOO\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Resetting" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "# Specific\n", "wr.config.reset(\"database\")\n", "# All\n", "wr.config.reset()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## wr.config" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "wr.config.database = \"default\"\n", "wr.config.ctas_approach = False\n", "wr.config.max_cache_seconds = 900\n", "wr.config.max_cache_query_inspections = 500\n", "wr.config.max_remote_cache_entries = 50\n", "wr.config.max_local_cache_entries = 100\n", "# Set botocore.config.Config that will be used for all boto3 calls\n", "wr.config.botocore_config = botocore.config.Config(\n", " retries={\"max_attempts\": 10},\n", " connect_timeout=20,\n", " max_pool_connections=20\n", ")" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
foo
01
\n", "
" ], "text/plain": [ " foo\n", "0 1" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "wr.athena.read_sql_query(\"SELECT 1 AS FOO\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Visualizing" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
nameEnv. Variabletypenullableenforcedconfiguredvalue
0catalog_idWR_CATALOG_ID<class 'str'>TrueFalseFalseNone
1concurrent_partitioningWR_CONCURRENT_PARTITIONING<class 'bool'>FalseFalseFalseNone
2ctas_approachWR_CTAS_APPROACH<class 'bool'>FalseFalseTrueFalse
3databaseWR_DATABASE<class 'str'>TrueFalseTruedefault
4max_cache_query_inspectionsWR_MAX_CACHE_QUERY_INSPECTIONS<class 'int'>FalseFalseTrue500
5max_cache_secondsWR_MAX_CACHE_SECONDS<class 'int'>FalseFalseTrue900
6max_remote_cache_entriesWR_MAX_REMOTE_CACHE_ENTRIES<class 'int'>FalseFalseTrue50
7max_local_cache_entriesWR_MAX_LOCAL_CACHE_ENTRIES<class 'int'>FalseFalseTrue100
8s3_block_sizeWR_S3_BLOCK_SIZE<class 'int'>FalseTrueFalseNone
9workgroupWR_WORKGROUP<class 'str'>FalseTrueFalseNone
10chunksizeWR_CHUNKSIZE<class 'int'>FalseTrueFalseNone
11s3_endpoint_urlWR_S3_ENDPOINT_URL<class 'str'>TrueTrueTrueNone
12athena_endpoint_urlWR_ATHENA_ENDPOINT_URL<class 'str'>TrueTrueTrueNone
13sts_endpoint_urlWR_STS_ENDPOINT_URL<class 'str'>TrueTrueTrueNone
14glue_endpoint_urlWR_GLUE_ENDPOINT_URL<class 'str'>TrueTrueTrueNone
15redshift_endpoint_urlWR_REDSHIFT_ENDPOINT_URL<class 'str'>TrueTrueTrueNone
16kms_endpoint_urlWR_KMS_ENDPOINT_URL<class 'str'>TrueTrueTrueNone
17emr_endpoint_urlWR_EMR_ENDPOINT_URL<class 'str'>TrueTrueTrueNone
18lakeformation_endpoint_urlWR_LAKEFORMATION_ENDPOINT_URL<class 'str'>TrueTrueTrueNone
19dynamodb_endpoint_urlWR_DYNAMODB_ENDPOINT_URL<class 'str'>TrueTrueTrueNone
20secretsmanager_endpoint_urlWR_SECRETSMANAGER_ENDPOINT_URL<class 'str'>TrueTrueTrueNone
21timestream_endpoint_urlWR_TIMESTREAM_ENDPOINT_URL<class 'str'>TrueTrueTrueNone
22botocore_configWR_BOTOCORE_CONFIG<class 'botocore.config.Config'>TrueFalseTrue<botocore.config.Config object at 0x14f313e50>
23verifyWR_VERIFY<class 'str'>TrueFalseTrueNone
24addressWR_ADDRESS<class 'str'>TrueFalseFalseNone
25redis_passwordWR_REDIS_PASSWORD<class 'str'>TrueFalseFalseNone
26ignore_reinit_errorWR_IGNORE_REINIT_ERROR<class 'bool'>TrueFalseFalseNone
27include_dashboardWR_INCLUDE_DASHBOARD<class 'bool'>TrueFalseFalseNone
28log_to_driverWR_LOG_TO_DRIVER<class 'bool'>TrueFalseFalseNone
29object_store_memoryWR_OBJECT_STORE_MEMORY<class 'int'>TrueFalseFalseNone
30cpu_countWR_CPU_COUNT<class 'int'>TrueFalseFalseNone
31gpu_countWR_GPU_COUNT<class 'int'>TrueFalseFalseNone
" ], "text/plain": [ "" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "wr.config" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.13" }, "vscode": { "interpreter": { "hash": "bd595004b250e5f4145a0d632609b0d8f97d1ccd278d58fafd6840c0467021f9" } } }, "nbformat": 4, "nbformat_minor": 4 }