# Index Configuration Tool

Python package that automates the creation of indices on a target cluster based on the contents of a source cluster. Index settings and index mappings are correctly copied over.

This tool seeks to automate [the steps outlined here](https://github.com/kartg/opensearch-migrations/tree/datastash/datastash#2-configure-index-templates-on-the-target-cluster) and eliminate the need for defining index templates on the target cluster when migrating data from one cluster to another. The tool currently supports ElasticSearch or OpenSearch as source and target.

If an identical index or an index with the same name but differing settings/mappings is already present on the target cluster, that index is skipped and left as-is on the target cluster. The tool completes by printing a report of all processed indices under 3 buckets:
* Successfully created on the target cluster
* Skipped due to a conflict in settings/mappings
* Skipped since the index configuration is identical on source and target

The input to the tool is a Logstash configuration file that is then parsed to obtain source and target endpoints.

## Current Limitations

* Only supports ElasticSearch and OpenSearch endpoints for source and target
* Only supports basic auth
* Type mappings for legacy indices are not handled
* Index templates and index aliases are not copied
* Index health is not validated after creation

## Usage

### Command-Line

#### Setup:

* [Clone](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository) this GitHub repo
* Install [Python](https://www.python.org/)
* Ensure that [pip](https://pip.pypa.io/en/stable/installation/#) is installed
* (Optional) Set up and activate a [virtual environment](https://packaging.python.org/en/latest/tutorials/installing-packages/#creating-and-using-virtual-environments)

Navigate to the cloned GitHub repo. Then, install the required Python dependencies by running:

```shell
python -m pip install -r index_configuration_tool/requirements.txt
```

#### Execution:

After [setup](#setup), the tool can be executed using:

```shell
python index_configuration_tool/main.py <path-to-Logstash-file>
```

Usage information can also be printed by supplying the `-h` flag.

### Docker

Replace `<path-to-Logstash-config-file>` in the command below with the path to your Logstash config file.

```shell
confPath=<path-to-Logstash-config-file>; docker run -v $confPath:/tmp/conf.json -t kartg/index-configuration-tool /tmp/conf.json
```

## Development

The source code for the tool is located under the `index_configuration_tool/` directory. Please refer to the [Setup](#setup) section to ensure that the necessary dependencies are installed prior to development.

Additionally, you'll also need to install development dependencies by running:

```shell
python -m pip install -r index_configuration_tool/dev-requirements.txt
```

### Unit Tests

Unit tests are located in a sub-directory named `tests`. Unit tests can be run using:

```shell
python -m unittest
```

### Coverage

Code coverage metrics can be generated by first running unit tests using _coverage run_:

```shell
python -m coverage run -m unittest
```

Then a report can either be printed on the command line or generated as HTML.
Note that the `--omit` parameter must be specified to avoid tracking code coverage on unit test code:

```shell
python -m coverage report --omit "*/tests/*"
python -m coverage html --omit "*/tests/*"
```

### Lark

The code uses [Lark](https://github.com/lark-parser/lark) for grammar definition, tree parsing and transformation.
The Logstash parser grammar is adapted from [node-logstash](https://github.com/bpaquet/node-logstash/blob/master/lib/logstash_config.jison).