- [Testing](#testing) - [Code Coverage Collecting](#code-coverage-collecting) - [OpenSearch plugin](#opensearch-plugin) - [Gradle project](#gradle-project) - [Maven project](#maven-project) - [Limitation](#limitation) - [OpenSearch Dashboards plugin](#opensearch-dashboards-plugin) - [Code Coverage Reporting](#code-coverage-reporting) - [Code coverage report upload through CI workflow](#code-coverage-report-upload-through-ci-workflow) - [A status badge in README file to show the code coverage](#a-status-badge-in-readme-file-to-show-the-code-coverage) - [Codecov integration in Pull Requests](#codecov-integration-in-pull-requests) - [Backwards Compatibility Testing](#backwards-compatibility-testing) - [Types of BWC tests](#types-of-bwc-tests) - [BWC tests on distribution bundle level](#bwc-tests-on-distribution-bundle-level) - [Hooking the BWC tests to CI](#hooking-the-bwc-tests-to-ci) - [BWC Version test matrix](#bwc-version-test-matrix) - [Gradle Plugins](#gradle-plugins) - [Distribution Download Plugin](#distribution-download-plugin) ## Testing ### Code Coverage Collecting Code coverage is a measurement of how many lines of code are executed under the test suites. #### OpenSearch plugin It is recommended to use [JaCoCo](https://www.eclemma.org/jacoco/) library to generate the code coverage report. The following guide may help you use JaCoCo in OpenSearch plugins with ease. ##### Gradle project 1. Add [JaCoCo Gradle plugin](https://docs.gradle.org/current/userguide/jacoco_plugin.html) to `build.gradle` in your repository. [example](https://github.com/opensearch-project/sql/blob/1.0/build.gradle#L48) 2. Set Jacoco to export coverage report in `xml` format. [example](https://github.com/opensearch-project/sql/blob/1.0/build.gradle#L101-L105) 3. Call gradle task "jacocoTestReport" to generate report (`./gradlew jacocoTestReport`) after running your tests. (Or add a `dependsOn` to make "jacocoTestReport" task depending on your test tasks. [example](https://github.com/opensearch-project/index-management/blob/1.0/build-tools/coverage.gradle)) ##### Maven project Since OpenSearch uses Gradle as the build tool, we don't recommend using Maven. Code coverage for a Maven project can be generated by [JaCoCo Maven plugin](https://www.eclemma.org/jacoco/trunk/doc/maven.html), and here is an [example](https://github.com/opensearch-project/security/blob/1.0/pom.xml#L636). ##### Limitation 1. JaCoCo Gradle plugin can not collect code coverage properly for integration tests (tests that extend `OpenSearchIntegTestCase` or `OpenSearchRestTestCase`, see [here](https://github.com/opensearch-project/OpenSearch/blob/1.0/TESTING.md#base-classes-for-test-cases) to learn more). However, there is a workaround to collect coverage when the integration test is running on a local single node cluster. For your reference, [this file](https://github.com/opensearch-project/index-management/blob/1.0/build-tools/coverage.gradle) is a custom Gradle plugin to help achieve, and it can be applied conditionally like [here](https://github.com/opensearch-project/index-management/blob/1.0/build.gradle#L90-L93). 2. Additional work is needed to get coverage report by JaCoCo for the test classes which use [PowerMock](https://github.com/powermock/powermock), because those classes must be instrumented before the test executes. Please see the [PowerMock wiki](https://github.com/powermock/powermock/wiki/Code-coverage-with-JaCoCo) for detail, and there is an example for using JaCoCo Maven plugin. But for JaCoCo Gradle plugin, there is no native support, see the [issue](https://github.com/gradle/gradle/issues/2429) in `gradle` repository for detail. 3. JaCoCo Gradle plugin can not merge code coverage reports from different sub-projects out-of-box, when the code repository contains a [Gradle multi-project build](https://docs.gradle.org/6.6.1/userguide/multi_project_builds.html). If you need the aggrectiated code coverage report from multiple sub-projects, please check the [example](https://github.com/opensearch-project/alerting/blob/1.0/build-tools/merged-coverage.gradle) of a custom Gradle plugin in OpenSearch Alerting plugin, or the [sample](https://docs.gradle.org/6.6.1/samples/sample_jvm_multi_project_with_code_coverage.html) from Gradle. #### OpenSearch Dashboards plugin OpenSearch Dashboards mainly uses [Jest](https://jestjs.io/) for unit testing, and plugins are recommended to utilize Jest as the unit testing framework. To generate code coverage report under Jest framework, adding the flag `--coverage` in the command to run unit tests. See [here](https://github.com/opensearch-project/security-dashboards-plugin/blob/1.0/.github/workflows/unit-test.yml#L66) for an example in the CI workflow. ### Code Coverage Reporting OpenSearch/Dashboards plugins are using [Codecov](https://about.codecov.io/) for code coverage reporting and analysis. The dashboard can be seen [here](https://app.codecov.io/gh/opensearch-project/). All the OpenSearch plugins are required to have the following items: #### Code coverage report upload through CI workflow 1. Generate code coverage report in `xml` format in a CI workflow of Github Actions 2. Upload the report to Codecov by adding a step of [codecov-action](https://github.com/codecov/codecov-action) in the workflow. 3. For private repositories, an upload token is needed, visit the repository page of Codecov to get your token: https://codecov.io/gh/opensearch-project/REPO-NAME-HERE. Please follow [Github docs](https://docs.github.com/en/actions/reference/encrypted-secrets#creating-encrypted-secrets-for-a-repository) to use Github Secrets to store the token and not expose to the public. 4. Set the workflow to be triggered by `push` and `pull request` on `main` and release branches. 5. To see your code coverage results from Codecov, visiting https://codecov.io/gh/opensearch-project/REPO-NAME-HERE after the CI workflow is triggered. See [CI Workflows](STANDARDS.md#ci-workflows) for example. #### A status badge in README file to show the code coverage See [here](https://github.com/opensearch-project/index-management/blob/main/README.md) for an example of the status badge. Add the following line at the header of the README markdown file of the repository: ``` [](https://codecov.io/gh/opensearch-project/REPO-NAME-HERE) ``` Link of the badge can also be accessed from the repository's settings page of Codecov: https://app.codecov.io/gh/opensearch-project/REPO-NAME-HERE/settings/badge #### Codecov integration in Pull Requests 1. Add a `codecov.yml` file into the repository ([example](https://github.com/opensearch-project/k-NN/commit/f7d1985230ce851cb97a7e41d8bce32127a4f33b)) 2. Set the target code coverage properly in case it blocks your PR in Github checks. See Codecov docs to learn more about [codecov yaml file](https://docs.codecov.com/docs/codecov-yaml) and [common configurations](https://docs.codecov.com/docs/common-recipe-list). ### Backwards Compatibility Testing The purpose of backwards compatibiity tests in plugins is to ensure that the plugin is safely upgraded when an OpenSearch node undergoes an upgrade. Backwards compatibility tests can be implemented in plugins by extending the OpenSearch bwc framework. See [OpenSearch#1051](https://github.com/opensearch-project/OpenSearch/pull/1051) for more information on the extended bwc framework. #### Types of BWC tests 1. Mixed cluster test: This tests a cluster running with three nodes of a bwc OpenSearch and plugin version and upgrades a single node to the current version of OpenSearch and the plugin while the other two nodes are still on the previous version. This creates a mixed cluster. The test checks that the correct version of the plugin is installed in the upgraded node and the plugin functionalities work as expected after the upgrade. Command: `./gradlew adBwcCluster#mixedClusterTask -Dtests.security.manager=false` 2. Rolling upgrade tests: This tests a cluster running with three nodes of a bwc OpenSearch and plugin version and performs a node by node upgrade to the current version of OpenSearch and the plugin. The test checks that the correct version of the plugin is installed in each upgraded node and the plugin functionalities work as expected after each node upgrade. Commmand: `./gradlew adBwcCluster#rollingUpgradeClusterTask -Dtests.security.manager=false` 3. Full restart upgrade tests: This tests a cluster running with three nodes of a bwc OpenSearch and plugin version and performs a full cluster upgrade for all the nodes to the current version of OpenSearch and the plugin. The test checks that the correct version of the plugin is installed in all upgraded nodes and the plugin functionalities work as expected after the upgrade. Command: `./gradlew adBwcCluster#fullRestartClusterTask -Dtests.security.manager=false` 4. Command: `./gradlew bwcTestSuite -Dtests.security.manager=false` is used to run all the above bwc tests together. The logs for the test runs can be found at `build/testclusters/{clustername}-{nodeid}/logs`. As mentioned above, the bwc tests are using the extended bwc framework from OpenSearch - the test cluster is contained within the `testclusters` module which is available as a plugin `opensearch.testclusters`. Each cluster is an instance of `OpenSearchCluster` containing a given number of `OpenSearchNode` nodes. The `OpenSearchNode` allows to install plugins to the node, start, stop, upgrade (along with plugin upgrade) and restart the node. The plugin bwc tests use these functionalities to spin up a test cluster with provided bwc versions and upgrade the nodes and plugins installed on the nodes to the current version using `upgradeNodeAndPluginToNextVersion` for a single node at a time and `upgradeAllNodesAndPluginsToNextVersion` for a full restart upgrade. See [anomaly-detection#158](https://github.com/opensearch-project/anomaly-detection/pull/158) and [anomaly-detection#185](https://github.com/opensearch-project/anomaly-detection/pull/185) for more information. Each plugin can test various functionalities in their bwc tests and add more scenarios if needed. #### BWC tests on distribution bundle level The BWC tests running on distribution level are using the same framework from OpenSearch. The test cluster is spin up with the `latest` distribution bundle of provided version exclusively when the project is initialized with property `-PcustomDistributionDownloadType=bundle`. The distribution bundle will be obtained with this format URL: `https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.2/latest/linux/x64/tar/dist/opensearch/opensearch-1.3.2-linux-x64.tar.gz` This URL pattern starts to be supported as far as `v1.3.2` so The distribution bundle set up in the test cluster can be all versions after version `v1.3.2`. On CI level for plugins, security certificates need to be manually imported when spinning up the test cluster as security plugin is included in the distribution bundle. When upgrading the version within the test cluster, `nextNodeToNextVersion` is used for a single node upgrade and `goToNextVersion` is for a full restart upgrade. See [anomaly-detection#766](https://github.com/opensearch-project/anomaly-detection/pull/766) or [observability#1366](https://github.com/opensearch-project/observability/pull/1366) for more information. #### Hooking the BWC tests to CI The bwc tests can be hooked to the CI workflow in the plugin to run it with each pull request and push. See [anomaly-detection#181](https://github.com/opensearch-project/anomaly-detection/pull/181) for more information. #### BWC Version test matrix The bwc tests should follow the OpenSearch's [support model](https://opensearch.org/faq#q3.28), in this way the last major version's most recent minor version should be tested. Additionally all minor versions of the current major version should also be tested. At the time of this writing there are OpenSearch releases for 1.0, 1.1, 1.2, 1.3, and 2.0. To ensure bwc tests have appropriate coverage when building components for 2.1, the bwc test matrix should include 1.3, and 2.0. When minor build version number bumps are made the bwc test matrix should include all previous minor versions of the same major version. When major build version number bumps are made the bwc test matrix will remove the previous major version, and only support the single most recently released major & minor version. ### Gradle Plugins #### Distribution Download Plugin The Distribution Download plugin downloads the latest version of OpenSearch by default, and supports overriding this behavior by setting `customDistributionUrl`. This will help to pull artifacts from custom location for testing during release process. ``` ./gradlew integTest -PcustomDistributionUrl="https://ci.opensearch.org/ci/dbc/bundle-build/1.2.0/1127/linux/x64/dist/opensearch-1.2.0-linux-x64.tar.gz" ```