# Media Insights on AWS Testing - How To

### Overview

MI Framework has 3 types of tests:


*Unit:* Tests of each functional component of the framework at the code level. Scope is the code itself, checks for introduction of bugs by syntax errors, whitespace issues, correct flow control and order of execution, etc

Unit tests can be run locally without an MI deployment. 

*Integration:* Tests of each functional component of the framework with the dependent cloud resources in order to ensure no bugs have been introduced by dependencies and to ensure that a change did not introduce a feature that is incompatible with dependencies.

* Ensures the workflowapi is able to create state machines and perform CRUD operations on dynamodb
* ensures boto3 proxy functions run successfully

Integration tests require MI to be deployed.  

*End to End:* Tests of each functional component of the framework with each other and all dependencies. Scope is the ensure all components work successfully to perform the expected function, e.g. ensure the workflowapi can successfully communicate with the dataplaneapi and successfully complete a workflow

End to end tests require MI to be deployed. 


You can find each of these within the `test` directory of the framework.


### Unit Tests

These tests are invoked by running the `run_unit.sh` script in the `test/unit` directory. The script optionally takes a
positional argument for what component to run the tests on: 
* `./run_unit.sh workflowapi` 
* `./run_unit.sh dataplaneapi`

Otherwise it runs all available unit tests when no arguments are passed


### Integration tests

Before these tests are run, you must have a healthy MI deployment in your
AWS account.

You also need to set the following environment variables:

* `REGION` - The AWS region your MI deployment is in
* `MI_STACK_NAME` - The name of your MI Cloudformation stack
* `AWS_ACCESS_KEY_ID` - A valid AWS Access Key
* `AWS_SECRET_ACCESS_KEY` - A valid AWS Secret Access Key

If you are using temporary STS credentials, you also need to set the session token:

* `AWS_SESSION_TOKEN` - For use with STS temporary credentials

*Note, the IAM credentials you specify must belong to an IAM principal that
has administrator permissions on the MI API's.  

These tests are invoked by running the `run_integ.sh` script in the `test/integ` directory. The script takes a
positional argument for what component to run the tests on: 
* `./run_integ.sh workflowapi` 
* `./run_integ.sh dataplaneapi` 


### End to End tests

Before these tests are run, you must have a healthy MI deployment in your
AWS account.

You also need to set the following environment variables:

* `REGION` - The AWS region your MI deployment is in
* `MI_STACK_NAME` - The name of your MI Cloudformation stack
* `AWS_ACCESS_KEY_ID` - A valid AWS Access Key
* `AWS_SECRET_ACCESS_KEY` - A valid AWS Secret Access Key

If you are using temporary STS credentials, you also need to set the session token:

* `AWS_SESSION_TOKEN` - For use with STS temporary credentials

*Note, the IAM credentials you specify must belong to an IAM principal that
has administrator permissions on the MI API's.  

These tests are invoked by running the `run_e2e.sh` script in the `test/e2e` directory.  
* `./run_e2e.sh`

The pytest command in that script will run all files of the form test_*.py or *_test.py in the current directory and its subdirectories.

### Coverage

#### Workflow API

| API Method | Unit | Integ | e2e |
| ------------- | ------------- | ---------- | -------- |
| `POST /workflow/execution`  | ✅ | ❌ | ✅
| `GET /workflow/execution/{AssetId}`  | ✅ | ❌ | ❌
| `GET /workflow/execution/{Id}`  | ✅ | ❌ | ✅
| `POST /system/configuration`  | ❌ | ❌ | ❌
| `GET /system/configuration`  | ❌ | ❌ | ❌
| `GET /workflow/operation/`  | ❌ | ✅ | ✅
| `POST /workflow/operation/`  | ❌ | ✅ | ❌
| `DELETE /workflow/operation/`  | ❌ | ✅ | ❌
| `GET /workflow/`  | ❌ | ❌ | ✅
| `POST /workflow/`  | ❌ | ✅ | ✅
| `DELETE /workflow/`  | ❌ | ✅ | ✅
| `GET /workflow/stage`  | ❌ | ✅ | ✅
| `POST /workflow/stage`  | ❌ | ✅ | ✅
| `DELETE /workflow/stage`  | ❌ | ✅ | ✅
| `GET /workflow/configuruation`  | ❌ | ❌ | ✅
| `POST /service/translate/get_terminology`  | ❌ | ✅ | ❌
| `POST /service/translate/download_terminology`  | ❌ | ✅ | ❌
| `GET /service/translate/list_terminologies`  | ❌ | ✅ | ❌
| `POST /service/translate/delete_terminology`  | ❌ | ✅ | ❌
| `POST /service/translate/create_terminology`  | ❌ | ✅ | ❌
| `POST /service/translate/get_parallel_data`  | ❌ | ✅ | ❌
| `GET /service/translate/list_parallel_data`  | ❌ | ✅ | ❌
| `POST /service/translate/download_parallel_data`  | ❌ | ✅ | ❌
| `POST /service/translate/delete_parallel_data`  | ❌ | ✅ | ❌
| `POST /service/translate/create_parallel_data`  | ❌ | ✅ | ❌
| `GET /service/transcribe/list_language_models`  | ❌ | ✅ | ❌
| `POST /service/transcribe/describe_language_model`  | ❌ | ✅ | ❌


#### Dataplane API

| API Method | Unit | Integ | e2e |
| ------------- | ------------- | ---------- | -------- |
| `POST /create`  | ✅ | ✅ | ✅
| `POST /metadata/{asset_id}`  | ✅ | ✅ | ✅
| `GET /metadata/{asset}/{operator}`  | ❌ | ✅ | ✅
| `GET /metadata/{asset}`  | ❌ | ❌ | ❌
| `DELETE /metadata/{asset}/{operator}`  | ❌ | ✅ | ✅
| `DELETE /metadata/{asset}`  | ❌ | ✅ | ✅