# Guard: Unit Testing ## Recommended Readings [Guard: Clauses](CLAUSES.md) [Guard: Query and Filtering](QUERY_AND_FILTERING.md) [Guard: Context-Aware Evaluations, this and Loops](CONTEXTAWARE_EVALUATIONS_AND_LOOPS.md) It is essential for rule authors to gain confidence that policies defined in a Guard file do indeed comply with expectations. With the Guard 2.0 tool you can leverage the built-in unit testing support that helps validate Guard policy files. ## Anatomy of a Unit Testing File All unit testing files are YAML/JSON formatted files. Each test file can contain multiple inputs along with the expected outcomes for rules written inside a Guard file. The anatomy of a unit testing file is as shown below (YAML format shown): ```yaml --- - name: input: expectations: rules: : [PASS|FAIL|SKIP] ``` There can be multiple samples to assess different expectations. It is recommended to start with testing for empty inputs and then progressively add information for assessing various rules and clauses that you are attempting to assess. Let’s illustrate this with an example. ## Example Rule Here is the intent for the rule: 1. Check that all [Amazon API Gateway resource types](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-apigateway-restapi.html) defined inside a CloudFormation template * are deployed for only private access. * and at least one policy statement that allows access from some VPC. ``` # # Select from Resources section of the template all ApiGateway resources # present in the template. # let api_gws = Resources.*[ Type == 'AWS::ApiGateway::RestApi'] # # Rule intent # a) All ApiGateway instances deployed must be private # b) All ApiGateway instances must have atleast one IAM policy condition key to allow access m a VPC # # Expectations: # 1) SKIP when there are not API Gateway instances in the plate # 2) PASS when ALL ApiGateway instances MUST be "PRIVATE" # ALL ApiGateway instances MUST have one IAM Condition key with aws:sourceVpc or :SourceVpc # 3) FAIL otherwise # # rule check_rest_api_is_private when %api_gws !empty %api_gws { Properties.EndpointConfiguration.Types[*] == "PRIVATE" } } rule check_rest_api_has_vpc_access when check_rest_api_is_private { %api_gws { Properties { # # ALL ApiGateways must have atleast one IAM statement that has Condition keys with # aws:sourceVpc # some Policy.Statement[*] { Condition.*[ keys == /aws:[sS]ource(Vpc|VPC|Vpce|VPCE)/ ] !empty } } } } ``` The rule is shown above. How do you test that the rules do work as intended? ### Testing the rule(s) First you don’t write complex rules. They get harder to maintain and test. The recommendation is you write smaller rules that you combine to specify larger rules. You will start with intent (1). ``` # # Select from Resources section of the template all ApiGateway resources # present in the template. # let api_gws = Resources.*[ Type == 'AWS::ApiGateway::RestApi' ] # # Rule intent # a) All ApiGateway instances deployed must be private # b) All ApiGateway instances must have atleast one IAM policy condition key to allow accessm a VPC # # Expectations: # 1) SKIP when there are not API Gateway instances in the template # 2) PASS when ALL ApiGateway instances MUST be "PRIVATE" and ALL ApiGateway instances MUST have one IAM Condition key with aws:sourceVpc or aws:SourceVpc # 3) FAIL otherwise # # rule check_rest_api_is_private when %api_gws !empty { %api_gws { Properties.EndpointConfiguration.Types[*] == "PRIVATE" } } ``` First, you should test expectations starting from empty input and progressively add properties needed to test. Start a file with the suffix `_tests.yaml`. If the name of the Guard policy file is `api_gateway_private.guard` then you should name the testing file `api_gateway_private_tests.yaml`. Here is the first test: ```yaml --- - name: MyTest input: {} expectations: rules: check_rest_api_is_private: SKIP ``` You `expect` that `rule` `check_rest_api_is_private` to skip. You can now run the test using: ```bash cfn-guard test \ --rules-file api_gateway_private.guard \ --test-data api_gateway_private_tests.yaml ``` The output you see is `PASS` for the test. ```bash Test Case #1 Name: "MyTest" PASS Rules: check_rest_api_is_private: Expected = SKIP, Evaluated = SKIP ``` Now let us extend the testing to include empty resources: ```yaml --- - name: MyTest1 input: {} expectations: rules: check_rest_api_is_private: SKIP - name: MyTest2 input: Resources: {} expectations: rules: check_rest_api_is_private: SKIP ``` Now you can re-run the test and should see: ```bash Test Case #1 Name: "MyTest1" PASS Rules: check_rest_api_is_private: Expected = SKIP, Evaluated = SKIP Test Case #2 Name: "MyTest2" PASS Rules: check_rest_api_is_private: Expected = SKIP, Evaluated = SKIP ``` You can now add an Amazon API Gateway resource type that was missing `Properties` (This isn’t a valid CFN template, but nonetheless testing that the rule works correctly even for these malformed inputs is useful.), and one that satisfies only `EndpointConfiguration` attribute and has no policy statements defined. You should expect this to `FAIL`. Here is the testing you should have: ```yaml --- - name: MyTest1 input: {} expectations: rules: check_rest_api_is_private: SKIP - name: MyTest2 input: Resources: {} expectations: rules: check_rest_api_is_private: SKIP - name: MyTest3 input: Resources: apiGw: Type: AWS::ApiGateway::RestApi expectations: rules: check_rest_api_is_private: FAIL - name: MyTest4 input: Resources: apiGw: Type: AWS::ApiGateway::RestApi Properties: EndpointConfiguration: Types: "PRIVATE" expectations: rules: check_rest_api_is_private: PASS ``` and a sample run you should see: ```bash Test Case #1 Name: "MyTest1" PASS Rules: check_rest_api_is_private: Expected = SKIP, Evaluated = SKIP Test Case #2 Name: "MyTest2" PASS Rules: check_rest_api_is_private: Expected = SKIP, Evaluated = SKIP Test Case #3 Name: "MyTest3" PASS Rules: check_rest_api_is_private: Expected = FAIL, Evaluated = FAIL Test Case #4 Name: "MyTest4" PASS Rules: check_rest_api_is_private: Expected = PASS, Evaluated = PASS ``` ### How do I know that `EndpointConfiguration` check did indeed succeed for PASS case? When testing you can specify the `--verbose` flag that lets you inspect evaluation results. [Before you ask, yes we plan to expose clause success failure summary like validate, but currently we have verbose as the option]. Often verbose context is needed to understand the evaluations. For this run, let us test only the last input, so you should comment out the earlier tests for this run (or create a file with this single input). Here is how it would look: ```yaml --- --- #- name: "MyTest1" # input: {} # expectations: # rules: # check_rest_api_is_private: SKIP #- name: "MyTest2" # input: # Resources: {} # expectations: # rules: # check_rest_api_is_private: SKIP #- name: "MyTest3" # input: # Resources: # apiGw: # Type: AWS::ApiGateway::RestApi # expectations: # rules: # check_rest_api_is_private: FAIL - name: "MyTest4" input: Resources: apiGw: Type: AWS::ApiGateway::RestApi Properties: EndpointConfiguration: Types: "PRIVATE" expectations: rules: check_rest_api_is_private: PASS ``` Now you re-run the test but with the verbose flag on: ```bash cfn-guard test \ --rules-file api_gateway_private.guard \ --test-data api_gateway_private_tests.yaml \ --verbose ``` Here is the output from that run: ```bash Test Case #1 Name: "MyTest4" PASS Rules: check_rest_api_is_private: Expected = PASS, Evaluated = PASS Rule(check_rest_api_is_private, PASS) | Message: DEFAULT MESSAGE(PASS) Condition(check_rest_api_is_private, PASS) | Message: DEFAULT MESSAGE(PASS) Clause(Clause(Location[file:api_gateway_private.guard, line:20, column:37], Check: %api_gws NOT EMPTY ), PASS) | From: Map((Path("/Resources/apiGw"), MapValue { keys: [String((Path("/Resources/apiGw/Type"), "Type")), String((Path("/Resources/apiGw/Properties"), "Properties"))], values: {"Type": String((Path("/Resources/apiGw/Type"), "AWS::ApiGateway::RestApi")), "Properties": Map((Path("/Resources/apiGw/Properties"), MapValue { keys: [String((Path("/Resources/apiGw/Properties/EndpointConfiguration"), "EndpointConfiguration"))], values: {"EndpointConfiguration": Map((Path("/Resources/apiGw/Properties/EndpointConfiguration"), MapValue { keys: [String((Path("/Resources/apiGw/Properties/EndpointConfiguration/Types"), "Types"))], values: {"Types": String((Path("/Resources/apiGw/Properties/EndpointConfiguration/Types"), "PRIVATE"))} }))} }))} })) | Message: (DEFAULT: NO_MESSAGE) Conjunction(cfn_guard::rules::exprs::GuardClause, PASS) | Message: DEFAULT MESSAGE(PASS) Clause(Clause(Location[file:api_gateway_private.guard, line:22, column:5], Check: Properties.EndpointConfiguration.Types[*] EQUALS String("PRIVATE")), PASS) | Message: (DEFAULT: NO_MESSAGE) ``` This is bit dense, but the key observation is the line that says `Clause(Location[file:api_gateway_private.guard, line:22, column:5], Check: Properties.EndpointConfiguration.Types[*] EQUALS String("PRIVATE")), PASS) ` that states that the check did PASS. The example also showed the case where `Types` was expected to be an array, but a single value was given. Guard will still evaluate and still provide a correct result. Now you should add a test case for `FAIL`ure. You can add this to the end of the test file. ```yaml - name: "MyTest" input: Resources: apiGw: Type: AWS::ApiGateway::RestApi Properties: EndpointConfiguration: Types: [PRIVATE, REGIONAL] expectations: rules: check_rest_api_is_private: FAIL ``` Now let us run the `test` command again: ```bash Test Case #1 Name: "MyTest" PASS Rules: check_rest_api_is_private: Expected = FAIL, Evaluated = FAIL Rule(check_rest_api_is_private, FAIL) | Message: DEFAULT MESSAGE(FAIL) Condition(check_rest_api_is_private, PASS) | Message: DEFAULT MESSAGE(PASS) Clause(Clause(Location[file:api_gateway_private.guard, line:3, column:37], Check: %api_gws NOT EMPTY ), PASS) | From: Map((Path("/Resources/apiGw"), MapValue { keys: [String((Path("/Resources/apiGw/Type"), "Type")), String((Path("/Resources/apiGw/Properties"), "Properties"))], values: {"Type": String((Path("/Resources/apiGw/Type"), "AWS::ApiGateway::RestApi")), "Properties": Map((Path("/Resources/apiGw/Properties"), MapValue { keys: [String((Path("/Resources/apiGw/Properties/EndpointConfiguration"), "EndpointConfiguration"))], values: {"EndpointConfiguration": Map((Path("/Resources/apiGw/Properties/EndpointConfiguration"), MapValue { keys: [String((Path("/Resources/apiGw/Properties/EndpointConfiguration/Types"), "Types"))], values: {"Types": List((Path("/Resources/apiGw/Properties/EndpointConfiguration/Types"), [String((Path("/Resources/apiGw/Properties/EndpointConfiguration/Types/0"), "PRIVATE")), String((Path("/Resources/apiGw/Properties/EndpointConfiguration/Types/1"), "REGIONAL"))]))} }))} }))} })) | Message: DEFAULT MESSAGE(PASS) BlockClause(Block[Location[file:api_gateway_private.guard, line:4, column:3]], FAIL) | Message: DEFAULT MESSAGE(FAIL) Conjunction(cfn_guard::rules::exprs::GuardClause, FAIL) | Message: DEFAULT MESSAGE(FAIL) Clause(Clause(Location[file:api_gateway_private.guard, line:5, column:5], Check: Properties.EndpointConfiguration.Types[*] EQUALS String("PRIVATE")), FAIL) | From: String((Path("/Resources/apiGw/Properties/EndpointConfiguration/Types/1"), "REGIONAL")) | To: String((Path("api_gateway_private.guard/5/5/Clause/"), "PRIVATE")) | Message: (DEFAULT: NO_MESSAGE) ``` The check fails as `REGIONAL` was not expected. ### Understanding the verbose output when testing The verbose output mostly follows the structure inside the Guard policy file. Every block in the Guard policy file is a block in the verbose output. The top-most is each rule. If there are `when` conditions against the rule then they would appear as a sibling `Condition` block. In this example the condition `%api_gws !empty` is being tested and it `PASS`es. ``` rule check_rest_api_is_private when %api_gws !empty { ``` Once the condition passes, we drop into the rule clauses. ``` %api_gws { Properties.EndpointConfiguration.Types[*] == "PRIVATE" } ``` `%api_gws` is a block Guard rule that corresponds to `BlockClause` level in the output (`line: 21`). The next is a set of conjunction (AND) clauses, where each conjunction clause is a set of disjunctions (ORs). The` Conjunction` has a single clause, `Properties.EndpointConfiguration.Types[*] == "PRIVATE"`, the output therefore shows a single `Clause`. The path `/Resources/apiGw/Properties/EndpointConfiguration/Types/1` shows which values in the input are getting compared, in this case the element for `Types` indexed at `1`.