# Fuzz testing Modern fuzz testers are very effective and we wish to use them to ensure that no silly bugs creep into BoringSSL. We use Clang's [libFuzzer](http://llvm.org/docs/LibFuzzer.html) for fuzz testing and there are a number of fuzz testing functions in `fuzz/`. They are not built by default because they require that the rest of BoringSSL be built with some changes that make fuzzing much more effective, but are completely unsafe for real use. In order to build the fuzz tests you will need at least Clang 6.0. Pass `-DFUZZ=1` on the CMake command line to enable building BoringSSL with coverage and AddressSanitizer, and to build the fuzz test binaries. You'll probably need to set the `CC` and `CXX` environment variables too, like this: ``` CC=clang CXX=clang++ cmake -GNinja -DFUZZ=1 -B build ninja -C build ``` From the `build/` directory, you can then run the fuzzers. For example: ``` ./fuzz/cert -max_len=10000 -jobs=32 -workers=32 ../fuzz/cert_corpus/ ``` The arguments to `jobs` and `workers` should be the number of cores that you wish to dedicate to fuzzing. By default, libFuzzer uses the largest test in the corpus (or 64 if empty) as the maximum test case length. The `max_len` argument overrides this. The recommended values of `max_len` for each test are: | Test | `max_len` value | |---------------|-----------------| | `bn_div` | 384 | | `bn_mod_exp` | 4096 | | `cert` | 10000 | | `client` | 20000 | | `pkcs8` | 2048 | | `privkey` | 2048 | | `server` | 4096 | | `session` | 8192 | | `spki` | 1024 | | `read_pem` | 512 | | `ssl_ctx_api` | 256 | These were determined by rounding up the length of the largest case in the corpus. There are directories in `fuzz/` for each of the fuzzing tests which contain seed files for fuzzing. Some of the seed files were generated manually but many of them are “interesting” results generated by the fuzzing itself. (Where “interesting” means that it triggered a previously unknown path in the code.) ## Minimising the corpora When a large number of new seeds are available, it's a good idea to minimise the corpus so that different seeds that trigger the same code paths can be deduplicated. In order to minimise all the corpora, build for fuzzing and run `./fuzz/minimise_corpora.sh`. Note that minimisation is, oddly, often not idempotent for unknown reasons. ## Fuzzer mode When `-DFUZZ=1` is passed into CMake, BoringSSL builds with `BORINGSSL_UNSAFE_FUZZER_MODE` and `BORINGSSL_UNSAFE_DETERMINISTIC_MODE` defined. This modifies the library to be more friendly to fuzzers. If `BORINGSSL_UNSAFE_DETERMINISTIC_MODE` is set, BoringSSL will: * Replace `RAND_bytes` with a deterministic PRNG. Call `RAND_reset_for_fuzzing()` at the start of fuzzers which use `RAND_bytes` to reset the PRNG state. * Use a hard-coded time instead of the actual time. Additionally, if `BORINGSSL_UNSAFE_FUZZER_MODE` is set, BoringSSL will: * Modify the TLS stack to perform all signature checks (CertificateVerify and ServerKeyExchange) and the Finished check, but always act as if the check succeeded. * Treat every cipher as the NULL cipher. * Tickets are unencrypted and the MAC check is performed but ignored. * renegotiation\_info checks are ignored. This is to prevent the fuzzer from getting stuck at a cryptographic invariant in the protocol. ## TLS transcripts The `client` and `server` corpora are seeded from the test suite. The test suite has a `-fuzzer` flag which mirrors the fuzzer mode changes above and a `-deterministic` flag which removes all non-determinism on the Go side. Not all tests pass, so `ssl/test/runner/fuzzer_mode.json` contains the necessary suppressions. The `run_tests` target will pass appropriate command-line flags. There are separate corpora, `client_corpus_no_fuzzer_mode` and `server_corpus_no_fuzzer_mode`. These are transcripts for fuzzers with only `BORINGSSL_UNSAFE_DETERMINISTIC_MODE` defined. To build in this mode, pass `-DNO_FUZZER_MODE=1` into CMake. This configuration is run in the same way but without `-fuzzer` and `-shim-config` flags. If both sets of tests pass, refresh the fuzzer corpora with `refresh_ssl_corpora.sh`: ``` cd fuzz ./refresh_ssl_corpora.sh /path/to/fuzzer/mode/build /path/to/non/fuzzer/mode/build ``` ## Adding New Fuzz Test Targets When adding new functionality, adding new fuzz tests are important to provide additional testing and verification that we are correct. ### Steps of generating Fuzz corpus 1. `NEW_FUNCTION='name_of_new_fuzzing_target'` 2. `touch fuzz/${NEW_FUNCTION}.cc` \ Write fuzzing code for libfuzzer to parse in `fuzz/${NEW_FUNCTION}.cc`. The code in this file will be how Libfuzzer identifies how to parse any data inputs the fuzzer gives. Fuzz tests test upon random data inputs, so in most cases we write code to test on data parsing (and reserializing the data again if applicable). 3. `mkdir fuzz/${NEW_FUNCTION}_corpus_raw` \ Import any existing unit test files into the `fuzz/${NEW_FUNCTION}_corpus_raw` directory. These will be used to create our new fuzzing corpus. 4. Add `fuzzer('name_of_new_fuzzing_target')` target in the `fuzz/CmakeLists.txt` file. 5. Build AWS-LC with fuzzing enabled (only buildable with clang) ``` mkdir test_build_dir cmake ../aws-lc -GNinja "-B`pwd`/test_build_dir" "-DCMAKE_INSTALL_PREFIX=`pwd`/test_build_dir/" \ -DCMAKE_BUILD_TYPE=RelWithDebInfo -DFUZZ=1 -DASAN=1 -DBUILD_TESTING=OFF ninja -C test_build_dir ``` 6. Run the fuzzer with our import test files by running:\ **Note**: * `max_len`: set for maximum test file size if needed * `max_total_time`: can be adjusted to how long you'd like the fuzzer to run. * `workers`: should be set accordingly to how many cpu processors are available. ``` mkdir fuzz/${NEW_FUNCTION}_corpus_temp NUM_CPU_THREADS=$(grep -c ^processor /proc/cpuinfo) ./test_build_dir/fuzz/${NEW_FUNCTION} -jobs=${NUM_CPU_THREADS} -workers=${NUM_CPU_THREADS} -timeout=5 \ -print_final_stats=1 -max_total_time=1800 fuzz/${NEW_FUNCTION}_corpus_temp fuzz/${NEW_FUNCTION}_corpus_raw ``` 7. By default, the fuzzing process will continue until `max_total_time` has been reached. When a bug is found, any crashes or sanitizer failures will be reported in the console and the particular corpus file that triggered the bug will be written into the root directory (as `crash-`, `leak-`, or `timeout-`). If any of these files appear, we should look into what is causing the failure. 8. If no failures emerge, we can merge the generated corpus to minimize the amount of files, while still providing full coverage. ``` mkdir fuzz/${NEW_FUNCTION}_corpus ./test_build_dir/fuzz/${NEW_FUNCTION} -merge=1 fuzz/${NEW_FUNCTION}_corpus fuzz/${NEW_FUNCTION}_corpus_temp fuzz/${NEW_FUNCTION}_corpus_raw ``` 9. Remove original directories, files, and fuzz logs used to generate the orginal corpus. ``` rmdir fuzz/${NEW_FUNCTION}_corpus_temp rmdir fuzz/${NEW_FUNCTION}_corpus_raw rm -r fuzz-*.log ``` ### Steps of verifying the generated Fuzz corpus. 1. Verify if the generated corpus has good coverage. - 1.1. Add temporary patch. For example, below patch adds some print statements to check if the fuzz inputs check the code blocks. ``` $ git diff fuzz/ssl_serialization.cc diff --git a/fuzz/ssl_serialization.cc b/fuzz/ssl_serialization.cc index 9b4bc804d..c8b3208d8 100644 --- a/fuzz/ssl_serialization.cc +++ b/fuzz/ssl_serialization.cc @@ -24,6 +24,7 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t *buf, size_t len) { // If the format was invalid, just return. if (!ssl) { + fprintf(stderr, "SSL Serialization fuzz executed code block 1.\n"); return 0; } @@ -31,6 +32,7 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t *buf, size_t len) { size_t encoded_len; uint8_t *encoded; if (!SSL_to_bytes(ssl.get(), &encoded, &encoded_len)) { + fprintf(stderr, "SSL Serialization fuzz executed code block 2.\n"); uint32_t e = ERR_get_error(); if (e == 0) { fprintf(stderr, "In Fuzz, SSL_to_bytes failed without giving a error code.\n"); @@ -46,6 +48,7 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t *buf, size_t len) { return 1; } + fprintf(stderr, "SSL Serialization fuzz executed code block 3.\n"); OPENSSL_free(encoded); return 0; } ``` - 1.2. Run fuzz script. ```sh rm -rf nohup.out && bash -c "nohup sh -c './tests/ci/run_fuzz_tests.sh' &" && sleep 2 && chmod a+rw nohup.out ``` - 1.3. Check the fuzz log. All the injected statements should be printed. That means the fuzz inputs cover the code blocks we expect. ``` $ cat nohup.out | grep "SSL Serialization fuzz executed code block 1" | wc -l 13545099 $ cat nohup.out | grep "SSL Serialization fuzz executed code block 2" | wc -l 12303 $ cat nohup.out | grep "SSL Serialization fuzz executed code block 3" | wc -l 68918 ``` ### Fuzz corpus added by aws-lc 1. `ssl_serialization`: detailed steps are in `CryptoAlg-850?selectedConversation=4ec5f34a-451e-4ae0-ae4c-0322b9f22108`.