## HTTP logs workload This workload is based on [Web server logs from the 1998 Football world cup](http://ita.ee.lbl.gov/html/contrib/WorldCup.html). Modifications: * Applied number to IP conversion as suggested in the original readme * Removed illegal characters in "object_mappings.sort" * Transformed the source data to a bulk-friendly JSON format (ignoring all entries that contained unrecognised / problematic characters and invalid IP addresses like "0"; around 0.001% of the source data was lost due to this approach) ### Example Document ```json { "@timestamp": 898459201, "clientip": "211.11.9.0", "request": "GET /english/index.html HTTP/1.0", "status": 304, "size": 0 } ``` Alternatively, an `unparsed` set of documents are also provided. The `unparsed` data set is identical to the standard data set, except the timestamp is ISO8601 and all the fields are unparsed via the `message` field. For example: ```json {"message" : "211.11.9.0 - - [1998-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0"} ``` ### Parameters This workload allows the following parameters to be specified using `--workload-params`: * `bulk_size` (default: 5000) * `bulk_indexing_clients` (default: 8): Number of clients that issue bulk indexing requests. * `ingest_percentage` (default: 100): A number between 0 and 100 that defines how much of the document corpus should be ingested. * `conflicts` (default: "random"): Type of id conflicts to simulate. Valid values are: 'sequential' (A document id is replaced with a document id with a sequentially increasing id), 'random' (A document id is replaced with a document id with a random other id). * `conflict_probability` (default: 25): A number between 0 and 100 that defines the probability of id conflicts. This requires to run the respective test_procedure. Combining ``conflicts=sequential`` and ``conflict-probability=0`` makes Benchmark generate index ids by itself, instead of relying on OpenSearch's `automatic id generation`. * `number_of_replicas` (default: 0) * `number_of_shards` (default: 5) * `source_enabled` (default: true): A boolean defining whether the `_source` field is stored in the index. * `index_settings`: A list of index settings. Index settings defined elsewhere (e.g. `number_of_replicas`) need to be overridden explicitly. * `cluster_health` (default: "green"): The minimum required cluster health. * `ingest_pipeline`: Only applicable for `--test-procedure=append-index-only-with-ingest-pipeline`, selects which ingest node pipeline to run. Valid options are `'baseline'` (default), `'grok'` and `'geoip'`. For example: `--test-procedure=append-index-only-with-ingest-pipeline --workload-params="ingest_pipeline:'baseline'" ` * `error_level` (default: "non-fatal"): Available for bulk operations only to specify ignore-response-error-level. * `target_throughput` (default: default values for each operation): Number of requests per second, `none` for no limit. * `search_clients`: Number of clients that issues search requests. ### Beta Feature: Increasing the size of the data corpus This workload provides for a feature to use a generated data corpus in lieu of the provided corpora files (which currently total ~31 GB.) The generated corpus could, for instance, be 100 GB or more. For more details on generating such a corpus, run the following command: ``` expand-data-corpus.py -h ``` Once a corpus has been generated, it can be used for a test by supplying the following parameter via `--workoad-params`: * `generated_corpus:t`: Use the generated data corpus instead of the corpora files packaged with this track If there are multiple generated corpora files, they are all used concurrently. Ingestion of the generated and the default corpora are mutually exclusive in any single OSB run. Once ingested, however, queries packaged with this workload will operate on the entire loaded data set. ### License Original license text: Copyright (C) 1997, 1998, 1999 Hewlett-Packard Company ALL RIGHTS RESERVED. The enclosed software and documentation includes copyrighted works of Hewlett-Packard Co. For as long as you comply with the following limitations, you are hereby authorized to (i) use, reproduce, and modify the software and documentation, and to (ii) distribute the software and documentation, including modifications, for non-commercial purposes only. 1. The enclosed software and documentation is made available at no charge in order to advance the general development of the Internet, the World-Wide Web, and Electronic Commerce. 2. You may not delete any copyright notices contained in the software or documentation. All hard copies, and copies in source code or object code form, of the software or documentation (including modifications) must contain at least one of the copyright notices. 3. The enclosed software and documentation has not been subjected to testing and quality control and is not a Hewlett-Packard Co. product. At a future time, Hewlett-Packard Co. may or may not offer a version of the software and documentation as a product. 4. THE SOFTWARE AND DOCUMENTATION IS PROVIDED "AS IS". HEWLETT-PACKARD COMPANY DOES NOT WARRANT THAT THE USE, REPRODUCTION, MODIFICATION OR DISTRIBUTION OF THE SOFTWARE OR DOCUMENTATION WILL NOT INFRINGE A THIRD PARTY'S INTELLECTUAL PROPERTY RIGHTS. HP DOES NOT WARRANT THAT THE SOFTWARE OR DOCUMENTATION IS ERROR FREE. HP DISCLAIMS ALL WARRANTIES, EXPRESS AND IMPLIED, WITH REGARD TO THE SOFTWARE AND THE DOCUMENTATION. HP SPECIFICALLY DISCLAIMS ALL WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. 5. HEWLETT-PACKARD COMPANY WILL NOT IN ANY EVENT BE LIABLE FOR ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES (INCLUDING LOST PROFITS) RELATED TO ANY USE, REPRODUCTION, MODIFICATION, OR DISTRIBUTION OF THE SOFTWARE OR DOCUMENTATION.