== Test the deployment // If steps are required to test the deployment, add them here. If not, remove the heading When the deployment is complete, you can verify the environment by running a {partner-product-short-name} job. This job reads the input datasets and the reference hash table from the S3 genomics data bucket you specified when you deployed the Quick Start, and outputs the analysis results into the designated output location in the S3 bucket. Note that the input datasets and reference hash table must be located in the same S3 bucket, the one specified during deployment. As a quick test, we recommend using an input dataset consisting of a smaller FASTQ pair, such as one of the following data samples: * https://ilmn-dragen-giab-samples.s3.amazonaws.com/WES/HG002/NA24385-AJ-Son-R1-NS_S33_L001_R1_001.fastq.gz[NA24385-AJ-Son-R1-NS_S33_L001_R1_001.fastq.gz^] * https://ilmn-dragen-giab-samples.s3.amazonaws.com/WES/HG002/NA24385-AJ-Son-R1-NS_S33_L001_R2_001.fastq.gz[NA24385-AJ-Son-R1-NS_S33_L001_R2_001.fastq.gz^] // Change link to? https://support.illumina.com/content/dam/illumina-support/help/Illumina_DRAGEN_Bio_IT_Platform_v3_7_1000000141465/Content/SW/Informatics/Dragen/SoftwareCommLine_fDG.htm Use AWS Batch to launch the job either through the AWS Command Line Interface (AWS CLI) or the console. Provide the {partner-product-short-name} job parameters as commands (in the command array within the containerOverrides field in the AWS CLI or the *Command* field in the console). For a full list of options, see the *{partner-product-name} Getting Started Guide* or *{partner-product-name} Bio-IT Platform User Guide* available on the https://sapac.support.illumina.com/sequencing/sequencing_software/dragen-bio-it-platform.html[*{partner-product-name} Bio-IT Platform Support Resources*^] website. Most of the options function exactly as they are described in the guide. However, for options that refer to local files or directories, provide the full path to the S3 bucket that contains the genomics data. In the following sections, we’ve provided instructions for running an end-to-end {partner-product-short-name} job using both methods (AWS CLI and console). In this example, the {partner-product-short-name} job handles mapping, aligning, sorting, deduplication, and variant calling. An input dataset with a paired FASTQ file is used to generate a variant call format (VCF) output file. [[option-1-use-the-aws-cli]] === Option 1: Use the AWS CLI The simplest way to run a {partner-product-short-name} job by using the AWS CLI is to create an input JSON file that describes the job. Here’s an example of a JSON input file named e2e-job.json: ``` { "jobName": "e2e-job", "jobQueue": "dragen-queue", "jobDefinition": "dragen", "containerOverrides": { "vcpus": 16, "memory": 120000, "command": [ "-f", "-r", "s3://", "-1", "s3://", "-2", "s3://", "--RGID", "1", "--RGSM", , "--enable-bam-indexing", "true", "--enable-map-align-output", "true", "--enable-sort", "true", "--output-file-prefix", , "--enable-map-align", "true", "--output-format", "BAM", "--output-directory", "s3://", "--enable-variant-caller", "true" ] }, "retryStrategy": { "attempts": 1 } } ``` You can then launch the job from the command line by using the https://docs.aws.amazon.com/cli/latest/reference/batch/submit-job.html[submit-job^] command and specifying the e2e-job.json file as input: ``` aws batch submit-job --cli-input-json file://e2e-job.json ``` You can submit multiple Batch jobs from the CLI. Here is an example with two JSON input files named `e2e-job1.json` and `e2e-job2.json`. ``` { "jobName": "e2e-job1", "jobQueue": "dragen-queue", "jobDefinition": "dragen", "containerOverrides": { "vcpus": 16, "memory": 120000, "command": [ "-f", "-r", "s3://", "-1", "s3://", "-2", "s3://", "--RGID", "1", "--RGSM", , "--enable-bam-indexing", "true", "--enable-map-align-output", "true", "--enable-sort", "true", "--output-file-prefix", , "--enable-map-align", "true", "--output-format", "BAM", "--output-directory", "s3://", "--enable-variant-caller", "true" ] }, "retryStrategy": { "attempts": 1 } } ``` ``` { "jobName": "e2e-job2", "jobQueue": "dragen-queue", "jobDefinition": "dragen", "containerOverrides": { "vcpus": 16, "memory": 120000, "command": [ "-f", "-r", "s3://", "-1", "s3://", "-2", "s3://", "--RGID", "1", "--RGSM", , "--enable-bam-indexing", "true", "--enable-map-align-output", "true", "--enable-sort", "true", "--output-file-prefix", , "--enable-map-align", "true", "--output-format", "BAM", "--output-directory", "s3://", "--enable-variant-caller", "true" ] }, "retryStrategy": { "attempts": 1 } } ``` The following bash script can be used to submit the above two jobs. ``` #!/bin/bash echo "Starting first job ..." aws batch submit-job --cli-input-json e2e-job1.json > ./job-output-log echo "Starting second job ..." aws batch submit-job --cli-input-json e2e-job1.json >> ./job-output-log echo "No more jobs pending!" ``` [[option-2-use-the-aws-batch-console]] === Option 2: Use the AWS Batch Console To run the {partner-product-short-name} job from the console:. 1. Open the AWS Batch console at https://console.aws.amazon.com/batch/[https://console.aws.amazon.com/batch/^] 2. From the navigation bar, choose the AWS Region you used for the Quick Start deployment. 3. In the navigation pane, choose *Jobs*, *Submit new job*. 4. Fill out these fields, as shown in <>: + ** *Job name*: Enter a unique name for the job. ** *Job definition*: Choose the {partner-product-short-name} job definition that was created by the Quick Start and displayed in the *Outputs* tab of the AWS CloudFormation console in step 3(9). ** *Job queue*: Choose dragen-queue, which was created by the Quick Start. ** *Job type*: Choose *Single*. ** *Command*: Specify the {partner-product-short-name}-specific parameters shown in the JSON command array in link:#option-1-use-the-aws-cli[option 1]. ** *vCPUs, Memory, Job attempts, Execution timeout*: Keep the defaults that are specified in the job definition. + For more information, see the https://docs.aws.amazon.com/batch/latest/userguide/submit_job.html[AWS Batch documentation^]. + 5. Choose *Submit*. + :xrefstyle: short [#runjob] .Running a {partner-product-short-name} job from the AWS Batch console [link=images/image5.png] image::../images/image5.png[runjob,width=733,height=427] + 6. Monitor the job status in the AWS Batch window to see if it succeeded or failed. For more information about job states and exit codes, see the https://docs.aws.amazon.com/batch/latest/userguide/job_states.html[AWS Batch documentation^]. [[best-practices-using-dragen-on-aws]] == Best practices for using {partner-product-short-name} on AWS // Provide post-deployment best practices for using the technology on AWS, including considerations such as migrating data, backups, ensuring high performance, high availability, etc. Link to software documentation for detailed information. For simplicity, we recommend that you create your S3 bucket in the AWS Region that you are deploying the Quick Start into. In some use cases, as outlined in the https://support.illumina.com/downloads/illumina-dragen-bio-it-platform-3-7-user-guide.html[DRAGEN User Guide^] on the Illumina website you might need to attach EBS volumes to instances.  The {partner-product-short-name} guides are available as links from the https://aws.amazon.com/marketplace/pp/B07CZ3F5HY[{partner-product-short-name} Bio-IT Platform webpage^] in AWS Marketplace (see the _Usage Information_ section on that page). == Security // Provide post-deployment best practices for using the technology on AWS, including considerations such as migrating data, backups, ensuring high performance, high availability, etc. Link to software documentation for detailed information. {partner-product-short-name} doesn’t enforce any specific security requirements. However, for security, this Quick Start deploys {partner-product-short-name} into private subnets that aren’t externally reachable from outside the VPC (they can access the internet only through NAT gateways). Please consult your IT and security teams for image hardening, encryption, and other security requirements.