= Parallelism and throughput using IOR
:toc:
:icons:
:linkattrs:
:imagesdir: ../resources/images


== Summary

This section will demonstrate how increasing the number of threads accessing an EFS file system will significantly improve throughput.

== Duration

NOTE: It will take approximately 15 minutes to complete this section.


== Step-by-step Guide

=== Parallelism

IMPORTANT: Read through all steps below and watch the quick video before continuing.

image::throughput-ior.gif[align="left", width=600]

. Return to the browser-based SSH connection of the *EFS Workshop Linux Instance 2*.
+
TIP: If the SSH connection has timed out, e.g. the session is unresponsive, refresh or reload the current browser tab. If that doesn't resolve the issue, close the browser-based SSH connection window and create a new one. Return to the link:https://console.aws.amazon.com/ec2/[Amazon EC2] console. *_Click_* the radio button next to the instance with the name *EFS Workshop Linux Instance 2*. *_Click_* the *Connect* button. *_Click_* the radio button next to  *EC2 Instance Connect (browser-based SSH connection)*. Leave the default user name as *ec2-user* and *_click_* *Connect*.


==== IOR single shared file (SSF) write tests

IOR is a commonly used file system benchmarking application typically used to evaluate the performance of distributed and parallel file systems. You can read more about IOR here - link:http://wiki.lustre.org/IOR[http://wiki.lustre.org/IOR].

. *_Run_* the following IOR command to generate one 2 GiB file using one thread on an EFS file system.
+
[source,bash]
----
module load mpi/openmpi-x86_64
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'
mpirun --npernode 1 --oversubscribe ior --posix.odirect -t 1m -b 1m -s 2048 -g -v -w -i 1 -k -D 0 -o /efs/ior/ior.bin

----
+
* The results should look similar to this.
+
[source,bash]
----
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Mon Jun  1 19:21:30 2020
Command line        : ior --posix.odirect -t 1m -b 1m -s 2048 -g -v -w -i 1 -D 0 -o /efs/ior/ior.bin
Machine             : Linux ip-10-0-0-11
Start time skew across all tasks: 0.00 sec
TestID              : 0
StartTime           : Mon Jun  1 19:21:30 2020
Path                : /efs/ior
FS                  : 8388608.0 TiB   Used FS: 0.0%   Inodes: 0.0 Mi   Used Inodes: -nan%
Participating tasks: 1

Options:
api                 : POSIX
apiVersion          :
test filename       : /efs/ior/ior.bin
access              : single-shared-file
type                : independent
segments            : 2048
ordering in a file  : sequential
ordering inter file : no tasks offsets
nodes               : 1
tasks               : 1
clients per node    : 1
repetitions         : 1
xfersize            : 1 MiB
blocksize           : 1 MiB
aggregate filesize  : 2 GiB

Results:

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
Commencing write performance test: Mon Jun  1 19:21:30 2020
write     24.38      24.39      83.98       1024.00    1024.00    0.004828   83.98      0.000897   83.99      0
Max Write: 24.38 MiB/sec (25.57 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write          24.38      24.38      24.38       0.00      24.38      24.38      24.38       0.00   83.99048         NA            NA     0      1   1    1   0     0        1         0    0   2048  1048576  1048576    2048.0 POSIX      0
Finished            : Mon Jun  1 19:22:54 2020
----
+
* Record the results somewhere (e.g. your favorite text editor).
* How long did it take (total seconds)?
* What was the bandwidth or throughput (MB/s)?
+
. *_Run_* the following IOR command to generate one 2 GiB file using two threads on an EFS file system.
+
[source,bash]
----
module load mpi/openmpi-x86_64
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'
mpirun --npernode 2 --oversubscribe ior --posix.odirect -t 1m -b 1m -s 1024 -g -v -w -i 1 -k -D 0 -o /efs/ior/ior.bin

----
+
* Record the results somewhere (e.g. your favorite text editor).
* How long did it take (total seconds)?
* What was the bandwidth or throughput (MB/s)?
+
. *_Run_* the following IOR command to generate one 2 GiB file using four threads on an EFS file system.
+
[source,bash]
----
module load mpi/openmpi-x86_64
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'
mpirun --npernode 4 --oversubscribe ior --posix.odirect -t 1m -b 1m -s 512 -g -v -w -i 1 -k -D 0 -o /efs/ior/ior.bin

----
+
* Record the results somewhere (e.g. your favorite text editor).
* How long did it take (total seconds)?
* What was the bandwidth or throughput (MB/s)?
+
. *_Run_* the following IOR command to generate one 2 GiB file using eight threads on an EFS file system.
+
[source,bash]
----
module load mpi/openmpi-x86_64
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'
mpirun --npernode 8 --oversubscribe ior --posix.odirect -t 1m -b 1m -s 256 -g -v -w -i 1 -k -D 0 -o /efs/ior/ior.bin

----
+
* Record the results somewhere (e.g. your favorite text editor).
* How long did it take (total seconds)?
* What was the bandwidth or throughput (MB/s)?
+
. *_Run_* the following IOR command to generate one 2 GiB file using sixteen threads on an EFS file system.
+
[source,bash]
----
module load mpi/openmpi-x86_64
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'
mpirun --npernode 16 --oversubscribe ior --posix.odirect -t 1m -b 1m -s 128 -g -v -w -i 1 -k -D 0 -o /efs/ior/ior.bin

----
+
* Record the results somewhere (e.g. your favorite text editor).
* How long did it take (total seconds)?
* What was the bandwidth or throughput (MB/s)?
+
. *_Run_* the following IOR command to generate one 2 GiB file using thirty-two threads on an EFS file system.
+
[source,bash]
----
module load mpi/openmpi-x86_64
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'
mpirun --npernode 32 --oversubscribe ior --posix.odirect -t 1m -b 1m -s 64 -g -v -w -i 1 -k -D 0 -o /efs/ior/ior.bin

----
+
* Record the results somewhere (e.g. your favorite text editor).
* How long did it take (total seconds)?
* What was the bandwidth or throughput (MB/s)?
+
. *_Run_* the following IOR command to generate one 2 GiB file using sixty-four threads on an EFS file system.
+
[source,bash]
----
module load mpi/openmpi-x86_64
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'
mpirun --npernode 64 --oversubscribe ior --posix.odirect -t 1m -b 1m -s 32 -g -v -w -i 1 -k -D 0 -o /efs/ior/ior.bin

----

==== IOR single shared file (SSF) read tests


. *_Run_* the following IOR command to read one 2 GiB file using one thread.
+
[source,bash]
----
module load mpi/openmpi-x86_64
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'
mpirun --npernode 1 --oversubscribe ior --posix.odirect -t 1m -b 1m -s 2048 -g -v -r -i 1 -k -D 0 -o /efs/ior/ior.bin

----
+
* Record the results somewhere (e.g. your favorite text editor).
* How long did it take (total seconds)?
* What was the bandwidth or throughput (MB/s)?
+
. *_Run_* the following IOR command to read one 2 GiB file using two threads.
+
[source,bash]
----
module load mpi/openmpi-x86_64
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'
mpirun --npernode 2 --oversubscribe ior --posix.odirect -t 1m -b 1m -s 1024 -g -v -r -i 1 -k -D 0 -o /efs/ior/ior.bin

----
+
* Record the results somewhere (e.g. your favorite text editor).
* How long did it take (total seconds)?
* What was the bandwidth or throughput (MB/s)?
+
. *_Run_* the following IOR command to read one 2 GiB file using four threads.
+
[source,bash]
----
module load mpi/openmpi-x86_64
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'
mpirun --npernode 4 --oversubscribe ior --posix.odirect -t 1m -b 1m -s 512 -g -v -r -i 1 -k -D 0 -o /efs/ior/ior.bin

----
+
* Record the results somewhere (e.g. your favorite text editor).
* How long did it take (total seconds)?
* What was the bandwidth or throughput (MB/s)?
+
. *_Run_* the following IOR command to read one 2 GiB file using eight threads.
+
[source,bash]
----
module load mpi/openmpi-x86_64
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'
mpirun --npernode 8 --oversubscribe ior --posix.odirect -t 1m -b 1m -s 256 -g -v -r -i 1 -k -D 0 -o /efs/ior/ior.bin

----
+
* Record the results somewhere (e.g. your favorite text editor).
* How long did it take (total seconds)?
* What was the bandwidth or throughput (MB/s)?
+
. *_Run_* the following IOR command to read one 2 GiB file using sixteen threads.
+
[source,bash]
----
module load mpi/openmpi-x86_64
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'
mpirun --npernode 16 --oversubscribe ior --posix.odirect -t 1m -b 1m -s 128 -g -v -r -i 1 -k -D 0 -o /efs/ior/ior.bin

----
+
* Record the results somewhere (e.g. your favorite text editor).
* How long did it take (total seconds)?
* What was the bandwidth or throughput (MB/s)?
+
. *_Run_* the following IOR command to read one 2 GiB file using thirty-two threads.
+
[source,bash]
----
module load mpi/openmpi-x86_64
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'
mpirun --npernode 32 --oversubscribe ior --posix.odirect -t 1m -b 1m -s 64 -g -v -r -i 1 -k -D 0 -o /efs/ior/ior.bin

----
+
* Record the results somewhere (e.g. your favorite text editor).
* How long did it take (total seconds)?
* What was the bandwidth or throughput (MB/s)?
+
. *_Run_* the following IOR command to read one 2 GiB file using sixty-four threads.
+
[source,bash]
----
module load mpi/openmpi-x86_64
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'
mpirun --npernode 64 --oversubscribe ior --posix.odirect -t 1m -b 1m -s 32 -g -v -r -i 1 -D 0 -o /efs/ior/ior.bin

----
* Record the results somewhere (e.g. your favorite text editor).
* How long did it take (total seconds)?
* What was the bandwidth or throughput (MB/s)?


==== IOR file per process (FPP) write tests

. *_Run_* the following IOR command to generate 2 GiBs of data with one file per thread per directory using one thread (e.g. one file one directory). Notice the new flags *-u* uniqueDir -- use unique directory name for each file-per-process, and *-F* filePerProc — file-per-process flag.
+
[source,bash]
----
module load mpi/openmpi-x86_64
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'
mpirun --npernode 1 --oversubscribe ior --posix.odirect -t 1m -b 1m -s 2048 -g -v -w -i 1 -u -F -k -D 0 -o /efs/ior/ior.bin

----
+
* Record the results somewhere (e.g. your favorite text editor).
* How long did it take (total seconds)?
* What was the bandwidth or throughput (MB/s)?
+
. *_Run_* the following IOR command to generate 2 GiBs of data with one file per thread per directory using two threads (e.g. two files two directories).
+
[source,bash]
----
module load mpi/openmpi-x86_64
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'
mpirun --npernode 2 --oversubscribe ior --posix.odirect -t 1m -b 1m -s 1024 -g -v -w -i 1 -u -F -k -D 0 -o /efs/ior/ior.bin

----
+
* Record the results somewhere (e.g. your favorite text editor).
* How long did it take (total seconds)?
* What was the bandwidth or throughput (MB/s)?
+
. *_Run_* the following IOR command to generate 2 GiBs of data with one file per thread per directory using four threads (e.g. four files four directories).
+
[source,bash]
----
module load mpi/openmpi-x86_64
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'
mpirun --npernode 4 --oversubscribe ior --posix.odirect -t 1m -b 1m -s 512 -g -v -w -i 1 -u -F -k -D 0 -o /efs/ior/ior.bin

----
+
* Record the results somewhere (e.g. your favorite text editor).
* How long did it take (total seconds)?
* What was the bandwidth or throughput (MB/s)?
+
. *_Run_* the following IOR command to generate 2 GiBs of data with one file per thread per directory using eight threads (e.g. eight files eight directories).
+
[source,bash]
----
module load mpi/openmpi-x86_64
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'
mpirun --npernode 8 --oversubscribe ior --posix.odirect -t 1m -b 1m -s 256 -g -v -w -i 1 -u -F -k -D 0 -o /efs/ior/ior.bin

----
+
* Record the results somewhere (e.g. your favorite text editor).
* How long did it take (total seconds)?
* What was the bandwidth or throughput (MB/s)?
+
. *_Run_* the following IOR command to generate 2 GiBs of data with one file per thread per directory using sixteen threads (e.g. sixteen files sixteen directories).
+
[source,bash]
----
module load mpi/openmpi-x86_64
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'
mpirun --npernode 16 --oversubscribe ior --posix.odirect -t 1m -b 1m -s 128 -g -v -w -i 1 -u -F -k -D 0 -o /efs/ior/ior.bin

----
+
* Record the results somewhere (e.g. your favorite text editor).
* How long did it take (total seconds)?
* What was the bandwidth or throughput (MB/s)?
+
. *_Run_* the following IOR command to generate 2 GiBs of data with one file per thread per directory using thirty-two threads (e.g. thirty-two files thirty-two directories).
+
[source,bash]
----
module load mpi/openmpi-x86_64
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'
mpirun --npernode 32 --oversubscribe ior --posix.odirect -t 1m -b 1m -s 64 -g -v -w -i 1 -u -F -k -D 0 -o /efs/ior/ior.bin

----
+
* Record the results somewhere (e.g. your favorite text editor).
* How long did it take (total seconds)?
* What was the bandwidth or throughput (MB/s)?
+
. *_Run_* the following IOR command to generate 2 GiBs of data with one file per thread per directory using sixty-four threads (e.g. sixty-four files sixty-four directories).
+
[source,bash]
----
module load mpi/openmpi-x86_64
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'
mpirun --npernode 64 --oversubscribe ior --posix.odirect -t 1m -b 1m -s 32 -g -v -w -i 1 -u -F -k -D 0 -o /efs/ior/ior.bin

----
+
* Record the results somewhere (e.g. your favorite text editor).
* How long did it take (total seconds)?
* What was the bandwidth or throughput (MB/s)?


==== IOR file per process (FPP) read tests

. *_Run_* the following IOR command to read 2 GiBs of data from the previous write test with one file per thread per directory using one thread (e.g. one file one directory).
+
[source,bash]
----
module load mpi/openmpi-x86_64
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'
mpirun --npernode 1 --oversubscribe ior --posix.odirect -t 1m -b 1m -s 2048 -g -v -r -i 1 -u -F -D 0 -o /efs/ior/ior.bin

----
+
* Record the results somewhere (e.g. your favorite text editor).
* How long did it take (total seconds)?
* What was the bandwidth or throughput (MB/s)?
+
. *_Run_* the following IOR command to read 2 GiBs of data from the previous write test with one file per thread per directory using two threads (e.g. two files two directories) on an EFS file system.
+
[source,bash]
----
module load mpi/openmpi-x86_64
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'
mpirun --npernode 2 --oversubscribe ior --posix.odirect -t 1m -b 1m -s 1024 -g -v -r -i 1 -u -F -D 0 -o /efs/ior/ior.bin

----
+
* Record the results somewhere (e.g. your favorite text editor).
* How long did it take (total seconds)?
* What was the bandwidth or throughput (MB/s)?
+
. *_Run_* the following IOR command to read 2 GiBs of data from the previous write test with one file per thread per directory using four threads (e.g. four files four directories) on an EFS file system.
+
[source,bash]
----
module load mpi/openmpi-x86_64
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'
mpirun --npernode 4 --oversubscribe ior --posix.odirect -t 1m -b 1m -s 512 -g -v -r -i 1 -u -F -D 0 -o /efs/ior/ior.bin

----
+
* Record the results somewhere (e.g. your favorite text editor).
* How long did it take (total seconds)?
* What was the bandwidth or throughput (MB/s)?
+
. *_Run_* the following IOR command to read 2 GiBs of data from the previous write test with one file per thread per directory using eight threads (e.g. eight files eight directories) on an EFS file system.
+
[source,bash]
----
module load mpi/openmpi-x86_64
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'
mpirun --npernode 8 --oversubscribe ior --posix.odirect -t 1m -b 1m -s 256 -g -v -r -i 1 -u -F -D 0 -o /efs/ior/ior.bin

----
+
* Record the results somewhere (e.g. your favorite text editor).
* How long did it take (total seconds)?
* What was the bandwidth or throughput (MB/s)?
+
. *_Run_* the following IOR command to read 2 GiBs of data from the previous write test with one file per thread per directory using sixteen threads (e.g. sixteen files sixteen directories) on an EFS file system.
+
[source,bash]
----
module load mpi/openmpi-x86_64
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'
mpirun --npernode 16 --oversubscribe ior --posix.odirect -t 1m -b 1m -s 128 -g -v -r -i 1 -u -F -D 0 -o /efs/ior/ior.bin

----
+
* Record the results somewhere (e.g. your favorite text editor).
* How long did it take (total seconds)?
* What was the bandwidth or throughput (MB/s)?
+
. *_Run_* the following IOR command to read 2 GiBs of data from the previous write test with one file per thread per directory using thirty-two threads (e.g. thirty-two files thirty-two directories) on an EFS file system.
+
[source,bash]
----
module load mpi/openmpi-x86_64
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'
mpirun --npernode 32 --oversubscribe ior --posix.odirect -t 1m -b 1m -s 64 -g -v -r -i 1 -u -F -D 0 -o /efs/ior/ior.bin

----
+
* Record the results somewhere (e.g. your favorite text editor).
* How long did it take (total seconds)?
* What was the bandwidth or throughput (MB/s)?
+
. *_Run_* the following IOR command to read 2 GiBs of data from the previous write test with one file per thread per directory using sixty-four threads (e.g. sixty-four files sixty-four directories) on an EFS file system.
+
[source,bash]
----
module load mpi/openmpi-x86_64
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'
mpirun --npernode 64 --oversubscribe ior --posix.odirect -t 1m -b 1m -s 32 -g -v -r -i 1 -u -F -D 0 -o /efs/ior/ior.bin

----
+
* Record the results somewhere (e.g. your favorite text editor).
* How long did it take (total seconds)?
* What was the bandwidth or throughput (MB/s)?

. Compare the results from the tests above.  Is there a big difference? Why?

=== Test results

The following table and graphs show the sample results of the IOR 2 GiB single shared file (SSF) tests. Look how increasing the number of threads (increasing parallelism) impacts the throughput and duration.


|=========================================================================
| Operation | Threads | File count| Throughput (MB/s) | Duration (seconds)
| Write     | 1       | 1         | 25.34             | 84.74
| Write     | 2       | 1         | 34.93             | 61.48
| Write     | 4       | 1         | 87.78             | 24.46
| Write     | 8       | 1         | 150.79            | 14.24
| Write     | 16      | 1         | 198.36            | 10.83
| Write     | 32      | 1         | 208.32            | 10.31
| Write     | 64      | 1         | 221.82            | 9.68
| Read      | 1       | 1         | 67.92             | 31.62
| Read      | 2       | 1         | 104.04            | 20.64
| Read      | 4       | 1         | 193.34            | 11.11
| Read      | 8       | 1         | 402.23            | 5.34
| Read      | 16      | 1         | 421.85            | 5.09
| Read      | 32      | 1         | 422.93            | 5.08
| Read      | 64      | 1         | 420.38            | 5.11
|=========================================================================

--
{empty} +
{empty} +
[.left]
.Single Shared File Throughput
image::ior-ssf-throughput.png[align="left"]
{empty} +
{empty} +
[.left]
.Single Shared File Duration
image::ior-ssf-duration.png[align="left"]
--

The following table and graphs show the sample results of the IOR 2 GiB file per process (FPP) tests. Look how increasing the number of threads (increasing parallelism) impacts the throughput and duration.


|==========================================================================
| Operation | Threads | File count | Throughput (MB/s) | Duration (seconds)
| Write     | 1       | 1          |  25.36            | 84.69
| Write     | 2       | 2          |  50.35            | 42.65
| Write     | 4       | 4          |  97.37            | 22.05
| Write     | 8       | 8          | 175.41            | 12.24
| Write     | 16      | 16         | 263.02            |  8.16
| Write     | 32      | 32         | 279.16            |  7.69
| Write     | 64      | 64         | 281.12            |  7.64
| Read      | 1       | 1          |  62.01            | 34.63
| Read      | 2       | 2          | 126.09            | 17.03
| Read      | 4       | 4          | 239.82            |  8.95
| Read      | 8       | 8          | 418.44            |  5.13
| Read      | 16      | 16         | 415.94            |  5.16
| Read      | 32      | 32         | 415.96            |  5.16
| Read      | 64      | 64         | 412.06            |  5.21
|==========================================================================


--
{empty} +
{empty} +
[.left]
.File Per Process Throughput
image::ior-fpp-throughput.png[align="left"]
{empty} +
{empty} +
[.left]
.File Per Process Duration
image::ior-fpp-duration.png[align="left"]
--


== Next section

Click the link below to go to the next section.

image::transfer-tools.png[link=../10-transfer-tools, align="left",width=420]

