man iotest
NAME
iotest - write or read files (IO test/benchmark or scrub)
SYNOPSIS
iotest
[-hH?CvVdDyYraRnwWetTp:P:i:I:o:O:s:S:b:B:c:z:Z:] [file]
DESCRIPTION
Write and/or read a file timing the IO and calculating speed. This program has
been used for IO benchmarking on a wide variety of UNIX and Linux variants includ-
ing many filesystem types such as native, NFS, GPFS, Lustre as well as many storage
sub-system configurations. This program has also been used for scrubbing/sanitiz-
ing files and filesystems.
Many systems use available memory to cache IO. Read rates may be false if file was
already buffered in memory. Some systems report write completion before a write is
committed. By default this program uses fsync() to avoid this but many "modern"
implementations continue to buffer. This program includes drop_caches and sync
options (see examples). For more accurate IO rates (benchmarks), specifying a file
size greater than system memory size and using a dedicated system is recommended.
Note, storage sub-systems can also cache data.
OPTIONS
-r | --read
Read, if also -w read after write.
-w | --write
Write, implied if neither -r or -w specified.
-W | -n | --no-sync
Write, does not fsync().
-e | --verify
Read after write and verify contents of last write. Cannot be used with
text|character writes.
-a | --all
Read after write and verify contents of all writes. Cannot be used with
text|character writes.
-t | --truncate
Initial open "w" (truncate) vs. "r+" (rewrite).
-s | --size-mb mb
Specify size(mb), default is 128mb.
-D | - | --drop_caches
Issue "drop_caches 3" before closing. Use -D to issue with sudo. Both bina-
ries must be in path.
-Y | -y | --sync
Issue "sync" before closing. Use -Y to issue with sudo. Both binaries must
be in path.
-b | --block-size bs
Specfify blocksize, default is 1kb, e.g., "-b 1". Blocksize must be a
either a divisor of 1024 (1k,2k,4k) or a multiple of 1024 (1mb,2mb,3mb,...).
For block sizes greater than 1mb the read|write size will be rounded up.
-p | --pattern pattern
Specify pattern(s):
0 is all zeros
1 is all ones
2 is text (ascii) and is default
3 is random
4 complement last pattern
=c is character
/ff is hex value
Even if --truncate is specified, multiple patterns will re-open with "r+" to
rewrite. To do a four pass write (sanitize) of a 1gb file with 0x00, 0xff,
0xf0, 0x0f use:
iotest -s 1024 -p 01/f04 file1gb
-i | --input file
Specify input file to read.
-o | --output file
Specify output file instead of providing as argument.
-z | --snooze seconds
Seconds to delay after write close.
-c | --comment comment
Specify comment to add to stdout.
-c | --curt
Specify curt stdout.
-v | --verbose
Specify verbose stdout.
-V | --version
Show iotest version and exit.
-h | -? | --help
Display usage information and exit.
EXAMPLES
The initial examples below were run on 'kcrh6l' which is a RHEL 6.7 dedicated sys-
tem with 4gb of memory utilizing an ext3 partition.
The final examples below were run on 'h15n*' which are RHEL 6.7 dedicated 16 core
compute nodes with 64gb of memory sharing a 2.4PB GPFS filesystem with 1236 nodes.
Write a 1gb file with default (ascii) pattern (ext3):
kcrh6l# iotest -s 1024 x
kcrh6l write 1024.0 mb in 1 kb blocks 0x45 at 103.79 mb/s in 9.9 seconds:
20160622.225020 x
kcrh6l# uals -y+K x
- 0644 kcarlson staff 1.0G 160622.2250 x
Rewrite file 5 times with 0x00, 0xff, random, 0x0f and complement 0xf0:
kcrh6l# rm x; iotest -Cp 013/0f4 -s 1024 x
kcrh6l write 1024 mb 1 kb 0x00 109.11 mb/s 9.4 secs
kcrh6l write 1024 mb 1 kb 0xff 38.66 mb/s 26.5 secs
kcrh6l write 1024 mb 1 kb 0x30 37.60 mb/s 27.2 secs
kcrh6l write 1024 mb 1 kb 0x0f 37.23 mb/s 27.5 secs
kcrh6l write 1024 mb 1 kb 0xf0 43.23 mb/s 23.7 secs
Note the time variances especially with initial write. Inspite of the fsync() the
file is being buffered. The initial write should be longer due to growing the
file.
Rewrite file 5 times again using 'sudo sync' (-Y) then read:
kcrh6l# iotest -Cp 013/0f4 -s 1024 x -Yr
kcrh6l write 1024 mb 1 kb 0x00 34.04 mb/s 30.1 secs
kcrh6l write 1024 mb 1 kb 0xff 38.83 mb/s 26.4 secs
kcrh6l write 1024 mb 1 kb 0x30 38.78 mb/s 26.4 secs
kcrh6l write 1024 mb 1 kb 0x0f 38.91 mb/s 26.3 secs
kcrh6l write 1024 mb 1 kb 0xf0 38.62 mb/s 26.5 secs
kcrh6l read 1024 mb 1 kb 0xf0 1059.63 mb/s 1.0 secs
Note more consistent times, but the read came from memory.
Rewrite file 5 times again using 'sudo drop_caches 3' (-D) then read:
kcrh6l# iotest -Cp 013/0f4 -s 1024 x -Dr
MemFree: 1.845 2.881
MemFree: 1.853 2.897
kcrh6l write 1024 mb 1 kb 0x00 28.39 mb/s 36.1 secs
MemFree: 1.859 2.894
kcrh6l write 1024 mb 1 kb 0xff 30.14 mb/s 34.0 secs
MemFree: 1.855 2.895
kcrh6l write 1024 mb 1 kb 0x30 30.08 mb/s 34.0 secs
MemFree: 1.854 2.892
kcrh6l write 1024 mb 1 kb 0x0f 30.41 mb/s 33.7 secs
MemFree: 1.854 2.895
kcrh6l write 1024 mb 1 kb 0xf0 30.54 mb/s 33.5 secs
MemFree: 1.889 2.895
kcrh6l read 1024 mb 1 kb 0xf0 44.50 mb/s 23.0 secs
Note the read time is more realistic since cache was dropped. The write time
increase is likely due to the time required to drop cache.
Write and read a 4gb file (larger than memory):
kcrh6l# iotest -C -s 4096 x -rw
kcrh6l write 4096 mb 1 kb 0x45 42.96 mb/s 95.4 secs
kcrh6l read 4096 mb 1 kb 0x45 51.18 mb/s 80.0 secs
Write and read a 4gb file with 'sudo drop_caches 3':
kcrh6l# iotest -C -s 4096 x -rw -D
MemFree: 0.389 3.338
MemFree: 0.144 3.337
kcrh6l write 4096 mb 1 kb 0x45 32.11 mb/s 127.5 secs
MemFree: 0.147 3.339
kcrh6l read 4096 mb 1 kb 0x45 52.52 mb/s 78.0 secs
Note memory increased in above example because a browser was closed.
Write and read a 4gb file with 'sudo sync':
kcrh6l# iotest -C -s 4096 x -rw -Y
kcrh6l write 4096 mb 1 kb 0x45 34.38 mb/s 119.1 secs
kcrh6l read 4096 mb 1 kb 0x45 84.08 mb/s 48.7 secs
Note read speed increase. Possibly due to not dropping cache and starting with no
cache to force out.
Two pass write with verify (all):
kcrh6l: iotest -C -p01 -s 4096 x -wa
kcrh6l write 4096 mb 1 kb 0x00 46.26 mb/s 88.5 secs
kcrh6l ver. 4096 mb 1 kb 0x00 56.14 mb/s 73.0 secs
kcrh6l write 4096 mb 1 kb 0xff 46.69 mb/s 87.7 secs
kcrh6l ver. 4096 mb 1 kb 0xff 55.40 mb/s 73.9 secs
Re-verify of file written above (must specify correct pattern):
kcrh6l: ./iotest -C -p1 -s 4096 x -er
kcrh6l ver. 4096 mb 1 kb 0xff 69.89 mb/s 58.6 secs
Write and read a 64gb file (GPFS):
h15n60# iotest /scr/kcarlson/64g -b 1024 -s 65536 -wrC
h15n60 write 65536 mb 1024 kb 0x45 2953.45 mb/s 22.2 secs
h15n60 read 65536 mb 1024 kb 0x45 2227.72 mb/s 29.4 secs
Note, GPFS has a pre-defined cache and does not consume available memory.
Build an mpi enabled iotest (mpi_io) and perform parallel writes and reads of eight
64gb files:
h15n60# mpicc -DUSE_MPI iotest.c -o mpi_io
h15n60# mpirun ./mpi_io /scr/kcarlson/64g -b 1024 -s 65536 -wrC
h15n54 write 65536 mb 1024 kb 0x45 2945.09 mb/s 22.3 secs
..
h15n58 write 65536 mb 1024 kb 0x45 2541.24 mb/s 25.8 secs
h15n54 read 65536 mb 1024 kb 0x45 1794.99 mb/s 36.5 secs
..
h15n59 read 65536 mb 1024 kb 0x45 1706.58 mb/s 38.4 secs
h15n60# uals -Ty+K /scr/kcarlson/64g.?
Total: 8 files 512.0G size, for: .
The output above was reduced from 8 to 2 (fastest and slowest). Note the parallel
write and read of 512gb was only slightly slower than 64gb serial write due to the
nature of the storage sub-system.
NOTES
Compile with -DUSE_MPI to enable mpi. Files written/read via mpi have .[rank]
appended to name.
Writing files larger than memory can have performance ramifications on shared sys-
tems as it fills available cache which can cause significant delays for competing
reads and writes in other applications. This can be especially troublesome for
some NFS implementations.
On large memory systems, default tuning may be to allow all/most memory to be used
for IO caching which can have performance ramifications if the workload does not
benefit from caching. Significant delays can occur with cache continuously being
forced out. If the filesystem permits reducing cache size that will improve per-
formance.
The current recommendations for media sanitization are covered in NIST Special Pub-
lication 800-88 Revision 1 (NIST SP 800-88r1) which states, "A single write pass
should suffice to Purge the media" for all modern disk media. The old military
7-pass "standard" is not applicable. However, 800-88 also references optional use
of three total write passes. Using verification for more than one write pass is
certainly overkill.
ACKNOWLEDGEMENTS
Written at the University of Alaska Arctic Region Supercomputing Center (ARSC).
Ongoing maintenance via SourceForge by Denali Sun Consulting.
Suggestions or bug reports can be directed to denalisun907@gmail.com.
RELATED INFORMATION
Commands: drop_caches(1), sync(1), fsync(2), usemem(1), uakpacct(1), dd(1).