Sstrhsrtj

I want to benchmark a ssd (possibly with encrypted filesystems) and compare it to benchmarks done by crystaldiskmark on windows.

So how can I measure approximately the same things as crystaldiskmark does?

For the first row (Seq) I think I could do something like

LC_ALL=C dd if=/dev/zero of=tempfile bs=1M count=1024 conv=fdatasync,notrunc



sudo su -c "echo 3 > /proc/sys/vm/drop_caches"

LC_ALL=C dd if=tempfile of=/dev/null bs=1M count=1024

But I am not sure about the dd parameters.

For the random 512KB, 4KB, 4KB (Queue Depth=32) reads/writes speed-tests I don't have any idea how to reproduce the measurements in linux? So how can I do this?

For testing reading speeds something like sudo hdparm -Tt /dev/sda doesn't seem to make sense for me since I want for example benchmark something like encfs mounts.

Edit

@Alko, @iain

Perhaps I should write something about the motivation about this question: I am trying to benchmark my ssd and compare some encryption solutions. But that's another question (Best way to benchmark different encryption solutions on my system). While surfing in the web about ssd's and benchmarking I have often seen users posting their CrystelDiskMark results in forums. So this is the only motivation for the question. I just want to do the same on linux. For my particular benchmarking see my other question.

I'd say fio would have no trouble producing those workloads. Note that despite its name CrystalDiskMark is actually a benchmark of a filesysystem on a particular disk - it can't do I/O raw to the disk alone. As such it will always have filesystem overhead in it (not necessarily a bad thing but something to be aware of e.g. because the filesystems being compared might not be the same).

An example based on replicating the output in the screenshot above supplemented by information from the CrystalDiskMark manual (this isn't complete but should give the general idea):

fio --loops=5 --size=1000m --filename=/mnt/fs/fiotest.tmp --stonewall --ioengine=libaio --direct=1 

  --name=Seqread --bs=1m --rw=read 

  --name=Seqwrite --bs=1m --rw=write 

  --name=512Kread --bs=512k --rw=randread 

  --name=512Kwrite --bs=512k --rw=randwrite 

  --name=4kQD32read --bs=4k --iodepth=32 --rw=randread 

  --name=4kQD32write --bs=4k --iodepth=32 --rw=randwrite

rm -f /mnt/fs/fiotest.tmp 

BE CAREFUL - this example permanently destroys the data in /mnt/fs/fiotest.tmp!

A list of fio parameters can be seen on http://fio.readthedocs.io/en/latest/fio_doc.html .

You can use iozone and bonnie. They can do what crystal disk mark can do and more.

I personally used iozone a lot while benchmarking and stress testing devices from personal computers to enterprise storage systems. It has an auto mode which does everything but you can tailor it to your needs.

I created a script that tries to replicate the behavior of crystaldiskmark 5 with fio. (Revised: Now does all tests available in the various versions of crystaldiskmark up to crystaldiskmark 6 including 512K and 4KQ8T8 tests + more improvements)

The script depends on fio and df. If you do not want to install df, erase line 19 through 21 (the script will no longer display which drive is being tested) or check the comments for a modified version.

#!/bin/bash



LOOPS=5 #How many times to run each test

SIZE=1024 #Size of each test, multiples of 32 recommended for Q32 tests to give the most accurate results.

WRITEZERO=0 #Set whether to write zeroes or randoms to testfile (random is the default for both fio and crystaldiskmark); dd benchmarks typically only write zeroes which is why there can be a speed difference.



QSIZE=$(($SIZE / 32)) #Size of Q32Seq tests

SIZE+=m

QSIZE+=m



if [ -z $1 ]; then

    TARGET=$HOME

    echo "Defaulting to $TARGET for testing"

else

    TARGET="$1"

    echo "Testing in $TARGET"

fi



DRIVE=$(df $TARGET | grep /dev | cut -d/ -f3 | cut -d" " -f1 | rev | cut -c 2- | rev)

DRIVEMODEL=$(cat /sys/block/$DRIVE/device/model)

DRIVESIZE=$(($(cat /sys/block/$DRIVE/size)*512/1024/1024/1024))GB



echo "Configuration: Size:$SIZE Loops:$LOOPS Write Only Zeroes:$WRITEZERO

Running Benchmark on: /dev/$DRIVE, $DRIVEMODEL ($DRIVESIZE), please wait...

"



fio --loops=$LOOPS --size=$SIZE --filename=$TARGET/.fiomark.tmp --stonewall --ioengine=libaio --direct=1 --zero_buffers=$WRITEZERO --output-format=json 

  --name=Bufread --loops=1 --bs=$SIZE --iodepth=1 --numjobs=1 --rw=readwrite 

  --name=Seqread --bs=$SIZE --iodepth=1 --numjobs=1 --rw=read 

  --name=Seqwrite --bs=$SIZE --iodepth=1 --numjobs=1 --rw=write 

  --name=512kread --bs=512k --iodepth=1 --numjobs=1 --rw=read 

  --name=512kwrite --bs=512k --iodepth=1 --numjobs=1 --rw=write 

  --name=SeqQ32T1read --bs=$QSIZE --iodepth=32 --numjobs=1 --rw=read 

  --name=SeqQ32T1write --bs=$QSIZE --iodepth=32 --numjobs=1 --rw=write 

  --name=4kread --bs=4k --iodepth=1 --numjobs=1 --rw=randread 

  --name=4kwrite --bs=4k --iodepth=1 --numjobs=1 --rw=randwrite 

  --name=4kQ32T1read --bs=4k --iodepth=32 --numjobs=1 --rw=randread 

  --name=4kQ32T1write --bs=4k --iodepth=32 --numjobs=1 --rw=randwrite 

  --name=4kQ8T8read --bs=4k --iodepth=8 --numjobs=8 --rw=randread 

  --name=4kQ8T8write --bs=4k --iodepth=8 --numjobs=8 --rw=randwrite > $TARGET/.fiomark.txt



SEQR="$(($(cat $TARGET/.fiomark.txt | grep -A15 '"name" : "Seqread"' | grep bw_bytes | cut -d: -f2 | sed s:,::g)/1024/1024))MB/s IOPS=$(cat $TARGET/.fiomark.txt | grep -A15 '"name" : "Seqread"' | grep -m1 iops | cut -d: -f2 | cut -d. -f1 | sed 's: ::g')"

SEQW="$(($(cat $TARGET/.fiomark.txt | grep -A80 '"name" : "Seqwrite"' | grep bw_bytes | sed '2!d' | cut -d: -f2 | sed s:,::g)/1024/1024))MB/s IOPS=$(cat $TARGET/.fiomark.txt | grep -A80 '"name" : "Seqwrite"' | grep iops | sed '7!d' | cut -d: -f2 | cut -d. -f1 | sed 's: ::g')"

F12KR="$(($(cat $TARGET/.fiomark.txt | grep -A15 '"name" : "512kread"' | grep bw_bytes | cut -d: -f2 | sed s:,::g)/1024/1024))MB/s IOPS=$(cat $TARGET/.fiomark.txt | grep -A15 '"name" : "512kread"' | grep -m1 iops | cut -d: -f2 | cut -d. -f1 | sed 's: ::g')"

F12KW="$(($(cat $TARGET/.fiomark.txt | grep -A80 '"name" : "512kwrite"' | grep bw_bytes | sed '2!d' | cut -d: -f2 | sed s:,::g)/1024/1024))MB/s IOPS=$(cat $TARGET/.fiomark.txt | grep -A80 '"name" : "512kwrite"' | grep iops | sed '7!d' | cut -d: -f2 | cut -d. -f1 | sed 's: ::g')"

SEQ32R="$(($(cat $TARGET/.fiomark.txt | grep -A15 '"name" : "SeqQ32T1read"' | grep bw_bytes | cut -d: -f2 | sed s:,::g)/1024/1024))MB/s IOPS=$(cat $TARGET/.fiomark.txt | grep -A15 '"name" : "SeqQ32T1read"' | grep -m1 iops | cut -d: -f2 | cut -d. -f1 | sed 's: ::g')"

SEQ32W="$(($(cat $TARGET/.fiomark.txt | grep -A80 '"name" : "SeqQ32T1write"' | grep bw_bytes | sed '2!d' | cut -d: -f2 | sed s:,::g)/1024/1024))MB/s IOPS=$(cat $TARGET/.fiomark.txt | grep -A80 '"name" : "SeqQ32T1write"' | grep iops | sed '7!d' | cut -d: -f2 | cut -d. -f1 | sed 's: ::g')"

FKR="$(($(cat $TARGET/.fiomark.txt | grep -A15 '"name" : "4kread"' | grep bw_bytes | cut -d: -f2 | sed s:,::g)/1024/1024))MB/s IOPS=$(cat $TARGET/.fiomark.txt | grep -A15 '"name" : "4kread"' | grep -m1 iops | cut -d: -f2 | cut -d. -f1 | sed 's: ::g')"

FKW="$(($(cat $TARGET/.fiomark.txt | grep -A80 '"name" : "4kwrite"' | grep bw_bytes | sed '2!d' | cut -d: -f2 | sed s:,::g)/1024/1024))MB/s IOPS=$(cat $TARGET/.fiomark.txt | grep -A80 '"name" : "4kwrite"' | grep iops | sed '7!d' | cut -d: -f2 | cut -d. -f1 | sed 's: ::g')"

FK32R="$(($(cat $TARGET/.fiomark.txt | grep -A15 '"name" : "4kQ32T1read"' | grep bw_bytes | cut -d: -f2 | sed s:,::g)/1024/1024))MB/s IOPS=$(cat $TARGET/.fiomark.txt | grep -A15 '"name" : "4kQ32T1read"' | grep -m1 iops | cut -d: -f2 | cut -d. -f1 | sed 's: ::g')"

FK32W="$(($(cat $TARGET/.fiomark.txt | grep -A80 '"name" : "4kQ32T1write"' | grep bw_bytes | sed '2!d' | cut -d: -f2 | sed s:,::g)/1024/1024))MB/s IOPS=$(cat $TARGET/.fiomark.txt | grep -A80 '"name" : "4kQ32T1write"' | grep iops | sed '7!d' | cut -d: -f2 | cut -d. -f1 | sed 's: ::g')"

FK8R="$(($(cat $TARGET/.fiomark.txt | grep -A15 '"name" : "4kQ8T8read"' | grep bw_bytes | sed 's/        "bw_bytes" : //g' | sed 's:,::g' | awk '{ SUM += $1} END { print SUM }')/1024/1024))MB/s IOPS=$(cat $TARGET/.fiomark.txt | grep -A15 '"name" : "4kQ8T8read"' | grep iops | sed 's/        "iops" : //g' | sed 's:,::g' | awk '{ SUM += $1} END { print SUM }' | cut -d. -f1)"

FK8W="$(($(cat $TARGET/.fiomark.txt | grep -A80 '"name" : "4kQ8T8write"' | grep bw_bytes | sed 's/        "bw_bytes" : //g' | sed 's:,::g' | awk '{ SUM += $1} END { print SUM }')/1024/1024))MB/s IOPS=$(cat $TARGET/.fiomark.txt | grep -A80 '"name" : "4kQ8T8write"' | grep '"iops" '| sed 's/        "iops" : //g' | sed 's:,::g' | awk '{ SUM += $1} END { print SUM }' | cut -d. -f1)"



echo -e "

Results from /dev/$DRIVE, $DRIVEMODEL ($DRIVESIZE):  

33[0;33m

Sequential Read: $SEQR

Sequential Write: $SEQW

33[0;32m

512KB Read: $F12KR

512KB Write: $F12KW

33[1;36m

Sequential Q32T1 Read: $SEQ32R

Sequential Q32T1 Write: $SEQ32W

33[0;36m

4KB Read: $FKR

4KB Write: $FKW

33[1;33m

4KB Q32T1 Read: $FK32R

4KB Q32T1 Write: $FK32W

33[1;35m

4KB Q8T8 Read: $FK8R

4KB Q8T8 Write: $FK8W

"



rm $TARGET/.fiomark.txt $TARGET/.fiomark.tmp

Which will output results like this:

Results from /dev/sdb, Corsair Force GT (111GB):  



Sequential Read: 533MB/s IOPS=0

Sequential Write: 125MB/s IOPS=0



512KB Read: 457MB/s IOPS=914

512KB Write: 133MB/s IOPS=267



Sequential Q32T1 Read: 534MB/s IOPS=16

Sequential Q32T1 Write: 134MB/s IOPS=4



4KB Read: 32MB/s IOPS=8224

4KB Write: 150MB/s IOPS=38460



4KB Q32T1 Read: 195MB/s IOPS=49951

4KB Q32T1 Write: 121MB/s IOPS=31148



4KB Q8T8 Read: 129MB/s IOPS=33149

4KB Q8T8 Write: 132MB/s IOPS=33796

(Revision: The results are now color coded, to remove the color coding remove all instances of 33[x;xxm (where x is a number) from the echo command at the bottom of the script.)

The script when run without arguments will test the speed of your home drive/partition. You can also enter a path to a directory on another hard drive if you would like to test that instead. While running the script creates hidden temporary files in the target directory which it cleans up after it finishes running (.fiomark.tmp and .fiomark.txt)

You cannot see the test results as they complete, but if you cancel the command while it is running before it finishes all the tests, you will get to see the results of completed tests and the temporary files will get deleted afterwards as well.

After some research, I found that the crystaldiskmark benchmark results on the same model of drive as I have seem to relatively closely match the results of this fio benchmark, at least at a glance. As I do not have a windows installation I can not verify how close they really are for certain on the same drive.

Note that you may sometimes get slightly off results, especially if you're doing something in the background while the tests are running, so running the test twice in a row to compare results is advisable.

These tests take a loong time to run. The default settings in the script currently are suitable for a regular (SATA) SSD.

Recommended SIZE setting for different drives:

• (SATA) SSD: 1024(default)


• (ANY) HDD: 256


• (High End NVME) SSD: 4096


• (Low-Mid End NVME) SSD: 1024(default)

A High End NVME typically has around ~2GB/s read speeds (Intel Optane and Samsung 960 EVO are examples; but in the latter's case I would recommend 2048 instead due to slower 4kb speeds.), a Low-Mid End can have anywhere between ~500-1800MB/s read speeds.

The major reason why these sizes should be adjusted is because of how long the tests would take otherwise, for older/weaker HDDs for example, you can have as low as 0.4MB/s 4kb Read speeds. You try waiting for 5 loops of 1GB at that speed, other 4kb tests typically have around 1MB/s speeds. We have 6 of them. Each running 5 loops, do you wait for 30GB of data to be transferred at those speeds? Or do you want to lower that to 7.5GB of Data instead (at 256MB/s it's a 2-3 hour test)

Of course, the ideal method for handling that situation would be to run sequential & 512k tests separate from the 4k tests ( so run the sequential and 512k tests with something like say 512m, and then run the 4k tests at 32m)

More recent HDD models are higher end though and can get much better results than that.

And there you have it. Enjoy!

I'm not sure the various deeper tests make any real sense when considering what you're doing in detail.

The settings like block size, and queue depth, are parameters for controlling the low level input/output parameters of the ATA interface your SSD is sitting on.

Thats all well and good when you're just running some basic test against a drive fairly directly, like to a large file in a simple partitioned filesystem.

Once you start talking about benchmarking an encfs, these parameters dont particularly apply to your filesystem any more, the filesystem is just an interface into something else that eventually backs onto a filesystem that backs onto a drive.

I think it would be helpful to understand what exactly you're trying to measure, because there are two factors in play here - the raw disk IO speed, which you can test by timing various DD commands (can give examples if this is what you want) /without/ encfs, or the process will be CPU bounded by the encryption and you're trying to test the relative throughput of the encryption algorithm. In which case the parameters for queue depth etc aren't particularly relevant.

In both regards, a timed DD command will give you the basic throughput statistics you seek, but you should consider what you're intending to measure and the relevant parameters for that.

This link seems to provide a good guide to disk speed testing using timed DD commands including the necessary coverage about 'defeating buffers/cache' and so forth. Likely this will provide the information you need. Decide which you're more interested in tho, disk performance or encryption performance, one of the two will be the bottleneck, and tuning the non-bottleneck isn't going to benefit anything.