SD Card vs. eMMC Benchmarks

sunil · July 23, 2019, 3:00am

I ran benchmarks on SD card and eMMC using Flexible I/O tester. The following results show that eMMC is an order of magnitude better than SD card for some use cases like application performance (involving random write).

SD Card - Random Write: 1690KiB/s

root@freedombox:/mnt# fio --name=randwrite --ioengine=libaio --iodepth=1 --rw=randwrite --bs=4k --direct=0 --size=512M --runtime=240
randwrite: (g=0): rw=randwrite, bs=® 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=56KiB/s][w=14 IOPS][eta 00m:00s]
randwrite: (groupid=0, jobs=1): err= 0: pid=1226: Tue Jul 23 02:41:46 2019
write: IOPS=422, BW=1690KiB/s (1731kB/s)(396MiB/240039msec); 0 zone resets
slat (usec): min=33, max=17945k, avg=2352.64, stdev=104531.54
clat (nsec): min=0, max=2334.5k, avg=5240.29, stdev=9819.29
lat (usec): min=38, max=17945k, avg=2359.23, stdev=104531.94
clat percentiles (usec):
| 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5],
| 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 5],
| 70.00th=[ 5], 80.00th=[ 5], 90.00th=[ 6], 95.00th=[ 13],
| 99.00th=[ 22], 99.50th=[ 28], 99.90th=[ 55], 99.95th=[ 80],
| 99.99th=[ 330]
bw ( KiB/s): min= 16, max=56008, per=100.00%, avg=3205.98, stdev=8813.08, samples=253
iops : min= 4, max=14002, avg=801.46, stdev=2203.28, samples=253
lat (nsec) : 2=0.01%
lat (usec) : 4=18.41%, 10=74.11%, 20=6.41%, 50=0.93%, 100=0.10%
lat (usec) : 250=0.03%, 500=0.01%, 750=0.01%
lat (msec) : 4=0.01%
cpu : usr=0.55%, sys=2.90%, ctx=42248, majf=0, minf=43
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,101421,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
WRITE: bw=1690KiB/s (1731kB/s), 1690KiB/s-1690KiB/s (1731kB/s-1731kB/s), io=396MiB (415MB), run=240039-240039msec

SD Card - Sequential Write: 11.0MiB/s

root@freedombox:/mnt# fio --name=write --ioengine=libaio --iodepth=1 --rw=write --bs=4k --direct=0 --size=512M --runtime=240 
write: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [W(1)][97.7%][w=19.9MiB/s][w=5093 IOPS][eta 00m:01s]
write: (groupid=0, jobs=1): err= 0: pid=1301: Tue Jul 23 02:45:15 2019
  write: IOPS=3064, BW=11.0MiB/s (12.6MB/s)(512MiB/42771msec); 0 zone resets
    slat (usec): min=29, max=430596, avg=315.49, stdev=2342.69
    clat (usec): min=3, max=315, avg= 3.65, stdev= 1.17
     lat (usec): min=33, max=430617, avg=320.10, stdev=2343.06
    clat percentiles (nsec):
     |  1.00th=[ 3376],  5.00th=[ 3408], 10.00th=[ 3472], 20.00th=[ 3504],
     | 30.00th=[ 3536], 40.00th=[ 3536], 50.00th=[ 3568], 60.00th=[ 3600],
     | 70.00th=[ 3632], 80.00th=[ 3664], 90.00th=[ 3760], 95.00th=[ 3888],
     | 99.00th=[ 4832], 99.50th=[ 5792], 99.90th=[13504], 99.95th=[19328],
     | 99.99th=[42240]
   bw (  KiB/s): min= 1024, max=40872, per=99.52%, avg=12198.66, stdev=4651.74, samples=85
   iops        : min=  256, max=10218, avg=3049.62, stdev=1162.94, samples=85
  lat (usec)   : 4=96.03%, 10=3.82%, 20=0.11%, 50=0.04%, 100=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=1.44%, sys=12.07%, ctx=14615, majf=0, minf=49
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,131072,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=11.0MiB/s (12.6MB/s), 11.0MiB/s-11.0MiB/s (12.6MB/s-12.6MB/s), io=512MiB (537MB), run=42771-42771msec

eMMC - Random Write: 23.1MiB/s

root@freedombox:/media/root/526376b5-9685-44b4-a3ad-fb0100da48d8# fio --name=randwrite --ioengine=libaio --iodepth=1 --rw=randwrite --bs=4k --direct=0 --size=512M --runtime=240
randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=29.0MiB/s][w=7427 IOPS][eta 00m:00s]
randwrite: (groupid=0, jobs=1): err= 0: pid=1262: Tue Jul 23 02:54:59 2019
  write: IOPS=5901, BW=23.1MiB/s (24.2MB/s)(512MiB/22211msec); 0 zone resets
    slat (usec): min=31, max=692392, avg=155.01, stdev=5013.11
    clat (usec): min=3, max=1785, avg= 4.82, stdev=13.31
     lat (usec): min=36, max=692417, avg=161.07, stdev=5013.49
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    4], 10.00th=[    4], 20.00th=[    4],
     | 30.00th=[    5], 40.00th=[    5], 50.00th=[    5], 60.00th=[    5],
     | 70.00th=[    5], 80.00th=[    5], 90.00th=[    5], 95.00th=[    6],
     | 99.00th=[   14], 99.50th=[   20], 99.90th=[  130], 99.95th=[  202],
     | 99.99th=[  619]
   bw (  KiB/s): min=   72, max=56904, per=100.00%, avg=23970.91, stdev=10487.72, samples=43
   iops        : min=   18, max=14226, avg=5992.67, stdev=2621.91, samples=43
  lat (usec)   : 4=24.05%, 10=74.37%, 20=1.14%, 50=0.22%, 100=0.11%
  lat (usec)   : 250=0.09%, 500=0.02%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=7.39%, sys=40.44%, ctx=39919, majf=0, minf=30
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,131072,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=23.1MiB/s (24.2MB/s), 23.1MiB/s-23.1MiB/s (24.2MB/s-24.2MB/s), io=512MiB (537MB), run=22211-22211msec

eMMC - Sequential Write: 74.5MiB/s

root@freedombox:/mnt/emmc# fio --name=write --ioengine=libaio --iodepth=1 --rw=write --bs=4k --direct=0 --size=512M --runtime=240 
write: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=73.1MiB/s][w=18.7k IOPS][eta 00m:00s]
write: (groupid=0, jobs=1): err= 0: pid=1347: Tue Jul 23 02:47:44 2019
  write: IOPS=19.1k, BW=74.5MiB/s (78.1MB/s)(512MiB/6877msec); 0 zone resets
    slat (usec): min=29, max=77610, avg=42.43, stdev=391.69
    clat (usec): min=3, max=850, avg= 3.75, stdev= 6.01
     lat (usec): min=33, max=77629, avg=47.10, stdev=391.96
    clat percentiles (nsec):
     |  1.00th=[ 3408],  5.00th=[ 3472], 10.00th=[ 3504], 20.00th=[ 3536],
     | 30.00th=[ 3568], 40.00th=[ 3600], 50.00th=[ 3632], 60.00th=[ 3664],
     | 70.00th=[ 3664], 80.00th=[ 3696], 90.00th=[ 3792], 95.00th=[ 3888],
     | 99.00th=[ 4448], 99.50th=[ 4768], 99.90th=[25216], 99.95th=[51456],
     | 99.99th=[69120]
   bw (  KiB/s): min=72512, max=96928, per=100.00%, avg=76298.85, stdev=6571.88, samples=13
   iops        : min=18128, max=24232, avg=19074.69, stdev=1642.98, samples=13
  lat (usec)   : 4=96.64%, 10=3.21%, 20=0.04%, 50=0.05%, 100=0.05%
  lat (usec)   : 250=0.01%, 750=0.01%, 1000=0.01%
  cpu          : usr=18.19%, sys=63.37%, ctx=270, majf=0, minf=24
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,131072,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=74.5MiB/s (78.1MB/s), 74.5MiB/s-74.5MiB/s (78.1MB/s-78.1MB/s), io=512MiB (537MB), run=6877-6877msec

These tests were done on an A64 Olinuxino and btrfs filesystem. The SD card a Samsung 32GB EVO Plus card. eMMC is 16GB shipped with A64-OLinuXino-1Ge16GW.

sunil · July 25, 2020, 6:56pm

Here is an update with a 320GB HDD (TOSHIBA MQ01ABD0) attached to A20-OLinuXino-Lime2.

HDD - Random Write: 3237KiB/s

admin@freedombox:~$ fio --name=randwrite --ioengine=libaio --iodepth=1 --rw=randwrite --bs=4k --direct=0 --size=512M --runtime=240
randwrite: (g=0): rw=randwrite, bs=® 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.21
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=10.7MiB/s][w=2740 IOPS][eta 00m:00s]
randwrite: (groupid=0, jobs=1): err= 0: pid=1213: Sat Jul 25 02:04:13 2020
write: IOPS=809, BW=3237KiB/s (3314kB/s)(512MiB/161985msec); 0 zone resets
slat (usec): min=53, max=568483, avg=1153.27, stdev=8460.15
clat (usec): min=5, max=40214, avg=25.70, stdev=329.48
lat (usec): min=62, max=568538, avg=1188.78, stdev=8476.09
clat percentiles (usec):
| 1.00th=[ 8], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 10],
| 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 13], 60.00th=[ 15],
| 70.00th=[ 17], 80.00th=[ 21], 90.00th=[ 29], 95.00th=[ 37],
| 99.00th=[ 83], 99.50th=[ 176], 99.90th=[ 2040], 99.95th=[ 4490],
| 99.99th=[14353]
bw ( KiB/s): min= 8, max=19984, per=100.00%, avg=3239.61, stdev=3046.63, samples=323
iops : min= 2, max= 4996, avg=809.68, stdev=761.67, samples=323
lat (usec) : 10=25.33%, 20=52.99%, 50=19.18%, 100=1.72%, 250=0.37%
lat (usec) : 500=0.13%, 750=0.06%, 1000=0.04%
lat (msec) : 2=0.08%, 4=0.05%, 10=0.04%, 20=0.01%, 50=0.01%
cpu : usr=4.12%, sys=16.67%, ctx=33375, majf=1, minf=34
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,131072,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
WRITE: bw=3237KiB/s (3314kB/s), 3237KiB/s-3237KiB/s (3314kB/s-3314kB/s), io=512MiB (537MB), run=161985-161985msec

HDD - Sequential Write: 33.4MiB/s

admin@freedombox:~$ fio --name=write --ioengine=libaio --iodepth=1 --rw=write --bs=4k --direct=0 --size=512M --runtime=240
write: (g=0): rw=write, bs=® 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.21
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=36.4MiB/s][w=9326 IOPS][eta 00m:00s]
write: (groupid=0, jobs=1): err= 0: pid=1221: Sat Jul 25 02:05:53 2020
write: IOPS=8550, BW=33.4MiB/s (35.0MB/s)(512MiB/15330msec); 0 zone resets
slat (usec): min=46, max=118899, avg=88.26, stdev=771.85
clat (usec): min=5, max=65971, avg= 9.77, stdev=259.47
lat (usec): min=54, max=118944, avg=101.12, stdev=824.24
clat percentiles (usec):
| 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 7], 20.00th=[ 7],
| 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 8], 60.00th=[ 8],
| 70.00th=[ 9], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 12],
| 99.00th=[ 20], 99.50th=[ 30], 99.90th=[ 149], 99.95th=[ 289],
| 99.99th=[ 857]
bw ( KiB/s): min=25639, max=41269, per=99.87%, avg=34156.13, stdev=4447.53, samples=30
iops : min= 6409, max=10317, avg=8538.77, stdev=1111.87, samples=30
lat (usec) : 10=86.64%, 20=12.39%, 50=0.66%, 100=0.05%, 250=0.19%
lat (usec) : 500=0.05%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 10=0.01%, 100=0.01%
cpu : usr=17.56%, sys=60.18%, ctx=1615, majf=1, minf=32
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,131072,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
WRITE: bw=33.4MiB/s (35.0MB/s), 33.4MiB/s-33.4MiB/s (35.0MB/s-35.0MB/s), io=512MiB (537MB), run=15330-15330msec

HDD - Random Read: 381KiB/s

admin@freedombox:~$ fio --name=read --ioengine=libaio --iodepth=1 --rw=randread --bs=4k --direct=0 --size=512M --runtime=240

read: (g=0): rw=randread, bs=® 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.21
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=228KiB/s][r=57 IOPS][eta 00m:00s]
read: (groupid=0, jobs=1): err= 0: pid=1302: Sat Jul 25 02:17:03 2020
read: IOPS=95, BW=381KiB/s (390kB/s)(89.3MiB/240011msec)
slat (usec): min=959, max=108969, avg=10269.11, stdev=4198.59
clat (usec): min=16, max=6721, avg=64.84, stdev=120.96
lat (usec): min=1006, max=109098, avg=10369.46, stdev=4201.37
clat percentiles (usec):
| 1.00th=[ 26], 5.00th=[ 35], 10.00th=[ 37], 20.00th=[ 39],
| 30.00th=[ 41], 40.00th=[ 44], 50.00th=[ 63], 60.00th=[ 66],
| 70.00th=[ 68], 80.00th=[ 71], 90.00th=[ 74], 95.00th=[ 77],
| 99.00th=[ 293], 99.50th=[ 594], 99.90th=[ 1516], 99.95th=[ 2638],
| 99.99th=[ 5669]
bw ( KiB/s): min= 135, max= 471, per=100.00%, avg=381.14, stdev=36.95, samples=478
iops : min= 33, max= 117, avg=94.71, stdev= 9.23, samples=478
lat (usec) : 20=0.07%, 50=42.88%, 100=54.87%, 250=0.70%, 500=0.84%
lat (usec) : 750=0.32%, 1000=0.06%
lat (msec) : 2=0.17%, 4=0.06%, 10=0.02%
cpu : usr=1.97%, sys=9.19%, ctx=25694, majf=0, minf=33
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=22851,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
READ: bw=381KiB/s (390kB/s), 381KiB/s-381KiB/s (390kB/s-390kB/s), io=89.3MiB (93.6MB), run=240011-240011msec

HDD - Sequential Read: 20.8MiB/s

admin@freedombox:~$ fio --name=read --ioengine=libaio --iodepth=1 --rw=read --bs=4k --direct=0 --size=512M --runtime=240
read: (g=0): rw=read, bs=® 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.21
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=23.0MiB/s][r=6140 IOPS][eta 00m:00s]
read: (groupid=0, jobs=1): err= 0: pid=1297: Sat Jul 25 02:12:21 2020
read: IOPS=5329, BW=20.8MiB/s (21.8MB/s)(512MiB/24594msec)
slat (usec): min=12, max=477916, avg=166.13, stdev=3299.99
clat (usec): min=3, max=8573, avg= 6.39, stdev=45.86
lat (usec): min=17, max=477963, avg=175.04, stdev=3304.32
clat percentiles (usec):
| 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 5],
| 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6],
| 70.00th=[ 6], 80.00th=[ 6], 90.00th=[ 7], 95.00th=[ 8],
| 99.00th=[ 25], 99.50th=[ 36], 99.90th=[ 178], 99.95th=[ 310],
| 99.99th=[ 1975]
bw ( KiB/s): min= 8510, max=37976, per=98.70%, avg=21039.19, stdev=6647.59, samples=48
iops : min= 2127, max= 9494, avg=5259.56, stdev=1661.99, samples=48
lat (usec) : 4=3.08%, 10=94.02%, 20=1.70%, 50=0.80%, 100=0.20%
lat (usec) : 250=0.12%, 500=0.04%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=8.68%, sys=26.94%, ctx=2171, majf=0, minf=323
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=131072,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
READ: bw=20.8MiB/s (21.8MB/s), 20.8MiB/s-20.8MiB/s (21.8MB/s-21.8MB/s), io=512MiB (537MB), run=24594-24594msec