## i4i.xlarge with attached 872 GB NVMe
### 1 min
```
random_writes: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.36
Starting 1 process
Jobs: 1 (f=1), 0-10 IOPS: [w(1)][100.0%][w=40KiB/s][w=10 IOPS][eta 00m:00s]
random_writes: (groupid=0, jobs=1): err= 0: pid=2607: Mon Apr 14 00:46:16 2025
write: IOPS=9, BW=40.0KiB/s (41.0kB/s)(2400KiB/60001msec); 0 zone resets
slat (nsec): min=2138, max=21001, avg=3897.31, stdev=1565.52
clat (nsec): min=25557, max=54363, avg=33711.65, stdev=3022.84
lat (nsec): min=28477, max=65020, avg=37608.96, stdev=3811.98
clat percentiles (nsec):
| 1.00th=[27264], 5.00th=[29312], 10.00th=[30080], 20.00th=[31360],
| 30.00th=[32384], 40.00th=[33024], 50.00th=[34048], 60.00th=[34560],
| 70.00th=[35072], 80.00th=[35584], 90.00th=[36608], 95.00th=[37632],
| 99.00th=[43776], 99.50th=[49408], 99.90th=[54528], 99.95th=[54528],
| 99.99th=[54528]
bw ( KiB/s): min= 40, max= 48, per=100.00%, avg=40.07, stdev= 0.73, samples=119
iops : min= 10, max= 12, avg=10.02, stdev= 0.18, samples=119
lat (usec) : 50=99.67%, 100=0.33%
cpu : usr=0.03%, sys=0.00%, ctx=1200, majf=0, minf=9
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,600,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=40.0KiB/s (41.0kB/s), 40.0KiB/s-40.0KiB/s (41.0kB/s-41.0kB/s), io=2400KiB (2458kB), run=60001-60001msec
Disk stats (read/write):
nvme1n1: ios=51/599, sectors=2126/4792, merge=0/0, ticks=3/18, in_queue=20, util=0.00%
```
### 10 min
```
random_writes: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.36
Starting 1 process
Jobs: 1 (f=1), 0-10 IOPS: [w(1)][100.0%][w=40KiB/s][w=10 IOPS][eta 00m:00s]
random_writes: (groupid=0, jobs=1): err= 0: pid=2646: Mon Apr 14 00:59:03 2025
write: IOPS=9, BW=40.0KiB/s (41.0kB/s)(23.4MiB/600001msec); 0 zone resets
slat (nsec): min=2167, max=39077, avg=4922.96, stdev=3880.19
clat (usec): min=5, max=4447, avg=37.79, stdev=115.86
lat (usec): min=27, max=4452, avg=42.71, stdev=115.90
clat percentiles (usec):
| 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 30], 20.00th=[ 31],
| 30.00th=[ 32], 40.00th=[ 33], 50.00th=[ 34], 60.00th=[ 35],
| 70.00th=[ 35], 80.00th=[ 37], 90.00th=[ 40], 95.00th=[ 44],
| 99.00th=[ 54], 99.50th=[ 59], 99.90th=[ 2802], 99.95th=[ 3720],
| 99.99th=[ 4424]
bw ( KiB/s): min= 40, max= 48, per=100.00%, avg=40.01, stdev= 0.33, samples=1199
iops : min= 10, max= 12, avg=10.00, stdev= 0.08, samples=1199
lat (usec) : 10=0.07%, 20=0.17%, 50=97.68%, 100=1.95%, 250=0.02%
lat (msec) : 4=0.10%, 10=0.02%
cpu : usr=0.01%, sys=0.02%, ctx=11992, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,6000,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=40.0KiB/s (41.0kB/s), 40.0KiB/s-40.0KiB/s (41.0kB/s-41.0kB/s), io=23.4MiB (24.6MB), run=600001-600001msec
Disk stats (read/write):
nvme1n1: ios=52/5999, sectors=2127/47992, merge=0/0, ticks=4/195, in_queue=199, util=0.13%
```
### 1 hour
```
random_writes: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.36
Starting 1 process
Jobs: 1 (f=1), 0-10 IOPS: [w(1)][100.0%][w=40KiB/s][w=10 IOPS][eta 00m:00s]
random_writes: (groupid=0, jobs=1): err= 0: pid=2682: Mon Apr 14 02:01:50 2025
write: IOPS=9, BW=40.0KiB/s (41.0kB/s)(141MiB/3600001msec); 0 zone resets
slat (nsec): min=1877, max=45961, avg=4043.56, stdev=1900.42
clat (usec): min=8, max=4989, avg=39.40, stdev=137.61
lat (usec): min=26, max=4991, avg=43.44, stdev=137.62
clat percentiles (usec):
| 1.00th=[ 28], 5.00th=[ 29], 10.00th=[ 30], 20.00th=[ 31],
| 30.00th=[ 31], 40.00th=[ 32], 50.00th=[ 33], 60.00th=[ 34],
| 70.00th=[ 34], 80.00th=[ 35], 90.00th=[ 36], 95.00th=[ 38],
| 99.00th=[ 51], 99.50th=[ 93], 99.90th=[ 2999], 99.95th=[ 3556],
| 99.99th=[ 4424]
bw ( KiB/s): min= 40, max= 48, per=100.00%, avg=40.00, stdev= 0.13, samples=7199
iops : min= 10, max= 12, avg=10.00, stdev= 0.03, samples=7199
lat (usec) : 10=0.01%, 20=0.01%, 50=98.91%, 100=0.58%, 250=0.06%
lat (usec) : 500=0.18%, 750=0.03%, 1000=0.01%
lat (msec) : 2=0.07%, 4=0.11%, 10=0.04%
cpu : usr=0.01%, sys=0.02%, ctx=71988, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,36000,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=40.0KiB/s (41.0kB/s), 40.0KiB/s-40.0KiB/s (41.0kB/s-41.0kB/s), io=141MiB (147MB), run=3600001-3600001msec
Disk stats (read/write):
nvme1n1: ios=51/35999, sectors=2126/287992, merge=0/0, ticks=3/1264, in_queue=1268, util=0.05%
```
## i7ie.xlarge with attached 2.3TB NVMe
### 1 min
```
random_writes: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.36
Starting 1 process
Jobs: 1 (f=1), 0-10 IOPS: [w(1)][100.0%][w=40KiB/s][w=10 IOPS][eta 00m:00s]
random_writes: (groupid=0, jobs=1): err= 0: pid=2326: Mon Apr 14 00:46:17 2025
write: IOPS=9, BW=40.0KiB/s (41.0kB/s)(2400KiB/60001msec); 0 zone resets
slat (nsec): min=9321, max=29202, avg=14392.04, stdev=4281.88
clat (usec): min=25, max=296, avg=37.51, stdev=27.12
lat (usec): min=37, max=307, avg=51.90, stdev=27.45
clat percentiles (usec):
| 1.00th=[ 28], 5.00th=[ 29], 10.00th=[ 30], 20.00th=[ 31],
| 30.00th=[ 31], 40.00th=[ 32], 50.00th=[ 33], 60.00th=[ 34],
| 70.00th=[ 36], 80.00th=[ 42], 90.00th=[ 45], 95.00th=[ 47],
| 99.00th=[ 229], 99.50th=[ 273], 99.90th=[ 297], 99.95th=[ 297],
| 99.99th=[ 297]
bw ( KiB/s): min= 40, max= 48, per=100.00%, avg=40.07, stdev= 0.73, samples=119
iops : min= 10, max= 12, avg=10.02, stdev= 0.18, samples=119
lat (usec) : 50=98.17%, 100=0.50%, 250=0.33%, 500=1.00%
cpu : usr=0.05%, sys=0.00%, ctx=1200, majf=0, minf=10
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,600,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=40.0KiB/s (41.0kB/s), 40.0KiB/s-40.0KiB/s (41.0kB/s-41.0kB/s), io=2400KiB (2458kB), run=60001-60001msec
Disk stats (read/write):
nvme1n1: ios=50/599, sectors=2116/4792, merge=0/0, ticks=2/18, in_queue=21, util=0.01%
```
### 10 min
```
random_writes: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.36
Starting 1 process
Jobs: 1 (f=1), 0-10 IOPS: [w(1)][100.0%][w=40KiB/s][w=10 IOPS][eta 00m:00s]
random_writes: (groupid=0, jobs=1): err= 0: pid=2408: Mon Apr 14 00:59:13 2025
write: IOPS=9, BW=40.0KiB/s (41.0kB/s)(23.4MiB/600001msec); 0 zone resets
slat (nsec): min=8856, max=47484, avg=14488.97, stdev=5030.84
clat (usec): min=3, max=444, avg=48.72, stdev=49.02
lat (usec): min=33, max=455, avg=63.21, stdev=48.98
clat percentiles (usec):
| 1.00th=[ 28], 5.00th=[ 30], 10.00th=[ 31], 20.00th=[ 32],
| 30.00th=[ 34], 40.00th=[ 36], 50.00th=[ 41], 60.00th=[ 43],
| 70.00th=[ 44], 80.00th=[ 45], 90.00th=[ 49], 95.00th=[ 61],
| 99.00th=[ 297], 99.50th=[ 314], 99.90th=[ 388], 99.95th=[ 408],
| 99.99th=[ 445]
bw ( KiB/s): min= 40, max= 48, per=100.00%, avg=40.01, stdev= 0.33, samples=1199
iops : min= 10, max= 12, avg=10.00, stdev= 0.08, samples=1199
lat (usec) : 4=0.03%, 10=0.02%, 20=0.03%, 50=90.78%, 100=4.82%
lat (usec) : 250=1.22%, 500=3.10%
cpu : usr=0.02%, sys=0.03%, ctx=11938, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,6000,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=40.0KiB/s (41.0kB/s), 40.0KiB/s-40.0KiB/s (41.0kB/s-41.0kB/s), io=23.4MiB (24.6MB), run=600001-600001msec
Disk stats (read/write):
nvme1n1: ios=50/5999, sectors=2116/47992, merge=0/0, ticks=2/236, in_queue=238, util=0.11%
```
### 1 hour
```
random_writes: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.36
Starting 1 process
Jobs: 1 (f=1), 0-10 IOPS: [w(1)][100.0%][w=40KiB/s][w=10 IOPS][eta 00m:00s]
random_writes: (groupid=0, jobs=1): err= 0: pid=2446: Mon Apr 14 02:01:55 2025
write: IOPS=9, BW=40.0KiB/s (41.0kB/s)(141MiB/3600001msec); 0 zone resets
slat (nsec): min=9254, max=61759, avg=21009.81, stdev=4651.05
clat (usec): min=3, max=437, avg=39.54, stdev=25.97
lat (usec): min=36, max=449, avg=60.55, stdev=26.19
clat percentiles (usec):
| 1.00th=[ 29], 5.00th=[ 31], 10.00th=[ 32], 20.00th=[ 33],
| 30.00th=[ 34], 40.00th=[ 35], 50.00th=[ 36], 60.00th=[ 37],
| 70.00th=[ 38], 80.00th=[ 42], 90.00th=[ 47], 95.00th=[ 49],
| 99.00th=[ 215], 99.50th=[ 285], 99.90th=[ 338], 99.95th=[ 363],
| 99.99th=[ 400]
bw ( KiB/s): min= 40, max= 48, per=100.00%, avg=40.00, stdev= 0.13, samples=7199
iops : min= 10, max= 12, avg=10.00, stdev= 0.03, samples=7199
lat (usec) : 4=0.01%, 10=0.03%, 20=0.01%, 50=96.48%, 100=2.41%
lat (usec) : 250=0.29%, 500=0.79%
cpu : usr=0.02%, sys=0.04%, ctx=71867, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,36000,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=40.0KiB/s (41.0kB/s), 40.0KiB/s-40.0KiB/s (41.0kB/s-41.0kB/s), io=141MiB (147MB), run=3600001-3600001msec
Disk stats (read/write):
nvme1n1: ios=50/35999, sectors=2116/287992, merge=0/0, ticks=2/1183, in_queue=1185, util=0.02%
```
## r7i with 550GB gp3 volume with 3k provisioned IOPS
### 1 min
```
random_writes: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.36
Starting 1 process
Jobs: 1 (f=1), 0-10 IOPS: [w(1)][100.0%][w=40KiB/s][w=10 IOPS][eta 00m:00s]
random_writes: (groupid=0, jobs=1): err= 0: pid=2411: Mon Apr 14 00:46:19 2025
write: IOPS=9, BW=40.0KiB/s (41.0kB/s)(2400KiB/60002msec); 0 zone resets
slat (nsec): min=10904, max=54728, avg=19953.58, stdev=7933.02
clat (usec): min=530, max=1390, avg=817.19, stdev=102.06
lat (usec): min=548, max=1417, avg=837.14, stdev=101.43
clat percentiles (usec):
| 1.00th=[ 619], 5.00th=[ 685], 10.00th=[ 709], 20.00th=[ 742],
| 30.00th=[ 758], 40.00th=[ 783], 50.00th=[ 799], 60.00th=[ 824],
| 70.00th=[ 848], 80.00th=[ 881], 90.00th=[ 947], 95.00th=[ 1012],
| 99.00th=[ 1139], 99.50th=[ 1205], 99.90th=[ 1385], 99.95th=[ 1385],
| 99.99th=[ 1385]
bw ( KiB/s): min= 40, max= 48, per=100.00%, avg=40.07, stdev= 0.73, samples=119
iops : min= 10, max= 12, avg=10.02, stdev= 0.18, samples=119
lat (usec) : 750=24.17%, 1000=69.83%
lat (msec) : 2=6.00%
cpu : usr=0.03%, sys=0.04%, ctx=1209, majf=0, minf=11
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,600,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=40.0KiB/s (41.0kB/s), 40.0KiB/s-40.0KiB/s (41.0kB/s-41.0kB/s), io=2400KiB (2458kB), run=60002-60002msec
Disk stats (read/write):
nvme0n1: ios=2/620, sectors=16/5128, merge=0/13, ticks=2/494, in_queue=496, util=0.32%
```
### 10 min
```
random_writes: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.36
Starting 1 process
Jobs: 1 (f=1), 0-10 IOPS: [w(1)][100.0%][w=40KiB/s][w=10 IOPS][eta 00m:00s]
random_writes: (groupid=0, jobs=1): err= 0: pid=2434: Mon Apr 14 00:59:21 2025
write: IOPS=9, BW=40.0KiB/s (41.0kB/s)(23.4MiB/600001msec); 0 zone resets
slat (nsec): min=10041, max=60115, avg=21054.20, stdev=8450.26
clat (usec): min=587, max=3730, avg=948.88, stdev=135.88
lat (usec): min=598, max=3762, avg=969.93, stdev=136.74
clat percentiles (usec):
| 1.00th=[ 676], 5.00th=[ 725], 10.00th=[ 758], 20.00th=[ 824],
| 30.00th=[ 914], 40.00th=[ 955], 50.00th=[ 979], 60.00th=[ 996],
| 70.00th=[ 1012], 80.00th=[ 1037], 90.00th=[ 1074], 95.00th=[ 1106],
| 99.00th=[ 1188], 99.50th=[ 1270], 99.90th=[ 2278], 99.95th=[ 2573],
| 99.99th=[ 3720]
bw ( KiB/s): min= 40, max= 48, per=100.00%, avg=40.01, stdev= 0.33, samples=1199
iops : min= 10, max= 12, avg=10.00, stdev= 0.08, samples=1199
lat (usec) : 750=9.47%, 1000=53.83%
lat (msec) : 2=36.58%, 4=0.12%
cpu : usr=0.02%, sys=0.04%, ctx=12007, majf=0, minf=10
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,6000,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=40.0KiB/s (41.0kB/s), 40.0KiB/s-40.0KiB/s (41.0kB/s-41.0kB/s), io=23.4MiB (24.6MB), run=600001-600001msec
Disk stats (read/write):
nvme0n1: ios=19/6117, sectors=784/49368, merge=79/35, ticks=15/5682, in_queue=5697, util=0.88%
```
### 1 hour
```
random_writes: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.36
Starting 1 process
Jobs: 1 (f=1), 0-10 IOPS: [w(1)][100.0%][w=40KiB/s][w=10 IOPS][eta 00m:00s]
random_writes: (groupid=0, jobs=1): err= 0: pid=2466: Mon Apr 14 02:02:00 2025
write: IOPS=9, BW=40.0KiB/s (41.0kB/s)(141MiB/3600002msec); 0 zone resets
slat (usec): min=11, max=135, avg=28.78, stdev= 7.44
clat (usec): min=623, max=5302, avg=1012.66, stdev=122.67
lat (usec): min=654, max=5334, avg=1041.44, stdev=123.46
clat percentiles (usec):
| 1.00th=[ 840], 5.00th=[ 906], 10.00th=[ 930], 20.00th=[ 955],
| 30.00th=[ 979], 40.00th=[ 988], 50.00th=[ 1004], 60.00th=[ 1020],
| 70.00th=[ 1037], 80.00th=[ 1057], 90.00th=[ 1090], 95.00th=[ 1123],
| 99.00th=[ 1205], 99.50th=[ 1270], 99.90th=[ 2769], 99.95th=[ 3556],
| 99.99th=[ 5145]
bw ( KiB/s): min= 32, max= 48, per=100.00%, avg=40.00, stdev= 0.19, samples=7199
iops : min= 8, max= 12, avg=10.00, stdev= 0.05, samples=7199
lat (usec) : 750=0.12%, 1000=46.00%
lat (msec) : 2=53.70%, 4=0.14%, 10=0.04%
cpu : usr=0.03%, sys=0.05%, ctx=72010, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,36000,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=40.0KiB/s (41.0kB/s), 40.0KiB/s-40.0KiB/s (41.0kB/s-41.0kB/s), io=141MiB (147MB), run=3600002-3600002msec
Disk stats (read/write):
nvme0n1: ios=50/36851, sectors=1728/297712, merge=166/266, ticks=40/36603, in_queue=36643, util=1.02%
```
## r7i with 550GB io2 volume with 16k provisioned IOPS
### 1 min
```
random_writes: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.36
Starting 1 process
Jobs: 1 (f=1), 0-10 IOPS: [w(1)][100.0%][w=40KiB/s][w=10 IOPS][eta 00m:00s]
random_writes: (groupid=0, jobs=1): err= 0: pid=2289: Mon Apr 14 00:46:20 2025
write: IOPS=9, BW=40.0KiB/s (41.0kB/s)(2400KiB/60001msec); 0 zone resets
slat (nsec): min=19110, max=57375, avg=30083.48, stdev=3185.95
clat (usec): min=269, max=1745, avg=387.91, stdev=98.22
lat (usec): min=298, max=1774, avg=417.99, stdev=98.35
clat percentiles (usec):
| 1.00th=[ 277], 5.00th=[ 289], 10.00th=[ 293], 20.00th=[ 306],
| 30.00th=[ 314], 40.00th=[ 330], 50.00th=[ 400], 60.00th=[ 429],
| 70.00th=[ 445], 80.00th=[ 457], 90.00th=[ 486], 95.00th=[ 506],
| 99.00th=[ 545], 99.50th=[ 562], 99.90th=[ 1745], 99.95th=[ 1745],
| 99.99th=[ 1745]
bw ( KiB/s): min= 40, max= 48, per=100.00%, avg=40.07, stdev= 0.73, samples=119
iops : min= 10, max= 12, avg=10.02, stdev= 0.18, samples=119
lat (usec) : 500=94.00%, 750=5.67%, 1000=0.17%
lat (msec) : 2=0.17%
cpu : usr=0.05%, sys=0.04%, ctx=1205, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,600,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=40.0KiB/s (41.0kB/s), 40.0KiB/s-40.0KiB/s (41.0kB/s-41.0kB/s), io=2400KiB (2458kB), run=60001-60001msec
Disk stats (read/write):
nvme0n1: ios=0/619, sectors=0/5096, merge=0/11, ticks=0/231, in_queue=231, util=0.05%
```
### 10 min
```
random_writes: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.36
Starting 1 process
Jobs: 1 (f=1), 0-10 IOPS: [w(1)][100.0%][w=40KiB/s][w=10 IOPS][eta 00m:00s]
random_writes: (groupid=0, jobs=1): err= 0: pid=2308: Mon Apr 14 00:59:27 2025
write: IOPS=9, BW=40.0KiB/s (41.0kB/s)(23.4MiB/600001msec); 0 zone resets
slat (nsec): min=15916, max=46502, avg=28349.64, stdev=3138.24
clat (usec): min=260, max=1352, avg=401.39, stdev=82.69
lat (usec): min=283, max=1383, avg=429.74, stdev=82.83
clat percentiles (usec):
| 1.00th=[ 277], 5.00th=[ 289], 10.00th=[ 297], 20.00th=[ 310],
| 30.00th=[ 326], 40.00th=[ 400], 50.00th=[ 424], 60.00th=[ 441],
| 70.00th=[ 457], 80.00th=[ 469], 90.00th=[ 490], 95.00th=[ 510],
| 99.00th=[ 553], 99.50th=[ 586], 99.90th=[ 979], 99.95th=[ 1303],
| 99.99th=[ 1352]
bw ( KiB/s): min= 40, max= 48, per=100.00%, avg=40.01, stdev= 0.33, samples=1199
iops : min= 10, max= 12, avg=10.00, stdev= 0.08, samples=1199
lat (usec) : 500=92.65%, 750=7.20%, 1000=0.08%
lat (msec) : 2=0.07%
cpu : usr=0.03%, sys=0.05%, ctx=12011, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,6000,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=40.0KiB/s (41.0kB/s), 40.0KiB/s-40.0KiB/s (41.0kB/s-41.0kB/s), io=23.4MiB (24.6MB), run=600001-600001msec
Disk stats (read/write):
nvme0n1: ios=23/6209, sectors=928/50400, merge=84/53, ticks=12/2398, in_queue=2409, util=0.25%
```
### 1 hour
```
random_writes: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.36
Starting 1 process
Jobs: 1 (f=1), 0-10 IOPS: [w(1)][100.0%][w=40KiB/s][w=10 IOPS][eta 00m:00s]
random_writes: (groupid=0, jobs=1): err= 0: pid=2343: Mon Apr 14 02:02:05 2025
write: IOPS=9, BW=40.0KiB/s (41.0kB/s)(141MiB/3600001msec); 0 zone resets
slat (nsec): min=15331, max=82212, avg=31628.22, stdev=4355.95
clat (usec): min=255, max=4879, avg=404.18, stdev=78.34
lat (usec): min=283, max=4908, avg=435.81, stdev=78.20
clat percentiles (usec):
| 1.00th=[ 277], 5.00th=[ 289], 10.00th=[ 297], 20.00th=[ 314],
| 30.00th=[ 338], 40.00th=[ 408], 50.00th=[ 424], 60.00th=[ 441],
| 70.00th=[ 453], 80.00th=[ 465], 90.00th=[ 486], 95.00th=[ 502],
| 99.00th=[ 545], 99.50th=[ 562], 99.90th=[ 668], 99.95th=[ 938],
| 99.99th=[ 1270]
bw ( KiB/s): min= 40, max= 48, per=100.00%, avg=40.00, stdev= 0.13, samples=7199
iops : min= 10, max= 12, avg=10.00, stdev= 0.03, samples=7199
lat (usec) : 500=94.25%, 750=5.67%, 1000=0.04%
lat (msec) : 2=0.04%, 10=0.01%
cpu : usr=0.03%, sys=0.06%, ctx=72015, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,36000,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=40.0KiB/s (41.0kB/s), 40.0KiB/s-40.0KiB/s (41.0kB/s-41.0kB/s), io=141MiB (147MB), run=3600001-3600001msec
Disk stats (read/write):
nvme0n1: ios=52/36863, sectors=1776/298040, merge=170/259, ticks=28/14355, in_queue=14382, util=0.05%
```
## r7i with 550GB io1 volume with 3k provisioned IOPS
### 1 min
```
random_writes: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.36
Starting 1 process
Jobs: 1 (f=1), 0-10 IOPS: [w(1)][100.0%][w=40KiB/s][w=10 IOPS][eta 00m:00s]
random_writes: (groupid=0, jobs=1): err= 0: pid=2061: Mon Apr 14 00:46:21 2025
write: IOPS=9, BW=40.0KiB/s (41.0kB/s)(2400KiB/60001msec); 0 zone resets
slat (nsec): min=20267, max=56100, avg=30584.11, stdev=4023.59
clat (usec): min=664, max=1393, avg=845.64, stdev=63.94
lat (usec): min=696, max=1423, avg=876.22, stdev=64.06
clat percentiles (usec):
| 1.00th=[ 693], 5.00th=[ 750], 10.00th=[ 775], 20.00th=[ 799],
| 30.00th=[ 816], 40.00th=[ 832], 50.00th=[ 840], 60.00th=[ 857],
| 70.00th=[ 873], 80.00th=[ 889], 90.00th=[ 914], 95.00th=[ 947],
| 99.00th=[ 988], 99.50th=[ 1057], 99.90th=[ 1401], 99.95th=[ 1401],
| 99.99th=[ 1401]
bw ( KiB/s): min= 40, max= 48, per=100.00%, avg=40.07, stdev= 0.73, samples=119
iops : min= 10, max= 12, avg=10.02, stdev= 0.18, samples=119
lat (usec) : 750=5.17%, 1000=94.00%
lat (msec) : 2=0.83%
cpu : usr=0.01%, sys=0.06%, ctx=1207, majf=0, minf=10
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,600,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=40.0KiB/s (41.0kB/s), 40.0KiB/s-40.0KiB/s (41.0kB/s-41.0kB/s), io=2400KiB (2458kB), run=60001-60001msec
Disk stats (read/write):
nvme0n1: ios=2/661, sectors=16/5520, merge=0/16, ticks=1/544, in_queue=545, util=0.97%
```
### 10 min
```
random_writes: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.36
Starting 1 process
Jobs: 1 (f=1), 0-10 IOPS: [w(1)][100.0%][w=40KiB/s][w=10 IOPS][eta 00m:00s]
random_writes: (groupid=0, jobs=1): err= 0: pid=2078: Mon Apr 14 00:59:37 2025
write: IOPS=9, BW=40.0KiB/s (41.0kB/s)(23.4MiB/600001msec); 0 zone resets
slat (nsec): min=18469, max=54701, avg=29488.92, stdev=2027.83
clat (usec): min=632, max=1314, avg=854.42, stdev=58.78
lat (usec): min=662, max=1342, avg=883.91, stdev=58.83
clat percentiles (usec):
| 1.00th=[ 709], 5.00th=[ 766], 10.00th=[ 791], 20.00th=[ 807],
| 30.00th=[ 824], 40.00th=[ 840], 50.00th=[ 848], 60.00th=[ 865],
| 70.00th=[ 881], 80.00th=[ 898], 90.00th=[ 922], 95.00th=[ 947],
| 99.00th=[ 1020], 99.50th=[ 1045], 99.90th=[ 1139], 99.95th=[ 1270],
| 99.99th=[ 1319]
bw ( KiB/s): min= 40, max= 48, per=100.00%, avg=40.01, stdev= 0.33, samples=1199
iops : min= 10, max= 12, avg=10.00, stdev= 0.08, samples=1199
lat (usec) : 750=3.17%, 1000=95.38%
lat (msec) : 2=1.45%
cpu : usr=0.03%, sys=0.05%, ctx=12010, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,6000,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=40.0KiB/s (41.0kB/s), 40.0KiB/s-40.0KiB/s (41.0kB/s-41.0kB/s), io=23.4MiB (24.6MB), run=600001-600001msec
Disk stats (read/write):
nvme0n1: ios=10/6138, sectors=320/49664, merge=30/36, ticks=6/5111, in_queue=5117, util=1.00%
```
### 1 hour
```
random_writes: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.36
Starting 1 process
Jobs: 1 (f=1), 0-10 IOPS: [w(1)][100.0%][w=40KiB/s][w=10 IOPS][eta 00m:00s]
random_writes: (groupid=0, jobs=1): err= 0: pid=2113: Mon Apr 14 02:02:10 2025
write: IOPS=9, BW=40.0KiB/s (41.0kB/s)(141MiB/3600001msec); 0 zone resets
slat (nsec): min=13941, max=78361, avg=29921.34, stdev=4658.63
clat (usec): min=298, max=5589, avg=853.66, stdev=82.26
lat (usec): min=331, max=5620, avg=883.58, stdev=82.28
clat percentiles (usec):
| 1.00th=[ 676], 5.00th=[ 750], 10.00th=[ 775], 20.00th=[ 807],
| 30.00th=[ 824], 40.00th=[ 840], 50.00th=[ 857], 60.00th=[ 873],
| 70.00th=[ 881], 80.00th=[ 906], 90.00th=[ 930], 95.00th=[ 955],
| 99.00th=[ 1020], 99.50th=[ 1045], 99.90th=[ 1188], 99.95th=[ 1434],
| 99.99th=[ 3064]
bw ( KiB/s): min= 40, max= 48, per=100.00%, avg=40.00, stdev= 0.13, samples=7199
iops : min= 10, max= 12, avg=10.00, stdev= 0.03, samples=7199
lat (usec) : 500=0.28%, 750=5.24%, 1000=92.90%
lat (msec) : 2=1.56%, 4=0.02%, 10=0.01%
cpu : usr=0.03%, sys=0.05%, ctx=72012, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,36000,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=40.0KiB/s (41.0kB/s), 40.0KiB/s-40.0KiB/s (41.0kB/s-41.0kB/s), io=141MiB (147MB), run=3600001-3600001msec
Disk stats (read/write):
nvme0n1: ios=360/36941, sectors=25128/325016, merge=345/339, ticks=202/30915, in_queue=31117, util=0.89%
```
## r7i with 550GB gp2 (1650 / 3000 iops)
### 1 min
```
random_writes: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.36
Starting 1 process
Jobs: 1 (f=1), 0-10 IOPS: [w(1)][100.0%][w=40KiB/s][w=10 IOPS][eta 00m:00s]
random_writes: (groupid=0, jobs=1): err= 0: pid=2054: Mon Apr 14 00:46:24 2025
write: IOPS=9, BW=40.0KiB/s (41.0kB/s)(2400KiB/60002msec); 0 zone resets
slat (nsec): min=13138, max=61088, avg=31328.72, stdev=8076.93
clat (usec): min=831, max=1253, avg=1023.18, stdev=57.65
lat (usec): min=864, max=1285, avg=1054.51, stdev=58.16
clat percentiles (usec):
| 1.00th=[ 865], 5.00th=[ 914], 10.00th=[ 955], 20.00th=[ 979],
| 30.00th=[ 996], 40.00th=[ 1012], 50.00th=[ 1020], 60.00th=[ 1037],
| 70.00th=[ 1045], 80.00th=[ 1074], 90.00th=[ 1090], 95.00th=[ 1123],
| 99.00th=[ 1172], 99.50th=[ 1172], 99.90th=[ 1254], 99.95th=[ 1254],
| 99.99th=[ 1254]
bw ( KiB/s): min= 40, max= 48, per=100.00%, avg=40.07, stdev= 0.73, samples=119
iops : min= 10, max= 12, avg=10.02, stdev= 0.18, samples=119
lat (usec) : 1000=31.00%
lat (msec) : 2=69.00%
cpu : usr=0.01%, sys=0.07%, ctx=1208, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,600,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=40.0KiB/s (41.0kB/s), 40.0KiB/s-40.0KiB/s (41.0kB/s-41.0kB/s), io=2400KiB (2458kB), run=60002-60002msec
Disk stats (read/write):
nvme0n1: ios=2/618, sectors=16/5080, merge=0/11, ticks=1/618, in_queue=620, util=1.04%
```
### 10 min
```
random_writes: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.36
Starting 1 process
Jobs: 1 (f=1), 0-10 IOPS: [w(1)][100.0%][w=40KiB/s][w=10 IOPS][eta 00m:00s]
random_writes: (groupid=0, jobs=1): err= 0: pid=2073: Mon Apr 14 00:59:44 2025
write: IOPS=9, BW=40.0KiB/s (41.0kB/s)(23.4MiB/600002msec); 0 zone resets
slat (nsec): min=12245, max=67152, avg=29648.21, stdev=4614.40
clat (usec): min=776, max=1983, avg=1031.03, stdev=60.47
lat (usec): min=807, max=1999, avg=1060.68, stdev=60.72
clat percentiles (usec):
| 1.00th=[ 873], 5.00th=[ 930], 10.00th=[ 963], 20.00th=[ 988],
| 30.00th=[ 1004], 40.00th=[ 1020], 50.00th=[ 1029], 60.00th=[ 1045],
| 70.00th=[ 1057], 80.00th=[ 1074], 90.00th=[ 1106], 95.00th=[ 1123],
| 99.00th=[ 1172], 99.50th=[ 1205], 99.90th=[ 1303], 99.95th=[ 1614],
| 99.99th=[ 1991]
bw ( KiB/s): min= 39, max= 48, per=100.00%, avg=40.01, stdev= 0.33, samples=1199
iops : min= 9, max= 12, avg=10.00, stdev= 0.09, samples=1199
lat (usec) : 1000=26.53%
lat (msec) : 2=73.47%
cpu : usr=0.03%, sys=0.06%, ctx=12010, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,6000,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=40.0KiB/s (41.0kB/s), 40.0KiB/s-40.0KiB/s (41.0kB/s-41.0kB/s), io=23.4MiB (24.6MB), run=600002-600002msec
Disk stats (read/write):
nvme0n1: ios=15/6186, sectors=536/50008, merge=52/39, ticks=7/6243, in_queue=6250, util=1.00%
```
### 1 hour
```
random_writes: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.36
Starting 1 process
Jobs: 1 (f=1), 0-10 IOPS: [w(1)][100.0%][w=40KiB/s][w=10 IOPS][eta 00m:00s]
random_writes: (groupid=0, jobs=1): err= 0: pid=2119: Mon Apr 14 02:02:17 2025
write: IOPS=9, BW=40.0KiB/s (41.0kB/s)(141MiB/3600002msec); 0 zone resets
slat (usec): min=11, max=110, avg=29.23, stdev= 8.79
clat (usec): min=319, max=8829, avg=1022.24, stdev=77.36
lat (usec): min=343, max=8848, avg=1051.47, stdev=78.00
clat percentiles (usec):
| 1.00th=[ 881], 5.00th=[ 938], 10.00th=[ 955], 20.00th=[ 979],
| 30.00th=[ 996], 40.00th=[ 1004], 50.00th=[ 1020], 60.00th=[ 1037],
| 70.00th=[ 1045], 80.00th=[ 1057], 90.00th=[ 1090], 95.00th=[ 1106],
| 99.00th=[ 1172], 99.50th=[ 1205], 99.90th=[ 1369], 99.95th=[ 1713],
| 99.99th=[ 2966]
bw ( KiB/s): min= 40, max= 48, per=100.00%, avg=40.00, stdev= 0.13, samples=7199
iops : min= 10, max= 12, avg=10.00, stdev= 0.03, samples=7199
lat (usec) : 500=0.05%, 750=0.08%, 1000=34.08%
lat (msec) : 2=65.76%, 4=0.02%, 10=0.01%
cpu : usr=0.03%, sys=0.06%, ctx=72007, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,36000,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=40.0KiB/s (41.0kB/s), 40.0KiB/s-40.0KiB/s (41.0kB/s-41.0kB/s), io=141MiB (147MB), run=3600002-3600002msec
Disk stats (read/write):
nvme0n1: ios=363/36958, sectors=24528/324656, merge=343/337, ticks=194/37106, in_queue=37301, util=1.01%
```
## r6id.4xlarge (th 884GB NVME volume
### 1 min
```
random_writes: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.36
Starting 1 process
Jobs: 1 (f=1), 0-10 IOPS: [w(1)][100.0%][w=40KiB/s][w=10 IOPS][eta 00m:00s]
random_writes: (groupid=0, jobs=1): err= 0: pid=2172: Mon Apr 14 00:46:26 2025
write: IOPS=9, BW=40.0KiB/s (41.0kB/s)(2400KiB/60001msec); 0 zone resets
slat (nsec): min=1965, max=21401, avg=3824.50, stdev=1203.42
clat (usec): min=26, max=183, avg=30.67, stdev= 6.68
lat (usec): min=29, max=205, avg=34.49, stdev= 7.50
clat percentiles (usec):
| 1.00th=[ 27], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 29],
| 30.00th=[ 30], 40.00th=[ 30], 50.00th=[ 31], 60.00th=[ 31],
| 70.00th=[ 32], 80.00th=[ 32], 90.00th=[ 34], 95.00th=[ 35],
| 99.00th=[ 39], 99.50th=[ 40], 99.90th=[ 184], 99.95th=[ 184],
| 99.99th=[ 184]
bw ( KiB/s): min= 40, max= 48, per=100.00%, avg=40.07, stdev= 0.73, samples=119
iops : min= 10, max= 12, avg=10.02, stdev= 0.18, samples=119
lat (usec) : 50=99.83%, 250=0.17%
cpu : usr=0.03%, sys=0.00%, ctx=1200, majf=0, minf=10
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,600,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=40.0KiB/s (41.0kB/s), 40.0KiB/s-40.0KiB/s (41.0kB/s-41.0kB/s), io=2400KiB (2458kB), run=60001-60001msec
Disk stats (read/write):
nvme1n1: ios=49/599, sectors=2126/4792, merge=0/0, ticks=4/16, in_queue=19, util=0.01%
```
### 10 min
```
random_writes: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.36
Starting 1 process
Jobs: 1 (f=1), 0-10 IOPS: [w(1)][100.0%][w=40KiB/s][w=10 IOPS][eta 00m:00s]
random_writes: (groupid=0, jobs=1): err= 0: pid=2208: Mon Apr 14 00:59:51 2025
write: IOPS=9, BW=40.0KiB/s (41.0kB/s)(23.4MiB/600001msec); 0 zone resets
slat (nsec): min=1848, max=37486, avg=4923.86, stdev=4278.82
clat (nsec): min=933, max=55991, avg=31481.88, stdev=4504.11
lat (nsec): min=25890, max=70917, avg=36405.74, stdev=6041.51
clat percentiles (nsec):
| 1.00th=[25984], 5.00th=[27264], 10.00th=[27520], 20.00th=[28288],
| 30.00th=[29056], 40.00th=[30080], 50.00th=[30848], 60.00th=[31360],
| 70.00th=[32128], 80.00th=[33024], 90.00th=[36096], 95.00th=[41728],
| 99.00th=[47872], 99.50th=[48896], 99.90th=[52480], 99.95th=[55552],
| 99.99th=[56064]
bw ( KiB/s): min= 40, max= 48, per=100.00%, avg=40.01, stdev= 0.33, samples=1199
iops : min= 10, max= 12, avg=10.00, stdev= 0.08, samples=1199
lat (nsec) : 1000=0.02%
lat (usec) : 2=0.05%, 10=0.20%, 20=0.10%, 50=99.25%, 100=0.38%
cpu : usr=0.01%, sys=0.02%, ctx=11964, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,6000,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=40.0KiB/s (41.0kB/s), 40.0KiB/s-40.0KiB/s (41.0kB/s-41.0kB/s), io=23.4MiB (24.6MB), run=600001-600001msec
Disk stats (read/write):
nvme1n1: ios=49/5999, sectors=2126/47992, merge=0/0, ticks=3/161, in_queue=164, util=0.34%
```
### 1 hour
```
random_writes: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.36
Starting 1 process
Jobs: 1 (f=1), 0-10 IOPS: [w(1)][100.0%][w=40KiB/s][w=10 IOPS][eta 00m:00s]
random_writes: (groupid=0, jobs=1): err= 0: pid=2256: Mon Apr 14 02:02:22 2025
write: IOPS=9, BW=40.0KiB/s (41.0kB/s)(141MiB/3600001msec); 0 zone resets
slat (nsec): min=1839, max=39382, avg=4753.26, stdev=3221.11
clat (nsec): min=999, max=14820k, avg=33119.01, stdev=129691.91
lat (usec): min=25, max=14822, avg=37.87, stdev=129.74
clat percentiles (usec):
| 1.00th=[ 27], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 29],
| 30.00th=[ 29], 40.00th=[ 30], 50.00th=[ 31], 60.00th=[ 32],
| 70.00th=[ 32], 80.00th=[ 33], 90.00th=[ 36], 95.00th=[ 38],
| 99.00th=[ 48], 99.50th=[ 55], 99.90th=[ 392], 99.95th=[ 545],
| 99.99th=[ 7767]
bw ( KiB/s): min= 40, max= 48, per=100.00%, avg=40.00, stdev= 0.13, samples=7199
iops : min= 10, max= 12, avg=10.00, stdev= 0.03, samples=7199
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.02%, 10=0.04%, 20=0.07%, 50=99.21%, 100=0.37%
lat (usec) : 250=0.12%, 500=0.10%, 750=0.04%, 1000=0.01%
lat (msec) : 2=0.01%, 10=0.01%, 20=0.01%
cpu : usr=0.01%, sys=0.02%, ctx=71940, majf=0, minf=11
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,36000,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=40.0KiB/s (41.0kB/s), 40.0KiB/s-40.0KiB/s (41.0kB/s-41.0kB/s), io=141MiB (147MB), run=3600001-3600001msec
Disk stats (read/write):
nvme1n1: ios=49/35999, sectors=2126/287992, merge=0/0, ticks=4/1053, in_queue=1056, util=0.07%
```