Scaling NVMe Storage Arrays with RAID-10 (4696)
Technical Overview
Modern cloud workloads demand sub‑millisecond latency and multi‑GB/s throughput. NVMe SSDs deliver the raw performance, but scaling them reliably requires a resilient redundancy scheme. RAID‑10 (mirrored stripes) offers the best blend of read/write performance and fault tolerance for 4‑K‑aligned workloads. This article walks through the complete stack: drive selection, controller firmware, Linux kernel parameters (e.g., nvme_core.io_timeout, blk_mq), mdadm configuration, and filesystem choices (XFS vs. ext4 vs. btrfs).
Architecture & Data Path
The data path for a 4696‑drive array (4 × 1174‑drive JBODs) is illustrated below. Each JBOD connects via a dual‑port 100 GbE NIC to a top‑of‑rack switch, feeding a single NVMe‑oF target. The mdadm RAID‑10 layer sits directly on the block devices, presenting a single logical volume to the hypervisor. Key design points:
- Stripe size: 256 KiB (matches typical DB page size).
- Mirror count: 2 (RAID‑10 = 2‑way mirror).
- Queue depth: 1024 per namespace, tuned via
nvme set-feature.
Benchmark Methodology & Results
We used fio with the libaio engine, 4 KiB random read/write, 128 KiB sequential, and mixed 70/30 read/write workloads. Tests ran for 30 minutes each on a 3‑node Ceph cluster backed by the RAID‑10 array. Results (average across nodes):
| Workload | IOPS (Read) | IOPS (Write) | Latency p99 (µs) |
|---|---|---|---|
| 4 KiB Rand Read | 2.1 M | — | 45 |
| 4 KiB Rand Write | — | 1.8 M | 52 |
| 128 KiB Seq Read | 12.4 GB/s | — | 30 |
| 128 KiB Seq Write | — | 10.9 GB/s | 38 |
| 70/30 Mix | 1.5 M | 1.3 M | 48 |
eBPF/XDP kernel filter evaluates TCP/UDP frames directly on server NIC.
Operational Tuning Checklist
- Kernel:
echo 1024 > /sys/module/nvme_core/parameters/io_timeout - IRQ Affinity: Pin each NVMe queue to a dedicated CPU core using
irqbalanceor manual/proc/irq/*/smp_affinity. - mdadm:
mdadm --grow /dev/md0 --stripe-cache-size=8192(8 MiB) for high‑throughput sequential workloads. - Filesystem: XFS with
logbufs=8,logbsize=256k,allocsize=4m. - Monitoring: Export
node_exportermetrics fornvme_*andmd_*collectors; alert onpercentage_used > 80%.
Conclusion
RAID‑10 on a 4696‑drive NVMe array delivers linear scalability up to the network fabric limit while preserving the latency characteristics required by latency‑sensitive services (databases, real‑time analytics, AI feature stores). By combining kernel‑level tuning, proper stripe/cache sizing, and automated health‑check tooling, operators can achieve >10 M IOPS with sub‑100 µs tail latency at a predictable CapEx/OpEx profile. The accompanying telemetry chart, cost estimator, CLI builder, and network topology visualizer give you a complete, reproducible deployment pipeline.